neuralmagic/deepsparse

Sparsity-aware deep learning inference runtime for CPUs

Structural Verdict

How Data Flows Through the System

Input data flows through task-specific pipelines that wrap the core sparse inference engine, with results formatted according to protocol requirements

1. Input Parsing — Parse JSON requests and extract model inputs (text, images, etc.)
2. Pipeline Creation — Create task-specific pipeline using Pipeline.create() with model path and configuration (config: model, batch_size, task)
3. Engine Compilation — Compile ONNX model for sparse execution on CPU using compile_model() (config: num_cores, engine)
4. Inference Execution — Run inputs through EngineOperator and core sparse inference engine (config: temperature, top_p, max_tokens)
5. Response Formatting — Format outputs according to protocol (OpenAI, custom JSON, etc.) (config: stream, object)

System Behavior

How the system actually operates at runtime — where data accumulates, what loops, what waits, and what controls what.

Data Pools

Feedback Loops

• Benchmark Iteration (polling, balancing) — Trigger: User benchmark request. Action: Run inference multiple times with warmup periods. Exit: Target iteration count reached.
• Streaming Response (polling, balancing) — Trigger: Stream=true in request. Action: Generate and yield partial responses. Exit: Generation complete or max_tokens reached.

Delays & Async Processing

• Model Compilation (async-processing, ~Variable based on model size) — First inference request has higher latency due to compilation overhead
• Cold Start Penalty (async-processing, ~5-30 seconds) — AWS Lambda cold starts require pipeline initialization
• Batch Job Queue (queue-drain, ~Variable) — Batch inference jobs wait in AWS Batch queue before execution

Control Points

• Batch Size (runtime-toggle) — Controls: Inference throughput vs memory usage tradeoff. Default: 1 for realtime, None for batch
• Temperature (threshold) — Controls: Text generation randomness. Default: 0.7
• Engine Selection (feature-flag) — Controls: Choice between DeepSparse and ONNXRuntime engines. Default: deepsparse
• Model Path (env-var) — Controls: Which model variant to load (sparse vs dense). Default: zoo: URIs or local paths

Technology Stack

Key Components

• Pipeline.create (factory) — Creates task-specific inference pipelines (sentiment_analysis, question-answering, etc.) src/deepsparse/pipelines/
• EngineOperator (class) — Wraps the core inference engine with input/output handling and configuration src/deepsparse/operators/engine_operator.py
• Benchmarker (class) — Orchestrates model benchmarking across different engines and configurations examples/benchmark-ui/benchmark.py
• compile_model (function) — Compiles ONNX models for optimized execution on the DeepSparse engine deepsparse module
• LoggerConfig (config) — Configures system logging and telemetry collection for inference monitoring src/deepsparse/loggers/config.py
• EvaluationResult (type-def) — Structured representation of model evaluation metrics and dataset information src/deepsparse/evaluation/results.py
• lambda_handler (handler) — AWS Lambda entry point for serverless inference requests examples/aws-serverless/realtime/app/app.py
• random_uuid (utility) — Generates unique identifiers for OpenAI protocol compliance examples/openai-server/protocol.py
• singlestream_benchmark (function) — Runs single-stream latency benchmarks for model performance evaluation deepsparse.benchmark.stream_benchmark
• Manager (config) — Central configuration for model paths, engines, and UI parameters in benchmark interface examples/benchmark-ui/settings.py

Configuration

examples/openai-server/protocol.py (python-pydantic)

• object (str, unknown) — default: "error"
• message (str, unknown)
• type (str, unknown)
• param (Optional[str], unknown) — default: None
• code (Optional[str], unknown) — default: None

examples/openai-server/protocol.py (python-pydantic)

• id (str, unknown) — default: Field(default_factory=lambda: f"modelperm-{random_uuid()}")
• object (str, unknown) — default: "model_permission"
• created (int, unknown) — default: Field(default_factory=lambda: int(time.time()))
• allow_create_engine (bool, unknown) — default: False
• allow_sampling (bool, unknown) — default: True
• allow_logprobs (bool, unknown) — default: True
• allow_search_indices (bool, unknown) — default: False
• allow_view (bool, unknown) — default: True
• +4 more parameters

examples/openai-server/protocol.py (python-pydantic)

• id (str, unknown)
• object (str, unknown) — default: "model"
• created (int, unknown) — default: Field(default_factory=lambda: int(time.time()))
• owned_by (str, unknown) — default: "neuralmagic"
• root (Optional[str], unknown) — default: None
• parent (Optional[str], unknown) — default: None
• permission (List[ModelPermission], unknown) — default: Field(default_factory=list)

examples/openai-server/protocol.py (python-pydantic)

• object (str, unknown) — default: "list"
• data (List[ModelCard], unknown) — default: Field(default_factory=list)

Science Pipeline

1. Model Loading — Load ONNX model from zoo: URI or local path src/deepsparse/pipelines/
2. Engine Compilation — Compile model for sparse execution with compile_model() [Model-dependent → Model-dependent] deepsparse.compile_model
3. Input Preprocessing — Task-specific input transformation (tokenization, normalization) [Raw text/images → Model input tensors] src/deepsparse/pipelines/
4. Sparse Inference — Execute through EngineOperator with sparse optimizations [Model input tensors → Model output tensors] src/deepsparse/operators/engine_operator.py
5. Output Postprocessing — Task-specific output formatting and protocol compliance [Model output tensors → Structured response objects] examples/openai-server/protocol.py

Assumptions & Constraints

• [warning] Assumes engine_inputs is a List but doesn't validate tensor shapes match model expectations (shape)
• [warning] Generates random inputs with assumed dtypes but may not match model's expected input types (dtype)
• [critical] Assumes ONNX model format but doesn't validate input/output node compatibility (format)
• [warning] Allows shape overrides without validating they're compatible with model architecture (shape)

Related Ml Training Repositories

Frequently Asked Questions

Analyzed on March 31, 2026 by CodeSea. Written by Karolina Sarna.