vllm-project/semantic-router
A programmable Mixture-of-Models router for heterogeneous LLM inference
13 hidden assumptions · 8-stage pipeline · 7 components
Like any codebase, this repository makes assumptions it never checks — most are routine. The ones worth your attention are below, in plain language with what to do about each.
Routes LLM requests across heterogeneous models using semantic signals and policies
A request arrives at the router as an OpenAI-compatible HTTP POST. Before forwarding it, the router passes the message text through the Rust FFI boundary into the ParallelLoRAEngine, which runs three transformer classifiers simultaneously: intent (sequence classification → label + confidence), PII (token classification → span list), and security (sequence classification → threat score). These signals are returned to Go as a RoutingSignals struct. The Go policy engine evaluates the configured RuleNode expression (e.g. 'intent("support") AND NOT pii("SSN") AND NOT security("jailbreak")') against the signals to select a target model and provider. The provider catalog supplies the concrete endpoint URL and auth headers. The router rewrites the request's model field, forwards it to the selected backend, and streams the response tokens back to the client. Simultaneously, request metadata (signals, routing decision, latency) is recorded for the dashboard's tracing and evaluation subsystems.
Under the hood, the system uses 3 feedback loops, 4 data pools, 6 control points to manage its runtime behavior.
A 7-component repository. 5968 files analyzed. Data flows through 8 distinct pipeline stages.
Hidden Assumptions
Most of what this code assumes is routine. These 3 are the ones most likely to cause trouble here — in plain terms, with what to do about each. The rest are minor; they're under "Show everything".
The paper is explicit that the headline benefit numbers were calculated from assumptions, not observed from real traffic. The authors themselves say these figures still need to be confirmed against actual production data. If you enable the load-adaptive caching and expect the promised savings, the real outcome could be quite different, and the software will not tell you.
What to do: Treat the load-based savings numbers as unverified estimates and measure actual hit rates and model-traffic reduction on your own workload before relying on them.
“The traffic reduction figures (9–17%) are theoretical projections based on assumed linear relationships between threshold relaxation and hit rate improvements.”
Read it in the paper · Discussion ↗When a downstream model gets busy, the method keeps cached answers around longer to boost hit rates — but this means users can receive out-of-date answers, and for fast-changing data the staleness can climb very high. The paper accepts this trade only when slow responses would be worse than stale ones, which may not hold for your use case.
What to do: For any category where freshness matters (financial, news, real-time data), cap the maximum time-to-live and confirm that serving somewhat stale answers is genuinely acceptable under load.
“Extending to t 1 = 15 t_{1}=15 minutes ( β = 3 \beta=3 ): staleness increases to 60%.”
Read it in the paper · Discussion ↗The paper draws a size window where this design pays off: roughly tens of thousands up to about ten million cached entries. Below that, a simpler cache is better; well above it, the fast local-search timing that the economics depend on starts to slip. Outside this window the claimed advantages may not apply.
What to do: Check that your expected cache size falls within the range the paper validated and reconsider the approach if it is much smaller or far larger.
“Pure in-memory caches remain simpler for small datasets ( < < 10K entries).”
Read it in the paper · Discussion ↗Show everything (10 more)
The claim that low-hit-rate categories become worth caching rests on specific timing numbers the authors assumed for search speed and model response time. If your servers are faster, slower, or your models respond differently, the point at which caching pays off shifts, and a category the paper says is viable might not be in your setup.
What to do: Measure your actual search latency and model response times, then recompute whether your low-hit-rate categories still clear the break-even point before enabling caching for them.
“Total cost: approximately 30ms per query (hit or miss) plus 5ms document fetch on hit.”
Read it in the paper · Vector Database Limitations ↗break-even / miss-cost assumptions embedded in response-cache economics
The router assumes all three AI model folders (for intent, personal-info detection, and security scanning) exist on disk and contain the right files in the right format. Nothing checks this at startup. If a model folder is missing, renamed, or points to the wrong type of model, the entire routing system fails silently on the first request — there is no early warning during setup.
What to do: Add a startup health check that verifies each model folder exists and contains the expected config file before accepting any traffic, so a missing model is caught immediately rather than on the first user request.
candle-binding/src/ffi/instances/mod.rs:Options
The labels your AI model outputs and the labels you type into routing rules must match exactly — including spelling and capitalization. Nothing checks that they agree. If you swap in a newly trained model whose label names changed even slightly, or if a rule is typed with a small difference, that rule will quietly stop working and all traffic will be sent to the fallback model. Everything looks fine; the routing policy is just silently doing nothing.
What to do: After loading the model, compare its known label list against all labels referenced in active routing rules, and log a clear warning at startup for any rule that references a label the model doesn't produce.
candle-binding/src/classifiers/lora/intent_lora.rs:IntentLoRAClassifier::classify_intent
If you configure the system to use a GPU, it assumes the GPU exists, has the right drivers, and has enough free memory for all three AI models at once. Nothing checks this before trying to load the models. On a shared server where another program is using the GPU, or on a machine without a compatible graphics card, the failure message you get will be technical and hard to interpret.
What to do: Document the minimum GPU memory requirement clearly, and make the startup log print which device was selected and whether model loading succeeded before accepting traffic.
candle-binding/src/ffi/instances/mod.rs:Options
The system records where in a message personal information was found, using position numbers. Those numbers are likely measured in the AI model's internal word-piece units, not in the actual character positions in the original text. If anything later uses those positions to highlight or redact the original text — for example in the dashboard traces — it will mark the wrong places, especially for messages with emoji, accented characters, or non-English text.
What to do: Clarify in the struct documentation whether positions are token indices or character offsets, and if they are token indices, add a conversion step before exposing them to the dashboard or any redaction pipeline.
candle-binding/src/classifiers/lora/pii_lora.rs:PIILoRAClassifier
All requests share the same pool of worker threads for running the three AI classifiers in parallel. Under heavy simultaneous traffic, those threads get divided among all requests at once. A single request that normally classifies in parallel may end up running its three classifiers one-at-a-time when the server is busy, making everything slower with no clear signal about why.
What to do: Consider setting a concurrency limit on the router (maximum simultaneous requests being classified) that matches the available thread count, so latency stays predictable under load.
candle-binding/src/classifiers/lora/parallel_engine.rs:ParallelLoRAEngine::classify_parallel
Routing rules are saved as text expressions. If a signal name — such as an intent category or personal-info type — ever contains a quotation mark or backslash, the saved rule will be corrupted. When the system loads that rule back, it will silently ignore it, meaning that routing policy quietly disappears with no error shown to the user.
What to do: Add quote escaping when building the expression text, and add unescaping when reading it back, so unusual label names don't silently break saved rules.
dashboard/frontend/src/components/ExpressionBuilderSupport.ts:serializeNode
The list of how to connect to each AI provider — including things like required version headers that some providers enforce — is baked in when the software starts and never updated while it runs. If a provider like Anthropic stops accepting old version headers or changes their authentication format, every request to that provider starts failing, and it looks like the provider is down rather than the configuration being out of date.
What to do: Track provider API version requirements in a changelog and add a startup log line showing which API versions are configured for each provider, so a sudden failure can be quickly diagnosed as a catalog update needed.
config/catalog/resources/providers/anthropic.yaml:ProviderDefinition
Configuration values like which hardware device to use and what numeric precision to run at are passed as plain text strings. If there is a typo — even just wrong capitalization — the system silently falls back to a default instead of warning you. You might expect GPU acceleration but get CPU performance, or expect high-precision math but get lower-precision results, with nothing in the logs to explain why.
What to do: Add validation after deserializing the configuration that checks each string field against its known valid values and logs a clear error for unrecognized options before trying to use them.
candle-binding/src/ffi/instances/mod.rs:Options
All management of the loaded AI models goes through a single internal lock. While actual classification runs in parallel just fine, creating, listing, or removing a model instance blocks everything else trying to do the same thing at that moment. Under very high traffic this becomes a bottleneck that is hard to diagnose because normal request processing is unaffected.
What to do: Be aware that hot-reloading models during peak traffic will cause brief management stalls; plan model updates for low-traffic windows until a sharded registry is implemented.
candle-binding/src/ffi/instances/mod.rs:FFI Instance Registry
Open the standalone hidden-assumptions report for semantic-router →
How Data Flows Through the System
A request arrives at the router as an OpenAI-compatible HTTP POST. Before forwarding it, the router passes the message text through the Rust FFI boundary into the ParallelLoRAEngine, which runs three transformer classifiers simultaneously: intent (sequence classification → label + confidence), PII (token classification → span list), and security (sequence classification → threat score). These signals are returned to Go as a RoutingSignals struct. The Go policy engine evaluates the configured RuleNode expression (e.g. 'intent("support") AND NOT pii("SSN") AND NOT security("jailbreak")') against the signals to select a target model and provider. The provider catalog supplies the concrete endpoint URL and auth headers. The router rewrites the request's model field, forwards it to the selected backend, and streams the response tokens back to the client. Simultaneously, request metadata (signals, routing decision, latency) is recorded for the dashboard's tracing and evaluation subsystems.
- Receive HTTP request — The Go router accepts an OpenAI-compatible HTTP POST to /v1/chat/completions (or /v1/responses). The JSON body is deserialized into an internal request struct preserving all fields. Stream mode is detected from the 'stream' field to determine whether to use SSE for the response. (config: compatibility.config_schema)
- Extract signals from request text — The concatenated message content is passed via C FFI to the Rust ParallelLoRAEngine. Rayon spawns three parallel tasks: IntentLoRAClassifier.classify_intent() runs sequence classification through a BERT or ModernBERT backbone to produce an intent label and confidence; PIILoRAClassifier detects PII token spans with per-occurrence confidence scores; SecurityLoRAClassifier scores for jailbreak/prompt-injection threats. Each classifier tokenizes the input using a UnifiedTokenizer, runs a forward pass on the loaded model weights, applies softmax, and returns results. Total latency is bounded by the slowest of the three. [IncomingChatRequest → RoutingSignals] (config: max_input_tokens, precision, device +1)
- Run parallel classifier engine — ParallelLoRAEngine.classify_parallel() uses rayon::join or par_iter to execute all three classifiers concurrently on the same text. Each classifier holds its model weights in an Arc, so multiple concurrent requests can use different classifiers simultaneously without contention. Results are aggregated into a DualPathResult indicating which model architecture was used (BERT vs ModernBERT) and the per-task outcomes. [IncomingChatRequest → DualPathResult] (config: model_type, adapters)
- Evaluate routing policy — The Go routing engine evaluates the user-configured RuleNode expression (stored as a text expression like 'intent("book_flight") AND NOT pii("SSN")') against the extracted RoutingSignals. Signal leaf nodes are resolved by looking up the corresponding field in RoutingSignals and applying a confidence threshold comparison. AND/OR/NOT operators combine boolean sub-results. The expression evaluator walks the parse tree depth-first and returns a boolean match per configured route rule. [RoutingSignals] (config: compatibility.required_features, schema_version)
- Select routing target — The first routing rule whose expression evaluates to true wins. Its associated target specifies a provider ID (e.g. 'anthropic', 'openai', 'bedrock') and model name. If no rule matches, a default fallback model is used. The router also applies cost, latency, and privacy constraints configured per route before finalizing the target. [ProviderDefinition] (config: channel, release, compatibility.router.min +1)
- Translate routing decision to HTTP call — The provider catalog entry for the selected provider ID is looked up. It supplies: the base URL, the correct path for the operation (possibly from path_overrides), the authentication header and strategy (bearer token, API key header, etc.), any fixed default_headers (e.g. 'anthropic-version: 2023-06-01'), and the protocol version. The router rewrites the request's model field to the target model name, injects auth credentials from environment/secrets, and constructs the outbound HTTP request. [ProviderDefinition → IncomingChatRequest] (config: default_base_url, auth.strategy, auth.header +5)
- Stream response to client — The router proxies the HTTP response from the selected backend. For streaming requests, it reads Server-Sent Events (SSE) chunks from the backend and forwards them directly to the client, accumulating token counts and timing. For non-streaming, it buffers the full response. The dashboard playground displays tokens via the Message.isStreaming flag and appends to Message.content incrementally.
- Record trace and signals for observability — After the request completes, the router writes a trace record containing: the extracted signals (intent, PII presence, security score), the routing decision (which rule fired, which provider was selected), latency per stage, and token counts. This feeds the dashboard's Tracing page, Logs page, and Evaluation system which compares quality across model variants. [RoutingSignals → EvaluationGate]
Data Models
The data structures that flow between stages — the contracts that hold the system together.
src/semantic-router/OpenAI-compatible JSON body: {model: string, messages: [{role: 'user'|'assistant'|'system', content: string}], stream?: bool, temperature?: float, ...passthrough fields}
Arrives over HTTP, is parsed and held in memory while signals are extracted, then forwarded (possibly with model field rewritten) to a backend provider.
candle-binding/src/classifiers/mod.rsStruct containing: IntentResult {intent: String, confidence: f32, processing_time_ms: u64}, PIIResult {has_pii: bool, pii_types: Vec<String>, occurrences: Vec<PIIOccurrence {pii_type, confidence, token, start_pos, end_pos}>, confidence: f32}, SecurityResult {is_threat: bool, threat_type: String, confidence: f32}
Produced by the Rust classifier layer for each request, passed via FFI to the Go router, consumed by the policy engine to determine which model to call, then discarded.
candle-binding/src/classifiers/mod.rsStruct: {path_used: ModelType (BERT|ModernBERT), results: Vec<TaskResult {task: ClassificationTask, result: String, confidence: f32}>, confidence: f32, processing_time_ms: f32}
Produced by the ParallelLoRAEngine when running all three classifiers simultaneously using Rayon; aggregates per-task results into a single response passed back to the routing layer.
config/catalog/resources/providers/anthropic.yamlYAML document: {id: string, display_name: string, category: 'model_api'|'start_here'|'private_runtime', support_tier: 'native'|'compatible'|'runtime', protocols: string[], default_protocol: string, default_base_url: string, auth: {strategy: 'bearer'|'api_key_header', header: string}, path_overrides: map<string,string>, reasoning_transport?: string}
Loaded from YAML at startup into an in-memory registry; read when a routing decision maps to a named provider to construct the outbound HTTP request with correct URL, headers, and auth.
dashboard/frontend/src/components/ExpressionBuilderSupport.tsRecursive union type: {operator: 'AND'|'OR', conditions: RuleNode[]} | {operator: 'NOT', conditions: [RuleNode]} | {signalType: string, signalName: string} — serializes to expression strings like: intent("book_flight") AND NOT pii("SSN")
Constructed interactively in the dashboard's visual expression builder, serialized to a string expression, stored in the router config, then parsed and evaluated at runtime against each request's signals.
dashboard/frontend/src/components/ChatComponentTypes.tsInterface: {id: string, role: 'user'|'assistant'|'system', content: string, timestamp: Date, isStreaming?: bool, choices?: Choice[], thinkingProcess?: string, toolCalls?: ToolCall[], toolResults?: ToolResult[], reasoning_mom_responses?: ReMoMRoundResponse[], attachments?: PlaygroundAttachmentSummary[]}
Created when the user submits a message in the playground, updated incrementally as streaming tokens arrive, and retained in the conversation history for display and evaluation.
candle-binding/src/ffi/instances/mod.rsDeserializable struct: {model_path: String, model_type: String, device: String ('cpu'|'cuda'|'metal'), precision: String ('f32'|'f16'|'bf16'), max_input_tokens: usize, document_max_input_tokens: usize, overflow: String, adapters: Vec<AdapterSpec {name, path}>, generation_max_tokens: usize}
Passed as JSON from Go to the Rust FFI layer when instantiating a classifier; controls which model weights are loaded, which hardware device to use, and token limits — stored in the instance registry for the lifetime of the process.
dashboard/frontend/src/types/evaluationReport.tsType referenced via EvaluationCoverage and GateVerdict; represents a named quality gate with pass/fail verdict applied across evaluation tracks (model comparison runs)
Created when an evaluation experiment is configured, populated with verdicts as the router runs requests against multiple model variants, displayed in the EvaluationPlane UI.
System Behavior
How the system operates at runtime — where data accumulates, what loops, what waits, and what controls what.
Data Pools
A global Mutex<HashMap<u64, Arc<ModelInstance>>> holding all loaded classifier models. Each model is loaded once into device memory (CPU/GPU/Metal) and reused for all subsequent requests. The AtomicU64 counter generates monotonically increasing resource IDs. This is the primary cache preventing repeated model weight deserialization.
~60 YAML files loaded at startup into an in-memory map from provider ID string to ProviderDefinition. Consulted on every request to translate a routing decision into a concrete HTTP endpoint.
Conversation history stored in browser localStorage under keys 'sr:playground:queue' and 'sr:playground:active-conversation'. Survives page refreshes. Holds the full Message array per conversation including streamed content, tool calls, and ReMoM (Recursive Mixture-of-Models) intermediate responses.
A WeakMap<RouteLoader, LazyExoticComponent> and Map<RouteLoader, Promise<unknown>> that ensure each dashboard page module is loaded only once. Prevents React from recreating lazy() references across navigation transitions, which would cause suspended renders to be discarded.
Feedback Loops
- Parallel classifier retry on device failure (retry, balancing) — Trigger: A Candle tensor operation fails on GPU/Metal (OOM or device error). Action: The error propagates through UnifiedError back to the FFI caller; the Go layer can re-invoke with a different Options.device value (falling back to CPU). Exit: Successful inference completion or explicit error return to client.
- Streaming token accumulation (async-processing, reinforcing) — Trigger: Backend provider starts returning SSE chunks. Action: Each chunk is forwarded to the client and appended to the in-flight Message object in the dashboard; the isStreaming flag remains true. Exit: Provider sends the [DONE] SSE sentinel or closes the connection.
- Evaluation experiment run (polling, balancing) — Trigger: User launches an evaluation experiment in the dashboard. Action: The evaluation system sends the same prompts to multiple model variants (tracks), collects responses, scores them against EvaluationGates, and updates GateVerdict state. Exit: All tracks complete and verdicts are finalized.
Delays
- Model weight loading at first use (warmup, ~seconds to tens of seconds depending on model size and device) — First request after process start experiences high latency while weights are deserialized from disk into device memory; subsequent requests use the cached Arc.
- Dashboard page code splitting (async-processing, ~network-dependent (typically <500ms on first visit)) — Each dashboard page is loaded as a separate JavaScript bundle on first navigation; the lazyRoutePage cache prevents re-fetching on subsequent visits.
- LLM provider response latency (async-processing, ~100ms–60s depending on model and prompt length) — The router holds the client connection open and streams tokens as they arrive; routing signal extraction adds overhead before the provider call begins.
Control Points
- device (CPU/CUDA/Metal) (device-selection) — Controls: Which hardware the classifier model runs on. 'cuda' uses GPU tensor operations, 'metal' uses Apple Silicon GPU, 'cpu' is the fallback. Affects throughput and latency of signal extraction.. Default: Configurable per instance via Options.device
- precision (f32/f16/bf16) (precision-mode) — Controls: Numeric precision for model weights and activations. Lower precision reduces memory footprint and can increase throughput on supported hardware at slight accuracy cost.. Default: Configurable per instance via Options.precision
- max_input_tokens / overflow strategy (hyperparameter) — Controls: How long inputs are allowed to be before being truncated or windowed. The overflow field controls the strategy when text exceeds the model's context window (e.g. truncate, sliding window).. Default: Configurable per instance via Options.max_input_tokens and Options.overflow
- confidence_threshold (threshold) — Controls: Minimum confidence score for an intent or PII classification to be treated as a positive signal. Below this threshold the classifier returns a low-confidence or 'unknown' result, preventing noisy signals from triggering routing rules.. Default: Loaded from model config at construction time
- catalog channel / release (feature-flag) — Controls: Which version of the provider catalog is loaded. The channel ('stable', 'latest') and release fields gate which provider definitions and compatibility constraints are active, preventing incompatible CLI/router combinations from loading.. Default: channel: latest, release: unreleased
- reasoning_transport (architecture-switch) — Controls: How extended thinking / chain-of-thought reasoning is communicated to a provider API. Values like 'output_config_effort' or 'top_level_effort' change the shape of the outbound request payload for reasoning-capable models.. Default: Per-provider, e.g. 'output_config_effort' for Anthropic
Technology Stack
Main router process: HTTP server, policy evaluation, provider proxy, fleet simulation
In-process ML inference for BERT/ModernBERT classifiers via the Candle tensor library (HuggingFace); exposed to Go via C FFI
Dashboard frontend: topology visualization, expression builder, playground, evaluation UI, tracing
Data-parallel execution of multiple classifier tasks simultaneously within the Rust process
Alternative inference backend for running classifier models on CPU without Candle/GPU dependencies
Alternative inference backend targeting Intel CPUs and integrated GPUs
Graph rendering library used for the visual expression builder (AND/OR/NOT gate canvas) in the dashboard
Kubernetes deployment manifests and an operator for managing router instances in production clusters
Key Components
- ParallelLoRAEngine (executor) — Runs intent classification, PII detection, and security/jailbreak detection simultaneously on a single request using Rayon's thread pool for data parallelism. Each of the three classifiers (IntentLoRAClassifier, PIILoRAClassifier, SecurityLoRAClassifier) is held as an Arc and executed in parallel, with results merged into a combined signal set. This is the hot path for signal extraction on every routed request.
candle-binding/src/classifiers/lora/parallel_engine.rs - IntentLoRAClassifier (processor) — Runs sequence classification on request text using either a BERT or ModernBERT transformer backbone with merged LoRA weights. Detects which conversational intent (e.g. 'book_flight', 'customer_support') the message belongs to. At construction it reads label mappings from the model directory's config.json, detects architecture type, and loads weights onto the selected device. Supports single-text and batch inference.
candle-binding/src/classifiers/lora/intent_lora.rs - PIILoRAClassifier (processor) — Runs token-level classification (NER-style, labeling each token) to locate Personally Identifiable Information spans in request text — SSNs, emails, phone numbers, etc. Returns both a boolean flag and a list of PIIOccurrence structs with character offsets and per-span confidence scores. Supports BERT and ModernBERT backends.
candle-binding/src/classifiers/lora/pii_lora.rs - FFI Instance Registry (registry) — Maintains a global, thread-safe map (protected by a Mutex<HashMap>) of named classifier instances keyed by u64 resource IDs. Go code calls into Rust via C ABI functions to create, query, and destroy instances. Each acquisition clones an Arc before leaving the lock, ensuring an in-flight inference cannot be interrupted by a concurrent removal. Stores all model types: BERT, ModernBERT, DebertaV3, Qwen3Guard, and rerankers.
candle-binding/src/ffi/instances/mod.rs - Provider Catalog (registry) — A directory of ~60 YAML files, one per LLM provider, declaring how to reach each API: base URL, authentication strategy, supported protocols (openai/chat-completions@1, anthropic/messages@1, etc.), path overrides, and reasoning transport conventions. The router reads this at startup to build an in-memory map from provider ID to connection parameters, which it consults when translating a routing decision into an outbound HTTP request.
config/catalog/resources/providers/ - ExpressionBuilderSupport + ExpressionBuilderNodes (transformer) — Implements the logic for a visual drag-and-drop rule editor in the dashboard. RuleNode trees (AND/OR/NOT gates over signal leaf nodes) are constructed interactively, then serialized by serializeNode() into text expressions like 'intent("book_flight") AND NOT pii("SSN")' that the router evaluates. parseExprText() does the reverse for loading saved rules. This is the primary interface through which operators encode routing policy.
dashboard/frontend/src/components/ExpressionBuilderSupport.ts - routeLoaders + lazyRoutePage (loader) — Manages code-split page loading for the dashboard. Each page module (Playground, Evaluation, Topology, Logs, etc.) is imported lazily; lazyRoutePage() caches the lazy() identity per loader function to prevent React from discarding suspended renders during navigation transitions. prefetching is triggered on route match before the user clicks.
dashboard/frontend/src/app/routeLoaders.ts
Explore the interactive analysis
See the full architecture map, data flow, and code patterns visualization.
Analyze on CodeSeaRelated Repository Repositories
Frequently Asked Questions
What is semantic-router used for?
Routes LLM requests across heterogeneous models using semantic signals and policies vllm-project/semantic-router is a 7-component repository written in Go. Data flows through 8 distinct pipeline stages. The codebase contains 5968 files.
How is semantic-router architected?
semantic-router is organized into 6 architecture layers: Routing Core (Go), ML Signal Extraction (Rust/Candle), Provider Catalog, Dashboard (React + Go backend), and 2 more. Data flows through 8 distinct pipeline stages. This layered structure keeps concerns separated and modules independent.
How does data flow through semantic-router?
Data moves through 8 stages: Receive HTTP request → Extract signals from request text → Run parallel classifier engine → Evaluate routing policy → Select routing target → .... A request arrives at the router as an OpenAI-compatible HTTP POST. Before forwarding it, the router passes the message text through the Rust FFI boundary into the ParallelLoRAEngine, which runs three transformer classifiers simultaneously: intent (sequence classification → label + confidence), PII (token classification → span list), and security (sequence classification → threat score). These signals are returned to Go as a RoutingSignals struct. The Go policy engine evaluates the configured RuleNode expression (e.g. 'intent("support") AND NOT pii("SSN") AND NOT security("jailbreak")') against the signals to select a target model and provider. The provider catalog supplies the concrete endpoint URL and auth headers. The router rewrites the request's model field, forwards it to the selected backend, and streams the response tokens back to the client. Simultaneously, request metadata (signals, routing decision, latency) is recorded for the dashboard's tracing and evaluation subsystems. This pipeline design reflects a complex multi-stage processing system.
What technologies does semantic-router use?
The core stack includes Go (Main router process: HTTP server, policy evaluation, provider proxy, fleet simulation), Rust + Candle (In-process ML inference for BERT/ModernBERT classifiers via the Candle tensor library (HuggingFace); exposed to Go via C FFI), React + TypeScript (Dashboard frontend: topology visualization, expression builder, playground, evaluation UI, tracing), Rayon (Data-parallel execution of multiple classifier tasks simultaneously within the Rust process), ONNX Runtime (Alternative inference backend for running classifier models on CPU without Candle/GPU dependencies), OpenVINO (Alternative inference backend targeting Intel CPUs and integrated GPUs), and 2 more. A focused set of dependencies that keeps the build manageable.
What system dynamics does semantic-router have?
semantic-router exhibits 4 data pools (FFI Instance Registry, Provider Catalog Registry), 3 feedback loops, 6 control points, 3 delays. The feedback loops handle retry and async-processing. These runtime behaviors shape how the system responds to load, failures, and configuration changes.
What design patterns does semantic-router use?
5 design patterns detected: Dual-path model architecture dispatch, Arc-protected global instance registry with atomic ID generation, Declarative provider catalog with protocol versioning, Visual expression tree for routing policy, Stable lazy route identity caching.
Analyzed on September 17, 2026 by CodeSea. Written by Karolina Sarna.