vllm-project/semantic-router
A programmable Mixture-of-Models router for heterogeneous LLM inference
15 hidden assumptions · 8-stage pipeline · 7 components
Like any codebase, this repository makes assumptions it never checks — most are routine. The ones worth your attention are below, in plain language with what to do about each.
Routes LLM inference requests across heterogeneous models using semantic signals and policies
A client sends an LLM chat request to the semantic router's HTTP endpoint. The router deserializes the request body, identifies the protocol (OpenAI chat-completions or Anthropic messages) from the path, then fans out the request text to the Rust ML classifier layer via FFI — running intent detection, PII scanning, and security analysis in parallel using Rayon. The resulting signal scores are fed into the policy evaluation engine, which walks the operator-defined RuleNode expression tree (compiled from the dashboard's expression builder) to produce a routing decision: a specific provider+model, a fallback chain, or a multi-model composition path. The router then looks up the target provider in the catalog, rewrites the request (base URL, auth headers, path overrides, protocol translation if needed), and proxies it to the backend. Streaming responses are forwarded token-by-token back to the client. All decisions, signals, and latencies are written to the audit/trace store for dashboard visibility.
Under the hood, the system uses 3 feedback loops, 5 data pools, 6 control points to manage its runtime behavior.
A 7-component repository. 5834 files analyzed. Data flows through 8 distinct pipeline stages.
Hidden Assumptions
Most of what this code assumes is routine. These 3 are the ones most likely to cause trouble here — in plain terms, with what to do about each. The rest are minor; they're under "Show everything".
The paper's headline claim that relaxing cache matching under load cuts traffic to busy models by roughly nine to seventeen percent is an on-paper calculation, not a measured result. The authors explicitly say real performance depends on the specific mix of queries and needs to be tested before it can be trusted.
What to do: Treat any projected traffic-reduction or savings number from load-based adaptation as an untested estimate, and measure actual hit rates on your own traffic before reporting benefits.
“The traffic reduction figures (9–17%) are theoretical projections based on assumed linear relationships between threshold relaxation and hit rate improvements.”
Read it in the paper · Discussion ↗The recommended similarity cutoffs and expiry times per query type come from the authors' assumptions about how tightly different query types cluster and how fast their content goes stale. If your traffic behaves differently, the cache may return semantically wrong answers or fail to reuse valid ones — and nothing flags it.
What to do: Before trusting cached answers, verify the threshold and expiry settings actually fit your query categories by A/B testing them on your own traffic as the paper suggests.
“Initial policies derive from category properties: dense spaces use tight thresholds ( ≥ \geq 0.88), sparse spaces use loose thresholds ( ≤ \leq 0.78)”
Read it in the paper · Discussion ↗The speed advantages that make this caching approach economical were characterized up to roughly ten million cached entries. The authors say beyond that you must split the index; if you run a much larger cache without doing so, the fast-response assumption underlying the whole benefit argument may no longer apply.
What to do: Keep the cache within the size range the paper characterizes, and plan to shard the index before you exceed roughly ten million entries.
“Practical latencies: 2–3ms for 1M entries, 5–8ms for 10M entries. Beyond 10M entries, consider sharding by category or vector space region.”
Read it in the paper · Discussion ↗Show everything (12 more)
Whether caching a given query type pays off depends on fixed example numbers for how slow the model is and how fast the cache search is. If your models or infrastructure are faster or slower than the paper's examples, the point at which caching helps versus hurts moves, and a category the paper calls worthwhile might not be for you.
What to do: Recompute the break-even hit rate using your own measured model and cache-search latencies rather than assuming the paper's example figures apply.
“Total cost: approximately 30ms per query (hit or miss) plus 5ms document fetch on hit. Without caching, LLM inference takes”
Read it in the paper · Methods ↗Provider/model selection and break-even economics (algorithm selection fragments, latency-aware routing)
All the per-category safety of this cache depends on each query being labeled with the right category. The cleanest options assume the client or endpoint already tells you the category; if you rely instead on an automatic classifier, misclassification silently applies the wrong freshness and matching rules.
What to do: Ensure query categories are assigned reliably — prefer explicit or endpoint-based labels — and watch for misclassifications that would apply the wrong staleness and threshold rules.
“Explicit routing and endpoint-based approaches add zero classification overhead.”
Read it in the paper · Discussion ↗Category classification path (prompt classifier / routing signals feeding cache category)
When the router starts up, it tries to load the AI models it needs to screen requests. If those model files aren't already sitting in the right folders on the same machine, nothing warns you — the router starts fine, happily accepts traffic, but quietly skips all the safety checks (PII detection, intent matching, security scanning) and just routes everything using its fallback rule. You could be running a 'secured' router with no active security for hours without knowing.
What to do: Before going live, verify that all three model folders exist and each contains the expected weight files and a config file; a startup health-check that tests each classifier with a short sample sentence would surface this immediately.
candle-binding/src/ffi/instances/mod.rs:Options
The system reads a list of category names from a settings file and assumes they line up perfectly with what the AI model learned. If that file ever gets edited, regenerated, or replaced with a version that has categories in a different order or with gaps, the model's answers get silently relabeled — it thinks it said 'booking' but the router records 'cancel' and routes accordingly. Everything looks normal in the logs; the wrong decisions just quietly accumulate.
What to do: After loading the model, do a one-time sanity check by running a handful of known test phrases and confirming the returned labels match what you expect; catching a mislabeled model takes seconds and prevents routing everything to the wrong destination.
candle-binding/src/classifiers/lora/intent_lora.rs:IntentLoRAClassifier::new
When the system finds sensitive information like an email address or ID number in a message, it records where in the message it found it using position numbers. Those position numbers count in the AI model's internal units, not in ordinary text characters — and for many languages or punctuation patterns, they don't match. Any feature that uses those positions to actually hide or mark the sensitive text may cut in the wrong place, leaving part of the sensitive data exposed while appearing to have done its job.
What to do: Check whether anything downstream actually uses the position numbers from PII results to redact text; if so, add a conversion step that maps token positions back to character positions before using them, and verify with a test that includes multi-byte characters and punctuation.
candle-binding/src/classifiers/lora/pii_lora.rs:PIILoRAClassifier
If you open a routing rule in the dashboard that was saved by a different version of the software, and the dashboard can't understand its format, it quietly shows you an empty canvas instead of an error. If you then hit save on that blank canvas, you overwrite your actual routing rules with nothing — and all traffic starts going to the default fallback without any warning.
What to do: Add a visible error message when a saved policy cannot be loaded into the editor, and require an explicit confirmation before allowing a save that would replace a non-empty stored policy with an empty one.
dashboard/frontend/src/components/ExpressionBuilderSupport.ts:parseExprText
The three AI screening models — for intent, privacy, and security — are loaded together as a group. If any one of them fails (say the security model file is corrupted or the wrong size), none of the three will work. You lose all screening, not just security screening, and everything gets routed by your fallback rule with no explanation beyond a startup error.
What to do: Consider logging a specific message for each model that fails to load, and decide whether to allow partial operation with the remaining models or fail loudly at startup so the problem is immediately visible.
candle-binding/src/classifiers/lora/parallel_engine.rs:ParallelLoRAEngine::new
Every AI model loaded by the router stays in memory permanently until the process restarts. If you run experiments or load multiple model variants through the dashboard, each one takes a permanent chunk of RAM or GPU memory. On a smaller machine this eventually causes the system to run out of memory and crash, with no warning beforehand that you were approaching the limit.
What to do: Before running evaluation experiments that load multiple model variants, check available system memory against the size of each model file, and restart the router process if memory grows unexpectedly large.
candle-binding/src/ffi/instances/mod.rs:FFI Instance Registry
The minimum confidence score required before the system acts on a classification result is set once when the system starts and never updated, even if you change it through the dashboard. So if you lower the threshold hoping to catch more cases, the AI layer keeps using the old, higher cutoff — and the dashboard shows you results calculated with the new threshold while the actual routing uses the old one.
What to do: Document clearly where the confidence threshold is set and whether it can be changed without a restart; if it cannot, add a note in the dashboard wherever the threshold is configured.
candle-binding/src/classifiers/lora/intent_lora.rs:IntentLoRAClassifier
The settings for each AI provider — their web addresses, required headers, API version strings — are read once when the router starts and never refreshed. If a provider like Anthropic or OpenAI changes something on their end (even just bumping a required version header), every request to that provider starts failing. Because the failure looks like a network error, it is easy to spend time debugging the wrong thing before realizing the catalog just needs a refresh.
What to do: When a specific provider suddenly starts failing while others work, check whether that provider has recently announced API changes, and restart the router after updating the relevant provider file.
config/catalog/manifest.yaml:Provider Catalog Loader
If you tell the router to use a GPU but the machine doesn't have one (or doesn't have the right software drivers installed), the router crashes when it tries to load the AI models. The opposite is also true: if you forget to set this, the router quietly uses the CPU, which is much slower, and there is no warning that a GPU is sitting idle.
What to do: Confirm the device setting matches what is actually available on your server before starting; on cloud deployments, check that GPU drivers are installed if you intend to use GPU acceleration.
candle-binding/src/core/device.rs:resolve_device
When many requests arrive at the same time, all of them compete for the same pool of worker threads that run the AI classification. There is no limit on how many can pile up waiting, and nothing tells the rest of the system that it is overloaded. During a traffic spike, AI screening for all requests can slow to a crawl at the same time, causing a wave of timeouts that looks like an outage.
What to do: During load testing, observe how classification latency behaves under sustained concurrent traffic and set a reasonable limit on how many simultaneous classification requests the system accepts.
candle-binding/src/classifiers/lora/parallel_engine.rs:ParallelLoRAEngine
Open the standalone hidden-assumptions report for semantic-router →
How Data Flows Through the System
A client sends an LLM chat request to the semantic router's HTTP endpoint. The router deserializes the request body, identifies the protocol (OpenAI chat-completions or Anthropic messages) from the path, then fans out the request text to the Rust ML classifier layer via FFI — running intent detection, PII scanning, and security analysis in parallel using Rayon. The resulting signal scores are fed into the policy evaluation engine, which walks the operator-defined RuleNode expression tree (compiled from the dashboard's expression builder) to produce a routing decision: a specific provider+model, a fallback chain, or a multi-model composition path. The router then looks up the target provider in the catalog, rewrites the request (base URL, auth headers, path overrides, protocol translation if needed), and proxies it to the backend. Streaming responses are forwarded token-by-token back to the client. All decisions, signals, and latencies are written to the audit/trace store for dashboard visibility.
- Receive LLM API request — The router's HTTP server accepts a POST to /v1/chat/completions or /v1/messages. It reads the raw JSON body and identifies the protocol from the path. The Authorization header is validated against the tenant's API key. The request is deserialized into an internal envelope that preserves the original body alongside routing metadata (tenant ID, timestamp, request ID). (config: compatibility.config_schema, compatibility.required_features)
- Classify request signals in parallel — The request text (concatenated system + user message content) is passed via C FFI to the Rust ParallelLoRAEngine. Using Rayon's parallel iterator, three classifiers execute concurrently: IntentLoRAClassifier runs sequence classification to get an intent label + confidence; PIILoRAClassifier runs token-level BIO tagging to find PII spans; SecurityLoRAClassifier checks for jailbreak/prompt-injection patterns. Each classifier tokenizes the text using its own UnifiedTokenizer (BERT WordPiece or ModernBERT tokenizer), truncates to max_input_tokens with configurable overflow strategy, runs a tensor forward pass on CPU or GPU, and applies softmax to produce scores. [IncomingLLMRequest → DualPathResult]
- Evaluate routing policy expression — The scored signals (intent='booking', confidence=0.92; has_pii=false; security=clean) are matched against the operator-configured RuleNode expression tree. The tree is walked recursively: leaf nodes check if a named signal meets its threshold, operator nodes (AND/OR/NOT) combine child results. The evaluation produces a boolean decision per candidate route. The highest-priority matching route is selected. If no route matches, a configured default route or fallback chain is used. [DualPathResult → RuleNode]
- Look up provider and rewrite request — The selected route names a provider ID (e.g. 'anthropic') and model (e.g. 'claude-opus-4'). The router looks up the ProviderDefinition in the catalog registry to get the base URL, auth strategy (bearer token, api_key_header, etc.), required headers (e.g. anthropic-version: 2023-06-01), and any path_overrides. If the incoming protocol differs from the target provider's native protocol (e.g. OpenAI-format request to an Anthropic backend), the request body is translated between schemas. Auth credentials are injected from the tenant's secret store. [ProviderDefinition → IncomingLLMRequest] (config: default_base_url, default_protocol, auth.strategy +4)
- Proxy request to backend and stream response — The rewritten HTTP request is sent to the target backend. If stream=true, the router opens a server-sent event (SSE) stream and forwards chunks token-by-token to the client as they arrive, maintaining the OpenAI streaming delta format. For multi-model (ReMoM — Reasoning Mixture-of-Models) routes, multiple backends are called and intermediate responses are accumulated into ReMoMRoundResponse structs before the final answer is composed. Non-streaming responses are buffered and returned as a single JSON body. [IncomingLLMRequest → Message] (config: supported_operations, default_protocol)
- Write decision trace to audit store — After the response completes, the router writes a trace record containing: the original request ID, timestamp, selected provider+model, all signal scores (intent, PII, security), the matched policy rule, total latency, token counts, and any error codes. These records feed the dashboard's Logs, Tracing, Insights, and Evaluation pages. [Message]
- Dashboard: build and save routing policy — Operators use the ExpressionBuilder React component to drag signal nodes (intent(), pii(), security()) and operator gates (AND, OR, NOT) onto a canvas. ExpressionBuilderSupport.serializeNode() converts the RuleNode tree to a string expression. Saving posts this config to the dashboard backend, which writes it to the router's configuration store, making it live on next policy reload. [RuleNode]
- Dashboard: run evaluation experiment — From the Evaluation page, operators submit test cases through EvaluationPlane. Each test case is routed through the live router, and the response is scored against expected outcomes. EvaluationGate checks pass/fail against configurable thresholds. Results accumulate in EvaluationRunStatus and EvaluationTrackStatus states, displayed with GateVerdict badges (pass/fail/warn) and EvaluationCoverage metrics.
Data Models
The data structures that flow between stages — the contracts that hold the system together.
src/semantic-router/HTTP POST body: either OpenAI chat-completions format {model, messages: [{role, content}], stream, ...} or Anthropic messages format {model, messages, system, max_tokens, ...}; protocol identified by path and provider config
Arrives as raw HTTP body, deserialized into a protocol-specific struct, annotated with routing metadata, then proxied (possibly rewritten) to the selected backend
candle-binding/src/classifiers/lora/intent_lora.rsstruct { intent: String, confidence: f32, processing_time_ms: u64 } — top predicted intent label with a 0.0–1.0 confidence score and wall-clock latency
Created by IntentLoRAClassifier.classify_intent() after tokenizing the request text, running a forward pass through BERT or ModernBERT, and applying softmax over the label vocabulary; returned via FFI to the Go routing core as a routing signal
candle-binding/src/classifiers/lora/pii_lora.rsstruct { has_pii: bool, pii_types: Vec<String>, confidence: f32, occurrences: Vec<PIIOccurrence>, processing_time_ms: u64 } where PIIOccurrence = { pii_type: String, confidence: f32, token: String, start_pos: usize, end_pos: usize }
Produced by PIILoRAClassifier running token-level classification; each token in the input gets a BIO label, consecutive matching tokens are merged into PIIOccurrence spans, then aggregated into PIIResult
config/catalog/resources/providers/anthropic.yamlYAML document: { id: string, display_name: string, category: string, support_tier: string, default_base_url: string, protocols: string[], default_protocol: string, supported_operations: string[], auth: { strategy: string, header: string }, reasoning_transport: string, path_overrides: map<string,string>, compatibility: { ... } }
Loaded from YAML files at startup into an in-memory registry; looked up by provider ID when building the outbound HTTP request to rewrite URLs, inject auth headers, and select the correct API path
dashboard/frontend/src/components/ExpressionBuilderSupport.tsRecursive union type: { operator: 'AND'|'OR'; conditions: RuleNode[] } | { operator: 'NOT'; conditions: [RuleNode] } | { signalType: string; signalName: string } — a boolean expression tree over named signals
Constructed interactively in the dashboard's ExpressionBuilder drag-and-drop UI, serialized to a string expression like 'intent("booking") AND NOT pii("SSN")' via serializeNode(), then saved to router configuration
dashboard/frontend/src/components/ChatComponentTypes.tsinterface { id: string, role: 'user'|'assistant'|'system', content: string, timestamp: Date, isStreaming?: boolean, choices?: Choice[], thinkingProcess?: string, toolCalls?: ToolCall[], toolResults?: ToolResult[], reasoning_mom_responses?: ReMoMRoundResponse[], attachments?: PlaygroundAttachmentSummary[] }
Created in the dashboard playground when a user sends a message; updated incrementally as streaming tokens arrive from the router; ReMoMRoundResponse[] fields are populated when multi-model (Mixture-of-Models) routing is active and intermediate model responses are shown
candle-binding/src/classifiers/mod.rsstruct { path_used: ModelType, results: Vec<TaskResult>, confidence: f32, processing_time_ms: f32 } where TaskResult = { task: ClassificationTask, result: String, confidence: f32 } and ModelType distinguishes BERT vs ModernBERT backend
Produced by DualPathUnifiedClassifier which tries the preferred model architecture first; if confidence falls below threshold it may fall back to the alternate path; result is returned via FFI
candle-binding/src/ffi/instances/mod.rsstruct { model_path: String, model_type: String, device: String, precision: String, max_input_tokens: usize, overflow: String, adapters: Vec<AdapterSpec>, generation_max_tokens: usize } — deserialized from JSON passed across the FFI boundary
Passed by the Go router when calling into the Rust FFI to load a named model; used to instantiate the correct ModelFactory backend, configure the tokenizer window, and register the handle in the global instance registry
System Behavior
How the system operates at runtime — where data accumulates, what loops, what waits, and what controls what.
Data Pools
A global LazyLock<Mutex<HashMap<u64, Arc<ModelInstance>>>> that holds every loaded ML model (BERT classifiers, ModernBERT classifiers, rerankers, Qwen3 guard models). Models are loaded once and stay resident in memory; handles are u64 tokens vended to Go callers. The AtomicU64 counter ensures each allocation gets a unique ID.
In-memory map of provider ID strings to ProviderDefinition structs, populated at startup from ~60+ YAML files. Used on every request to resolve base URLs, auth strategies, and protocol translation rules.
Persisted store of operator-defined routing policies (serialized RuleNode expression trees plus metadata). The router reads policies at startup and on hot-reload signals; the dashboard writes to it when operators save policy changes.
Append-only store of per-request trace records including signal scores, routing decisions, latency, and token counts. Queried by the dashboard's Logs, Tracing, and Insights pages for observability.
WeakMap and Map caching lazy() React component identities and prefetch Promises per route loader function. Prevents re-creating lazy() on each suspended render (which would cause infinite retry loops in React Router v7's startTransition navigation).
Feedback Loops
- Confidence-based classifier path fallback (self-correction, balancing) — Trigger: DualPathUnifiedClassifier gets a confidence score below its threshold from the preferred model architecture. Action: Switches to the alternate classifier backend (e.g. from LoRA-adapted BERT to TraditionalModernBertClassifier) and re-runs inference on the same input. Exit: Second path returns a result regardless of confidence, or an error is propagated.
- Evaluation experiment polling (polling, balancing) — Trigger: Operator submits an evaluation run from the dashboard Evaluation page. Action: Dashboard polls the backend for EvaluationRunStatus / EvaluationTrackStatus updates, rendering progress and gate verdicts as they arrive. Exit: All tracks reach a terminal status (completed, failed, cancelled).
- Routing policy hot-reload (polling, balancing) — Trigger: Operator saves a new routing policy through the dashboard. Action: Router detects configuration change, reloads RuleNode policy trees without restarting, begins applying new rules to subsequent requests. Exit: Policy version stabilizes.
Delays
- ML model cold-start (warmup, ~Seconds to tens of seconds depending on model size and device) — First request after router startup incurs model load time (reading safetensors weights, building the compute graph, moving tensors to device). Subsequent requests hit the warm registry instantly.
- Streaming LLM response drain (queue-drain, ~Variable — depends on backend model speed and output length) — Router holds the client connection open while forwarding SSE chunks from the backend. Slow backends or long outputs increase end-to-end latency perceived by the client.
- Dashboard route lazy-load (async-processing, ~One network round-trip per unvisited page) — Each dashboard page (Evaluation, Topology, Logs, etc.) is a separate JS bundle loaded on first visit. routePreloads prefetches on hover/link-proximity to hide this latency.
Control Points
- model_type (architecture-switch) — Controls: Which classifier architecture (BERT, ModernBERT, DebertaV3, Qwen3Guard, MatryoshkaReranker) is instantiated for a given named resource. Changes which weights and tokenizer are loaded.. Default: Detected from model directory contents; overridable via Options.model_type
- precision (precision-mode) — Controls: Whether model weights are loaded in f32, f16, or bf16. Affects GPU memory usage, throughput, and numerical precision of classifier outputs.. Default: Configurable via Options.precision; defaults to f32 if empty
- device (device-selection) — Controls: Whether classifier inference runs on CPU, CUDA GPU, or Metal (Apple Silicon). resolve_device() maps the Options.device string to a candle_core::Device. Determines throughput and latency of signal classification.. Default: Configurable per instance; defaults to CPU
- max_input_tokens / overflow (hyperparameter) — Controls: Maximum token count passed to the classifier. The overflow strategy (truncate, sliding-window via tokenization_window) determines how inputs longer than the model's context window are handled — truncation loses signal; windowing increases latency.. Default: Set per instance via Options; 0 means use the model's architectural max
- confidence_threshold (threshold) — Controls: Minimum softmax probability for a classifier to report a label as matched. Requests scoring below this are treated as unclassified and routed to the default path. Loaded from global config at classifier construction time.
- catalog.channel / catalog.release (feature-flag) — Controls: Which version of the provider catalog is active and which features (entrypoints, isolated_recipes, decision_algorithms, effective_model_registry) are required to be present in the router before the catalog is accepted.. Default: channel: latest, release: unreleased
Technology Stack
Main routing server language — handles HTTP ingestion, policy evaluation, request proxying, streaming, and provider catalog management
On-device ML inference for routing signal classifiers; Candle is a Hugging Face tensor framework (like PyTorch but in Rust) that runs BERT/ModernBERT models on CPU/GPU without Python
Dashboard SPA for operator configuration, policy building, evaluation, and observability
Renders the routing policy expression builder as an interactive node graph in the dashboard
Rust data-parallelism library used by ParallelLoRAEngine to run intent, PII, and security classifiers concurrently on a shared thread pool
Alternative ML inference backend (onnx-binding/) for running classifier models in ONNX format, complementing the candle-binding Rust path
Intel-accelerated inference backend (openvino-binding/) for running classifiers on Intel CPUs and integrated GPUs
Manages router deployments in Kubernetes clusters via CRDs; handles scaling, configuration sync, and lifecycle management
Key Components
- ParallelLoRAEngine (executor) — Runs intent, PII, and security classifiers concurrently on a single input text using Rayon (a Rust data-parallelism library). Holds Arc-wrapped (reference-counted, thread-safe) handles to all three classifiers and dispatches them in parallel, collecting results into a combined signal set before returning to the routing core.
candle-binding/src/classifiers/lora/parallel_engine.rs - IntentLoRAClassifier (processor) — Tokenizes input text and runs a forward pass through either a BERT or ModernBERT sequence-classification model (with LoRA adapters merged into weights at load time) to predict the user's intent. Automatically detects model architecture by inspecting the model directory, loads label mappings from config.json, and applies a configurable confidence threshold before returning the top-1 label.
candle-binding/src/classifiers/lora/intent_lora.rs - PIILoRAClassifier (processor) — Runs token-level classification (where every input token gets a BIO entity label like B-EMAIL, I-SSN, O) to detect personally identifiable information spans in request text. Merges consecutive same-type tokens into PIIOccurrence spans with character offsets, enabling the router to redact or reroute requests containing sensitive data.
candle-binding/src/classifiers/lora/pii_lora.rs - FFI Instance Registry (registry) — A global, mutex-protected HashMap that maps string resource IDs to Arc-wrapped live model instances (BERT classifiers, ModernBERT classifiers, rerankers, generative Qwen3 guard models). Go code calls into this registry via C FFI to load models by name and get back an opaque u64 handle. Every FFI call acquires an Arc clone before releasing the lock, so removing a handle cannot unload a model that is mid-inference.
candle-binding/src/ffi/instances/mod.rs - Provider Catalog Loader (loader) — The catalog manifest declares which YAML provider files are valid for a given router+CLI version range (compatibility.router.min/max_exclusive). At startup the router reads this manifest, validates version compatibility, then loads each provider YAML from config/catalog/resources/providers/ to build an in-memory map of provider IDs to ProviderDefinition structs used for request rewriting and auth injection.
config/catalog/manifest.yaml - ExpressionBuilderSupport (RuleNode serializer/parser) (serializer) — Converts between the dashboard's tree-structured routing policy (RuleNode) and a human-readable string expression (e.g. 'intent("booking") AND NOT pii("SSN")'). serializeNode() walks the tree recursively and emits the string; parseExprText() does the reverse using a recursive-descent parser. The serialized form is what gets written to router config files.
dashboard/frontend/src/components/ExpressionBuilderSupport.ts - DualPathUnifiedClassifier (adapter) — Abstracts over the two classification paths (traditional BERT vs LoRA-adapted ModernBERT) behind a single interface. Selects which backend to use based on model architecture detection and EmbeddingRequirements, then returns a DualPathResult that records which path was taken — enabling the routing core to trust classifier output without knowing which model ran.
candle-binding/src/classifiers/unified/
Explore the interactive analysis
See the full architecture map, data flow, and code patterns visualization.
Analyze on CodeSeaRelated Repository Repositories
Frequently Asked Questions
What is semantic-router used for?
Routes LLM inference requests across heterogeneous models using semantic signals and policies vllm-project/semantic-router is a 7-component repository written in Go. Data flows through 8 distinct pipeline stages. The codebase contains 5834 files.
How is semantic-router architected?
semantic-router is organized into 6 architecture layers: Routing Core, ML Classifier Layer, Provider Catalog, Management Dashboard, and 2 more. Data flows through 8 distinct pipeline stages. This layered structure keeps concerns separated and modules independent.
How does data flow through semantic-router?
Data moves through 8 stages: Receive LLM API request → Classify request signals in parallel → Evaluate routing policy expression → Look up provider and rewrite request → Proxy request to backend and stream response → .... A client sends an LLM chat request to the semantic router's HTTP endpoint. The router deserializes the request body, identifies the protocol (OpenAI chat-completions or Anthropic messages) from the path, then fans out the request text to the Rust ML classifier layer via FFI — running intent detection, PII scanning, and security analysis in parallel using Rayon. The resulting signal scores are fed into the policy evaluation engine, which walks the operator-defined RuleNode expression tree (compiled from the dashboard's expression builder) to produce a routing decision: a specific provider+model, a fallback chain, or a multi-model composition path. The router then looks up the target provider in the catalog, rewrites the request (base URL, auth headers, path overrides, protocol translation if needed), and proxies it to the backend. Streaming responses are forwarded token-by-token back to the client. All decisions, signals, and latencies are written to the audit/trace store for dashboard visibility. This pipeline design reflects a complex multi-stage processing system.
What technologies does semantic-router use?
The core stack includes Go (Main routing server language — handles HTTP ingestion, policy evaluation, request proxying, streaming, and provider catalog management), Rust + Candle (On-device ML inference for routing signal classifiers; Candle is a Hugging Face tensor framework (like PyTorch but in Rust) that runs BERT/ModernBERT models on CPU/GPU without Python), React (TypeScript) (Dashboard SPA for operator configuration, policy building, evaluation, and observability), ReactFlow (Renders the routing policy expression builder as an interactive node graph in the dashboard), Rayon (Rust data-parallelism library used by ParallelLoRAEngine to run intent, PII, and security classifiers concurrently on a shared thread pool), ONNX Runtime (Alternative ML inference backend (onnx-binding/) for running classifier models in ONNX format, complementing the candle-binding Rust path), and 2 more. A focused set of dependencies that keeps the build manageable.
What system dynamics does semantic-router have?
semantic-router exhibits 5 data pools (FFI Model Instance Registry, Provider Catalog Registry), 3 feedback loops, 6 control points, 3 delays. The feedback loops handle self-correction and polling. These runtime behaviors shape how the system responds to load, failures, and configuration changes.
What design patterns does semantic-router use?
5 design patterns detected: Dual-path classifier with architecture auto-detection, Versioned provider catalog with compatibility gating, Arc-based FFI model registry preventing use-after-free, Recursive tree expression builder for routing policies, Lazy route loading with stable identity cache.
Analyzed on September 17, 2026 by CodeSea. Written by Karolina Sarna.