Hidden Assumptions in semantic-router

The claim that low-hit-rate categories become worth caching rests on specific timing numbers the authors assumed for search speed and model response time. If your servers are faster, slower, or your models respond differently, the point at which caching pays off shifts, and a category the paper says is viable might not be in your setup.

What to do: Measure your actual search latency and model response times, then recompute whether your low-hit-rate categories still clear the break-even point before enabling caching for them.

The router assumes all three AI model folders (for intent, personal-info detection, and security scanning) exist on disk and contain the right files in the right format. Nothing checks this at startup. If a model folder is missing, renamed, or points to the wrong type of model, the entire routing system fails silently on the first request — there is no early warning during setup.

What to do: Add a startup health check that verifies each model folder exists and contains the expected config file before accepting any traffic, so a missing model is caught immediately rather than on the first user request.

The labels your AI model outputs and the labels you type into routing rules must match exactly — including spelling and capitalization. Nothing checks that they agree. If you swap in a newly trained model whose label names changed even slightly, or if a rule is typed with a small difference, that rule will quietly stop working and all traffic will be sent to the fallback model. Everything looks fine; the routing policy is just silently doing nothing.

What to do: After loading the model, compare its known label list against all labels referenced in active routing rules, and log a clear warning at startup for any rule that references a label the model doesn't produce.

If you configure the system to use a GPU, it assumes the GPU exists, has the right drivers, and has enough free memory for all three AI models at once. Nothing checks this before trying to load the models. On a shared server where another program is using the GPU, or on a machine without a compatible graphics card, the failure message you get will be technical and hard to interpret.

What to do: Document the minimum GPU memory requirement clearly, and make the startup log print which device was selected and whether model loading succeeded before accepting traffic.

The system records where in a message personal information was found, using position numbers. Those numbers are likely measured in the AI model's internal word-piece units, not in the actual character positions in the original text. If anything later uses those positions to highlight or redact the original text — for example in the dashboard traces — it will mark the wrong places, especially for messages with emoji, accented characters, or non-English text.

What to do: Clarify in the struct documentation whether positions are token indices or character offsets, and if they are token indices, add a conversion step before exposing them to the dashboard or any redaction pipeline.

All requests share the same pool of worker threads for running the three AI classifiers in parallel. Under heavy simultaneous traffic, those threads get divided among all requests at once. A single request that normally classifies in parallel may end up running its three classifiers one-at-a-time when the server is busy, making everything slower with no clear signal about why.

What to do: Consider setting a concurrency limit on the router (maximum simultaneous requests being classified) that matches the available thread count, so latency stays predictable under load.

Routing rules are saved as text expressions. If a signal name — such as an intent category or personal-info type — ever contains a quotation mark or backslash, the saved rule will be corrupted. When the system loads that rule back, it will silently ignore it, meaning that routing policy quietly disappears with no error shown to the user.

What to do: Add quote escaping when building the expression text, and add unescaping when reading it back, so unusual label names don't silently break saved rules.

The list of how to connect to each AI provider — including things like required version headers that some providers enforce — is baked in when the software starts and never updated while it runs. If a provider like Anthropic stops accepting old version headers or changes their authentication format, every request to that provider starts failing, and it looks like the provider is down rather than the configuration being out of date.

What to do: Track provider API version requirements in a changelog and add a startup log line showing which API versions are configured for each provider, so a sudden failure can be quickly diagnosed as a catalog update needed.

Configuration values like which hardware device to use and what numeric precision to run at are passed as plain text strings. If there is a typo — even just wrong capitalization — the system silently falls back to a default instead of warning you. You might expect GPU acceleration but get CPU performance, or expect high-precision math but get lower-precision results, with nothing in the logs to explain why.

What to do: Add validation after deserializing the configuration that checks each string field against its known valid values and logs a clear error for unrecognized options before trying to use them.

All management of the loaded AI models goes through a single internal lock. While actual classification runs in parallel just fine, creating, listing, or removing a model instance blocks everything else trying to do the same thing at that moment. Under very high traffic this becomes a bottleneck that is hard to diagnose because normal request processing is unaffected.

What to do: Be aware that hot-reloading models during peak traffic will cause brief management stalls; plan model updates for low-traffic windows until a sharded registry is implemented.