Hidden Assumptions in semantic-router

Whether caching a given query type pays off depends on fixed example numbers for how slow the model is and how fast the cache search is. If your models or infrastructure are faster or slower than the paper's examples, the point at which caching helps versus hurts moves, and a category the paper calls worthwhile might not be for you.

What to do: Recompute the break-even hit rate using your own measured model and cache-search latencies rather than assuming the paper's example figures apply.

All the per-category safety of this cache depends on each query being labeled with the right category. The cleanest options assume the client or endpoint already tells you the category; if you rely instead on an automatic classifier, misclassification silently applies the wrong freshness and matching rules.

What to do: Ensure query categories are assigned reliably — prefer explicit or endpoint-based labels — and watch for misclassifications that would apply the wrong staleness and threshold rules.

When the router starts up, it tries to load the AI models it needs to screen requests. If those model files aren't already sitting in the right folders on the same machine, nothing warns you — the router starts fine, happily accepts traffic, but quietly skips all the safety checks (PII detection, intent matching, security scanning) and just routes everything using its fallback rule. You could be running a 'secured' router with no active security for hours without knowing.

What to do: Before going live, verify that all three model folders exist and each contains the expected weight files and a config file; a startup health-check that tests each classifier with a short sample sentence would surface this immediately.

The system reads a list of category names from a settings file and assumes they line up perfectly with what the AI model learned. If that file ever gets edited, regenerated, or replaced with a version that has categories in a different order or with gaps, the model's answers get silently relabeled — it thinks it said 'booking' but the router records 'cancel' and routes accordingly. Everything looks normal in the logs; the wrong decisions just quietly accumulate.

What to do: After loading the model, do a one-time sanity check by running a handful of known test phrases and confirming the returned labels match what you expect; catching a mislabeled model takes seconds and prevents routing everything to the wrong destination.

When the system finds sensitive information like an email address or ID number in a message, it records where in the message it found it using position numbers. Those position numbers count in the AI model's internal units, not in ordinary text characters — and for many languages or punctuation patterns, they don't match. Any feature that uses those positions to actually hide or mark the sensitive text may cut in the wrong place, leaving part of the sensitive data exposed while appearing to have done its job.

What to do: Check whether anything downstream actually uses the position numbers from PII results to redact text; if so, add a conversion step that maps token positions back to character positions before using them, and verify with a test that includes multi-byte characters and punctuation.

If you open a routing rule in the dashboard that was saved by a different version of the software, and the dashboard can't understand its format, it quietly shows you an empty canvas instead of an error. If you then hit save on that blank canvas, you overwrite your actual routing rules with nothing — and all traffic starts going to the default fallback without any warning.

What to do: Add a visible error message when a saved policy cannot be loaded into the editor, and require an explicit confirmation before allowing a save that would replace a non-empty stored policy with an empty one.

The three AI screening models — for intent, privacy, and security — are loaded together as a group. If any one of them fails (say the security model file is corrupted or the wrong size), none of the three will work. You lose all screening, not just security screening, and everything gets routed by your fallback rule with no explanation beyond a startup error.

What to do: Consider logging a specific message for each model that fails to load, and decide whether to allow partial operation with the remaining models or fail loudly at startup so the problem is immediately visible.

Every AI model loaded by the router stays in memory permanently until the process restarts. If you run experiments or load multiple model variants through the dashboard, each one takes a permanent chunk of RAM or GPU memory. On a smaller machine this eventually causes the system to run out of memory and crash, with no warning beforehand that you were approaching the limit.

What to do: Before running evaluation experiments that load multiple model variants, check available system memory against the size of each model file, and restart the router process if memory grows unexpectedly large.

The minimum confidence score required before the system acts on a classification result is set once when the system starts and never updated, even if you change it through the dashboard. So if you lower the threshold hoping to catch more cases, the AI layer keeps using the old, higher cutoff — and the dashboard shows you results calculated with the new threshold while the actual routing uses the old one.

What to do: Document clearly where the confidence threshold is set and whether it can be changed without a restart; if it cannot, add a note in the dashboard wherever the threshold is configured.

The settings for each AI provider — their web addresses, required headers, API version strings — are read once when the router starts and never refreshed. If a provider like Anthropic or OpenAI changes something on their end (even just bumping a required version header), every request to that provider starts failing. Because the failure looks like a network error, it is easy to spend time debugging the wrong thing before realizing the catalog just needs a refresh.

What to do: When a specific provider suddenly starts failing while others work, check whether that provider has recently announced API changes, and restart the router after updating the relevant provider file.

If you tell the router to use a GPU but the machine doesn't have one (or doesn't have the right software drivers installed), the router crashes when it tries to load the AI models. The opposite is also true: if you forget to set this, the router quietly uses the CPU, which is much slower, and there is no warning that a GPU is sitting idle.

What to do: Confirm the device setting matches what is actually available on your server before starting; on cloud deployments, check that GPU drivers are installed if you intend to use GPU acceleration.

When many requests arrive at the same time, all of them compete for the same pool of worker threads that run the AI classification. There is no limit on how many can pile up waiting, and nothing tells the rest of the system that it is overloaded. During a traffic spike, AI screening for all requests can slow to a crawl at the same time, causing a wave of timeouts that looks like an outage.

What to do: During load testing, observe how classification latency behaves under sustained concurrent traffic and set a reasonable limit on how many simultaneous classification requests the system accepts.