Semantic Routing for LLMs and When It Beats Rules or Classifiers

A developer types “debug the refund endpoint in our django app” into your support assistant. Semantic routing is meant to prevent what happens next: the keyword rule sees “refund”, sends it to billing, and the developer gets your refund policy instead of help with a stack trace.

So the team swaps keywords for embeddings. At a cautious threshold most traffic falls through to the fallback, so someone loosens it to 0.55. In our replay the refund question then scores 0.683 for billing and lands in billing again. Paraphrases got easier, and near misses did not.

Semantic routing handles rewording well, but it has its own failure modes, and few guides say how to pick the threshold. Below you get the mechanics, the cases where it beats rules or classifiers, what two 2026 projects do with it, and a runnable router we replayed on labelled traffic.

TL;DR

• Semantic routing embeds each request, matches it against example utterances per route, and uses the closest route above a threshold, or a fallback.
• Keyword rules are cheapest and predictable, classifiers win once you have labels, and semantic routing wins when phrasing varies and labels are scarce.
• vLLM Semantic Router v0.4.0 (September 27, 2026) treats embedding similarity as one signal in a hybrid; LiteLLM’s Auto Router v2 (beta) makes it optional.
• In our 32-query replay, 0.80 sent 84% of traffic to the fallback; 0.55 reached 0.78 accuracy but let 4 of 7 fallback queries into real routes.
• After fixes: 0.94 on the 32 queries it was tuned on, 0.85 on 26 held-out ones. Trust the held-out number.

How Semantic Routing Works, From Embedding to Route

Semantic routing turns the incoming request into an embedding, scores it against a short list of example utterances for each route, and picks the route with the highest cosine similarity. If that best score clears a threshold, the request goes to the route’s model or handler; if not, it goes to a fallback.

A semantic router sits in front of models, tools or RAG indexes. Libraries combine utterance scores differently: our script takes the single best match, vLLM Semantic Router’s max aggregation leads with the best match and weighs close prototypes for text rules, while Aurelio’s semantic-router averages each route’s scores among the 5 nearest utterances by default.

For round-robin, latency and cost strategies, see our guide to LLM routing in general.

What a route holds

A route is a name, a target and a handful of example utterances. The target can be a model, a prompt, a tool or a handler. The utterances act as a tiny few-shot classifier: a request inherits the label of whichever example it sits closest to in embedding space.

Varied wording beats near-duplicates. Five versions of “refund my order” teach the router one point; five different money questions cover the region users write in. The vLLM Semantic Router docs warn that near-duplicate candidates add little recall and can make a rule look better calibrated than it is.

Similarity threshold and the fallback route

The threshold is the minimum best-match score a request needs before it may use a route. The fallback is the explicit “none of the above”, usually a general model or a clarifying reply. Without one, every out-of-scope question is forced into some route, and each wrong answer still looks confident.

Semantic Routing vs Keyword Rules, Classifiers and Complexity Scoring

Keyword rules match literal strings, trained classifiers predict a label from features learned on labelled data, complexity scorers grade how hard a prompt is, and semantic routing matches meaning against examples. They differ most on what a new route costs to add, how they handle paraphrases, how accurate they are, and how they fail.

Latency stays qualitative in the table, because it depends on your encoder, hardware and batch size. Per-request cost means extra model work before the main LLM call, not the price of that call. Treat the rows as a starting point for your own replay.

Where keyword rules still win

Keyword rules win on exact identifiers, compliance triggers, product SKUs and slash commands, where a literal match is the whole point. They are deterministic, cheap and easy to audit line by line. The vLLM Semantic Router v0.4 release post adds a caution for mixing rules with scores: a keyword hit is a policy gate, not 100% confidence.

Where a trained classifier wins

Once you hold hundreds of labelled examples per class, a fine-tuned encoder separates overlapping intents that similarity to five utterances cannot. It learns which words matter for each label instead of weighing them all. The price is the labelling and retraining loop whenever a route changes or a new intent appears.

LLM routing strategies that score complexity instead of topic

Complexity scoring picks a model tier by difficulty, not intent, using heuristics such as length, code presence and reasoning markers, or a small LLM grader. It pairs with semantic routing: one decides what the request is about, the other how much model it needs.

When Semantic Routing Beats Rules or Classifiers

Semantic routing beats keyword rules when users phrase the same intent many ways, and beats a trained classifier when you have a handful of examples per route instead of a labelled dataset. It loses when routes share vocabulary, when out-of-scope traffic looks like an existing route, or when difficulty matters more than topic.

The strongest case is paraphrase-heavy traffic such as support questions, FAQ lookups and workflow triggers, where “I was charged twice” and “you billed me two times” must land together. Adding a varied utterance takes seconds with no retraining, which also makes it a good cold-start router before labels exist.

Billing questions and code that implements billing share vocabulary and meaning, so they sit close in embedding space. Open-domain traffic lands near whichever route is most generic. And when the real question is how hard a prompt is rather than what it is about, topic matching is the wrong tool.

Both projects in the next section land on a hybrid design. Keyword rules catch exact identifiers, embeddings catch paraphrases, a classifier or scorer handles what similarity cannot separate, and a request that clears no threshold goes to the fallback model instead.

vLLM Semantic Router and LiteLLM Auto Router v2 in 2026

The vLLM Semantic Router v0.4.0, released on GitHub on September 27, 2026 and its latest tag as of November 11, treats embedding similarity as one signal among keywords and classifiers. LiteLLM’s Auto Router v2, in beta as of November 2026, did the same and deprecated its utterance-only router.

Readers choosing a product can start with our comparison of LLM routers and load balancers, since this post does not rank products.

Many tutorials use Aurelio Labs semantic-router, an MIT-licensed Python library whose latest release, as of November 11, 2026, is v1.1.2 from November 9.

vLLM Semantic Router v0.4.0

The Apache-2.0 project’s release post, named Hermes, walks through a routing path of Signal, Projection, Decision, Algorithm and Model, where rules combine signals with AND, OR and NOT. It also covers Vela 1.0, now 15 open models including embedding, PII and jailbreak models. The README since lists Vela 2.0 and Decision 2.0 and 3.0.

The v0.4 embedding signal docs set per-rule thresholds (0.75 and 0.72 in the example) over candidate utterances, with max aggregation that also weighs close prototypes for text rules. They say to calibrate on labelled positive and negative traffic, because thresholds do not transfer reliably between models.

The When to Reason paper, an October 2025 workshop paper (NeurIPS ML for Systems) on an earlier router version, reports a 10.2 percentage point MMLU-Pro gain with 47.1% lower latency and 48.5% fewer tokens than direct vLLM inference. Those numbers predate v0.4.0 and cover reasoning-mode routing only.

LiteLLM Auto Router v2 (beta)

LiteLLM’s Auto Routing, in beta since v1.94.0, puts one complexity router behind a choice of classifiers. The options are a seven-dimension heuristic scorer (the default), an LLM classifier, JEV through TypeSafe, keyword rules with optional semantic matching (an embedding model plus a match_threshold), or a custom plugin.

Build a Small Semantic Router in Python

The full script runs 253 lines across three passes, and its routing logic is 13 non-blank lines. It uses a local embedding model and NumPy, three routes with five utterances each, a 0.80 threshold, a fallback route, and a replay loop that scores 32 labelled queries, including near misses, per route.

The encoder is bge-small-en-v1.5, loaded through fastembed, so it runs locally with no API key.

Every vector is normalised, so one matrix multiply scores each query against each utterance, and route_all(threshold) takes the best match or falls back. Below is the first pass, the script’s first 138 lines; save it as semantic_router.py and run uv run --with fastembed==0.9.1 --with numpy==2.5.4 python semantic_router.py.

"""Minimal semantic router: local embeddings, 3 routes, threshold, fallback, replay.

Run: uv run --with fastembed --with numpy python semantic_router.py
"""
import numpy as np
from fastembed import TextEmbedding

MODEL = "BAAI/bge-small-en-v1.5"
THRESHOLD = 0.80  # first guess; the sweep below picks the value from labelled traffic
FALLBACK = "fallback"

ROUTES = {
    "billing": [
        "I was charged twice this month",
        "how do I update my credit card",
        "can I get a refund for my subscription",
        "why is my invoice higher than usual",
        "cancel my plan and stop billing me",
    ],
    "code_help": [
        "my python script throws a KeyError",
        "how do I write a SQL join across three tables",
        "fix this TypeError in my React component",
        "explain what this regex does",
        "my docker build fails on pip install",
    ],
    "small_talk": [
        "hi there, how are you",
        "thanks, that was helpful",
        "tell me a joke",
        "good morning",
        "what's your name",
    ],
}

# Labelled replay sample: (query, expected route). Near misses share words with
# a route but belong elsewhere (or to the fallback).
SAMPLE = [
    ("I got billed two times for the same order", "billing"),
    ("please refund last month's charge", "billing"),
    ("where can I download my invoices", "billing"),
    ("switch me to the annual plan", "billing"),
    ("my card was declined at checkout", "billing"),
    ("how much does the pro tier cost", "billing"),
    ("getting IndexError: list index out of range", "code_help"),
    ("how do I reverse a linked list in java", "code_help"),
    ("my unit tests pass locally but fail in CI", "code_help"),
    ("what does async await do in javascript", "code_help"),
    ("segfault when I free this pointer in C", "code_help"),
    ("convert this bash loop to python", "code_help"),
    ("hello!", "small_talk"),
    ("thank you so much", "small_talk"),
    ("how's your day going", "small_talk"),
    ("you're funny", "small_talk"),
    ("hey, good evening", "small_talk"),
    ("bye for now", "small_talk"),
    # near misses
    ("write a python function that calculates a monthly invoice total", "code_help"),
    ("my billing webhook handler returns 500 in node", "code_help"),
    ("debug the refund endpoint in our django app", "code_help"),
    ("tell me a joke about python programmers", "small_talk"),
    ("thanks, but I was still charged after cancelling", "billing"),
    ("hi, I need a refund", "billing"),
    ("how are you calculating the tax on my invoice", "billing"),
    ("explain how credit card processing works in general", FALLBACK),
    # out of scope, should fall back
    ("what is the capital of australia", FALLBACK),
    ("summarise this article about climate policy", FALLBACK),
    ("draft a polite email declining a meeting", FALLBACK),
    ("what's the weather in paris tomorrow", FALLBACK),
    ("recommend a good sci-fi novel", FALLBACK),
    ("translate good night into spanish", FALLBACK),
]


def unit(v):
    return v / np.linalg.norm(v, axis=-1, keepdims=True)


embedder = TextEmbedding(MODEL)
names, utter = [], []
for route, examples in ROUTES.items():
    names += [route] * len(examples)
    utter += examples
U = unit(np.array(list(embedder.embed(utter))))
Q = unit(np.array(list(embedder.embed([q for q, _ in SAMPLE]))))
sims = Q @ U.T  # cosine similarity, queries x utterances


def route_all(threshold):
    best = sims.argmax(axis=1)
    return [names[j] if sims[i, j] >= threshold else FALLBACK for i, j in enumerate(best)]


def report(threshold, verbose=False):
    pred = route_all(threshold)
    gold = [g for _, g in SAMPLE]
    if verbose:
        top = [names[j] for j in sims.argmax(axis=1)]
        for (q, g), p, s, t in zip(SAMPLE, pred, sims.max(axis=1), top):
            mark = "ok  " if p == g else "MISS"
            print(f"{mark} score={s:.3f} top={t:<10} gold={g:<10} pred={p:<10} {q}")
    per = {}
    for r in list(ROUTES) + [FALLBACK]:
        idx = [i for i, g in enumerate(gold) if g == r]
        per[r] = (sum(pred[i] == r for i in idx), len(idx))
    overall = sum(p == g for p, g in zip(pred, gold)) / len(gold)
    fb_rate = pred.count(FALLBACK) / len(pred)
    return per, overall, fb_rate


print(f"model={MODEL} routes={len(ROUTES)} utterances={len(utter)} sample={len(SAMPLE)}")


def show(t, verbose):
    per, overall, fb = report(t, verbose=verbose)
    print(f"threshold={t:.2f}")
    for r, (ok, n) in per.items():
        print(f"  {r:<10} accuracy {ok}/{n} = {ok / n:.2f}")
    print(f"  overall accuracy {overall:.2f} | fallback rate {fb:.2f}")


show(THRESHOLD, verbose=False)

print("\nthreshold sweep")
grid = [round(0.55 + 0.05 * k, 2) for k in range(7)]  # 0.55 .. 0.85
best_t, best_acc = None, -1.0
for t in grid:
    per, overall, fb = report(t)
    cells = " ".join(f"{r}={ok}/{n}" for r, (ok, n) in per.items())
    print(f"  t={t:.2f} overall={overall:.2f} fallback_rate={fb:.2f} {cells}")
    if overall > best_acc:
        best_t, best_acc = t, overall

print(f"\nbest threshold on this sample: {best_t:.2f}\n")
show(best_t, verbose=True)

assert len(SAMPLE) >= 30

model=BAAI/bge-small-en-v1.5 routes=3 utterances=15 sample=32
threshold=0.80
  billing    accuracy 3/9 = 0.33
  code_help  accuracy 0/9 = 0.00
  small_talk accuracy 2/7 = 0.29
  fallback   accuracy 7/7 = 1.00
  overall accuracy 0.38 | fallback rate 0.84

Those are the first lines of our November 10, 2026 run, and a November 11 re-run printed identical output. A threshold that sounds reasonable sent 84% of the traffic to the fallback. Nothing crashed or logged an error, because the fallback answered everything, which is why a router needs a replay before it ships.

The labelled replay sample

Near misses belong in the sample because that is where routers fail: queries that share words with one route but belong to another, and polite out-of-scope requests. Our 32 queries hold 18 clean cases, 8 near misses and 6 plainly out of scope. In production, pull near misses from logs where users rephrased a question.

Testing Semantic Routing Thresholds on Labelled Traffic

Test semantic routing by replaying labelled traffic at several thresholds and reading per-route accuracy next to the fallback rate. In our replay, moving from 0.80 to 0.55 lifted overall accuracy from 0.38 to 0.78 and cut the fallback rate from 0.84 to 0.09, but fallback accuracy fell from 7/7 to 3/7.

The refund query “debug the refund endpoint in our django app” went to billing at 0.683 in pass 1 at 0.55, and to code_help at 0.690 in pass 2. That row is the opening scenario, fixed by utterances about code for billing systems.

The script calls 0.55 the best threshold, but 0.55 is the lowest value on this grid, so extend the sweep lower before trusting it. Overall accuracy alone also picked a value that leaks out-of-scope traffic into real routes, as the fallback column shows.

Fallback rate counts out_of_scope predictions as fallback, so its rise from 0.09 to 0.19 in pass 2 is intended: 6 of 7 fallback queries now reach it. The held-out 0.35 sits above its gold share of 7 in 26 because 3 in-scope queries scored under their route thresholds.

Re-run the replay whenever the embedding model, an utterance or a threshold changes, since a threshold tuned for one setup means little in another. Answer quality, realised cost per route and shadow routes before a switch belong to the full router evaluation in our guide to evaluating LLM routing policies.

Near-miss checks before every change

Keep a fixed near-miss set, here 8 queries, that must pass before any utterance or threshold edit ships. Track it apart from the clean set, because a global gain can hide a regression. In pass 2, accuracy rose to 0.94 while a clean billing query, “how much does the pro tier cost”, broke and moved to code_help.

Near-Miss Queries, Threshold Drift and Overlapping Routes

Two failure modes explain all 7 misroutes at 0.55: near-miss queries that borrow another route’s vocabulary, and overlapping routes that sit too close in embedding space for one threshold. A third, threshold drift, shows up over time when the embedding model or the traffic changes.

The fixes follow from the misses: billing-aware code_help utterances, an out-of-scope route or a higher small_talk threshold, or a keyword gate for code markers. Pass 2 (script lines 145 to 158, below) applied the first two plus per-route thresholds. It left 2 misses: credit card processing (billing, 0.703) and the pro tier price question (code_help, 0.633).

ROUTES_V2 = {k: list(v) for k, v in ROUTES.items()}
ROUTES_V2["code_help"] += [
    "my stripe payment webhook throws an exception in flask",
    "unit test for the function that prorates subscription charges",
    "the checkout API endpoint returns a 500 error",
]
ROUTES_V2["out_of_scope"] = [
    "who won the world cup in 2018",
    "write a short poem about autumn",
    "what time is it in tokyo right now",
    "help me plan a trip to italy",
    "give me a recipe for banana bread",
]
THRESHOLDS_V2 = {"billing": 0.65, "code_help": 0.55, "small_talk": 0.70, "out_of_scope": 0.55}

Near-miss queries

Three coding questions about billing systems went to billing at 0.55: “my billing webhook handler returns 500 in node” at 0.660, “write a python function that calculates a monthly invoice total” at 0.672, and the Django refund question at 0.683. At 0.80 all three fell to the fallback instead.

Threshold drift

A threshold belongs to one embedding model and one traffic mix. Swap bge-small-en-v1.5 for another encoder and the score distribution shifts, so 0.55 may mean nothing there. Record the model, the utterances and the thresholds together as one versioned config.

Overlapping routes

Small_talk became the catch-all for polite out-of-scope requests. “Draft a polite email declining a meeting” scored 0.574, “what’s the weather in paris tomorrow” 0.640 and “translate good night into spanish” 0.668, all routed to small_talk. One global threshold cannot fix that without starving the other routes, which is why per-route thresholds exist.

Getting the Refund Bug to the Code Model

After billing-aware code_help utterances, an out-of-scope route and per-route thresholds, the Django refund question moved from billing at 0.683 to code_help at 0.690. Those changes were tuned on the same sample, so its 0.94 is in-sample. A held-out check on 26 hand-written queries scored 0.85, a better estimate; replay your own labelled traffic before switching.

The held-out misses show where the config still bends. “Do you offer a student discount” scored 0.645 for billing, just under its 0.65 threshold, and “how do I center a div with flexbox” (0.525) and “hiya” (0.638) also fell to the fallback. “How do credit scores work” went to billing at 0.666.

Export a week of requests, label a few hundred including near misses, sweep past both ends of your grid, and set per-route thresholds. Keep the near-miss set as a gate on every change, then shadow the new router against the old one before it takes live traffic.

When you are ready to place the router in your stack, see where the router sits in an AI gateway.