We Ran 1,341 Prompts Through Our Router to Find Out What Misroutes

Every legal-AI product with a chat box has a classifier sitting in front of it, deciding what you meant before anything gets answered. Almost nobody publishes what happens when you push on that classifier at scale.

We built a benchmark that does. It expands 17 labeled prompt categories into exactly 1,341 concrete prompts, pushes every one of them through our document-task router, and counts a single, narrowly defined failure. It costs nothing to run, it calls no models, and it finishes in milliseconds. We run it before every rollout that touches routing.

Today's result: 1,341 prompts, 1,266 exact matches, and zero severe misroutes. The zero is the number that matters, and the rest of this post is what it means and how we get to keep it.

This post is about the benchmark itself: how it is constructed, why it is free, what it measures, and why the metric it measures deserves a name that nobody in this space has given it yet. The categories are legal because our product is legal. The shape is not. If you have an LLM classifier in front of anything, you can build this in an afternoon, and you probably should.

The failure this benchmark exists to catch

Start with the concrete thing, because abstractions about "routing accuracy" are how this problem stays invisible.

Here is a request a lawyer typed into our product with a document pinned:

"Rewrite the entire transcript in third person, this is not a summary but a complete recitation."

The interesting property of that sentence is that it contains its own negation. The user said what they wanted, then said in plain words which thing they did not want. A classifier that reads "transcript," reads the general shape of a whole-document operation, and reaches for the summarizer has executed a different job than the one it was given, flawlessly, at speed, with confidence.

That sentence is now prompt number one in the benchmark's rewrite category, verbatim, along with its sibling: "Put this in third person and keep every detail, do not summarize." That is the design principle underneath the whole corpus. A real prompt that exposed a real weakness in a classifier does not get patched and forgotten. It becomes a permanent, free, deterministic test.

Naming the metric: the severe false-whole-doc misroute

Overall routing accuracy is a comfortable number and a nearly useless one. It averages together mistakes that cost a user nothing with mistakes that make a user close the tab.

So the headline metric is not accuracy. It is the count of severe false-whole-doc misroutes: a prompt whose intent is targeted, meaning answer this specific thing inline, that gets hijacked into a whole-document engine such as summarize, extract, matrix, or review.

That is a distinct, nameable failure class, and naming it is the point. It has a direction (targeted intent going the wrong way, never the reverse). It has a severity argument (the user's literal instruction is discarded and replaced with a different task). And it is countable, which means you can drive it toward zero and know when you have.

Its mirror image gets tracked separately and treated as minor: a genuine whole-document request that falls back to a plain targeted answer. That is the recall side, and it is a lesser harm by construction. A user who asks "summarize this lease" and gets a good direct answer got something adjacent to what they wanted. A user who asks for a verbatim recitation and gets a summary got the opposite of what they asked for, and they said so in the prompt.

Those two mistakes are not the same size, so the benchmark refuses to average them into one number. That single decision is most of what makes the metric useful.

How 17 categories become 1,341 prompts

The corpus is generated, not hand-listed. Hand-listing a thousand prompts produces a thousand prompts a human found interesting, which is exactly the set the classifier's authors already thought about.

Instead, each category carries a small set of templates with a {doc} slot, and each template is fanned across 25 document nouns: document, transcript, contract, agreement, NDA, lease, memo, brief, motion, report, deposition, policy, filing, letter, settlement agreement, employment contract, term sheet, complaint, affidavit, will, trust deed, invoice, MSA, SOW, waiver.

So "Rewrite this {doc} in third person" is not one test. It is 25, one of which is "Rewrite this will in third person" and another of which is "Rewrite this SOW in third person," and the classifier has to hold the same route across all of them. That fan-out is where a lot of the value comes from, because a rule that quietly keys off "contract" and misses "term sheet" is invisible in a hand-written suite and obvious in a fanned one.

Categories can also carry extra prompts, which are literal one-offs that stay exactly as written. That is where real incident prompts live, unfanned, in their original words.

Here is the whole pipeline, from one category to the number we watch.

The expected route on each category is set from first principles, deliberately. It is what the route should be, argued from what a lawyer typing that sentence wants, not a snapshot of what the code currently returns. That distinction is easy to get wrong and it matters enormously. A benchmark whose labels were generated by recording the current behavior can only ever tell you that your system still does what it used to do. It can never tell you that what it used to do was the wrong thing.

The category families, and which ones are severe

Ten of the 17 categories are marked severe. Those are the ones where a targeted intent getting hijacked is a betrayal of the instruction rather than a slightly imperfect answer.

Read that table as two halves. The top ten are the adversarial half, aimed straight at the ways a plausible-looking classifier gets a plainly worded instruction wrong. The bottom six are the recall half, which exist so a router cannot game the severe metric by routing literally everything to targeted. A router that never fires a whole-document engine scores a perfect zero on severe misroutes and is completely useless, and the recall categories are what make that cheating visible.

Notice the weighting is intentional and not uniform. rewrite and targeted_keyword carry 152 prompts each, the largest blocks in the corpus, because those are the two families where a classifier is most likely to grab a surface keyword and run with it. Benchmarks should be lopsided in the direction of the danger.

Notice also what targeted_keyword includes as literals: "What is the standard for summary judgment" and "Explain the business judgment rule." Neither is a request to summarize anything. Both contain the exact tokens that a keyword-matcher would love to claim. The word "summary" inside a term of art is not an instruction, and a router that cannot tell the difference will confidently summarize a document in response to a question about a legal standard.

The prompts that actually make this hard

A router that only saw clean requests would be easy. The value is in the ones that look like one route and mean another. These are real prompts from the set, verbatim.

The first trap is a task word sitting inside an ordinary question. Every one of these has to stay targeted, and a naive keyword rule sends them to a whole-document engine instead:

• "What does the summary section of this contract say" (contains "summary," but it is a question about a section)
• "What is the standard for summary judgment" ("summary judgment" is a legal term, not a request to summarize)
• "List the risks in this MSA as bullet points" ("list" is output shape, not a task)
• "Is there an indemnity clause in this agreement" (an ordinary lookup that happens to name a clause)

The second trap is an explicit negation the router has to honor:

• "This is not a summary, just tell me the parties in the NDA"
• "Do not summarize the contract, quote its notice provision"
• "Don't review the whole lease, only answer about the payment term"

And the original incident that started the whole benchmark, unfanned and word for word:

• "Rewrite the entire transcript in third person, this is not a summary but a complete recitation"
• "Put this in third person and keep every detail, do not summarize"

On the other side sit the legitimate whole-document requests that must still route to their engine. They are the recall test, and they are the reason the metric cannot be gamed:

• "Give me an executive summary of this brief" routes to summarize
• "Build a chronology of the deposition" routes to chronology
• "List every duty and deadline in this MSA" routes to obligations
• "Is this policy compliant with GDPR" routes to compliance
• "Is this NDA safe to sign" routes to triage
• "Redline this contract and flag risky clauses" routes to full_review

That last group is the safeguard against a cheat. A router that simply refused to ever pick a whole-document engine would post zero severe misroutes and then fail every recall category on this page. You have to get both halves right at once.

Why it is deterministic and free, and why that changes everything

The reason this benchmark can be 1,341 prompts instead of 30 is that it does not call a language model.

The document-task decision happens before retrieval and before generation. At the bottom of our router is a pure classifier: pattern work, no model call, no network, deterministic by construction. The benchmark drives exactly that layer. Every one of the 1,341 prompts resolves in milliseconds, the whole sweep costs nothing, and running it twice produces byte-identical output.

Free and deterministic is not a minor operational detail. It is the reason the benchmark is any good.

A suite that costs real money per run gets run before releases, if you are disciplined, and quarterly if you are honest. A suite that costs nothing gets run in the inner loop: change a guard, rerun in seconds, watch the severe count, change it again. That converts routing from something you reason about into something you measure while you type.

Determinism is what makes the number a gate rather than a vibe. If a benchmark produces a different result on identical inputs, a change in the number is ambiguous forever, and an ambiguous number cannot block a release. 1,341 prompts through a deterministic classifier gives you a result you can put a hard threshold on, because any movement in it was caused by your diff and nothing else. That is also why a small team can afford to be rigorous here: we spend milliseconds to learn whether the router still honors "do not summarize," so we spend them constantly.

The layer above, and the discipline that makes fixes permanent

Our shipped router does not rely on a language model alone to make this decision, which is precisely because of what benchmarks like this one reveal about classifiers that do.

There is a model in our router, and it is genuinely good at the ambiguous middle, the sentences where "go through this" could reasonably be a review or a summary. But it sits between two deterministic layers. Below it is the pattern classifier, which is the prior and the fallback. Above it are guards the model is not permitted to overrule: a rewrite, reformat, or verbatim-recitation request, or an explicit negation, short-circuits to the targeted path before the model is ever consulted. The full design is a separate story, and I wrote it up in "Sending 'Rewrite This Transcript, NOT a Summary' to the Right Engine".

What matters here is the loop connecting the two. Alongside the free sweep there is a deliberately tiny probe, about ten curated content-generation prompts, run through the full model-backed resolver at the cost of roughly one model call each. Its only job is to catch a case where the model tries to override a correct deterministic prior.

And then the rule that makes the whole system compound:

Every confirmed misroute becomes a permanent deterministic guard.

Not a prompt tweak. Not a note in a tracker. A rule in the router that settles that case before the model is consulted, plus a prompt in the free corpus that would fail the instant it regressed.

The economics of that rule are what make it worth adopting. A misroute found by the expensive probe gets fixed by the cheap layer, and from that moment it is caught for free, forever, by a sweep that costs nothing to run. The fragile, expensive judgment gets replaced by a permanent one exactly once per failure class. You do not solve the same problem twice, which is the only sustainable strategy when your surface is large and your team is not.

Today's number

A benchmark post should contain the benchmark's output, so here it is, run against the router as it stands on the day of writing.

ROUTING STRESS (deterministic): 1341 prompts | pass=1266 (94%) | SEVERE false-whole-doc=0 | minor false-targeted=48 | other=27

The headline is the zero. Across 1,341 prompts, not one targeted instruction was hijacked into a whole-document action. That is the number we gate on, because it is the one that maps to a user being ignored.

The 75 non-matches deserve precision, because every one of them falls in the safe direction.

Forty-eight are whole-document intent that resolved to targeted instead. "Review and redline this contract" gets answered inline rather than opening a full review pass. The harness classifies that as minor by design, because the failure is conservative: the user gets an answer to the thing they asked about instead of having a large, opinionated action taken on their document without being asked. Falling back to the smaller action is the direction you want a router to fail in.

Twenty-seven are adjacent-family mismatches. "List every duty and deadline in this document" routes to the general extraction engine rather than the narrower obligations one. Both extract, and the user gets their list.

One honest note on how to read those two numbers. The category labels in this corpus encode what should happen by principle, not what the code currently does. A non-match is therefore a gap between a principle and current behavior, and some of those gaps are deliberate: the whole-document routes are held to a higher confidence bar precisely so they do not fire on an ambiguous phrase. The severe count is the one that is unambiguously a defect, and it is zero.

The corpus counts (1,341 prompts, 17 categories, 25 document nouns, 10 severe families) are properties of the code and do not move unless someone edits the corpus. The pass and misroute counts are a property of the router on the day you run it, which is why the harness prints them fresh on every run rather than baking a number into a README that goes stale the moment someone edits a guard. A benchmark's job is to produce today's number, not to memorialize a flattering one.

Build this for your own router

The categories in our corpus are legal. The shape is completely portable, and if you have a classifier deciding which engine handles a user's request, here is the whole recipe.

1. Write your categories as templates with a slot, and fan the slot across your domain's nouns. Generated beats hand-listed, because hand-listed only contains the cases you already thought of.

2. Label each category with the route it should take, argued from first principles. If you label from current behavior, you have built a change detector, not a benchmark.

3. Pick the one direction of error that actually burns a user, and name it. For us it is the severe false-whole-doc misroute. For you it is something else, but it is one thing, and it goes on its own line.

4. Include the reverse direction as recall categories, so a router cannot score perfectly by refusing to route at all.

5. Test the deterministic layer, not the model. That is what makes 1,341 prompts cost nothing and finish in milliseconds, and it is what lets you run the thing in your inner loop.

6. Put every real incident into the corpus verbatim, and turn every confirmed misroute into a permanent rule.

That is the entire artifact. It took less effort to build than one afternoon of manually clicking through routing cases, and unlike that afternoon, it runs again for free every time anyone touches the router.

What to ask any vendor about how they test routing

Buyers ask about model quality and accuracy percentages. Almost nobody asks about the classifier that decides what job gets done in the first place, which is where a precise instruction quietly becomes a different task.

Four questions, and they are worth more than any benchmark number a vendor volunteers.

1. Do you test your router separately from your generation, and how many prompts do you run through it? Routing and answering fail in completely different ways. A team that only tests them together is testing neither cleanly, and "our end-to-end evals look good" is not an answer to this question.

2. What is your headline routing metric, and which direction of error does it count? If the answer is "accuracy," ask what it averages. A team that has thought about this can name the one failure that enrages users and tell you they report it on its own.

3. Is your routing test deterministic and free, or does it cost money per run? This tells you how often it actually runs. A suite that costs money runs before releases at best. A free one runs while an engineer is typing, which is the only place it prevents anything.

4. When you find a misroute, does it become a permanent test, or do you patch a prompt? This is the one that matters most. A prompt tweak fixes today and silently reopens on the next model update. A deterministic guard plus a regression prompt fixes forever.

A vendor who answers all four concretely has built the discipline. A vendor who cannot tell you whether a routing mistake they already fixed can come back is telling you it will, and that you will be the one who finds it.

Priyansh Khodiyar

Co-Founder & CTO

Priyansh leads engineering and AI at Vaquill, from the matter workbench to drafting, document comparison, document matrix, and citation-verified research.