How We Think About Hallucination in Legal AI

The word "hallucination" does a lot of quiet damage in legal AI, because it makes a family of very different failures sound like one problem with one fix. A tool "hallucinates," the story goes, so you buy the tool that "hallucinates less," and everyone moves on. That framing is comfortable and wrong. A fabricated case citation and a subtly overstated holding are both called hallucinations, but they fail for different reasons, at different moments, and they need different defenses. Treating them as one thing is how you end up with a green "verified" badge that catches the easy failure and waves the dangerous one straight through.

We think about it differently. Hallucination is not a single defect to be solved with a clever trick. It is a set of distinct failure modes, and the only durable answer is a layered system where each mode has an owner: a specific mechanism whose job is to catch that specific way of being wrong. This is the synthesis post. The deep dives it points to are already published, so this one stays above them: it names the taxonomy, maps each failure to the defense that owns it, and shows why the layers have to be arranged the way they are.

Why the stakes justify the engineering

Independent research is blunt about how common the problem is. The Stanford HAI and RegLab study of leading legal research tools (Magesh et al., 2024, summary) found hallucinations on the order of 17 to 34 percent of queries, more than 17 percent for Lexis+ AI and Ask Practical Law and more than 34 percent for Westlaw's AI-Assisted Research, in products marketed as built to prevent exactly that. These were not toy chatbots. They were purpose-built legal tools with retrieval and citation features, and they still put fabrications in front of lawyers a meaningful fraction of the time.

The consequence has a name. In Mata v. Avianca, a lawyer filed a brief citing cases that did not exist, a chatbot having produced and formatted them convincingly, and the court sanctioned the lawyers. That case is now the reference point every general counsel reaches for when they ask whether an AI tool can be trusted near a filing. The lesson is not "models make things up," because models will always make things up. The lesson is that nothing stood between the fabrication and the filing. Closing that gap is an engineering problem, and it does not have one solution: it has one solution per failure mode.

The taxonomy

Here is how we break the problem down, from the crudest and most catchable failure to the subtlest, and then the two that live one layer deeper than the answer text itself.

The fabricated citation. A marker pointing at a source that does not exist: an answer built on five retrieved sources that cites "[7]." This is the Mata failure in miniature, and it is the one class of fabrication a machine can catch perfectly, because you do not need judgment to know that a seventh source is not among five. You need to count.

The invented quotation. Exact words dressed as a verbatim block quote that appear in no retrieved source. This is more dangerous than a stray marker, because a block quote is a promise to the reader that these are the actual words, copied faithfully. Showing a lawyer language that was never in the record is a citation-grade fabrication even when no bracketed number is involved.

The unsupported claim. A plausible, well-formed statement that the cited source does not actually back: a "may" quietly promoted to a "must," a date shifted by a year, a holding overstated past what the passage supports. The citation is real, the sentence reads correctly, and the source simply does not say what the sentence claims.

The ungrounded synthesis. A fluent sentence with no source behind it that trips none of the usual wires: no quotation, no number, no citation marker for a matcher to grab. "The docket shows service irregularities" is the shape of it. This is the confident aside that rides along inside an otherwise sourced answer, indistinguishable in tone from the sentences around it.

The confident guess. An answer the model is not actually sure of, delivered in exactly the same calm, articulate register as one it is sure of. The surface confidence of a language model tells you almost nothing about reliability, because it is equally fluent when it is recalling and when it is improvising.

The retrieval-driven error. A grounded but wrong answer, faithfully built on the passages it was given, where the right passage was never retrieved in the first place. This one is not an invention at all. It is a research miss wearing the clothes of a good answer, and it is why retrieval quality sets the ceiling on everything above it.

Stale or overruled authority. A real case, correctly quoted, that is no longer good law. Every earlier defense can pass this one cleanly, because the quote is accurate and the source genuinely supports the claim, and the only thing wrong is that the authority has since been overruled, limited, or questioned.

Defense in depth

The organizing idea is that these failures happen at different moments in an answer's life, so the defenses have to sit at those different moments. There are four layers, and each owns a stage.

Grounding at retrieval time decides what the model is even allowed to reason over, because a wrong answer is very often a retrieval miss rather than an invention. Prevention at generation time catches the crude, mechanically detectable fabrications before the reader ever sees them. Detection at verification time reads the finished answer back against the exact sources and grades what actually holds. Honest uncertainty is the last layer: when the system cannot be sure, it says so by pulling the displayed confidence down rather than dressing a guess as a finding.

The shape of that diagram is the whole thesis. No single box is the answer. Each layer assumes the ones before it did their job and were not enough on their own, which is exactly what defense in depth means.

Mapping each failure to its owner

The value of the taxonomy is that every failure mode has a named mechanism accountable for it, and the mechanisms are deliberately different in kind: deterministic text operations that cannot themselves hallucinate, model-graded checks that read for meaning, and statistical signals about stability. The match between the failure and the tool matters more than the raw power of any one tool.

Prevention: the two failures a machine catches perfectly

The fabricated citation and the invented quotation are the crudest failures and, not coincidentally, the only two we can catch with certainty, so we catch them before display rather than grading them after. A deterministic gate counts the retrieved sources and walks every citation marker in the answer. A "[7]" against five sources is fabricated by definition, so it is removed, and a grouped "[2,7]" becomes "[2]," keeping the real citation and dropping only the invented index. The rule is strict in exactly one direction: only out of range indices are ever touched, so a valid citation is never second-guessed, which is what keeps lawyers trusting the citations that survive. The same gate takes every verbatim block quote, checks it against the combined source text, and if the quoted words appear nowhere in what was retrieved, it demotes the quote to ordinary prose so it loses the visual authority of an exact excerpt it did not earn. Neither check costs a model call, both run in milliseconds, and crucially neither can invent, which is the property you want in your last line of defense against invention. The full sequence, including the one narrow case where every citation is fabricated and the answer is re-generated within the valid range, is the subject of "Catching a Fabricated Citation Before It Reaches the Lawyer."

Detection: reading the answer back against the record

The unsupported claim and the ungrounded synthesis are fuzzy, so they need judgment, and that is the verifier's job. The verifier answers a deliberately narrow question: not "is this true in the world," but "does this answer faithfully represent the specific passages it was given." For legal work that is the more useful question, because a lawyer wants "this sentence is supported by that passage, and here is the passage," not "the model is fairly sure." It runs cheap deterministic matching first and spends a model only on the uncertain middle and on anything carrying a number, because numbers are where a lawyer gets burned. Two measurement choices do more for honesty than any model upgrade: the denominator excludes claims that had no source to check against, so "I could not verify this" is never scored as "the model lied," and analytical framing is separated from source-checkable fact so a thoughtful answer is not punished for thinking out loud. Sentence level groundedness is the coverage layer underneath claim checking: it scores every sentence for whether any source supports it, which is how the confident aside that trips no claim pattern still gets a number and gets flagged. Both mechanisms are grounded against the exact retrieved passages, and the design decisions behind them, including why corpus case law is only ever verified against what the model actually saw, are in "What Our Citation Verifier Actually Checks."

Honest uncertainty: measuring stability instead of trusting tone

The confident guess is the failure that defeats every check that reads the answer once, because the answer looks fine. So we stop trusting the model's tone and measure its stability directly. We resample the same question against the same sources several times and measure how much the concrete facts overlap across samples, because a model that keeps landing on the same specifics is recalling, and a model that invents different specifics each time is improvising. Then calibration starts from distrust: it discounts the raw confidence on the assumption that the unadjusted number runs hot, and it only earns the discount back when independent methods, the string matcher, the model verifier, the resampler, and the groundedness scorer, converge on the same verdict. The default is doubt, and confidence has to be corroborated to survive, which is the opposite of how a raw model score behaves. These signals cost real time and money, so they run on the deep tier and only inside the uncertain band where a second look can move the verdict, and the full reasoning is in "Knowing When You Do Not Know."

Grounding: the ceiling under everything

The retrieval-driven error is the one that humbles the whole stack, because none of the layers above can fix an answer that was faithfully built on the wrong passages. If the right subsection was never retrieved, a perfectly grounded, perfectly verified answer can still be wrong, which is why we treat retrieval quality as the ceiling on answer quality rather than a preprocessing step. A lawyer rarely asks what a document is about; a lawyer asks for a specific thing, and pure meaning-based search is structurally blind to exact identifiers, treating "Section 512(c)" and "Section 512(a)" as near twins. So every query runs as a hybrid search that fuses meaning-based retrieval with exact keyword matching, tilts toward keyword when the query names an identifier, filters rather than ranks when it can parse an exact section, and reranks the over-fetched candidates with a sharper second reader, which is "Hybrid Retrieval and the Citation to a Subsection Problem." On top of that default the retriever adapts per question, because one blend is not correct for a vague question, a broad question, a two-part question, and a clean lookup all at once, which is "One Retriever Is Not Enough." Getting retrieval right is not glamorous, and it is the single highest-leverage thing we do against hallucination, because it moves the ceiling everything else operates under.

The next capability

Stale or overruled authority is the failure mode we are building toward next, and it is worth being precise about why it is genuinely harder than the rest. Every defense above can pass a real, correctly quoted case whose holding the passage genuinely supports: the quote is accurate, the source backs the claim, and the only thing wrong is that the case was later overruled, limited, or questioned. "The quote is accurate" and "the authority is still good" are two different promises, and we currently keep the first with real rigor. Keeping the second, treatment and good law checking, is the next capability we are building on this foundation, precisely because the foundation, verified quotes and grounded claims, is what a treatment layer has to sit on top of. We would rather ship it as a promise we can actually keep than paint a badge over a check we have not built.

The through line

The reason we refuse the "hallucination, singular" framing is that it hides the most important design decision in the whole system, which is the match between a failure and its defense. A deterministic counter is the perfect tool for a fabricated marker and useless for an overstated holding. A model-graded read is the right tool for an overstated holding and the wrong tool for a fabricated marker, because a model asked to police invention can invent while doing it. A stability signal is the only tool that catches a confident guess, and it catches nothing that a single read would flag. Retrieval is the tool that prevents the error the other three cannot touch. You do not solve hallucination. You decompose it, assign each piece to the mechanism suited to it, and arrange the mechanisms so that the crude failures die early and cheaply and the subtle ones meet a check built for exactly them.

Test any legal AI yourself: one prompt per failure mode

You do not need our code to see whether a tool is defending in depth or decorating with a badge. You need seven prompts and fifteen minutes, one per failure mode, run in any legal-AI product including ours, watching the behavior rather than the confidence label.

1. Fabricated citation. Ask for three cases supporting a narrow proposition, each with its citation, then check every citation on a free source. A strong tool returns real cases or says it cannot find good authority; the tell is three tidy citations, at least one of which resolves to nothing.

2. Invented quotation. Upload a short document, ask a question, and copy anything presented as a verbatim quote back into the file to search for it character for character. A strong tool only block-quotes words that actually appear; the tell is a smooth quotable sentence in quotation marks that exists nowhere in the document.

3. Unsupported claim. Ask a question whose answer sits near, but not exactly on, what a real passage says, then compare the qualifier the tool used against the source: did a "may" become a "must," a "generally" become an "always." A strong tool tracks the qualifier; the tell is a confident tightening the source does not support.

4. Ungrounded synthesis. Ask something answerable, then read the answer for the extra sentence with no citation behind it. A tool scoring groundedness at the sentence level flags or hedges that sentence; the tell is one unsupported aside riding along, indistinguishable in tone from the rest.

5. Confident guess. Ask the same genuinely hard question three or four times in fresh sessions. A tool that measures stability gives you a steady answer or tells you the question is unsettled; the tell is three fluent answers quietly asserting different specifics, each delivered with equal certainty.

6. Retrieval-driven error. Ask for a specific subsection that has a near neighbor, when subsections (a), (b), and (c) all exist and say different things. A tool with real hybrid retrieval returns the exact provision you named; the tell is a confident answer about the wrong subsection, one identifier off.

7. Stale authority. Ask whether a real but later-treated case is still good law, and whether it has been overruled, limited, or questioned. A strong tool checks treatment or is explicit that it did not; the tell is a clean "yes, it is good law" with no basis, and a tool that admits the limit is being more honest than one that does not.

If a tool passes these, it is defending in depth. If it fails several and still shows a green "verified," you have learned the most important thing about it, which is that the badge and the behavior are not the same promise. A vendor who can tell you which mechanism owns each of these seven failures has thought about the problem as a system. A vendor who answers all seven with one word, "verified," is selling you the word, not the architecture.

Priyansh Khodiyar

Co-Founder & CTO

Priyansh leads engineering and AI at Vaquill, from the matter workbench to drafting, document comparison, document matrix, and citation-verified research.