In 2023 a lawyer filed a brief in the Southern District of New York that cited a string of cases (Mata v. Avianca, No. 1:22-cv-01461 (S.D.N.Y.)). The cases did not exist. A chatbot had produced them, formatted them convincingly, and the lawyer had passed them along. The court in Mata v. Avianca sanctioned the lawyers, and the episode became the reference point every general counsel now quotes when they ask whether an AI tool can be trusted around a filing.
The lesson people took from Mata was "check the citations." The sharper lesson is about timing. By the time a fabricated citation is on screen, formatted like every real one next to it, the damage is already one copy-paste away. The honest place to catch an invented citation is before it is ever shown, not after a human notices.
Our previous post walked through the verifier that scores an answer after it is generated: it grades how faithfully the answer represents its sources and reports a number. This post is about a different mechanism that runs at a different moment. It is a deterministic gate that sits between the model finishing its answer and the reader seeing it, and its only job is to make sure a fabricated citation never reaches the page in the first place. A scorer tells you an answer was bad. A gate stops the bad part from being displayed.
There are two small code snippets below, simplified for readability, because the design decisions here are visible in a few lines and worth seeing rather than trusting.
The one kind of fabrication a machine can catch perfectly
Most hallucinations are fuzzy. "May" quietly becomes "must," a date shifts by a year, a holding gets subtly overstated. Those need judgment to catch, which is why the after-the-fact verifier spends a model on them.
But one class of citation fabrication is not fuzzy at all. When an answer draws on a handful of retrieved sources, those sources are numbered. If the model retrieved five sources and then writes "[7]," there is no seventh source. That marker points at nothing. It is fabricated by definition, and you do not need a model, a probability, or a judgment call to know it. You need to count.
This is the part of the problem where a deterministic check beats a language model outright, and it is worth being precise about why. A regex is fast: this pass runs in well under a millisecond for the quick check and single-digit milliseconds for the full one, with zero model cost, so it can run on every answer without anyone weighing latency against safety. A regex is also incapable of the very failure we are guarding against. A model asked to police citations can hallucinate while doing it. Text-matching code cannot invent a source that is not in the list it was handed. When the whole point is to catch invention, you do not want your last line of defense to be something that can itself invent.
Step one: strip the out-of-range marker, and nothing else
The core check counts the retrieved sources and walks every citation marker in the answer.
A marker like [7] when only five sources exist is removed before display.
A grouped marker is cleaned surgically: [2,7] with five sources becomes [2], keeping the real citation and dropping only the fabricated index.
The rule we hold ourselves to is strict in exactly one direction. Only out-of-range indices are ever removed. An in-range citation is never touched, never rewritten, never second-guessed. This is a deliberate zero-false-positive design, and the direction matters. A gate that occasionally strips a valid citation would train lawyers to distrust the citations that survive, which is the opposite of the point. So the gate is allowed to be certain or to do nothing. It is never allowed to guess.
Simplified, the decision for a single marker looks like this:
# Simplified from strip_fabricated_citations().
valid = [n for n in nums if 1 <= n <= source_count]
invalid = [n for n in nums if n < 1 or n > source_count]
if not invalid:
return marker # every index real: leave it exactly as written
removed.extend(invalid)
if valid:
return "[" + ",".join(map(str, valid)) + "]" # keep the real ones
return "" # all fabricated: drop the marker
In plain terms: if every number in the marker is a real source, the gate does not touch it. If some are real and some are not, it keeps the real ones and drops the rest. Only if every number is fabricated does the whole marker disappear.
The unglamorous details are where a gate like this earns or loses trust.
When a marker is dropped, its leading space goes with it, so removing [7] does not leave a double space behind.
When a citation stood alone inside parentheses, the empty () it would have left is cleaned up too.
And when a marker sits directly against the next word, the space is preserved so "the tenant [9]must" becomes "the tenant must" and never "tenantmust."
Critically, the cleanup stays local to each removed marker.
We deliberately do not run a document-wide whitespace pass, because that would collapse intentional formatting elsewhere, like the indentation inside a code block or the spacing in a table.
The gate fixes exactly what it broke and leaves the rest of the document alone.
Step two: demote a quotation that was never in the record
The second fabrication is more dangerous because it looks more authoritative. A model can present invented or paraphrased language as a verbatim block quote, formatted as an exact excerpt lifted from a source. To a reader, a block quote is a promise: these are the actual words, copied faithfully. Showing a lawyer exact words that were never in any retrieved source is a citation-grade fabrication, even when no bracketed number is involved.
So the gate takes every verbatim block quote in the answer and checks it against the combined text of the actual sources. If the quoted language is genuinely present, whether as a clean substring or a close match allowing for minor formatting differences, the quote stands. If it appears nowhere in what was retrieved, the gate demotes it: it strips the block-quote formatting so the text renders as ordinary prose instead of masquerading as an exact excerpt. The claim can remain in the answer, but it loses the visual authority of a direct quotation it did not earn.
Two guardrails keep this from over-firing. Callout boxes, the "note" and "warning" style blocks a model uses for its own commentary, are never treated as quotations and are left untouched. And very short fragments are skipped entirely, because a handful of common words is too little to adjudicate and too easy to match by accident. Like the marker check, this runs with no model call at all.
Step three, only when the whole answer ignored its sources
There is a worse case than a single stray marker. Sometimes every citation in an answer is fabricated: the model wrote a fluent, confident response and cited sources that all point past the end of the real list. That is not a typo. It is a signal that the model ignored the material it was given and answered from its own memory instead.
For that specific case, the gate does one more thing before it starts deleting. It counts the markers and the fabricated markers, and only when every citation in the answer is fabricated does it trigger a single, stricter re-answer. The model is told plainly how many real sources it has, that it must cite only within that range, and that any claim it cannot support should be stated as unsupported rather than dressed in a fake citation. That corrected answer then goes through the same deterministic stripping as everything else.
This trigger is deliberately narrow. An earlier, always-on regeneration step was retired precisely because it fired on nearly every complex query and cost latency for little gain. The rule now is that a full re-answer only happens in the total-failure case, where the alternative is showing the reader an answer whose every citation is invented. Everywhere short of that, the cheaper deterministic passes do the work.
Where the gate sits, and why that order matters
The order of operations is part of the design. Before the gate runs, a separate grounding step re-maps citation numbers, fixing the markers it can legitimately fix by matching the answer's claims back to the right source. The gate runs after that. It acts only on the citations grounding could not rescue, so the last thing standing between a fabricated marker and the reader is the check that cannot itself hallucinate.
The full sequence, from the model finishing to the reader seeing anything, looks like this.
Each stage narrows what the next one has to handle. Grounding fixes the fixable, so the total-failure re-answer sees a genuinely hopeless case rather than a fixable one. The re-answer, when it fires, produces text that flows back through the same deterministic stripping as everything else, so a corrective pass never becomes a way to smuggle a fresh out-of-range marker past the floor. The stripping and the blockquote demotion cost nothing and always run. Only the middle stage, the re-answer, spends a model call, and only in the one case where every citation is invented.
When any of these passes changes the answer, the corrected text is what gets displayed and what gets saved. On the live chat surface, the cleaned answer is re-emitted so the reader sees the corrected version, not the raw one. The same deterministic passes also run on the non-chat surfaces that share this pipeline, so an answer delivered over email or a connected channel gets the identical floor, and any correction there flips the answer's status to unverified so the reader sees that a citation was adjusted rather than a silent edit.
None of this is where our citation safety ends. The after-the-fact verifier still scores what remains, and it still catches the fuzzy errors a text match cannot. The gate is the floor, not the ceiling: it guarantees that the crudest and most dangerous fabrication, a citation or quotation pointing at something that was never retrieved, does not reach a lawyer's screen intact. A model reads and reasons and sometimes invents. The last check before the reader should be one that reads, counts, and cannot.
Test it yourself: three prompts that expose a fabricated citation
You do not need our code to see whether a tool guards the display or only decorates it. You need a few prompts and a couple of minutes. Run these in any legal-AI product, including ours, and watch the output itself, not the confidence badge next to it.
-
The out-of-range marker. Ask a question that forces the tool to cite retrieved sources, then read every bracketed number in the answer. Count the sources it says it used, and check that no citation points past that count. The tell of a weak tool: a "[6]" in an answer built on four sources, sitting there formatted exactly like the real ones.
-
The quotation that was never there. Upload a short document, ask a question, and look at anything the tool presents as a verbatim quote or block quote. Copy that quoted text and search for it, character for character, in your document. A strong tool only block-quotes words that actually appear. The tell: a smooth, quotable sentence in quotation marks that exists nowhere in the file.
-
The whole-cloth answer. Ask about something just outside what the tool could have retrieved, a narrow proposition it is unlikely to have real sources for. A strong tool tells you it lacks support, or answers with citations you can verify. The tell: a fluent answer with several tidy citations, none of which resolve to anything real.
If a tool passes these, its citations are load-bearing. If it fails and still shows a green "verified," you have learned the most important thing about it, which is that the badge and the behavior are not the same promise. Mata v. Avianca did not happen because a model made things up. Models make things up. It happened because nothing stood between the fabrication and the filing. That gap is the thing worth closing, and it is worth closing before the answer is ever shown, not after someone gets sanctioned for trusting it.
