A lawyer does not usually ask a search system what a document is about. A lawyer asks for a specific thing. Quote subsection (c) of that section. Pull the indemnification clause, not the limitation-of-liability clause next to it. Find the definition of Confidential Information, the one with the capital letters, not the ordinary English phrase.
These are the queries that quietly break the kind of semantic search most legal-AI tools are built on. Semantic search is very good at meaning and structurally blind to exact identifiers. This post is about why that happens, and the specific machinery we use to make retrieval survive a citation to a particular subsection. There are two small code snippets, both simplified for readability, because the decisions here are easier to see than to describe.
Why pure semantic search misses the exact one
A dense embedding model turns a passage into a vector, a long list of numbers that encodes what the passage means. Two passages that mean similar things land near each other in that space, even when they share no words. That is the whole magic, and for legal research it is genuinely useful: a question about "an employee's duty not to compete after leaving" can find a clause that never says "non-compete."
But meaning is exactly the wrong lens for an identifier. To a dense model, "Section 512(c)" and "Section 512(a)" are nearly identical. They are the same statute, the same structure, one character apart, and they sit almost on top of each other in vector space. The model has no reason to prefer the subsection you actually asked for over the one next to it, because as meaning they are near-twins. The same failure hits a docket number, a defined term, and a pin cite. Ask for the exact provision and a purely semantic system will happily return a semantically-similar-but-wrong one, which in legal work is not a near miss. It is a wrong answer with a confident citation attached.
Keyword search is the exact-match rescue
The fix is old and boring and it works: keyword search, specifically BM25.
BM25 does not care what a passage means. It scores documents by the query terms they literally contain, weighted so that rare, discriminating terms count for more than common ones and so that a term appearing many times does not run away with the score. "512(c)" is a rare, discriminating token. BM25 will rank the passage that literally contains it above the passage that merely contains "512(a)," which is precisely the discrimination the dense model could not make. Keyword search is blind in the opposite direction from semantic search: it cannot find the non-compete clause that never says "non-compete."
So we do not choose. We run both and fuse them.
Fusing dense and keyword
Every query runs as a hybrid search: a dense vector for meaning and a sparse BM25 vector for exact terms, executed together against our vector store and combined with Reciprocal Rank Fusion. Reciprocal Rank Fusion is a merge that looks at each result's rank in each list rather than its raw score, which matters because dense scores and keyword scores are on different scales and cannot be added directly. A passage that ranks high on both meaning and exact terms rises to the top.
Our default gives roughly equal weight to the two. The reason is plain: a balanced split keeps semantic understanding without giving up exact legal-term matching. That default alone rescues most identifier queries, because the keyword half pulls the literally-correct passage up even when the semantic half is ambivalent about it.
Here is the whole path a query travels, from the words you typed to the bounded set of passages the model finally reads.
The weight tilt is not just a scoring knob; it changes how deep each half searches. The prefetch budget is a fixed pool split in proportion to the two weights, so a structured query does not merely score the keyword half higher, it also fetches more BM25 candidates and fewer dense ones before fusion ever runs. That means the half more likely to hold the literally-correct passage gets the wider net, not just the louder vote.
But a fixed split leaves value on the table. A natural-language question ("what standard of care governs the receiving party") wants the meaning half to lead. A reference query ("Section 512(c)") wants the keyword half to lead. So before we search, we look at the query itself.
Detecting an identifier and tilting the weights
We classify each query as structured or semantic. A structured query is one that names an exact legal reference: a section, an article, a rule, a statutory citation. A semantic query is ordinary natural language. The classifier is deterministic pattern matching, not a model call, because this decision runs on every query and needs to be free and predictable.
# Simplified from our detect_query_type() + weight selection.
if looks_like_a_citation(query): # names a section, article, rule, or reporter cite
dense_weight, sparse_weight = STRUCTURED # tilt toward keyword (BM25)
else:
dense_weight, sparse_weight = SEMANTIC # tilt toward meaning (dense)
In plain terms: when the query names an exact identifier, we shift the balance toward keyword matching, so the literal token does more of the work. When it reads as a natural question, we shift toward meaning. We do not paste our exact weight numbers here, but the shape is the point. The system asks "is this a lookup or a question?" and leans the retrieval accordingly, instead of applying one blend to two very different jobs.
When the identifier is exact, filter, do not rank
There is a stronger move than reweighting, and for exact identifiers it is the right one.
If a query truly pins down one provision, ranking is the wrong tool entirely. Ranking asks "which passages are most relevant," and relevance will always leave a little probability mass on the neighboring subsection. So when we can parse an exact identifier out of the query, we stop ranking on it and start filtering on it. Our statute passages carry structured metadata: the section number, the title number, the jurisdiction. When the query names a section, we hand the vector store a hard equality filter on that field, and only passages with that exact section number survive to be searched at all.
# Simplified from our statute filter builder.
conditions = []
if section_number:
conditions.append(exact_match("section_number", section_number)) # not similarity
if title_number is not None:
conditions.append(exact_match("title_number", title_number))
if jurisdiction:
conditions.append(exact_match("jurisdiction", jurisdiction)) # only the picked states
This is the difference between "find me passages that look like section 512" and "find me section 512." Semantic similarity, left to itself, will return section 237 when you asked for section 23, because as meaning they are close. An exact-match filter cannot make that mistake, because it is not reasoning about meaning at all. The reweighting handles the fuzzy middle, where a query mentions an identifier but is really a question about it. The hard filter handles the precise case, where the identifier is the query.
Reranking: a second, sharper reader
Fusion and filtering decide what to fetch. They do not, on their own, put the single best passage first, and "first" matters because the passages nearest the top get read most carefully by the model that writes the answer.
So we over-fetch and then re-sort. The hybrid search returns a wide net of candidates, deliberately more than we will use. Then a cross-encoder reranker reads each candidate against the actual query and scores it directly. A cross-encoder is slower and much more discerning than the first-pass vectors, because instead of comparing two pre-computed embeddings it looks at the query and the passage together and judges the fit. It is the difference between a first-pass sort and a careful second read. We keep the top of that re-sorted list, and a relevance gate drops candidates whose rerank score is too low, so a thin query returns a few strong passages rather than a padded list.
The reranker is where an identifier query gets its final polish. The correct subsection, once the filter and the keyword half have surfaced it, is what the cross-encoder confirms belongs at the top.
Contextual retrieval: so a lonely clause keeps its context
There is a subtler failure that neither fusion nor reranking fixes, and it is specific to how legal documents are written.
We split documents into chunks so we can retrieve the relevant part instead of a whole contract. But a clause pulled out of its document loses the thing that made it meaningful. "The Receiving Party shall protect Confidential Information" is a fine chunk and a useless one on its own. Which agreement? Who is the Receiving Party? Where was Confidential Information defined? The chunk has been amputated from the definitions and cross-references that give its capitalized terms meaning, and a defined term is exactly the kind of exact identifier this whole post is about.
So before we embed a chunk, we contextualize it. Following a documented contextual-retrieval technique, we generate a short situating note for each chunk and prepend it to the chunk's own text before it is embedded and indexed. The note records where the chunk sits in the document's hierarchy, which defined terms it uses and where they were defined, and what legal function it serves. Published research on this technique reports a large reduction in retrieval misses, and it is intuitive why: the embedding now encodes "this is the Receiving Party's confidentiality obligation under this NDA, and Confidential Information is defined in Article 1," not just a floating sentence. A search for the defined term now lands on the clause that actually uses it.
We reinforce the same idea at retrieval time by pulling a chunk's immediate neighbors along with it, so a provision arrives with the sentences that surround it rather than as a bare fragment.
Budgeting the final context
All of that produces a ranked, contextualized set of passages. The last step is deciding how many to actually hand the model.
More context is not free and not always better. So we enforce a budget: a fixed token ceiling in the tens of thousands and a hard cap on the number of passages, whichever binds first. The reranker's ordering is what makes this safe. Because the passages are already sorted best-first and gated for relevance, the budget trims from the bottom, dropping the weakest candidates rather than the ones that matter. The model receives a bounded, high-signal set instead of everything the search returned.
The shape, not the vendor
One deliberate design choice underneath all of this: every component sits behind a provider-agnostic interface, with automatic fallback to a secondary provider if the primary one fails. The embedding model, the reranker, and the vector store are all swappable. That is partly operational resilience and partly a bet that the retrieval architecture, the fusing and filtering and reranking and contextualizing, is what actually determines answer quality, more than which specific model is plugged into each slot. We tune the slots. We do not build the system around any one of them.
Test it yourself: the exact-identifier probe
You do not need our internals to see whether a tool has solved this. You need one query that hinges on an exact identifier, and one minute.
Pick a statute or a contract with clearly numbered subsections, and ask for a specific one that has a near neighbor. "What does subsection (c) require," when subsections (a), (b), and (c) all exist and say different things. Or, in a contract, "quote the definition of Confidential Information," when the document also uses the phrase in ordinary, unfrozen ways.
Then check what came back against the source. A tool with real hybrid retrieval returns the exact provision you named. The tell of a purely semantic one is subtle and dangerous: it returns a passage that is obviously about the right topic, formatted confidently, that is quietly the wrong subsection, or the ordinary-English use of a term instead of its defined meaning. It looks right. It cites something real. It is one identifier off.
That last failure, the confident near-twin, is the one that ends up in a filing. If a tool drifts to the semantically-similar-but-wrong provision on a query this simple, you have learned exactly how much to trust it on a hard one.
