Hybrid Search Over 20 Million Legal Passages: Why Semantic Alone Fails on Law

Pure semantic search fails on legal text because a large share of legal queries aren't semantic at all. They are symbol lookups, defined terms, and conditions, and an embedding flattens all three. The fix isn't a better embedding model. It is an exact-identifier fast path in front, keyword and vector retrieval fused in the middle, a cross-encoder rerank at the end, and the caller given control over the trade-off.

This post walks through how retrieval over a large legal corpus is designed, using a corpus of more than 5.3 million sections of US primary law, indexed as more than 20 million passages, as the working example. Every choice below prevents a specific failure.

Reviewed and corrected on 12 October 2026. This is general information, not legal advice.

TL;DR

• Three failure modes break vector-only search on law: exact identifiers, terms of art whose legal meaning diverges from the everyday one, and conditions that flip meaning without moving the vector. Keyword-only search fails the mirror case, where a plain-words question shares no tokens with the statute.
• A citation fast path is what makes rank-1 exact. A query shaped like a citation resolves straight to that section at rank 1, with relevanceScore: 1.0. With no parse there is no shortcut, and a symbolic string has to compete as prose, which embeddings handle badly.
• Hybrid retrieval fuses two rankings by position, not by score. A cosine similarity and a keyword relevance score live on different scales, so reciprocal rank fusion combines ranks instead of averaging numbers.
• Chunking is why the passage count is several times the section count. Retrieval runs at passage level while a citation lookup returns one whole section, so chunking never leaks into what a user reads.
• An unknown filter value returns 422, not an empty list. A filter that silently matches nothing produces a confident zero, which downstream reads as "this law does not exist."

This one is part of our legal data infrastructure series.

For related coverage, see Legal Citation Resolution: Turning a Cite Into the Right Section, Every Time and Parsing Legal Citations in Code: Bluebook Forms, State Variants, and the Ones You Must Refuse, which together cover the fast path in step 1, and 128 Fields on One Statute: What Rich Legal Metadata Actually Buys You for the filter fields.

Why pure semantic search fails on law

Vector search is good at one job: finding text similar in meaning to other text. Legal retrieval keeps handing it other jobs.

Failure 1: an identifier is a symbol, not a meaning

Ask a vector index for 1950.5 and look at what you are really asking: the nearest neighbors of a short numeric token in a space trained to capture meaning. There is no meaning in 1950.5, only identity.

That embedding lands somewhere generic, near other numbers and text about quantities. It doesn't land near California Civil Code section 1950.5, whose text is about residential security deposits and never says "1950.5" in a way that dominates its embedding. Same for a docket number.

A user pastes a cite, gets a plausible neighbor, and stops trusting the tool.

Failure 2: terms of art collide with their everyday senses

Legal English reuses ordinary words with meanings that aren't the ordinary ones. A model trained mostly on general text learns the ordinary sense, because that is what its training data holds.

Search "what counts as consideration" and the nearest neighbors skew toward text about deliberation, with the contract-law provisions below them. The model is answering in the space it has.

Keyword matching has no such problem. "Consideration" is one string in both senses, and keyword matching cares about tokens, not senses.

Failure 3: negation and conditions barely move the vector

This one is the quietest and does the most damage. Statutory text is dense with conditions that change what a provision does while barely changing where it sits in embedding space. Two passages:

• "A person who violates this section shall be liable for a civil penalty."
• "Except as provided in subsection (c), a person who violates this section shall be liable for a civil penalty."

Near-identical to a cosine similarity, not at all near-identical to a lawyer. The second says the rule you just read isn't the whole rule. Other phrases carry the same weight: "notwithstanding any other provision of law", "this section shall not apply to", "unless the Secretary determines".

Embeddings compress, and compression drops the low-frequency, high-consequence tokens that legal drafting uses to flip a rule. More dimensions don't fix it. Similarity is the wrong operator for a conditional.

The mirror failure: keyword search alone loses on ordinary language

Now flip it. Someone asks: "how long does a landlord have to return my deposit."

The statute that answers it may say "lessor" instead of "landlord", "shall furnish" instead of "return", and won't say "how long" at all. Token overlap is close to zero, so keyword search returns noise.

So neither works alone. Vector search handles the question and fails the symbol. Keyword search does the reverse. Legal traffic mixes both, through one endpoint, with no reliable way to tell in advance which is which.

The actual pipeline

Here is the path a query takes, in order.

Step 1: the citation fast path

A query shaped like a citation is resolved to that exact section and returned at rank 1 with relevanceScore: 1.0. That is where rank-1 exactness comes from. It isn't the retriever being good at citations. It is the retriever being bypassed. What the resolver does with a parsed string is a post of its own.

The corollary matters more than the feature. When a citation does not parse, there is no shortcut, and the query falls through to the general path, where a symbolic string is scored as prose by models that have almost nothing to work with. A system with no fast path lives in that fallback for every citation query.

Two consequences follow:

• The parser's coverage is the feature. Every citation form a jurisdiction publishes has to be one the parser reads. A corpus that publishes a citation string it can't read back will hand a caller a citation and then fail on that exact input.
• Refusing to parse is sometimes correct. Some bare forms name no jurisdiction. In our 12 October 2026 tests Code § 8.01-243 (the form Virginia prints) and RS 14:30 (a Louisiana form) both came back resolved: false with missReason: ambiguous_jurisdiction, and the hint was to write the code out in full or pass state. The honest answer there is resolved: false, because a guess would state another state's law under the caller's citation. The forms that must be refused are catalogued in Bluebook forms, state variants, and the ones you must refuse.

Step 2: hybrid retrieval

Two retrieval legs run in parallel over the same corpus: a semantic leg that matches on meaning, and a keyword leg that matches on the terms themselves. The two rankings are then fused by reciprocal rank, which the search documentation describes.

The semantic leg catches meaning, like the deposit question. The keyword leg catches exact strings: terms of art, defined phrases, identifiers. Running both means a query never has to declare which kind it is.

Step 3: cross-encoder rerank

Fusion produces a good candidate set that isn't ordered well enough for a lawyer. Those candidates are then reranked by a cross-encoder that reads each candidate against your query, rather than comparing two separately computed vectors.

Reciprocal rank fusion, in plain English

You have two lists of results, one from each side, and you need one list.

The obvious idea is to add the scores. It is broken. A cosine similarity runs from about 0 to 1, while a keyword score such as BM25 has no upper bound and shifts with the corpus and the query length. Add them and the keyword score swamps the cosine number every time, so the "hybrid" is keyword search with extra steps.

Reciprocal rank fusion throws the scores away and keeps the positions. For each list, a document at position r earns 1 / (k + r), where k is a small constant that stops the top result dominating. Add up what a document earns across both lists and sort by the total.

A worked example with k = 60, the value most often used in the literature. It comes from the 2009 paper that introduced the method, where the authors say k = 60 "was fixed during a pilot investigation and not altered during subsequent validation" (Cormack, Clarke and Büttcher, SIGIR 2009). Their pilot runs found it "near-optimal" and the choice "not critical." Implementations pick their own, so treat the number as illustrative and the shape of the result as the point:

Document B wins even though it topped neither list. Agreement between two independent methods beats a high score from one. Document A was the vector index's favorite while the keyword index barely liked it, the classic shape of a semantic near-miss. Document C is a pure vector hit with no keyword support, and ranks last.

RRF also needs no per-collection score calibration. It has one constant.

What the research says, and what builders report

The claim that keyword and vector search each fail alone isn't only ours. A few outside sources are worth knowing.

The benchmark papers. BEIR, a 2021 benchmark that tested ten retrieval systems on 18 datasets, reported that BM25 is a strong baseline and that "re-ranking and late-interaction-based models on average achieve the best zero-shot performances, however, at high computational costs." It added that dense and sparse retrieval models "often underperform other approaches" when they meet data unlike what they were trained on (BEIR, Thakur et al., arXiv 2104.08663). Plain keyword matching is a strong baseline, and that's the part people forget.

Legal text got its own retrieval benchmark in 2024. LegalBench-RAG describes itself as "the first benchmark specifically designed to evaluate the retrieval step of RAG pipelines within the legal space." It has 6,858 query-answer pairs over a corpus of more than 79 million characters, annotated by legal experts, and it rewards pulling back "minimal, highly relevant text segments" rather than whole documents, partly because "precise results allow LLMs to generate citations for the end user." (LegalBench-RAG, Pipitone and Alami, arXiv 2408.10343) That is the case for retrieving at passage level and still returning the whole section for a citation.

What builders say. In November 2025 a developer who said they were building RAG on Lucene reported that embeddings added "only like +10% boost compared to BM25 alone (on average)" on their evaluation sets. A proper reranker plus query rewriting, they said, moved retrieval from 65% to over 80%. Their conclusion: embeddings are best treated as "a lightweight filtering/screening tool to quickly find a pool of candidates as a first stage." (kgeist, Hacker News, November 2025) Another builder in March 2024 found the opposite for keyword-style queries: "the lexical search was adding nothing; the embeddings alone produced almost identical results," and added that "The re-ranker, however, made a big difference." (anon373839, Hacker News, March 2024)

So the evidence is mixed, and that is useful. Neither was working with legal text, and both say the reranker earns its keep. Whether the keyword side adds anything depends on your queries. Law is a case where it tends to, because so many queries are section numbers and defined terms, which is the whole argument of the failure modes above. Run the four query sets below on your own traffic before you decide.

Chunking, and why two counts differ

The corpus holds more than 5.3 million distinct sections, indexed as more than 20 million retrieval passages. Two different units, both numbers real.

A section is one citable item with a stable actId. A passage is a unit of retrieval. The search documentation says sections are split into overlapping passages, each carrying a header with its corpus, jurisdiction, status, hierarchy and section title, which is why the passage count runs several times the section count. Long documents give a retriever the same problem: no single embedding represents a whole preamble, comment discussion and rule text usefully.

The trade-off cuts both ways:

There is no universal right answer, so anyone quoting a chunk size as a best practice is guessing. What is defensible is the boundary rule: split at the structure the publisher already uses, because a statute was chunked by whoever drafted it.

What matters for an API consumer: retrieval happens at passage level, but a citation lookup still resolves to one section with its full text. GET /us/statutes/section/{actId}/body returns the whole section text rather than the passage that matched.

matchType: the trade-off, handed to the caller

Most retrieval systems pick one point on the precision/recall curve and hide it. POST /us/statutes/search exposes three, because the right point depends on the query and only the caller knows what it is for.

A natural-language question. Use any.

{
  "query": "how long does a landlord have to return a security deposit",
  "corpusType": "STATE",
  "state": "ca",
  "matchType": "any",
  "limit": 10
}

The semantic leg does the work. Nothing in the question matches the operative text on tokens. On 12 October 2026 the top hits were Cal. Civ. Code § 1950.7 and § 1950.5, the second of which is the deposit-return section.

A defined term you must not have paraphrased. Use all.

{
  "query": "covered financial institution beneficial ownership",
  "corpusType": ["CFR", "AGENCY_GUIDANCE"],
  "matchType": "all",
  "limit": 25
}

A near-synonym is a wrong answer here. "Financial institution" and "covered financial institution" are different sets, and the difference is the question.

A statutory phrase. Use phrase.

{
  "query": "arbitrary, capricious, an abuse of discretion",
  "corpusType": "USC",
  "matchType": "phrase",
  "limit": 20
}

That is the standard-of-review language from 5 U.S.C. 706, which was the first hit on 12 October 2026, followed by 2 U.S.C. § 1406 and 31 U.S.C. § 755, which borrow the formula. You aren't after provisions about arbitrariness. You want every place that exact string appears, so you can see where Congress borrowed the formula.

Rule of thumb: start at any, move to all when results are topically right and legally wrong, move to phrase when you know the words and are hunting for occurrences. Going straight to phrase on a question you phrased yourself is the fastest way to get zero results and conclude the corpus is thin. All three run in the playground.

Filters are the other half of precision

Ranking decides the order. Filters decide what is eligible at all. Pre-filtering runs before ranking, so it shrinks the candidate set the ranker has to be right about.

Available on POST /us/statutes/search: corpusType, state, code, titleNumber, chapter, part, source, agency, documentType, yearFrom, yearTo, publishedFrom, publishedTo.

"Beneficial ownership" across 20 million passages is a hard ranking problem. The same query with corpusType=CFR and titleNumber=31 is easy, because the wrong answers are gone before the reranker sees them. Filters are the cheapest precision available.

The behavior that matters more than the list: an unknown filter value is rejected with 422. It does not silently match nothing.

Consider what a silent no-match does. You send state=calfornia, a typo, instead of state=ca. A permissive API filters to an empty set and returns [] with a 200. Your code sees a successful call with zero results and reports, correctly as far as it can tell, that no matching law exists.

A confident empty result is indistinguishable from an answer. Zero results reads downstream as "the law does not exist." An agent will say that out loud, and a compliance report will show a clean row. A 422 listing the valid codes is loud and fixable in a minute. An empty 200 is a bug you find months later in a customer's output.

How to evaluate any legal retrieval system in an afternoon

You need no eval framework and no labeled dataset. Four query sets and a spreadsheet. Suggested sizes in brackets.

Set 1: exact citations [20 queries]. Use citations you already know, in the forms your users paste. Mix federal (42 U.S.C. § 1983) and state. Include a trailing period, since 42 U.S.C. § 1983. is what copying from a brief produces. Include a doubled section symbol (§§). Include one pasted straight out of a PDF, because PDFs carry dash characters that look like a hyphen and aren't (U+2010, U+2011, U+2012, U+2015, U+2212, U+FF0D). Score: right section at rank 1, not top 5.

Set 2: terms of art [20 queries]. Use words whose legal sense diverges from the everyday one: consideration, sanction, service, instrument. Score: of the top 5 results, how many use the legal sense.

Set 3: natural-language questions [20 queries]. Phrase them as a non-lawyer would, sharing as few tokens with the statute as you can. This set exposes a keyword system pretending to be semantic. Score: is a correct answer in the top 5.

Set 4: queries whose right answer is nothing [10 queries]. Everyone skips this set, and it is the only one that measures honesty. Build it from a section number that doesn't exist in a title that does, bare forms that name no jurisdiction (RS 14:30, Code § 8.01-243), a jurisdiction the vendor doesn't cover, and a body of law they told you they don't hold.

Score: nothing, an explicit refusal, or a confident wrong answer? A system that never returns zero results is not comprehensive. It is incapable of saying no. Only set 4 finds that.

Run all four against every vendor on your list, including the incumbent. The results usually reorder it.

What to design around

• limit is capped at 50 and offset at 70, so the ranked candidate pool is at most 120 results deep (offset 70 plus limit 50). If you need to survey more than that, run narrower queries with a tighter filter.
• Paging is cut from one ranking, under the pagination rules. Results never repeat between pages and never go missing, and a later page costs no more than the first. Check this in any vendor by diffing the actId sets on pages 1 and 2.
• Earlier text is available through asOf=YYYY-MM-DD on the section body endpoint. One citation resolves to the current text by default. With asOf, a corpus where we hold the publisher's printed editions (the CFR annual editions, for one) returns the edition that governs your date, and a date that falls between held editions comes back unavailable rather than being answered from a neighbouring edition. The other corpora rebuild the earlier text from the changes we recorded, flagging isBounded when the answer rests on when capture began, and a section we know changed but cannot rebuild comes back unavailable. There is also amendment history and a last-amended year per section, a yearFrom/yearTo currency filter, and change events with per-section diffs on watched sources. Amendment History and Point-in-Time Law sets out how far to trust each.

Legal retrieval is a mixed workload. A large share of queries are lookups rather than searches, and treating everything as a search produces a system that is quietly wrong on the queries users trust most. Vaquill AI's US primary law API is built around that split, the grounding guide covers handing its output to a model, and GET /us/statutes/coverage is free, so you can check what sits behind it first.

FAQ

Is hybrid search always better than pure semantic search?

For legal text, yes, because legal queries include identifiers and defined terms that embeddings handle badly. On a purely conversational workload with no citations and no terms of art, the keyword side contributes less. Running both means one more ranking to fuse.

What is reciprocal rank fusion and why not just add the scores?

RRF combines two ranked lists by position rather than by score. Each list gives a document 1 / (k + rank) points, and the totals are summed. Scores can't be added directly because a cosine similarity and a keyword score sit on different scales, so a raw sum lets the larger-scaled number decide every result.

Why do you need a reranker if hybrid retrieval already ranks results?

Retrieval compares separately computed representations, which is fast over millions of passages but coarse. A cross-encoder reads query and passage together, which is far more accurate and far too slow to run corpus-wide. Retrieve wide and cheap, then rerank narrow and expensive.

Why is the passage count several times the section count?

Because sections are split into several overlapping passages. More than 5.3 million sections are indexed as more than 20 million passages. Retrieval works on passages, while a citation lookup returns one whole section.

What happens if I send a filter value that does not exist?

The request is rejected with a 422, and for the source filter the error lists every valid code. An empty 200 would be indistinguishable from a genuine finding of no matching law, a far more expensive failure than a rejected request.

Can I retrieve a statute as it read on a specific past date?

Yes, through asOf=YYYY-MM-DD on the section body endpoint. The response says which engine answered: a held published edition for corpora such as the CFR annual editions (a date between held editions comes back unavailable), or a reconstruction from recorded changes for the rest, where isBounded flags an answer resting on when change capture began and a section we know changed but cannot rebuild comes back unavailable. A citation with no date maps to the text in force now. Around it sit amendment history per section, a yearFrom/yearTo filter working on the last amendment year the publisher credits, change events captured at each refresh, and per-section diffs on watched sources.

Why would a legal search API ever refuse to resolve a citation?

Because some bare forms name no jurisdiction. In our 12 October 2026 tests Code § 8.01-243 and RS 14:30 both returned resolved: false with the reason ambiguous_jurisdiction. Guessing would state one state's law under a citation meaning another, and confidently wrong is worse than resolved: false.

How do I test whether a legal retrieval vendor is actually good?

Run four query sets: exact citations scored at rank 1, terms of art scored on how many top-5 results use the legal sense, natural-language questions scored on whether a correct answer lands in the top 5, and queries whose correct answer is nothing. Almost nobody runs that last set, and it is the one that shows whether a system can say no.

Priyansh Khodiyar

Co-Founder & CTO

Priyansh leads engineering and AI at Vaquill AI: the pipelines that pull statutes, regulations and court rules from every US jurisdiction's official publisher, and the REST API, MCP server and open dataset that serve them.