Hybrid Search Over 12 Million Legal Passages: Why Semantic Alone Fails on Law

Pure semantic search fails on legal text because a large share of legal queries are not semantic at all. They are symbol lookups, defined terms, and conditions, and an embedding flattens all three. The fix is not a better embedding model. It is an exact-identifier fast path in front, keyword and vector retrieval fused in the middle, a cross-encoder rerank at the end, and the caller given control over the trade-off.

This post walks the retrieval path we run over 4,150,839 sections of US primary law, indexed as 12,003,716 passages. Every choice below prevents a specific failure.

TL;DR

• Three failure modes break vector-only search on law: exact identifiers, terms of art whose legal meaning diverges from the everyday one, and conditions that flip meaning without moving the vector. Keyword-only search fails the mirror case, where a plain-words question shares no tokens with the statute.
• A citation fast path is what makes rank-1 exact. A parsed citation resolves to one section and is spliced in at relevanceScore: 1.0 before ranking. With no parse there is no splice, and a symbolic string scored as prose ranks close to randomly.
• Hybrid retrieval fuses two rankings by position, not by score. A cosine similarity and a BM25 score live on different scales, so reciprocal rank fusion combines ranks instead of averaging numbers.
• Chunking is why the passage count is roughly three times the section count. Retrieval runs at passage level while a citation lookup returns one whole section, so chunking never leaks into what a user reads.
• An unknown filter value returns 422, not an empty list. A filter that silently matches nothing produces a confident zero, which downstream reads as "this law does not exist."

This one is part of our legal data infrastructure series.

For related coverage, see Legal Citation Resolution: Turning a Cite Into the Right Section, Every Time and Parsing Legal Citations in Code: Bluebook Forms, State Variants, and the Ones You Must Refuse, which together cover the fast path in step 1, and 128 Fields on One Statute: What Rich Legal Metadata Actually Buys You for the filter fields.

Why pure semantic search fails on law

Vector search is good at one job: finding text similar in meaning to other text. Legal retrieval keeps handing it other jobs.

Failure 1: an identifier is a symbol, not a meaning

Ask a vector index for 1950.5 and look at what you are really asking: the nearest neighbors of a short numeric token in a space trained to capture meaning. There is no meaning in 1950.5, only identity.

That embedding lands somewhere generic, near other numbers and text about quantities. It does not land near California Civil Code section 1950.5, whose text is about residential security deposits and never says "1950.5" in a way that dominates its embedding. Same for a docket number.

This is the most common complaint about legal AI tools. A user pastes a cite, gets a plausible neighbor, and stops trusting the tool.

Failure 2: terms of art collide with their everyday senses

Legal English reuses ordinary words with meanings that are not the ordinary ones. A model trained mostly on general text learns the ordinary sense, because that is what its training data holds.

Search "what counts as consideration" and the nearest neighbors skew toward text about deliberation, with the contract-law provisions below them. The model is answering in the space it has.

Keyword matching has no such problem. "Consideration" is one string in both senses, and BM25 cares about tokens, not senses.

Failure 3: negation and conditions barely move the vector

This one is the quietest and does the most damage. Statutory text is dense with conditions that change what a provision does while barely changing where it sits in embedding space. Two passages:

• "A person who violates this section shall be liable for a civil penalty."
• "Except as provided in subsection (c), a person who violates this section shall be liable for a civil penalty."

Near-identical to a cosine similarity, not at all near-identical to a lawyer. The second says the rule you just read is not the whole rule. Other phrases carry the same weight: "notwithstanding any other provision of law", "this section shall not apply to", "unless the Secretary determines".

Embeddings compress, and compression drops the low-frequency, high-consequence tokens that legal drafting uses to flip a rule. More dimensions do not fix it. Similarity is the wrong operator for a conditional.

The mirror failure: BM25 alone loses on ordinary language

Now flip it. Someone asks: "how long does a landlord have to return my deposit."

The statute that answers it may say "lessor" instead of "landlord", "shall furnish" instead of "return", and will not say "how long" at all. Token overlap is close to zero, so BM25 returns noise.

So neither works alone. Vector search handles the question and fails the symbol. Keyword search does the reverse. Legal traffic mixes both, through one endpoint, with no reliable way to tell in advance which is which.

The actual pipeline

Here is the path a query takes, in order.

Step 1: the citation fast path

Before retrieval, the query is tested against a citation parser. If it parses, the resolver looks up that exact section and splices the result in at relevanceScore: 1.0, before ranking begins. That is where rank-1 exactness comes from. It is not the retriever being good at citations. It is the retriever being skipped. What the resolver does with a parsed string is a post of its own.

The corollary matters more than the feature. When a citation does not parse, there is no splice, and the query falls through to the general path, where a symbolic string is scored as prose by models that have almost nothing to work with. A system with no fast path lives in that fallback for every citation query.

Two consequences follow:

• The parser's coverage is the feature. Every citation form a jurisdiction publishes has to be one the parser reads. A corpus that publishes a citation string it cannot read back will hand a caller a citation and then fail on that exact input.
• Refusing to parse is sometimes correct. Several acronyms are claimed by two states, and Virginia and Connecticut print bare forms that name no jurisdiction at all. The honest answer there is resolved: false, because a guess would state another state's law under the caller's citation. The forms that must be refused are catalogued in Bluebook forms, state variants, and the ones you must refuse.

Step 2: hybrid retrieval

Every passage carries two vectors: a dense embedding vector and a sparse BM25 vector. Both are queried in one request and fused server-side by the vector store using reciprocal rank fusion.

Dense catches meaning, like the deposit question. Sparse catches exact strings: terms of art, defined phrases, identifiers. Running both means a query never has to declare which kind it is.

Step 3: cross-encoder rerank

Fusion produces a good candidate set that is not ordered well enough for a lawyer. Those candidates are rescored by a cross-encoder, which reads the query and the passage together rather than comparing two separately computed vectors.

The part worth copying is how the passage is prepared. The text handed to the reranker is built around the matching region rather than the first N characters, then capped to fit the model's context window. Truncate from the start instead and you routinely cut the matched language out before the model sees it, so the reranker scores a passage that no longer contains the reason it was retrieved.

Reciprocal rank fusion, in plain English

You have two lists of results, one from each side, and you need one list.

The obvious idea is to add the scores. It is broken. A cosine similarity runs from about 0 to 1, while a BM25 score has no upper bound and shifts with the corpus and the query length. Add them and BM25 swamps the cosine number every time, so the "hybrid" is keyword search with extra steps.

Reciprocal rank fusion throws the scores away and keeps the positions. For each list, a document at position r earns 1 / (k + r), where k is a small constant that stops the top result dominating. Add up what a document earns across both lists and sort by the total.

A worked example with k = 60, the value most often used in the literature. Implementations pick their own, so treat the number as illustrative and the shape of the result as the point:

Document B wins even though it topped neither list. Agreement between two independent methods beats a high score from one. Document A was the vector index's favorite while the keyword index barely liked it, the classic shape of a semantic near-miss. Document C is a pure vector hit with no keyword support, and ranks last.

RRF also needs no per-corpus tuning. One constant, and it does not change when you add a state.

Chunking, and why two counts differ

The corpus holds 4,150,839 distinct sections, indexed as 12,003,716 retrieval passages. Two different units, both numbers real.

A section is one citable item with a stable actId. A passage is a unit of retrieval. Short sections are one passage each, and long documents split into several, which is why the passage count runs about three times the section count. Federal Register agency rules are the extreme case: one rule carries a long preamble, a response-to-comments discussion, and the regulatory text, and no single embedding represents all of that usefully.

The trade-off cuts both ways:

There is no universal right answer, so anyone quoting a chunk size as a best practice is guessing. What is defensible is the boundary rule: split at the structure the publisher already uses, because a statute was chunked by whoever drafted it.

What matters for an API consumer: retrieval happens at passage level, but a citation lookup still resolves to one section with its full text. GET /us/statutes/section/{actId} returns the section, not the passage that matched.

matchType: the trade-off, handed to the caller

Most retrieval systems pick one point on the precision/recall curve and hide it. POST /us/statutes/search exposes three, because the right point depends on the query and only the caller knows what it is for.

A natural-language question. Use any.

{
  "query": "how long does a landlord have to return a security deposit",
  "corpusType": "STATE",
  "state": "ca",
  "matchType": "any",
  "limit": 10
}

The dense side does the work. Nothing in the question matches the operative text on tokens.

A defined term you must not have paraphrased. Use all.

{
  "query": "covered financial institution beneficial ownership",
  "corpusType": ["CFR", "AGENCY_GUIDANCE"],
  "matchType": "all",
  "limit": 25
}

A near-synonym is a wrong answer here. "Financial institution" and "covered financial institution" are different sets, and the difference is the question.

A statutory phrase. Use phrase.

{
  "query": "arbitrary, capricious, an abuse of discretion",
  "corpusType": "USC",
  "matchType": "phrase",
  "limit": 20
}

That is the standard-of-review language from 5 U.S.C. 706. You are not after provisions about arbitrariness. You want every place that exact string appears, so you can see where Congress borrowed the formula.

Rule of thumb: start at any, move to all when results are topically right and legally wrong, move to phrase when you know the words and are hunting for occurrences. Going straight to phrase on a question you phrased yourself is the fastest way to get zero results and conclude the corpus is thin. All three run in the playground.

Filters are the other half of precision

Ranking decides the order. Filters decide what is eligible at all. Pre-filtering runs before ranking, so it shrinks the candidate set the ranker has to be right about.

Available on POST /us/statutes/search: corpusType, state, code, titleNumber, chapter, part, source, agency, documentType, yearFrom, yearTo, publishedFrom, publishedTo.

"Beneficial ownership" across 12 million passages is a hard ranking problem. The same query with corpusType=CFR and titleNumber=31 is easy, because the wrong answers are gone before the reranker sees them. Filters are the cheapest precision available.

The behavior that matters more than the list: an unknown filter value is rejected with 422. It does not silently match nothing.

Consider what a silent no-match does. You send state=california instead of state=ca. A permissive API filters to an empty set and returns [] with a 200. Your code sees a successful call with zero results and reports, correctly as far as it can tell, that no matching law exists.

A confident empty result is indistinguishable from an answer. Zero results reads downstream as "the law does not exist." An agent will say that out loud, and a compliance report will show a clean row. A 422 listing the valid codes is loud and fixable in a minute. An empty 200 is a bug you find months later in a customer's output.

How to evaluate any legal retrieval system in an afternoon

You need no eval framework and no labeled dataset. Four query sets and a spreadsheet. Suggested sizes in brackets.

Set 1: exact citations [20 queries]. Use citations you already know, in the forms your users paste. Mix federal (42 U.S.C. § 1983) and state. Include a trailing period, since 42 U.S.C. § 1983. is what copying from a brief produces. Include a doubled section symbol (§§). Include one pasted straight out of a PDF, because PDFs carry dash characters that look like a hyphen and are not (U+2010, U+2011, U+2012, U+2015, U+2212, U+FF0D). Score: right section at rank 1, not top 5.

Set 2: terms of art [20 queries]. Use words whose legal sense diverges from the everyday one: consideration, sanction, service, instrument. Score: of the top 5 results, how many use the legal sense.

Set 3: natural-language questions [20 queries]. Phrase them as a non-lawyer would, sharing as few tokens with the statute as you can. This set exposes a keyword system pretending to be semantic. Score: is a correct answer in the top 5.

Set 4: queries whose right answer is nothing [10 queries]. Everyone skips this set, and it is the only one that measures honesty. Build it from a section number that does not exist in a title that does, an ambiguous abbreviation claimed by two states (MCA, IC, GS, GL, a bare RS), a jurisdiction-free form (Code § 8.01-243), a jurisdiction the vendor does not cover, and a body of law they told you they do not hold.

Score: nothing, an explicit refusal, or a confident wrong answer? A system that never returns zero results is not comprehensive. It is incapable of saying no. Only set 4 finds that.

Run all four against every vendor on your list, including the incumbent. The results usually reorder it.

What to design around

• limit is capped at 50 and offset at 70, so the deepest reachable result is bounded. Past that depth, ranking quality has decayed enough that page 40 would be selling noise. If you need more, use a tighter filter.
• Paging is cut from one ranking, under the pagination rules. Results never repeat between pages and never go missing, and a later page costs no more than the first. Check this in any vendor by diffing the actId sets on pages 1 and 2.
• Embeddings are cached, keyed by content hash, so refresh cost tracks actual change rather than corpus size.
• The vector store is one that treats server-side fusion as first-class, not a managed search product with a query DSL bolted on. Hybrid then runs in one round trip rather than two calls and a merge in app code.
• Versioning here is the amendment record, not a date parameter. One citation resolves to one stored text, the current one. There is amendment history and a last-amended year per section, a yearFrom/yearTo currency filter, and change events with per-section diffs on watched sources. Versioning runs off the amendment record. Reconstructing the text as it stood on an arbitrary past date is a separate problem, and Amendment History and Point-in-Time Law sets out why.

Legal retrieval is a mixed workload. A large share of queries are lookups rather than searches, and treating everything as a search produces a system that is quietly wrong on the queries users trust most. Vaquill AI's US primary law API is built around that split, the grounding guide covers handing its output to a model, and GET /us/statutes/coverage is free, so you can check what sits behind it first.

FAQ

Is hybrid search always better than pure semantic search?

For legal text, yes, because legal queries include identifiers and defined terms that embeddings handle badly. On a purely conversational workload with no citations and no terms of art, the sparse side contributes less. Running both costs one extra vector per passage.

What is reciprocal rank fusion and why not just add the scores?

RRF combines two ranked lists by position rather than by score. Each list gives a document 1 / (k + rank) points, and the totals are summed. Scores cannot be added directly because a cosine similarity and a BM25 score sit on different scales, so a raw sum lets the larger-scaled number decide every result.

Why do you need a reranker if hybrid retrieval already ranks results?

Retrieval compares separately computed representations, which is fast over millions of passages but coarse. A cross-encoder reads query and passage together, which is far more accurate and far too slow to run corpus-wide. Retrieve wide and cheap, then rerank narrow and expensive.

Why is the passage count three times the section count?

Because long documents split into several passages. 4,150,839 sections are indexed as 12,003,716 passages, with Federal Register rules the extreme case since one rule carries a preamble, a comment response, and regulatory text. Retrieval works on passages, while a citation lookup returns one whole section.

What happens if I send a filter value that does not exist?

The request is rejected with a 422, and for the source filter the error lists every valid code. An empty 200 would be indistinguishable from a genuine finding of no matching law, a far more expensive failure than a rejected request.

Can I retrieve a statute as it read on a specific past date?

Versioning here runs off the amendment record rather than a date parameter. A citation maps to a single stored text, the one in force now. What exists is amendment history per section, a yearFrom/yearTo filter working on the last amendment year the publisher credits, change events captured at each refresh, and per-section diffs on watched sources. There is no as_of=DATE parameter.

Why would a legal search API ever refuse to resolve a citation?

Because some abbreviations are genuinely ambiguous. MCA names both the Montana and Mississippi codes, IC names both the Indiana and Idaho codes, and Virginia and Connecticut publish bare forms naming no jurisdiction. Guessing would state one state's law under a citation meaning another, and confidently wrong is worse than resolved: false.

How do I test whether a legal retrieval vendor is actually good?

Run four query sets: exact citations scored at rank 1, terms of art scored on how many top-5 results use the legal sense, natural-language questions scored on whether a correct answer lands in the top 5, and queries whose correct answer is nothing. Almost nobody runs that last set, and it is the one that shows whether a system can say no.

Priyansh Khodiyar

Co-Founder & CTO

Priyansh leads engineering and AI at Vaquill, from the matter workbench to drafting, document comparison, document matrix, and citation-verified research.