AI Case Law Search Explained: How Semantic Search Finds the Right Precedent

Retrieval first, then grounding, then verification. The model never answers from memory.

The demo that hides the machinery

Watch any legal-AI demo and you will see the same move. Someone types a plain-English question, hits enter, and three perfectly relevant opinions appear, none of which share a single keyword with the query.

The room nods. The word "semantic" gets used like a spell. Nobody asks what happened in the half second between the question and the answer.

That gap is the whole story. AI case law search is not the model "reading the law for you" or "understanding" your client's problem the way a senior associate would.

Underneath the clean interface is a retrieval pipeline with named, inspectable parts: text gets turned into vectors, the system measures distance between your query and millions of stored passages, a second pass re-orders the winners, and only then does a language model write anything.

Once you can see those parts, two things happen. You stop being impressed by the magic, and you start catching the misses that the magic was hiding.

Short answer: AI case law search works by converting your question and millions of court opinions into vectors (embeddings), retrieving the passages that sit closest in meaning, re-ranking the top few with a heavier model, then letting a language model write a grounded answer with citations you verify. That is the whole loop: embed, retrieve, re-rank, ground, verify. Everything below is each step in plain English, plus where each one fails.

This post is the mechanics, one layer below the marketing. If you want the higher-level question of when to reach for semantic search versus a Boolean query, our companion guide on AI law search vs keyword search is the decision tree. This one is about what is actually moving when you press enter.

TL;DR

• Semantic search is vector distance, not comprehension. Queries and opinions become embeddings (long lists of numbers), and the system ranks results by how close those numbers sit in meaning-space.
• Serious legal search is hybrid, not pure vector. Production systems run dense embeddings for concepts plus BM25 keyword matching for exact terms, then a cross-encoder re-ranker on top. Pure semantic alone fumbles citations, party names, and statute numbers.
• Retrieval reduces hallucination, it does not erase it. The Stanford RegLab study found leading RAG legal tools still hallucinate 17 to 33 percent of the time. "Hallucination-free" is marketing, not a property of the pipeline.
• The pipeline has five named steps: embed, retrieve, re-rank, ground, verify. The lawyer who knows all five is the one who catches the wrong precedent before it reaches a brief.
• Mata v. Avianca is what the same question looks like with no retrieval at all: confident, fluent, and entirely made up.

Part of our legal AI verification and hallucination guide series.

For related verification / hallucination / vendor-trust coverage, see How AI Legal Research Works: RAG, Grounding, and Citations and AI Law Search Engine vs Keyword Search: Find Case Law Faster.

What an embedding actually is

Start with the piece everyone hand-waves past. An embedding is a way of turning text into a point in space.

A model reads a chunk of an opinion, say a paragraph of a Fourth Amendment ruling, and outputs a list of numbers, often several hundred to a couple thousand of them. That list is the embedding.

The useful property is that passages with similar meaning land near each other, and passages about unrelated things land far apart. "The officer searched the vehicle without a warrant" and "police conducted a warrantless car search" produce nearby points even though they share almost no words. "The court awarded attorney fees" lands somewhere else entirely.

So when you type a question, the system embeds your query the same way, then asks a simple geometric question: which stored passages sit closest to this point? Closest usually means smallest cosine distance, which is just the angle between two of those number-lists.

That is the entire trick behind "it found the case even though my words never appear in it." Your words did not match. Your meaning did, because meaning is what the embedding encodes.

This is also why semantic search shines on the problem keyword search cannot touch. Take Carpenter v. United States, 585 U.S. 296 (2018). A lawyer who knows the facts (the government pulled months of cell-site location data without a warrant) but does not yet know the doctrine can describe the situation in plain English.

Semantic search maps that description near the Carpenter line of reasoning even if the lawyer never types "third-party doctrine" or "reasonable expectation of privacy." A keyword search for the words the lawyer actually used would have returned noise. The embedding followed the concept.

Why pure semantic search quietly fails

Here is the part the demos skip. If you build a legal search engine on embeddings alone, it breaks in predictable, embarrassing ways.

Vectors are wonderful at concepts and terrible at exact strings. Ask for "18 U.S.C. 922(g)" and a pure semantic system may hand you passages that are thematically about felon-in-possession firearm law while missing the one section you actually cited, because "922(g)" as a precise token is not where embeddings are strong.

Same failure for a docket number, a party name spelled an unusual way, or a defined term of art that has to match exactly. The model knows the neighborhood. It does not always know the street address.

This is why every credible legal search system in 2026 is hybrid. The standard architecture, the one you see across recent legal information-retrieval research and production legaltech guides, runs two retrievers in parallel:

• Dense retrieval (embeddings) for meaning. This is the part that finds Carpenter from a plain-English fact pattern.
• Sparse retrieval (BM25 or similar) for exact terms. BM25 is the workhorse keyword algorithm that has powered search for decades. It is the part that nails "922(g)," the party name, the precise citation. If you want the keyword half explained on its own, our Boolean search for legal databases guide covers it.

The two result sets get fused, and that fused list is usually still not the final answer. A third stage, the re-ranker, takes the top candidates and scores them again with a heavier, slower model (a cross-encoder) that reads the query and each candidate together rather than comparing pre-computed vectors.

The first pass is fast and approximate so it can scan millions of passages. The re-ranker is slow and precise so it can sort the few dozen that survived. Speed first, judgment second.

Case law search AI: keyword vs semantic vs hybrid

The three approaches are not ranked best to worst. They are good at different jobs, which is exactly why production systems combine them.

The takeaway most demos hide: a pure semantic engine looks magical on the easy concept query and quietly drops the citation you typed verbatim. Hybrid is the boring answer that actually holds up.

As a concrete, public example of hybrid retrieval in legal text, our own US Statutes and Regulations search (the part exposed through the public statutes API) is documented as hybrid semantic plus BM25 over the full US legal corpus, roughly 2.6 million sections spanning the U.S. Code, the CFR, and 49 state codes.

That is the mechanic in the open: concept matching and exact-token matching running together over statutory text. Case-law search lives inside the product rather than the public API, running over millions of US federal and state court opinions (8M+ opinions) so you can find precedent by describing a situation.

Try Vaquill AI on your matterBook a demo

Embed, retrieve, re-rank, ground, verify

Put the stages in order and the whole pipeline becomes legible. Five steps, each with a job, each with a failure mode.

1. Embed

Both your query and the corpus get turned into vectors. The corpus part happens ahead of time: millions of opinion passages are chunked and embedded once, then stored in a vector index. Your query is embedded live. Failure mode: if opinions are chunked badly (a holding split across two chunks, a footnote glued to unrelated text), the right passage may never be retrievable no matter how good the query is. Chunking is unglamorous and decides more than people admit.

2. Retrieve

The hybrid pass runs: dense vectors for meaning, BM25 for exact terms, fused into one candidate list. Failure mode: recall. If the relevant case is not in the candidate set at this stage, nothing downstream can rescue it. A re-ranker can only sort what retrieval handed it. This is the silent miss, and it is the one lawyers should fear most, because the output looks complete.

3. Re-rank

A cross-encoder re-scores the top candidates by reading query and passage together. Failure mode: a plausible-but-wrong case ranks above the controlling one, usually because it is textually similar but factually or procedurally off point.

4. Ground

The top passages get fed to a language model as context, and the model writes its answer anchored to those passages with citations. This is the "RAG" part, retrieval-augmented generation. The full grounding pipeline is worth its own read in how AI legal research works with RAG. Failure mode: the model summarizes a real, retrieved case but states its holding slightly wrong, or stretches it to fit your question.

5. Verify

You open the cited opinion and read it. Not the snippet, the opinion. Failure mode: skipping this step, which is the failure mode that ends careers.

The "hallucination-free" claim does not survive contact with the data

Now the opinionated part, the one the brochures will not print.

A lot of vendors sell retrieval as the cure for hallucination. The pitch goes: because the answer is grounded in real retrieved documents, it cannot make things up. That is a comforting story and it is not true. Grounding lowers the rate of fabrication. It does not zero it out.

The cleanest evidence is the Stanford RegLab and HAI study, "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools," first released as a preprint in 2024 and later published in the Journal of Empirical Legal Studies. The researchers benchmarked the marquee RAG-based legal products and found that leading tools still produced hallucinations between 17 and 33 percent of the time.

Tools like Lexis+ AI and Ask Practical Law AI hallucinated more than one in six queries; one Westlaw offering tested higher still. The paper's blunt conclusion is that providers' "hallucination-free" claims are overstated, and that while RAG meaningfully reduces error compared to a raw model like GPT-4, it does not eliminate it. (Stanford HAI's summary is here.)

Sit with that. These are not toy systems. These are the most heavily resourced legal search products on the market, built by Thomson Reuters and LexisNexis, the same companies that wrote the rules of legal research for a century.

They run real retrieval over real case law. And roughly one answer in five still goes wrong, somewhere along embed, retrieve, re-rank, ground.

Map the failures back to the pipeline and you understand why. A grounded answer can fail because retrieval missed the controlling case (step 2), because the re-ranker preferred a similar-but-wrong one (step 3), or because the model mis-stated a holding it genuinely retrieved (step 4). Retrieval fixed the worst category, the case that does not exist at all, but it left three other ways to be wrong standing.

Why this matters more than the marketing

Contrast two failure modes and the value of understanding the machine becomes obvious.

The first is Mata v. Avianca (2023), the now-infamous matter where a lawyer filed a brief full of citations that ChatGPT had invented wholesale, complete with fake quotes and fake docket numbers, and got sanctioned. That is what zero retrieval looks like: a fluent model improvising from training data with nothing to ground it. No embed, no retrieve, no source to open. Pure confident fiction.

A modern RAG legal tool would almost certainly not invent Mata's phantom cases, because retrieval anchors it to opinions that actually exist. That is real progress and worth saying plainly. But the RegLab data shows the failure mode does not vanish, it migrates.

Instead of a case that does not exist, you get a real case cited for a proposition it does not stand for. That error is harder to catch precisely because everything checks out at a glance: the citation is real, the case is real, the link opens.

Only reading the opinion reveals that the holding does not say what the answer claims. The duty to do exactly that reading sits squarely on the lawyer under ABA Formal Opinion 512, and our practical walkthrough on verifying AI legal citations before filing is the checklist version.

So the lawyer who treats "semantic" as magic verifies nothing, because magic does not need checking. The lawyer who knows the pipeline verifies the right things: did retrieval surface the controlling authority, did the re-ranker put it on top, and does the cited opinion actually hold what the summary says.

Same tool. Completely different risk profile. The understanding is the safety feature.

Where AI for case law research genuinely wins

Beyond a ranked list of hits, a citation graph shows a case's authorities and the later cases that cite it, which is how you check whether a precedent still holds.

None of this is an argument against AI case law search. It is an argument for using it with your eyes open. Used well, semantic retrieval does something Boolean never could: it lets you find precedent by describing a situation instead of guessing the vocabulary the court happened to use.

That is the Carpenter scenario from earlier, and it generalizes. A new factual wrinkle, a doctrine you cannot name yet, a "has this been overruled" question after Loper Bright v. Raimondo, 603 U.S. 369 (2024), knocked out Chevron deference: these are exactly the queries where meaning-matching beats keyword-matching, because you do not yet know the keywords.

Litigators living in this terrain feel it most, which is why semantic research pairs naturally with downstream tools like a chronology builder that turns the cases and facts you surface into a defensible timeline. Our litigation-focused use case walks through how research and chronology feed each other on an active matter.

The honest framing is the one the RegLab authors land on. Semantic retrieval is a powerful instrument that reduces a specific, dangerous category of error and introduces subtler ones in its place.

It earns a permanent spot in the legal research workflow. It does not earn blind trust. Know the five steps, verify at the fifth, and the machine works for you instead of against you.

FAQ

How does AI case law search work?

It runs a five-step retrieval pipeline. Your question and millions of court opinions are turned into embeddings (lists of numbers), the system retrieves the passages closest in meaning while a keyword pass (BM25) catches exact terms, a re-ranker re-scores the top candidates, a language model writes a grounded answer with citations, and you verify by opening the cited opinion. The model never answers from memory alone.

What is the difference between AI case law search and keyword search?

Keyword (Boolean) search matches exact terms, so it nails citations, statute numbers, and party names but misses cases phrased differently from your query. AI case law search matches meaning through embeddings, so it can surface the right precedent from a plain-English fact pattern even when none of your words appear in the opinion. Serious tools run both together because each covers the other's blind spot.

Is AI case law search accurate, or does it hallucinate?

It reduces hallucination but does not remove it. The Stanford RegLab study (published in the Journal of Empirical Legal Studies, preprint 2024) found leading retrieval-based legal tools still hallucinated 17 to 33 percent of the time. Grounding stops the worst error (a case that does not exist) but a real case can still be cited for a holding it does not support, which is why you read the opinion before relying on it.

Can AI search find a case if I do not know the legal term for it?

Yes, that is semantic search's main advantage. You can describe the facts in plain English (for example, the government pulling months of cell-site location data without a warrant) and the system maps your description near the relevant line of reasoning, such as Carpenter v. United States, without you ever typing "third-party doctrine."

Does AI for case law research replace Westlaw or Lexis?

Not yet, and not cleanly. Lexis+ AI and Westlaw's AI offerings were two of the tools the Stanford study found hallucinating on real queries, so an AI layer is not automatic reliability. Treat AI case law research as a faster first pass over the same primary law, with citation verification still mandatory under ABA Formal Opinion 512.

What data does AI case law search run on?

It depends on the tool. Vaquill AI's case law research runs over millions of US federal and state court opinions (8M+ opinions). The quality of any answer is capped by the corpus underneath it, so coverage and freshness of the case data matter as much as the model.

Why do I still have to read the cited opinion?

Because retrieval can hand the model the wrong case, the re-ranker can rank a similar-but-off-point case first, or the model can mis-state a holding it genuinely retrieved. All three failures produce an answer where the citation is real and the link opens, so only reading the opinion reveals whether it actually holds what the summary claims.

Try it on your next question

Vaquill AI is an AI legal research and drafting suite built for US firms, not AmLaw price tags. Ask a question and get an answer grounded in real federal and state court opinions, with citations you can open and verify. Compare documents, build chronologies, and keep every matter organized in one workspace.

Start free at app.vaquill.ai or see how Vaquill AI works.

Vaquill AI

Product & Content

Legal AI suite for US working lawyers: research, drafting, document comparison, document matrix, matters, and citation-verified answers, in one tool.