How AI Legal Research Works: RAG, Grounding, and Citations

How AI legal research works: the tool retrieves the actual statutes and opinions that match your question first, a pattern called retrieval-augmented generation (RAG), grounds the answer in that retrieved text, then verifies every citation against the source before you see it. It does not ask the model to recall the law from memory. That single design choice is the difference between a tool that cites real authority and one that invents cases.

Those three steps, retrieval, grounding, and verification, are what make RAG legal research reliable enough to file from. This guide walks through each layer of the pipeline and shows exactly where each one breaks.

The Setup

In 2023 a New York lawyer filed a brief citing six cases that did not exist. ChatGPT had written them, names, citations, and quoted holdings, and nobody checked.

The judge in Mata v. Avianca sanctioned the firm, and the screenshot of that order has shown up in CLE decks ever since. That is the failure mode this entire field exists to prevent.

The rest of this guide is how AI legal research works when it works, and why grounding and citation verification, the two stages most explainers gloss over, are what actually decide whether you can file the output.

I'm Priyansh, CTO of Vaquill AI. I've spent two years building this pipeline for US case law and statutes, and this is the explainer I wish existed when I started.

TL;DR

• AI legal research that you can trust is built on RAG (retrieval-augmented generation): the model fetches real opinions first, then answers from them, instead of answering from memory.
• A raw LLM with no retrieval will confidently invent citations. That is not a bug you prompt away. It is why Mata v. Avianca happened.
• The pipeline has six stages: ingestion, chunking, embedding, vector storage, retrieval and reranking, then generation with citation extraction.
• Embedding similarity is not legal correctness. Terry, Katz, and Carpenter all cluster as "Fourth Amendment search" but hold very different things. Reranking and hybrid keyword search exist to fix that.
• The grounding layer (mapping every claim back to a real source passage with a pin cite, and surfacing whether the case is still good law) is what separates a demo from something a lawyer can file.

Part of our legal AI verification and hallucination guide series.

For related verification / hallucination / vendor-trust coverage, see AI Case Law Search Explained: How Semantic Search Finds the Right Precedent and What Legal AI Actually Avoids Hallucinating Cases (and Why Most Do Not).

Retrieval first, then grounding, then verification. The model never answers from memory.

What is RAG, and why law breaks without it

RAG stands for retrieval-augmented generation. The idea is simple: an LLM does not know your data, so you fetch relevant documents first and hand them to the model along with the question.

It is an open-book exam. The LLM is the student. The case law is the textbook. Retrieval is flipping to the right pages before answering.

Without that step you get hallucination. Ask a bare LLM "what did the Supreme Court hold in Carpenter v. United States?" and it will often produce something fluent and mostly right on the law (cell-site location records get Fourth Amendment protection) but wrong on the specifics that matter.

It might give you "138 S. Ct. 2026" when the real reporter cite is 138 S. Ct. 2206, or assign the wrong year, or attribute the majority opinion to the wrong Justice. In a product recommendation, a fabricated detail is annoying. In a brief, it is the Mata problem, and it is sanctionable.

Grounding the answer in retrieved text does not magically make the model honest. It gives you something to check the model against. That distinction runs through everything below.

RAG cuts hallucinations, it does not erase them

RAG is the reason legal-specific tools beat a bare chatbot, and the gap is measurable. In Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (Magesh et al., Stanford RegLab and HAI, preprint May 2024, later published in the Journal of Empirical Legal Studies, 2025), researchers hand-scored 202 legal queries and found GPT-4 with no retrieval hallucinated on about 43% of them. The RAG-based commercial tools did much better: Lexis+ AI hallucinated roughly 17% of the time and Westlaw AI-Assisted Research roughly 33%.

So retrieval roughly halves the error rate, and it still leaves one wrong or unsupported answer in every three to six. The study's point is the one this guide is built around: RAG is necessary, but the vendor claim of "hallucination-free" is marketing. What closes the remaining gap is the verification and treatment work in the back half of the pipeline, not the retrieval step alone.

The RAG legal research pipeline, stage by stage

Here is what a production retrieval-augmented generation system for law looks like at a high level.

Six stages. Each has its own failure modes. Let me walk through them with real US case law.

One step sits in front of all six and the diagram leaves out: intent routing. Before a production system retrieves anything, it decides whether your message even needs retrieval. Paste a brief and ask it to proofread, and firing case-law search is the wrong move: you want text editing, not authority. A good system classifies the ask (a research question, a text-processing task, or a plain conversation) and only runs the pipeline below when the question actually calls for law. In the product, that same routing is what lets the chat search your own uploaded documents or the web when those fit better than the case-law corpus, and hand a "compare these two versions" request off to the right tool instead of answering in prose.

1. Document ingestion

First you turn opinions into machine-readable text. Modern federal opinions are clean, text-based PDFs, so a parser like PyMuPDF or pdfplumber gives you decent text.

Older material is where it gets ugly. Pull a 1960s opinion out of a scanned reporter volume, Mapp v. Ohio, 367 U.S. 643 (1961), or Gideon v. Wainwright, 372 U.S. 335 (1963), and you are dealing with faded print, two-column layouts, and pages someone fed through the scanner at an angle.

Those need OCR (Tesseract, Google Vision, or Amazon Textract), and quality varies. I have watched OCR turn "Section 1983" into "Section 1983" on a good day and "Section 1933" on a bad one. When your whole system depends on text accuracy, a 2% error rate compounds fast.

2. Chunking

You cannot feed an entire 90-page opinion into an embedding model. Embedding models cap out around 512 tokens, so you split documents into passages.

Sounds trivial. It is not. A single sentence in a Supreme Court opinion can run fifteen lines, with nested parentheticals, string cites to four other cases, and a "but see" buried in the middle. Split it in the wrong place and the meaning falls apart. Split too coarsely and the embeddings become too generic to match a specific question.

We tried fixed-size chunks (512 tokens with overlap), paragraph splitting, and recursive character splitting. None worked well out of the box on judicial prose.

We built a custom chunker that respects opinion structure: it keeps the syllabus separate from the majority, holds a footnote with its anchor, and never severs a citation from the sentence that gives it meaning.

3. Embedding

Embeddings are vector representations of text. You turn each chunk into a high-dimensional vector and later turn the query into a vector in the same space. Similar meaning, nearby vectors.

Which model? OpenAI's text-embedding-3-large is the default and fine for general text. Legal language is a different animal. "Reasonable suspicion," "probable cause," and "exigent circumstances" are distinct standards with distinct thresholds, and a general-purpose model can blur them.

Models adapted to legal text (something in the Legal-BERT family or a model fine-tuned on US opinions) hold the distinctions better, but they are harder to deploy and maintain.

We tested several embedding models with hard negative mining, training on case pairs that look similar but reach opposite outcomes, before landing on our current approach.

4. Vector database

Your vectors have to live somewhere: Pinecone, Qdrant, Weaviate, or Milvus.

At scale this is not a hobby deployment. Millions of opinions at roughly 50 chunks each runs into hundreds of millions of vectors, which means real RAM and real monthly spend.

Treat the numbers as order of magnitude, not a quote: a managed index at that size is comfortably in the hundreds of dollars per month, and a self-hosted HNSW index wants machines with a lot of memory to keep the graph resident.

The database also shapes retrieval quality. HNSW is an approximate nearest-neighbor algorithm. It is fast, and it misses results. For legal research, where a missed case can flip an argument, you tune ef_search and ef_construction toward higher recall and accept slower queries in return.

5. Retrieval and reranking

Vector search returns your top-K chunks. Similarity alone is not enough, and Fourth Amendment law is the cleanest way to see why.

Search "warrantless search of digital location data." A semantic embedding will happily pull back Terry v. Ohio, 392 U.S. 1 (1968), Katz v. United States, 389 U.S. 347 (1967), and Carpenter v. United States, 585 U.S. 296 (2018). All three live in the same neighborhood of vector space because they are all "Fourth Amendment search" cases.

But Terry is a stop-and-frisk on a sidewalk, Katz is a wiretap on a phone booth, and Carpenter is seven days of cell-site location records. For the digital-location question, Carpenter is on point and the other two are background. Generic embeddings cannot tell you that. The vectors are close; the law is not.

So you split retrieval into two stages. The first stage is recall-optimized: cast a wide net, pull 50 to 100 candidate chunks. Then a cross-encoder reranker scores each candidate against the query with full attention, and Carpenter rises to the top while Terry and Katz settle into context.

This two-stage pattern is standard in information retrieval, and it matters more in law than almost anywhere else.

Legal queries also need hybrid retrieval: vector search for meaning plus BM25 keyword search for exact strings. If someone searches "138 S. Ct. 2206" or runs a Boolean query like "Fourth Amendment" AND "warrantless search", you want exact matching, not fuzzy semantic similarity. Pure vector search is bad at literal citations; BM25 is built for them.

Try Vaquill AI on your matterBook a demo

6. Citation normalization: one case, three reporters

Before you can verify a citation you have to recognize it, and US citations are not one format. Carpenter alone shows up as:

• 585 U.S. 296 (United States Reports)
• 138 S. Ct. 2206 (Supreme Court Reporter)
• 201 L. Ed. 2d 507 (Lawyers' Edition, Second)

Same opinion, three reporters, plus pincite variants and the occasional docket-number reference. Lower-court work adds F.2d / F.3d for the Courts of Appeals and F. Supp. for the District Courts.

Your system has to know all three Carpenter cites point at one case. Get it wrong and you either tell a user that two cites are two different cases or you merge two different cases into one. Building a normalization layer that maps every alias back to a single canonical case took us months.

7. Generation and grounding

Now you have your relevant chunks. You build a prompt:

Given the following excerpts from US court opinions and statutes:

[Chunk 1: From Carpenter v. United States, 585 U.S. 296 (2018), syllabus]
[Chunk 2: 42 U.S.C. § 1983, current text]
...

Answer the question below. Cite specific cases, statutory sections,
and page or paragraph numbers. If the answer is not in the excerpts,
say so. Do not supply citations that are not in the provided sources.
{user_question}

The model writes an answer. Here is the hard part: how do you guarantee the citations in that answer map back to real passages in real sources?

The model might write "as the Court held at 585 U.S. at 313." Did the source chunk actually come from page 313, or did the model interpolate a plausible-looking pin cite?

You need a post-processing layer that checks every citation against the retrieved chunks, pulls the exact page or paragraph, and flags any claim that is not grounded in the context you supplied. Anything the model asserts without a backing passage gets dropped or marked low-confidence rather than shipped.

That verification layer, plus a confidence threshold that makes the system say "I am not sure" instead of guessing, is what separates a demo from a system a lawyer can file. Without it, you have built a faster way to produce a Mata brief.

A worked example: question in, grounded answer out

Here is the full path for one query, so the abstract pipeline above has something concrete attached to it.

Question: "Does the government need a warrant to get a week of someone's cell-site location records?"

Retrieval pulls candidate chunks. Vector search returns Terry, Katz, and Carpenter; the BM25 pass catches the literal phrase "cell site"; the reranker pushes Carpenter v. United States, 585 U.S. 296 (2018) to the top because it is the only one of the three about cell-site location data.

Generation writes an answer using only those chunks. Verification then checks each cite against the retrieved passage and attaches the pin cite. The shipped output looks like this:

Yes. The government generally needs a warrant supported by probable cause to obtain historical cell-site location information spanning seven days. The Supreme Court held that acquiring this record is a Fourth Amendment search, so the third-party doctrine does not strip it of protection. Carpenter v. United States, 585 U.S. 296, 310-11 (2018).

Treatment check: Carpenter is good law as of this writing, with no negative subsequent history on the holding.

Every load-bearing claim in that paragraph traces to a retrieved passage, the cite is a real, checkable one (you can pull 585 U.S. 296 and read it), and the treatment line is a separate signal the pipeline adds, not something the model guessed. That last line is the part the diagram up top does not show, and the next section is why it matters most.

The part most explainers miss: is it still good law?

Text similarity tells you a passage is relevant. It does not tell you the passage is still authoritative. This is the most important and most overlooked piece of legal RAG.

Run a query about agency deference and your retrieval will surface Chevron U.S.A. v. NRDC, 467 U.S. 837 (1984). For forty years that was the controlling framework, and the opinion text is a perfect semantic match for any deference question.

It is also, as of 2024, no longer good law: the Supreme Court overruled it in Loper Bright Enterprises v. Raimondo, 603 U.S. 369 (2024). An AI tool that retrieves Chevron, summarizes its two-step test, and hands it to you without flagging Loper Bright has given you a confident, well-cited, dangerous answer.

Embedding similarity cannot catch this. Chevron and Loper Bright are similar in topic, but the relationship that matters is one of treatment: overruled by. That lives in a citation graph, not a vector space.

A serious system runs every retrieved case through a validity check, surfaces negative treatment, and tells the model (and the user) when controlling authority has shifted.

Relevance without treatment is how you cite a dead case in a live brief.

The alternative: don't rebuild what's already built

You can build all seven stages yourself. It is real work: months of engineering, OCR pipelines, a custom chunker, embedding evaluation, a normalization layer, a verification layer, and a treatment graph, before you write a single feature your users actually see.

The detailed cost comparison of building the pipeline versus buying access lays out the order-of-magnitude numbers.

On the statutes side, the legwork has been done already too. A public statutes API covering the U.S. Code, the CFR, and all 50 state codes lets an agent pin the operative statutory language with one call:

curl -X POST https://api.vaquill.ai/api/v1/statutes/search \
  -H "Authorization: Bearer $VAQUILL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "fair use defense", "corpusType": "USC", "limit": 5}'

The response returns matching sections with citation, title, and excerpt fields, plus a GET /api/v1/statutes/section companion endpoint for the full normalized section text. That is the statutes half of the pipeline above (parsing, chunking, normalization, indexing) replaced by a single HTTP call.

The case-law half (opinion search, citation resolution across reporter formats, treatment for the "is it still good law" check) is the harder problem. There is no public REST API that solves it end-to-end in 2026.

Production case-law grounding lives inside research suites (Harvey, Legora, CoCounsel, and the suite our team builds) as a product feature, not as an exposed endpoint, because the verification and treatment layers are tightly coupled to a specific corpus and UI.

If you need that in your own product, the realistic options are: license a vendor's full product, build the case-law pipeline yourself against an opinion corpus, or constrain the use case to the statutory side that an API does cover today.

When to build your own

I'll be honest about when rolling your own makes sense:

• Private data. Internal memos, a contract database, client correspondence that cannot leave your infrastructure. You need your own pipeline.
• A narrow niche with custom embeddings. If you live entirely in, say, patent claims or municipal code and need embeddings fine-tuned to that, owning the stack gives you more control.
• Research. If you are studying legal NLP or inventing new retrieval methods, building from scratch is the point.

For statute lookups (U.S. Code, CFR, state codes), a public API already gives you parsing, chunking, and normalization for free. For grounded case-law research, the realistic path is licensing a vendor or building your own opinion pipeline; the verification and treatment layers are not yet a commodity.

Either way, spend the engineering time on what makes your product different.

The real takeaway

RAG is conceptually simple: fetch relevant documents, feed them to an LLM, generate an answer. You can demo it in a weekend.

The distance from that demo to something a lawyer can file is enormous, and almost all of it is in the parts nobody screenshots: OCR on scanned reporters, a chunker that respects opinion structure, citation normalization across three reporters per case, a verification layer that maps every claim to a real passage, and a treatment graph that catches a Loper Bright before you cite a dead Chevron.

Get those right and AI legal research stops being a liability and starts being faster than the old way. Skip them and you have built a confident hallucination machine pointed at a federal docket.

If you want to see what happens when this goes wrong, read our breakdown of AI hallucinations and legal sanctions.

FAQ

How does AI legal research work?

It works in three stages. The tool retrieves the statutes and opinions that match your question from a real legal corpus, a pattern called retrieval-augmented generation (RAG). It then grounds its answer in that retrieved text instead of the model's memory. Finally it verifies every citation against the source passage before showing you the answer. The retrieval step is what lets the tool cite authority that actually exists.

What is retrieval-augmented generation in legal research?

Retrieval-augmented generation (RAG) is the method of fetching relevant documents first and handing them to a language model along with the question, so the model answers from supplied sources rather than guessing. In legal research it means pulling the real opinion or statute section, then generating an answer grounded in that text. It is the open-book version of an AI answer, which is why RAG legal tools hallucinate far less than a bare chatbot.

Is AI legal research accurate?

It is useful but not turnkey. A 2025 study from Stanford RegLab and HAI (Magesh et al.) found the leading RAG-based tools, Lexis+ AI and Westlaw AI-Assisted Research, still gave an incorrect or unsupported answer on roughly 17% to 33% of queries, versus about 43% for GPT-4 with no retrieval. RAG cuts the error rate sharply, but every citation still has to be verified before you file.

Does RAG stop AI from hallucinating legal citations?

It reduces hallucination, it does not eliminate it. Grounding the answer in retrieved text gives you a real passage to check each cite against, which is the whole value, but the same Stanford study showed even RAG tools mischaracterize cases and cite inapplicable authority. The fix is a verification layer that drops any claim with no backing passage, plus a treatment check for whether the case is still good law.

Why do AI tools invent fake cases?

A language model with no retrieval predicts plausible text, and a citation that looks real (right reporter format, believable volume and page) is easy to predict and impossible to trust. That is what produced the fabricated cases in Mata v. Avianca. RAG fixes the root cause by forcing the model to answer from documents that were actually fetched, then verifying the cites against those documents.

Can AI tell if a case is still good law?

Not from text similarity alone. A retrieval system will happily surface an overruled case like Chevron U.S.A. v. NRDC because it is a perfect semantic match for a deference question, even though Loper Bright Enterprises v. Raimondo (2024) overruled it. Catching that requires a citation graph that tracks treatment (overruled, reversed, distinguished), which is a separate layer from retrieval.

What is the difference between vector search and keyword search in legal RAG?

Vector (semantic) search matches meaning, so it finds relevant cases even when they use different words than your query. Keyword search (BM25) matches exact strings, which you need for literal reporter citations like "138 S. Ct. 2206" or Boolean queries. Good legal RAG runs both, called hybrid retrieval, because pure semantic search is poor at exact citations and pure keyword search misses paraphrased concepts.

Should I build my own legal RAG pipeline or use a tool?

Build it when your data is private, your niche needs custom embeddings, or you are doing research. For statute lookups, a public API already handles parsing, chunking, and normalization. For grounded case-law research, the realistic path is licensing a vendor or building your own opinion pipeline, because the verification and treatment layers are not yet a commodity you can call over an API.

Try it on your next question

If you want a US legal AI suite that has already built the grounded research pipeline so you don't have to, Vaquill AI is one option. Start at app.vaquill.ai or see the statutes API, or email contact@vaquill.ai.

Priyansh Khodiyar

Co-Founder & CTO

Priyansh leads engineering and AI at Vaquill, from the matter workbench to drafting, document comparison, document matrix, and citation-verified research.