How to Verify a Legal AI Accuracy Claim (Before You Trust the Number)

Short answer: A legal AI accuracy claim is only worth as much as your ability to reproduce it. Before you trust a number like "95 percent accurate," ask for three things: the exact questions, the vendor's actual answers, and a scoring script you can run yourself. If any of the three is missing, the number is marketing, not evidence. A real accuracy claim survives you rerunning the math and getting the same result.

TL;DR

• A number with no released test set is a slogan. If you cannot see the questions, you cannot judge whether they were easy, cherry-picked, or graded loosely.
• Demand three artifacts: the committed questions, the vendor's committed answers, and a public scorer. With all three you reproduce the number in minutes. With fewer, you are trusting a press release.
• Watch for the easy-question trap. A model that ties a specialist on famous black-letter law tells you nothing. The test has to include law the model could not have memorized.
• Prefer deterministic metrics over an AI grader. "Did the answer contain the right statute section" needs no model and reproduces exactly. An LLM judging another LLM is noisy and gameable.
• The strongest proof is a real, checkable citation. An accuracy claim you can open (a statute section, a case name) beats any percentage you cannot.
• Independent third-party checks help, but the cheapest audit is running the vendor's own committed answers through the vendor's own scorer.

Why this matters more in legal than anywhere else

Most software accuracy claims are low stakes. A recommendation engine that is 90 percent accurate is fine. A brief built on an AI-invented case gets you sanctioned.

That asymmetry is why legal buyers cannot take an accuracy number on faith. Stanford's RegLab tested the leading commercial legal-research tools and found they still misstate or miscite the law 17 to 33 percent of the time, even after vendors marketed retrieval as the fix (RegLab, 2024). The lesson is not that legal AI does not work. It is that vendor accuracy claims and measured accuracy are two different things, and you need a way to tell them apart.

The three artifacts a real accuracy claim ships with

You do not need a data-science team to audit a legal AI accuracy claim. You need the vendor to hand you three files, and you need to run one command.

1. The questions, committed and dated

Ask for the actual test set as a file, not a description. Then check two things.

Are the questions hard enough to matter? A benchmark where a general model ties the specialist tool is measuring memory, not the product. The set has to include law a model could not have memorized: figures that changed this year, recently amended statutes, obscure state thresholds. If every question is famous black-letter law, a strong general model already knows it and the vendor's edge is invisible.

Are time-varying questions dated? "The SALT deduction cap" has a different correct answer in 2025 than in 2026. A serious test set stamps each answer with a "verified as of" date, so you know the key is current.

2. The vendor's answers, committed so you can re-score them

This is the artifact almost no vendor gives you, and the one that matters most. A percentage with no answers behind it cannot be checked. The vendor's committed answers, including the sources each answer cited, let you re-score the run yourself and confirm the number.

3. A scorer you can run, deterministic where it counts

The scoring code should be public and, for the headline metrics, need no model at all. "Did the answer contain the correct statute section or dollar figure" is a substring check. It reproduces exactly, every time, on any machine. That is a metric you can trust.

Be skeptical of any headline number produced by an AI grader scoring another AI. It is noisy run to run and easy to tune. An LLM judge is fine as a secondary signal, but only when the vendor also publishes how it agreed with human graders.

A worked example you can reproduce in five minutes

Here is what "reproducible" looks like in practice, using the open benchmark we publish. The point is not the score. It is that you can get the same score yourself.

The test set has two tranches: famous settled law (the control) and recent or changed US law (the hard set, screened so a plain model gets it wrong from memory). Both arms run the same questions: a retrieval-grounded engine, and the same underlying model with no sources.

Run the scorer on the committed answers and you get the correct-fact rate of 93 percent on recent law, versus 33 percent for the same model with no sources, and the governing authority retrieved on 97 percent of questions. No API key, no model, no randomness.

The proof that beats any percentage is a single checkable fact. Ask a plain model for the 2026 SALT deduction cap and it tends to answer $10,000, the old cap. The current figure is $40,400 under 26 U.S.C. 164(b)(6), as amended in 2025. A grounded answer cites that section so you can open it. A guess cannot.

The questions to send a vendor

Paste these into your next vendor call. The answers, and the speed of the answers, tell you a lot.

• Can you send the exact questions your accuracy number was measured on? A yes, as a file, is a strong signal. A "we cannot share that" is a red flag.
• Are the answers committed so we can re-score them ourselves? This separates a reproducible claim from a slide.
• How many of your test questions would a general model already get right? If the honest answer is "most," the benchmark is measuring memory, not their product.
• Is the headline metric deterministic, or graded by another AI model? Deterministic first, AI judge second.
• How do you date time-sensitive answers? Tax figures and thresholds go stale; a good test stamps them.
• Where does your own tool fall short on this test? A vendor that publishes its own misses is more credible than one that reports a clean sweep.

Where a plain accuracy number misleads

Even an honest percentage hides things a legal buyer needs to know.

• It averages over easy and hard. A single number blends black-letter law (where any tool wins) with current law (where most fail). Ask for the split.
• It does not tell you what happens on unanswerable questions. The dangerous failure is not a wrong answer. It is a confident answer to a question with no answer, like a fictitious case. Ask how the tool behaves on invented authorities.
• It ignores whether you can verify the answer. A correct answer you cannot check and a wrong answer you cannot check look identical on the page. Citations you can open are the difference between a research tool and a liability.

This is the same discipline the broader buyer's framework for evaluating legal AI vendors applies across security, pricing, and workflow. Accuracy is the dimension where "trust me" is most tempting and least earned.

See the open, reproducible benchmarkBook a demo

FAQ

What does "accuracy" even mean for legal AI?

There is no single number. The useful ones are: did the answer contain the correct fact (a statute section, a figure, a case name), did it cite the governing authority, and did it decline when there was no real answer. A vendor that reports one blended percentage and nothing else is hiding the parts that matter.

Is a 90 percent accuracy claim good?

It depends entirely on the questions. Ninety percent on famous settled law is unremarkable, because a general model scores near that from memory. Ninety percent on recent, changed law that a plain model gets wrong is meaningful. Always ask what the questions were.

How do I reproduce a vendor's accuracy number myself?

Ask for three files: the questions, the vendor's answers, and the scoring script. Run the script on the answers. If you get the number the vendor published, it is real. Our benchmark is set up this way at github.com/Vaquill-AI/vaquill-legal-benchmark.

Why can I not just run the vendor's tool on my own test?

You can, and you should, in a pilot. But most legal AI products are hosted apps with no public answer API, so you cannot script thousands of questions through them. Committed answers plus a public scorer let you verify the vendor's claim without that access.

Are AI-graded benchmarks trustworthy?

Treat them as secondary. An LLM grading another LLM is noisy between runs and can be tuned to flatter. It is acceptable as a soft signal only when the vendor also reports how the AI grader agreed with human lawyers on a sample.

What is the single biggest red flag?

A refusal to share the questions. If a vendor will not show you what its number was measured on, assume the number was measured on the questions most flattering to the tool.

Does grounding (RAG) guarantee accuracy?

No. It makes wrong answers catchable, because every claim carries a source you can open, and it keeps answers current on law that changed. It does not remove your duty to check every cite before filing. See how AI legal research works with RAG and how to verify AI legal citations before filing.

Priyansh Khodiyar

Co-Founder & CTO

Priyansh leads engineering and AI at Vaquill, from the matter workbench to drafting, document comparison, document matrix, and citation-verified research.