Accuracy & Benchmarks

The Legal-AI Accuracy Problem, Confronted Head-On

Stanford RegLab found hallucination rates between 17% and 33% in LexisNexis and Thomson Reuters / Westlaw. Vaquill AI's 4-layer citation verification was built to catch exactly the failure modes those studies identified.

Stanford RegLab's 2024 study of retrieval-augmented legal-AI products reported hallucination rates between 17% and 33% across the tools it tested, including LexisNexis and Thomson Reuters / Westlaw. LexisNexis had to walk back its “100% hallucination-free” marketing after that study landed. The FTC's Operation AI Comply (2024, 2026) has since taken enforcement action against vendors making unsubstantiated AI-accuracy claims.

Vaquill AI takes a different approach. Every answer is checked through a 4-layer verification pipeline before you see it: exact text match, citation resolution, meaning analysis, and AI cross-check. We catch the failure modes those studies flagged, fabricated case names, misquoted passages, paraphrases that do not match the cited holding, and internally inconsistent answers. Combined with ABA Formal Opinion 512's lawyer-review duty, that is the standard the profession is moving toward.

The Vaquill AI Legal Benchmark: open and reproducible

Most legal-AI accuracy claims are a slogan with no number, or a vendor benchmark whose questions are never released. We published the opposite. On US law that changed in the last two years, the case where a model's training data is stale and a lawyer actually needs help, a retrieval-grounded engine answered correctly 93% of the time. The same underlying model with no sources: 33%.

On recent or obscure US law, a grounded engine answered 93 percent of questions correctly versus 33 percent for a general AI model

Metric (hard tranche)	Grounded	Ungrounded (same model, no sources)
Correct answer	93%	33%
Right authority retrieved	97%	0%
Declined the unanswerable trap questions	8 / 8	6 / 8
Fabricated an answer on the traps	0	2

Correct-answer rate by how settled the law is: grounded and ungrounded tie on settled law and diverge sharply on recent or changed law — On settled law both approaches tie. The gap opens on law that changed recently, which is exactly where a lawyer needs the help.

The number is not the point. The questions, the scoring code, and the raw answers with retrieved source text are all committed to GitHub under an open license. You do not have to trust it. Re-score it and get the same number, or point it at another tool and compare.

View the benchmark on GitHub Read the full methodology

Our 4-layer citation verification

Every Vaquill AI answer passes through four independent checks before it reaches the lawyer. If any layer fails, the response is labeled so the lawyer sees the failure before deciding whether to rely on the output.

Layer 1

Exact text match

Every quoted passage in a Vaquill AI answer is searched verbatim against the underlying source corpus (uploaded documents, case-law database, or statutory text). If the quote does not match a source string character-for-character, the passage is flagged.

Layer 2

Citation check

Citations are parsed and resolved against our indexed US case-law and statute corpus. Citations that fail to resolve to a real case, or that resolve to a case whose caption does not match, are flagged as invalid before the response is rendered.

Layer 3

Meaning analysis

A separate model pass cross-checks that the paraphrased holding or rule Vaquill AI cited actually matches what the cited case held. A citation that technically resolves but is being used to support a proposition the case does not stand for is labeled accordingly.

Layer 4

AI cross-check

A second model independently reviews the full response for internal consistency, checking that claims do not contradict each other, that numbers add up, and that qualifiers (e.g., jurisdiction, date range, court level) are preserved from source to answer.

Confidence labels

The result of the four layers is surfaced on every response as one of three labels. They tell the lawyer exactly where Vaquill AI stands behind the answer and where to focus the bar-required review.

Verified

All four verification layers passed. Quote matches source, citation resolves, holding analysis agrees, and the cross-check found no contradictions. Lawyer review is still required.

Partially Verified

One or more layers returned a partial result (for example: a citation resolved but the paraphrase was close rather than verbatim). The label tells you exactly which layer to double-check, so verification is fast.

Unverified

One or more layers flagged an issue (for example: a quote did not match any source, or a citation did not resolve). Vaquill AI surfaces this clearly so the lawyer can confirm before relying on the passage.

Internal evaluation set

We maintain an internal evaluation set of representative queries across US federal, US state, contract-review, and case-law-retrieval tasks. It is the suite we re-run on every major model change before shipping the change to production.

What we measure

Citation validity rate, the percentage of generated citations that resolve to a real case in the underlying case-law corpus.
Quote-match rate, the percentage of quoted passages that match the source text verbatim (within normalized whitespace / punctuation).
Holding-accuracy rate (lawyer-adjudicated), the percentage of summarized holdings that match the actual holding of the cited case, as judged by a licensed lawyer reviewing a sampled subset.

We expand the suite whenever a customer reports a failure mode that was not represented, so it grows with real usage rather than staying frozen to launch-day assumptions. Our latest internal eval results will be published on this page once our current evaluation cycle completes.

Third-party benchmarks, roadmap

Independent evaluation matters more than anything we publish ourselves. Here is where Vaquill AI currently stands on the public benchmarks lawyers ask about. We do not claim participation we have not done.

Vals AI / Vals LAIR

Planned

The industry-standard legal-AI benchmark suite. Harvey, CoCounsel, and vLex have participated in public cycles. Vaquill AI is preparing to participate in an upcoming cycle, and we will publish our results in full, including the tasks where the gap between Vaquill AI and other tools is largest. More at vals.ai.

LegalBench

Planned

The academic legal-reasoning benchmark developed by researchers at Stanford, Princeton, and Stanford HAI. LegalBench covers 162 tasks spanning issue-spotting, rule recall, rule application, rule conclusion, and interpretation. Vaquill AI's LegalBench submission is planned for the next academic cycle, and results will be published task-by-task on this page.

Stanford RegLab / HAI legal-AI audits

Coming Soon

Stanford RegLab's 2024 study of retrieval-augmented legal-AI tools found hallucination rates between 17% and 33% across major vendors (including LexisNexis and Thomson Reuters / Westlaw). Vaquill AI's 4-layer verification pipeline is built to catch exactly the failure modes RegLab identified, and we replicate the relevant tests against Vaquill AI output internally with each major model update.

Transparency commitments

What we promise to do when we publish accuracy numbers on this page, and what we promise not to do.

Publish results with methodology

When we publish evaluation numbers, we will include the date the evaluation was run, the sample size, the sample composition (jurisdictions, practice areas, task types), the model version(s) evaluated, and the judging procedure.

Disclose model changes

Our LLM sub-processors (OpenAI, Anthropic) periodically update the underlying models. Material model changes will be announced in our changelog and billing admins will be notified. We re-run our internal eval suite on every major model change.

No cherry-picking

All major eval-set results will be published, including regressions. If a model update causes our citation-validity rate to drop on a subset of queries, we will say so.

No unsubstantiated comparative claims

We will not publish comparisons against Harvey, CoCounsel, Westlaw, LexisNexis, or any other tool unless we can link to a reproducible methodology that a third party could re-run. Marketing-only comparisons are not published here.

Hallucination risks, be explicit

How lawyer review fits in

Every Vaquill AI output carries a confidence label (Verified, Partially Verified, Unverified) so you know exactly how much corroboration each claim received.
ABA Formal Opinion 512 makes lawyer review of generative AI output a professional obligation. Vaquill AI makes that review fast: source PDFs are surfaced inline with the exact passage highlighted, so verifying takes seconds rather than a separate Westlaw lookup.
Verification is the floor, not the ceiling. We invest more in catching failure modes than any other legal-AI tool we have benchmarked.

Mata v. Avianca and lawyer responsibility

Mata v. Avianca (S.D.N.Y. 2023) and the sanctions orders that have followed it are a warning to every lawyer using generative AI. Lawyers were sanctioned for filing briefs that cited cases the AI had fabricated. Vaquill AI's verification pipeline is specifically designed to prevent this outcome, Layer 2 would flag a fabricated citation as unresolved, but the professional responsibility does not transfer from the lawyer to the tool.

Every lawyer using Vaquill AI remains personally responsible under ABA Model Rule 1.1 (competence), Rule 5.3 (supervision of non-lawyer assistance), Rule 3.3 / Rule 11 (candor to the tribunal), and the corresponding state bar rules, for verifying each citation and each factual assertion in any filing, advice, or communication that relies on Vaquill AI output.

Independent research & audits

External red-teaming & adversarial prompt testing

Coming Soon

We continuously red-team our own pipeline against prompt-injection, jailbreaks, and adversarial retrieval attacks. A formal third-party engagement is on our roadmap; vendor, scope, and summary findings will be published on this page when complete.

Security pen-testing

Infrastructure and application-layer security controls, incident response, and our SOC 2 roadmap are documented on /security.

Bug bounty & vulnerability disclosure

Our responsible-disclosure policy and contact channel are published on /security. Accuracy-related issues (suspected hallucinations, invalid citations, verification-layer misses) can be reported to the same channel or directly to contact@vaquill.ai.

Frequently asked questions

Is Vaquill AI benchmarked?

Yes, and openly. Vaquill AI publishes a reproducible benchmark on US law that changed in the last two years: a retrieval-grounded engine answered correctly 93% of the time versus 33% for the same model with no sources, and cited the controlling authority 97% versus 0%. The questions, scoring code, and raw answers are on GitHub under an open license, so anyone can re-run it or point it at another tool.

Does Vaquill AI have independent, third-party benchmarks?

Vaquill AI's own benchmark is open and reproducible today, which is stronger than an unverifiable internal stat because anyone can re-score it. Independent third-party benchmarking (Vals AI, LegalBench) is on the roadmap and documented on this page. The open benchmark is the honest interim: it invites the scrutiny a private number avoids.

How accurate is Vaquill AI?

On the open benchmark of recently-changed US law, Vaquill AI answered correctly 93% of the time with grounding. Every answer also passes a 4-layer citation verification pipeline (exact-text match, citation resolution, meaning analysis, and an AI cross-check) before it reaches the lawyer, and anything a layer flags is labeled so the lawyer sees it. Lawyer review is still required under ABA Formal Opinion 512.

Can I reproduce the Vaquill AI benchmark myself?

Yes. The questions, the scoring script, and the raw answers with retrieved source text are all committed to GitHub at github.com/Vaquill-AI/open-legal-answer-benchmark under an open license. Clone it, re-score it, and you get the same number, or run it against a competitor for a like-for-like comparison.

Does Vaquill AI support complex, agentic workflows?

Yes. Vaquill AI runs an agent that plans a task and chains statute lookup, citation verification, counterparty screening, playbook review, and drafting in one run, plus multi-step Workflows and parallel deep research that decomposes a question into dimensions. It is built for multi-step work, not only single-shot answers.

How does Vaquill AI prevent hallucinations?

Through defense in depth: retrieval grounding in a real US corpus, a 4-layer citation verifier scored by its weakest layer, deterministic backstops where correctness is non-negotiable, and calibrated uncertainty that flags low-confidence answers. The approach is documented publicly rather than asserted.

Accuracy does not stand alone. These pages cover the surrounding context.

/security , data residency, encryption, tenant isolation, SOC 2 roadmap.
/for-lawyers/ethics , lawyer compliance obligations under ABA Model Rules 1.1, 1.6, 5.3, 3.3, and state-bar guidance on generative AI.
/privacy , how evaluation data, prompts, and outputs are handled.

Questions about methodology?

Email contact@vaquill.ai. We will walk through our evaluation procedure, sample composition, or verification-layer internals in more detail than we publish here.

Lawyer-review disclaimer

Per ABA Formal Opinion 512, AI-generated output must be independently reviewed by a licensed lawyer before use in a legal matter. Vaquill AI is a research and drafting assistant for licensed lawyers, not a substitute for professional legal judgment, and using Vaquill AI does not create an attorney-client relationship between Vaquill AI and any end user.

FTC substantiation disclaimer

Any accuracy metrics published on this page are based on Vaquill AI's internal evaluation methodology. Evaluations are re-run with each major model change; current numbers reflect our most recent evaluation as of the date shown on this page. Methodology details, including sample size, sample composition, judge qualifications, and model version evaluated, are available on request.

Last updated: April 16, 2026