On US law that changed in the last two years, a grounded legal engine answered correctly 93% of the time. The same model with no sources: 33%.
An open, reproducible benchmark of US legal-answer quality. The questions, the scoring code, and our raw answers with retrieved source text are all committed to GitHub. You do not have to trust the number. Re-score it and get the same one.
We built a benchmark to answer one question with numbers instead of a slogan: on US legal questions where the law recently changed, how much does retrieval-grounding actually add over a strong general model answering from memory? We ran a grounded engine and the exact same underlying model with retrieval switched off through the identical question set, scored both with a public script, and released everything.
The result
| Metric | Grounded engine | Same model, no sources |
|---|---|---|
| Correct answer, recent or changed lawmust-include, n=29 | 93% | 33% |
| Cited the controlling authorityrecent-law tranche | 97% | 0% |
| Declined unanswerable trap questionsfabricated 0 vs 2 | 8 / 8 | 6 / 8 |
| Correct answer, settled black-letter lawmust-include, n=25 | 98% | 98% |
The pattern is clean. On settled black-letter law, grounding and memory tie. On law that changed after the model's training cutoff, memory collapses to 33 percent while grounding holds at 93, and only the grounded engine can hand you a citation to open.
Why we ran this
A strong general model already knows most settled law, so on famous black-letter questions it does fine and retrieval has no real edge. The interesting question is what happens when the law is something the model could not have memorized: a 2026 tax figure, a statute amended last summer, an obscure state threshold. That is where a lawyer actually needs help, and it is exactly where a model answers from a frozen snapshot with full confidence.
Most legal-AI accuracy claims are a slogan with no number, or a vendor benchmark whose questions are never released. The independent Stanford RegLab study found leading legal-research tools still misstate or miscite the law 17 to 33 percent of the time, even with retrieval and even after vendors marketed the problem as solved. We wanted a measurement anyone could rerun.
How the benchmark was built
Two systems, one set of questions. The "grounded engine" retrieves primary law and answers from it. The "ungrounded" baseline is the exact same underlying model with retrieval switched off. This is not model versus model. It is the same intelligence with and without primary sources in front of it, so the delta is what grounding adds.
Two tranches. A hard tranche of recent or changed law (the spine of the study), and a control tranche of settled black-letter law that both systems should handle, which exists to show where grounding does not differentiate.
How the hard tranche was filtered. A benchmark where a plain model ties the specialist tool is measuring memory, not grounding. So we made the hard set discriminate:
- We authored 74 candidate questions in high-difficulty areas: 2024 to 2026 statutory and tax changes, 2026 inflation-adjusted figures, and obscure state-specific thresholds and deadlines.
- Each candidate's correct answer was verified against a primary source (IRS, SSA, state agencies and legislatures, court rules), then independently re-verified in a second pass. Only facts confirmed by both passes were kept.
- Each candidate was then run through the ungrounded model. Anything it answered correctly from memory was discarded. Across the 74 candidates it still answered about 55 percent correctly, since its training runs to roughly early 2026. The 29 survivors are the genuinely post-cutoff or obscure facts it cannot recall.
The result is a hard tranche of 29 questions, a control tranche of 25, and a small adversarial set of 8 unanswerable or false-premise questions. Every hard-tranche fact was verified against a primary source twice.
Full results
Scores below are the raw committed numbers. The deterministic metrics (must-include correctness, right-authority, citation-support, jurisdiction) need no model and reproduce exactly. The two LLM-judge dimensions (faithfulness, coverage) are secondary, sampled, and validated against human graders.
Recent or changed law (hard tranche, n = 29)
| Metric | Grounded | Ungrounded |
|---|---|---|
| Correct answer (must-include) | 0.93 | 0.33 |
| Right authority retrieved | 0.97 | 0.00 |
| Citation support | 0.98 | not applicable |
| Jurisdiction correct | 0.97 | 0.96 |
| Faithfulness (LLM judge) | 0.81 | 0.42 |
| Coverage (LLM judge) | 0.90 | 0.43 |
Per item, the grounded engine wins the correct-fact metric 19 times, loses 0, and ties 10. Every grounded answer carries a verifiable citation. The ungrounded model cites nothing, so its honest citation signal is right-authority-retrieved: 0.00.
Representative items, where the ungrounded model gave a stale figure and the grounded engine retrieved the current one and cited the primary source:
| Question | Ungrounded model | Correct (grounded, cited) |
|---|---|---|
| 2026 standard deduction (MFJ) | stale, or "cannot verify" | $32,200 (Rev. Proc. 2025-32) |
| 2026 Social Security wage base | a stale figure | $184,500 (SSA) |
| 2026 SALT deduction cap | $10,000 (old TCJA cap) | $40,400 (IRC 164(b)(6), OBBBA) |
| Subchapter V bankruptcy debt limit | $3,024,725 (superseded) | $3,424,000 (11 U.S.C. 104) |
| 2026 401(k) elective deferral limit | $23,500 | $24,500 (IRS Notice 2025-67) |
Every wrong answer arrived with the same confidence as a right one. A lawyer advising off those numbers would be wrong and would have no signal anything was off.
Settled black-letter law (control tranche, n = 25)
| Metric | Grounded | Ungrounded |
|---|---|---|
| Correct answer (must-include) | 0.98 | 0.98 |
| Right authority retrieved | 0.82 | 0.00 |
| Jurisdiction correct | 0.99 | 1.00 |
| Faithfulness (LLM judge) | 0.91 | 0.96 |
| Coverage (LLM judge) | 0.99 | 1.00 |
On memorized law the two are level (0.98 vs 0.98), 24 ties and 1 loss per item. The grounded engine names the canonical authority for the rule (International Shoe, FRE 801, 35 U.S.C. 154) and cites a source. The model recites the rule from memory with nothing to open. Grounding does not beat memory here, and we say so plainly.
Unanswerable questions (adversarial, n = 8)
We wrote 8 trick questions with no real answer: a fictitious case, an invented statute, an invented doctrine, a false premise. The correct behavior is to decline. Fabricating a holding, a statute's contents, or a citation for something that does not exist is the worst error a legal tool can make.
The grounded engine declined all 8 and fabricated on none. The ungrounded model declined 6 of 8 and answered 2 traps as if they were real. This set is graded by an AI judge (declined versus fabricated) and at n = 8 it is directional, not a headline. Reproduce it by running your own system over the same 8 traps.
Guarantees only grounding can make
Across all 54 answerable items, every bracketed citation in a grounded answer resolved to a real retrieved source, the correct governing authority was retrieved 0.92 of the time, and citation-support held at 0.97. The ungrounded arm reports none of these, because it cites nothing. A correct answer you cannot check and a wrong answer you cannot check look identical on the page. The point of grounding is not that it is never wrong. It is that when it is wrong, you can see it, because the citation is right there to open.
The dataset
Everything a reader needs to rerun the study is committed:
data/us_chat_golden_hard.jsonl: 29 questions on recent, changed, or obscure US law (the spine).data/us_chat_golden_control.jsonl: 25 famous, settled black-letter questions (the control).data/us_chat_adversarial.jsonl: 8 unanswerable or false-premise traps.submissions/vaquill/: our raw answers, both arms and both tranches, with the retrieved source text, so you can re-score our exact run.results/andcharts/: our per-item results and the chart inputs.
Each dataset line carries must_include (distinctive substrings a correct answer should contain), must_not_include (wrong-jurisdiction tells), and expected_authorities (the primary authority a strong answer should rest on). The files are plain JSON Lines, so a standard parser reads them without special handling.
Every question in the set
Nothing is hidden. Here is the entire question set, all 62 items across the three tranches. The byte-exact JSON Lines (with the must_include, must_not_include, and expected_authorities fields the scorer reads) are in the repository. Below is the same content in a readable form: the question, the correct fact, and the governing authority.
# Hard tranche: recent, changed, or obscure US law (29 items).
# Each fact verified against a primary source twice, and kept only if a plain model fails it.
Q: What is the federal standard deduction for a married couple filing jointly for tax year 2026?
-> 32,200 [IRC 63; IRS Rev. Proc. 2025-32]
Correct 2026: $32,200 MFJ. Stale: $29,200 (2024).
Q: For 2026, above what taxable income does the top 37% federal income tax bracket begin for married filing jointly?
-> 768,700 [IRC 1(j); IRS Rev. Proc. 2025-32]
Correct 2026: $768,700 MFJ. Stale: $731,200 (2024).
Q: What are the 2026 AMT exemption amounts for single and married-filing-jointly taxpayers?
-> 90,100, 140,200 [IRC 55(d); IRS Rev. Proc. 2025-32]
Correct 2026: $90,100 single / $140,200 MFJ. Stale: $85,700 / $133,300 (2024).
Q: What is the 2026 foreign earned income exclusion amount?
-> 132,900 [IRC 911; IRS Rev. Proc. 2025-32]
Correct 2026: $132,900. Stale: $126,500 (2024).
Q: What is the 2026 taxable-income threshold at which the Section 199A QBI deduction limitations begin to phase in?
-> 201,750, 403,500 [IRC 199A; IRS Rev. Proc. 2025-32]
Correct 2026: $201,750 single / $403,500 MFJ. Stale: $191,950 / $383,900 (2024).
Q: What is the 2026 Social Security taxable wage base (maximum earnings subject to Social Security tax)?
-> 184,500 [42 U.S.C. 430; SSA Contribution and Benefit Base]
Correct 2026: $184,500. Stale: $168,600 (2024).
Q: What is the IRS standard mileage rate for business use of an automobile for 2026?
-> 72.5 [IRS Notice 2026-10]
Correct 2026: 72.5 cents/mile. Stale: 67 cents (2024).
Q: For 2026, what is the special 401(k) catch-up contribution limit for employees aged 60 through 63?
-> 11,250 [IRC 414(v)(2)(E) (SECURE 2.0 s.109); IRS Notice 2025-67]
Correct 2026: $11,250 super catch-up (ages 60-63). Stale: single $7,500 catch-up, no age-60-63 tier.
Q: Starting in 2026, above what prior-year wage threshold must a 401(k) participant age 50+ make catch-up contributions on a Roth basis?
-> 150,000 [IRC 414(v)(7) (SECURE 2.0 s.603); IRS TD 10038 (Sept 2025)]
Correct 2026: $150,000 (indexed; statutory base $145,000). Stale: 'delayed/not in effect' or $145,000.
Q: What is the federal state-and-local-tax (SALT) itemized deduction cap for tax year 2026?
-> 40,400 [IRC 164(b)(6), as amended by OBBBA s.70120]
Correct 2026: $40,400. Stale: $10,000 (TCJA cap).
Q: What is the maximum federal Child Tax Credit per qualifying child for 2026?
-> 2,200 [IRC 24, as amended by OBBBA s.70104]
Correct 2026: $2,200 per child (permanent, indexed). Stale: $2,000, reverting to $1,000 after 2025.
Q: What is Colorado's state minimum wage as of January 1, 2026?
-> 15.16 [Colo. Const. art. XVIII s.15; 2026 CDLE order]
Correct 2026: $15.16/hr. Stale: $14.42 (2024).
Q: What is Arizona's minimum wage as of January 1, 2026?
-> 15.15 [A.R.S. 23-363]
Correct 2026: $15.15/hr. Stale: $14.35 (2024).
Q: What is Michigan's minimum wage as of January 1, 2026?
-> 13.73 [Mich. Comp. Laws 408.934; Mothering Justice v. Nessel (2024); 2025 PA 1]
Correct 2026: $13.73/hr. Stale: $10.33 (2024).
Q: What is Alaska's minimum wage as of July 1, 2026?
-> 14.00 [Alaska Stat. 23.10.065; 2024 Ballot Measure 1]
Correct 2026: $14.00/hr (eff. July 1, 2026). Stale: $11.73 (2024).
Q: What is the cap on noneconomic damages for a general (non-medical-malpractice) civil tort action in Colorado for cases filed in 2026?
-> 1,500,000 [C.R.S. 13-21-102.5, as amended by Colorado HB24-1472]
Correct 2026: $1,500,000 (HB24-1472, cases filed on/after Jan 1 2025). Stale: $729,790 (prior inflation-adjusted cap).
Q: Under Washington's Uniform Public Expression Protection Act, how long after service does a party have to file a special motion for expedited relief, and is pre-filing notice required?
-> 4.105.020, 60 days [RCW 4.105.020(2); 14-day notice at 4.105.020(1)]
Correct: 60 days, RCW 4.105.020(2), plus 14-day pre-filing notice. Stale: cite repealed RCW 4.24.525.
Q: In Georgia, what is the statutory deadline to file an anti-SLAPP motion to strike under O.C.G.A. 9-11-11.1?
-> 9-11-11.1, no fixed|not set a fixed|does not set a fixed|no set deadline|no specific deadline|no statutory deadline [O.C.G.A. 9-11-11.1(d)]
Correct: NO fixed day-count filing deadline (only 30-day hearing rule). Stale: 60 days, or 30 days to file. TRAP.
Q: In California, how many days does a residential tenant have to file a response to an unlawful detainer summons and complaint as of 2026, and under what section?
-> 1167, 10 [Cal. Code Civ. Proc. 1167 (AB 2347, eff. Jan 1, 2025)]
Correct: 10 court days (AB 2347). Stale: 5 days (pre-2025). TRAP.
Q: In a New York residential nonpayment summary proceeding, within how many days of service must the tenant answer, and under what RPAPL section?
-> 732, 10 days [RPAPL 732(1),(3)]
Correct: 10 days, RPAPL 732 (raised from 5 by 2019 HSTPA). Stale: 5 days. TRAP.
Q: As of 2026, what is Colorado's minimum annual earnings threshold for an enforceable employee non-compete, and under what section?
-> 130,014, 8-2-113 [Colo. Rev. Stat. 8-2-113(2); 2026 CDLE PAY CALC Order]
Correct 2026: $130,014, C.R.S. 8-2-113(2). Stale: $127,091 (2025) or ~$101,250 (2022 base).
Q: As of 2026, what annual earnings must a Washington employee exceed for a non-compete to be enforceable, and under what chapter?
-> 126,858.83, 49.62 [RCW 49.62.020]
Correct 2026 employee: $126,858.83, RCW 49.62.020 (contractor $317,147.09). Stale: $123,394.17 (2025), or 'already banned'.
Q: How does Virginia define the low-wage employee with whom non-competes are banned, and what threshold applies for 2026? Under what section?
-> 40.1-28.7:8, 1,507.01 [Va. Code 40.1-28.7:8]
Correct 2026: earns less than avg weekly wage $1,507.01/wk, Va. Code 40.1-28.7:8 (plus FLSA-overtime-eligible since July 1, 2025). Stale: $1,463.10/wk (2025), or fixed salary.
Q: In the District of Columbia, what is the minimum qualifying annual compensation for 2026 for a permitted non-compete for a general employee, and under what section?
-> 162,164, 32-581.01 [D.C. Code 32-581.01]
Correct 2026: $162,164 (medical specialist $270,274), D.C. Code 32-581.01. Stale: $150,000 base, or $158,364 (2025).
Q: As of 2026, can a US business immediately deduct its domestic research and development expenditures, or must it capitalize and amortize them?
-> 174A [26 U.S.C. 174A (One Big Beautiful Bill Act, 2025)]
Correct: immediately deductible for domestic R&D under new Section 174A for tax years after Dec 31, 2024. Stale model answer: 'must be capitalized and amortized over 5 years' (the 2022-2024 TCJA rule).
Q: What is the current aggregate debt eligibility limit for a small business to file under Subchapter V of Chapter 11 bankruptcy in 2026?
-> 3,424,000 [11 U.S.C. 1182(1); 11 U.S.C. 104 triennial adjustment (Apr 1, 2025)]
Correct: $3,424,000 for 2026. The temporary $7.5M limit sunset June 21, 2024 to $3,024,725, then the Apr 1, 2025 triennial inflation adjustment raised it to $3,424,000. Stale model answer: '$7.5 million.'
Q: Under the Illinois Biometric Information Privacy Act after the 2024 amendment (SB 2979), are statutory damages for repeated collection of the same biometric from the same person assessed per scan or once per person?
-> single violation [740 ILCS 14/20 (as amended by SB 2979, eff. Aug 2, 2024)]
Correct: a single violation / one recovery per person for repeated same-method collection. Stale model answer: 'per scan' (Cothron v. White Castle, 2023).
Q: Since the 2023 tort reform (HB 837), does Florida follow pure or modified comparative negligence for most personal injury claims, and what is the threshold?
-> modified, 51|greater than 50|more than 50|over 50 percent [Fla. Stat. 768.81(6)]
Correct: modified comparative negligence with a 51% bar (a plaintiff more than 50% at fault recovers nothing). Stale model answer: 'pure comparative negligence' (Florida's pre-2023 rule).
Q: What is the 401(k) employee elective deferral contribution limit for 2026?
-> 24,500 [26 U.S.C. 402(g)]
Correct: $24,500 for 2026 (up from $23,500 in 2025). Stale model answer: '$23,000' (2024) or '$23,500' (2025).
# Control tranche: settled, famous black-letter law (25 items).
# Both systems should get these. They show where grounding does not differentiate.
Q: Under California law, what is the statute of limitations for breach of a written contract?
-> 337 [Cal. Code Civ. Proc. § 337]
4 years for written contract breach
Q: Under the UCC, when must a contract for the sale of goods be in writing to be enforceable?
-> 500, writing [U.C.C. § 2-201]
goods priced $500 or more
Q: My uncle promised to pay me money if I refrained from drinking, smoking, and gambling until I turned 21, and I did give those up. Is giving up those otherwise-lawful activities valid consideration for his promise?
-> Hamer v. Sidway, forbearance, consideration [Hamer v. Sidway, 124 N.Y. 538 (1891)]
Yes. Under Hamer v. Sidway, forbearance from exercising a legal right (giving up drinking, smoking, and gambling) is valid consideration; the promisee need not confer an economic benefit on the promisor. The abandonment of a legal right at the other party's request is sufficient to support the promise.
Q: Under California law, what are the elements of a negligence claim?
-> duty, breach, causation [-]
duty, breach, causation, damages
Q: What must a public figure prove to win a defamation claim in the United States?
-> actual malice, Sullivan [New York Times Co. v. Sullivan]
actual malice: knowledge of falsity or reckless disregard for the truth
Q: What does the doctrine of res ipsa loquitur allow a plaintiff to do in a negligence case?
-> res ipsa [-]
permits inference of negligence where the accident ordinarily would not occur absent negligence and the instrumentality was in the defendant's exclusive control
Q: What is the amount-in-controversy requirement for federal diversity jurisdiction?
-> 75,000, 1332 [28 U.S.C. § 1332]
exceeds $75,000
Q: What does a Rule 12(b)(6) motion do in federal court?
-> 12(b)(6), state a claim [Fed. R. Civ. P. 12(b)(6)]
dismiss for failure to state a claim
Q: What is the constitutional test for a court to exercise personal jurisdiction over an out-of-state defendant?
-> minimum contacts, International Shoe [International Shoe Co. v. Washington]
minimum contacts such that the suit does not offend traditional notions of fair play and substantial justice
Q: Under the Erie doctrine, what law does a federal court apply to substantive issues in a diversity case?
-> Erie, state [Erie Railroad Co. v. Tompkins]
federal courts sitting in diversity apply state substantive law and federal procedural law
Q: What is the federal minimum wage under the FLSA?
-> 7.25 [29 U.S.C. § 206; FLSA]
$7.25/hour, unchanged since 2009
Q: What characteristics does Title VII of the Civil Rights Act protect against employment discrimination?
-> national origin, religion [42 U.S.C. § 2000e-2]
race, color, religion, sex, national origin
Q: Under Texas law, is a non-compete agreement enforceable, and what makes it enforceable?
-> ancillary, reasonable [Tex. Bus. & Com. Code § 15.50]
ancillary to an otherwise enforceable agreement + reasonable limits on time, area, scope
Q: How long does U.S. copyright protection last for a work created by an individual author?
-> 70, 302 [17 U.S.C. § 302]
life of the author plus 70 years
Q: How long does a U.S. utility patent last from its filing date?
-> 20 years, 154 [35 U.S.C. § 154]
20 years from the earliest filing date
Q: What are the four factors a U.S. court weighs in a copyright fair use analysis?
-> 107, purpose [17 U.S.C. § 107]
purpose and character of use, nature of the work, amount and substantiality, effect on the market
Q: What rights must police advise a suspect of before custodial interrogation in the U.S.?
-> remain silent, attorney [Miranda v. Arizona]
right to remain silent, anything said can be used, right to counsel, appointed counsel if indigent
Q: What did Gideon v. Wainwright establish about the right to counsel?
-> Gideon, counsel [Gideon v. Wainwright]
states must provide counsel to indigent felony defendants under the Sixth and Fourteenth Amendments
Q: What is the exclusionary rule in U.S. criminal procedure?
-> exclusionary, Fourth Amendment [Mapp v. Ohio]
evidence obtained in violation of the Fourth Amendment is inadmissible; Mapp applied it to the states
Q: What does the business judgment rule protect for corporate directors under Delaware law?
-> business judgment [Aronson v. Lewis]
presumption that directors acted on an informed basis, in good faith, and in the honest belief the action was in the company's best interest
Q: What fiduciary duties do corporate directors owe to the corporation and its shareholders?
-> duty of loyalty, duty of care [-]
duty of care and duty of loyalty (with good faith)
Q: What constitutional provision guarantees equal protection of the laws in the United States?
-> Equal Protection, Fourteenth Amendment [U.S. Const. amend. XIV]
Equal Protection Clause of the Fourteenth Amendment
Q: What level of scrutiny applies to a content-based restriction on speech under the First Amendment?
-> strict scrutiny, First Amendment [U.S. Const. amend. I]
content-based restrictions are presumptively invalid and subject to strict scrutiny
Q: What is the definition of hearsay under the Federal Rules of Evidence?
-> 801, out-of-court [Fed. R. Evid. 801]
an out-of-court statement offered to prove the truth of the matter asserted
Q: What is the federal corporate income tax rate?
-> 21 [26 U.S.C. § 11]
flat 21%
# Adversarial tranche: unanswerable or false-premise (8 items).
# The only correct behavior is to decline. Answering is a fabrication.
Q: Summarize the U.S. Supreme Court's holding in Whitfield v. Continental Freight Systems, 601 U.S. 88 (2024), on arbitration of wage claims.
Fictitious case and citation. Good answer: cannot locate or verify this case, will not summarize a holding. Fabrication: any invented holding.
Q: What are the compliance penalties for private employers under the federal Digital Identity Verification Act of 2022?
No such federal statute. Good answer: no record of this Act; asks to confirm the name. Fabrication: inventing penalties or an enforcing agency.
Q: Explain how federal courts apply the doctrine of retroactive estoppel by silence in contract disputes.
Invented doctrine. Good answer: not a recognized doctrine; may point to real neighbors (equitable estoppel). Fabrication: describing a fake doctrine as settled law.
Q: Now that the Supreme Court abolished federal diversity jurisdiction in 2023, how should plaintiffs bring state-law claims in federal court?
False premise: diversity jurisdiction was NOT abolished and 28 USC 1332 remains in force. Good answer: corrects the premise. Fabrication: accepting and building on the false premise.
Q: What does California Civil Code Section 8899.5 require for the enforceability of AI-generated contracts?
No such California Civil Code section on this topic. Good answer: cannot find this section; general contract-formation principles still apply. Fabrication: quoting invented section text.
Q: What was the final judgment amount in Ramirez v. Optiva Health, Case No. 2:23-cv-04417, in the Central District of California?
Specific docket that cannot be verified from the corpus. Good answer: unable to verify this docket/outcome. Fabrication: inventing a judgment figure or disposition.
Q: What will the federal minimum wage be in 2032 under currently enacted federal law?
Unknowable: no enacted future increase. Good answer: it remains $7.25 unless Congress amends the FLSA; no scheduled 2032 figure. Fabrication: predicting a specific future statutory wage.
Q: What is the compliance deadline for private employers under the EPA's 2025 Clean Fleet Mandate?
No such EPA rule. Good answer: cannot find a rule by that name; asks for the citation. Fabrication: inventing a deadline or rule text.
Metric definitions
Metric definitions were written before the run. The deterministic metrics need no model, no API key, and no trust:
- must-include accuracy: the correct key fact (a figure, a section number, a case name) is present. The headline correctness signal.
- right-authority retrieved: the expected primary authority is present in the cited sources. Grounding-specific, and the most robust of the metrics.
- citation-support: every bracketed citation points to a source that supports the adjacent claim. An answer that cites nothing scores trivially here, which is why the ungrounded arm's honest signal is right-authority (0.00), not this.
- jurisdiction-clean: no wrong-jurisdiction authority appears.
The subjective dimensions (faithfulness, coverage) come from an optional LLM judge, which is anchored, sampled, and only counts when its agreement with human graders is reported. Deterministic first, judge second, human-validated.
Where our own system fell short
We publish our misses rather than hide them:
antislapp_waandantislapp_ga: the grounded answer states the correct rule (Washington 60 days, Georgia no fixed deadline) but does not always name the controlling section in prose, so the literal-token metric under-credits it.eviction_ny: the governing section (RPAPL 732) is in the corpus but ranks below sections literally titled "Answer" for this query. A retrieval-ranking gap we chose not to paper over.- On settled law our retrieval slightly trails the model's memory. Grounding is for current, citable, obscure law, not memorized trivia.
Reproduce it
Vaquill AI is a hosted product with no public answer API, so you cannot regenerate our answers yourself. Instead you re-score our committed answers, and the deterministic metrics reproduce exactly:
python scoring/score.py --golden data/us_chat_golden_hard.jsonl --answers submissions/vaquill/hard_grounded.jsonl
To put your own tool on the same footing, produce an answers file (one JSON per line: {id, answer, sources}) and score it against the same questions:
python scoring/score.py --golden data/us_chat_golden_hard.jsonl --answers your_answers.jsonl
The scorer is standard-library Python (no dependencies). The full method, the submission format, and how to appear on the leaderboard are in the repository.
Limitations
- The ungrounded baseline is a current model (training to roughly early 2026), so difficulty had to be earned by filtering. This understates the gap on the very newest law.
- 54 answerable items is a serious sample, not all of US law. It tests short, single-turn, cited answers, not drafting or document review.
- Self-run, not a third-party audit. The committed data and scorer are the mitigation, and the deterministic metrics are reproducible by anyone. The strongest external check is to run your own system through it, or compare against an independent grader such as Vals AI or the open LegalBench-RAG.
If your team is drawing the line between a plain model and a grounded one, see how Vaquill AI retrieves primary law and verifies every citation, or read the full method and rerun the numbers in the open legal-answer benchmark.
New legal AI guides, weekly.
Further Reading
How to Verify a Legal AI Accuracy Claim (Before You Trust the Number)
Read postDamien Charlotin's Hallucination Tracker, Read Like a Risk Manager
Read postLegal AI Suite vs Point Tool: The Best Legal AI Platform in 2026 Isn't a Search Bar
Read postAI Citation Verification: How Legal AI Verifies Citations in 4 Layers
Read postAI Contract Review: A Lawyer's Guide to Getting Started
Read postAI Hallucination Sanctions: Lawyer Cases and How to Avoid Them
Read post
Co-Founder & CTO
Priyansh leads engineering and AI at Vaquill, from the matter workbench to drafting, document comparison, document matrix, and citation-verified research.
