Fetching a state statute page is an afternoon of work. Proving that the text you stored is the text the publisher actually published, and being able to answer six months later what happened to a document you expected and did not get, is the part that never ends. That gap is the difference between a scraper and a corpus, and it is almost entirely made of gates and audit records. This post walks the whole path, layer by layer, with the real module names from an open-source pipeline that carries 4,150,839 sections across 53 jurisdictions. If you are deciding whether to build this yourself, the honest answer is at the end.
TL;DR
- A scraper is easy. A corpus is not. The hard part is not the fetch, it is proving what you stored matches what the publisher published.
- Eleven layers sit between a government web page and an API row: discovery, fetch, extraction, mirroring, identity, enrichment, chunking, payload build, embedding, upsert, reconcile. Each has one specific way it goes wrong.
- Nine named gates default to FAIL. A gate that defaults to pass will wave through a bad run on the exact day it matters most, so refusing to publish is the correct default.
- Ledgers beat logs. A per-candidate audit record is the only thing that can answer "the publisher's index lists 520 documents and we serve 480, so what happened to the other 40."
- Two rules learned the hard way: a fetch failure must never become a deletion, and a corrective re-scrape can mint new identifiers for content you already had.
- Char and page offsets into the publisher's own stored artifact are what turn a provenance claim into something a reader can check.
What should a validation gate in a legal ingest pipeline do by default?
This post opens our legal data infrastructure series.
The pipeline is where provenance is manufactured, so read it next to Legal Data Provenance: How to Tell Where a Statute Actually Came From, and then What a Legal Data API Should Guarantee: The Schema Contract for the shape it finally emits.
The honest framing
Nobody who has built one of these thinks the fetch is the problem. HTTP is solved. The problem is that a legal corpus makes a strong claim on every single row: this text is what this government publisher published, at this citation, as of this currency statement.
That claim has to survive a publisher redesigning a page without telling anyone, a PDF whose text layer came off a scanner two decades ago, and a run that returned an error page with HTTP 200 and a body that parsed cleanly into empty sections.
None of those failures announce themselves. They look like a successful run with a slightly different document count, which is why the count is the only thing most pipelines check and why most pipelines are wrong in ways nobody has noticed yet.
The eleven layers
Here is the whole path, stage by stage, with the failure each stage exists to catch.
| Layer | What it does | How it goes wrong |
|---|---|---|
| 1. TOC discovery | Walks the publisher's index to enumerate every document that should exist. | A lazy-loading or paginated index ends early, and the run reports success on a partial enumeration. |
| 2. Fetch | Pulls each document from the publisher with retries and backoff. | A bot wall returns HTTP 200 with a challenge page, so the retry logic sees success and stores it. |
| 3. Extraction | Pulls the legal text out of HTML or PDF, dropping navigation and page chrome. | Site chrome leaks into section bodies, or a PDF with a bad text layer yields plausible-looking garbage. |
| 4. Mirror | Stores the publisher's own artifact byte for byte, in every format they offer. | The pipeline mirrors its own extracted text instead of the publisher's file, so the provenance chain points at itself. |
| 5. Identity | Assigns the stable actId and point identifiers that everything downstream keys on. | Identity derived from position in the publisher's tree changes when the publisher renumbers, minting a duplicate. |
| 6. Enrichment | Parses amendment credits, effective dates, cross references and currency statements out of the text. | A four-digit number that is not a year gets read as one, and a section starts asserting an amendment that never happened. |
| 7. Chunking | Splits long sections into retrieval passages while keeping the citation intact. | A pathological document explodes into thousands of chunks and swamps the run. |
| 8. Payload build | Constructs the served object against a declared schema. | A field renamed upstream lands as null on every row and nothing errors, because null is a legal value. |
| 9. Embed | Generates dense and sparse vectors and writes them to the vector store. | A write is accepted and then discarded while the store is mid-optimize, so the point is missing but the run says completed. |
| 10. Upsert | Writes payload plus vectors under content-addressed ids. | A resume flag skips a point that already exists, so a corrective run keeps the old broken payload. |
| 11. Reconcile | Compares what came back against what is already stored, and retires what the publisher removed. | A partial fetch looks like a mass deletion, and the corpus deletes real law. |
That last row is the one that keeps people up at night, and it gets its own rule further down. Reconcile is also where the customer-visible output of the pipeline gets made, because a difference it finds becomes a change event, which is the subject of How We Know a Law Changed.
The gates, and why they default to FAIL
Nine named gates sit between a finished run and a published corpus. Each one refuses to publish rather than publish something wrong.
| Gate | What it refuses to let through |
|---|---|
| Identity check | A section whose identifier does not agree with its own citation and hierarchy. Prevents a run from serving Texas Property Code text under a Transportation Code identifier. |
| Text quality | Text that fails basic sanity: empty bodies, mojibake, encoding damage, page furniture where legal text should be. |
| Truncation | A section whose stored text ends before the publisher's does. The quiet one, because a truncated section reads perfectly until the missing clause is the one that mattered. |
| Chunk cap | A document that produced an implausible number of retrieval passages, which usually means the splitter misread the structure. |
| Excluded material | Content that should never enter the corpus, such as commercial annotations or editorial matter sitting beside the government text. |
| Third-party render | Output from a rendering service or intermediary rather than the publisher, so an intermediary's view is never mistaken for the official one. |
| Identifier guard | A newly minted identifier that collides with, or degrades, one already in service. |
| Publisher check | A run whose bytes did not come from the publisher we claim they came from. |
| Record preflight | A malformed or schema-violating record, checked before a single vector is generated. |
The list matters less than the default. A gate that defaults to pass is not a gate. It is a log line with ambition. On an ordinary day it changes nothing, and on the one day the publisher shipped a broken export it does exactly what a missing gate would have done: nothing.
Defaulting to fail costs real time. Runs stop, someone has to look at why, and that cost is the product. A corpus that publishes a bad run on Tuesday and fixes it on Friday served wrong law for three days, and every customer who cached a response still has it.
The reconcile guard is the highest-stakes rule in the system, and it is a few lines:
DROP_CEILING = 0.02 # fraction of a jurisdiction a single run may retire
def reconcile(fetched: set, stored: set):
missing = stored - fetched
if len(missing) > len(stored) * DROP_CEILING:
raise RunHalted(
f"run proposes retiring {len(missing)} of {len(stored)} sections; "
"treating as a failed fetch, not a publisher deletion"
)
return missing # only now is retirement allowed
Fail closed. A source that times out returns fewer documents, and a reconcile step without this guard reads that as removal and deletes real law.
The ledgers: why an audit record beats a log line
A gate tells you a run failed. A ledger tells you what happened to one specific document. Four audit records run alongside the pipeline:
- A candidate ledger records what each candidate document became, kept or discarded and why, keyed to its source URL.
- A structural ledger records the shape a run found, so a later run can tell "the publisher removed this" from "our parser stopped seeing it."
- A repair ledger records every corrective write applied to stored data, so a repair cannot be silently re-applied or silently undone.
- A write receipt records what a run actually wrote, which is not the same as what it intended to write.
Here is the question that motivates all four. Say the publisher's index lists 520 bulletins for a state and your API serves 480. Where did the other 40 go?
A log file cannot answer this. Logs are keyed by time and they rotate. To answer from logs you would need to know which run touched those 40 documents, that the run still has logs, and what to grep for, which requires already knowing the answer.
A candidate ledger answers it as a join. Take the publisher's URLs, join against the ledger on source URL, and every missing row comes back with a disposition: kept, or discarded with a reason. Some are duplicates of documents already served under a different URL. Some are index or container pages that were never documents. Some failed extraction and are queued for repair. Some the publisher withdrew. That is an answer. "We do not know" is not.
The distinction generalizes past legal data. Logs record what the program did. Ledgers record what happened to each item, and if your users will ever ask about one specific item, you need the second kind written at the time, because it cannot be reconstructed later.
Two rules, stated as rules
Rule 1: a fetch failure must never become a deletion
The reconcile step compares what a run retrieved against what is stored, and retires anything the publisher no longer publishes. That is correct behavior and it is also the most dangerous code in the pipeline.
If a source times out, rate limits you, or returns an error page, the run comes back with fewer documents. Naive reconcile reads that as removal and deletes real law.
Absence of evidence is not evidence of absence. The correct response to a failed or partial fetch is to keep everything you have, mark the run failed, and make the failure visible. A corpus that deletes on a bad fetch destroys itself over enough refresh cycles, and it accelerates, because each smaller corpus makes the next drop look less anomalous.
The practical guard is a drop threshold that fails closed. If a run proposes retiring more than a small fraction of a jurisdiction, it stops and produces a report instead of executing.
Rule 2: a corrective re-scrape can mint new identifiers for content you already had
This one is less obvious and it catches teams late. You find a defect, fix the parser, and re-scrape the affected source. The fixed run produces better text. It also produces identifiers.
If identity is derived from position in the publisher's tree, or from a number parsed off the page, then a parser change changes the identity. You now hold two generations of the same section under two ids, both live, both returned by search. An identifier-scoped cleanup is blind to the generation it replaced, because it only knows the ids it just minted.
The fix is structural. Identity has to be content-addressed and guarded rather than derived from position. Point ids are keyed on content, so an unchanged section produces an unchanged id no matter how many times you re-run, which is why an actId is not guessable. The identifier guard then refuses a new identifier that degrades one already in service.
There is a corollary worth internalizing: a deletion ledger only protects you if the ingest path reads it. A ledger consulted by the cleanup job but not by the ingest job will let the next scheduled run re-mint exactly the identifier you deleted last week.
The multi-format mirror is the provenance backbone
Sixteen of the 128 fields on a search result are provenance. The links are the obvious part: externalUrl, stateHtmlUrl, htmlUrl, pdfUrl, xmlUrl, textUrl, docxUrl, and for federal material govInfoHtmlUrl, govInfoPdfUrl, packageId and granuleId. Where the publisher offers five formats, all five are kept, and the formats reference says which mirror each corpus has.
The interesting four are sourceCharStart, sourceCharEnd, sourcePageStart and sourcePageEnd. Those offsets point at the exact span inside the publisher's own stored artifact that this section was extracted from. Not "this text came from this website." This text is a named character range inside this specific stored file, or a named page range inside this specific PDF.
That is the difference between a provenance claim and a provenance promise. A promise you either trust or do not. A claim with offsets you can open and check, and so can opposing counsel, or a regulator asking where a compliance system got its rule text.
Build versus buy, answered fairly
If you build this, here is what you own forever. Not for a quarter. Forever.
| What you own | Why it never finishes |
|---|---|
| 52 publishers, 52 formats | Statutes, regulations, court rules and constitutions are published 52 different ways, and no two states agree on structure, numbering, or what an amendment credit looks like. |
| Layout changes with no notice | Publishers redesign. Your parser breaks. You find out from a defect report or a gate, and only if you wrote the gate. |
| robots.txt and access policy | Some publishers prohibit automated access. Honoring that means declared gaps rather than a corpus you crawled anyway, which is a product decision, not an engineering one. |
| PDF text layers of varying quality | Some state PDFs OCR cleanly. Some are scans of typewritten pages. Both look identical until you read the extracted text. |
| Identity that survives renumbering | Sections move, get renumbered, get transferred. Your ids have to follow without ever pointing two citations at one row. |
| Reconcile logic | Distinguishing "the publisher removed this" from "our fetch failed" is the single highest-stakes decision in the system, and it runs unattended. |
| Embedding cost at scale | 12,003,716 retrieval passages is the current index size. Re-embedding is the real ceiling on how often a full re-pull can run, and caching by content hash is why it is affordable at all. |
| Monitoring that notices nothing happened | The failure that hurts is a run that completes, reports success, and changed nothing. That needs its own alerting, because every other signal says green. |
How often you can afford to run all of that end to end is the real cadence question, and it is worked through source by source in Legal Data Freshness: What Refresh Cadence Actually Means, Source by Source.
The question that actually decides it is what you are differentiating on.
If the coverage decisions are your product, meaning the sourcing policy and the judgment calls about which gaps to carry rather than fill, then that work is your value and it belongs in your company.
For everyone else the corpus is an input to something else, and the engineering above is a tax on the product you actually meant to build. Most teams are in that second group and misjudge themselves into the first, usually around week three, when the first state parses cleanly and the estimate for the other 51 looks linear. It is not linear.
Either way, start by reading the collection layer rather than guessing at it. Vaquill AI publishes it openly at open-us-law alongside the dataset, so you can see exactly how each government source is read before you decide what you want to own.
What to design around
Gates catch structural and identity defects. They do not catch every semantic error, and the difference matters.
A truncation gate can tell that stored text ends earlier than the publisher's. It cannot tell that a complete, correctly extracted section was filed under the wrong chapter because the publisher's own breadcrumb was wrong. Structure is checkable. Meaning mostly is not.
A gate's measured defect rate is a floor, not a ceiling. It only sees the shapes it was written to see, so the number it reports is "defects of the kinds we already know about." A new failure mode reads as zero until somebody writes the check that finds it.
Our own amendment-history census, run on 2026-08-30 across 3,518,180 sections in the four state corpora, makes the point concretely: 912,388 sections (25.9%) carry no amendment year at all, and three unrelated causes sit inside that one number. The full breakdown, and why a missing year is usually a publisher property rather than a pipeline failure, is in Amendment History and Point-in-Time Law.
The transferable lesson is the shape, not the figure. A single "amendment history coverage" percentage from any provider mixes several unrelated things. Ask which of them it measures. The answer tells you whether they ran the census or are quoting a field-populated rate.
FAQ
What is the difference between a legal scraper and a legal corpus?
A scraper fetches pages. A corpus makes a per-row claim that the stored text is what a named government publisher published at a named citation, and can prove it later. The proof layer, gates plus per-candidate audit records, is most of the engineering and essentially all of the maintenance.
Why should ingestion gates default to fail instead of pass?
Because a gate that defaults to pass does nothing on the day it was written for. A default-fail gate stops a run and forces a human to look, which costs time on ordinary days and prevents serving wrong law on bad ones. Serving a bad run for three days is worse than blocking a good run for three hours, because downstream caches keep the bad answer.
What is a candidate ledger and why does a log not replace it?
A candidate ledger records what each candidate document became, kept or discarded and why, keyed to its source URL. Logs are keyed by time and rotate, so answering "what happened to this specific document" from logs requires already knowing which run touched it. A ledger answers it as a join against the publisher's index.
What should a pipeline do when a government source times out mid-run?
Keep everything already stored, mark the run failed, and surface the failure. Never let a partial fetch flow into a deletion, because absence of evidence is not evidence of absence. A drop threshold that fails closed, so any large proposed retirement produces a report instead of executing, is the practical guard.
Why can a corrective re-scrape create duplicate records?
If identifiers are derived from position in the publisher's tree or from text parsed off the page, then fixing the parser changes the identifiers. The fixed run mints a second generation of ids for content you already had, and an id-scoped cleanup cannot see the generation it replaced. Content-addressed identity plus a guard that refuses degrading ids is the structural fix.
What are sourceCharStart and sourcePageStart used for?
They mark the exact span inside the publisher's own mirrored artifact that a section was extracted from, by character offset for text formats and page number for PDFs. That turns provenance into something a reader can verify by opening the file, rather than a claim they have to accept. It is the field pair to ask any legal data vendor about first.
How many formats should a legal API mirror per document?
Every format the publisher offers. Vaquill AI's search result carries htmlUrl, pdfUrl, xmlUrl, textUrl and docxUrl, plus govInfoHtmlUrl, govInfoPdfUrl, packageId and granuleId on federal material. XML gives you structure, PDF gives you the pagination a court expects in a citation, and plain text is what most parsers actually want.
Does a legal data API let me query text as of a past date?
Not in Vaquill AI's case, and you should ask any vendor claiming it to demonstrate it live. Storage is one text per citation, the current one, so nothing in the request accepts a date to rewind to. The four things that come closest are amendment history and lastAmendedYear per section, a yearFrom and yearTo currency filter, change events captured on each refresh, and per-section diffs on watched boards.
Should my team build its own US legal corpus?
Build it if the corpus is your product and the coverage and sourcing decisions are your differentiation. Buy it or start from open source if the corpus is an input to something else, which is most teams. The cost that gets underestimated is not the first state, it is maintaining 52 publishers who change layouts without telling you.
How can I verify a vendor's corpus claims before signing?
Ask three things: for the disposition of one document you can see on a government site but not in their API, for which validation gates exist and what their default is, and for a free machine-readable coverage endpoint you can poll. Vaquill AI exposes GET /us/statutes/coverage, documented in the coverage guide, with per-jurisdiction counts, a freshness array declaring every paused corpus with its reason, and a measuredAt timestamp.
New legal AI guides, weekly.
Further Reading
Hybrid Search Over 12 Million Legal Passages: Why Semantic Alone Fails on Law
Read postParsing Legal Citations in Code: Bluebook Forms, State Variants, and the Ones You Must Refuse
Read postLegal Citation Resolution: Turning a Cite Into the Right Section, Every Time
Read postWhat a Legal Data API Should Guarantee: The Schema Contract
Read post128 Fields on One Statute: What Rich Legal Metadata Actually Buys You
Read postAmendment History and Point-in-Time Law: What a Legal API Can and Cannot Tell You
Read post
Co-Founder & CTO
Priyansh leads engineering and AI at Vaquill, from the matter workbench to drafting, document comparison, document matrix, and citation-verified research.