Fetching a state statute page is an afternoon of work. Proving that the text you stored is the text the publisher actually published, and being able to answer six months later what happened to a document you expected and didn't get, is the part that never ends. That gap is the difference between a scraper and a corpus, and it is almost entirely made of gates and audit records. This post walks the whole path, layer by layer, for a pipeline that carries more than 5.3 million sections across 56 jurisdictions (50 states, DC and five territories, plus federal law). If you are deciding whether to build this yourself, the honest answer is at the end.
Reviewed and corrected on 12 October 2026. This is general information, not legal advice. The layer, gate and ledger inventory below describes our internal collection pipeline. The public API does not expose it, and we did not re-count it in our October 2026 review. The open-source collection code is at open-us-law.
TL;DR
- A scraper is easy. A corpus is not. The hard part isn't the fetch, it is proving what you stored matches what the publisher published.
- Eleven layers sit between a government web page and an API row: discovery, fetch, extraction, mirroring, identity, enrichment, chunking, payload build, embedding, upsert, reconcile. Each has one specific way it goes wrong.
- Nine named gates default to FAIL. A gate that defaults to pass will wave through a bad run on the exact day it matters most, so refusing to publish is the correct default.
- Ledgers beat logs. A per-candidate audit record is the only thing that can answer "the publisher's index lists 520 documents and we serve 480, so what happened to the other 40."
- Two rules learned the hard way: a fetch failure must never become a deletion, and a corrective re-scrape can mint new identifiers for content you already had.
- Where populated, char and page offsets into the source document turn a provenance claim into something a reader can check. They are null on many rows.
Your nightly job pulls a state's statutes. One night the state's site is half down and the job comes back with about 40% of the usual sections. What should the sync do about the 60% it didn't get?
This post opens our legal data infrastructure series.
The pipeline is where provenance is manufactured, so read it next to Legal Data Provenance: How to Tell Where a Statute Actually Came From, and then What a Legal Data API Should Guarantee: The Schema Contract for the shape it finally emits.
The honest framing
Nobody who has built one of these thinks the fetch is the problem. HTTP is solved. The problem is that a legal corpus makes a strong claim on every single row: this text is what this government publisher published, at this citation, as of this currency statement.
That claim has to survive a publisher redesigning a page without telling anyone, a PDF whose text layer came off a scanner two decades ago, and a run that returned an error page with HTTP 200 and a body that parsed cleanly into empty sections.
None of those failures announce themselves. They look like a successful run with a slightly different document count, which is why the count is the only thing most pipelines check and why most pipelines are wrong in ways nobody has noticed yet.
The eleven layers
Here is the whole path, stage by stage, with the failure each stage exists to catch.
| Layer | What it does | How it goes wrong |
|---|---|---|
| 1. TOC discovery | Walks the publisher's index to enumerate every document that should exist. | A lazy-loading or paginated index ends early, and the run reports success on a partial enumeration. |
| 2. Fetch | Pulls each document from the publisher with retries and backoff. | A bot wall returns HTTP 200 with a challenge page, so the retry logic sees success and stores it. |
| 3. Extraction | Pulls the legal text out of HTML or PDF, dropping navigation and page chrome. | Site chrome leaks into section bodies, or a PDF with a bad text layer yields plausible-looking garbage. |
| 4. Mirror | Stores the publisher's own artifact byte for byte, in every format they offer. | The pipeline mirrors its own extracted text instead of the publisher's file, so the provenance chain points at itself. |
| 5. Identity | Assigns the stable actId and point identifiers that everything downstream keys on. | Identity derived from position in the publisher's tree changes when the publisher renumbers, minting a duplicate. |
| 6. Enrichment | Parses amendment credits, effective dates, cross references and currency statements out of the text. | A four-digit number that isn't a year gets read as one, and a section starts asserting an amendment that never happened. |
| 7. Chunking | Splits long sections into retrieval passages while keeping the citation intact. | A pathological document explodes into thousands of chunks and swamps the run. |
| 8. Payload build | Constructs the served object against a declared schema. | A field renamed upstream lands as null on every row and nothing errors, because null is a legal value. |
| 9. Embed | Generates the search representations and writes them to the search index. | A write is accepted and then dropped while the index is reorganizing, so the passage is missing but the run says completed. |
| 10. Upsert | Writes payload plus vectors under content-addressed ids. | A resume flag skips a point that already exists, so a corrective run keeps the old broken payload. |
| 11. Reconcile | Compares what came back against what is already stored, and retires what the publisher removed. | A partial fetch looks like a mass deletion, and the corpus deletes real law. |
That last row is the one that keeps people up at night, and it gets its own rule further down. Reconcile is also where the customer-visible output of the pipeline gets made, because a difference it finds becomes a change event, which is the subject of How We Know a Law Changed.
The gates, and why they default to FAIL
Nine named gates sit between a finished run and a published corpus. Each one refuses to publish rather than publish something wrong.
| Gate | What it refuses to let through |
|---|---|
| Identity check | A section whose identifier doesn't agree with its own citation and hierarchy. Prevents a run from serving Texas Property Code text under a Transportation Code identifier. |
| Text quality | Text that fails basic sanity: empty bodies, mojibake, encoding damage, page furniture where legal text should be. |
| Truncation | A section whose stored text ends before the publisher's does. The quiet one, because a truncated section reads perfectly until the missing clause is the one that mattered. |
| Chunk cap | A document that produced an implausible number of retrieval passages, which usually means the splitter misread the structure. |
| Excluded material | Content that should never enter the corpus, such as commercial annotations or editorial matter sitting beside the government text. |
| Third-party render | Output from a rendering service or intermediary rather than the publisher, so an intermediary's view is never mistaken for the official one. |
| Identifier guard | A newly minted identifier that collides with, or degrades, one already in service. |
| Publisher check | A run whose bytes didn't come from the publisher we claim they came from. |
| Record preflight | A malformed or schema-violating record, checked before a single vector is generated. |
The list matters less than the default. A gate that defaults to pass is not a gate. It is a log line with ambition. On an ordinary day it changes nothing, and on the one day the publisher shipped a broken export it does exactly what a missing gate would have done: nothing.
Defaulting to fail costs real time. Runs stop, someone has to look at why, and that cost is the product. A corpus that publishes a bad run on Tuesday and fixes it on Friday served wrong law for three days, and every customer who cached a response still has it.
The reconcile guard is the highest-stakes rule in the system. This is an illustrative sketch of the rule, not our production code:
DROP_CEILING = 0.02 # fraction of a jurisdiction a single run may retire
def reconcile(fetched: set, stored: set):
missing = stored - fetched
if len(missing) > len(stored) * DROP_CEILING:
raise RunHalted(
f"run proposes retiring {len(missing)} of {len(stored)} sections; "
"treating as a failed fetch, not a publisher deletion"
)
return missing # only now is retirement allowed
Fail closed. A source that times out returns fewer documents, and a reconcile step without this guard reads that as removal and deletes real law.
The ledgers: why an audit record beats a log line
A gate tells you a run failed. A ledger tells you what happened to one specific document. Four audit records run alongside the pipeline:
- A candidate ledger records what each candidate document became, kept or discarded and why, keyed to its source URL.
- A structural ledger records the shape a run found, so a later run can tell "the publisher removed this" from "our parser stopped seeing it."
- A repair ledger records every corrective write applied to stored data, so a repair can't be silently re-applied or silently undone.
- A write receipt records what a run actually wrote, which isn't the same as what it intended to write.
Here is the question that motivates all four. Say the publisher's index lists 520 bulletins for a state and your API serves 480. Where did the other 40 go?
A log file can't answer this. Logs are keyed by time and they rotate. To answer from logs you would need to know which run touched those 40 documents, that the run still has logs, and what to grep for, which requires already knowing the answer.
A candidate ledger answers it as a join. Take the publisher's URLs, join against the ledger on source URL, and every missing row comes back with a disposition: kept, or discarded with a reason. Some are duplicates of documents already served under a different URL. Some are index or container pages that were never documents. Some failed extraction and are queued for repair. Some the publisher withdrew. That is an answer. "We do not know" isn't.
The distinction generalizes past legal data. Logs record what the program did. Ledgers record what happened to each item, and if your users will ever ask about one specific item, you need the second kind written at the time, because it can't be reconstructed later.
Two rules, stated as rules
Rule 1: a fetch failure must never become a deletion
The reconcile step compares what a run retrieved against what is stored, and retires anything the publisher no longer publishes. That is correct behavior and it is also the most dangerous code in the pipeline.
If a source times out, rate limits you, or returns an error page, the run comes back with fewer documents. Naive reconcile reads that as removal and deletes real law.
Absence of evidence is not evidence of absence. The correct response to a failed or partial fetch is to keep everything you have, mark the run failed, and make the failure visible. A corpus that deletes on a bad fetch destroys itself over enough refresh cycles, and it accelerates, because each smaller corpus makes the next drop look less anomalous.
The practical guard is a drop threshold that fails closed. If a run proposes retiring more than a small fraction of a jurisdiction, it stops and produces a report instead of executing.
Rule 2: a corrective re-scrape can mint new identifiers for content you already had
This one is less obvious and it catches teams late. You find a defect, fix the parser, and re-scrape the affected source. The fixed run produces better text. It also produces identifiers.
If identity is derived from position in the publisher's tree, or from a number parsed off the page, then a parser change changes the identity. You now hold two generations of the same section under two ids, both live, both returned by search. An identifier-scoped cleanup is blind to the generation it replaced, because it only knows the ids it just minted.
The fix is structural. Identity has to be content-addressed and guarded rather than derived from position. Point ids are keyed on content, so an unchanged section produces an unchanged id no matter how many times you re-run, which is why an actId is not guessable. The identifier guard then refuses a new identifier that degrades one already in service.
There is a corollary worth internalizing: a deletion ledger only protects you if the ingest path reads it. A ledger consulted by the cleanup job but not by the ingest job will let the next scheduled run re-mint exactly the identifier you deleted last week.
The multi-format mirror is the provenance backbone
Sixteen of the 148 fields on a search result are provenance. The links are the obvious part: externalUrl, stateHtmlUrl, htmlUrl, pdfUrl, xmlUrl, textUrl, docxUrl, and for federal material govInfoHtmlUrl, govInfoPdfUrl, packageId and granuleId. Most are null on any given row, and the formats reference says which mirror each corpus has. sourceUrl is the link to cite: the government's own page for the section.
The interesting four are sourceCharStart, sourceCharEnd, sourcePageStart and sourcePageEnd. Where populated, those offsets say where in the source document this section's text begins and ends, for reconciling our extraction against the original PDF. Not "this text came from this website." This text is a named character range, or a named page range inside a specific PDF.
That is the difference between a provenance claim and a provenance promise. A promise you either trust or don't. A claim with offsets you can open and check. The API reference documents the page fields as 100 percent populated on the Federal Rules, though on 12 October 2026 Fed. R. Civ. P. 12 and Fed. R. Crim. P. 12 returned null. On the three rows we pulled on 12 October 2026 (a US Code section, a Statutes at Large section and a Maryland insurance bulletin) all four offsets were null, so check which corpora populate them before you build on them.
A short history: who gets to call a copy official
Everything above assumes you can say which copy of a law is the real one. For a long time that was easy, because the real one was printed. Online, copying is free and altering is easy. A Library of Congress digital archivist put the problem plainly in 2013: "Digital technology makes documents easy to alter or copy, leading to multiple non-identical versions that can be used in unauthorized or illegitimate ways." (Library of Congress, April 2013)
States saw it coming. The Uniform Law Commission began looking at a uniform law for authenticating online legal materials in early 2008, and approved the Uniform Electronic Legal Material Act at its annual meeting in July 2011. The act doesn't prescribe a technology. It says that when a state designates an electronic version of its legal material as official, that version has to be authenticated "by providing a method to determine that it is unaltered," preserved, and kept accessible to the public on a permanent basis (same Library of Congress source). Put that next to the mirror layer and the offsets later in this post and you can see the shared idea: somebody has to be able to check a copy against something.
The other question is who owns the words. In Georgia v. Public.Resource.Org, decided April 27, 2020, the state of Georgia had sued a nonprofit for posting the Official Code of Georgia Annotated online. That code held every Georgia statute in force plus non-binding annotations under each one: summaries of court opinions, attorney general opinions and related articles. A commercial publisher drafted the annotations under a work-for-hire agreement, supervised by the state's Code Revision Commission, which is made up mostly of legislators. The Supreme Court held the annotations weren't eligible for copyright, and wrote that "The animating principle behind this rule is that no one can own the law." (Georgia v. Public.Resource.Org, Supreme Court, 2020)
Two details matter if you build a corpus. The Court's rule rests on who the author is: it "does not apply, however, to works created by government officials (or private parties) who lack the authority to make or interpret the law." So a private publisher's annotations, written with no legislative body behind them, can stand on different ground. And it's one reason the gate list above has an "excluded material" row. Which material belongs in a corpus is a sourcing decision, and it's a legal one too.
Build versus buy, answered fairly
If you build this, here is what you own forever. Not for a quarter. Forever.
| What you own | Why it never finishes |
|---|---|
| 56 jurisdictions, 56 publishers' formats | Statutes, regulations, court rules and constitutions are published 56 different ways, and no two states agree on structure, numbering, or what an amendment credit looks like. |
| Layout changes with no notice | Publishers redesign. Your parser breaks. You find out from a defect report or a gate, and only if you wrote the gate. |
| robots.txt and access policy | Some publishers prohibit automated access. Honoring that means declared gaps rather than a corpus you crawled anyway, which is a product decision, not an engineering one. |
| PDF text layers of varying quality | Some state PDFs OCR cleanly. Some are scans of typewritten pages. Both look identical until you read the extracted text. |
| Identity that survives renumbering | Sections move, get renumbered, get transferred. Your ids have to follow without ever pointing two citations at one row. |
| Reconcile logic | Distinguishing "the publisher removed this" from "our fetch failed" is the single highest-stakes decision in the system, and it runs unattended. |
| Embedding cost at scale | More than 20 million retrieval passages is the current index size. Re-embedding is the real ceiling on how often a full re-pull can run, and caching by content hash is why it is affordable at all. |
| Monitoring that notices nothing happened | The failure that hurts is a run that completes, reports success, and changed nothing. That needs its own alerting, because every other signal says green. |
People who run scrapers against government sites for a living say much the same thing. One long-time maintainer of open-source legislative scrapers, writing about bill data (a close cousin of statute data) in March 2026, said "the sources constantly change and break in goofy ways." The commenter added that this work doesn't behave like web search, where you can ignore broken pages, because "the penalty for missing things is too high." (showerst, Hacker News, March 2026) The same comment describes the author as someone who has "been working on legislative data for 15 years now, on open source scrapers with OpenStates." That is the reconcile rule from a practitioner's chair: when a missing record can be a missed law, a quiet gap is the expensive failure.
How often you can afford to run all of that end to end is the real cadence question, and it is worked through source by source in Legal Data Freshness: What Refresh Cadence Actually Means, Source by Source.
The question that actually decides it is what you are differentiating on.
If the coverage decisions are your product, meaning the sourcing policy and the judgment calls about which gaps to carry rather than fill, then that work is your value and it belongs in your company.
For everyone else the corpus is an input to something else, and the engineering above is a tax on the product you actually meant to build. Most teams are in that second group and misjudge themselves into the first, usually around week three, when the first state parses cleanly and the estimate for the other states looks linear. It isn't linear.
Either way, start by reading the collection layer rather than guessing at it. Vaquill AI publishes it openly at open-us-law, so you can see how each government source is read before you decide what you want to own.
What to design around
Gates catch structural and identity defects. They don't catch every semantic error, and the difference matters.
A truncation gate can tell that stored text ends earlier than the publisher's. It can't tell that a complete, correctly extracted section was filed under the wrong chapter because the publisher's own breadcrumb was wrong. Structure is checkable. Meaning mostly isn't.
A gate's measured defect rate is a floor, not a ceiling. It only sees the shapes it was written to see, so the number it reports is "defects of the kinds we already know about." A new failure mode reads as zero until somebody writes the check that finds it.
Our own amendment-history census, run on 2026-08-30 across 3,518,180 sections in the four state corpora (not re-measured in our October 2026 review), makes the point concretely: 912,388 sections (25.9%) carry no amendment year at all, and three unrelated causes sit inside that one number. The full breakdown, and why a missing year is usually a publisher property rather than a pipeline failure, is in Amendment History and Point-in-Time Law.
The transferable lesson is the shape, not the figure. A single "amendment history coverage" percentage from any provider mixes several unrelated things. Ask which of them it measures. The answer tells you whether they ran the census or are quoting a field-populated rate.
FAQ
What is the difference between a legal scraper and a legal corpus?
A scraper fetches pages. A corpus makes a per-row claim that the stored text is what a named government publisher published at a named citation, and can prove it later. The proof layer, gates plus per-candidate audit records, is most of the engineering and essentially all of the maintenance.
Why should ingestion gates default to fail instead of pass?
Because a gate that defaults to pass does nothing on the day it was written for. A default-fail gate stops a run and forces a human to look, which costs time on ordinary days and prevents serving wrong law on bad ones. Serving a bad run for three days is worse than blocking a good run for three hours, because downstream caches keep the bad answer.
What is a candidate ledger and why does a log not replace it?
A candidate ledger records what each candidate document became, kept or discarded and why, keyed to its source URL. Logs are keyed by time and rotate, so answering "what happened to this specific document" from logs requires already knowing which run touched it. A ledger answers it as a join against the publisher's index.
What should a pipeline do when a government source times out mid-run?
Keep everything already stored, mark the run failed, and surface the failure. Never let a partial fetch flow into a deletion, because absence of evidence isn't evidence of absence. A drop threshold that fails closed, so any large proposed retirement produces a report instead of executing, is the practical guard.
Why can a corrective re-scrape create duplicate records?
If identifiers are derived from position in the publisher's tree or from text parsed off the page, then fixing the parser changes the identifiers. The fixed run mints a second generation of ids for content you already had, and an id-scoped cleanup can't see the generation it replaced. Content-addressed identity plus a guard that refuses degrading ids is the structural fix.
What are sourceCharStart and sourcePageStart used for?
Where populated, they mark where a section's text begins and ends in the source document, by character offset and by page number for PDFs, so a reader can reconcile the extraction against the original. They are null on many rows, and the API reference documents the page fields as 100 percent populated on the Federal Rules. Ask any legal data vendor which corpora populate them.
How many formats should a legal API mirror per document?
Every format the publisher offers. Vaquill AI's search result can carry htmlUrl, pdfUrl, xmlUrl, textUrl and docxUrl, plus govInfoHtmlUrl, govInfoPdfUrl, packageId and granuleId on federal material, populated where the publisher provides them. XML gives you structure, PDF gives you the pagination a court expects in a citation, and plain text is what most parsers actually want.
Does a legal data API let me query text as of a past date?
Yes, with a caveat, and you should ask any vendor claiming it to show it live. The section body endpoint takes asOf=YYYY-MM-DD. Where we hold the publisher's printed editions (the CFR annual editions, for one), it returns the edition that governs your date, a date in a gap between editions comes back out of coverage, and for the US Code most historical dates fall outside the editions we hold. For every other corpus the earlier text is rebuilt from the changes we recorded, the asOf block reports isBounded when the answer rests on when capture began, and a date before capture can come back unavailable. Alongside it sit amendment history and lastAmendedYear per section, a yearFrom and yearTo currency filter, change events captured on each refresh, and per-section diffs on watched boards.
Should my team build its own US legal corpus?
Build it if the corpus is your product and the coverage and sourcing decisions are your differentiation. Buy it if the corpus is an input to something else, which is most teams. The cost that gets underestimated isn't the first state, it is maintaining dozens of publishers who change layouts without telling you.
How can I verify a vendor's corpus claims before signing?
Ask three things: for the disposition of one document you can see on a government site but not in their API, for which validation gates exist and what their default is, and for a free machine-readable coverage endpoint you can poll. Vaquill AI exposes GET /us/statutes/coverage, documented in the coverage guide, with per-jurisdiction counts by corpus, a statusBreakdown and a measuredAt timestamp, alongside the free GET /boards call that returns cadence and lastRetrievedAt for every watchable source.
New legal AI guides, weekly.
Further Reading
Hybrid Search Over 20 Million Legal Passages: Why Semantic Alone Fails on Law
Read postParsing Legal Citations in Code: Bluebook Forms, State Variants, and the Ones You Must Refuse
Read postLegal Citation Resolution: Turning a Cite Into the Right Section, Every Time
Read postWhat a Legal Data API Should Guarantee: The Schema Contract
Read post128 Fields on One Statute: What Rich Legal Metadata Actually Buys You
Read postAmendment History and Point-in-Time Law: What a Legal API Can and Cannot Tell You
Read post
Co-Founder & CTO
Priyansh leads engineering and AI at Vaquill AI: the pipelines that pull statutes, regulations and court rules from every US jurisdiction's official publisher, and the REST API, MCP server and open dataset that serve them.