I am building a search and monitoring product over Panama's Gaceta Oficial, the government's journal of record. Decades of issues, most of them scanned rather than born digital. The hard problem was never search. It was deciding which law a passage belongs to, and then the harder problem underneath it: knowing whether that decision was right.
Every norm in this corpus is identified by a number. Resolución 525. Decreto 174. That number is how citizens cite it, how lawyers argue about it, and how any sane database keys it.
It is also the part of the page that scanning reliably destroys. Digits carry no linguistic context to fall back on, so the same scan that renders legal Spanish almost perfectly turns the heading number into noise. Three real headings from this corpus: RESOLUCIÓN NÚMERO JETA is Resolución 525, RESOLUCIÓN NÚMERO GP? is 526, and RESOLUCIÓN NÚMERO 1/4 is 174. Meanwhile the prose alongside them reads cleanly: EXPEDIR CARTA DE NATURALEZA a favor de YENEANSKI IVÁN CABRERA.
Supplying the number from outside does not rescue this. Knowing from the publisher that Resolución 525 sits in the issue does not help you find NÚMERO JETA, because there is nothing legible in the target to match it against.
This is era-dependent, and worth being precise about. For the years where the entire issue is born-digital text, locating a norm by type and number on a line-anchored heading works well: 0.62% inclusion error, 0.62% exclusion error, and zero misattributions across 53 edge cases. The failure is specific to scanned bodies, which is most of the archive.
Which leaves the problem that shaped everything else. A wrong crop is indistinguishable from a right one downstream. Nothing further along the pipeline can tell you which one you produced, so accuracy has to be established at the point of attribution or not at all.
The contents page is an ordered list and the bodies appear in that order, so each norm is searched for only after the previous one has been placed. A number appearing inside an earlier body cannot drag a boundary backwards.
Locating a body by number alone misattributed 36.00%, because in a municipal issue the digit 9 is a date, an article, a lot number and a sum long before it is Acuerdo 9. Requiring the declared type word at the start of the line took that to 0.69%.
If the next declared norm cannot be located, this body's end is unknown, so it is not emitted. A body containing a foreign heading is refused outright. Ambiguity costs a document rather than producing a citation.
Title tokens are restricted to those appearing in exactly one title in the issue, filtered again for rarity in the body text, then tested by position. Distinctive-looking words fail all three tests routinely.
Known misattributions are planted and the instrument has to find them. Current standing is 300 of 300 detected against 0 of 295 false positives. An instrument that cannot report the defect will report none.
Every finding, every trap and its measured cost lives in one document that anyone reads before touching ingestion.
Inclusion is not exclusion, and the difference was twenty-fold. Asking whether the right norm is inside a span and asking whether anything else is inside it are different questions, and a wider span satisfies the first for free. Measuring only inclusion reported 0.34% misattribution. The true edge-level figure was 6.40%. The flattering number came from the instrument, not the pipeline.
A locator that cannot fail is not a locator. Contents pages are cleanly typeset and OCR far better than bodies, so a search that starts at offset zero finds every norm in the contents page and none in any body. It reports a magnificent hit rate while locating nothing.
Most apparent corruption was the law working correctly. A norm that amends another quotes its title, so a body legitimately containing another norm's words is the normal case rather than contamination. 97.7% of flagged bodies were lawful citation. The instrument was measuring the wrong thing and reporting it with total confidence.
Template titles measure the template. One issue holds thirty municipal acuerdos all reading EL MUNICIPIO DE BOCAS DEL TORO VENDE UN LOTE DE TERRENO. Scoring on distinctive title words scores the boilerplate, and every one of the thirty matches every other.
Seven instruments, seven confident wrong answers. In one investigation, seven separate measurements returned 74%, 63%, 44%, 61%, 9.09%, 26.71% and 84.3%. Each looked reasonable. Every one was caught by reading the rows it flagged or by someone questioning the premise, never by the number looking wrong.
Do not mark your own homework. If the splitter anchors on title prose, verifying with title prose proves only that the splitter ran. Hold half the tokens back, or verify with a signal the rule under test never consulted.
Compare proportions, not confidence intervals. Zero of 53 against 13 of 203 produces overlapping intervals and a Fisher exact p of 0.045. Overlap is not the absence of a difference.
The engineering result I would defend anywhere is type-awareness: 36.00% misattribution down to 0.69%, by requiring the declared type word at the head of a line instead of hunting for a number that means five other things in a municipal document.
The methodological result matters more, and it is less flattering. My own instrument reported 0.34% misattribution on the scanned path. The honest figure, once I measured exclusion instead of only inclusion, was 6.40%. Nothing in the output looked wrong either time. Finding that gap was the work, and it is why calibration against planted positives is now a precondition here rather than a nicety.
I am not claiming the corpus is solved. The pre-2006 era is out of reach today. The titles-not-numbers locator for the largest era of the archive is specified and not yet built. There are documents the pipeline declines to attribute rather than guess at, and those abstentions are the design rather than a coverage gap.
Next: build the titles-not-numbers locator for the scanned-body era, which is the largest part of the archive and currently unserved. Then a public accuracy page, because a legal search product that will not state its own misattribution rate is asking for trust it has not earned.
For focused AI systems architecture or production workflow work, write directly. I take on limited, non-conflicting fractional builds.