Does it get the numbers right?

Methodology · stated up front

How every number here was produced.

• Ground truth first. Every correct entity, figure, date, and contradiction was labeled by hand before the engine saw the documents — outputs are scored against an answer key, not opinion.
• Three test beds. A controlled synthetic benchmark (wire-transfer record, deposition with a planted contradiction, corporate-ownership filing); a real-case run on an OCR'd 324-page Electronic Funds Transfer Analysis; and — new in v4 — a famous-case run on the U.S. DOJ's 2016 civil forfeiture complaint from the 1MDB kleptocracy case, where the ground truth is public record.
• Automated, repeatable scoring. A scripted harness compares engine output to the answer key — the same test re-runs on every engine change, so error rates stay current instead of frozen in marketing copy.
• Defined error terms. Grounding = a reported figure or name is traceable to the source — verbatim, or via a credited normalization (acronym expansion, OCR repair, summed-total labeling) whose evidence is recorded; verbatim and credited are always reported separately. Honesty = grounded, or explicitly labeled as the engine's own computation. Recall = the known items were found. Containment = knowledge from outside the documents stays out of the evidence sections (new in v4).

Famous-case validation · new in v4

Run against the most famous money trail in the world.

The engine analyzed a 45,000-character section of the Department of Justice's 2016 forfeiture complaint from the 1MDB case — the "Good Star phase," in which more than $1 billion left a Malaysian sovereign fund. Because the case is public and extensively adjudicated, every claim the engine makes can be checked against established record.

Every headline fact of the phase was recovered: the $1 billion transfer split into $700M to Good Star and $300M to the joint venture; the $330M follow-on transfers; and the $1.03B total — which the source never states as one number, and which the engine correctly reported as its own computation, labeled derived, with both components shown. The central deception of the case — a 1MDB officer telling a bank that "Good Star is owned 100% by PetroSaudi" when its sole signatory was Jho Low — was caught as a contradiction.

A new measured property · containment

It knows who "MALAYSIAN OFFICIAL 1" is — and kept that out of the evidence.

The DOJ complaint anonymizes its subjects. The engine — like any modern AI model — knows from years of public reporting who they are. That creates a measurable risk: does outside knowledge leak into the analysis as if it were in the document? Scored across the full output: zero real-world identifications appeared in the entities, money flows, timeline, or contradictions. Names from public reporting surfaced only in the investigative-leads section, each explicitly attributed — "widely reported in open sources as…" — where a suggestion is supposed to live.

One honest miss inside the clearly-fenced leads section: the engine described an executive as serving during "the relevant period" who in fact joined the company years after it. Contained, attributed, and speculative by design — but factually wrong, and reported here because reporting it is what this document is for.

A new measured property · the challenge pass · new in v9

It argues against its own findings — and the arguing is measured too.

Every finding above comes from a prompt built to hunt, and that instinct cannot be argued out of it: handed its own findings and told to defend them, the forensic engine returned nine more adverse findings and no defence. So the other side now gets its own pass. Challenge mode re-reads the documents as counsel for the subject and returns, for each finding, the strongest innocent explanation, any passage that weakens the claim, the finding's unstated assumption, what outside evidence would settle it, and a verdict — refuted, weakened, or survives.

An adversarial pass has its own ways to fail, so it has its own eval: a document with the defence planted in it — a controller's note explaining a "missing" $32,000 as an accrued future installment, and a policy cap giving two $9,900 withdrawals an innocent reading — plus one documentary-absence finding constructed to be un-refutable. Ten live runs:

The eval's first run caught a defect in the eval, not the engine: it demanded "survives" on the un-refutable finding, and the challenger returned "weakened" by invoking the limitation that absence from a produced file does not establish absence everywhere — which is exactly what its instructions define as weakened. The metric assumed advocacy stops at literal truth; real advocacy attacks significance. The metric was corrected to score direction; the boundary calls are reported, not scored — across ten runs direction never flipped, while the boundary did (the structuring finding: refuted once, weakened nine times; the absence finding: survives five, weakened five).

A finding that survives a challenge is un-refuted from inside the documents — not verified. The demo labels it that way, the exported report labels it that way, and so does this one.

New in v11 · the review panel

The reviewers are becoming buttons — measured as they arrive.

The findings in this report were hardened by a human review method: adversarial passes under different lenses — counsel arguing the other side, an auditor checking that compared figures share a basis, a statistician demanding denominators. Across five real investigation packages in July 2026 that method went sixteen-for-sixteen: every pass either killed a claim or armored it — including five corrections to our own memos. The automatable lenses are now endpoints.

Four run today: counsel (the challenge pass above), basis (do the two sides of every comparison share unit, accounting basis, and column definition as the document defines them — the request-vs-actual class of quiet error), tiering (is each claim stated verbatim, arithmetic-derivable, partially supported, or an inference beyond the document — with the pivotal evidence named), and denominators (every "only / unique / unprecedented" must carry its population, computed from the document where possible). Lenses that need the outside world — prior coverage, prior-period control documents, the recipient's own work — are deliberately not automated, and the product says so rather than pretending.

Same epistemic rule as the challenge pass, printed on the output: a lens verdict is review, not verification — it narrows, labels, and argues; the documents decide.

New in v10 · compare mode

Two editions of the same document — the diff is a census, the reading is not.

Four investigations in a row, the strongest finding came from comparing two editions of the same document — and the sharpest of them was invisible to every other method: between the May and June editions of a county's FY 2025-26 budget book, five line items each rose by exactly $143,035, offset to the dollar by one cut elsewhere, leaving every total unchanged. Totals hide offsetting changes by construction; only lines expose them.

So compare mode splits the work by what each part can promise. A deterministic differ parses every numeric line of both editions and matches them exactly — complete by construction, no model involved, identical on every run. A model layer then interprets only that diff: offsetting pairs, uniform per-unit changes, changes that bypass the documents' own change-tracking columns. If the model layer fails, the exact rows still come back.

Both layers are measured, each against the claim it actually makes. The ground truth is the real pair of county budget books above, where the correct answer is known to the dollar from independent hand verification: exactly 10 changed rows, five of them the uniform +$143,035.

The ground-truth eval earned its keep before the feature shipped: the differ's first run against the real pair silently mismatched 189 rows — bare-zero column values were being left inside row labels, which broke matching whenever the two editions print different column layouts — and the five $143,035 rows were among the missing. A synthetic test would have passed; the real answer key did not.

Same-day correction, from our own adversarial review: when editions print different column layouts, "changed" is measured on each row's first amount — and rows whose change sat only in a later column were being counted as unchanged. On the ground-truth pair that silently hid all 20 rows carrying the adopted budget's augmentation data. Those rows now surface in their own labeled category ("later columns differ — alignment unknown, no delta computed"), while rows differing only by an added all-zero column stay unchanged: on the ground-truth pair the bucket contains exactly the 20 augmentation rows and zero noise. A matching tripwire was added the same day: when a large share of rows fail to pair at all, the result carries a warning that the inputs may not be two editions of the same document, or that an inserted section has shifted the row matching.

New benchmarks since v3 · coverage & tables

Did it find everything — and read the right column?

Recall tells you what was found of what you checked; completeness asks the harder question — of every dollar amount in the document, how many did the engine surface at all? And financial tables add a distinct failure mode: pulling a figure from the wrong column (requested vs. approved vs. disbursed) produces a number that is in the document but means the wrong thing.

In the column-trap case the engine also reported the sums of the other columns — but labeled them as its own derived totals with full breakdowns, rather than presenting them as stated figures. The first automated check scored those as errors; they weren't. Third time this evaluation has had to learn the same lesson: the metric must credit labeled derivation, or the score punishes honesty.

Real-case validation

Known error rates on a real bank statement — and it found the red flag itself.

The engine was run against a real Electronic Funds Transfer Analysis — 324 pages of OCR'd business bank statements. On the tested statements, every dollar figure it reported was scored against the source text:

Resolved in v5 · from caveat to result

Entity grounding, measured properly: 96%.

Earlier versions of this report carried a caveat: naive exact-match scoring read entity grounding at 58%, because the engine fixes what it reads — expanding acronyms (IRS → Internal Revenue Service), repairing OCR damage ("MERCIAL" → "340 Commercial Street") — and a literal string comparison scores those corrections as inventions.

v5 replaces that metric with a three-tier score in which every credit is auditable: verbatim (name appears in the source as-is), normalized (a credited transformation — token overlap, acronym ↔ expansion, OCR-fragment match — with the matching evidence recorded per entity), and ungrounded. Crediting never hides: the tiers are always reported separately.

The single ungrounded entity is itself worth reading: "LLC (unnamed)" — the document's LLC name is redacted, and the engine described it as unnamed rather than guessing. The one measured entity error is the engine declining to invent a name. It is counted as an error anyway, because a metric that starts making exceptions stops being a metric.

New in v7 · the number nobody publishes

Run the same document five times, and 49% of findings appear every time.

Every vendor reports what their tool found once. Nobody reports what happens when you run it again. We did: the same document, five identical runs, nothing changed between them. Re-measured 29 Jul on the engine's current model; the previous model measured 46% stable and 30% single-run, so the property survived the upgrade — it is not a quirk of one model.

What this means practically. One pass is a lead generator, not a census. If you search a report for an organization and do not find it, that is not evidence it is absent from the document — unless it sits in a parsed schedule, which is complete by construction and identical every run. For anything the model extracted, run it twice.

This is not an accuracy measurement. A finding that appears five times is consistent, not necessarily correct, and a finding that appears once may be perfectly true — on one filing, a single run surfaced a Bermuda entity and $33.3M of controlled-entity transfers that the other run missed entirely. Both runs were right; both were incomplete, in different places.

What we could not measure, and are not going to guess. Contradictions and leads are prose, and the engine rewords the same finding each run, so scoring them requires a similarity threshold — and the threshold decides the answer. Worse, automated matching disagrees with reading them: one lead appears in all five runs, phrased "Identity of…" three times and "True identity of…" twice, and no workable threshold matches the pair. Every threshold we tried returned 0–4% stability, which is not what the documents say. So that number is reported as unmeasured rather than published. A tuned metric that contradicts a manual read looks rigorous and is wrong.

Measured on a 44,807-character extract of the 1MDB/Good Star complaint, five runs, re-run 2026-07-29 on the current engine. Entity name variants count as separate entities, which depresses the stable share rather than inflating it. Harness: eval/variance.py.

Controlled benchmark · synthetic first pass

Three documents with a known answer key.

Three synthetic-but-realistic forensic documents — a wire-transfer record, a deposition with one deliberately planted contradiction, and a corporate-ownership filing describing an interconnected shell-company structure. Every correct entity, figure, date, and contradiction was labeled by hand before the engine saw the text, so each output can be scored against ground truth rather than opinion.

Headline results · aggregate

Accurate — and honest about its own limits.

✓ Planted contradiction caught

Per-document

Scores by document.

What it changes

From a metric to a feature.

• Label every figure extracted (stated in the source) vs. derived (computed).
• Give each derived total a breakdown — its component figures and their dates — so the arithmetic is auditable.
• Re-score grounding so a labeled, sourced derivation counts as correct, not as a miss.
• New in v4: treat containment — outside knowledge fenced into clearly-attributed suggestions — as a first-class measured property, alongside accuracy.

Limitations · stated plainly

What this does not yet prove.

• Real-case validation covers a slice of one 324-page document and one section of the 1MDB complaint — not full files, and not yet many cases.
• Containment is measured on one document so far — and within the clearly-fenced leads section, one outside-knowledge statement was factually wrong. Fencing worked; the fence's contents still need their own accuracy bar.
• Compare mode's ground truth is one real document pair; its row-matching assumes the two editions share structure (the Nth occurrence of a repeated label is paired across editions — a shifted section trips a warning but is not repaired), rows printed as dashes rather than zeros can appear as removals, and when editions print different column layouts, per-column deltas exist only for the first amount — later-column differences are surfaced without deltas. Model-layer pattern stability is measured on five runs of that one pair. And the "four productive cases" behind this feature were selected, not sampled — they show the diff finds the finding when one exists, not how often editions contain findings.
• The review-panel lenses are measured on one synthetic planted case (five runs per lens); label boundaries between adjacent tiers are judgment and are direction-scored, not exact-scored. Prior-art, control-period, and recipient-facts review are not automated and remain human passes.
• The challenge-pass eval is one synthetic case — two planted defences and one un-refutable finding, over ten runs. The refuted/weakened/survives boundary is judgment and varies run to run; direction is the measured claim. A survived challenge is un-refuted from inside the documents, not verified.
• Figure grounding is auto-scored and objective; full recall (did it miss anything?) still needs expert adjudication.
• The 96% entity-grounding figure comes from one fresh 15,000-character run of the real bank-statement document; the credited-normalization matcher is heuristic and its rules are published with the evaluation code.
• Published evaluations exist elsewhere in legal AI (document review, research). The claim here is narrower and, to our knowledge, currently unique: published error rates for forensic financial extraction — figures, flows, and reconciliation.
• This study has not been independently replicated, and no court has yet ruled on evidence produced with this method. FRE 707 is proposed, not enacted — the Advisory Committee declined to recommend it for adoption in May 2026 and sent it back for further study, so its timeline is uncertain and later than originally projected. Nothing here should be read as a claim about admissibility; the claim is a documented methodology and measured error rates.