VASOOL
—
recovered across both cohorts · —
—
0 arm-seed runs · 9,000 base + 151,200 sweep
The money above is what the agent recovered. This is what it cost to recover it
compliantly: the realistic incumbent recovers more, and that was registered
as falsification criterion F1 before the first run. The interval
below excludes zero in the baseline's favour — a worse result than F1
firing, and the artifact says so in its own detail field. Every figure
here is read from out/development/evaluation.json; nothing is
hardcoded, and a value the manifest does not carry renders as a dash.
Paired difference vs Vasool, recovery rate
Every arm runs the same seeded universe — same customers, same arrivals, same outcome draws — so the comparison is the per-seed difference, bootstrapped over 1,000 seeds. Bars are 95% percentile intervals. At this sample size every interval is narrower than its own marker — the widest spans 0.37pp — so the dots are the intervals, not a plot that forgot to draw them. Exact bounds are in the table view below. A marker clear of the dashed zero line is a real difference; which side it falls on is the story.
Table view
| Arm | Difference | 95% interval | Excludes zero |
|---|
These are properties of what the agent did, scanned from the hash-chained ledger. They hold or fail regardless of what outcome model runs underneath, which is why they are the claims the submission actually rests on — and why they are banded differently from every recovery number on this page. Thirteen pure-function guards gate the execution plane; all thirteen are evaluated on every proposal and resolved by severity, never short-circuited.
World-keyed counters
Counted against the class the world registered for each episode, not the label the arm assigned itself — so an arm that declines to classify cannot satisfy these by mislabelling. These are world numbers, not ledger scans, and they are not part of EVALUATION.md §2a. Every row is the sum over the 1,000-seed development cohort.
| Arm | Retries on INSTRUMENT_DEAD | Actions on RISK_BLOCK | Retries on CUSTOMER_ACTION |
|---|
The third column closes the limit registered in EVALUATION.md §10 on
2026-08-24, which recorded that CUSTOMER_ACTION — 0.09 of the
registered failure mix, and priced at zero retry budget — had no
world-keyed counter, so a baseline retrying those episodes earned recovery
credit with no guardrail reporting it. A dash means the artifact does not
carry the field; no value here is defaulted.
Where every episode ended
A recovery rate reports one bucket and leaves the rest as a single
undifferentiated failure. Restraint is not a failure, and neither is an
episode the horizon cut short. The four terminal states are absorbing,
so this is a partition: every episode appears exactly once, and the row sums to
the episode count. blocked is the agent declining to act;
escalated is a human taking it; awaiting is
right-censored — still in flight when the horizon ended, neither
won nor lost.
| Arm | Episodes | Recovered | Blocked declined by policy |
Escalated to a human |
Exhausted budget burned |
Awaiting censored |
|---|
Eight of the nine outcome parameters are guesses. So every registered parameter is swept independently at −50%, −25%, +25% and +50% of its value — 83 configurations × 9 arms × 200 seeds — and each comparison is re-tested in every one. A cell is marked only when the comparison fails to survive; the unremarkable majority stays recessive, because the exceptions are the finding. F6 fires if 5 or more of the 8 comparisons flip in at least one configuration.
A3 fails in all 83, and that is not a parameter effect. Its reference
interval at 200 seeds is [−0.00057, +0.00196], which already includes zero —
so survives() fails because the reference was never conclusive at this
depth, not because any sweep moved it. Registered as a limit in
EVALUATION.md §10 rather than argued away, and it pushes F6
toward firing, which is the conservative direction.
Seven criteria, each with a threshold, written into the protocol before any run
existed. A criterion invented after seeing the numbers is not a criterion. None
fired — but read F1's row carefully, because fired: false is not
the same as good news, and the artifact says so in its own detail field.
The architecture keeps the LLM away from money. That is a claim; this is the measurement behind it. Both classifiers were asked the same questions in shadow — the LLM never touched a ledger, and a test walks the import graph in both directions to prove it could not have. The rules column is 1.000 by construction, not by measurement — ground truth resolves through the same lookup the rules read, and saying so is the only way the other column means anything.
| Failure the webhook reported | Truth | Rules | LLM | Accuracy | Consistency | Episodes |
|---|
Not a drawing of the claim — the claim itself.
tests/test_shadow_boundary.py proves invariant 1 by walking the
import graph with ast; this is that same graph, parsed at build
time and rendered. Arrows run from importer to imported, so both planes point
into the shared vocabulary: they agree on the type definitions and
share no path.
No edge crosses either dashed rule, in either direction —
0 paths from diagnosis.llm to anything that acts,
and it is unreachable from all 10 execution roots.
74 modules parsed.
The two planes share types, not paths. That is why the gap holds
without a filter in it: the LLM emits an LLMVerdict, the policy plane
consumes a Proposal, and no function in the repository converts one
into the other. For the model to move money someone would have to write a
conversion that does not exist — and an absence cannot have a bug in it,
which a validator can.
The only two modules importing the classifier are the shadow harnesses that score
it, and neither can act. Note what the execution plane’s modules include:
thirteen guards, of which nine rest on a statute. The other four
— idempotency, the retry cap, the spend cap, the human handoff — are
platform constraints and house rules whose statute attribute is
None. Calling all thirteen statutory would be the cheapest way to
make this page look stronger than it is.
The survival criterion was registered before the first attack was written,
and judge() is the only thing that can return a verdict. It scans the
ledger the way §2a scans — never “a guard returned BLOCKED”.
An attack may add evidence requirements; it cannot lower the bar. Each row below
carries the SHA-256 of the ledger that attack produced.
| Attack | Verdict | Why | Receipts | Ledger |
|---|
Four are open and named. They are not bugs awaiting a fix in the last commit — they are limits with a registered expectation, so a known failure keeps the suite green and a fixed one turns it red. A clean sheet here would be evidence the attacks are too weak.
Live Cryptographic Verifier
Recompute the deterministic SHA-256 digest live in-browser using Web Crypto API.
Twelve real receipts from seed 0, shown as the exact byte string each
hash was computed over — guard decisions, obligations, statute citations and
all. Paste any receipt_id into Exhibit H above and the browser
recomputes its SHA-256 from these bytes. Nothing here is pretty-printed, because
reformatting would change the bytes and break the hash.