A dumber agent beat mine by twenty-two points

I built an agent that recovers failed payments, and before I pointed it at anything I wrote seven ways it could fail into a file, with thresholds, and dated it.

Then I ran it. Nine variants, a thousand seeded universes, five hundred customers each. The incumbent baseline — retry everything, then text the customer a payment link — recovered 64.8% of failed payments. Mine recovered 43.0%.

Strictly, it didn't fire. I'd written it to trip when the interval included zero — when I couldn't tell my agent apart from the baseline. Mine excluded zero comfortably. On the wrong side. I had carefully specified the failure where my work turns out to be indistinguishable from nothing, and not the one where it is decisively worse. That's its own lesson about writing criteria.

That number went in the second paragraph of my README and the first thirty seconds of my demo video. Here is why.

Start with what the baseline does to earn those twenty-two points. Across those thousand universes it took 18,803 automated actions on payments a fraud engine had declined. It sent 60,972 retries on a failure class where the correct number of attempts is zero.

Sit with the first number. A payment is declined for suspected fraud. Maybe the card is stolen. Maybe it's yours and the person holding it isn't you. The baseline's response is to send the cardholder an automated message: your payment failed, click here to complete it.

If that customer is a victim, you have just sent them a link, in the voice of a brand they trust, about a transaction they never made. That isn't adjacent to the phishing pattern. It is the phishing pattern, executed by the merchant, at scale, automatically.

I check this with a predicate that scans the ledger rather than asking the agent how it did — eight claims, all read out of receipts. It holds for my agent on 1,000 of 1,000 seeds. It holds for the baseline on 0 of 1,000.

So the baseline is not a better agent that happens to score higher. It is an agent that cannot be deployed, scoring higher because of the actions that make it undeployable.

Which invites the fair question: how does it do against something that isn't breaking the law? Against an ablation with my guards intact and the taxonomy stripped out, I win by 1.00 point, interval [+0.85, +1.14]. Against a retry-only baseline that never contacts anyone, I used to win by 0.90; I now lose by 2.74, for a reason I'll come to. Every arm that beats me breaks a rule to get there — the retry-only one included, which acts on 16,631 payments a fraud engine declined, where mine acts on none. Every arm that obeys the rules, I beat or match. That is a narrower claim than "my agent recovers more money," and it is the one the data supports. Hold on to that retry-only number, though. It has already told me two things.

The twenty-two points are not all compliance either, though it would flatter me to say so. Run my own policy with the guard chain removed and everything else held fixed, and it recovers 49.8% — so the guards themselves cost 6.77 points. The remaining fifteen belong to a policy that contacts more people, more often, which is a different thing from obeying the rules. I conflated the two for a while, because the bigger number made a better story about regulation.

None of which a single reported number can express. Had I built the baseline and posted "65% recovery," every word would be true, and no reader could tell what it cost.

Now look at what produces a number like that: a search that rewards whatever scores highest, run by someone who never wrote down in advance what would count as failing. What scored highest was the illegal thing. That isn't a coincidence, and it isn't really a story about safety. It's a story about when you decided what counts.

Almost no agent demo I've seen was evaluated against criteria fixed before the run. I don't think that's fraud. I think it's what the process does to you by default. You run something. It disappoints. You adjust a threshold, change the population, extend the horizon, run it again. Eventually a run looks good, and that's the run in the post. Nobody lied at any point — but the number you're reading is a maximum over a search whose length goes unreported. And it isn't only the reader who can't see that. You can't either, not reliably, not about your own work.

Pre-registration fixes it, and it's embarrassingly cheap. It's a file. Dated, append-only, thresholds written down with a rule for what each one means. Mine carries sixty-six amendments, each flagged with whether I made the change with the relevant output already visible — because I amended the protocol and I amended the protocol after seeing the numbers are different sentences, and only one of them is a confession.

What it costs isn't technical. It costs you the option to quietly not mention the run that went badly. That is the entire price, and it's why almost nobody pays it.

Before this gets too pleased with itself: the outcomes are modelled. I'm a student, and no student has live merchant traffic. Eleven parameters govern whether an intervention succeeds; ten are my judgement, tagged [guess] in the source, where a parameter with no provenance tag fails a test. The eleventh isn't a measurement either: it's the registered zero for retrying a dead instrument, a structural fact rather than an estimate. No one publishes those rates at this granularity, and inventing a citation would have been the first dishonest sentence in the repository.

So the claim was never "₹41 crore was recovered." It was that under a stated outcome model, this policy compares to its controls by stated margins — and, as you'll see, even that claim did not survive its own test. The methodology is the deliverable. Point it at real traffic and it gives you a real number.

When I wrote that sentence the first time, it ended differently. It said the direction survives a ±50% sweep of every parameter. I had registered that sweep and not run it — it costs about a hundred times the main run — so what I was really reporting was an intention. Nobody would have caught it. The criterion that covers it, F6, read not evaluated in every manifest I published, which is honest and also empty: a criterion that has never been evaluated has never risked anything.

So I ran it. 185,400 runs, just under eight hours: every guessed parameter moved to −50%, −25%, +25% and +50% of its registered value, one at a time, plus six shifts of the two failure-reason mixes — 102 configurations, nine variants, two hundred seeded universes each.

The first time, the headline survived all of it. So did the price of the guards, and the two ablations with the largest effects. Under every single move of every single guess, the incumbent still beat me by about eighteen points and the guard chain still cost about seven. Those are properties of the agent.

Two claims were not. The one-point lead over the retry-only baseline — the number I told you to hold on to — failed to survive 24 of the 102 configurations, under fourteen different parameters. And my flagship claim, the one the whole taxonomy is built around, failed to survive two.

A lead that small is a property of the world I built, not of the agent in it.

The flagship claim is that retrying a dead card is worse than useless: it burns one of the four attempts the gateway allows before it cancels the subscription, and that attempt is the one a re-authorisation link needed. Measuring it means comparing my agent against a copy of itself that gives dead instruments one retry. For months that difference was indistinguishable from zero, because expired cards are five percent of the population. It only became measurable when I added UPI mandates, where a third of failures are an instrument that cannot be debited as it stands.

Which is exactly what that sweep found. Halve the share of customers on mandates, or shift the mandate failure mix toward reasons that are recoverable, and the effect stops excluding zero. The claim holds in the world I registered. It does not hold in a world with half as many mandates, and I would not have known that without spending the eight hours.

F6 fires if five or more of eight comparisons flip. That time, four did. It did not fire, by one — and one of the four flips for a structural reason that has nothing to do with sensitivity, which I've written up rather than argued away, because the tempting move here is to adjust the denominator now that I can see the count.

It fired the next day. I closed the last open attack in the suite — a customer who pays through a channel my agent can't see, which it went on chasing for money the merchant already had — and that changed what the agent does, so every one of those 185,400 runs described an agent that no longer existed. I ran the grid again, all eight hours of it. This time five comparisons flipped.

Before the first run I had written down what that case means: the evaluation does not support conclusions and is presented as an instrument-building exercise instead. So that is what the repository now says, at the top, above the numbers. It fires under the older version of the rule as well — the one that set aside the arm that can never pass — four of seven, against a threshold of four. No choice of denominator rescues it, and I didn't go looking for one. I did recompute the configuration that tipped it, from nothing, to make sure it wasn't a stale file. Byte for byte, the same.

The pattern matters more than the verdict. The incumbent's lead and the price of the guards survived every move of every guess, again. The taxonomy, the salary-aware timing and the flagship claim each failed somewhere. If I had been tuning toward a good story, that is the one pattern the tuning would not have left behind.

The findings that are bad news for me survived everything. The ones that argue for my design did not.

And the retry-only number, the one I told you to hold on to, turned over. Closing that attack cost me 3.6 points of recovery, and the baseline that retries everything now leads me by 2.7. That reads like a loss until you look at what the 3.6 points were: of 1,240 recoveries I checked with the fix switched off, 91 were money the customer had already paid some other way. The agent now stops and hands those to a person instead of collecting twice. The strawman has no way to notice, so its number didn't move.

There's a second thing I'd been deferring. The protocol seals sixty percent of the population and allows exactly one look at it; I spent that look in August, on a version of the agent three revisions old. So the honest position was that my numbers had never been checked against anything unseen. Last week I registered a fresh set of a thousand universes, with the tolerances written down first, and ran them once. Every conclusion replicated: my rate moved a fifth of a point, the gap, eighteen points at the time, moved a tenth, and every difference that excluded zero excluded it again on the same side — including the flagship claim, at a quarter of a point, on worlds the agent had never seen. Then, the next day, I changed the agent — the fix above — so that holdout now confirms a version one revision old, and it cannot be run again. A holdout is spent per agent, which means the next one belongs at the end, when the agent stops changing.

Here is where all of that stopped being theoretical.

Every test passed. 1,353 of them. The safety predicate was clean across a thousand seeds. No guard misbehaved. No receipt was missing. No exception was raised.

And a third of my population was doing nothing at all.

The system has a guard for India's e-mandate rules: you cannot debit a customer's standing mandate without serving twenty-four hours' notice first. When it sees a mandate debit with no notice on file, it defers the debit and hands back an obligation — send the notice.

Obligations were honoured on the execute path. And a deferred proposal never executes. So no notice was built. So the timestamp stayed empty. So the guard deferred again, and again, five times, and then blocked the debit for good.

The one thing that could satisfy the guard was an execution the guard was blocking.

I didn't find it with a test. I found it writing an adversarial attack against the mandate flow, which came back passing — and then noticing it could never have failed, because the thing it attacked had never happened. Of 707 retries executed on one seed, zero landed on any of the 275 mandate episodes. Two hundred and nine of those episodes ended blocked. That was the third: thirty-one percent of my population had a retry ladder that never fired once.

Fixing it moved my headline recovery rate from 0.344 to 0.491.

Which means three quarters of what I had been calling the price of compliance was my own deadlock. I'd been telling a tidy story about what it costs to do this properly, and most of that cost was a bug I wrote.

Had it fired, I would have published a conclusion about regulation that was actually a conclusion about me.

Every test I had written asked whether the agent did something wrong. Not one asked whether it did anything at all.

I don't think that gap is mine alone. Safety tests get written as this must not happen, and a system that does nothing passes every one of them, perfectly, forever. A guard that defers indefinitely is indistinguishable from a guard that works. Liveness needs its own assertions, and they feel redundant right up until they aren't.

About those points: I sat on them for a day. Not deciding whether to publish — I'd written down in advance that I would, which is the only thing writing it down is for. I was deciding how to phrase it so it stung less. Every draft had a softener in it. While the baseline scores higher on raw recovery… The comparison is arguably unfair, since… Both true. Both also the sound of someone negotiating with a number.

The version I shipped states it flat, in the second paragraph, before I've earned any goodwill from the reader. It's the worse-reading version. It's the only one I'd defend.

I run twenty-four adversarial attacks against this thing. Twenty-three survive. One is open, named in the repo, with the conditions under which it wins: the payment-channel attack from earlier, against a deployment that hasn't wired the lookup that closes it.

The claim was never that the agent is correct. It's that being wrong is discoverable — and the evidence for that isn't the twenty-three. It's how much of that list the apparatus found instead of me.

So write down what would make you abandon the thing you're building, before you build it, with a number attached. You probably won't fail the way you predicted. I didn't: I specified the wrong failure, and the one that actually mattered got caught by a test I'd written for something else. That is still the apparatus working. What never works is deciding afterwards.