A dumber agent beat mine by sixteen points

I built an agent that recovers failed payments, and before I pointed it at anything I wrote seven ways it could fail into a file, with thresholds, and dated it.

Then I ran it. Nine variants, a thousand seeded universes, five hundred customers each. The naive baseline — retry everything, then text the customer a payment link — recovered 65.4% of failed payments. Mine recovered 49.1%.

Strictly, it didn't fire. I'd written it to trip when the interval included zero — when I couldn't tell my agent apart from the dumb one. Mine excluded zero comfortably. On the wrong side. I had carefully specified the failure where my work turns out to be indistinguishable from nothing, and not the one where it is decisively worse. That's its own lesson about writing criteria.

That number went in the second paragraph of my README and the first thirty seconds of my demo video. Here is why.

Start with what the baseline does to earn those sixteen points. Across those thousand universes it took 20,988 automated actions on payments a fraud engine had declined. It sent 66,040 retries on a failure class where the correct number of attempts is zero.

Sit with the first number. A payment is declined for suspected fraud. Maybe the card is stolen. Maybe it's yours and the person holding it isn't you. The baseline's response is to send the cardholder an automated message: your payment failed, click here to complete it.

If that customer is a victim, you have just sent them a link, in the voice of a brand they trust, about a transaction they never made. That isn't adjacent to the phishing pattern. It is the phishing pattern, executed by the merchant, at scale, automatically.

I check this with a predicate that scans the ledger rather than asking the agent how it did — eight claims, all read out of receipts. It holds for my agent on 1,000 of 1,000 seeds. It holds for the baseline on 0 of 1,000.

So the baseline is not a better agent that happens to score higher. It is an agent that cannot be deployed, scoring higher because of the actions that make it undeployable.

None of which a single reported number can express. Had I built the baseline and posted "65% recovery," every word would be true, and no reader could tell what it cost.

Now look at what produces a number like that: a search that rewards whatever scores highest, run by someone who never wrote down in advance what would count as failing. What scored highest was the illegal thing. That isn't a coincidence, and it isn't really a story about safety. It's a story about when you decided what counts.

Almost no agent demo I've seen was evaluated against criteria fixed before the run. I don't think that's fraud. I think it's what the process does to you by default. You run something. It disappoints. You adjust a threshold, change the population, extend the horizon, run it again. Eventually a run looks good, and that's the run in the post. Nobody lied at any point — but the number you're reading is a maximum over a search whose length goes unreported. And it isn't only the reader who can't see that. You can't either, not reliably, not about your own work.

Pre-registration fixes it, and it's embarrassingly cheap. It's a file. Dated, append-only, thresholds written down with a rule for what each one means. Mine carries thirty-six amendments, each flagged with whether I made the change with the relevant output already visible — because I amended the protocol and I amended the protocol after seeing the numbers are different sentences, and only one of them is a confession.

What it costs isn't technical. It costs you the option to quietly not mention the run that went badly. That is the entire price, and it's why almost nobody pays it.

Before this gets too pleased with itself: the outcomes are modelled. I'm a student, and no student has live merchant traffic. Nine parameters govern whether an intervention succeeds; eight are my judgement, tagged [guess] in the source, where a parameter with no provenance tag fails a test. The ninth isn't a measurement either: it's the registered zero for retrying a dead instrument, a structural fact rather than an estimate. No one publishes those rates at this granularity, and inventing a citation would have been the first dishonest sentence in the repository.

So the claim isn't "₹116 crore was recovered." It's that under a stated outcome model, swept ±50% on every parameter across 83 configurations, this policy beats the control by a stated margin and the direction survives all of them. The methodology is the deliverable. Point it at real traffic and it gives you a real number.

Here is where all of that stopped being theoretical.

Every test passed. 1,353 of them. The safety predicate was clean across a thousand seeds. No guard misbehaved. No receipt was missing. No exception was raised.

And a third of my population was doing nothing at all.

The system has a guard for India's e-mandate rules: you cannot debit a customer's standing mandate without serving twenty-four hours' notice first. When it sees a mandate debit with no notice on file, it defers the debit and hands back an obligation — send the notice.

Obligations were honoured on the execute path. And a deferred proposal never executes. So no notice was built. So the timestamp stayed empty. So the guard deferred again, and again, five times, and then blocked the debit for good.

The one thing that could satisfy the guard was an execution the guard was blocking.

I didn't find it with a test. I found it writing an adversarial attack against the mandate flow, which came back passing — and then noticing it could never have failed, because the thing it attacked had never happened. Of 707 retries executed on one seed, zero landed on any of the 275 mandate episodes. Two hundred and nine of those episodes ended blocked. That was the third: thirty-one percent of my population had a retry ladder that never fired once.

Fixing it moved my headline recovery rate from 0.344 to 0.491.

Which means three quarters of what I had been calling the price of compliance was my own deadlock. I'd been telling a tidy story about what it costs to do this properly, and most of that cost was a bug I wrote.

Had it fired, I would have published a conclusion about regulation that was actually a conclusion about me.

Every test I had written asked whether the agent did something wrong. Not one asked whether it did anything at all.

I don't think that gap is mine alone. Safety tests get written as this must not happen, and a system that does nothing passes every one of them, perfectly, forever. A guard that defers indefinitely is indistinguishable from a guard that works. Liveness needs its own assertions, and they feel redundant right up until they aren't.

About those sixteen points: I sat on them for a day. Not deciding whether to publish — I'd written down in advance that I would, which is the only thing writing it down is for. I was deciding how to phrase it so it stung less. Every draft had a softener in it. While the baseline scores higher on raw recovery… The comparison is arguably unfair, since… Both true. Both also the sound of someone negotiating with a number.

The version I shipped states it flat, in the second paragraph, before I've earned any goodwill from the reader. It's the worse-reading version. It's the only one I'd defend.

I run twenty-two adversarial attacks against this thing. Nineteen survive. Three are open, named in the repo, each with the conditions under which it wins.

The claim was never that the agent is correct. It's that being wrong is discoverable — and the evidence for that isn't the nineteen. It's how much of that list the apparatus found instead of me.

So write down what would make you abandon the thing you're building, before you build it, with a number attached. You probably won't fail the way you predicted. I didn't: I specified the wrong failure, and the one that actually mattered got caught by a test I'd written for something else. That is still the apparatus working. What never works is deciding afterwards.