The Backtest Just Works. That's the Problem.

A Short Form Note — the segue into this month’s Digest.

A database CTO wrote a piece this week called “The End of Programming.” Paul Dix — he built InfluxDB — argues the act of writing code by hand and having other people read it is on its way out. Not because the models got clever enough to trust blindly, but because of one move: if you can build a verification system and give proper direction, AI can produce sophisticated software and refine it until it just works. His proof is the Bun 1.4 rewrite — a million lines translated from one language to another by agents, then hardened over months into software running on millions of machines.

He’s right about the mechanism. The verification system is the load-bearing part — I’ve said a version of this all year. But I live one domain over, where the same sentence turns dangerous, and it’s worth making that crossing myself before the Digest goes long tomorrow.

“Refine until it just works” assumes the thing checking the work is on your side. In the Bun case it was. A language rewrite has an oracle — the original code to compare against, and millions of users to break what’s wrong. That oracle is the verification system, and it’s the strongest one there is. The model couldn’t quietly pass itself; the old code and the real world graded every line.

Trading doesn’t hand you that oracle. The grader is the market, and the market is not on your side.

Here’s what that looks like when you forget it. This week an influencer waved a real Swiss paper around — Zarattini, Barbon, and Aziz, an Opening Range Breakout study, “prop firms don’t want you reading this,” 1,600% returns. The paper is real, and it’s free on SSRN; nobody’s hiding it, and one of the three authors runs a paid day-trading school, which is worth knowing before you trust the number. The 1,600% is real too — as a backtest.

And a backtest is a friendly grader. You run the strategy against the past and refine it until it just works. That’s Dix’s loop with the adversary taken out. When someone independently replicated the authors’ sibling paper — same method, different index — and added the parts the backtest left out, the edge broke even around two cents a share of slippage, most of the profit came from a single year, and a flashier variant fell out of a parameter search. Refine-until-it-works, against data that can’t fight back, produces exactly that: a number that’s true about the past and says nothing about your fills.

So the principle holds, and it sharpens. Verification is everything — Dix is right. But verification only means something when the verifier is trying to make you fail. In software you often get that for free: the oracle, the users, the crash that won’t let you lie. In trading you have to build it on purpose. Out-of-sample data you genuinely didn’t peek at. Costs modeled where they actually bite — for news-driven small-caps, that’s brutal slippage and borrow you can’t get. A live-shadow period before a dollar is at risk. A demotion path that pulls capital the moment the edge stops confirming, no ego about it.

I run this split in my own book. One strategy is a market maker built from a 2008 paper — it earns a spread for absorbing inventory, not a forecast of the next candle, so its verifier is honest about what it’s actually claiming. The other is a pattern strategy, and it sits in ghost trading — generating signals, touching no orders — precisely because it fails that bar. It’s a directional bet dressed in pattern-recognition language, the exact shape a friendly backtest flatters and the market selects against. No amount of “refine until it works” fixes a strategy the market is quietly grading down.

“It just works” is a verification claim. The only question that matters is what did the verifying. If the answer is the past, you didn’t test the strategy — you fit it. The market will try to break it either way — make sure your own tests get there first.

Tomorrow, the Digest goes long on it: why access to all this stopped being the constraint… and method never was.

Sources: Paul Dix, “The End of Programming” (2026-08-25). Carlo Zarattini, Andrea Barbon & Andrew Aziz, “A Profitable Day Trading Strategy for the U.S. Equity Market” (SSRN 4729284). Independent cost/robustness replication of the authors’ sibling QQQ study: giovannibrusco/zarattini-2023-orb-qqq. And my earlier field notes, The Backtest Isn’t the Edge and Adversarial Dynamical Systems and the Ceiling on AI Trading.