> ## Content Index > Fetch the complete content index at: https://thomasadair.ghost.io/llms.txt > Use this file to discover other available public pages before exploring further. # The Karpathy Loop Finds Mirages - URL: https://thomasadair.ghost.io/karpathy-loop-finds-mirages-short-form/ - Published: 2026-09-07T21:30:00.000Z - Updated: 2026-09-07T21:29:59.000Z - Author: Thomas Arthur Adair - Tags: short-form-note, Trading, Trident, #short-form-note Andrej Karpathy keeps a CLAUDE.md — one page of rules for getting a language model to write code you won’t have to rewrite. Read the files before you touch them. Keep the diff small. When something breaks, find out *why* the value is null instead of adding a null check, or “the bug just moves somewhere quieter.” The rules are good. I run that loop every day. But the reason it works is narrower than it looks, and the gap matters the moment you point the loop at a market. The engine of the loop is a test: write the failing test, watch it fail, fix the cause, watch it pass. That cycle is trustworthy because of one property nobody states out loud — **the model did not author the world the test runs against.** A function returns the right value or it doesn’t. Run the loop two hundred times and the two-hundredth pass is exactly as real as the first. Iterate fast, keep what passes, discard the rest: a virtue — *when the grader is a fact outside the searcher’s reach.* Now replace the test with a backtest. A backtest is not outside the searcher’s reach. It’s a number the searcher can push on, and it moves in two directions the loop cannot see. **The first is counting.** Every configuration you try is a draw. Run enough draws against a fixed history and something looks excellent for no reason at all. Bailey, Borwein, López de Prado and Zhu proved the sharp version of this: under memory effects, an overfit backtest doesn’t just fail — it produces *negative* expected out-of-sample returns. It is worse than no backtest, because it points you confidently at a loser. And nobody can assess the damage, because researchers almost never report how many configurations they tried. **The second is memory.** The model already read the period you’re testing on. A recent paper gives it a name — *parametric look-ahead bias*. A model trained in 2024 already knows which way 2018 moved, and no amount of care with your data pipeline removes knowledge that is sitting in the weights. When five named LLM trading systems were run past their training cutoff, every one lost more than half its Sharpe — on a window deliberately chosen so the market return was near-identical. The edge wasn’t skill. It was recall. Read that next to Karpathy’s loop and the collision is exact. The loop’s great strength in software — cheap iteration, keep what passes — is, in trading research, the precise mechanism of the failure. In code, iteration costs nothing. Here you pay for every draw, and the bill arrives out of sample. There are two fixes, and they are not the same fix. The first is arithmetic, and it already exists. Count your trials and deflate for them — the Deflated Sharpe Ratio corrects an observed Sharpe for the number of trials that produced it; the Probability of Backtest Overfitting tells you the odds the winner is in-sample noise. None of it is hard. All of it needs one input almost nobody keeps: an honest count of how many configurations you ran. An automated loop will happily run thousands overnight, and if that number isn’t written down as it happens, it cannot be recovered afterward. The second is architectural, and arithmetic will not substitute for it: **fix the splits, the definitions, and the scoring *outside* the loop’s reach.** When the thing being graded cannot touch the grader, the same class of model that produces contaminated backtests produces real held-out numbers instead. The variable isn’t capability. It’s whether the searcher can reach the answer key. Here’s what I do with it. Two rules carry the weight. **The backtest ranks; forward judges.** In my loop the backtest is just a cheap pre-screen — it kills the obvious losers for almost nothing, and decides nothing else. Whatever survives is then tested on fresh, out-of-sample data — a stretch it was never tuned on. Only if it holds up there does it earn the right to be proposed. And even then, a human gate — mine — stands between a proposal and live capital. That step never automates. **The trial count gets written down as it happens.** Every proposed change lands in a trace store with its outcome, so the deflation has a real denominator instead of a guess. A loop that can’t tell you how many draws it took can’t be corrected for them. I’ve paid for both. The one that taught me most: a strategy of mine backtested at a Sharpe near +19 — until its backtests were routed through a proper fill model. The same trades, priced honestly against what the market would actually have given me, came out negative. Nothing was wrong with the code. The number was measuring a fill that was never available. The mirage wasn’t in the strategy. It was in the instrument I was reading it with. Karpathy’s rules about the *searcher* hold everywhere — read before you write, keep the diff small, find out why it’s null. The one rule that doesn’t carry across is the loop itself, and only because of a property of software so ordinary nobody says it out loud: the test tells the truth no matter how many times you run it. In markets you have to *build* that property, deliberately, and pay for it in speed. Fix the splits outside the loop’s reach. Count the draws. Let the forward window decide. A backtest run two hundred times is not two hundred pieces of evidence. It is one piece of evidence and one hundred and ninety-nine chances to fool yourself. And you don’t dispel a mirage by staring harder — you dispel it by walking toward it. The forward window is the walk. --- ## Sources - Andrej Karpathy, *CLAUDE.md* (coding rules), June 2026. - Bailey, Borwein, López de Prado & Zhu, *Pseudo-Mathematics and Financial Charlatanism*, Notices of the AMS, 2014. - Li, Zeng, Xing, Xu & Xu, *Profit Mirage: Revisiting Information Leakage in LLM-based Financial Agents*, arXiv:2510.07920. - Li, Wang & Ma, *Summoning the Oracle to Slay It: Mitigating Look-Ahead Bias in Financial Backtesting with Large Language Models*, arXiv:2605.24564. - Bailey & López de Prado, *The Deflated Sharpe Ratio*, Journal of Portfolio Management, 2014. - Bailey, Borwein, López de Prado & Zhu, *The Probability of Backtest Overfitting*, Journal of Computational Finance, 2017.