The Backtest Isn't the Edge

Every strategy looks brilliant on the data it was fit to. That’s not a discovery — that’s arithmetic. The question that actually matters is whether it survives data it never saw, tested in a way that doesn’t quietly leak the future into the past.

I found a library this week that bakes that discipline in, and it’s worth knowing about even if you never run it. skfolio does portfolio optimization on scikit-learn’s API — but the part I care about isn’t the optimizer. It’s that it ships Combinatorial Purged Cross-Validation and Walk-Forward as first-class tools. Those are Marcos López de Prado’s answer to the most common way a backtest lies: overlapping train and test windows that let information bleed across the split, so your “out-of-sample” Sharpe is really in-sample wearing a disguise.

Purged CV cuts the samples around each test fold so the leak can’t happen. Combinatorial CV then runs many train/test partitions instead of one, and hands you a distribution of outcomes instead of a single hopeful number… and the spread is where the truth lives. A strategy with a great mean and an ugly spread isn’t an edge — it’s a coin that landed your way.

I don’t run skfolio’s allocation engine; my work is single-instrument, not a book of assets. But the validation spine is asset-count-agnostic, and it’s the exact discipline I already hold myself to before anything touches live money: a forecast is not an edge, and a backtest is not a result. What the library does is make that discipline a default — an import, not an act of willpower.

Here’s the through-line I keep coming back to, across trading and everything else I build: don’t trust the “done.” An agent that reports a task finished and a backtest that reports a strategy has an edge are the same claim — a self-graded pass. The fix is the same too. Don’t ask the thing that did the work whether the work is good. Make it survive a test it can’t game.

The edge was never in the backtest. It’s in whether the backtest could have been fair and still said yes.

Sources: skfolio (github skfolio/skfolio; arXiv 2507.04176) — sklearn-native portfolio optimization with Combinatorial Purged CV + Walk-Forward. Marcos López de Prado, “Advances in Financial Machine Learning” — purged & combinatorial-purged cross-validation.