A Pilot Beats a Prediction
When you have more ideas than you can afford to test, the pull is to get clever about picking the winner before you spend a dime — build something that reads all your candidates and tells you which one to run. A team at FAIR just tested that idea head-on, and the result is the one I’d have bet on from my own work: the version that runs a small, cheap experiment before it decides beats the version that only predicts.[1]
An AI research agent can now propose, write, and run its own machine-learning experiments. The catch is cost — a candidate idea takes minutes to write and hours to days of GPU time to actually run. So the agent dreams up far more experiments than it can afford to execute, and the whole game becomes what the paper calls research preference: how you spend a fixed budget across a pile of candidates. Their tool for it is a Research Preference Model — an RPM — a model that predicts which candidates are worth running without paying to run them all.
They built it two ways. One reads the candidate plans, the code, the past results, and reasons about which looks best — pure prediction. The other runs a small pilot version of the experiment first, then decides. Both helped. But the one that ran the pilot helped more: it lifted the average score further than the pure-prediction version, reached the unguided agent’s 24-hour result in about 15 hours, and did it on under two-thirds of the compute budget.[1] The probe won.
I’ve been running a version of that rule for a while, in a place where the bill is real. I build and run a live-money trading stack, and I turn out more strategy ideas than I can properly test — every honest backtest costs time and compute. The rule I landed on, the hard way, has the same shape as the paper’s result: don’t build a fortune-teller that ranks your ideas and crowns a winner on paper. Try the cheap thing first. Let a verify step catch what fails. Spend the big compute only when the cheap signal has earned it. Escalate, don’t predict.
On the surface an RPM sounds like the exact opposite of that — it’s a prediction model, and I just said don’t predict. But look at which version won. It wasn’t the one that guesses from the plan. It was the one that pays a little to look before it leaps. The pure-prediction model is real, it’s cheaper, and it helps — but the one that reaches further is the one that runs the pilot. A prediction is a story you tell about a candidate… a pilot is the smallest true thing you can learn about it.
What that hands me is a cleaner way to run strategy search. A preference model as a pre-screen: rank the candidates cheaply, run small pilots on the handful worth a closer look, and commit a full backtest only to what survives. It’s the same pre-screen I already put in front of the expensive tests — now with a smarter way to spend the middle step.
One thing it does not do is let you skip the proof. A pilot that looks good is not an edge, the same way a backtest that looks good is not an edge — I’ve said that before and I still mean it. The pilot doesn’t tell you a strategy works. It tells you which strategy is worth the cost of finding out. That’s a budgeting move, not a validation move, and keeping those two apart is most of the discipline.
So the lesson isn’t guess better. It’s make the test cheap enough that you don’t have to guess. Pay a little to look, then pay a lot to commit — in that order, the cheap probe first, every time.
[1] Thomas Simon Foster et al., “AI Research Preference Models,” arXiv:2608.13940 (FAIR at Meta / University of Oxford / UCL, 14 Aug 2026). RPMs are built from frozen pretrained language models with no task-specific training. The inference-only variant raised the average normalized score from 0.684 to 0.711; the agentic variant, which runs small pilot experiments before deciding, raised it to 0.729 — and reached the unguided agent’s 24-hour performance in roughly 15 hours on under two-thirds of the execution budget. https://arxiv.org/abs/2608.13940