Why AI coding agents p-hack backtests, and how a chained tries log catches it
Give a coding agent a folder of historical prices, a strategy script and a common prompt: "Test a short-term pullback strategy on S&P 500 stocks and improve its risk-adjusted return."
Twenty minutes later the agent returns a clean script with a smooth equity curve, a tidy git diff and a strong Sharpe ratio. You can read every line, confirm there is no look-ahead bias, verify the commissions, and still be looking at noise.
What disappeared along the way are the 180 parameter combinations, alternative indicators, regime filters and exit rules the agent tried in its loop before keeping the one that survived. An agent does not need to introduce a bug to mislead you: automating iteration compresses a month of human data-mining into an afternoon of selection bias under multiple testing.
The maths of an afternoon of iteration
Under i.i.d. normal returns with no autocorrelation or regime shift (Lo, 2002), the standard error of an annualised Sharpe ratio over ten years is roughly 1 / sqrt(10) ≈ 0.32. On a single honest hypothesis, an observed Sharpe ratio of 0.95 sits roughly three standard errors above zero (Phi(z) > 99%).
Now let an agent run 300 variations on that same ten-year history. Even if every variation has zero true edge, picking the maximum of 300 noisy draws pushes the best observed Sharpe upward rapidly.
David Bailey and Marcos López de Prado formalised this in the Deflated Sharpe Ratio (2014). Given N trials with average return correlation ρ and cross-trial Sharpe standard deviation σ, the effective number of independent trials is:
N_eff = ρ + (1 - ρ) * N
SR0 = σ * ((1 - γ) * Phi^-1(1 - 1 / N_eff) + γ * Phi^-1(1 - 1 / (N_eff * e)))
z = (SR - SR0) * sqrt(T - 1) / sqrt(1 - g3 * SR + (g4 - 1) / 4 * SR^2)
DSR = Phi(z)where γ ≈ 0.5772 is the Euler-Mascheroni constant, T is the number of return observations, g3 is daily skewness and g4 is daily non-excess kurtosis (3 for a normal distribution). Note the scale: SR, SR0 and σ enter the z denominator in per-period (daily) units (annual ÷ sqrt(252) for daily stock returns), not annualised units.
Using one fixed return profile across all three rows (T = 2,520 daily returns, g3 = -0.4, g4 = 6.0, with SR, SR0 and σ shown annualised):
| Variations tried (N) | Effective trials (N_eff) | Luck’s best Sharpe (SR0) | Observed Sharpe | Deflated Sharpe (DSR) | Verdict |
|---|---|---|---|---|---|
| 1 (first try) | 1.0 | 0.00 | 0.95 | 99.8% | Probably not luck |
50 tweaks (σ=0.35, ρ=0.30) | 35.3 | 0.75 | 1.20 | 91.9% | Can’t tell |
300 agent tries (σ=0.50, ρ=0.15) | 255.2 | 1.42 | 1.30 | 35.3% | Probably luck |
One important subtlety: SR0 is sensitive to the cross-trial spread σ. Under pure zero-skill noise over ten years (SE ≈ 0.32), 300 random tries have a smaller spread around σ = 0.32, which puts SR0 near 0.91 and leaves a 1.30 Sharpe at 88.6% (Can’t tell). In real agent exploration, where the loop mixes working ideas, inverted rules and high-turnover variants that bleed commissions, cross-trial spread widens quickly (σ = 0.50 or more), lifting SR0 to 1.42 and dropping that same 1.30 Sharpe to 35.3% (Probably luck). Remember too that Phi(z) is a frequentist test score under i.i.d. assumptions (real serial correlation and regime shifts widen uncertainty further), not an unconditional probability that a strategy will make money live.
You can test how sensitive DSR is to N, σ and ρ in our browser-based Deflated Sharpe Ratio calculator.
Why Deflated Sharpe alone is not enough
Even when a rule family clears Deflated Sharpe, the specific parameter setting you choose can still be noise.
In Rearview’s synthetic sample project (10 years of generated prices, not a real market), down_days_overtuned searches 50 combinations of two parameters. Its Deflated Sharpe is 97% (the underlying rule family earns something on the synthetic series even after counting all 50 trials), yet its Probability of Backtest Overfitting (PBO) is 82%: across all half-and-half combinations of time blocks (CSCV, Bailey et al., 2017), the setting that looked best in one half ranked below the median in the other half 82% of the time.
Because you trade the winning setting rather than the average of all 50, Rearview pairs both checks and gives one verdict: Probably luck.
Why manual trial counting fails
The Deflated Sharpe formula takes twenty lines of code. The hard part in practice is knowing N.
Humans forget last Tuesday’s runs, delete weak equity curves, and undercount how many ideas they glanced at. AI coding agents amplify that gap: in a typical agent loop, intermediate runs live in /tmp, terminal scrollback, or overwritten notebooks, so by the time you review the winning script, the trial count is gone.
A pattern for automatic trial accounting (and how Rearview implements it)
Any systematic research setup that uses coding agents needs three properties: the agent runs the same engine as the human, every try is appended to a persistent log outside the run folders, and the most recent years are held back in a lockbox until the loop ends.
Here is how we built that into Rearview. Strategies are plain-text .rvw files in a regular folder, the language reference fits on one page (File ▸ Save Language Reference to Project), and rvw check <script.rvw> --json gives an agent structured line-and-column diagnostics so it can fix syntax errors on its own. When the agent tests a script inside the project with:
rvw run --project . --script pullback.rvw(or rvw search / rvw walkforward with --project . --script ... --space ...), the engine appends the run to the project’s tries.jsonl under an exclusive file lock, right beside every run and preview made in the Mac app.
Four engineering details keep the log useful during messy iteration:
- Token-hashed try keys. Each try is keyed by the first 32 hex digits of the SHA-256 of
{"code", "params", "run"}.codehashes the script’s lexed token stream without comments, spaces or repeated blank lines, andrunomits the benchmark (which changes no trade). Adding comments, adjusting spaces or blank lines, or re-running the exact same setting after adding new market days keeps the same try key; changing a rule, threshold or stop adds a new try. - Lineage events and conservative correlation. Deleting a run folder from disk never erases its tries from
tries.jsonl. Renaming or duplicating a script inside Rearview (or viarvw tries) recordsrenameandcopyevents so the script’s trial history follows it. Becausetries.jsonlstores only[key, sharpe]rather than every historical daily return series, single-runEvery tryDSR conservatively setsN_eff = N(no correlation discount), whereas an in-memory search computes pairwiseρacross its own trials. - SHA-256 hash chaining. Every line in
tries.jsonlrecords a sequence number (seq), UTC timestamp andprev, the SHA-256 hex digest of the previous line. The Mac app stores the last seen head (tries_head) inrearview-project.json. Editing an earlier line breaks theseq/prevchain and permanently flags the log asedited; deleting or truncating the file sotries_headis missing causes the next append to write a permanentgapevent (missingorcut) and flag the count as potentially low from then on. - Sealed lockbox with embargo. Before starting an agent loop, you can seal the most recent years of data in a lockbox. Rearview suggests the smallest embargo gap needed by the project’s scripts (lookback plus holding period) so indicators cannot reach across the boundary. Every in-app run, search and
--projectCLI invocation (or standalone CLI run given--lockbox-start) stops short of the lockbox until you reveal it once at the end.
Honest limits
A tries log is a research discipline aid, not security. Anyone with filesystem access can edit both files, though casual tampering shows. More importantly, Rearview is explicit about what it still cannot count:
- Runs made outside the project (
rvw run --store ...without--project) or in another backtester. - Scripts copied or renamed outside Rearview with shell
cp/mvor Finder, or brand-new.rvwfiles written from scratch (each separate file tracks its own lineage). - Ideas taken from papers or forums that were themselves selected from dozens of unpublished tests.
For that reason, every Rearview verdict labels its trial count "at least N trials". Pairing an agent loop with --project logging, a sealed lockbox, PBO and walk-forward tests lets you keep the speed of AI-assisted iteration without quietly losing track of how hard you searched.
Read the full specification in Is this result luck?, or test your own trial count in the Deflated Sharpe Ratio calculator.