How Rearview is checked

A backtest is only as good as the engine behind it. This page says how Rearview's engine is checked before every release, what each check catches and what it can't, and the latest results. The figures come from the checks themselves, recorded on the date shown below.

The written rules

The engine follows written rules: how orders fill, how cash and shares move, how splits, dividends, delistings, margin, borrowing and several strategies work, sharing cash or in sleeves that sell to rebalance, with weights that can change by date, how coins trade every day beside stocks, in steps finer than a share, how futures settle every day and roll from one contract to the next, how each exchange trades on its own days, and how an account in one currency holds securities priced in others and forex pairs that earn carry. Small scenarios are worked out by hand from those rules, every fill, cash and share figure written down, and the engine must reproduce each one byte for byte.

Catches: an engine that doesn't do what the rules say. Can't catch: a rule that is itself wrong. The outside engines below check that.

A second engine

A second engine, written separately in Python from the written rules and the hand-worked cases alone, without seeing the main engine's code, runs the same scenarios: the hand-worked ones, and thousands of random ones with random data, signals and settings. Every money and share figure must agree exactly, and every other figure within one part in a trillion. Both must also reject the same invalid settings.

Catches: mistakes in either engine's code, including in the rare combinations no one would think to write a case for. Can't catch: a mistake in the rules, because both engines follow them.

Live accounts

A live account's replay of your fills follows written rules of its own, with hand-worked cases: fills that follow the model exactly, slippage, fewer and more shares, a fill that turns a position around, a missed sell carried and made late, fills on ex-dates and on days without a bar, orders sized from your account after a deposit, and a margin breach. The second engine replays them too. So do over a thousand random accounts made from random runs, with fills dropped, made late, at other prices and sizes, added and turned around, and with deposits and withdrawals: every file must agree byte for byte. Fills that follow the model exactly must give the backtest's own files.

Catches: a replay that applies a fill, carries a sell or compares a fill otherwise than the rules say. Can't catch: your broker booking something differently, such as a corporate action your data doesn't have. The statement check shows those against your statement.

Outside engines

Independent open-source backtesters run the scenarios they model the same way as Rearview, and their results are compared:

• vectorbt: entries, exits and costs on long positions, on daily and 5-minute bars;
• bt: rebalancing to target weights;
• QuantConnect LEAN: short positions, margin accounts, splits, dividends and delistings.

Where an outside engine works differently by design (it sizes orders another way, or handles a margin call differently), the scenarios avoid that difference, and it is written down instead of tested. vectorbt and bt also check runs in fractional shares, and vectorbt checks coin runs over 365-day years and an account in Australian dollars holding US stocks, on prices converted beforehand.

Catches: a rule both of Rearview's engines get wrong in the same way. Can't catch: a difference in what the scenarios leave out.

No look-ahead

A backtest that uses tomorrow's prices looks wonderful and means nothing. Three checks look for it:

• Replay: the whole pipeline (split and dividend adjustment, index membership by date, weekly and monthly bars, ranks, the engine) runs on the data cut off at a date, and its decisions up to that date must be the same as on all the data. Cut dates fall on splits, dividends, delistings, index changes, month ends and the days settings change. Portfolios and whole walk-forwards are replayed the same way, and so is a market whose stocks all list after its index, cut on their first day and the day after. On intraday data the data is also cut inside sessions, at their first and last bars and on half days. A scan is replayed too: on data cut at a day, every example strategy's candidates and every example scan's rows on that day must be those on all the data.
• Perturbation: changing every price after a date must not change a single decision before it, in thousands of random runs.
• Planted leaks: inputs that cheat on purpose (signals that read tomorrow's close, adjustments computed from the whole future, a universe that knows which stocks will be delisted) must be caught, which shows the checks above can see a leak.

Can't catch: a leak in your own data, such as an index list of today's members or a price history already adjusted by its vendor. The data-quality report and the import guide warn about those.

The same answer every run

Cash and shares are counted in whole millionths of a dollar and whole shares, never in rounded decimals. The same inputs must give the same files, byte for byte, at any number of threads, run after run, and on data generated again from scratch.

Genetic and CMA-ES searches choose each generation from how the earlier ones did, so their choices are checked the same way: the same seed gives the same trials and the same folder at any number of threads, and every saved search and walk-forward runs again to the same bytes.

Golden replays

Rearview keeps a set of saved runs, searches and walk-forwards, the reports made from them, two data updates' reports, and a run and a search on data whose index starts before its stocks. It runs each one again from its own folder, and every file must come out byte for byte the same. When a change to Rearview changes any of them on purpose, or changes the results on some kinds of data, the change is listed below with the reason. Your own saved runs can be checked the same way with Check It Reproduces, or all of a project's at once with Check Runs Reproduce.

The entries dated 2026-11-06 and earlier were made before Rearview went on sale, so they affect no buyer's saved runs.

2026-11-06 · engine 0.9.0

The engine's version moves to 0.9.0, in step with the app's, and every run, search and walk-forward records it, so these files name 0.9.0. No figure changed.

• adx_breakout (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• bollinger_reversion (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• breakout_atr (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• hull_trend (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• ibs_reversion (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• keltner_breakout (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• ladder_scale_out (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• long_short_decile (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• low_beta_rotation (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• low_volatility_rank (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• macd_regime (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• momentum_rotation (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• monthly_rebalanced_momentum (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• near_52w_high (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• pullback_scale_out (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• pyramid_breakout (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• rotation_with_buffer (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• rsi2_pullback (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• sector_momentum (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• short_meanrev (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• sma_cross_trend (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• stochastic_pullback (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• supertrend_trend (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• three_down_days (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• trend_timing_etf (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• uncorrelated_pullback (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• volatility_sized_trend (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• volatility_targeted_weights (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• weekly_rsi_trend (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• long_short_margin (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• short_meanrev_margin (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• same_close (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• volume_cap (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• fractional_rebalance (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• lockbox_sealed (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• shared (portfolio):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• sleeves (portfolio):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• sleeves_rebalanced (portfolio):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• sleeves_selling (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• search_grid (search):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• search_formula (search):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• search_genetic (search):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• search_cmaes (search):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• search_random (search):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• search_portfolio (search):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• search_weights (search):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• search_benchmark (search):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• walkforward_stop (walkforward):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"
  • oos/manifest.json: "engine_version" "0.1.0" → "0.9.0"

• walkforward_genetic (walkforward):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"
  • oos/manifest.json: "engine_version" "0.1.0" → "0.9.0"

• walkforward_portfolio (walkforward):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"
  • oos/manifest.json: "engine_version" "0.1.0" → "0.9.0"

• walkforward_weights (walkforward):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"
  • oos/manifest.json: "engine_version" "0.1.0" → "0.9.0"

• search_signals (derived):

  • manifest.json: line 3: "engine_version": "0.1.0", → "engine_version": "0.9.0",

• walkforward_signals (derived):

  • manifest.json: line 3: "engine_version": "0.1.0", → "engine_version": "0.9.0",
  • oos/manifest.json: line 2: "engine_version": "0.1.0", → "engine_version": "0.9.0",

• report_run (derived):

  • report.html: line 57, from column 321: class="mono">55b011f4</span> · engine 0.1.0</p><div class="hero"><div><b>+4.7%</b><span>CAGR</span></div><div><b>+9.4%<… → class="mono">55b011f4</span> · engine 0.9.0</p><div class="hero"><div><b>+4.7%</b><span>CAGR</span></div><div><b>+9.4%<…

• report_portfolio (derived):

  • report.html: line 57, from column 309: class="mono">55b011f4</span> · engine 0.1.0</p><div class="hero"><div><b>+8.2%</b><span>CAGR</span></div><div><b>+16.9%… → class="mono">55b011f4</span> · engine 0.9.0</p><div class="hero"><div><b>+8.2%</b><span>CAGR</span></div><div><b>+16.9%…

• report_search (derived):

  • report.html: line 68, from column 327: class="mono">55b011f4</span> · engine 0.1.0</p><h2>Is this result luck?</h2><p class="sub">The best of 36 trials, by Sh… → class="mono">55b011f4</span> · engine 0.9.0</p><h2>Is this result luck?</h2><p class="sub">The best of 36 trials, by Sh…

• report_search_benchmark (derived):

  • report.html: line 68, from column 330: class="mono">55b011f4</span> · engine 0.1.0</p><h2>Is this result luck?</h2><p class="sub">The best of 4 trials, by Sha… → class="mono">55b011f4</span> · engine 0.9.0</p><h2>Is this result luck?</h2><p class="sub">The best of 4 trials, by Sha…

• report_search_genetic (derived):

  • report.html: line 68, from column 330: class="mono">55b011f4</span> · engine 0.1.0</p><h2>Is this result luck?</h2><p class="sub">The best of 60 trials, by Sh… → class="mono">55b011f4</span> · engine 0.9.0</p><h2>Is this result luck?</h2><p class="sub">The best of 60 trials, by Sh…

• report_walkforward (derived):

  • report.html: line 68, from column 415: class="mono">55b011f4</span> · engine 0.1.0</p><h2>Did it hold up out of sample?</h2><p class="sub">6 folds, 4 settings… → class="mono">55b011f4</span> · engine 0.9.0</p><h2>Did it hold up out of sample?</h2><p class="sub">6 folds, 4 settings…

• benchmark_run (run):

  • manifest.json: "engine_version" "0.1.0" → "0.9.0"

• report_run_tries (derived):

  • report.html: line 57, from column 303: class="mono">55b011f4</span> · engine 0.1.0</p><div class="hero"><div><b>+8.1%</b><span>CAGR</span></div><div><b>+16.7%… → class="mono">55b011f4</span> · engine 0.9.0</p><div class="hero"><div><b>+8.1%</b><span>CAGR</span></div><div><b>+16.7%…

• late_listing (derived):

  • run/manifest.json: line 2: "engine_version": "0.1.0", → "engine_version": "0.9.0",

• search_late (derived):

  • search/manifest.json: line 3: "engine_version": "0.1.0", → "engine_version": "0.9.0",

• report_stress (derived):

  • report.html: line 57, from column 305: class="mono">55b011f4</span> · engine 0.1.0</p><div class="hero"><div><b>+7.4%</b><span>CAGR</span></div><div><b>+15.1%… → class="mono">55b011f4</span> · engine 0.9.0</p><div class="hero"><div><b>+7.4%</b><span>CAGR</span></div><div><b>+15.1%…

• live_replay (derived):

  • acct/backtest/manifest.json: line 2: "engine_version": "0.1.0", → "engine_version": "0.9.0",
  • acct/replay/manifest.json: line 2: "engine_version": "0.1.0", → "engine_version": "0.9.0",

• report_live (derived):

  • report.html: line 57, from column 41435: le><tr><td>Engine</td><td class="mono">0.1.0</td></tr><tr><td>Data fingerprint</td><td class="mono">55b011f48d8d0a9bd1e2… → le><tr><td>Engine</td><td class="mono">0.9.0</td></tr><tr><td>Data fingerprint</td><td class="mono">55b011f48d8d0a9bd1e2…

• coins_trend (derived):

  • run/manifest.json: line 2: "engine_version": "0.1.0", → "engine_version": "0.9.0",

• stocks_and_coins (derived):

  • run/manifest.json: line 2: "engine_version": "0.1.0", → "engine_version": "0.9.0",

• search_coins (derived):

  • search/manifest.json: line 3: "engine_version": "0.1.0", → "engine_version": "0.9.0",

• report_coins (derived):

  • report.html: line 57, from column 301: class="mono">4f50212e</span> · engine 0.1.0</p><div class="hero"><div><b>+4.9%</b><span>CAGR</span></div><div><b>+61.6%… → class="mono">4f50212e</span> · engine 0.9.0</p><div class="hero"><div><b>+4.9%</b><span>CAGR</span></div><div><b>+61.6%…

• futures_trend (derived):

  • run/manifest.json: line 2: "engine_version": "0.1.0", → "engine_version": "0.9.0",

• stocks_and_futures (derived):

  • run/manifest.json: line 2: "engine_version": "0.1.0", → "engine_version": "0.9.0",

• report_futures (derived):

  • report.html: line 57, from column 303: class="mono">12de05e6</span> · engine 0.1.0</p><div class="hero"><div><b>+7.6%</b><span>CAGR</span></div><div><b>+108.3… → class="mono">12de05e6</span> · engine 0.9.0</p><div class="hero"><div><b>+7.6%</b><span>CAGR</span></div><div><b>+108.3…

• aud_account_us_stocks (derived):

  • run/manifest.json: line 2: "engine_version": "0.1.0", → "engine_version": "0.9.0",

• two_exchanges (derived):

  • run/manifest.json: line 2: "engine_version": "0.1.0", → "engine_version": "0.9.0",

• forex_trend (derived):

  • run/manifest.json: line 2: "engine_version": "0.1.0", → "engine_version": "0.9.0",

• sleeves_forex (derived):

  • run/manifest.json: line 2: "engine_version": "0.1.0", → "engine_version": "0.9.0",

• report_currencies (derived):

  • report.html: line 57, from column 296: class="mono">bd268026</span> · engine 0.1.0</p><div class="hero"><div><b>+0.6%</b><span>CAGR</span></div><div><b>+6.4%<… → class="mono">bd268026</span> · engine 0.9.0</p><div class="hero"><div><b>+0.6%</b><span>CAGR</span></div><div><b>+6.4%<…

• intraday_orb (derived):

  • run/manifest.json: line 2: "engine_version": "0.1.0", → "engine_version": "0.9.0",

• intraday_vwap (derived):

  • run/manifest.json: line 2: "engine_version": "0.1.0", → "engine_version": "0.9.0",

• intraday_overnight (derived):

  • run/manifest.json: line 2: "engine_version": "0.1.0", → "engine_version": "0.9.0",

• report_intraday (derived):

  • report.html: line 57, from column 300: class="mono">4fcc87b2</span> · engine 0.1.0</p><div class="hero"><div><b>-47.6%</b><span>CAGR</span></div><div><b>-26.5… → class="mono">4fcc87b2</span> · engine 0.9.0</p><div class="hero"><div><b>-47.6%</b><span>CAGR</span></div><div><b>-26.5…

2026-11-04 · engine 0.1.0

Analyses and run reports add how much of the closed trades' P&L the best five trades made, and the profit factor and expectancy without them. No figure changed.

• report_run (derived):

  • report.html: line 57, from column 34881: 8 in interest and rebates</p><div class="grid"><div class="card"><h3>By sector</h3><p class="def">Your data's sectors no… → 8 in interest and rebates</p><div class="card"><p>The best 5 trades made 16528% of the closed trades' P&amp;L; without t…

• report_portfolio (derived):

  • report.html: line 57, from column 29350: from interest or rebates</p><div class="grid"><div class="card"><h3>By sector</h3><p class="def">Your data's sectors no… → from interest or rebates</p><div class="card"><p>The best 5 trades made 149% of the closed trades' P&amp;L; without the…

• report_walkforward (derived):

  • report.html: line 68, from column 26808: from interest or rebates</p><div class="grid"><div class="card"><h3>By sector</h3><p class="def">Your data's sectors no… → from interest or rebates</p><div class="card"><p>The best 5 trades made 71% of the closed trades' P&amp;L; without them…

• analyze_run (derived):

  • analyze.json: line 723: "costs": { → "concentration": {

• analyze_benchmark (derived):

  • analyze.json: line 535: "costs": { → "concentration": {

• analyze_tries (derived):

  • analyze.json: line 528: "costs": { → "concentration": {

• report_run_tries (derived):

  • report.html: line 57, from column 28676: from interest or rebates</p><div class="grid"><div class="card"><h3>By sector</h3><p class="def">Your data's sectors no… → from interest or rebates</p><div class="card"><p>The best 5 trades made 76% of the closed trades' P&amp;L; without them…

• report_stress (derived):

  • report.html: line 57, from column 28677: from interest or rebates</p><div class="grid"><div class="card"><h3>By sector</h3><p class="def">Your data's sectors no… → from interest or rebates</p><div class="card"><p>The best 5 trades made 82% of the closed trades' P&amp;L; without them…

• report_live (derived):

  • report.html: line 57, from column 24152: deposited less withdrawn</p><div class="grid"><div class="card"><h3>By sector</h3><p class="def">Your data's sectors no… → deposited less withdrawn</p><div class="card"><p>The best 5 trades made 206% of the closed trades' P&amp;L; without the…

• report_coins (derived):

  • report.html: line 57, from column 113919: from interest or rebates</p><div class="grid"><div class="card"><h3>By sector</h3><p class="def">Your data's sectors no… → from interest or rebates</p><div class="card"><p>The best 5 trades made 187% of the closed trades' P&amp;L; without the…

• report_futures (derived):

  • report.html: line 57, from column 105140: from interest or rebates</p><div class="grid"><div class="card"><h3>By sector</h3><p class="def">Your data's sectors no… → from interest or rebates</p><div class="card"><p>The best 5 trades made 107% of the closed trades' P&amp;L; without the…

• report_currencies (derived):

  • report.html: line 57, from column 102520: from interest or rebates</p><div class="grid"><div class="card"><h3>By sector</h3><p class="def">Your data's sectors no… → from interest or rebates</p><div class="card"><p>The best 5 trades made 25803% of the closed trades' P&amp;L; without t…

• report_intraday (derived):

  • report.html: line 57, from column 12942: from interest or rebates</p><div class="grid"><div class="card"><h3>By sector</h3><p class="def">Your data's sectors no… → from interest or rebates</p><div class="card"><p>The best 5 trades made $1,057, and all 954 closed trades -$26,471; wit…

2026-11-02 · engine 0.1.0

where the money came from: analyze and the run report add results by security and sector, the worst falls and figures against the benchmark

• report_run (derived):

  • report.html: line 57, from column 34717: $22,343</td></tr></table></div></div><h2>Returns by year and month</h2><p class="sub">Each period from the previous peri… → $22,343</td></tr></table></div></div><h2>Where the money came from</h2><p class="sub">From $100,000 to $109,399: the tra…

• report_portfolio (derived):

  • report.html: line 57, from column 29197: </td><td>1.00</td></tr></table></div><h2>Returns by year and month</h2><p class="sub">Each period from the previous peri… → </td><td>1.00</td></tr></table></div><h2>Where the money came from</h2><p class="sub">From $100,000 to $116,856: the tra…

• report_walkforward (derived):

  • report.html: line 68, from column 26655: $17,935</td></tr></table></div></div><h2>Returns by year and month</h2><p class="sub">Each period from the previous peri… → $17,935</td></tr></table></div></div><h2>Where the money came from</h2><p class="sub">From $100,000 to $117,794: the tra…

• analyze_run (derived):

  • analyze.json: line 2: "costs": { → "benchmark": null,

• analyze_tries (derived):

  • analyze.json: line 2: "costs": { → "benchmark": null,

• report_run_tries (derived):

  • report.html: line 57, from column 28523: $16,747</td></tr></table></div></div><h2>Returns by year and month</h2><p class="sub">Each period from the previous peri… → $16,747</td></tr></table></div></div><h2>Where the money came from</h2><p class="sub">From $100,000 to $116,665: the tra…

• report_live (derived):

  • report.html: line 57, from column 23957: n">$383</td></tr></table></div></div><h2>Returns by year and month</h2><p class="sub">Each period from the previous peri… → n">$383</td></tr></table></div></div><h2>Where the money came from</h2><p class="sub">From $50,000 to $55,200: the trade…

• report_coins (derived):

  • report.html: line 57, from column 113766: </td><td>1.00</td></tr></table></div><h2>Returns by year and month</h2><p class="sub">Each period from the previous peri… → </td><td>1.00</td></tr></table></div><h2>Where the money came from</h2><p class="sub">From $100,000 to $161,555: the tra…

• report_futures (derived):

  • report.html: line 57, from column 104979: </td><td>1.00</td></tr></table></div><h2>Returns by year and month</h2><p class="sub">Each period from the previous peri… → </td><td>1.00</td></tr></table></div><h2>Where the money came from</h2><p class="sub">From $5,000,000 to $10,413,903: th…

• report_currencies (derived):

  • report.html: line 57, from column 102360: A$2,416</td></tr></table></div></div><h2>Returns by year and month</h2><p class="sub">Each period from the previous peri… → A$2,416</td></tr></table></div></div><h2>Where the money came from</h2><p class="sub">From A$1,000,000 to A$1,064,480: t…

• report_intraday (derived):

  • report.html: line 57, from column 12789: $26,471</td></tr></table></div></div><h2>Returns by year and month</h2><p class="sub">Each period from the previous peri… → $26,471</td></tr></table></div></div><h2>Where the money came from</h2><p class="sub">From $100,000 to $73,529: the trad…

2026-10-31 · engine 0.1.0

Runs warn FILL_OVER_VOLUME when, with no volume cap, a fill is larger than all its bar traded. No figure changed.

• intraday_vwap (derived):

  • run/warnings.json: line 1: [] → [

2026-10-31 · engine 0.1.0

Runs warn SUSPECT_ACTION when a held position takes a split or dividend its prices don't back. No figure changed.

• sleeves_selling (run):

  • warnings.json: "" [{"code":"DELIST_ASSUMED","date":"2017-01-09","id":20,"message":"delisting reason is OTHER or UNKNOWN; the exchange h…

2026-10-26 · engine 0.1.0

Searches record the Reality Check: whether the best setting's Sharpe ratio beats the benchmark's after counting every try, or why there's no comparison (no benchmark in the run settings). No figure changed.

• search_grid (search):

  • luck.json: "reality_check" null → null; "reality_check_skipped" null → "no benchmark is set in the run settings"

• search_formula (search):

  • luck.json: "reality_check" null → null; "reality_check_skipped" null → "no benchmark is set in the run settings"

• search_random (search):

  • luck.json: "reality_check" null → null; "reality_check_skipped" null → "no benchmark is set in the run settings"

• search_portfolio (search):

  • luck.json: "reality_check" null → null; "reality_check_skipped" null → "no benchmark is set in the run settings"

• search_weights (search):

  • luck.json: "reality_check" null → null; "reality_check_skipped" null → "no benchmark is set in the run settings"

• search_late (derived):

  • search/luck.json: line 29, from column 26: ds at least 30 trials; this search has 4" → ds at least 30 trials; this search has 4",

2026-10-26 · engine 0.1.0

Run reports and analyses gain the trade order: the closed trades replayed in 2,000 other orders, for the drawdowns their order alone could give. No figure changed.

• report_run (derived):

  • report.html: line 57, from column 45686: n"><b>-14.6%</b></td></tr></table></div><h2>Trades</h2><p class="sub">103 closed trades: P&amp;L over entry value</p><di… → n"><b>-14.6%</b></td></tr></table></div><p class="sub">The 103 closed trades one after another, each as a share of the e…

• report_portfolio (derived):

  • report.html: line 57, from column 40247: "n"><b>-8.1%</b></td></tr></table></div><h2>Trades</h2><p class="sub">127 closed trades: P&amp;L over entry value</p><di… → "n"><b>-8.1%</b></td></tr></table></div><p class="sub">The 127 closed trades one after another, each as a share of the e…

• report_walkforward (derived):

  • report.html: line 68, from column 33471: "n"><b>-3.7%</b></td></tr></table></div><h2>Walk-forward folds</h2><p class="sub">Each window's best setting on its in-s… → "n"><b>-3.7%</b></td></tr></table></div><p class="sub">The 105 closed trades one after another, each as a share of the e…

• analyze_run (derived):

  • analyze.json: line 1017: "trades": { → "trade_order": {

2026-10-23 · engine 0.1.0

Report axes give their ticks the decimals the step needs, as the app does: a scatter axis in 2.5% steps read -8% for -7.5%. Years and months on a time axis thin out as in the app, and a walk-forward under a year gives its length in trading days. No figure changed.

• report_walkforward (derived):

  • report.html: line 68, from column 37982: "8" y2="240"/><text x="121.9" y="255.0">-8%</text><line class="grid-line" x1="217.9" x2="217.9" y1="8" y2="240"/><text x… → "8" y2="240"/><text x="121.9" y="255.0">-7.5%</text><line class="grid-line" x1="217.9" x2="217.9" y1="8" y2="240"/><text…

2026-10-22 · engine 0.1.0

Reports label their charts better: an equity axis within one decade gets round dollar steps rather than a single label, and a run under a year gives its length in trading days and marks months or days along its dates. No figure changed.

• report_run (derived):

  • report.html: line 57, from column 773: ne class="grid-line" x1="0" x2="936" y1="145.1" y2="145.1"/><text x="942.0" y="149.1">$100,000</text><line class="grid-l… → ne class="grid-line" x1="0" x2="936" y1="251.3" y2="251.3"/><text x="942.0" y="255.3">$90,000</text><line class="grid-li…

• report_portfolio (derived):

  • report.html: line 57, from column 849: 00,000</text><line class="grid-line" x1="240.1" x2="240.1" y1="8" y2="260"/><line class="grid-line" x1="712.8" x2="712.8… → 00,000</text><line class="grid-line" x1="0" x2="936" y1="122.7" y2="122.7"/><text x="942.0" y="126.7">$110,000</text><li…

• report_walkforward (derived):

  • report.html: line 68, from column 3627: 00,000</text><line class="grid-line" x1="616.8" x2="616.8" y1="8" y2="260"/><polyline class="s-strategy" points="0.0,218… → 00,000</text><line class="grid-line" x1="0" x2="936" y1="160.3" y2="160.3"/><text x="942.0" y="164.3">$105,000</text><li…

2026-10-22 · engine 0.1.0

Runs now start on the first day a stock or ETF has a bar, not on the first day of the data's calendar (rule C78). None of the saved results changed. A run of your own changes only when its data has an index whose bars start before every stock and ETF, and the run has no later start date. It now starts later, so Check It Reproduces says it differs; run it again for the new figures.

Updates

When your data is updated with new days and nothing earlier changed, runs made before the update must still reproduce, on the new data cut at the old data's last day. A property test checks it on generated markets cut at random days, as an export made that day would hold them: every example strategy, with random settings and behind a lockbox, must give the same result on the old data and on the updated data cut at that day, through listings, delistings and ticker changes that came later. Every kind of change to history (a price, a corporate action, a delisting, a ticker, index membership, a calendar day, a series value, a name or a sector) must be found and dated on the first day a run could see it.

A file of only the newest days (a daily file) is laid over the data under the same rule: a row on or before the data's last day must be the one the data has. The property test cuts the synthetic market at five days, takes the rest as a daily file that repeats none, one or five of the days before, and requires the updated data to be the data a whole export gives, with the same fingerprint and factors. A file that changes history, or adds nothing, must be refused without writing anything.

Can't catch: a change your data source makes without saying so to days after its last export, which aren't history yet.

Imports

The importer and the converter read whatever files they are given, so they are fed hundreds of thousands of broken ones: valid files with bytes flipped, cut short or added, lines repeated, dropped or swapped, fields added, removed or replaced by odd values (huge, negative, not a number, impossible dates), columns renamed, files emptied or missing, and damaged stores. Nothing may crash; every refusal must say which file, line and rule; every import that still passes must load back unchanged; and the same data written differently (rows shuffled, other line endings, needless quotes) must import the same.

The Sharadar converter is checked by round trip. A store is written out as Sharadar-shaped tables, with prices adjusted for splits as the vendor's are, and the converter must give back that store, with the same fingerprint: its delistings and their reasons, its splits and dividends, its index changes and its as-traded prices. It also lists every step in the adjustment that no split in its actions explains, and every split whose step is missing from the prices.

Can't catch: wrong data in a valid file. The data-quality report looks for that.

Invariants

On every day of thousands of random runs: equity equals cash plus positions; buying power is never exceeded and short proceeds never add to it; nothing trades on a day without a bar; in a shared pool, two strategies never hold the same stock.

The day's choices rest on the run's own orders. At every close of every saved run and every cross-checked scenario, the candidates marked ordered are exactly the entry and rebalance-buy orders the run made at that close, and a run makes the same files with its choices kept or not. Every candidate has one outcome; places run 1, 2, 3 and so on; and the ranked that found no free slot are the ranked less those ordered or skipped. Hand-worked cases cover each outcome. A scan of a strategy on any day lists exactly its signals' entry rows, for every example script, and the candidates at a run's last close.

Indicators and statistics

Each indicator is checked against values computed separately from its published formula, including its warm-up. The Deflated Sharpe Ratio, the Probability of Backtest Overfitting and the Monte Carlo bootstrap are checked against separate implementations of the papers' procedures.

Genetic and CMA-ES searches are tested on surfaces whose highest point is known, a bowl and a many-peaked surface of 125,000 settings each: within 1,000 trials both find the top for every one of 20 seeds, and neither runs a setting twice. Each window of a genetic walk-forward picks the same winner as a search of that window's days alone.

The luck checks of a run and a search are checked the same way. A second implementation, written from their rules and hand-worked cases alone, draws the same random entries bit for bit for every example script, and those entries give the same trades in both engines. It also gives the same trade order for every saved run, and the same Reality Check for searches rebuilt one trial at a time.

So are the stress tests. Each is an ordinary run with its costs doubled or its signals changed, so the written rules cover it as they cover any run. Hand-worked cases set out doubled costs (minimums and per-contract commissions too), signals made a day late (across a halt, on the last bar, rows that land on the same day, intraday moments), entries missed for two strategies on both sides, and later starts' days and figures. The second implementation changes every example script's signals the same way, missing the same entries bit for bit; the changed signals give the same trades in both engines, and the figures are summed up the same way. A strategy that peeks at tomorrow's prices loses its edge when it trades a day late.

Where the money came from

The figures by security, sector and industry, the worst falls and the figures against the benchmark are worked out again by a separate implementation from their written definitions, for every saved run, money to the cent and every other figure within one part in a trillion. On every saved run, futures, forex and live accounts included, the securities add up to the trades, the worst fall is the run's max drawdown, and what no trade made is exactly the interest on cash, margin interest and short rebates the run booked.

Speed

Fixed budgets for a cold run, a warm run, a scan of the whole market on a day and over a year, a run's choices, 10,000 parameter variants, 1,000-trial grid, genetic and CMA-ES searches, counting a script's tries, a script that stacks the slowest indicators, adding a day to the data, stress tests of runs on 500 and on 5,000 stocks, runs on the same stocks beside 200 coins and beside 100 futures chains, 5,000 euro stocks in a dollar account, 2,500 stocks on each of two exchanges' days, and converting 30 years of futures contracts are measured on synthetic data of 5,000 stocks over ten years, and a release must stay within them.

What isn't checked yet

• A rule both engines and every outside engine get wrong in the same way. The hand-worked cases are the only check on those.
• Other backtesters on the same data. Running classic strategies in RealTest and AmiBroker on identical data, with every difference explained, is still to come.
• A real Sharadar download. The converter has only read tables made from a synthetic store; its column names are those of the vendor's published tables, and it assumes the open, high, low and volume are adjusted for splits as the close is.
• A real broker statement. The statement check is built, but it hasn't yet reconciled a real month of margin interest and borrow fees, and the Interactive Brokers reader hasn't yet read a real export.
• Real futures contracts. The futures converter has only read contract files made from synthetic data (Yahoo serves no expired contracts), and no outside engine checks a futures run yet: both engines and the hand-worked cases are the check on them.
• Forex and other exchanges against outside engines. No outside engine checks a forex pair's settlement and carry, or a run on two exchanges' days; both engines and the hand-worked cases are the check on them. Whether a broker's carry matches the rate difference depends on its spread.
• The short-sale price test (Reg SHO Rule 201) isn't modelled.
• Tries made outside the app, or in another project, can't be counted in the Deflated Sharpe, so it is shown as "at least".
• The lockbox and the tries log are discipline aids, not security: anyone can edit their files, though an edit shows.

Latest results

Recorded on 2026-11-06 for engine 0.9.0, at commit 3862bd7, on an Apple M5 Max, 18 cores, 128 GB.

Speed, at 4 threads: