Is this result luck?

Try enough variations of a strategy on the same history and one of them will look good by chance. Rearview counts what you tried, every run, preview and search of a script, and says in plain words whether the result is likely to be luck. A single run can be set against runs whose entries were drawn at random. Rearview also lets you hold back recent data, so the last test is on days the strategy has never seen.

None of this can catch a strategy that peeks at the future. The engine prevents that itself: each day it decides from what was known at that day's close (see How Rearview is checked).

Searching a script's params

A param is a number with a range, such as param rsi_entry = 10 in 5..20 step 5 (see Params). To search, select the script, choose Optimize in Run Settings, and press ⌥⌘R (Run ▸ Optimize).

• Parameters to vary: each one you tick runs over its range; the rest keep their values.
• Method: Every combination runs the whole grid. Random sample runs a seeded sample of it, and the same seed picks the same sample every time. Genetic and CMA-ES search spaces far too big for a grid, picking each generation of trials from how the earlier ones did (see Genetic and CMA-ES searches).
• Rank by sets the fitness that picks the best trial: Sharpe (the default), Sortino, CAGR, MAR, UPI or total return, or with Formula… a formula of cagr, total_return, volatility, sharpe, sortino, max_drawdown (a positive fraction), mar, ulcer and upi, such as cagr / max_drawdown. The luck statistics judge the search by the same fitness.
• A search runs up to 10,000 trials, on every core, with the project's run settings.

A portfolio can be searched the same way: each strategy's params, and the strategies' weights. See Searching weights.

The search appears in the project navigator under Searches and Walk-Forwards. Verdict answers the question, Stability maps the fitness over the params, and Trials lists every trial. Run This Setting runs the best trial as an ordinary run, and Use These Values saves its values as the script's.

Genetic and CMA-ES searches

A grid of three params with 50 values each is 125,000 settings, far over the limit. A genetic or CMA-ES search explores a space like that with a few hundred trials. Choose Genetic or CMA-ES as the Method, and set Trials, the most it runs (500 by default, at least 30), and Seed.

• Both start from a random sample of the space: 10 settings for each param you vary, from 30 to 100. They are the same draws a random search with that seed makes.
• Genetic then breeds each generation from the best settings so far: each new setting takes every param from one of two good parents, and now and then moves one a few steps. It stops at its trials, after ten first generations' worth of trials without a better result, or when it finds nothing new to try.
• CMA-ES samples each generation around a centre that moves towards the better settings, its spread learning which way the good region runs. When it settles, it starts again from a random point with twice as many samples. It runs to its trials, unless it finds nothing new to try.
• The same seed gives the same trials every time, and a space whose every combination fits in the trials runs as a grid.
• Trials gains a Generation column, and Stability shows only the settings the search tried.

On the sample data, a 300-trial genetic search found the best setting of a 2,160-setting grid. On a rugged surface either may stop on a good setting that isn't the very best, so try another seed before trusting the peak, and look for a plateau on the stability map.

Their luck figures. The Deflated Sharpe counts every trial they ran. It takes the spread and correlation of the trials from the random first generation, because later generations crowd around the best and would make luck's bar look lower than it is. There's no PBO and no Reality Check: the search chose its trials by how they did over the whole history, so splitting that history to test the pick wouldn't be fair. The fair test is a walk-forward with the same method, where each window runs its own search of its own days: Walk Forward… on the verdict page sets one up.

The verdict

The two statistics ask different questions, so a search can pass one and fail the other. On the sample data, down_days_overtuned has a Deflated Sharpe of 97% and a PBO of 82%: the rule family seems to earn something, but which setting is best is noise. Its verdict is Probably luck, because the setting you would pick is the part that doesn't hold up.

Deflated Sharpe

The Deflated Sharpe is the chance that the best trial's true Sharpe ratio beats what luck alone would give the best of that many tries. The more trials, and the more they differ, the higher that bar.

The verdict lists every input:

• Trials: how many settings were tried, labelled "at least". It includes the script's earlier tries in the project (see Every try counted), so running it by hand with ten values before searching it still counts them, even after you delete those runs.
• Effective trials: trials that move together count as fewer independent ones, from the average correlation of their daily returns.
• Luck's best: the Sharpe ratio the best of that many independent tries would reach by chance.
• Best Sharpe, daily returns, skew and kurtosis of the best trial: a short history, or returns with fat tails, make a high Sharpe ratio less convincing.

The trial count is a minimum. Tries made outside Rearview, by hand, in other tools or from ideas you read about, can't be counted, so treat a borderline result with more doubt than the number shows.

Every try counted

Every run of a script, every preview once its results show, every search trial and every walk-forward goes into the project's tries log, tries.jsonl. A try is the script's code (comments and spacing don't count), every param's value and the run settings, so the same setting run again, or on data that has grown, is the same try.

Each run's Summary says how many ways its script has been tried, such as Tried 37 ways · Can't tell, and Robustness opens with Every try: the run's Deflated Sharpe counting those tries, with every input. On a script's first try nothing is deflated, and it is simply the chance the true Sharpe is above zero.

• The count follows the script. Renaming it in Rearview keeps its tries, a duplicate starts with the original's, and editing it keeps counting. Deleting a run or a search leaves its tries counted. Deleting the script, then making a new one at the same path, starts fresh.
• Tries on other data count separately, and so does each script and portfolio.
• A search counts the script's tries it doesn't run itself, so the trials count shows both: "at least 540 trials (500 in this search, 40 tried before)".
• The run's report shows the same figures, and rvw analyze --tries gives them in Terminal.
• Runs in Terminal count too when they're made with rvw run --project (or search, walkforward), as an AI agent's should be (see Writing scripts with an AI agent).
• A project that ran things before it had a tries log adds its saved runs and searches when it first opens.

Each line of tries.jsonl is chained to the last by SHA-256, as the lockbox's log is, and the project file keeps the last one, so a log that was edited or cut short is flagged where the count shows. Later tries don't clear the flag: the log records what it lost and keeps saying the count may be low. It keeps research honest; it isn't security.

Probability of Backtest Overfitting

PBO cuts the history into blocks and tries every way of using half of them to pick the best setting and the other half to test it. It is how often the setting that looked best in one half ranked below the median in the other. It needs at least 30 trials, and uses the same fitness that picked the winner.

PBO is not the chance the strategy is overfit. It measures how far picking the best setting from this search can be trusted. The verdict also shows how often the pick lost money out of sample, and the slope from in-sample to out-of-sample fitness: a negative slope means the settings that did best in one half tended to do worse in the other.

Stability

The stability map shows the fitness of every setting, two params at a time, with the best marked. A good strategy usually sits on a plateau, where nearby settings do almost as well. A lone spike, with poor neighbours, is the shape of luck.

Neighbours keep is the average fitness of the settings one step from the best, as a share of the best's.

Against the benchmark

When the run settings name a benchmark, a search also runs the Reality Check: after counting every setting it tried, is the best Sharpe ratio above the benchmark's by more than luck would give? It resamples the history 1,000 times in blocks of about 20 days, as the Monte Carlo does, and counts how often luck alone would put the best of that many settings this far ahead.

The verdict page says Better than the benchmark when that happens at most 5% of the time, and Not clearly better than it otherwise. It sits beside the verdict, not inside it: a search can be probably not luck and still not beat simply holding the benchmark. Like the Deflated Sharpe, it can't count tries made before the search.

On the sample data, down_days_overtuned's best Sharpe ratio is 0.74 a year, below SYNSPY's 0.85: not clearly better than SYNSPY.

Walk-forward tests

A walk-forward test repeats the search on a moving window of history and trades the next window with the winner, as you would in real time. Choose Walk-forward in Run Settings, then Run ▸ Walk Forward.

• First in-sample: the days the first search sees, 756 trading days (three years) by default.
• Out-of-sample: the days each winner trades before the next search, 252 by default.
• On intraday data both count days, not bars, and each window runs from its first day's first bar to its last day's last.
• In a run that trades every day, because it can hold coins, both count every day rather than trading days, so the defaults cover about two years and eight months of history.
• Method: with Genetic or CMA-ES, each window runs its own search of its in-sample days, with the same seed and up to its trials, so its winner is the one a search of those days alone would pick.
• Rolling searches the same number of days before each window. Anchored searches everything from the start.
• Open positions carry from one window to the next, and a param that sets a stop, a size or the number of positions switches to the new winner's value on the window's first day.
• A portfolio's weights can vary as well: turn on a strategy's Weight, %, as for a search. See Searching weights.

The traded windows join into one out-of-sample run you can open like any other. The whole procedure counts as one trial, so its question is simpler: did it hold up?

Settings that won shows how often the winner changed. The same setting winning every window is a good sign; a different one every time suggests the search is chasing noise.

Your own signals

If you compute entries and exits elsewhere, in Python say, Rearview can run them, search their settings and walk them forward, with the same luck figures a script gets.

A signals folder holds one CSV per kind of row, each with a header:

id is the security's id in your data (securities.csv), and a file you don't need can be left out. rvw signals writes a script's signals in this form, which is a good way to see one.

The rest comes from a config, a JSON file with the strategy's settings and the run settings:

{"strategy": {"sides": "long", "exits": {"signal": true, "stop": {"type": "pct", "pct": 80000}, "max_bars": 10},
              "sizing": {"type": "pct_equity", "pct": 100000}, "max_positions": 10},
 "run": {"initial_cash": 100000000000}}

Money is in millionths of a dollar and percentages in millionths (80000 is 8%). For a portfolio, the config is {"portfolio": {"mode": …, "strategies": [{"strategy": …, "weight": …}, …]}, "run": …} and the folder has a sub-folder per strategy, 0/, 1/ and so on.

rvw run --store data --signals my_signals --config my_config.json --out runs/mine
rvw search --store data --signals my_signals --config my_config.json \
  --space stop.pct=4..12:2 --space max_positions=5..20:5 --out searches/mine
rvw walkforward --store data --signals my_signals --config my_config.json \
  --space stop.pct=4..12:2 --out searches/mine-wf

• What a search varies: the numbers a walk-forward may change by date, named by their place in the config and written as a script writes them: stop.pct, stop.mult, trail.pct, trail.mult, target, max_bars, size.amount (dollars), size.pct, size.risk, size.mult, max_positions, max_position, rebalance.band, rebalance.days, rebalance.keep, adds.max, scale_out.pct and scale_out.at (a ladder's steps are scale_out.0.pct, scale_out.1.pct, …). Each exists only when the config has that setting. A portfolio's are 0.stop.pct, 1.max_positions and so on, and its weights 0.weight, as for scripts.
• The signals stay as they are. Each trial runs the engine on the same rows with other settings, so a search is quick. To search a signal's own parameters, write a folder for each and compare them yourself, or write the signal as a script.
• Its figures are a script search's: the Deflated Sharpe, PBO and stability map, and the walk-forward's verdict. Tries made outside the search can't be counted, since the tries log follows scripts; give their number with --prior-trials.
• The search remembers the folder by its path, relative to the search's folder, and a digest of its files. Rerunning it with --spec, or Check It Reproduces, refuses if the folder has changed since.
• In the app, a search written into the project's searches folder opens like any other: Run This Setting runs the folder with that trial's settings. There's no Use These Values, since the settings live in your config file.
• Random entries aren't offered for signals, which have no universe to draw from, nor are stress tests, which need the script that makes them. The Chart shows a signals run's prices and fills but no script lines.

Checking your signals for look-ahead

A script can't see past the day it's on, but a program of yours can, through a shift(-1) or an average over the whole history. Select a run made from a signals folder and choose Run ▸ Check Signals for Look-Ahead…. Give the command that makes the folder, with {data} where it reads the data and {out} where it writes the signals:

python3 make_signals.py {data} {out}

Rearview writes your data into {data} in the import format (securities.csv, tickers.csv, bars.csv, actions.csv, membership.csv and series/, as in The import format), makes {out} an empty folder and runs the command with /bin/sh, in the project's folder. Then it does the same at six days spread over the data, twice for each:

• on the data cut at that day, as it stood at that day's close;
• on the whole data with every price after that day doubled, or halved at every other day. A rule that reads a later price only when it jumps does the same whether that price is missing or ordinary; a price far out of line shows it.

Every row dated on or before the day must come out as it did from the whole data, file by file, with the same score or value. Rows after it don't count. Each day is ok, or names the first row that isn't: "entries_long.csv 2021-06-30 id 412 appears only when later data exists: this signal reads past its date."

• It reads what your program writes, not its code. A leak that changes no row on those days goes unseen, so a pass is evidence, not proof.
• A command that fails stops the check, with its exit status and the end of what it printed.
• In the app, the command doesn't see your shell's settings: give a program's full path when it isn't in /usr/bin, such as /opt/homebrew/bin/python3.
• Behind a sealed lockbox, your program sees the data only up to it, as runs do.
• The copies are made in your temporary folder and removed afterwards, whether or not the check passes.
• On the command line, rvw lookahead --store data --command 'python3 make_signals.py {data} {out}' runs the command in the current folder and prints the same as JSON, and exits 1 when a day differs. --cuts 12 checks twelve days, --work <folder> makes the copies there, and --lockbox-start with --embargo holds back a lockbox, as they do for rvw run.

Random entries

Would entries drawn at random have done as well? Select a run, open Robustness (⌘7) and choose Compare with Random Entries (also Run ▸ Compare with Random Entries).

Rearview runs the strategy 500 more times with random entries at its own rate. On each day, every security the strategy could have entered (in its universe with a bar, meeting its filter, its rank and size inputs known) has the same chance of an entry as the strategy's own rule had on average over the run. The exits, stops, sizing, slots and costs stay the strategy's own, so only the entries differ. When more random candidates qualify than there are free slots, they're taken in random order.

p is 1 plus the number of random runs whose Sharpe ratio is at least the run's, divided by 501. The page shows the spread of the random runs' Sharpe ratio, CAGR and deepest fall with the run's marked.

On the sample data, near_52w_high beats 499 of the 500 random runs (p = 0.004): better than random entries. rsi2_pullback beats 152 of them (p = 0.70): no better than random entries. The sample data is synthetic, with no pullbacks for that rule to find.

Read Like for like too. Random entries at the same rate fall evenly across the days, while a rule's entries usually bunch together, so random runs often trade more and spend more days in the market: rsi2_pullback trades 2,975 times on the sample data, its random runs about 7,100. The verdict uses the Sharpe ratio, which allows for time in the market, but costs weigh more on the busier runs.

• A condition in filter counts as who may be traded, so random entries keep it. Put the signal you want tested in enter.
• The comparison is saved in the run's folder, and Check It Reproduces checks it too. The same seed draws the same entries every time.
• It isn't offered for a run made from a signals folder, which has no universe to draw from, or for a walk-forward's joined run, whose settings change from window to window.

Stress tests

Would the result survive the frictions of trading it? Select a run, open Robustness (⌘7) and choose Run Stress Tests (also Run ▸ Stress Test). Rearview runs the strategy again four ways, each changing one thing:

Each test sits beside the run's own figures (growth per year, Sharpe ratio, max drawdown, exposure and trades) with a sentence on what changed. The two drawn tests give their median and the range nine in ten runs fall in. There is no verdict: how much worse is too much depends on how you'll trade the strategy.

On the sample data, with a commission of $0.005 a share ($1 at least) and 5 bp of slippage, rsi2_pullback grows −0.3% a year; with costs doubled, −5.5%. It trades almost 3,000 times in ten years for a thin edge, so costs are most of its story. near_52w_high, half as busy, goes from +21.7% to +20.4%. A day late, neither changes much: the synthetic data has no next-day pattern for them to miss. Both grow the same whichever of their next 20 days they start on, because their holdings soon converge.

• A run with no commission and no slippage has nothing to double, and says so. A strategy with no entries has none to miss, and later starts need at least 41 days.
• The tests are saved in the run's folder (stress.json), and Check It Reproduces checks them too. The same seed misses the same entries every time.
• They aren't offered for a run made from a signals folder, a walk-forward's joined run, or a live account. On the command line: rvw stress <run folder> --store <store>.

Capacity

Would the result hold with more money? Small stocks that are easy to trade with $100,000 can be too thin for $1 million. Select a run, open Robustness (⌘7) and choose Run Capacity Test. Rearview runs the strategy again at 1, 3, 10, 30 and 100 times its starting cash, on the same signals.

• Every dollar amount scales with the cash: the initial cash, deposits and withdrawals, a fixed size such as size $5000 per position, and the commission per trade and its minimum. Amounts per share or contract, and every percentage, stay as they are: they grow with the shares already.
• Every rerun has a volume cap, 1× too: the run's own, or a test cap of 1% of each bar's volume when it has none. The page says which, and under a test cap its first row is the run as it was, with no cap. The rest of an entry over the cap is cancelled, and the rest of an exit is sold at the next open, as in any run with a cap.

Each scale shows the run's figures (growth per year, Sharpe ratio, max drawdown and trades) and how its orders met the market:

Under the table, the page counts the fills larger than all their bar traded. Under a cap of 100% or less there are none.

Slippage is a flat cost, with no model of market impact: a large order pays the same share of its price as a small one. So the curve shows only what the volume cap cuts, and a strategy well within the cap looks the same at every scale. In a real market an order that size would also move the price against you. There is no verdict.

On the sample data, near_52w_high has no volume cap of its own. Under the test cap it grows 22.6% a year with $100,000 and 17.7% with $10 million, where the cap cuts 23% of what it means to buy: its median order is 0.2% of its bar's volume, and one in ten is over 2.6%.

• The test is saved in the run's folder (capacity.json), and Check It Reproduces checks it too.
• It isn't offered for a run made from a signals folder, a walk-forward's joined run, or a live account. On the command line: rvw capacity <run folder> --store <store>.

Monte Carlo

Every run with at least 20 days of returns gets a Monte Carlo of its daily returns under Robustness: many replays of them in other orders, in blocks of random length, so streaks stay together. It shows how much of the growth and the deepest fall came from the order the days happened to come in. See Robustness.

Trade order

Robustness also replays a run's closed trades one after another, in 2,000 other orders, each trade's profit or loss taken as a share of the equity it started from. Every order ends at the same equity, so this shows sequence risk only: how deep the falls and how long the losing streaks could have been had the same trades come in another order. It needs at least 20 closed trades.

Shuffling treats the trades as independent, so trades that lose together, as they do in a market fall, are broken apart. The fall as traded is often deeper than in most orders: on the sample data, near_52w_high fell 19.3% as traded, more than in 95% of the orders. The Monte Carlo, which keeps streaks of days together, is the better guide to how deep a fall could go.

The lockbox

The lockbox holds back the most recent data while you research, so you have one honest test left at the end.

1. Open the project's data (⌃⌘4) and set Hold back from, such as two years before the end.
2. The embargo is a gap of trading days (the exchange's days, in data with coins too) just before that date, also held back, so no indicator or open trade can reach into the lockbox. Rearview suggests the smallest one the project's scripts need: the longest lookback, plus the longest holding period (max bars, or 20 days when a script has none). On intraday data the embargo is in days too, whatever a session holds. The need it shows for a script is its window's bars and its holding, divided by the bars in the data's longest session and rounded up, plus the days a daily, weekly or monthly view reaches back, plus one.
3. Choose Seal Lockbox. Every run, search and walk-forward now sees only the days before the embargo, and says so in its assumptions. A sealed lockbox can hold back more, never less. Rearview refuses a seal that would leave no data before the embargo, and nothing is logged. If data that's sealed later won't open behind its lockbox, because it was replaced by a shorter version, say, the Data view says why, and Reveal… is what opens it.
4. When you're done, choose Reveal…. This is final: a project's lockbox is used once.
5. Run ▸ Test on the Lockbox runs a script with its saved params from the lockbox's start to the end of the data.

From then on, every result is marked Lockbox used: the revealed data is in it, and tuning on it makes it in-sample too.

Each seal and the reveal are written to lockbox.jsonl in the project folder, each line chained to the last by SHA-256, so an edit shows. It keeps research honest; it isn't security, and anyone can edit the files.

What none of this can count

• Ideas you tried in your head, on paper, in other tools or in another project. Random entries test one rule; they don't count how many rules you tried before it.
• A variant written as a new script file: each file counts its own tries. Edit or duplicate the script instead.
• A script renamed or copied in Finder or Terminal, which starts a new count, and a tries log deleted or cut short (Rearview says so, and goes on saying so for that log).
• A history that won't repeat. A result that passes every check here still assumes the future looks enough like the past.