Rearview Download

Why your live trades don’t match last week’s backtest: point-in-time data vintages via reverse patches

Every systematic trader who updates market data daily runs into two quiet problems sooner or later.

First, saved backtests stop reproducing. You run a strategy on Monday, save the result, update your database on Friday, and rerun it, only to watch three trades change because your data vendor corrected a historical close, delivered a dividend two days late, or back-adjusted a stock after a split.

Second, live-order forensics breaks down. On Tuesday evening your system told you to buy 46 shares of a stock, so you placed the order Wednesday morning. A week later you check the backtest through Tuesday and it says you should have bought 52 shares, or never bought the stock at all. Was there a bug in your strategy, a mistake in your broker import, or did the data underneath you change after Tuesday night?

Keeping a full copy of a 5,000-security, 10-year binary store after every daily update multiplies disk usage rapidly. Throwing older versions away leaves you guessing whenever live trading and research diverge.

In Rearview 1.1, we solved both problems with data versions and Replay That Evening (rule C117): a four-tier vintage store built on a 180-line deterministic reverse byte-delta encoder, paired with a side-by-side live replay that traces differing orders back to the data changes behind them.

1. Four ways to keep an old version (and spend zero bytes when history didn’t move)

Most daily market updates do not change history at all: they simply append one new trading day (and maybe a new dividend or delisting on that day). Keeping a second copy of the entire 10-year store just to remember yesterday’s version would be wasteful.

Each Rearview store folder maintains a version index at vintages/log.jsonl, oldest first, updated atomically via temp-file rename (log.jsonl.tmp → log.jsonl). Every line records the store’s content fingerprint, the timestamp it was recorded, its last trading day (through), whether it was an import, an append-only days update, or a history restatement (with the first date that changed), its payload SHA-256, and its byte length.

Each earlier version is stored in the smallest representation that rebuilds it cleanly:

Storage mode (kept)When Rearview chooses itDisk costHow it rebuilds
currentThe active store file itselfExisting storeRead directly
extendedThe next update only added newer days and changed nothing up to the old last day0 bytesTruncate the newer base store at the old through date (exact logical data and fingerprint)
patchHistory changed, and a reverse byte delta against the truncated new store is at most half the old file’s sizeCompact delta (<fp>.rvwpatch)Truncate the base store at the old through date, apply the reverse delta, verify SHA-256 (exact store file bytes)
storeHistory changed so broadly that a delta would exceed 50% of the old fileFull file (<fp>.rvwstore, hard-linked when possible)Read directly and verify payload SHA-256 (exact store file bytes)

Two lifecycle rules keep the version graph consistent as updates accumulate:

  • Re-anchoring patches onto byte-exact bases. Notice the distinction in the table above: patch and store rebuild the exact .rvwstore file bytes, while extended rebuilds the exact logical data and fingerprint via truncation rather than guaranteeing identical file bytes. So if version V1 is stored as a reverse patch against V2, and the next evening V3 arrives as an append-only update that turns V2 into extended, Rearview reconstructs V1 from V2, re-encodes V1 directly against V3, and removes the old patch. Every patch stays anchored to a byte-exact base.
  • Reverting cleanly. If a later data update restores the exact fingerprint of an earlier version, Rearview removes that older version’s kept file and old log line, and records it at the end of vintages/log.jsonl as current with the new timestamp.

2. Why reverse patches on a flat binary file?

When designing data versioning for a local backtester, there are two obvious alternatives, and both carry real trade-offs:

  • Bitemporal databases (ArcticDB, SQLite, DuckDB, Parquet tables). Storing valid-time and transaction-time columns per row answers point-in-time queries natively, but adds indexing and query overhead to every ordinary backtest read. Rearview’s engine reads a flat, aligned columnar binary store (market.rvwstore) straight into memory, running a 5,000-security × 10-year strategy cold from disk in 0.49 seconds (cold_seconds = 0.487s). Keeping versioning as external reverse patches leaves everyday backtesting at zero versioning overhead.
  • Off-the-shelf binary diff libraries (bsdiff, xdelta3, zstd dictionaries). External binary-diff codecs either pull in heavy C dependencies, change output across library versions, or spend too much memory and CPU on 100 MB+ files. Because Rearview truncates the new store to the old version’s through date before diffing, the two columnar binary files share the same calendar range and align almost everywhere, making a simple, deterministic 64-byte block matcher both fast and tiny.

3. Inside the reverse byte-delta format (RVWPATCH)

crates/rearview-data/src/delta.rs implements a self-contained reverse byte-delta codec in ~180 lines of Rust (excluding tests).

Before diffing, Rearview truncates the new store at the old version’s last trading day (through). The encoder indexes the truncated base in 64-byte blocks using a deterministic open-addressed hash table (fixed multiplier 0x0000_0100_0000_01b3 and Fibonacci slot hashing, storing the first offset per block hash, with no random hasher seed). It then scans the old file with a rolling hash, and whenever a 64-byte block matches, extends the match both backward and forward byte by byte.

The resulting patch file (vintages/<fingerprint>.rvwpatch) has an 88-byte little-endian header followed by a stream of just two opcodes with unsigned LEB128 varints:

offset  size  field
     0     8  magic "RVWPATCH"
     8     4  format version (1)
    12     4  reserved (0)
    16     8  old store file length
    24    32  SHA-256 of the old store file
    56    32  SHA-256 of the truncated base store file
    88     .  ops:  0, offset, length         (COPY from truncated base)
                    1, length, literal bytes  (ADD literal bytes)

Three invariants keep patches safe to rely on months later:

  1. Strict 50% size cutoff. The encoder aborts early and returns None if added_bytes exceeds half the old store’s length, and keep checks again that the final encoded patch (88-byte header + all COPY/ADD ops) is at most half the old store’s length. If hash collisions or widespread edits ever blow up a patch, Rearview silently falls back to keeping the whole .rvwstore file.
  2. Round-trip proof before saving. A patch is never written to disk on faith. Before keeping it, Rearview runs apply_patch(&source, &patch) in memory and checks that the output equals the original old store file byte for byte.
  3. Dual SHA-256 checks on rebuild. When rebuilding an old version (for Check It Reproduces, rvw rebuild, or Replay That Evening), Rearview verifies both the SHA-256 of the truncated base input (patch[56..88]) and the exact byte length plus SHA-256 of the reconstructed output (patch[16..56]), with a recursion-depth guard (versions.len() + 1) against cyclic base references.

When you click Check It Reproduces on a saved run whose data was later restated, Rearview matches the run’s recorded data fingerprint against vintages/log.jsonl, rebuilds the exact vintage the run was made on, and checks that every output file still comes out byte for byte identical.

4. Recording time vs. market date (and why timezones matter)

To answer "what did my data say on Tuesday evening?", the store needs two different dates for every version:

  • The market date (through): the latest bar inside the database.
  • The recording timestamp (time): when that version was recorded, formatted as YYYY-MM-DDTHH:MM:SS with Z or a ±HH:MM offset.

Selecting the version known on day (known_on) filters versions whose recorded date is on or before day, then picks the one with the latest UTC instant. That timezone distinction matters in practice: an evening update run at 9:00 PM in Los Angeles on November 19 (2026-11-19T21:00:00-08:00) is already 05:00 UTC on November 20, yet it belongs to the evening of November 19. The Mac app automatically records your local date and UTC offset. On the command line, rvw import and rvw update default to UTC (Z), so pass --at with your local offset for evening automation or when backfilling past exports you saved yourself.

5. Replay That Evening, and the two bugs real Yahoo data caught

With point-in-time vintages on disk, Replay That Evening (rvw live evening <dir> --store <store> --day YYYY-MM-DD --out <folder>) takes any past trading day of a live account and runs two replays side by side:

  1. Then: rebuilds the version recorded last by the end of that evening, cuts it at day (if it extended past day), replays your actual broker fills through day, and computes the next session’s orders.
  2. Now: cuts today’s store at day, replays those exact same broker fills through day, and computes the next session’s orders.

When the two order lists match, whatever your vendor changed since that evening made no difference to your live trading. When an order differs in presence or share count (keyed by security, action, order type, limit and stop price), Rearview explains why.

Our first implementation of that explanation looked clean in unit tests, and failed two real-world checks the moment we dogfooded it on actual Yahoo Finance exports:

  1. A 2004 re-adjustment hid a split three days earlier. The first version printed the earliest historical change on the differing order’s security. Because Yahoo Finance’s adjusted files rescale prices all the way back to a ticker’s start when a split arrives, the UI blamed a price change in 2004 instead of the GOOGL split that happened three days before that evening.
  2. An untouched stock’s buy size changed because a different held stock split. When a stock the account already held that evening had a late split or corrected close, the account’s closing equity shifted. Because the strategy sized new buys as a percentage of equity, the share count of an unrelated buy order moved, and looking only at changes on the buy order’s own security reported no changes at all.

Fixing both taught us to collect every data change that can reach an order across four explicit causal scopes, sorted most recent first by the day a run could first observe it:

Causal scope (via)What it coversTie-break priority
ownPrice, split, dividend, delisting, membership, ticker or series changes on the differing order’s own security1
heldChanges on any security the account held at the close of day on either the "then" or "now" replay (marked Held that evening in the app)2
marketStore-wide changes not tied to a single security (CALENDAR_CHANGED, RATES_CHANGED, named series added/removed, or unitemised summary changes)3
lateDAYS_LATE: trading days today’s data has through day that were missing on that evening because the data hadn’t been updated yet4

Re-running the same real-data dogfood test immediately surfaced the GOOGL split on both the GOOGL order (own) and the resized buy order (held, marked Held that evening), while keeping older 2004 re-adjustments neatly tucked behind it in the full change list.

6. Speed and randomized verification

Point-in-time versioning is only practical if rebuilding an older store feels instantaneous during an evening check. On Rearview 1.1.0, our single-threaded rebuild_vintage_seconds benchmark cuts a 5,000-security × 10-year (2,520-day) store to four-fifths of its history (2,016 days), applies a reverse .rvwpatch, verifies both SHA-256 digests, and reads the rebuilt store into memory in 0.79 seconds (0.793s measured on an Apple M5 Max; speed budget 2.5s, see How Rearview is checked).

Before every release, xcheck/vintages.py stress-tests the vintage index and reverse-patch engine across 40 randomized update sequences spanning 303 successful import/update steps (plus refused partial updates) and 242 kept historical versions (282 total versions including the 40 current stores), mixing corrected bars, dropped bars, rescaled prices, late splits, doubled prices, partial daily updates and refused updates. After every step, the harness verifies the index’s ordering and storage modes, rebuilds every historical version (checking exact file bytes for every patch and whole store, and exact fingerprint for every extended version), and checks known_on selection for every date.

7. Honest limits

Stating what a mechanism cannot do is as important as documenting what it does:

  • No retroactive time travel without saved exports. A store only knows versions recorded from the time versioning began (or older exports you backfill yourself with --at). If a vendor silently corrected a close last year before you imported it, no local versioning system can recover what the vendor never gave you.
  • Consecutive history restatements chain patches. While append-only updates re-anchor older patches directly onto the new store, a sequence of back-to-back history restatements (common if you update back-adjusted per-symbol files every week as new dividends arrive) builds a chain of reverse patches until an edit crosses the 50% cutoff and writes a whole .rvwstore file. Using as-traded bars with a separate actions.csv keeps almost every daily update append-only (extended, 0 bytes).
  • Causal candidates, not counterfactual proof. Replay That Evening lists every change on the order’s security (own), held securities (held), store-wide changes (market) and late days (late). It does not run an ablation over every individual bar edit to prove which of two simultaneous edits flipped a signal. Moreover, changes to a third security that was neither ordered nor held (such as an unheld market-regime ETF like SPY whose restated close flipped a regime filter, or an unheld stock whose rank leapfrogged your candidate) appear in the top-level history array of evening.json rather than directly inside that order’s changes list.
  • Market data is only half of live reality. Versioning proves what your data said that evening; it cannot detect a broker-specific corporate-action booking or an intraday fill discrepancy, which is what the live account’s Fills and Statement tabs track separately.

Read the full documentation in Keeping your data up to date and Replay That Evening, or see every release check at How Rearview is checked.