Backtest Determinism And Reproducibility

When to Use

Invoke this skill when building backtest simulation engines or CI regression suites. If backtest runs produce varying P&L curves or Sharpe ratios across identical code and data, parameter optimization becomes impossible to trust — you cannot tell an improvement from noise. Non-determinism comes from unseeded RNGs, set iteration order, unordered event streams, wall-clock reads, and floating-point accumulation order.

When NOT to Use

• As a tamper-evident audit record. The checksum is an unkeyed SHA256: anyone can edit a trade log and recompute it. For an authenticated record see backtest-audit-trail-for-regulatory-review.
• To compare runs across different environments. Differing Python/NumPy/BLAS versions or thread counts will legitimately diverge; that is an environment-pinning problem, not something a checksum can fix.
• To validate strategy correctness. A perfectly reproducible backtest can be reproducibly wrong — see lookahead-bias-elimination.

Prerequisites

• A master integer seed (e.g. 42).
• PYTHONHASHSEED set in the environment before the interpreter starts.
• Pinned library versions and a fixed thread configuration for any run you intend to compare.

Workflow

1. Set PYTHONHASHSEED before launching Python. This is the trap that catches people: hash randomization is fixed at interpreter startup, so os.environ["PYTHONHASHSEED"] = "0" inside running code does nothing while looking like it works. Launch as PYTHONHASHSEED=0 python backtest.py. check_hash_seed() reports whether the current process is safe; it cannot fix it.
2. Seed the RNGs — and know what seeding misses. apply_master_seeds() seeds random, NumPy's legacy global RandomState, and torch when present. NumPy documents np.random.seed as a legacy function and recommends a dedicated Generator; code calling np.random.default_rng() draws fresh OS entropy and stays non-deterministic no matter how often you seed the global. Use make_numpy_generator() and thread it through explicitly.
3. Sort the event stream by (timestamp, symbol, sequence_id). All three keys are required — a missing timestamp is rejected rather than defaulted to 0.0, which would silently move that event to the front. Duplicate sort keys are also rejected: Python's sort is stable, so tied events would keep their input order and the result would depend on how the file happened to be read.
4. Replace wall-clock reads with SimulatedClock. Inject it rather than monkey-patching time.time() — patching a global affects every library in the process, including ones whose correctness depends on real time. The clock refuses to move backwards, which surfaces an unsorted stream immediately.
5. Checksum with exact float bits and compare runs. audit_reproducibility() reports is_bit_identical and, on failure, first_divergence_index so you can go straight to the offending trade instead of staring at two mismatched hex strings.

Full procedure: see references/workflows.md. Standards and sources: see references/standards.md. Printable pre-flight checklist: see assets/checklist.md.

Common Pitfalls

• Rounding before hashing. Any tolerance applied before comparison erases exactly the sub-ulp divergences that signal non-determinism. float_precision exists for callers who explicitly want tolerant comparison, and it logs a warning saying what it costs.
• Setting PYTHONHASHSEED from inside Python. It has no effect. The assignment succeeds, the variable reads back correctly, and set iteration order stays random.
• Assuming dicts and sets behave alike. Dicts have preserved insertion order since Python 3.7 and are deterministic. Sets are not — iterating a symbol universe as a set gives a different order per process. With PYTHONHASHSEED at 0, 1, and 2, {'AAPL','MSFT','GOOG','TSLA'} iterates in three different orders. Iterate a sorted list.
• Seeding only the NumPy global. np.random.seed() does not affect any Generator created by default_rng().
• NaN in a trade log. json.dumps renders every NaN as the same token, so two runs that both corrupted to NaN hash identically and are declared reproducible. Non-finite values are now rejected outright.
• Unsigned notional mistaken for P&L. Summing quantity * price across a round trip gives gross turnover, not equity: BUY 100@150 then SELL 100@155 sums to 30,500 while the realized P&L is 500. Cash flow must carry the side's sign.
• Monte Carlo without a fixed seed. Slippage or execution-timing perturbations reseeded per run make every comparison meaningless.
• Treating a matching checksum as proof of correctness. It proves two runs agreed, nothing more.

Verification

• Run python -m unittest discover -s skills/backtest-determinism-and-reproducibility/scripts — 40 tests, 100% pass rate.
• The test that matters: feed sum([0.1]*10) and 0.1*10 as prices for otherwise identical runs and confirm is_bit_identical is False. Pre-2.0 this returned True.
• Run a backtest twice with identical inputs and confirm matching checksums; change the master seed and confirm the checksum shifts.
• Confirm a missing timestamp or a duplicate (timestamp, symbol, sequence_id) raises DeterminismError rather than sorting silently.
• Confirm SimulatedClock.advance_to() rejects a backwards jump.

Related Skills

• walk-forward-validation-setup
• monte-carlo-strategy-robustness-testing
• vectorized-vs-event-driven-backtest-tradeoffs
• backtest-audit-trail-for-regulatory-review — for an authenticated audit record, which this skill's unkeyed checksum is not
• reproducible-ml-training-pipelines — the same problem for model training
• dependency-pinning-and-reproducible-builds — pinning the environment determinism depends on