Backtesting Methodology

Lookahead bias elimination, walk-forward validation, realistic slippage/fee/latency simulation, synthetic data generation, standardized tearsheets.

Skills: 31
Hand off: 206
Handed off from: 282

Hands off most often to

Brokers, venues and frameworks named here

Index

All 31 skills in this domain, alphabetically. Hover a row for its situation; open it for the full playbook.

A

adjusted-vs-unadjusted-price-series-pitfallsUse before loading historical OHLCV into a backtest, to declare and verify the adjustment mode; separates raw, split-adjusted and total-return conventions and catches split, dividend and continuity errors in vendor adjustment factors.price-adjustment

B

backtest-audit-trail-for-regulatory-reviewUse when a backtest result needs a durable record of exactly what produced it: git commit SHA, per-file data checksums, hyperparameter manifest and environment, signed with HMAC for integrity against outsiders.regulatory-compliance backtest-database-schema-for-point-in-time-queriesUse when designing the storage layer for backtest data, so as-of queries are native: two independent time axes let a query ask what was known on a date rather than what the table holds today. The join engine itself is point-in-time-database-for-ml-training-data.point-in-time backtest-determinism-and-reproducibilityUse when identical backtest code and data produce different P&L or Sharpe across runs, making optimisation untrustworthy: deterministic seeding, strict input sort order, a simulated clock replacing wall-clock reads, and run-to-run divergence detection.reproducibility backtest-infrastructure-cost-budgetingUse before launching a large grid search or tick-level sweep, to forecast cloud compute and storage spend with spot interruption overhead, per-task billing minimums and non-linear scaling, rather than discovering it on the invoice.cost backtest-look-ahead-in-universe-selectionUse when the backtest picks its universe by a rule such as top 50 by market cap or current index membership; audits membership timestamps so 2015 is not traded with 2026 constituents. Feature-level leakage belongs to lookahead-bias-elimination.universe-selection backtest-outlier-and-bad-tick-filteringUse when ingesting raw tick or bar files that contain fat-finger prints, test messages, zero prices and out-of-sequence ticks, so one bad print does not create a phantom signal. Every purge also removes a price a stop might genuinely have hit.bad-tick-filtering backtest-parameter-sensitivity-analysisUse after optimising strategy parameters, to perturb them across a grid and measure how fast Sharpe decays; separates a fragile overfit peak from a genuine plateau. It does not deflate the selected Sharpe for trial count.parameter-sensitivity backtest-reporting-standardized-tearsheetUse at the end of a backtest run to produce one standard performance sheet (Sharpe, Sortino, Calmar, max drawdown, hit rate, profit factor) from a per-period returns array, so strategies are compared on identical metrics.tearsheet backtest-vs-live-performance-divergence-trackingUse after promoting a strategy to live trading, to measure and decompose the gap between backtested and realised Sharpe, hit rate and slippage, and alert when the divergence exceeds what execution friction explains.backtest-live-divergence benchmark-relative-performance-attributionUse when a strategy must be judged against a benchmark index rather than on raw return: alpha, beta, tracking error, information ratio and Brinson-Fachler allocation and selection effects. Multi-factor attribution with inference is strategy-performance-attribution-vs-market-beta.performance-attribution benchmark-selection-for-strategy-evaluationUse when choosing which index to evaluate a strategy against, before any attribution runs; screens candidate benchmarks by tracking error, beta and R-squared so a tech-heavy book is not flattered by a broad index or a cash rate.benchmark

C

corporate-action-adjusted-backtestingUse when a stock split, reverse split or cash dividend puts artificial gaps in a backtest's price history and fabricates signals; builds a CRSP-convention backward-adjusted series for signals while keeping dividend cash accounted separately.stock-splits cross-validation-of-commission-schedules-over-timeUse when a multi-year US equity backtest applies today's zero commissions to trades from 2016; models the schedule that actually applied on each date. Volume-tiered rates belong to exchange-fee-tier-and-rebate-structure-analysis.commission-schedule

D

data-vendor-cross-validation-for-backtestsUse before trusting a single vendor's history, to cross-check OHLCV bars against a genuinely independent second source for price discrepancies, phantom volume, stale prints and missing-bar coverage.data-validation

E

execution-realistic-simulationUse when a backtest reports fills at the signal bar close with no slippage or fees; models a marketable order crossing the spread with latency, impact and commission. Passive resting orders need a queue-position model instead.

I

intraday-vs-eod-backtest-granularity-tradeoffsUse when choosing bar resolution for a backtest or buying the data to feed one; an OHLC bar records four prices and no path, so stop-and-target strategies are ambiguous inside every bar regardless of bar length.data-granularity

L

lookahead-bias-eliminationUse when writing or auditing backtest code, to ensure no signal or decision uses information unavailable at the decision timestamp, starting with a bar's own close. Universe-selection lookahead is backtest-look-ahead-in-universe-selection.lookahead-bias

M

monte-carlo-strategy-robustness-testingUse when a promising equity curve is a single realised path and you need the distribution around it; trade-sequence shuffling, IID bootstrap resampling and execution-noise perturbation with confidence intervals on drawdown.monte-carlo multi-asset-backtest-currency-normalizationUse when a backtest holds assets in several currencies and P&L must be reported in one; converts cash flows and valuations at dated rates instead of summing unconverted numbers, which silently mixes currencies.multi-currency multi-timeframe-backtest-consistency-checksUse when a strategy trades bars it did not receive natively, resampled or vendor-supplied, to rebuild them from higher-resolution data and compare; catches boundary-anchor, gap and aggregation errors.multi-timeframe multi-year-regime-coverage-requirementUse before promoting a strategy, to check its backtest actually spans bull, bear, high-volatility and range regimes rather than one long trend, and to enforce a minimum coverage of each. Retrospective labels only, never a live signal.regime-classification

O

options-backtesting-with-realistic-iv-surfaceUse when backtesting straddles, verticals or condors and every leg is currently priced at one at-the-money volatility; prices each leg off a parametric surface across moneyness and term. European exercise only.options-backtesting

S

short-selling-borrow-cost-and-availability-modelingUse when a strategy shorts equities and the securities-lending leg must be priced: borrow fee accrual on an ACT/360 basis against daily-marked collateral, plus availability. Not a Regulation SHO locate.borrow-cost survivorship-bias-free-universe-constructionUse when a backtest needs the instruments actually tradable on each historical date, keeping delisted, bankrupt and acquired names in history and settling their terminal cash value rather than dropping them.survivorship-bias synthetic-data-generation-for-backtest-augmentationUse when a strategy rests on one short price history and you need more paths to judge it: geometric Brownian motion, GARCH(1,1) volatility clustering and circular block bootstrap, none of which invent tails beyond the sample.synthetic-data

T

transaction-cost-analysis-tca-integrationUse when validating whether a backtested edge survives execution, decomposing implementation shortfall into delay, spread cross, square-root impact, commission and opportunity cost, and comparing estimated against realised.tca-integration

V

vectorized-vs-event-driven-backtest-tradeoffsUse when choosing between a vectorized and an event-driven backtest engine, or when measuring how much of a vectorized result is an artefact of its fill assumption rather than the strategy.vectorized-backtest

W

walk-forward-hyperparameter-search-budgetUse when walk-forward optimisation would otherwise run tens of thousands of parameter combinations per window; bounds and audits the cumulative trial count that drives selection bias. It never sees a performance number.hyperparameter-budget walk-forward-optimization-window-managementUse when generating the rolling or anchored in-sample and out-of-sample windows for parameter optimisation, separated by a purge and embargo gap, and computing walk-forward efficiency across them.walk-forward-optimization walk-forward-validation-setupUse when validating a strategy or ML model on time-series data, because shuffled k-fold trains on the future; generates chronological train and test folds with a boundary gap and aggregates out-of-sample results.walk-forward-validation

Search the whole catalog