When to Use
Machine learning models inherently generate highly active signals (e.g., changing predictions from slightly positive to slightly negative on every bar). Without strict Transaction Cost Analysis (TCA) and confidence thresholding, these models achieve massive theoretical Sharpe ratios that instantly collapse in live trading due to Turnover Drag. Invoke this skill to evaluate ML model predictions against realistic slippage and commission models, filtering out low-confidence trades that do not cover their own execution costs.
When NOT to Use
- Order-level cost attribution. This skill models cost as a flat bps charge per unit of turnover. To decompose realised shortfall into delay, spread, and market impact, use
transaction-cost-analysis-tca-integration. - Size-dependent impact. The flat-rate model has no ADV or participation term, so it will understate cost for orders that are large relative to available liquidity. Pair it with
liquidity-adjusted-position-sizingandstrategy-capacity-estimation-before-scaling-capitalbefore scaling capital. - Continuous or leveraged target weights. Positions here are discrete
{-1, 0, +1}. A continuous-weight portfolio needs turnover computed on the weight vector — seeportfolio-construction-with-transaction-cost-awareness.
Prerequisites
- An array of numerical ML predictions (e.g., predicted forward returns or continuous signal strength).
- An array of actual underlying returns, already aligned so that element
iis the return realised afterpredictions[i]was observable. The backtester cannot detect misalignment. - Estimated transaction costs in basis points (bps) per half-turn (e.g., 5 bps for slippage + fee), calibrated against your own execution data rather than a folklore constant.
Workflow
- Align predictions to forward returns: pair
predictions[t]with the return earned by acting on it (t → t+1, ort+1 → t+2if your execution lags the signal by a bar). Getting this wrong is the single most common way to produce a spectacular, meaningless backtest. - Clean the inputs: drop or impute NaN/Inf predictions before the backtest. A NaN prediction silently classified as "flat" is a hidden regime change in the strategy, not a neutral default —
MlTcaBacktesterrejects non-finite input rather than absorbing it. - Set the hurdle: the entry threshold must clear the round-trip cost. A position that is entered and later exited pays 2 units of turnover, so require
signal_threshold >= 2 * bps_cost_half_turn / 10_000in decimal-return units. Below this line, a correct prediction is still a losing trade. - Generate positions: map predictions to
{+1, -1, 0}against the threshold. - Decide whether to apply a buy/hold spread: if the model churns around the threshold boundary, set
exit_threshold < signal_thresholdso an open position is held until conviction genuinely decays, rather than being closed and re-entered on noise. - Calculate turnover: costs are charged on position changes only. Flat→long is 1 unit; long→short is 2 units; a position still open on the final bar owes its exit half-turn (
liquidate_at_end, on by default). - Apply cost drag and compute net returns: subtract
turnover × bps_cost_half_turn / 10_000from gross returns each period. - Sweep the threshold out-of-sample: pick
signal_threshold/exit_thresholdon training folds and confirm them on held-out data. Choosing the threshold that maximises net return on the same sample you report is parameter mining, not cost control — seewalk-forward-optimization-window-management. - Stress the cost assumption: re-run at 2–3× your estimated bps. A strategy whose net edge disappears when costs double has no margin for a bad execution day.
Full procedure: see
references/workflows.md. Standards reference: seereferences/standards.md. Printable pre-flight checklist: seeassets/checklist.md.
Common Pitfalls
- Ignoring State Persistence: Charging a transaction cost on every bar simply because the model predicted "Buy", even though the portfolio is already in a "Buy" state. Costs are only incurred on position changes (Turnover).
- Zero Thresholding: Taking trades on microscopic predictions (e.g., model predicts +0.0001% return) which guarantees a net loss when the transaction cost is 0.05%. A threshold of exactly
0is worse than useless — it makes the long and short conditions overlap, so a neutral prediction can be classified as a short. - Symmetric Costs: Forgetting that crossing the spread (market order) costs money on both entry and exit (half-turns).
- The Free Exit: Ending the backtest with an open position and never charging the closing half-turn. On a low-trade-count strategy this can hide a meaningful slice of total cost.
- Contemporaneous Returns: Multiplying
predictions[t]by the return of the bar the prediction was made on. This is look-ahead bias, and the resulting equity curve survives any cost assumption you throw at it — seelookahead-bias-elimination. - NaN as Neutral: Letting missing predictions default to "flat". The backtest then reports a strategy nobody can trade, and a NaN in the return series silently poisons every compounded metric.
- Threshold Mined In-Sample: Sweeping
signal_thresholdacross the full history and reporting the best net result. The threshold is a free parameter, and cost-aware backtests overfit it as readily as any other.
Verification
- Simulate an ML model that perfectly predicts 10% returns but flips direction every day. Apply a 6% (600 bps) per-half-turn cost. The gross return will be massive, but the net return must be decisively negative — covered by
test_perfect_but_churning_model_is_net_negative. - Confirm a held position is charged once, not once per bar: two consecutive identical long signals must produce 1 unit of turnover, not 2.
- Confirm a neutral (
0.0) prediction maps to flat, never to short. - Run
python -m unittest discover -s skills/backtesting-ml-models-against-transaction-costs/scriptsand confirm a 100% pass rate.