Reinforcement Learning Safety Constraints For Execution

When to Use

Use this skill when a reinforcement-learning policy decides how much to trade on an execution task — optimal liquidation, scheduled slicing, market making — and its raw output would otherwise reach an order router. An unconstrained policy in an out-of-distribution state proposes quantities that are merely improbable in training and catastrophic in production: a size ten times its normal slice, an inventory build past every limit, an aggressive order into a spread that has just gapped.

SafeRLExecutionGuard is a post-posed shield in the sense of Alshiekh et al. (AAAI 2018): it lets the policy choose freely, monitors what it chose, and corrects the choice only when a hard constraint would be violated. Corrections are deterministic, ordered, and individually attributable, and each one can carry a penalty back into training so the policy learns which of its own proposals were unsafe.

ESMA's February 2026 supervisory briefing names reinforcement learning directly and tells firms to "consider risks posed by trading signals generated by more advanced AI-related technology when designing PTCs" (pre-trade controls). This skill is one such control.

When NOT to Use

• As your SEC Rule 15c3-5 market-access control. Rule 15c3-5(d)(1) requires the financial and regulatory risk-management controls to be "under the direct and exclusive control of the broker or dealer." A limit living inside your own agent is by definition not under the broker's exclusive control. This shield sits in addition to broker-side controls, never instead of them.
• As the firm's complete RTS 6 pre-trade control set. It implements an order-volume limit (RTS 6 Art. 15(1)(c)) and a firm-risk-threshold block (Art. 15(5)). It implements no price collar (Art. 15(1)(a)), no order-value limit in currency (Art. 15(1)(b)), no message-rate limit (Art. 15(1)(d)), and no repeated-execution throttle (Art. 15(3)). See references/standards.md for the full mapping.
• As a price or execution-quality control. The guard shields quantity only. It never inspects limit price, order type, venue, or aggressiveness. A shielded quantity sent as a market order into a thin book is still a bad order.
• As a substitute for a kill switch. The shield constrains each action; it cannot stop a strategy, cancel resting orders, or withdraw from a venue. That is execution-algorithm-kill-switch-integration and RTS 6 Art. 12.
• Where the policy controls something other than quantity. A policy that outputs a limit-price offset or a participation rate needs a shield over that action space; this one will pass it through untouched while appearing to guard it.
• As a reason to skip pre-deployment testing. A shield bounds the worst case; it does not make an unvalidated policy safe. Retraining an RL policy is itself a material change requiring re-testing — ESMA lists "Retraining or modifying machine learning components" as a change type that warrants it.

Prerequisites

• An execution state per decision step: current_inventory, max_inventory, bid, ask, time_remaining_sec, max_spread. max_inventory and max_spread must be finite and non-negative — ExecutionState raises RLSafetyError otherwise.
• bid/ask from the same snapshot as the state the policy observed. A quote from a different instant makes the spread veto measure a spread nobody could have traded.
• current_inventory reconciled against the broker's position, not the agent's internal belief. The cap is only as sound as the inventory fed to it.
• Guard limits agreed with whoever owns risk, not chosen by whoever writes the policy. Under ESMA's guidance a limit a trader can move unilaterally is not a hard block at all.
• time_remaining_sec measured against the parent order's actual deadline, which may be zero or negative once the window has closed.

Workflow

Guards run in this fixed order on every step. The order is a property of the shield, never of what the policy proposed.

1. Data-integrity gate (fail closed):
   • Veto to safe_qty = 0.0 on a non-finite proposed quantity, a non-finite inventory, quote or clock, or a crossed book (ask < bid).
   • Decision point — bad data must veto, not fall through. Every guard below is a value > limit comparison, and every such comparison is False when the value is NaN. A single NaN quote does not trip the spread veto; it silently disables it, and the action routes unshielded. Failing closed is the only safe reading.
   • Decision point — do not punish the policy for this. The environment produced the bad quote, not the policy. Charging penalty_lambda here teaches the agent to avoid states it does not control. is_data_integrity_failure marks the step; drop the transition rather than training on it.
2. Terminal inventory clearance:
   • Inside terminal_horizon_sec with non-zero inventory, override the policy with a liquidating order, itself clipped to max_order_size.
   • Decision point — "clearance is in force" is a fact about the state, not about the correction. Deriving it from "did the shield have to change the action" suspends the spread veto for a policy that proposed nothing while applying it to a policy that proposed the correct liquidation — the exact inversion of what you want.
3. Spread veto:
   • Veto when spread > max_spread, unless a terminal clearance is in force and terminal_clearance_overrides_spread_veto is set (the default).
   • Decision point — this default trades a known cost for an unbounded one. Crossing a wide spread to flatten costs a measurable amount; carrying inventory past the execution deadline is an open-ended risk. Firms that would rather hold set the flag to False and must then handle the residual inventory themselves.
4. Max order size clip: clip |qty| to max_order_size, preserving sign.
5. Position cap:
   • Clamp projected inventory into [-max_inventory, +max_inventory], widened to include the current inventory, then derive the order as clamped - current.
   • Decision point — the band is widened so an over-limit position stays reducible. Sizing the order from same-side headroom (cap - |inventory|) returns zero for any order once the position is already outside the cap, including the order that would bring it back in. A lowered limit, a manual position or an external fill would trap exposure the shield exists to shed.
   • Decision point — clamp the target, don't cap the order. A sell from +900 against a 1000 cap may run to −1900 (down to the −1000 floor). Sizing it from the long-side headroom yields −100 and under-executes by 95%.
   • After the deadline (time_remaining_sec <= 0) the band tightens to the span between flat and the current inventory: reduce-only, and no overshoot through zero.
6. Cumulative quantity budget (optional, max_cumulative_qty):
   • Clip against the episode's remaining traded-quantity budget. Forced terminal clearance is exempt so a spent budget can never strand inventory.
   • Decision point — the per-order clip alone bounds nothing cumulative. A policy denied 5,000 shares simply proposes 100 fifty times. ESMA is explicit that a hard block must not be circumventable "indirectly (e.g. by slicing blocked orders to circumvent the set parameters)." This defaults to None, i.e. unconstrained — set it.
7. Reward penalty shaping:
   • Deduct penalty_lambda once per intercepted step, not once per violated constraint.
   • Decision point — attribute the penalty to proposed_qty, never to safe_qty.

Full procedure: see references/workflows.md. Standards reference: see references/standards.md. Printable pre-flight checklist: see assets/checklist.md.

Common Pitfalls

• Storing safe_qty in the replay buffer next to the penalised reward: this is the single most destructive misuse of a post-posed shield. The punishment exists to teach the policy that its proposal was unsafe; pairing it with the corrected action teaches the policy that the safe action is bad, and the policy learns to avoid safety. Store proposed_qty with shaped_reward, or shield without punishing (penalty_lambda=0.0).
• Assuming a NaN will be caught by a limit check: float('nan') > 100 is False, so every threshold in an unguarded shield passes and the NaN quantity reaches the router. Bad data does not trip guards, it removes them.
• Treating a per-order clip as a bound on activity: it caps one message, not the episode. Without a cumulative budget, an interception is only a delay.
• Letting the position cap block risk reduction: a cap computed as remaining same-side headroom returns zero once the position is outside the limit, so the shield refuses the de-risking order precisely when the position is worst.
• Trading a crossed book as if it were a tight one: ask < bid gives a negative spread, which passes any spread > max_spread test comfortably. A crossed top-of-book is a data fault, not an opportunity.
• Opening fresh exposure after the execution window closes: with inventory already flat and the deadline passed, nothing in a naive horizon check stops a brand-new position.
• Reading the terminal horizon as a flat-at-deadline guarantee: the forced liquidation is clipped to max_order_size like any other order, and nothing checks that the horizon leaves enough steps to finish. 800 units at 100 per slice needs 8 slices; a 60-second horizon polled every 30 seconds offers 4, and 400 units quietly survive the deadline. Feed quality makes it worse — a data-integrity veto inside the terminal window burns a slice.
• Tuning penalty_lambda against the reward scale by feel: a penalty far smaller than typical step rewards is ignored by the policy, which learns to propose unsafe actions and let the shield fix them; far larger, and it dominates the objective and suppresses legitimate trading. It is a hyperparameter of the reward scale, not a risk limit — the risk limit is the clip, which binds regardless.
• Reading a falling interception rate as a safer policy: it also falls when the policy learns to sit just inside the limits, or when the market stops producing the states that trigger vetoes. Track interceptions by reason code, not in aggregate.
• Letting recalibration drift past review: ESMA warns that "a series of minor or small changes due to recalibrations could accumulate over time... into a material change in the model output without it being tested." An online-learning policy changes continuously by construction.
• Silently widening a limit to stop the interception alerts: the alerts are the control working. Changing these thresholds is itself a material change, and ESMA expects revisions to involve the risk-management and compliance functions.

Verification

• Instantiate SafeRLExecutionGuard(max_order_size=5000.0) and propose +500 with current_inventory=800.0, max_inventory=1000.0. Expect safe_qty == 200.0 and reason_codes == ('POSITION_CAP',) — headroom is 200, and the size clip is deliberately held out of the way so the cap is what is being tested. With the default max_order_size=100.0 the same proposal yields 100.0 under MAX_ORDER_SIZE, and the cap never binds at all.
• Propose -1950 at current_inventory=900.0, max_inventory=1000.0, max_order_size=5000.0. Expect -1900.0: the target is clamped to the −1000 floor, not sized from long-side headroom (which would give −100).
• Propose -100 at current_inventory=1200.0, max_inventory=1000.0. Expect it to pass unintercepted — reduction from an over-cap position is always admissible. Propose +50 in the same state and expect 0.0 under POSITION_CAP.
• Propose NaN, inf, a NaN bid, a NaN inventory and a crossed book (bid=100.50, ask=100.00). Each must return safe_qty == 0.0, is_data_integrity_failure is True, and shaped_reward == base_reward (no penalty). A locked book (bid == ask) must trade.
• With current_inventory=500.0, time_remaining_sec=30.0 and a 2.50 spread against max_spread=1.0, confirm the routed quantity is -500.0 for every proposal in {0, 50, -500, -250, 900} — identical regardless of what the policy proposed. Repeat with terminal_clearance_overrides_spread_veto=False and confirm it is 0.0 for all five.
• With time_remaining_sec=-10.0 and flat inventory, propose +500 and expect 0.0 under HORIZON_EXPIRED.
• With max_order_size=100.0, max_cumulative_qty=250.0, propose +100 five times and expect [100, 100, 50, 0, 0]. Confirm a forced terminal liquidation still routes with the budget fully spent.
• Confirm SafeRLExecutionGuard(max_order_size=0) and ExecutionState(max_spread=-0.5) both raise RLSafetyError.
• Run python -m unittest discover -s skills/reinforcement-learning-safety-constraints-for-execution/scripts — 45 tests, 100% pass rate.

Related Skills

• execution-algorithm-kill-switch-integration
• kill-switch-and-drawdown-circuit-breakers
• risk-control-unit-testing-framework
• order-to-trade-ratio-fee-penalty-avoidance
• model-versioning-and-rollback
• explainability-for-live-trading-signals
• mifid-ii-algo-trading-compliance-eu
• sec-rule-15c3-5-risk-controls-us