When to Use

Invoke this skill when a trading system holds a backup broker relationship and needs order flow to continue after the primary degrades — connection loss, sustained 5xx, throttling, or an outage mid-session. It provides a circuit breaker over two broker legs, with the failover decision gated on why the primary failed.

A generic circuit breaker guards idempotent reads, where retrying elsewhere is free. This one guards live order submission across two accounts, which breaks three of the pattern's usual assumptions:

A failed call is not a failed order. Not every error is an outage. A probe is a real order.

And one assumption that endpoint failover never has to think about at all: positions do not net across accounts. Failing a closing order over to the backup account does not flatten anything — it leaves you long in one account and short in the other, with double the gross exposure.

When NOT to Use

• As a retry mechanism. It routes each order once, to one place. Retrying the same intent is a different problem with different safety rules — see order-placement-idempotency.
• As reconciliation. It detects that an outcome is ambiguous and refuses to act; resolving "did the primary take my order?" needs the broker's order-state stream or a status query, which you supply. See webhook-based-order-fill-notifications.
• As a risk control. Pre-trade limits belong above the router and must apply to both accounts. A second broker is a second market-access path, not an exemption — sec-rule-15c3-5-risk-controls-us.
• As a kill switch. Opening the circuit stops new flow to the primary; it does not withdraw working orders. That is kill-switch-and-drawdown-circuit-breakers.
• For failover between endpoints of the same account (a regional DR flip, a second gateway). There the position-integrity problem does not exist and the routing is simpler — see multi-region-failover-for-broker-connectivity and smart-order-router-failover-on-venue-outage.

Prerequisites

• A funded, entitled, tested secondary account with the same instruments enabled. A backup you have never placed an order through is not a backup.
• Stable client_order_ids per order intent. This is the key you reconcile on when a call is ambiguous. Note what it does not buy you here: a client order id de-duplicates at the broker that issued it, and the secondary broker has never seen it. Cross-broker duplicates cannot be prevented by the broker — only by not sending the second order.
• An order-status resolver, ideally. Without one, every ambiguous primary failure becomes a hard stop for that order. With one, the router can ask "is it working?" and keep going.
• Seeded positions per account. The router's position map is a cache. Seed it from each broker's own position report at session start, or a REDUCE order will be judged on stale information.
• Symbol mappings for every tradable instrument, and strict_symbol_mapping=True in production.
• Documented business continuity arrangements where required. EU-authorised investment firms are subject to MiFID II RTS 6 (Reg (EU) 2017/589) Article 14, which requires business continuity arrangements for algorithmic trading systems, proportionate to the business and documented in a durable medium.

Workflow

1. Classify the failure before deciding anything. This is the whole skill. Four classes, three different answers:

   Class	Meaning	Failover?
   UNAVAILABLE	Connection refused, DNS failure — the request never left	Yes, safe
   RATE_LIMITED	429/418 — not accepted, but back off this leg	Yes, and honour Retry-After
   AMBIGUOUS	Timeout, connection reset, 5xx — it may be working	No — resolve first
   REJECTED	Bad symbol, insufficient buying power, size limit	No — the order is dead

   Anything unrecognised is AMBIGUOUS. The errors are not symmetric: a wrong "ambiguous" costs one status query, a wrong "failed" costs a duplicate order in an account that will never net it off.

2. Never fail over an ambiguous outcome. Binance's REST documentation states the general case for 5xx: "It is important to NOT treat this as a failure operation; the execution status is UNKNOWN and could have been a success." Either resolve it through a status query, or stop and make the caller reconcile. A resolver that itself errors means still ambiguous — never read a failed status check as "not there".

3. Never fail over a terminal rejection. "Insufficient buying power" is the primary's pre-trade risk control working. Routing it to a second account is shopping for a broker that will say yes. Re-raise it, and do not count it against broker health — otherwise the strategy's own bad orders trip the breaker.

4. Pin reducing orders to the account that holds the position. Mark every closing or reducing order PositionEffect.REDUCE. In an account with no position the identical instruction opens new exposure. In US equities it is also a compliance event: Reg SHO Rule 200(c) deems a person to own securities "only to the extent that he has a net long position", Rule 200(g) permits a "long" mark only where the seller owns the security, and Rule 203(b)(1) requires a locate before a short sale order is accepted. Netting the two accounts requires the independent-trading-unit conditions of Rule 200(f), including a written plan. If the holding account is down, the correct answer is to refuse and escalate — not to open a position somewhere else.

5. Bound the half-open probes. Each probe is capital at risk against a broker you believe is down. Generic implementations permit several concurrent trial calls (resilience4j's permittedNumberOfCallsInHalfOpenState defaults to 10); here the default is one, claimed under an explicit permit so concurrent callers cannot all probe at once.

6. Ignore stale successes. A slow call that started before the breaker tripped can return after it. Honouring it closes a circuit that other threads opened for good reason. Tag each attempt with the circuit generation and discard successes from an older one.

7. Measure the recovery timeout on a monotonic clock. time.time() moves when NTP steps it; a backward jump holds the circuit open indefinitely.

8. Handle "both legs down" as its own outcome. It is the case that matters most and the one most likely to be untested. The caller needs a typed error naming the order, not whatever the secondary happened to raise.

Full failure-classification tables and phase procedure: see references/workflows.md. Cited standards, Reg SHO analysis, and thresholds: see references/standards.md. Printable go-live checklist: see assets/checklist.md.

Common Pitfalls

• Failing over on a timeout. The primary may have accepted the order. The secondary cannot de-duplicate it, because it has never seen the client order id. You end up with two live orders in two accounts and no automatic way to net them.
• Treating every exception as a clean failure. except Exception: route_to_secondary is the single most damaging line this pattern attracts.
• Failing over a business rejection. An order the primary refused on buying power or size gets filled at the backup — the failover has quietly become a risk-control bypass.
• Counting rejections toward the breaker. A strategy sending bad orders then trips the circuit and pushes healthy flow to the backup for no reason.
• Failing over a closing order. It opens new exposure instead of reducing any, and in US equities produces a sale in an account that owns nothing.
• Assuming positions net across accounts. They do not, absent a Rule 200(f) aggregation-unit plan. Two accounts flat against each other is double gross exposure, double margin, and two settlement obligations.
• Unbounded half-open probing. Every concurrent caller sends a live order to a broker believed to be down.
• Letting a stale success close the circuit. One slow call returning late resets the breaker and the failure counter, and flow returns to a broker that is still broken.
• Using the wall clock for the recovery timeout. An NTP step forward probes early; a step backward never probes at all.
• Silently passing an unmapped symbol through. The same ticker can denote a different instrument at the other broker. Use strict mapping in production.
• Leaving the secondary leg untested. A backup that has never taken an order is an assumption, not a control. Exercise it on a schedule.
• Forgetting the backup has its own limits. Different rate limits, different order types, different margin treatment, different pattern-day-trade and margin accounting per account. Flow that fits the primary may not fit the backup.

Verification

• Run python -m unittest discover -s skills/broker-failover-secondary-account-routing/scripts and confirm all tests pass.
• Make the primary accept an order and then raise TimeoutError. Confirm the router raises AmbiguousOrderStateError carrying the client_order_id, and that the secondary received nothing.
• Supply an order_status_resolver that finds the order; confirm the router returns the real primary fill and still does not touch the secondary. Then make the resolver raise and confirm the outcome stays ambiguous rather than becoming "not found".
• Make the primary raise a REJECTED BrokerError; confirm it is re-raised, the secondary is untouched, and the circuit stays CLOSED with primary_failures == 0.
• Make the primary raise UNAVAILABLE; confirm the order reaches the secondary with the secondary's symbol mapping applied.
• Fail both legs; confirm AllBrokersUnavailableError names the order.
• Seed a long position in the primary, take the primary down, and submit a PositionEffect.REDUCE sell. Confirm PositionAffinityError and that no short was opened at the secondary.
• Trip the circuit, wait past the recovery timeout, then fire ten concurrent orders and confirm exactly one reached the primary.
• Start a slow successful call, trip the circuit from other threads while it is in flight, then let it return. Confirm the circuit is still OPEN.
• Patch time.time to a fixed value and confirm recovery still happens on schedule.
• Call manual_open, wait past the recovery timeout, and confirm the primary is still not probed until manual_reset.
• With strict_symbol_mapping=True, submit an unmapped symbol and confirm SymbolMappingError — not an ambiguous outcome, and no failover.

Related Skills

• order-placement-idempotency
• broker-api-idempotent-cancel-requests
• broker-agnostic-adapter-interface
• multi-region-failover-for-broker-connectivity
• smart-order-router-failover-on-venue-outage
• multi-broker-consolidated-position-view
• multi-broker-rate-limit-handling
• kill-switch-and-drawdown-circuit-breakers
• webhook-based-order-fill-notifications
• us-reg-sho-short-sale-locate-requirements
• sec-rule-15c3-5-risk-controls-us
• counterparty-and-broker-concentration-risk