Real-Time Architecture

Producer-consumer tick pipelines, burst-safe buffering, explicit backpressure policy, WebSocket subscription reconciliation after a reconnect.

Skills: 31
Hand off: 238
Handed off from: 257

Index

All 31 skills in this domain, alphabetically. Hover a row for its situation; open it for the full playbook.

A

adaptive-batch-size-tuning-under-loadUse when writing high-volume ticks or order logs into TimescaleDB, ClickHouse or Kafka and one static batch size either stalls in quiet markets or floods the sink in a burst; adapts batch size and flush timeout from EWMA sink latency.adaptive-batching adaptive-sampling-under-extreme-tick-ratesUse when a trade-tick stream exceeds a consumer's measured processing ceiling and you must emit 1:N samples that still preserve traded volume and notional. Not for order-book, quote or compliance feeds that need every message.adaptive-sampling

B

backpressure-drop-degrade-policyUse when a bounded tick buffer sits at capacity for sustained periods and you must choose a drop, sample or degrade policy per stream instead of inheriting a queue library's blocking default. For momentary spikes see tick-buffering-burst-handling.backpressure binary-protocol-parsing-for-low-latency-feedsUse when decoding fixed-layout binary market data by byte offset (NASDAQ ITCH, CME MDP 3.0 SBE, Eurex T7 EMDI): frame validation, message-type dispatch, fixed-point tick prices and zero-copy buffer walking that fails loud.binary-protocol

C

circuit-breaker-for-downstream-service-callsUse when a trading process calls a non-order dependency (reference data, an alt-data vendor, an internal microservice) that can hang and exhaust threads and connection pools. Never wrap order submission or cancels with it.circuit-breaker clock-skew-correction-for-tick-timestampsUse when captured ticks carry both a venue send time and a local receive time and the local clock drifts. Estimates rate offset from windowed minimum one-way delays and re-expresses local stamps on the venue timescale, monotonically.minimum-delay-filtering consumer-group-rebalance-safetyUse when trading workers consume order or tick events from a Kafka consumer group and a rebalance can move a partition mid-flight. Fences revoked partitions, drains in-flight work, and commits the next offset rather than the last processed one.consumer-group cross-datacenter-clock-sync-validationUse when a strategy merges ticks from more than one datacenter (NY4, LD4, a cloud region) and must prove those sites agree closely enough to order events correctly. Measures pairwise inter-region drift with its uncertainty and fails closed.clock-sync

E

exchange-multicast-feed-handlingUse when building the sequencing layer of a co-located UDP multicast feed handler (CME MDP 3.0, MoldUDP64, Eurex T7 EMDI): A/B line arbitration, datagram re-sequencing, gap arbitration windows and venue retransmission escalation.line-arbitration

F

feed-handler-canary-deploymentUse when releasing a new market data parser and first live exposure should cover a slice of symbols only. Both decoders run in parallel, prices are audited tick by tick against the incumbent, and divergence triggers auto-rollback.canary-deployment feed-handler-cpu-pinning-and-numa-awarenessUse when a feed handler or execution loop shows latency jitter unrelated to market activity. Pins workers to chosen cores and confirms pages are resident on the local NUMA node, removing scheduler migration and cross-socket interconnect hops.cpu-pinning

G

graceful-degradation-to-polling-fallbackUse when a strategy holds positions behind a streaming feed that can go silent without ever firing on_close. Detects a dead stream, fails over to rate-limit-aware REST polling, dedupes across the handover, and declares itself blind when both fail.feed-degradation graceful-shutdown-draining-in-flight-ticksUse when redeploying or restarting tick workers under SIGTERM and in-flight queue entries, sink writes and consumer offsets must not be lost. Drains data queues; open orders and positions need an explicit unwind decision first.graceful-shutdown grpc-streaming-for-internal-service-communicationUse when internal trading services poll each other over REST JSON and the poll interval sets the latency floor. Replaces it with gRPC bi-directional HTTP/2 streaming, typed protobuf payloads and push semantics.grpc

K

kafka-based-tick-distribution-at-scaleUse when fanning ticks to many consumers over Kafka and per-symbol ordering must be real. Reproduces each client library's partition hash (librdkafka CRC32 versus murmur2), pins producer settings, and audits lag, staleness and hot partitions.tick-distribution

M

market-data-feed-arbitration-across-vendorsUse when a strategy prices off two independent vendors for the same instrument and must decide per tick whether a tradeable, cross-verified price exists. Covers divergence tolerance, stale and frozen feed quarantine, and total blackout.feed-arbitration market-data-replay-harness-for-integration-testingUse when integration-testing the real feed handler, strategy and risk pipeline against a recorded session. Replays captured ticks in a reproducible order at 1x or a speed multiplier and measures the scheduling lag of the replay itself.tick-replay market-data-snapshot-plus-delta-reconciliationUse when initialising an L2 or L3 book from a WebSocket delta stream plus a REST snapshot: buffers deltas, aligns on the snapshot sequence id, discards stale updates, and flags the gaps that force a re-sync.order-book memory-mapped-ring-buffer-for-ultra-low-latencyUse when handing ticks between two processes without a language-level queue: fixed binary slots in a shared memory-mapped file, split head and tail ownership, explicit drop-on-full, single producer and single consumer only.memory-mapped multi-exchange-feed-normalizationUse when trade prints arrive from several venues and downstream code must not branch per venue. Maps payloads onto one canonical tick with a single symbol namespace, aggressor-side convention and timestamp timescale, failing loudly on unreadable fields.feed-normalization multi-region-active-active-tick-ingestionUse when the same feed is ingested from two or more cloud regions for availability and something must forward the first-arriving copy of each tick, dropping redundant copies by signature within a bounded dedup window.active-active

N

network-interface-level-tick-timestampingUse when a Linux feed handler must stamp ticks at wire arrival rather than in the application. Decodes SO_TIMESTAMPING ancillary messages and reads the NIC hardware stamp without mistaking a kernel software timestamp for it.nic-timestamping

O

order-book-depth-processing-l2-l3Use when maintaining a local L2 or L3 book from an already-sequenced depth feed, and reading top-of-book, weighted mid-price and depth imbalance off it under thread-safe mutation, with crossed and locked book detection.order-book-l2-l3 order-book-imbalance-signal-pipelineUse when a strategy trades on the relative size of resting queues. Computes signed top-of-book and depth-aggregated imbalance plus the imbalance-weighted mid, rejecting crossed books and out-of-order updates before a signal reaches execution.queue-imbalance

P

producer-consumer-tick-pipelineUse when non-trivial work runs inside a WebSocket on_message callback and can stall the socket read loop. Moves processing off the read path so a slow database write, GC pause or inference call cannot cause a broker disconnect.websocket-ingestion

R

redis-streams-multi-consumer-tick-fanoutUse when one feed must reach several independent consumers with a bounded replay window. Covers Redis Streams consumer groups, XACK as the only thing that drains the pending entries list, and XAUTOCLAIM recovery of a crashed worker's backlog.tick-fanout

S

sequence-number-gap-detection-for-feedsUse when downstream state is only correct if every feed message arrived exactly once and in order. Tracks one expected sequence per stream, buffers out-of-order frames, computes the missing ranges, and withholds trading authorization until repaired.sequence-number

T

tick-buffering-burst-handlingUse when sizing in-memory tick buffers so an expiry or news burst neither drops data silently nor grows unbounded into an OOM kill. Pairs with producer-consumer-tick-pipeline; sustained backlog belongs to backpressure-drop-degrade-policy.tick-buffering tick-data-schema-versioningUse when an internal tick schema changes while consumers run at mixed versions. Stamps a schema_version envelope, chains upgrade and downgrade adapters, and marks the fields a migration synthesised rather than received.schema-versioning

W

websocket-reconnection-with-state-recoveryUse when a long-lived venue WebSocket must reconnect and then prove downstream state still matches: bounded jittered backoff, deterministic re-subscription from desired state, and fail-closed sequence-gap recovery that withholds messages until filled.websocket websocket-subscription-reconciliation-after-reconnectUse when reconnect logic must restore exactly the intended subscription set. Reconciles the broker's own subscribe acknowledgements rather than assuming success, backfills the gap window in order, and dedupes ticks replayed across the reconnect.websocket

Search the whole catalog