Custody Solution Uptime And Liveness Guarantees

When to Use

Use this skill on institutional crypto desks whose automated strategies depend on a custody provider being able to sign right now. In 24/7 crypto markets a custody outage or an MPC quorum failure during a volatility spike blocks margin top-ups and liquidation defence, and the loss is realized before a human notices.

The engine consumes health-probe telemetry and answers one question — is the primary custodian live enough to sign, and if not, should we fail over? It reports:

• Uptime against your contractual target, compared without a rounding step that could mask a breach.
• MPC quorum for a k-of-n signing cluster, including a QUORUM_AT_RISK warning at zero remaining redundancy — before the halt, not during it.
• P99 signing latency, gated on having enough samples for a 99th percentile to mean anything.

It fails closed. No telemetry, stale telemetry, and malformed telemetry are all non-healthy outcomes that recommend failover. Silence is not health.

When NOT to Use

• As the failover executor. This engine sets is_failover_recommended. It does not move keys, re-route signing, or touch the secondary provider. Failover is a capital-moving action; keep the decision and the execution separate and human-reviewable.
• As a source of SLA numbers. Every threshold is an input, not a constant. target_uptime_pct and max_signing_latency_ms come from your executed agreement with the provider. No numeric SLA is hard-coded or implied here.
• As a time-weighted availability calculator for SLA credits. Uptime is computed as a probe-success ratio, which equals contractual availability only when probes are evenly spaced. Claims for service credits should be computed from the provider's own incident record and your contract's measurement method.
• As a substitute for the provider's status page. A probe pipeline can be healthy while the custodian is degraded in ways your probes do not exercise (for example, withdrawals halted while reads succeed).

Prerequisites

• Contractual SLA terms from the executed provider agreement: target_uptime_pct, max_signing_latency_ms, and the measurement window they are defined over.
• Cluster shape: mpc_threshold_k (shares required to sign) and mpc_total_n (shares provisioned). A k-of-n cluster tolerates exactly n - k unavailable nodes.
• Continuous health-probe telemetry carrying timestamp_ms, is_api_healthy, signing_latency_ms, and active_mpc_nodes.
• A freshness bound (max_probe_age_ms) and a wall-clock reference (as_of_timestamp_ms) if staleness is to be detected at all.

Workflow

1. Configure from the contract, then validate. CustodyLivenessMonitorEngine(config) rejects an unsatisfiable cluster (k > n), a target_uptime_pct outside [0, 100], and non-positive latency or freshness bounds at construction. A 4-of-3 quorum is dead on arrival and should fail at startup, not at the first signing attempt.
2. Feed probes; do not pre-sort them. audit_liveness orders by timestamp_ms itself. Do not rely on list order to identify the current state — concurrent collectors routinely deliver an older probe last, and taking the final list element as "now" is how a real quorum loss goes unseen.
3. Pass as_of_timestamp_ms whenever you have a clock. Without it, freshness is not evaluated and the report says so explicitly. With it, a newest probe older than max_probe_age_ms yields STALE_TELEMETRY and recommends failover — a frozen collector must not read as a healthy custodian.
4. Read breached_conditions, not just status. status is the single most severe condition; breached_conditions and recommendations carry all of them. A quorum halt and an uptime breach commonly occur together and the operator needs both.
5. Check percentiles_reliable before acting on P99. When latency_sample_count < min_latency_samples the P99 is reported but explicitly not gated, because a 99th percentile drawn from a handful of samples is the maximum wearing a percentile's name. Widen the window rather than lowering the bar.
6. Decide failover deliberately. Quorum loss, uptime breach, stale telemetry, and no telemetry all set is_failover_recommended. A latency breach does so only when failover_on_latency_breach=True, because slow signing is usually better than a custody migration mid-volatility — make that trade explicit rather than inheriting it.

Full procedure: see references/workflows.md. Standards reference: see references/standards.md. Printable pre-flight checklist: see assets/checklist.md.

Common Pitfalls

• Treating "no data" as "no problem". An empty probe list is the signature of a dead collector, not a perfect custodian. Any monitor that returns healthy when blind will report healthy through the exact incident it exists to catch.
• Rounding before comparing. Rounding uptime to two decimals and then testing it against a 99.9% target lets a true 99.896% clear the gate. Round for display, compare on the raw value.
• Trusting list order for "current" state. probes[-1] is the last element, not the newest observation. Sort by timestamp before reading the current node count.
• Letting NaN through. Every > comparison against NaN is False, so one corrupt latency reading passes an SLA gate instead of tripping it. Reject non-finite telemetry at ingestion.
• Alerting only after quorum is lost. In a 2-of-3 cluster, two active nodes is not healthy — it is one node from a signing halt, and routine maintenance on either node causes it. Warn at zero remaining redundancy.
• Gating on an under-sampled P99. A 99th percentile needs at least 100 observations to resolve the top 1% at all; below that it is just the maximum and one unlucky request looks like an SLA breach.
• Reporting one breach at a time. An if/elif chain hides concurrent failures; the quorum alarm masks the uptime breach that explains it.
• Reading a 99.9% figure as a standard. SOC 2 does not set an uptime number — see references/standards.md. The number is contractual, and it is only meaningful alongside its measurement window.
• Confusing probe-ratio uptime with time-weighted availability. If probe frequency rises during incidents (retries, denser polling), the ratio is biased relative to the contractual measure.

Verification

• Construct ProviderSlaConfig(target_uptime_pct=99.9, max_signing_latency_ms=2000.0, mpc_threshold_k=2, mpc_total_n=3). Feed 100 healthy probes at 450 ms with 3 active nodes → HEALTHY, redundant_nodes == 1, no failover.
• Feed 24,974 healthy probes and 26 unhealthy (99.896%) → DEGRADED_SLA_BREACH with failover. Rounding to two decimals would have shown 99.9% and passed.
• Feed probes where the newest by timestamp reports 1 active node but arrives mid-list → QUORUM_LOST_LIVENESS_HALT, current_active_mpc_nodes == 1.
• Feed an empty list → UNKNOWN_NO_TELEMETRY with is_failover_recommended is True.
• Set mpc_threshold_k=4, mpc_total_n=3 → ValueError at construction.

python -m unittest discover -s skills/custody-solution-uptime-and-liveness-guarantees/scripts

Related Skills

• custodial-vs-non-custodial-tradeoff-assessment
• multi-party-computation-mpc-custody-solutions
• custody-solution-vendor-due-diligence-checklist
• broker-failover-secondary-account-routing