Backtest Audit Trail For Regulatory Review

When to Use

Invoke this skill when performing strategy validation prior to live deployment or institutional client presentation, and you need a durable record of exactly what code, data, and parameters produced a set of backtested numbers. It captures git commit SHA, per-file input data checksums, hyperparameters, performance metrics, and execution environment, then authenticates the record with HMAC-SHA256 so later modification by anyone without the signing key is detectable.

When NOT to Use

• As proof to a regulator that your firm did not alter its own results. HMAC is symmetric: whoever can sign can also forge. This gives integrity against outsiders, not non-repudiation against the issuer. See the trust-boundary note below.
• As a substitute for compliant recordkeeping storage. Under 17 CFR 240.17a-4(f) the durability obligation is met by the storage system (WORM or the audit-trail alternative), not by a hash embedded in a file.
• For live order/trade event capture. Order and execution event reporting is a different obligation with different systems — see the CAT and RTS 25 notes in references/standards.md.
• When you have not yet eliminated lookahead bias. An impeccably signed manifest for a biased backtest is a precisely documented wrong answer.

Prerequisites

• Git repository with clean working tree — a commit SHA taken against uncommitted changes documents code that never existed.
• Input data files accessible for checksumming.
• Strategy parameter configuration (all tunable hyperparameters).
• A signing key of ≥32 random bytes from a secrets manager or environment variable — never committed to the repository and never stored alongside the manifests it authenticates.
• Standard library only: hashlib, hmac, json, platform.

Workflow

1. Capture Lineage Metadata: Record git commit SHA (git rev-parse HEAD), UTC timestamp as integer nanoseconds, OS/architecture, and Python version. Confirm the working tree is clean first — if git status --porcelain is non-empty, stop; the SHA does not describe the code that ran.
2. Compute Per-File Data Checksums: Stream each input file to compute SHA256 individually. Never hash a concatenation of files: a single aggregate digest tells you something changed but not which file, which is exactly what an auditor asks.
3. Assemble Manifest: Organization, strategy ID, git SHA, per-file checksum map, all hyperparameters, performance metrics, execution metadata, data sources, notes. Any field left out of the signed payload is a field an attacker can change undetected — including the key ID.
4. Digest and Authenticate: Serialize as canonical JSON (sorted keys, no whitespace, allow_nan=False), then compute both the unkeyed digest and the keyed HMAC. Reject non-finite metrics rather than serializing them — Python writes bare NaN, which is invalid JSON and unparseable by strict verifiers in other languages.
5. Verify on Load: Check the HMAC, not just the digest. A manifest whose digest matches but whose HMAC does not has been re-digested by someone without the key — treat it as forged, not as corrupted.
6. Store & Archive: Write to storage whose retention controls match your regulatory status. If you are a registered broker-dealer, 17 CFR 240.17a-4(f) requires either WORM or an audit-trail system that can recreate a modified or deleted record. Determine the applicable retention tier from the record's classification under 17a-3 — do not assume six years applies (see references/standards.md).

Full procedure: see references/workflows.md. Standards reference: see references/standards.md. Printable pre-flight checklist: see assets/checklist.md.

Common Pitfalls

• Calling an Unkeyed Hash a Signature: A plain SHA256 embedded in the record it describes is a checksum. Recomputing it after editing takes three lines. Use a keyed MAC or an asymmetric signature.
• Testing Tamper Detection Without Re-Signing: Mutating a field and asserting verification fails only proves the digest covers that field. The real adversary recomputes the digest. A tamper test that never re-signs tests nothing about forgery.
• Storing the Signing Key With the Manifests: A key in the same bucket, repo, or backup set as the records it authenticates provides no protection against anyone who can reach the records.
• Untracked Code Modifications: Running a backtest against a dirty working tree — the SHA is meaningless for audit.
• Single Aggregate Checksum: Hashing concatenated data instead of per-file checksums masks which input changed.
• Non-Canonical JSON Serialization: Unsorted keys, incidental whitespace, or NaN/Infinity values break verification across implementations.
• Aliasing Caller State: Holding a reference to the caller's parameter dict rather than a copy means the "immutable" manifest silently changes after signing.
• Float Formatting Across Languages: Metrics stored as JSON floats may re-serialize differently in another runtime and break byte-level canonical comparison. Where cross-toolchain verification matters, encode metrics as decimal strings.
• Assuming a Retention Period: 17a-4 sets different tiers for different record classes and applies to registered broker-dealers. Asserting "six years" for a research artifact is unsupported.

Verification

• Run python -m unittest discover -s skills/backtest-audit-trail-for-regulatory-review/scripts — 100% pass rate (32 tests).
• Forgery test (the one that matters): modify a metric, recompute content_digest_sha256, then verify. The digest will match and verification must still fail on the HMAC.
• Confirm a manifest signed with a different key, or a different signing_key_id, fails verification.
• Confirm mutating the caller's parameter dict after build_manifest does not change the manifest.
• Confirm non-finite metrics are rejected at build time rather than serialized.

Related Skills

• backtest-determinism-and-reproducibility — ensures bit-identical runs; this skill records what was run
• paper-to-live-promotion-checklist — final gate before live capital; requires a valid audit trail
• lookahead-bias-elimination — must pass before an audit trail is meaningful
• walk-forward-validation-setup — validation methodology whose results belong in the manifest
• centralized-secrets-management-vault-integration — where the signing key should live
• record-retention-periods-by-jurisdiction — determining the applicable retention tier