Dependency Pinning And Reproducible Builds

When to Use

Use this skill on the requirements files behind quantitative research environments, backtesting frameworks, and live trading deployments. Floating dependency specifiers (pandas>=2.0, or a bare numpy) let a third-party release change floating-point behaviour, deprecate an API, or shift a calculation between two installs of the "same" code — which invalidates historical backtests and produces production model drift that looks like alpha decay. This module parses requirements lines, reports which are pinned to an exact version and carry a valid strong hash, and drafts a lockfile skeleton for remediation.

When NOT to Use

• As a resolver. It cannot tell you which version to pin, and it cannot discover transitive dependencies. Use pip-compile --generate-hashes or uv pip compile --generate-hashes for that; use this to audit the result.
• As proof that pip install --require-hashes will succeed. Hash-checking mode requires hashes for all dependencies and errors on any that is not spelled out. This tool sees only the lines you hand it, so it cannot confirm transitive completeness — a file can pass this audit and still fail the install.
• As a reproducible-build guarantee. Pinning plus hashes gives a reproducible install. A reproducible build (bit-for-bit identical artifacts) additionally requires the same build environment and instructions, plus handling of timestamps, locales, build paths and file ordering.
• On non-requirements formats. It does not parse poetry.lock, Pipfile.lock, or the PEP 751 pylock.toml standard lock format. Export to requirements format first.
• As a vulnerability scanner. A pinned, hashed dependency can still be a known-vulnerable one — see dependency-vulnerability-scanning-in-ci.

Prerequisites

• Requirements-file lines as a List[str] (requirements.txt, or a poetry export / pip freeze rendering).
• The target Python runtime version, recorded in the generated lockfile header for traceability.

Workflow

1. Line Reassembly:
   • Join backslash continuations before parsing. pip-compile --generate-hashes emits one requirement across several physical lines; parsing them independently reads the --hash lines as separate nameless packages and mis-audits a correctly locked file.
2. Directive Filtering:
   • Skip pip control directives (-r, -c, -e, --index-url, --require-hashes, ...). They are options, not requirements; counting them as packages produces false unpinned findings.
3. Specifier Classification — an exact pin is ==X.Y.Z, ===X.Y.Z, or a direct URL/path reference. Specifically not exact:
   • ==2.2.* — PEP 440 prefix matching, which accepts any 2.2.x release. This is the failure mode the audit exists to catch, and it looks like a pin.
   • ~=, >=, <, !=, or a bare package name.
   • Strip environment markers (; python_version >= "3.9") and extras ([security]) before reading the version, or the parsed version is corrupt.
4. Hash Validation — presence is not sufficient; the algorithm and the digest must both be checked:
   • Accept sha256 (recommended) and stronger. Reject md5, sha1, sha224 — pip excludes these from hash-checking mode "to avoid giving a false sense of security".
   • Verify the digest is hex of the algorithm's full length (64 characters for sha256). A short or non-hex digest is not a hash.
   • Retain every hash on a requirement: a package with wheels for several platforms legitimately carries several.
5. Proportional Scoring ($0.0$ to $100.0$):
   • $\text{Score} = 100 \times \left(w_{\text{pin}} \frac{N_{\text{pinned}}}{N} + w_{\text{hash}} \frac{N_{\text{hashed}}}{N}\right)$, defaults $w_{\text{pin}} = w_{\text{hash}} = 0.5$.
   • Proportional rather than a fixed penalty per package: hash-checking mode requires every transitive dependency to be listed, so production files routinely hold hundreds of entries and any absolute penalty saturates at zero.
   • The score is an internal engineering heuristic. It is not a standard — see references/standards.md.
6. Remediation Drafting:
   • Compliant requirements pass through verbatim. Deficient ones are emitted as commented # TODO(...) lines naming the command that resolves them. A version or a hash is never invented.

Full procedure: see references/workflows.md. Standards reference: see references/standards.md. Printable pre-flight checklist: see assets/checklist.md.

Common Pitfalls

• Never synthesise a hash to make an audit pass. A fabricated digest either fails --require-hashes outright or, if it were ever trusted, defeats the exact integrity check the hash exists to provide. Emit an unresolved TODO instead — a remediation tool that invents hashes is worse than no tool at all.
• ==2.2.* is not a pin. PEP 440 treats a trailing .* as prefix matching, so it silently accepts any 2.2.x. It passes a naive "==" in line check, which is why an auditor must parse the specifier rather than substring-match it.
• Presence of --hash= is not hash verification. --hash=md5:... is rejected by pip as too weak, and --hash=sha256:nothex is not a digest at all. Validate the algorithm and the digest, or the audit reports a false green.
• Pinning top-level packages only. scipy==1.12.0 with unpinned transitive dependencies is not locked; --require-hashes errors on the first dependency that is not spelled out.
• Auditing physical lines instead of logical requirements. Real hashed lockfiles use backslash continuations, so a line-by-line audit mis-reads the most common lockfile shape there is.
• Assuming pinned Python packages pin the whole environment. The C/C++ and CUDA libraries beneath the wheels (openblas, libgomp) are not covered by any --hash and can change numerical results across otherwise-identical installs. Pin the base image too.

Verification

• Instantiate ReproducibleBuildPinnerEngine(). Audit ["pandas>=2.0", "numpy", "ccxt==4.2.0 --hash=sha256:<64 hex digest>"]: expect total_packages_audited=3, unpinned_packages=['pandas', 'numpy'], missing_hashes_count=2, reproducibility_score=33.33, all_requirements_pinned_and_hashed=False.
• Confirm the generated lockfile contains no uncommented line for pandas or numpy, and that no fabricated digest appears anywhere in it.
• Audit a real pip-compile --generate-hashes block (numpy==1.26.4 \ followed by two indented --hash=sha256: lines): expect total_packages_audited=1 and reproducibility_score=100.0, not three packages.
• Confirm --hash=md5:<digest> yields missing_hashes_count=1 and all_requirements_pinned_and_hashed=False.
• Confirm pandas==2.2.* is reported in unpinned_packages.
• Run python -m unittest discover -s skills/dependency-pinning-and-reproducible-builds/scripts.

Related Skills

• research-environment-vs-production-environment-parity
• execution-algorithm-regression-testing-suite
• dependency-vulnerability-scanning-in-ci
• backtest-determinism-and-reproducibility
• immutable-infrastructure-for-trading-bots