Fix Pipenv matrix flake on 503/DNS transport blips

[Mikola Lysenko (mikolalysenko)]

Problem

Pipenv compatibility is the flakiest PR workflow by re-run count. Between 2026-09-18 and 2026-10-01, 10 runs went red on attempt 1 and green on re-run of the same SHA, covering 15 failed matrix legs (9 on the 8-release ubuntu direct leg, 3 on the macOS leg, 3 on the ubuntu crlf/marker-excluded/extras/category leg). Each re-run costs a full matrix leg (5–7 min each, on top of the cargo build) plus a human noticing and re-running it.

Examples of red-then-green runs:

• https://github.com/SocketDev/socket-patch/actions/runs/36699362953 (3 legs red on attempt 1)
• https://github.com/SocketDev/socket-patch/actions/runs/36549335640
• https://github.com/SocketDev/socket-patch/actions/runs/36472053681
• https://github.com/SocketDev/socket-patch/actions/runs/36428895598
• https://github.com/SocketDev/socket-patch/actions/runs/35632449754 (3 legs red on attempt 1)

Clustering the attempt-1 logs of all 15 legs:

Signature	Legs	Covered by this PR
A hosted case ERRORs with Max retries exceeded ... /patch/pypi/urllib3/1.26.18/... (Caused by ResponseError('too many 503 error responses')) (pip fetching the hosted patch wheel)	3 (one incident, run 36699362953)	yes
macOS hosted case ERRORs with NewConnectionError ... [Errno 8] nodename nor servname provided	1 (job 108949673510)	yes
A single hosted cell FAILs warmInstallReplacesUpstream plus one lock-only/dry-run check, with lock-only applied=0 and no exception text in the summary table	7	only if the CLI log in the case dir carries the request error or 5xx. Not confirmed from the summary tails
Old Pipenv (2020.11.15, 2021.11.23) resolver SystemExit: 1, root cause truncated	2	no
Agent / agent-oot rescanIdempotent / scanApplied	2	no

Example (job 109834889963):

| 2023.12.1 | direct | hosted | in-dir | ERROR | ...urllib3-1.26.18-py2.py3-none-any.whl (Caused by ResponseError('too many 503 error responses')) ERROR: Couldn't install package ...

Root cause (confirmed cluster)

scripts/backtest-pipenv.py installs real wheels, both from PyPI and from the public patch service's hosted wheel URLs. Unlike the Bun (retry_network_cell), PDM (NETWORK_RE + Run(retry=True)) and vlt (run_with_retries) harnesses, it has no transport retry. One 503 burst or DNS blip fails the case, so the leg exits 1.

The applied=0 cluster looks like the same outage reaching the CLI's hosted fetch, but the summary table hides the CLI's output. The case logs are not uploaded as an artifact; only summary.* is. The retry recorded here (transportRetries) will show in summary.json whether those rows were transport failures. The other two clusters are left for a follow-up.

Fix

retry_transport wraps each case in the scheduler. When a case raises or fails its checks and its error text or one of its *.log* files matches a narrow transport signature, it re-runs the case from a fresh case directory, with at most 3 attempts and a 10s/20s backoff. backtest_case already rmtrees the case dir, and the shared native lock is only cached after pipenv lock succeeds. Signatures: pip's too many 5xx error responses / Max retries exceeded with url / NewConnectionError / ConnectTimeoutError / ReadTimeoutError / DNS (Temporary failure in name resolution, macOS nodename nor servname provided) / connection reset, plus the CLI's own error sending request for url ( / API request failed with status 5xx (the same regex the Bun and vlt harnesses use).

• Functional failures (hash mismatch, resolution failures, wrong bytes) are never retried, and they still fail on the first attempt.
• A transport failure that persists through all 3 attempts still fails the leg.
• The logs from failed attempts are kept under attempts/<case>/<n>/, and the final row records transportRetries, so a retried pass shows up in summary.json.
• case_dir was factored out so the scheduler and the case runner agree on the path.

No assertions were weakened and no tests were removed or moved. The workflow and job names are unchanged.

Proof

• New unit tests (PipenvTransportRetryTests in scripts/tests/test_backtest_harnesses.py, run by the workflow's "Harness unit tests" step):
  • A PyPI 503 error is retried from a fresh case and its evidence is kept.
  • A failed check whose log shows a 503 is retried.
  • A patch API 5xx is retried.
  • A macOS DNS failure is retried.
  • Hash-mismatch and resolution failures are not retried.
  • A persistent 503 stays red after 3 attempts.
  • A passing case runs exactly once.
• python3 -B -m unittest scripts/tests/test_backtest_harnesses.py: 59 tests OK (re-run after merging main at acbce34).
• ruff check: same 4 errors as on main, none new.
• Real Pipenv matrix on this PR: all 3 legs green (https://github.com/SocketDev/socket-patch/actions/runs/36817010228), e.g. 16/16 macOS cases PASS.
• Cursor Bugbot: no issues (on head 3ef2997).
• CI: green on 3ef2997. main was merged in at acbce34 because Fix Poetry venv discovery to match Poetry (#327, #329) #330 changed python_crawler.rs venv discovery, which the Pipenv matrix exercises. CI is re-running on that head.

Where tests run

The tests are unchanged. The Pipenv matrix still runs on PRs that touch its paths and on every push to main.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TrR5uqMbj9sJcU2Gpm9y9R

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

[burn-down agent] Ready for review on 3ef2997: mergeable, 0 commits behind main.

• CI: 97/97 completed checks green on 3ef2997 (3 skipped by workflow conditions).
• Bugbot: reviewed 3ef2997, no new issues; 0 unresolved review threads.
• Reviewer note: Scripts/CI-only change (Pipenv backtest harness transport retry + its unit tests); no Rust code touched.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit acbce34. Configure here.}