Skip to content

fix(ci): raise test job timeout for instrumented main lane - #1625

Merged
mldangelo-oai merged 1 commit into
mainfrom
fix/main-ci-coverage-lane-timeout
Jun 10, 2026
Merged

mldangelo-oai merged 1 commit into
mainfrom
fix/main-ci-coverage-lane-timeout

Conversation

@mldangelo

Copy link
Copy Markdown
Member

Summary

  • raise the test job timeout from 35 to 75 minutes so the main-branch Python 3.12 coverage lane can complete

Root cause

The main-only "fast tests with coverage" step measures the suite under branch coverage, which costs ~3x the uninstrumented lanes (measured locally: 16:36 instrumented vs ~6 min uninstrumented; the uninstrumented CI siblings take 22-26 min). The overhead concentrates in CPU-bound detector replay tests (e.g. tests/detectors/test_jit_script_detector.py, 1,818 tests) that compile fresh code objects per iteration, which defeats both the C tracer and the sysmon core (benchmarked 3.0x vs 2.7x — switching cores does not recover the budget).

Since the detector suites grew, every push to main has had Test Python 3.12 killed at the 35-minute cap (most recently run 27274096829, cancelled at 35m13s while the sibling 3.10/3.11/3.13 lanes passed in 22-26 min). The failure was previously masked because earlier jobs in the workflow failed first.

Why not alternatives

  • COVERAGE_CORE=sysmon: benchmarked — only ~10% faster because the replay tests constantly execute fresh code objects; branch coverage also forces a fallback to the C tracer on Python ≤3.13.
  • Excluding the detector file from instrumentation: loses its coverage data and still leaves a ~50 min lane; not worth the fidelity loss.
  • The durable fix is reducing the intrinsic cost of the replay tests — left as a follow-up.

Validation

  • Local: full fast suite under branch coverage completes with the same failure set as uninstrumented (no coverage-induced failures), 996s locally → projected ~55-65 min on a 4-vCPU runner, within the new 75-minute cap.
  • Uninstrumented matrix versions are unaffected (they finish in ~25 min regardless of the cap).

🤖 Generated with Claude Code

The main-branch Python 3.12 step runs the fast suite under branch coverage,
which measures ~3x the cost of the uninstrumented lanes (~25 min each).
The tracer overhead concentrates in CPU-bound detector replay tests that
compile fresh code objects per iteration, so neither the sysmon core nor
line-only coverage recovers the budget. The 35-minute cap has cancelled the
lane on every main push since the detector suites grew; raise it to 75 so
the coverage lane completes while uninstrumented lanes remain ~25 min.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mldangelo-oai
mldangelo-oai merged commit 94e98ca into main Jun 10, 2026
25 checks passed
@mldangelo-oai
mldangelo-oai deleted the fix/main-ci-coverage-lane-timeout branch June 10, 2026 13:25
@github-actions github-actions Bot mentioned this pull request Jun 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants