Skip to content

feat(ci): reserve self-hosted runners for Linux wheels - #6806

Open
zhanglistar wants to merge 15 commits into
mainfrom
feat/ci-linux-pools-official
Open

zhanglistar wants to merge 15 commits into
mainfrom
feat/ci-linux-pools-official

Conversation

@zhanglistar

@zhanglistar zhanglistar commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Linux release wheels can serialize other builds on ov-ci-runner. Reserve that capacity for official Linux x86_64 wheel builds, while Studio, API native preparation, Rust CLI builds and other checks use hosted runners.

  • Route only official branch pushes and same-repository PR Linux x86_64 wheels to ov-ci-runner. Fork PRs, releases/tags, manual runs and schedules stay hosted. CI_DISABLE_SELF_HOSTED=true moves new Linux wheel jobs to hosted capacity.
  • Preserve the Ubuntu 20.04/glibc 2.31 wheel baseline, native ARM runners and the 60-minute Linux wheel execution limit. Disposable build containers use job-local HOME/Cargo/Rustup directories and restore workspace ownership after the job.
  • Keep API/cuVS tests reusing the same-run stable-ABI native artifact.
  • The branch head is directly restored to dbed5ef34. Compiler-cache separation, component-specific Cargo target directories and Windows cache warmup experiments have been removed from this branch's history. Their historical timing measurements, including the intermediate fade53aaa revert, remain documented below.

Validation

  • Fresh CI after directly restoring dbed5ef34: PR Checks passed all 17 jobs and API/CLI passed all four jobs.
  • The original dbed5ef34 PR Checks and API/CLI passed. The measurements below are historical runs on the indicated commits.
  • 34 local packaging, release-selection and Linux runner-routing tests passed.
  • actionlint and whitespace checks passed.
  • Rollback API native preparation passed in 7:23, versus 7:51 originally and 15:33 with cache separation. Cargo and C++ caches again hit the existing namespaces.
  • Rollback PR CI passed all 17 jobs, including all five wheel platforms and their smoke tests.
  • Rollback API CI passed all four jobs, including API/CLI integration and cuVS.

macOS Intel history (job execution time, excluding queue time; all completed builds below passed):

Stage Original: dbed5ef Cache split: attempt 1 Cache split: attempt 2 Rollback: fade53a
Rust CLI step 6:22 12:42 13:50 11:58
Wheel build step 16:22 23:27 21:01 20:52
RAGFS compile, within wheel step 11:47 17:34 Not extracted 15:25
C++ configure/build, within wheel step 3:53 5:10 Not extracted 4:26
macOS smoke test 4:40 6:28 5:43 6:32
Entire Intel job 28:40 44:20 42:36 41:33

Both cache-split runs used 4f61b3784, the same PR merge checkout, runner image 20260824.0482.1, Rust 1.99.1 and compiler-cache keys. CLI, RAGFS and C++ caches all missed, as they did in the original baseline. Attempt 2 reproduced the slowdown. Separate Cargo targets duplicated some RAGFS build dependencies, but this alone does not explain the CLI slowdown. The rollback run provides a control using the earlier implementation under current conditions.

The rollback CI checks out merge commit 29eb5db33, while the original checked out a9f158dd3; main advanced between runs. Git tree hashes confirm identical crates/, src/, third_party/, Cargo manifests/lockfile, setup.py, pyproject.toml, workflows and actions across these two checkouts.

The complete rollback Intel log confirms the same runner image 20260824.0482.1, Rust 1.99.1, original compiler fingerprints and cache misses as the original build. CLI compiled the same 329 entries; shared Cargo output restored the RAGFS compile count from 345 to the original 330. Despite this, Intel execution remained 44.9% above the original baseline, improving only 6.3% over split attempt 1 and 2.5% over attempt 2. Reverting the split does not remove most of the slowdown; the available evidence cannot attribute the full regression to that implementation.

Other completed job execution times:

Job Original Cache split Rollback
Windows wheel 32:16 31:01 26:04
macOS ARM wheel 16:06 17:11 13:30
Linux x86_64 wheel 14:29 14:19 14:07
Linux ARM wheel 15:35 15:59 15:12
API native preparation 7:51 15:33 7:23

Windows compiler caches still missed in the rollback run, so its improvement is not evidence of a cache-hit benefit. API native preparation again restored Cargo/C++ caches, recovering the previous behavior. These timings describe the observed runs; they are not a guarantee of runner performance.

Same-commit rollback rerun

Attempt 2, initiated by the author, passed all 17 PR jobs on the same fade53aaa head and 29eb5db33 merge checkout. The complete Intel log confirms runner image 20260824.0482.1 and the same original Cargo/C++ cache fingerprints, with both caches missing again.

Job/stage Rollback attempt 1 Rollback attempt 2
Intel CLI step 11:58 10:26
Intel wheel step 20:52 21:37
Intel smoke step 6:32 6:16
Entire Intel job 41:33 40:36
Windows job 26:04 31:41
macOS ARM job 13:30 14:22
Linux x86_64 job 14:07 14:09
Linux ARM job 15:12 15:08

Intel improved by 57 seconds versus rollback attempt 1 but remained 11:56 (41.6%) above the original 28:40 baseline. Both rollback attempts reproduce the persistent slowdown; its root cause remains undetermined.

Fresh run after directly restoring dbed5ef

Job/stage Original dbed5ef run Fresh dbed5ef run
Linux x86_64 job 14:29 14:05
Linux ARM job 15:35 16:27
macOS ARM job 16:06 11:42
Windows job 32:16 25:24
Intel CLI step 6:22 11:17
Intel wheel step 16:22 19:59
Intel smoke step 4:40 5:41
Entire Intel job 28:40 39:10
API native preparation 7:51 6:56
API/CLI test job 24:43 24:11

The fresh PR merge checkout is afe5479ec, against a newer main, rather than the original a9f158dd3. Native source, third-party native dependencies, Cargo manifests/lockfile, setup.py and pyproject.toml tree hashes still match the original. Complete Intel logs confirm the same image, Rust version, original cache fingerprints and cache misses, and identical CLI/RAGFS crate/version sets. Intel remains 10:30 (36.6%) above the historical baseline even with the original branch head directly restored. Its root cause remains undetermined; queue time was at most six seconds for this PR run.

…ools

Use prepared self-hosted hosts for ordinary Linux jobs with isolated build environments, shared pool scheduling without inventory credentials, and hosted fallback when selection is unavailable.
Preserve the Ubuntu 20.04 wheel baseline while routing x64 wheels and shared Studio/source builds through the Linux pool selector.
Default the selector to GitHub-hosted and retain an explicit runner-label override for prepared self-hosted machines.
Register setup-python libraries with the container loader so Hermes can clear its environment without losing the shared Python runtime.
The hosted image supplied zip implicitly, but archive installation tests also need it in the isolated test container.
Install jq in Linux test containers and check archive and marketplace tooling before running the plugin suites.
@zhanglistar zhanglistar changed the title feat(ci): route Linux jobs across available compute pools feat(ci): route Linux jobs to ov-ci-runner or GitHub-hosted Oct 30, 2026
@zhanglistar zhanglistar changed the title feat(ci): route Linux jobs to ov-ci-runner or GitHub-hosted feat(ci): split Linux jobs 2:1 across self-hosted and hosted Oct 30, 2026
Run independent tests and checks on hosted runners so they do not queue behind compilation on ov-ci-runner.
@zhanglistar zhanglistar changed the title feat(ci): split Linux jobs 2:1 across self-hosted and hosted feat(ci): route heavy Linux builds to self-hosted capacity Oct 30, 2026
Select runners directly for heavy builds and keep hosted tests unchanged, removing the selector, pool configuration and unrelated container migration.
@zhanglistar zhanglistar changed the title feat(ci): route heavy Linux builds to self-hosted capacity feat(ci): assign heavy Linux builds to ov-ci-runner Oct 30, 2026
Limit self-hosted builds to trusted PRs and branch pushes, move ARM64 musl to hosted capacity, and provide execution limits plus an emergency hosted override.
Keep Studio, API native runtime and musl CLI builds hosted so they do not serialize behind the wheel build.
@zhanglistar zhanglistar changed the title feat(ci): assign heavy Linux builds to ov-ci-runner feat(ci): reserve ov-ci-runner for Linux x64 wheels Oct 30, 2026
@zhanglistar zhanglistar changed the title feat(ci): reserve ov-ci-runner for Linux x64 wheels feat(ci): isolate compiler caches and reserve Linux wheel runners Oct 30, 2026
@zhanglistar zhanglistar changed the title feat(ci): isolate compiler caches and reserve Linux wheel runners feat(ci): reserve self-hosted runners for Linux wheels Oct 31, 2026
@zhanglistar
zhanglistar force-pushed the feat/ci-linux-pools-official branch from fade53a to dbed5ef Compare October 31, 2026 04:07
Record platform timings and runner cost tradeoffs for later CI optimization.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

1 participant