Repository navigation
feat(ci): reserve self-hosted runners for Linux wheels - #6806
Open
zhanglistar wants to merge 15 commits into
Open
zhanglistar wants to merge 15 commits into
zhanglistar wants to merge 15 commits into
Conversation
…ools Use prepared self-hosted hosts for ordinary Linux jobs with isolated build environments, shared pool scheduling without inventory credentials, and hosted fallback when selection is unavailable.
4 of 8 tasks
Preserve the Ubuntu 20.04 wheel baseline while routing x64 wheels and shared Studio/source builds through the Linux pool selector.
Default the selector to GitHub-hosted and retain an explicit runner-label override for prepared self-hosted machines.
Register setup-python libraries with the container loader so Hermes can clear its environment without losing the shared Python runtime.
The hosted image supplied zip implicitly, but archive installation tests also need it in the isolated test container.
Install jq in Linux test containers and check archive and marketplace tooling before running the plugin suites.
Run independent tests and checks on hosted runners so they do not queue behind compilation on ov-ci-runner.
Select runners directly for heavy builds and keep hosted tests unchanged, removing the selector, pool configuration and unrelated container migration.
Limit self-hosted builds to trusted PRs and branch pushes, move ARM64 musl to hosted capacity, and provide execution limits plus an emergency hosted override.
Keep Studio, API native runtime and musl CLI builds hosted so they do not serialize behind the wheel build.
zhanglistar
force-pushed
the
feat/ci-linux-pools-official
branch
from
October 31, 2026 04:07
fade53a to
dbed5ef
Compare
Record platform timings and runner cost tradeoffs for later CI optimization.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Linux release wheels can serialize other builds on
ov-ci-runner. Reserve that capacity for official Linux x86_64 wheel builds, while Studio, API native preparation, Rust CLI builds and other checks use hosted runners.ov-ci-runner. Fork PRs, releases/tags, manual runs and schedules stay hosted.CI_DISABLE_SELF_HOSTED=truemoves new Linux wheel jobs to hosted capacity.dbed5ef34. Compiler-cache separation, component-specific Cargo target directories and Windows cache warmup experiments have been removed from this branch's history. Their historical timing measurements, including the intermediatefade53aaarevert, remain documented below.Validation
dbed5ef34: PR Checks passed all 17 jobs and API/CLI passed all four jobs.dbed5ef34PR Checks and API/CLI passed. The measurements below are historical runs on the indicated commits.macOS Intel history (job execution time, excluding queue time; all completed builds below passed):
Both cache-split runs used
4f61b3784, the same PR merge checkout, runner image20260824.0482.1, Rust1.99.1and compiler-cache keys. CLI, RAGFS and C++ caches all missed, as they did in the original baseline. Attempt 2 reproduced the slowdown. Separate Cargo targets duplicated some RAGFS build dependencies, but this alone does not explain the CLI slowdown. The rollback run provides a control using the earlier implementation under current conditions.The rollback CI checks out merge commit
29eb5db33, while the original checked outa9f158dd3; main advanced between runs. Git tree hashes confirm identicalcrates/,src/,third_party/, Cargo manifests/lockfile,setup.py,pyproject.toml, workflows and actions across these two checkouts.The complete rollback Intel log confirms the same runner image
20260824.0482.1, Rust1.99.1, original compiler fingerprints and cache misses as the original build. CLI compiled the same 329 entries; shared Cargo output restored the RAGFS compile count from 345 to the original 330. Despite this, Intel execution remained 44.9% above the original baseline, improving only 6.3% over split attempt 1 and 2.5% over attempt 2. Reverting the split does not remove most of the slowdown; the available evidence cannot attribute the full regression to that implementation.Other completed job execution times:
Windows compiler caches still missed in the rollback run, so its improvement is not evidence of a cache-hit benefit. API native preparation again restored Cargo/C++ caches, recovering the previous behavior. These timings describe the observed runs; they are not a guarantee of runner performance.
Same-commit rollback rerun
Attempt 2, initiated by the author, passed all 17 PR jobs on the same
fade53aaahead and29eb5db33merge checkout. The complete Intel log confirms runner image20260824.0482.1and the same original Cargo/C++ cache fingerprints, with both caches missing again.Intel improved by 57 seconds versus rollback attempt 1 but remained 11:56 (41.6%) above the original 28:40 baseline. Both rollback attempts reproduce the persistent slowdown; its root cause remains undetermined.
Fresh run after directly restoring dbed5ef
The fresh PR merge checkout is
afe5479ec, against a newer main, rather than the originala9f158dd3. Native source, third-party native dependencies, Cargo manifests/lockfile, setup.py and pyproject.toml tree hashes still match the original. Complete Intel logs confirm the same image, Rust version, original cache fingerprints and cache misses, and identical CLI/RAGFS crate/version sets. Intel remains 10:30 (36.6%) above the historical baseline even with the original branch head directly restored. Its root cause remains undetermined; queue time was at most six seconds for this PR run.