Skip to content
ci

record(ENG-UPSTREAM-LTX2-PIN): the LTX-2.5 lane's own reference gets a pin, and the run it has never had (#1433) #5111

Sign in to view logs

record(ENG-UPSTREAM-LTX2-PIN): the LTX-2.5 lane's own reference gets a pin, and the run it has never had (#1433)

record(ENG-UPSTREAM-LTX2-PIN): the LTX-2.5 lane's own reference gets a pin, and the run it has never had (#1433) #5111

Workflow file for this run

.github/workflows/ci.yml at e74b017
name: ci
on:
push:
branches: [main]
# `closed` so a merged or abandoned pull request ENTERS the concurrency
# group and supersedes its own in-flight run (#822). Every job below is
# skipped on it -- directly through its own `if:`, or transitively by
# `needs:`-ing a job that is -- except the two `windows-msvc-*` PR proofs,
# whose whole job mapping is pinned byte-for-byte by
# `scripts/check-release-workflow.py` and can carry neither key. Those two
# therefore still start on a close; recorded as owed in
# .agents/specs/ci-concurrency.md and tracked by #874.
pull_request:
types: [opened, synchronize, reopened, closed]
# THE BASELINE LANE (issue #274, spec .agents/specs/main-verifiability.md).
#
# The push lane below cannot answer "is main green?", by construction. Every
# expensive job carries a job-level group keyed on `github.ref`, which is the
# constant `refs/heads/main` for every push, so consecutive pushes cancel each
# other's long jobs. Measured over 40 consecutive main runs at 0eb049f7: 26
# cancelled, 12 failure, 1 success. The cancel instant equals the next push's
# start instant (run 31485402200 died 11:46:33, run 31488132224 started
# 11:46:32; 31482845117 died 11:05:12, 31485054749 started 11:05:11). At 55
# pushes/day and a 99-minute suite, that is structural, not bad luck.
#
# This lane runs the SAME jobs on a cadence, in its own concurrency groups so
# a push cannot cancel it, and publishes a verdict via `baseline-summary`.
# Every 4h: at 55 pushes/day a red baseline then names ~9 commits (three
# bisect steps), and successive runs still cannot overlap a 99-minute suite.
# Making every MERGE non-cancellable was rejected: ~55 overlapping runs/day,
# ~190 h/day of job time on a pool where run 31485402200 already sat queued 37
# minutes without starting a single job -- and at that rate the newest
# complete baseline would be several commits stale anyway.
schedule:
- cron: '17 */4 * * *'
# The hybrid half: pin a baseline on a SHA you care about right after merging
# it, without paying for a full run on all 55 of the day's pushes.
# gh workflow run ci.yml --ref main
workflow_dispatch:
# Workflow-level: dedupe PR pushes only. The group is keyed on the SHA for a
# push, so two pushes to main never share a group and never cancel each other.
# That is deliberate: `documentation-checkpoint` and `commit-protocol-tag` below
# are DIFF-scoped over `github.event.before..github.sha`, so a cancelled main run
# leaves its own commit range permanently unvalidated (no later run re-covers it,
# because the next run's `before` is this run's `sha`).
#
# The expensive TREE-scoped jobs carry their own job-level group instead, which
# does collapse superseded main pushes. Net effect: a superseded push to main
# keeps only its 2 per-commit gates and drops the other 6 jobs.
#
# 2026-08-04: those two jobs DID carry `ci-doc-`/`ci-commit-` groups keyed on
# `github.ref` with `cancel-in-progress: true`, directly contradicting their own
# comments and this one. It was not theoretical: two consecutive pushes to main
# had their documentation-checkpoint and commit-protocol-tag cancelled by the
# next push, so commits cdec2d10 and 0435746d's ranges went unvalidated by the
# gates written to cover them. The groups are removed; these two jobs must never
# regain one. They are a checkout plus a Python script, so the cost of always
# running them is negligible against a silently skipped gate.
#
# 2026-08-11 (#274): `github.event_name` joins the key. `cancel-in-progress` is
# already false for every non-PR event, so this cancels nothing new -- it stops
# a `workflow_dispatch` baseline fired on a SHA that was just pushed from
# sharing group `ci-<sha>-<repo>` with that push's run and QUEUEING behind it.
# For `push` and `pull_request` the added token is a constant, so their
# partition into groups is exactly what it was.
# 2026-08-15 (#822): the push lane becomes LATEST-ONLY. The group keys on
# `github.ref` for a push rather than `github.sha`, so consecutive pushes to main
# share a group and the older one is cancelled.
#
# This was previously unsafe and is now safe. It was unsafe because a cancelled
# run's diff-scoped range was never re-covered: the next run's `before` is this
# run's `sha`. The gates now walk from the last SUCCESSFULLY gated commit
# (`last-gated-commit`), so a cancelled run is lossless -- the next run walks a
# wider range and reports the same red. Reverting that base to `before` without
# reverting this reintroduces #863.
#
# `schedule` and `workflow_dispatch` keep their own partition through the
# event_name token and stay non-cancellable, so the baseline lane (#274) is
# untouched.
#
# 2026-08-19 (#274): THE BASELINE EVENTS ADMIT `github.run_id` INTO THE KEY, and
# every job-level group below takes the same conditional.
#
# `cancel-in-progress: false` is only half of GitHub's concurrency contract. The
# other half is the queue, and it holds exactly ONE pending run: a third run
# joining a group that has one run in progress and one pending CANCELS the
# pending one. Non-cancellable therefore never meant "cannot be cancelled".
#
# It cost two of the last 39 scheduled baselines, and neither executed a single
# job. Runs 32140419182 and 32206456661 return `startedAt: null` for EVERY job
# and died at 16:46:51 and 04:49:54 -- the seconds their successors 32162114781
# and 32217173498 were created, 2 out of 2. `cancel-in-progress` resolves false
# for `schedule`, so it cancelled nothing; the shared key
# `ci-schedule-refs/heads/main-<repo>` did.
#
# Queue depth two is reachable because the suite outgrew the cron. Run
# 32118587477 took 9 h 28 min with NO predecessor to wait for, on 345
# job-minutes of work: `agent-record` is a 3.8-minute ubuntu job and waited 3 h
# 59 min for a runner, `baseline-summary` is a 0.2-minute job and waited 4 h 33
# min. The four-hour cron laps that, and 8 of 39 runs ran over four hours.
#
# A queue that discards its own contents costs a verdict and reclaims nothing,
# because a discarded run never ran anything to reclaim. So a group that never
# cancels must never queue: `github.run_id` enters for `schedule` and
# `workflow_dispatch` ONLY. For `push` the conditional resolves to `github.ref`
# and for `pull_request` to the pull request number, so both keys are what they
# were and #822's latest-only push lane is untouched.
#
# Widening the cron was REJECTED as the fix and left as a separate decision: it
# is probabilistic (the wall is set by pool contention this file does not
# control), it spends the ~9-commit red range the 4h cadence was chosen for, and
# the contention it answers is one day old. See
# .agents/specs/baseline-lane-eviction.md.
#
# `tests/scripts/test_main_baseline.py::GroupEvictionTests` holds the invariant
# over every concurrency block in this file, derived from `cancel-in-progress`
# rather than from a list, so a job that joins later is covered by the pull
# request that adds it: never-cancels implies varies-per-run, and cancels
# implies shared. Both directions, because a key that varies per run cancels
# nothing.
concurrency:
group: ci-${{ github.event_name }}-${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && github.run_id || github.event.pull_request.number || github.ref }}-${{ github.repository }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || github.event_name == 'push' }}
jobs:
agent-record:
# Tree-scoped: validates the record as it stands at HEAD, so only the newest
# push to a ref is meaningful. Cancellable.
concurrency:
group: ci-agent-record-${{ github.event_name }}-${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && github.run_id || github.ref }}-${{ github.repository }}
cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
# NO job-level `if:`, deliberately, and no `always()`.
#
# `check-release-binary-contract.py` and `check-test-registration.py` count
# a checker as wired into CI only when the job owning its step carries no
# `if:` at all: a registration behind a condition is not a registration.
# #865 gave this job one, and both checkers plus three of their suites went
# RED on `main` (#873).
#
# The closed-pull-request skip #822 needs is carried by `needs:` instead.
# `last-gated-commit` excludes the closed action and runs on every other
# lane, so a closed pull request skips it and skips this job with it, while
# push, pull_request and the baseline lane all still run it. That is also
# why `always()` must NOT come back here: it would defeat the skip.
#
# The asymmetry that buys is deliberate but real: without `always()` the
# implicit `success()` couples this job to `last-gated-commit` SUCCEEDING,
# not merely running, while `documentation-checkpoint` and
# `commit-protocol-tag` on the same `needs:` keep `always()` and run
# regardless. A failure in the sha resolver therefore silences this job's
# checkers rather than reding them -- which is why that job does nothing but
# resolve a string and falls back instead of failing.
needs: [last-gated-commit]
runs-on: ubuntu-latest
env:
# The SAME pin as `.github/workflows/gh-pages.yml`, which publishes the
# site this job's suite renders. Two files carrying one version is a drift
# risk, so `tests/scripts/test_ci_site_lane.py` holds them equal rather
# than trusting whoever bumps one to remember the other. Read that file's
# comment before changing this value: 0.153 deprecates `excludeFiles`,
# which `website/hugo.toml` uses.
HUGO_VERSION: "0.146.3"
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
# The docs-site suite RENDERS the site, so this lane needs the renderer.
# Without it `test_rendered_benchmark_index_links_resolve_to_emitted_pages`
# cannot run at all -- and neither of its two ways of not running is a
# verdict on this repository: unguarded it raises `FileNotFoundError` and
# reds the job (#1722, #1754), guarded it skips and the job goes green
# having rendered nothing. Installing the binary is what makes the
# assertion execute; the guard only decides which of the two non-verdicts
# an absent binary produces.
- uses: peaceiris/actions-hugo@v3
with:
hugo-version: ${{ env.HUGO_VERSION }}
extended: true
- name: Canonical roadmap tables and links are consistent
# --report, wired the way check-device-leakage's is below: the record
# anchor ratchet (ENG-RECORD-ANCHOR-RATCHET, #632) gates on a baseline,
# and printing every STALE/BROKEN offender on every run is what keeps
# the backlog legible instead of a number nobody can act on. It is the
# same gate -- --report adds output, never leniency.
run: |
python3 scripts/check-agent-record.py --report
python3 tests/scripts/test_agent_record.py
python3 tests/scripts/test_check_issue_index_append_only.py
- name: Accepted binary-release design and record anchors stay in sync
run: |
python3 scripts/check-release-binary-contract.py
python3 tests/scripts/test_check_release_binary_contract.py
- name: Binary release manifests are deterministic and fail closed
run: |
python3 tests/scripts/test_release_manifest.py
- name: Extracted release archives fail closed on content and metadata drift
run: |
python3 tests/scripts/test_release_archive.py
- name: Release workflow is dry-run safe and least privilege
run: |
python3 scripts/check-release-workflow.py
python3 scripts/check-windows-release-state.py
python3 scripts/check-container-matrix.py
python3 scripts/check-container-workflow.py
python3 scripts/check-build-runtime-deps.py
python3 tests/scripts/test_release_pipeline.py
python3 tests/scripts/test_check_windows_release_state.py
python3 tests/scripts/test_release_postpublish_audit.py
python3 tests/scripts/test_check_container_matrix.py
python3 tests/scripts/test_check_container_workflow.py
python3 tests/scripts/test_check_build_runtime_deps.py
python3 tests/scripts/test_validate_container_image.py
python3 tests/scripts/test_release_index.py
- name: CPU release metadata and tier execution gates fail closed
run: |
python3 tests/scripts/test_release_metadata.py
python3 tests/scripts/test_release_accelerator_metadata.py
python3 tests/scripts/test_release_macos_metadata.py
python3 tests/scripts/test_release_windows_metadata.py
python3 tests/scripts/test_cpu_release_gates.py
- name: README stays a human-readable user-facing document
run: |
python3 scripts/check-readme-structure.py
python3 tests/scripts/test_check_readme_structure.py
- name: Quickstart recipes name a published tag and a parseable model
run: |
python3 scripts/check-quickstart-recipes.py
python3 tests/scripts/test_check_quickstart_recipes.py
- name: Benchmark index links each ID to one detail file
run: |
python3 scripts/check-benchmark-index.py
python3 tests/scripts/test_check_benchmark_index.py
- name: Architecture-support checklist matches the row states
run: |
python3 scripts/check-model-checklist.py
python3 tests/scripts/test_check_model_checklist.py
- name: Public supported-model list matches the C++ registry
run: |
python3 scripts/check-supported-models.py
python3 tests/scripts/test_check_supported_models.py
- name: Every production env var is documented or classified
run: |
python3 scripts/check-env-doc.py
python3 tests/scripts/test_check_env_doc.py
- name: The gate bring-up script resolves its values or refuses
run: |
python3 tests/scripts/test_gate_bringup.py
- name: scripts/ and tools/ resolve their values, refuse, and keep records
# Registered in TWO places, like the bring-up suite above: here and in
# the `SUITES` array of `scripts/agent-preflight.sh`. Half of it guards
# the SECOND class of the #1190 rule -- the dated measurements that must
# STAY literal -- and that half is what a later blind sweep of the tree
# would trip, so it has to run on a lane nobody has to remember.
run: |
python3 tests/scripts/test_env_agnostic_tooling.py
- name: The docs site's content invariants hold
# `hugo version` first, deliberately. It is the one line that fails
# LEGIBLY when the setup step above stopped providing a binary, and it
# fails BEFORE the suite gets the chance to report that absence as
# either a red it did not cause or a skip nobody reads.
run: |
hugo version
python3 scripts/check-site.py
python3 tests/scripts/test_check_site.py
- name: Model add+RMSNorm glue routes through the fusion catalog
run: |
python3 scripts/check-fusion-consistency.py
python3 tests/scripts/test_check_fusion_consistency.py
- name: A model on the naive attention kernel says why (#1544)
run: |
python3 scripts/check-attention-rung-consistency.py
python3 tests/scripts/test_check_attention_rung_consistency.py
- name: The lane that renders the docs site actually provisions Hugo
run: |
python3 tests/scripts/test_ci_site_lane.py
- name: Structural checkers ignore text the compiler never sees
run: |
python3 tests/scripts/test_checker_text.py
- name: Every fp4 resident upload keeps its post-upload residency step
run: |
python3 scripts/check-fp4-resident-consistency.py
python3 tests/scripts/test_check_fp4_resident_consistency.py
- name: An arch-independent CUDA op is registered from an unconditional TU
run: |
python3 scripts/check-cuda-op-arch-gate.py --report
python3 tests/scripts/test_check_cuda_op_arch_gate.py
- name: Model decode is born on the runner (device-resident logits)
run: |
python3 scripts/check-runner-routing-consistency.py
python3 tests/scripts/test_check_runner_routing_consistency.py
- name: Windows source contract, and no `[[noreturn]]` on a non-void return
# THE CHECKER RAN ON NO LANE AT ALL until now: absent from this file, from
# `scripts/agent-preflight.sh` and from every hook (#646, #680). #1829 is
# what that costs -- `[[noreturn]] static ForwardLogits ForwardDevice(` is
# MSVC C4646, `/W4 /WX` makes it C2220, and `main` failed to COMPILE under
# MSVC while every POSIX lane stayed green, because `windows-msvc-cpu` and
# `windows-msvc-vulkan` are pull-request-only and give `main` no verdict.
#
# `shipped_server_sources` derives its set from the CMake file API and
# forces `-G Ninja`. `ubuntu-latest` is not assumed to carry Ninja -- every
# other job in this file installs `ninja-build` explicitly -- so it is
# installed here, and both tool versions print first: a missing tool must
# read as a missing tool, not as a mystery configure failure.
#
# Cost, read off the runner class that pays it rather than estimated
# twice: 17.4 s for the checker and 25.2 s for the suite on
# `ubuntu-latest`, from this job in run 32691600796, log stamps
# 05:34:01 -> 05:34:18 -> 05:34:43. The same two commands on a 20-core
# Ryzen 9 9950X3D workstation are 6.2 s and 18.0 s, median of five at
# load average 6-8 -- and 24.8 s for the suite at load average 11. That
# spread is why this line and the spec once carried 24.3 s and 18.1 s
# for one quantity: both were the workstation under different loads,
# and neither said which machine it was.
run: |
cmake --version
sudo apt-get update
sudo apt-get install -y ninja-build
ninja --version
python3 scripts/check-windows-portability.py
python3 -m unittest tests.scripts.test_check_windows_portability
- name: Examples are thin clients of the public ABI, capabilities reach it
run: |
python3 scripts/check-surface-coverage.py
python3 tests/scripts/test_check_surface_coverage.py
- name: No harness launches the pinned oracle on a handicapped path (#414)
# Without `--language-model-only` a Qwen3.6 checkpoint gives the oracle
# `text_only == False` (qwen3_next.py:324-331 at the pin), so it runs its
# UNFUSED QK-norm+RoPE+gate path against our fused one and every ratio it
# produces is FLATTERED. The 2026-08-13 series repaired the RUN; this
# repairs the HARNESS, which is what the next campaign reaches for.
run: |
python3 scripts/check-oracle-denominator-flags.py
python3 tests/scripts/test_check_oracle_denominator_flags.py
- name: The tower-skip RSS reporter decides correctly on fabricated runs
# `scripts/mm/tower_skip_rss.sh` needs a 56 G or an 8.3 G checkpoint, two
# Release builds and a quiet box, so CI cannot run it -- but its REPORTER
# reads four `.time` files and four logs and needs none of that. This
# pins MET/FAILING/every VOID for both declared model kinds, and the
# property the L3 review found missing: the swapped second pair reaches
# the verdict, so a binary-shaped bias cannot pass as a saving.
run: |
python3 tests/scripts/test_tower_skip_rss_report.py
- name: The tower-skip RSS harness runs a leg it can attribute to itself
# The other half of the same harness, and the half #1844 was in: the
# readiness poll answered from the PREVIOUS leg's server, so every
# measured leg of the first real run was killed mid-load and all five
# `.time` files came back 0 bytes while the reporter suite was 60/60
# green. A fake server on a scratch port exercises the occupied-port
# refusal, the leg's own banner, the SIGTERM that reaches the server
# rather than `/usr/bin/time`, and both halves restored as mutations.
run: |
python3 tests/scripts/test_tower_skip_rss_arm.py
- name: The x86_64 CPU floor harness runs from a clean checkout
# It shipped unable to run at all: `OUT=evi` with no `mkdir -p`, so every
# redirection failed, every leg was discarded for a non-zero exit, and
# the series died blaming contention it had never reached. Stub engines,
# so this needs no model, no GPU and no quiet box.
run: |
python3 tests/scripts/test_cpu_x86_llamacpp_floor.py
- name: The IndexTTS-2.5 port still matches the shipped checkpoint (#634)
# These three read committed copies of the checkpoint's own config and
# safetensors header, so they need no network, no weights and no GPU.
# They landed in #727 and earlier WITHOUT a CI line, which made them
# run-if-you-remember rather than gates; wiring them is the point.
run: |
python3 tests/scripts/test_indextts2_config_contract.py
python3 tests/scripts/test_indextts2_emotion_arch_covered.py
python3 tests/scripts/test_indextts2_pth_manifest.py
python3 tests/scripts/test_indextts2_convert.py
python3 tests/scripts/test_indextts2_reference_path.py
python3 tests/scripts/test_vocoder1d_single_home.py
- name: Critical regression tests remain executable and CTest-registered
run: |
python3 scripts/check-test-registration.py
python3 tests/scripts/test_check_test_registration.py
- name: No gate resolves its checkpoint by readdir order (#471)
# Registered HERE and not only in `scripts/agent-preflight.sh`: a checker
# reachable through a hook gates whoever chooses to run the hook, and
# AGENTS.md is explicit that hooks are bypassable convenience, never
# proof. The `--self-test` leg sweeps the checker's own fixture corpus in
# both directions, so a narrowed pattern fails here rather than passing
# quietly on a tree it no longer inspects.
run: |
python3 scripts/check-snapshot-pins.py
python3 scripts/check-snapshot-pins.py --self-test
python3 tests/scripts/test_check_snapshot_pins.py
- name: Every oracle is named, pinned, and honest about gateability (#647)
# The fallback rule (AGENTS.md "When vLLM has no implementation") admits
# a secondary oracle only at a recorded pin. Ungated, the registry decays
# the way the scattered pins it replaced already had: an upstream gets
# compared against at whatever revision happened to be checked out, and
# the pin that does exist lands in one spec where no other spec can see
# it. `--self-test` sweeps the fixture corpus in both directions.
run: |
python3 scripts/check-oracle-pins.py
python3 scripts/check-oracle-pins.py --self-test
python3 tests/scripts/test_check_oracle_pins.py
- name: The SGLang lease identity gate detects a tree that is not the pin (#1265)
# `sglang/_version.py` in the PyPI wheel sets `__commit_id__ = None`, so
# the installed package carries NO runtime assertion of the commit it was
# built from and `.agents/specs/sglang-wheel-in-lease.json` is that
# assertion instead. The gate itself runs inside an `rc` lease, which CI
# has none of; what CI holds is that it goes RED on each defect it
# promises to catch, and that the committed manifest agrees with the pin
# block it stands in for. It is also the only executing code that opens
# that manifest, so this step is what stops it landing unread.
run: |
python3 tests/scripts/test_sglang_lease_identity.py
- name: The Nemotron oracle golden says whether it can be regenerated (#926)
# A golden whose provenance is one person's home directory on one box is
# not a pin. `af8170154` committed this reference with no generator and
# no engine configuration, and the host was reimaged two days later, so
# nobody can re-derive it or say what produced it -- while a 95/96 is
# being scored against it. The contract does not demand the lost
# configuration back. It demands that the file state which of the two it
# is, so silence stops being an option. Runs with no vLLM, no GPU and no
# checkpoint, deliberately: the defect is a records defect and its gate
# must not need the hardware whose absence caused it.
run: |
python3 scripts/nemotron-h-oracle-capture.py --check \
tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json
python3 tests/scripts/test_nemotron_h_oracle_capture.py
- name: cuBLASLt GEMM invocation stays dtype-faithful (template parity)
run: |
python3 scripts/check-gemv-invocation-consistency.py
python3 tests/scripts/test_check_gemv_invocation_consistency.py
- name: The main baseline lane and its reader stay honest
# Registered in TWO places, deliberately: here and in the `SUITES` array
# of `scripts/agent-preflight.sh`. It shipped in neither, so the 24 tests
# that guard the baseline lane ran on no machine -- the same class of
# defect as an unregistered CTest target, one layer up. #408 tracks
# whether a checker should catch the class; this closes the instance.
run: |
python3 tests/scripts/test_main_baseline.py
- name: Preflight never reports green over a block that did not run
# Registered in TWO places, like the baseline suite above: here and in
# the `SUITES` array of `scripts/agent-preflight.sh`. This lane is the
# one that matters more of the two, because the defect it pins (#998)
# is preflight lying about ITSELF, and a suite that only ever runs from
# the script it audits is checked by exactly the run it cannot trust.
run: |
python3 tests/scripts/test_agent_preflight_skip_report.py
- name: Protocol prose matches the checkers that enforce it
# PART tree-scoped, PART diff-scoped, and the two must be told apart on
# the baseline lane (#274). `documentation-checkpoint` and
# `commit-protocol-tag` opt OUT of `schedule`/`workflow_dispatch`
# wholesale because they are diff-scoped end to end; this job cannot,
# because most of it is tree-scoped and the baseline needs it. So the
# RANGE-SCOPED call is guarded here instead of the whole job.
#
# Without that guard this step DID kill the lane: a `schedule` payload
# has no `github.event.before`, `PUSH_BASE` renders empty, and under
# `set -eu` the walk aborts with
# `--range: range must be exactly BASE..HEAD` (exit 2)
# so `agent-record` could never be green and `baseline-summary`, which
# `needs:` it, could never publish GREEN. Reproduced by replaying this
# body with EVENT_NAME=schedule, PUSH_BASE="".
env:
EVENT_NAME: ${{ github.event_name }}
run: |
set -eu
python3 scripts/check-prompt-contract.py
python3 -m unittest \
tests.scripts.test_check_commit_trailers
python3 tests/scripts/test_check_prompt_contract.py
python3 tests/scripts/test_agent_gates.py
python3 tests/scripts/test_agent_pr_body.py
python3 tests/scripts/test_ab_arms_differ.py
- name: The LTX-2.5 pixel comparison and its harness run on a lane (#1612)
# Registered in TWO places, like the baseline and preflight suites above:
# here and in `scripts/agent-preflight.sh`. Both suites ran on NO lane
# when they landed -- absent from that array, from this file and from
# `tests/CMakeLists.txt` -- while section 8 of
# `.agents/specs/ltx25-dit-attn-flash.md` registered the first as a gate.
#
# numpy is installed HERE and only here. `ltx25-render-compare.py` reads
# PPM and WAV by hand so that a leased GPU worker needs no image or audio
# library, and numpy is the one dependency that remains. Preflight SKIPS
# that suite when numpy is absent, which is honest on a developer box and
# would be silence on a lane, so this lane installs it instead.
#
# The harness suite needs neither numpy nor a GPU: it extracts the shell
# helpers from `ltx25-dit-attn-flash-pixel-ab.sh` between its marker
# comments and runs them against a fabricated /proc/meminfo.
#
# `test_ltx25_ab_memwatch.py` (#1734) needs neither either, and it spans
# all THREE A/B harnesses rather than one: it extracts their shared
# `memwatch-helpers` block, asserts the three copies are byte-identical,
# and runs each harness's own poll and report lines over a fabricated
# /proc/meminfo. The defect it pins printed an EMPTY `memavail low-water`
# -- a missing measurement in the shape of a reported one -- in two of
# the three, so a gate over one harness would not have seen it.
run: |
sudo apt-get update -qq
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends python3-numpy
python3 tests/scripts/test_ltx25_render_compare.py
python3 tests/scripts/test_ltx25_pixel_ab_harness.py
- name: Every LTX-2.5 A/B arm sets a knob value the dispatch parses (#1751)
# The `VLLM_LTX2_DIT_FLASH_ATTN` values a harness exports and the values
# `ltx2_device.cpp` matches are two literals in two languages, and nothing
# connected them. #1551 renamed the flash rung and two #1549-era harnesses
# kept the old spelling, so each ran an arm it did not name. The suite
# reads the accepted set out of the dispatch's own `strcmp` calls rather
# than restating it, and needs no GPU, lease or toolchain.
run: |
python3 tests/scripts/test_ltx2_dit_attn_knob_arms.py
python3 tests/scripts/test_ltx25_ab_memwatch.py
- name: No tracked file carries a merge conflict marker (#1417)
# Registered HERE and not only in preflight. The four record gates that
# missed this all return 0 on a `docs/STATUS.md` carrying a spliced
# conflict hunk, because each measures its own budget and none asks
# whether the file is well-formed. A person caught the real instance by
# reading `git diff --stat`.
#
# The bare `=======` separator is refused ONLY inside an open hunk. Five
# shipped files carry a rule of eight or more `=` characters, so an
# unconditional rule would have been red on arrival, and a gate that
# fires on ordinary work is the defect rather than the discipline.
run: |
python3 scripts/check-conflict-markers.py
python3 tests/scripts/test_check_conflict_markers.py
- name: The pre-push hook refuses a checker it names but cannot find
# #1779. The hook named six checkers, three of which had been deleted,
# and skipped each missing one in silence while still exiting 0. The
# suite executes the hook against a scratch repository, so what it pins
# is the loop's control flow rather than the text of the list.
run: |
python3 tests/scripts/test_prepush_checker_names.py
- name: The benchmark-tool suites run in THIS lane too (#1646, #1648)
# CORRECTION. #1646 landed claiming `tests/tools/` had "no workflow, no
# CTest registration and no preflight line" running it. THE CTEST
# REGISTRATION EXISTS: `tests/CMakeLists.txt` registers
# `test_serve_low_tools`, which runs exactly this discover command with
# PYTHONPATH set, and has since `e58858a91`; `build-test-cpu` runs
# `ctest --test-dir build` on every pull request, so these suites were
# already gating there. The error came from grepping `tests.tools`, the
# dotted module path, and concluding absence -- CMake spells it
# `tests/tools`. A null grep proves the terms wrong, never that the
# thing is absent. #1648 carries the correction; the #1646 index row is
# append-only and stands as written.
#
# WHAT IS STILL WORTH HAVING. This step runs the suites WITHOUT a C++
# build, so a Python-only change gets the verdict in ~21 s instead of
# behind a full configure-and-compile, and it fails on its own name
# rather than inside a ctest summary. The preflight line beside it
# closes the one gap that was real: preflight genuinely ran none of
# these suites.
#
# DISCOVERED, not enumerated: an enumeration is a shared file every new
# suite must edit, which is the record-lock shape AGENTS.md §Records
# forbids. Standard library only, no GPU, no vLLM wheel.
run: |
python3 -m unittest discover -s tests/tools -t . -p "test_*.py"
- name: A citation that names a symbol still finds it (#1143, #1139)
# The tree already carried 539 `path::Symbol` citations and had never
# checked one. Line anchors are what rot: one 45-line insertion near the
# top of `model_loader.cpp` moved 203 references at once, in files that
# change never opened (#1143), and a pin advance left three upstream
# line anchors on one row pointing at unrelated code (#1139).
#
# NOT the #911 shape. The expectation is the symbol name written by the
# citing author and read from the CITING file; only the evidence comes
# from the cited one. `--self-test` sweeps the fixture corpus in both
# directions. The upstream half needs an oracle checkout this runner
# does not have, so it stays opt-in behind `--upstream-root`.
run: |
python3 scripts/check-symbol-anchors.py
python3 scripts/check-symbol-anchors.py --self-test
python3 tests/scripts/test_check_symbol_anchors.py
- name: NOW.md stays a short one-Read resume surface
run: |
python3 scripts/check-now-current.py
python3 tests/scripts/test_check_now_current.py
- name: A gated row may never lose a gate command that can FAIL
# An EXACT PIN over the SET of rows whose spec names a runnable command
# -- not a count, and not shrink-only. Not a count, because a count
# cannot tell a row that LOST its command from one that legitimately
# left the gated population, and the fix for the second reads as
# lowering the number for the first. Not shrink-only, because the unit
# test asserts the set EQUALS RUNNABLE_BASELINE, so GROWTH reds this job
# too: any movement, up or down, re-pins the baseline in the same
# change, naming the rows and the reason.
run: |
python3 scripts/check-gate-commands.py --check
python3 tests/scripts/test_check_gate_commands.py
- name: The GPU mutex has exactly one truth (#777)
# Registered HERE and not only in preflight: a `flock` on the wrong file
# SUCCEEDS, so a second mutex is silent at runtime and surfaces only as
# timing noise attributed to whoever else was on the box. Nothing at run
# time can catch it; this is the only place it can be caught.
run: python3 tests/scripts/test_gpu_lock_one_truth.py
- name: Agent role machinery and role discipline
# Same split as the step above: the two suites are tree-scoped and run
# on every lane; `check-role-discipline.py --base/--head` is DIFF-scoped
# and is skipped where there is no range.
#
# It would not have aborted on its own -- passing `--base ""` makes
# `commits_in_range` fall back to the head commit alone
# (scripts/check-role-discipline.py:328-333), so the baseline lane would
# have reported a PASS covering one commit while looking like it covered
# a range. Skipping loudly beats passing vacuously. Recorded as risk 10
# in .agents/specs/main-verifiability.md.
env:
EVENT_NAME: ${{ github.event_name }}
PR_BASE: ${{ github.event.pull_request.base.sha }}
PR_HEAD: ${{ github.event.pull_request.head.sha }}
PUSH_BASE: ${{ github.event.before }}
LAST_GREEN: ${{ needs.last-gated-commit.outputs.sha }}
PUSH_HEAD: ${{ github.sha }}
run: |
set -eu
pending_args=()
if [ "$EVENT_NAME" = "pull_request" ]; then
head="$PR_HEAD"
pending_args=(--pending-pr-head "$PR_HEAD")
else
head="$PUSH_HEAD"
fi
# ONE resolver for every diff-scoped base in this file (#1809). The base
# is still the last SUCCESSFULLY gated commit, so a cancelled run stays
# lossless (#822, #863). It is now clamped from below by the recorded
# enforcement floor, `scripts/ci-enforcement-floor.txt`, so a commit
# already on `main` that no contributor can repair cannot freeze the
# base and turn the walk into a ratchet.
base="$(python3 scripts/ci-walk-base.py \
--event "$EVENT_NAME" --head "$head" \
--pr-base "${PR_BASE:-}" --push-base "${PUSH_BASE:-}" \
--last-green "${LAST_GREEN:-}")"
if [ -z "$base" ]; then
echo "no diff range on the $EVENT_NAME lane: role discipline is"
echo "diff-scoped and every commit's own push already ran it."
else
python3 scripts/check-role-discipline.py \
--base "$base" --head "$head" "${pending_args[@]}"
fi
python3 tests/scripts/test_agent_role.py
python3 tests/scripts/test_agent_onboard.py
- name: The diff-scoped gates resolve their base through one tested rule (#1809)
# Registered in TWO places, like the baseline and preflight suites above:
# here and in the `SUITES` array of `scripts/agent-preflight.sh`.
#
# The rule it covers WAS four byte-similar copies of inline shell in this
# file. `test_main_baseline.py::AgentRecordDiffRangeTests` replayed ONE of
# them, `agent-record`'s, under a shim that stubs every `python3` call --
# so it pinned which checker ran with which range string and could not
# see the base rule -- and the other three were executed by nothing. That
# is why the ratchet in #1809, an unrepairable commit freezing the base so
# every later push re-walked the same 41 violations over a range one
# commit wider, could only be found by reading a job log. The suite replays a
# cancelled-run sequence against a real throwaway repository, with the
# naive `github.event.before` base as its positive control, so the
# property #822 and #863 bought cannot be lost silently.
run: |
python3 tests/scripts/test_ci_walk_base.py
- name: Claim view, helper queue and PR reviewability
run: |
python3 scripts/claim-view.py --check
python3 scripts/ready-for-helper.py --check
python3 scripts/upstream-inventory.py --check
python3 tests/scripts/test_claim_view.py
python3 tests/scripts/test_upstream_inventory.py
- name: Live-state rows are reconciled against Git reality
# The classifier reads its evidence from `origin/main`. This job's
# checkout already uses `fetch-depth: 0`, which DOES leave
# `refs/remotes/origin/*` behind, so the ref is present today -- the
# fetch makes that independent of the checkout block above rather than
# a standing assumption about it, and refreshes the ref besides.
# Without a resolvable `origin/main` the audit aborts by design
# (require_origin_main): "no origin/main" would otherwise read as
# "every ACTIVE row is abandoned", absence of information wearing
# absence of work's face.
#
# The refspec is EXPLICIT and forced, not a bare `git fetch origin
# main`. A bare fetch only updates a remote-tracking ref when a
# CONFIGURED `remote.origin.fetch` refspec matches, and checkout sets
# that config from its own inputs (at shallower settings it narrows to
# the triggering ref, on a PR `refs/pull/N/merge`); with no match it
# writes FETCH_HEAD only. Naming the refspec creates the ref whatever
# the checkout is configured to do.
#
# The `row/*` refspec is the OTHER half of the same argument (#726).
# The classifier's IN-FLIGHT verdict is reached only through a ref named
# `row/<ID>`, and fetching `main` alone left none in the checkout -- so
# IN-FLIGHT was unreachable and a row whose work is genuinely in flight
# produced the same verdict as one nobody is working on. Every PR that
# moves a row to ACTIVE before its code lands failed here, which is the
# normal shape of the work. `require_branch_information` now aborts
# rather than misreport if this refspec is ever dropped again.
#
# It does not cover a FORK, whose branch is a ref `origin` cannot hold;
# that case is covered inside the audit, by reading `origin/main..HEAD`.
#
# Cost measured at `af026e524`: 267 `row/*` heads carrying 648 commits
# not on `origin/main`, over a `fetch-depth: 0` checkout that already
# holds main's history.
run: |
git fetch -q origin +refs/heads/main:refs/remotes/origin/main \
'+refs/heads/row/*:refs/remotes/origin/row/*'
python3 scripts/audit-live-rows.py --check
python3 tests/scripts/test_audit_live_rows.py
documentation-checkpoint:
# Gate: every code/test/benchmark/spike/lifecycle iteration refreshes both
# user-facing status surfaces in that same commit, including void attempts.
#
# DIFF-scoped: deliberately carries NO concurrency group. Its verdict covers
# this push's own `before..sha` range and nothing re-covers that range later,
# so cancelling it would silently exempt those commits from the gate.
#
# Not on the baseline lane (#274): `schedule` and `workflow_dispatch`
# payloads have no `github.event.before`, so the range below would be empty
# and the gate vacuous. The push that landed each commit already ran this,
# which is exactly why it carries no group. The baseline's subject is the
# TREE at a SHA, and `scripts/main-baseline.py` names the jobs it covered.
# always(): `last-gated-commit` is skipped on the PR lane, and a
# skipped dependency would otherwise skip this gate too.
if: always() && github.event.action != 'closed' && (github.event_name != 'schedule' && github.event_name != 'workflow_dispatch')
needs: [last-gated-commit]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Every feature checkpoint updates STATUS, BENCHMARKS and NOW
env:
EVENT_NAME: ${{ github.event_name }}
PR_BASE: ${{ github.event.pull_request.base.sha }}
PR_HEAD: ${{ github.event.pull_request.head.sha }}
PUSH_BASE: ${{ github.event.before }}
LAST_GREEN: ${{ needs.last-gated-commit.outputs.sha }}
PUSH_HEAD: ${{ github.sha }}
run: |
set -eu
if [ "$EVENT_NAME" = "pull_request" ]; then
head="$PR_HEAD"
else
head="$PUSH_HEAD"
fi
# ONE resolver for every diff-scoped base in this file (#1809). The base
# is still the last SUCCESSFULLY gated commit, so a cancelled run stays
# lossless (#822, #863). It is now clamped from below by the recorded
# enforcement floor, `scripts/ci-enforcement-floor.txt`, so a commit
# already on `main` that no contributor can repair cannot freeze the
# base and turn the walk into a ratchet.
base="$(python3 scripts/ci-walk-base.py \
--event "$EVENT_NAME" --head "$head" \
--pr-base "${PR_BASE:-}" --push-base "${PUSH_BASE:-}" \
--last-green "${LAST_GREEN:-}")"
python3 scripts/check-now-current.py --base "$base" --head "$head"
pending_args=()
if [ "$EVENT_NAME" = "pull_request" ]; then
pending_args=(--pending-pr-head "$PR_HEAD")
fi
python3 scripts/check-role-discipline.py \
--base "$base" --head "$head" "${pending_args[@]}"
last-gated-commit:
# The head of the most recent SUCCESSFUL push run of this workflow on this
# branch. It is the base the diff-scoped gates walk from, instead of
# `github.event.before`.
#
# `before` is the previous push's sha whether or not that push was gated, so
# a cancelled run's commits are skipped and NOTHING re-covers them -- the
# reason a superseded main run could not be cancelled at all (#822, #863).
# Basing on the last GREEN commit makes a cancelled run lossless: the next
# run simply walks a wider range.
#
# Carries NO concurrency group, like the gates that consume it.
#
# This is workflow plumbing resolving an input, not a checker. The
# protocol's ban on network calls is about a CHECKER failing on
# connectivity; this falls back to `before` when the query returns nothing,
# so a failed query is the status quo rather than a skipped gate.
#
# It runs on EVERY lane except a closed pull request, and resolves a sha
# only on the push lane -- there is no gated `main` run to find for a pull
# request or a schedule. The wider condition is deliberate: `agent-record`
# may not carry an `if:` of its own without un-registering three checkers
# (#873), so this job's condition is the closed-PR guard it inherits through
# `needs:`. A skipped dependency skips its dependents, which is the same
# skip #822 asked for, expressed where a checker can still see the gate.
if: github.event.action != 'closed'
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
outputs:
sha: ${{ steps.resolve.outputs.sha }}
steps:
- id: resolve
env:
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
BRANCH: ${{ github.ref_name }}
EVENT_NAME: ${{ github.event_name }}
run: |
set -eu
# Only the push lane has a gated `main` history to resolve against.
# Every other lane leaves this EMPTY and its consumers fall back,
# which is what they already do when the query finds nothing.
sha=""
if [ "$EVENT_NAME" = "push" ]; then
sha="$(gh api \
"repos/$REPO/actions/workflows/ci.yml/runs?branch=$BRANCH&event=push&status=success&per_page=1" \
--jq '.workflow_runs[0].head_sha' 2>/dev/null || true)"
fi
# NOT `[ ... ] && sha=""`: that returns non-zero when the test is
# false, which aborts the step under `set -e`.
if [ "$sha" = "null" ]; then sha=""; fi
echo "last successfully gated commit: ${sha:-<none, falling back to before>}"
echo "sha=$sha" >> "$GITHUB_OUTPUT"
commit-protocol-tag:
# Gate: every NEW commit must carry the FOLLOWING_AGENTS_PROTOCOL trailer,
# asserting the contributor read AGENTS.md. See .agents/ai-coding-assistants.md.
#
# DIFF-scoped: deliberately carries NO concurrency group, same reasoning as
# documentation-checkpoint above. It walks `before..sha` per push, so a
# cancelled run means those commits are never checked for the trailer.
#
# Not on the baseline lane (#274), same reasoning: no `before` on a
# `schedule` or `workflow_dispatch` payload, so the walk has no range.
needs: [last-gated-commit]
if: always() && github.event.action != 'closed' && github.event_name != 'schedule' && github.event_name != 'workflow_dispatch'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Every new commit carries FOLLOWING_AGENTS_PROTOCOL
env:
EVENT_NAME: ${{ github.event_name }}
PR_BASE: ${{ github.event.pull_request.base.sha }}
PR_HEAD: ${{ github.event.pull_request.head.sha }}
PUSH_BASE: ${{ github.event.before }}
LAST_GREEN: ${{ needs.last-gated-commit.outputs.sha }}
PUSH_HEAD: ${{ github.sha }}
run: |
set -eu
if [ "$EVENT_NAME" = "pull_request" ]; then
head="$PR_HEAD"
else
head="$PUSH_HEAD"
fi
# ONE resolver for every diff-scoped base in this file (#1809). The base
# is still the last SUCCESSFULLY gated commit, so a cancelled run stays
# lossless (#822, #863). It is now clamped from below by the recorded
# enforcement floor, `scripts/ci-enforcement-floor.txt`, so a commit
# already on `main` that no contributor can repair cannot freeze the
# base and turn the walk into a ratchet.
base="$(python3 scripts/ci-walk-base.py \
--event "$EVENT_NAME" --head "$head" \
--pr-base "${PR_BASE:-}" --push-base "${PUSH_BASE:-}" \
--last-green "${LAST_GREEN:-}")"
# Enforce on the FIRST-PARENT mainline (the commits the submitter
# directly lands on the target branch, incl. merge commits which are
# skipped below). Merged-in feature-branch commits are owned by the
# submitter via the merge and are not re-checked here.
# New branch / unreachable base (all-zero SHA): only check the tip commit.
if ! git cat-file -e "${base}^{commit}" 2>/dev/null; then
commits="$head"
else
commits="$(git rev-list --first-parent "${base}..${head}")"
fi
fail=0
for c in $commits; do
# Skip merge commits (>1 parent) — they are not authored content.
if [ "$(git rev-list --parents -n1 "$c" | wc -w)" -gt 2 ]; then continue; fi
if ! git log -1 --format=%B "$c" | grep -q 'FOLLOWING_AGENTS_PROTOCOL'; then
echo "::error::commit $c is missing the FOLLOWING_AGENTS_PROTOCOL trailer — read AGENTS.md"
git log -1 --oneline "$c"
fail=1
fi
done
if [ "$fail" -ne 0 ]; then
echo "One or more commits lack FOLLOWING_AGENTS_PROTOCOL. See AGENTS.md / .agents/ai-coding-assistants.md."
exit 1
fi
echo "OK: all new commits carry FOLLOWING_AGENTS_PROTOCOL."
- name: Every new commit satisfies the strict trailer contract
# DIFF-scoped, and therefore HERE rather than in agent-record. It lived
# in that job until #863: agent-record carries a cancellable group keyed
# on `github.ref`, which for a push is the constant `refs/heads/main`, so
# consecutive pushes cancelled it. Measured on run 31851003245, the push
# of 51e0cb5b1: agent-record cancelled, this job succeeded, and the
# commit fails the strict walk. The grep above passed it because the
# marker was PRESENT three times -- presence is not parseability, and
# only the strict walk knows the difference.
env:
EVENT_NAME: ${{ github.event_name }}
PR_BASE: ${{ github.event.pull_request.base.sha }}
PR_HEAD: ${{ github.event.pull_request.head.sha }}
PUSH_BASE: ${{ github.event.before }}
LAST_GREEN: ${{ needs.last-gated-commit.outputs.sha }}
PUSH_HEAD: ${{ github.sha }}
# Passed as an ENV VAR, never interpolated into the script: a body is
# attacker-controlled text on a fork pull request.
PR_BODY: ${{ github.event.pull_request.body }}
run: |
set -eu
if [ "$EVENT_NAME" = "pull_request" ]; then
head="$PR_HEAD"
else
head="$PUSH_HEAD"
fi
# ONE resolver for every diff-scoped base in this file (#1809). The base
# is still the last SUCCESSFULLY gated commit, so a cancelled run stays
# lossless (#822, #863). It is now clamped from below by the recorded
# enforcement floor, `scripts/ci-enforcement-floor.txt`, so a commit
# already on `main` that no contributor can repair cannot freeze the
# base and turn the walk into a ratchet.
base="$(python3 scripts/ci-walk-base.py \
--event "$EVENT_NAME" --head "$head" \
--pr-base "${PR_BASE:-}" --push-base "${PUSH_BASE:-}" \
--last-green "${LAST_GREEN:-}")"
if [ -z "$base" ] || ! git cat-file -e "${base}^{commit}" 2>/dev/null; then
echo "no usable diff range on the $EVENT_NAME lane: base=${base:-<empty>}"
echo "checking the tip commit alone rather than passing vacuously."
python3 scripts/check-commit-trailers.py --range "$head~1..$head"
else
echo "strict trailer walk over ${base}..${head}"
python3 scripts/check-commit-trailers.py --range "$base..$head"
fi
# `squash_merge_commit_message = PR_BODY` makes the pull request body
# the landed commit message, so it is held to the contract its commit
# will be held to, by the same checker (#848). `${PR_BODY:-}` because
# this runs under `set -eu` and the variable is UNSET, not empty, on a
# lane with no pull request. An empty body is not skipped: it lands a
# commit carrying no trailers at all.
if [ "$EVENT_NAME" = "pull_request" ]; then
body_file="$(mktemp)"
printf '%s' "${PR_BODY:-}" > "$body_file"
python3 scripts/check-commit-trailers.py \
--message-file "$body_file" --filled
fi
pr-size:
# PR-only. The per-class LINE BUDGETS this job used to enforce were retired
# 2026-08-10, and the fail-closed BINARY GUARD was retired 2026-08-13 (see
# scripts/check-pr-size.py for both); the job name is kept because it is a
# required check. What it enforces now: explicit path classification, the
# checker-change mutation-evidence contract, and the role check that keeps
# product paths on a PR. Nothing here measures the size of a diff.
if: github.event.action != 'closed' && (github.event_name == 'pull_request')
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: PR path classification and checker evidence
run: |
python3 scripts/check-pr-size.py \
--base "${{ github.event.pull_request.base.sha }}" \
--head "${{ github.event.pull_request.head.sha }}" \
--branch "${{ github.event.pull_request.head.ref }}" \
--pr-number "${{ github.event.pull_request.number }}"
cuda-arch-features:
# Configure-tier assertions on the CUDA per-arch FEATURE TABLE. Needs neither
# a CUDA toolkit nor a GPU: it drives cmake/CudaArchFeatures.cmake directly,
# so the resolution that decides which architectures get fp4-mma /
# cutlass-nvfp4 / cutlass-fp8 / marlin-nvfp4 / fa2 (whose historical failure
# mode was a SILENT capability drop) is checked on every push.
#
# Tree-scoped (drives cmake at HEAD): cancellable.
concurrency:
group: ci-cuda-arch-features-${{ github.event_name }}-${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && github.run_id || github.ref }}-${{ github.repository }}
cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
# Skip when the pull request was CLOSED: the run exists only to enter
# the concurrency group and supersede the in-flight one (#822).
if: github.event.action != 'closed'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: CUDA feature table and per-source gencode resolve exactly
run: |
cmake -P cmake/CudaArchFeaturesTest.cmake
cmake -P cmake/CudaSourceGencodeTest.cmake
cmake -P cmake/TritonAOTMultiArchTest.cmake
cmake -P cmake/TritonAOTDefaultTest.cmake
# Same tier, no CUDA involved: the in-source build guard (issue #85) is a
# configure-time predicate, so it is asserted the same way — by driving the
# module directly, with no compiler and no writes to the tree.
- name: In-source build guard fires on `cmake .` and only on `cmake .`
run: |
cmake -P cmake/InSourceGuardTest.cmake
python3 -m unittest tests.scripts.test_check_cuda_fat_gencode
python3 -m unittest tests.scripts.test_check_triton_aot_multiarch
cuda-fat-build:
# W1 release prerequisite: compile the primary heterogeneous CUDA archive,
# then inspect both compile_commands.json and the linked archive. No GPU is
# used; runtime/correctness evidence remains independent.
permissions:
contents: read
concurrency:
group: ci-cuda-fat-build-${{ github.event_name }}-${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && github.run_id || github.ref }}-${{ github.repository }}
cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
# Skip when the pull request was CLOSED: the run exists only to enter
# the concurrency group and supersede the in-flight one (#822).
if: github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 180
container: nvidia/cuda:13.3.0-devel-ubuntu24.04
steps:
- name: Install build tools
run: |
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
binutils ca-certificates cmake g++ git ninja-build python3
rm -rf /var/lib/apt/lists/*
- uses: actions/checkout@v4
- name: Configure ten-SM static library
run: |
cmake -S . -B build-cuda-fat -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
-DVLLM_CPP_CUDA=ON \
-DVLLM_CPP_CUDA_ARCHITECTURES='80;86;87;89;90a;100a;103a;110;120a;121a' \
-DVLLM_CPP_CUTLASS_FETCH=ON \
-DVLLM_CPP_TRITON=ON \
-DVLLM_CPP_BUILD_TESTS=OFF \
-DVLLM_CPP_BUILD_EXAMPLES=OFF \
-DVLLM_CPP_SERVER=OFF
- name: Build and audit exact per-source gencode
run: |
cmake --build build-cuda-fat --target vllm --parallel 2
python3 scripts/check-cuda-fat-gencode.py \
--compile-commands build-cuda-fat/compile_commands.json \
--library build-cuda-fat/libvllm.a
python3 scripts/check-triton-aot-multiarch.py \
--vendored-root src/vt/cuda/triton_aot_vendored \
--library build-cuda-fat/libvllm.a
# BUILD-TRITON-DEFAULT-ON (#219), spec Tests item 3. The configure above
# passes -DVLLM_CPP_TRITON=ON explicitly, so it cannot witness the DEFAULT.
# This one passes no such flag: the option must resolve ON by itself AND
# reach every CUDA translation unit as VLLM_CPP_TRITON=1 /
# VLLM_CPP_TRITON_CHUNKO_BF16=1, which is the row's actual claim. Configure
# only, and it reuses the CUTLASS the job already fetched (FETCH stays ON
# as the fallback if that path ever moves).
- name: The computed Triton default reaches the CUDA translation units
run: |
cmake -S . -B build-cuda-default -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
-DVLLM_CPP_CUDA=ON \
-DVLLM_CPP_CUDA_ARCHITECTURES='80;86;87;89;90a;100a;103a;110;120a;121a' \
-DVLLM_CPP_CUTLASS_DIR="$PWD/build-cuda-fat/_deps/cutlass-src" \
-DVLLM_CPP_CUTLASS_FETCH=ON \
-DVLLM_CPP_BUILD_TESTS=OFF \
-DVLLM_CPP_BUILD_EXAMPLES=OFF \
-DVLLM_CPP_SERVER=OFF
VLLM_CPP_DEFAULT_BUILD_DIR=build-cuda-default \
python3 -m unittest tests.scripts.test_triton_default_definitions
vulkan-spirv-freshness:
# The Vulkan backend commits its SPIR-V (src/vt/vulkan/vulkan_spirv.h) instead
# of compiling shaders at build time, so the build needs no shader toolchain on
# any machine -- strictly better than llama.cpp, which requires glslc on every
# build box. The trade is an obligation to regenerate by hand, and NOTHING
# ENFORCED IT: `gen-vulkan-spirv.py --check` existed but no job ran it, so a
# .comp edit without a regenerate shipped silently. That is the failure mode
# this job closes, before the shader surface grows (BACKEND-VULKAN, VK-A1).
#
# The glslang DOWNLOAD URL is the pin. An exact version-string assertion was
# deliberately NOT used: the committed SPIR-V was measured byte-identical under
# both the 16.4.0 that produced it and the 16.5.0 pinned here, so gating on the
# string would be brittle without buying anything. The URL being fixed is what
# makes the run deterministic.
#
# Tree-scoped (checks HEAD's artifact): cancellable.
concurrency:
group: ci-vulkan-spirv-freshness-${{ github.event_name }}-${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && github.run_id || github.ref }}-${{ github.repository }}
cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
# Skip when the pull request was CLOSED: the run exists only to enter
# the concurrency group and supersede the in-flight one (#822).
if: github.event.action != 'closed'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install pinned glslang
run: |
set -euo pipefail
curl -fsSL -o /tmp/glslang.tar.gz \
https://github.com/KhronosGroup/glslang/releases/download/16.5.0/glslang-16.5.0-linux-x86_64-release.tar.gz
mkdir -p /tmp/glslang
tar xzf /tmp/glslang.tar.gz -C /tmp/glslang
/tmp/glslang/bin/glslang --version | head -1
echo "/tmp/glslang/bin" >> "$GITHUB_PATH"
- name: Committed SPIR-V is not stale
run: python3 scripts/gen-vulkan-spirv.py --check
build-test-vulkan:
# The Vulkan backend had NO CI leg at all: `VLLM_CPP_VULKAN=ON` appeared
# nowhere in this workflow, so tests/vt/test_vulkan_backend.cpp ran on no
# machine. It rotted exactly as you would expect. Accelerator-seam row S5
# (af0b21ba) gave unified-memory devices a portable CPU reference tier, which
# means a missed GetOp no longer throws; the Metal sibling test was updated for
# that as Metal work continued, and the Vulkan one was not, so its "the
# unimplemented ops throw" assertion sat RED and INVISIBLE from S5 until
# VK-A1 built the backend by hand (2026-08-06). That is the gap this closes.
#
# Runs GPU-FREE on `mesa-vulkan-drivers`' llvmpipe software ICD, which is the
# arrangement the fan-out spike already recorded as working. The suite is
# scoped to the Vulkan-specific targets rather than all of ctest: the full
# suite is already covered by build-test-cpu, and this lane exists to keep the
# Vulkan-only TUs compiled and executed, not to duplicate it.
#
# Tree-scoped: cancellable.
concurrency:
group: ci-build-test-vulkan-${{ github.event_name }}-${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && github.run_id || github.ref }}-${{ github.repository }}
cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
# Skip when the pull request was CLOSED: the run exists only to enter
# the concurrency group and supersede the in-flight one (#822).
if: github.event.action != 'closed'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install the software Vulkan ICD
# Loader + llvmpipe only. No shader toolchain: the backend consumes the
# COMMITTED SPIR-V, which is the whole point of the committed-artifact
# route, and the vulkan-spirv-freshness job above is what keeps it honest.
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get install -y --no-install-recommends libvulkan1 mesa-vulkan-drivers
- name: Configure
run: cmake -S . -B build-vulkan -DVLLM_CPP_BUILD_TESTS=ON -DVLLM_CPP_VULKAN=ON
- name: Build
# Bounded parallelism for the same reason as build-test-cpu: an unbounded
# parallel link OOM-kills the runner.
run: cmake --build build-vulkan -j 2 --target test_vulkan_backend test_backend_cross_device test_platform
- name: Vulkan backend gate
run: ./build-vulkan/tests/test_vulkan_backend
- name: Cross-device numerics vs the CPU oracle
run: ./build-vulkan/tests/test_backend_cross_device
- name: Platform seam gate, with a non-CUDA accelerator REGISTERED
# #1823. `test_platform` also runs on build-test-cpu, but that tier has
# only kCPU registered, so the capability-UNIT contract it asserts
# (`get_device_capability()` is an NVIDIA SM value or it is absent) walks
# exactly one platform there and cannot catch the defect it exists for.
# This lane registers kVULKAN, which is where that walk has teeth — and
# kVULKAN is one of the two platforms that carried the defect.
#
# THE GREP IS THE POINT, not the exit status. `VulkanPresent()` is a
# silent early return in the suite, so on a runner where the ICD failed
# to install BOTH executables report `Status: SUCCESS!` with rc 0 while
# measuring nothing about Vulkan: test_vulkan_backend drops from 2109
# assertions to 344, and test_platform walks kCPU alone. A lane whose
# only claim is "kVULKAN is registered here" must prove that, or the
# claim is the thing being tested and nothing else is.
run: |
set -euo pipefail
./build-vulkan/tests/test_platform | tee platform-gate.log
grep -q 'platform vulkan get_device_capability()' platform-gate.log
grep -q 'Status: SUCCESS!' platform-gate.log
- name: The Vulkan suite MEASURED a device rather than skipping past one
# The same positive control for the suite itself: assert the platform
# case ran its #1823 assertions, which it does not do when
# `VulkanPresent()` is false.
run: |
set -euo pipefail
./build-vulkan/tests/test_vulkan_backend \
-tc="Vulkan platform is registered and reports unified/no-pool residency" -s \
| tee vulkan-platform-case.log
grep -q 'kVULKAN Platform::get_device_capability() present=false' vulkan-platform-case.log
grep -q 'SelectAttentionBackendName(p) == "FLASH_ATTN" ) is correct' vulkan-platform-case.log
grep -qE 'test cases: *1 \| *1 passed' vulkan-platform-case.log
device-leakage:
# The DSR RATCHET (work row `S1` of .agents/specs/accelerator-seam-audit.md).
# Counts device-specific references in `src/vllm/` + `include/vllm/` — the
# layer that is supposed to be device-agnostic — and fails on ANY increase
# over scripts/device-leakage-baseline.json. A reduction must lower the
# baseline in the SAME commit, so the number can only ever move down.
#
# It exists because the audit re-measured the leakage and found it had DRIFTED
# UPWARD with no bad commit: DeepSeek-V2, Qwen3-Coder and the attention-registry
# work each added a device test in passing. Leakage grows silently under
# well-executed work, which is a job for a ratchet, not a cleanup.
#
# Needs neither a CUDA toolkit nor a GPU — pure static analysis, like the
# cuda-arch-features job above. The mutation suite proves the checker actually
# catches a planted leak; an unpoliced checker is worse than none.
#
# Tree-scoped: the ratchet compares HEAD against the committed baseline, so
# only the newest push matters. Cancellable.
concurrency:
group: ci-device-leakage-${{ github.event_name }}-${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && github.run_id || github.ref }}-${{ github.repository }}
cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
# Skip when the pull request was CLOSED: the run exists only to enter
# the concurrency group and supersede the in-flight one (#822).
if: github.event.action != 'closed'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Shared-layer device leakage does not grow (DSR ratchet)
run: |
python3 scripts/check-device-leakage.py --report
python3 tests/scripts/test_device_leakage.py
windows-msvc-cpu:
# Native Windows release portability must be proven before merge. This PR
# lane calls the authoritative driver but retains no artifact and has no
# release, upload, write-token, or OIDC authority (#117).
#
# IT RUNS ON THE BASELINE LANE TOO (#503). The condition was
# `github.event_name == 'pull_request'` alone until 2026-08-17, so `main`
# could establish neither green nor red under MSVC. `scripts/main-baseline.py`
# then printed GREEN *because* the lane it grades never started this job --
# absence wearing success's face, which is the exact defect #274 was filed
# about, reached by a different route. Measured on the deliberate
# workflow_dispatch baseline 32044993401 (`conclusion=success`): both
# windows-msvc-* jobs `skipped`. The cost of that was not theoretical --
# #503, #603, #729, #965, #968 and #1068 each landed on `main` unseen and
# then surfaced as a red on an unrelated contributor's pull request.
#
# `push` stays EXCLUDED. That lane is cancellable by construction and cannot
# answer "is main green" (see the `schedule` comment above), and at the 55
# pushes/day measured there, two `windows-2022` runners per push is a bill
# rather than a baseline.
if: github.event_name == 'pull_request' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
permissions:
contents: read
runs-on: windows-2022
timeout-minutes: 180
steps:
- uses: actions/checkout@v4
- name: Prove PowerShell, static CRT, and unsupported-tier contracts
run: ./scripts/build-windows-release.ps1 -ContractTest
- name: Build and execute the native Windows CPU focused gate
env:
EVIDENCE_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
SOURCE_SHA: ${{ github.sha }}
run: |
$env:VERSION = (Get-Content release/release-version.json -Raw | ConvertFrom-Json).version
$env:SOURCE_DATE_EPOCH = (git show -s --format=%ct HEAD).Trim()
./scripts/build-windows-release.ps1 `
-Backend cpu `
-ArtifactId windows-x86_64-msvc-cpu `
-BuildDir $env:GITHUB_WORKSPACE/build-pr-windows-cpu
windows-msvc-vulkan:
# Independent Vulkan build/loader/backend proof on the native MSVC ABI.
# Like the CPU lane it is read-only and retains no release asset, and it
# answers for the same three events for the same reason (#503): a baseline
# that silently excludes a compiling gate certifies less than it claims.
if: github.event_name == 'pull_request' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
permissions:
contents: read
runs-on: windows-2022
timeout-minutes: 180
steps:
- uses: actions/checkout@v4
- name: Prove PowerShell, static CRT, and unsupported-tier contracts
run: ./scripts/build-windows-release.ps1 -ContractTest
- name: Build and execute the native Windows Vulkan focused gate
env:
EVIDENCE_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
SOURCE_SHA: ${{ github.sha }}
run: |
$env:VERSION = (Get-Content release/release-version.json -Raw | ConvertFrom-Json).version
$env:SOURCE_DATE_EPOCH = (git show -s --format=%ct HEAD).Trim()
./scripts/build-windows-release.ps1 `
-Backend vulkan `
-ArtifactId windows-x86_64-msvc-vulkan `
-BuildDir $env:GITHUB_WORKSPACE/build-pr-windows-vulkan
build-test-cpu:
# Tree-scoped and the most expensive lane in the workflow. Cancellable.
concurrency:
group: ci-build-test-cpu-${{ github.event_name }}-${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && github.run_id || github.ref }}-${{ github.repository }}
cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
# Skip when the pull request was CLOSED: the run exists only to enter
# the concurrency group and supersede the in-flight one (#822).
if: github.event.action != 'closed'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure
run: |
cmake -S . -B build \
-DVLLM_CPP_BUILD_TESTS=ON \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON
- name: Build
# Bounded parallelism: a bare `-j` lets Make link ALL test executables at
# once, which OOM-kills the runner (ld signal 9) during the parallel link.
run: cmake --build build -j 2
- name: Installed server archive is reproducible and runnable
env:
VLLM_CPP_PACKAGE_TEST_BUILD_DIR: ${{ github.workspace }}/build
run: python3 tests/scripts/test_server_package.py
- name: CPU ISA objects use only their declared feature flags
run: |
python3 scripts/check-cpu-isa-build.py \
--compile-commands build/compile_commands.json
- name: Test
run: ctest --test-dir build --output-on-failure
- name: Force every x86 ISA tier available on this runner
run: |
tiers=(portable)
if [[ "$(uname -m)" == "x86_64" ]]; then
tiers+=(sse2)
if grep -qiw f16c /proc/cpuinfo && grep -qiw avx /proc/cpuinfo; then
tiers+=(sse2+f16c)
fi
if grep -qiw avx2 /proc/cpuinfo; then
tiers+=(avx2)
fi
if grep -qiw avx512f /proc/cpuinfo && \
grep -qiw avx512bw /proc/cpuinfo && \
grep -qiw avx512vl /proc/cpuinfo; then
tiers+=(avx512)
fi
fi
for tier in "${tiers[@]}"; do
echo "Running test_ops_matmul_elem with VT_CPU_MATMUL_TIER=${tier}"
VT_CPU_MATMUL_TIER="${tier}" build/tests/test_ops_matmul_elem
done
build-newest-gcc:
# COMPILE-ONLY coverage on a compiler NEWER than any other lane's. Every
# other Linux lane installs the distro `g++`, which on ubuntu-latest is
# gcc 13, so a header that only compiles because of a TRANSITIVE include
# goes green here and red on a current distro. That is not hypothetical:
# ::getpid without <unistd.h> was fixed once and came back in five more
# files, one of them src/vllm/entrypoints/openai/server_main.cpp — a
# SHIPPED binary that does not build on gcc 16 (reported on issue #41).
#
# Deliberately does NOT run ctest: this lane exists to catch compile-time
# portability, and build-test-cpu already owns execution. Keeping it to a
# build is roughly one extra compile per PR.
concurrency:
group: ci-build-newest-gcc-${{ github.event_name }}-${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && github.run_id || github.ref }}-${{ github.repository }}
cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
if: github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 90
container: gcc:16
steps:
- name: Install build tools
# The gcc image ships the toolchain only; git is needed BEFORE checkout
# so actions/checkout uses git rather than the slower tarball path.
run: |
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
ca-certificates cmake git ninja-build python3
rm -rf /var/lib/apt/lists/*
- uses: actions/checkout@v4
- name: Report the compiler this lane is actually pinning
run: g++ --version
- name: Configure
run: |
cmake -S . -B build-newest-gcc -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DVLLM_CPP_BUILD_TESTS=ON
- name: Build
# Same bounded parallelism as build-test-cpu: a bare -j OOM-kills the
# runner during the parallel link of the test executables.
run: cmake --build build-newest-gcc -j 2
build-test-cpu-arm64:
# Independent Arm execution evidence: the x86 lane cannot prove HWCAP
# dispatch, Arm instructions, or the host ABI. The native runner exercises
# every available rich tier; qemu's Cortex-A53 model supplies the required
# feature-poor execution and illegal-tier refusal gate.
concurrency:
group: ci-build-test-cpu-arm64-${{ github.event_name }}-${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && github.run_id || github.ref }}-${{ github.repository }}
cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
# Skip when the pull request was CLOSED: the run exists only to enter
# the concurrency group and supersede the in-flight one (#822).
if: github.event.action != 'closed'
runs-on: ubuntu-24.04-arm
steps:
- uses: actions/checkout@v4
- name: Configure focused Arm CPU gate
run: |
cmake -S . -B build-arm \
-DVLLM_CPP_BUILD_TESTS=ON \
-DVLLM_CPP_BUILD_EXAMPLES=OFF \
-DVLLM_CPP_SERVER=OFF \
-DVLLM_CPP_CUDA=OFF \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON
- name: Build Arm ISA and kernel gates
# The nine tokenizer targets joined this list on 2026-08-23
# (GATE-CI-AARCH64-COVERAGE W1b, #1385). They are NOT ISA-dispatch
# gates; they are here because `char` is UNSIGNED on aarch64 and SIGNED
# on x86-64, so byte-oriented pretokenization, UTF-8 handling and BPE
# merge ordering can diverge on the shipping architecture in a way the
# x86-64 lane cannot falsify -- and until this change NO tokenizer test
# executed on aarch64 anywhere in CI. `test_tokenizer_parity` in
# particular is hermetic (its goldens are committed under
# tests/parity/goldens/) and compares IDS, not counts, so it is a real
# gate here rather than a smoke test. See
# .agents/specs/prompt-token-divergence.md `## Owed`, first item.
run: |
cmake --build build-arm \
--target test_cpu_isa_arm test_ops_matmul_elem \
test_ops_quant_dot test_ops_quant_repack \
test_pretokenizer test_bpe test_bpe_equivalence \
test_detokenizer test_tiktoken_bpe \
test_tokenizer_parity test_tokenizer_parity_deepseek \
test_tokenizer_parity_mistral test_tokenizer_parity_gpt4o \
-j 2
- name: Arm ISA objects use only their declared feature flags
run: |
python3 scripts/check-arm-isa-build.py \
--compile-commands build-arm/compile_commands.json
- name: Force baseline and every rich tier available on the native runner
run: |
build-arm/tests/test_cpu_isa_arm
VT_CPU_MATMUL_TIER=portable build-arm/tests/test_ops_matmul_elem
VT_CPU_MATMUL_TIER=neon build-arm/tests/test_ops_matmul_elem
VT_CPU_Q8_DOT=portable \
VT_CPU_QUANT_MMLA=portable \
build-arm/tests/test_ops_quant_dot
VT_CPU_QUANT_REPACK=portable build-arm/tests/test_ops_quant_repack
if grep -qiw asimddp /proc/cpuinfo; then
VT_CPU_Q8_DOT=sdot build-arm/tests/test_ops_quant_dot
fi
if grep -qiw asimddp /proc/cpuinfo && grep -qiw i8mm /proc/cpuinfo; then
VT_CPU_QUANT_MMLA=i8mm build-arm/tests/test_ops_quant_dot
VT_CPU_QUANT_REPACK=i8mm build-arm/tests/test_ops_quant_repack
fi
- name: Feature-poor Cortex-A53 executes baseline and refuses rich tiers
run: |
sudo apt-get update
sudo apt-get install --yes qemu-user
qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_cpu_isa_arm
VT_CPU_MATMUL_TIER=portable \
qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_matmul_elem
VT_CPU_MATMUL_TIER=neon \
qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_matmul_elem
VT_CPU_Q8_DOT=portable VT_CPU_QUANT_MMLA=portable \
qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_quant_dot
if VT_CPU_Q8_DOT=sdot \
qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_quant_dot; then
echo "Cortex-A53 unexpectedly accepted the DotProd tier" >&2
exit 1
fi
if VT_CPU_QUANT_MMLA=i8mm \
qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_quant_dot; then
echo "Cortex-A53 unexpectedly accepted the i8mm tier" >&2
exit 1
fi
- name: Tokenizers agree with the HF oracle on the shipping architecture
# Executed, not merely built: a target that compiles proves nothing
# about `char` signedness at run time. Each binary is run by name so a
# MISSING one is a "no such file" failure rather than a silently empty
# selection -- `ctest -R` that matches nothing exits 0.
run: |
set -u
for t in test_pretokenizer test_bpe test_bpe_equivalence \
test_detokenizer test_tiktoken_bpe \
test_tokenizer_parity test_tokenizer_parity_deepseek \
test_tokenizer_parity_mistral test_tokenizer_parity_gpt4o; do
echo "== aarch64 native: ${t}"
"build-arm/tests/${t}"
done
build-test-cpu-arm64-full:
# THE WHOLE SUITE ON THE SHIPPING ARCHITECTURE (#1385,
# .agents/specs/ci-aarch64-coverage.md). `build-test-cpu-arm64` above
# builds FOUR test targets and runs no `ctest` at all, and all four are
# ISA-dispatch and kernel-tier gates. The spec carries the ratio measured
# against a NAMED SHA; a live count kept here would be stale within a day
# and no gate would say so. Until this job, no engine, model, loader, sampler,
# tokenizer or KV-cache test executed on aarch64 anywhere, while every GPU
# box in the fleet is aarch64 and release.yml ships `cpu_arm64` and
# `cuda_arm64` bundles. x86-64 cannot falsify the divergences the standard
# and the two ABIs permit: the weak memory model, `char` signedness,
# floating-point contraction, and address-dependent iteration order.
#
# SCHEDULE AND DISPATCH ONLY, and `continue-on-error: true`, for its first
# landing. The duration is MODELLED at 45-55 minutes and measured at
# nothing, so it lands where a wrong model costs zero pull-request time,
# and where a pre-existing aarch64 finding cannot block unrelated work --
# the shape `sanitize-cpu` below already uses. Promotion to every event and
# removal of the flag is gated on the measurement (spec `## Gates` G2:
# median duration plus the arm pool's median queue must stay under
# `cuda-fat-build`'s measured 123.0-minute finish) and on triaging every
# finding into its own issue and row. A red lane must be triaged, never
# muted.
#
# It is a SIBLING of the fast arm job, not an extension of it, so that the
# 11.2-minute per-pull-request ISA gate keeps its duration.
continue-on-error: true
concurrency:
group: ci-build-test-cpu-arm64-full-${{ github.event_name }}-${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && github.run_id || github.ref }}-${{ github.repository }}
cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
# Skip when the pull request was CLOSED: the run exists only to enter
# the concurrency group and supersede the in-flight one (#822).
if: github.event.action != 'closed' && (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch')
runs-on: ubuntu-24.04-arm
# Above the 45-55 minute estimate and below the point where a hung job
# holds an arm slot for a whole 4-hour cron interval. `cuda-fat-build`
# uses 180.
timeout-minutes: 120
steps:
- uses: actions/checkout@v4
- name: Configure
# The SAME four flags as build-test-cpu-arm64, so the two aarch64 lanes
# configure one tree and cannot drift. Every test target configures
# under them except those the `VLLM_CPP_SERVER`, HIP, Metal, Vulkan and
# Tenstorrent guards exclude, and NO list is kept here: a new test joins
# this lane by existing. `examples/` and the server path staying OFF is
# recorded as owed in the spec.
run: |
cmake -S . -B build-arm-full \
-DVLLM_CPP_BUILD_TESTS=ON \
-DVLLM_CPP_BUILD_EXAMPLES=OFF \
-DVLLM_CPP_SERVER=OFF \
-DVLLM_CPP_CUDA=OFF
- name: Build
# Same bounded parallelism as build-test-cpu: a bare -j lets Make link
# every test executable at once and OOM-kills the runner (ld signal 9).
run: cmake --build build-arm-full -j 2
- name: Enumerate the suite, and refuse a selection that matched nothing
id: enumerate
# A ctest invocation that selects ZERO tests exits 0. A wrong build
# directory, a failed configure or a stray filter would therefore
# publish this lane as green while executing nothing at all. This step
# states, in words and in its own output, WHAT it counted and refuses a
# zero. The floor is not a stored number: it is derived here and
# compared against the run below, so no future test addition touches it.
run: |
set -u
ctest --test-dir build-arm-full -N > ctest-enumeration.txt
rc=$?
cat ctest-enumeration.txt
if [ "${rc}" -ne 0 ]; then
echo "REFUSED: ctest -N itself failed (rc=${rc})" >&2
exit 1
fi
count=$(sed -n 's/^Total Tests: \([0-9][0-9]*\)$/\1/p' ctest-enumeration.txt | tail -1)
echo "ctest -N enumerated '${count:-}' CTest entries in build-arm-full"
if [ -z "${count:-}" ] || [ "${count}" -eq 0 ]; then
echo "REFUSED: ctest enumerated no tests, and a ctest that selects" >&2
echo "nothing exits 0. This lane certifies nothing in that state." >&2
exit 1
fi
echo "count=${count}" >> "$GITHUB_OUTPUT"
- name: Test
# SERIAL, deliberately. This tree has tests that starve under `ctest -j`
# and .agents/verification.md requires a serial re-run before calling
# such a failure a regression. A lane whose first duty is to find
# genuine aarch64 defects must not add a second source of red.
#
# No pipe: `$?` after `cmd | tee` reports tee, not ctest.
run: |
set -u
ctest --test-dir build-arm-full --output-on-failure > ctest-run.txt 2>&1
status=$?
cat ctest-run.txt
ran=$(sed -n 's/^.*tests passed, .* out of \([0-9][0-9]*\)$/\1/p' ctest-run.txt | tail -1)
echo "ctest EXECUTED '${ran:-}' entries; ctest -N had enumerated '${{ steps.enumerate.outputs.count }}'"
if [ "${ran:-0}" != "${{ steps.enumerate.outputs.count }}" ]; then
echo "REFUSED: the run and the enumeration disagree, so the number" >&2
echo "of tests this lane actually executed is not the suite." >&2
exit 1
fi
exit "${status}"
sanitize-cpu:
# The DYNAMIC detector lanes (VLLM_CPP_SANITIZE, see CMakeLists.txt). The
# plain build-test-cpu job above proves the suite PASSES; it cannot see a
# read one past a tensor row, a use-after-free of an engine-owned container,
# a signed-overflow index computation, or an unsynchronized access between
# the serving threads — every one of which stays green under -O2 until it
# silently corrupts a token stream. These jobs run the SAME suite with
# ASan+UBSan and with TSan.
#
# Two jobs, not one: the runtimes are mutually exclusive. Both are CPU-tier
# (the lane refuses to configure with the CUDA backend on) and both are
# `continue-on-error` for their FIRST landing so a pre-existing finding
# cannot block unrelated work — the finding is triaged, then this flag is
# removed and the lane becomes binding. Removing it is tracked as the
# closing step of the hardening-adoption row.
strategy:
fail-fast: false
matrix:
lane: ["address,undefined", "thread"]
continue-on-error: true
# Tree-scoped: cancellable. `matrix.lane` MUST be part of the group key, or
# the two mutually-exclusive sanitizer legs of the SAME run would land in one
# group and cancel each other, leaving only whichever started second.
concurrency:
group: ci-sanitize-cpu-${{ matrix.lane }}-${{ github.event_name }}-${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && github.run_id || github.ref }}-${{ github.repository }}
cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
# Skip when the pull request was CLOSED: the run exists only to enter
# the concurrency group and supersede the in-flight one (#822).
if: github.event.action != 'closed'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure
run: |
cmake -S . -B build-sanitize \
-DVLLM_CPP_BUILD_TESTS=ON \
-DVLLM_CPP_CUDA=OFF \
-DVLLM_CPP_SANITIZE='${{ matrix.lane }}'
- name: Build
run: cmake --build build-sanitize -j 2
- name: Test
# ctest runs the suite serially: the sanitizer runtimes multiply peak RSS,
# and a parallel run OOM-kills the runner before it reports a finding.
env:
UBSAN_OPTIONS: print_stacktrace=1
ASAN_OPTIONS: detect_leaks=1:strict_string_checks=1
# The production DevicePool deliberately retains scratch blocks. Its
# detector lane uses exact allocations and real frees so ASan can
# distinguish that cache from a leak and can see use-after-free.
VT_POOL_BYPASS: "1"
run: ctest --test-dir build-sanitize --output-on-failure
macos-metal-mlx:
# THE ONLY BUILD OF THE FOUR METAL TRANSLATION UNITS OUTSIDE A RELEASE
# (GATE-METAL-MLX-COMPILE wave 2, issue #1765), and since #1692 the only
# EXECUTION of the Metal/MLX op-provider routing outside one either.
#
# `src/vt/metal/metal_{context,backend,ops}.mm` are named in CMakeLists.txt
# and resolve into NO build rule anywhere else: on a configured Linux tree at
# be432e8e3 each appears 0 times in `build.ninja` and 0 times in
# `compile_commands.json`, against 6 for `src/vt/op_provider.cpp` as a
# control. They are Objective-C++ against Metal.framework, so no C++ compiler
# reaches them -- `-x c++` with the wave-1 stub include path fails with 22 /
# 46 / 166 errors, `metal_context.mm` on `stray '@' in program` -- and the
# wave-1 OBJECT-library technique therefore does not extend to them.
#
# Until this job, the only compiler that ever read them was `metal_arm64` in
# .github/workflows/release.yml, which fires on a `v*` tag or a manual
# dispatch. Between the last successful release build (v0.0.2, 7020de936,
# 2026-08-11) and be432e8e3 that left 955 commits unbuilt, 2 of them editing
# these TUs directly and 28 editing seam headers they include (vt/ops.h 23,
# vt/backend.h 4, vt/dtype.h 1). A break presented as a blocked release
# rather than as a red check; #1584 is the case that proved it.
#
# POST-MERGE BY DESIGN, and the trade is recorded rather than glossed. No
# `pull_request`: a macOS runner is roughly 10x a Linux one and this repo
# takes ~55 pushes/day, so the pre-merge lane would pay that on every
# contributor push for a surface only Apple hardware can compile. On `push`
# to `main` plus the 4-hourly baseline the exposure window is at most the
# commits since the last completed run instead of "until somebody cuts a
# release" -- 955 commits down to roughly one. What this job CANNOT do is
# stop a break from landing; it names it one commit later.
#
# MLX is ON because it is nearly free here and covers a surface nothing else
# can. Measured on release run 31466516224: `metal_arm64` 5m59s,
# `mlx_arm64` 6m09s -- one `pip install` and ~10s buys the fourth TU
# compiled against the REAL mlx headers, which the wave-1 stubs are blind to
# by construction (src/vt/metal/stubs/README.md). A pip failure reds this job
# under its own step name, so it can never be read as a verdict on the code.
#
# AND IT EXECUTES, since #1692. The open question was whether a `macos-15`
# runner exposes a Metal device at all -- `MlxSupports` gates on
# `MetalContext::Available()`, and a device-less runner registers nothing and
# would SKIP rather than exercise the provider, which is a pass wearing a
# skip's face. It is answered, from the log of `mlx_arm64` in release run
# 31466516224 (job 93700466685, macos-15-arm64 image 20260727.0256.1,
# 2026-08-11): `test_metal_backend` ran 25 cases / 112327 assertions /
# `Status: SUCCESS!` with 3 skipped, and those 3 are the file's own
# `doctest::skip(true)` benchmarks, not a device guard. The GEMM case printed
# real MLX numerics -- `mlx-vs-cpu=1.66168e-13` and `mlx-vs-msl=1.48631e-13`
# on the square f32 arm -- so MLX genuinely computed on a genuine device.
# What that run could NOT cover is the #1584 repair, which landed in
# 944d7d947, after it.
#
# THIS JOB'S FIRST RUN WAS RED, ON #1823, and that is what a lane which had
# never executed anything is for. It reported `test_metal_backend.cpp:170`
# THREW `"No valid attention backend for device type 2 from {FLASH_ATTN:
# [compute capability not supported]}"`, because
# `FlashAttentionBackend::supports_compute_capability` is upstream's NVIDIA
# `>= (8,0)` while `MetalPlatform::get_device_capability` answered with the
# Apple GPU FAMILY. Red since 369ea7fd4 (2026-08-19), which is not an
# ancestor of 7020de936, so nothing could see it for four days. It was
# deliberately NOT hidden behind `continue-on-error`, a skip or a tuned
# floor, and the exactness step runs FIRST so that row's own gate was not
# hostage to it. FIXED by `.agents/specs/attn-capability-unit.md`: the Metal
# platform now reports an ABSENT capability, mirroring upstream's own answer
# for a foreign capability format (xpu.py:228-236), so the SM predicate is
# skipped rather than misapplied. The four days this went unseen are the
# exposure window this job exists to close, measured.
concurrency:
group: ci-macos-metal-mlx-${{ github.event_name }}-${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && github.run_id || github.ref }}-${{ github.repository }}
cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
# The closed-PR clause is carried directly even though `pull_request` is
# already excluded: a job whose `if:` never names `closed` cannot be
# shown to execute no gate on a closed pull request (#822, #874), and
# the proof is the shape rather than the reasoning.
if: github.event.action != 'closed' && (github.event_name == 'push' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch')
permissions:
contents: read
runs-on: macos-15
timeout-minutes: 60
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.13'
- name: Install the exact redistributable MLX runtime
run: python3 -m pip install --disable-pip-version-check 'mlx==0.32.0'
- name: Compile every Metal translation unit against the real SDK
# The configure flags are `scripts/build-macos-release.sh`'s, minus the
# release-only ones: same generator, same build type, same backend
# tri-states, same MLX_ROOT resolution. `BUILD_TESTS=ON` and the second
# `--target` are the whole difference -- this lane compiles and runs the
# Metal suite, it does not package.
run: |
mlx_root=$(python3 -c 'import importlib.metadata as m; d=m.distribution("mlx"); print(d.locate_file("mlx"))')
test -f "$mlx_root/lib/libmlx.dylib"
cmake -S . -B build-metal -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DVLLM_CPP_METAL=ON \
-DVLLM_CPP_MLX=ON \
-DMLX_ROOT="$mlx_root" \
-DVLLM_CPP_BUILD_TESTS=ON \
-DVLLM_CPP_BUILD_EXAMPLES=OFF \
-DVLLM_CPP_SERVER=OFF \
-DVLLM_CPP_CUDA=OFF \
-DVLLM_CPP_HIP=OFF \
-DVLLM_CPP_TRITON=OFF \
-DVLLM_CPP_VULKAN=OFF
cmake --build build-metal --target vllm test_metal_backend test_platform -j 3
- name: Every Metal translation unit produced an object
# The postcondition, not the exit status. A green build proves nothing
# if VLLM_CPP_METAL resolved OFF, if a TU left `target_sources`, or if
# the object layout moved: the compiler would then have read none of
# these files and this job would publish success for a lane covering
# nothing -- absence wearing success's face, which is the shape #274 and
# #503 were both filed about.
run: |
rc=0
for tu in metal_context metal_backend metal_ops metal_mlx_provider; do
obj="build-metal/CMakeFiles/vllm.dir/src/vt/metal/${tu}.mm.o"
if [ -f "$obj" ]; then
echo "compiled $obj"
else
echo "MISSING $obj"
rc=1
fi
done
exit $rc
- name: The MLX decline count is exact from the FIRST decline of a process
# FIRST OF THE THREE EXECUTION STEPS, and the order is the point. This is
# #1692's gate and it runs one case in its own process, so an unrelated
# red anywhere else in `test_metal_backend` cannot skip it -- which is
# exactly what happened on run 32668677681, where the suite step failed on
# issue #1823 and this step reported `skipped`.
# ISSUE #1692 O2, row KERNEL-ACCEL-PROVIDER-DECLINE-EXACT. `MlxFallback`
# in src/vt/metal/metal_mlx_provider.mm caches its fallback through
# `GetOpFallbackUncounted`; the counting spelling would add one decline
# of its own, but ONLY while the function-local static is cold. Reverting
# that one call site must red THIS step.
#
# ITS OWN PROCESS, and that is the guarantee -- not the line number. The
# case is also first by line order in the suite below, so the static is
# cold there too, but #1812's repair argues correctly that ordering is not
# a repair: `--order-by=file` is a default, `--order-by=rand` exists, and
# first is one file edit away from not-first. A dedicated process has no
# ordering to depend on. The CUDA arm of this row is the case that proves
# it matters: its full suite stayed green with the whole repair reverted,
# because an earlier case warmed the static.
#
# `--test-case=` SPLITS ITS ARGUMENT ON COMMAS and a filter that matches
# nothing prints `0 cases ran` and exits 0, so the case name carries no
# comma and the `test cases: 1 | 1 passed` line is asserted rather than
# the exit status.
run: |
set -o pipefail
rc=0
./build-metal/tests/test_metal_backend --no-colors=1 \
'--test-case=*EXACTLY one decline*' 2>&1 \
| tee metal-decline.log || rc=$?
echo "filtered run rc=${rc}"
grep -E 'test cases:|assertions:|Status:' metal-decline.log || true
echo "SKIP lines: $(grep -c 'SKIP' metal-decline.log || true)"
test "${rc}" -eq 0
grep -qE 'test cases:[[:space:]]+1[[:space:]]+\|[[:space:]]+1 passed \|[[:space:]]+0 failed' metal-decline.log
grep -qE 'assertions:[[:space:]]+[1-9][0-9]*[[:space:]]+\|' metal-decline.log
grep -qE 'assertions:.*\|[[:space:]]+0 failed' metal-decline.log
grep -qE 'Status: SUCCESS!' metal-decline.log
# The positive signal: the provider that declined names itself and the
# two readings agree. A green filtered run without this line would be
# measuring the native MSL path with MLX never registered.
grep -E 'MLX decline accounting:' metal-decline.log
grep -q 'MLX decline accounting: first=1 second=1 provider=mlx' metal-decline.log
- name: CTest resolves the per-case entry
# ORDER: before the full-suite step, because this asserts that
# tests/CMakeLists.txt's entry RESOLVES, and that question must not be
# answerable only on a day when every other case is green. `-R` names the
# per-case entry alone for the same reason -- while #1823 is open the
# `test_metal_backend` entry is red, and an assertion about registration
# would then be unrunnable rather than merely inconvenient.
#
# The entry's own exit status cannot report this: an `--test-case=`
# filter that matches nothing prints `0 cases ran` and `Status: SUCCESS!`,
# and CMake splits an UNQUOTED `COMMAND` argument on its spaces, which is
# exactly how this entry was written for one commit (744abeca3). So `-V`,
# and assert the doctest summary out of the ctest log.
run: |
set -o pipefail
rc=0
ctest --test-dir build-metal -R '^test_metal_backend_mlx_decline_exact$' \
-V -j 1 2>&1 | tee metal-ctest.log || rc=$?
echo "ctest rc=${rc}"
grep -E 'test cases:|assertions:|Status:|MLX decline accounting:' metal-ctest.log || true
test "${rc}" -eq 0
# `100% tests passed out of 1` -- and NOT `tests passed, 0 tests failed
# out of 1`, which is what CTest prints when something DID fail. Run
# 32670585579 failed this step on that exact wording while `ctest rc=0`
# and the entry `Passed`, so the assertion is written from the log this
# runner produces rather than from memory.
grep -qE '100% tests passed' metal-ctest.log
grep -qE 'Test #[0-9]+: test_metal_backend_mlx_decline_exact .*Passed' metal-ctest.log
grep -qE 'test cases:[[:space:]]+1[[:space:]]+\|[[:space:]]+1 passed' metal-ctest.log
grep -q 'MLX decline accounting: first=1 second=1 provider=mlx' metal-ctest.log
- name: Execute the Metal suite on the runner's Metal device
# A COMPILE IS NOT A RUN, and this repository has already shipped a green
# gate that executed nothing. So the exit status is not the verdict here:
# doctest exits 0 after printing `assertions: 0 | 0 passed` and
# `Status: SUCCESS!`, which is what a device-less runner would produce
# once every case guarded itself away. The floors below are what separate
# that from a run. Measured baseline, release run 31466516224 job
# 93700466685: 25 cases, 112327 assertions, 3 skipped, SUCCESS!. The
# three skips are the file's own `doctest::skip(true)` benchmarks
# (`Metal strided-read bandwidth probe`, `Metal GEMM microbenchmark`,
# `Metal GEMM per-row diagnostic`); the count is echoed, not gated, so
# adding a benchmark does not red the lane, while a case that starts
# skipping itself shows up in the `passed` floor.
run: |
# `rc=0; ... || rc=$?` rather than `rc=$?` on the next line: the step
# shell is `bash -e`, so a bare failing pipeline would exit BEFORE the
# status could be printed, and the log would carry no rc at all.
# `--no-colors=1` keeps ANSI escapes out of the summary lines the greps
# below read.
set -o pipefail
rc=0
./build-metal/tests/test_metal_backend --no-colors=1 2>&1 \
| tee metal-suite.log || rc=$?
echo "test_metal_backend rc=${rc}"
grep -E 'test cases:|assertions:|Status:' metal-suite.log || true
echo "SKIP lines: $(grep -c 'SKIP' metal-suite.log || true)"
# The parser runs BEFORE the rc check on purpose: it prints the numbers,
# so a red run still leaves the counts in the log and the floors are
# exercised rather than skipped past. Its own exit status is a gate too.
python3 - <<'EOF'
import re, sys
log = open('metal-suite.log', encoding='utf-8', errors='replace').read()
cases = re.search(r'test cases:\s*(\d+)\s*\|\s*(\d+) passed \|\s*(\d+) failed \|\s*(\d+) skipped', log)
asserts = re.search(r'assertions:\s*(\d+)\s*\|\s*(\d+) passed \|\s*(\d+) failed', log)
if not cases or not asserts:
sys.exit('no doctest summary in the log: the suite did not run')
total, passed, failed, skipped = (int(x) for x in cases.groups())
a_total, a_passed, a_failed = (int(x) for x in asserts.groups())
print(f'cases total={total} passed={passed} failed={failed} skipped={skipped}')
print(f'assertions total={a_total} passed={a_passed} failed={a_failed}')
rc = 0
# Floors, not equalities: growth is normal, collapse is the defect.
if failed or a_failed:
print('FAIL: a case or assertion failed'); rc = 1
if passed < 25:
print(f'FAIL: only {passed} cases ran; 25 ran on 2026-08-11'); rc = 1
if a_passed < 100000:
print(f'FAIL: only {a_passed} assertions ran; 112327 ran on 2026-08-11'); rc = 1
sys.exit(rc)
EOF
test "${rc}" -eq 0
grep -qE 'Status: SUCCESS!' metal-suite.log
- name: Platform seam gate, with kMETAL REGISTERED
# #1823. `test_platform` asserts the capability-UNIT contract
# (`Platform::get_device_capability()` is an NVIDIA SM value, or it is
# ABSENT) for every platform REGISTERED in the build it runs in. On
# build-test-cpu that is kCPU alone, and on build-test-vulkan it is kCPU
# plus kVULKAN. Without this step the contract had NO gate on the Metal
# tier at all, and the case's own comment claimed otherwise -- the
# forward-protection claim ("a platform added later is covered with no
# edit") was false for the one accelerator tier that is not Vulkan.
#
# The grep is the point, for the same reason as everywhere else in this
# job: a device-less runner registers no Metal platform, the walk covers
# kCPU only, and the executable still prints `Status: SUCCESS!` with rc 0.
run: |
set -euo pipefail
./build-metal/tests/test_platform --no-colors=1 2>&1 | tee platform-gate.log
grep -q 'platform metal get_device_capability()' platform-gate.log
grep -q 'Status: SUCCESS!' platform-gate.log
baseline-summary:
# THE PUBLISHED VERDICT (issue #274, spec .agents/specs/main-verifiability.md).
#
# Baseline lane ONLY. It can never run on a `pull_request` or a `push`, so it
# blocks no contributor and changes nothing about what a PR must pass.
#
# It fails when any covered job is red, `sanitize-cpu` INCLUDED. That job is
# `continue-on-error: true` for the push/PR lanes and stays that way -- its
# removal is the closing step of the hardening row, not this one. What this
# job refuses is a BASELINE that reports green while a job it covered is red,
# which would rebuild the very defect #274 was filed about.
#
# It reads per-job conclusions from the Actions API for its own run, NOT from
# `needs.<job>.result`: `continue-on-error` makes that context report
# `success` for a failed job, which is the trap the whole row is about. Run
# 31448896841 is the live proof -- run conclusion `success`, both sanitizer
# lanes `failure`.
#
# On the day this lands the first run is EXPECTED RED, on the six sanitizer
# failures of #274 finding #1 / #301. That is the correct first verdict. A
# baseline that hid them would certify nothing.
#
# The same applies to `windows-msvc-cpu`, joined here on 2026-08-17 (#503).
# It is red on #584 -- `test_openai_api_server.exe` fast-fails with
# 0xC0000409 -- so the baseline reports RED from the moment it can see that
# lane at all. It reported GREEN before only because it never ran it.
if: always() && github.event.action != 'closed' && (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch')
needs:
- agent-record
- cuda-arch-features
- cuda-fat-build
- vulkan-spirv-freshness
- build-test-vulkan
- device-leakage
- windows-msvc-cpu
- windows-msvc-vulkan
- build-test-cpu
- build-test-cpu-arm64
- build-test-cpu-arm64-full
- macos-metal-mlx
- sanitize-cpu
permissions:
contents: read
actions: read
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Publish the per-job baseline verdict for this SHA
env:
GH_TOKEN: ${{ github.token }}
run: |
python3 scripts/main-baseline.py \
--run-id "${{ github.run_id }}" --emit-summary