Skip to content

gate(GATE-METAL-MLX-COMPILE): W2 — compile all four Metal TUs on the post-merge lane, against the real SDK (#1765) - #1792

Merged
localai-bot merged 4 commits into
mainfrom
row/GATE-METAL-MLX-COMPILE-W2
Aug 23, 2026
Merged

gate(GATE-METAL-MLX-COMPILE): W2 — compile all four Metal TUs on the post-merge lane, against the real SDK (#1765)#1792
localai-bot merged 4 commits into
mainfrom
row/GATE-METAL-MLX-COMPILE-W2

Conversation

@localai-bot

@localai-bot localai-bot commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

src/vt/metal/metal_{context,backend,ops}.mm are named in CMakeLists.txt and
resolve into no build rule anywhere. On a configured Linux tree at be432e8e3
each appears 0 times in build.ninja and 0 times in
compile_commands.json, against 3 for the wave-1 provider target and 6 for
src/vt/op_provider.cpp as a control.

The break goes undetected today, measured rather than argued

One #1584-class seam rename per file, all three at once: RegisterBackend( from
include/vt/backend.h (1 hunk), VT_CHECK( from include/vt/dtype.h (6),
RegisterOp( from include/vt/ops.h (19).

  • cmake --build build -j 12 returns rc 0 in 9m24.65s over 1667 edges, with
    zero error: lines. The build log mentions metal_backend 0 times,
    metal_context 0 and metal_ops 0, and metal_mlx_provider once, which is
    the wave-1 control proving the instrument works.
  • ctest --test-dir build -j 4: 593/595 passed, 3 skipped. The two reds
    (test_serve_low_tools, test_async_llm) pass on a serial re-run, so they are
    -j starvation under verification.md, and neither could be caused by a file
    in no build rule anyway.
  • Restored with git checkout -- plus touch; sha256sum -c OK on all three.

Wave 1's technique cannot be stretched over them, and that is measured too: with
the same -x c++ recipe and stub include path they fail with 22 / 46 / 166
errors, metal_context.mm on stray '@' in program. No stub fixes that. They
need an Objective-C++ front end, which the development host does not have
(cannot execute 'cc1objplus', no clang anywhere under /, /opt,
/usr/local or $HOME).

What lands

macos-metal-mlx in ci.yml is release.yml's proven metal_arm64 /
mlx_arm64 shape with the release-only parts removed: macos-15, checkout, the
same pip install 'mlx==0.32.0' and importlib.metadata resolution of
MLX_ROOT, scripts/build-macos-release.sh's configure flags, and
--target vllm. It compiles; it does not package or execute.

MLX is ON because the difference is ten seconds. On release run
31466516224, metal_arm64 took 5m59s and mlx_arm64 6m09s. That buys the
fourth TU compiled against the real mlx headers, which the wave-1 stubs are
blind to by construction, so O2 is discharged as well. A pip failure reds the
job under its own step name and can never read as a verdict on the code.

Post-merge deliberately. No pull_request: a macOS runner is roughly 10x a
Linux one and this repository takes ~55 pushes/day. On push to main plus the
4-hourly baseline, the exposure window falls from "until somebody cuts a
release"
— 955 commits since 7020de936 (v0.0.2, 2026-08-11), 2 of them editing
these TUs and 28 editing seam headers they include — to the commits since the
last completed run. What it gives up is stated as a limit: it names a break a
commit or two after it lands, not before. The wave-1 Linux syntax check stays,
because it is the only Metal signal a pull request gets and it is free.

Two things the job asserts about itself. The last step requires all four
.mm.o objects to exist and names the missing one, because a green build proves
nothing if VLLM_CPP_METAL resolved OFF or a TU left target_sources — absence
wearing success's face, the #274 / #503 shape. And the job joins
baseline-summary's needs: and main-baseline.py's EXPECTED_JOBS (count pin
11 → 12), because a compiling gate the published baseline never grades is #503
exactly. Dropping it from needs: reds
test_expected_jobs_is_pinned_against_the_workflow_needs_list, mutation-checked.

Measured on the lane itself

No Linux host can run this job, so its two halves are dispatched runs of
ci.yml, one probe branch per file, each carrying the shipped job and exactly
one mutated file.

Run Branch Mutation macos-metal-mlx
32647402016 this branch none success, 5m03s — compiled .../{metal_context,metal_backend,metal_ops,metal_mlx_provider}.mm.o, all four, after Successfully installed mlx-0.32.0 and -- MLX GEMM provider enabled: .../libmlx.dylib
32647406515 probe/metal-w2-red-backend RegisterBackend(, 1 hunk failuremetal_backend.mm:141:5: error: use of undeclared identifier
32650354752 probe/metal-w2-red-context VT_CHECK(, 6 hunks failuremetal_context.mm:44:5: error: use of undeclared identifier
32650358209 probe/metal-w2-red-ops RegisterOp(, 19 hunks failuremetal_ops.mm:1092:5: error: use of undeclared identifier

Every red fails at the compile step and skips the object-existence step, so
the compiler produced it and not the assertion. The same three breaks leave the
full Linux build and ctest green. On the pull-request run of this very change
(32647478561) the job is completed/skipped in 0 s with runner_name: null,
which is the post-merge design priced at zero.

The expected first red did not happen: 955 commits after the last macOS build,
main plus this change compiles all four TUs on the first attempt. The drift was
real exposure that had not yet become a break.

Nothing is stubbed here, so nothing is claimed that is not built

src/vt/metal/stubs/README.md is untouched and still exactly right about the
wave-1 Linux target. This job builds against the real Apple SDK, the real
Foundation and Metal headers, AppleClang's ObjC++ front end and the real mlx
wheel.

#1765's two owed items, O2 and O3, are discharged: all four TUs are compiled
against the real SDK and the real mlx wheel. It is deliberately NOT closed by a
keyword here, because what it buys is post-merge coverage rather than the
pre-merge lane its title asks for — that trade is the developer's recorded
decision C, and whoever merges this should close it knowingly.

Still owed, and unchanged by this: the runtime arms — test_metal_backend on a
real Metal device and test_ops_attention_cross on CUDA — which are #1692's and
stay open there.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

mudler added 4 commits August 23, 2026 14:57
…nner that compiles them is post-merge (#1765)

`§10` O3 recorded `metal_ops.mm`, `metal_backend.mm` and `metal_context.mm` as
owed and owned by nobody, and O2 left the MLX API to release time. This wave
takes both, on a `macos-15` job rather than a Linux one, and the spec section
says why that was the only technique left.

Measured at `be432e8e3`, before any of it: each of the three appears 0 times in
`build.ninja` and 0 times in `compile_commands.json`, against 3 for the wave-1
provider target and 6 for `src/vt/op_provider.cpp` as a control. Broken all
three at once with one seam rename each -- `RegisterBackend`, `VT_CHECK`,
`RegisterOp`, 1/6/19 hunks -- the full CPU build returns rc 0 in 9m24s with zero
`error:` lines and zero mentions of the three files, and `ctest` is green
(593/595; the two reds pass on a serial re-run). A break in these files is
invisible to everything that runs today.

Wave 1's technique cannot be extended to them, and that is measured rather than
argued: with the same `-x c++` recipe and stub include path they fail with 22,
46 and 166 errors, `metal_context.mm` on `stray '@' in program`. No stub fixes
that. They need an Objective-C++ front end, which the development host does not
have at all (`cannot execute 'cc1objplus'`, no clang anywhere).

The section costs the three designs and records the developer's choice of C: a
Linux clang gate would invent ~30 Apple selectors, 14 types and 5 constants and
compile a different ObjC dialect than AppleClang, for 3 of 4 TUs; the same macOS
job on every pull request buys pre-merge at ~10x a Linux runner on ~55 pushes a
day; the post-merge lane buys the same 4-of-4 coverage for neither. What it
gives up is stated as a limit rather than glossed -- it names a break a commit
or two after it lands, not before.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
…post-merge lane, against the real SDK (#1765)

`src/vt/metal/metal_{context,backend,ops}.mm` are named in `CMakeLists.txt` and
resolve into no build rule anywhere: 0 occurrences each in `build.ninja` and
`compile_commands.json` on a configured Linux tree, against 6 for
`src/vt/op_provider.cpp` as a control. Break all three with a seam rename and
the full CPU build still returns rc 0 and `ctest` still passes. The only
compiler that ever read them was `metal_arm64` in `release.yml`, which fires on
a `v*` tag -- 955 commits after the last one, 2 of them editing these files and
28 editing seam headers they include.

`macos-metal-mlx` is `release.yml`'s proven `metal_arm64`/`mlx_arm64` shape with
the release-only parts removed: `macos-15`, checkout, the same `mlx==0.32.0`
install and `MLX_ROOT` resolution, `build-macos-release.sh`'s configure flags,
and `--target vllm`. It compiles; it does not package or execute. MLX is ON
because run 31466516224 measured the difference at ten seconds (5m59s vs 6m09s)
and it buys the fourth TU against the real headers, which the wave-1 stubs are
blind to by construction.

POST-MERGE deliberately. No `pull_request`: a macOS runner is roughly 10x a
Linux one and this repository takes ~55 pushes/day. On `push` to `main` plus the
4-hourly baseline the exposure window is the commits since the last completed
run instead of "until somebody cuts a release". The wave-1 Linux syntax check
stays -- it is the only Metal signal a pull request gets, and it is free.

Two things the job asserts about itself. The last step requires all four
`.mm.o` objects to exist and names a missing one, because a green build proves
nothing if `VLLM_CPP_METAL` resolved OFF or a TU left `target_sources`; that is
absence wearing success's face, the #274/#503 shape. And the job joins
`baseline-summary`'s `needs:` and `main-baseline.py`'s `EXPECTED_JOBS` (count
pin 11 -> 12), because a compiling gate the baseline never grades is #503
exactly.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
… own file, one green compiling all four (#1765)

The gate cannot be run from a Linux host, so its two halves are dispatched runs
of `ci.yml` on one probe branch per file, each carrying the shipped job and
exactly one mutated file.

Green: run 32647402016 on this branch, `macos-metal-mlx` success in 5m03s, with
`compiled .../{metal_context,metal_backend,metal_ops,metal_mlx_provider}.mm.o`
for all four and `MLX GEMM provider enabled` against the installed
`mlx-0.32.0`. Red: 32647406515 `metal_backend.mm:141:5`, 32650354752
`metal_context.mm:44:5`, 32650358209 `metal_ops.mm:1092:5`, each `use of
undeclared identifier` on the renamed seam symbol, each `FAILED: [code=1]` on
its own object. Every red stops at the compile step and SKIPS the
object-existence step, so the compiler produced it rather than the assertion.

Two things the runs measured that were not asked for. The expected first red did
not happen: 955 commits after the last macOS build, all four TUs still compile,
so the drift was exposure that had not yet become a break. And a `macos-15`
runner took 45 and 50 minutes to allocate, which prices design B -- the same job
pre-merge -- in wall time on every contributor's pull request rather than only
in dollars.

`§11 Outcome` is recorded with the rejected designs and the reason each default
has its value.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
Routine integration before landing the macOS compile gate. No conflicts.

FOLLOWING_AGENTS_PROTOCOL

Issue: #1765

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants