| `BACKEND-GATE-METAL-MLXLM` | **MLX is the NAMED COMPETITOR FLOOR for Metal (USER DIRECTIVE 2026-07-22: "we should do benchmark testing against MLX for the same model").** Binds exactly as AGENTS.md Acceptance rule defines for vLLM on CUDA: SAME model, SAME workload, SAME box, ours >= MLX on EVERY axis (total + output throughput, req/s, TTFT, TPOT/ITL, peak memory), correctness a precondition never traded off | [MLX-LM](https://github.com/ml-explore/mlx-lm), commit pinned per run | - | **NOT YET RUNNABLE — nothing to benchmark: the Metal backend is a W0 SKELETON with no model** (`BACKEND-METAL-MLX`). No number is claimed or owed. **Do NOT conflate MLX's two roles:** as an IMPLEMENTATION path it stays DEMOTED to work row M5 (bring-up is native MSL, zero installs); as a BENCHMARK ARM it is binding from the first Metal model onward. **HARD CONSTRAINT ON M3:** the first Metal bring-up model must be one MLX can also run, or the arms are incomparable — this composes with the spike's "OPT or Qwen3-dense first, NEVER Qwen3.5-Next", and since MLX-LM ships Qwen3 dense support, **Qwen3-dense is the pairing satisfying both**. **M4 readiness probed 2026-07-22 (read-only, NOT installed):** `brew info mlx` = 0.32.0, not installed, sole dep `python@3.14`; system python `/usr/bin/python3` 3.9.6 (CLT) with `venv` present; `import mlx` fails; ~28 GiB free. Deliberately NOT installed during W0 because the brew formula puts `python@3.14` in `/opt/homebrew/bin`, first on the PATH our macOS builds use, which would change `find_package(Python3)` and perturb the very gates W0 was proving. **Recommended (venv, avoids that hazard):** `python3 -m venv ~/mlx-venv && ~/mlx-venv/bin/pip install -U pip mlx-lm`; record the exact resolved version — an unpinned competitor arm is not a floor. **Any M4 timing is VOID unless the `com.localai.worker` root LaunchDaemon is booted out first** (needs interactive sudo; commands in [environment.md](environment.md)). **MLX INSTALLED AND AN UNOPPOSED BASELINE MEASURED 2026-07-22** ([study §7](specs/metal-mlx-reuse-study.md), [BENCHMARKS](../docs/BENCHMARKS.md)) — venv route as recommended, brew NOT used, so `python@3.14` never touched the build PATH. Resolved **`mlx` 0.29.3 / `mlx-metal` 0.29.3 / `mlx-lm` 0.29.1** (CLT python 3.9.6 caps the resolve below brew's 0.32.0). Model **`mlx-community/Qwen3-1.7B-bf16` @ `9cd6692855d3e06772228e9a962b2606359b2d24`** — 1.7B not 4B/8B because the desktop session already holds ~13 of 17.2 GB and a b=16 arm must not page. Harness is **MLX-LM's OWN** `python -m mlx_lm.benchmark`, p=512 g=128, 1 warmup + 3 timed trials. **MLX numbers (aggregate over batch):** b=1 27.57 tok/s decode / 470 ms TTFT / 36.3 ms ITL / 3.78 GB peak; b=2 48.91 / 900 ms / 40.9 ms / 3.97 GB; b=4 90.15 / 1738 ms / 44.4 ms / 4.18 GB; b=8 156.95 / 3416 ms / 51.0 ms / 4.47 GB; b=16 **213.39** / 6857 ms / 75.0 ms / 5.28 GB. Prefill saturates ~1200 tok/s by b=8 (compute roof); decode scales **7.74x** from b=1 to b=16 (bandwidth-bound, weight traffic amortized). Trial spread **0.12%-0.63%** across all five arms. **STATUS = `BLOCKED-ON-SUDO`, INDICATIVE NOT BINDING:** the worker could not be stopped (`sudo -n true` -> "a password is required"), so per the standing contended-run rule these may NOT be cited as a bound floor — they are a DESIGN TARGET plus an exact re-run recipe. The worker was nevertheless MEASURED genuinely idle (0.0% CPU; `ioreg` Device/Renderer/Tiler Utilization all **0**), consistent with the sub-1% spread; a second unanticipated contender WAS found — the desktop aerial video wallpaper (`WallpaperAerialsExtension` 8.2% CPU + `VTDecoderXPCService`), which must also be disabled for a binding run. **NO "ours" column exists or was manufactured — no Metal model runs, and no Metal speed result is claimed. THE ROW STAYS `INVENTORIED` DELIBERATELY:** a competitor baseline is not a gate. There is no implementation code and no ours-vs-MLX evidence to anchor, so promoting the state would be exactly the ungrounded transition `scripts/check-agent-record.py` exists to reject — the MLX arm is now READY and PINNED, and the row moves only when a Metal model runs (work row `M3b`). **=== OURS-VS-MLX PRODUCED 2026-07-23 (M3b) — INDICATIVE / `BLOCKED-ON-SUDO`, and the row moves `INVENTORIED` -> `ACTIVE`. ===** Qwen3-dense now runs on Metal, so both arms exist for the SAME model. Both = Qwen3-1.7B bf16 (ours = `Qwen/Qwen3-1.7B` HF safetensors on our Metal engine, device=2 confirmed via `VT_OP_PROVIDER_STATS` — every op `vt-native`; MLX = `mlx-community/Qwen3-1.7B-bf16`), SAME box, SAME session, p=512 g=128, b∈{1,2,4,8,16}. **Ours (`vllm-bench`, Metal), decode tok/s per stream · TTFT · peak:** b=1 4.29 · 4.86 s · 7.36 GB; b=2 4.32 · 9.08 s · 7.36; b=4 4.16 · 17.66 s · 7.37; b=8 3.25 · 28.52 s · 7.64; b=16 2.14 · 47.17 s · 8.78. Aggregate output tok/s 3.71 -> 19.14. **MLX (re-run same session, matches the committed §7 baseline):** gen 27.77 -> 211.55 tok/s, TTFT 0.47 -> 7.16 s, peak 3.78 -> 5.28 GB. **VERDICT: ours LOSES on every axis — ~6-11x slower decode, ~7-10x slower TTFT, ~2x peak memory.** This is a knowingly-unoptimised FLOOR, NOT our best: one command buffer per op (commit+wait), a plain threadgroup-tiled GEMM with no simdgroup-matrix, no batched encoders (`M3c`) — vs MLX's `steel` kernels. It is the FIRST ours-vs-MLX number and it sets the optimisation target. **STILL INDICATIVE, NOT BINDING:** the Mac could NOT be quieted (`sudo -n true` -> password required; the root `com.localai.worker` daemon and the aerial wallpaper `WallpaperAerialsExtension` ~9.8% CPU are both up), so per the standing contended-run rule the numbers are void as a bound comparison — though memory pressure read 83% free and MLX's trial spread stayed sub-1%. **The row is `ACTIVE` (not `DONE`): a native-competitor gate is `DONE` only when a model runs token-exact AND at MLX-parity-or-better on a genuinely quiet box; ours meets neither the parity nor the quiet-box condition. Exact residual: close the ~6-11x decode / ~7-10x TTFT / ~2x memory gap (start with `M3c` + a simdgroup GEMM) AND re-run once the user boots out the daemon + wallpaper (commands in [environment.md](environment.md)).** **=== MLX-LM FLOOR RE-MEASURED 2026-07-27, AND IT REPRODUCES. ===** Same box, same model revision, byte-identical toolchain (`mlx` 0.29.3 / `mlx-metal` 0.29.3 / `mlx-lm` 0.29.1), MLX-LM's own harness, p=512 g=128 n=3, this time with the three `actions.runner` LaunchAgents verified job-idle, booted out and restored, and the sweep under the GPU lock. **gen tok/s: b=1 27.89, b=2 49.69, b=4 91.10, b=8 158.65, b=16 213.71**; prompt tok/s 1091.2 / 1139.5 / 1180.2 / 1200.2 / 1197.7; peak memory 3.776 / 3.974 / 4.184 / 4.466 / 5.279 GB. Against the committed 2026-07-22 baseline this is **+0.15% to +1.6% on generation, within 0.3% on prefill, and peak memory IDENTICAL TO THREE DECIMALS at every batch size** — the ~1% is consistent with the paused runners. Trial spread tightened to **0.05%-0.28%** (was 0.12%-0.63%). **The competitor floor is therefore REPRODUCED, not merely restated.** **OURS, SAME SESSION:** MLX-provider arm 5.79 / 10.19 / 15.75 / 25.70 / 38.65 aggregate tok/s; native MSL arm 3.08 / 6.08 / 10.24 / 13.69 / 17.69. **Gap vs MLX-LM: 4.8x-6.2x with the MLX GEMM provider, 8.2x-12.1x without.** TTFT 2.7x-7.1x behind, peak memory 1.4x-1.8x higher. **WHY THE GAP PERSISTS IS NOW ANSWERED BY MEASUREMENT, NOT NARRATIVE** ([Metal dispatch attribution](specs/metal-dispatch-attribution.md)): the per-op `commit`+`waitUntilCompleted` model costs ~186 us x ~395 dispatches per decode token, a **~13.6 tok/s hard ceiling with infinitely fast kernels** versus MLX-LM's measured 27.9. **This row cannot pass on kernel work alone; `M3c` (batched encoders) is the precondition.** MLX commits at the `eval()` boundary rather than the op boundary, paying the same round trip a handful of times per token where we pay it ~395 times. Row stays `ACTIVE`: still short on every axis, and still not a quiet box. | [competitive benchmark spike](specs/competitive-benchmarks.md); **[Metal dispatch attribution](specs/metal-dispatch-attribution.md)**; [Metal/MLX reuse study](specs/metal-mlx-reuse-study.md) | `ACTIVE` | `CLAIM-BACKEND-METAL-M3B-1` |
0 commit comments