fix(FIX-NAS-PATH-1073): resolve every live checkpoint default through CHECKPOINT_ROOT

.agents/environment.md

@@ -66,6 +66,23 @@ environment:
   never share a build tree between agents.
   - Non-interactive SSH does not put nvcc on PATH — prepend
     `export PATH=/usr/local/cuda/bin:$PATH` in remote build commands.
+  - **The NAS mounts at `/usr/local/nas_share`, and `/mnt/nas_share` is GONE
+    (re-verified 2026-08-16).** `.env` sets
+    `CHECKPOINT_ROOT=/usr/local/nas_share/checkpoints`, where 18 checkpoint
+    directories resolve, `nemotron-3.5-lightning-30b-nvfp4` and
+    `nemotron-3.5-lightning-30b-gguf` among them. **Do not restore the old path
+    as a convenience symlink.** `/mnt` is on the EPHEMERAL root overlay of this
+    immutable Kairos OS, so anything created there is gone after the next
+    reboot; `/usr/local` is `COS_PERSISTENT` and survives. That is the same
+    property that made an earlier `/oem` `rootfs`-stage change cost a boot (see
+    [[kairos-oem-rw-paths-change-cost-a-boot]]). Measured 2026-08-16, after the
+    box returned from an 8 h 19 min outage: the mount itself came back because
+    the `/oem` boot-stage unit worked and `findmnt /usr/local/nas_share` was
+    clean, and `/mnt/nas_share` did not come back. Every path built on `/mnt`
+    broke while `.env` still declared it, which blocks a checkpoint-loading gate
+    silently — a gate that reads a path `.env` does not declare is not the gate
+    its spec names. Check `findmnt /usr/local/nas_share` before you conclude
+    that a checkpoint is missing (#1073).
   - **MANDATORY gate-build flags on this box (re-proven 2026-07-29).** A model
     gate configured WITHOUT `-DVLLM_CPP_CUTLASS_DIR=$HOME/cutlass-4.5.0` and
     `-DVLLM_CPP_TRITON=ON` is NOT the production stack: cutlass-off silently

.agents/issue-index.md

@@ -306,3 +306,6 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#989](https://github.com/mudler/vllm.cpp/issues/989) | `VT-FP8-QUANT-ARCH-GATE` | `scripts/check-pr-size.py`'s `classify_path` has no entry for `.agents/reachability.md` (added by `POLICY-NOTHING-LANDS-DEAD`, [#888](https://github.com/mudler/vllm.cpp/issues/888) @ `8f49ac3be`), and it FAILS CLOSED, so `pr-size` — a REQUIRED check — refuses every pull request that touches that guide, and `tests/scripts/test_check_pr_size.py` has been red on `main` ever since. Red SILENTLY: that suite is wired into no CI job and is not in `agent-preflight.sh`'s `SUITES`, so the only thing that ever loads it is `check-pr-size`'s own executable-evidence contract, which fires only when a PR edits a checker — the red is reachable exclusively by the next person who must touch that file, and presents to them as their own breakage (the [#584](https://github.com/mudler/vllm.cpp/issues/584)/[#965](https://github.com/mudler/vllm.cpp/issues/965) shape). Third instance of the class after [#856](https://github.com/mudler/vllm.cpp/issues/856) (`issue-index.md` + the style guides) and [#668](https://github.com/mudler/vllm.cpp/issues/668) (`.agents/oracles/*`), both fixed in flow by the row that tripped over them. FIXED IN FLOW while landing [#960](https://github.com/mudler/vllm.cpp/issues/960), which could not register its new checker's creation mutation without touching `check-pr-size.py` at all. NOT fixed: wiring that suite into CI, which is its own change and would red `main` until this landed | bug |
 | [#1066](https://github.com/mudler/vllm.cpp/issues/1066) | `ENG-EXPERT-STREAM` | `Qwen35ExpertStream` (`src/vllm/model_executor/models/qwen3_5.cpp`) is a **process-lifetime singleton** and keyed its slot cache on `(TowerId(base), expert)`, where `base` is the expert tower's host buffer **ADDRESS**. Its own comment stated the premise and drew the wrong conclusion: "A tower's identity is its base pointer, which is stable for the model's life". The premise is true; the conclusion does not follow, because the CACHE is not scoped to one model's life. Free a model, load another, and the allocator hands the new towers addresses the old ones held, so the new model's expert resolves to an entry filled from a DIFFERENT checkpoint — returned as a HIT, which by contract moves no bytes, so no counter moves and nothing downstream has anything to observe. MEASURED on two synthetic 4-layer/4-expert MoE models in one process, instrumenting `KqExpertSlice` to `memcmp` each returned slot against the tower slice it claims to be: **24 towers occupied 21 distinct addresses, and 20 of 222 slices returned another tower's bytes**; end to end the two arms disagreed on all 160 logits while each arm was internally deterministic (0 differing values on a repeat), which rules out nondeterminism. Invisible to every existing test of this row by construction, because all of them build the cache, store and streamer by hand and none runs two models through the production seam. Reachable by any process that loads a model, releases it, and loads another. Fixed by `OwnedTensor::TowerUid()`, a lazily assigned process-unique counter stamped on the tensor and re-stamped when `bytes` moves (so a copy cannot inherit an identity along with a different buffer); a counter cannot collide because it never goes backwards. Found and fixed while repairing the F1-F11 wiring review for [#912](https://github.com/mudler/vllm.cpp/issues/912). Spec [`expert-streaming.md`](specs/expert-streaming.md) | bug |
 | [#1088](https://github.com/mudler/vllm.cpp/issues/1088) | `LTX25-RESOLUTION-ENVELOPE` | `docs/USAGE.md` published "Measured to complete on one GB10 = 320x192 at 25 frames" beside "Measured NOT to complete = 448x256 at 25 frames, the denoise finishes, then the decode loses about 59 GB in 24 s", and `docs/BENCHMARKS.md` said the same. Two renders on `dgx.casa` on 16 to 17 August 2026 against `main` `0b0b8900f`, which carries [#1041](https://github.com/mudler/vllm.cpp/issues/1041) threaded decode, [#1032](https://github.com/mudler/vllm.cpp/issues/1032) T2A and [#1036](https://github.com/mudler/vllm.cpp/issues/1036) f32 decode accumulators, in container `vllmcpp-build:gb10`, `Release`, `VLLM_CPP_CUDA=ON`, arch `121a`, `TRITON=ON`, CUTLASS absent so FlashAttention-2 was not built, `VLLM_CPP_CPU_THREADS=20`, NVFP4 transformer, no `--allow-unported`, **COMPLETED 448x256/25f in 3085 s and 704x448/25f in 4231 s**. 1024x576/25f was not attempted to completion because another session claimed the box. That is scheduling and not an envelope, so 704x448 is NOT a ceiling. The ~59 GiB cliff did NOT recur under a 2 s memory guard that would have seen it. `MemAvailable` floors were 38.96 GiB over 1289 samples at 448x256 and 38.89 GiB over 1743 samples at 704x448, with ZERO samples under 34 GiB on either, peak use of 80 of 119 GiB, and no reboot. The 704x448 artifact was verified rather than inferred from an exit code: 25/25 distinct frame md5s, 0 near-uniform and 0 near-black frames, adjacent-frame mean absolute difference 4.381 against a uniform-noise reference of 85.3 on the same shape, 0/24 zero-motion pairs, and audio at 48 kHz stereo, 1.010 s, -37.29 dBFS, 20/20 windows above threshold. NOT claimed: ONE run per geometry on a contended shared box with no oracle on either side, so two points establish no scaling law. The 59 GB observation is KEPT and attributed to its own run, which is rung F1 in `benchmark-record.md`, a prompt-embeds render with no text tower that an ARMED WATCHDOG ended at 13.77 GiB against an 18 GiB floor rather than the engine failing. Attributing it stays open as [#1014](https://github.com/mudler/vllm.cpp/issues/1014). The claim that most of a 320x192/25f render is spent in the host VAE decode is superseded too: after #1041 a resolution-independent ~1731 s single-threaded phase, measured at 1731 s and 1732 s across two rungs whose voxel counts differ 2.75x, is 57 to 66% of wall, filed as [#1087](https://github.com/mudler/vllm.cpp/issues/1087). FIXED IN FLOW in `docs/USAGE.md`, `docs/BENCHMARKS.md` and [`ltx25-resolution-envelope.md`](specs/ltx25-resolution-envelope.md) section 4.1 | bug |
+| [#1073](https://github.com/mudler/vllm.cpp/issues/1073) | `FIX-NAS-PATH-1073` | The NAS moved to `/usr/local/nas_share` and `/mnt/nas_share` is gone, so every tracked default built on `/mnt` broke. `/mnt` is the EPHEMERAL root overlay of the gate box's immutable Kairos OS and does not survive a reboot; `/usr/local` is `COS_PERSISTENT` and does. Observed 2026-08-16 after an 8 h 19 min outage: the mount came back because the `/oem` boot-stage unit worked, `/mnt/nas_share` did not, and the untracked `.env` still declared `CHECKPOINT_ROOT=/mnt/nas_share/checkpoints` — a gate that reads a path `.env` does not declare is not the gate its spec names. `.agents/environment.md` documented NO NAS location at all (measured: the file held no `/mnt` string), so the repair adds the path AND the `COS_PERSISTENT` reason, because a bare path correction invites the next reader to restore the dead location as a symlink. The seven live defaults now derive from `CHECKPOINT_ROOT`, which four sibling scripts already did: `scripts/gen-minimax-music3-manifest.py:17`, `scripts/gen-ltx2-quant-goldens.py:48`, `tools/parity/dump_tokenizer_gpt4o.py:36,39,57`, `tools/gen_pretok_goldens.py:57`, `src/vllm/tokenizer/pretokenizer.cpp:319`, `tests/parity/test_minimax_music3_quant_real.cpp:133,144` and `docs/USAGE.md:3069,3453`. The 41 hits were classified before any edit and the records that cite the old path KEEP it: `.agents/benchmark-record.md`, the LTX-2.5/Nemotron-H specs, `.agents/model-matrix.md`, the captured goldens and the generated `.inc` headers state where a past measurement read its bytes, which is provenance. Spec [`nas-mount-path.md`](specs/nas-mount-path.md) | bug |
+| [#1077](https://github.com/mudler/vllm.cpp/issues/1077) | — | `.env.example:37`, `.agents/environment.md:29` and `tests/vllm/multimodal/test_ltx2_video.cpp:2128-2132` each state that nothing in the tree reads `CHECKPOINT_ROOT`, and six gates read it: `tests/parity/test_minimax_music3_ar_real.cpp:162`, `_e2e_real.cpp:170`, `_llm_real.cpp:137`, `_quant_real.cpp:130,140`, `tests/vllm/models/test_ltx2_text_encoder.cpp:2299`, and `test_nemotron_h_loader.cpp:161` tells the reader to export it. No product code under `src/` or `include/` reads it, so the accurate statement is that the LIBRARY never reads it while several gates do. It costs more than tidiness: `test_ltx2_video.cpp` reasons FROM the claim when it chooses a separate `LTX2_CHECKPOINT_ROOT` ("this would be its first reader"), and that reasoning is void. Found while repairing [#1073](https://github.com/mudler/vllm.cpp/issues/1073) and NOT fixed there, because reversing a design decision needs its own review rather than a path substitution. Listed under `## Owed` in [`nas-mount-path.md`](specs/nas-mount-path.md) | bug |
+| [#1079](https://github.com/mudler/vllm.cpp/issues/1079) | `FIX-NAS-PATH-1073` | All four skip messages in `tests/parity/test_minimax_music3_quant_real.cpp` streamed the case name as a `const char*`, and doctest 2.5.2 stringifies that through its bool overload, so every one printed `SKIP 1` and named no case. The comment above the helpers states the obligation the messages then failed: a gate that silently passes when its asset is absent has not reported. It matters here because the binary reports `6 passed` with `assertions: 0` when the checkpoint is absent, so the message text is all that separates a skipped run from a gated one. Pre-existing on `main` at `100026481`. FIXED IN FLOW while landing [#1073](https://github.com/mudler/vllm.cpp/issues/1073), which rewrote those exact messages and would have carried the defect forward under a changed line; the fix streams `std::string(what)`. Scope measured before fixing: 4 hits, all in this one file | bug |