diff --git a/.agents/environment.md b/.agents/environment.md index e884ade79..bd857e645 100644 --- a/.agents/environment.md +++ b/.agents/environment.md @@ -66,6 +66,23 @@ environment: never share a build tree between agents. - Non-interactive SSH does not put nvcc on PATH — prepend `export PATH=/usr/local/cuda/bin:$PATH` in remote build commands. + - **The NAS mounts at `/usr/local/nas_share`, and `/mnt/nas_share` is GONE + (re-verified 2026-08-16).** `.env` sets + `CHECKPOINT_ROOT=/usr/local/nas_share/checkpoints`, where 18 checkpoint + directories resolve, `nemotron-3.5-lightning-30b-nvfp4` and + `nemotron-3.5-lightning-30b-gguf` among them. **Do not restore the old path + as a convenience symlink.** `/mnt` is on the EPHEMERAL root overlay of this + immutable Kairos OS, so anything created there is gone after the next + reboot; `/usr/local` is `COS_PERSISTENT` and survives. That is the same + property that made an earlier `/oem` `rootfs`-stage change cost a boot (see + [[kairos-oem-rw-paths-change-cost-a-boot]]). Measured 2026-08-16, after the + box returned from an 8 h 19 min outage: the mount itself came back because + the `/oem` boot-stage unit worked and `findmnt /usr/local/nas_share` was + clean, and `/mnt/nas_share` did not come back. Every path built on `/mnt` + broke while `.env` still declared it, which blocks a checkpoint-loading gate + silently — a gate that reads a path `.env` does not declare is not the gate + its spec names. Check `findmnt /usr/local/nas_share` before you conclude + that a checkpoint is missing (#1073). - **MANDATORY gate-build flags on this box (re-proven 2026-07-29).** A model gate configured WITHOUT `-DVLLM_CPP_CUTLASS_DIR=$HOME/cutlass-4.5.0` and `-DVLLM_CPP_TRITON=ON` is NOT the production stack: cutlass-off silently diff --git a/.agents/issue-index.md b/.agents/issue-index.md index d32f263f6..865338249 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -306,3 +306,6 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#989](https://github.com/mudler/vllm.cpp/issues/989) | `VT-FP8-QUANT-ARCH-GATE` | `scripts/check-pr-size.py`'s `classify_path` has no entry for `.agents/reachability.md` (added by `POLICY-NOTHING-LANDS-DEAD`, [#888](https://github.com/mudler/vllm.cpp/issues/888) @ `8f49ac3be`), and it FAILS CLOSED, so `pr-size` — a REQUIRED check — refuses every pull request that touches that guide, and `tests/scripts/test_check_pr_size.py` has been red on `main` ever since. Red SILENTLY: that suite is wired into no CI job and is not in `agent-preflight.sh`'s `SUITES`, so the only thing that ever loads it is `check-pr-size`'s own executable-evidence contract, which fires only when a PR edits a checker — the red is reachable exclusively by the next person who must touch that file, and presents to them as their own breakage (the [#584](https://github.com/mudler/vllm.cpp/issues/584)/[#965](https://github.com/mudler/vllm.cpp/issues/965) shape). Third instance of the class after [#856](https://github.com/mudler/vllm.cpp/issues/856) (`issue-index.md` + the style guides) and [#668](https://github.com/mudler/vllm.cpp/issues/668) (`.agents/oracles/*`), both fixed in flow by the row that tripped over them. FIXED IN FLOW while landing [#960](https://github.com/mudler/vllm.cpp/issues/960), which could not register its new checker's creation mutation without touching `check-pr-size.py` at all. NOT fixed: wiring that suite into CI, which is its own change and would red `main` until this landed | bug | | [#1066](https://github.com/mudler/vllm.cpp/issues/1066) | `ENG-EXPERT-STREAM` | `Qwen35ExpertStream` (`src/vllm/model_executor/models/qwen3_5.cpp`) is a **process-lifetime singleton** and keyed its slot cache on `(TowerId(base), expert)`, where `base` is the expert tower's host buffer **ADDRESS**. Its own comment stated the premise and drew the wrong conclusion: "A tower's identity is its base pointer, which is stable for the model's life". The premise is true; the conclusion does not follow, because the CACHE is not scoped to one model's life. Free a model, load another, and the allocator hands the new towers addresses the old ones held, so the new model's expert resolves to an entry filled from a DIFFERENT checkpoint — returned as a HIT, which by contract moves no bytes, so no counter moves and nothing downstream has anything to observe. MEASURED on two synthetic 4-layer/4-expert MoE models in one process, instrumenting `KqExpertSlice` to `memcmp` each returned slot against the tower slice it claims to be: **24 towers occupied 21 distinct addresses, and 20 of 222 slices returned another tower's bytes**; end to end the two arms disagreed on all 160 logits while each arm was internally deterministic (0 differing values on a repeat), which rules out nondeterminism. Invisible to every existing test of this row by construction, because all of them build the cache, store and streamer by hand and none runs two models through the production seam. Reachable by any process that loads a model, releases it, and loads another. Fixed by `OwnedTensor::TowerUid()`, a lazily assigned process-unique counter stamped on the tensor and re-stamped when `bytes` moves (so a copy cannot inherit an identity along with a different buffer); a counter cannot collide because it never goes backwards. Found and fixed while repairing the F1-F11 wiring review for [#912](https://github.com/mudler/vllm.cpp/issues/912). Spec [`expert-streaming.md`](specs/expert-streaming.md) | bug | | [#1088](https://github.com/mudler/vllm.cpp/issues/1088) | `LTX25-RESOLUTION-ENVELOPE` | `docs/USAGE.md` published "Measured to complete on one GB10 = 320x192 at 25 frames" beside "Measured NOT to complete = 448x256 at 25 frames, the denoise finishes, then the decode loses about 59 GB in 24 s", and `docs/BENCHMARKS.md` said the same. Two renders on `dgx.casa` on 16 to 17 August 2026 against `main` `0b0b8900f`, which carries [#1041](https://github.com/mudler/vllm.cpp/issues/1041) threaded decode, [#1032](https://github.com/mudler/vllm.cpp/issues/1032) T2A and [#1036](https://github.com/mudler/vllm.cpp/issues/1036) f32 decode accumulators, in container `vllmcpp-build:gb10`, `Release`, `VLLM_CPP_CUDA=ON`, arch `121a`, `TRITON=ON`, CUTLASS absent so FlashAttention-2 was not built, `VLLM_CPP_CPU_THREADS=20`, NVFP4 transformer, no `--allow-unported`, **COMPLETED 448x256/25f in 3085 s and 704x448/25f in 4231 s**. 1024x576/25f was not attempted to completion because another session claimed the box. That is scheduling and not an envelope, so 704x448 is NOT a ceiling. The ~59 GiB cliff did NOT recur under a 2 s memory guard that would have seen it. `MemAvailable` floors were 38.96 GiB over 1289 samples at 448x256 and 38.89 GiB over 1743 samples at 704x448, with ZERO samples under 34 GiB on either, peak use of 80 of 119 GiB, and no reboot. The 704x448 artifact was verified rather than inferred from an exit code: 25/25 distinct frame md5s, 0 near-uniform and 0 near-black frames, adjacent-frame mean absolute difference 4.381 against a uniform-noise reference of 85.3 on the same shape, 0/24 zero-motion pairs, and audio at 48 kHz stereo, 1.010 s, -37.29 dBFS, 20/20 windows above threshold. NOT claimed: ONE run per geometry on a contended shared box with no oracle on either side, so two points establish no scaling law. The 59 GB observation is KEPT and attributed to its own run, which is rung F1 in `benchmark-record.md`, a prompt-embeds render with no text tower that an ARMED WATCHDOG ended at 13.77 GiB against an 18 GiB floor rather than the engine failing. Attributing it stays open as [#1014](https://github.com/mudler/vllm.cpp/issues/1014). The claim that most of a 320x192/25f render is spent in the host VAE decode is superseded too: after #1041 a resolution-independent ~1731 s single-threaded phase, measured at 1731 s and 1732 s across two rungs whose voxel counts differ 2.75x, is 57 to 66% of wall, filed as [#1087](https://github.com/mudler/vllm.cpp/issues/1087). FIXED IN FLOW in `docs/USAGE.md`, `docs/BENCHMARKS.md` and [`ltx25-resolution-envelope.md`](specs/ltx25-resolution-envelope.md) section 4.1 | bug | +| [#1073](https://github.com/mudler/vllm.cpp/issues/1073) | `FIX-NAS-PATH-1073` | The NAS moved to `/usr/local/nas_share` and `/mnt/nas_share` is gone, so every tracked default built on `/mnt` broke. `/mnt` is the EPHEMERAL root overlay of the gate box's immutable Kairos OS and does not survive a reboot; `/usr/local` is `COS_PERSISTENT` and does. Observed 2026-08-16 after an 8 h 19 min outage: the mount came back because the `/oem` boot-stage unit worked, `/mnt/nas_share` did not, and the untracked `.env` still declared `CHECKPOINT_ROOT=/mnt/nas_share/checkpoints` — a gate that reads a path `.env` does not declare is not the gate its spec names. `.agents/environment.md` documented NO NAS location at all (measured: the file held no `/mnt` string), so the repair adds the path AND the `COS_PERSISTENT` reason, because a bare path correction invites the next reader to restore the dead location as a symlink. The seven live defaults now derive from `CHECKPOINT_ROOT`, which four sibling scripts already did: `scripts/gen-minimax-music3-manifest.py:17`, `scripts/gen-ltx2-quant-goldens.py:48`, `tools/parity/dump_tokenizer_gpt4o.py:36,39,57`, `tools/gen_pretok_goldens.py:57`, `src/vllm/tokenizer/pretokenizer.cpp:319`, `tests/parity/test_minimax_music3_quant_real.cpp:133,144` and `docs/USAGE.md:3069,3453`. The 41 hits were classified before any edit and the records that cite the old path KEEP it: `.agents/benchmark-record.md`, the LTX-2.5/Nemotron-H specs, `.agents/model-matrix.md`, the captured goldens and the generated `.inc` headers state where a past measurement read its bytes, which is provenance. Spec [`nas-mount-path.md`](specs/nas-mount-path.md) | bug | +| [#1077](https://github.com/mudler/vllm.cpp/issues/1077) | — | `.env.example:37`, `.agents/environment.md:29` and `tests/vllm/multimodal/test_ltx2_video.cpp:2128-2132` each state that nothing in the tree reads `CHECKPOINT_ROOT`, and six gates read it: `tests/parity/test_minimax_music3_ar_real.cpp:162`, `_e2e_real.cpp:170`, `_llm_real.cpp:137`, `_quant_real.cpp:130,140`, `tests/vllm/models/test_ltx2_text_encoder.cpp:2299`, and `test_nemotron_h_loader.cpp:161` tells the reader to export it. No product code under `src/` or `include/` reads it, so the accurate statement is that the LIBRARY never reads it while several gates do. It costs more than tidiness: `test_ltx2_video.cpp` reasons FROM the claim when it chooses a separate `LTX2_CHECKPOINT_ROOT` ("this would be its first reader"), and that reasoning is void. Found while repairing [#1073](https://github.com/mudler/vllm.cpp/issues/1073) and NOT fixed there, because reversing a design decision needs its own review rather than a path substitution. Listed under `## Owed` in [`nas-mount-path.md`](specs/nas-mount-path.md) | bug | +| [#1079](https://github.com/mudler/vllm.cpp/issues/1079) | `FIX-NAS-PATH-1073` | All four skip messages in `tests/parity/test_minimax_music3_quant_real.cpp` streamed the case name as a `const char*`, and doctest 2.5.2 stringifies that through its bool overload, so every one printed `SKIP 1` and named no case. The comment above the helpers states the obligation the messages then failed: a gate that silently passes when its asset is absent has not reported. It matters here because the binary reports `6 passed` with `assertions: 0` when the checkpoint is absent, so the message text is all that separates a skipped run from a gated one. Pre-existing on `main` at `100026481`. FIXED IN FLOW while landing [#1073](https://github.com/mudler/vllm.cpp/issues/1073), which rewrote those exact messages and would have carried the defect forward under a changed line; the fix streams `std::string(what)`. Scope measured before fixing: 4 hits, all in this one file | bug | diff --git a/.agents/specs/nas-mount-path.md b/.agents/specs/nas-mount-path.md new file mode 100644 index 000000000..a33ab748b --- /dev/null +++ b/.agents/specs/nas-mount-path.md @@ -0,0 +1,142 @@ +# The tracked checkpoint paths name `/mnt/nas_share`, which a reboot deletes + +Row: `FIX-NAS-PATH-1073` +Issue: [#1073](https://github.com/mudler/vllm.cpp/issues/1073) +Baseline: `origin/main` @ `100026481` + +## 1. Scope + +Point the live checkpoint defaults at the declared checkpoint root, and record +in [`../environment.md`](../environment.md) where the NAS mounts on `dgx.casa` +and why the old location cannot be restored. + +**In scope:** `.agents/environment.md`, the four script and tool defaults that a +fresh run reads, the one product comment that names the source checkpoint, the +one gate whose fallback path is a literal, and the two `docs/USAGE.md` recipes a +reader copies. + +**Out of scope:** every record that cites `/mnt/nas_share` as the path a past +measurement used. `.agents/benchmark-record.md`, the LTX-2.5 and Nemotron-H +specs, `.agents/model-matrix.md`, the captured goldens under +`tests/parity/goldens/`, and the generated `.inc` headers state what was read at +the time. That is provenance. AGENTS.md is explicit that rewriting an existing +file to satisfy a rule is out of scope unless a row asks for the rewrite, and +this row does not ask. + +## 2. The fact, and why it is not a one-off + +The NAS mounts at `/usr/local/nas_share` on `dgx.casa`. `/mnt/nas_share` is +gone. The developer confirmed that `/usr/local` is canonical. + +`/mnt` sits on the ephemeral root overlay of the immutable Kairos OS, so a +directory created there does not survive a reboot. `/usr/local` is +`COS_PERSISTENT` and does survive. This is the same property that made an +earlier `/oem` `rootfs`-stage change cost a boot. + +Observed 2026-08-16, after the box returned from an 8 h 19 min outage: the mount +itself came back because the `/oem` boot-stage unit worked, and `/mnt/nas_share` +did not. Every checkpoint path built on `/mnt` broke while the untracked `.env` +still declared `CHECKPOINT_ROOT=/mnt/nas_share/checkpoints`. A gate that reads a +path `.env` does not declare is not the gate its spec names, so this state +blocks any checkpoint-loading gate until a person notices. + +`.env` is untracked. It was repointed by hand and verified against 18 checkpoint +directories under `/usr/local/nas_share/checkpoints`, including +`nemotron-3.5-lightning-30b-nvfp4` and `nemotron-3.5-lightning-30b-gguf`. The +tracked surfaces are the part this row repairs. + +## 3. The change + +`.env` declares the location once. Every tracked default now derives from +`CHECKPOINT_ROOT` instead of repeating an absolute path, so the next mount move +costs one untracked line rather than another sweep. The convention already +exists in the tree: `scripts/measure-ltx2-keyframes-meta.py:30`, +`scripts/gen-ltx2-prompt-tokens-goldens.py:28`, +`scripts/measure-ltx2-prompt-adaln.py:27` and `tools/oracle/music3_oracle.py:31` +all write `$CHECKPOINT_ROOT/…` already. The files below were the outliers. + +| File | Was | Now | +|---|---|---| +| `.agents/environment.md` | the DGX profile named no NAS location at all | a profile bullet gives `/usr/local/nas_share`, the `COS_PERSISTENT` reason, and the instruction not to restore the old path | +| `scripts/gen-minimax-music3-manifest.py:17` | `--checkpoint /mnt/nas_share/checkpoints/minimax-music3` | `--checkpoint "$CHECKPOINT_ROOT/minimax-music3"` | +| `scripts/gen-ltx2-quant-goldens.py:48` | `--checkpoint-root /mnt/nas_share/checkpoints` | `--checkpoint-root "$CHECKPOINT_ROOT"` | +| `tools/parity/dump_tokenizer_gpt4o.py:36,39` | two literal paths in the by-hand recipe | `$CHECKPOINT_ROOT/…` | +| `tools/parity/dump_tokenizer_gpt4o.py:57` | `DEFAULT_TOKENIZER_JSON`, a literal | read from `CHECKPOINT_ROOT`; `--tokenizer-json` becomes required when the variable is unset | +| `tools/gen_pretok_goldens.py:57` | the regex source path | `$CHECKPOINT_ROOT/muse-glimmer-30b/tokenizer.json` | +| `src/vllm/tokenizer/pretokenizer.cpp:319` | the same source path, in a comment | the same substitution | +| `tests/parity/test_minimax_music3_quant_real.cpp:133,144` | fallback `std::string("/mnt/nas_share/checkpoints")` | no fallback; an unset root skips and names the two variables | +| `docs/USAGE.md:3069,3453` | two literal paths in copyable recipes | `$CHECKPOINT_ROOT` and `CHECKPOINT_ROOT=…`, which is what the rest of the file already writes | + +The `.agents/environment.md` bullet carries the reason, not only the path. The +issue asks for that explicitly, because a bare path correction invites the next +reader to restore `/mnt/nas_share` as a convenience symlink, and that symlink +disappears at the next reboot. + +The issue reports that `.agents/environment.md` documented the `/mnt` location +and a "canonical symlink". Measured against `100026481`, it documents neither: +the file holds no `/mnt/nas_share` string, and its only "canonical symlink" at +`:87` is `~/venvs/vllm-oracle`, which is the oracle venv and is unrelated. The +defect is therefore an absence rather than a wrong value, and the repair is to +add the fact. + +## 4. Two paths that are deliberately left + +`tests/vllm/multimodal/test_qwen3_5_moe_vl_hw.cpp:66-67` probes +`/usr/local/nas_share/checkpoints/qwen3.6-35b-a3b-bf16` FIRST and +`/mnt/nas_share/…` second. A fresh run on `dgx.casa` already resolves, so the +second entry costs one `fs::exists` call on an absent path and breaks nothing. +The fact in this row covers `dgx.casa`. It does not cover the cluster nodes, and +deleting a tolerated fallback for a host nobody measured would trade a harmless +probe for a possible refusal. + +`tests/vllm/test_pretokenizer.cpp:377` records that the GPT-4o regex was +"transcribed verbatim from `/mnt/nas_share/…/tokenizer.json` into +`tools/gen_pretok_goldens.py`". That sentence is the same shape as the two +comments this row does change, and the issue lists those two and not this one. +It reads as a statement about a past transcription rather than as a pointer a +reader follows, because the pointer it gives is the generator script. Left +unchanged and reported, so the owner decides rather than an implementer guessing. + +## 5. Evidence + +No behavior changes except in one gate, so the gate is the accuracy of the text +and the reachability of each default. + +| Claim | How it was checked at `100026481` | +|---|---| +| every `/mnt/nas_share` hit is classified | `grep -rn '/mnt/nas_share'` returns 41 lines in 26 files; each is named in §3, §4, or the out-of-scope list in §1 | +| `.agents/environment.md` holds no `/mnt` path | `grep -n '/mnt' .agents/environment.md` returns nothing | +| `$CHECKPOINT_ROOT` is an existing convention, not a new one | four sibling scripts already write it; listed in §3 | +| the gate tests do read `CHECKPOINT_ROOT` | `getenv("CHECKPOINT_ROOT")` at `tests/parity/test_minimax_music3_ar_real.cpp:162`, `_e2e_real.cpp:170`, `_llm_real.cpp:137`, `tests/vllm/models/test_ltx2_text_encoder.cpp:2299` | +| the changed gate still skips loudly | `test_minimax_music3_quant_real` builds warning-free and runs. With the root unset, 6 cases, `SKIP music3 q4_k artifact identity: VLLM_CPP_MUSIC3_GGUF and CHECKPOINT_ROOT are both unset`. With `CHECKPOINT_ROOT=/usr/local/nas_share/checkpoints`, the same 6 name the composed path `…/minimax-music3-gguf/rvq_depth_decoder_q4_k.gguf` | +| the tokenizer tool still refuses cleanly | with `CHECKPOINT_ROOT` unset and no arguments, `error: the following arguments are required: --tokenizer-json`; with it set, `--help` prints the `$CHECKPOINT_ROOT/muse-glimmer-30b/tokenizer.json` default | + +## 5b. One defect found in flow and fixed here + +[#1079](https://github.com/mudler/vllm.cpp/issues/1079). Those six skip +messages printed `SKIP 1` and named no case, because the helper streamed its +`const char*` argument into doctest `MESSAGE` and doctest 2.5.2 stringifies a +`const char*` through its bool overload. It is pre-existing at `100026481` and +is repaired here rather than left, because this row rewrites those exact +messages and would otherwise carry the defect forward under a changed line. The +binary reports `6 passed` with `assertions: 0` when the checkpoint is absent, so +the message text is what separates a skipped run from a gated one. Scope was +measured before the fix: `grep -rn 'MESSAGE("SKIP " << what' tests/` returns 4 +hits, all in this file. The fix streams `std::string(what)`, and the output +above is the after. + +## 6. Now + +The tracked defaults and the DGX profile name the location that survives a +reboot. The records that cite the old location keep it, because they record +where a past measurement read its bytes. + +## Owed + +[#1077](https://github.com/mudler/vllm.cpp/issues/1077): `.env.example:37`, +`.agents/environment.md:29` and `tests/vllm/multimodal/test_ltx2_video.cpp:2129` +each state that nothing in the tree reads `CHECKPOINT_ROOT`. Six gates read it +today, listed in §5. This row does not repair that claim, because +`test_ltx2_video.cpp:2129` reasons FROM the false premise when it chooses a +separate variable, and reversing that reasoning is a design decision with its own +review rather than a path substitution. Filed and left owned here. diff --git a/docs/USAGE.md b/docs/USAGE.md index 2ff0dcc24..9e7cacbdd 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -3095,7 +3095,7 @@ checkout (the two nibble-order authorities); it reads a few hundred bytes at their own offsets and never a payload: ```sh -python3 scripts/gen-ltx2-quant-goldens.py --vllm ~/_git/vllm --ltx2 ~/_git/LTX-2 --checkpoint-root /mnt/nas_share/checkpoints --out tests/vllm/models/ltx2_quant_goldens.inc +python3 scripts/gen-ltx2-quant-goldens.py --vllm ~/_git/vllm --ltx2 ~/_git/LTX-2 --checkpoint-root "$CHECKPOINT_ROOT" --out tests/vllm/models/ltx2_quant_goldens.inc cmake --build build --target test_ltx2_loader && ./build/tests/test_ltx2_loader ``` @@ -3522,7 +3522,7 @@ The Q4_K arm's own gate needs the pinned GGUF and the bf16 checkpoint, and skips loudly without them: ```sh -CHECKPOINT_ROOT=/mnt/nas_share/checkpoints \ +CHECKPOINT_ROOT=... \ ./build/tests/test_minimax_music3_quant_real ``` diff --git a/scripts/gen-ltx2-quant-goldens.py b/scripts/gen-ltx2-quant-goldens.py index a54eeb766..c3256de01 100644 --- a/scripts/gen-ltx2-quant-goldens.py +++ b/scripts/gen-ltx2-quant-goldens.py @@ -45,7 +45,7 @@ Usage: python3 scripts/gen-ltx2-quant-goldens.py \\ --vllm ~/_git/vllm \\ - --checkpoint-root /mnt/nas_share/checkpoints \\ + --checkpoint-root "$CHECKPOINT_ROOT" \\ --out tests/vllm/models/ltx2_quant_goldens.inc Needs torch + numpy (CPU only). diff --git a/scripts/gen-minimax-music3-manifest.py b/scripts/gen-minimax-music3-manifest.py index 441c1cc34..ce3a29907 100644 --- a/scripts/gen-minimax-music3-manifest.py +++ b/scripts/gen-minimax-music3-manifest.py @@ -14,7 +14,7 @@ Usage: scripts/gen-minimax-music3-manifest.py \ - --checkpoint /mnt/nas_share/checkpoints/minimax-music3 \ + --checkpoint "$CHECKPOINT_ROOT/minimax-music3" \ --output tests/vllm/models/minimax_music3_manifest.inc """ diff --git a/src/vllm/tokenizer/pretokenizer.cpp b/src/vllm/tokenizer/pretokenizer.cpp index 0171a250f..2bbca4479 100644 --- a/src/vllm/tokenizer/pretokenizer.cpp +++ b/src/vllm/tokenizer/pretokenizer.cpp @@ -316,7 +316,7 @@ size_t MatchWs(std::string_view t, size_t pos) { // 2294-2299 @ 153d324bcf). // // The pattern is transcribed VERBATIM from the checkpoint that forced it in, -// /mnt/nas_share/checkpoints/muse-glimmer-30b/tokenizer.json (read 2026-08-11), +// $CHECKPOINT_ROOT/muse-glimmer-30b/tokenizer.json (read 2026-08-11), // pre_tokenizer.pretokenizers[0].pattern.Regex, behavior=Isolated. It is the // same string llama.cpp records as "original regex from tokenizer.json" above // its own GPT4O entry (llama.cpp/src/llama-vocab.cpp:432 @ 153d324bcf): diff --git a/tests/parity/test_minimax_music3_quant_real.cpp b/tests/parity/test_minimax_music3_quant_real.cpp index 44a56d015..682bec5ce 100644 --- a/tests/parity/test_minimax_music3_quant_real.cpp +++ b/tests/parity/test_minimax_music3_quant_real.cpp @@ -123,14 +123,23 @@ constexpr double kQ4KIdenticalFloor = 0.02; // measured 0.0284 // is "too good" here is not a better port; it is a different set of weights. constexpr double kQ4KMeanAbsFloor = 5e-3; // bf16 control is 1.659e-3 +// The checkpoint root comes from the environment, never from a literal. The +// literal that stood here named `/mnt/nas_share/checkpoints`, which sits on the +// ephemeral root overlay of the gate box's immutable OS and was deleted by a +// reboot (issue #1073); `.agents/environment.md` records the live location and +// why it cannot move back. An undeclared root now yields an EMPTY path, so the +// skips below name the variables to set rather than a path nobody declared. +std::string CheckpointRoot() { + const char* root = std::getenv("CHECKPOINT_ROOT"); + return root != nullptr && *root != '\0' ? std::string(root) : std::string(); +} + std::string GgufPath() { if (const char* direct = std::getenv("VLLM_CPP_MUSIC3_GGUF")) { if (*direct != '\0') return direct; } - const char* root = std::getenv("CHECKPOINT_ROOT"); - const std::string base = root != nullptr && *root != '\0' - ? std::string(root) - : std::string("/mnt/nas_share/checkpoints"); + const std::string base = CheckpointRoot(); + if (base.empty()) return {}; return (fs::path(base) / "minimax-music3-gguf" / "rvq_depth_decoder_q4_k.gguf").string(); } @@ -138,19 +147,27 @@ std::string SafetensorsRoot() { if (const char* direct = std::getenv("VLLM_CPP_MUSIC3_CHECKPOINT")) { if (*direct != '\0') return direct; } - const char* root = std::getenv("CHECKPOINT_ROOT"); - const std::string base = root != nullptr && *root != '\0' - ? std::string(root) - : std::string("/mnt/nas_share/checkpoints"); + const std::string base = CheckpointRoot(); + if (base.empty()) return {}; return (fs::path(base) / "minimax-music3").string(); } // Skip loudly. A gate that silently passes when its asset is absent has not // reported (AGENTS.md); this says which file it wanted. +// +// `what` is streamed as a `std::string`, never as the `const char*` it arrives +// as: doctest stringifies a `const char*` through its bool overload, so every +// message here printed "SKIP 1" and named no case at all (issue #1079). bool SkipIfMissing(const char* what) { + const std::string gguf = GgufPath(); + if (gguf.empty()) { + MESSAGE("SKIP " << std::string(what) + << ": VLLM_CPP_MUSIC3_GGUF and CHECKPOINT_ROOT are both unset"); + return true; + } std::error_code ec; - if (!fs::exists(GgufPath(), ec)) { - MESSAGE("SKIP " << what << ": no GGUF at " << GgufPath() + if (!fs::exists(gguf, ec)) { + MESSAGE("SKIP " << std::string(what) << ": no GGUF at " << gguf << " (set VLLM_CPP_MUSIC3_GGUF or CHECKPOINT_ROOT)"); return true; } @@ -158,11 +175,17 @@ bool SkipIfMissing(const char* what) { } bool SkipIfNoSafetensors(const char* what) { + const std::string root = SafetensorsRoot(); + if (root.empty()) { + MESSAGE("SKIP " << std::string(what) + << ": VLLM_CPP_MUSIC3_CHECKPOINT and CHECKPOINT_ROOT are both unset"); + return true; + } std::error_code ec; const fs::path shard = - fs::path(SafetensorsRoot()) / "rvq_depth_decoder" / "diffusion_pytorch_model.safetensors"; + fs::path(root) / "rvq_depth_decoder" / "diffusion_pytorch_model.safetensors"; if (!fs::exists(shard, ec)) { - MESSAGE("SKIP " << what << ": no bf16 reference at " << shard.string()); + MESSAGE("SKIP " << std::string(what) << ": no bf16 reference at " << shard.string()); return true; } return false; diff --git a/tools/gen_pretok_goldens.py b/tools/gen_pretok_goldens.py index d2263bf85..c5154db37 100644 --- a/tools/gen_pretok_goldens.py +++ b/tools/gen_pretok_goldens.py @@ -54,7 +54,7 @@ # GPT-4o / o200k family (llama.cpp LLAMA_VOCAB_PRE_TYPE_GPT4O; GGUF pre names # "gpt-4o", "llama4", "kanana2", "talkie"). VERBATIM from -# /mnt/nas_share/checkpoints/muse-glimmer-30b/tokenizer.json +# $CHECKPOINT_ROOT/muse-glimmer-30b/tokenizer.json # (pre_tokenizer.pretokenizers[0].pattern.Regex, read 2026-08-11) — byte-equal # to the string llama.cpp records as "original regex from tokenizer.json" at # src/llama-vocab.cpp:432 @ 153d324bcf. diff --git a/tools/parity/dump_tokenizer_gpt4o.py b/tools/parity/dump_tokenizer_gpt4o.py index ea55498c3..a93439243 100644 --- a/tools/parity/dump_tokenizer_gpt4o.py +++ b/tools/parity/dump_tokenizer_gpt4o.py @@ -33,19 +33,20 @@ and by hand, against the GGUF itself: build-cpu/examples/tokenize \ - /mnt/nas_share/checkpoints/muse-glimmer-30b-gguf/muse-glimmer-30B-kquant-17gb.gguf \ + "$CHECKPOINT_ROOT/muse-glimmer-30b-gguf/muse-glimmer-30B-kquant-17gb.gguf" \ tests/parity/goldens/tokenizer_muse_glimmer/corpus.txt > /tmp/gguf_cpp.txt python3 tools/parity/verify_tokenizer_gguf.py \ - /mnt/nas_share/checkpoints/muse-glimmer-30b/tokenizer.json \ + "$CHECKPOINT_ROOT/muse-glimmer-30b/tokenizer.json" \ tests/parity/goldens/tokenizer_muse_glimmer/corpus.txt > /tmp/hf_py.txt diff /tmp/hf_py.txt /tmp/gguf_cpp.txt # expect no output Regenerate: - python3 tools/parity/dump_tokenizer_gpt4o.py + CHECKPOINT_ROOT=... python3 tools/parity/dump_tokenizer_gpt4o.py """ import argparse import hashlib import json +import os import pathlib import sys @@ -53,8 +54,18 @@ REPO = pathlib.Path(__file__).resolve().parents[2] GOLDEN_DIR = REPO / "tests/parity/goldens/tokenizer_muse_glimmer" -DEFAULT_TOKENIZER_JSON = pathlib.Path( - "/mnt/nas_share/checkpoints/muse-glimmer-30b/tokenizer.json" + +# The checkpoint root comes from `CHECKPOINT_ROOT` (`.env`), never from a +# literal here. The literal this replaced named `/mnt/nas_share`, which sits on +# the ephemeral root overlay of the gate box's immutable OS and disappeared at a +# reboot (issue #1073); `.agents/environment.md` records the live location and +# the reason. With the variable unset, `--tokenizer-json` is required, so the +# tool refuses by name instead of reading a path nobody declared. +_CHECKPOINT_ROOT = os.environ.get("CHECKPOINT_ROOT") or "" +DEFAULT_TOKENIZER_JSON = ( + pathlib.Path(_CHECKPOINT_ROOT) / "muse-glimmer-30b" / "tokenizer.json" + if _CHECKPOINT_ROOT + else None ) DEFAULT_LABEL = "meta/muse-glimmer-30b (HF snapshot on the NAS)" @@ -70,7 +81,10 @@ def merge_pair(m) -> tuple[str, str]: def main() -> int: ap = argparse.ArgumentParser(description=__doc__) ap.add_argument("--tokenizer-json", type=pathlib.Path, - default=DEFAULT_TOKENIZER_JSON) + default=DEFAULT_TOKENIZER_JSON, + required=DEFAULT_TOKENIZER_JSON is None, + help="the checkpoint's tokenizer.json; defaults to " + "$CHECKPOINT_ROOT/muse-glimmer-30b/tokenizer.json") ap.add_argument("--golden-dir", type=pathlib.Path, default=GOLDEN_DIR) ap.add_argument("--label", default=DEFAULT_LABEL) args = ap.parse_args()