fix(BACKEND-TENSTORRENT-HOST-FREE-FORWARD): refresh the stale TT Qwen3-0.6B paged-engine golden pair (#1488, #1508)

.agents/issue-index.md

@@ -517,6 +517,7 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1487](https://github.com/mudler/vllm.cpp/issues/1487) | `BACKEND-TENSTORRENT-HOST-FREE-FORWARD` | **`test_release_metadata` is red on every aarch64 host, so `agent-preflight` cannot go green on the TT dev fleet.** `tests/scripts/test_release_metadata.py:55` stages the HOST arch — `shutil.copy2("/bin/true", ...)` — into the hardcoded `vllm.cpp-0.0.1-linux-x86_64-glibc-cpu.tar.gz` (`:143`); on x86_64 the ELF matches the manifest by accident, on aarch64 the validator correctly refuses: `ELF host architecture does not match manifest: ARM aarch64`. Measured at `206afb63` == `origin/main` with zero local commits and an uncommitted src-only diff; NOT the #1353 disk-full mode (902G free, no scratch-repo failures, the message is the mismatch itself). Repair direction: make the fixture self-consistent (stage an ELF matching the declared arch, or derive the declared arch from the staged ELF) so the validator is exercised for a manifest that lies. Listed under `## Owed` in [`tenstorrent-host-free-forward.md`](specs/tenstorrent-host-free-forward.md) | bug |
 | [#1488](https://github.com/mudler/vllm.cpp/issues/1488) | `BACKEND-TENSTORRENT-HOST-FREE-FORWARD` | **The TT `test_qwen3_paged_engine` golden is stale: `REQUIRE(anchor_ok)` fails with anchor drift prompt[1] tok=10 (engine=14126, committed anchor=62901).** Measured IDENTICALLY before and after the #1476 fix (same engine token both arms, ctest exit 8 both arms) — that test runs the DEFAULT path with no `VT_TT_HOST_FREE_DECODE`, which the fix does not touch, so the drift is not the captured-decode defect. The golden was committed `971d5506` (2026-08-09) and 13 TT ops commits landed Aug 9-18 with no on-card golden re-run. Owed: the `VT_DUMP_IDS` + `qwen3-neartie-gap.py` re-adjudication, then either a re-derived golden or an escalation to a real divergence. Split out of #1476 so the fidelity fix can close without orphaning the finding. Listed under `## Owed` in [`tenstorrent-host-free-forward.md`](specs/tenstorrent-host-free-forward.md) | bug |
 | [#1504](https://github.com/mudler/vllm.cpp/issues/1504) | `SPEC-DFLASH2` | **`test_audit_live_rows` is red on `main`: `KERNEL-DFLASH2-SELECTOR-EDGES` and `KERNEL-TOPK-PAIRS` remain `ACTIVE` after their implementation landed.** Measured at `origin/main` `cffe59b0` in a clean detached worktree: `scripts/audit-live-rows.py` reports `248 live rows; 2 abandoned ACTIVE` and `test_audit_live_rows` fails `test_shipped_record_has_no_abandoned_active_row` with exactly those two IDs. Both kernel-matrix rows (.agents/kernel-matrix.md:144-145) record their implementation as landed and gated on 2026-08-20 via SPEC-DFLASH2 W3 (PR #1497) yet still carry `ACTIVE`, so every preflight on current main inherits the red. Fourth instance of the #1469 class. NOT FIXED IN FLOW: the repair is a lifecycle move on `SPEC-DFLASH2`-owned rows and owes `docs/STATUS.md`, `docs/BENCHMARKS.md` and the moved spec's `## Now`; `KERNEL-TOPK-PAIRS` also carries the open CUDA NaN-tie disagreement #1489 that the move should reconcile or name as its open gate. Found while rebasing `BACKEND-TENSTORRENT-HOST-FREE-FORWARD` onto main; filed in flow | bug |
+| [#1508](https://github.com/mudler/vllm.cpp/issues/1508) | `BACKEND-TENSTORRENT-HOST-FREE-FORWARD` | doctest `MessageBuilder` streams `const char*` as bool, so `test_qwen3_paged_engine` prints `committed anchor=62901` for a golden holding `6290` (and `96251` for `9625`) and a useless `1` label; proven with a 7-line harness against the pinned header, the garbled value was misread during #1488 as golden-buffer corruption before byte-level npy verification cleared the buffers; `label`/`ids_name`/`gap_name` now `std::string` and message ternaries wrapped, in the same flow | bug |
 | [#1499](https://github.com/mudler/vllm.cpp/issues/1499) | `ENG-EXPERT-STREAM-DEVICE` | **`main` was RED on `sanitize-cpu (thread)` and the finding was a heap-BUFFER-OVERFLOW in the test, not a use-after-free on the alias path.** `test_resident_weight_host_addressable` case *a MISALIGNED BORROW is not re-homed, and stages instead* sized its backing vector `nb + kDeviceAliasAlignment` and then wrote `nb` bytes at `base + (kDeviceAliasAlignment - base % kDeviceAliasAlignment) + 8`. That padding expression yields a WHOLE `kDeviceAliasAlignment` when `base` is already 256-aligned rather than zero, so the largest offset it can produce is `256 + 8` and the write runs 8 bytes past the end. CONFIRMED by address rather than by argument: reproduced locally under the CI lane's own configuration (`-DVLLM_CPP_SANITIZE=thread -DVLLM_CPP_CUDA=OFF`, `VT_POOL_BYPASS=1`), TSan reports the block at `0x724800000000` of size 352 -- `nb`=96 from `kN*kK*2`, plus 256 -- and the write at `0x724800000160`, which is `base + 352`, exactly one past the end for 8 bytes. Neither reading the issue offered is right. It is not a dangling alias, because nothing on the `OwnedBytes::Borrow` path is involved; and it is not a sanitizer artefact of allocator reuse, because the write is genuinely out of bounds. TSan named it `heap-use-after-free` because the 8 overrun bytes landed in the block an earlier `CHECK_THROWS_WITH_AS` case had freed -- the `VT_CHECK` message string destroyed at `qwen3_5.cpp:1205` during unwind -- so the detector reported the free it could see instead of the overflow it could not. It only fires when the allocator returns a 256-aligned block, which is why the plain lane never saw it. FIXED IN FLOW by sizing the allocation for the misalignment it deliberately creates (`nb + kDeviceAliasAlignment + kMisalign`) and adding TWO assertions beside it. (a) `REQUIRE(backing->size() >= nb + kDeviceAliasAlignment + kMisalign)` is deterministic -- it reads no pointer, so it reds on every run in every lane, measured 40/40 in the plain lane and 6/6 under TSan with the allocation shrunk back. (b) `REQUIRE(static_cast<size_t>(off - base) + nb <= backing->size())` binds the write to the block this run got, and is honest about its reach: it fires only when `base % kDeviceAliasAlignment < kMisalign`, which is not merely lane-dependent but BUILD-dependent, since the residue is deterministic inside one binary and arbitrary across binaries -- the same mutation reds 40/40 in one plain-lane build and stays GREEN 20/20 at `base % 256 == 176` in a build differing by one `MESSAGE` line, while TSan reds 6/6 at `base % 256 == 0`. (b) is written as an integer comparison because doctest stringifies the pointer form as bools and printed `REQUIRE( 1 <= 1 ) is NOT correct!`. The deliberate misalignment is the point of the case -- it proves the alias path DECLINES a badly-aligned borrow -- and is preserved unchanged. Evidence: before, exit 66 with the UAF and `72` assertions passing; after, exit 0 with no TSan finding on 3/3 runs and `73` assertions. Mutation: restoring the old size reds at the new `REQUIRE` (exit 1, 13 cases / 1 failed), which is the assertion proving the buffer is sized for the offset | bug |
 | [#1501](https://github.com/mudler/vllm.cpp/issues/1501) | `SPEC-DFLASH2` | **The row recorded DFlash2's T>0 walk as "inverse CDF" from its opening brief; upstream draws GUMBEL-MAX at BOTH pull-request heads it has cited.** `.agents/specs/dflash2-spec-decode.md` `## Upstream chain` mechanism 2 said "At T>0 the walk is by inverse CDF and returns q over the K candidates", and the `SPEC-DFLASH2` engine-matrix row said the same. At `19c9351904df4c63042671bc67a866ca48dc7d6f` the non-greedy branch of `_selector_walk_kernel` drew `uniform = tl_rand32(gumbel_seed, candidates, includes_zero=False)`, `noise = -tl.log(-tldevice.log1p(-uniform))` and took the argmax of `scores / temperature + noise` — Gumbel-max; at `66e5414c6d75a8529473d977f7458c140bbab8a0` that branch is replaced by one call to `gumbel_noised_argmax`, the same draw. No inverse-CDF walk exists upstream. BLAST RADIUS is a mis-scoped wave rather than a shipped defect: W4 ships the GREEDY arm, which is byte-for-byte upstream's `SAMPLE_PROBABILISTIC=False` arm, and the noised arm is not ported (`ParseSpeculativeConfigJson` refuses `draft_sample_method: "probabilistic"` by name against an accept-iff-equal verify) — but an implementer scoped to write an inverse-CDF walk would have written the wrong algorithm with no oracle to catch it, because the acceptance gate that would notice is itself owed. FIXED IN THE SAME FLOW by W4: both records corrected in place rather than annotated, `## Risks/decisions` D13 records why the noised arm is unreachable here, and `## Owed` O12 records its layout and the Triton Philox stream (`tl.randint4x`, Philox 4x32-10, keyed by the candidate token ids) a bit-parity port would need. Also corrected there: the six-item enumeration of the speculator head move was verified item by item against the two blobs and is correct, but it was missing a SEVENTH change in the base class (`DraftModelSpeculator.__init__` now calls a virtual `draft_logits_spec`) and the `+24 / -6` file it lives in was absent from the delta table. Listed under `## Risks/decisions` D13 in [dflash2-spec-decode.md](specs/dflash2-spec-decode.md) | bug |
 | [#1490](https://github.com/mudler/vllm.cpp/issues/1490) | `VT-MATMUL-FP8-BLOCK-CUDA` | **`tests/vt/test_ops_matmul_fp8_block_cuda.cpp` asserted in six places that it has never run against a device, and the loudest is its own header.** The suite ran UNPATCHED on `dgx:gpu0` (NVIDIA GB10, driver 580.173.02, compute capability 12.1) in an `rc` lease on 2026-08-20, at tree `7481a2eecbb26b3d5c977e8707b0384994caf136` — an ancestor of `main` — reporting `5 cases / 136 assertions / 0 failed`, `REFERENCE_TIER_LINES=0` and `TEST_RC=0`, against a device-free baseline of 41; recorded in `5870cb2bf` (PR [#1472](https://github.com/mudler/vllm.cpp/pull/1472)) and confirmed in [#1437](https://github.com/mudler/vllm.cpp/issues/1437). The header opened "THIS FILE HAS NEVER RUN AGAINST A DEVICE" and added that "no number produced here appears in any document as a measurement" — doubly false, because the 5/136/0 result appears in both the public documentation (`docs/USAGE.md` at the time, `docs/models/qwen3-8-27b.md` after [#1491](https://github.com/mudler/vllm.cpp/pull/1491) moved it) and the row's spec. The other five were G2's comment block and the four `NO CUDA DEVICE ... #1189 M5's on-hardware leg is OWED, not passed` `MESSAGE` strings in G2, G7, G8 and G9. The four messages are CORRECT in what they do — on a device-free host the case genuinely did not run, and a skip is not a pass — so the repair is that they say the case did not run HERE and name where the on-hardware result is recorded, not that they are deleted or weakened. Three caveats survive unweakened: NO token gate ([#1189](https://github.com/mudler/vllm.cpp/issues/1189) still owns `Qwen/Qwen3.8-27B-FP8` against the pinned oracle), NO speed claim of any kind, and correctness established on the SEVEN shapes actually run rather than on a class; DSV3's `kv_a_proj_with_mqa` `N=576` capability gap stands as a CUTLASS sm120 limitation. Same class as [#1396](https://github.com/mudler/vllm.cpp/issues/1396) and [#1411](https://github.com/mudler/vllm.cpp/issues/1411), which cover other anchors of the same row; no checker here compares a comment against the measurement it annotates. Listed under `## Owed` in [`vt-matmul-fp8-block-cuda.md`](specs/vt-matmul-fp8-block-cuda.md) | bug |

.agents/specs/tenstorrent-host-free-forward.md

@@ -308,11 +308,32 @@ not a process-global `GraphCapturesCounter`. Tracked on
   the host `/bin/true` into an `x86_64`-named archive, so `agent-preflight`
   cannot go green on the TT dev fleet. Found while running this row's preflight.
   Owned by [#1487](https://github.com/mudler/vllm.cpp/issues/1487).
-- **The TT `test_qwen3_paged_engine` golden is stale**: anchor drift
-  prompt[1] tok=10 (engine=14126, committed=62901), identical before and after
-  the #1476 fix (that test runs the default path). Needs the `VT_DUMP_IDS` +
-  near-tie-gap re-adjudication. Owned by
+- **The TT `test_qwen3_paged_engine` golden was stale** — RESOLVED 2026-08-20
+  in this flow: the anchor drift prompt[1] tok=10 (engine=14126, committed
+  6290 — the logged `62901` was a print artifact, #1508) came from the default
+  decode path having moved since the 2026-08-10 capture. The before/after-#1476
+  comparison recorded in #1488 compared exit codes and only the p1-tok-10
+  engine token, so it does not establish that pre-fix TT matches the refreshed
+  cells (prompt 5 tail, prompt 7); the golden is derived from this branch's TT
+  output and the refresh stacks here. Re-adjudicated with `VT_DUMP_IDS` (eager
+  `VLLM_CPP_CUDAGRAPH=0` and captured dumps byte-identical, md5
+  `b5307e33…`) and `qwen3-neartie-gap-transformers.py` (transformers 4.57.1,
+  torch 2.10.0a0+cpu): 53 cells refreshed across 7 prompts — single
+  near-tie divergences at one token each (p1 tok10, p5 tok10, p10 tok5,
+  p11 tok4, p12 tok13, p15 tok12) whose greedy continuations then follow
+  the new prefix, plus prompt 7 rewritten from tok0; the new p7 row
+  matches the vLLM greedy sequence exactly, max gap 375 mnats, zero cells
+  above the 500-mnat band, zero outside-top-K. Fixed by
   [#1488](https://github.com/mudler/vllm.cpp/issues/1488).
+- **doctest `MessageBuilder` streams `const char*` as bool**: every
+  separately-bound `const char*` in a `MESSAGE`/`REQUIRE_MESSAGE` renders as
+  `1`, so the anchor-drift message printed `committed anchor=62901` for a
+  golden holding `6290` (and `96251` for `9625`), and `label` printed as `1`
+  instead of `qwen3-0.6B`. Reproduced against the pinned header with a
+  7-line harness; fixed by passing `std::string` in
+  `test_qwen3_paged_engine.cpp`. Found during the #1488 re-adjudication after
+  the garbled value had been misread as golden-buffer corruption. Fixed by
+  [#1508](https://github.com/mudler/vllm.cpp/issues/1508).
 
 The operator must still rerun the 80-token no-hang gate and
 `test_qwen3_paged_engine` on a Blackhole P150. An implementer run is an
@@ -383,4 +404,5 @@ ambient `VT_TT_HOST_FREE_DECODE` (its exit-time segfault is pre-existing,
 
 Next: operator rerun of the 80-token captured-vs-eager gate and
 `test_qwen3_paged_engine` on card; the paged-engine golden re-adjudication is
-[#1488](https://github.com/mudler/vllm.cpp/issues/1488).
+done ([#1488](https://github.com/mudler/vllm.cpp/issues/1488) closed by the
+stacked golden-refresh commit on this branch).

tests/parity/test_qwen3_paged_engine.cpp

@@ -142,7 +142,7 @@ const int32_t* AsI32(const parity::NpyArray& a) {
 // vLLM's argmax in vLLM's own logits (strict where our token IS vLLM's argmax).
 // Reports the strict token-exact count and the max gap.
 void RunGate(const std::string& repo_dir, const std::string& golden_subdir,
-             const char* label) {
+             const std::string& label) {
   const std::string snap = FindSnapshot(repo_dir);
   if (snap.empty()) {
     MESSAGE(label << " checkpoint absent; skipping (dgx-only) — " << repo_dir
@@ -262,10 +262,13 @@ void RunGate(const std::string& repo_dir, const std::string& golden_subdir,
   const int32_t* anchor_ids = od;   // hard anchor for THIS device
   const int32_t* gap_ids = gapd;    // vLLM teacher-forced gaps for THIS device
   parity::NpyArray o_dev, gap_dev;  // keep device arrays alive for the loop
-  const char* ids_name =
+  // std::string, not const char*: doctest's MessageBuilder stream has no
+  // const char* overload and would render a bare pointer as bool "1" in the
+  // REQUIRE_MESSAGE below (#1508).
+  const std::string ids_name =
       metal ? "our_ids_metal.npy"
             : (rocm ? "our_ids_rocm.npy" : "our_ids_tenstorrent.npy");
-  const char* gap_name =
+  const std::string gap_name =
       metal ? "neartie_gap_mnats_metal.npy"
             : (rocm ? "neartie_gap_mnats_rocm.npy"
                     : "neartie_gap_mnats_tenstorrent.npy");
@@ -277,7 +280,8 @@ void RunGate(const std::string& repo_dir, const std::string& golden_subdir,
       // qwen3-neartie-gap.py then teacher-forces vLLM on that sequence.
       bootstrap_only = true;
       MESSAGE(label << ": BOOTSTRAP dump (device golden absent) for "
-              << (metal ? "Metal" : (rocm ? "ROCm" : "Tenstorrent")) << "...");
+              << std::string(metal ? "Metal" : (rocm ? "ROCm" : "Tenstorrent"))
+              << "...");
     } else {
       REQUIRE_MESSAGE(have_dev,
                       label << ": device oracle golden absent (" << ids_name << " / "
@@ -338,7 +342,7 @@ void RunGate(const std::string& repo_dir, const std::string& golden_subdir,
                     << " engine=" << (first_div < 0 ? -1 : got[static_cast<size_t>(first_div)])
                     << " committed anchor="
                     << (first_div < 0 ? -1 : anchor_ids[i * T + first_div])
-                    << (device_golden
+                    << std::string(device_golden
                             ? " — re-capture the device golden pair via "
                               "qwen3-neartie-gap.py"
                             : " — re-run qwen3-neartie-gap.py to refresh the gap "
@@ -427,7 +431,8 @@ void RunGate(const std::string& repo_dir, const std::string& golden_subdir,
           << strict_exact << "/" << N << "; near-tie-band only: " << neartie_only
           << "/" << N << "; max gap " << (worst_gap / 1000.0) << " nats @ prompt["
           << worst_i << "] tok=" << worst_j << "; " << fail << " forward-divergent"
-          << (device_golden ? "; anchor+gaps = device oracle-backed golden" : "")
+          << std::string(device_golden ? "; anchor+gaps = device oracle-backed golden"
+                                        : "")
           << ")");
   REQUIRE(fail == 0);
 }