fix(SAMPLER-HOST-LOGITS-DEVICE-TAG): the on-device sampler was handed a host address, and the CUDA arm dereferences it

[localai-bot]

fix(SAMPLER-HOST-LOGITS-DEVICE-TAG): the on-device sampler was handed a host address, and the CUDA arm dereferences it

GPUModelRunner built the [rows, vocab] logits tensor the on-device sampler
runs on from a host std::vector<float> pointer and stamped queue_.device on
it. The issue asked whether vt::GreedyArgmax copies before touching that
pointer, because if it did this would be mislabelling rather than a defect. It
does not. src/vt/cuda/cuda_sample.cu::GreedyArgmaxCuda passes
logits.Ptr<float>() straight into a kernel launch on both arms --
src/vt/cuda/cuda_sample.cu::GreedyArgmaxKernelSlow and
src/vt/cuda/cuda_sample.cu::ArgmaxPartialKernel -- and ApplyTemperature,
ApplyTopKTopP, ComputeProbs and ComputeLogprobs reach the same pointer
through the same gate, src/vt/ops.cpp::CheckSamplingLogits. The address IS
dereferenced on device.

Every citation in this change is the SYMBOL form, and that is a repair rather
than a style choice. This description previously cited src/vt/ops.cpp:3605-3665
for that gate. The gate is at :3863 now, so the cited range had drifted into
MlaPrefillAttention -- it still resolved, and it had stopped saying anything.
The same dead anchors were sitting in the two other places AGENTS.md requires to
agree with this text, the #1313 row in .agents/issue-index.md and the spec,
and five more runner.cpp line anchors in the spec were stale too. The index row
is the one that could not wait: that file is append-only, so a row may never be
edited after it lands, and scripts/check-symbol-anchors.py lists it in
FROZEN_FILES for exactly that reason -- nothing would ever have reported the rot
and nothing could ever have fixed it. All ten symbols are verified by whole-word
search against the file each names, at the head this pull request pushes.

It survives here because CudaBackend::UnifiedMemory() is
caps.pageable_memory_access && caps.integrated
(src/vt/cuda/cuda_backend.cu::UnifiedMemory). GB10 is integrated, so the driver services
an ordinary host pointer through ATS and the NemotronH A3 gate reads
96/96 STRICT PASS over it at host-memory latency. A discrete GPU reports
integrated == false and the same address is illegal.

The audit found four such sites, not the one the issue names: the (A')
VT_GPU_SAMPLE=0 download-then-sample A/B leg, the reported host-logits leg,
the (B) VT_LOGITS_GATHER=0 re-gather leg, and the collect_prompt_logprobs
prompt-row slice. All four are fixed together, which is what covers all three
affected models -- nemotron_h, laguna and qwen3_vl, exactly the
ForwardLogits.host producers listed in
scripts/runner-routing-allowlist.txt. A2-Q2b removing NemotronH's host-logits
path leaves the other two fixed.

include/vllm/v1/sample/device_scratch.h already states the contract this
needed -- unified backends wrap the host buffer in place at zero copy, discrete
backends allocate and copy up -- and applies it to the small derived sampling
tensors. The largest tensor in the path, the one every sampling kernel reads,
bypassed it. HostBufferStaging puts it back, with one grow-only allocation
reused across steps rather than DeviceScratch's alloc/free per construction,
because this sits on the per-token decode path. Its unified branch is the
expression it replaces, unchanged, so GB10 does not move.

Two staging buffers, not one: collect_prompt_logprobs stages the prompt rows
while the assembled sample-logits tensor is still live, so a shared buffer would
invalidate the tensor the sampler is about to read.

The user-facing consequence is one bullet under ## First-line troubleshooting
in docs/USAGE.md: which three models stage, that a unified-memory device
samples the logits where they are, that a discrete GPU pays one host-to-device
transfer of rows x vocab x 4 bytes per step, that an older build aborted on an
illegal address there instead, and that the discrete arm is gated at the seam
with no hardware run behind it. #1491 rewrote that page while this branch sat,
so the note sits in the restructured page's own idiom rather than in the section
it was originally written against, which no longer exists.

Rejected, and recorded in the spec because both look cheaper than they are. A
sampler that refuses a host pointer carrying a device tag converts a repairable
path into a refusal -- the three models would stop decoding on a discrete GPU
rather than start working -- and it cannot separate a GB10-valid host pointer
from a discrete-invalid one without the same per-backend probe the repair would
need. Building the tensor with the host device instead needs a host queue and
the whole pipeline re-dispatched to CPU, because CheckSamplingLogits requires
logits.device == q.device for every sampling op; that regresses GB10 from
on-device sampling to host sampling.

The issue's suggested residency assertion in vt::Tensor::Contiguous is
rejected too, and this is the part worth keeping: two registered backends
legitimately stamp a host-dereferenceable pointer with a non-unified device.
Tenstorrent's Alloc returns aligned_alloc host memory by design
(src/vt/tenstorrent/tenstorrent_backend.cpp::Alloc, with
src/vt/tenstorrent/tenstorrent_backend.cpp::UnifiedMemory returning false), and a discrete Vulkan device's buffers are HOST_VISIBLE and
persistently mapped (include/vt/backend.h::DeviceMemoryIsHostAddressable). Both register a
kGreedyArgmax provider, so a blanket assert false-fires on both.

The test uses fake backends rather than hardware, and says so in its header:
every backend registered on this fleet reports UnifiedMemory() == true, so the
defect is invisible here by construction. A Backend over host memory on the
otherwise-unused kXPU slot -- the idiom tests/vt/test_reference_tier.cpp
already uses to gate the reference tier's discrete/unified split without a GPU
-- gives one unified instance and one discrete instance. The unified case pins
the zero-copy wrap, which is the executable form of "A3 stays 96/96".

RED, against the seam holding the expression main has today:

[doctest] test cases:   4 |   2 passed | 2 failed | 0 skipped
[doctest] assertions: 913 | 908 passed | 5 failed |
[doctest] Status: FAILURE!
RC=1

GREEN:

[doctest] test cases:   4 |   4 passed | 0 failed | 0 skipped
[doctest] assertions: 913 | 913 passed | 0 failed |
[doctest] Status: SUCCESS!
RC=0

commit-protocol-tag was RED on this branch, and the first of its two steps was
not the one that failed. The grep step reported that all new commits carry the
protocol marker, because it skips merge commits; the STRICT step,
scripts/check-commit-trailers.py, does not skip them, and it named the commit:
3a883f81999f, which carried Git's default Merge remote-tracking branch ...
subject and no body at all, because it was made with git merge --no-edit. The
five commits from it to the tip were rebuilt with git commit-tree, which is
why their SHAs move. Only that one message changed. Every rebuilt commit reuses
its original tree object and its original author and committer identity and
date, verified pairwise, and git diff <old-tip> <new-tip> is empty.

Closes #1313

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]