fix(VT-REFTIER-HOST-ADDRESSABLE): the reference tier gated on unified memory, not on whether the host can address it (#844, #1435)

[localai-bot]

ReferenceTierEligible decides whether a device may receive the portable CPU
reference tier. It asked Backend::UnifiedMemory(). A host kernel does not need
unified memory; it needs to be able to dereference what Backend::Alloc
returned. include/vt/backend.h already carried the refutation, beside the
predicate that asks the right question:

CUDA on GB10 reports unified memory because host and device address the same
physical RAM, yet a plain cudaMalloc pointer is still not
host-dereferenceable.

Default false: a backend must OPT IN, because being wrong here hands a device
pointer to a host memcpy and segfaults.

CudaBackend::Alloc calls cudaMalloc. So every CUDA op without a native
kernel installed the CPU host kernel as a vt-cpu-ref provider, that kernel
dereferenced device pointers, and the process took SIGSEGV — after printing a
banner that claimed the opposite:

[vt reference-tier] op=MatmulFp8BlockScaled device=cuda has NO native kernel;
                    running the PORTABLE CPU fallback (correct but slow)

This is #844, measured twice on GB10: vt::QuantFp8Static on sm_110 (#960),
and vt::MatmulFp8BlockScaled on a CUTLASS-less CUDA build (#1435), where
tests/vt/test_ops_matmul_fp8_block_cuda exits 139 with
test cases: 0 assertions: 0. The second is reachable on a default build:
VLLM_CPP_CUTLASS_FETCH defaults OFF and CUTLASS is not a submodule.

Why this got a row and a spec, and not an in-flow fix

The reference tier is shared dispatch. Every op that can fall back, on every
backend, resolves through this one predicate, so the blast radius is the whole
op table rather than one FP8 kernel. .agents/issue-index.md had already
recorded the same judgement on #1435 — "both are code changes on a CUDA path
needing their own row, spec and hardware re-gate" — and
.agents/specs/vt-fp8-quant-arch-gate.md ## Outcome recorded that #960
deliberately did not take this on: "making the reference tier refuse ... is
#844's class, which is a larger change to the tier and is deliberately still
open".

So: row VT-REFTIER-HOST-ADDRESSABLE, spec
.agents/specs/vt-reference-tier-host-addressable.md,
committed in 447cb5c9e before the implementation in 817e1769c. The commit
order is the proof. The row is deliberately not added to an area matrix:
BACKEND-ACCEL-PROVIDER already carries an ACTIVE row for this seam, and a
second row would put a shared-file edit in the path of a one-predicate fix.

The change

ReferenceTierEligible asks Backend::DeviceMemoryIsHostAddressable(). No new
seam, no new virtual, and no DeviceType branch — the audit's rule that
eligibility never keys on the device type is kept.

Two backend cells move, and the spec enumerates all five to show which:

Backend	Alloc	unified	host addressable	tier before	tier after
CPU	host	true	the tier's source, never a target	no	no
CUDA	cudaMalloc	true on GB10	false	yes	no
Vulkan	host-visible, host-coherent, persistently mapped	true only where a combined host-visible, host-coherent, DEVICE_LOCAL type exists	true, unconditionally, already overridden	yes, except where unified is false	yes, always
Metal	MTLResourceStorageModeShared	hasUnifiedMemory	true, once it answers	yes	yes
ROCm	managed when integrated, else hipMalloc	see below	equal to unified	yes	yes

CUDA loses the tier — that is the crash. Vulkan widens, and the first draft
of this description got that wrong by claiming only one cell moved.
VulkanBackend::DeviceMemoryIsHostAddressable() already returned true
unconditionally, while UnifiedMemory() reads VulkanContext::unified_memory(),
which is set from
FindMemoryType(mem, ~0u, kHostFlags | VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT) >= 0.
Where no such combined type exists that is false, the context falls back to
FindMemoryType(mem, ~0u, kHostFlags) and then VT_CHECKs that one was found.
So on that device every allocation is host-visible and host-coherent anyway, and
a tier that used to be withheld is now installed. Sound because of that
VT_CHECK, not because a portable tier is harmless.

Metal and ROCm gain a one-line override so they do not lose the tier as a side
effect. Both return the value those backends already report for unified memory,
so their behaviour is byte-identical. ROCm answers unified_memory_ rather than
managed_alloc_ on purpose: the registrar sets
unified_memory_ = managed_alloc || (pageable_memory_access && integrated), and
both branches give the host a pointer it may dereference — the first by the
hipMallocManaged contract, the second by the PageableMemoryAccess attribute
that the backend's own comment records as measured on gfx1151 and gfx1103.

Two messages change with it. The refusal names which precondition the device
failed, so a reader does not have to open op_provider.cpp to learn that a
portable fallback exists and was withheld. The banner names the fact that makes
it valid instead of asserting correctness in three words — those three words are
what #844 quotes as having misled a reader past a
SIGSEGV.

Red first, then green

The condition is reproducible with no GPU: it is a property pair on a backend,
not a hardware event. tests/vt/test_reference_tier.cpp gains a fake backend
that reports UnifiedMemory() == true and
DeviceMemoryIsHostAddressable() == false — the GB10 CUDA shape — on its own
device slot, because a reference-tier provider cannot be uninstalled once
registered and the case must observe an empty table. That slot is skipped, not
asserted empty; see the review repairs below for why that distinction was a
must-fix.

RED, on the unfixed tree at 447cb5c9e (ANSI stripped, both lines read):

test cases:  6 |  5 passed | 1 failed | 0 skipped
assertions: 34 | 26 passed | 8 failed |
Status: FAILURE!            TEST_RC=1

Failing for the intended reason, not an incidental one:

ERROR: CHECK_FALSE( vt::ReferenceTierEligible(kSlot) ) is NOT correct!
  values: CHECK_FALSE( true )
ERROR: CHECK( vt::RegisterReferenceTier(kSlot) == 0 ) is NOT correct!
  values: CHECK( 107 == 0 )
ERROR: CHECK( thrown_fn != vt::GetOp(OpId::kRelu, DeviceType::kCPU) ) is NOT correct!
  values: CHECK( 1 != 1 )

107 host kernels installed for a device the host cannot address, and GetOp
returned the CPU function pointer rather than throwing. That last assertion is
what separates a refusal from a fallback: the tier installs src->fn, the very
same pointer, so an equal answer is the defect even when nothing throws.

GREEN, at 817e1769c:

test cases:  6 |  6 passed | 0 failed | 0 skipped
assertions: 34 | 34 passed | 0 failed |
Status: SUCCESS!            TEST_RC=0

The rest of the suite

Full CPU build clean (-Werror, 1130/1130, BUILD_RC=0, zero error: lines),
then ctest -j 6 over the whole tree:

99% tests passed, 1 tests failed out of 574
The following tests FAILED:
	309 - test_engine_core_proc (Failed)

That one is #1052, which the issue index already records at this exact
assertion — tests/vllm/v1/test_engine_core_proc.cpp:481 "searches for the abort
frame over a FIXED bud[get]" and needs the parallel harness to reproduce. It is
not reachable from this change: grep -c 'UnifiedMemory\|RegisterBackend\|ReferenceTier'
over that test file is 0, so it registers no backend, and on a CPU-only
build TryGetBackend returns nullptr for every non-CPU device, which makes the
old and new predicates return identically false. Rerun standalone three times
here: SUCCESS, FAILURE, SUCCESS, with assertion counts of 109, 1104 and
116 — a race, not a verdict. Another session was running its own ctest on this
box at the same time.

The seam's own suites, run together afterwards: test_reference_tier,
test_op_provider, test_backend_cross_device (4 variants),
test_backend_multidevice — 7/7 passed.

Mutation

The fix mutated in a scratch copy — the predicate put back to UnifiedMemory(),
one line:

$ diff -u op_provider.cpp.orig src/vt/op_provider.cpp
-  return b != nullptr && b->DeviceMemoryIsHostAddressable();
+  return b != nullptr && b->UnifiedMemory();
 1 file changed, 1 insertion(+), 1 deletion(-)

MUT_BUILD_RC=0        compile_err: (none)
MUT_TEST_RC=1
test cases:  6 |  5 passed | 1 failed | 0 skipped
assertions: 34 | 26 passed | 8 failed |
Status: FAILURE!

Both the diffstat and the compile result are printed because a mutation that
never applied and a mutation that failed to build each read as a passing test.
The tree was then restored and re-verified byte-for-byte:
sha256 e9ad9fd6e69124153b9bd34a2993c05f5442acaa8a507cb9437e0a463d711a39 before
and after, green again at 6/6 and 34/34.

The one .cu file, compiled by a real nvcc

src/vt/cuda/cuda_attention_cross.cu carried a header note whose premise this
change removes: it said GB10 would have run the DiT's cross-attention on the
host, "running, correct, and making 'the forward ran on the GPU' false". It was
never correct — the host kernel dereferences device pointers — and after this
change it is not possible. The edit is comment-only.

Comment-only is not a licence to skip nvcc here, so it did not:
rc run -d thor:gpu0, job 645bf395-23fc-408f-a9ad-b9823885622c, NVIDIA Thor,
CUDA 13.0 V13.0.88, aarch64. Configure with -DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a and no CUTLASS, then the single object:

[1/1] Building CUDA object CMakeFiles/vllm.dir/src/vt/cuda/cuda_attention_cross.cu.o
NVCC_TU_RC=0
object bytes: 162576
ELF file    1: cuda_attention_cross.cu.1.sm_121a.cubin
RESULT=PASS

The postcondition is asserted, not inferred from rc=0: the object exists and
carries sm_121a SASS.

A second, unplanned measurement

That same configure printed, on a build containing no CUTLASS at all:

--   CUDA feature cutlass-nvfp4: ENABLED for [121a]
--   CUDA feature cutlass-fp8: ENABLED for [121a]

#1435 measured that on dgx:gpu0 for one feature cell. A second box and a
second cell make it a property of vt_cuda_report_feature reading the
architecture intersection before the CUTLASS detection, not a property of either
host. Recorded in ## Owed by 7c07270b6, where that half of #1435 lives.

Fresh review: five findings, all repaired in b2c29e8f9

F1, and it was a must-fix. The new case took kTENSTORRENT believing the
enumerator was unused. It is not: src/vt/tenstorrent/tenstorrent_ops.cpp
registers 21 op sites on it, OpId::kRelu among them, and vllm carries an
INTERFACE --whole-archive, so on -DVLLM_CPP_TENSTORRENT=ON that registrar
runs inside this test binary. The old REQUIRE(OpProviderCount(kRelu, kSlot) == 0)
was commented as failing loudly if something were wrong. Nothing has to be wrong:
it fails unconditionally on a supported configuration no CI lane builds, so it
would have landed as a permanent red for whoever next enabled that backend, and
RegisterBackend would have displaced a live backend on the way.

Verified here both ways, with a scratch static registrar reproducing what
--whole-archive does:

(a) pre-guard code + simulated Tenstorrent registrar
    FATAL ERROR: REQUIRE( vt::OpProviderCount(OpId::kRelu, kSlot) == 0 )
    test cases:  6 |  5 passed | 1 failed        BUILD_RC=0  compile_err: none
    assertions: 23 | 22 passed | 1 failed        git diff --stat: 12 insertions(+)
    Status: FAILURE!                             TEST_RC=1

(b) guarded code + the SAME simulated registrar
    MESSAGE: SKIP: DeviceType::kTENSTORRENT carries a backend or providers ...
    test cases:  6 |  6 passed | 0 failed        BUILD_RC=0  compile_err: none
    assertions: 22 | 22 passed | 0 failed        git diff --stat: 32 insertions(+)
    Status: SUCCESS!                             TEST_RC=0

(a) reproduces the reviewer's numbers exactly. The tree was restored
byte-for-byte afterwards (sha256 0735aeb9…f02741b) and rebuilt green. The slot
itself stays: kXPU is the only DeviceType with no production registration and
this case cannot share it, because a reference-tier provider cannot be
uninstalled once registered. What was missing was the guard, not a different
slot.

F2. include/vt/backend.h still described the tier as reading
UnifiedMemory(), at the FlushPending() comment and at TryGetBackend. That
is the header a reader opens for exactly this distinction.

F3. "Exactly one cell moves" was wrong; corrected above, in the spec's
Design table and Risks item 1, and in the comment in op_provider.cpp.

F4. Split the remainder of #844 into #1482 rather than leaving it owned by an
issue this PR closes. Reasoning under Issues below.

F5. Three Mamba2 guards justified themselves with "ReferenceTierEligible(kCUDA)
is TRUE … GetOp does not throw". Both halves are now false. The guards are
worth keeping — they still catch a run-time decline, and a future backend
answering the narrow predicate true restores the hazard exactly — so each
paragraph now reads as the reason the guard exists, and says not to delete it.
Comment-only; suites unchanged and green:
test_ops_mamba2_ssd 8/8 · 1175 assertions, test_ops_mamba2_state_update 6/6 ·
2469, test_ops_mamba2_gated_norm 9/9 · 2107.

Reachability, run by the reviewer and not by me. Deleting the refusal clause
from the VT_CHECK inside Resolve — the production path every vt::GetOp
takes — reds test_reference_tier.cpp. So the new message is measured through a
production entry point, not by calling the helper directly. Recorded in the
spec's gates, because "nothing lands dead" asks for exactly this.

Focused suite after the repairs: 6 cases, 6 passed, 33 assertions, 33 passed,
Status: SUCCESS! — one assertion fewer than the 34 quoted above, because the
precondition REQUIRE became the skip guard. Seam suites re-run together: 8/8.

What this does NOT claim

Nothing about the block-wise FP8 arm. It has no token gate, no speed claim,
and its correctness is established only on the seven shapes actually run under
#1437. This change turns one crash into one named refusal and claims nothing
else about that arm.

No CUDA run. The defect was measured on GB10 and the fix is verified on the
host tier against a fake backend carrying the exact property pair. A CUDA re-run
of tests/vt/test_ops_matmul_fp8_block_cuda on a CUTLASS-less build, to record
the refusal replacing exit 139, is owed by this row and named in the spec.

Metal and ROCm are not compiled anywhere this change can reach. CI has no
macOS or ROCm job and this host has neither toolchain. That is why each override
is one line, placed beside the existing UnifiedMemory() override in the same
class, returning a value those backends already compute.

Issues

Fixes #844.

#1435 stays OPEN, deliberately. Its vt_cuda_report_feature half needs a CUDA
configure and is listed under ## Owed in this row's spec, with the thor
reproduction above.

#1482 is new, and it is why Fixes #844 is safe. Two of #844's four "what done
looks like" items are not done here: item 1 also wants the refusal to name the
BUILD FEATURE that would have supplied the kernel (the landed message names the
op, the device and the failed precondition, not this build has no cutlass-fp8), and item 4 wants a warning at engine construction rather than
only at configure time. The spec originally named #844 itself as the owner of
that remainder, which would have pointed at a closed issue the moment this
merged. #1482 carries it, owned by this row, indexed, and referenced from
## Owed.

That sentence and the one above are worded to keep a closing keyword away from
every number except 844: PR #1155 auto-closed issue #584 from inside ordinary
prose.

NOTE ON THE MERGE. docs/USAGE.md is absent from this change's file list.
It carried the same misleading guidance and was going to be corrected here,
but #1491 RETIRED that section outright rather than relocating it -- its
migration manifest routes it to docs/BUILD.md as deduplicated, and no
prose matching correct but slow, portable CPU fallback or NO native kernel survives anywhere under docs/. Re-applying the edit would have
resurrected deleted content, so the file resolves to origin/main byte for
byte and the debt is discharged by the deletion.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]