A full test sweep on dgx.casa found five red binaries. A control build proves all five are pre-existing and not caused by the change that found them.
Method
Two CUDA builds on the same host, the same container toolchain (cmake 3.28.3, nvcc 13.0.88) and the same flags (-DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_TRITON=ON -DVLLM_CPP_CUTLASS_FETCH=ON). Both passed the degraded-build guards, so both carry CUTLASS NVFP4 and FP8, marlin-nvfp4, Triton AOT sm_121a and FlashAttention-2.
- Experiment:
98f8e046d (main with ENG-WEIGHT-OFFLOAD W0a to the totality guard).
- Control:
10b8bbdaa, the commit before that row's first code change. Verified to contain zero weight-offload files.
Result
| Binary |
Experiment |
Control |
Same |
test_capi |
rc=139, 47 of 47, SIGSEGV |
rc=139, 47 of 47, SIGSEGV |
yes |
test_cuda_ops |
439 of 440 |
439 of 440 |
yes |
test_linear_method |
83 of 85 |
83 of 85 |
yes |
test_ops_gdn |
4899 of 4900 |
4899 of 4900 |
yes |
test_qwen3_5_gdn_spec_routing |
119 of 123 |
119 of 123 |
yes |
Identical exit codes, assertion counts, failure counts and crash site. The remaining 446 binaries pass.
test_capi crashes in an ABI v8 path
tests/capi/test_capi.cpp:487
TEST CASE: capi: custom logits processor forces the generated token (ABI v8)
FATAL ERROR: test case CRASHED: SIGSEGV
A crash aborts the whole binary, so 59 of its 63 cases never run. This is likely the same defect as #547 (portable CPU reference tier segfaults on CUDA and GB10) and #844 (the vt reference-tier fallback segfaults on device tensors): a custom logits processor is a host callback over what are device tensors on this hardware. That link is a hypothesis and needs confirming before either issue is closed.
Owners
test_ops_gdn already has #614. test_capi plausibly belongs to #547 or #844. test_cuda_ops, test_linear_method and test_qwen3_5_gdn_spec_routing have no issue, which is why this one exists.
Two notes for whoever picks this up
Two further binaries, test_modelopt_mixed_precision_checkpoint and test_voxtral_e2e, exit 77 with zero assertions. They refuse to run for want of checkpoints, because the tree was transferred with git archive and carries no weights. That is a harness limit, not a defect, and they are excluded from the five above.
The box is shared. During this work another session held 55 to 67 GiB of GPU memory on the 119 GiB unified pool, which OOM-killed one build at -j16. Build at -j4 while that is true, and note that the other job does not appear to take $HOME/gpu.lock.
A full test sweep on dgx.casa found five red binaries. A control build proves all five are pre-existing and not caused by the change that found them.
Method
Two CUDA builds on the same host, the same container toolchain (cmake 3.28.3, nvcc 13.0.88) and the same flags (
-DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_TRITON=ON -DVLLM_CPP_CUTLASS_FETCH=ON). Both passed the degraded-build guards, so both carry CUTLASS NVFP4 and FP8, marlin-nvfp4, Triton AOT sm_121a and FlashAttention-2.98f8e046d(main withENG-WEIGHT-OFFLOADW0a to the totality guard).10b8bbdaa, the commit before that row's first code change. Verified to contain zero weight-offload files.Result
test_capitest_cuda_opstest_linear_methodtest_ops_gdntest_qwen3_5_gdn_spec_routingIdentical exit codes, assertion counts, failure counts and crash site. The remaining 446 binaries pass.
test_capicrashes in an ABI v8 pathA crash aborts the whole binary, so 59 of its 63 cases never run. This is likely the same defect as #547 (portable CPU reference tier segfaults on CUDA and GB10) and #844 (the vt reference-tier fallback segfaults on device tensors): a custom logits processor is a host callback over what are device tensors on this hardware. That link is a hypothesis and needs confirming before either issue is closed.
Owners
test_ops_gdnalready has #614.test_capiplausibly belongs to #547 or #844.test_cuda_ops,test_linear_methodandtest_qwen3_5_gdn_spec_routinghave no issue, which is why this one exists.Two notes for whoever picks this up
Two further binaries,
test_modelopt_mixed_precision_checkpointandtest_voxtral_e2e, exit 77 with zero assertions. They refuse to run for want of checkpoints, because the tree was transferred withgit archiveand carries no weights. That is a harness limit, not a defect, and they are excluded from the five above.The box is shared. During this work another session held 55 to 67 GiB of GPU memory on the 119 GiB unified pool, which OOM-killed one build at
-j16. Build at-j4while that is true, and note that the other job does not appear to take$HOME/gpu.lock.