main is RED on sanitize-cpu (thread) with a heap-use-after-free, and nothing tracks it
test_resident_weight_host_addressable (ctest #260) fails under ThreadSanitizer
on main, reproducibly, and it is the only failing test in that lane:
260/574 Test #260: test_resident_weight_host_addressable ... ***Failed 0.97 sec
WARNING: ThreadSanitizer: heap-use-after-free (pid=24554)
SUMMARY: ThreadSanitizer: heap-use-after-free ... in memcpy
99% tests passed, 1 tests failed out of 574
It is main, not a pull request
| run |
job |
when |
test #260 |
32394151274 (main) |
96506970159 |
2026-08-20 18:26Z |
failed, same UAF |
32381718043 (main) |
96466616543 |
2026-08-20 15:41Z |
failed, same UAF |
32395464146 (PR #1477) |
96511340456 |
2026-08-20 19:09Z |
failed, same UAF |
#1477 touches no resident_weight file and no qwen3_5.cpp, so it inherits the
red rather than causing it. Same test number, same signature, three runs.
The frames
Write of size 8 by main thread:
#0 memcpy
#2 tests/vllm/model_executor/test_resident_weight_host_addressable.cpp:428
Previous write of size 8 at the same address by main thread:
#0 operator delete(void*, unsigned long)
#5 std::__cxx11::basic_string<...>::_M_dispose()
#7 ResidentWeight src/vllm/model_executor/models/qwen3_5.cpp:1205
#8 vllm::detail::StageWeightForTest(vt::Queue&, vllm::OwnedTensor const&)
src/vllm/model_executor/models/qwen3_5.cpp:7954
#9 tests/vllm/model_executor/test_resident_weight_host_addressable.cpp:351
Location is heap block of size 352 allocated by main thread:
std::__new_allocator<unsigned char>::allocate (the test's `backing` vector)
qwen3_5.cpp:1205 is the VT_CHECK(!w.repacked, ...) whose message string is
destroyed when the exception unwinds. The test case at :351 deliberately
triggers that throw via CHECK_THROWS_WITH_AS. The 352-byte block is the
backing vector that a LATER case allocates and then memcpys into at :428,
where OwnedBytes::Borrow(off, nb, backing) builds an aliased device view.
What this does NOT establish, and why it needs an owner rather than a guess
Two readings fit the trace and they have very different fixes:
- A genuine dangling alias. Something retains a view into a block that has
been freed, and the allocator handing that address to backing is what makes
it visible. If so this is a real memory-safety defect on the device-alias
path, and it is exactly the class OwnedBytes::Borrow's keepalive
shared_ptr exists to prevent.
- A sanitizer artefact of allocator reuse across an exception boundary. The
string freed during unwind and the vector allocated afterwards share an
address, and the shadow state was not cleared.
I have not isolated which, and I am not going to assert one. It reproduces
deterministically in CI, so whoever owns it can bisect it cheaply — and reading 2
is the one that would make this safe to ignore, which is precisely the reason not
to assume it.
Why it matters now
sanitize-cpu (thread) is one of two sanitizer lanes and it is red on main,
so every pull request inherits a red there and must argue its way past it. That
is the condition in which a genuine sanitizer regression becomes invisible: the
lane is already red, so the next person attributes their red to this and merges.
build-test-cpu is in the same state on main (#1439, #1464, #1470, #1494), so
two of the fourteen reporting lanes now carry a standing red.
Found while holding #1477 for a CI verdict rather than merging on local gates.
mainis RED onsanitize-cpu (thread)with a heap-use-after-free, and nothing tracks ittest_resident_weight_host_addressable(ctest #260) fails under ThreadSanitizeron
main, reproducibly, and it is the only failing test in that lane:It is
main, not a pull request32394151274(main)9650697015932381718043(main)9646661654332395464146(PR #1477)96511340456#1477 touches no
resident_weightfile and noqwen3_5.cpp, so it inherits thered rather than causing it. Same test number, same signature, three runs.
The frames
qwen3_5.cpp:1205is theVT_CHECK(!w.repacked, ...)whose message string isdestroyed when the exception unwinds. The test case at
:351deliberatelytriggers that throw via
CHECK_THROWS_WITH_AS. The 352-byte block is thebackingvector that a LATER case allocates and thenmemcpys into at:428,where
OwnedBytes::Borrow(off, nb, backing)builds an aliased device view.What this does NOT establish, and why it needs an owner rather than a guess
Two readings fit the trace and they have very different fixes:
been freed, and the allocator handing that address to
backingis what makesit visible. If so this is a real memory-safety defect on the device-alias
path, and it is exactly the class
OwnedBytes::Borrow's keepaliveshared_ptrexists to prevent.string freed during unwind and the vector allocated afterwards share an
address, and the shadow state was not cleared.
I have not isolated which, and I am not going to assert one. It reproduces
deterministically in CI, so whoever owns it can bisect it cheaply — and reading 2
is the one that would make this safe to ignore, which is precisely the reason not
to assume it.
Why it matters now
sanitize-cpu (thread)is one of two sanitizer lanes and it is red onmain,so every pull request inherits a red there and must argue its way past it. That
is the condition in which a genuine sanitizer regression becomes invisible: the
lane is already red, so the next person attributes their red to this and merges.
build-test-cpuis in the same state onmain(#1439, #1464, #1470, #1494), sotwo of the fourteen reporting lanes now carry a standing red.
Found while holding #1477 for a CI verdict rather than merging on local gates.