Skip to content

main is RED on sanitize-cpu (thread): a heap-use-after-free in test_resident_weight_host_addressable, untracked #1499

Description

@localai-bot

main is RED on sanitize-cpu (thread) with a heap-use-after-free, and nothing tracks it

test_resident_weight_host_addressable (ctest #260) fails under ThreadSanitizer
on main, reproducibly, and it is the only failing test in that lane:

260/574 Test #260: test_resident_weight_host_addressable ... ***Failed  0.97 sec
WARNING: ThreadSanitizer: heap-use-after-free (pid=24554)
SUMMARY: ThreadSanitizer: heap-use-after-free ... in memcpy
99% tests passed, 1 tests failed out of 574

It is main, not a pull request

run job when test #260
32394151274 (main) 96506970159 2026-08-20 18:26Z failed, same UAF
32381718043 (main) 96466616543 2026-08-20 15:41Z failed, same UAF
32395464146 (PR #1477) 96511340456 2026-08-20 19:09Z failed, same UAF

#1477 touches no resident_weight file and no qwen3_5.cpp, so it inherits the
red rather than causing it. Same test number, same signature, three runs.

The frames

Write of size 8 by main thread:
  #0 memcpy
  #2 tests/vllm/model_executor/test_resident_weight_host_addressable.cpp:428

Previous write of size 8 at the same address by main thread:
  #0 operator delete(void*, unsigned long)
  #5 std::__cxx11::basic_string<...>::_M_dispose()
  #7 ResidentWeight   src/vllm/model_executor/models/qwen3_5.cpp:1205
  #8 vllm::detail::StageWeightForTest(vt::Queue&, vllm::OwnedTensor const&)
                     src/vllm/model_executor/models/qwen3_5.cpp:7954
  #9 tests/vllm/model_executor/test_resident_weight_host_addressable.cpp:351

Location is heap block of size 352 allocated by main thread:
  std::__new_allocator<unsigned char>::allocate  (the test's `backing` vector)

qwen3_5.cpp:1205 is the VT_CHECK(!w.repacked, ...) whose message string is
destroyed when the exception unwinds. The test case at :351 deliberately
triggers that throw via CHECK_THROWS_WITH_AS. The 352-byte block is the
backing vector that a LATER case allocates and then memcpys into at :428,
where OwnedBytes::Borrow(off, nb, backing) builds an aliased device view.

What this does NOT establish, and why it needs an owner rather than a guess

Two readings fit the trace and they have very different fixes:

  1. A genuine dangling alias. Something retains a view into a block that has
    been freed, and the allocator handing that address to backing is what makes
    it visible. If so this is a real memory-safety defect on the device-alias
    path, and it is exactly the class OwnedBytes::Borrow's keepalive
    shared_ptr exists to prevent.
  2. A sanitizer artefact of allocator reuse across an exception boundary. The
    string freed during unwind and the vector allocated afterwards share an
    address, and the shadow state was not cleared.

I have not isolated which, and I am not going to assert one. It reproduces
deterministically in CI, so whoever owns it can bisect it cheaply — and reading 2
is the one that would make this safe to ignore, which is precisely the reason not
to assume it.

Why it matters now

sanitize-cpu (thread) is one of two sanitizer lanes and it is red on main,
so every pull request inherits a red there and must argue its way past it. That
is the condition in which a genuine sanitizer regression becomes invisible: the
lane is already red, so the next person attributes their red to this and merges.
build-test-cpu is in the same state on main (#1439, #1464, #1470, #1494), so
two of the fourteen reporting lanes now carry a standing red.

Found while holding #1477 for a CI verdict rather than merging on local gates.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions