Skip to content

qwen3_5 decode-graph poison hook wrote past the end of the pinned seq_lens block on a spec step #1319

Description

@localai-bot

Found while migrating the Qwen3.5 drivers onto the break-point capture seam (#1307, parent #1163). NOT caused by that work — the shape is present at its base commit 5c8671c50.

MaybePoisonStagedInputs (src/vllm/model_executor/models/qwen3_5.cpp:9848-9855 @ 5c8671c50) zeroed FOUR pinned staging blocks over pin.S elements each:

std::fill(pin.positions,    pin.positions + pin.S, 0);
std::fill(pin.slot_mapping, pin.slot_mapping + pin.S, 0);
std::fill(pin.seq_lens,     pin.seq_lens + pin.S, 0);   // <-- allocated with R
std::fill(pin.token_ids,    pin.token_ids + pin.S, 0);

But seq_lens is allocated with R elements, not S (:9733, AllocPinned(sizeof(int32_t) * R)), and PinnedStepInputs keeps the two counts apart deliberately — SPEC-DSPARK W8 (#442) separated them because a speculative verify arrives with S = R * (1 + k). On a pure-decode step S == R and the fill is exact; on a spec step it writes (S - R) int32s past the end of a cudaHostAlloc'd block.

Severity is bounded and stated. The hook only runs under VT_ASYNC_EXECUTOR_POISON=1, which is a test-only escape hatch (scripts/env-doc-allowlist.txt:6, "Never set in production"), so no production path reaches it. It is the deterministic RED arm of tests/parity/test_qwen36_async_serving.cpp:170-173, which is checkpoint-gated and dgx-only, so the overrun would corrupt pinned memory belonging to another allocation exactly where the gate is trying to prove an event boundary sound — a defect in the instrument, in the arm that is supposed to fail loudly.

Fixed in flow by #1307, as a consequence of routing the staging through vt::PersistentStepInput: the hook now memsets each cell over its own capacity(), which is per-cell and cannot disagree with the allocation. Two of the four fills also disappear — block_table and qsl were never poisoned, and token_ids' pinned block is gone entirely because nothing ever uploaded it.

Close when #1307 merges.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions