A token-exact gate on Qwen/Qwen3.6-35B-A3B bf16 vs the pinned oracle came back 6/7 prompts strict 16/16, 108/112 positions. The single divergence is not a numerical defect — it is a tie-break convention difference.
prompt[6] "import numpy as np", position 7
oracle 464 "import" logprob -0.8293954133987427
ours 1445 "from" logprob -0.8293954133987427
top2_gap_mnats 0.0
oracle_minus_ours_mnats 0.0
our rank in oracle top-20: 2
The oracle holds both candidates at a bit-identical logprob — a 0.0-millinat gap. torch.argmax returns the lowest maximal index, so vLLM deterministically takes 464 "import". Our on-device argmax took 1445 "from". Same logits, different rule for resolving the tie.
Why this is worth fixing even though the gate passes
Under the projects near-tie doctrine an exact tie is a pass, and this one is. But AGENTS.md is explicit that where vLLM defines behavior we mirror it rather than re-invent — and this is a behavioral divergence from the reference that happens to be benign here and would not be benign everywhere:
- It is deterministic, not random: the same prompt diverges every run, so it is a permanent difference in output for any input that produces an exact tie.
- Ties are not rare in low-entropy positions — code prompts, repeated structure, and greedy decode after a strong prefix all produce them.
- It costs a strict gate. This run would be 7/7 STRICT rather than 6/7-plus-a-ratified-tie, and every future greedy gate inherits the same avoidable noise.
- Anyone debugging a future divergence has to re-derive that ties are involved before they can dismiss it, exactly as happened here.
Fix
Break ties toward the lower token id in the on-device argmax, matching torch.argmax. Cite the upstream behavior at the call site so the convention is not re-litigated.
Needs a RED-first test that constructs an exact tie and asserts the lower id wins — a test that merely checks "argmax picks a maximum" passes both conventions and proves nothing. Inertness for existing greedy goldens must be shown: any golden whose tokens move was previously recording our tie-break, and that is worth knowing.
Found while adjudicating the 35B bf16 MoE token gate for #864.
A token-exact gate on
Qwen/Qwen3.6-35B-A3Bbf16 vs the pinned oracle came back 6/7 prompts strict 16/16, 108/112 positions. The single divergence is not a numerical defect — it is a tie-break convention difference.The oracle holds both candidates at a bit-identical logprob — a 0.0-millinat gap.
torch.argmaxreturns the lowest maximal index, so vLLM deterministically takes464 "import". Our on-device argmax took1445 "from". Same logits, different rule for resolving the tie.Why this is worth fixing even though the gate passes
Under the projects near-tie doctrine an exact tie is a pass, and this one is. But AGENTS.md is explicit that where vLLM defines behavior we mirror it rather than re-invent — and this is a behavioral divergence from the reference that happens to be benign here and would not be benign everywhere:
Fix
Break ties toward the lower token id in the on-device argmax, matching
torch.argmax. Cite the upstream behavior at the call site so the convention is not re-litigated.Needs a RED-first test that constructs an exact tie and asserts the lower id wins — a test that merely checks "argmax picks a maximum" passes both conventions and proves nothing. Inertness for existing greedy goldens must be shown: any golden whose tokens move was previously recording our tie-break, and that is worth knowing.
Found while adjudicating the 35B bf16 MoE token gate for #864.