[Bug] Qwen-VL patch bypasses SGLang token-ID preservation and breaks multi-turn TITO

[dirtyDan0]

Bug Description

Slime changes Qwen-VL from load_mm_data() to a direct legacy_load_mm_data() call:

slime/docker/patch/latest/sglang.patch

Lines 1248 to 1260 in 4c193f1

	diff --git a/python/sglang/srt/multimodal/processors/qwen_vl.py b/python/sglang/srt/multimodal/processors/qwen_vl.py
	index b8774ebade56891fe0aef2f54aa26e39b0c63b23..fa01537b2010c6ee002de514281c0d36a2c3bad2 100644
	--- a/python/sglang/srt/multimodal/processors/qwen_vl.py
	+++ b/python/sglang/srt/multimodal/processors/qwen_vl.py
	@@ -678,7 +678,7 @@ class QwenVLImageProcessor(SGLangBaseProcessor):
	**kwargs,
	):
	entry_time = time.perf_counter()
	- base_output = await self.load_mm_data(
	+ base_output = await self.legacy_load_mm_data(
	prompt=input_text,
	image_data=image_data,
	video_data=request_obj.video_data,

This bypasses the code that preserves caller-provided token IDs.

In SGLang v0.5.15.post1, load_mm_data() first saves a list[int] prompt as input_ids, then
decodes the prompt for multimodal processing:

https://github.com/sgl-project/sglang/blob/0b3bb0cbe31873994c9f989fddfe2f87ca839fdd/python/sglang/srt/multimodal/processors/base_processor.py#L785-L821

input_ids = prompt if isinstance(prompt, list) else None
prompt = self._tokenizer.decode(prompt)

If load_mm_data() needs the legacy loader, it forwards the saved IDs through the separate
input_ids=input_ids argument:

https://github.com/sgl-project/sglang/blob/0b3bb0cbe31873994c9f989fddfe2f87ca839fdd/python/sglang/srt/multimodal/processors/base_processor.py#L833-L864

The Slime patch calls legacy_load_mm_data(prompt=input_text, ...) directly and does not pass its
separate input_ids argument. That argument therefore remains None:

https://github.com/sgl-project/sglang/blob/0b3bb0cbe31873994c9f989fddfe2f87ca839fdd/python/sglang/srt/multimodal/processors/base_processor.py#L952-L977

The caller IDs are still present in prompt, but legacy_load_mm_data() immediately decodes that
list into text. It later returns the decoded/reconstructed text together with input_ids=None:

https://github.com/sgl-project/sglang/blob/0b3bb0cbe31873994c9f989fddfe2f87ca839fdd/python/sglang/srt/multimodal/processors/base_processor.py#L1065-L1070

For raw images, process_and_combine_mm_data() then invokes the HF processor on that text and takes
the processor's newly generated ret["input_ids"]:

https://github.com/sgl-project/sglang/blob/0b3bb0cbe31873994c9f989fddfe2f87ca839fdd/python/sglang/srt/multimodal/processors/base_processor.py#L1188-L1204

SGLang's protection against this retokenization requires base_output.input_ids is not None:

https://github.com/sgl-project/sglang/blob/0b3bb0cbe31873994c9f989fddfe2f87ca839fdd/python/sglang/srt/multimodal/processors/base_processor.py#L1328-L1383

The caller must send pre-expansion rollout IDs: one image placeholder per image, together with the
exact token IDs returned by earlier generation turns. SGLang owns image-placeholder expansion for
its forward pass. The processor-expanded sequence can be retained separately for training.

Because the Slime patch leaves that field as None, the protection branch can never run. The raw
image path therefore always follows:

caller input_ids
  -> decode to text
  -> HF processor
  -> newly tokenized input_ids

This means retokenization is executed for every affected request. The resulting IDs do not
necessarily differ on every request: canonical token sequences may round-trip unchanged. For a
non-canonical sequence, encode(decode(input_ids)) can differ from the caller's IDs, causing token
drift.

SGLang upstream added the preservation path specifically for this problem in
sgl-project/sglang#26555

The fix is already present in the SGLang revision pinned by Slime, but the Slime patch bypasses it.

Steps to Reproduce

Run this CPU-only comparison in an SGLang v0.5.15.post1 source environment with the
Qwen/Qwen3.6-35B-A3B tokenizer. The first response ends with <|im_end|> and contains the valid
but non-canonical IDs [479, 3770] (["Ġtr", "uly"]) before it. Those two IDs decode to
" truly" and re-tokenize as the single ID [9149] (["Ġtruly"]). No SGLang source change or
model weights are required.

import asyncio
from difflib import SequenceMatcher

from transformers import AutoConfig, AutoProcessor

from sglang.srt.managers.schedule_batch import MultimodalInputFormat
from sglang.srt.multimodal.processors.base_processor import MultimodalSpecialTokens
from sglang.srt.multimodal.processors.qwen_vl import QwenVLImageProcessor
from sglang.srt.server_args import ServerArgs


async def main():
    model_name = "Qwen/Qwen3.6-35B-A3B"
    hf_processor = AutoProcessor.from_pretrained(model_name)
    tokenizer = hf_processor.tokenizer
    processor = QwenVLImageProcessor(
        AutoConfig.from_pretrained(model_name),
        ServerArgs(model_path=model_name),
        hf_processor,
        "default",
        skip_mm_pool=True,
    )
    # Turn 1 prompt, including one image placeholder.
    prompt_ids = tokenizer.encode(
        "<|im_start|>user\nWhat is shown?"
        "<|vision_start|><|image_pad|><|im_end|>\n"
        "<|im_start|>assistant\n",
        add_special_tokens=False,
    )

    # Turn 1 response exactly as generated by SGLang, including the assistant
    # turn terminator. Each ID is annotated with its token and decoded text:
    response_ids = [
        479,     # token "Ġtr", decoded fragment " tr"
        3770,    # token "uly", decoded fragment "uly"
        248046,  # token "<|im_end|>", decoded assistant-turn terminator
    ]

    # Turn 2 starts after the complete response above.
    next_prompt_ids = tokenizer.encode(
        "\n<|im_start|>user\nExplain your answer.<|im_end|>\n"
        "<|im_start|>assistant\n",
        add_special_tokens=False,
    )
    accumulated_ids = prompt_ids + response_ids + next_prompt_ids

    tokens = MultimodalSpecialTokens(
        image_token="<|image_pad|>",
        image_token_id=tokenizer.convert_tokens_to_ids("<|image_pad|>"),
    )
    tokens.parse_regex()
    # Valid preprocessed input keeps this loader comparison CPU-only.
    image_data = [{"format": MultimodalInputFormat.PROCESSOR_OUTPUT}]
    try:
        normal = await processor.load_mm_data(
            prompt=accumulated_ids,
            multimodal_tokens=tokens,
            image_data=image_data,
        )
        legacy = await processor.legacy_load_mm_data(
            prompt=accumulated_ids,
            multimodal_tokens=tokens,
            image_data=image_data,
        )
        _, normal_ids, _ = processor.process_and_combine_mm_data(normal, tokens)
        _, legacy_ids, _ = processor.process_and_combine_mm_data(legacy, tokens)
    finally:
        processor.io_executor.shutdown()
        processor.cpu_executor.shutdown()

    normal_ids = normal_ids.tolist()
    legacy_ids = legacy_ids.tolist()
    print("response tokens:")
    for index, token_id in enumerate(response_ids):
        print(
            f"  {index}: id={token_id}, "
            f"token={tokenizer.convert_ids_to_tokens(token_id)!r}, "
            f"text={tokenizer.decode([token_id])!r}"
        )
    print("load_mm_data preserved caller IDs:", normal.input_ids == accumulated_ids)
    print("legacy_load_mm_data input_ids:", legacy.input_ids)
    print("normal final IDs preserved caller IDs:", normal_ids == accumulated_ids)
    print(
        "legacy decoded text matches caller:",
        legacy.input_text == tokenizer.decode(accumulated_ids),
    )

    differences = []
    for tag, caller_start, caller_end, legacy_start, legacy_end in SequenceMatcher(
        a=accumulated_ids, b=legacy_ids, autojunk=False
    ).get_opcodes():
        if tag == "equal":
            continue
        caller_span = accumulated_ids[caller_start:caller_end]
        legacy_span = legacy_ids[legacy_start:legacy_end]
        differences.append((tag, caller_span, legacy_span))
        print(f"{tag}:")
        print(
            f"  caller/load_mm_data[{caller_start}:{caller_end}]",
            list(zip(caller_span, tokenizer.convert_ids_to_tokens(caller_span))),
        )
        print(
            f"  legacy retokenized[{legacy_start}:{legacy_end}]",
            list(zip(legacy_span, tokenizer.convert_ids_to_tokens(legacy_span))),
        )

    assert normal.input_ids == accumulated_ids
    assert legacy.input_ids is None
    assert normal_ids == accumulated_ids
    # Expected: the prior response's two IDs become one ID after retokenization.
    assert differences == [("replace", [479, 3770], [9149])]


asyncio.run(main())

Expected output:

response tokens:
  0: id=479, token='Ġtr', text=' tr'
  1: id=3770, token='uly', text='uly'
  2: id=248046, token='<|im_end|>', text='<|im_end|>'
load_mm_data preserved caller IDs: True
legacy_load_mm_data input_ids: None
normal final IDs preserved caller IDs: True
legacy decoded text matches caller: True
replace:
  caller/load_mm_data[14:16] [(479, 'Ġtr'), (3770, 'uly')]
  legacy retokenized[14:15] [(9149, 'Ġtruly')]

This matches the Slime patch exactly: it calls legacy_load_mm_data(prompt=input_text, ...) without
input_ids=input_text, so the original IDs are not retained for the anti-retokenization branch.

The original Qwen3-VL diagnosis captured the same pattern in three real multi-turn failures:

[350, 3140] ["ĠT", "CL"]       -> [65231] ["ĠTCL"]
[1760, 529] ["Ġcount", "ert"] -> [60110] ["Ġcountert"]
[1841, 424] ["Ġsign", "age"]  -> [79080] ["Ġsignage"]

Expected Behavior

For pre-tokenized image requests, the caller should send pre-expansion rollout IDs with one image
placeholder per image. SGLang should preserve every caller-provided non-image ID, including prior
response IDs, and expand the image placeholder for its forward pass.

Actual Behavior

The Slime patch loses the original-ID side channel. SGLang decodes the caller IDs and adopts the HF
processor's re-tokenized IDs, even though SGLANG_MM_AVOID_RETOKENIZE is enabled by default.

Environment

• slime commit: 4c193f1f37509cca70f0e88807a9305b70f63f4e
• SGLang version: v0.5.15.post1
• OS: Linux

Logs

Additional Context

Suggested Fix

Remove the Qwen-VL load_mm_data() to legacy_load_mm_data() override from the active Slime SGLang
patches. load_mm_data() already falls back to legacy_load_mm_data() when needed while preserving
the original IDs through input_ids=input_ids.

The multi-turn VLM rollout should also keep pre-expansion rollout IDs separately from the
processor-expanded training IDs and send the rollout IDs to SGLang. The current recipe writes
processor output directly into sample.tokens and later sends that same sequence as input_ids:

slime/examples/geo3k_vlm_multi_turn/rollout.py

Lines 154 to 179 in 4c193f1

	def _prepare_initial_inputs(sample: Sample, processor, tokenizer):
	if processor:
	processor_output = processor(text=sample.prompt, **(sample.multimodal_inputs or {}))
	prompt_ids = processor_output["input_ids"][0]
	sample.multimodal_train_inputs = {
	k: v for k, v in processor_output.items() if k not in ["input_ids", "attention_mask"]
	} or None
	else:
	prompt_ids = tokenizer.encode(sample.prompt, add_special_tokens=False)
	image_data = []
	if sample.multimodal_inputs and sample.multimodal_inputs.get("images"):
	image_data = [encode_image_for_rollout_engine(img) for img in sample.multimodal_inputs["images"]]
	return prompt_ids, image_data, sample.multimodal_train_inputs
	def _prepare_start_state(sample: Sample, state, args: Any, sampling_params: dict):
	prompt_ids, image_data, init_mm_train = _prepare_initial_inputs(sample, state.processor, state.tokenizer)
	current_image_data = image_data
	multimodal_train_inputs_buffer: list[dict | None] = []
	if init_mm_train:
	multimodal_train_inputs_buffer.append(init_mm_train)
	if not sample.tokens:
	sample.tokens = list(prompt_ids)
	response_tokens: list[int] = sample.tokens[len(prompt_ids) :] if len(sample.tokens) >= len(prompt_ids) else []

slime/examples/geo3k_vlm_multi_turn/rollout.py

Lines 192 to 200 in 4c193f1

	async def _run_inference_step(url: str, tokens: list[int], sampling_params: dict, image_data, tokenizer):
	payload = {
	"input_ids": tokens,
	"sampling_params": sampling_params,
	"return_logprob": True,
	}
	if image_data:
	payload["image_data"] = image_data

Pre-submission Checklist

• I have read the CONTRIBUTING.md and understand the collaboration scope.
• I have read the documentation and my issue is not addressed there.
• I have searched for existing issues and this is not a duplicate.
• I have provided a minimal, reproducible example.