fix(server): skip the prompt cache for VLM-loaded models (b769 text chat failures)

[solderzzc]

Problem

On b769, text chat with an auto-detected VLM goes through the prompt cache and breaks:

• Qwen3.5 / 3.6 (e.g. mlx-community/Qwen3.5-0.8B-4bit): every text request fails with HTTP 500 unsupportedBatchContinuation(model: "Qwen35"). This covers single-turn, multi-turn and streaming requests. It comes from the hybrid prompt cache added in feat: prompt cache for hybrid recurrent/attention models (Qwen3.5/3.6) #189.
• Qwen3-VL (e.g. mlx-community/Qwen3-VL-2B-Instruct-4bit): the second request that shares a prefix fails with missingState(model: "Qwen3VL", key: "qwen35vl.ropeDeltas"). It comes from the generic cache hit path plus the fail-closed continuation check that came in with the mlx-swift-lm bump in b769.
• Gemma 3/4, Mistral3, Idefics3/SmolVLM, Pixtral and others (pre-existing): a generic cache hit silently re-feeds the whole prompt on top of the restored KV.

Image requests are not affected, because they already skip the prompt cache.

Root cause

Most VLM processors return [1, T] tokens, even for text-only input. The prompt cache paths slice on axis 0 and use MLXArray.count (dim(0), so 1):

• The hybrid split tokens[boundary...] yields [0, T].
• A generic hit sets startIndex = 0, trims 1 token and feeds all T tokens again.

Qwen3.5 and Qwen3-VL also need the LMOutput.State (ropeDeltas) of the cached prefix, and the prompt cache does not store it.

Fix

Skip the prompt cache, both restore and save, when context.model is any VLMModel. The check is on the model type rather than config.isVision, so --audio (OmniModelFactory) loads are covered too. LLM-factory models are unchanged, including text-only hybrid models on the #189 path.

A startup note is printed for VLM/Omni loads.

Cost

Text chat on VLM loads re-prefills the full prompt every turn. For Qwen3.5 this is the pre-#189 behaviour. Because of auto-detection, this includes default launches of mlx-community Qwen3.5/3.6 and Gemma 4. Qwen2-VL, Qwen2.5-VL and LFM2-VL, which use 1-D tokens, also lose cache reuse that did work before.

Testing

• swift build -c release, SwiftLMTests (194, 0 failures) and SwiftBuddyTests (127, 9 existing skips, 0 failures).

• End to end at temp 0 on port 5440/5441:

  Model	Loaded as	Before (b769)	After
  Qwen3.5-0.8B-4bit	VLM	all text requests: HTTP 500	single-turn, multi-turn ×2, streaming and text-after-image all answer correctly; image request answers "A dog."
  Qwen3-VL-2B-Instruct-4bit	VLM	2nd and 3rd shared-prefix requests: missingState	"Paris", "Berlin", "Berlin"
  gemma-4-e2b-it-4bit	VLM	answers, with context duplicated on cache hits	"Paris", "Berlin", "Berlin"; no cache hits
  text-only view of Qwen3.5-0.8B (vision_config removed)	LLM		Prompt cache HIT (hybrid) still logged on the repeated request (feat: prompt cache for hybrid recurrent/attention models (Qwen3.5/3.6) #189 path intact)

• New Test 7 in tests/test-speculative.sh: loads mlx-community/Qwen3.5-2B-4bit without a draft. It asserts:

  • the model is served as a VLM;
  • single-turn and multi-turn text chat both complete;
  • there are no prompt cache hits.

  Every other Qwen3.5 launch in CI passes a draft model, which turns off VLM auto-detection. That is why CI missed this.

  Local run: passes with the fix; the two chat checks fail on b769.

Follow-ups (not in this PR)

• Restore prompt reuse for VLMs:
  • slice on the last axis;
  • store LMOutput.State with the cache entry and pass it to generate(state:);
  • then drop this gate.
• Make PromptCacheTests test the production skip decision instead of a copied formula.
• Add a CI check for a non-hybrid VLM, such as Qwen3-VL or SmolVLM, with shared-prefix text chat.

AI usage: Claude Code (Claude Opus 5.5) found the bug, wrote the fix and Test 7, ran the verification and wrote this description.

🤖 Generated with Claude Code