Investigate prompt-cache hit-rate regression (DeepSeek cost up): per-turn <turn_meta> block may be busting the cacheable prefix

[Hmbown]

Summary

A user reports CodeWhale is costing them more than before — strongly suggesting a prompt-cache hit-rate regression. With DeepSeek context caching, a stable prompt prefix should yield high cache-hit rates (cached input tokens are ~10x cheaper). If recent changes perturb the cacheable prefix, cost rises even with identical usage.

Why this likely regressed

DeepSeek (and most providers) cache on the longest common prefix of the request. Anything dynamic inserted early in the message order invalidates the cache for everything after it.

Prime suspects (recent per-turn additions):

1. <turn_meta> block (fix(tui): surface mode policy in turn metadata #3623, commit 1494481). Active mode + runtime mode policy is now injected into every user turn. If this block (or anything near it) varies per turn — timestamps, changing mode label, counters, dynamic policy text — it shifts the prefix and busts the cache. Check exact placement and whether any field is non-constant turn-to-turn.
2. System-prompt / tool-definition ordering. If tool schemas, model catalog data, or skills/MCP descriptors are injected in a non-deterministic order or with dynamic content, the cacheable system prefix breaks.
3. Mode policy text reordering from the recent mode-sync work (fix(tui): keep mode policy in sync with engine #3722) — verify the per-turn policy snippet is stable when the mode hasn't changed.

Investigation steps (distributed prompt)

• Use the existing /cache debug surface (per-turn cache-telemetry ring, see crates/tui/src/tui/app.rs ~1019) to measure cached vs. uncached input tokens across turns on DeepSeek.
• Diff the actual request bodies of two consecutive turns (same mode, no file changes) and confirm the prefix is byte-identical up to the new user message. Any diff before the last user message = cache breaker.
• Bisect: compare cache-hit rate on a build before fix(tui): surface mode policy in turn metadata #3623 vs. after, then before/after fix(tui): keep mode policy in sync with engine #3722.
• Confirm cache_hit_tokens / prompt_cache_hit_tokens are being read from the DeepSeek usage payload and surfaced in cost math (verify the cost calc applies the cached-token discount).

Acceptance

• Two consecutive same-mode turns show a high cached-prefix ratio on DeepSeek.
• Cost-per-turn with caching returns to pre-regression levels.
• Add a regression/telemetry assertion that the cacheable prefix is stable across turns when mode/context are unchanged.

Refs #3623, #3722.

[Hmbown]

Update — static trace done; obvious suspects RULED OUT

Traced the cache-critical path. The first hypotheses are wrong — the standard path is cache-correct:

• turn_meta is frozen, not re-wrapped. messages_with_turn_metadata() just clones stored session messages (crates/tui/src/core/engine/turn_loop.rs:2635); each user message keeps the turn_meta it was built with. Test messages_with_turn_metadata_preserves_stored_messages_for_prefix_cache guards this. So historical prefix is byte-stable. ✅ not the buster.
• Static system prefix is deliberately kept stable — pwd/workspace were moved OUT of the system block INTO per-turn turn_meta precisely to protect the cache prefix (crates/tui/src/core/engine.rs:1927-1930, prompts.rs:2024). ✅
• Cost accounting reads cache tokens — prompt_cache_hit_tokens/prompt_cache_miss_tokens parsed at crates/tui/src/client.rs:1909-1944. ✅
• Anthropic-style cache_control breakpoints exist — prompt_zones.rs:663, compaction.rs:1095, tools/registry.rs:256. ✅

Remaining candidates (need a LIVE 2-turn DeepSeek measurement to confirm)

1. Per-model cache fragmentation via auto-routing. turn_meta logs Auto model route. If auto_model re-routes across turns, each model has a separate server-side cache → systematic misses. Check whether routing changed defaults recently. (verifiable: log routed model per turn.)
2. Anthropic-compatible DeepSeek endpoint breakpoint placement. The provider list includes "DeepSeek (Anthropic-compatible)". Verify the cache_control breakpoint sits after the stable prefix and that the per-turn turn_meta (date/mode/working-set) is positioned AFTER the last breakpoint, not before it. If a breakpoint moved relative to dynamic content, cache from that point is invalidated.
3. Compaction churn. compaction.rs rewrites history; if compaction triggers more often now, it rewrites the prefix and resets the cache. Check compaction frequency vs. prior versions.
4. Cost math may double-count. Confirm cached input tokens are billed at the cached (discounted) rate, not full input rate, in the displayed $ (the prompt_cache_hit_tokens value must feed a discounted line in the cost calc, not just telemetry).

How to confirm (use the /cache surface)

• Run two consecutive same-mode, no-file-change turns on DeepSeek. Capture prompt_cache_hit_tokens vs prompt_cache_miss_tokens for turn 2. A healthy turn 2 should be mostly HITS.
• If turn 2 is mostly MISSES → dump the two request bodies and diff the prefix up to the last user message; the first differing byte is the buster (suspects 1–3).
• If turn 2 is mostly HITS but $ is still high → it's the cost math (suspect 4).

Next action: add per-turn logging of routed model + hit/miss token split so the live measurement is one command, then fix whichever suspect it points to.