Checkpoint: consolidated runtime + native image work + Gemma-4 KV-cache wiring lock#72
Draft
jjang-ai wants to merge 10 commits into
Draft
Checkpoint: consolidated runtime + native image work + Gemma-4 KV-cache wiring lock#72jjang-ai wants to merge 10 commits into
jjang-ai wants to merge 10 commits into
Conversation
added 10 commits
May 25, 2026 12:58
…a-4 KV-cache wiring lock
Consolidates accumulated, individually-proven work into one reviewable
checkpoint for main:
- Runtime: MiMo v2 hybrid cache topology/ABI, BatchEngine scheduler updates,
distributed TP/JACCL sharding plans, model config-init guards across the
LLM model family, slider-governed KV cap, memory-safety plan cache.
- Native image generation (vMLXFlux{,Kit,Models,Video} targets): z-image-turbo,
flux-schnell, qwen-image native pipelines + local model store, wired into
Package.swift. Spec/handoff docs under docs/.
- Release guard scripts (scripts/vmlx-*-guard-check.sh) and focused tests.
Gemma-4 tool-call KV-cache wiring (osaurus prefill→0 after tool call):
- osaurus root cause already fixed upstream in osaurus-ai/osaurus#1525
(removed finalizingPostToolChoice; tool_choice/tools held byte-stable per
iteration; transient notices ride as tool-role feedback).
- vmlx side: lock the Gemma4WithTools.jinja prefix-stability contract with two
pure-template regression tests in Gemma4ChatTemplateProbeTests, and document
the full chain in docs/GEMMA4_TOOLCALL_KV_CACHE_WIRING.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Large consolidation checkpoint of accumulated, individually-proven vmlx-swift work, plus the vmlx-side lock for the Gemma-4 tool-call KV-cache wiring. Opened as a draft — this is a save-point for review, not for immediate merge. CI is expected to be the gate; un-draft when the team is ready to land.
What's in here
Runtime
scripts/vmlx-*-guard-check.shNative image generation (
vMLXFlux{,Kit,Models,Video}targets, wired intoPackage.swift)docs/(MFLUX_HANDOFF.md,OSAURUS_VMLX_FLUX_INTEGRATION_SPEC.md, port plans)Gemma-4 tool-call KV-cache wiring (the "prefill resets to 0 after a tool call" report)
osaurus-ai/osaurus#1525:finalizingPostToolChoicedeleted (no more gemma-4/qat/mxfp4tool_choice→noneflip),tool_choice/toolsheld byte-stable per iteration, transient[System Notice]lines now ride as tool-role feedback.Gemma4WithTools.jinja's system turn (system content +<|tool>declaration:block) must stay byte-identical across a tool-call → tool-result iteration so the KV prefix is reused. Two pure-template regression tests inTests/MLXLMTests/Gemma4ChatTemplateProbeTests.swift(no weights, CI-safe) assert both prefix-stability and the busting failure mode.docs/GEMMA4_TOOLCALL_KV_CACHE_WIRING.md.Verification
swift build(full package, 1876 steps) green on this branch.testGemma4WithToolsSystemPrefixStableAcrossToolResultTurntestGemma4WithToolsDroppingToolsBustsSystemPrefixNotes
goalnew.mdand local scratch (.agents/,docs/local/,.logs/) are intentionally excluded;.gitignoreupdated to cover.logs/,codex/logs/,.sentry.local.env,.claude/*.local.md.