Skip to content

docs(llama-cpp): clarify multimodal speculative decoding - #11700

Open
aim9sour wants to merge 2 commits into
mudler:masterfrom
aim9sour:docs/fix-mmproj-speculative-decoding
Open

docs(llama-cpp): clarify multimodal speculative decoding#11700
aim9sour wants to merge 2 commits into
mudler:masterfrom
aim9sour:docs/fix-mmproj-speculative-decoding

Conversation

@aim9sour

@aim9sour aim9sour commented Aug 24, 2026

Copy link
Copy Markdown

Description

Upstream llama.cpp removed the former general multimodal/speculative restriction in ggml-org/llama.cpp#19493, while ggml-org/llama.cpp#22673 later added MTP support and explicitly documented compatibility with vision input. This page still said LocalAI automatically disabled speculation whenever mmproj was active.

This PR:

  • documents that current llama.cpp backends can use mmproj with speculative decoding, including MTP;
  • keeps the backend-version and model-architecture compatibility caveats;
  • tells operators to verify projector/speculative initialization and the draft acceptance counts;
  • corrects the mmproj_use_gpu guidance so vision users do not remove the projector unnecessarily;
  • adds a combined vision + MTP model configuration example; and
  • removes the adjacent unsupported claim that n_draft can be overridden through each OpenAI request.

No issue was opened because this is a focused correction to one stale documentation section and does not require a code change.

Runtime evidence

Tested the official qwythos-9b-v2 Gallery configuration, which combines an embedded MTP head with mmproj, using a real base64 image request on an NVIDIA RTX PRO 2000 Blackwell (16,311 MiB):

  • LocalAI v4.8.2-gpu-nvidia-cuda-12 (sha256:90b8eb9505b4ea5f60e6f1057621a96100cde559fb4d92dd743ed5091b6b4b9f) with its default installed llama.cpp backend.
  • LocalAI master-gpu-nvidia-cuda-12, commit 38ba3fec636e684f81320e5d8ab497f5f1e985a0 (sha256:65487335cacd5395fd16c1bf78d3d23f90a41911bc7e98f663212352a4a84dd7).
  • The same master image with the explicitly installed cuda12-llama-cpp-development backend (master-gpu-nvidia-cuda-12-llama-cpp, amd64 digest sha256:805c89619771eb12ac0261f9c1a60c2105d38f816e603a287d242c3c9f15ec3c).

All three runs logged both:

  • creating MTP draft context against the target model
  • loaded multimodal model, '...mmproj-Qwythos-9B-v2-BF16.gguf'

The image request completed successfully, and both recorded inference runs reported:

draft acceptance = 0.97143 (102 accepted / 105 generated)

Verification

  • hugo v0.146.3+extended ... --minify — built 223 documentation pages successfully (matching the version pinned in Pages CI)
  • go test ./core/config --count=1
  • git diff --check upstream/master..HEAD
  • Independent read-only review: no remaining Critical, Important, or Minor findings

Notes for Reviewers

The current LocalAI gRPC bridge forwards MMProj and the speculative settings independently. Upstream #19493 removed the old general multimodal/speculative exclusion; #22673 later added MTP with explicit vision-input compatibility.

AI assistance is disclosed in the commit's Assisted-by trailer in accordance with the repository policy.

Signed commits

  • Yes, I signed my commits.
  • Documentation updated (docs/content/) for user-facing changes, or not applicable

Update the speculative decoding guidance now that modern llama.cpp backends can combine mmproj-based vision with speculative decoding, including MTP. Document compatibility checks, draft acceptance statistics, VRAM tradeoffs, and a combined configuration example.

Assisted-by: Codex:GPT-5.6-Sol [gh] [OpenStack] [Docker]
Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com>
@aim9sour

Copy link
Copy Markdown
Author

@localai-org-maint-bot please review this documentation correction and its runtime validation.

@localai-org-maint-bot

Copy link
Copy Markdown
Collaborator

Reviewed the documentation correction against the current LocalAI llama.cpp bridge and pin. The mmproj and speculative settings are forwarded independently, the pinned upstream revision contains the multimodal/speculative support change, and the mmproj_use_gpu guidance matches the option parser. The example and compatibility caveats look accurate, git diff --check passes, and DCO is green. Good to merge, @mudler.

Distinguish the upstream change that removed the general multimodal speculative restriction from the later change that added MTP with explicit vision compatibility.

Assisted-by: Codex:GPT-5.6-Sol [gh] [Docker]
Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com>
@aim9sour

Copy link
Copy Markdown
Author

Follow-up verification of the upstream attribution:

  • llama.cpp #19493 removed the former general multimodal/speculative disable and rejection paths.
  • llama.cpp #22673 later added MTP support and explicitly documented compatibility with vision input.

Updated the documentation and PR description in f9c2859 so each PR is cited only for its exact role. Fresh verification: Hugo 0.146.3 built all 223 pages, go test ./core/config passed, and git diff --check passed.

@aim9sour

Copy link
Copy Markdown
Author

@localai-org-maint-bot please re-review the follow-up attribution clarification in f9c2859. It distinguishes #19493 (removal of the general multimodal/speculative restriction) from #22673 (MTP support with explicit vision-input compatibility).

@aim9sour

Copy link
Copy Markdown
Author

@localai-org-maint-bot please submit an updated review for current head f9c2859. The follow-up precisely distinguishes #19493 (general multimodal/speculative restriction removal) from #22673 (MTP with explicit vision-input compatibility); no review threads or CI failures are open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants