fix(litertlm): pin the vision encoder to the CPU backend on iOS - #413
Open
yjinjo wants to merge 1 commit into
Open
fix(litertlm): pin the vision encoder to the CPU backend on iOS#413yjinjo wants to merge 1 commit into
yjinjo wants to merge 1 commit into
Conversation
Passing `PreferredBackend.gpu` also routes the vision encoder to Metal,
where it cannot be prepared:
ERROR: Node number 41 (STABLEHLO_COMPOSITE) failed to prepare.
Failed to create conversation: INTERNAL:
[runtime/executor/vision_litert_compiled_model_executor.cc:275]
Reproduced on an iPhone 17 Pro Max (iOS 26.6) with
gemma-4-E2B-it.litertlm, supportImage: true, maxTokens: 2048.
In debug builds the failure is silent and looks like a hang; only a
profile build surfaces the error.
The .litertlm file already ships `section_backend_constraint: cpu` for
vision_adapter and audio_encoder, so mixed backends are the intended
configuration -- the vision encoder is simply missing that constraint.
The audio backend here is already hardcoded to CPU for the same reason;
this applies the same treatment to vision.
Measured on the same device (profile build, image + Korean prompt),
time to first token:
all CPU 3799 ms cold / 3727 ms warm
GPU LLM + CPU vision 6013 ms first ever (Metal shader
compile), 3487 ms cold after restart,
1499-1636 ms warm
A follow-up could expose `visionBackend` as a parameter instead of
hardcoding it, once the Metal delegate gains STABLEHLO_COMPOSITE
support.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Passing
PreferredBackend.gpualso routes the vision encoder to Metal, where it cannot be prepared:Reproduced on an iPhone 17 Pro Max (iOS 26.6) with gemma-4-E2B-it.litertlm, supportImage: true, maxTokens: 2048. In debug builds the failure is silent and looks like a hang; only a profile build surfaces the error.
The .litertlm file already ships
section_backend_constraint: cpufor vision_adapter and audio_encoder, so mixed backends are the intended configuration -- the vision encoder is simply missing that constraint. The audio backend here is already hardcoded to CPU for the same reason; this applies the same treatment to vision.Measured on the same device (profile build, image + Korean prompt), time to first token:
A follow-up could expose
visionBackendas a parameter instead of hardcoding it, once the Metal delegate gains STABLEHLO_COMPOSITE support.