Fine-tuned FunctionGemma-270M on iOS: no working GPU path (int8 `<pad>` / fp16 OOM), and CPU is ~10s/call

[Matt2012]

Plugin: flutter_gemma 0.16.4 · Device: iPhone 13 (A15), iOS 26 · Path: .litertlm FFI (LiteRT-LM)

What I'm doing: running a custom-fine-tuned FunctionGemma-270M as an on-device function-call router. Model bundled to .litertlm (int8 + fp16) with litert-lm-builder from a litert-torch .tflite. Loaded via ModelType.functionGemma, ToolChoice.required.

The problem in one line: I can't find a usable on-device configuration — the GPU backend won't run the model at all, and the CPU backend works correctly but is ~10s per call on an iPhone 13.

Results (4 experiments):

Build	Backend	Where	Result
int8 .litertlm	gpu	iPhone 13 (Metal)	Degenerate <pad> spam — wrong output, no crash
fp16 .litertlm	gpu	iPhone 13 (Metal)	OOM crash — EXC_RESOURCE high-watermark 2348 MB in dequantize::DequantizeImpl
int8 .litertlm	cpu	iOS Simulator	✅ Clean, correct routing — p50 ~960ms
int8 .litertlm	cpu	iPhone 13 (A15)	✅ Correct routing, but ~10s/call (first call ~5s, drifting up to ~10s — looks like thermal throttling)

The simulator-CPU run is the control: the container/tokenizer/metadata are correct, so the GPU failures are backend-specific and the on-device CPU latency is a hardware/runtime question, not a bad bundle.

On the ~10s/call: each call uses a fresh createChat(tools: …) and so re-prefills the full ~700-token tool prompt every time (the user query itself is only ~15 tokens), with maxTokens=2048. I suspect most of the wall-clock is re-prefilling that constant tool prefix on every call — which leads to question 4 below.

My read:

• int8 is litert-torch dynamic_int8 (CPU/XNNPACK). The Metal partitioner seems to lack kernels for some quantized ops (embedding lookup / gather), desyncing prefill/decode → <pad>. Possibly related to Custom INT4 .litertlm conversion report for Phi-4-mini and Qwen2.5-1.5B google-ai-edge/LiteRT-LM#1748.
• fp16 reaches the GPU compute path but weight conversion appears to materialize ~fp32 copies of the 543 MB weights → peak exceeds the iPhone-13 cap. Runner.entitlements has increased-memory-limit + extended-virtual-addressing (confirmed in the signed binary), but the crash still hit the un-entitled ~2348 MB ceiling.

Questions:

1. Recommended quant/format for iOS Metal GPU with a custom fine-tune? Is Q4_0 (QAT) the only Metal-safe small format, and is there any path to produce it for a custom checkpoint (vs. only the litert-community pre-quantized models)? Will weight-only int4 (wi4_afp32) avoid the op-coverage gap, or fall back to CPU?
2. fp16 OOM under a debugger: does com.apple.developer.kernel.increased-debugging-memory-limit need to be present and granted via the provisioning profile for the GPU path to get the raised cap? Any guidance in the plugin's iOS setup for this?
3. Sampler precedence on the iOS .litertlm FFI path: are temperature/topK passed to createChat() honored, or does the bundle's LlmMetadata.sampler_params take precedence? (We saw int8-CPU in-scope accuracy drop ~4 pts vs the .task baseline and suspect default TOP_P sampling overriding our greedy intent.)
4. CPU latency / prompt-prefix reuse: for a router whose tool prompt is constant across calls, is there a supported way to prefill the tool prompt once and reuse the KV cache (prefix caching), rather than re-prefilling ~700 tokens on every createChat()? Is ~10s/call on an A15 expected for a 270M model under this pattern, or are we missing a faster path (e.g. session reuse, smaller maxTokens)?

Happy to share the bundle or a minimal repro.

[DenisovAV]

Hi @Matt2012, thanks for the detailed report.

Quick question first — did you see my FunctionGemma article? → On-Device Function Calling with FunctionGemma

There are ready-to-use conversion notebooks in the repo, including one for .litertlm: colabs/functiongemma_to_litertlm.ipynb.

Your setup looks different from the pipeline there — maybe the pipeline from those notebooks will work for you. Could you check your bundle against them?

[Matt2012]

@DenisovAV I didn't see those. I'll take a quick look now.