macOS 26.5.1 (M4): Gemma 4 E2B audio inference fails — CPU: "Failed to invoke compiled model", GPU: heap corruption crash

[noxymon]

Environment

• flutter_gemma: 1.1.1
• flutter_gemma_litertlm: 1.0.2
• Platform: macOS 26.5.1 (25F80) — Apple Silicon arm64
• Hardware: Mac16,12 (MacBook Air M4, 16 GB)
• Metal: Metal 4, AGXMetalG16G_B0 v351.2
• Model: gemma-4-E2B-it.litertlm (~2.5 GB from litert-community HuggingFace)
• Flutter: 3.41.6 (stable)

Description

Gemma 4 E2B inference fails on macOS 26.5.1 with two different failure modes depending on backend:

CPU backend (PreferredBackend.cpu)

Model loads successfully. Inference fails on first chunk with:

SEVERE: ERROR during chunk 0 inference: Exception: Stream error: INTERNAL: ERROR: [runtime/executor/llm_litert_compiled_model_executor.cc:734] Failed to invoke the compiled model

This is 100% reproducible — every chunk, every session.

GPU backend (PreferredBackend.gpu)

Model loads successfully. Within ~30s of starting inference, entire process crashes:

Exception Type: EXC_BREAKPOINT (SIGTRAP)
BUG IN CLIENT OF LIBMALLOC: memory corruption of free block

Crashed thread (DartWorker) in dart::malloc() → _xzm_xzone_malloc_from_freelist_chunk.cold.1 — victim of prior heap corruption from LiteRtMetalAccelerator.framework. Multiple LiteRT threads active (execution_threa, callback_thread, worker pool) at crash time.

Full crash report attached below.

Context from existing issues

• #268: Gemma 4 E2B vision+text works on macOS 26.2 / M4 Mini. This issue is specifically about audio inference (supportAudio: true) on macOS 26.5.1.
• #218: Similar crash pattern on iOS 26.2 — libLiteRtMetalAccelerator mispackaged, fixed in 0.14.0 FFI rewrite.
• LiteRT#6745: Upstream root cause of Continual crashes with Gemma 4 E2B litertlm on iOS (works on Android emulator) #218.

Steps to Reproduce

// Initialize with audio support
await FlutterGemma.initialize(inferenceEngines: const [LiteRtLmEngine()]);
await FlutterGemma.installModel(
  modelType: ModelType.gemma4,
  fileType: ModelFileType.litertlm,
).fromFile(modelPath).install();

final model = await FlutterGemma.getActiveModel(
  maxTokens: 1024,
  supportAudio: true,    // ← triggers failure
  supportImage: true,
  preferredBackend: PreferredBackend.cpu,
);

final chat = await model.createChat(
  supportAudio: true,
  supportImage: true,
  modelType: ModelType.gemma4,
);

// Send audio message — fails here
final audioBytes = await File('audio_chunk.wav').readAsBytes();
await chat.addQuery(Message.withAudio(
  text: 'Transcribe this audio to English.',
  audioBytes: audioBytes,
));

await for (final token in chat.generateChatResponseAsync()) {
  // Never reaches here — stream errors with:
  // Exception: Stream error: INTERNAL: ERROR:
  // [runtime/executor/llm_litert_compiled_model_executor.cc:734]
  // Failed to invoke the compiled model
}

Error Log (CPU backend)

2026-06-25 15:54:53.063659: [GemmaPipelineIsolate] SEVERE: ERROR during chunk 0 inference: Exception: Stream error: INTERNAL: ERROR: [runtime/executor/llm_litert_compiled_model_executor.cc:734] Failed to invoke the compiled model
2026-06-25 15:54:53.063993: [GemmaPipeline] SEVERE: chunk 0 error: Exception: Stream error: INTERNAL: ERROR: [runtime/executor/llm_litert_compiled_model_executor.cc:734]

Model initializes without errors (model path resolved, LiteRT engine created). Failure occurs only when generateChatResponseAsync / addQuery is called with audio data.

Crash Report (GPU backend)

Process:             gemplay [77688]
Hardware Model:      Mac16,12
OS Version:          macOS 26.5.1 (25F80)

Exception Type:    EXC_BREAKPOINT (SIGTRAP)
Exception Codes:   0x0000000000000001, 0x000000018eee9bac

Application Specific Information:
BUG IN CLIENT OF LIBMALLOC: memory corruption of free block
Abort Cause 38188534784

Triggered by Thread: 33  DartWorker

Thread 33 Crashed:: DartWorker
0   libsystem_malloc.dylib  _xzm_xzone_malloc_from_freelist_chunk.cold.1 + 36
1   libsystem_malloc.dylib  _xzm_xzone_malloc_from_freelist_chunk + 1004
2   libsystem_malloc.dylib  _xzm_xzone_find_and_malloc_from_freelist_chunk + 336
3   libsystem_malloc.dylib  _xzm_xzone_malloc_freelist_outlined + 324
4   FlutterMacOS            dart::malloc(unsigned long) + 32
5   FlutterMacOS            dart::VMHandles::AllocateHandle(dart::Zone*) + 68
...
10  FlutterMacOS            dart::CallSiteInliner::TryInliningImpl(...)
...
21  FlutterMacOS            dart::BackgroundCompiler::Run() + 252

Binary Images (relevant):
  LiteRtLm.framework               (native-v0.13.1 equivalent, arm64)
  LiteRtMetalAccelerator.framework (arm64, 7.5 MB)
  GemmaModelConstraintProvider.framework (arm64, 5.5 MB)
  AGXMetalG16G_B0                 (Apple M4 GPU driver, v351.2)

Active LiteRT threads at crash:
  Thread 18-26:  LiteRtLm thread_main (worker pool, 9 threads)
  Thread 41:     execution_threa in absl::Mutex::AwaitCommon
  Thread 42:     callback_thread in absl::Mutex::AwaitCommon

Additional Notes

• supportAudio: true is the differentiating factor from [Bug] FastVLM-0.5B outputs raw <start_of_*> tokens on macOS (M4) — text and vision both broken #268 where Gemma 4 E2B worked on macOS 26.2 M4 with vision+text only
• Background isolate used for inference (via Isolate.spawn) — same as gemplay's production path
• Same model file works in flutter_gemma example app for text chat (no audio)
• FFmpeg audio extraction verified correct: 16kHz mono WAV chunks, ~896 KB per 28s chunk

Related

• #268 — Gemma 4 E2B works on macOS 26.2 M4 (vision+text, no audio)
• #218 — iOS 26.2 Gemma 4 E2B SIGSEGV (fixed in 0.14.0)
• LiteRT#6745 — Upstream Metal accelerator mispackaging
• #219 — macOS 26.4.1 SIGSEGV with Qwen 2.5

[DenisovAV]

Thanks @noxymon — and I think I found the most likely culprit while trying to reproduce.

First, the repro: I ran your exact pattern on macOS 26.5 (M4) — getActiveModel(maxTokens: 1024, supportAudio: true) → createChat → addQuery(Message.withAudio) → generateChatResponseAsync, Gemma 4 E2B. CPU @ 1024, CPU @ 4096, and GPU @ 4096 all passed and transcribed correctly, no executor.cc:734, no heap corruption. So maxTokens and the chat-audio path itself aren't the trigger.

But here's the thing — your toolchain can't actually resolve flutter_gemma: 1.1.1. You're on Flutter 3.41.6, which bundles Dart 3.11.4, and flutter_gemma 1.1.x requires Dart ≥3.12 / Flutter ≥3.44 (raised at the 1.0 split). I verified: flutter pub get on 3.41.6 fails with "version solving failed... Try using the Flutter SDK version: 3.44.4."

So either your pubspec.lock is actually pinning an older flutter_gemma (whose audio path differs from 1.1.1), or there's an override in play. Could you share:

flutter --version
flutter pub deps --style=compact | grep flutter_gemma     # the ACTUAL resolved version

and then try upgrading to Flutter ≥3.44 + a clean flutter clean && flutter pub get? My strong suspicion is the audio failure is tied to running on an unsupported toolchain / an older resolved flutter_gemma, not 1.1.1 itself. Once you're on a supported Flutter and we know the real resolved version, if it still fails I'll dig into the macOS 26.5.1 Metal angle.

[DenisovAV]

Update — I re-verified this on the current release and can't reproduce either failure.

On flutter_gemma_litertlm 1.2.0 (LiteRT-LM v0.14.0), macOS 26.5.1 (25F80) — the same build you're on — on Apple M4 Pro, Gemma 4 E2B, getActiveModel(maxTokens: 4096, supportAudio: true) → audio chat → generateChatResponseAsync:

• CPU backend: engine creates in ~3.6s, audio transcribes correctly, no executor.cc:734.
• GPU backend: engine creates in ~4.9s, audio transcribes correctly, no heap corruption.

Both are the exact failure modes from your report, and both pass on current code, on the same macOS build you reported.

The most likely explanation is still the toolchain. You reported flutter_gemma 1.1.1 on Flutter 3.41.6, but 3.41.6 bundles Dart 3.11.4, and flutter_gemma 1.1.x needs Dart ≥3.12 / Flutter ≥3.44 — pub get can't actually resolve 1.1.1 there, so your build was running an older resolved flutter_gemma whose audio path predates several fixes.

Could you:

• upgrade to Flutter ≥3.44 with flutter_gemma: ^1.3.2 + flutter_gemma_litertlm: ^1.2.0,
• flutter clean && flutter pub get,
• share flutter pub deps --style=compact | grep flutter_gemma (the actual resolved versions),

and retest? I'll keep this open until you confirm. If audio still fails on a supported toolchain with the real resolved version, I'll dig into the macOS 26.5.1 Metal angle specifically.