`InferenceChat.close()` doesn't release GPU (Graphics) + native memory — RSS ratchets up per inference → `lowmemorykiller` OOM on sustained use (Android, GPU backend)

[basictech01]

Summary

When repeatedly opening a per-turn chat (model.openChat(...) → generateChatResponseAsync() → chat.close()) against a single loaded model, device memory is not released after close(). Two things accumulate:

1. Graphics (GPU delegate) jumps to ~1.77 GB on the first inference and is never released while the model stays loaded.
2. Native Heap ratchets up ~150–300 MB per inference and is not reclaimed by close().

The resting footprint climbs ~0.7 GB → ~2.9 GB → ~3.7 GB over the first few inferences, each inference peaks at ~5 GB PSS (with ~1 GB pushed to swap), and a 6 GB device then thrashes and the OS lowmemorykiller SIGKILLs the app after a handful of inferences. This happens with a 0.5B model, so it is accumulation, not model size.

Environment

flutter_gemma	1.1.0
flutter_gemma_litertlm	1.0.2
flutter_gemma_mediapipe	1.0.2
flutter_gemma_embeddings	1.0.1
Flutter	3.44.2 (stable)
Dart	3.12.2
Device	Google Pixel 4a (sunfish), Android 13, 6 GB RAM, Snapdragon 730G
Model	Qwen 2.5 0.5B, .litertlm (LiteRT-LM engine)
Backend	PreferredBackend.gpu
Engines registered	LiteRtLmEngine(), MediaPipeEngine(); embeddings via LiteRtEmbeddingBackend()

Initialized once at startup:

await FlutterGemma.initialize(
  inferenceEngines: const [LiteRtLmEngine(), MediaPipeEngine()],
  embeddingBackends: const [LiteRtEmbeddingBackend()],
);

Our usage pattern

We open a throwaway chat per inference turn and close it in finally (the model itself stays loaded across turns):

final model = await FlutterGemma.getActiveModel(
  maxTokens: 4096,                 // KV-cache size
  preferredBackend: PreferredBackend.gpu,
);
InferenceChat? chat;
try {
  chat = await model.openChat(
    temperature: 0.2, topK: 20, tokenBuffer: 256,
    modelType: /* model's type */, isThinking: false,
  );
  await chat.addQueryChunk(Message.text(text: prompt));
  await for (final r in chat.generateChatResponseAsync()) {
    if (r is TextResponse) { /* collect r.token */ }
  }
} finally {
  await chat?.close();             // <-- memory is NOT reclaimed here
}

Expected behavior

After chat.close() completes, the per-inference GPU/native buffers (KV cache, activations, delegate scratch) should be released, so repeated open→generate→close cycles return to roughly the steady "model loaded, idle" footprint. Memory should not ratchet upward across inferences.

Actual behavior

adb shell dumpsys meminfo <pkg> sampled every 4 s while issuing one chat at a time (waiting for completion + idle between each), Qwen 2.5 0.5B, GPU backend:

Phase	TOTAL PSS	Native Heap	Graphics
Model loaded, idle (pre-inference)	0.70 GB	0.12 GB	0.06 GB
Rests here after inference #1	2.88 GB	0.67 GB	1.77 GB
Rests here after inference #2	3.67 GB	0.82 GB	1.77 GB
Rests here after #3–#4	3.68 GB	1.13 GB	1.77 GB
Peak during an inference	~5.0 GB	1.77 GB	2.20 GB (+~1 GB swap)
→ lowmemorykiller SIGKILL

Key points:

• Graphics goes 0.06 GB → 1.77 GB on the first inference and stays there at rest (spiking to ~2.2 GB during active generation, then settling back to ~1.77 GB — never back to ~0.06 GB).
• Native Heap at rest ratchets 0.12 → 0.67 → 0.82 → 1.13 GB across inferences — it does not return to baseline after close().

Raw dumpsys meminfo App Summary excerpts

Model loaded, idle (before first inference):

 App Summary
                       Pss(KB)                        Rss(KB)
         Native Heap:   127200                         128208
            Graphics:    59944                          59944
           TOTAL PSS:   705053            TOTAL RSS:   794324       TOTAL SWAP PSS:      155

At rest after the first inference completed and the chat was closed (memory did not return):

 App Summary
                       Pss(KB)                        Rss(KB)
         Native Heap:   672436                         673444
            Graphics:  1767024                        1767024
           TOTAL PSS:  2875268            TOTAL RSS:  2955368       TOTAL SWAP PSS:      153

Peak during a later inference (just before the kill):

 App Summary
                       Pss(KB)                        Rss(KB)
         Native Heap:  1602020                        1602252
            Graphics:  2220388                        2220388
           TOTAL PSS:  5014426            TOTAL RSS:  3951516       TOTAL SWAP PSS:  1105546

OOM kill (logcat)

lowmemorykiller: Kill 'in.uniun.app' (14456), uid 10249, oom_score_adj 0 to free
  4167864kB rss, 1514684kB swap; reason: device is low on swap (0kB < 209712kB)
  and thrashing (310%)
Zygote: Process 14456 exited due to signal 9 (Killed)

Steps to reproduce

1. FlutterGemma.initialize([LiteRtLmEngine(), MediaPipeEngine()]).
2. Download/activate a .litertlm model (we used Qwen 2.5 0.5B).
3. getActiveModel(maxTokens: 4096, preferredBackend: PreferredBackend.gpu).
4. In a loop (~6×): openChat(...) → addQueryChunk → drain generateChatResponseAsync() → await chat.close(). Wait for completion + ~8 s idle between iterations.
5. Between iterations run adb shell dumpsys meminfo <pkg> and watch the App Summary → Graphics and Native Heap climb and not return; on a 6 GB device the app is OOM-killed within a handful of iterations.

Note: observed inside our app's normal usage; we have not yet reduced it to a standalone minimal sample. The loop above mirrors exactly what our code does. Happy to build a minimal repro app if useful.

Questions

1. Is InferenceChat.close() expected to release the GPU-delegate (Graphics) and native buffers, or does it only tear down Dart-side state?
2. Is there a recommended way to fully release per-inference memory between turns without unloading/reloading the whole model (which costs several seconds)?
3. Is per-turn openChat()/close() the intended pattern, or should a single long-lived InferenceChat be reused? (We re-open per turn because our RAG context changes every turn and we don't want stale context in history.)
4. Does the CPU backend avoid the ~1.77 GB resident Graphics allocation, and does it also avoid the native-heap ratchet?
5. Does maxTokens (KV-cache size, 4096 here) scale this — i.e., would a smaller cache reduce the per-inference and resident cost?

Impact

On a 6 GB Android phone this makes sustained on-device inference unreliable — the app OOM-crashes after a few inferences even with a 0.5B model — which limits multi-turn / background-agent use cases.

[DenisovAV]

Thanks for the unusually thorough report — the dumpsys meminfo breakdown made this much faster to pin down. I reproduced it and traced the layer. Findings:

1. The resident ~1.77 GB "Graphics" is expected, not a leak. That's the GPU/OpenCL context owned by the engine (the loaded model), allocated on first inference. It's released only when the model is closed (model.close()), not when a chat/session closes. InferenceChat.close() only closes the session — by design, so the loaded model can be reused across chats. So that number staying resident while the model is loaded is correct behavior.

2. The Native Heap ratchet is a real leak — but it's in the upstream OpenCL GPU delegate, not in flutter_gemma's teardown. I verified by layer:

layer / backend	result
openChat→close, no generation	flat
C loop vs the LiteRT-LM C API, CPU (macOS)	flat
C loop, Metal GPU (macOS)	flat (RSS even falls)
your loop, Android OpenCL GPU — at both chat and session level (bypassing our chat layer entirely)	+~110 MB/turn, identical

So the per-turn growth is OpenCL-delegate-specific GPU scratch that isn't reclaimed on conversation delete — our Dart/FFI teardown is balanced (one conversation create / one delete per turn, verified). Filed upstream: google-ai-edge/LiteRT-LM#2699.

Answers to your questions:

1. InferenceChat.close() releases the per-session state, not the engine's GPU context (that's model.close()). The native-heap growth you see is the upstream OpenCL leak above, which close() doesn't currently reclaim.
2. To fully release per-inference memory between turns today: periodically model.close() + re-getActiveModel() to force an engine teardown — that's the only call that returns the accumulated OpenCL buffers. Costs a model reload, but it's the workaround until the upstream fix lands.
3. Per-turn openChat/close is a fine pattern — re-opening for fresh RAG context is correct, and not what causes the leak.
4. CPU avoids both the resident ~1.77 GB Graphics and the native-heap ratchet (verified — CPU is flat). Metal (iOS/macOS) also doesn't leak. This is OpenCL-specific, so if you can tolerate CPU latency on Android it sidesteps the issue entirely.
5. Yes — a smaller maxTokens shrinks the KV cache, so both the per-turn and resident cost scale down with it. It slows the ratchet but doesn't stop it.

I'll update here when the upstream fix lands. Thanks again for the detailed repro.