feat(genai): genai_primitives parallel input/output path (#181) by DenisovAV · Pull Request #363 · DenisovAV/flutter_gemma

DenisovAV · 2026-07-09T12:14:50Z

Adds a parallel genai_primitives path per the agreed #181 course, running alongside the existing Message API (untouched).

New public API

import 'package:flutter_gemma/genai.dart'; — an extension on InferenceChat:

sendMessage(ChatMessage) / sendMessageStream(ChatMessage)
generateContent(List<ChatMessage>) / generateContentStream(List<ChatMessage>)

All return a role:model ChatMessage whose parts carry text + tool calls + thinking together. Both stage each message via the existing addQueryChunk (buffers without generating) then generate once.

Design

Zero engine changes. litertlm / mediapipe / android / ios / hook untouched. The only edit to an existing file is a 5-line Mutex field in chat.dart; the old Message / addQueryChunk path is fully intact — both paths work side by side.
Input converter ChatMessage → List<Message> (1→N; tool results become sibling messages), output converter ModelResponse → ChatMessage, a conditional-import LinkPart reader (data/file/http, wasm-safe), a capability guard (no silent image/audio/tool drop), and a per-chat mutex with terminal-path release on the stream variants.
generateContent is stateful (stages into this chat's history, then generates once) — isolation = a fresh model.createChat(). This is on-device reality, not firebase's stateless model.
genai_primitives + http become direct core deps, re-exported from a side barrel (lib/genai.dart, not the main barrel) so a 0.x bump only touches the opt-in surface.

Verification

22 unit tests (converters, capability guard, every throw path)
10 integration tests on a real model (macOS, gemma-4-E2B) — all green: every method × text / vision / audio / thinking / tool round-trip / multi-turn
flutter build web --wasm ✓, APK release ✓, flutter analyze clean

Closes #181.

snapsl

LGTM

snapsl · 2026-07-09T18:40:38Z

+        textBuffer.write(text);
+      case DataPart(:final bytes, :final mimeType):
+        _routeMedia(bytes, mimeType, images, () => audio, (v) => audio = v);
+      case LinkPart(:final url, :final mimeType):


link is not required to contain media. We could also support other types in the future.

snapsl · 2026-07-09T19:24:39Z

+  /// Wrap the generate stream, mapping each event to a ChatMessage and holding
+  /// the chat mutex until a terminal event (done/error/cancel).
+  Stream<ChatMessage> _lockedStream(Future<void> Function() stage) {
+    late StreamController<ChatMessage> controller;


snapsl · 2026-07-09T19:33:40Z

+  Uint8List bytes,
+  String? mimeType,
+  List<Uint8List> images,
+  Uint8List? Function() getAudio,


unnecessary getAudio and setAudio

snapsl · 2026-07-09T19:38:11Z

  hooks: ^2.0.0
  code_assets: ^1.2.0
  crypto: ^3.0.6
+  genai_primitives: ^0.2.3


pin version?

snapsl · 2026-07-09T19:46:32Z

+    final msg = ChatMessage(
+      role: ChatMessageRole.user,
+      parts: [
+        DataPart(Uint8List.fromList([1, 2]), mimeType: 'application/pdf'),


could be supported in future :)

snapsl · 2026-07-09T20:46:04Z

+    messages.insert(
+      0,
+      Message(
+        text: textBuffer.toString(),


ChatMessage.text contains the text of all Textparts. No need to parse it.

snapsl · 2026-07-09T21:18:39Z

+    );
+  }
+  final parts = message.parts;
+  if (parts.isEmpty) {


Create an empty Message?

snapsl · 2026-07-09T21:29:01Z

+///
+/// One ChatMessage may yield >1 Message (tool results become sibling messages).
+/// Async because LinkPart resolution may need I/O. Throws — never silently drops.
+Future<List<Message>> messagesFromChatMessage(


Looks good. Strictly speaking, this could change the order of the parts compared to the order of the messages, since the final message is inserted at first, followed by the tool messages.

snapsl · 2026-07-09T21:30:47Z

+import 'genai_output_converter.dart';
+
+/// Guard: a model-role ChatMessage is output, not a sendMessage input.
+void rejectModelRole(ChatMessage message) {


reject model + system so there is no checking required later on?

snapsl · 2026-07-09T21:41:02Z

+/// A side barrel (NOT re-exported from `flutter_gemma.dart`) so genai_primitives
+/// 0.x churn stays contained. Import `package:flutter_gemma/genai.dart` to use
+/// [ChatMessage] with [InferenceChat.sendMessage] / [generateContent].
+library;


Do the converters need to be exported?

… re-input, simplify converters (#181)

…import (#181)

…181)

…ceChat (#181)

…#181)

…181)

…ls/multi-turn (#181)

#181) - _lockedStream: a subscriber cancelling during the onListen await (before sub is assigned) triggered onCancel while onListen was suspended, releasing the mutex early; onListen then resumed and started a second concurrent turn with an un-cancellable subscription. Guard with a cancelled flag checked after stage() — bail before attaching the subscription. Verified with an isolated StreamController race repro. - integration test: generateContent multi-turn now closes chat.session (was the one of 10 tests that leaked).

… re-input, simplify converters (#181)

…ging nothing (#181)

onCancel released genaiLock even while onListen was still queued on acquire() or inside stage(), freeing another turn's critical section (package:mutex is ownership-blind) — letting a second turn interleave into the shared session, or permanently wedging the chat. Lock is now owned by onListen: release() guards on lockHeld, and onCancel only tears down once generation is running. Also route a throw in the chunk-mapping into cleanup (was escaping to the Zone and leaking the lock), and log stopGeneration teardown failures.

- keep preamble text alongside a tool call in the batch fold (was silently dropped, diverging from the streaming path) - link_reader: reject an empty 2xx body (blank media) and bound the fetch with a timeout so a hung server can't wedge the chat mutex - throw on a tool result carried by a model-role message (symmetric with the tool-call-on-user guard) - mark InferenceChat.genaiLock @internal — external acquire/release deadlocks the chat - document that callId mirrors toolName (flutter_gemma's tool protocol is name-keyed)

…ives # Conflicts: # CLAUDE.md # packages/flutter_gemma/CHANGELOG.md # packages/flutter_gemma/DESKTOP_SUPPORT.md # website/content/docs/migration.md

- Guard onCancel teardown on !released: Dart fires onCancel after a normal or errored 'done' too, and the generating-only guard issued a spurious stopGeneration() on the shared session that could cancel the next back-to-back turn. - Scope LinkPart to bytes-only: the inference layer no longer fetches URLs or reads files (SSRF / arbitrary local-file read); LinkPart throws with guidance to resolve links caller-side into a DataPart. Removes link_reader and the http dependency; link handling belongs in flutter_gemma_agent. - Regression tests: no stopGeneration after a successful stream, back-to-back turn isolation, LinkPart throws. - migration.md core pin -> ^1.3.1.

DenisovAV · 2026-07-17T16:06:38Z

Thanks for the review, @snapsl — heads-up that the PR has moved a lot since (34 commits), and your comments drove a good chunk of it. Status:

Addressed: ChatMessage.text used directly (no re-parsing); getAudio/setAudio folded into _routeMedia; doc added on messagesFromChatMessages; converters no longer exported (genai.dart only show GenAiChat); model and system roles rejected up front.

LinkPart — removed entirely (your "not required to contain media" + File.fromUri notes): you were right a link isn't necessarily media, and since an on-device model needs the actual bytes and we can't safely resolve an arbitrary URL/file (SSRF / local-file read), I dropped link fetching. LinkPart now throws with guidance to resolve it caller-side into a DataPart; URL/web handling lives in flutter_gemma_agent behind a permission gate. link_reader + the http dep are gone — so your two link_reader comments are now moot.

Chain-of-thoughts with tools: yes — chatMessageFromParts coalesces ThinkingPart → TextPart → ToolPart.call… into one model turn, preamble preserved.

Kept deliberately: empty-parts ChatMessage throws (caller misuse); part-order documented; genai_primitives: ^0.2.3 intentional caret.

Also fixed a concurrency bug a later pass missed (a post-completion onCancel could stop the next back-to-back turn). Since your LGTM predates all of this, a fresh look is very welcome if you have a minute — no worries if not.

DenisovAV mentioned this pull request Jul 9, 2026

Consider using genai_primitives. #181

Open

snapsl reviewed Jul 9, 2026

View reviewed changes

DenisovAV added a commit that referenced this pull request Jul 12, 2026

fix(genai): address #363 review — accept 2xx links, strip thoughts on…

db4326d

… re-input, simplify converters (#181)

DenisovAV added 15 commits July 12, 2026 16:47

feat(genai): add genai_primitives + http deps and side barrel (#181)

bad4576

feat(genai): conditional-import LinkPart reader (data/file/http) (#181)

0e53cbb

feat(genai): ChatMessage → List<Message> input converter (#181)

0436481

test(genai): cover tool-call-in-user + non-media-mime throw paths (#181)

a2b5f07

feat(genai): ModelResponse → ChatMessage output converter (#181)

7efd84f

refactor(genai): drop unused finishStatus + private genai_primitives …

556a61c

…import (#181)

feat(genai): capability guard prevents silent image/audio/tool drop (#…

cc48310

…181)

feat(genai): sendMessage/sendMessageStream/generateContent on Inferen…

eed4923

…ceChat (#181)

feat(genai): export sendMessage/generateContent API from genai barrel (…

4c150c2

…#181)

test(genai): E2E sendMessage/sendMessageStream round-trip (#181)

78222ef

refactor(genai): drop dead import + cancel/stop on stream error path (#…

46335a6

…181)

test(genai): E2E coverage for all methods + vision/audio/thinking/too…

0520d91

…ls/multi-turn (#181)

fix(genai): address #363 review — accept 2xx links, strip thoughts on…

c651111

… re-input, simplify converters (#181)

fix(genai): throw on a user-role ThinkingPart instead of silently sta…

c68dfb8

…ging nothing (#181)

DenisovAV force-pushed the feat/181-genai-primitives branch from a788575 to c68dfb8 Compare July 12, 2026 14:48

DenisovAV added 6 commits July 12, 2026 17:29

docs(website): add genai_primitives adoption guide + sidebar link (#181)

27b30e7

chore(release): flutter_gemma 1.3.0 — genai_primitives support (#181)

67bec57

Merge remote-tracking branch 'origin/main' into feat/181-genai-primit…

2212900

…ives # Conflicts: # CLAUDE.md # packages/flutter_gemma/CHANGELOG.md # packages/flutter_gemma/DESKTOP_SUPPORT.md # website/content/docs/migration.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(genai): genai_primitives parallel input/output path (#181)#363

feat(genai): genai_primitives parallel input/output path (#181)#363
DenisovAV wants to merge 21 commits into
mainfrom
feat/181-genai-primitives

DenisovAV commented Jul 9, 2026

Uh oh!

snapsl left a comment

Uh oh!

snapsl Jul 9, 2026

Uh oh!

snapsl Jul 9, 2026

Uh oh!

snapsl Jul 9, 2026

Uh oh!

snapsl Jul 9, 2026

Uh oh!

snapsl Jul 9, 2026

Uh oh!

snapsl Jul 9, 2026

Uh oh!

snapsl Jul 9, 2026

Uh oh!

snapsl Jul 9, 2026

Uh oh!

snapsl Jul 9, 2026

Uh oh!

snapsl Jul 9, 2026

Uh oh!

DenisovAV commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

DenisovAV commented Jul 9, 2026

New public API

Design

Verification

Uh oh!

snapsl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DenisovAV commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants