feat(agent): image input in an agent turn + MCP 2025-11-25 - #407
Open
DenisovAV wants to merge 3 commits into
Open
feat(agent): image input in an agent turn + MCP 2025-11-25#407DenisovAV wants to merge 3 commits into
DenisovAV wants to merge 3 commits into
Conversation
AgentLoop.run / AgentSession.ask take optional imageBytes and build the turn with Message.withImage, so a multimodal chat (supportImage: true) can dispatch skills based on what the model saw. McpClient now advertises protocol revision 2025-11-25, pinned by a test.
- AgentLoop.run throws when imageBytes is passed to a chat without image support, or when the bytes are empty. The engines disagreed on this case: FFI, mobile MediaPipe and built-in AI dropped the image and answered text-only, web MediaPipe threw. A silent drop is the worse half — the model answers confidently about a photo it never saw. - McpClient verifies the revision the server echoes in initialize and throws on a downgrade, instead of proceeding blind. - McpClient sends MCP-Protocol-Version on every request after initialize; without it a stateless server must assume 2025-03-26, so advertising a newer revision in the handshake alone changed nothing on the wire. - _notify surfaces a rejected notifications/initialized instead of swallowing it (and its HTTP status). - clientVersion follows the package to 0.2.0; example lock and the CLAUDE.md version line updated. - Tests: guard cases, negotiated-revision mismatch, and the header on post-initialize requests. The fixture now echoes the client's revision rather than silently downgrading it.
agent.md had no coverage of ask(imageBytes:) — including the supportImage: true precondition and the ArgumentError it now throws.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
AgentLoop.run(chat, userMessage, {Uint8List? imageBytes})andAgentSession.ask(userMessage, {imageBytes})now accept an optional image. When present, the turn is built withMessage.withImageinstead of a plain textMessage, so the model sees the photo alongside the text and can dispatch skills based on what it saw.McpClientadvertises MCP protocol revision2025-11-25(was2025-06-18), pinned by a test that asserts what goes out in theinitializerequest.Requirements
The image only reaches the model when the chat is multimodal — build the session with
supportImage: true(seeAgentSession.fromModel) and back it with a vision-capable model. Without that the image is ignored downstream; this is documented on both new parameters.Notes
2025-06-18. That is deliberate: the client advertises a revision and the server may negotiate down, so the test asserts the outgoing value, not the reply.flutter_gemma_agentbumped 0.1.0 → 0.2.0.Verification
flutter testinpackages/flutter_gemma_agent: 198 passed.flutter analyze: no issues.