Skip to content

Latest commit

 

History

History
120 lines (83 loc) · 7.14 KB

multimodal-io.md

File metadata and controls

120 lines (83 loc) · 7.14 KB
title Multimodal I/O over A2A
description Sending images and PDFs to a forge agent, and receiving media back, over the A2A protocol.
order 9

Multimodal I/O over A2A

Forge agents accept images and PDF documents as input and can return media as output — over the existing A2A message schema. No envelope change was required: A2A already modeled media as a file part; forge now honors those parts instead of dropping them (#255).

Backward compatible. Text-only messages are unchanged and serialize byte-for-byte as before. Media support is purely additive.

The A2A shape (unchanged)

A message is a list of typed parts. The media type is file:

// a2a.Part
{ "kind": "file", "file": { "name": "photo.png", "mimeType": "image/png", "bytes": "<base64>" } }

FileContent fields: name (optional), mimeType (required for media), bytes (the raw file standard-base64-encoded), or uri (by reference).

Inline bytes only, today. Forge feeds media to the model from inline bytes. A file part with only a uri (no bytes) is not fetched — it's rejected. URI-fetch (with egress control) is a tracked follow-up. Send bytes.

Input — send an image or PDF

JSON-RPC (POST /)

{
  "jsonrpc": "2.0", "id": 1, "method": "tasks/send",
  "params": {
    "id": "t-1",
    "message": {
      "role": "user",
      "parts": [
        { "kind": "text", "text": "What's in this image?" },
        { "kind": "file", "file": { "name": "photo.png", "mimeType": "image/png", "bytes": "iVBORw0KGgo..." } }
      ]
    }
  }
}

REST (POST /tasks/send)

Same message, wrapped in the REST envelope:

{ "task": { "id": "t-1", "message": { "role": "user", "parts": [ /* …same parts… */ ] } } }

A PDF is identical with "mimeType": "application/pdf".

What the model must support

Media the resolved model can't consume is rejected loudly (a 4xx / JSON-RPC error naming the part + reason) — never silently dropped and answered anyway.

Media MIME Capable models
Image image/png, image/jpeg, image/gif, image/webp vision models — OpenAI gpt-4o/gpt-4.1/gpt-5/o1/o3/o4, Anthropic Claude 3+, Gemini 1.5/2/3.x/4
PDF application/pdf Anthropic Sonnet 3.5+, Opus 4+, Haiku 4.5+, Fable 5; Gemini 1.5/2.x/3.x/4; OpenAI gpt-4o/gpt-4.1/gpt-5/o1/o3/o4 on the openai-responses provider only (via the Responses input_file block)

OpenAI PDF is Responses-only. PDF input for OpenAI works solely through provider: openai-responses (the Responses API has a native input_file document block). The plain provider: openai chat-completions path has no document block, so a PDF there is still rejected (model_not_document_capable) — switch the agent to openai-responses. Images work on both OpenAI providers.

Limits (enforced at ingest)

Bound Limit Reject reason
Per-image bytes 5 MiB image_limit_exceeded
Image dimensions 100 000 px/side, 50 MP total image_limit_exceeded
Images per message 20 too_many_image_parts
Per-PDF bytes 32 MiB (+%PDF- sniff) document_limit_exceeded
PDFs per message 5 too_many_document_parts
Request body 32 MiB (both transports) HTTP 413
Concurrent media requests 4 429 / unavailable (shed)

A rejection emits an input_media_rejected audit event. Note media bytes aren't text-scannable, so guardrail/intent scanning applies to the text/data parts only.

Output — receive media back

Also the existing A2A shape: the response message.parts can carry file parts alongside text.

{
  "role": "agent",
  "parts": [
    { "kind": "text", "text": "Here's the chart:" },
    { "kind": "file", "file": { "mimeType": "image/png", "bytes": "iVBORw0KGgo..." } }
  ]
}

File parts come from two sources:

  • Tools that produce files (file_create, browser_screenshot) — always on.

  • Model-generated images — opt-in via model.image_generation: true on an image-capable model (see forge.yaml schema):

    • provider: openai-responses — sends the image_generation built-in tool.
    • provider: gemini (native client) — sets responseModalities: ["TEXT","IMAGE"]; use an image model such as gemini-3-pro-image.

    Both forge init (the Image generation wizard step) and the web dashboard create form (the Image generation checkbox) offer this toggle, but only for providers/models that support it (runtime.ModelSupportsImageGen) — elsewhere the flag is a no-op and both surfaces hide it.

    Video generation (Veo) is not supported — it's a long-running async operation that doesn't fit the synchronous chat loop; a future tool will cover it.

Persistence

Inbound uploads and model-generated output are written under the agent's files dir — .forge/files/inbound/ and .forge/files/generated/ respectively — content-addressed. Session history stores only the on-disk path (never base64), and media is reloaded per turn, so multi-turn conversations keep their media without bloating the session file. See Runtime Engine → Image and document input.

In the web dashboard

The chat UI has a 📎 attach button: select one or more images/PDFs (validated against the accepted types and size caps above), send them with your message, and see images rendered inline and documents as download links — in both your message and the agent's reply. Model-generated images come back the same way and render inline in the reply. (The dashboard relays the agent's SSE stream frame-by-frame; a generated image is a single large base64 frame, so the relay buffer is sized to carry a full media-bearing response — a frame too large to relay surfaces an error in the chat rather than ending silently.)

Each generated image also emits a generated_media audit event carrying its {mime, bytes, sha256}, where sha256 matches the .forge/files/generated/<sha256>.<ext> filename — so image output is visible in the audit trail, not just as llm_call output tokens.

Reopening a past session (or switching between sessions) also re-renders its media. History stores media as a URI reference only (the bytes live on disk, not in the session file), so the dashboard serves them on demand from GET /api/agents/{id}/files/{name} — a basename-only endpoint, restricted to the agent's .forge/files/{generated,inbound}/ store. It rejects path separators / traversal and re-confines resolved symlinks to the store, sends X-Content-Type-Options: nosniff, and serves only an allowlist of media types inline — anything else is forced to a download attachment so a direct hit can't render active content in the dashboard origin.

Follow-ups (not yet supported)

URI-fetch input (egress-gated) · OpenAI Responses input_file documents · Gemini/Bedrock media · text-extraction fallback for non-native document models · remote/distributed session media replay · streaming (partial-image) generated output.