| title | Multimodal I/O over A2A |
|---|---|
| description | Sending images and PDFs to a forge agent, and receiving media back, over the A2A protocol. |
| order | 9 |
Forge agents accept images and PDF documents as input and can return media as output — over the existing A2A message schema. No envelope change was required: A2A already modeled media as a file part; forge now honors those parts instead of dropping them (#255).
Backward compatible. Text-only messages are unchanged and serialize byte-for-byte as before. Media support is purely additive.
A message is a list of typed parts. The media type is file:
FileContent fields: name (optional), mimeType (required for media), bytes (the raw file standard-base64-encoded), or uri (by reference).
Inline
bytesonly, today. Forge feeds media to the model from inlinebytes. Afilepart with only auri(no bytes) is not fetched — it's rejected. URI-fetch (with egress control) is a tracked follow-up. Sendbytes.
{
"jsonrpc": "2.0", "id": 1, "method": "tasks/send",
"params": {
"id": "t-1",
"message": {
"role": "user",
"parts": [
{ "kind": "text", "text": "What's in this image?" },
{ "kind": "file", "file": { "name": "photo.png", "mimeType": "image/png", "bytes": "iVBORw0KGgo..." } }
]
}
}
}Same message, wrapped in the REST envelope:
{ "task": { "id": "t-1", "message": { "role": "user", "parts": [ /* …same parts… */ ] } } }A PDF is identical with "mimeType": "application/pdf".
Media the resolved model can't consume is rejected loudly (a 4xx / JSON-RPC error naming the part + reason) — never silently dropped and answered anyway.
| Media | MIME | Capable models |
|---|---|---|
| Image | image/png, image/jpeg, image/gif, image/webp |
vision models — OpenAI gpt-4o/gpt-4.1/gpt-5/o1/o3/o4, Anthropic Claude 3+, Gemini 1.5/2/3.x/4 |
application/pdf |
Anthropic Sonnet 3.5+, Opus 4+, Haiku 4.5+, Fable 5; Gemini 1.5/2.x/3.x/4; OpenAI gpt-4o/gpt-4.1/gpt-5/o1/o3/o4 on the openai-responses provider only (via the Responses input_file block) |
OpenAI PDF is Responses-only. PDF input for OpenAI works solely through
provider: openai-responses(the Responses API has a nativeinput_filedocument block). The plainprovider: openaichat-completions path has no document block, so a PDF there is still rejected (model_not_document_capable) — switch the agent toopenai-responses. Images work on both OpenAI providers.
| Bound | Limit | Reject reason |
|---|---|---|
| Per-image bytes | 5 MiB | image_limit_exceeded |
| Image dimensions | 100 000 px/side, 50 MP total | image_limit_exceeded |
| Images per message | 20 | too_many_image_parts |
| Per-PDF bytes | 32 MiB (+%PDF- sniff) |
document_limit_exceeded |
| PDFs per message | 5 | too_many_document_parts |
| Request body | 32 MiB (both transports) | HTTP 413 |
| Concurrent media requests | 4 | 429 / unavailable (shed) |
A rejection emits an input_media_rejected audit event. Note media bytes aren't text-scannable, so guardrail/intent scanning applies to the text/data parts only.
Also the existing A2A shape: the response message.parts can carry file parts alongside text.
{
"role": "agent",
"parts": [
{ "kind": "text", "text": "Here's the chart:" },
{ "kind": "file", "file": { "mimeType": "image/png", "bytes": "iVBORw0KGgo..." } }
]
}File parts come from two sources:
-
Tools that produce files (
file_create,browser_screenshot) — always on. -
Model-generated images — opt-in via
model.image_generation: trueon an image-capable model (see forge.yaml schema):provider: openai-responses— sends theimage_generationbuilt-in tool.provider: gemini(native client) — setsresponseModalities: ["TEXT","IMAGE"]; use an image model such asgemini-3-pro-image.
Both
forge init(the Image generation wizard step) and the web dashboard create form (the Image generation checkbox) offer this toggle, but only for providers/models that support it (runtime.ModelSupportsImageGen) — elsewhere the flag is a no-op and both surfaces hide it.Video generation (Veo) is not supported — it's a long-running async operation that doesn't fit the synchronous chat loop; a future tool will cover it.
Inbound uploads and model-generated output are written under the agent's files dir — .forge/files/inbound/ and .forge/files/generated/ respectively — content-addressed. Session history stores only the on-disk path (never base64), and media is reloaded per turn, so multi-turn conversations keep their media without bloating the session file. See Runtime Engine → Image and document input.
The chat UI has a 📎 attach button: select one or more images/PDFs (validated against the accepted types and size caps above), send them with your message, and see images rendered inline and documents as download links — in both your message and the agent's reply. Model-generated images come back the same way and render inline in the reply. (The dashboard relays the agent's SSE stream frame-by-frame; a generated image is a single large base64 frame, so the relay buffer is sized to carry a full media-bearing response — a frame too large to relay surfaces an error in the chat rather than ending silently.)
Each generated image also emits a generated_media audit event carrying its {mime, bytes, sha256}, where sha256 matches the .forge/files/generated/<sha256>.<ext> filename — so image output is visible in the audit trail, not just as llm_call output tokens.
Reopening a past session (or switching between sessions) also re-renders its media. History stores media as a URI reference only (the bytes live on disk, not in the session file), so the dashboard serves them on demand from GET /api/agents/{id}/files/{name} — a basename-only endpoint, restricted to the agent's .forge/files/{generated,inbound}/ store. It rejects path separators / traversal and re-confines resolved symlinks to the store, sends X-Content-Type-Options: nosniff, and serves only an allowlist of media types inline — anything else is forced to a download attachment so a direct hit can't render active content in the dashboard origin.
URI-fetch input (egress-gated) · OpenAI Responses input_file documents · Gemini/Bedrock media · text-extraction fallback for non-native document models · remote/distributed session media replay · streaming (partial-image) generated output.