Skip to content

feat(dart-agent): call renderer functions from the agent - #3941

Open
nan-yu wants to merge 13 commits into
feat/dart-agent-function-rpcfrom
feat/dart-renderer-function-calls
Open

nan-yu wants to merge 13 commits into
feat/dart-agent-function-rpcfrom
feat/dart-renderer-function-calls

Conversation

@nan-yu

@nan-yu nan-yu commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #3861. Review the top nine commits: feat(dart-agent): call renderer functions from the agent (#3535), followed by the review-fix and conformance commits covering RendererFunctionCalls.resolve declining unknown responses, catalog target checks, the shared validation order, isPending, and routing conformance responses through RendererToAgentMessageProcessor. Draft until #3861 lands and the open questions below are settled.
Part of #3535.

Description

#3861 first shipped the agent's calls to renderer functions (callRendererFunction) together with agent functions, as the Python stack did. The review on #3793 questioned whether in-memory pending calls fit stateless HTTP servers, where the rendererFunctionResponse can arrive in a later request on another replica. To keep agent functions moving, renderer function calls were split out of #3861 into this PR in every SDK (Python #3939, TypeScript #3940). The code is the code that was on #3861, renamed to fit RendererToAgentMessageProcessor, with the review fixes below.

  1. RendererFunctionCalls in a2ui_agent, one per conversation. call({required catalogId, required name, args, surfaceId, timeout, functionCallId}) checks the call and returns a record of the CallRendererFunctionMessage to send and a Future of the result. resolve completes the matching future with the renderer's value, or with an A2uiRpcError carrying the renderer's code. It returns true when it settled or failed a pending call, and false otherwise (an unknown, late or duplicate response).

  2. A call times out after defaultRendererCallTimeout (30 seconds, as on the renderer side) unless given another timeout; defaultTimeout: null disables the default. Unknown, duplicate and late responses are ignored. maxPending caps the calls pending at once (TOO_MANY_PENDING), and dispose fails the pending calls with CANCELLED and refuses later calls with DISPOSED.

  3. a2ui_core gets RpcErrorCode.tooManyPending (TOO_MANY_PENDING), and package:a2ui_agent/a2ui_agent.dart also re-exports CallRendererFunctionMessage. fake_async is a new dev dependency for the timer tests.

  4. How it plugs into the processor: pass calls.resolve to RendererToAgentMessageProcessor as onRendererFunctionResponse, so the processor validates the response and hands it over. Since callbacks may return false to decline (from the second review of feat(python-agent): answer agent functions through RendererToAgentMessageProcessor #3793), a response resolve doesn't own, such as one to a call the model wrote, comes back handled: false and can go on to the model. The processor's API from feat(dart): agent functions and RendererToAgentMessageProcessor #3861 doesn't change:

    final calls = RendererFunctionCalls();
    final processor = RendererToAgentMessageProcessor(
      catalogs: catalogs,
      onRendererFunctionResponse: calls.resolve,
    );
  5. The README gets a "Calling renderer functions" section, and both CHANGELOGs list the API.

Review fixes from #3793, applied here as in Python:

  1. An empty function name is rejected with A2uiRpcError (invalidFunctionCall), like an empty catalogId.
  2. An empty surfaceId counts as omitted: the message leaves it out.
  3. A malformed rendererFunctionResponse that names a pending call fails that call right away with unknownError, instead of leaving it to time out. One that names no pending call is ignored.
  4. The test for settling a call checks that its timer is cancelled (FakeAsync.pendingTimers is empty afterwards), the counterpart of Python's timer.cancelled() assertion.

Open design questions, kept for discussion on #3939:

  1. maxPending and the TOO_MANY_PENDING code, which the protocol doesn't define. The agent's own code starts these calls, so a limit may not be needed.
  2. Stateless servers. Pending calls live in memory, so this suits an agent that keeps a connection to the renderer open, such as a WebSocket. A stateless multi-replica server would handle rendererFunctionResponse in its own onRendererFunctionResponse callback instead.

Catalog checks (after review on #3793, #3793 (comment)): RendererFunctionCalls takes an optional catalogs argument. With it, call() checks the target before it builds a message, registers a pending call or starts a timer. An unknown catalogId, or a name that isn't one of that catalog's renderer functions (agent functions included), gets UNKNOWN_FUNCTION. A rendererOnly function (or one that leaves out allowedCallers) gets INVALID_FUNCTION_CALL, and so does one with requiresUserActivation: true, since a call from the agent has no user gesture. Two catalogs with the same ID are a catalog error. Without catalogs, nothing changes. The conformance runner supports the new catalogs cases. Checked locally against renderer_function_calls.yaml from #3939: 31/31 (30 cases and the non-empty check). call() runs its checks in the same order in all three SDKs:

  1. an invalid timeout
  2. disposed
  3. an empty given functionCallId
  4. a duplicate id
  5. max_pending
  6. an empty name, then an empty catalogId
  7. a name that isn't an identifier
  8. the args checks: a JSON object, identifier keys, and JSON values (no NaN or infinity)
  9. these catalog checks

Only the instance default timeout can be "no limit". isPending(functionCallId) inspects whether a call is awaiting a response without settling it, matching Python and TypeScript.

Changes after the latest review: Review: _parseResponse reads responses as strict v1.0 (version "1.0", exactly one of value and error, no extra keys), so a malformed response fails a pending call with UNKNOWN_ERROR as in Python and TypeScript; tests updated (a void response sends value: null); the conformance runner enforces the closed RendererCallError and RendererCallsBuildError shapes.

Tests

  1. renderer_function_calls_test.dart: constructor checks, the message and its value, void and error results, call validation, duplicate IDs, maxPending, timeouts and dispose, the timer cancelled on settle, an empty name, an empty surfaceId, malformed responses with and without a pending call, resolve plugged into RendererToAgentMessageProcessor, what resolve returns, and a response it declines coming back unhandled from the processor.
  2. agent_function_validation_end_to_end_test.dart round-trips both callAgentFunction and callRendererFunction between a core MessageProcessor and the agent. agent_functions_test.dart in a2ui_core checks the TOO_MANY_PENDING code.
  3. renderer_function_calls_conformance_test.dart runs conformance/agent/renderer_function_calls.yaml. That suite lands with feat(python-agent): call renderer functions from the agent #3939, so on this branch the runner registers one skipped test saying the suite isn't in the checkout. With the suite from feat(python-agent): call renderer functions from the agent #3939 copied into a copy of this branch, all 30 cases and the non-empty check pass (31 tests), delivering responses through RendererToAgentMessageProcessor and checking calls.isPending.
  4. Commands:
    1. cd dart/a2ui_core && dart analyze --fatal-infos && dart test: clean, 2265 passed (20 skips, none in the agent function tests).
    2. cd dart/a2ui_agent && dart analyze --fatal-infos && dart test: clean, 1691 passed. The renderer calls conformance test registers one skipped placeholder until renderer_function_calls.yaml lands with feat(python-agent): call renderer functions from the agent #3939.
    3. dart format --output=none --set-exit-if-changed . is clean in both packages.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces RendererFunctionCalls to allow agents to call renderer functions and wait for their results, managing timeouts, concurrent call caps, and disposal. It also adds the TOO_MANY_PENDING RPC error code to a2ui_core and includes comprehensive unit, conformance, and end-to-end tests. The review feedback suggests retaining explicit type annotations on loop variables when iterating over non-literal collections in renderer_function_calls.dart and renderer_function_calls_conformance_test.dart to comply with the project's style guidelines.

@nan-yu
nan-yu added this pull request to stack #3972 November 4, 2026 05:17
@nan-yu
nan-yu marked this pull request as ready for review November 4, 2026 05:22
@nan-yu
nan-yu force-pushed the feat/dart-renderer-function-calls branch from af13dcc to 1d69e36 Compare November 4, 2026 18:21
@nan-yu
nan-yu force-pushed the feat/dart-renderer-function-calls branch from 1d69e36 to 935fa91 Compare November 4, 2026 19:22
nan-yu added a commit that referenced this pull request Nov 4, 2026
Audit the agent functions feature blueprint against the Python, TS and Dart
branches and fix the drift:
- The version check runs first for every message kind, so a
  callAgentFunction with a missing or other version raises even with a
  functionCallId; a schema-failing call with a readable id still gets an
  INVALID_FUNCTION_CALL reply in all three SDKs.
- The timeout option is agent_function_timeout (seconds) in Python,
  agentFunctionTimeoutMs in TypeScript and Duration agentFunctionTimeout in
  Dart; the sketch now uses a neutral name.
- The callback signatures per SDK, including Dart's FutureOr<void>
  callbacks that still decline with false.
- Only Python's a2ui.rpc re-exports the message classes; a validation error
  raised by a handler is INVALID_FUNCTION_CALL in Dart only.

Replace the follow-up note with section 5, the agent calling renderer
functions as implemented by #3939, #3940 and #3941: RendererFunctionCalls,
its checks and error codes, timeouts, resolve returning whether it owned
the response, malformed responses, disposal, the optional catalog checks,
the in-memory limitation on stateless servers, and the ADK integration.
Requirements, links, conformance (renderer_function_calls.yaml, 23 cases)
and implementation steps follow. The Python agent codebase blueprint says
renderer calls come in the follow-up #3939.

Signed-off-by: Nan Yu <nanyu@google.com>
@nan-yu
nan-yu force-pushed the feat/dart-renderer-function-calls branch from 935fa91 to f451496 Compare November 4, 2026 20:23
nan-yu added a commit that referenced this pull request Nov 4, 2026
Audit the agent functions feature blueprint against the Python, TS and Dart
branches and fix the drift:
- The version check runs first for every message kind, so a
  callAgentFunction with a missing or other version raises even with a
  functionCallId; a schema-failing call with a readable id still gets an
  INVALID_FUNCTION_CALL reply in all three SDKs.
- The timeout option is agent_function_timeout (seconds) in Python,
  agentFunctionTimeoutMs in TypeScript and Duration agentFunctionTimeout in
  Dart; the sketch now uses a neutral name.
- The callback signatures per SDK, including Dart's FutureOr<void>
  callbacks that still decline with false.
- Only Python's a2ui.rpc re-exports the message classes; a validation error
  raised by a handler is INVALID_FUNCTION_CALL in Dart only.

Replace the follow-up note with section 5, the agent calling renderer
functions as implemented by #3939, #3940 and #3941: RendererFunctionCalls,
its checks and error codes, timeouts, resolve returning whether it owned
the response, malformed responses, disposal, the optional catalog checks,
the in-memory limitation on stateless servers, and the ADK integration.
Requirements, links, conformance (renderer_function_calls.yaml, 23 cases)
and implementation steps follow. The Python agent codebase blueprint says
renderer calls come in the follow-up #3939.

Signed-off-by: Nan Yu <nanyu@google.com>
nan-yu added a commit that referenced this pull request Nov 4, 2026
Audit the agent functions feature blueprint against the Python, TS and Dart
branches and fix the drift:
- The version check runs first for every message kind, so a
  callAgentFunction with a missing or other version raises even with a
  functionCallId; a schema-failing call with a readable id still gets an
  INVALID_FUNCTION_CALL reply in all three SDKs.
- The timeout option is agent_function_timeout (seconds) in Python,
  agentFunctionTimeoutMs in TypeScript and Duration agentFunctionTimeout in
  Dart; the sketch now uses a neutral name.
- The callback signatures per SDK, including Dart's FutureOr<void>
  callbacks that still decline with false.
- Only Python's a2ui.rpc re-exports the message classes; a validation error
  raised by a handler is INVALID_FUNCTION_CALL in Dart only.

Replace the follow-up note with section 5, the agent calling renderer
functions as implemented by #3939, #3940 and #3941: RendererFunctionCalls,
its checks and error codes, timeouts, resolve returning whether it owned
the response, malformed responses, disposal, the optional catalog checks,
the in-memory limitation on stateless servers, and the ADK integration.
Requirements, links, conformance (renderer_function_calls.yaml, 23 cases)
and implementation steps follow. The Python agent codebase blueprint says
renderer calls come in the follow-up #3939.

Signed-off-by: Nan Yu <nanyu@google.com>
@nan-yu
nan-yu force-pushed the feat/dart-renderer-function-calls branch from f451496 to 47fdc16 Compare November 4, 2026 22:04
nan-yu added a commit that referenced this pull request Nov 4, 2026
Audit the agent functions feature blueprint against the Python, TS and Dart
branches and fix the drift:
- The version check runs first for every message kind, so a
  callAgentFunction with a missing or other version raises even with a
  functionCallId; a schema-failing call with a readable id still gets an
  INVALID_FUNCTION_CALL reply in all three SDKs.
- The timeout option is agent_function_timeout (seconds) in Python,
  agentFunctionTimeoutMs in TypeScript and Duration agentFunctionTimeout in
  Dart; the sketch now uses a neutral name.
- The callback signatures per SDK, including Dart's FutureOr<void>
  callbacks that still decline with false.
- Only Python's a2ui.rpc re-exports the message classes; a validation error
  raised by a handler is INVALID_FUNCTION_CALL in Dart only.

Replace the follow-up note with section 5, the agent calling renderer
functions as implemented by #3939, #3940 and #3941: RendererFunctionCalls,
its checks and error codes, timeouts, resolve returning whether it owned
the response, malformed responses, disposal, the optional catalog checks,
the in-memory limitation on stateless servers, and the ADK integration.
Requirements, links, conformance (renderer_function_calls.yaml, 23 cases)
and implementation steps follow. The Python agent codebase blueprint says
renderer calls come in the follow-up #3939.

Signed-off-by: Nan Yu <nanyu@google.com>
@nan-yu
nan-yu force-pushed the feat/dart-renderer-function-calls branch from 47fdc16 to 8d78baf Compare November 4, 2026 22:56
nan-yu added a commit that referenced this pull request Nov 4, 2026
Audit the agent functions feature blueprint against the Python, TS and Dart
branches and fix the drift:
- The version check runs first for every message kind, so a
  callAgentFunction with a missing or other version raises even with a
  functionCallId; a schema-failing call with a readable id still gets an
  INVALID_FUNCTION_CALL reply in all three SDKs.
- The timeout option is agent_function_timeout (seconds) in Python,
  agentFunctionTimeoutMs in TypeScript and Duration agentFunctionTimeout in
  Dart; the sketch now uses a neutral name.
- The callback signatures per SDK, including Dart's FutureOr<void>
  callbacks that still decline with false.
- Only Python's a2ui.rpc re-exports the message classes; a validation error
  raised by a handler is INVALID_FUNCTION_CALL in Dart only.

Replace the follow-up note with section 5, the agent calling renderer
functions as implemented by #3939, #3940 and #3941: RendererFunctionCalls,
its checks and error codes, timeouts, resolve returning whether it owned
the response, malformed responses, disposal, the optional catalog checks,
the in-memory limitation on stateless servers, and the ADK integration.
Requirements, links, conformance (renderer_function_calls.yaml, 23 cases)
and implementation steps follow. The Python agent codebase blueprint says
renderer calls come in the follow-up #3939.

Signed-off-by: Nan Yu <nanyu@google.com>
@nan-yu
nan-yu force-pushed the feat/dart-renderer-function-calls branch from 8d78baf to c6e6ba9 Compare November 4, 2026 23:18
nan-yu added a commit that referenced this pull request Nov 5, 2026
Audit the agent functions feature blueprint against the Python, TS and Dart
branches and fix the drift:
- The version check runs first for every message kind, so a
  callAgentFunction with a missing or other version raises even with a
  functionCallId; a schema-failing call with a readable id still gets an
  INVALID_FUNCTION_CALL reply in all three SDKs.
- The timeout option is agent_function_timeout (seconds) in Python,
  agentFunctionTimeoutMs in TypeScript and Duration agentFunctionTimeout in
  Dart; the sketch now uses a neutral name.
- The callback signatures per SDK, including Dart's FutureOr<void>
  callbacks that still decline with false.
- Only Python's a2ui.rpc re-exports the message classes; a validation error
  raised by a handler is INVALID_FUNCTION_CALL in Dart only.

Replace the follow-up note with section 5, the agent calling renderer
functions as implemented by #3939, #3940 and #3941: RendererFunctionCalls,
its checks and error codes, timeouts, resolve returning whether it owned
the response, malformed responses, disposal, the optional catalog checks,
the in-memory limitation on stateless servers, and the ADK integration.
Requirements, links, conformance (renderer_function_calls.yaml, 23 cases)
and implementation steps follow. The Python agent codebase blueprint says
renderer calls come in the follow-up #3939.

Signed-off-by: Nan Yu <nanyu@google.com>
@nan-yu
nan-yu force-pushed the feat/dart-renderer-function-calls branch from c6e6ba9 to 7a30bd6 Compare November 5, 2026 00:21
nan-yu added a commit that referenced this pull request Nov 5, 2026
Audit the agent functions feature blueprint against the Python, TS and Dart
branches and fix the drift:
- The version check runs first for every message kind, so a
  callAgentFunction with a missing or other version raises even with a
  functionCallId; a schema-failing call with a readable id still gets an
  INVALID_FUNCTION_CALL reply in all three SDKs.
- The timeout option is agent_function_timeout (seconds) in Python,
  agentFunctionTimeoutMs in TypeScript and Duration agentFunctionTimeout in
  Dart; the sketch now uses a neutral name.
- The callback signatures per SDK, including Dart's FutureOr<void>
  callbacks that still decline with false.
- Only Python's a2ui.rpc re-exports the message classes; a validation error
  raised by a handler is INVALID_FUNCTION_CALL in Dart only.

Replace the follow-up note with section 5, the agent calling renderer
functions as implemented by #3939, #3940 and #3941: RendererFunctionCalls,
its checks and error codes, timeouts, resolve returning whether it owned
the response, malformed responses, disposal, the optional catalog checks,
the in-memory limitation on stateless servers, and the ADK integration.
Requirements, links, conformance (renderer_function_calls.yaml, 23 cases)
and implementation steps follow. The Python agent codebase blueprint says
renderer calls come in the follow-up #3939.

Signed-off-by: Nan Yu <nanyu@google.com>
@nan-yu
nan-yu force-pushed the feat/dart-renderer-function-calls branch from 7a30bd6 to 833a091 Compare November 5, 2026 00:40

@gspencergoog gspencergoog left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request adds RendererFunctionCalls in dart/a2ui_agent and RpcErrorCode.tooManyPending in dart/a2ui_core to build outbound v1.0 callRendererFunction messages, validate call inputs and renderer catalog targets synchronously in the shared cross-SDK order, correlate incoming rendererFunctionResponse envelopes, and manage timeouts and disposal. Static analysis, formatting, unit and end-to-end tests, and all 31 conformance cases from conformance/agent/renderer_function_calls.yaml pass, with one schema-alignment fix needed in _parseResponse (and its paired unit test) when resolve is called directly without RendererToAgentMessageProcessor.

🤖 Agent-generated

nan-yu added 13 commits November 5, 2026 17:58
…n call

RendererFunctionCalls.resolve returns true when it settled or failed a
pending call and false otherwise, so as onRendererFunctionResponse it
declines responses it doesn't own and they reach the model.

Signed-off-by: Nan Yu <nanyu@google.com>
RendererFunctionCalls takes the renderer's catalogs; call then refuses a
function the catalogs don't declare (UNKNOWN_FUNCTION), and one that is
rendererOnly or requires a user activation (INVALID_FUNCTION_CALL),
before anything is sent. Two catalogs with one id are a catalog error.

Signed-off-by: Nan Yu <nanyu@google.com>
…ormance runner

Build args.catalogs in code as v1.0 catalogs, and treat a top-level
expect_error as building the helper failing.

Signed-off-by: Nan Yu <nanyu@google.com>
An agent function in the agent's catalog can't run on the renderer, so a
call to it gets UNKNOWN_FUNCTION.

Signed-off-by: Nan Yu <nanyu@google.com>
call() runs its checks in the order the other SDKs use. args must have
identifier keys and JSON values; the argument-count limit and the nested
function-call checks are gone, leaving nested @call and @path objects for
the renderer.

Signed-off-by: Nan Yu <nanyu@google.com>
Pass functionCallId from a call step, and meet the case with args that
aren't an object by showing Dart's typed call can't take them.

Signed-off-by: Nan Yu <nanyu@google.com>
Tells whether a call with a functionCallId is awaiting a response, as
is_pending and isPending do in Python and TypeScript.

Signed-off-by: Nan Yu <nanyu@google.com>
…rocessor

Signed-off-by: Nan Yu <nanyu@google.com>
RendererFunctionCalls.resolve now holds a response to the v1.0
FunctionResponse schema, as RendererToAgentMessageProcessor and the
Python SDK do: version '1.0' on raw maps and typed messages, exactly one
of value and error (a void function sends value: null; error: null is
not an error), and no keys beyond the schema's. A response that breaks
it fails its pending call with UNKNOWN_ERROR. An error whose code is
empty, which the schema allows, now settles the call with that code
instead of failing it as malformed. Also pin that a response after
dispose settles nothing.

Signed-off-by: Nan Yu <nanyu@google.com>
The renderer function calls conformance runner now requires an
expect_error to be a non-empty map of the keys the suite uses (category,
code and message on a case; code and message on a step), so a typo or an
empty expectation fails instead of passing silently. message is matched
as a literal substring.

Signed-off-by: Nan Yu <nanyu@google.com>
Match conformance_schema.json: a step's expect_error is a
RendererCallError (required code, optional message) and a case's is a
RendererCallsBuildError (required category, only CatalogError, optional
message), with no other keys. Also cover a version '0.9' response as
malformed.

Signed-off-by: Nan Yu <nanyu@google.com>
nan-yu added a commit that referenced this pull request Nov 5, 2026
Audit the agent functions feature blueprint against the Python, TS and Dart
branches and fix the drift:
- The version check runs first for every message kind, so a
  callAgentFunction with a missing or other version raises even with a
  functionCallId; a schema-failing call with a readable id still gets an
  INVALID_FUNCTION_CALL reply in all three SDKs.
- The timeout option is agent_function_timeout (seconds) in Python,
  agentFunctionTimeoutMs in TypeScript and Duration agentFunctionTimeout in
  Dart; the sketch now uses a neutral name.
- The callback signatures per SDK, including Dart's FutureOr<void>
  callbacks that still decline with false.
- Only Python's a2ui.rpc re-exports the message classes; a validation error
  raised by a handler is INVALID_FUNCTION_CALL in Dart only.

Replace the follow-up note with section 5, the agent calling renderer
functions as implemented by #3939, #3940 and #3941: RendererFunctionCalls,
its checks and error codes, timeouts, resolve returning whether it owned
the response, malformed responses, disposal, the optional catalog checks,
the in-memory limitation on stateless servers, and the ADK integration.
Requirements, links, conformance (renderer_function_calls.yaml, 23 cases)
and implementation steps follow. The Python agent codebase blueprint says
renderer calls come in the follow-up #3939.

Signed-off-by: Nan Yu <nanyu@google.com>
@nan-yu
nan-yu force-pushed the feat/dart-renderer-function-calls branch from 833a091 to b40f00d Compare November 5, 2026 18:01
@nan-yu
nan-yu requested a review from gspencergoog November 5, 2026 18:33

@gspencergoog gspencergoog left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request adds RendererFunctionCalls in dart/a2ui_agent and RpcErrorCode.tooManyPending (TOO_MANY_PENDING) in dart/a2ui_core to construct outbound v1.0 callRendererFunction messages, validate call inputs and renderer catalog targets synchronously in the shared cross-SDK order, correlate incoming rendererFunctionResponse messages, and manage per-call timeouts and disposal. Both Round 1 review comments (M1 strict v1.0 FunctionResponse validation in _parseResponse and L1 void/malformed response unit test updates) and both earlier style threads have been verified as resolved at HEAD, and dart analyze --fatal-infos and dart test pass with zero issues.

🤖 Agent-generated

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants