diff --git a/conformance/README.md b/conformance/README.md index 07ae89ed17..22b108ee1f 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -37,6 +37,7 @@ The suites that do not depend on an inference format sit directly under `agent/` - `agent/skill.yaml`: Generating skill documents and skill sets from a format and a catalog. - `agent/multi_catalog_formats.yaml`: Resolving components and function calls across active catalogs (omitting `createSurface.catalogId` when multiple catalogs are active, resolving un-annotated components and function calls by name when unique across active catalogs, and requiring an explicit `catalogId` override when ambiguous), across `direct_json` (and later `express`): compiling, decompiling (including omitting `catalogId` when name lookup is unambiguous), parsing a response, rendering a prompt snippet, and building a format. Component order inside a message carries no meaning, so a harness may compare the `components` of a compiled message by `id`. - `agent/agent_functions.yaml`: Agent functions (v1.0) and the agent's side of the messages a renderer sends. It covers declaring agent functions on the agent's copy of a catalog (including building one from the catalog's own declaration), answering `callAgentFunction` (the response values and error codes, schema defaults, the handler's context, and which catalog a call runs in), validating model output that calls agent functions, and how `RendererToAgentMessageProcessor` routes `action`, `error` and `rendererFunctionResponse` messages to its callbacks. See [Writing cases for the agent function suite](#writing-cases-for-the-agent-function-suite). +- `agent/renderer_function_calls.yaml`: The agent calling renderer functions with `callRendererFunction` (v1.0): the message it builds, settling calls with responses, ignoring unknown, late and duplicate responses, timeouts, disposal and the limit on pending calls. See [Writing cases for the agent function suite](#writing-cases-for-the-agent-function-suite). - `agent/streamed_answer.yaml`: Reading a streamed answer into a renderer's message processor, for an app that is both the agent and the renderer, and keeping the answer's text, surfaces and rejected blocks in the order the model wrote them. It runs on `direct_json`, the format that streams. The suite belongs to the optional `agent_renderer_app` feature, and only an SDK that implements it runs the suite. A case names its format with `args.format`, an `InferenceFormatName`: `direct_json` (or `express`). @@ -173,7 +174,33 @@ The suite has four actions: In all four actions, `expect_error` is an `AgentFunctionExpectError`: a required `category`, `CatalogError` or `ValidationError`, and an optional `message`. The harness checks the error's category, and matches `message` as a literal substring of the error's text, not as a regular expression. A harness fails a case whose `expect_error` has any other key, so that an unrelated exception can't pass it. -The TypeScript and Dart agent SDKs do not implement agent functions yet, so their harnesses skip every case of this suite with that reason. +`agent/renderer_function_calls.yaml` has one action: + +- `call_renderer_function` makes calls to renderer functions and settles them, in `steps`. Each step does one thing: `call` makes a call named by its `label` (with `catalogId`, `name`, and optionally `args`, `surfaceId`, `functionCallId` and `timeout_ms`; `args` may be any JSON value, so a case can check that a non-object is refused), `respond` hands over a `rendererFunctionResponse` for a labelled `call` or a literal `functionCallId`, `advance_ms` moves the clock, and `dispose` disposes the calls with an optional `reason`. `args` may set `default_timeout_ms` (otherwise the SDK's default of 30 seconds applies), `max_pending`, and `catalogs`, the agent's catalogs that each `call` is checked against. A case may have `expect_error` (`RendererCallsBuildError`: a required `category`, which is `CatalogError`, and an optional literal-substring `message`) instead of `steps`, when building the calls helper must fail. A harness that hands responses over through a `RendererToAgentMessageProcessor`, with `resolve` as `on_renderer_function_response`, also checks that a response is handled exactly when its call is pending: `resolve` declines unknown, late and duplicate responses. + +A `call_renderer_function` harness runs the calls on a fake clock that only `advance_ms` moves, so the cases never wait on real time and a timer fires exactly when the clock reaches it. After each step it lets pending asynchronous work run, then checks: + +- `expect_message`, on a `call` step: the `callRendererFunction` message without its `functionCallId`. Separately, the harness checks that every call in a case got a different, non-empty `functionCallId`. +- `expect_error`, on a `call` step (`RendererCallError`): the call is refused with that RPC error `code`, and, when `message` is set, a message containing it as a literal substring. No other keys are allowed. +- `expect_calls`: the state of each labelled call, which is `pending`, `{value}`, or `{error: {code, message?}}` with `message` a substring. +- `expect_pending_count`: how many calls are waiting for a response. + +Each entry of `catalogs` (`RendererCallCatalog` in `conformance_schema.json`) is a v1.0 catalog that the harness builds in code, not from a catalog document, so the catalog schema's rules don't apply to it: + +```yaml +catalogs: + - catalogId: conformance/device + functions: + getLocation: {allowedCallers: rendererOrAgent} + vibrate: {allowedCallers: rendererOrAgent, requiresUserActivation: true} + formatLocal: {} # allowedCallers omitted, so rendererOnly +``` + +Each function takes any object as its arguments (schema `{type: object}`) and returns `any`; `allowedCallers` and `requiresUserActivation` are left unset when the case leaves them out. With `catalogs`, `call` fails with `UNKNOWN_FUNCTION` when `catalogId` names no catalog or the catalog lacks the function, and with `INVALID_FUNCTION_CALL` when the function is `rendererOnly` (or leaves `allowedCallers` out) or requires user activation. Two catalogs with one id make building the helper fail with a `CatalogError`. + +Before the catalog checks, `call` checks its own arguments, in this order: an empty given `functionCallId`, then (after the duplicate and pending-limit checks) an empty `name`, an empty `catalogId`, a `name` that isn't a UAX #31 identifier or starts with `@`, and `args` that aren't an object with identifier keys and JSON values. Each is `INVALID_FUNCTION_CALL`. Two cases pin the order with `expect_error.message`: an empty name and catalogId must give a message containing `function name`, and a bad argument key sent to a `rendererOnly` function must give a message containing the key. An SDK's message for a bad argument key names the key. A statically typed SDK whose `call` can't take a non-object `args` (Dart) meets `test_call_renderer_function_with_args_that_are_not_an_object` by showing that such a value can't be passed (a type error), with nothing sent or pending, instead of raising `INVALID_FUNCTION_CALL`. + +Only the Python agent harness runs this suite so far: the TypeScript and Dart agent harnesses (`typescript/agents/a2ui_agent` and `dart/a2ui_agent`) don't load it yet. ### Writing cases for the agent SDK suites diff --git a/conformance/agent/renderer_function_calls.yaml b/conformance/agent/renderer_function_calls.yaml new file mode 100644 index 0000000000..b9dd1ec0e1 --- /dev/null +++ b/conformance/agent/renderer_function_calls.yaml @@ -0,0 +1,452 @@ +# Copyright 2024 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The agent's calls to renderer functions with `callRendererFunction` (v1.0): +# building the message, settling calls with the renderer's responses, +# timeouts, disposal and the limit on pending calls. Renderer calls run on a +# fake clock that only an `advance_ms` step moves, so no case waits on real +# time. + +# --- Calling renderer functions --- + +- name: test_call_renderer_function_builds_the_message_without_waiting + description: >- + A call returns its callRendererFunction message at once, with a generated functionCallId and the catalogId the message requires, and stays pending until the renderer answers. + action: call_renderer_function + steps: + - call: + label: a + catalogId: conformance/agent_functions + name: notify + args: {message: Saved} + surfaceId: s1 + expect_message: + version: "1.0" + callRendererFunction: + surfaceId: s1 + callFunction: + "@call": notify + catalogId: conformance/agent_functions + args: {message: Saved} + expect_calls: + a: pending + expect_pending_count: 1 + +- name: test_call_renderer_function_without_surface_or_args + description: >- + A call made without a surface leaves surfaceId out of the message, and one made without arguments sends empty args. + action: call_renderer_function + steps: + - call: + label: a + catalogId: conformance/agent_functions + name: now + expect_message: + version: "1.0" + callRendererFunction: + callFunction: + "@call": now + catalogId: conformance/agent_functions + args: {} + expect_calls: + a: pending + +- name: test_call_renderer_function_with_an_empty_surface_id + description: >- + An empty surfaceId counts as no surface: it is left out of the message, which a renderer would otherwise reject. + action: call_renderer_function + steps: + - call: + label: a + catalogId: conformance/agent_functions + name: now + surfaceId: "" + expect_message: + version: "1.0" + callRendererFunction: + callFunction: + "@call": now + catalogId: conformance/agent_functions + args: {} + expect_calls: + a: pending + +- name: test_call_renderer_function_with_an_empty_name + description: >- + A call needs a function name: an empty one fails with INVALID_FUNCTION_CALL before anything is registered. + action: call_renderer_function + steps: + - call: {label: a, catalogId: conformance/agent_functions, name: ""} + expect_error: {code: INVALID_FUNCTION_CALL} + expect_pending_count: 0 + +- name: test_call_renderer_function_settled_by_its_response + description: >- + A rendererFunctionResponse settles the call whose functionCallId it copies with its value, and leaves the other calls pending. + action: call_renderer_function + steps: + - call: {label: a, catalogId: conformance/agent_functions, name: getLocale} + - call: {label: b, catalogId: conformance/agent_functions, name: getTimezone} + expect_pending_count: 2 + - respond: {call: b, value: Europe/Paris} + expect_calls: + a: pending + b: {value: Europe/Paris} + expect_pending_count: 1 + - respond: {call: a, value: {language: fr, region: FR}} + expect_calls: + a: {value: {language: fr, region: FR}} + b: {value: Europe/Paris} + expect_pending_count: 0 + +- name: test_call_renderer_function_settled_with_a_null_value + description: A response whose value is null settles the call with null. + action: call_renderer_function + steps: + - call: {label: a, catalogId: conformance/agent_functions, name: getSelection} + - respond: {call: a, value: null} + expect_calls: + a: {value: null} + +- name: test_call_renderer_function_failed_by_an_error_response + description: A response that carries an error fails the call with the renderer's code. + action: call_renderer_function + steps: + - call: {label: a, catalogId: conformance/agent_functions, name: getLocation} + - respond: + call: a + error: {code: PERMISSION_DENIED, message: "The user declined."} + expect_calls: + a: {error: {code: PERMISSION_DENIED, message: "The user declined."}} + expect_pending_count: 0 + +- name: test_call_renderer_function_ignores_an_unknown_response + description: >- + A response whose functionCallId matches no call comes from the network, so it is ignored without failing, and the pending calls stay pending. + action: call_renderer_function + steps: + - call: {label: a, catalogId: conformance/agent_functions, name: getLocale} + - respond: {functionCallId: never-sent, value: en} + expect_calls: + a: pending + expect_pending_count: 1 + +- name: test_call_renderer_function_ignores_a_duplicate_response + description: A second response to a settled call is ignored, and the call keeps the first value. + action: call_renderer_function + steps: + - call: {label: a, catalogId: conformance/agent_functions, name: getLocale} + - respond: {call: a, value: en} + - respond: {call: a, value: fr} + expect_calls: + a: {value: en} + +- name: test_call_renderer_function_times_out_after_thirty_seconds_by_default + description: >- + A call waits 30 seconds for its response by default, the same as the renderer waits for the agent, then fails with TIMEOUT and stops being pending. + action: call_renderer_function + steps: + - call: {label: a, catalogId: conformance/agent_functions, name: getLocale} + - advance_ms: 29999 + expect_calls: + a: pending + - advance_ms: 1 + expect_calls: + a: {error: {code: TIMEOUT}} + expect_pending_count: 0 + +- name: test_call_renderer_function_ignores_a_late_response + description: A response that arrives after its call timed out is ignored, and the call stays failed. + action: call_renderer_function + steps: + - call: {label: a, catalogId: conformance/agent_functions, name: getLocale, timeout_ms: 100} + - advance_ms: 100 + expect_calls: + a: {error: {code: TIMEOUT}} + - respond: {call: a, value: en} + expect_calls: + a: {error: {code: TIMEOUT}} + +- name: test_call_renderer_function_with_its_own_timeout + description: A timeout given to one call replaces the default for that call only. + action: call_renderer_function + steps: + - call: {label: short, catalogId: conformance/agent_functions, name: getLocale, timeout_ms: 500} + - call: {label: default, catalogId: conformance/agent_functions, name: getTimezone} + - advance_ms: 500 + expect_calls: + short: {error: {code: TIMEOUT}} + default: pending + +- name: test_call_renderer_function_with_a_default_timeout + description: A default timeout given when the calls are set up applies to every call that sets none of its own. + action: call_renderer_function + args: + default_timeout_ms: 2000 + steps: + - call: {label: a, catalogId: conformance/agent_functions, name: getLocale} + - advance_ms: 1999 + expect_calls: + a: pending + - advance_ms: 1 + expect_calls: + a: {error: {code: TIMEOUT}} + +- name: test_call_renderer_function_settled_before_its_timeout + description: A call that is answered in time does not time out later. + action: call_renderer_function + steps: + - call: {label: a, catalogId: conformance/agent_functions, name: getLocale, timeout_ms: 1000} + - advance_ms: 400 + - respond: {call: a, value: en} + - advance_ms: 1000 + expect_calls: + a: {value: en} + +- name: test_call_renderer_function_dispose_fails_pending_calls + description: >- + Disposing the calls fails every pending call with CANCELLED and the reason given, leaves settled calls alone, refuses later calls with DISPOSED, and ignores later responses. + action: call_renderer_function + steps: + - call: {label: done, catalogId: conformance/agent_functions, name: getLocale} + - call: {label: a, catalogId: conformance/agent_functions, name: getTimezone} + - call: {label: b, catalogId: conformance/agent_functions, name: getLocation} + - respond: {call: done, value: en} + - dispose: {reason: "the session ended"} + expect_calls: + done: {value: en} + a: {error: {code: CANCELLED, message: "the session ended"}} + b: {error: {code: CANCELLED, message: "the session ended"}} + expect_pending_count: 0 + - respond: {call: a, value: Europe/Paris} + expect_calls: + a: {error: {code: CANCELLED}} + - call: {label: c, catalogId: conformance/agent_functions, name: getLocale} + expect_error: {code: DISPOSED} + +- name: test_call_renderer_function_limits_pending_calls + description: >- + With a limit on pending calls, a call beyond it fails with TOO_MANY_PENDING instead of piling up, and settling a call makes room for another. + action: call_renderer_function + args: + max_pending: 2 + steps: + - call: {label: a, catalogId: conformance/agent_functions, name: getLocale} + - call: {label: b, catalogId: conformance/agent_functions, name: getTimezone} + - call: {label: c, catalogId: conformance/agent_functions, name: getLocation} + expect_error: {code: TOO_MANY_PENDING} + expect_pending_count: 2 + - respond: {call: a, value: en} + - call: {label: d, catalogId: conformance/agent_functions, name: getLocation} + expect_calls: + b: pending + d: pending + expect_pending_count: 2 + +# --- Checking calls against the agent's catalogs --- +# With `catalogs`, `call` checks the target before anything is registered or +# sent. Each catalog is built in code from its id and its functions. + +- name: test_call_renderer_function_checked_rejects_a_renderer_only_function + description: >- + A function whose allowedCallers is rendererOnly can't be called by the agent: the call fails with INVALID_FUNCTION_CALL, and nothing is sent or pending. + action: call_renderer_function + args: + catalogs: + - catalogId: conformance/device + functions: + formatLocal: {allowedCallers: rendererOnly} + steps: + - call: {label: a, catalogId: conformance/device, name: formatLocal} + expect_error: {code: INVALID_FUNCTION_CALL, message: formatLocal} + expect_pending_count: 0 + +- name: test_call_renderer_function_checked_rejects_a_function_without_allowed_callers + description: >- + A function that leaves allowedCallers out is rendererOnly, so the agent's call fails with INVALID_FUNCTION_CALL, and nothing is sent or pending. + action: call_renderer_function + args: + catalogs: + - catalogId: conformance/device + functions: + formatLocal: {} + steps: + - call: {label: a, catalogId: conformance/device, name: formatLocal} + expect_error: {code: INVALID_FUNCTION_CALL, message: formatLocal} + expect_pending_count: 0 + +- name: test_call_renderer_function_checked_rejects_a_function_that_needs_user_activation + description: >- + An agent call has no user gesture, so a function with requiresUserActivation fails with INVALID_FUNCTION_CALL even when its allowedCallers lets the agent call it. The catalog is built in code, so the schema rule that pairs activation with rendererOnly doesn't apply. + action: call_renderer_function + args: + catalogs: + - catalogId: conformance/device + functions: + vibrate: {allowedCallers: rendererOrAgent, requiresUserActivation: true} + steps: + - call: {label: a, catalogId: conformance/device, name: vibrate} + expect_error: {code: INVALID_FUNCTION_CALL, message: vibrate} + expect_pending_count: 0 + +- name: test_call_renderer_function_checked_rejects_a_function_the_catalog_lacks + description: >- + A function the named catalog doesn't have fails with UNKNOWN_FUNCTION, and nothing is sent or pending. + action: call_renderer_function + args: + catalogs: + - catalogId: conformance/device + functions: + getLocation: {allowedCallers: rendererOrAgent} + steps: + - call: {label: a, catalogId: conformance/device, name: getAltitude} + expect_error: {code: UNKNOWN_FUNCTION} + expect_pending_count: 0 + +- name: test_call_renderer_function_checked_rejects_an_unknown_catalog + description: >- + A catalogId that names none of the agent's catalogs fails with UNKNOWN_FUNCTION, and nothing is sent or pending. + action: call_renderer_function + args: + catalogs: + - catalogId: conformance/device + functions: + getLocation: {allowedCallers: rendererOrAgent} + steps: + - call: {label: a, catalogId: conformance/other, name: getLocation} + expect_error: {code: UNKNOWN_FUNCTION} + expect_pending_count: 0 + +- name: test_call_renderer_function_checked_allows_a_renderer_or_agent_function + description: >- + A rendererOrAgent function without user activation passes the check: the message is built, and the call settles from its response. + action: call_renderer_function + args: + catalogs: + - catalogId: conformance/device + functions: + getLocation: {allowedCallers: rendererOrAgent} + steps: + - call: + label: a + catalogId: conformance/device + name: getLocation + surfaceId: s1 + expect_message: + version: "1.0" + callRendererFunction: + surfaceId: s1 + callFunction: + "@call": getLocation + catalogId: conformance/device + args: {} + expect_pending_count: 1 + - respond: {call: a, value: Paris} + expect_calls: + a: {value: Paris} + expect_pending_count: 0 + +- name: test_call_renderer_function_checked_rejects_two_catalogs_with_one_id + description: >- + Two catalogs with the same id are a catalog error when the calls helper is built. + action: call_renderer_function + args: + catalogs: + - catalogId: conformance/device + functions: + getLocation: {allowedCallers: rendererOrAgent} + - catalogId: conformance/device + functions: + vibrate: {allowedCallers: agentOnly} + expect_error: {category: CatalogError} + +# --- The checks `call` makes on its own arguments --- +# Each check runs in a fixed order before anything is built, registered or +# timed. A rejected call sends nothing and leaves nothing pending. + +- name: test_call_renderer_function_with_an_empty_function_call_id + description: >- + A functionCallId given as an empty string is refused with INVALID_FUNCTION_CALL rather than replaced by a generated one. + action: call_renderer_function + steps: + - call: {label: a, catalogId: conformance/agent_functions, name: getLocale, functionCallId: ""} + expect_error: {code: INVALID_FUNCTION_CALL} + expect_pending_count: 0 + +- name: test_call_renderer_function_with_a_name_that_is_not_an_identifier + description: >- + A function name that isn't a UAX #31 identifier is refused with INVALID_FUNCTION_CALL. + action: call_renderer_function + steps: + - call: {label: a, catalogId: conformance/agent_functions, name: get-location} + expect_error: {code: INVALID_FUNCTION_CALL} + expect_pending_count: 0 + +- name: test_call_renderer_function_with_a_reserved_name + description: >- + A function name that starts with @, such as the @index system function, is refused with INVALID_FUNCTION_CALL. + action: call_renderer_function + steps: + - call: {label: a, catalogId: conformance/agent_functions, name: "@index"} + expect_error: {code: INVALID_FUNCTION_CALL} + expect_pending_count: 0 + +- name: test_call_renderer_function_with_args_that_are_not_an_object + description: >- + Arguments given as a list instead of an object are refused with INVALID_FUNCTION_CALL. + action: call_renderer_function + steps: + - call: {label: a, catalogId: conformance/agent_functions, name: notify, args: [Saved]} + expect_error: {code: INVALID_FUNCTION_CALL} + expect_pending_count: 0 + +- name: test_call_renderer_function_with_an_argument_key_that_is_not_an_identifier + description: >- + An argument key that isn't a UAX #31 identifier is refused with INVALID_FUNCTION_CALL, and the message names the key. + action: call_renderer_function + steps: + - call: + label: a + catalogId: conformance/agent_functions + name: notify + args: {bad-key: Saved} + expect_error: {code: INVALID_FUNCTION_CALL, message: bad-key} + expect_pending_count: 0 + +- name: test_call_renderer_function_checks_the_name_before_the_catalog_id + description: >- + With both the name and the catalogId empty, the error is about the missing function name: the name is checked first. + action: call_renderer_function + steps: + - call: {label: a, catalogId: "", name: ""} + expect_error: {code: INVALID_FUNCTION_CALL, message: function name} + expect_pending_count: 0 + +- name: test_call_renderer_function_checks_the_args_before_the_catalogs + description: >- + A call to a rendererOnly function whose arguments have a bad key is refused for the argument, which is checked before the catalogs: the message names the key. + action: call_renderer_function + args: + catalogs: + - catalogId: conformance/device + functions: + formatLocal: {allowedCallers: rendererOnly} + steps: + - call: + label: a + catalogId: conformance/device + name: formatLocal + args: {bad-key: 1} + expect_error: {code: INVALID_FUNCTION_CALL, message: bad-key} + expect_pending_count: 0 diff --git a/conformance/conformance_harness_implementation_guide.md b/conformance/conformance_harness_implementation_guide.md index 7c6e52f459..1dea1bdf04 100644 --- a/conformance/conformance_harness_implementation_guide.md +++ b/conformance/conformance_harness_implementation_guide.md @@ -159,7 +159,7 @@ Implement the suites in stages so each layer is tested before higher-level compo | A4. Multi-catalog and session pipeline | `agent/multi_catalog_formats.yaml`, `agent/request_processor.yaml`, `agent/streamed_answer.yaml` | Cross-catalog lookup, `InferenceFormatFactory`, `A2uiGenerator`, `A2uiRequestProcessor`, and `StreamedAnswer` | | A5. Specification round-trips | `catalogs/basic/v1/examples/*.json` (and v0.9 examples if supported) | End-to-end `decompile` followed by `compile` across every published basic catalog example | | A6. Skills, macros, and scaling | `agent/skill.yaml`, `agent/builder/builder.yaml`, `agent/macros/macros.yaml`, `agent/*/scaling.yaml` | Skill generation, typesafe fluent builders, macros, and linear-time parsing | -| A7. Agent functions | `agent/agent_functions.yaml` | Agent functions on a catalog, answering `callAgentFunction`, and routing renderer messages through `RendererToAgentMessageProcessor` | +| A7. Agent functions | `agent/agent_functions.yaml`, `agent/renderer_function_calls.yaml` | Agent functions on a catalog, answering `callAgentFunction`, routing renderer messages, and calling renderer functions on a fake clock | | F1. Framework adapter | `core/adapter_conformance.yaml`, `core/accessibility.yaml` | Rendered view trees, repeaters, two-way bindings, action and user-activation gating, validation state, accessibility, and unmount cleanup | Notice that `agent/express/` has no `response_streaming.yaml` file. Express parses whole buffered blocks and does not implement `openStream`. @@ -294,6 +294,8 @@ All four actions in `response_parser.yaml` run against `format.createParser()`: `agent/agent_functions.yaml` (`declare_agent_functions`, `handle_agent_function`, `validate_payload` and `process_renderer_message`) is described in [Writing cases for the agent function suite](README.md#writing-cases-for-the-agent-function-suite). Turn each declared `behavior` into a handler in the harness (for `raises`, throw the SDK's RPC error with `code`, its validation error with `validation: true`, or an ordinary error otherwise), build a declaration with `from_api: true` from the loaded catalog's own entry for it (or, when the catalog has none, from the declaration's `returnType`, `args` and `description`), and run every `callAgentFunction` through `RendererToAgentMessageProcessor`. For `process_renderer_message`, build the processor from `args.catalog` or `args.catalogs` inside the error check (building it can fail), and register a recording callback for each entry in `args.callbacks`: a plain kind returns nothing (handled), and `{kind, declines: true}` returns false (declined). An SDK that does not implement agent functions yet registers every case as skipped with that reason. +`agent/renderer_function_calls.yaml` (`call_renderer_function`) is described in the same section. Build each `args.catalogs` entry in code as a v1.0 catalog of schema-only functions (arguments `{type: object}`, return type `any`, `allowedCallers` and `requiresUserActivation` only when the entry sets them), and pass the list to the calls helper; a case with a top-level `expect_error` expects building the helper to fail. Pass a `call` step's `functionCallId` to `call` when the step sets it. In a statically typed SDK whose `call` can't take a non-object `args` (Dart), `test_call_renderer_function_with_args_that_are_not_an_object` passes by showing that such a value can't be passed (a type error), with nothing sent or pending. Run its cases on a fake clock: in Python, an event loop whose `time()` returns a counter that `advance_ms` steps add to; in JavaScript, fake timers. Only the Python agent harness runs the suite so far; the TypeScript and Dart agent harnesses don't load it yet. A harness that loads it before its SDK can call renderer functions registers every case as skipped with that reason. Reject an `expect_error` with keys outside its schema shape (`RendererCallError` on a step, `RendererCallsBuildError` on a case), so a case can't pass without checking the error. + ### Skills, builders, macros, round-trips, and timing suites 1. Skills (`agent/skill.yaml`): Dispatch five actions against `SkillGenerator` and `SkillSet`: `from_format` (`generateSkill`), `core_syntax` (`generateCoreSkill`), `from_catalog` (`generateCatalogSkill`), `skill_set` (`generateSkillSet`), and `export_to_directory` (`SkillSet.exportToDirectory`). Resolve all file paths in this suite from `REPO_ROOT`. Run `export_to_directory` in a temporary directory populated with `args.pre_existing_files` to check that exporting deletes only managed skill subdirectories and preserves sibling files (even on `expect_error`). diff --git a/conformance/conformance_schema.json b/conformance/conformance_schema.json index 96f7779286..ab3102246b 100644 --- a/conformance/conformance_schema.json +++ b/conformance/conformance_schema.json @@ -37,7 +37,8 @@ {"$ref": "#/$defs/DeclareAgentFunctionsTest"}, {"$ref": "#/$defs/HandleAgentFunctionTest"}, {"$ref": "#/$defs/ValidatePayloadTest"}, - {"$ref": "#/$defs/ProcessRendererMessageTest"} + {"$ref": "#/$defs/ProcessRendererMessageTest"}, + {"$ref": "#/$defs/CallRendererFunctionTest"} ] }, "$defs": { @@ -2566,6 +2567,216 @@ "required": ["category"], "additionalProperties": false }, + "CallRendererFunctionTest": { + "type": "object", + "description": "Calling renderer functions from the agent with callRendererFunction, and settling the calls with the renderer's responses, on a fake clock that only `advance_ms` steps move.", + "properties": { + "name": {"type": "string"}, + "description": {"type": "string"}, + "action": {"const": "call_renderer_function"}, + "args": { + "type": "object", + "properties": { + "default_timeout_ms": { + "type": ["integer", "null"], + "exclusiveMinimum": 0, + "description": "How long a call waits for its response unless it sets its own timeout. Without it, the SDK's default, 30000. Null waits without a limit." + }, + "max_pending": { + "type": "integer", + "minimum": 1, + "description": "The most calls that may wait for a response at once." + }, + "catalogs": { + "type": "array", + "description": "The agent's catalogs, which `call` checks each target against. Without them, calls aren't checked.", + "items": {"$ref": "#/$defs/RendererCallCatalog"} + } + }, + "additionalProperties": false + }, + "steps": { + "type": "array", + "items": {"$ref": "#/$defs/RendererCallStep"}, + "minItems": 1 + }, + "expect_error": { + "$ref": "#/$defs/RendererCallsBuildError", + "description": "Building the calls helper fails with this error, such as a CatalogError for two catalogs with one id. The case then has no `steps`." + } + }, + "required": ["name", "action"], + "oneOf": [{"required": ["steps"]}, {"required": ["expect_error"]}], + "additionalProperties": false + }, + "RendererCallCatalog": { + "type": "object", + "description": "A v1.0 catalog the harness builds in code (not from a catalog document, so it isn't checked against the catalog schema): its id and its functions, by name. Each function takes any object as `args` and returns `any`.", + "properties": { + "catalogId": {"type": "string"}, + "functions": { + "type": "object", + "additionalProperties": { + "type": "object", + "properties": { + "allowedCallers": { + "enum": ["rendererOnly", "agentOnly", "rendererOrAgent"], + "description": "Omitted means rendererOnly." + }, + "requiresUserActivation": {"type": "boolean"} + }, + "additionalProperties": false + } + } + }, + "required": ["catalogId", "functions"], + "additionalProperties": false + }, + "RendererCallState": { + "description": "The state of a call: still pending, settled with a value, or failed with an RPC error code. `error.message` is a substring of the failure's message.", + "oneOf": [ + {"const": "pending"}, + { + "type": "object", + "properties": {"value": {}}, + "required": ["value"], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "error": { + "type": "object", + "properties": { + "code": {"type": "string"}, + "message": {"type": "string"} + }, + "required": ["code"], + "additionalProperties": false + } + }, + "required": ["error"], + "additionalProperties": false + } + ] + }, + "RendererCallsBuildError": { + "type": "object", + "description": "The error building the calls helper fails with: its category and, optionally, a literal substring of its message.", + "properties": { + "category": {"enum": ["CatalogError"]}, + "message": { + "type": "string", + "description": "A literal substring of the error's message, not a regular expression." + } + }, + "required": ["category"], + "additionalProperties": false + }, + "RendererCallError": { + "type": "object", + "description": "The RPC error a `call` step is refused with: its code and, optionally, a literal substring of its message.", + "properties": { + "code": {"type": "string"}, + "message": { + "type": "string", + "description": "A literal substring of the error's message, not a regular expression." + } + }, + "required": ["code"], + "additionalProperties": false + }, + "RendererCallStep": { + "type": "object", + "description": "One step of a call_renderer_function case: exactly one of `call`, `respond`, `advance_ms` or `dispose`, then its expectations, checked once pending asynchronous work has run.", + "properties": { + "call": { + "type": "object", + "description": "Makes a call. `label` names it for later steps.", + "properties": { + "label": {"type": "string"}, + "catalogId": {"type": "string"}, + "name": {"type": "string"}, + "args": { + "description": "The call's arguments. Usually an object; any other JSON value tests that `call` refuses it." + }, + "surfaceId": {"type": "string"}, + "functionCallId": { + "type": "string", + "description": "The ID to give the call, in place of a generated one." + }, + "timeout_ms": { + "type": "integer", + "exclusiveMinimum": 0, + "description": "How long this call waits for its response, in place of the default." + } + }, + "required": ["label", "catalogId", "name"], + "additionalProperties": false + }, + "respond": { + "type": "object", + "description": "Hands the calls a rendererFunctionResponse, for the call `call` labels or for a literal `functionCallId`.", + "properties": { + "call": {"type": "string"}, + "functionCallId": {"type": "string"}, + "value": {}, + "error": { + "type": "object", + "properties": { + "code": {"type": "string"}, + "message": {"type": "string"} + }, + "required": ["code", "message"], + "additionalProperties": false + } + }, + "allOf": [ + {"oneOf": [{"required": ["call"]}, {"required": ["functionCallId"]}]}, + {"oneOf": [{"required": ["value"]}, {"required": ["error"]}]} + ], + "additionalProperties": false + }, + "advance_ms": { + "type": "integer", + "minimum": 0, + "description": "Moves the fake clock forward." + }, + "dispose": { + "type": "object", + "description": "Disposes the calls, with an optional reason.", + "properties": { + "reason": {"type": "string"} + }, + "additionalProperties": false + }, + "expect_message": { + "type": "object", + "description": "On a `call` step, the callRendererFunction message, less its functionCallId." + }, + "expect_error": { + "$ref": "#/$defs/RendererCallError", + "description": "On a `call` step, the call is refused with this RPC error instead of being made." + }, + "expect_calls": { + "type": "object", + "description": "The state of the calls it names, by label.", + "additionalProperties": {"$ref": "#/$defs/RendererCallState"} + }, + "expect_pending_count": { + "type": "integer", + "minimum": 0, + "description": "How many calls are waiting for a response." + } + }, + "oneOf": [ + {"required": ["call"]}, + {"required": ["respond"]}, + {"required": ["advance_ms"]}, + {"required": ["dispose"]} + ], + "additionalProperties": false + }, "ExpectError": { "description": "Expected error specification (error message string, regex, or structured error object).", "oneOf": [ diff --git a/python/a2ui_adk/CHANGELOG.md b/python/a2ui_adk/CHANGELOG.md index 71c9501a83..60db7f3879 100644 --- a/python/a2ui_adk/CHANGELOG.md +++ b/python/a2ui_adk/CHANGELOG.md @@ -1,5 +1,27 @@ ## Unreleased +- `a2ui_rpc_callback` takes `renderer_calls`, a provider of the session's + `RendererFunctionCalls` such as `A2uiRendererCallsStore.get`, and `catalogs` + becomes optional (pass one or both). The callback processes each message + with a `RendererToAgentMessageProcessor` whose + `on_renderer_function_response` is `RendererFunctionCalls.resolve`, and + answers the request without running the agent only when every message is + handled. A response to a call the session didn't make, such as one the + model wrote, is declined and goes to the agent. Responses that answer + pending calls always settle them, even in a request that goes to the agent, + and a malformed one fails the call it names. It needs the `a2a` + extra. [#3535](https://github.com/a2ui-project/a2ui/issues/3535) +- Add `a2ui_rpc_interceptor`, an `A2aAgentExecutor` interceptor that ends a + task `completed` when its request held only `callAgentFunction` and + `rendererFunctionResponse` messages and ADK would end it `working` because + there was nothing to send back. Other final states are left as they are. It + needs the `a2a` extra. + [#3535](https://github.com/a2ui-project/a2ui/issues/3535) +- Add `A2uiRendererCallsStore`, which keeps a `RendererFunctionCalls` for each + session. `get` returns the session's instance, creating it on first use, and + is the `renderer_calls` provider of `a2ui_rpc_callback`. `remove` disposes a + session's instance, which fails its pending calls. + [#3535](https://github.com/a2ui-project/a2ui/issues/3535) - Add `a2ui_rpc_callback`, which returns a `before_agent_callback` that answers the renderer's A2UI v1.0 `callAgentFunction` messages without running the agent. When the user content holds only such calls, it builds a diff --git a/python/a2ui_adk/README.md b/python/a2ui_adk/README.md index 6d07c83933..28da55a4f7 100644 --- a/python/a2ui_adk/README.md +++ b/python/a2ui_adk/README.md @@ -44,6 +44,10 @@ Available with the base install: since ADK's session services save state as JSON and can't hold one. Its `is_enabled` and `get` methods are the callbacks for the toolset, `A2uiInstruction` and `A2uiEventConverter`. +- `A2uiRendererCallsStore` keeps each session's `RendererFunctionCalls` (from + `a2ui.rpc`), which holds the agent's pending `callRendererFunction` calls. + Its `get` method returns the session's instance, creating it on first use, + and `remove` disposes it when the session ends. Available with the `a2a` extra: @@ -60,7 +64,18 @@ Available with the `a2a` extra: catalogs, and returns the `agentFunctionResponse` replies as the agent's response. For any other request, including the renderer's `action`, `error` and `rendererFunctionResponse` messages, it returns None, so the agent runs. - With `A2aAgentExecutor`, the replies reach the client as A2UI parts. + With `A2aAgentExecutor`, the replies reach the client as A2UI parts. With + `renderer_calls`, its processor's `on_renderer_function_response` is the + session's `RendererFunctionCalls.resolve`, which settles a response to a + pending call and declines any other, such as one to a call the model wrote. + The request is answered without the agent only when every message is + handled. A request that goes to the agent still settles the pending calls + its responses answer. +- `a2ui_rpc_interceptor` (also in `a2ui.adk.a2a`) returns an + `A2aAgentExecutor` interceptor that ends a task `completed` when its request + held only `callAgentFunction` and `rendererFunctionResponse` messages. ADK's + executor would otherwise end a task with nothing to send back, such as one + holding only `rendererFunctionResponse`, in the `working` state. - `exclude_a2ui_rpc_from_prompt` (also in `a2ui.adk.a2a`) is a `before_model_callback` that removes the `callAgentFunction` and `agentFunctionResponse` messages from the contents sent to the model. The @@ -110,6 +125,35 @@ can't answer, such as one without a `functionCallId` or with a `version` other than `"1.0"`, is logged and dropped. A request whose calls are all dropped gets a response with no parts, which `A2aAgentExecutor` ends in the `working` state. +To settle the agent's calls to renderer functions, also pass the session's +`RendererFunctionCalls`, and add the interceptor to the executor so a task that +held only responses ends `completed`: + +```python +from a2ui.adk import A2uiEventConverter, A2uiRendererCallsStore, a2ui_rpc_interceptor + +renderer_calls = A2uiRendererCallsStore() +agent = LlmAgent( + ..., + before_agent_callback=a2ui_rpc_callback( + catalogs=active_catalogs, renderer_calls=renderer_calls.get + ), +) +config = A2aAgentExecutorConfig( + event_converter=A2uiEventConverter(a2ui_processor=processors.get), + execute_interceptors=[a2ui_rpc_interceptor()], +) + +# In a tool +message, result = renderer_calls.get(tool_context).call( + catalog_id=CATALOG_ID, name="getLocation" +) +``` + +The pending calls live in this process's memory, so a deployment whose +requests for one session may reach different replicas can't settle them this +way. + ADK stores the calls and their replies in the session's events like any other turn. `SendA2uiToClientToolset` removes them from the model's prompt; an agent without the toolset passes `before_model_callback=exclude_a2ui_rpc_from_prompt`. diff --git a/python/a2ui_adk/src/a2ui/adk/__init__.py b/python/a2ui_adk/src/a2ui/adk/__init__.py index 3074f99ef1..a72647a31f 100644 --- a/python/a2ui_adk/src/a2ui/adk/__init__.py +++ b/python/a2ui_adk/src/a2ui/adk/__init__.py @@ -14,10 +14,11 @@ """A2UI extension for the Agent Development Kit (ADK). -`SendA2uiToClientToolset`, `A2uiInstruction` and `A2uiProcessorStore` only -need ADK. The ADK-to-A2A converters (`A2uiPartConverter`, -`A2uiEventConverter`), the A2UI RPC helpers (`a2ui_rpc_callback`, -`exclude_a2ui_rpc_from_prompt`), the `a2a` subpackage they live in, and the A2A subagent orchestration helpers +`SendA2uiToClientToolset`, `A2uiInstruction`, `A2uiProcessorStore` and +`A2uiRendererCallsStore` only need ADK. The ADK-to-A2A converters +(`A2uiPartConverter`, `A2uiEventConverter`), the A2UI RPC helpers +(`a2ui_rpc_callback`, `a2ui_rpc_interceptor`, `exclude_a2ui_rpc_from_prompt`), +the `a2a` subpackage they live in, and the A2A subagent orchestration helpers (`A2uiSubagentMap`, `SurfaceIdAlreadyExistsError`, the `orchestration` subpackage) need the `a2a` extra: @@ -33,6 +34,7 @@ from .instruction import A2uiInstruction from .processor_store import A2uiProcessorStore +from .renderer_calls_store import A2uiRendererCallsStore from .send_a2ui_to_client_toolset import ( A2uiEnabledProvider, A2uiProcessorProvider, @@ -47,7 +49,9 @@ A2uiPartConverter, A2uiRpcCallback, CatalogsProvider, + RendererFunctionCallsProvider, a2ui_rpc_callback, + a2ui_rpc_interceptor, exclude_a2ui_rpc_from_prompt, ) from .orchestration import A2uiSubagentMap, SurfaceIdAlreadyExistsError @@ -62,7 +66,9 @@ "A2uiPartConverter": (".a2a", "A2uiPartConverter"), "A2uiRpcCallback": (".a2a", "A2uiRpcCallback"), "CatalogsProvider": (".a2a", "CatalogsProvider"), + "RendererFunctionCallsProvider": (".a2a", "RendererFunctionCallsProvider"), "a2ui_rpc_callback": (".a2a", "a2ui_rpc_callback"), + "a2ui_rpc_interceptor": (".a2a", "a2ui_rpc_interceptor"), "exclude_a2ui_rpc_from_prompt": (".a2a", "exclude_a2ui_rpc_from_prompt"), "A2uiSubagentMap": (".orchestration", "A2uiSubagentMap"), "SurfaceIdAlreadyExistsError": ( @@ -114,13 +120,16 @@ def __getattr__(name: str) -> Any: "A2uiPartConverter", "A2uiProcessorProvider", "A2uiProcessorStore", + "A2uiRendererCallsStore", "A2uiRpcCallback", "A2uiSubagentMap", "CatalogsProvider", + "RendererFunctionCallsProvider", "SendA2uiToClientToolset", "SurfaceIdAlreadyExistsError", "a2a", "a2ui_rpc_callback", + "a2ui_rpc_interceptor", "exclude_a2ui_rpc_from_prompt", "orchestration", ] diff --git a/python/a2ui_adk/src/a2ui/adk/a2a/__init__.py b/python/a2ui_adk/src/a2ui/adk/a2a/__init__.py index 63a13ab717..7c04949332 100644 --- a/python/a2ui_adk/src/a2ui/adk/a2a/__init__.py +++ b/python/a2ui_adk/src/a2ui/adk/a2a/__init__.py @@ -16,7 +16,13 @@ from .event_converter import A2uiEventConverter from .part_converter import A2uiPartConverter -from .rpc_callback import A2uiRpcCallback, CatalogsProvider, a2ui_rpc_callback +from .rpc_callback import ( + A2uiRpcCallback, + CatalogsProvider, + RendererFunctionCallsProvider, + a2ui_rpc_callback, +) +from .rpc_interceptor import a2ui_rpc_interceptor from .rpc_prompt_filter import exclude_a2ui_rpc_from_prompt __all__ = [ @@ -24,6 +30,8 @@ "A2uiPartConverter", "A2uiRpcCallback", "CatalogsProvider", + "RendererFunctionCallsProvider", "a2ui_rpc_callback", + "a2ui_rpc_interceptor", "exclude_a2ui_rpc_from_prompt", ] diff --git a/python/a2ui_adk/src/a2ui/adk/a2a/_rpc.py b/python/a2ui_adk/src/a2ui/adk/a2a/_rpc.py index e860c186d2..db3b6b2a05 100644 --- a/python/a2ui_adk/src/a2ui/adk/a2a/_rpc.py +++ b/python/a2ui_adk/src/a2ui/adk/a2a/_rpc.py @@ -19,6 +19,7 @@ from collections.abc import Mapping from typing import Any +from a2a import types as a2a_types from google.adk.a2a.converters import part_converter from google.genai import types as genai_types @@ -26,6 +27,11 @@ CALL_AGENT_FUNCTION = "callAgentFunction" AGENT_FUNCTION_RESPONSE = "agentFunctionResponse" +RENDERER_FUNCTION_RESPONSE = "rendererFunctionResponse" + +# The renderer's messages that `a2ui_rpc_callback` may answer without the +# model, and which `a2ui_rpc_interceptor` completes. +RENDERER_RPC_KINDS = frozenset({CALL_AGENT_FUNCTION, RENDERER_FUNCTION_RESPONSE}) # The agent function messages, in both directions, which `a2ui_rpc_callback` # answers without the model. The renderer's other messages (`action`, `error`, @@ -39,6 +45,16 @@ def message_kind(message: Mapping[str, Any]) -> str | None: return keys[0] if len(keys) == 1 else None +def is_renderer_rpc_only(parts: list[a2a_types.Part]) -> bool: + """Whether every part is a renderer message that the callback may answer.""" + messages = get_a2ui_messages(parts) + return ( + bool(messages) + and len(messages) == len(parts) + and all(message_kind(message) in RENDERER_RPC_KINDS for message in messages) + ) + + def genai_part_a2ui_messages(part: genai_types.Part) -> list[dict[str, Any]]: """The A2UI messages in a GenAI part that ADK converted from an A2A part. diff --git a/python/a2ui_adk/src/a2ui/adk/a2a/rpc_callback.py b/python/a2ui_adk/src/a2ui/adk/a2a/rpc_callback.py index 8ba69105cd..83db072b01 100644 --- a/python/a2ui_adk/src/a2ui/adk/a2a/rpc_callback.py +++ b/python/a2ui_adk/src/a2ui/adk/a2a/rpc_callback.py @@ -12,15 +12,17 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Module for `a2ui_rpc_callback`, which answers agent function calls in ADK. - -From v1.0, a renderer sends `callAgentFunction` to run a function on the agent. -Answering it is deterministic, and the renderer shows a pending value until -the answer arrives, so it shouldn't wait for a model turn. -`a2ui_rpc_callback` returns a `before_agent_callback` that answers a request -holding only such calls, through a `RendererToAgentMessageProcessor` built -from the agent's catalogs, and skips the agent. The renderer's other messages -(`action`, `error`, `rendererFunctionResponse`) go to the agent's turn. +"""Module for `a2ui_rpc_callback`, which answers A2UI RPC messages in ADK. + +From v1.0, a renderer sends `callAgentFunction` to run a function on the agent, +and `rendererFunctionResponse` to answer the agent's `callRendererFunction`. +Answering them is deterministic, and the renderer shows a pending value until +the answer arrives, so they shouldn't wait for a model turn. +`a2ui_rpc_callback` returns a `before_agent_callback` that handles a request +holding only such messages, through a `RendererToAgentMessageProcessor`, and +skips the agent. The renderer's other messages (`action`, `error`), and +responses to calls the session's `RendererFunctionCalls` didn't make, go to +the agent's turn. It works with ADK's `A2aAgentExecutor`. The executor turns each A2UI `DataPart` of the request into a GenAI part, and the callback turns it back @@ -32,6 +34,7 @@ ```python processors = A2uiProcessorStore() + renderer_calls = A2uiRendererCallsStore() def active_catalogs(ctx: ReadonlyContext) -> list[CatalogApi] | None: processor = processors.get(ctx) @@ -39,14 +42,15 @@ def active_catalogs(ctx: ReadonlyContext) -> list[CatalogApi] | None: agent = LlmAgent( ..., - before_agent_callback=a2ui_rpc_callback(catalogs=active_catalogs), + before_agent_callback=a2ui_rpc_callback( + catalogs=active_catalogs, renderer_calls=renderer_calls.get + ), ) ``` """ from __future__ import annotations -import asyncio from collections.abc import Awaitable, Callable, Iterable, Mapping import inspect import logging @@ -62,9 +66,14 @@ def active_catalogs(ctx: ReadonlyContext) -> list[CatalogApi] | None: from a2ui.a2a import create_a2ui_parts from a2ui.core import A2uiValidationError, CatalogApi from a2ui.core.schema.v1_0 import AgentToRendererMessage -from a2ui.rpc import RendererToAgentMessageProcessor +from a2ui.rpc import RendererFunctionCalls, RendererToAgentMessageProcessor -from ._rpc import CALL_AGENT_FUNCTION, genai_part_a2ui_messages, message_kind +from ._rpc import ( + CALL_AGENT_FUNCTION, + RENDERER_FUNCTION_RESPONSE, + genai_part_a2ui_messages, + message_kind, +) logger = logging.getLogger(__name__) @@ -74,32 +83,54 @@ def active_catalogs(ctx: ReadonlyContext) -> list[CatalogApi] | None: ] """Returns the agent's catalogs for a context, or None for none.""" +RendererFunctionCallsProvider: TypeAlias = Callable[ + [ReadonlyContext], + RendererFunctionCalls | None | Awaitable[RendererFunctionCalls | None], +] +"""Returns the session's `RendererFunctionCalls`, such as `A2uiRendererCallsStore.get`.""" + A2uiRpcCallback: TypeAlias = Callable[ [CallbackContext], Awaitable[genai_types.Content | None] ] -"""A `before_agent_callback` that answers A2UI agent function calls.""" +"""A `before_agent_callback` that answers A2UI RPC messages.""" @experimental def a2ui_rpc_callback( *, - catalogs: Iterable[CatalogApi] | CatalogsProvider, + catalogs: Iterable[CatalogApi] | CatalogsProvider | None = None, + renderer_calls: RendererFunctionCallsProvider | None = None, agent_function_timeout: float | None = None, ) -> A2uiRpcCallback: - """Returns a `before_agent_callback` that answers agent function calls. - - The callback reads the user content that started the invocation. When - every part of it is an A2UI `callAgentFunction` message, it builds a - `RendererToAgentMessageProcessor` from `catalogs`, has it run each call's - agent function, and returns the `agentFunctionResponse` replies as the - agent's response, so the agent doesn't run. - - For any other content it returns None and the agent runs as usual. That - includes text, the renderer's `action`, `error` and - `rendererFunctionResponse` messages, and a mix of calls and other parts. - A `callAgentFunction` that the processor can't answer, such as one - without a `functionCallId` or with a `version` other than `"1.0"`, is - logged and dropped. + """Returns a `before_agent_callback` that answers A2UI RPC messages. + + The callback reads the user content that started the invocation and + processes each A2UI message in it, one at a time, with a + `RendererToAgentMessageProcessor` built from `catalogs`, whose + `on_renderer_function_response` is the session's + `RendererFunctionCalls.resolve`, from `renderer_calls`: + + 1. `callAgentFunction` runs the agent function of `catalogs` and replies + with its `agentFunctionResponse`. + 2. `rendererFunctionResponse` that answers a pending call of the session + settles the call and sends no reply. `resolve` declines any other + response, such as one to a call the model wrote itself (which the + inference formats compile into `callRendererFunction`), so it isn't + handled. + 3. `action` and `error` aren't handled. + + When every part is an A2UI message and every message is handled, the + callback returns the replies as the agent's response, so the agent doesn't + run. A request holding only responses gets a response with no parts, + which `A2aAgentExecutor` ends in the `working` state; + `a2ui_rpc_interceptor` ends it `completed` instead. For any other content + it returns None and the agent runs as usual, with the whole request. + Responses are processed first, so a pending call is always settled, even + in a request that goes to the agent, and `callAgentFunction` runs only + when the request is answered. A `callAgentFunction` that the + processor can't answer, such as one without a `functionCallId` or with a + `version` other than `"1.0"`, is logged and dropped. A + malformed `rendererFunctionResponse` fails the call it names. To combine it with a callback of your own, pass both as a list: `before_agent_callback=[a2ui_rpc_callback(...), my_callback]`. ADK runs @@ -109,8 +140,11 @@ def a2ui_rpc_callback( catalogs: The catalogs whose agent functions answer `callAgentFunction`, or a sync or async provider that returns them, or None, per context. A provider can return the session's - `A2uiRequestProcessor.active_catalogs`. When it returns None, - the calls go to the agent. + `A2uiRequestProcessor.active_catalogs`. Without catalogs, + `callAgentFunction` goes to the agent. + renderer_calls: A sync or async provider of the session's + `RendererFunctionCalls`, such as `A2uiRendererCallsStore.get`. + Without one, `rendererFunctionResponse` goes to the agent. agent_function_timeout: Seconds an agent function may run, as for `RendererToAgentMessageProcessor`. None, the default, sets no limit. @@ -119,9 +153,11 @@ def a2ui_rpc_callback( The callback, for `before_agent_callback` of an ADK agent. Raises: - ValueError: If `agent_function_timeout` is not a finite positive - number. + ValueError: If neither `catalogs` nor `renderer_calls` is given, or + `agent_function_timeout` is not a finite positive number. """ + if catalogs is None and renderer_calls is None: + raise ValueError("Pass catalogs, renderer_calls, or both.") if agent_function_timeout is not None and ( isinstance(agent_function_timeout, bool) or not isinstance(agent_function_timeout, (int, float)) @@ -133,35 +169,65 @@ def a2ui_rpc_callback( fixed: list[CatalogApi] | None = None if callable(catalogs): provider = catalogs - else: + elif catalogs is not None: fixed = list(catalogs) async def callback(callback_context: CallbackContext) -> genai_types.Content | None: - messages = _rpc_messages(callback_context.user_content) - if not messages or not all( - message_kind(message) == CALL_AGENT_FUNCTION for message in messages + content = callback_context.user_content + if content is None or not content.parts: + return None + per_part = [genai_part_a2ui_messages(part) for part in content.parts] + messages = [message for part in per_part for message in part] + if not messages: + return None + calls_processor = RendererToAgentMessageProcessor([]) + calls: RendererFunctionCalls | None = None + if renderer_calls is not None and any( + message_kind(message) == RENDERER_FUNCTION_RESPONSE for message in messages ): + calls = await _resolve(renderer_calls, callback_context) + if calls is not None: + calls_processor = RendererToAgentMessageProcessor( + [], on_renderer_function_response=calls.resolve + ) + + # Everything but agent function calls first: this settles the + # session's pending calls whatever happens to the request, and has no + # other side effects. + handled = [ + await _process_other(calls_processor, calls, message) + for message in messages + if message_kind(message) != CALL_AGENT_FUNCTION + ] + agent_calls = [ + message + for message in messages + if message_kind(message) == CALL_AGENT_FUNCTION + ] + if not all(handled) or not all(per_part): + # Text or another part that isn't A2UI, or a message the processor + # didn't handle: the agent runs, with the whole request. return None - resolved = ( - fixed - if provider is None - else await _resolve_catalogs(provider, callback_context) - ) - if resolved is None: - return None - # The processor keeps no state, so building one per request is cheap. - processor = RendererToAgentMessageProcessor( - resolved, agent_function_timeout=agent_function_timeout - ) + replies: list[AgentToRendererMessage] = [] + if agent_calls: + resolved: Iterable[CatalogApi] | None = ( + fixed + if provider is None + else await _resolve(provider, callback_context) + ) + if resolved is None: + return None + # The processor keeps no state, so building one per request is + # cheap. + processor = RendererToAgentMessageProcessor( + resolved, agent_function_timeout=agent_function_timeout + ) + for message in agent_calls: + replies.extend(await _process_call(processor, message)) - replies = await asyncio.gather( - *(_process(processor, message) for message in messages) - ) parts: list[genai_types.Part] = [] - for a2a_part in create_a2ui_parts( - reply for message_replies in replies for reply in message_replies - ): + for a2a_part in create_a2ui_parts(replies): genai_part = part_converter.convert_a2a_part_to_genai_part(a2a_part) if genai_part is not None: parts.append(genai_part) @@ -170,33 +236,50 @@ async def callback(callback_context: CallbackContext) -> genai_types.Content | N return callback -def _rpc_messages(content: genai_types.Content | None) -> list[dict[str, Any]]: - """The A2UI messages of `content`, or none if any part isn't A2UI.""" - if content is None or not content.parts: - return [] - messages: list[dict[str, Any]] = [] - for part in content.parts: - part_messages = genai_part_a2ui_messages(part) - if not part_messages: - return [] - messages.extend(part_messages) - return messages - - -async def _resolve_catalogs( - provider: CatalogsProvider, ctx: ReadonlyContext -) -> Iterable[CatalogApi] | None: - """Resolves a sync or async catalogs provider for `ctx`.""" +async def _resolve( + provider: Callable[[ReadonlyContext], Any], ctx: ReadonlyContext +) -> Any: + """Calls a sync or async provider for `ctx`.""" resolved = provider(ctx) if inspect.isawaitable(resolved): resolved = await resolved return resolved -async def _process( +async def _process_other( + processor: RendererToAgentMessageProcessor, + calls: RendererFunctionCalls | None, + message: Mapping[str, Any], +) -> bool: + """Processes a message that isn't an agent function call. + + Returns whether it was handled. A message that can't be processed isn't, + except a malformed response that `calls` takes, which fails the pending + call it names at once. + """ + try: + return (await processor.process_message(message)).handled + except A2uiValidationError as e: + if ( + calls is not None + and message_kind(message) == RENDERER_FUNCTION_RESPONSE + and calls.resolve(message) + ): + # `resolve` logged the malformed response and failed its call. + return True + # The error may quote the payload, so the details are logged only at + # debug level. + logger.warning( + "Passing an invalid A2UI message on to the agent: %s", type(e).__name__ + ) + logger.debug("Invalid A2UI message error: %s", e) + return False + + +async def _process_call( processor: RendererToAgentMessageProcessor, message: Mapping[str, Any] ) -> list[AgentToRendererMessage]: - """Processes one call, logging and dropping one that can't be answered.""" + """Answers one agent function call, logging and dropping one that can't be.""" try: result = await processor.process_message(message) except A2uiValidationError as e: diff --git a/python/a2ui_adk/src/a2ui/adk/a2a/rpc_interceptor.py b/python/a2ui_adk/src/a2ui/adk/a2a/rpc_interceptor.py new file mode 100644 index 0000000000..793f9819cb --- /dev/null +++ b/python/a2ui_adk/src/a2ui/adk/a2a/rpc_interceptor.py @@ -0,0 +1,107 @@ +# Copyright 2024 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Module for `a2ui_rpc_interceptor`, which completes A2UI RPC tasks. + +ADK's `A2aAgentExecutor` marks a task `completed` only when the agent sent +something. A request holding only `rendererFunctionResponse` messages has +nothing to send back, so `a2ui_rpc_callback` answers it with an empty +response, and the executor would end the task in the `working` state. The +interceptor ends such tasks `completed` instead. + +Usage Example: + + ```python + config = A2aAgentExecutorConfig( + event_converter=A2uiEventConverter(a2ui_processor=processors.get), + execute_interceptors=[a2ui_rpc_interceptor()], + ) + ``` +""" + +from __future__ import annotations + +from a2a.server.agent_execution.context import RequestContext +from a2a.types import TaskState, TaskStatusUpdateEvent +from google.adk.a2a.executor.config import ExecuteInterceptor +from google.adk.a2a.executor.executor_context import ExecutorContext +from google.adk.utils.feature_decorator import experimental + +from ._rpc import is_renderer_rpc_only + +# Tasks whose request held only RPC messages, between `before_agent` and +# `after_agent`. A request that fails in between never reaches +# `after_agent`, so the oldest entries are dropped past this size. +_MAX_TRACKED_TASKS = 1024 + + +class _RpcTaskTracker: + """Remembers which tasks' requests held only renderer RPC messages.""" + + def __init__(self) -> None: + # A dict keeps insertion order, so the first key is the oldest. + self._rpc_task_ids: dict[str, None] = {} + + async def before_agent(self, context: RequestContext) -> RequestContext: + if context.task_id is None: + return context + # Forgets an entry left by an earlier request on this task that never + # reached `after_agent`, so a later request that isn't RPC-only keeps + # its status. Re-adding the key moves it to the end of the order. + self._rpc_task_ids.pop(context.task_id, None) + if context.message is not None and is_renderer_rpc_only(context.message.parts): + self._rpc_task_ids[context.task_id] = None + while len(self._rpc_task_ids) > _MAX_TRACKED_TASKS: + del self._rpc_task_ids[next(iter(self._rpc_task_ids))] + return context + + async def after_agent( + self, executor_context: ExecutorContext, final_event: TaskStatusUpdateEvent + ) -> TaskStatusUpdateEvent: + if final_event.task_id not in self._rpc_task_ids: + return final_event + del self._rpc_task_ids[final_event.task_id] + status = final_event.status + if status.state != TaskState.working or ( + status.message is not None and status.message.parts + ): + return final_event + return final_event.model_copy( + update={"status": status.model_copy(update={"state": TaskState.completed})} + ) + + +@experimental +def a2ui_rpc_interceptor() -> ExecuteInterceptor: + """Returns an `A2aAgentExecutor` interceptor that completes RPC-only tasks. + + Its `before_agent` hook checks the request message: when every part is an + A2UI `callAgentFunction` or `rendererFunctionResponse` message, it notes + the task. Its `after_agent` hook then changes that task's final status + from `working`, which ADK's executor reports when the agent sent nothing, + to `completed`. Other final states, such as `failed`, `input_required` or + a `completed` task with a result, and every other task, are left as they + are. + + It reads the request itself, so it doesn't depend on `a2ui_rpc_callback`, + though the two are meant to be used together. Use one interceptor per + executor. + + Returns: + The interceptor, for `A2aAgentExecutorConfig.execute_interceptors`. + """ + tracker = _RpcTaskTracker() + return ExecuteInterceptor( + before_agent=tracker.before_agent, after_agent=tracker.after_agent + ) diff --git a/python/a2ui_adk/src/a2ui/adk/renderer_calls_store.py b/python/a2ui_adk/src/a2ui/adk/renderer_calls_store.py new file mode 100644 index 0000000000..dd56a725b2 --- /dev/null +++ b/python/a2ui_adk/src/a2ui/adk/renderer_calls_store.py @@ -0,0 +1,131 @@ +# Copyright 2024 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Module for the A2uiRendererCallsStore. + +A `RendererFunctionCalls` holds the agent's pending `callRendererFunction` +calls of one conversation, as asyncio futures, so like an +`A2uiRequestProcessor` it can't be kept in ADK session state. +`A2uiRendererCallsStore` keeps one per session in memory. A tool gets the +session's instance to call a renderer function, and the callback from +`a2ui_rpc_callback` gets the same instance to settle the call when the +renderer's `rendererFunctionResponse` arrives. + +Usage Example: + + ```python + renderer_calls = A2uiRendererCallsStore() + + # In the agent + agent = LlmAgent( + ..., + before_agent_callback=a2ui_rpc_callback( + catalogs=catalogs, renderer_calls=renderer_calls.get + ), + ) + + # In a tool + message, result = renderer_calls.get(tool_context).call( + catalog_id=CATALOG_ID, name="getLocation" + ) + + # When the session ends + renderer_calls.remove(session.id) + ``` +""" + +from __future__ import annotations + +from google.adk.agents.readonly_context import ReadonlyContext +from google.adk.utils.feature_decorator import experimental + +from a2ui.rpc import DEFAULT_RENDERER_CALL_TIMEOUT, RendererFunctionCalls + + +@experimental +class A2uiRendererCallsStore: + """Keeps a `RendererFunctionCalls` for each session, outside session state. + + `get` creates the session's instance the first time it is asked for, with + the store's `default_timeout` and `max_pending`. Call `remove` when a + session ends: it disposes the session's instance, which fails its pending + calls. Nothing is evicted on its own. + """ + + def __init__( + self, + *, + default_timeout: float | None = DEFAULT_RENDERER_CALL_TIMEOUT, + max_pending: int | None = None, + ) -> None: + """Initializes the store. + + Args: + default_timeout: The `default_timeout` of each session's + `RendererFunctionCalls`. + max_pending: The `max_pending` of each session's + `RendererFunctionCalls`. + + Raises: + ValueError: If `default_timeout` is not a finite positive number + or None, or `max_pending` is less than 1, as for + `RendererFunctionCalls`. + """ + # Checks the settings the way each session's instance will. + RendererFunctionCalls(default_timeout=default_timeout, max_pending=max_pending) + self._default_timeout = default_timeout + self._max_pending = max_pending + self._calls: dict[str, RendererFunctionCalls] = {} + + def get(self, ctx: ReadonlyContext) -> RendererFunctionCalls: + """Returns the `RendererFunctionCalls` of the context's session. + + It has the signature of a `RendererFunctionCallsProvider`, so it can be + passed as the `renderer_calls` of `a2ui_rpc_callback`. + """ + return self.get_for_session(ctx.session.id) + + def get_for_session(self, session_id: str) -> RendererFunctionCalls: + """Returns the session's `RendererFunctionCalls`, creating it if needed. + + An instance that was disposed outside the store is replaced. + """ + calls = self._calls.get(session_id) + if calls is None or calls.disposed: + calls = RendererFunctionCalls( + default_timeout=self._default_timeout, max_pending=self._max_pending + ) + self._calls[session_id] = calls + return calls + + def remove(self, session_id: str, reason: str = "The session was removed.") -> None: + """Removes and disposes the session's instance, if it has one. + + Its pending calls fail with `A2uiRpcError(reason)`. + """ + calls = self._calls.pop(session_id, None) + if calls is not None: + calls.dispose(reason) + + def dispose(self, reason: str = "The store was disposed.") -> None: + """Removes and disposes every session's instance, as on shutdown.""" + calls, self._calls = self._calls, {} + for instance in calls.values(): + instance.dispose(reason) + + def __contains__(self, session_id: object) -> bool: + return session_id in self._calls + + def __len__(self) -> int: + return len(self._calls) diff --git a/python/a2ui_adk/tests/a2a/test_rpc_callback.py b/python/a2ui_adk/tests/a2a/test_rpc_callback.py index 40d74020b7..778e612019 100644 --- a/python/a2ui_adk/tests/a2a/test_rpc_callback.py +++ b/python/a2ui_adk/tests/a2a/test_rpc_callback.py @@ -35,6 +35,8 @@ Part, Role, TaskArtifactUpdateEvent, + TaskState, + TaskStatus, TaskStatusUpdateEvent, TextPart, ) @@ -43,6 +45,7 @@ A2aAgentExecutor, A2aAgentExecutorConfig, ) +from google.adk.a2a.executor.config import ExecuteInterceptor from google.adk.agents.base_agent import BaseAgent from google.adk.agents.callback_context import CallbackContext from google.adk.agents.invocation_context import InvocationContext @@ -53,9 +56,14 @@ import pytest from a2ui.a2a import A2UI_MIME_TYPE, create_a2ui_part, get_a2ui_messages -from a2ui.adk import A2uiProcessorStore -from a2ui.adk.a2a import A2uiEventConverter, A2uiPartConverter, a2ui_rpc_callback -from a2ui.core import AgentFunction, AgentFunctionContext, CatalogApi +from a2ui.adk import A2uiProcessorStore, A2uiRendererCallsStore +from a2ui.adk.a2a import ( + A2uiEventConverter, + A2uiPartConverter, + a2ui_rpc_callback, + a2ui_rpc_interceptor, +) +from a2ui.core import A2uiRpcError, AgentFunction, AgentFunctionContext, CatalogApi from a2ui.core.basic_catalog import BasicCatalog from a2ui.core.schema import ProtocolVersion from a2ui.processor import A2uiRequestProcessor @@ -215,6 +223,11 @@ def _error_code(message: dict[str, Any]) -> str: } +def test_needs_catalogs_or_renderer_calls() -> None: + with pytest.raises(ValueError): + a2ui_rpc_callback() + + @pytest.mark.parametrize("timeout", [0, -1, float("nan"), float("inf"), True, "1"]) def test_rejects_a_timeout_that_is_not_a_finite_positive_number(timeout: Any) -> None: with pytest.raises(ValueError): @@ -549,7 +562,10 @@ def other(callback_context: CallbackContext) -> genai_types.Content: async def _execute( - agent: BaseAgent, request_parts: list[Part], session_id: str + agent: BaseAgent, + request_parts: list[Part], + session_id: str, + interceptors: list[ExecuteInterceptor] | None = None, ) -> list[Any]: """Runs one A2A request through ADK's `A2aAgentExecutor`. @@ -564,7 +580,8 @@ async def _execute( executor = A2aAgentExecutor( runner=runner, config=A2aAgentExecutorConfig( - event_converter=A2uiEventConverter(a2ui_processor=processors.get) + event_converter=A2uiEventConverter(a2ui_processor=processors.get), + execute_interceptors=interceptors, ), ) request = RequestContext( @@ -648,3 +665,410 @@ async def test_a2a_agent_executor_passes_a_renderer_function_response_to_the_age if isinstance(part.root, TextPart) ] == [AGENT_TEXT] _assert_completed(events) + + +class _SilentAgent(BaseAgent): + """Sends nothing.""" + + async def _run_async_impl( + self, ctx: InvocationContext + ) -> AsyncGenerator[Event, None]: + return + yield + + +def _renderer_response(call_id: str, value: Any = "Paris") -> dict[str, Any]: + return { + "version": "1.0", + "rendererFunctionResponse": {"functionCallId": call_id, "value": value}, + } + + +def _final_state(events: list[Any]) -> str: + final = events[-1] + assert isinstance(final, TaskStatusUpdateEvent) + assert final.final + return final.status.state + + +@pytest.mark.asyncio +async def test_renderer_function_response_settles_the_pending_call() -> None: + store = A2uiRendererCallsStore() + calls = store.get_for_session("session-1") + message, result = calls.call(catalog_id=CATALOG_ID, name="getLocation") + response = _renderer_response(message.call_renderer_function.function_call_id) + callback = a2ui_rpc_callback(renderer_calls=store.get) + + content = await callback(_callback_context(_content(_genai_part(response)))) + + assert content is not None + assert content.parts == [] + assert await result == "Paris" + + +@pytest.mark.asyncio +async def test_answers_calls_and_settles_responses_together() -> None: + store = A2uiRendererCallsStore() + message, result = store.get_for_session("session-1").call( + catalog_id=CATALOG_ID, name="getLocation" + ) + response = _renderer_response(message.call_renderer_function.function_call_id) + callback = a2ui_rpc_callback(catalogs=[CATALOG], renderer_calls=store.get) + + content = await callback( + _callback_context(_content(_genai_part(_call()), _genai_part(response))) + ) + + assert content is not None + assert _messages(content) == [_reply()] + assert await result == "Paris" + + +@pytest.mark.asyncio +async def test_response_to_an_unknown_call_goes_to_the_agent() -> None: + # Such as a response to a call the model wrote, which the inference + # formats compile into callRendererFunction without RendererFunctionCalls. + store = A2uiRendererCallsStore() + _, result = store.get_for_session("session-1").call( + catalog_id=CATALOG_ID, name="getLocation" + ) + callback = a2ui_rpc_callback(renderer_calls=store.get) + + content = await callback( + _callback_context(_content(_genai_part(_renderer_response("call_1")))) + ) + + assert content is None + assert not result.done() + result.cancel() + + +@pytest.mark.asyncio +async def test_mixed_responses_settle_the_pending_call_and_go_to_the_agent() -> None: + store = A2uiRendererCallsStore() + message, result = store.get_for_session("session-1").call( + catalog_id=CATALOG_ID, name="getLocation" + ) + pending = _renderer_response(message.call_renderer_function.function_call_id) + callback = a2ui_rpc_callback(renderer_calls=store.get) + + content = await callback( + _callback_context( + _content(_genai_part(pending), _genai_part(_renderer_response("call_1"))) + ) + ) + + assert content is None + assert await result == "Paris" + + +@pytest.mark.asyncio +async def test_pending_response_next_to_text_settles_the_call() -> None: + store = A2uiRendererCallsStore() + message, result = store.get_for_session("session-1").call( + catalog_id=CATALOG_ID, name="getLocation" + ) + pending = _renderer_response(message.call_renderer_function.function_call_id) + callback = a2ui_rpc_callback(renderer_calls=store.get) + + content = await callback( + _callback_context( + _content(genai_types.Part.from_text(text="hi"), _genai_part(pending)) + ) + ) + + assert content is None + assert await result == "Paris" + + +@pytest.mark.asyncio +async def test_pending_response_with_a_call_and_no_catalogs_settles_the_call() -> None: + store = A2uiRendererCallsStore() + message, result = store.get_for_session("session-1").call( + catalog_id=CATALOG_ID, name="getLocation" + ) + pending = _renderer_response(message.call_renderer_function.function_call_id) + callback = a2ui_rpc_callback(catalogs=lambda _ctx: None, renderer_calls=store.get) + + content = await callback( + _callback_context(_content(_genai_part(_call()), _genai_part(pending))) + ) + + assert content is None + assert await result == "Paris" + + +@pytest.mark.asyncio +async def test_a_settled_call_ignores_its_response_when_it_comes_again() -> None: + store = A2uiRendererCallsStore() + message, result = store.get_for_session("session-1").call( + catalog_id=CATALOG_ID, name="getLocation" + ) + call_id = message.call_renderer_function.function_call_id + callback = a2ui_rpc_callback(renderer_calls=store.get) + unknown = _genai_part(_renderer_response("call_1")) + + await callback( + _callback_context(_content(_genai_part(_renderer_response(call_id)), unknown)) + ) + again = await callback( + _callback_context( + _content(_genai_part(_renderer_response(call_id, "Rome")), unknown) + ) + ) + + assert again is None + assert await result == "Paris" + + +@pytest.mark.asyncio +async def test_malformed_response_fails_the_pending_call() -> None: + store = A2uiRendererCallsStore() + message, result = store.get_for_session("session-1").call( + catalog_id=CATALOG_ID, name="getLocation" + ) + malformed = { + "version": "1.0", + "rendererFunctionResponse": { + "functionCallId": message.call_renderer_function.function_call_id, + }, + } + callback = a2ui_rpc_callback(renderer_calls=store.get) + + content = await callback(_callback_context(_content(_genai_part(malformed)))) + + assert content is not None + assert content.parts == [] + with pytest.raises(A2uiRpcError) as excinfo: + await result + assert excinfo.value.code == "UNKNOWN_ERROR" + + +@pytest.mark.asyncio +async def test_calls_go_to_the_agent_without_catalogs() -> None: + callback = a2ui_rpc_callback(renderer_calls=A2uiRendererCallsStore().get) + + assert await callback(_callback_context(_content(_genai_part(_call())))) is None + + +@pytest.mark.asyncio +async def test_a2a_agent_executor_settles_a_renderer_function_call() -> None: + session_id = str(uuid.uuid4()) + store = A2uiRendererCallsStore() + message, result = store.get_for_session(session_id).call( + catalog_id=CATALOG_ID, name="getLocation" + ) + agent = _StubAgent( + name="agent", + before_agent_callback=a2ui_rpc_callback(renderer_calls=store.get), + ) + response = _renderer_response(message.call_renderer_function.function_call_id) + + events = await _execute(agent, [create_a2ui_part(response)], session_id) + + assert await result == "Paris" + assert _agent_status_parts(events) == [] + # With no reply to send, ADK's executor ends the task without a result, as + # for any turn whose agent sends nothing. + final = events[-1] + assert isinstance(final, TaskStatusUpdateEvent) + assert final.final + + +@pytest.mark.asyncio +async def test_interceptor_completes_an_rpc_only_task() -> None: + session_id = str(uuid.uuid4()) + store = A2uiRendererCallsStore() + message, result = store.get_for_session(session_id).call( + catalog_id=CATALOG_ID, name="getLocation" + ) + agent = _StubAgent( + name="agent", + before_agent_callback=a2ui_rpc_callback(renderer_calls=store.get), + ) + response = _renderer_response(message.call_renderer_function.function_call_id) + + events = await _execute( + agent, [create_a2ui_part(response)], session_id, [a2ui_rpc_interceptor()] + ) + + assert await result == "Paris" + assert _final_state(events) == "completed" + assert _agent_status_parts(events) == [] + + +@pytest.mark.asyncio +async def test_interceptor_leaves_a_task_with_a_result_unchanged() -> None: + agent = _StubAgent( + name="agent", before_agent_callback=a2ui_rpc_callback(catalogs=[CATALOG]) + ) + + events = await _execute( + agent, + [create_a2ui_part(_call())], + str(uuid.uuid4()), + [a2ui_rpc_interceptor()], + ) + + assert get_a2ui_messages(_agent_status_parts(events)) == [_reply()] + assert _final_state(events) == "completed" + + +@pytest.mark.asyncio +async def test_interceptor_leaves_other_requests_unchanged() -> None: + interceptor = a2ui_rpc_interceptor() + callback = a2ui_rpc_callback(renderer_calls=A2uiRendererCallsStore().get) + + # A text request whose agent sends nothing stays `working`, as ADK reports. + text_events = await _execute( + _SilentAgent(name="agent", before_agent_callback=callback), + [Part(root=TextPart(text="hi"))], + str(uuid.uuid4()), + [interceptor], + ) + assert _final_state(text_events) == "working" + + # So does a request that mixes text and an RPC message, which the agent + # handles. + mixed_events = await _execute( + _SilentAgent(name="agent", before_agent_callback=callback), + [Part(root=TextPart(text="hi")), create_a2ui_part(_renderer_response("c1"))], + str(uuid.uuid4()), + [interceptor], + ) + assert _final_state(mixed_events) == "working" + + # A normal turn with a reply completes as usual. + normal_events = await _execute( + _StubAgent(name="agent", before_agent_callback=callback), + [Part(root=TextPart(text="hi"))], + str(uuid.uuid4()), + [interceptor], + ) + assert _final_state(normal_events) == "completed" + assert [ + part.root.text + for part in _agent_status_parts(normal_events) + if isinstance(part.root, TextPart) + ] == [AGENT_TEXT] + + +@pytest.mark.asyncio +async def test_interceptor_leaves_a_failed_rpc_task_unchanged() -> None: + session_id = str(uuid.uuid4()) + + def failing_provider(ctx: Any) -> Any: + raise RuntimeError("no calls") + + agent = _StubAgent( + name="agent", + before_agent_callback=a2ui_rpc_callback(renderer_calls=failing_provider), + ) + + events = await _execute( + agent, + [create_a2ui_part(_renderer_response("c1"))], + session_id, + [a2ui_rpc_interceptor()], + ) + + assert _final_state(events) == "failed" + + +def _request_context(task_id: str, *parts: Part) -> MagicMock: + context = MagicMock(spec=RequestContext) + context.task_id = task_id + context.message = Message( + message_id=str(uuid.uuid4()), role=Role.user, parts=list(parts) + ) + return context + + +def _working_event(task_id: str) -> TaskStatusUpdateEvent: + return TaskStatusUpdateEvent( + task_id=task_id, + context_id="ctx", + status=TaskStatus(state=TaskState.working), + final=True, + ) + + +@pytest.mark.asyncio +async def test_interceptor_forgets_an_rpc_request_that_never_finished() -> None: + interceptor = a2ui_rpc_interceptor() + assert interceptor.before_agent is not None + assert interceptor.after_agent is not None + rpc_part = create_a2ui_part(_renderer_response("c1")) + + # The RPC-only request fails before `after_agent` runs. + await interceptor.before_agent(_request_context("t1", rpc_part)) + # A later request on the task that isn't RPC-only keeps its `working`. + await interceptor.before_agent( + _request_context("t1", Part(root=TextPart(text="hi"))) + ) + event = await interceptor.after_agent(MagicMock(), _working_event("t1")) + + assert event.status.state == TaskState.working + + +@pytest.mark.asyncio +async def test_interceptor_keeps_a_task_it_sees_again_past_older_ones() -> None: + interceptor = a2ui_rpc_interceptor() + assert interceptor.before_agent is not None + assert interceptor.after_agent is not None + rpc_part = create_a2ui_part(_renderer_response("c1")) + + await interceptor.before_agent(_request_context("again", rpc_part)) + # The tracker keeps 1024 tasks, so it is full after 1023 more. "again" + # comes once more before the next ones, so they drop older tasks instead. + for index in range(1030): + if index == 1023: + await interceptor.before_agent(_request_context("again", rpc_part)) + await interceptor.before_agent(_request_context(f"t{index}", rpc_part)) + event = await interceptor.after_agent(MagicMock(), _working_event("again")) + + assert event.status.state == TaskState.completed + + +@pytest.mark.asyncio +async def test_malformed_response_to_a_pending_call_logs_no_pass_on_warning( + caplog: pytest.LogCaptureFixture, +) -> None: + store = A2uiRendererCallsStore() + message, result = store.get_for_session("session-1").call( + catalog_id=CATALOG_ID, name="getLocation" + ) + malformed = { + "version": "1.0", + "rendererFunctionResponse": { + "functionCallId": message.call_renderer_function.function_call_id, + "secret": "hunter2", + }, + } + callback = a2ui_rpc_callback(renderer_calls=store.get) + + with caplog.at_level("WARNING"): + await callback(_callback_context(_content(_genai_part(malformed)))) + + assert "Passing an invalid A2UI message on to the agent" not in caplog.text + assert "hunter2" not in caplog.text + with pytest.raises(A2uiRpcError): + await result + + +@pytest.mark.asyncio +async def test_malformed_response_to_no_pending_call_warns_without_the_payload( + caplog: pytest.LogCaptureFixture, +) -> None: + malformed = { + "version": "1.0", + "rendererFunctionResponse": {"functionCallId": "c1", "secret": "hunter2"}, + } + callback = a2ui_rpc_callback(renderer_calls=A2uiRendererCallsStore().get) + + with caplog.at_level("WARNING"): + content = await callback(_callback_context(_content(_genai_part(malformed)))) + + assert content is None + assert "Passing an invalid A2UI message on to the agent" in caplog.text + assert "hunter2" not in caplog.text diff --git a/python/a2ui_adk/tests/test_lazy_a2a_exports.py b/python/a2ui_adk/tests/test_lazy_a2a_exports.py index bbe44523e0..986f45bf04 100644 --- a/python/a2ui_adk/tests/test_lazy_a2a_exports.py +++ b/python/a2ui_adk/tests/test_lazy_a2a_exports.py @@ -28,9 +28,11 @@ "A2uiRpcCallback", "A2uiSubagentMap", "CatalogsProvider", + "RendererFunctionCallsProvider", "SurfaceIdAlreadyExistsError", "a2a", "a2ui_rpc_callback", + "a2ui_rpc_interceptor", "exclude_a2ui_rpc_from_prompt", "orchestration", ] @@ -70,6 +72,7 @@ def test_base_exports_work_without_a2a(adk_without_a2a) -> None: assert adk_without_a2a.SendA2uiToClientToolset is not None assert adk_without_a2a.A2uiInstruction is not None assert adk_without_a2a.A2uiProcessorStore is not None + assert adk_without_a2a.A2uiRendererCallsStore is not None @pytest.mark.parametrize("name", A2A_EXPORTS) diff --git a/python/a2ui_adk/tests/test_renderer_calls_store.py b/python/a2ui_adk/tests/test_renderer_calls_store.py new file mode 100644 index 0000000000..679762ac48 --- /dev/null +++ b/python/a2ui_adk/tests/test_renderer_calls_store.py @@ -0,0 +1,113 @@ +# Copyright 2024 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for A2uiRendererCallsStore.""" + +from __future__ import annotations + +from unittest.mock import MagicMock + +from google.adk.agents.readonly_context import ReadonlyContext +import pytest + +from a2ui.adk import A2uiRendererCallsStore +from a2ui.core import A2uiRpcError + +CATALOG_ID = "https://a2ui.org/specification/v1_0/catalogs/basic/catalog.json" + + +def _ctx(session_id: str) -> MagicMock: + ctx = MagicMock(spec=ReadonlyContext) + ctx.session.id = session_id + return ctx + + +def test_store_keeps_one_instance_per_session() -> None: + store = A2uiRendererCallsStore() + + first = store.get(_ctx("s1")) + + assert store.get(_ctx("s1")) is first + assert store.get_for_session("s1") is first + assert store.get(_ctx("s2")) is not first + assert len(store) == 2 + assert "s1" in store + + +@pytest.mark.asyncio +async def test_store_passes_its_settings_to_each_instance() -> None: + store = A2uiRendererCallsStore(default_timeout=0.01, max_pending=1) + calls = store.get(_ctx("s1")) + + _, result = calls.call(catalog_id=CATALOG_ID, name="getLocation") + with pytest.raises(A2uiRpcError): + calls.call(catalog_id=CATALOG_ID, name="getLocation") + with pytest.raises(A2uiRpcError) as excinfo: + await result + assert excinfo.value.code == "TIMEOUT" + + +@pytest.mark.parametrize( + "kwargs", + [ + {"default_timeout": 0}, + {"default_timeout": -1.0}, + {"default_timeout": float("nan")}, + {"default_timeout": float("inf")}, + {"default_timeout": True}, + {"max_pending": 0}, + ], +) +def test_store_rejects_invalid_settings(kwargs: dict[str, float]) -> None: + with pytest.raises(ValueError): + A2uiRendererCallsStore(**kwargs) + + +@pytest.mark.asyncio +async def test_remove_disposes_the_session_instance() -> None: + store = A2uiRendererCallsStore() + calls = store.get(_ctx("s1")) + _, result = calls.call(catalog_id=CATALOG_ID, name="getLocation") + + store.remove("s1", reason="session ended") + + assert calls.disposed + assert "s1" not in store + with pytest.raises(A2uiRpcError, match="session ended"): + await result + # Removing an unknown session does nothing. + store.remove("unknown") + + +@pytest.mark.asyncio +async def test_dispose_disposes_every_instance() -> None: + store = A2uiRendererCallsStore() + first = store.get(_ctx("s1")) + second = store.get(_ctx("s2")) + + store.dispose() + + assert first.disposed and second.disposed + assert len(store) == 0 + + +def test_get_replaces_an_instance_disposed_outside_the_store() -> None: + store = A2uiRendererCallsStore() + calls = store.get(_ctx("s1")) + calls.dispose() + + replacement = store.get(_ctx("s1")) + + assert replacement is not calls + assert not replacement.disposed diff --git a/python/a2ui_agent/CHANGELOG.md b/python/a2ui_agent/CHANGELOG.md index dad7e07120..4972a69df2 100644 --- a/python/a2ui_agent/CHANGELOG.md +++ b/python/a2ui_agent/CHANGELOG.md @@ -1,5 +1,30 @@ ## Unreleased +- `RendererFunctionCalls.call` checks its arguments in a fixed order shared + with the TypeScript and Dart SDKs. An empty `function_call_id` is now + refused with `INVALID_FUNCTION_CALL` instead of replaced by a generated + one, and so are a `name` that isn't a UAX #31 identifier or starts with + `@`, and `args` that aren't a mapping with identifier keys and JSON values + (no NaN or infinity). `timeout` and `default_timeout` must be finite + positive numbers (not bools); only `default_timeout=None` turns the limit + off. ([#3939](https://github.com/a2ui-project/a2ui/pull/3939)) +- `RendererFunctionCalls` takes optional `catalogs`. With them, `call` raises + `UNKNOWN_FUNCTION` for a catalog they lack or a name that isn't one of its + renderer functions (agent functions included), and + `INVALID_FUNCTION_CALL` for a `rendererOnly` function or one that requires + user activation, before anything is registered. Two catalogs with one ID + raise `A2uiCatalogError`. + ([#3939](https://github.com/a2ui-project/a2ui/pull/3939)) +- Add `RendererFunctionCalls` to `a2ui.rpc`, which sends v1.0 + `callRendererFunction` messages for one conversation and settles each call's + future from the renderer's `rendererFunctionResponse`, with timeouts (30 + seconds by default), `max_pending` and `dispose()`. Its `resolve` method is + the `on_renderer_function_response` callback of + `RendererToAgentMessageProcessor`: it returns `True` when it settles (or, for + a malformed response, fails) a pending call and `False` otherwise, so the + processor declines responses to calls it didn't make. `is_pending` tells + whether a response answers one of its calls. `agent_development.md` describes it + ([#3535](https://github.com/a2ui-project/a2ui/issues/3535)). - **BREAKING CHANGE**: (v1_0) The `TextField` builder in `a2ui.builder.v1_0.catalogs.basic` requires `value`, following the basic catalog. Recommended migration: pass `value`, such as diff --git a/python/a2ui_agent/README.md b/python/a2ui_agent/README.md index aef2d9ce06..8b1bae0b75 100644 --- a/python/a2ui_agent/README.md +++ b/python/a2ui_agent/README.md @@ -71,8 +71,9 @@ The following directories contain the base protocol logic, parsing, and schema o ### Messages from the renderer (`src/a2ui/rpc`) - **`processor.py`**: `RendererToAgentMessageProcessor` processes the messages a renderer sends to the agent. It answers `callAgentFunction` by running the agent functions of the agent's catalogs, and passes `action`, `error` and `rendererFunctionResponse` to optional callbacks. +- **`renderer_function_calls.py`**: `RendererFunctionCalls` builds `callRendererFunction` messages for one conversation and settles their futures when the `rendererFunctionResponse` arrives. Its `resolve` method is the processor's `on_renderer_function_response` callback. `call` refuses an empty `functionCallId`, a name that isn't an identifier, and `args` that aren't an object of JSON values, and its timeouts must be finite positive numbers. Given the agent's `catalogs`, it also refuses a name that isn't one of the catalog's renderer functions, agent functions included (`UNKNOWN_FUNCTION`), a `rendererOnly` one, or one that requires user activation (`INVALID_FUNCTION_CALL`). -See [Agent Functions](agent_development.md#5-agent-functions) for how to use it. +See [Agent Functions](agent_development.md#5-agent-functions) for how to use them. ## Agent framework integrations diff --git a/python/a2ui_agent/agent_development.md b/python/a2ui_agent/agent_development.md index 2b702ffd85..71cb77847f 100644 --- a/python/a2ui_agent/agent_development.md +++ b/python/a2ui_agent/agent_development.md @@ -752,7 +752,9 @@ From v1.0, a renderer that evaluates a function it has no implementation for sends `callAgentFunction` to the agent, and the agent answers with `agentFunctionResponse`. `RendererToAgentMessageProcessor` in `a2ui.rpc` answers those calls, and is the entry point for the other messages a renderer -sends. It doesn't depend on an agent framework. +sends. In the other direction, the agent sends `callRendererFunction` with +`RendererFunctionCalls`, and the renderer answers with +`rendererFunctionResponse`. Neither depends on an agent framework. #### 5a. Declaring Agent Functions @@ -882,7 +884,101 @@ processor can serve many conversations, or you can build one per request from the active catalogs. Each catalog needs its own ID: two catalogs with one ID raise `A2uiCatalogError`. -#### 5c. What Renderers Need +#### 5c. Calling Renderer Functions + +`RendererFunctionCalls` sends `callRendererFunction` and matches up the +responses. Use one instance per conversation, and pass its `resolve` method as +the processor's `on_renderer_function_response` callback. `call` returns the +message to send and a future, without waiting: + +```python +from a2ui.rpc import RendererFunctionCalls, RendererToAgentMessageProcessor + +calls = RendererFunctionCalls(catalogs=request_processor.active_catalogs) +processor = RendererToAgentMessageProcessor( + request_processor.active_catalogs, + on_renderer_function_response=calls.resolve, +) + +message, result = calls.call( + catalog_id=CATALOG_ID, name="getLocation", args={}, surface_id="s1" +) +await send_to_renderer(message.model_dump(by_alias=True, exclude_unset=True)) +location = await result # settled when the processor gets the response +``` + +1. `call` always sets `catalogId`, which `callRendererFunction` requires, + and generates the `functionCallId` unless you pass one. Before anything is + built, registered or timed, it checks, in this order: + 1. `timeout`: anything but None or a finite positive number (a bool, NaN, + infinity, zero or a negative number) raises `ValueError`; + 2. a disposed instance raises `A2uiRpcError` with code `DISPOSED`; + 3. an empty `function_call_id` raises with code `INVALID_FUNCTION_CALL`; + 4. an ID that is already pending raises with code `DUPLICATE`; + 5. `max_pending` reached raises with code `TOO_MANY_PENDING`; + 6. an empty `name`, then an empty `catalog_id`, raises with code + `INVALID_FUNCTION_CALL`; + 7. a `name` that isn't a UAX #31 identifier or starts with `@` raises with + code `INVALID_FUNCTION_CALL`; + 8. `args` that aren't a mapping with identifier keys and JSON values (null, + booleans, strings, finite numbers, and lists and string-keyed mappings of + these; NaN and infinity aren't JSON) raise with code + `INVALID_FUNCTION_CALL`. Values aren't otherwise interpreted: a nested + `@call` or `@path` object is just an object, which the renderer checks; + 9. last, the `catalogs` checks (item 7 below). + + An empty `surface_id` counts as none: the message leaves `surfaceId` out, + and the renderer runs the function against an empty data model. + +2. The instance belongs to the event loop where it is first used. `call` must + run on that loop. `resolve` may be called from any thread. +3. A call fails with `A2uiRpcError` if the renderer answers with an error (with + the renderer's code) or no answer arrives in time (code `TIMEOUT`). The + timeout is 30 seconds by default; set `default_timeout` (None for no + limit, which only the instance can set) or pass `timeout` + to `call`. +4. Cancelling the future forgets the call. Responses to calls that are unknown, + already answered, timed out or cancelled are logged at debug level and + ignored. A malformed response is logged as a warning; if it names a pending + call, that call fails at once with code `UNKNOWN_ERROR`. +5. `dispose(reason)` fails every pending call (code `CANCELLED`) and makes later + `call`s raise. Dispose the instance when the conversation ends, or use it as + `async with RendererFunctionCalls() as calls:`. +6. `max_pending` makes `call` raise instead of letting a looping model pile up + calls that wait for an answer. +7. `catalogs`, such as `request_processor.active_catalogs`, makes `call` check + the target before anything is registered or sent: + - a `catalog_id` that names none of the catalogs, or a name that isn't one + of that catalog's renderer functions (an agent function included, since + it runs on the agent), raises `A2uiRpcError` with code + `UNKNOWN_FUNCTION`; + - a `rendererOnly` function (including one that leaves `allowedCallers` + out), which the agent may not call, raises with code + `INVALID_FUNCTION_CALL`; + - a function that requires user activation raises with code + `INVALID_FUNCTION_CALL`, since a call from the agent has no user + gesture. + + The arguments aren't checked against the function's schema. Two catalogs + with the same ID raise `A2uiCatalogError` when the instance is built. + +8. `resolve` returns `True` when the response named a pending call and + `False` for an unknown, late or duplicate one. As the processor's + callback, it therefore declines responses it doesn't own, and + `process_message` reports them as not handled. That matters because the + inference formats also compile a function call the model writes on its + own, such as `getLocation()`, into a `callRendererFunction` + message that `RendererFunctionCalls` never saw: pass an unhandled response + on to the model's turn. `is_pending(function_call_id)` checks a response + without settling it. + +The pending calls live in memory, so this suits a server that keeps a +conversation on one process, such as one holding a streaming connection. A +stateless server whose next request may reach another replica handles +`rendererFunctionResponse` in its own `on_renderer_function_response` callback +instead, for example by looking up saved state by `functionCallId`. + +#### 5d. What Renderers Need The renderer's catalog doesn't list the agent's functions, so how it treats a payload that calls one depends on its validation settings: @@ -899,7 +995,7 @@ payload that calls one depends on its validation settings: the agent's catalog document: `catalog.to_json()` writes agent functions like any other function, without the handler. -#### 5d. With A2A and ADK +#### 5e. With A2A and ADK Over A2A, each A2UI message is one `DataPart` of the A2A message. `a2ui-a2a` reads them with `get_a2ui_messages(message)` and wraps the @@ -920,8 +1016,15 @@ agent = LlmAgent(..., before_agent_callback=a2ui_rpc_callback(catalogs=catalogs) ``` `catalogs` can also be a provider that returns the catalogs per context, such -as the session's `A2uiRequestProcessor.active_catalogs`. The calls and their -replies stay in the session's events but not in the model's prompt: -`SendA2uiToClientToolset` removes them, and agents without it pass -`exclude_a2ui_rpc_from_prompt` as their `before_model_callback`. See the +as the session's `A2uiRequestProcessor.active_catalogs`. The agent function +calls and their replies stay in the session's events but not in the model's +prompt: `SendA2uiToClientToolset` removes them, and agents without it pass +`exclude_a2ui_rpc_from_prompt` as their `before_model_callback`. + +To settle the agent's calls to renderer functions too, pass +`renderer_calls=A2uiRendererCallsStore().get`, which keeps one +`RendererFunctionCalls` per session, and add `a2ui_rpc_interceptor()` to the +executor's `execute_interceptors`, so a task whose request held only responses +ends `completed`. Only responses to pending calls are settled; any other +response goes to the agent's turn. See the [`a2ui-adk` README](../a2ui_adk/README.md) for details. diff --git a/python/a2ui_agent/src/a2ui/rpc/__init__.py b/python/a2ui_agent/src/a2ui/rpc/__init__.py index 222c8d26c0..edebab5e1c 100644 --- a/python/a2ui_agent/src/a2ui/rpc/__init__.py +++ b/python/a2ui_agent/src/a2ui/rpc/__init__.py @@ -17,6 +17,9 @@ `RendererToAgentMessageProcessor` answers the renderer's `callAgentFunction` messages by running the catalogs' agent functions, and passes `action`, `error` and `rendererFunctionResponse` messages to optional callbacks. +`RendererFunctionCalls` sends `callRendererFunction` messages and matches up +the renderer's responses; its `resolve` method is the processor's +`on_renderer_function_response` callback. The v1.0 message classes it takes and returns are re-exported here. """ @@ -38,6 +41,11 @@ RendererToAgentMessageInput, RendererToAgentMessageProcessor, ) +from .renderer_function_calls import ( + DEFAULT_RENDERER_CALL_TIMEOUT, + RendererFunctionCalls, + RendererFunctionResponseInput, +) __all__ = [ "A2uiRendererActionMessage", @@ -45,9 +53,12 @@ "ActionCallback", "AgentFunctionResponseMessage", "CallAgentFunctionMessage", + "DEFAULT_RENDERER_CALL_TIMEOUT", "ErrorCallback", "ProcessResult", + "RendererFunctionCalls", "RendererFunctionResponseCallback", + "RendererFunctionResponseInput", "RendererFunctionResponseMessage", "RendererToAgentMessage", "RendererToAgentMessageInput", diff --git a/python/a2ui_agent/src/a2ui/rpc/renderer_function_calls.py b/python/a2ui_agent/src/a2ui/rpc/renderer_function_calls.py new file mode 100644 index 0000000000..b095bee85d --- /dev/null +++ b/python/a2ui_agent/src/a2ui/rpc/renderer_function_calls.py @@ -0,0 +1,622 @@ +# Copyright 2024 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Calls renderer functions from the agent and matches up their responses.""" + +from __future__ import annotations + +import asyncio +from collections.abc import Callable, Iterable, Mapping +from dataclasses import dataclass +import logging +import math +from types import TracebackType +from typing import Any +import uuid + +from pydantic import ValidationError + +from a2ui.core import ( + A2uiCatalogError, + A2uiRpcError, + CatalogApi, + RpcErrorCode, + is_valid_uax31_identifier, +) +from a2ui.core.schema.v1_0 import ( + CallRendererFunction, + CallRendererFunctionMessage, + FunctionCall, + RendererFunctionResponseMessage, +) + +logger = logging.getLogger(__name__) + +DEFAULT_RENDERER_CALL_TIMEOUT = 30.0 +"""Seconds a renderer function call waits for its response by default.""" + +RendererFunctionResponseInput = RendererFunctionResponseMessage | Mapping[str, Any] +"""A `rendererFunctionResponse` message, as a schema model or as parsed JSON.""" + + +@dataclass +class _PendingCall: + name: str + future: asyncio.Future[Any] + timer: asyncio.TimerHandle | None + + +class RendererFunctionCalls: + """The agent's pending `callRendererFunction` calls in one conversation. + + `call` builds the message to send and returns it with a future, without + waiting; `resolve` settles the future when the renderer's + `rendererFunctionResponse` arrives: + + ```python + message, result = calls.call(catalog_id=CATALOG_ID, name="getLocation") + await send_to_renderer(message) + value = await result + ``` + + An instance belongs to the event loop it is first used on. `call` and + `dispose` must run on that loop; `resolve` may be called from any thread + and hands the work to the loop. A call fails with `A2uiRpcError` when it + times out (code `TIMEOUT`), when the renderer answers with an error (the + renderer's code), or when the instance is disposed (code `CANCELLED`). + Responses to calls that are unknown, already settled, timed out or + cancelled are logged at debug level and ignored. + """ + + def __init__( + self, + *, + default_timeout: float | None = DEFAULT_RENDERER_CALL_TIMEOUT, + max_pending: int | None = None, + catalogs: Iterable[CatalogApi] | None = None, + ) -> None: + """Initializes the call table. + + Args: + default_timeout: Seconds a call waits for its response unless + `call` sets another timeout. None waits without a limit. + max_pending: The most calls that may wait for a response at once. + `call` raises beyond it. None, the default, sets no limit. + catalogs: The agent's catalogs, such as + `A2uiRequestProcessor.active_catalogs`. With them, `call` + checks that the catalog has a renderer function (not an agent + function) of that name and that the agent may call it before + anything is registered. None, the default, checks + nothing. + + Raises: + ValueError: If `default_timeout` is not a finite positive number + or None, or `max_pending` is less than 1. + A2uiCatalogError: If two catalogs have the same ID. + """ + if default_timeout is not None: + _check_timeout("default_timeout", default_timeout) + if max_pending is not None and max_pending < 1: + raise ValueError("max_pending must be at least 1.") + self._default_timeout = default_timeout + self._max_pending = max_pending + self._catalogs: dict[str, CatalogApi] | None = None + if catalogs is not None: + self._catalogs = {} + for catalog in catalogs: + if catalog.catalog_id in self._catalogs: + raise A2uiCatalogError( + f"Duplicate catalog ID: '{catalog.catalog_id}'. Pass each" + " catalog once." + ) + self._catalogs[catalog.catalog_id] = catalog + self._pending: dict[str, _PendingCall] = {} + self._loop: asyncio.AbstractEventLoop | None = None + self._disposed = False + + @property + def disposed(self) -> bool: + """Whether `dispose` has been called.""" + return self._disposed + + @property + def pending_count(self) -> int: + """The number of calls waiting for a response.""" + return len(self._pending) + + def call( + self, + *, + catalog_id: str, + name: str, + args: Mapping[str, Any] | None = None, + surface_id: str | None = None, + timeout: float | None = None, + function_call_id: str | None = None, + ) -> tuple[CallRendererFunctionMessage, asyncio.Future[Any]]: + """Registers a renderer function call and returns its message and future. + + Send the message to the renderer, then await the future for the + function's value. Cancelling the future forgets the call. Serialize + the message with `model_dump(by_alias=True, exclude_unset=True)`, as + every A2UI RPC message. + + Args: + catalog_id: The catalog that defines the function, which + `callRendererFunction` requires. + name: The function's name. + args: The function's arguments. + surface_id: The surface whose data model the call runs against. + None, or an empty string, leaves `surfaceId` out of the + message, so the renderer runs it against an empty data model. + timeout: Seconds to wait for the response, in place of the + default timeout. None uses the default, which may be no + limit; a call can't turn the limit off on its own. + function_call_id: The call's ID. None generates one; an empty + string is refused. + + Returns: + The `callRendererFunction` message and the future of its value. + + The checks run in this order, each before anything is built, + registered or timed: the timeout, disposal, the ID, a duplicate ID, + `max_pending`, an empty `name`, an empty `catalog_id`, the `name` + as an identifier, the `args`, and last the `catalogs`. + + Raises: + ValueError: If `timeout` is not None or a finite positive number + (a bool, NaN, infinity, zero or a negative number). + A2uiRpcError: If the instance is disposed (code `DISPOSED`), the + ID is already pending (code `DUPLICATE`), `max_pending` calls + are pending (code `TOO_MANY_PENDING`); with code + `INVALID_FUNCTION_CALL` if `function_call_id` is an empty + string, `name` or `catalog_id` is empty, `name` isn't a + UAX #31 identifier or starts with `@`, `args` isn't a mapping + with identifier keys and JSON values (null, booleans, + strings, finite numbers, and lists and string-keyed mappings + of these), or the message fails the schema. With `catalogs`, + also if `catalog_id` names none of them or that catalog has + no renderer function `name`, such as when `name` is an agent + function (code `UNKNOWN_FUNCTION`), or the function is + `rendererOnly` (also when `allowedCallers` is unset) or + requires user activation (code `INVALID_FUNCTION_CALL`). + RuntimeError: If no event loop is running, or the instance + belongs to another loop. + """ + if timeout is not None: + _check_timeout("timeout", timeout) + if self._disposed: + raise A2uiRpcError( + "RendererFunctionCalls has been disposed.", code=RpcErrorCode.DISPOSED + ) + loop = self._bind_loop() + if function_call_id is not None and not function_call_id: + raise A2uiRpcError( + "functionCallId must not be empty.", + code=RpcErrorCode.INVALID_FUNCTION_CALL, + ) + call_id = function_call_id or f"call-{uuid.uuid4().hex}" + if call_id in self._pending: + raise A2uiRpcError( + f"A call with functionCallId '{call_id}' is already pending.", + code=RpcErrorCode.DUPLICATE, + function_call_id=call_id, + ) + if self._max_pending is not None and len(self._pending) >= self._max_pending: + raise A2uiRpcError( + f"{len(self._pending)} renderer function calls are already" + " pending, the most allowed.", + code=RpcErrorCode.TOO_MANY_PENDING, + function_call_id=call_id, + ) + if not name: + raise A2uiRpcError( + "The call needs a function name.", + code=RpcErrorCode.INVALID_FUNCTION_CALL, + function_call_id=call_id, + ) + if not catalog_id: + raise A2uiRpcError( + f"The call to '{name}' needs a catalogId.", + code=RpcErrorCode.INVALID_FUNCTION_CALL, + function_call_id=call_id, + ) + if name.startswith("@") or not is_valid_uax31_identifier(name): + raise A2uiRpcError( + f"The function name '{name}' must be a UAX #31 identifier that" + " doesn't start with '@'.", + code=RpcErrorCode.INVALID_FUNCTION_CALL, + function_call_id=call_id, + ) + if args is not None: + problem = _args_problem(args) + if problem: + raise A2uiRpcError( + f"Invalid arguments for '{name}': {problem}", + code=RpcErrorCode.INVALID_FUNCTION_CALL, + function_call_id=call_id, + ) + self._check_target(catalog_id, name, call_id) + try: + # Left unset, not None or empty, without a surface, so that + # `model_dump(exclude_unset=True)` omits it: the schema types + # `surfaceId` as a string and rejects null, and a renderer + # rejects an empty one. + surface = {"surfaceId": surface_id} if surface_id else {} + message = CallRendererFunctionMessage( + version="1.0", + callRendererFunction=CallRendererFunction( + **surface, + functionCallId=call_id, + callFunction=FunctionCall.model_validate({ + "@call": name, + "catalogId": catalog_id, + "args": dict(args or {}), + }), + ), + ) + except ValidationError as e: + raise A2uiRpcError( + f"Invalid callRendererFunction message for '{name}': {e}", + code=RpcErrorCode.INVALID_FUNCTION_CALL, + function_call_id=call_id, + ) from e + + future: asyncio.Future[Any] = loop.create_future() + effective_timeout = timeout if timeout is not None else self._default_timeout + timer = ( + loop.call_later( + effective_timeout, self._time_out, call_id, effective_timeout + ) + if effective_timeout is not None + else None + ) + self._pending[call_id] = _PendingCall(name=name, future=future, timer=timer) + future.add_done_callback(lambda _: self._forget(call_id, future)) + return message, future + + def resolve(self, message: RendererFunctionResponseInput) -> bool: + """Settles the call that a `rendererFunctionResponse` answers. + + The response comes from the network, so this never raises for it: a + malformed response is logged as a warning, and a response that + matches no pending call is logged at debug level. A malformed response + whose `functionCallId` names a pending call fails that call at once + (code `UNKNOWN_ERROR`), rather than leaving it to time out. It may be + called from any thread. + + Pass it as `on_renderer_function_response` of a + `RendererToAgentMessageProcessor`, which validates the message first. + It then declines a response to a call it didn't make, or one that is + late or a duplicate, so the processor reports it as not handled and + the response can go on to the model's turn. + + Args: + message: The `rendererFunctionResponse` message, as a schema model + or as parsed JSON. + + Returns: + True if the response named a pending call, which it settles or, + when malformed, fails. False if no call with its `functionCallId` + is pending (unknown, late or duplicate) or it has none. Called + from another thread, it reports whether the call was pending at + the time; the instance's loop settles it. + """ + try: + response = ( + message + if isinstance(message, RendererFunctionResponseMessage) + else RendererFunctionResponseMessage.model_validate(message) + ) + except ValidationError as e: + # The response comes from the network and the error may quote it, + # so the details are logged only at debug level. + logger.warning( + "Ignoring a malformed rendererFunctionResponse: %s", type(e).__name__ + ) + logger.debug("Malformed rendererFunctionResponse error: %s", e) + call_id = _raw_call_id(message) + if not call_id: + return False + pending = self.is_pending(call_id) + self._on_loop( + call_id, + self._fail_malformed, + call_id, + f"Malformed rendererFunctionResponse: {e}", + ) + return pending + call_id = response.renderer_function_response.function_call_id + pending = self.is_pending(call_id) + self._on_loop(call_id, self._settle, response) + return pending + + def is_pending(self, function_call_id: str) -> bool: + """Whether a call with this ID is waiting for its response. + + An agent framework can use it to take only the responses to its own + calls, and pass any other `rendererFunctionResponse`, such as one + answering a call the model wrote, on to the model's turn. + + Args: + function_call_id: The `functionCallId` of a response. + + Returns: + True if `resolve` would settle a call with this response. False + once the instance is disposed, even before `dispose` has failed + the pending calls on the instance's loop. + """ + if self._disposed: + return False + pending = self._pending.get(function_call_id) + return pending is not None and not pending.future.done() + + def dispose(self, reason: str = "RendererFunctionCalls was disposed.") -> None: + """Fails every pending call and refuses later ones. + + Pending futures fail with `A2uiRpcError` (code `CANCELLED`) carrying + `reason`. Later `call`s raise, and later responses are ignored. + + Args: + reason: Why the calls were dropped, such as "the session ended". + """ + if self._disposed: + return + self._disposed = True + loop = self._loop + if loop is None or _running_loop() is loop or loop.is_closed(): + self._fail_all(reason) + return + try: + loop.call_soon_threadsafe(self._fail_all, reason) + except RuntimeError: + # The loop closed after the check above. + self._fail_all(reason) + + async def __aenter__(self) -> RendererFunctionCalls: + return self + + async def __aexit__( + self, + exc_type: type[BaseException] | None, + exc: BaseException | None, + tb: TracebackType | None, + ) -> None: + self.dispose("RendererFunctionCalls was closed.") + + def _check_target(self, catalog_id: str, name: str, call_id: str) -> None: + """Checks that the agent may call the function, if catalogs were given.""" + if self._catalogs is None: + return + catalog = self._catalogs.get(catalog_id) + if catalog is None: + raise A2uiRpcError( + f"The call to '{name}' names catalog '{catalog_id}', which isn't" + " one of the agent's catalogs.", + code=RpcErrorCode.UNKNOWN_FUNCTION, + function_call_id=call_id, + ) + # Agent functions run on the agent, so the renderer can't run them. + function = catalog.renderer_functions.get(name) + if function is None: + raise A2uiRpcError( + f"Catalog '{catalog_id}' has no renderer function '{name}'.", + code=RpcErrorCode.UNKNOWN_FUNCTION, + function_call_id=call_id, + ) + if function.allowed_callers in (None, "rendererOnly"): + raise A2uiRpcError( + f"Function '{name}' is rendererOnly, so the agent may not call it.", + code=RpcErrorCode.INVALID_FUNCTION_CALL, + function_call_id=call_id, + ) + if function.requires_user_activation: + raise A2uiRpcError( + f"Function '{name}' requires user activation, and a call from the" + " agent has no user gesture.", + code=RpcErrorCode.INVALID_FUNCTION_CALL, + function_call_id=call_id, + ) + + def _bind_loop(self) -> asyncio.AbstractEventLoop: + loop = asyncio.get_running_loop() + if self._loop is None: + self._loop = loop + elif self._loop is not loop: + raise RuntimeError( + "RendererFunctionCalls belongs to another event loop; use one" + " instance per conversation, on one loop." + ) + return loop + + def _on_loop(self, call_id: str, callback: Callable[..., None], *args: Any) -> None: + """Runs `callback(*args)` on the instance's loop, now if this is it.""" + loop = self._loop + if loop is None: + logger.debug( + "Ignoring rendererFunctionResponse %s: no call was made.", call_id + ) + return + if _running_loop() is loop: + callback(*args) + return + try: + loop.call_soon_threadsafe(callback, *args) + except RuntimeError: + logger.debug( + "Ignoring rendererFunctionResponse %s: the event loop is closed.", + call_id, + ) + + def _fail_malformed(self, call_id: str, reason: str) -> None: + if self._disposed: + # `dispose` cancels every pending call, even if it ran on another + # thread and its `_fail_all` hasn't run on the loop yet. + return + pending = self._pending.pop(call_id, None) + if pending is None or pending.future.done(): + return + if pending.timer is not None: + pending.timer.cancel() + pending.future.set_exception( + A2uiRpcError( + reason, code=RpcErrorCode.UNKNOWN_ERROR, function_call_id=call_id + ) + ) + + def _settle(self, response: RendererFunctionResponseMessage) -> None: + body = response.renderer_function_response + call_id = body.function_call_id + if self._disposed: + logger.debug( + "Ignoring rendererFunctionResponse %s: the calls were disposed.", + call_id, + ) + return + pending = self._pending.pop(call_id, None) + if pending is None or pending.future.done(): + logger.debug( + "Ignoring rendererFunctionResponse %s: no pending call has that" + " ID (unknown, late or duplicate).", + call_id, + ) + return + if pending.timer is not None: + pending.timer.cancel() + if body.error is not None: + pending.future.set_exception( + A2uiRpcError( + f"Renderer function '{pending.name}' failed" + f" ({body.error.code}): {body.error.message}", + code=body.error.code, + function_call_id=call_id, + ) + ) + else: + pending.future.set_result(body.value) + + def _time_out(self, call_id: str, timeout: float) -> None: + if self._disposed: + # `_fail_all` cancels the call. + return + pending = self._pending.pop(call_id, None) + if pending is None or pending.future.done(): + return + pending.future.set_exception( + A2uiRpcError( + f"Renderer function '{pending.name}' timed out after {timeout}s.", + code=RpcErrorCode.TIMEOUT, + function_call_id=call_id, + ) + ) + + def _forget(self, call_id: str, future: asyncio.Future[Any]) -> None: + """Drops a settled or cancelled call and its timer.""" + pending = self._pending.get(call_id) + if pending is None or pending.future is not future: + return + del self._pending[call_id] + if pending.timer is not None: + pending.timer.cancel() + + def _fail_all(self, reason: str) -> None: + pending_calls = list(self._pending.items()) + self._pending.clear() + for call_id, pending in pending_calls: + if pending.timer is not None: + pending.timer.cancel() + if pending.future.done(): + continue + try: + pending.future.set_exception( + A2uiRpcError( + reason, code=RpcErrorCode.CANCELLED, function_call_id=call_id + ) + ) + except RuntimeError: + # Failing a future schedules its callbacks on its loop, which + # raises once the loop is closed. Nothing can await the future + # then, so the call is just dropped. + logger.debug( + "Dropping renderer function call %s: the event loop is closed.", + call_id, + ) + + +def _check_timeout(label: str, timeout: Any) -> None: + """Raises ValueError unless `timeout` is a finite positive number.""" + if ( + isinstance(timeout, bool) + or not isinstance(timeout, (int, float)) + or not math.isfinite(timeout) + or timeout <= 0 + ): + raise ValueError(f"{label} must be a finite positive number, got {timeout!r}.") + + +def _args_problem(args: Any) -> str | None: + """What makes `args` unfit to send, or None if nothing does.""" + if not isinstance(args, Mapping): + return f"they must be an object, got {type(args).__name__}." + for key in args: + if not isinstance(key, str) or not is_valid_uax31_identifier(key): + return f"the key {key!r} is not a UAX #31 identifier." + for key, value in args.items(): + problem = _json_value_problem(value, key) + if problem: + return problem + return None + + +def _json_value_problem(value: Any, path: str) -> str | None: + """What makes `value` (at `path`) not a JSON value, or None.""" + if value is None or isinstance(value, (bool, str)): + return None + if isinstance(value, (int, float)): + if isinstance(value, float) and not math.isfinite(value): + return f"'{path}' is {value!r}, which isn't a JSON number." + return None + if isinstance(value, (list, tuple)): + for index, item in enumerate(value): + problem = _json_value_problem(item, f"{path}[{index}]") + if problem: + return problem + return None + if isinstance(value, Mapping): + for key, item in value.items(): + if not isinstance(key, str): + return f"'{path}' has the key {key!r}, which isn't a string." + problem = _json_value_problem(item, f"{path}.{key}") + if problem: + return problem + return None + return f"'{path}' is a {type(value).__name__}, which isn't a JSON value." + + +def _raw_call_id(message: Any) -> str | None: + """The `functionCallId` of a response that failed its schema, if it has one.""" + if not isinstance(message, Mapping): + return None + body = message.get("rendererFunctionResponse") + if not isinstance(body, Mapping): + return None + call_id = body.get("functionCallId") + return call_id if isinstance(call_id, str) and call_id else None + + +def _running_loop() -> asyncio.AbstractEventLoop | None: + try: + return asyncio.get_running_loop() + except RuntimeError: + return None diff --git a/python/a2ui_agent/tests/conformance/test_renderer_function_calls_conformance.py b/python/a2ui_agent/tests/conformance/test_renderer_function_calls_conformance.py new file mode 100644 index 0000000000..99eff8cb7a --- /dev/null +++ b/python/a2ui_agent/tests/conformance/test_renderer_function_calls_conformance.py @@ -0,0 +1,258 @@ +# Copyright 2024 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Runs the shared `conformance/agent/renderer_function_calls.yaml` suite. + +The cases run on `_FakeClockLoop`, an event loop whose clock moves only when a +case's `advance_ms` step moves it, so their timeouts don't depend on real +time. Responses go through a `RendererToAgentMessageProcessor` whose +`on_renderer_function_response` callback is `RendererFunctionCalls.resolve`. +""" + +from __future__ import annotations + +import asyncio +from collections.abc import Mapping +import selectors +from typing import Any + +import pytest + +from a2ui.core import A2uiCatalogError, A2uiRpcError, Catalog, CatalogApi +from a2ui.core.catalog import FunctionApi +from a2ui.core.schema import ProtocolVersion +from a2ui.rpc import RendererFunctionCalls, RendererToAgentMessageProcessor + +from .conformance_helpers import load_conformance_yaml + +SUITE = "agent/renderer_function_calls.yaml" + +CASES = [pytest.param(case, id=case["name"]) for case in load_conformance_yaml(SUITE)] + + +def test_suite_is_not_empty() -> None: + assert CASES + + +CATEGORY_TO_EXCEPTION: dict[str, type[Exception]] = {"CatalogError": A2uiCatalogError} + + +def _assert_build_error_matches( + error: BaseException, expected: Mapping[str, Any] +) -> None: + """Checks an error against a `RendererCallsBuildError`. + + `category` is required, and `message` is a literal substring of the + error's text. + """ + assert isinstance(expected, Mapping), expected + unknown = set(expected) - {"category", "message"} + assert not unknown, f"Unhandled expect_error keys: {sorted(unknown)}" + assert isinstance(error, CATEGORY_TO_EXCEPTION[expected["category"]]), error + if "message" in expected: + assert expected["message"] in str(error), error + + +def _assert_call_error_matches( + error: A2uiRpcError, expected: Mapping[str, Any] +) -> None: + """Checks a refused call's error against a `RendererCallError`. + + `code` is required, and `message` is a literal substring of the error's + text. + """ + assert isinstance(expected, Mapping), expected + unknown = set(expected) - {"code", "message"} + assert not unknown, f"Unhandled expect_error keys: {sorted(unknown)}" + assert error.code == expected["code"], error + if "message" in expected: + assert expected["message"] in str(error), error + + +class _FakeClockLoop(asyncio.SelectorEventLoop): + """An event loop whose clock moves only when `advance` moves it.""" + + def __init__(self) -> None: + # Set before the base constructor, which may already read the clock. + self._now_ms = 0 + super().__init__(selectors.DefaultSelector()) + + def time(self) -> float: + return self._now_ms / 1000 + + def advance(self, ms: int) -> None: + self._now_ms += ms + + +async def _let_pending_work_run() -> None: + for _ in range(5): + await asyncio.sleep(0) + + +def _call_state(future: asyncio.Future[Any]) -> Any: + """A call's state in the suite's vocabulary.""" + if not future.done(): + return "pending" + error = future.exception() + if error is None: + return {"value": future.result()} + assert isinstance(error, A2uiRpcError), error + return {"error": {"code": error.code, "message": str(error)}} + + +def _assert_call_state(label: str, actual: Any, expected: Any) -> None: + if expected == "pending" or actual == "pending": + assert actual == expected, f"call {label}: {actual}" + return + if "value" in expected: + assert actual == expected, f"call {label}: {actual}" + return + assert set(actual) == {"error"}, f"call {label}: {actual}" + error = actual["error"] + assert set(error) == {"code", "message"}, f"call {label}: {error}" + assert ( + isinstance(error["message"], str) and error["message"] + ), f"call {label}: {error}" + assert error["code"] == expected["error"]["code"], f"call {label}" + if "message" in expected["error"]: + assert expected["error"]["message"] in error["message"] + + +def _catalog(entry: Mapping[str, Any]) -> CatalogApi: + """Builds a `RendererCallCatalog` entry in code, as the suite requires.""" + functions = [] + for name, spec in entry["functions"].items(): + kwargs: dict[str, Any] = {"allowed_callers": spec.get("allowedCallers")} + if "requiresUserActivation" in spec: + kwargs["requires_user_activation"] = spec["requiresUserActivation"] + functions.append( + FunctionApi(name, return_type="any", schema={"type": "object"}, **kwargs) + ) + return Catalog(entry["catalogId"], ProtocolVersion.V1_0, functions=functions) + + +async def _run_calls_on_loop(case: Mapping[str, Any], loop: _FakeClockLoop) -> None: + args = case.get("args", {}) + options: dict[str, Any] = {} + if "catalogs" in args: + options["catalogs"] = [_catalog(entry) for entry in args["catalogs"]] + if "expect_error" in case: + with pytest.raises(Exception) as excinfo: + RendererFunctionCalls(**options) + _assert_build_error_matches(excinfo.value, case["expect_error"]) + return + if "default_timeout_ms" in args: + timeout_ms = args["default_timeout_ms"] + options["default_timeout"] = None if timeout_ms is None else timeout_ms / 1000 + if "max_pending" in args: + options["max_pending"] = args["max_pending"] + calls = RendererFunctionCalls(**options) + # Responses reach the calls the way an agent receives them: through the + # processor, with `resolve` as its callback. + processor = RendererToAgentMessageProcessor( + [], on_renderer_function_response=calls.resolve + ) + futures: dict[str, asyncio.Future[Any]] = {} + call_ids: dict[str, str] = {} + + for index, step in enumerate(case["steps"]): + where = f"step {index}" + if "call" in step: + spec = step["call"] + kwargs: dict[str, Any] = { + "catalog_id": spec["catalogId"], + "name": spec["name"], + } + if "args" in spec: + kwargs["args"] = spec["args"] + if "surfaceId" in spec: + kwargs["surface_id"] = spec["surfaceId"] + if "timeout_ms" in spec: + kwargs["timeout"] = spec["timeout_ms"] / 1000 + if "functionCallId" in spec: + kwargs["function_call_id"] = spec["functionCallId"] + if "expect_error" in step: + with pytest.raises(A2uiRpcError) as excinfo: + calls.call(**kwargs) + _assert_call_error_matches(excinfo.value, step["expect_error"]) + else: + message, future = calls.call(**kwargs) + dumped = message.model_dump(by_alias=True, exclude_unset=True) + body = dumped["callRendererFunction"] + call_id = body.pop("functionCallId") + assert isinstance(call_id, str) and call_id, where + assert call_id not in call_ids.values(), f"{where}: reused {call_id}" + call_ids[spec["label"]] = call_id + futures[spec["label"]] = future + if "expect_message" in step: + assert dumped == step["expect_message"], where + elif "respond" in step: + spec = step["respond"] + call_id = ( + call_ids[spec["call"]] if "call" in spec else spec["functionCallId"] + ) + response: dict[str, Any] = {"functionCallId": call_id} + if "error" in spec: + response["error"] = spec["error"] + else: + response["value"] = spec["value"] + owned = calls.is_pending(call_id) + result = await processor.process_message( + {"version": "1.0", "rendererFunctionResponse": response} + ) + # `resolve` declines a response to a call that isn't pending. + assert result.handled == owned and not result.replies, where + elif "advance_ms" in step: + loop.advance(step["advance_ms"]) + elif "dispose" in step: + reason = step["dispose"].get("reason") + if reason is None: + calls.dispose() + else: + calls.dispose(reason) + else: + raise ValueError(f"{where}: unknown step {step}") + + await _let_pending_work_run() + for label, expected in step.get("expect_calls", {}).items(): + _assert_call_state(label, _call_state(futures[label]), expected) + if "expect_pending_count" in step: + assert calls.pending_count == step["expect_pending_count"], where + + calls.dispose() + await _let_pending_work_run() + # Reads each failure, so asyncio doesn't log "exception was never + # retrieved". `dispose` settles every pending call, but a future that is + # somehow still pending is skipped rather than raising InvalidStateError. + for future in futures.values(): + if future.done() and not future.cancelled(): + future.exception() + + +def _run_calls(case: Mapping[str, Any]) -> None: + loop = _FakeClockLoop() + asyncio.set_event_loop(loop) + try: + loop.run_until_complete(_run_calls_on_loop(case, loop)) + finally: + loop.close() + asyncio.set_event_loop(None) + + +@pytest.mark.parametrize("case", CASES) +def test_renderer_function_calls_conformance(case: Mapping[str, Any]) -> None: + action = case["action"] + if action != "call_renderer_function": + raise ValueError(f"Unhandled conformance action: {action}") + _run_calls(case) diff --git a/python/a2ui_agent/tests/rpc/test_renderer_function_calls.py b/python/a2ui_agent/tests/rpc/test_renderer_function_calls.py new file mode 100644 index 0000000000..dbd324a928 --- /dev/null +++ b/python/a2ui_agent/tests/rpc/test_renderer_function_calls.py @@ -0,0 +1,996 @@ +# Copyright 2024 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for RendererFunctionCalls, the agent's `callRendererFunction` table.""" + +from __future__ import annotations + +import asyncio +import logging +import threading +from typing import Any + +import pytest + +from a2ui.core import A2uiCatalogError, A2uiRpcError, AgentFunction, Catalog +from a2ui.core.catalog import FunctionApi +from a2ui.core.schema import ProtocolVersion +from a2ui.core.schema.v1_0 import FunctionResponse, RendererFunctionResponseMessage +from a2ui.rpc import ( + DEFAULT_RENDERER_CALL_TIMEOUT, + RendererFunctionCalls, + RendererToAgentMessageProcessor, +) + +CATALOG = "https://example.com/device/v1/catalog.json" + + +def _value(call_id: str, value: Any) -> dict[str, Any]: + return { + "version": "1.0", + "rendererFunctionResponse": {"functionCallId": call_id, "value": value}, + } + + +def _failure(call_id: str, code: str, message: str) -> dict[str, Any]: + return { + "version": "1.0", + "rendererFunctionResponse": { + "functionCallId": call_id, + "error": {"code": code, "message": message}, + }, + } + + +def _id(message: Any) -> str: + return message.call_renderer_function.function_call_id + + +async def _settle(future: asyncio.Future[Any]) -> Any: + return await asyncio.wait_for(future, timeout=5) + + +class TestCall: + + @pytest.mark.asyncio + async def test_builds_the_message_without_waiting(self) -> None: + calls = RendererFunctionCalls() + + message, future = calls.call( + catalog_id=CATALOG, + name="getResolution", + args={"unit": "px"}, + surface_id="s1", + ) + + assert not future.done() + dumped = message.model_dump(by_alias=True, exclude_unset=True) + call_id = dumped["callRendererFunction"]["functionCallId"] + assert call_id + assert dumped == { + "version": "1.0", + "callRendererFunction": { + "surfaceId": "s1", + "functionCallId": call_id, + "callFunction": { + "@call": "getResolution", + "catalogId": CATALOG, + "args": {"unit": "px"}, + }, + }, + } + assert calls.pending_count == 1 + future.cancel() + + @pytest.mark.asyncio + async def test_leaves_out_surface_id_when_not_given(self) -> None: + calls = RendererFunctionCalls() + + message, future = calls.call(catalog_id=CATALOG, name="now") + + dumped = message.model_dump(by_alias=True, exclude_unset=True) + assert "surfaceId" not in dumped["callRendererFunction"] + assert dumped["callRendererFunction"]["callFunction"]["args"] == {} + future.cancel() + + @pytest.mark.asyncio + async def test_generates_distinct_ids(self) -> None: + calls = RendererFunctionCalls() + + pairs = [calls.call(catalog_id=CATALOG, name="now") for _ in range(3)] + + assert len({_id(message) for message, _ in pairs}) == 3 + for _, future in pairs: + future.cancel() + + @pytest.mark.asyncio + async def test_leaves_out_an_empty_surface_id(self) -> None: + calls = RendererFunctionCalls() + + message, future = calls.call(catalog_id=CATALOG, name="now", surface_id="") + + dumped = message.model_dump(by_alias=True, exclude_unset=True) + assert "surfaceId" not in dumped["callRendererFunction"] + future.cancel() + + @pytest.mark.asyncio + async def test_requires_a_function_name(self) -> None: + calls = RendererFunctionCalls() + + with pytest.raises(A2uiRpcError) as excinfo: + calls.call(catalog_id=CATALOG, name="") + + assert excinfo.value.code == "INVALID_FUNCTION_CALL" + assert calls.pending_count == 0 + + @pytest.mark.asyncio + async def test_requires_a_catalog_id(self) -> None: + calls = RendererFunctionCalls() + + with pytest.raises(A2uiRpcError) as excinfo: + calls.call(catalog_id="", name="now") + + assert excinfo.value.code == "INVALID_FUNCTION_CALL" + assert calls.pending_count == 0 + + @pytest.mark.asyncio + async def test_rejects_a_pending_call_id(self) -> None: + calls = RendererFunctionCalls() + _, future = calls.call(catalog_id=CATALOG, name="now", function_call_id="c1") + + with pytest.raises(A2uiRpcError) as excinfo: + calls.call(catalog_id=CATALOG, name="now", function_call_id="c1") + + assert excinfo.value.code == "DUPLICATE" + future.cancel() + + @pytest.mark.asyncio + async def test_max_pending(self) -> None: + calls = RendererFunctionCalls(max_pending=2) + _, first = calls.call(catalog_id=CATALOG, name="now", function_call_id="a") + _, second = calls.call(catalog_id=CATALOG, name="now") + + with pytest.raises(A2uiRpcError) as excinfo: + calls.call(catalog_id=CATALOG, name="now") + assert excinfo.value.code == "TOO_MANY_PENDING" + + calls.resolve(_value("a", 1)) + assert await _settle(first) == 1 + _, third = calls.call(catalog_id=CATALOG, name="now") + second.cancel() + third.cancel() + + def test_needs_a_running_loop(self) -> None: + with pytest.raises(RuntimeError): + RendererFunctionCalls().call(catalog_id=CATALOG, name="now") + + def test_belongs_to_the_first_loop(self) -> None: + calls = RendererFunctionCalls() + + async def call() -> None: + _, future = calls.call(catalog_id=CATALOG, name="now") + future.cancel() + + asyncio.run(call()) + with pytest.raises(RuntimeError, match="another event loop"): + asyncio.run(call()) + + def test_rejects_bad_settings(self) -> None: + with pytest.raises(ValueError): + RendererFunctionCalls(max_pending=0) + + @pytest.mark.parametrize("timeout", [0, -1, float("nan"), float("inf"), True, "30"]) + def test_rejects_a_default_timeout_that_is_not_finite_and_positive( + self, timeout: Any + ) -> None: + with pytest.raises(ValueError, match="default_timeout"): + RendererFunctionCalls(default_timeout=timeout) + + def test_accepts_no_default_timeout(self) -> None: + RendererFunctionCalls(default_timeout=None) + + +class TestCallChecks: + """The checks `call` makes, in order, before anything is registered.""" + + @pytest.mark.asyncio + @pytest.mark.parametrize( + "timeout", [0, -1, float("nan"), float("inf"), float("-inf"), True, False] + ) + async def test_rejects_a_timeout_that_is_not_finite_and_positive( + self, timeout: Any + ) -> None: + calls = RendererFunctionCalls() + + with pytest.raises(ValueError, match="timeout"): + calls.call(catalog_id=CATALOG, name="now", timeout=timeout) + + assert calls.pending_count == 0 + + @pytest.mark.asyncio + async def test_checks_the_timeout_before_disposal(self) -> None: + calls = RendererFunctionCalls() + calls.dispose() + + with pytest.raises(ValueError): + calls.call(catalog_id=CATALOG, name="now", timeout=float("nan")) + + @pytest.mark.asyncio + async def test_rejects_an_empty_function_call_id(self) -> None: + calls = RendererFunctionCalls() + + with pytest.raises(A2uiRpcError) as excinfo: + calls.call(catalog_id=CATALOG, name="now", function_call_id="") + + assert excinfo.value.code == "INVALID_FUNCTION_CALL" + assert calls.pending_count == 0 + + @pytest.mark.asyncio + async def test_checks_disposal_before_the_function_call_id(self) -> None: + calls = RendererFunctionCalls() + calls.dispose() + + with pytest.raises(A2uiRpcError) as excinfo: + calls.call(catalog_id=CATALOG, name="now", function_call_id="") + + assert excinfo.value.code == "DISPOSED" + + @pytest.mark.asyncio + async def test_checks_the_name_before_the_catalog_id(self) -> None: + calls = RendererFunctionCalls() + + with pytest.raises(A2uiRpcError, match="function name"): + calls.call(catalog_id="", name="") + + @pytest.mark.asyncio + async def test_checks_the_pending_limit_before_the_name(self) -> None: + calls = RendererFunctionCalls(max_pending=1) + _, future = calls.call(catalog_id=CATALOG, name="now") + + with pytest.raises(A2uiRpcError) as excinfo: + calls.call(catalog_id=CATALOG, name="") + + assert excinfo.value.code == "TOO_MANY_PENDING" + future.cancel() + + @pytest.mark.asyncio + @pytest.mark.parametrize("name", ["get-location", "@index", "1st", "a b"]) + async def test_rejects_a_name_that_is_not_an_identifier(self, name: str) -> None: + calls = RendererFunctionCalls() + + with pytest.raises(A2uiRpcError) as excinfo: + calls.call(catalog_id=CATALOG, name=name) + + assert excinfo.value.code == "INVALID_FUNCTION_CALL" + assert calls.pending_count == 0 + + @pytest.mark.asyncio + async def test_checks_the_catalog_id_before_the_identifier(self) -> None: + calls = RendererFunctionCalls() + + with pytest.raises(A2uiRpcError, match="needs a catalogId"): + calls.call(catalog_id="", name="get-location") + + @pytest.mark.asyncio + @pytest.mark.parametrize( + ("args", "text"), + [ + (["Saved"], "must be an object"), + ("Saved", "must be an object"), + ({"bad-key": 1}, "bad-key"), + ({1: "one"}, "1"), + ({"x": float("nan")}, "'x'"), + ({"x": float("inf")}, "'x'"), + ({"x": [1, float("-inf")]}, "x[1]"), + ({"x": {"y": float("nan")}}, "x.y"), + ({"x": {1: "one"}}, "isn't a string"), + ({"x": object()}, "isn't a JSON value"), + ({"x": {1, 2}}, "isn't a JSON value"), + ], + ids=[ + "list", + "string", + "bad-key", + "non-string-key", + "nan", + "inf", + "nested-inf", + "nested-nan", + "nested-non-string-key", + "object", + "set", + ], + ) + async def test_rejects_args_that_are_not_a_json_object( + self, args: Any, text: str + ) -> None: + calls = RendererFunctionCalls() + + with pytest.raises(A2uiRpcError) as excinfo: + calls.call(catalog_id=CATALOG, name="notify", args=args) + + assert excinfo.value.code == "INVALID_FUNCTION_CALL" + assert text in str(excinfo.value) + assert calls.pending_count == 0 + + @pytest.mark.asyncio + async def test_accepts_json_args_and_leaves_reserved_objects_alone(self) -> None: + calls = RendererFunctionCalls() + args = { + "a": None, + "b": True, + "c": "text", + "d": 1, + "e": 1.5, + "f": [1, "two", {"g": None}], + "h": {"@path": "/x"}, + "i": {"@call": "now", "args": {}}, + } + + message, future = calls.call(catalog_id=CATALOG, name="notify", args=args) + + dumped = message.model_dump(by_alias=True, exclude_unset=True) + assert dumped["callRendererFunction"]["callFunction"]["args"] == args + future.cancel() + + @pytest.mark.asyncio + async def test_checks_the_args_before_the_catalogs(self) -> None: + calls = RendererFunctionCalls(catalogs=[_device_catalog()]) + + with pytest.raises(A2uiRpcError) as excinfo: + calls.call(catalog_id=CATALOG, name="formatLocal", args={"bad-key": 1}) + + assert "bad-key" in str(excinfo.value) + assert "may not call" not in str(excinfo.value) + + @pytest.mark.asyncio + async def test_checks_the_name_identifier_before_the_catalogs(self) -> None: + calls = RendererFunctionCalls(catalogs=[_device_catalog()]) + + with pytest.raises(A2uiRpcError) as excinfo: + calls.call(catalog_id="https://example.com/other.json", name="@index") + + assert excinfo.value.code == "INVALID_FUNCTION_CALL" + + @pytest.mark.asyncio + async def test_no_limit_comes_only_from_the_instance(self) -> None: + loop = asyncio.get_running_loop() + delays: list[float] = [] + call_later = loop.call_later + + def recording_call_later(delay: float, *args: Any) -> asyncio.TimerHandle: + delays.append(delay) + return call_later(delay, *args) + + unlimited = RendererFunctionCalls(default_timeout=None) + with pytest.MonkeyPatch.context() as patch: + patch.setattr(loop, "call_later", recording_call_later) + _, first = unlimited.call(catalog_id=CATALOG, name="now") + _, second = unlimited.call(catalog_id=CATALOG, name="now", timeout=5) + _, third = RendererFunctionCalls(default_timeout=7).call( + catalog_id=CATALOG, name="now", timeout=None + ) + + assert delays == [5, 7] + for future in (first, second, third): + future.cancel() + + +def _device_catalog(catalog_id: str = CATALOG) -> Catalog: + def fn(name: str, **kwargs: Any) -> FunctionApi: + return FunctionApi(name, return_type="any", schema={"type": "object"}, **kwargs) + + return Catalog( + catalog_id, + ProtocolVersion.V1_0, + functions=[ + fn("getLocation", allowed_callers="rendererOrAgent"), + fn("notify", allowed_callers="agentOnly"), + fn("formatLocal", allowed_callers="rendererOnly"), + fn("unset", allowed_callers=None), + fn( + "vibrate", + allowed_callers="rendererOrAgent", + requires_user_activation=True, + ), + ], + ) + + +class TestCatalogChecks: + + @pytest.mark.asyncio + @pytest.mark.parametrize("name", ["getLocation", "notify"]) + async def test_allows_functions_the_agent_may_call(self, name: str) -> None: + calls = RendererFunctionCalls(catalogs=[_device_catalog()]) + + message, future = calls.call(catalog_id=CATALOG, name=name) + + assert message.call_renderer_function.call_function.call == name + assert calls.pending_count == 1 + future.cancel() + + @pytest.mark.asyncio + @pytest.mark.parametrize( + ("catalog_id", "name", "code", "text"), + [ + ( + "https://example.com/other.json", + "getLocation", + "UNKNOWN_FUNCTION", + "isn't one", + ), + ( + CATALOG, + "getAltitude", + "UNKNOWN_FUNCTION", + "no renderer function 'getAltitude'", + ), + (CATALOG, "formatLocal", "INVALID_FUNCTION_CALL", "may not call"), + (CATALOG, "unset", "INVALID_FUNCTION_CALL", "may not call"), + (CATALOG, "vibrate", "INVALID_FUNCTION_CALL", "user gesture"), + ], + ids=[ + "unknown-catalog", + "unknown-function", + "renderer-only", + "unset", + "activation", + ], + ) + async def test_rejects_before_registering( + self, catalog_id: str, name: str, code: str, text: str + ) -> None: + calls = RendererFunctionCalls(catalogs=[_device_catalog()]) + loop = asyncio.get_running_loop() + timers: list[Any] = [] + call_later = loop.call_later + + def recording_call_later(*args: Any) -> asyncio.TimerHandle: + timers.append(args) + return call_later(*args) + + with pytest.MonkeyPatch.context() as patch: + patch.setattr(loop, "call_later", recording_call_later) + with pytest.raises(A2uiRpcError) as excinfo: + calls.call(catalog_id=catalog_id, name=name, function_call_id="c1") + + assert excinfo.value.code == code + assert text in str(excinfo.value) + assert excinfo.value.function_call_id == "c1" + assert calls.pending_count == 0 + assert not calls.is_pending("c1") + assert timers == [] + + @pytest.mark.asyncio + async def test_empty_name_is_checked_first(self) -> None: + calls = RendererFunctionCalls(catalogs=[_device_catalog()]) + + with pytest.raises(A2uiRpcError, match="needs a function name"): + calls.call(catalog_id="https://example.com/other.json", name="") + + @pytest.mark.asyncio + async def test_rejects_an_agent_function(self) -> None: + catalog = _device_catalog().copy_with( + agent_functions=[AgentFunction("lookUp", handler=lambda args: 1)] + ) + calls = RendererFunctionCalls(catalogs=[catalog]) + + with pytest.raises(A2uiRpcError) as excinfo: + calls.call(catalog_id=CATALOG, name="lookUp") + + assert excinfo.value.code == "UNKNOWN_FUNCTION" + assert "no renderer function 'lookUp'" in str(excinfo.value) + assert calls.pending_count == 0 + + @pytest.mark.asyncio + async def test_rejects_an_agent_function_built_from_renderer_or_agent( + self, + ) -> None: + source = _device_catalog() + agent_function = AgentFunction.from_api( + source.functions["getLocation"], lambda args: "here" + ) + assert agent_function.allowed_callers == "rendererOrAgent" + catalog = source.copy_with(agent_functions=[agent_function]) + calls = RendererFunctionCalls(catalogs=[catalog]) + + with pytest.raises(A2uiRpcError) as excinfo: + calls.call(catalog_id=CATALOG, name="getLocation", function_call_id="c1") + + assert excinfo.value.code == "UNKNOWN_FUNCTION" + assert "no renderer function 'getLocation'" in str(excinfo.value) + assert calls.pending_count == 0 + assert not calls.is_pending("c1") + + def test_two_catalogs_with_one_id_are_a_catalog_error(self) -> None: + with pytest.raises(A2uiCatalogError, match="Duplicate catalog ID"): + RendererFunctionCalls(catalogs=[_device_catalog(), _device_catalog()]) + + @pytest.mark.asyncio + async def test_without_catalogs_nothing_is_checked(self) -> None: + calls = RendererFunctionCalls() + + _, future = calls.call(catalog_id="https://example.com/any.json", name="any") + + assert calls.pending_count == 1 + future.cancel() + + @pytest.mark.asyncio + async def test_reads_an_iterable_once(self) -> None: + calls = RendererFunctionCalls(catalogs=iter([_device_catalog()])) + + _, future = calls.call(catalog_id=CATALOG, name="getLocation") + _, other = calls.call(catalog_id=CATALOG, name="getLocation") + + future.cancel() + other.cancel() + + +class TestResolve: + + @pytest.mark.asyncio + async def test_value_settles_the_matching_call(self) -> None: + calls = RendererFunctionCalls() + message_a, future_a = calls.call(catalog_id=CATALOG, name="a") + message_b, future_b = calls.call(catalog_id=CATALOG, name="b") + + calls.resolve(_value(_id(message_b), [1920, 1080])) + + assert await _settle(future_b) == [1920, 1080] + assert not future_a.done() + assert calls.pending_count == 1 + future_a.cancel() + + @pytest.mark.asyncio + async def test_accepts_the_schema_model(self) -> None: + calls = RendererFunctionCalls() + message, future = calls.call(catalog_id=CATALOG, name="now") + + calls.resolve( + RendererFunctionResponseMessage( + version="1.0", + rendererFunctionResponse=FunctionResponse( + functionCallId=_id(message), value="12:00" + ), + ) + ) + + assert await _settle(future) == "12:00" + + @pytest.mark.asyncio + async def test_none_value(self) -> None: + calls = RendererFunctionCalls() + message, future = calls.call(catalog_id=CATALOG, name="clear") + + calls.resolve(_value(_id(message), None)) + + assert await _settle(future) is None + + @pytest.mark.asyncio + async def test_error_fails_the_call_with_the_renderer_code(self) -> None: + calls = RendererFunctionCalls() + message, future = calls.call(catalog_id=CATALOG, name="getResolution") + + calls.resolve( + _failure(_id(message), "EXECUTION_ERROR", "Failed to query the screen.") + ) + + with pytest.raises(A2uiRpcError) as excinfo: + await _settle(future) + assert excinfo.value.code == "EXECUTION_ERROR" + assert excinfo.value.function_call_id == _id(message) + assert "getResolution" in str(excinfo.value) + assert "Failed to query the screen." in str(excinfo.value) + + @pytest.mark.asyncio + async def test_unknown_late_and_duplicate_responses_are_ignored( + self, caplog: pytest.LogCaptureFixture + ) -> None: + calls = RendererFunctionCalls() + message, future = calls.call(catalog_id=CATALOG, name="now") + call_id = _id(message) + + with caplog.at_level(logging.DEBUG, logger="a2ui.rpc"): + assert calls.resolve(_value("never-sent", 0)) is False + assert calls.resolve(_value(call_id, "first")) is True + assert calls.resolve(_value(call_id, "duplicate")) is False + assert calls.resolve(_failure(call_id, "EXECUTION_ERROR", "late")) is False + + assert await _settle(future) == "first" + ignored = [r for r in caplog.records if "Ignoring" in r.getMessage()] + assert len(ignored) == 3 + assert all(r.levelno == logging.DEBUG for r in ignored) + + @pytest.mark.asyncio + async def test_malformed_response_is_logged_and_ignored( + self, caplog: pytest.LogCaptureFixture + ) -> None: + calls = RendererFunctionCalls() + message, future = calls.call(catalog_id=CATALOG, name="now") + + with caplog.at_level(logging.WARNING, logger="a2ui.rpc"): + assert not calls.resolve({"version": "1.0", "rendererFunctionResponse": {}}) + assert not calls.resolve({"something": "else"}) + + assert len(caplog.records) == 2 + assert not future.done() + future.cancel() + + @pytest.mark.asyncio + async def test_malformed_response_to_a_pending_call_fails_it(self) -> None: + calls = RendererFunctionCalls() + message, future = calls.call(catalog_id=CATALOG, name="now") + _, other = calls.call(catalog_id=CATALOG, name="other") + + # Both `value` and `error`, which the schema forbids. + assert ( + calls.resolve({ + "version": "1.0", + "rendererFunctionResponse": { + "functionCallId": _id(message), + "value": 1, + "error": {"code": "EXECUTION_ERROR", "message": "boom"}, + }, + }) + is True + ) + + with pytest.raises(A2uiRpcError) as excinfo: + await _settle(future) + assert excinfo.value.code == "UNKNOWN_ERROR" + assert excinfo.value.function_call_id == _id(message) + assert "Malformed rendererFunctionResponse" in str(excinfo.value) + assert not other.done() + assert calls.pending_count == 1 + other.cancel() + + @pytest.mark.asyncio + @pytest.mark.parametrize( + "version, body, extra", + [ + ("1.0", {"error": None}, {}), + ("1.0", {}, {}), + ("1.0", {"value": 1, "extra": 2}, {}), + ("1.0", {"value": 1}, {"extra": 2}), + ("0.9", {"value": 1}, {}), + ("v1.0", {"value": 1}, {}), + ], + ids=[ + "null-error", + "neither-value-nor-error", + "extra-body-key", + "extra-message-key", + "version-0.9", + "version-v1.0", + ], + ) + async def test_malformed_response_shapes_fail_the_pending_call( + self, version: str, body: dict[str, Any], extra: dict[str, Any] + ) -> None: + calls = RendererFunctionCalls() + message, future = calls.call(catalog_id=CATALOG, name="now") + + response = { + "version": version, + "rendererFunctionResponse": {"functionCallId": _id(message), **body}, + **extra, + } + assert calls.resolve(response) is True + + with pytest.raises(A2uiRpcError) as excinfo: + await _settle(future) + assert excinfo.value.code == "UNKNOWN_ERROR" + assert calls.pending_count == 0 + + @pytest.mark.asyncio + async def test_malformed_response_is_logged_without_its_payload( + self, caplog: pytest.LogCaptureFixture + ) -> None: + calls = RendererFunctionCalls() + message, future = calls.call(catalog_id=CATALOG, name="now") + + with caplog.at_level(logging.WARNING, logger="a2ui.rpc"): + calls.resolve({ + "version": "1.0", + "rendererFunctionResponse": { + "functionCallId": _id(message), + "value": 1, + "secret": "hunter2", + }, + }) + + assert "Ignoring a malformed rendererFunctionResponse" in caplog.text + assert "hunter2" not in caplog.text + with pytest.raises(A2uiRpcError): + await _settle(future) + + @pytest.mark.asyncio + async def test_malformed_response_to_an_unknown_call_is_ignored(self) -> None: + calls = RendererFunctionCalls() + _, future = calls.call(catalog_id=CATALOG, name="now") + + assert ( + calls.resolve({ + "version": "1.0", + "rendererFunctionResponse": {"functionCallId": "unknown"}, + }) + is False + ) + + assert not future.done() + future.cancel() + + @pytest.mark.asyncio + async def test_is_pending(self) -> None: + calls = RendererFunctionCalls() + message, future = calls.call(catalog_id=CATALOG, name="now") + + assert calls.is_pending(_id(message)) + assert not calls.is_pending("unknown") + calls.resolve(_value(_id(message), 1)) + assert await _settle(future) == 1 + assert not calls.is_pending(_id(message)) + + @pytest.mark.asyncio + async def test_resolve_is_the_processor_callback(self) -> None: + calls = RendererFunctionCalls() + processor = RendererToAgentMessageProcessor( + [], on_renderer_function_response=calls.resolve + ) + message, future = calls.call(catalog_id=CATALOG, name="now") + + result = await processor.process_message(_value(_id(message), "Paris")) + + assert result.handled + assert result.replies == [] + assert await _settle(future) == "Paris" + + @pytest.mark.asyncio + async def test_processor_declines_responses_it_does_not_own(self) -> None: + calls = RendererFunctionCalls() + processor = RendererToAgentMessageProcessor( + [], on_renderer_function_response=calls.resolve + ) + message, future = calls.call(catalog_id=CATALOG, name="now") + assert (await processor.process_message(_value(_id(message), 1))).handled + + for call_id in ("unknown", _id(message)): # unknown, then duplicate + result = await processor.process_message(_value(call_id, 2)) + assert not result.handled + assert result.replies == [] + assert await _settle(future) == 1 + + def test_response_before_any_call_is_ignored(self) -> None: + assert RendererFunctionCalls().resolve(_value("c1", 1)) is False + + @pytest.mark.asyncio + async def test_resolve_from_another_thread(self) -> None: + calls = RendererFunctionCalls() + message, future = calls.call(catalog_id=CATALOG, name="now") + + thread = threading.Thread( + target=calls.resolve, args=(_value(_id(message), "from thread"),) + ) + thread.start() + thread.join() + + assert await _settle(future) == "from thread" + assert calls.pending_count == 0 + + +class TestTimeouts: + + def test_default_timeout_matches_the_renderer_side(self) -> None: + assert DEFAULT_RENDERER_CALL_TIMEOUT == 30.0 + + @pytest.mark.asyncio + async def test_timeout_fails_the_call_and_forgets_it(self) -> None: + calls = RendererFunctionCalls(default_timeout=0.01) + message, future = calls.call(catalog_id=CATALOG, name="slow") + + with pytest.raises(A2uiRpcError) as excinfo: + await _settle(future) + + assert excinfo.value.code == "TIMEOUT" + assert calls.pending_count == 0 + calls.resolve(_value(_id(message), "too late")) + + @pytest.mark.asyncio + async def test_per_call_timeout(self) -> None: + calls = RendererFunctionCalls() + _, future = calls.call(catalog_id=CATALOG, name="slow", timeout=0.01) + + with pytest.raises(A2uiRpcError) as excinfo: + await _settle(future) + assert excinfo.value.code == "TIMEOUT" + + @pytest.mark.asyncio + async def test_settling_cancels_the_timer(self) -> None: + loop = asyncio.get_running_loop() + timers: list[asyncio.TimerHandle] = [] + call_later = loop.call_later + + def recording_call_later(*args: Any) -> asyncio.TimerHandle: + timer = call_later(*args) + timers.append(timer) + return timer + + calls = RendererFunctionCalls(default_timeout=0.01) + with pytest.MonkeyPatch.context() as patch: + patch.setattr(loop, "call_later", recording_call_later) + message, future = calls.call(catalog_id=CATALOG, name="now") + (timer,) = timers + calls.resolve(_value(_id(message), 1)) + + assert await _settle(future) == 1 + assert timer.cancelled() + await asyncio.sleep(0.03) + assert future.result() == 1 + + @pytest.mark.asyncio + async def test_no_timeout(self) -> None: + calls = RendererFunctionCalls(default_timeout=None) + message, future = calls.call(catalog_id=CATALOG, name="now") + + await asyncio.sleep(0.02) + assert not future.done() + calls.resolve(_value(_id(message), 1)) + assert await _settle(future) == 1 + + +class TestCancellationAndDisposal: + + @pytest.mark.asyncio + async def test_cancelling_the_future_forgets_the_call(self) -> None: + calls = RendererFunctionCalls(default_timeout=0.01) + message, future = calls.call(catalog_id=CATALOG, name="now") + + future.cancel() + await asyncio.sleep(0) + + assert calls.pending_count == 0 + calls.resolve(_value(_id(message), "ignored")) + await asyncio.sleep(0.03) + assert future.cancelled() + + @pytest.mark.asyncio + async def test_cancelling_the_awaiting_task_forgets_the_call(self) -> None: + calls = RendererFunctionCalls() + _, future = calls.call(catalog_id=CATALOG, name="now") + waiter = asyncio.ensure_future(future) + await asyncio.sleep(0) + + waiter.cancel() + with pytest.raises(asyncio.CancelledError): + await waiter + await asyncio.sleep(0) + + assert calls.pending_count == 0 + + @pytest.mark.asyncio + async def test_dispose_fails_pending_calls_and_refuses_new_ones(self) -> None: + calls = RendererFunctionCalls() + message, future = calls.call(catalog_id=CATALOG, name="now") + + calls.dispose("the session ended") + calls.dispose("again") + + with pytest.raises(A2uiRpcError) as excinfo: + await _settle(future) + assert excinfo.value.code == "CANCELLED" + assert "the session ended" in str(excinfo.value) + assert calls.disposed + assert calls.pending_count == 0 + with pytest.raises(A2uiRpcError) as refused: + calls.call(catalog_id=CATALOG, name="now") + assert refused.value.code == "DISPOSED" + calls.resolve(_value(_id(message), "ignored")) + + @pytest.mark.asyncio + async def test_dispose_from_another_thread(self) -> None: + calls = RendererFunctionCalls() + _, future = calls.call(catalog_id=CATALOG, name="now") + + thread = threading.Thread(target=calls.dispose, args=("closed",)) + thread.start() + thread.join() + + with pytest.raises(A2uiRpcError) as excinfo: + await _settle(future) + assert excinfo.value.code == "CANCELLED" + + @pytest.mark.asyncio + @pytest.mark.parametrize("malformed", [False, True]) + async def test_response_after_dispose_from_another_thread_is_ignored( + self, malformed: bool + ) -> None: + calls = RendererFunctionCalls() + message, future = calls.call(catalog_id=CATALOG, name="now") + call_id = _id(message) + + # `dispose` hands `_fail_all` to the loop, which hasn't run it yet. + thread = threading.Thread(target=calls.dispose, args=("closed",)) + thread.start() + thread.join() + response = ( + {"version": "1.0", "rendererFunctionResponse": {"functionCallId": call_id}} + if malformed + else _value(call_id, "too late") + ) + + assert not calls.is_pending(call_id) + assert calls.resolve(response) is False + with pytest.raises(A2uiRpcError) as excinfo: + await _settle(future) + assert excinfo.value.code == "CANCELLED" + + def test_dispose_after_the_loop_closes_does_not_raise(self) -> None: + loop = asyncio.new_event_loop() + calls = RendererFunctionCalls() + + async def start() -> asyncio.Future[Any]: + _, future = calls.call(catalog_id=CATALOG, name="now") + # A callback makes failing the future schedule work on the loop, + # which raises once the loop is closed. + future.add_done_callback(lambda _: None) + return future + + future = loop.run_until_complete(start()) + loop.close() + + thread = threading.Thread(target=calls.dispose, args=("closed",)) + thread.start() + thread.join() + + assert calls.disposed + assert calls.pending_count == 0 + del future + + def test_dispose_when_the_loop_closes_during_the_call( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + loop = asyncio.new_event_loop() + calls = RendererFunctionCalls() + + async def start() -> asyncio.Future[Any]: + _, future = calls.call(catalog_id=CATALOG, name="now") + return future + + future = loop.run_until_complete(start()) + + def closed(*args: Any) -> None: + raise RuntimeError("Event loop is closed") + + # The loop is open when `dispose` checks it, then closes before the + # handoff, as during teardown on another thread. + monkeypatch.setattr(loop, "call_soon_threadsafe", closed) + thread = threading.Thread(target=calls.dispose, args=("closed",)) + thread.start() + thread.join() + + assert calls.pending_count == 0 + assert future.done() + assert isinstance(future.exception(), A2uiRpcError) + loop.close() + + @pytest.mark.asyncio + async def test_async_with_disposes(self) -> None: + async with RendererFunctionCalls() as calls: + _, future = calls.call(catalog_id=CATALOG, name="now") + + assert calls.disposed + with pytest.raises(A2uiRpcError) as excinfo: + await _settle(future) + assert excinfo.value.code == "CANCELLED" diff --git a/python/a2ui_core/CHANGELOG.md b/python/a2ui_core/CHANGELOG.md index d6a18ffec1..0b9882d7ba 100644 --- a/python/a2ui_core/CHANGELOG.md +++ b/python/a2ui_core/CHANGELOG.md @@ -230,6 +230,10 @@ and the bare `protocolVersion` (for example `'1.0'`) on each inline catalog even when the source catalog document omits them. [#3382](https://github.com/a2ui-project/a2ui/pull/3382) +- (v1_0) Add `RpcErrorCode.TOO_MANY_PENDING`, for an agent's call to a + renderer function refused because too many are pending + ([#3535](https://github.com/a2ui-project/a2ui/issues/3535), + [#3939](https://github.com/a2ui-project/a2ui/pull/3939)). ## 0.4.0 (2026-10-17) diff --git a/python/a2ui_core/src/a2ui/core/exceptions.py b/python/a2ui_core/src/a2ui/core/exceptions.py index 3756a64204..d1a02acf81 100644 --- a/python/a2ui_core/src/a2ui/core/exceptions.py +++ b/python/a2ui_core/src/a2ui/core/exceptions.py @@ -180,6 +180,7 @@ class RpcErrorCode(str, enum.Enum): DISPOSED = "DISPOSED" NO_LISTENER = "NO_LISTENER" DUPLICATE = "DUPLICATE" + TOO_MANY_PENDING = "TOO_MANY_PENDING" class A2uiRpcError(A2uiError): diff --git a/python/a2ui_core/tests/test_state.py b/python/a2ui_core/tests/test_state.py index fb7aeff022..e53428ee32 100644 --- a/python/a2ui_core/tests/test_state.py +++ b/python/a2ui_core/tests/test_state.py @@ -439,6 +439,7 @@ def test_exception_hierarchy_normalization(): "DISPOSED", "NO_LISTENER", "DUPLICATE", + "TOO_MANY_PENDING", } actual_codes = {member.value for member in RpcErrorCode} assert actual_codes == expected_codes diff --git a/typescript/core/web_core/tests/conformance/conformance_test.mjs b/typescript/core/web_core/tests/conformance/conformance_test.mjs index cbf200373f..cb8296132f 100644 --- a/typescript/core/web_core/tests/conformance/conformance_test.mjs +++ b/typescript/core/web_core/tests/conformance/conformance_test.mjs @@ -296,6 +296,7 @@ const UNIMPLEMENTED_ACTIONS = new Map([ ['handle_agent_function', 'agent functions are agent-side only'], ['validate_payload', 'validating model output is agent-side only'], ['process_renderer_message', 'processing renderer messages is agent-side only'], + ['call_renderer_function', "the agent's calls to renderer functions are agent-side only"], [ 'agent_to_renderer_schema', 'web_core does not generate the agent_to_renderer schema from its own models',