diff --git a/docs/architecture/README.md b/docs/architecture/README.md index e2f922528..39fb42280 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -1,15 +1,92 @@ # Architecture -Kagent defines portable agent behavior with `AgentTemplate` and compiles it -through an admitted `Harness`. The resulting revision is applied to a Substrate -ActorTemplate. PostgreSQL-backed `AgentInstance` records own lifecycle and public -A2A contexts; they are not Kubernetes resources. +Kagent is a Kubernetes-native control plane for defining, compiling, running, +and invoking agents. Kubernetes stores desired agent configuration. PostgreSQL +stores runtime identity, A2A history, and lifecycle state. Substrate Actors run +the agent processes. -Detailed documents: +## Resource ownership -- [Human in the loop](human-in-the-loop.md) -- [Prompt templates](prompt-templates.md) +| Resource | Owner | Purpose | +| --- | --- | --- | +| `Harness` | Kubernetes (`kagent.dev/v1alpha3`) | Runtime implementation, workload, credentials, capacity, and admission policy | +| `AgentTemplate` | Kubernetes (`kagent.dev/v1alpha3`) | Portable agent behavior: model, prompt, tools, skills, and plugins | +| prepared revision | PostgreSQL and ate-api | Immutable compiled runtime input and its Substrate ActorTemplate | +| `AgentInstance` | PostgreSQL, exposed by gRPC | Ephemeral compute identity and lifecycle | +| A2A context, task, and events | PostgreSQL, exposed by A2A | Durable interaction and audit history | +| checkpoint | PostgreSQL plus a Substrate snapshot tag | Immutable, named restart boundary | +| Actor and durable directory | Substrate | Process lifecycle and private runtime state | + +`AgentInstance` is not a Kubernetes resource. A2A owns public interaction +semantics; kagent does not maintain a parallel session or task API. + +## Public surfaces + +| Surface | Role | +| --- | --- | +| Kubernetes API | Author Harnesses, AgentTemplates, models, prompts, and remote MCP servers | +| gRPC / gRPC-Web | Manage AgentInstances, sharing, checkpoints, and control-plane reads | +| A2A | Invoke agents and manage durable tasks and streams | +| MCP | Discover, invoke, checkpoint, and fork AgentInstances through A2A semantics | + +## End-to-end flow + +```mermaid +flowchart LR + AT[AgentTemplate] --> R[resolve tree] + H[Harness] --> R + R --> B[build harness inputs] + B --> C[registered harness compiler] + C --> REV[immutable revision] + REV --> ATE[ate-api ActorTemplate] + ATE --> SNAP[golden snapshot ready] + SNAP --> AI[AgentInstance] + AI --> ACTOR[Substrate Actor] + CLIENT[A2A client] --> GW[public A2A gateway] + GW --> ACTOR + GW --> DB[(tasks and events)] + GW --> QUIESCE[auto-suspend at quiescence] + QUIESCE --> CKPT[checkpoint tag] + CKPT --> FORK[forked AgentInstance] +``` + +Compilation and application are separate. The translator produces an immutable +revision; the controller applies it through ate-api. At runtime, the public A2A +gateway is the sole owner of task ingestion, durable event ordering, and +quiescence. It reaches Actors through the private runtime network. + +## Component boundaries + +- API types describe agent behavior without exposing backend mechanics. +- The v2 translator resolves references and compiles explicit runtime inputs. +- The controller reconciles compiled revisions to ate-api ActorTemplates. +- AgentInstance services and workflows own lifecycle orchestration. +- The A2A gateway owns public task routing, persistence, streaming, and + auto-suspend boundaries. +- The store owns transactional invariants and never performs network work. +- Substrate adapters own Actor, snapshot, and private-network operations. + +## Documents + +- [Configuration and compilation](configuration-and-compilation.md) +- [Runtime and lifecycle](runtime-and-lifecycle.md) +- [A2A gateway](a2a-gateway.md) +- [Persistence, checkpoints, and forks](persistence-checkpoints-and-forks.md) +- [MCP](mcp.md) - [A2A agent tools](a2a-subagents.md) +- [Human in the loop](human-in-the-loop.md) +- [Prompt resolution](prompt-templates.md) + +The documents describe implemented behavior. Deferred work, including full +cross-AgentInstance delegation and Dedicated agents, belongs in the +[API v2 execution plan](../plans/api-v2-execution-plan.md). + +## Current boundaries + +Implemented end to end: kagent, Codex, Claude, and BYO compilation; ate-api +ActorTemplates; AgentInstance lifecycle; durable A2A tasks; auto-suspend; +checkpoint/fork; and MCP Tasks continuation. -The implementation roadmap and dependency graph live in -[the API v2 execution plan](../plans/api-v2-execution-plan.md). +Not implemented: Dedicated agent bindings, policy-enforced public +cross-AgentInstance delegation, checkpoint sharing, and multi-replica gateway +coordination. diff --git a/docs/architecture/a2a-gateway.md b/docs/architecture/a2a-gateway.md new file mode 100644 index 000000000..e904390f7 --- /dev/null +++ b/docs/architecture/a2a-gateway.md @@ -0,0 +1,49 @@ +# A2A Gateway + +The public gateway implements the upstream A2A handler for message send/stream, +task get/list/cancel, and subscription. It also serves the extended Agent Card +compiled into the instance's prepared revision. + +## Routing and execution + +Authentication establishes namespace and AgentInstance authority. The gateway +loads the instance and prepared revision, derives the private Actor route, and +forwards upstream A2A requests. Actor addresses and runtime credentials remain +internal. + +Each running task has one event ingester. It alone owns runtime event consumption, +durable persistence, and the final quiescence transition; client streams and +subscribers only observe its queue. This permits multiple observers without +creating multiple Actor readers or suspending the same turn twice. + +```mermaid +flowchart LR + ACTOR[private Actor stream] --> INGEST[one task event ingester] + INGEST -->|1. append event and update task| DB[(PostgreSQL)] + DB -->|2. committed| INGEST + INGEST -->|3. publish| Q[event queue] + Q --> SEND[original send stream] + Q --> SUB1[subscriber] + Q --> SUB2[subscriber] + INGEST -->|at quiescence| SUSPEND[AgentInstance workflow] +``` + +## Durable ordering + +The gateway persists the task and every ordered event before publishing the event +to observers. The store atomically applies an event to materialized task state and +appends its history row. Malformed durable events fail rather than being silently +discarded. + +The persistence model enforces: + +- one non-quiescent task per A2A context; +- message-ID idempotency using the request hash; +- conflict rejection when an ID is reused for different content; and +- an exact snapshot identity and history sequence at each quiescent boundary. + +Tasks contain current materialized A2A state. Complete message history is rebuilt +from ordered event rows, not stored as one history blob. + +The implementation is in +[`go/core/v2/a2agateway`](../../go/core/v2/a2agateway). diff --git a/docs/architecture/a2a-subagents.md b/docs/architecture/a2a-subagents.md index bc1b0e38a..4b6becd81 100644 --- a/docs/architecture/a2a-subagents.md +++ b/docs/architecture/a2a-subagents.md @@ -1,19 +1,41 @@ # A2A Agent Tools -Kagent runtimes can expose another A2A agent as a tool. Each invocation sends an -A2A message with a context ID and returns the child result together with that -context ID. A binding may reuse one context for consecutive calls or isolate each -call in a fresh context. +Kagent has two distinct subagent mechanisms. They should not be confused. -If the child task enters `input_required`, the tool records the child task and -context IDs in the parent's approval state. Continuing the parent forwards the -answer to that same child task. Authentication, user identity, and lineage headers -are forwarded by A2A client interceptors. +```mermaid +flowchart TB + PARENT[Parent agent] + PARENT -->|Shared binding compiled into one runtime| LOCAL[Native in-process subagent] + PARENT -->|remote A2A tool call| REMOTE[Addressable A2A agent] + REMOTE -->|task + context IDs retained| CONTINUE[input-required continuation] + DEDICATED[Dedicated binding] -. deferred .-> SEPARATE[separate AgentInstance] + PUBLIC[Public cross-instance delegation] -. deferred .-> POLICY[credential and lineage policy] +``` -The Go and Python implementations are: +## Shared agent tools -- `go/adk/pkg/tools/remote_a2a_tool.go` -- `python/packages/kagent-adk/src/kagent/adk/_remote_a2a_tool.py` +An `AgentTemplate` can bind another template as a `Shared` agent tool. The +translator resolves the referenced template in the same compilation tree and +the selected harness compiler emits its native, in-process representation. +Kagent, Codex, and Claude support Shared bindings according to their runtime +capabilities. -Public cross-AgentInstance delegation remains tracked in the API v2 execution -plan; this runtime tool does not replace gateway-level delegation policy. +Tree resolution detects missing references and cycles before compilation. +`Dedicated` bindings are represented in the API but are currently rejected; +they do not create a separate AgentInstance today. + +## Runtime remote A2A tools + +The Go and Python ADKs also contain a remote A2A tool. Each call sends an A2A +message to an already-addressable remote agent and preserves the child task and +context IDs. If the child enters `input-required`, the parent can retain those +identifiers and continue the same child task after receiving human input. + +Implementations: + +- [`go/adk/pkg/tools/remote_a2a_tool.go`](../../go/adk/pkg/tools/remote_a2a_tool.go) +- [`python/packages/kagent-adk/src/kagent/adk/_remote_a2a_tool.py`](../../python/packages/kagent-adk/src/kagent/adk/_remote_a2a_tool.py) + +This runtime helper is not public cross-AgentInstance delegation. Gateway-level +delegation still requires scoped credentials, lineage/depth/cycle enforcement, +and streamed child execution; that work is deferred in the execution plan. diff --git a/docs/architecture/configuration-and-compilation.md b/docs/architecture/configuration-and-compilation.md new file mode 100644 index 000000000..791ba5d56 --- /dev/null +++ b/docs/architecture/configuration-and-compilation.md @@ -0,0 +1,77 @@ +# Configuration and Compilation + +## Public configuration + +`Harness` describes how to run a class of agents. It selects exactly one runtime +variant—kagent, Codex, Claude, or BYO—and contains workload image/command/args, +environment and credential references, WorkerPool configuration, snapshot +location, and an admission selector. + +`AgentTemplate` describes what the agent does. It contains model configuration, +description and prompt, MCP tool bindings, skills, plugins, and Shared or +Dedicated agent bindings. Model configuration may be omitted for BYO images; +pair compilation rejects managed harness combinations without one. + +Both are `kagent.dev/v1alpha3` Kubernetes resources. Infrastructure-derived +values such as runtime addresses and inferred egress do not belong in the public +API. + +## Prepared revision pipeline + +The v2 controller collects admitted Harness/AgentTemplate pairs and compiles each +pair through one pipeline: + +```mermaid +flowchart TD + H[Harness] --> MATCH{admission selector matches} + AT[AgentTemplate] --> MATCH + MATCH --> RESOLVE[resolve template tree and references] + RESOLVE --> INPUTS[build explicit inputs] + INPUTS --> REGISTRY{runtime type} + REGISTRY --> K[kagent compiler] + REGISTRY --> X[Codex compiler] + REGISTRY --> C[Claude compiler] + REGISTRY --> B[BYO compiler] + K --> REV[immutable revision and digest] + X --> REV + C --> REV + B --> REV + REV --> ATE[ate-api ActorTemplate] + ATE --> GOLDEN[golden snapshot] + GOLDEN -->|ready| LATEST[latest successful revision] + RESOLVE -->|error| STATUS[pair status] + ATE -->|error| STATUS +``` + +1. Resolve the template tree and referenced Kubernetes objects. +2. Build explicit, harness-independent inputs. +3. Select the harness compiler from the runtime-type registration map. +4. Produce an immutable revision containing workload, configuration, Agent Card, + capacity, snapshot, provenance, and inferred egress inputs. +5. Hash the revision and apply it as an ate-api ActorTemplate. +6. Wait for the golden snapshot to become ready. +7. Persist the revision and advance the pair's latest-successful pointer. + +A failed compile or apply leaves the previous successful revision available. +AgentInstances pin a prepared revision, so later template edits do not mutate a +running instance. + +Harness compilers only translate inputs. The controller and Substrate adapter own +application and readiness. The central entry points are +[`translator/compiler.go`](../../go/core/v2/translator/compiler.go) and +[`controller/reconciler.go`](../../go/core/v2/controller/reconciler.go). + +## Harness-specific output + +- **kagent** emits Go ADK configuration, Shared native subagents, and the kagent + HITL extension. +- **Codex** emits native App Server configuration, OpenAI or Bedrock model setup, + Streamable HTTP MCP servers, Shared agents, and skills. Approvals are currently + disabled by policy. +- **Claude** emits Anthropic, Bedrock, or Vertex model setup, HTTP/SSE MCP + servers, Shared agents, and skills. +- **BYO** runs a digest-pinned user image that implements private A2A gRPC and + `/readyz`. Optional model, prompt, tool, skill, and plugin configuration is + supplied in the ADK-shaped format when requested. + +Dedicated agent bindings are not compiled yet. diff --git a/docs/architecture/human-in-the-loop.md b/docs/architecture/human-in-the-loop.md index 06452d6e2..2953e027b 100644 --- a/docs/architecture/human-in-the-loop.md +++ b/docs/architecture/human-in-the-loop.md @@ -1,52 +1,8 @@ -# Human-in-the-Loop +# Human in the Loop -Kagent Human-in-the-Loop (HITL) lets an agent pause an A2A task, ask a human -for a decision, and resume the same task after the decision arrives. The public -contract is the framework-agnostic A2A Extension, defined in the [A2A spec](https://a2a-protocol.org/latest/specification/#46-extensions) -Google ADK confirmation events are implementation details of an adapter and are -documented separately in the appendices. - -The version 1 profile of the A2A Extension supports the complete minimum HITL feature set: - -- approve one or more tool calls; -- reject one or more tool calls, optionally with reasons; -- resolve several pending calls independently in one response; -- answer one or more `ask_user` questions; and -- propagate any of those interactions through a remote subagent. - -An implementation advertising `hitl/v1` supports this profile as a whole. The -AgentCard does not contain separate feature flags for approval, rejection, -multi-tool responses, or `ask_user`. - -## Mental model - -HITL is a two-message exchange attached to an ordinary A2A task: - -1. The agent sends an `input-required` status whose status Message contains a - HITL request. -2. The client sends a user Message to the same `taskId` and `contextId` whose - HITL payload contains explicit approval results or question answers. -3. The runtime validates the response, translates it into its local continuation - primitive, and resumes the paused task. - -No special RPC or task state is added. The task remains the unit of routing and -durability; the extension only defines the structured request and decision. -The human decision is not a new prompt asking the model to reconstruct the -call. A framework adapter resumes the exact paused operation using its stored -continuation state. The model may run again after the tool result, as it would -after any other tool call. - -## Responsibilities - -| Component | Responsibility | -|---|---| -| AgentCard | Declare support for the exact versioned extension URI. | -| A2A client | Opt in, render requests, collect a complete decision, and resume the same task. | -| A2A server | Negotiate the extension and route the decision to the pending task. | -| Framework adapter | Translate between the public extension and the framework's local pause/resume mechanism. | -| Tool or subagent adapter | Preserve the continuation needed to resume the exact tool or child task. | - -## Extension discovery and activation +Kagent human-in-the-loop (HITL) lets an agent pause an A2A task, request a human +decision, and resume the exact paused operation. It is a framework-independent +A2A extension; framework confirmation events are adapter details. The extension URI is: @@ -54,588 +10,86 @@ The extension URI is: https://kagent.dev/extensions/hitl/v1 ``` -An agent declares it in its AgentCard: +An Agent Card advertises that URI as an optional capability. A client activates it +with the standard `A2A-Extensions` header. -```json -{ - "capabilities": { - "streaming": true, - "extensions": [ - { - "uri": "https://kagent.dev/extensions/hitl/v1", - "description": "Tool approval, ask_user, and nested subagents", - "required": false - } - ] - } -} -``` +## Protocol -The client opts in on the request: +HITL is an exchange on an ordinary A2A task: -```http -A2A-Extensions: https://kagent.dev/extensions/hitl/v1 -``` +1. The agent emits an `input-required` status whose status message metadata + contains a HITL request. +2. The gateway persists the event and quiesces the Actor at that exact boundary. +3. The client sends a user message to the same task and context with a structured + decision. +4. The runtime adapter validates the decision and resumes its stored continuation. -The server activates only the exact requested URI and echoes the activated URI -on its response or event stream. A future incompatible contract uses a new URI, -for example `.../hitl/v2`; there is no silent version fallback. +No HITL RPC, session, or second task model exists. The response continues the +same A2A task; it is not a prompt asking the model to reconstruct the pending +operation. -**Because version 1 is optional (`required: false`), a client that does not opt in can still receive an ordinary `input-required` task with human-readable text. It cannot safely submit a structured HITL decision and must use a client that supports the extension or avoid invoking HITL features.** +## Request forms -## Where the payload lives +Version 1 supports two request kinds: -HITL data is a [Message extension](https://a2a-protocol.org/latest/specification/#462-extensions-points). -This is because Kagent uses Messages in `TaskStatusUpdateEvents` on transition to `input-required` -states to signal HITL events instead of artifacts. The Message must contain both: +- **tool approval** identifies one or more pending tool calls. Each decision + explicitly approves or rejects a call and may include a rejection reason. +- **ask user** identifies one or more questions and returns an answer for each. -```json -{ - "extensions": ["https://kagent.dev/extensions/hitl/v1"], - "metadata": { - "https://kagent.dev/extensions/hitl/v1": { - "type": "..." - } - } -} -``` - -The `extensions` entry declares that the Message uses the extension. The -metadata entry contains its payload. Clients must require both; they must not -infer HITL from text or from private framework-shaped DataParts. - -The Message can also contain a TextPart for display, logging, and accessibility. -That text is descriptive only and must never be parsed as the decision. - -## Identifier model - -Several IDs participate in a HITL flow. They are not interchangeable. - -| Identifier | Scope | Used by | -|---|---|---| -| `taskId` | The paused A2A task | Client and server route the resume to the correct pending task. | -| `contextId` | The A2A conversation/session | Client resumes in the same conversation; nested calls preserve the child's context independently. | -| `id` | One resumable approval | Opaque correlation token copied unchanged from request to response. | -| `call_id` | One original tool call | UI identity, display, and audit correlation; it is not needed in the response. | - -The client must return each opaque `id`, but must not interpret it. The adapter -validates every returned ID against the stored `input-required` request before -resuming anything. `name`, `args`, and `call_id` are request-only and are never -echoed back as authoritative data. - -## Server to client: pausing a task +Requests and responses live in the A2A status message metadata under the +extension URI key. The message also includes the extension URI in its `extensions` +array. IDs correlate each decision with its exact pending call or question. +Clients must preserve unknown metadata and return a complete, unambiguous +response. -The server sends a final status update for the current stream segment: - -```json -{ - "kind": "status-update", - "taskId": "task-123", - "contextId": "conversation-456", - "final": true, - "status": { - "state": "input-required", - "message": { - "messageId": "message-789", - "role": "agent", - "taskId": "task-123", - "contextId": "conversation-456", - "extensions": ["https://kagent.dev/extensions/hitl/v1"], - "metadata": { - "https://kagent.dev/extensions/hitl/v1": { - "type": "tool_approval_request", - "hint": "Deleting this file requires approval", - "tools": [ - { - "id": "approval-1", - "call_id": "call-1", - "name": "delete_file", - "args": {"path": "/tmp/example"} - } - ] - } - }, - "parts": [ - {"kind": "text", "text": "Approval required for delete_file"} - ] - } - } -} -``` - -The client should persist the status Message as part of the task state. On page -reload, it can reconstruct an unresolved HITL interaction from a task whose -current state is still `input-required`. - -### Case 1: one tool approval - -For a single entry in `tools`, render the tool name and arguments and offer -Approve and Reject. A rejection UI may optionally collect a free-text reason. - -The client must treat `args` as untrusted display data. Rendering an approval -card must not execute the tool or interpret arguments as HTML. - -### Case 2: several pending tools - -Parallel framework confirmations are flattened into one list of independently -resumable approvals: - -```json -{ - "type": "tool_approval_request", - "hint": "Two operations require approval", - "tools": [ - { - "id": "approval-21", - "call_id": "call-delete", - "name": "delete_file", - "args": {"path": "/tmp/old"} - }, - { - "id": "approval-22", - "call_id": "call-restart", - "name": "restart_deployment", - "args": {"name": "api", "namespace": "production"} - } - ] -} -``` - -This is a flat list of two independent approvals. “Approve all” is only a UI -shortcut: the response still contains one explicit result per `id`. - -### Case 3: ask the user - -`ask_user` is a separate request type because answers are not approve/reject -decisions: - -```json -{ - "type": "ask_user_request", - "id": "approval-question-1", - "questions": [ - { - "question": "Which database should be used?", - "choices": ["PostgreSQL", "MySQL", "SQLite"], - "multiple": false - }, - { - "question": "Which optional features should be enabled?", - "choices": ["Authentication", "Caching", "Audit logging"], - "multiple": true - }, - { - "question": "Any additional requirements?", - "choices": [], - "multiple": false - } - ] -} -``` - -The user can select listed choices or provide free text. Answers are positional: -answer zero corresponds to question zero, and so on. An individual answer is an -array because a multiple-choice question may select several values. - -### Case 4: a remote subagent needs input - -A parent agent can pause because a child A2A task returned `input-required`. -The top-level `tools` entry represents the parent's remote-agent tool -continuation. The `nested` object describes what the child is waiting for: +The payload types are: ```json { "type": "tool_approval_request", - "hint": "Remote agent k8s_agent requires approval", + "hint": "Allow these calls?", "tools": [ - { - "id": "parent-approval-1", - "call_id": "parent-call-1", - "name": "k8s_agent", - "args": {"request": "Remove the obsolete pod"} - } - ], - "nested": { - "subagent_name": "k8s_agent", - "task_id": "child-task-1", - "context_id": "child-context-1", - "tools": [ - { - "id": "child-approval-1", - "call_id": "child-call-1", - "name": "delete_pod", - "args": {"name": "obsolete", "namespace": "production"} - } - ] - } -} -``` - -The UI displays `nested.tools`, because those are the operations the human is -actually authorizing, and returns their opaque IDs. The parent adapter retains -the top-level ID so it can resume the remote-agent tool, then forwards the -child results using the child IDs. - -While the child is paused, the Kagent UI also renders the parent AgentCall card -and uses `nested.context_id` to open that child session in its Activity panel. -This is especially important for isolated subagent sessions, where the session -ID exists only for that individual remote call. - -A nested `ask_user_request` has the same `questions` field as a direct request -and includes `nested` for attribution and child task routing. - -## Client to server: resuming a task - -The client sends a user Message using `message/stream` (or the corresponding -non-streaming send operation) with the same `taskId` and `contextId`. It opts in -again with the `A2A-Extensions` header and attaches a response payload. - -```json -{ - "messageId": "approval-response-1", - "role": "user", - "taskId": "task-123", - "contextId": "conversation-456", - "extensions": ["https://kagent.dev/extensions/hitl/v1"], - "metadata": { - "https://kagent.dev/extensions/hitl/v1": { - "type": "tool_approval_response", - "approvals": [ - {"id": "approval-1", "approved": true} - ] - } - }, - "parts": [ - {"kind": "text", "text": "Approved"} + {"id": "tool-1", "call_id": "call-1", "name": "deploy", "args": {}} ] } ``` -### Tool approval response - ```json { "type": "tool_approval_response", "approvals": [ - { - "id": "approval-21", - "approved": false, - "rejection_reason": "The file is still used by the migration job" - }, - { - "id": "approval-22", - "approved": true - } + {"id": "tool-1", "approved": false, "rejection_reason": "not production"} ] } ``` -Every `id` exposed by the applicable `tools` list must appear exactly once. -Missing, duplicate, and unknown IDs are invalid; omission must never silently -mean approval. A rejection reason is optional and belongs directly to its -rejected approval. For nested HITL, return IDs from `nested.tools`, not the -parent remote-tool entry. +Ask-user requests use `type: "ask_user_request"`, an `id`, and an array of +framework-neutral question objects. Responses use `type: "ask_user_response"`, +the same `id`, and an `answers` array whose entries contain string arrays. A +request may also carry `nested` child-agent correlation: `subagent_name`, +`task_id`, `context_id`, and the child's pending tools. -### Answer `ask_user` +The server rejects responses that use the wrong task/context, omit required +decisions, duplicate IDs, or answer an operation that is no longer pending. -```json -{ - "type": "ask_user_response", - "id": "approval-question-1", - "answers": [ - {"answer": ["PostgreSQL"]}, - {"answer": ["Authentication", "Caching"]}, - {"answer": ["Use the existing backup policy"]} - ] -} -``` - -The number and order of answer objects must match the questions. For a nested -question, the response uses the child question ID from `nested.tools`. - -## Validation and stale decisions - -A server should reject a decision when any of these conditions is true: - -- the Message does not declare the HITL extension; -- its extension payload is missing or has an unknown `type`; -- the target task is absent or is no longer `input-required`; -- the response type does not match the pending request type; -- an approval response omits an ID, repeats an ID, or contains an unknown ID; -- an `ask_user` response has missing or malformed answers; or -- the stored pause has no usable approval correlation. - -Clients should guard against two tabs answering the same pause. Before sending, -refresh or compare the current task state. Once one decision has been accepted, -later submissions are stale and must not resume another operation. - -An accepted decision may be retained in task history for audit and UI display, -but the framework adapter consumes a translated continuation message rather -than treating the human-readable TextPart as framework input. - -## Direct end-to-end flow - -```mermaid -sequenceDiagram - participant Human - participant Client as A2A Client - participant Server as A2A Server - participant Adapter as Framework Adapter - participant Runtime as Agent Runtime - participant Tool - - Client->>Server: Send user Message + HITL extension opt-in - Server->>Runtime: Start or continue task - Runtime->>Tool: Proposed tool call - Tool-->>Runtime: Local confirmation required - Runtime-->>Adapter: Framework pause event(s) - Adapter->>Adapter: Flatten approvals into tools[] - Adapter-->>Client: input-required + tool_approval_request - Human->>Client: Approve, reject, or choose per tool - Client->>Server: tool_approval_response for same taskId/contextId - Server->>Adapter: Stored pause + explicit approval results - Adapter->>Runtime: One local continuation response per id - Runtime->>Tool: Resume exact paused call - Tool-->>Runtime: Result or rejection result - Runtime-->>Client: Working updates and final result -``` - -## Nested end-to-end flow - -```mermaid -sequenceDiagram - participant Human - participant Client - participant Parent as Parent Agent - participant RemoteTool as Remote A2A Tool - participant Child as Child Agent - participant Tool as Child Tool - - Parent->>RemoteTool: Delegate request - RemoteTool->>Child: A2A Message with HITL opt-in - Child->>Tool: Proposed child tool call - Tool-->>Child: Confirmation required - Child-->>RemoteTool: Child task input-required + HITL request - RemoteTool->>Parent: Pause parent remote-tool call with child task/context - Parent-->>Client: Parent input-required + nested request - Human->>Client: Decision over nested tools - Client->>Parent: tool_approval_response with child ids - Parent->>RemoteTool: Resume parent remote-tool continuation - RemoteTool->>Child: Forward extension decision to child task/context - Child->>Tool: Resume exact child call - Tool-->>Child: Result or rejection - Child-->>RemoteTool: Completed child task - RemoteTool-->>Parent: Child result - Parent-->>Client: Final parent result -``` - -Each hop owns only its local continuation. The public response remains the same -A2A extension shape at every remote boundary. This permits recursive nesting, -although deep agent chains are harder to operate and debug. - -## Building another framework adapter - -A framework adapter implementing `hitl/v1` needs five capabilities: - -1. Detect local pauses for tool approval and user questions. -2. Convert every local approval into a public tool with a stable opaque `id`, - original `call_id`, name, and arguments. -3. Persist enough local continuation state to resume the exact operation after - the A2A task is stored and the original request has ended. -4. Validate a response against the stored public pause and create one local - continuation response per approval. -5. For remote agents, preserve child `task_id` and `context_id` and forward the - same A2A response shape to that child. - -The adapter should not ask the model to recreate a paused call, scan display -text, or make the client return private framework state. If its framework -already owns durable confirmation matching, delegate matching to that upstream -mechanism. - -## Operational and debugging guide - -When a client does not show an approval card, verify in order: - -1. The AgentCard declares the exact HITL URI. -2. The request includes `A2A-Extensions` with that URI. -3. The response echoes the activated URI. -4. The task state is `input-required`. -5. `status.message.extensions` includes the URI. -6. `status.message.metadata[URI]` has a recognized request type. - -When a response does not resume execution, verify: - -1. The decision uses the same `taskId` and `contextId` as the pause. -2. The task is still `input-required` and has not already been answered. -3. The response Message declares the extension and has the matching response type. -4. Its approval IDs exactly match the IDs displayed by the pending request. -5. The framework adapter can resolve every returned opaque `id`. -6. For a subagent, the child task and context still identify its pending task. - -Useful logs should include task ID, context ID, request type, approval count, -response type, and subagent name. Avoid logging secrets contained in tool -arguments or free-text answers. - ---- - -## Appendix A: Google ADK concepts - -Google ADK tools call `request_confirmation()` when a proposed operation needs -human approval. ADK represents the pause as an -`adk_request_confirmation` function call containing: - -- the original function call name, arguments, and call ID; -- a separate confirmation function-call ID; and -- a `ToolConfirmation` containing a hint and optional tool-owned payload. +## Ownership -These objects are internal to the ADK adapter. At the A2A boundary: - -| ADK internal value | A2A HITL value | -|---|---| -| Confirmation function-call ID | `id` | -| Original function-call ID | `call_id` | -| Original function name and arguments | `tools[].name` and `tools[].args` | -| Confirmation hint | `hint` or the status TextPart | -| Several confirmation calls | Flat `tools[]` list | -| Confirmation FunctionResponse | Adapter-generated local continuation response | - -For a direct approval, ADK reinvokes the original tool with the resulting -`ToolConfirmation`. The before-tool approval callback allows execution when -confirmed and returns a rejection result when denied. Rejection reasons can be -included in the confirmation payload so the model receives useful context. - -The built-in `ask_user` tool uses the same local confirmation mechanism, but -the adapter stores structured answers in the confirmation payload. The tool -returns question/answer pairs to the model. - -## Appendix B: Go ADK adapter - -The Go adapter is intentionally thin around upstream Go ADK 2.x. - -### Request path - -1. Upstream ADK emits one `adk_request_confirmation` call per pending tool. -2. The upstream A2A executor converts those calls into internal long-running - function-call parts and produces `input-required`. -3. Kagent's `BuildHITLStatusMessage` removes the ADK-shaped parts from the - public status Message, flattens them into `tools[]`, and attaches the HITL - extension when it was negotiated. -4. Each ADK confirmation call ID becomes that tool's opaque `id`. - -### Response path - -1. A2A routing supplies both the stored `input-required` task and the incoming - response to `KAgentExecutor`. -2. `BuildResumeHITLMessage` validates the response against the stored public - request and builds one ADK confirmation FunctionResponse per - returned `id`. -3. The ordinary upstream ADK executor receives those responses. -4. Upstream ADK looks up its own session, matches confirmation responses to - pending calls, and resumes the original tools. - -Kagent Go code does **not** scan ADK session history for pending confirmations. -It owns only A2A negotiation and translation; upstream ADK owns session -matching and tool continuation. - -### Go remote subagent - -`KAgentRemoteA2ATool` activates the HITL extension on outbound child requests. -If the child returns `input-required`, it translates the child's public request -into local confirmation state containing the child task, context, subagent -name, and inner tools. The parent exposes that state as `nested`. - -After the parent confirmation resumes, the remote tool reconstructs an A2A -`tool_approval_response` from the retained child IDs and sends it to the saved -child task and context. `ask_user` answers and rejection reasons are preserved. - -## Appendix C: Python ADK adapter - -The Python runtime currently uses Google ADK before 2.0, whose A2A executor does -not yet provide the same upstream confirmation-resume behavior as Go ADK 2.x. -Its adapter therefore duplicates a limited amount of private ADK semantics as a -temporary compatibility layer. - -### Request path - -1. The Python runner emits long-running ADK confirmation events. -2. `_split_hitl_artifact_parts` separates confirmation parts from ordinary - artifact output. -3. `build_hitl_status_message` converts those parts into a public A2A HITL request - and emits an `input-required` status. -4. ADK-shaped confirmation DataParts do not cross the public A2A boundary. - -### Response path - -1. The executor parses the public tool-approval or ask-user response. -2. `_find_pending_confirmations` scans the Python ADK session for the most - recent unanswered confirmation calls. -3. `_process_hitl_response` builds the corresponding ADK - `ToolConfirmation` FunctionResponses and preserves tool-owned payload state. -4. The Python runner resumes with those FunctionResponses. - -This session scan is Python-specific compatibility code, not part of the A2A -extension and not a pattern to copy into the Go adapter or a new framework -adapter. The intended follow-up is to remove it when Python migrates to an -upstream ADK version that owns confirmation matching. - -Python and Go now expose the same request and response models. Each public tool -has an opaque `id`; each response returns that ID with an explicit boolean. - -### Python remote subagent - -`KAgentRemoteA2ATool` uses a two-phase invocation. On the first invocation it -sends an A2A request to the child. When the child pauses, the tool calls -`request_confirmation()` on the parent and stores the child task/context and -the validated public request in its private ADK payload. The parent executor -stores the validated public response alongside that request. When ADK -reinvokes the tool with a confirmation, the tool forwards that response to the -saved child task unchanged. - -Python still scans its private ADK session as a temporary compatibility layer, -but its public edge—including nested subagents—uses the same framework-neutral -ID-based contract as Go. - -`ToolConfirmation` and session scanning remain adapter internals. The stable -interface for clients and non-ADK runtimes is the A2A extension described in -the main body of this document. The remote adapter deliberately stores the -public `hitl_request` and `hitl_response` objects instead of reconstructing an -ADK-shaped `hitl_parts` representation. - -## Appendix D: Built-in `ask_user` tool - -The built-in tool accepts one or more questions in one call: - -```python -ask_user(questions=[ - { - "question": "Which database should I use?", - "choices": ["PostgreSQL", "MySQL", "SQLite"], - "multiple": False, - }, - { - "question": "Which features do you want?", - "choices": ["Auth", "Logging", "Caching"], - "multiple": True, - }, - { - "question": "Any additional requirements?", - "choices": [], - "multiple": False, - }, -]) -``` - -After the client returns positional answers, the tool result supplied to the -model is conceptually: - -```python -[ - {"question": "Which database should I use?", "answer": ["PostgreSQL"]}, - {"question": "Which features do you want?", "answer": ["Auth", "Caching"]}, - {"question": "Any additional requirements?", "answer": ["Add rate limiting"]}, -] -``` - -`ask_user` is a structured human-input operation, not a tool approval with an -implicit approval value. That distinction is why the public protocol uses -`ask_user_request` and `ask_user_response`. +| Component | Responsibility | +| --- | --- | +| Agent Card | Advertise the exact versioned extension URI | +| A2A client | Activate the extension, render requests, and continue the same task | +| Public gateway | Route and durably persist the interaction | +| Harness adapter | Translate public request/response data to native pause/resume state | +| Remote A2A tool | Preserve child task/context IDs for nested continuation | + +The kagent Go harness maps the contract to ADK confirmations. Remote A2A tools in +the Go and Python ADKs can retain a child continuation when a subagent requests +input. + +The current v2 MCP tool-binding API does not expose per-tool approval policy, so +users cannot configure generic MCP approval gating through `AgentTemplate` today. +The extension remains valid for runtimes and tools that actually produce a pause, +including ask-user and nested A2A continuation. + +For the base extension mechanism, see the +[A2A extension specification](https://a2a-protocol.org/latest/specification/#46-extensions). diff --git a/docs/architecture/mcp.md b/docs/architecture/mcp.md new file mode 100644 index 000000000..7546adbc5 --- /dev/null +++ b/docs/architecture/mcp.md @@ -0,0 +1,37 @@ +# MCP + +Kagent exposes an authenticated, stateless Streamable HTTP MCP endpoint at +`/mcp` on the HTTP port (`8083`). It is another client of the public control-plane +semantics, not a private path to Actors. + +Current tools can: + +- list accessible AgentInstances; +- invoke an AgentInstance; +- create and list checkpoints; and +- fork an AgentInstance from a checkpoint. + +Invocation calls the in-process public A2A gateway. Streaming MCP clients receive +updates from the same durable A2A task; synchronous clients drain the same stream +to completion. + +```mermaid +flowchart LR + CLIENT[MCP client] -->|Streamable HTTP /mcp| MCP[MCP server] + MCP -->|in-process A2A request| GW[public A2A gateway] + GW --> DB[(A2A task and events)] + GW --> ACTOR[private Actor] + ACTOR --> GW + GW --> MCP + MCP -->|updates or final result| CLIENT +``` + +## MCP Tasks + +The server implements the MCP Tasks extension. An opaque base64 task reference +contains the authorized namespace, AgentInstance, and A2A task identity. +`tasks/get`, `tasks/update`, and `tasks/cancel` translate to operations on that +same durable A2A task, including `input-required` continuation. There is no +separate MCP task or session store. + +The implementation is in [`go/core/v2/mcp`](../../go/core/v2/mcp). diff --git a/docs/architecture/persistence-checkpoints-and-forks.md b/docs/architecture/persistence-checkpoints-and-forks.md new file mode 100644 index 000000000..cb3ab4c79 --- /dev/null +++ b/docs/architecture/persistence-checkpoints-and-forks.md @@ -0,0 +1,70 @@ +# Persistence, Checkpoints, and Forks + +## Durable interaction model + +`AgentInstance` represents ephemeral compute. An A2A context durably owns its +tasks and ordered events, allowing interaction history to remain as an audit trail +after compute is removed. Normally a newly created context ID equals its +AgentInstance ID, but they are separate identities. + +The core PostgreSQL records are: + +| Record | Purpose | +| --- | --- | +| `runtime_revision` | Immutable compiled input and ate-api identity | +| `agent_template_harness_pair` | Pair status and latest successful revision | +| `agent_instance` | Compute identity, pinned revision, lifecycle phase, and Actor identity | +| `agent_instance_share` | Instance authorization grants | +| `a2a_context` | Durable owner of interaction history | +| `agent_instance_task` | Materialized current A2A task state | +| `agent_instance_task_event` | Append-only ordered task and message events | +| `agent_instance_checkpoint` | Named immutable snapshot/history boundary | + +Identity columns use PostgreSQL's native UUID type. Other framework-specific +tables are runtime implementation details, not part of this ownership model. + +```mermaid +flowchart TD + PAIR[Harness + AgentTemplate pair] --> REV[runtime revision] + REV --> INSTANCE[AgentInstance] + INSTANCE -. normally same initial ID .-> CONTEXT[A2A context] + CONTEXT --> TASK[materialized tasks] + TASK --> EVENT[ordered task events] + CONTEXT --> CHECKPOINT[checkpoint boundary] + REV --> CHECKPOINT + CHECKPOINT --> TAG[Substrate snapshot tag] + CHECKPOINT --> FORK[forked AgentInstance] + FORK --> NEWCTX[new A2A context] + CONTEXT -->|bounded history copy| NEWCTX +``` + +## Checkpoint creation + +A checkpoint names a quiescent boundary already recorded by the gateway. Creating +one does not suspend the Actor again: + +1. Reserve the checkpoint in PostgreSQL. +2. Verify the exact Substrate snapshot UID and scope recorded on the boundary. +3. Create an immutable snapshot tag. +4. Persist the tag UID and mark the checkpoint ready. + +The checkpoint retains source-instance provenance, source context, prepared +revision, labels, head task, and history sequence. The source AgentInstance may be +deleted while its context and checkpoint remain. + +Deletion first hides the checkpoint, then deletes its snapshot tag, then removes +the row. A checkpoint referenced by a fork cannot be deleted. Snapshot garbage +collection is a separate concern. + +## Forking + +Forking creates a new AgentInstance and A2A context. It copies task/event history +through the checkpoint sequence, deterministically remapping task and message IDs, +then creates the Actor from the checkpoint's snapshot tag. New work appends only +to the fork's context; source history and the checkpoint remain immutable. + +Checkpoint sharing is not implemented. Future sharing must be restricted to data +snapshots without process state. + +The workflow lives in +[`go/core/v2/checkpoint`](../../go/core/v2/checkpoint). diff --git a/docs/architecture/prompt-templates.md b/docs/architecture/prompt-templates.md index 3a4de0fec..df444eb3d 100644 --- a/docs/architecture/prompt-templates.md +++ b/docs/architecture/prompt-templates.md @@ -1,22 +1,28 @@ -# Prompt Templates +# Prompt Resolution -`AgentTemplate.spec.systemPrompt` may be rendered as a Go `text/template` when -`spec.promptTemplate` is present. A template can interpolate these values: +An `AgentTemplate` can provide its system prompt in one of three forms: + +- literal `spec.systemPrompt`; +- a complete value selected by `spec.systemPromptFrom`; or +- `spec.systemPrompt` rendered as a Go `text/template` when + `spec.promptTemplate` is configured. + +The mutually exclusive forms are enforced by the API schema. + +Prompt templates can reference: - `AgentTemplateName` - `AgentTemplateNamespace` - `Description` - `ToolNames` -It can also call `{{include "source/key"}}`. Each source is a same-namespace -ConfigMap named in `spec.promptTemplate.dataSources`; an optional alias replaces -the ConfigMap name in the include path. Included values are inserted as plain text -and are not recursively rendered. - -`spec.systemPromptFrom` instead reads the complete prompt from one key in a -same-namespace ConfigMap. It is mutually exclusive with `spec.systemPrompt`. +They can also call `{{include "source/key"}}`. Each source is a same-namespace +ConfigMap named by `spec.promptTemplate.dataSources`; an optional alias replaces +the ConfigMap name in the include path. Included values are inserted as text and +are not recursively rendered. -Compilation resolves ConfigMaps, rejects missing keys and duplicate include -identifiers, renders the prompt, and passes the result to the selected Harness -compiler. The semantic implementation and tests are in -`go/core/v2/translator/template.go`. +Resolution happens before harness compilation. Missing ConfigMaps or keys, +duplicate include identifiers, and invalid templates fail the prepared revision +rather than producing a partially configured runtime. The semantic implementation +and focused tests live in +[`go/core/v2/translator/template.go`](../../go/core/v2/translator/template.go). diff --git a/docs/architecture/runtime-and-lifecycle.md b/docs/architecture/runtime-and-lifecycle.md new file mode 100644 index 000000000..d99993088 --- /dev/null +++ b/docs/architecture/runtime-and-lifecycle.md @@ -0,0 +1,74 @@ +# Runtime and Lifecycle + +An `AgentInstance` is PostgreSQL-backed control-plane state exposed through gRPC. +It pins one prepared revision and names one Substrate Actor. It is not a +Kubernetes resource. + +## Creation and state + +Creation selects the latest successful revision for the Harness/AgentTemplate +pair, creates a deterministic Actor initially suspended, and marks the instance +ready after Substrate accepts it. Readiness of the image was already established +while preparing the ate-api ActorTemplate; AgentInstance creation does not resume +an Actor merely to probe `/readyz`. + +Lifecycle operations are implemented as retryable workflows: + +- database compare-and-set operations claim a transition; +- network work happens without holding a database transaction or lock; +- completion records the resulting state; +- retries observe and continue the durable phase. + +Explicit suspend and resume update the logical lifecycle state. Deletion fences +the instance, deletes the Actor, then removes control-plane state. The workflow +entry points are in +[`go/core/v2/agentinstance`](../../go/core/v2/agentinstance). + +## Automatic quiescence + +After an A2A task reaches a quiescent boundary—terminal, `input-required`, or +`auth-required`—the gateway asks the lifecycle workflow to quiesce the Actor. +Quiescence suspends compute and returns the exact snapshot identity while leaving +the AgentInstance logically ready. Substrate ingress resumes a suspended Actor +automatically when the next interaction arrives. + +Runtime calls and quiescence are serialized by an in-memory coordinator so a +late suspend cannot race a new turn in one process. This intentionally limits the +gateway to one replica until coordination is moved to a shared store. + +```mermaid +sequenceDiagram + participant Client + participant Gateway + participant DB as PostgreSQL + participant Workflow as AgentInstance workflow + participant Actor as Substrate Actor + Client->>Gateway: send or continue A2A task + Gateway->>Actor: invoke (ingress resumes if suspended) + Actor-->>Gateway: quiescent event + Gateway->>Actor: close runtime stream + Gateway->>Workflow: quiesce instance + Workflow->>Actor: suspend + Actor-->>Workflow: exact snapshot identity + Workflow-->>Gateway: snapshot boundary + Gateway->>DB: store task + event + snapshot atomically + DB-->>Gateway: committed + Gateway-->>Client: publish quiescent event + Note over Workflow,Actor: AgentInstance remains logically ready +``` + +## Runtime boundaries + +- Public native gRPC listens on port `8084`. +- Port `8083` serves health, gRPC-Web, and authenticated MCP. +- Actor A2A gRPC is private on port `80`. +- Runtime readiness is private HTTP `/readyz` on port `8081`. +- ate-api defaults to `dns:///api.ate-system.svc:443`. + +Clients never receive Actor addresses. The gateway derives and dials them through +the private atenetwork router. + +Every Actor mounts a Substrate `DurableDir` at `/data`. Harnesses keep private +state there—local framework state, workspaces, and downloaded assets that must +survive Actor replacement. This state is runtime-private; public task history +remains in PostgreSQL.