From 07b4f9abab875bdc377f603f05ccb48eba023452 Mon Sep 17 00:00:00 2001 From: Rui Conti Date: Tue, 7 Jul 2026 23:40:30 -0400 Subject: [PATCH 1/5] fix(eve): own HITL approval closure end-to-end in the harness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A parked local tool call is a provider-enforced obligation: its tool_use id must be closed by exactly one terminal tool-result before any replay. Closure was split across two owners — eve synthesized deny-side results while the approve side was delegated to AI SDK collectToolApprovals (which only inspects messages.at(-1)) with result persistence riding stream capture. Fixes #236, #460, #529, #533; turns the stacked red evals green. Two mechanisms, both harness-owned: - approved-tool-execution.ts: resolvePendingInput returns the approved calls (approvedActions) and closeApprovedActionBatch executes them at resume time via the same wrapToolExecute + toModelOutput normalization the SDK path uses (extracted as resolveExecutedToolModelOutput). Every approved call yields exactly one terminal closure — real output, tool-error on throw, error result when the tool is missing or has no local execute — appended to the same tool message as the approval response before any model request is assembled. action.result is emitted against the parked batch's turn coordinates, and an AuthorizationSignal parks via the shared parkOnAuthorization (also adopted by the post-step in-stream path). Closure no longer depends on the approval message being the request tail, so the post-resolution message deferral (and the channel-context ordering hazard from #529) is deleted; a message sent with an approval response now rides the same model call. The unanswered-approval deferral (unrelated follow-ups never auto-deny) is unchanged. - transcript-obligations.ts: one pure reconciliation pass over the assembled messages immediately before every model call closes any remaining dangling local tool-call with a synthetic error-text result placed adjacent to its assistant message. No exemptions: anything still dangling at that point is an orphan nothing else will close. The repair is durable, so histories already poisoned by a missed closure heal on their next turn instead of replaying the same 400 forever. Signed-off-by: Rui Conti --- .changeset/hitl-approval-owned-closure.md | 5 + docs/tools/human-in-the-loop.md | 2 +- .../harness/approved-tool-execution.test.ts | 180 +++++++++++++ .../src/harness/approved-tool-execution.ts | 244 ++++++++++++++++++ .../eve/src/harness/input-requests.test.ts | 18 +- packages/eve/src/harness/input-requests.ts | 89 ++++--- packages/eve/src/harness/tool-loop.test.ts | 240 +++++++++++++---- packages/eve/src/harness/tool-loop.ts | 134 +++++++--- packages/eve/src/harness/tools.ts | 60 +++-- .../harness/transcript-obligations.test.ts | 237 +++++++++++++++++ .../eve/src/harness/transcript-obligations.ts | 113 ++++++++ research/approval-resume-owned-execution.md | 2 +- 12 files changed, 1160 insertions(+), 164 deletions(-) create mode 100644 .changeset/hitl-approval-owned-closure.md create mode 100644 packages/eve/src/harness/approved-tool-execution.test.ts create mode 100644 packages/eve/src/harness/approved-tool-execution.ts create mode 100644 packages/eve/src/harness/transcript-obligations.test.ts create mode 100644 packages/eve/src/harness/transcript-obligations.ts diff --git a/.changeset/hitl-approval-owned-closure.md b/.changeset/hitl-approval-owned-closure.md new file mode 100644 index 000000000..2c0bdefc8 --- /dev/null +++ b/.changeset/hitl-approval-owned-closure.md @@ -0,0 +1,5 @@ +--- +"eve": patch +--- + +Approval-gated tools now execute during the resume itself: eve runs the approved tool and records its result before the next model call, and every outbound model request is checked so a tool call can never replay without a result. This fixes approve-resume failing with provider 400s (Anthropic `tool_use` without `tool_result`, OpenAI `No tool output found for function call`) — including when a channel attaches context to the approving prompt — and heals sessions whose history already carries a dangling tool call. A message sent together with an approval response now lands in the same model call instead of a deferred follow-up step. diff --git a/docs/tools/human-in-the-loop.md b/docs/tools/human-in-the-loop.md index 3d0e1db4f..e01cbe438 100644 --- a/docs/tools/human-in-the-loop.md +++ b/docs/tools/human-in-the-loop.md @@ -102,7 +102,7 @@ Approvals and questions share one protocol: 3. The turn parks at `session.waiting`, durably, for as long as it takes. 4. The client answers with `inputResponses` (structured, keyed by `requestId`) or a normal follow-up `message`. A follow-up whose text matches an option ID, option label, or numeric option index resolves automatically, including approval options such as `approve` and `deny`. -The run picks back up exactly where it parked. Because the pause is durable, nothing is held in memory while it waits — the process can restart and the parked turn survives. +The run picks back up exactly where it parked. On an approval grant, eve executes the tool during the resume itself and records its result before the next model call; on a denial it records the denial the same way. A parked tool call is always resolved before the model sees the transcript again. Because the pause is durable, nothing is held in memory while it waits — the process can restart and the parked turn survives. For approval requests, unrelated follow-up text does not deny the tool call. eve keeps the approval pending and holds that text until the approval is answered, then replays it as the next message in the session. diff --git a/packages/eve/src/harness/approved-tool-execution.test.ts b/packages/eve/src/harness/approved-tool-execution.test.ts new file mode 100644 index 000000000..d539a6c21 --- /dev/null +++ b/packages/eve/src/harness/approved-tool-execution.test.ts @@ -0,0 +1,180 @@ +import { jsonSchema, type ModelMessage } from "ai"; +import { describe, expect, it, vi } from "vitest"; + +import type { HarnessToolDefinition } from "#harness/execute-tool.js"; +import { + appendExecutedToolResults, + executeApprovedToolCalls, +} from "#harness/approved-tool-execution.js"; + +function createDefinition(overrides: Partial = {}): HarnessToolDefinition { + return { + description: "Echo", + inputSchema: jsonSchema({ type: "object" }), + name: "echo", + ...overrides, + }; +} + +const call = { + callId: "call-1", + input: { note: "hi" }, + kind: "tool-call", + toolName: "echo", +} as const; + +describe("executeApprovedToolCalls", () => { + it("executes the approved call and returns a durable result part plus action.result payload", async () => { + const execute = vi.fn().mockResolvedValue({ echoed: "hi" }); + const outcome = await executeApprovedToolCalls({ + calls: [call], + messages: [], + resolveTool: () => createDefinition({ execute }), + }); + + expect(execute).toHaveBeenCalledWith( + { note: "hi" }, + expect.objectContaining({ toolCallId: "call-1" }), + ); + expect(outcome.executed).toHaveLength(1); + expect(outcome.executed[0]?.part).toEqual({ + output: { type: "json", value: { echoed: "hi" } }, + toolCallId: "call-1", + toolName: "echo", + type: "tool-result", + }); + expect(outcome.executed[0]?.actionResult).toEqual({ + callId: "call-1", + kind: "tool-result", + output: { echoed: "hi" }, + toolName: "echo", + }); + }); + + it("applies the author's toModelOutput to the durable result", async () => { + const outcome = await executeApprovedToolCalls({ + calls: [call], + messages: [], + resolveTool: () => + createDefinition({ + execute: () => ({ echoed: "hi" }), + toModelOutput: () => ({ type: "text", value: "echoed hi" }), + }), + }); + + expect(outcome.executed[0]?.part.output).toEqual({ type: "text", value: "echoed hi" }); + }); + + it("closes a throwing execute with an error result instead of leaving the call dangling", async () => { + const outcome = await executeApprovedToolCalls({ + calls: [call], + messages: [], + resolveTool: () => + createDefinition({ + execute: () => { + throw new Error("boom"); + }, + }), + }); + + expect(outcome.executed[0]?.part.output).toEqual({ type: "error-text", value: "boom" }); + expect(outcome.executed[0]?.actionResult).toMatchObject({ + callId: "call-1", + isError: true, + output: "boom", + }); + }); + + it("closes an approved call whose tool no longer resolves", async () => { + const outcome = await executeApprovedToolCalls({ + calls: [call], + messages: [], + resolveTool: () => undefined, + }); + + expect(outcome.executed[0]?.part.output).toEqual({ + type: "error-text", + value: 'Tool "echo" is no longer available.', + }); + expect(outcome.executed[0]?.actionResult).toMatchObject({ isError: true }); + }); + + it("closes an execute-less tool with an error result instead of leaving it dangling", async () => { + const outcome = await executeApprovedToolCalls({ + calls: [call], + messages: [], + resolveTool: () => createDefinition(), + }); + + expect(outcome.executed[0]?.part.output).toEqual({ + type: "error-text", + value: 'Tool "echo" has no local execution and cannot run.', + }); + expect(outcome.executed[0]?.actionResult).toMatchObject({ isError: true }); + }); + + it("executes calls sequentially in batch order", async () => { + const order: string[] = []; + const outcome = await executeApprovedToolCalls({ + calls: [call, { ...call, callId: "call-2", input: { note: "second" } }], + messages: [], + resolveTool: () => + createDefinition({ + execute: async (input: { note: string }) => { + order.push(input.note); + return input.note; + }, + }), + }); + + expect(order).toEqual(["hi", "second"]); + expect(outcome.executed.map((entry) => entry.part.output)).toEqual([ + { type: "text", value: "hi" }, + { type: "text", value: "second" }, + ]); + }); +}); + +describe("appendExecutedToolResults", () => { + const part = { + output: { type: "text", value: "ok" }, + toolCallId: "call-1", + toolName: "echo", + type: "tool-result", + } as const; + + it("merges results into the trailing tool message", () => { + const messages: ModelMessage[] = [ + { content: "hi", role: "user" }, + { + content: [{ approvalId: "approval-1", approved: true, type: "tool-approval-response" }], + role: "tool", + }, + ]; + + const result = appendExecutedToolResults(messages, [part]); + + expect(result).toHaveLength(2); + expect(result[1]).toEqual({ + content: [{ approvalId: "approval-1", approved: true, type: "tool-approval-response" }, part], + role: "tool", + }); + }); + + it("appends a new tool message when the transcript does not end in one", () => { + const messages: ModelMessage[] = [{ content: "hi", role: "user" }]; + + const result = appendExecutedToolResults(messages, [part]); + + expect(result).toEqual([ + { content: "hi", role: "user" }, + { content: [part], role: "tool" }, + ]); + }); + + it("returns a copy when there is nothing to append", () => { + const messages: ModelMessage[] = [{ content: "hi", role: "user" }]; + + expect(appendExecutedToolResults(messages, [])).toEqual(messages); + }); +}); diff --git a/packages/eve/src/harness/approved-tool-execution.ts b/packages/eve/src/harness/approved-tool-execution.ts new file mode 100644 index 000000000..b20505626 --- /dev/null +++ b/packages/eve/src/harness/approved-tool-execution.ts @@ -0,0 +1,244 @@ +import type { ModelMessage } from "ai"; + +import { createLogger, logError } from "#internal/logging.js"; +import { buildDynamicTools } from "#context/build-dynamic-tools.js"; +import type { AlsContext } from "#context/container.js"; +import { createActionResultEvent } from "#protocol/message.js"; +import type { + RuntimeToolCallActionRequest, + RuntimeToolResultActionResult, +} from "#runtime/actions/types.js"; +import { toError } from "#shared/errors.js"; +import { createRuntimeToolResultFromValue } from "#harness/action-result-helpers.js"; +import { type AuthorizationSignal, isAuthorizationSignal } from "#harness/authorization.js"; +import type { HarnessToolDefinition } from "#harness/execute-tool.js"; +import type { ApprovedActionBatch } from "#harness/input-requests.js"; +import { readToolInterrupt } from "#harness/tool-interrupts.js"; +import { resolveExecutedToolModelOutput, wrapToolExecute } from "#harness/tools.js"; +import type { HarnessEmitFn, HarnessToolMap } from "#harness/types.js"; + +type ToolMessage = Extract; +type ToolResponsePart = ToolMessage["content"][number]; +type ToolResultPart = Extract; + +const log = createLogger("harness.approved-tool-execution"); + +/** One approved tool call the harness executed (or failed) at resume time. */ +export interface ExecutedApprovedToolCall { + /** `action.result` projection, attributed to the parked batch's turn. */ + readonly actionResult: RuntimeToolResultActionResult; + /** Durable transcript closure for the call's `tool_use` id. */ + readonly part: ToolResultPart; +} + +/** Outcome of executing one resolved approval batch's approved calls. */ +export interface ApprovedToolExecutionOutcome { + readonly executed: readonly ExecutedApprovedToolCall[]; +} + +/** + * Closes one resolved approval batch: executes the approved calls, appends + * their durable results to the transcript, and emits `action.result` events + * against the originating turn's stream coordinates. Returns the closed + * transcript plus the first {@link AuthorizationSignal} an executed tool + * raised, so the caller can park exactly like in-stream execution. + * + * Dynamic tools take precedence over authored tools of the same name, + * mirroring the toolset override order used for model calls. + */ +export async function closeApprovedActionBatch(input: { + readonly abortSignal?: AbortSignal; + readonly batch: ApprovedActionBatch; + readonly ctx: AlsContext | undefined; + readonly emit?: HarnessEmitFn; + readonly messages: readonly ModelMessage[]; + readonly tools: HarnessToolMap; +}): Promise<{ + readonly authorizationSignal?: AuthorizationSignal; + readonly messages: ModelMessage[]; +}> { + const { batch, ctx } = input; + + const { executed } = await executeApprovedToolCalls({ + abortSignal: input.abortSignal, + calls: batch.calls, + messages: input.messages, + resolveTool: (toolName) => { + const dynamicDefinition = + ctx === undefined + ? undefined + : buildDynamicTools(ctx).find((definition) => definition.name === toolName); + return dynamicDefinition ?? input.tools.get(toolName); + }, + }); + + if (executed.length === 0) { + return { messages: [...input.messages] }; + } + + const messages = appendExecutedToolResults( + input.messages, + executed.map((outcome) => outcome.part), + ); + + if (input.emit !== undefined && batch.event !== undefined) { + for (const outcome of executed) { + await input.emit( + createActionResultEvent({ + result: outcome.actionResult, + sequence: batch.event.sequence, + stepIndex: batch.event.stepIndex, + turnId: batch.event.turnId, + }), + ); + } + } + + return { + authorizationSignal: findStashedAuthorizationSignal(ctx, executed), + messages, + }; +} + +/** + * Reads back the first authorization signal stashed by `wrapToolExecute` + * during resume-time execution. + */ +function findStashedAuthorizationSignal( + ctx: AlsContext | undefined, + executed: readonly ExecutedApprovedToolCall[], +): AuthorizationSignal | undefined { + if (ctx === undefined) { + return undefined; + } + for (const outcome of executed) { + const stashed = readToolInterrupt(ctx, outcome.actionResult.callId); + if (stashed !== undefined && isAuthorizationSignal(stashed)) { + return stashed; + } + } + return undefined; +} + +/** + * Executes approved tool calls at resume time, in batch order, so the parked + * `tool_use` obligations are closed by the harness itself — durably and + * before any model request is assembled — instead of being delegated to the + * AI SDK's last-message approval scan and stream capture. + * + * Every approved call yields exactly one terminal closure — no exceptions: + * the tool's real output, an `error-text` result when `execute` throws, an + * `error-text` result when the approved tool no longer resolves, and an + * `error-text` result when the tool has no local `execute` (nothing else in + * the pipeline reliably closes an execute-less call, and an unclosed call is + * a guaranteed provider rejection). A tool that signals an authorization + * park keeps its existing `wrapToolExecute` semantics: the redacted pending + * output closes the call and the full signal is stashed for the caller's + * park detector. + */ +export async function executeApprovedToolCalls(input: { + readonly abortSignal?: AbortSignal; + readonly calls: readonly RuntimeToolCallActionRequest[]; + readonly messages: readonly ModelMessage[]; + readonly resolveTool: (toolName: string) => HarnessToolDefinition | undefined; +}): Promise { + const executed: ExecutedApprovedToolCall[] = []; + + for (const call of input.calls) { + const definition = input.resolveTool(call.toolName); + + if (definition === undefined) { + executed.push( + buildErrorOutcome(call, new Error(`Tool "${call.toolName}" is no longer available.`)), + ); + continue; + } + + const execute = wrapToolExecute(definition); + if (execute === undefined) { + executed.push( + buildErrorOutcome( + call, + new Error(`Tool "${call.toolName}" has no local execution and cannot run.`), + ), + ); + continue; + } + + try { + const output = await execute(call.input, { + abortSignal: input.abortSignal, + messages: [...input.messages], + toolCallId: call.callId, + }); + + executed.push({ + actionResult: createRuntimeToolResultFromValue({ + callId: call.callId, + output, + toolName: call.toolName, + }), + part: { + output: await resolveExecutedToolModelOutput({ + definition, + output, + toolCallId: call.callId, + }), + toolCallId: call.callId, + toolName: call.toolName, + type: "tool-result", + }, + }); + } catch (error) { + logError(log, "approved tool execution failed", error, { + toolCallId: call.callId, + toolName: call.toolName, + }); + executed.push(buildErrorOutcome(call, toError(error))); + } + } + + return { executed }; +} + +function buildErrorOutcome( + call: RuntimeToolCallActionRequest, + error: Error, +): ExecutedApprovedToolCall { + return { + actionResult: createRuntimeToolResultFromValue({ + callId: call.callId, + isError: true, + output: error, + toolName: call.toolName, + }), + part: { + output: { type: "error-text", value: error.message }, + toolCallId: call.callId, + toolName: call.toolName, + type: "tool-result", + }, + }; +} + +/** + * Merges resume-time closures into the trailing tool message (the one + * carrying the approval responses) so every `tool_use` and its result stay + * within adjacent messages, as providers require. Appends a new tool message + * when the transcript does not end in one. + */ +export function appendExecutedToolResults( + messages: readonly ModelMessage[], + parts: readonly ToolResultPart[], +): ModelMessage[] { + if (parts.length === 0) { + return [...messages]; + } + + const last = messages.at(-1); + if (last !== undefined && last.role === "tool" && Array.isArray(last.content)) { + return [...messages.slice(0, -1), { ...last, content: [...last.content, ...parts] }]; + } + + return [...messages, { content: [...parts], role: "tool" }]; +} diff --git a/packages/eve/src/harness/input-requests.test.ts b/packages/eve/src/harness/input-requests.test.ts index 08485bd9a..4171437b1 100644 --- a/packages/eve/src/harness/input-requests.test.ts +++ b/packages/eve/src/harness/input-requests.test.ts @@ -252,7 +252,7 @@ describe("resolvePendingInput", () => { }); }); - it("defers a follow-up message until after tool approvals are resolved", () => { + it("resolves approvals and keeps a simultaneous follow-up message on the same step", () => { const session = setPendingInputBatch({ requests: [ { @@ -305,18 +305,10 @@ describe("resolvePendingInput", () => { // The approval should be resolved immediately. expect(result.outcome).toBe("resolved"); - // The follow-up message should be deferred. - expect(result.deferredMessage).toBe(true); - expect(hasDeferredStepInput(result.session)).toBe(true); - - const deferred = consumeDeferredStepInput({ - session: result.session, - }); - - expect(deferred.input).toEqual({ - message: "Ignore that and say hi instead.", - }); - expect(hasDeferredStepInput(deferred.session)).toBe(false); + // The harness closes the denied call inline, so the follow-up message + // rides the same model call instead of being deferred a step. + expect(result.deferredMessage).toBeUndefined(); + expect(hasDeferredStepInput(result.session)).toBe(false); }); it("resolves approval when follow-up text matches an option", () => { diff --git a/packages/eve/src/harness/input-requests.ts b/packages/eve/src/harness/input-requests.ts index 631b372de..3f418ee74 100644 --- a/packages/eve/src/harness/input-requests.ts +++ b/packages/eve/src/harness/input-requests.ts @@ -54,6 +54,17 @@ export interface RejectedActionBatch { readonly results: readonly RuntimeToolResultActionResult[]; } +/** + * Approved tool-call actions from one resolved batch. The caller (the tool + * loop) executes them and appends their durable results before assembling + * the next model request; `event` carries the originating turn's stream + * coordinates for the resulting `action.result` emissions. + */ +export interface ApprovedActionBatch { + readonly calls: readonly RuntimeToolCallActionRequest[]; + readonly event?: PendingInputBatchEvent; +} + type ApprovalTerminalStatus = "approved" | "denied" | "ignored" | "invalid"; /** @@ -121,13 +132,14 @@ export function hasDeferredStepInput(session: HarnessSession): boolean { /** * Resolves pending input at the start of a harness step. * - * When the pending batch contains tool-approval requests and the step input - * also carries a follow-up user message, the message is deferred to the next - * internal harness step rather than appended to the current turn. This is - * necessary because AI SDK cannot process tool-approval responses and a new - * user message in the same request -- the approval must be resolved in - * isolation first, and the user message replayed on the subsequent step via - * {@link consumeDeferredStepInput}. + * When the pending batch contains tool-approval requests that the step input + * does not answer, the input is deferred to a later step via + * {@link consumeDeferredStepInput} and the batch stays parked — an unrelated + * follow-up must never auto-deny a pending approval. Once every approval is + * answered, the batch resolves in one pass: denials are closed inline with + * `execution-denied` results, and approved calls are returned on + * `approvedActions` for the tool loop to execute and close before the next + * model request. */ export function resolvePendingInput(input: { readonly history?: readonly ModelMessage[]; @@ -206,31 +218,11 @@ export function resolvePendingInput(input: { } const rejectedActions = buildRejectedActionBatch(pendingBatch, responses); + const approvedActions = buildApprovedActionBatch(pendingBatch, responses); session = clearPendingInputBatch(session); - // AI SDK cannot process tool-approval responses and a new user message - // in the same request. Defer the message so the approval is resolved in - // isolation; `consumeDeferredStepInput` replays it on the next step. - if ( - resolvedStepInput?.message !== undefined && - pendingBatch.requests.some((request) => isApprovalRequest(request)) - ) { - session = queueDeferredStepInput(session, { - message: resolvedStepInput.message, - }); - - return { - consumedMessage: resolvedStepInput?.messageConsumed, - deferredMessage: true, - limitContinuation, - outcome: "resolved", - messages, - rejectedActions, - session, - }; - } - return { + approvedActions, consumedMessage: resolvedStepInput?.messageConsumed, limitContinuation, outcome: "resolved", @@ -306,6 +298,11 @@ function hasUnansweredApproval(input: { } type ResolvePendingInputResult = { + /** + * Approved tool-call actions the caller must execute and close before the + * next model request. Present only on a resolved approval batch. + */ + readonly approvedActions?: ApprovedActionBatch; readonly consumedMessage?: boolean; readonly deferredMessage?: boolean; /** @@ -551,6 +548,25 @@ function buildRejectedActionBatch( return results.length > 0 ? { event: batch.event, results } : undefined; } +/** + * Extracts the approved tool-call actions from one resolved batch so the + * caller can execute them. + */ +function buildApprovedActionBatch( + batch: PendingInputBatch, + responses: readonly InputResponse[], +): ApprovedActionBatch | undefined { + const approvedIds = new Set( + responses.filter((r) => r.optionId === "approve").map((r) => r.requestId), + ); + + const calls = batch.requests + .filter((request) => isApprovalRequest(request) && approvedIds.has(request.requestId)) + .map((request) => request.action); + + return calls.length > 0 ? { calls, event: batch.event } : undefined; +} + function buildToolResponseParts( batch: PendingInputBatch, responses: readonly InputResponse[], @@ -589,17 +605,12 @@ function buildToolResponsePartsForRequest( }, ]; /* - * On denial (explicit "deny" or auto-deny when the user continues - * without responding), splice in the matching `execution-denied` - * tool-result. AI SDK's `streamText` synthesizes this for the - * current turn's `initialResponseMessages`, but that synthesis is - * gated on the input messages' last entry being a tool message — - * on subsequent turns (when a new user message is the tail of - * history) the synthesis is skipped, and the persisted + * The harness owns closing every parked tool call: providers reject a + * replayed `tool_use` without a `tool_result`, and the persisted * `tool-approval-response` is stripped during provider prompt - * conversion. Without an own `tool-result` in history, the prior - * `tool_use` block replays unmatched and some providers reject - * the request with 400. + * conversion, so it is not a closure. Denials are closed here with the + * matching `execution-denied` result; approved calls are executed and + * closed by the tool loop (see `approvedActions`). */ if (!approved) { parts.push({ diff --git a/packages/eve/src/harness/tool-loop.test.ts b/packages/eve/src/harness/tool-loop.test.ts index f355bf280..ba694a977 100644 --- a/packages/eve/src/harness/tool-loop.test.ts +++ b/packages/eve/src/harness/tool-loop.test.ts @@ -404,6 +404,7 @@ function finalOutputResult(text: string, structured: unknown): Record { }); }); - it("persists the inline approval-resume tool-result into session history so the next turn replays a balanced tool_use / tool_result pair", async () => { + it("executes an approved tool call at resume and persists its result before the model call, so the next turn replays a balanced tool_use / tool_result pair", async () => { /* - * When a previously-parked tool call is approved, the AI SDK - * enqueues its tool-result onto the parent stream before re- - * entering the LLM call. The result is absent from - * `stepResult.response.messages` / `toolCalls` / `toolResults`, - * so the harness must capture it from the stream and splice it - * into persisted history. Without this, the next turn replays a + * Closing a parked tool call is owned by the harness: the approved + * tool executes at resume time and its durable result lands in the + * same tool message as the approval response, before any model + * request is assembled. Without this, the next turn replays a * `tool_use` block with no matching `tool_result` and Anthropic * rejects the request with 400. */ @@ -4816,12 +4815,6 @@ describe("createToolLoopHarness", () => { content: [], finishReason: "stop", fullStreamParts: [ - { - output: { exitCode: 0, stderr: "", stdout: "/workspace\n", truncated: false }, - toolCallId: "call-1", - toolName: "bash", - type: "tool-result", - }, { id: "text-1", text: "`/workspace`", type: "text-delta" }, { finishReason: "stop", type: "finish-step" }, ], @@ -4833,43 +4826,179 @@ describe("createToolLoopHarness", () => { toolResults: [], }); - const { emit } = createEventCollector(); + const { emit, events } = createEventCollector(); const session = createPendingBashApprovalSession(); + const bashExecute = vi + .fn() + .mockResolvedValue({ exitCode: 0, stderr: "", stdout: "/workspace\n", truncated: false }); const harness = createToolLoopHarness( - createTestConfig("conversation", emit, { tools: new Map() }), + createTestConfig("conversation", emit, { + tools: new Map([ + [ + "bash", + { + description: "Run shell commands", + execute: bashExecute, + inputSchema: jsonSchema({ type: "object" }), + name: "bash", + }, + ], + ]), + }), ); const result = await harness(session, { inputResponses: [{ optionId: "approve", requestId: "approval-1" }], }); + expect(bashExecute).toHaveBeenCalledWith( + { command: "pwd" }, + expect.objectContaining({ toolCallId: "call-1" }), + ); + expect(result.session.history.map((msg) => msg.role)).toEqual([ "assistant", "tool", - "tool", "assistant", ]); - const approvalMessage = result.session.history[1]; - expect(Array.isArray(approvalMessage?.content)).toBe(true); - const approvalParts = approvalMessage?.content as Array>; - expect(approvalParts).toHaveLength(1); - expect(approvalParts[0]).toEqual({ + const resolutionMessage = result.session.history[1]; + expect(Array.isArray(resolutionMessage?.content)).toBe(true); + const resolutionParts = resolutionMessage?.content as Array>; + expect(resolutionParts).toHaveLength(2); + expect(resolutionParts[0]).toEqual({ approvalId: "approval-1", approved: true, reason: undefined, type: "tool-approval-response", }); - - const toolResultMessage = result.session.history[2]; - expect(Array.isArray(toolResultMessage?.content)).toBe(true); - const toolResultParts = toolResultMessage?.content as Array>; - expect(toolResultParts).toHaveLength(1); - expect(toolResultParts[0]).toMatchObject({ + expect(resolutionParts[1]).toMatchObject({ toolCallId: "call-1", toolName: "bash", type: "tool-result", }); expect(result.session.history.at(-1)?.role).toBe("assistant"); + + const actionResults = events.filter((event) => event.type === "action.result"); + expect(actionResults).toHaveLength(1); + expect(actionResults[0]?.data).toMatchObject({ + result: { callId: "call-1", kind: "tool-result", toolName: "bash" }, + status: "completed", + }); + }); + + it("executes an approved tool call even when channel context rides the approving prompt", async () => { + // https://github.com/vercel/eve/issues/529 — the Linear channel sends + // `context` on every prompt, including the one answering an approval. + // Owned resume-time execution must not depend on the approval message + // being the request tail. + setupMockAgent({ + content: [], + finishReason: "stop", + response: { messages: [{ content: "Done.", role: "assistant" }] }, + text: "Done.", + toolCalls: [], + toolResults: [], + }); + + const session = createPendingBashApprovalSession(); + const bashExecute = vi.fn().mockResolvedValue("ok"); + const harness = createToolLoopHarness( + createTestConfig("conversation", undefined, { + tools: new Map([ + [ + "bash", + { + description: "Run shell commands", + execute: bashExecute, + inputSchema: jsonSchema({ type: "object" }), + name: "bash", + }, + ], + ]), + }), + ); + + const result = await harness(session, { + context: ["Issue ENG-123"], + inputResponses: [{ optionId: "approve", requestId: "approval-1" }], + }); + + expect(bashExecute).toHaveBeenCalledTimes(1); + + // The closed resolution precedes the context; the context stays on the + // same call. + const roles = result.session.history.map((msg) => msg.role); + expect(roles).toEqual(["assistant", "tool", "user", "assistant"]); + const resolutionParts = result.session.history[1]?.content as Array>; + expect(resolutionParts.map((part) => part.type)).toEqual([ + "tool-approval-response", + "tool-result", + ]); + expect(result.session.history[2]).toEqual({ + content: "Issue ENG-123", + role: "user", + }); + }); + + it("heals a history poisoned by a dangling local tool call before the model call", async () => { + // A dangling `tool_use` in durable history (the #460/#533 poison shape) + // must be closed with a synthetic error result instead of being replayed + // verbatim and 400ing forever. + const generateCalls: unknown[][] = []; + vi.mocked(ToolLoopAgent).mockImplementation(function ( + this: MockAgentInstance, + settings: MockAgentSettings, + ) { + const result = { + finishReason: "stop", + response: { messages: [{ content: "Recovered.", role: "assistant" }] }, + text: "Recovered.", + toolCalls: [], + toolResults: [], + }; + const { onStepFinish } = settings; + this.generate = vi.fn().mockImplementation(async (input: { messages: unknown[] }) => { + generateCalls.push(input.messages); + if (onStepFinish) await onStepFinish(result); + return result; + }); + return this; + } as MockAgentConstructor); + + const session = createTestSession({ + history: [ + { content: "create the object", role: "user" }, + { + content: [ + { + input: { command: "pwd" }, + toolCallId: "dangling-1", + toolName: "bash", + type: "tool-call", + }, + ], + role: "assistant", + }, + ], + }); + + const harness = createToolLoopHarness(createTestConfig("conversation", undefined)); + const result = await harness(session, { message: "try again" }); + + const sent = generateCalls[0] as Array<{ role: string; content: unknown }>; + const toolMessage = sent.find((msg) => msg.role === "tool"); + expect(toolMessage?.content).toEqual([ + { + output: { type: "error-text", value: expect.stringContaining("did not complete") }, + toolCallId: "dangling-1", + toolName: "bash", + type: "tool-result", + }, + ]); + + // The repair is durable: the closed transcript persists into history. + const persistedTool = result.session.history.find((msg) => msg.role === "tool"); + expect(persistedTool).toBeDefined(); }); it("does not persist provider-executed deferred tool-results as generic tool messages", async () => { @@ -6314,7 +6443,7 @@ describe("createToolLoopHarness", () => { expect(events.filter((event) => event.type === "action.result")).toHaveLength(1); }); - it("queues a follow-up user message until the pending tool approval resolves", async () => { + it("resolves a queued follow-up message in the same step as the approval denial", async () => { const generateCalls: unknown[] = []; const agentResults = [ { @@ -6433,7 +6562,9 @@ describe("createToolLoopHarness", () => { inputResponses: [{ requestId: "approval-1", optionId: "deny" }], }); - expect(typeof deniedResult.next).toBe("function"); + // The denial is closed inline, so the queued message rides the same + // model call instead of costing another step. + expect(deniedResult.next).toBeNull(); expect(generateCalls[0]).toEqual([ { content: [ @@ -6471,21 +6602,18 @@ describe("createToolLoopHarness", () => { ], role: "tool", }, + { + content: "Hi instead.", + role: "user", + }, ]); - - const secondResult = await createToolLoopHarness(config)(deniedResult.session); - - expect(secondResult.next).toBeNull(); - expect((generateCalls[1] as { role: string; content: unknown }[]).at(-1)).toEqual({ + expect(hasDeferredStepInput(deniedResult.session)).toBe(false); + expect(deniedResult.session.history.at(-2)).toEqual({ content: "Hi instead.", role: "user", }); - expect(secondResult.session.history.at(-2)).toEqual({ - content: "Hi instead.", - role: "user", - }); - expect(secondResult.session.history.at(-1)).toEqual({ - content: "Hello!", + expect(deniedResult.session.history.at(-1)).toEqual({ + content: "Okay, I will not run that command.", role: "assistant", }); }); @@ -6617,17 +6745,24 @@ describe("createToolLoopHarness", () => { reason: undefined, type: "tool-approval-response", }, + { + output: { type: "text", value: "ok" }, + toolCallId: "call-1", + toolName: "guarded_echo", + type: "tool-result", + }, ], role: "tool", }, ]); }); - it("deferred message lands as last non-system message after explicit approval denial", async () => { + it("deferred message lands as last non-system message in the approval-denial step", async () => { // Step 1: pending approval + user sends a follow-up message. The approval // remains pending and the message is deferred. Step 2: the user denies the - // approval. Step 3: the deferred message is consumed and appears as the - // last message the model sees. + // approval; the denial is closed inline, so the deferred message is + // consumed in the same step and appears as the last message the model + // sees. const generateCalls: Array> = []; const agentResults = [ { @@ -6745,25 +6880,18 @@ describe("createToolLoopHarness", () => { expect(firstResult.next).toBeNull(); expect(generateCalls).toEqual([]); - // Step 2: user denies the approval; the deferred message is NOT in this call. + // Step 2: user denies the approval; the deferred message is consumed in + // the same step and is the last message the model sees. const deniedResult = await createToolLoopHarness(config)(firstResult.session, { inputResponses: [{ requestId: "approval-1", optionId: "deny" }], }); - expect(typeof deniedResult.next).toBe("function"); + expect(deniedResult.next).toBeNull(); const step2Last = generateCalls[0]?.at(-1); - expect(step2Last?.role).toBe("tool"); - - // Step 3: harness consumes the deferred message. - const secondResult = await createToolLoopHarness(config)(deniedResult.session); - expect(secondResult.next).toBeNull(); - - // The deferred user message is the last message the model sees. - const step3Last = generateCalls[1]?.at(-1); - expect(step3Last).toEqual({ content: "Do something else", role: "user" }); + expect(step2Last).toEqual({ content: "Do something else", role: "user" }); // History reflects the full conversation. - expect(secondResult.session.history.at(-1)).toEqual({ - content: "Sure, here you go.", + expect(deniedResult.session.history.at(-1)).toEqual({ + content: "I will not run that command.", role: "assistant", }); }); diff --git a/packages/eve/src/harness/tool-loop.ts b/packages/eve/src/harness/tool-loop.ts index dc7cc06a5..8daed0b2c 100644 --- a/packages/eve/src/harness/tool-loop.ts +++ b/packages/eve/src/harness/tool-loop.ts @@ -97,6 +97,8 @@ import { extractToolApprovalInputRequests, } from "#harness/input-extraction.js"; import { createToolResultMessagePartFromToolError } from "#harness/action-result-helpers.js"; +import { closeApprovedActionBatch } from "#harness/approved-tool-execution.js"; +import { reconcileToolTranscript } from "#harness/transcript-obligations.js"; import { buildTelemetryRuntimeContext } from "#harness/instrumentation-runtime-context.js"; import { consumeDeferredStepInput, @@ -570,6 +572,40 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn { session = pending.session; let messages: ModelMessage[] = pending.messages; + // Direct harness unit tests may run without an ambient context. + const ctx = contextStorage.getStore(); + + // --- Owned closure of approved tool calls ------------------------------- + // + // The harness executes approved parked tool calls itself and appends + // their durable results before assembling the model request. Closure of + // a parked `tool_use` must not depend on the AI SDK's last-message + // approval scan (anything appended after the approval message silently + // disables it) nor on stream capture for result persistence. + if (pending.approvedActions !== undefined) { + const closure = await closeApprovedActionBatch({ + abortSignal: config.abortSignal, + batch: pending.approvedActions, + ctx, + emit, + messages, + tools: config.tools, + }); + messages = closure.messages; + + // Owned execution parks on authorization exactly like in-stream + // execution: the redacted pending output already closed the call. + if (closure.authorizationSignal !== undefined) { + return parkOnAuthorization({ + challenges: closure.authorizationSignal.challenges, + emit, + emissionState, + history: messages, + session, + }); + } + } + // A resolved session-limit continuation prompt grants a fresh token // budget or ends the session; see session-limit-enforcement. const continuation = await applySessionLimitContinuation({ @@ -606,8 +642,6 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn { // --- Model + tools ------------------------------------------------------ - // Direct harness unit tests may run without an ambient context. - const ctx = contextStorage.getStore(); if (ctx !== undefined && config.dispatchDynamicModelEvent !== undefined) { await config.dispatchDynamicModelEvent({ ctx, @@ -649,6 +683,23 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn { telemetry: enrichTelemetry(telemetryConfig, agentName) ?? undefined, })); + // --- Transcript-closure invariant ---------------------------------------- + // + // The single guard: no local `tool-call` may reach a provider without a + // terminal `tool-result`. Runs after all message assembly (pending + // resolution, context/message appends, compaction) so any dangling call + // here is an orphan; closing it durably also heals histories poisoned + // before this invariant existed. + const reconciliation = reconcileToolTranscript(messages); + if (reconciliation.repaired.length > 0) { + log.warn("closed dangling tool calls before model call", { + repaired: reconciliation.repaired, + sessionId: session.sessionId, + turnId: emissionState.turnId, + }); + messages = reconciliation.messages; + } + const approvedTools = getApprovedTools(session); const emptyDeliveryEnabled = @@ -1851,35 +1902,13 @@ async function handleStepResult(input: { const authSignal = findAuthorizationSignalFromToolResults(result.toolResults); if (authSignal) { - const { challenges } = authSignal; - - if (emit) { - for (const ch of challenges) { - await emit( - createAuthorizationRequiredEvent({ - authorization: ch.challenge, - name: ch.name, - description: ch.challenge.instructions ?? `Authorization required for ${ch.name}`, - webhookUrl: ch.hookUrl, - sequence: emissionState.sequence, - stepIndex: emissionState.stepIndex, - turnId: emissionState.turnId, - }), - ); - } - } - - return { - next: null, - session: setHarnessEmissionState( - { - ...baseSession, - history: [...promptMessages], - state: setPendingAuthorization(baseSession.state, { challenges }), - }, - emissionState, - ), - }; + return parkOnAuthorization({ + challenges: authSignal.challenges, + emit, + emissionState, + history: promptMessages, + session: baseSession, + }); } // --- Continue or terminate ------------------------------------------------ @@ -2384,6 +2413,49 @@ async function runModelCallWithRetries( } } +/** + * Emits `authorization.required` for each challenge and parks the session on + * the pending authorization. Shared by resume-time owned execution and the + * post-step in-stream path. + */ +async function parkOnAuthorization(input: { + readonly challenges: AuthorizationSignal["challenges"]; + readonly emit?: ToolLoopHarnessConfig["handleEvent"]; + readonly emissionState: ReturnType; + readonly history: readonly ModelMessage[]; + readonly session: HarnessSession; +}): Promise { + const { challenges, emissionState } = input; + + if (input.emit) { + for (const ch of challenges) { + await input.emit( + createAuthorizationRequiredEvent({ + authorization: ch.challenge, + name: ch.name, + description: ch.challenge.instructions ?? `Authorization required for ${ch.name}`, + webhookUrl: ch.hookUrl, + sequence: emissionState.sequence, + stepIndex: emissionState.stepIndex, + turnId: emissionState.turnId, + }), + ); + } + } + + return { + next: null, + session: setHarnessEmissionState( + { + ...input.session, + history: [...input.history], + state: setPendingAuthorization(input.session.state, { challenges }), + }, + emissionState, + ), + }; +} + function findAuthorizationSignalFromToolResults( toolResults: readonly TypedToolResult[] | undefined, ): AuthorizationSignal | undefined { diff --git a/packages/eve/src/harness/tools.ts b/packages/eve/src/harness/tools.ts index 4431365e9..2b38e5098 100644 --- a/packages/eve/src/harness/tools.ts +++ b/packages/eve/src/harness/tools.ts @@ -89,29 +89,7 @@ export function buildToolSet(input: { }: { readonly output: unknown; readonly toolCallId?: string; - }) => { - if (isAuthorizationPendingModelOutput(output)) { - return { - type: "text" as const, - value: authorizationPendingModelText(output.connections), - }; - } - if (authorToModelOutput !== undefined) { - return normalizeToolModelOutput({ - output: await authorToModelOutput(output), - toolCallId, - toolName: definition.name, - }); - } - if (typeof output === "string") { - return { type: "text" as const, value: output }; - } - return normalizeToolModelOutput({ - output: { type: "json" as const, value: output ?? null }, - toolCallId, - toolName: definition.name, - }); - }, + }) => resolveExecutedToolModelOutput({ definition, output, toolCallId }), } : authorToModelOutput !== undefined ? { @@ -192,6 +170,42 @@ export function wrapToolExecute( }; } +/** + * Model-facing output projection for an executed tool's raw output. The + * single normalization point shared by the AI SDK `toModelOutput` hook (via + * {@link buildToolSet}) and the harness's own resume-time execution of + * approved tool calls, so both paths record identical durable results. + */ +export async function resolveExecutedToolModelOutput(input: { + readonly definition: HarnessToolDefinition; + readonly output: unknown; + readonly toolCallId?: string; +}): Promise { + const { definition, output, toolCallId } = input; + + if (isAuthorizationPendingModelOutput(output)) { + return { + type: "text", + value: authorizationPendingModelText(output.connections), + }; + } + if (definition.toModelOutput !== undefined) { + return normalizeToolModelOutput({ + output: await definition.toModelOutput(output), + toolCallId, + toolName: definition.name, + }); + } + if (typeof output === "string") { + return { type: "text", value: output }; + } + return normalizeToolModelOutput({ + output: { type: "json", value: output ?? null }, + toolCallId, + toolName: definition.name, + }); +} + function normalizeToolJsonOutput(input: { readonly boundary: "execute" | "toModelOutput"; readonly output: unknown; diff --git a/packages/eve/src/harness/transcript-obligations.test.ts b/packages/eve/src/harness/transcript-obligations.test.ts new file mode 100644 index 000000000..1b8b93274 --- /dev/null +++ b/packages/eve/src/harness/transcript-obligations.test.ts @@ -0,0 +1,237 @@ +import type { ModelMessage } from "ai"; +import { describe, expect, it } from "vitest"; + +import { + INTERRUPTED_TOOL_CALL_RESULT, + reconcileToolTranscript, +} from "#harness/transcript-obligations.js"; + +function assistantToolCall(toolCallId: string, providerExecuted?: boolean): ModelMessage { + const part: { + input: unknown; + providerExecuted?: boolean; + toolCallId: string; + toolName: string; + type: "tool-call"; + } = { + input: { command: "pwd" }, + toolCallId, + toolName: "bash", + type: "tool-call", + }; + if (providerExecuted !== undefined) { + part.providerExecuted = providerExecuted; + } + return { content: [part], role: "assistant" }; +} + +function toolResult(toolCallId: string): ModelMessage { + return { + content: [ + { + output: { type: "text", value: "ok" }, + toolCallId, + toolName: "bash", + type: "tool-result", + }, + ], + role: "tool", + }; +} + +describe("reconcileToolTranscript", () => { + it("passes a balanced transcript through unchanged", () => { + const messages: ModelMessage[] = [ + { content: "run pwd", role: "user" }, + assistantToolCall("call-1"), + toolResult("call-1"), + { content: "done", role: "assistant" }, + ]; + + const result = reconcileToolTranscript(messages); + + expect(result.repaired).toEqual([]); + expect(result.messages).toEqual(messages); + }); + + it("closes a dangling local tool call with a synthetic error result in the adjacent message", () => { + const messages: ModelMessage[] = [ + { content: "run pwd", role: "user" }, + assistantToolCall("call-1"), + { content: "anything else?", role: "user" }, + ]; + + const result = reconcileToolTranscript(messages); + + expect(result.repaired).toEqual([{ toolCallId: "call-1", toolName: "bash" }]); + expect(result.messages).toEqual([ + { content: "run pwd", role: "user" }, + assistantToolCall("call-1"), + { + content: [ + { + output: { type: "error-text", value: INTERRUPTED_TOOL_CALL_RESULT }, + toolCallId: "call-1", + toolName: "bash", + type: "tool-result", + }, + ], + role: "tool", + }, + { content: "anything else?", role: "user" }, + ]); + }); + + it("merges the closure into an existing adjacent tool message", () => { + const messages: ModelMessage[] = [ + { + content: [ + { + input: {}, + toolCallId: "call-1", + toolName: "bash", + type: "tool-call", + }, + { + input: {}, + toolCallId: "call-2", + toolName: "bash", + type: "tool-call", + }, + ], + role: "assistant", + }, + toolResult("call-2"), + ]; + + const result = reconcileToolTranscript(messages); + + expect(result.repaired).toEqual([{ toolCallId: "call-1", toolName: "bash" }]); + expect(result.messages).toHaveLength(2); + const closure = result.messages[1]; + expect(closure?.role).toBe("tool"); + expect(closure?.content).toEqual([ + { + output: { type: "error-text", value: INTERRUPTED_TOOL_CALL_RESULT }, + toolCallId: "call-1", + toolName: "bash", + type: "tool-result", + }, + { + output: { type: "text", value: "ok" }, + toolCallId: "call-2", + toolName: "bash", + type: "tool-result", + }, + ]); + }); + + it("closes a dangling call at the end of the transcript", () => { + const messages: ModelMessage[] = [assistantToolCall("call-1")]; + + const result = reconcileToolTranscript(messages); + + expect(result.repaired).toEqual([{ toolCallId: "call-1", toolName: "bash" }]); + expect(result.messages.at(-1)).toEqual({ + content: [ + { + output: { type: "error-text", value: INTERRUPTED_TOOL_CALL_RESULT }, + toolCallId: "call-1", + toolName: "bash", + type: "tool-result", + }, + ], + role: "tool", + }); + }); + + it("ignores provider-executed tool calls", () => { + const messages: ModelMessage[] = [assistantToolCall("call-1", true)]; + + const result = reconcileToolTranscript(messages); + + expect(result.repaired).toEqual([]); + expect(result.messages).toEqual(messages); + }); + + it("counts inline assistant tool-results as closures", () => { + const messages: ModelMessage[] = [ + { + content: [ + { + input: {}, + toolCallId: "call-1", + toolName: "bash", + type: "tool-call", + }, + { + output: { type: "text", value: "ok" }, + toolCallId: "call-1", + toolName: "bash", + type: "tool-result", + }, + ], + role: "assistant", + }, + ]; + + const result = reconcileToolTranscript(messages); + + expect(result.repaired).toEqual([]); + expect(result.messages).toEqual(messages); + }); + + it("closes a call whose approval response never produced a result — no exemptions", () => { + // An approval-response is not a closure: providers strip it, so a call + // with only a response replays as a dangling tool_use. The harness + // closes every approved call at resume; anything still dangling here is + // an orphan. + const messages: ModelMessage[] = [ + { + content: [ + { + input: {}, + toolCallId: "call-1", + toolName: "client_tool", + type: "tool-call", + }, + { + approvalId: "approval-1", + toolCallId: "call-1", + type: "tool-approval-request", + }, + ], + role: "assistant", + }, + { + content: [ + { + approvalId: "approval-1", + approved: true, + type: "tool-approval-response", + }, + ], + role: "tool", + }, + ]; + + const result = reconcileToolTranscript(messages); + + expect(result.repaired).toEqual([{ toolCallId: "call-1", toolName: "client_tool" }]); + const closure = result.messages[1]; + expect(closure?.role).toBe("tool"); + const closureParts = (closure?.content ?? []) as Array>; + expect(closureParts.map((part) => part.type)).toEqual([ + "tool-result", + "tool-approval-response", + ]); + }); + + it("closes each dangling call exactly once when it appears twice", () => { + const messages: ModelMessage[] = [assistantToolCall("call-1"), assistantToolCall("call-1")]; + + const result = reconcileToolTranscript(messages); + + expect(result.repaired).toEqual([{ toolCallId: "call-1", toolName: "bash" }]); + }); +}); diff --git a/packages/eve/src/harness/transcript-obligations.ts b/packages/eve/src/harness/transcript-obligations.ts new file mode 100644 index 000000000..c0d51c2a2 --- /dev/null +++ b/packages/eve/src/harness/transcript-obligations.ts @@ -0,0 +1,113 @@ +import type { ModelMessage } from "ai"; + +type ToolMessage = Extract; +type ToolResponsePart = ToolMessage["content"][number]; +type ToolResultPart = Extract; + +/** + * Synthetic result recorded for a local tool call that reached a model call + * without a terminal result. Phrased for the model: the call must be treated + * as never executed. + */ +export const INTERRUPTED_TOOL_CALL_RESULT = + "Tool execution did not complete: the call was interrupted before a result was recorded. Treat this call as not executed."; + +/** One tool call the reconciler closed with a synthetic result. */ +export interface RepairedToolCall { + readonly toolCallId: string; + readonly toolName: string; +} + +/** + * Enforces the transcript-closure invariant on an assembled model request: + * every non-provider-executed `tool-call` must have a terminal `tool-result` + * before the transcript reaches a provider, or the provider rejects the + * replay (Anthropic: `tool_use` ids without `tool_result` blocks; OpenAI + * Responses: `No tool output found for function call`). + * + * Dangling calls are closed durably with a synthetic error result placed in + * the message immediately after the assistant message that carries the call + * (merged into an existing tool message, or inserted as a new one), so the + * repair satisfies provider adjacency rules. This also heals sessions whose + * durable history was already poisoned by a missed closure. + * + * There are no exemptions: the harness closes every parked local call at + * resume (see `approved-tool-execution.ts`), so anything still dangling here + * is an orphan nothing else will close. + */ +export function reconcileToolTranscript(messages: readonly ModelMessage[]): { + readonly messages: ModelMessage[]; + readonly repaired: readonly RepairedToolCall[]; +} { + const closedCallIds = collectClosedCallIds(messages); + + const repaired: RepairedToolCall[] = []; + const result: ModelMessage[] = []; + let pendingClosures: ToolResultPart[] = []; + + const flushPendingClosures = (next: ModelMessage | undefined): ModelMessage | undefined => { + if (pendingClosures.length === 0) { + return next; + } + const closures = pendingClosures; + pendingClosures = []; + if (next !== undefined && next.role === "tool" && Array.isArray(next.content)) { + return { ...next, content: [...closures, ...next.content] }; + } + result.push({ content: closures, role: "tool" }); + return next; + }; + + for (const message of messages) { + const merged = flushPendingClosures(message); + if (merged !== undefined) { + result.push(merged); + } + + if (message.role !== "assistant" || !Array.isArray(message.content)) { + continue; + } + + for (const part of message.content) { + if (part.type !== "tool-call" || part.providerExecuted === true) { + continue; + } + if (closedCallIds.has(part.toolCallId)) { + continue; + } + + closedCallIds.add(part.toolCallId); + repaired.push({ toolCallId: part.toolCallId, toolName: part.toolName }); + pendingClosures.push({ + output: { type: "error-text", value: INTERRUPTED_TOOL_CALL_RESULT }, + toolCallId: part.toolCallId, + toolName: part.toolName, + type: "tool-result", + }); + } + } + + flushPendingClosures(undefined); + + return { messages: result, repaired }; +} + +function collectClosedCallIds(messages: readonly ModelMessage[]): Set { + const callIds = new Set(); + + for (const message of messages) { + if (message.role !== "tool" && message.role !== "assistant") { + continue; + } + if (!Array.isArray(message.content)) { + continue; + } + for (const part of message.content) { + if (part.type === "tool-result") { + callIds.add(part.toolCallId); + } + } + } + + return callIds; +} diff --git a/research/approval-resume-owned-execution.md b/research/approval-resume-owned-execution.md index 1039ae0ee..5cc305463 100644 --- a/research/approval-resume-owned-execution.md +++ b/research/approval-resume-owned-execution.md @@ -1,6 +1,6 @@ --- issue: https://github.com/vercel/eve/issues/460 -status: proposed +status: in-review last_updated: "2026-07-07" --- From 32214dcc79112399d1113f773c45d2ff0df2fb48 Mon Sep 17 00:00:00 2001 From: Rui Conti Date: Wed, 8 Jul 2026 17:14:06 -0400 Subject: [PATCH 2/5] refactor(eve): fold invalid-input closure into the transcript primitive MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Generalize the reconciler into closeDanglingToolCalls(messages, resolveClosure): one primitive that finds every local tool-call without a terminal tool-result and records the supplied closure adjacent to the calling assistant message. Calls whose closure resolves to undefined pass through untouched, so legitimately open obligations stay open. Every synthesized transcript closure now flows through it: - handleStepResult closes invalid-input tool calls (#576's sibling obligation) with their specific, model-actionable parse error, placed adjacently instead of via the insertInlineToolResultMessages trailing append. The emission-layer trailingInlineToolResultParts push for invalid input is dropped — that site only emits the action.result event; closure is re-derived from the step result via getInvalidToolCallInputErrors. - reconcileToolTranscript becomes a thin caller that closes every remaining dangler with the generic interrupted text. Also trims the red-eval descriptions and updates the research doc's point-fix history to record the fold. Signed-off-by: Rui Conti --- .../hitl/approve-with-client-context.eval.ts | 2 +- .../dynamic-approve-then-followup.eval.ts | 2 +- .../hitl/slow-once-approve-replay.eval.ts | 2 +- packages/eve/src/harness/emission.test.ts | 11 +-- packages/eve/src/harness/emission.ts | 4 +- packages/eve/src/harness/tool-loop.ts | 21 +++-- .../harness/transcript-obligations.test.ts | 45 +++++++++++ .../eve/src/harness/transcript-obligations.ts | 81 ++++++++++++++----- research/approval-resume-owned-execution.md | 37 ++++++--- 9 files changed, 152 insertions(+), 53 deletions(-) diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/approve-with-client-context.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/approve-with-client-context.eval.ts index b80798c62..6fb6c0f60 100644 --- a/e2e/fixtures/agent-tools-hitl/evals/hitl/approve-with-client-context.eval.ts +++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/approve-with-client-context.eval.ts @@ -14,7 +14,7 @@ import { GUARDED_ECHO_TOKEN } from "./shared.js"; * prevent the approved tool from executing. */ export default defineEval({ - description: "HITL red (#529): approval answered with channel context still executes the tool.", + description: "#529: approval answered with channel context still executes the tool.", async test(t) { const parked = await t.send('Call the guarded-echo tool with note "context-approve".'); parked.calledTool("guarded-echo", { status: "pending", count: 1 }); diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/dynamic-approve-then-followup.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/dynamic-approve-then-followup.eval.ts index a14da2725..454584f02 100644 --- a/e2e/fixtures/agent-tools-hitl/evals/hitl/dynamic-approve-then-followup.eval.ts +++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/dynamic-approve-then-followup.eval.ts @@ -15,7 +15,7 @@ const TOOL_NAME = "dynamic_guarded_echo"; * Expected once fixed: the follow-up turn succeeds. */ export default defineEval({ - description: "HITL red (#533): a resolved approval park replays cleanly on the next turn.", + description: "#533: resolved approval park replays cleanly on the next turn.", async test(t) { const parked = await t.send(`Call the \`${TOOL_NAME}\` tool with note "replay-probe".`); parked.calledTool(TOOL_NAME, { status: "pending", count: 1 }); diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/slow-once-approve-replay.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/slow-once-approve-replay.eval.ts index 316ad8725..40e9b7602 100644 --- a/e2e/fixtures/agent-tools-hitl/evals/hitl/slow-once-approve-replay.eval.ts +++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/slow-once-approve-replay.eval.ts @@ -16,7 +16,7 @@ const TOOL_NAME = "guarded-slow-echo"; * fixed: the follow-up turn succeeds. */ export default defineEval({ - description: "HITL red (#460): approved slow tool's result survives into replayed history.", + description: "#460: approved slow tool's result survives into replayed history.", async test(t) { const parked = await t.send( `Call the ${TOOL_NAME} tool with note "alpha". After its result arrives, call it again with note "beta" — strictly one call at a time, never in parallel. When both results are in, reply with exactly SLOW-DONE.`, diff --git a/packages/eve/src/harness/emission.test.ts b/packages/eve/src/harness/emission.test.ts index 602c41573..b27db19de 100644 --- a/packages/eve/src/harness/emission.test.ts +++ b/packages/eve/src/harness/emission.test.ts @@ -546,14 +546,9 @@ describe("emitStreamContent action requests", () => { }), ]); expect([...result.invalidInputToolCallIds]).toEqual(["call-bad"]); - expect(result.trailingInlineToolResultParts).toEqual([ - { - output: { type: "error-text", value: message }, - toolCallId: "call-bad", - toolName: "web_search", - type: "tool-result", - }, - ]); + // Transcript closure is owned by the step-result handler via + // `closeDanglingToolCalls`; the stream layer only emits the event. + expect(result.trailingInlineToolResultParts).toEqual([]); }); }); diff --git a/packages/eve/src/harness/emission.ts b/packages/eve/src/harness/emission.ts index 69593e05a..a06b72c08 100644 --- a/packages/eve/src/harness/emission.ts +++ b/packages/eve/src/harness/emission.ts @@ -478,7 +478,9 @@ export async function emitStreamContent( } await emitActionResult(createRuntimeToolResultFromToolError(toolError)); handledInlineToolResultCallIds.add(toolCall.toolCallId); - trailingInlineToolResultParts.push(createToolResultMessagePartFromToolError(toolError)); + // Transcript closure is owned by `handleStepResult`, which re-derives + // the same error via `getInvalidToolCallInputErrors` and closes the + // call through `closeDanglingToolCalls`. This site only observes. return; } throw error; diff --git a/packages/eve/src/harness/tool-loop.ts b/packages/eve/src/harness/tool-loop.ts index 8daed0b2c..f86dc3e53 100644 --- a/packages/eve/src/harness/tool-loop.ts +++ b/packages/eve/src/harness/tool-loop.ts @@ -98,7 +98,10 @@ import { } from "#harness/input-extraction.js"; import { createToolResultMessagePartFromToolError } from "#harness/action-result-helpers.js"; import { closeApprovedActionBatch } from "#harness/approved-tool-execution.js"; -import { reconcileToolTranscript } from "#harness/transcript-obligations.js"; +import { + closeDanglingToolCalls, + reconcileToolTranscript, +} from "#harness/transcript-obligations.js"; import { buildTelemetryRuntimeContext } from "#harness/instrumentation-runtime-context.js"; import { consumeDeferredStepInput, @@ -1751,15 +1754,17 @@ async function handleStepResult(input: { ...(result.invalidInputToolCallIds ?? []), ...invalidInputToolErrors.map((toolError) => toolError.toolCallId), ]); + const invalidInputClosures = new Map( + invalidInputToolErrors.map((toolError) => [ + toolError.toolCallId, + createToolResultMessagePartFromToolError(toolError).output, + ]), + ); const rawResponseMessages = emptyDelivery ? [] - : insertInlineToolResultMessages({ - append: invalidInputToolErrors.map((toolError) => - createToolResultMessagePartFromToolError(toolError), - ), - prepend: [], - responseMessages: result.response.messages, - }); + : closeDanglingToolCalls(result.response.messages, (call) => + invalidInputClosures.get(call.toolCallId), + ).messages; const stepOutput = emptyDelivery ? null : resolvedStepOutput; const providerExecutedOutcomeIds = new Set(); diff --git a/packages/eve/src/harness/transcript-obligations.test.ts b/packages/eve/src/harness/transcript-obligations.test.ts index 1b8b93274..e213c6f8c 100644 --- a/packages/eve/src/harness/transcript-obligations.test.ts +++ b/packages/eve/src/harness/transcript-obligations.test.ts @@ -2,6 +2,7 @@ import type { ModelMessage } from "ai"; import { describe, expect, it } from "vitest"; import { + closeDanglingToolCalls, INTERRUPTED_TOOL_CALL_RESULT, reconcileToolTranscript, } from "#harness/transcript-obligations.js"; @@ -235,3 +236,47 @@ describe("reconcileToolTranscript", () => { expect(result.repaired).toEqual([{ toolCallId: "call-1", toolName: "bash" }]); }); }); + +describe("closeDanglingToolCalls", () => { + it("closes only the calls the resolver supplies an output for, with that output", () => { + const messages: ModelMessage[] = [ + assistantToolCall("call-invalid"), + assistantToolCall("call-parked"), + ]; + + const result = closeDanglingToolCalls(messages, (call) => + call.toolCallId === "call-invalid" + ? { type: "error-text", value: "Failed to parse tool-call arguments" } + : undefined, + ); + + expect(result.closed).toEqual([{ toolCallId: "call-invalid", toolName: "bash" }]); + expect(result.messages).toEqual([ + messages[0], + { + content: [ + { + output: { type: "error-text", value: "Failed to parse tool-call arguments" }, + toolCallId: "call-invalid", + toolName: "bash", + type: "tool-result", + }, + ], + role: "tool", + }, + messages[1], + ]); + }); + + it("does not re-close a call that already has a tool result", () => { + const messages: ModelMessage[] = [assistantToolCall("call-1"), toolResult("call-1")]; + + const result = closeDanglingToolCalls(messages, () => ({ + type: "error-text", + value: "should not appear", + })); + + expect(result.closed).toEqual([]); + expect(result.messages).toEqual(messages); + }); +}); diff --git a/packages/eve/src/harness/transcript-obligations.ts b/packages/eve/src/harness/transcript-obligations.ts index c0d51c2a2..97f8d96a0 100644 --- a/packages/eve/src/harness/transcript-obligations.ts +++ b/packages/eve/src/harness/transcript-obligations.ts @@ -3,6 +3,7 @@ import type { ModelMessage } from "ai"; type ToolMessage = Extract; type ToolResponsePart = ToolMessage["content"][number]; type ToolResultPart = Extract; +type ToolResultOutput = ToolResultPart["output"]; /** * Synthetic result recorded for a local tool call that reached a model call @@ -12,36 +13,38 @@ type ToolResultPart = Extract; export const INTERRUPTED_TOOL_CALL_RESULT = "Tool execution did not complete: the call was interrupted before a result was recorded. Treat this call as not executed."; -/** One tool call the reconciler closed with a synthetic result. */ -export interface RepairedToolCall { +/** One local tool call found without a terminal `tool-result`. */ +export interface DanglingToolCall { readonly toolCallId: string; readonly toolName: string; } /** - * Enforces the transcript-closure invariant on an assembled model request: - * every non-provider-executed `tool-call` must have a terminal `tool-result` - * before the transcript reaches a provider, or the provider rejects the - * replay (Anthropic: `tool_use` ids without `tool_result` blocks; OpenAI - * Responses: `No tool output found for function call`). - * - * Dangling calls are closed durably with a synthetic error result placed in - * the message immediately after the assistant message that carries the call - * (merged into an existing tool message, or inserted as a new one), so the - * repair satisfies provider adjacency rules. This also heals sessions whose - * durable history was already poisoned by a missed closure. + * The one primitive that closes a local `tool-call`'s transcript obligation. + * Walks the messages, finds every non-provider-executed `tool-call` without + * a `tool-result`, and — when `resolveClosure` supplies an output for it — + * records that closure in the message immediately after the assistant + * message carrying the call (merged into an existing tool message, or + * inserted as a new one), satisfying provider adjacency rules. Calls whose + * closure resolves to `undefined` are left dangling, so legitimately open + * obligations (parked approvals, pending runtime actions) pass through + * untouched. * - * There are no exemptions: the harness closes every parked local call at - * resume (see `approved-tool-execution.ts`), so anything still dangling here - * is an orphan nothing else will close. + * Every transcript closure the harness synthesizes goes through here: the + * step-time closure of invalid-input tool calls (detailed, model-actionable + * error text) and the request-assembly guard + * {@link reconcileToolTranscript} (generic interrupted text). */ -export function reconcileToolTranscript(messages: readonly ModelMessage[]): { +export function closeDanglingToolCalls( + messages: readonly ModelMessage[], + resolveClosure: (call: DanglingToolCall) => ToolResultOutput | undefined, +): { + readonly closed: readonly DanglingToolCall[]; readonly messages: ModelMessage[]; - readonly repaired: readonly RepairedToolCall[]; } { const closedCallIds = collectClosedCallIds(messages); - const repaired: RepairedToolCall[] = []; + const closed: DanglingToolCall[] = []; const result: ModelMessage[] = []; let pendingClosures: ToolResultPart[] = []; @@ -76,10 +79,16 @@ export function reconcileToolTranscript(messages: readonly ModelMessage[]): { continue; } + const call: DanglingToolCall = { toolCallId: part.toolCallId, toolName: part.toolName }; + const output = resolveClosure(call); + if (output === undefined) { + continue; + } + closedCallIds.add(part.toolCallId); - repaired.push({ toolCallId: part.toolCallId, toolName: part.toolName }); + closed.push(call); pendingClosures.push({ - output: { type: "error-text", value: INTERRUPTED_TOOL_CALL_RESULT }, + output, toolCallId: part.toolCallId, toolName: part.toolName, type: "tool-result", @@ -89,7 +98,35 @@ export function reconcileToolTranscript(messages: readonly ModelMessage[]): { flushPendingClosures(undefined); - return { messages: result, repaired }; + return { closed, messages: result }; +} + +/** + * Enforces the transcript-closure invariant on an assembled model request: + * every non-provider-executed `tool-call` must have a terminal `tool-result` + * before the transcript reaches a provider, or the provider rejects the + * replay (Anthropic: `tool_use` ids without `tool_result` blocks; OpenAI + * Responses: `No tool output found for function call`). + * + * Dangling calls are closed durably with a synthetic error result via + * {@link closeDanglingToolCalls}. This also heals sessions whose durable + * history was already poisoned by a missed closure. + * + * There are no exemptions: the harness closes every parked local call at + * resume (see `approved-tool-execution.ts`) and every invalid-input call at + * step time (see `handleStepResult` in `tool-loop.ts`), so anything still + * dangling here is an orphan nothing else will close. + */ +export function reconcileToolTranscript(messages: readonly ModelMessage[]): { + readonly messages: ModelMessage[]; + readonly repaired: readonly DanglingToolCall[]; +} { + const { closed, messages: reconciled } = closeDanglingToolCalls(messages, () => ({ + type: "error-text", + value: INTERRUPTED_TOOL_CALL_RESULT, + })); + + return { messages: reconciled, repaired: closed }; } function collectClosedCallIds(messages: readonly ModelMessage[]): Set { diff --git a/research/approval-resume-owned-execution.md b/research/approval-resume-owned-execution.md index 5cc305463..27c5073fc 100644 --- a/research/approval-resume-owned-execution.md +++ b/research/approval-resume-owned-execution.md @@ -33,6 +33,18 @@ message-assembly ordering in the tool loop. Because the logic is decoupled, the invariant is never stated and never checked — each seam holds only as long as its neighbors behave. +Tiny examples of the mismatch: + +- Deny: eve writes `tool-result(call_1, execution-denied)` during resume. +- Approve: eve writes `tool-approval-response(call_1)` during resume and + waits for the AI SDK to later produce `tool-result(call_1, output)`. +- Approve with a follow-up `message`: eve defers the message so the approval + tool message stays last, because the SDK only scans the tail. +- Approve with `context`: eve appends the context after the approval, so the + SDK tail scan sees a user message and runs no tool. +- Approved tool execution: the durable `tool-result` reaches history through + stream capture, not through the resume code that accepted the approval. + ### Seam 1: the deny arm closes in eve, the approve arm closes in the AI SDK Denial is closed by eve, inline, at resume @@ -168,25 +180,28 @@ Each prior fix patched one seam without owning the obligation: _message_ to keep the approval message the tail (seam 3, `message` only — `context` was missed). - `bd287b17` (#576) — spliced synthetic results for invalid-input tool calls - (a sibling obligation, closed in yet another place). + through a separate append path. This branch folds that sibling obligation + into the same `closeDanglingToolCalls` primitive used by the replay guard, + while preserving the invalid-input error text the model can act on. -The pattern is the diagnosis: every fix adds one more closure site, in a -different module, guarding one more entry point. The invariant — one terminal -result per parked call before any replay — still exists nowhere in code, so -each new entry point (a channel adding context, a slower `execute`, a -stricter provider) breaks it again. +The pattern is the diagnosis: each point fix added one more closure site, in a +different module, guarding one more entry point. Before this branch, the +invariant — one terminal result per parked call before any replay — existed +nowhere as an owned harness rule, so each new entry point (a channel adding +context, a slower `execute`, a stricter provider) could break it again. ## Direction Centralize both halves of the obligation in the harness: -1. **One closure moment.** eve closes every parked local tool call itself at +1. **One closure primitive.** eve closes every parked local tool call itself at resume — executing approved tools directly (the execution wrapper already exists: `wrapToolExecute` in `harness/tools.ts`) and appending the durable - `tool-result`/`tool-error` alongside the existing denial synthesis. No - dependency on the SDK's tail scan or on stream capture for - obligation-closing results. This also removes the reason the - message/context deferral machinery exists. + `tool-result`/`tool-error` alongside the existing denial synthesis. The + same primitive closes invalid-input calls at step time with their specific + parse error. No dependency on the SDK's tail scan or on stream capture for + obligation-closing results. This also removes the reason the message/context + deferral machinery exists. 2. **One guard.** A single pure reconciliation pass over the assembled messages, immediately before every model call: no local `tool-call` without a terminal `tool-result` may reach the wire. Dangling calls are From cfb7d5b5bef1df725babc849f9015407cbd8d422 Mon Sep 17 00:00:00 2001 From: Rui Conti Date: Wed, 8 Jul 2026 18:50:32 -0400 Subject: [PATCH 3/5] refactor(eve): flatten the transcript-closure primitive Review follow-ups, behavior-preserving: - Delete the reconcileToolTranscript wrapper: it had become a constant resolver plus a closed->repaired field rename. The tool loop's guard calls closeDanglingToolCalls directly with the interrupted-text policy; the invariant documentation moves onto the primitive. - Replace the flush-callback walk with an indexed lookahead loop so the provider adjacency rule reads literally: push the message, then merge its closures into the next tool message or insert one. - Define 'approved response' once: collectApprovedRequestIds feeds both recordApprovedTools and buildApprovedActionBatch instead of each re-deriving the predicate. Signed-off-by: Rui Conti --- packages/eve/src/harness/input-requests.ts | 27 ++-- packages/eve/src/harness/tool-loop.ts | 11 +- .../harness/transcript-obligations.test.ts | 16 +- .../eve/src/harness/transcript-obligations.ts | 145 ++++++++---------- 4 files changed, 99 insertions(+), 100 deletions(-) diff --git a/packages/eve/src/harness/input-requests.ts b/packages/eve/src/harness/input-requests.ts index 3f418ee74..65c2b0772 100644 --- a/packages/eve/src/harness/input-requests.ts +++ b/packages/eve/src/harness/input-requests.ts @@ -201,11 +201,13 @@ export function resolvePendingInput(input: { responses, }); + const approvedRequestIds = collectApprovedRequestIds(responses); + // Record approved tools before clearing the batch. session = recordApprovedTools({ + approvedRequestIds, pendingBatch, resolveApprovalKey: input.resolveApprovalKey, - responses, session, }); @@ -218,7 +220,7 @@ export function resolvePendingInput(input: { } const rejectedActions = buildRejectedActionBatch(pendingBatch, responses); - const approvedActions = buildApprovedActionBatch(pendingBatch, responses); + const approvedActions = buildApprovedActionBatch(pendingBatch, approvedRequestIds); session = clearPendingInputBatch(session); return { @@ -427,23 +429,24 @@ export function getApprovedTools(session: HarnessSession): ReadonlySet { return new Set(value as string[]); } +/** The single definition of an approved response in one resolved batch. */ +function collectApprovedRequestIds(responses: readonly InputResponse[]): ReadonlySet { + return new Set(responses.filter((r) => r.optionId === "approve").map((r) => r.requestId)); +} + /** * Resolves the approval key for a request. When a `resolveApprovalKey` * function is provided and returns a string, that compound key is recorded * instead of the bare tool name. */ function recordApprovedTools(input: { + readonly approvedRequestIds: ReadonlySet; readonly pendingBatch: PendingInputBatch; readonly resolveApprovalKey?: (request: InputRequest) => string | undefined; - readonly responses: readonly InputResponse[]; readonly session: HarnessSession; }): HarnessSession { - const approvedIds = new Set( - input.responses.filter((r) => r.optionId === "approve").map((r) => r.requestId), - ); - const newKeys = input.pendingBatch.requests - .filter((r) => approvedIds.has(r.requestId)) + .filter((r) => input.approvedRequestIds.has(r.requestId)) .map((r) => input.resolveApprovalKey?.(r) ?? r.action.toolName); if (newKeys.length === 0) { @@ -554,14 +557,10 @@ function buildRejectedActionBatch( */ function buildApprovedActionBatch( batch: PendingInputBatch, - responses: readonly InputResponse[], + approvedRequestIds: ReadonlySet, ): ApprovedActionBatch | undefined { - const approvedIds = new Set( - responses.filter((r) => r.optionId === "approve").map((r) => r.requestId), - ); - const calls = batch.requests - .filter((request) => isApprovalRequest(request) && approvedIds.has(request.requestId)) + .filter((request) => isApprovalRequest(request) && approvedRequestIds.has(request.requestId)) .map((request) => request.action); return calls.length > 0 ? { calls, event: batch.event } : undefined; diff --git a/packages/eve/src/harness/tool-loop.ts b/packages/eve/src/harness/tool-loop.ts index f86dc3e53..b8df50022 100644 --- a/packages/eve/src/harness/tool-loop.ts +++ b/packages/eve/src/harness/tool-loop.ts @@ -100,7 +100,7 @@ import { createToolResultMessagePartFromToolError } from "#harness/action-result import { closeApprovedActionBatch } from "#harness/approved-tool-execution.js"; import { closeDanglingToolCalls, - reconcileToolTranscript, + INTERRUPTED_TOOL_CALL_RESULT, } from "#harness/transcript-obligations.js"; import { buildTelemetryRuntimeContext } from "#harness/instrumentation-runtime-context.js"; import { @@ -693,10 +693,13 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn { // resolution, context/message appends, compaction) so any dangling call // here is an orphan; closing it durably also heals histories poisoned // before this invariant existed. - const reconciliation = reconcileToolTranscript(messages); - if (reconciliation.repaired.length > 0) { + const reconciliation = closeDanglingToolCalls(messages, () => ({ + type: "error-text", + value: INTERRUPTED_TOOL_CALL_RESULT, + })); + if (reconciliation.closed.length > 0) { log.warn("closed dangling tool calls before model call", { - repaired: reconciliation.repaired, + repaired: reconciliation.closed, sessionId: session.sessionId, turnId: emissionState.turnId, }); diff --git a/packages/eve/src/harness/transcript-obligations.test.ts b/packages/eve/src/harness/transcript-obligations.test.ts index e213c6f8c..440e245e8 100644 --- a/packages/eve/src/harness/transcript-obligations.test.ts +++ b/packages/eve/src/harness/transcript-obligations.test.ts @@ -3,8 +3,8 @@ import { describe, expect, it } from "vitest"; import { closeDanglingToolCalls, + type DanglingToolCall, INTERRUPTED_TOOL_CALL_RESULT, - reconcileToolTranscript, } from "#harness/transcript-obligations.js"; function assistantToolCall(toolCallId: string, providerExecuted?: boolean): ModelMessage { @@ -40,7 +40,19 @@ function toolResult(toolCallId: string): ModelMessage { }; } -describe("reconcileToolTranscript", () => { +/** The request-assembly guard's closure policy, as wired in the tool loop. */ +function reconcileToolTranscript(messages: ModelMessage[]): { + messages: ModelMessage[]; + repaired: readonly DanglingToolCall[]; +} { + const { closed, messages: reconciled } = closeDanglingToolCalls(messages, () => ({ + type: "error-text", + value: INTERRUPTED_TOOL_CALL_RESULT, + })); + return { messages: reconciled, repaired: closed }; +} + +describe("closeDanglingToolCalls with the guard policy", () => { it("passes a balanced transcript through unchanged", () => { const messages: ModelMessage[] = [ { content: "run pwd", role: "user" }, diff --git a/packages/eve/src/harness/transcript-obligations.ts b/packages/eve/src/harness/transcript-obligations.ts index 97f8d96a0..1b5459dcd 100644 --- a/packages/eve/src/harness/transcript-obligations.ts +++ b/packages/eve/src/harness/transcript-obligations.ts @@ -21,19 +21,26 @@ export interface DanglingToolCall { /** * The one primitive that closes a local `tool-call`'s transcript obligation. - * Walks the messages, finds every non-provider-executed `tool-call` without - * a `tool-result`, and — when `resolveClosure` supplies an output for it — - * records that closure in the message immediately after the assistant - * message carrying the call (merged into an existing tool message, or - * inserted as a new one), satisfying provider adjacency rules. Calls whose - * closure resolves to `undefined` are left dangling, so legitimately open - * obligations (parked approvals, pending runtime actions) pass through - * untouched. + * + * Providers reject a replayed `tool_use` without a `tool_result` (Anthropic: + * `tool_use` ids without `tool_result` blocks; OpenAI Responses: `No tool + * output found for function call`), so every non-provider-executed + * `tool-call` must be closed before the transcript reaches a provider. + * + * Walks the messages, finds every local `tool-call` without a `tool-result`, + * and — when `resolveClosure` supplies an output for it — records that + * closure in the message immediately after the assistant message carrying + * the call (merged into an existing tool message, or inserted as a new one), + * satisfying provider adjacency rules. Calls whose closure resolves to + * `undefined` are left dangling, so legitimately open obligations (parked + * approvals, pending runtime actions) pass through untouched. * * Every transcript closure the harness synthesizes goes through here: the * step-time closure of invalid-input tool calls (detailed, model-actionable - * error text) and the request-assembly guard - * {@link reconcileToolTranscript} (generic interrupted text). + * error text) and the request-assembly guard in the tool loop, which closes + * any remaining orphan with {@link INTERRUPTED_TOOL_CALL_RESULT} — durably, + * so histories poisoned by a missed closure heal instead of replaying the + * same provider rejection forever. */ export function closeDanglingToolCalls( messages: readonly ModelMessage[], @@ -43,90 +50,68 @@ export function closeDanglingToolCalls( readonly messages: ModelMessage[]; } { const closedCallIds = collectClosedCallIds(messages); - const closed: DanglingToolCall[] = []; const result: ModelMessage[] = []; - let pendingClosures: ToolResultPart[] = []; - const flushPendingClosures = (next: ModelMessage | undefined): ModelMessage | undefined => { - if (pendingClosures.length === 0) { - return next; - } - const closures = pendingClosures; - pendingClosures = []; - if (next !== undefined && next.role === "tool" && Array.isArray(next.content)) { - return { ...next, content: [...closures, ...next.content] }; - } - result.push({ content: closures, role: "tool" }); - return next; - }; - - for (const message of messages) { - const merged = flushPendingClosures(message); - if (merged !== undefined) { - result.push(merged); - } + for (let index = 0; index < messages.length; index += 1) { + const message = messages[index] as ModelMessage; + result.push(message); - if (message.role !== "assistant" || !Array.isArray(message.content)) { + const closures = collectClosures(message, closedCallIds, resolveClosure, closed); + if (closures.length === 0) { continue; } - for (const part of message.content) { - if (part.type !== "tool-call" || part.providerExecuted === true) { - continue; - } - if (closedCallIds.has(part.toolCallId)) { - continue; - } - - const call: DanglingToolCall = { toolCallId: part.toolCallId, toolName: part.toolName }; - const output = resolveClosure(call); - if (output === undefined) { - continue; - } - - closedCallIds.add(part.toolCallId); - closed.push(call); - pendingClosures.push({ - output, - toolCallId: part.toolCallId, - toolName: part.toolName, - type: "tool-result", - }); + // Providers require a call's result in the message immediately after the + // assistant message that carries the call. + const next = messages[index + 1]; + if (next !== undefined && next.role === "tool" && Array.isArray(next.content)) { + result.push({ ...next, content: [...closures, ...next.content] }); + index += 1; + } else { + result.push({ content: closures, role: "tool" }); } } - flushPendingClosures(undefined); - return { closed, messages: result }; } -/** - * Enforces the transcript-closure invariant on an assembled model request: - * every non-provider-executed `tool-call` must have a terminal `tool-result` - * before the transcript reaches a provider, or the provider rejects the - * replay (Anthropic: `tool_use` ids without `tool_result` blocks; OpenAI - * Responses: `No tool output found for function call`). - * - * Dangling calls are closed durably with a synthetic error result via - * {@link closeDanglingToolCalls}. This also heals sessions whose durable - * history was already poisoned by a missed closure. - * - * There are no exemptions: the harness closes every parked local call at - * resume (see `approved-tool-execution.ts`) and every invalid-input call at - * step time (see `handleStepResult` in `tool-loop.ts`), so anything still - * dangling here is an orphan nothing else will close. - */ -export function reconcileToolTranscript(messages: readonly ModelMessage[]): { - readonly messages: ModelMessage[]; - readonly repaired: readonly DanglingToolCall[]; -} { - const { closed, messages: reconciled } = closeDanglingToolCalls(messages, () => ({ - type: "error-text", - value: INTERRUPTED_TOOL_CALL_RESULT, - })); +function collectClosures( + message: ModelMessage, + closedCallIds: Set, + resolveClosure: (call: DanglingToolCall) => ToolResultOutput | undefined, + closed: DanglingToolCall[], +): ToolResultPart[] { + if (message.role !== "assistant" || !Array.isArray(message.content)) { + return []; + } + + const closures: ToolResultPart[] = []; + for (const part of message.content) { + if (part.type !== "tool-call" || part.providerExecuted === true) { + continue; + } + if (closedCallIds.has(part.toolCallId)) { + continue; + } + + const call: DanglingToolCall = { toolCallId: part.toolCallId, toolName: part.toolName }; + const output = resolveClosure(call); + if (output === undefined) { + continue; + } + + closedCallIds.add(part.toolCallId); + closed.push(call); + closures.push({ + output, + toolCallId: part.toolCallId, + toolName: part.toolName, + type: "tool-result", + }); + } - return { messages: reconciled, repaired: closed }; + return closures; } function collectClosedCallIds(messages: readonly ModelMessage[]): Set { From a9ade2cd6b808c0e8bc7cd8aa5b107513dcd8af3 Mon Sep 17 00:00:00 2001 From: Rui Conti Date: Thu, 9 Jul 2026 14:58:44 -0400 Subject: [PATCH 4/5] fix(eve): preserve approval batch authorization state Classify resume-time authorization parks before emitting terminal action results, and retain challenges from every approved call. Route approved, denied, and answered input results through call-ID transcript closure so results stay adjacent to their originating calls. Signed-off-by: Rui Conti --- .../harness/approved-tool-execution.test.ts | 114 ++++++++++++----- .../src/harness/approved-tool-execution.ts | 91 +++++++------- packages/eve/src/harness/input-requests.ts | 119 ++++++++---------- .../tool-loop-compaction-accounting.test.ts | 14 ++- packages/eve/src/harness/tool-loop.test.ts | 1 + .../eve/src/harness/transcript-obligations.ts | 20 +-- 6 files changed, 202 insertions(+), 157 deletions(-) diff --git a/packages/eve/src/harness/approved-tool-execution.test.ts b/packages/eve/src/harness/approved-tool-execution.test.ts index d539a6c21..379d10580 100644 --- a/packages/eve/src/harness/approved-tool-execution.test.ts +++ b/packages/eve/src/harness/approved-tool-execution.test.ts @@ -1,9 +1,11 @@ import { jsonSchema, type ModelMessage } from "ai"; import { describe, expect, it, vi } from "vitest"; +import { ContextContainer, contextStorage } from "#context/container.js"; +import { requestAuthorization } from "#harness/authorization.js"; import type { HarnessToolDefinition } from "#harness/execute-tool.js"; import { - appendExecutedToolResults, + closeApprovedActionBatch, executeApprovedToolCalls, } from "#harness/approved-tool-execution.js"; @@ -135,46 +137,90 @@ describe("executeApprovedToolCalls", () => { }); }); -describe("appendExecutedToolResults", () => { - const part = { - output: { type: "text", value: "ok" }, - toolCallId: "call-1", - toolName: "echo", - type: "tool-result", - } as const; - - it("merges results into the trailing tool message", () => { - const messages: ModelMessage[] = [ - { content: "hi", role: "user" }, +describe("closeApprovedActionBatch", () => { + it("preserves every authorization challenge raised by an approved batch", async () => { + const firstSignal = requestAuthorization([ { - content: [{ approvalId: "approval-1", approved: true, type: "tool-approval-response" }], - role: "tool", + challenge: { url: "https://idp.example/first" }, + hookUrl: "https://app.example/first/callback", + name: "first_connection", + resume: { nonce: "first" }, }, - ]; - - const result = appendExecutedToolResults(messages, [part]); - - expect(result).toHaveLength(2); - expect(result[1]).toEqual({ - content: [{ approvalId: "approval-1", approved: true, type: "tool-approval-response" }, part], - role: "tool", - }); - }); - - it("appends a new tool message when the transcript does not end in one", () => { - const messages: ModelMessage[] = [{ content: "hi", role: "user" }]; + ]); + const secondSignal = requestAuthorization([ + { + challenge: { url: "https://idp.example/second" }, + hookUrl: "https://app.example/second/callback", + name: "second_connection", + resume: { nonce: "second" }, + }, + ]); + const ctx = new ContextContainer(); + const tools = new Map([ + ["first", createDefinition({ execute: () => firstSignal, name: "first" })], + ["second", createDefinition({ execute: () => secondSignal, name: "second" })], + ]); - const result = appendExecutedToolResults(messages, [part]); + const result = await contextStorage.run(ctx, () => + closeApprovedActionBatch({ + batch: { + calls: [ + { ...call, callId: "call-first", toolName: "first" }, + { ...call, callId: "call-second", toolName: "second" }, + ], + }, + ctx, + messages: [], + tools, + }), + ); - expect(result).toEqual([ - { content: "hi", role: "user" }, - { content: [part], role: "tool" }, + expect(result.authorizationSignal?.challenges).toEqual([ + ...firstSignal.challenges, + ...secondSignal.challenges, ]); }); - it("returns a copy when there is nothing to append", () => { - const messages: ModelMessage[] = [{ content: "hi", role: "user" }]; + it("places a result beside its matching call when later input already follows", async () => { + const assistant: ModelMessage = { + content: [ + { + input: { note: "hi" }, + toolCallId: "call-1", + toolName: "echo", + type: "tool-call", + }, + ], + role: "assistant", + }; + const approval: ModelMessage = { + content: [{ approvalId: "approval-1", approved: true, type: "tool-approval-response" }], + role: "tool", + }; + const laterInput: ModelMessage = { content: "additional context", role: "user" }; + + const result = await closeApprovedActionBatch({ + batch: { calls: [call] }, + ctx: undefined, + messages: [assistant, approval, laterInput], + tools: new Map([["echo", createDefinition({ execute: () => "ok" })]]), + }); - expect(appendExecutedToolResults(messages, [])).toEqual(messages); + expect(result.messages).toEqual([ + assistant, + { + content: [ + { approvalId: "approval-1", approved: true, type: "tool-approval-response" }, + { + output: { type: "text", value: "ok" }, + toolCallId: "call-1", + toolName: "echo", + type: "tool-result", + }, + ], + role: "tool", + }, + laterInput, + ]); }); }); diff --git a/packages/eve/src/harness/approved-tool-execution.ts b/packages/eve/src/harness/approved-tool-execution.ts index b20505626..52b7e20f6 100644 --- a/packages/eve/src/harness/approved-tool-execution.ts +++ b/packages/eve/src/harness/approved-tool-execution.ts @@ -10,10 +10,15 @@ import type { } from "#runtime/actions/types.js"; import { toError } from "#shared/errors.js"; import { createRuntimeToolResultFromValue } from "#harness/action-result-helpers.js"; -import { type AuthorizationSignal, isAuthorizationSignal } from "#harness/authorization.js"; +import { + type AuthorizationSignal, + isAuthorizationSignal, + requestAuthorization, +} from "#harness/authorization.js"; import type { HarnessToolDefinition } from "#harness/execute-tool.js"; import type { ApprovedActionBatch } from "#harness/input-requests.js"; import { readToolInterrupt } from "#harness/tool-interrupts.js"; +import { closeDanglingToolCalls } from "#harness/transcript-obligations.js"; import { resolveExecutedToolModelOutput, wrapToolExecute } from "#harness/tools.js"; import type { HarnessEmitFn, HarnessToolMap } from "#harness/types.js"; @@ -38,10 +43,11 @@ export interface ApprovedToolExecutionOutcome { /** * Closes one resolved approval batch: executes the approved calls, appends - * their durable results to the transcript, and emits `action.result` events - * against the originating turn's stream coordinates. Returns the closed - * transcript plus the first {@link AuthorizationSignal} an executed tool - * raised, so the caller can park exactly like in-stream execution. + * their durable results to the transcript, and emits terminal `action.result` + * events against the originating turn's stream coordinates. Returns the + * closed transcript plus one combined {@link AuthorizationSignal} for every + * authorization challenge the batch raised, so the caller can park exactly + * like in-stream execution. * * Dynamic tools take precedence over authored tools of the same name, * mirroring the toolset override order used for model calls. @@ -76,16 +82,22 @@ export async function closeApprovedActionBatch(input: { return { messages: [...input.messages] }; } - const messages = appendExecutedToolResults( - input.messages, - executed.map((outcome) => outcome.part), + const classified = classifyExecutedToolCalls(ctx, executed); + const closures = new Map( + executed.map((outcome) => [outcome.part.toolCallId, outcome.part.output]), ); + const messages = closeDanglingToolCalls(input.messages, (call) => closures.get(call.toolCallId), { + placement: "after-existing", + }).messages; if (input.emit !== undefined && batch.event !== undefined) { - for (const outcome of executed) { + for (const result of classified) { + if (result.kind === "authorization-pending") { + continue; + } await input.emit( createActionResultEvent({ - result: outcome.actionResult, + result: result.outcome.actionResult, sequence: batch.event.sequence, stepIndex: batch.event.stepIndex, turnId: batch.event.turnId, @@ -94,30 +106,37 @@ export async function closeApprovedActionBatch(input: { } } + const challenges = classified.flatMap((result) => + result.kind === "authorization-pending" ? result.signal.challenges : [], + ); + return { - authorizationSignal: findStashedAuthorizationSignal(ctx, executed), + authorizationSignal: challenges.length > 0 ? requestAuthorization(challenges) : undefined, messages, }; } -/** - * Reads back the first authorization signal stashed by `wrapToolExecute` - * during resume-time execution. - */ -function findStashedAuthorizationSignal( +type ClassifiedExecutedToolCall = + | { + readonly kind: "authorization-pending"; + readonly outcome: ExecutedApprovedToolCall; + readonly signal: AuthorizationSignal; + } + | { readonly kind: "completed"; readonly outcome: ExecutedApprovedToolCall }; + +/** Separates terminal results from authorization parks before event emission. */ +function classifyExecutedToolCalls( ctx: AlsContext | undefined, executed: readonly ExecutedApprovedToolCall[], -): AuthorizationSignal | undefined { - if (ctx === undefined) { - return undefined; - } - for (const outcome of executed) { - const stashed = readToolInterrupt(ctx, outcome.actionResult.callId); +): ClassifiedExecutedToolCall[] { + return executed.map((outcome) => { + const stashed = + ctx === undefined ? undefined : readToolInterrupt(ctx, outcome.actionResult.callId); if (stashed !== undefined && isAuthorizationSignal(stashed)) { - return stashed; + return { kind: "authorization-pending", outcome, signal: stashed }; } - } - return undefined; + return { kind: "completed", outcome }; + }); } /** @@ -220,25 +239,3 @@ function buildErrorOutcome( }, }; } - -/** - * Merges resume-time closures into the trailing tool message (the one - * carrying the approval responses) so every `tool_use` and its result stay - * within adjacent messages, as providers require. Appends a new tool message - * when the transcript does not end in one. - */ -export function appendExecutedToolResults( - messages: readonly ModelMessage[], - parts: readonly ToolResultPart[], -): ModelMessage[] { - if (parts.length === 0) { - return [...messages]; - } - - const last = messages.at(-1); - if (last !== undefined && last.role === "tool" && Array.isArray(last.content)) { - return [...messages.slice(0, -1), { ...last, content: [...last.content, ...parts] }]; - } - - return [...messages, { content: [...parts], role: "tool" }]; -} diff --git a/packages/eve/src/harness/input-requests.ts b/packages/eve/src/harness/input-requests.ts index 65c2b0772..8a478ba4e 100644 --- a/packages/eve/src/harness/input-requests.ts +++ b/packages/eve/src/harness/input-requests.ts @@ -12,6 +12,7 @@ import { isSessionLimitContinuationRequest, resolveSessionLimitContinuation, } from "#harness/session-limit-continuation.js"; +import { closeDanglingToolCalls } from "#harness/transcript-obligations.js"; import type { HarnessSession, SessionStateMap, StepInput } from "#harness/types.js"; const PENDING_INPUT_BATCH_KEY = "eve.runtime.pendingInputBatch"; @@ -25,6 +26,7 @@ const TOOL_EXECUTION_DENIED_MESSAGE = "Tool execution was denied."; const TOOL_EXECUTION_INVALID_APPROVAL_MESSAGE = "Invalid approval response."; type ToolResponsePart = Extract["content"][number]; +type ToolResultOutput = Extract["output"]; /** * Stream-emit coordinates carried so a parked batch's resolution can attribute @@ -178,11 +180,7 @@ export function resolvePendingInput(input: { // A follow-up message arrived for question-only input with no explicit // responses. Keep the existing question semantics: mark unanswered // question requests ignored so the model can continue with the message. - const toolParts = buildToolResponseParts(pendingBatch, []); - const messages: ModelMessage[] = [...baseHistory, ...pendingBatch.responseMessages]; - if (toolParts.length > 0) { - messages.push({ content: toolParts, role: "tool" }); - } + const messages = buildResolvedInputMessages(baseHistory, pendingBatch, []); const rejectedActions = buildRejectedActionBatch(pendingBatch, []); session = clearPendingInputBatch(session); @@ -211,13 +209,7 @@ export function resolvePendingInput(input: { session, }); - // Build tool result messages from responses. - const toolParts = buildToolResponseParts(pendingBatch, responses); - - const messages: ModelMessage[] = [...baseHistory, ...pendingBatch.responseMessages]; - if (toolParts.length > 0) { - messages.push({ content: toolParts, role: "tool" }); - } + const messages = buildResolvedInputMessages(baseHistory, pendingBatch, responses); const rejectedActions = buildRejectedActionBatch(pendingBatch, responses); const approvedActions = buildApprovedActionBatch(pendingBatch, approvedRequestIds); @@ -566,76 +558,67 @@ function buildApprovedActionBatch( return calls.length > 0 ? { calls, event: batch.event } : undefined; } -function buildToolResponseParts( +/** Resolves input metadata and closes every answered or denied tool call. */ +function buildResolvedInputMessages( + baseHistory: readonly ModelMessage[], batch: PendingInputBatch, responses: readonly InputResponse[], -): ToolResponsePart[] { - const responseMap = new Map(responses.map((r) => [r.requestId, r])); - - const parts: ToolResponsePart[] = []; - for (const request of batch.requests) { - parts.push(...buildToolResponsePartsForRequest(request, responseMap.get(request.requestId))); +): ModelMessage[] { + const { closures, responseParts } = buildInputResolutionParts(batch, responses); + const messages: ModelMessage[] = [...baseHistory, ...batch.responseMessages]; + if (responseParts.length > 0) { + messages.push({ content: [...responseParts], role: "tool" }); } - return parts; + + return closeDanglingToolCalls(messages, (call) => closures.get(call.toolCallId), { + placement: "after-existing", + }).messages; } -function buildToolResponsePartsForRequest( - request: InputRequest, - response: InputResponse | undefined, -): ToolResponsePart[] { - // A session-limit continuation prompt is harness-authored: no matching - // tool call exists in model history, so resolving it must not append a - // tool message the provider would reject as unmatched. This is currently - // the only harness-authored request type; if another appears, replace this - // toolName predicate with a generic synthetic-request marker instead of - // stacking a second special case here. - if (isSessionLimitContinuationRequest(request)) { - return []; - } +function buildInputResolutionParts( + batch: PendingInputBatch, + responses: readonly InputResponse[], +): { + readonly closures: ReadonlyMap; + readonly responseParts: readonly ToolResponsePart[]; +} { + const responseMap = new Map(responses.map((r) => [r.requestId, r])); + const closures = new Map(); + const responseParts: ToolResponsePart[] = []; - if (isApprovalRequest(request)) { - const { approved, reason } = resolveApprovalOutcome(response); - const parts: ToolResponsePart[] = [ - { + for (const request of batch.requests) { + // A session-limit continuation prompt is harness-authored: no matching + // tool call exists in model history, so resolving it must not append a + // tool message the provider would reject as unmatched. + if (isSessionLimitContinuationRequest(request)) { + continue; + } + + const response = responseMap.get(request.requestId); + if (isApprovalRequest(request)) { + const { approved, reason } = resolveApprovalOutcome(response); + responseParts.push({ approvalId: request.requestId, approved, reason, type: "tool-approval-response", - }, - ]; - /* - * The harness owns closing every parked tool call: providers reject a - * replayed `tool_use` without a `tool_result`, and the persisted - * `tool-approval-response` is stripped during provider prompt - * conversion, so it is not a closure. Denials are closed here with the - * matching `execution-denied` result; approved calls are executed and - * closed by the tool loop (see `approvedActions`). - */ - if (!approved) { - parts.push({ - output: { type: "execution-denied", reason }, - toolCallId: request.action.callId, - toolName: request.action.toolName, - type: "tool-result", }); + if (!approved) { + closures.set(request.action.callId, { type: "execution-denied", reason }); + } + continue; } - return parts; + + closures.set(request.action.callId, { + type: "json", + value: + response !== undefined + ? { optionId: response.optionId, text: response.text, status: "answered" } + : { status: "ignored" }, + }); } - return [ - { - output: { - type: "json", - value: - response !== undefined - ? { optionId: response.optionId, text: response.text, status: "answered" } - : { status: "ignored" }, - }, - toolCallId: request.action.callId, - toolName: request.action.toolName, - type: "tool-result", - }, - ]; + return { closures, responseParts }; } function isApprovalRequest(request: InputRequest): boolean { diff --git a/packages/eve/src/harness/tool-loop-compaction-accounting.test.ts b/packages/eve/src/harness/tool-loop-compaction-accounting.test.ts index 0ff388cfb..efd693c19 100644 --- a/packages/eve/src/harness/tool-loop-compaction-accounting.test.ts +++ b/packages/eve/src/harness/tool-loop-compaction-accounting.test.ts @@ -250,7 +250,19 @@ describe("tool-loop structured compaction accounting", () => { requestId: "question-call", }, ], - responseMessages: [], + responseMessages: [ + { + content: [ + { + input: { prompt: "Pick one." }, + toolCallId: "question-call", + toolName: "ask_question", + type: "tool-call", + }, + ], + role: "assistant", + }, + ], session: createTestSession({ compaction: { lastKnownInputTokens: 100, diff --git a/packages/eve/src/harness/tool-loop.test.ts b/packages/eve/src/harness/tool-loop.test.ts index ba694a977..4d13f3291 100644 --- a/packages/eve/src/harness/tool-loop.test.ts +++ b/packages/eve/src/harness/tool-loop.test.ts @@ -455,6 +455,7 @@ function createPendingBashApprovalSession(): HarnessSession { function createPendingProtectedActionApprovalSession(): HarnessSession { return setPendingInputBatch({ + event: { sequence: 7, stepIndex: 0, turnId: "turn-1" }, requests: [ { action: { diff --git a/packages/eve/src/harness/transcript-obligations.ts b/packages/eve/src/harness/transcript-obligations.ts index 1b5459dcd..952ece33d 100644 --- a/packages/eve/src/harness/transcript-obligations.ts +++ b/packages/eve/src/harness/transcript-obligations.ts @@ -35,16 +35,18 @@ export interface DanglingToolCall { * `undefined` are left dangling, so legitimately open obligations (parked * approvals, pending runtime actions) pass through untouched. * - * Every transcript closure the harness synthesizes goes through here: the - * step-time closure of invalid-input tool calls (detailed, model-actionable - * error text) and the request-assembly guard in the tool loop, which closes - * any remaining orphan with {@link INTERRUPTED_TOOL_CALL_RESULT} — durably, - * so histories poisoned by a missed closure heal instead of replaying the - * same provider rejection forever. + * Every transcript closure the harness synthesizes goes through here: + * answered input requests, approved and denied calls, invalid-input calls, + * and the request-assembly guard, which closes any remaining orphan with + * {@link INTERRUPTED_TOOL_CALL_RESULT}. Recording the guard's closure durably + * heals histories poisoned by a missed result. */ export function closeDanglingToolCalls( messages: readonly ModelMessage[], resolveClosure: (call: DanglingToolCall) => ToolResultOutput | undefined, + options: { readonly placement: "after-existing" | "before-existing" } = { + placement: "before-existing", + }, ): { readonly closed: readonly DanglingToolCall[]; readonly messages: ModelMessage[]; @@ -66,7 +68,11 @@ export function closeDanglingToolCalls( // assistant message that carries the call. const next = messages[index + 1]; if (next !== undefined && next.role === "tool" && Array.isArray(next.content)) { - result.push({ ...next, content: [...closures, ...next.content] }); + const content = + options.placement === "after-existing" + ? [...next.content, ...closures] + : [...closures, ...next.content]; + result.push({ ...next, content }); index += 1; } else { result.push({ content: closures, role: "tool" }); From 166ef3f36d5d6a5bd2ea0b3582155ac5891d3d18 Mon Sep 17 00:00:00 2001 From: Rui Conti Date: Thu, 9 Jul 2026 15:54:47 -0400 Subject: [PATCH 5/5] revert(eve): defer HITL approval execution to AI SDK Remove the Eve-owned executor and transcript-closure changes from #590. Approval collection with trailing context belongs in AI SDK and is now tracked in vercel/ai#17033; the stacked red-eval base remains unchanged. Signed-off-by: Rui Conti --- .changeset/hitl-approval-owned-closure.md | 5 - docs/tools/human-in-the-loop.md | 2 +- .../hitl/approve-with-client-context.eval.ts | 2 +- .../dynamic-approve-then-followup.eval.ts | 2 +- .../hitl/slow-once-approve-replay.eval.ts | 2 +- .../harness/approved-tool-execution.test.ts | 226 -------------- .../src/harness/approved-tool-execution.ts | 241 -------------- packages/eve/src/harness/emission.test.ts | 11 +- packages/eve/src/harness/emission.ts | 4 +- .../eve/src/harness/input-requests.test.ts | 18 +- packages/eve/src/harness/input-requests.ts | 211 +++++++------ .../tool-loop-compaction-accounting.test.ts | 14 +- packages/eve/src/harness/tool-loop.test.ts | 241 ++++---------- packages/eve/src/harness/tool-loop.ts | 156 +++------- packages/eve/src/harness/tools.ts | 60 ++-- .../harness/transcript-obligations.test.ts | 294 ------------------ .../eve/src/harness/transcript-obligations.ts | 141 --------- research/approval-resume-owned-execution.md | 39 +-- 18 files changed, 265 insertions(+), 1404 deletions(-) delete mode 100644 .changeset/hitl-approval-owned-closure.md delete mode 100644 packages/eve/src/harness/approved-tool-execution.test.ts delete mode 100644 packages/eve/src/harness/approved-tool-execution.ts delete mode 100644 packages/eve/src/harness/transcript-obligations.test.ts delete mode 100644 packages/eve/src/harness/transcript-obligations.ts diff --git a/.changeset/hitl-approval-owned-closure.md b/.changeset/hitl-approval-owned-closure.md deleted file mode 100644 index 2c0bdefc8..000000000 --- a/.changeset/hitl-approval-owned-closure.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -"eve": patch ---- - -Approval-gated tools now execute during the resume itself: eve runs the approved tool and records its result before the next model call, and every outbound model request is checked so a tool call can never replay without a result. This fixes approve-resume failing with provider 400s (Anthropic `tool_use` without `tool_result`, OpenAI `No tool output found for function call`) — including when a channel attaches context to the approving prompt — and heals sessions whose history already carries a dangling tool call. A message sent together with an approval response now lands in the same model call instead of a deferred follow-up step. diff --git a/docs/tools/human-in-the-loop.md b/docs/tools/human-in-the-loop.md index e01cbe438..3d0e1db4f 100644 --- a/docs/tools/human-in-the-loop.md +++ b/docs/tools/human-in-the-loop.md @@ -102,7 +102,7 @@ Approvals and questions share one protocol: 3. The turn parks at `session.waiting`, durably, for as long as it takes. 4. The client answers with `inputResponses` (structured, keyed by `requestId`) or a normal follow-up `message`. A follow-up whose text matches an option ID, option label, or numeric option index resolves automatically, including approval options such as `approve` and `deny`. -The run picks back up exactly where it parked. On an approval grant, eve executes the tool during the resume itself and records its result before the next model call; on a denial it records the denial the same way. A parked tool call is always resolved before the model sees the transcript again. Because the pause is durable, nothing is held in memory while it waits — the process can restart and the parked turn survives. +The run picks back up exactly where it parked. Because the pause is durable, nothing is held in memory while it waits — the process can restart and the parked turn survives. For approval requests, unrelated follow-up text does not deny the tool call. eve keeps the approval pending and holds that text until the approval is answered, then replays it as the next message in the session. diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/approve-with-client-context.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/approve-with-client-context.eval.ts index 6fb6c0f60..b80798c62 100644 --- a/e2e/fixtures/agent-tools-hitl/evals/hitl/approve-with-client-context.eval.ts +++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/approve-with-client-context.eval.ts @@ -14,7 +14,7 @@ import { GUARDED_ECHO_TOKEN } from "./shared.js"; * prevent the approved tool from executing. */ export default defineEval({ - description: "#529: approval answered with channel context still executes the tool.", + description: "HITL red (#529): approval answered with channel context still executes the tool.", async test(t) { const parked = await t.send('Call the guarded-echo tool with note "context-approve".'); parked.calledTool("guarded-echo", { status: "pending", count: 1 }); diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/dynamic-approve-then-followup.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/dynamic-approve-then-followup.eval.ts index 454584f02..a14da2725 100644 --- a/e2e/fixtures/agent-tools-hitl/evals/hitl/dynamic-approve-then-followup.eval.ts +++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/dynamic-approve-then-followup.eval.ts @@ -15,7 +15,7 @@ const TOOL_NAME = "dynamic_guarded_echo"; * Expected once fixed: the follow-up turn succeeds. */ export default defineEval({ - description: "#533: resolved approval park replays cleanly on the next turn.", + description: "HITL red (#533): a resolved approval park replays cleanly on the next turn.", async test(t) { const parked = await t.send(`Call the \`${TOOL_NAME}\` tool with note "replay-probe".`); parked.calledTool(TOOL_NAME, { status: "pending", count: 1 }); diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/slow-once-approve-replay.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/slow-once-approve-replay.eval.ts index 40e9b7602..316ad8725 100644 --- a/e2e/fixtures/agent-tools-hitl/evals/hitl/slow-once-approve-replay.eval.ts +++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/slow-once-approve-replay.eval.ts @@ -16,7 +16,7 @@ const TOOL_NAME = "guarded-slow-echo"; * fixed: the follow-up turn succeeds. */ export default defineEval({ - description: "#460: approved slow tool's result survives into replayed history.", + description: "HITL red (#460): approved slow tool's result survives into replayed history.", async test(t) { const parked = await t.send( `Call the ${TOOL_NAME} tool with note "alpha". After its result arrives, call it again with note "beta" — strictly one call at a time, never in parallel. When both results are in, reply with exactly SLOW-DONE.`, diff --git a/packages/eve/src/harness/approved-tool-execution.test.ts b/packages/eve/src/harness/approved-tool-execution.test.ts deleted file mode 100644 index 379d10580..000000000 --- a/packages/eve/src/harness/approved-tool-execution.test.ts +++ /dev/null @@ -1,226 +0,0 @@ -import { jsonSchema, type ModelMessage } from "ai"; -import { describe, expect, it, vi } from "vitest"; - -import { ContextContainer, contextStorage } from "#context/container.js"; -import { requestAuthorization } from "#harness/authorization.js"; -import type { HarnessToolDefinition } from "#harness/execute-tool.js"; -import { - closeApprovedActionBatch, - executeApprovedToolCalls, -} from "#harness/approved-tool-execution.js"; - -function createDefinition(overrides: Partial = {}): HarnessToolDefinition { - return { - description: "Echo", - inputSchema: jsonSchema({ type: "object" }), - name: "echo", - ...overrides, - }; -} - -const call = { - callId: "call-1", - input: { note: "hi" }, - kind: "tool-call", - toolName: "echo", -} as const; - -describe("executeApprovedToolCalls", () => { - it("executes the approved call and returns a durable result part plus action.result payload", async () => { - const execute = vi.fn().mockResolvedValue({ echoed: "hi" }); - const outcome = await executeApprovedToolCalls({ - calls: [call], - messages: [], - resolveTool: () => createDefinition({ execute }), - }); - - expect(execute).toHaveBeenCalledWith( - { note: "hi" }, - expect.objectContaining({ toolCallId: "call-1" }), - ); - expect(outcome.executed).toHaveLength(1); - expect(outcome.executed[0]?.part).toEqual({ - output: { type: "json", value: { echoed: "hi" } }, - toolCallId: "call-1", - toolName: "echo", - type: "tool-result", - }); - expect(outcome.executed[0]?.actionResult).toEqual({ - callId: "call-1", - kind: "tool-result", - output: { echoed: "hi" }, - toolName: "echo", - }); - }); - - it("applies the author's toModelOutput to the durable result", async () => { - const outcome = await executeApprovedToolCalls({ - calls: [call], - messages: [], - resolveTool: () => - createDefinition({ - execute: () => ({ echoed: "hi" }), - toModelOutput: () => ({ type: "text", value: "echoed hi" }), - }), - }); - - expect(outcome.executed[0]?.part.output).toEqual({ type: "text", value: "echoed hi" }); - }); - - it("closes a throwing execute with an error result instead of leaving the call dangling", async () => { - const outcome = await executeApprovedToolCalls({ - calls: [call], - messages: [], - resolveTool: () => - createDefinition({ - execute: () => { - throw new Error("boom"); - }, - }), - }); - - expect(outcome.executed[0]?.part.output).toEqual({ type: "error-text", value: "boom" }); - expect(outcome.executed[0]?.actionResult).toMatchObject({ - callId: "call-1", - isError: true, - output: "boom", - }); - }); - - it("closes an approved call whose tool no longer resolves", async () => { - const outcome = await executeApprovedToolCalls({ - calls: [call], - messages: [], - resolveTool: () => undefined, - }); - - expect(outcome.executed[0]?.part.output).toEqual({ - type: "error-text", - value: 'Tool "echo" is no longer available.', - }); - expect(outcome.executed[0]?.actionResult).toMatchObject({ isError: true }); - }); - - it("closes an execute-less tool with an error result instead of leaving it dangling", async () => { - const outcome = await executeApprovedToolCalls({ - calls: [call], - messages: [], - resolveTool: () => createDefinition(), - }); - - expect(outcome.executed[0]?.part.output).toEqual({ - type: "error-text", - value: 'Tool "echo" has no local execution and cannot run.', - }); - expect(outcome.executed[0]?.actionResult).toMatchObject({ isError: true }); - }); - - it("executes calls sequentially in batch order", async () => { - const order: string[] = []; - const outcome = await executeApprovedToolCalls({ - calls: [call, { ...call, callId: "call-2", input: { note: "second" } }], - messages: [], - resolveTool: () => - createDefinition({ - execute: async (input: { note: string }) => { - order.push(input.note); - return input.note; - }, - }), - }); - - expect(order).toEqual(["hi", "second"]); - expect(outcome.executed.map((entry) => entry.part.output)).toEqual([ - { type: "text", value: "hi" }, - { type: "text", value: "second" }, - ]); - }); -}); - -describe("closeApprovedActionBatch", () => { - it("preserves every authorization challenge raised by an approved batch", async () => { - const firstSignal = requestAuthorization([ - { - challenge: { url: "https://idp.example/first" }, - hookUrl: "https://app.example/first/callback", - name: "first_connection", - resume: { nonce: "first" }, - }, - ]); - const secondSignal = requestAuthorization([ - { - challenge: { url: "https://idp.example/second" }, - hookUrl: "https://app.example/second/callback", - name: "second_connection", - resume: { nonce: "second" }, - }, - ]); - const ctx = new ContextContainer(); - const tools = new Map([ - ["first", createDefinition({ execute: () => firstSignal, name: "first" })], - ["second", createDefinition({ execute: () => secondSignal, name: "second" })], - ]); - - const result = await contextStorage.run(ctx, () => - closeApprovedActionBatch({ - batch: { - calls: [ - { ...call, callId: "call-first", toolName: "first" }, - { ...call, callId: "call-second", toolName: "second" }, - ], - }, - ctx, - messages: [], - tools, - }), - ); - - expect(result.authorizationSignal?.challenges).toEqual([ - ...firstSignal.challenges, - ...secondSignal.challenges, - ]); - }); - - it("places a result beside its matching call when later input already follows", async () => { - const assistant: ModelMessage = { - content: [ - { - input: { note: "hi" }, - toolCallId: "call-1", - toolName: "echo", - type: "tool-call", - }, - ], - role: "assistant", - }; - const approval: ModelMessage = { - content: [{ approvalId: "approval-1", approved: true, type: "tool-approval-response" }], - role: "tool", - }; - const laterInput: ModelMessage = { content: "additional context", role: "user" }; - - const result = await closeApprovedActionBatch({ - batch: { calls: [call] }, - ctx: undefined, - messages: [assistant, approval, laterInput], - tools: new Map([["echo", createDefinition({ execute: () => "ok" })]]), - }); - - expect(result.messages).toEqual([ - assistant, - { - content: [ - { approvalId: "approval-1", approved: true, type: "tool-approval-response" }, - { - output: { type: "text", value: "ok" }, - toolCallId: "call-1", - toolName: "echo", - type: "tool-result", - }, - ], - role: "tool", - }, - laterInput, - ]); - }); -}); diff --git a/packages/eve/src/harness/approved-tool-execution.ts b/packages/eve/src/harness/approved-tool-execution.ts deleted file mode 100644 index 52b7e20f6..000000000 --- a/packages/eve/src/harness/approved-tool-execution.ts +++ /dev/null @@ -1,241 +0,0 @@ -import type { ModelMessage } from "ai"; - -import { createLogger, logError } from "#internal/logging.js"; -import { buildDynamicTools } from "#context/build-dynamic-tools.js"; -import type { AlsContext } from "#context/container.js"; -import { createActionResultEvent } from "#protocol/message.js"; -import type { - RuntimeToolCallActionRequest, - RuntimeToolResultActionResult, -} from "#runtime/actions/types.js"; -import { toError } from "#shared/errors.js"; -import { createRuntimeToolResultFromValue } from "#harness/action-result-helpers.js"; -import { - type AuthorizationSignal, - isAuthorizationSignal, - requestAuthorization, -} from "#harness/authorization.js"; -import type { HarnessToolDefinition } from "#harness/execute-tool.js"; -import type { ApprovedActionBatch } from "#harness/input-requests.js"; -import { readToolInterrupt } from "#harness/tool-interrupts.js"; -import { closeDanglingToolCalls } from "#harness/transcript-obligations.js"; -import { resolveExecutedToolModelOutput, wrapToolExecute } from "#harness/tools.js"; -import type { HarnessEmitFn, HarnessToolMap } from "#harness/types.js"; - -type ToolMessage = Extract; -type ToolResponsePart = ToolMessage["content"][number]; -type ToolResultPart = Extract; - -const log = createLogger("harness.approved-tool-execution"); - -/** One approved tool call the harness executed (or failed) at resume time. */ -export interface ExecutedApprovedToolCall { - /** `action.result` projection, attributed to the parked batch's turn. */ - readonly actionResult: RuntimeToolResultActionResult; - /** Durable transcript closure for the call's `tool_use` id. */ - readonly part: ToolResultPart; -} - -/** Outcome of executing one resolved approval batch's approved calls. */ -export interface ApprovedToolExecutionOutcome { - readonly executed: readonly ExecutedApprovedToolCall[]; -} - -/** - * Closes one resolved approval batch: executes the approved calls, appends - * their durable results to the transcript, and emits terminal `action.result` - * events against the originating turn's stream coordinates. Returns the - * closed transcript plus one combined {@link AuthorizationSignal} for every - * authorization challenge the batch raised, so the caller can park exactly - * like in-stream execution. - * - * Dynamic tools take precedence over authored tools of the same name, - * mirroring the toolset override order used for model calls. - */ -export async function closeApprovedActionBatch(input: { - readonly abortSignal?: AbortSignal; - readonly batch: ApprovedActionBatch; - readonly ctx: AlsContext | undefined; - readonly emit?: HarnessEmitFn; - readonly messages: readonly ModelMessage[]; - readonly tools: HarnessToolMap; -}): Promise<{ - readonly authorizationSignal?: AuthorizationSignal; - readonly messages: ModelMessage[]; -}> { - const { batch, ctx } = input; - - const { executed } = await executeApprovedToolCalls({ - abortSignal: input.abortSignal, - calls: batch.calls, - messages: input.messages, - resolveTool: (toolName) => { - const dynamicDefinition = - ctx === undefined - ? undefined - : buildDynamicTools(ctx).find((definition) => definition.name === toolName); - return dynamicDefinition ?? input.tools.get(toolName); - }, - }); - - if (executed.length === 0) { - return { messages: [...input.messages] }; - } - - const classified = classifyExecutedToolCalls(ctx, executed); - const closures = new Map( - executed.map((outcome) => [outcome.part.toolCallId, outcome.part.output]), - ); - const messages = closeDanglingToolCalls(input.messages, (call) => closures.get(call.toolCallId), { - placement: "after-existing", - }).messages; - - if (input.emit !== undefined && batch.event !== undefined) { - for (const result of classified) { - if (result.kind === "authorization-pending") { - continue; - } - await input.emit( - createActionResultEvent({ - result: result.outcome.actionResult, - sequence: batch.event.sequence, - stepIndex: batch.event.stepIndex, - turnId: batch.event.turnId, - }), - ); - } - } - - const challenges = classified.flatMap((result) => - result.kind === "authorization-pending" ? result.signal.challenges : [], - ); - - return { - authorizationSignal: challenges.length > 0 ? requestAuthorization(challenges) : undefined, - messages, - }; -} - -type ClassifiedExecutedToolCall = - | { - readonly kind: "authorization-pending"; - readonly outcome: ExecutedApprovedToolCall; - readonly signal: AuthorizationSignal; - } - | { readonly kind: "completed"; readonly outcome: ExecutedApprovedToolCall }; - -/** Separates terminal results from authorization parks before event emission. */ -function classifyExecutedToolCalls( - ctx: AlsContext | undefined, - executed: readonly ExecutedApprovedToolCall[], -): ClassifiedExecutedToolCall[] { - return executed.map((outcome) => { - const stashed = - ctx === undefined ? undefined : readToolInterrupt(ctx, outcome.actionResult.callId); - if (stashed !== undefined && isAuthorizationSignal(stashed)) { - return { kind: "authorization-pending", outcome, signal: stashed }; - } - return { kind: "completed", outcome }; - }); -} - -/** - * Executes approved tool calls at resume time, in batch order, so the parked - * `tool_use` obligations are closed by the harness itself — durably and - * before any model request is assembled — instead of being delegated to the - * AI SDK's last-message approval scan and stream capture. - * - * Every approved call yields exactly one terminal closure — no exceptions: - * the tool's real output, an `error-text` result when `execute` throws, an - * `error-text` result when the approved tool no longer resolves, and an - * `error-text` result when the tool has no local `execute` (nothing else in - * the pipeline reliably closes an execute-less call, and an unclosed call is - * a guaranteed provider rejection). A tool that signals an authorization - * park keeps its existing `wrapToolExecute` semantics: the redacted pending - * output closes the call and the full signal is stashed for the caller's - * park detector. - */ -export async function executeApprovedToolCalls(input: { - readonly abortSignal?: AbortSignal; - readonly calls: readonly RuntimeToolCallActionRequest[]; - readonly messages: readonly ModelMessage[]; - readonly resolveTool: (toolName: string) => HarnessToolDefinition | undefined; -}): Promise { - const executed: ExecutedApprovedToolCall[] = []; - - for (const call of input.calls) { - const definition = input.resolveTool(call.toolName); - - if (definition === undefined) { - executed.push( - buildErrorOutcome(call, new Error(`Tool "${call.toolName}" is no longer available.`)), - ); - continue; - } - - const execute = wrapToolExecute(definition); - if (execute === undefined) { - executed.push( - buildErrorOutcome( - call, - new Error(`Tool "${call.toolName}" has no local execution and cannot run.`), - ), - ); - continue; - } - - try { - const output = await execute(call.input, { - abortSignal: input.abortSignal, - messages: [...input.messages], - toolCallId: call.callId, - }); - - executed.push({ - actionResult: createRuntimeToolResultFromValue({ - callId: call.callId, - output, - toolName: call.toolName, - }), - part: { - output: await resolveExecutedToolModelOutput({ - definition, - output, - toolCallId: call.callId, - }), - toolCallId: call.callId, - toolName: call.toolName, - type: "tool-result", - }, - }); - } catch (error) { - logError(log, "approved tool execution failed", error, { - toolCallId: call.callId, - toolName: call.toolName, - }); - executed.push(buildErrorOutcome(call, toError(error))); - } - } - - return { executed }; -} - -function buildErrorOutcome( - call: RuntimeToolCallActionRequest, - error: Error, -): ExecutedApprovedToolCall { - return { - actionResult: createRuntimeToolResultFromValue({ - callId: call.callId, - isError: true, - output: error, - toolName: call.toolName, - }), - part: { - output: { type: "error-text", value: error.message }, - toolCallId: call.callId, - toolName: call.toolName, - type: "tool-result", - }, - }; -} diff --git a/packages/eve/src/harness/emission.test.ts b/packages/eve/src/harness/emission.test.ts index b27db19de..602c41573 100644 --- a/packages/eve/src/harness/emission.test.ts +++ b/packages/eve/src/harness/emission.test.ts @@ -546,9 +546,14 @@ describe("emitStreamContent action requests", () => { }), ]); expect([...result.invalidInputToolCallIds]).toEqual(["call-bad"]); - // Transcript closure is owned by the step-result handler via - // `closeDanglingToolCalls`; the stream layer only emits the event. - expect(result.trailingInlineToolResultParts).toEqual([]); + expect(result.trailingInlineToolResultParts).toEqual([ + { + output: { type: "error-text", value: message }, + toolCallId: "call-bad", + toolName: "web_search", + type: "tool-result", + }, + ]); }); }); diff --git a/packages/eve/src/harness/emission.ts b/packages/eve/src/harness/emission.ts index a06b72c08..69593e05a 100644 --- a/packages/eve/src/harness/emission.ts +++ b/packages/eve/src/harness/emission.ts @@ -478,9 +478,7 @@ export async function emitStreamContent( } await emitActionResult(createRuntimeToolResultFromToolError(toolError)); handledInlineToolResultCallIds.add(toolCall.toolCallId); - // Transcript closure is owned by `handleStepResult`, which re-derives - // the same error via `getInvalidToolCallInputErrors` and closes the - // call through `closeDanglingToolCalls`. This site only observes. + trailingInlineToolResultParts.push(createToolResultMessagePartFromToolError(toolError)); return; } throw error; diff --git a/packages/eve/src/harness/input-requests.test.ts b/packages/eve/src/harness/input-requests.test.ts index 4171437b1..08485bd9a 100644 --- a/packages/eve/src/harness/input-requests.test.ts +++ b/packages/eve/src/harness/input-requests.test.ts @@ -252,7 +252,7 @@ describe("resolvePendingInput", () => { }); }); - it("resolves approvals and keeps a simultaneous follow-up message on the same step", () => { + it("defers a follow-up message until after tool approvals are resolved", () => { const session = setPendingInputBatch({ requests: [ { @@ -305,10 +305,18 @@ describe("resolvePendingInput", () => { // The approval should be resolved immediately. expect(result.outcome).toBe("resolved"); - // The harness closes the denied call inline, so the follow-up message - // rides the same model call instead of being deferred a step. - expect(result.deferredMessage).toBeUndefined(); - expect(hasDeferredStepInput(result.session)).toBe(false); + // The follow-up message should be deferred. + expect(result.deferredMessage).toBe(true); + expect(hasDeferredStepInput(result.session)).toBe(true); + + const deferred = consumeDeferredStepInput({ + session: result.session, + }); + + expect(deferred.input).toEqual({ + message: "Ignore that and say hi instead.", + }); + expect(hasDeferredStepInput(deferred.session)).toBe(false); }); it("resolves approval when follow-up text matches an option", () => { diff --git a/packages/eve/src/harness/input-requests.ts b/packages/eve/src/harness/input-requests.ts index 8a478ba4e..631b372de 100644 --- a/packages/eve/src/harness/input-requests.ts +++ b/packages/eve/src/harness/input-requests.ts @@ -12,7 +12,6 @@ import { isSessionLimitContinuationRequest, resolveSessionLimitContinuation, } from "#harness/session-limit-continuation.js"; -import { closeDanglingToolCalls } from "#harness/transcript-obligations.js"; import type { HarnessSession, SessionStateMap, StepInput } from "#harness/types.js"; const PENDING_INPUT_BATCH_KEY = "eve.runtime.pendingInputBatch"; @@ -26,7 +25,6 @@ const TOOL_EXECUTION_DENIED_MESSAGE = "Tool execution was denied."; const TOOL_EXECUTION_INVALID_APPROVAL_MESSAGE = "Invalid approval response."; type ToolResponsePart = Extract["content"][number]; -type ToolResultOutput = Extract["output"]; /** * Stream-emit coordinates carried so a parked batch's resolution can attribute @@ -56,17 +54,6 @@ export interface RejectedActionBatch { readonly results: readonly RuntimeToolResultActionResult[]; } -/** - * Approved tool-call actions from one resolved batch. The caller (the tool - * loop) executes them and appends their durable results before assembling - * the next model request; `event` carries the originating turn's stream - * coordinates for the resulting `action.result` emissions. - */ -export interface ApprovedActionBatch { - readonly calls: readonly RuntimeToolCallActionRequest[]; - readonly event?: PendingInputBatchEvent; -} - type ApprovalTerminalStatus = "approved" | "denied" | "ignored" | "invalid"; /** @@ -134,14 +121,13 @@ export function hasDeferredStepInput(session: HarnessSession): boolean { /** * Resolves pending input at the start of a harness step. * - * When the pending batch contains tool-approval requests that the step input - * does not answer, the input is deferred to a later step via - * {@link consumeDeferredStepInput} and the batch stays parked — an unrelated - * follow-up must never auto-deny a pending approval. Once every approval is - * answered, the batch resolves in one pass: denials are closed inline with - * `execution-denied` results, and approved calls are returned on - * `approvedActions` for the tool loop to execute and close before the next - * model request. + * When the pending batch contains tool-approval requests and the step input + * also carries a follow-up user message, the message is deferred to the next + * internal harness step rather than appended to the current turn. This is + * necessary because AI SDK cannot process tool-approval responses and a new + * user message in the same request -- the approval must be resolved in + * isolation first, and the user message replayed on the subsequent step via + * {@link consumeDeferredStepInput}. */ export function resolvePendingInput(input: { readonly history?: readonly ModelMessage[]; @@ -180,7 +166,11 @@ export function resolvePendingInput(input: { // A follow-up message arrived for question-only input with no explicit // responses. Keep the existing question semantics: mark unanswered // question requests ignored so the model can continue with the message. - const messages = buildResolvedInputMessages(baseHistory, pendingBatch, []); + const toolParts = buildToolResponseParts(pendingBatch, []); + const messages: ModelMessage[] = [...baseHistory, ...pendingBatch.responseMessages]; + if (toolParts.length > 0) { + messages.push({ content: toolParts, role: "tool" }); + } const rejectedActions = buildRejectedActionBatch(pendingBatch, []); session = clearPendingInputBatch(session); @@ -199,24 +189,48 @@ export function resolvePendingInput(input: { responses, }); - const approvedRequestIds = collectApprovedRequestIds(responses); - // Record approved tools before clearing the batch. session = recordApprovedTools({ - approvedRequestIds, pendingBatch, resolveApprovalKey: input.resolveApprovalKey, + responses, session, }); - const messages = buildResolvedInputMessages(baseHistory, pendingBatch, responses); + // Build tool result messages from responses. + const toolParts = buildToolResponseParts(pendingBatch, responses); + + const messages: ModelMessage[] = [...baseHistory, ...pendingBatch.responseMessages]; + if (toolParts.length > 0) { + messages.push({ content: toolParts, role: "tool" }); + } const rejectedActions = buildRejectedActionBatch(pendingBatch, responses); - const approvedActions = buildApprovedActionBatch(pendingBatch, approvedRequestIds); session = clearPendingInputBatch(session); + // AI SDK cannot process tool-approval responses and a new user message + // in the same request. Defer the message so the approval is resolved in + // isolation; `consumeDeferredStepInput` replays it on the next step. + if ( + resolvedStepInput?.message !== undefined && + pendingBatch.requests.some((request) => isApprovalRequest(request)) + ) { + session = queueDeferredStepInput(session, { + message: resolvedStepInput.message, + }); + + return { + consumedMessage: resolvedStepInput?.messageConsumed, + deferredMessage: true, + limitContinuation, + outcome: "resolved", + messages, + rejectedActions, + session, + }; + } + return { - approvedActions, consumedMessage: resolvedStepInput?.messageConsumed, limitContinuation, outcome: "resolved", @@ -292,11 +306,6 @@ function hasUnansweredApproval(input: { } type ResolvePendingInputResult = { - /** - * Approved tool-call actions the caller must execute and close before the - * next model request. Present only on a resolved approval batch. - */ - readonly approvedActions?: ApprovedActionBatch; readonly consumedMessage?: boolean; readonly deferredMessage?: boolean; /** @@ -421,24 +430,23 @@ export function getApprovedTools(session: HarnessSession): ReadonlySet { return new Set(value as string[]); } -/** The single definition of an approved response in one resolved batch. */ -function collectApprovedRequestIds(responses: readonly InputResponse[]): ReadonlySet { - return new Set(responses.filter((r) => r.optionId === "approve").map((r) => r.requestId)); -} - /** * Resolves the approval key for a request. When a `resolveApprovalKey` * function is provided and returns a string, that compound key is recorded * instead of the bare tool name. */ function recordApprovedTools(input: { - readonly approvedRequestIds: ReadonlySet; readonly pendingBatch: PendingInputBatch; readonly resolveApprovalKey?: (request: InputRequest) => string | undefined; + readonly responses: readonly InputResponse[]; readonly session: HarnessSession; }): HarnessSession { + const approvedIds = new Set( + input.responses.filter((r) => r.optionId === "approve").map((r) => r.requestId), + ); + const newKeys = input.pendingBatch.requests - .filter((r) => input.approvedRequestIds.has(r.requestId)) + .filter((r) => approvedIds.has(r.requestId)) .map((r) => input.resolveApprovalKey?.(r) ?? r.action.toolName); if (newKeys.length === 0) { @@ -543,82 +551,81 @@ function buildRejectedActionBatch( return results.length > 0 ? { event: batch.event, results } : undefined; } -/** - * Extracts the approved tool-call actions from one resolved batch so the - * caller can execute them. - */ -function buildApprovedActionBatch( - batch: PendingInputBatch, - approvedRequestIds: ReadonlySet, -): ApprovedActionBatch | undefined { - const calls = batch.requests - .filter((request) => isApprovalRequest(request) && approvedRequestIds.has(request.requestId)) - .map((request) => request.action); - - return calls.length > 0 ? { calls, event: batch.event } : undefined; -} - -/** Resolves input metadata and closes every answered or denied tool call. */ -function buildResolvedInputMessages( - baseHistory: readonly ModelMessage[], - batch: PendingInputBatch, - responses: readonly InputResponse[], -): ModelMessage[] { - const { closures, responseParts } = buildInputResolutionParts(batch, responses); - const messages: ModelMessage[] = [...baseHistory, ...batch.responseMessages]; - if (responseParts.length > 0) { - messages.push({ content: [...responseParts], role: "tool" }); - } - - return closeDanglingToolCalls(messages, (call) => closures.get(call.toolCallId), { - placement: "after-existing", - }).messages; -} - -function buildInputResolutionParts( +function buildToolResponseParts( batch: PendingInputBatch, responses: readonly InputResponse[], -): { - readonly closures: ReadonlyMap; - readonly responseParts: readonly ToolResponsePart[]; -} { +): ToolResponsePart[] { const responseMap = new Map(responses.map((r) => [r.requestId, r])); - const closures = new Map(); - const responseParts: ToolResponsePart[] = []; + const parts: ToolResponsePart[] = []; for (const request of batch.requests) { - // A session-limit continuation prompt is harness-authored: no matching - // tool call exists in model history, so resolving it must not append a - // tool message the provider would reject as unmatched. - if (isSessionLimitContinuationRequest(request)) { - continue; - } + parts.push(...buildToolResponsePartsForRequest(request, responseMap.get(request.requestId))); + } + return parts; +} + +function buildToolResponsePartsForRequest( + request: InputRequest, + response: InputResponse | undefined, +): ToolResponsePart[] { + // A session-limit continuation prompt is harness-authored: no matching + // tool call exists in model history, so resolving it must not append a + // tool message the provider would reject as unmatched. This is currently + // the only harness-authored request type; if another appears, replace this + // toolName predicate with a generic synthetic-request marker instead of + // stacking a second special case here. + if (isSessionLimitContinuationRequest(request)) { + return []; + } - const response = responseMap.get(request.requestId); - if (isApprovalRequest(request)) { - const { approved, reason } = resolveApprovalOutcome(response); - responseParts.push({ + if (isApprovalRequest(request)) { + const { approved, reason } = resolveApprovalOutcome(response); + const parts: ToolResponsePart[] = [ + { approvalId: request.requestId, approved, reason, type: "tool-approval-response", + }, + ]; + /* + * On denial (explicit "deny" or auto-deny when the user continues + * without responding), splice in the matching `execution-denied` + * tool-result. AI SDK's `streamText` synthesizes this for the + * current turn's `initialResponseMessages`, but that synthesis is + * gated on the input messages' last entry being a tool message — + * on subsequent turns (when a new user message is the tail of + * history) the synthesis is skipped, and the persisted + * `tool-approval-response` is stripped during provider prompt + * conversion. Without an own `tool-result` in history, the prior + * `tool_use` block replays unmatched and some providers reject + * the request with 400. + */ + if (!approved) { + parts.push({ + output: { type: "execution-denied", reason }, + toolCallId: request.action.callId, + toolName: request.action.toolName, + type: "tool-result", }); - if (!approved) { - closures.set(request.action.callId, { type: "execution-denied", reason }); - } - continue; } - - closures.set(request.action.callId, { - type: "json", - value: - response !== undefined - ? { optionId: response.optionId, text: response.text, status: "answered" } - : { status: "ignored" }, - }); + return parts; } - return { closures, responseParts }; + return [ + { + output: { + type: "json", + value: + response !== undefined + ? { optionId: response.optionId, text: response.text, status: "answered" } + : { status: "ignored" }, + }, + toolCallId: request.action.callId, + toolName: request.action.toolName, + type: "tool-result", + }, + ]; } function isApprovalRequest(request: InputRequest): boolean { diff --git a/packages/eve/src/harness/tool-loop-compaction-accounting.test.ts b/packages/eve/src/harness/tool-loop-compaction-accounting.test.ts index efd693c19..0ff388cfb 100644 --- a/packages/eve/src/harness/tool-loop-compaction-accounting.test.ts +++ b/packages/eve/src/harness/tool-loop-compaction-accounting.test.ts @@ -250,19 +250,7 @@ describe("tool-loop structured compaction accounting", () => { requestId: "question-call", }, ], - responseMessages: [ - { - content: [ - { - input: { prompt: "Pick one." }, - toolCallId: "question-call", - toolName: "ask_question", - type: "tool-call", - }, - ], - role: "assistant", - }, - ], + responseMessages: [], session: createTestSession({ compaction: { lastKnownInputTokens: 100, diff --git a/packages/eve/src/harness/tool-loop.test.ts b/packages/eve/src/harness/tool-loop.test.ts index 4d13f3291..f355bf280 100644 --- a/packages/eve/src/harness/tool-loop.test.ts +++ b/packages/eve/src/harness/tool-loop.test.ts @@ -404,7 +404,6 @@ function finalOutputResult(text: string, structured: unknown): Record { }); }); - it("executes an approved tool call at resume and persists its result before the model call, so the next turn replays a balanced tool_use / tool_result pair", async () => { + it("persists the inline approval-resume tool-result into session history so the next turn replays a balanced tool_use / tool_result pair", async () => { /* - * Closing a parked tool call is owned by the harness: the approved - * tool executes at resume time and its durable result lands in the - * same tool message as the approval response, before any model - * request is assembled. Without this, the next turn replays a + * When a previously-parked tool call is approved, the AI SDK + * enqueues its tool-result onto the parent stream before re- + * entering the LLM call. The result is absent from + * `stepResult.response.messages` / `toolCalls` / `toolResults`, + * so the harness must capture it from the stream and splice it + * into persisted history. Without this, the next turn replays a * `tool_use` block with no matching `tool_result` and Anthropic * rejects the request with 400. */ @@ -4816,6 +4816,12 @@ describe("createToolLoopHarness", () => { content: [], finishReason: "stop", fullStreamParts: [ + { + output: { exitCode: 0, stderr: "", stdout: "/workspace\n", truncated: false }, + toolCallId: "call-1", + toolName: "bash", + type: "tool-result", + }, { id: "text-1", text: "`/workspace`", type: "text-delta" }, { finishReason: "stop", type: "finish-step" }, ], @@ -4827,179 +4833,43 @@ describe("createToolLoopHarness", () => { toolResults: [], }); - const { emit, events } = createEventCollector(); + const { emit } = createEventCollector(); const session = createPendingBashApprovalSession(); - const bashExecute = vi - .fn() - .mockResolvedValue({ exitCode: 0, stderr: "", stdout: "/workspace\n", truncated: false }); const harness = createToolLoopHarness( - createTestConfig("conversation", emit, { - tools: new Map([ - [ - "bash", - { - description: "Run shell commands", - execute: bashExecute, - inputSchema: jsonSchema({ type: "object" }), - name: "bash", - }, - ], - ]), - }), + createTestConfig("conversation", emit, { tools: new Map() }), ); const result = await harness(session, { inputResponses: [{ optionId: "approve", requestId: "approval-1" }], }); - expect(bashExecute).toHaveBeenCalledWith( - { command: "pwd" }, - expect.objectContaining({ toolCallId: "call-1" }), - ); - expect(result.session.history.map((msg) => msg.role)).toEqual([ "assistant", "tool", + "tool", "assistant", ]); - const resolutionMessage = result.session.history[1]; - expect(Array.isArray(resolutionMessage?.content)).toBe(true); - const resolutionParts = resolutionMessage?.content as Array>; - expect(resolutionParts).toHaveLength(2); - expect(resolutionParts[0]).toEqual({ + const approvalMessage = result.session.history[1]; + expect(Array.isArray(approvalMessage?.content)).toBe(true); + const approvalParts = approvalMessage?.content as Array>; + expect(approvalParts).toHaveLength(1); + expect(approvalParts[0]).toEqual({ approvalId: "approval-1", approved: true, reason: undefined, type: "tool-approval-response", }); - expect(resolutionParts[1]).toMatchObject({ + + const toolResultMessage = result.session.history[2]; + expect(Array.isArray(toolResultMessage?.content)).toBe(true); + const toolResultParts = toolResultMessage?.content as Array>; + expect(toolResultParts).toHaveLength(1); + expect(toolResultParts[0]).toMatchObject({ toolCallId: "call-1", toolName: "bash", type: "tool-result", }); expect(result.session.history.at(-1)?.role).toBe("assistant"); - - const actionResults = events.filter((event) => event.type === "action.result"); - expect(actionResults).toHaveLength(1); - expect(actionResults[0]?.data).toMatchObject({ - result: { callId: "call-1", kind: "tool-result", toolName: "bash" }, - status: "completed", - }); - }); - - it("executes an approved tool call even when channel context rides the approving prompt", async () => { - // https://github.com/vercel/eve/issues/529 — the Linear channel sends - // `context` on every prompt, including the one answering an approval. - // Owned resume-time execution must not depend on the approval message - // being the request tail. - setupMockAgent({ - content: [], - finishReason: "stop", - response: { messages: [{ content: "Done.", role: "assistant" }] }, - text: "Done.", - toolCalls: [], - toolResults: [], - }); - - const session = createPendingBashApprovalSession(); - const bashExecute = vi.fn().mockResolvedValue("ok"); - const harness = createToolLoopHarness( - createTestConfig("conversation", undefined, { - tools: new Map([ - [ - "bash", - { - description: "Run shell commands", - execute: bashExecute, - inputSchema: jsonSchema({ type: "object" }), - name: "bash", - }, - ], - ]), - }), - ); - - const result = await harness(session, { - context: ["Issue ENG-123"], - inputResponses: [{ optionId: "approve", requestId: "approval-1" }], - }); - - expect(bashExecute).toHaveBeenCalledTimes(1); - - // The closed resolution precedes the context; the context stays on the - // same call. - const roles = result.session.history.map((msg) => msg.role); - expect(roles).toEqual(["assistant", "tool", "user", "assistant"]); - const resolutionParts = result.session.history[1]?.content as Array>; - expect(resolutionParts.map((part) => part.type)).toEqual([ - "tool-approval-response", - "tool-result", - ]); - expect(result.session.history[2]).toEqual({ - content: "Issue ENG-123", - role: "user", - }); - }); - - it("heals a history poisoned by a dangling local tool call before the model call", async () => { - // A dangling `tool_use` in durable history (the #460/#533 poison shape) - // must be closed with a synthetic error result instead of being replayed - // verbatim and 400ing forever. - const generateCalls: unknown[][] = []; - vi.mocked(ToolLoopAgent).mockImplementation(function ( - this: MockAgentInstance, - settings: MockAgentSettings, - ) { - const result = { - finishReason: "stop", - response: { messages: [{ content: "Recovered.", role: "assistant" }] }, - text: "Recovered.", - toolCalls: [], - toolResults: [], - }; - const { onStepFinish } = settings; - this.generate = vi.fn().mockImplementation(async (input: { messages: unknown[] }) => { - generateCalls.push(input.messages); - if (onStepFinish) await onStepFinish(result); - return result; - }); - return this; - } as MockAgentConstructor); - - const session = createTestSession({ - history: [ - { content: "create the object", role: "user" }, - { - content: [ - { - input: { command: "pwd" }, - toolCallId: "dangling-1", - toolName: "bash", - type: "tool-call", - }, - ], - role: "assistant", - }, - ], - }); - - const harness = createToolLoopHarness(createTestConfig("conversation", undefined)); - const result = await harness(session, { message: "try again" }); - - const sent = generateCalls[0] as Array<{ role: string; content: unknown }>; - const toolMessage = sent.find((msg) => msg.role === "tool"); - expect(toolMessage?.content).toEqual([ - { - output: { type: "error-text", value: expect.stringContaining("did not complete") }, - toolCallId: "dangling-1", - toolName: "bash", - type: "tool-result", - }, - ]); - - // The repair is durable: the closed transcript persists into history. - const persistedTool = result.session.history.find((msg) => msg.role === "tool"); - expect(persistedTool).toBeDefined(); }); it("does not persist provider-executed deferred tool-results as generic tool messages", async () => { @@ -6444,7 +6314,7 @@ describe("createToolLoopHarness", () => { expect(events.filter((event) => event.type === "action.result")).toHaveLength(1); }); - it("resolves a queued follow-up message in the same step as the approval denial", async () => { + it("queues a follow-up user message until the pending tool approval resolves", async () => { const generateCalls: unknown[] = []; const agentResults = [ { @@ -6563,9 +6433,7 @@ describe("createToolLoopHarness", () => { inputResponses: [{ requestId: "approval-1", optionId: "deny" }], }); - // The denial is closed inline, so the queued message rides the same - // model call instead of costing another step. - expect(deniedResult.next).toBeNull(); + expect(typeof deniedResult.next).toBe("function"); expect(generateCalls[0]).toEqual([ { content: [ @@ -6603,18 +6471,21 @@ describe("createToolLoopHarness", () => { ], role: "tool", }, - { - content: "Hi instead.", - role: "user", - }, ]); - expect(hasDeferredStepInput(deniedResult.session)).toBe(false); - expect(deniedResult.session.history.at(-2)).toEqual({ + + const secondResult = await createToolLoopHarness(config)(deniedResult.session); + + expect(secondResult.next).toBeNull(); + expect((generateCalls[1] as { role: string; content: unknown }[]).at(-1)).toEqual({ content: "Hi instead.", role: "user", }); - expect(deniedResult.session.history.at(-1)).toEqual({ - content: "Okay, I will not run that command.", + expect(secondResult.session.history.at(-2)).toEqual({ + content: "Hi instead.", + role: "user", + }); + expect(secondResult.session.history.at(-1)).toEqual({ + content: "Hello!", role: "assistant", }); }); @@ -6746,24 +6617,17 @@ describe("createToolLoopHarness", () => { reason: undefined, type: "tool-approval-response", }, - { - output: { type: "text", value: "ok" }, - toolCallId: "call-1", - toolName: "guarded_echo", - type: "tool-result", - }, ], role: "tool", }, ]); }); - it("deferred message lands as last non-system message in the approval-denial step", async () => { + it("deferred message lands as last non-system message after explicit approval denial", async () => { // Step 1: pending approval + user sends a follow-up message. The approval // remains pending and the message is deferred. Step 2: the user denies the - // approval; the denial is closed inline, so the deferred message is - // consumed in the same step and appears as the last message the model - // sees. + // approval. Step 3: the deferred message is consumed and appears as the + // last message the model sees. const generateCalls: Array> = []; const agentResults = [ { @@ -6881,18 +6745,25 @@ describe("createToolLoopHarness", () => { expect(firstResult.next).toBeNull(); expect(generateCalls).toEqual([]); - // Step 2: user denies the approval; the deferred message is consumed in - // the same step and is the last message the model sees. + // Step 2: user denies the approval; the deferred message is NOT in this call. const deniedResult = await createToolLoopHarness(config)(firstResult.session, { inputResponses: [{ requestId: "approval-1", optionId: "deny" }], }); - expect(deniedResult.next).toBeNull(); + expect(typeof deniedResult.next).toBe("function"); const step2Last = generateCalls[0]?.at(-1); - expect(step2Last).toEqual({ content: "Do something else", role: "user" }); + expect(step2Last?.role).toBe("tool"); + + // Step 3: harness consumes the deferred message. + const secondResult = await createToolLoopHarness(config)(deniedResult.session); + expect(secondResult.next).toBeNull(); + + // The deferred user message is the last message the model sees. + const step3Last = generateCalls[1]?.at(-1); + expect(step3Last).toEqual({ content: "Do something else", role: "user" }); // History reflects the full conversation. - expect(deniedResult.session.history.at(-1)).toEqual({ - content: "I will not run that command.", + expect(secondResult.session.history.at(-1)).toEqual({ + content: "Sure, here you go.", role: "assistant", }); }); diff --git a/packages/eve/src/harness/tool-loop.ts b/packages/eve/src/harness/tool-loop.ts index b8df50022..dc7cc06a5 100644 --- a/packages/eve/src/harness/tool-loop.ts +++ b/packages/eve/src/harness/tool-loop.ts @@ -97,11 +97,6 @@ import { extractToolApprovalInputRequests, } from "#harness/input-extraction.js"; import { createToolResultMessagePartFromToolError } from "#harness/action-result-helpers.js"; -import { closeApprovedActionBatch } from "#harness/approved-tool-execution.js"; -import { - closeDanglingToolCalls, - INTERRUPTED_TOOL_CALL_RESULT, -} from "#harness/transcript-obligations.js"; import { buildTelemetryRuntimeContext } from "#harness/instrumentation-runtime-context.js"; import { consumeDeferredStepInput, @@ -575,40 +570,6 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn { session = pending.session; let messages: ModelMessage[] = pending.messages; - // Direct harness unit tests may run without an ambient context. - const ctx = contextStorage.getStore(); - - // --- Owned closure of approved tool calls ------------------------------- - // - // The harness executes approved parked tool calls itself and appends - // their durable results before assembling the model request. Closure of - // a parked `tool_use` must not depend on the AI SDK's last-message - // approval scan (anything appended after the approval message silently - // disables it) nor on stream capture for result persistence. - if (pending.approvedActions !== undefined) { - const closure = await closeApprovedActionBatch({ - abortSignal: config.abortSignal, - batch: pending.approvedActions, - ctx, - emit, - messages, - tools: config.tools, - }); - messages = closure.messages; - - // Owned execution parks on authorization exactly like in-stream - // execution: the redacted pending output already closed the call. - if (closure.authorizationSignal !== undefined) { - return parkOnAuthorization({ - challenges: closure.authorizationSignal.challenges, - emit, - emissionState, - history: messages, - session, - }); - } - } - // A resolved session-limit continuation prompt grants a fresh token // budget or ends the session; see session-limit-enforcement. const continuation = await applySessionLimitContinuation({ @@ -645,6 +606,8 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn { // --- Model + tools ------------------------------------------------------ + // Direct harness unit tests may run without an ambient context. + const ctx = contextStorage.getStore(); if (ctx !== undefined && config.dispatchDynamicModelEvent !== undefined) { await config.dispatchDynamicModelEvent({ ctx, @@ -686,26 +649,6 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn { telemetry: enrichTelemetry(telemetryConfig, agentName) ?? undefined, })); - // --- Transcript-closure invariant ---------------------------------------- - // - // The single guard: no local `tool-call` may reach a provider without a - // terminal `tool-result`. Runs after all message assembly (pending - // resolution, context/message appends, compaction) so any dangling call - // here is an orphan; closing it durably also heals histories poisoned - // before this invariant existed. - const reconciliation = closeDanglingToolCalls(messages, () => ({ - type: "error-text", - value: INTERRUPTED_TOOL_CALL_RESULT, - })); - if (reconciliation.closed.length > 0) { - log.warn("closed dangling tool calls before model call", { - repaired: reconciliation.closed, - sessionId: session.sessionId, - turnId: emissionState.turnId, - }); - messages = reconciliation.messages; - } - const approvedTools = getApprovedTools(session); const emptyDeliveryEnabled = @@ -1757,17 +1700,15 @@ async function handleStepResult(input: { ...(result.invalidInputToolCallIds ?? []), ...invalidInputToolErrors.map((toolError) => toolError.toolCallId), ]); - const invalidInputClosures = new Map( - invalidInputToolErrors.map((toolError) => [ - toolError.toolCallId, - createToolResultMessagePartFromToolError(toolError).output, - ]), - ); const rawResponseMessages = emptyDelivery ? [] - : closeDanglingToolCalls(result.response.messages, (call) => - invalidInputClosures.get(call.toolCallId), - ).messages; + : insertInlineToolResultMessages({ + append: invalidInputToolErrors.map((toolError) => + createToolResultMessagePartFromToolError(toolError), + ), + prepend: [], + responseMessages: result.response.messages, + }); const stepOutput = emptyDelivery ? null : resolvedStepOutput; const providerExecutedOutcomeIds = new Set(); @@ -1910,13 +1851,35 @@ async function handleStepResult(input: { const authSignal = findAuthorizationSignalFromToolResults(result.toolResults); if (authSignal) { - return parkOnAuthorization({ - challenges: authSignal.challenges, - emit, - emissionState, - history: promptMessages, - session: baseSession, - }); + const { challenges } = authSignal; + + if (emit) { + for (const ch of challenges) { + await emit( + createAuthorizationRequiredEvent({ + authorization: ch.challenge, + name: ch.name, + description: ch.challenge.instructions ?? `Authorization required for ${ch.name}`, + webhookUrl: ch.hookUrl, + sequence: emissionState.sequence, + stepIndex: emissionState.stepIndex, + turnId: emissionState.turnId, + }), + ); + } + } + + return { + next: null, + session: setHarnessEmissionState( + { + ...baseSession, + history: [...promptMessages], + state: setPendingAuthorization(baseSession.state, { challenges }), + }, + emissionState, + ), + }; } // --- Continue or terminate ------------------------------------------------ @@ -2421,49 +2384,6 @@ async function runModelCallWithRetries( } } -/** - * Emits `authorization.required` for each challenge and parks the session on - * the pending authorization. Shared by resume-time owned execution and the - * post-step in-stream path. - */ -async function parkOnAuthorization(input: { - readonly challenges: AuthorizationSignal["challenges"]; - readonly emit?: ToolLoopHarnessConfig["handleEvent"]; - readonly emissionState: ReturnType; - readonly history: readonly ModelMessage[]; - readonly session: HarnessSession; -}): Promise { - const { challenges, emissionState } = input; - - if (input.emit) { - for (const ch of challenges) { - await input.emit( - createAuthorizationRequiredEvent({ - authorization: ch.challenge, - name: ch.name, - description: ch.challenge.instructions ?? `Authorization required for ${ch.name}`, - webhookUrl: ch.hookUrl, - sequence: emissionState.sequence, - stepIndex: emissionState.stepIndex, - turnId: emissionState.turnId, - }), - ); - } - } - - return { - next: null, - session: setHarnessEmissionState( - { - ...input.session, - history: [...input.history], - state: setPendingAuthorization(input.session.state, { challenges }), - }, - emissionState, - ), - }; -} - function findAuthorizationSignalFromToolResults( toolResults: readonly TypedToolResult[] | undefined, ): AuthorizationSignal | undefined { diff --git a/packages/eve/src/harness/tools.ts b/packages/eve/src/harness/tools.ts index 2b38e5098..4431365e9 100644 --- a/packages/eve/src/harness/tools.ts +++ b/packages/eve/src/harness/tools.ts @@ -89,7 +89,29 @@ export function buildToolSet(input: { }: { readonly output: unknown; readonly toolCallId?: string; - }) => resolveExecutedToolModelOutput({ definition, output, toolCallId }), + }) => { + if (isAuthorizationPendingModelOutput(output)) { + return { + type: "text" as const, + value: authorizationPendingModelText(output.connections), + }; + } + if (authorToModelOutput !== undefined) { + return normalizeToolModelOutput({ + output: await authorToModelOutput(output), + toolCallId, + toolName: definition.name, + }); + } + if (typeof output === "string") { + return { type: "text" as const, value: output }; + } + return normalizeToolModelOutput({ + output: { type: "json" as const, value: output ?? null }, + toolCallId, + toolName: definition.name, + }); + }, } : authorToModelOutput !== undefined ? { @@ -170,42 +192,6 @@ export function wrapToolExecute( }; } -/** - * Model-facing output projection for an executed tool's raw output. The - * single normalization point shared by the AI SDK `toModelOutput` hook (via - * {@link buildToolSet}) and the harness's own resume-time execution of - * approved tool calls, so both paths record identical durable results. - */ -export async function resolveExecutedToolModelOutput(input: { - readonly definition: HarnessToolDefinition; - readonly output: unknown; - readonly toolCallId?: string; -}): Promise { - const { definition, output, toolCallId } = input; - - if (isAuthorizationPendingModelOutput(output)) { - return { - type: "text", - value: authorizationPendingModelText(output.connections), - }; - } - if (definition.toModelOutput !== undefined) { - return normalizeToolModelOutput({ - output: await definition.toModelOutput(output), - toolCallId, - toolName: definition.name, - }); - } - if (typeof output === "string") { - return { type: "text", value: output }; - } - return normalizeToolModelOutput({ - output: { type: "json", value: output ?? null }, - toolCallId, - toolName: definition.name, - }); -} - function normalizeToolJsonOutput(input: { readonly boundary: "execute" | "toModelOutput"; readonly output: unknown; diff --git a/packages/eve/src/harness/transcript-obligations.test.ts b/packages/eve/src/harness/transcript-obligations.test.ts deleted file mode 100644 index 440e245e8..000000000 --- a/packages/eve/src/harness/transcript-obligations.test.ts +++ /dev/null @@ -1,294 +0,0 @@ -import type { ModelMessage } from "ai"; -import { describe, expect, it } from "vitest"; - -import { - closeDanglingToolCalls, - type DanglingToolCall, - INTERRUPTED_TOOL_CALL_RESULT, -} from "#harness/transcript-obligations.js"; - -function assistantToolCall(toolCallId: string, providerExecuted?: boolean): ModelMessage { - const part: { - input: unknown; - providerExecuted?: boolean; - toolCallId: string; - toolName: string; - type: "tool-call"; - } = { - input: { command: "pwd" }, - toolCallId, - toolName: "bash", - type: "tool-call", - }; - if (providerExecuted !== undefined) { - part.providerExecuted = providerExecuted; - } - return { content: [part], role: "assistant" }; -} - -function toolResult(toolCallId: string): ModelMessage { - return { - content: [ - { - output: { type: "text", value: "ok" }, - toolCallId, - toolName: "bash", - type: "tool-result", - }, - ], - role: "tool", - }; -} - -/** The request-assembly guard's closure policy, as wired in the tool loop. */ -function reconcileToolTranscript(messages: ModelMessage[]): { - messages: ModelMessage[]; - repaired: readonly DanglingToolCall[]; -} { - const { closed, messages: reconciled } = closeDanglingToolCalls(messages, () => ({ - type: "error-text", - value: INTERRUPTED_TOOL_CALL_RESULT, - })); - return { messages: reconciled, repaired: closed }; -} - -describe("closeDanglingToolCalls with the guard policy", () => { - it("passes a balanced transcript through unchanged", () => { - const messages: ModelMessage[] = [ - { content: "run pwd", role: "user" }, - assistantToolCall("call-1"), - toolResult("call-1"), - { content: "done", role: "assistant" }, - ]; - - const result = reconcileToolTranscript(messages); - - expect(result.repaired).toEqual([]); - expect(result.messages).toEqual(messages); - }); - - it("closes a dangling local tool call with a synthetic error result in the adjacent message", () => { - const messages: ModelMessage[] = [ - { content: "run pwd", role: "user" }, - assistantToolCall("call-1"), - { content: "anything else?", role: "user" }, - ]; - - const result = reconcileToolTranscript(messages); - - expect(result.repaired).toEqual([{ toolCallId: "call-1", toolName: "bash" }]); - expect(result.messages).toEqual([ - { content: "run pwd", role: "user" }, - assistantToolCall("call-1"), - { - content: [ - { - output: { type: "error-text", value: INTERRUPTED_TOOL_CALL_RESULT }, - toolCallId: "call-1", - toolName: "bash", - type: "tool-result", - }, - ], - role: "tool", - }, - { content: "anything else?", role: "user" }, - ]); - }); - - it("merges the closure into an existing adjacent tool message", () => { - const messages: ModelMessage[] = [ - { - content: [ - { - input: {}, - toolCallId: "call-1", - toolName: "bash", - type: "tool-call", - }, - { - input: {}, - toolCallId: "call-2", - toolName: "bash", - type: "tool-call", - }, - ], - role: "assistant", - }, - toolResult("call-2"), - ]; - - const result = reconcileToolTranscript(messages); - - expect(result.repaired).toEqual([{ toolCallId: "call-1", toolName: "bash" }]); - expect(result.messages).toHaveLength(2); - const closure = result.messages[1]; - expect(closure?.role).toBe("tool"); - expect(closure?.content).toEqual([ - { - output: { type: "error-text", value: INTERRUPTED_TOOL_CALL_RESULT }, - toolCallId: "call-1", - toolName: "bash", - type: "tool-result", - }, - { - output: { type: "text", value: "ok" }, - toolCallId: "call-2", - toolName: "bash", - type: "tool-result", - }, - ]); - }); - - it("closes a dangling call at the end of the transcript", () => { - const messages: ModelMessage[] = [assistantToolCall("call-1")]; - - const result = reconcileToolTranscript(messages); - - expect(result.repaired).toEqual([{ toolCallId: "call-1", toolName: "bash" }]); - expect(result.messages.at(-1)).toEqual({ - content: [ - { - output: { type: "error-text", value: INTERRUPTED_TOOL_CALL_RESULT }, - toolCallId: "call-1", - toolName: "bash", - type: "tool-result", - }, - ], - role: "tool", - }); - }); - - it("ignores provider-executed tool calls", () => { - const messages: ModelMessage[] = [assistantToolCall("call-1", true)]; - - const result = reconcileToolTranscript(messages); - - expect(result.repaired).toEqual([]); - expect(result.messages).toEqual(messages); - }); - - it("counts inline assistant tool-results as closures", () => { - const messages: ModelMessage[] = [ - { - content: [ - { - input: {}, - toolCallId: "call-1", - toolName: "bash", - type: "tool-call", - }, - { - output: { type: "text", value: "ok" }, - toolCallId: "call-1", - toolName: "bash", - type: "tool-result", - }, - ], - role: "assistant", - }, - ]; - - const result = reconcileToolTranscript(messages); - - expect(result.repaired).toEqual([]); - expect(result.messages).toEqual(messages); - }); - - it("closes a call whose approval response never produced a result — no exemptions", () => { - // An approval-response is not a closure: providers strip it, so a call - // with only a response replays as a dangling tool_use. The harness - // closes every approved call at resume; anything still dangling here is - // an orphan. - const messages: ModelMessage[] = [ - { - content: [ - { - input: {}, - toolCallId: "call-1", - toolName: "client_tool", - type: "tool-call", - }, - { - approvalId: "approval-1", - toolCallId: "call-1", - type: "tool-approval-request", - }, - ], - role: "assistant", - }, - { - content: [ - { - approvalId: "approval-1", - approved: true, - type: "tool-approval-response", - }, - ], - role: "tool", - }, - ]; - - const result = reconcileToolTranscript(messages); - - expect(result.repaired).toEqual([{ toolCallId: "call-1", toolName: "client_tool" }]); - const closure = result.messages[1]; - expect(closure?.role).toBe("tool"); - const closureParts = (closure?.content ?? []) as Array>; - expect(closureParts.map((part) => part.type)).toEqual([ - "tool-result", - "tool-approval-response", - ]); - }); - - it("closes each dangling call exactly once when it appears twice", () => { - const messages: ModelMessage[] = [assistantToolCall("call-1"), assistantToolCall("call-1")]; - - const result = reconcileToolTranscript(messages); - - expect(result.repaired).toEqual([{ toolCallId: "call-1", toolName: "bash" }]); - }); -}); - -describe("closeDanglingToolCalls", () => { - it("closes only the calls the resolver supplies an output for, with that output", () => { - const messages: ModelMessage[] = [ - assistantToolCall("call-invalid"), - assistantToolCall("call-parked"), - ]; - - const result = closeDanglingToolCalls(messages, (call) => - call.toolCallId === "call-invalid" - ? { type: "error-text", value: "Failed to parse tool-call arguments" } - : undefined, - ); - - expect(result.closed).toEqual([{ toolCallId: "call-invalid", toolName: "bash" }]); - expect(result.messages).toEqual([ - messages[0], - { - content: [ - { - output: { type: "error-text", value: "Failed to parse tool-call arguments" }, - toolCallId: "call-invalid", - toolName: "bash", - type: "tool-result", - }, - ], - role: "tool", - }, - messages[1], - ]); - }); - - it("does not re-close a call that already has a tool result", () => { - const messages: ModelMessage[] = [assistantToolCall("call-1"), toolResult("call-1")]; - - const result = closeDanglingToolCalls(messages, () => ({ - type: "error-text", - value: "should not appear", - })); - - expect(result.closed).toEqual([]); - expect(result.messages).toEqual(messages); - }); -}); diff --git a/packages/eve/src/harness/transcript-obligations.ts b/packages/eve/src/harness/transcript-obligations.ts deleted file mode 100644 index 952ece33d..000000000 --- a/packages/eve/src/harness/transcript-obligations.ts +++ /dev/null @@ -1,141 +0,0 @@ -import type { ModelMessage } from "ai"; - -type ToolMessage = Extract; -type ToolResponsePart = ToolMessage["content"][number]; -type ToolResultPart = Extract; -type ToolResultOutput = ToolResultPart["output"]; - -/** - * Synthetic result recorded for a local tool call that reached a model call - * without a terminal result. Phrased for the model: the call must be treated - * as never executed. - */ -export const INTERRUPTED_TOOL_CALL_RESULT = - "Tool execution did not complete: the call was interrupted before a result was recorded. Treat this call as not executed."; - -/** One local tool call found without a terminal `tool-result`. */ -export interface DanglingToolCall { - readonly toolCallId: string; - readonly toolName: string; -} - -/** - * The one primitive that closes a local `tool-call`'s transcript obligation. - * - * Providers reject a replayed `tool_use` without a `tool_result` (Anthropic: - * `tool_use` ids without `tool_result` blocks; OpenAI Responses: `No tool - * output found for function call`), so every non-provider-executed - * `tool-call` must be closed before the transcript reaches a provider. - * - * Walks the messages, finds every local `tool-call` without a `tool-result`, - * and — when `resolveClosure` supplies an output for it — records that - * closure in the message immediately after the assistant message carrying - * the call (merged into an existing tool message, or inserted as a new one), - * satisfying provider adjacency rules. Calls whose closure resolves to - * `undefined` are left dangling, so legitimately open obligations (parked - * approvals, pending runtime actions) pass through untouched. - * - * Every transcript closure the harness synthesizes goes through here: - * answered input requests, approved and denied calls, invalid-input calls, - * and the request-assembly guard, which closes any remaining orphan with - * {@link INTERRUPTED_TOOL_CALL_RESULT}. Recording the guard's closure durably - * heals histories poisoned by a missed result. - */ -export function closeDanglingToolCalls( - messages: readonly ModelMessage[], - resolveClosure: (call: DanglingToolCall) => ToolResultOutput | undefined, - options: { readonly placement: "after-existing" | "before-existing" } = { - placement: "before-existing", - }, -): { - readonly closed: readonly DanglingToolCall[]; - readonly messages: ModelMessage[]; -} { - const closedCallIds = collectClosedCallIds(messages); - const closed: DanglingToolCall[] = []; - const result: ModelMessage[] = []; - - for (let index = 0; index < messages.length; index += 1) { - const message = messages[index] as ModelMessage; - result.push(message); - - const closures = collectClosures(message, closedCallIds, resolveClosure, closed); - if (closures.length === 0) { - continue; - } - - // Providers require a call's result in the message immediately after the - // assistant message that carries the call. - const next = messages[index + 1]; - if (next !== undefined && next.role === "tool" && Array.isArray(next.content)) { - const content = - options.placement === "after-existing" - ? [...next.content, ...closures] - : [...closures, ...next.content]; - result.push({ ...next, content }); - index += 1; - } else { - result.push({ content: closures, role: "tool" }); - } - } - - return { closed, messages: result }; -} - -function collectClosures( - message: ModelMessage, - closedCallIds: Set, - resolveClosure: (call: DanglingToolCall) => ToolResultOutput | undefined, - closed: DanglingToolCall[], -): ToolResultPart[] { - if (message.role !== "assistant" || !Array.isArray(message.content)) { - return []; - } - - const closures: ToolResultPart[] = []; - for (const part of message.content) { - if (part.type !== "tool-call" || part.providerExecuted === true) { - continue; - } - if (closedCallIds.has(part.toolCallId)) { - continue; - } - - const call: DanglingToolCall = { toolCallId: part.toolCallId, toolName: part.toolName }; - const output = resolveClosure(call); - if (output === undefined) { - continue; - } - - closedCallIds.add(part.toolCallId); - closed.push(call); - closures.push({ - output, - toolCallId: part.toolCallId, - toolName: part.toolName, - type: "tool-result", - }); - } - - return closures; -} - -function collectClosedCallIds(messages: readonly ModelMessage[]): Set { - const callIds = new Set(); - - for (const message of messages) { - if (message.role !== "tool" && message.role !== "assistant") { - continue; - } - if (!Array.isArray(message.content)) { - continue; - } - for (const part of message.content) { - if (part.type === "tool-result") { - callIds.add(part.toolCallId); - } - } - } - - return callIds; -} diff --git a/research/approval-resume-owned-execution.md b/research/approval-resume-owned-execution.md index 27c5073fc..1039ae0ee 100644 --- a/research/approval-resume-owned-execution.md +++ b/research/approval-resume-owned-execution.md @@ -1,6 +1,6 @@ --- issue: https://github.com/vercel/eve/issues/460 -status: in-review +status: proposed last_updated: "2026-07-07" --- @@ -33,18 +33,6 @@ message-assembly ordering in the tool loop. Because the logic is decoupled, the invariant is never stated and never checked — each seam holds only as long as its neighbors behave. -Tiny examples of the mismatch: - -- Deny: eve writes `tool-result(call_1, execution-denied)` during resume. -- Approve: eve writes `tool-approval-response(call_1)` during resume and - waits for the AI SDK to later produce `tool-result(call_1, output)`. -- Approve with a follow-up `message`: eve defers the message so the approval - tool message stays last, because the SDK only scans the tail. -- Approve with `context`: eve appends the context after the approval, so the - SDK tail scan sees a user message and runs no tool. -- Approved tool execution: the durable `tool-result` reaches history through - stream capture, not through the resume code that accepted the approval. - ### Seam 1: the deny arm closes in eve, the approve arm closes in the AI SDK Denial is closed by eve, inline, at resume @@ -180,28 +168,25 @@ Each prior fix patched one seam without owning the obligation: _message_ to keep the approval message the tail (seam 3, `message` only — `context` was missed). - `bd287b17` (#576) — spliced synthetic results for invalid-input tool calls - through a separate append path. This branch folds that sibling obligation - into the same `closeDanglingToolCalls` primitive used by the replay guard, - while preserving the invalid-input error text the model can act on. + (a sibling obligation, closed in yet another place). -The pattern is the diagnosis: each point fix added one more closure site, in a -different module, guarding one more entry point. Before this branch, the -invariant — one terminal result per parked call before any replay — existed -nowhere as an owned harness rule, so each new entry point (a channel adding -context, a slower `execute`, a stricter provider) could break it again. +The pattern is the diagnosis: every fix adds one more closure site, in a +different module, guarding one more entry point. The invariant — one terminal +result per parked call before any replay — still exists nowhere in code, so +each new entry point (a channel adding context, a slower `execute`, a +stricter provider) breaks it again. ## Direction Centralize both halves of the obligation in the harness: -1. **One closure primitive.** eve closes every parked local tool call itself at +1. **One closure moment.** eve closes every parked local tool call itself at resume — executing approved tools directly (the execution wrapper already exists: `wrapToolExecute` in `harness/tools.ts`) and appending the durable - `tool-result`/`tool-error` alongside the existing denial synthesis. The - same primitive closes invalid-input calls at step time with their specific - parse error. No dependency on the SDK's tail scan or on stream capture for - obligation-closing results. This also removes the reason the message/context - deferral machinery exists. + `tool-result`/`tool-error` alongside the existing denial synthesis. No + dependency on the SDK's tail scan or on stream capture for + obligation-closing results. This also removes the reason the + message/context deferral machinery exists. 2. **One guard.** A single pure reconciliation pass over the assembled messages, immediately before every model call: no local `tool-call` without a terminal `tool-result` may reach the wire. Dangling calls are