feat(interaction): readiness wait for replay and the Node client on a shared observation engine (#3072)

* proto(reliability): observation engine, readiness poll on promotedTarget, two loops migrated

Prototype only. Shared observeUntil loop in capture-kit with one ride-out
classifier in contracts; promotedTarget declares a 2 s / 200 ms readiness
budget consumed by selector resolution with the first capture unbounded;
post-gesture stability and scroll movement run as verdict predicates over
the engine; targetReadiness and outcomeObservation guarantee cells.

* feat(interaction): operator-only readiness budget for press/click/longpress

promotedTarget.poll becomes {intervalMs, maxTimeoutMs} (no row default). A
new operator-only readinessTimeoutMs common input field (no CLI flag, no MCP
exposure) gates resolveSelectorInteractionTarget's poll: a positive integer
caps at the row's maxTimeoutMs and runs the readiness loop; absent (MCP,
CLI, and every caller today) takes the one-attempt path, so a wrong-selector
miss fails on the first capture instead of paying a 2s wait. Replay defaults
readinessTimeoutMs: 2_000 onto dispatched press/click/longpress steps unless
already set.

Split resolution.ts (1,411 lines) three ways: selector-readiness.ts owns the
readiness poll and its capture-attempt/failure machinery,
interaction-snapshot-capture.ts and covered-interaction-error.ts are new
leaves shared with resolution.ts without a value-import cycle (verified via
the layering scan). interaction-guarantees.ts's targetReadiness via now
points at selector-readiness.ts#pollForSelectorReadiness.

* test(client): split the readinessTimeoutMs SDK test out of client.test.ts

client.test.ts is already over the test-file-size-ratchet tripwire and may
not grow; the new press/click/longpress readinessTimeoutMs test moves to its
own small file instead.

* feat(interaction): end a readiness wait on a sparse capture; widen the request envelope by the readiness budget

* docs(interaction): document the replay readiness wait and readinessTimeoutMs

* refactor(interaction): keep readiness comments to their constraints

* refactor(move): split interaction target resolution by domain question

resolution.ts (1,131 lines) keeps the entry and the point/selector target
kinds (249 lines). The rest moves, function bodies unchanged:

- ref-target-resolution.ts (262): how an @ref becomes a node, and when it
  is refused (adopt/read, frame reconciliation, tryResolveRefNode,
  refMissRefusal).
- resolution-disclosure.ts (179): what a resolution records and reports
  (EXACT_REF_RESOLUTION, buildRefResolution and its ResolvedRefNode type,
  selector disclosure, non-hittable hint, press recording retarget).
- target-visibility-stages.ts (219): is the resolved target reachable on
  screen: the node-stage runner, the covered refusal (covered-interaction-
  error.ts folded in and deleted) and the off-screen stage.
- native-ref-interaction.ts (135): the native-ref preflight and dispatch.
- resolution-touch-point.ts (48): node to touch point, shared by the ref and
  selector kinds, so neither kind module owns the other.
- replay-target-guard.ts (65): assertExpectedResolvedTarget and
  assertReplayTargetResolution, used by the ref and selector kinds and by
  selector-is/selector-read.
- interaction-resolution-request.ts (73): InteractionAction,
  ResolveInteractionTargetParams and ExpectedResolvedTarget. The layering
  scan (R9/R10) refused the split while sibling modules type-imported
  resolution.ts: the largest type cycle grew from 6 to 8 files. The request
  types now sit below every module that reads them.

Not a pure move: `export` added to symbols now crossing a module boundary;
two comments that pointed at "above" now name the file; the stale
fallow-ignore on EXACT_REF_RESOLUTION (now statically imported) and on
pollForSelectorReadiness (already statically imported by resolution.ts)
are removed. interaction-guarantees.ts `via` paths and three cross-package
comments follow the symbols.

resolution.test.ts (706) splits along the same seams: resolution.test.ts
350, ref-target-resolution.test.ts 126, resolution-disclosure.test.ts 19,
target-visibility-stages.test.ts 227. Test bodies unchanged.

git diff -M90% --stat: 24 files changed, 1399 insertions(+), 1312
deletions(-). No file passes the 90% rename threshold (it is a split);
resolution.ts +18/-900, resolution.test.ts +1/-357. A line-multiset
comparison of old against new bodies (imports and blank lines excluded)
differs only in the export keywords and comments listed above. No fallow
baseline entry was keyed on the moved paths.

* refactor(replay): read the readiness-budgeted commands from their declaration

Replay hardcoded press/click/longpress as the commands that get a default
readinessTimeoutMs. Nothing declared that set per command:
readinessTimeoutMs is a common input field every command reads.

press, click and longpress now declare `targetReadiness: 'budgeted'` in the
command registry. Replay reads it through commandAcceptsReadinessBudget.
resolveCommandTimeoutPolicy attaches the trait to the resolved timeout
policy, and resolveCommandRequestTimeoutMs widens the envelope by the
readiness budget only when that trait is present. Before, it widened for
any command whose flags carried readinessTimeoutMs; only these three
commands forward the field to request flags, so no request changes.

A test asserts that the commands with the trait equal the commands the
`runtime` targetReadiness guarantee cells in interaction-guarantees.ts
apply to. The replay default test iterates the declared set.

* refactor(interaction): keep the engine and the readiness wait; loop migrations move to a stacked PR

The post-gesture stability and scroll-movement loops return to their main
versions, together with the scroll-movement observation provider scenario and
the outcomeObservation guarantee cell that described them. They move to a
stacked PR: both loops run captures that may not honor a per-capture deadline
signal, so the engine needs a declared per-capture deadline mode and
slow-capture scenarios before they can migrate without turning a late capture
into a stalled end.

The shared ride-out classifier had no production reader (readiness rides out
isUnreadableCaptureContentError), so it is deleted with the contracts
observation subpath. ObservationEnd has one reader, the engine, and now lives
in observe-until.ts.

* feat(interaction): declare a per-capture deadline mode and report ridden-out errors

observeUntil schedules now declare captureDeadline. 'cancel' keeps the
existing contract: each capture is armed with the remaining budget and a
capture that ends at or past its deadline ends the loop stalled. 'none' hands
the capture no deadline, so a capture that cannot be cancelled is still judged
when it finishes late and the loop ends done or expired, never stalled.

A ridden-out poll now records its error, so a loop whose every capture was
ridden out ends expired with lastError set; the readiness wait raises that
typed unreadable-content error instead of a selector miss against an empty
tree. A terminal capture failure is labelled 'failed' in the poll timeline
instead of 'observed'.

* fix(interaction): scope readinessTimeoutMs to budgeted commands and cap its envelope

readinessTimeoutMs was a common input every structured command accepted and
ignored. Only commands whose descriptor declares targetReadiness: 'budgeted'
now carry it, through targetReadinessFields; the common input reader refuses
the key with INVALID_ARGS for every command whose fields do not declare it.

The request envelope widened by the raw readinessTimeoutMs, so a caller value
of 999000 stretched it by 999 s while the poll itself stops at the
promotedTarget row's ceiling. The widening now uses the same capped budget.

The press readiness provider scenario claimed to exercise forceFresh through
the selector capture cache; the press route captures through the interaction
backend, which keeps no such cache, so the comments now say so.

* fix(replay): wait for a late target at the pre-dispatch gate and report readiness

A recorded press/click/longpress step carries target evidence, and replay
classified that target from one capture before dispatch. A target that had not
rendered yet diverged as a selector miss before the dispatch's own readiness
wait could run, so the replay default budget never applied to annotated steps.

The engine now asks the port to observeTarget, which captures and classifies
in one capability. For a step whose dispatch would wait (a readiness-budgeted
command resolving a selector), the port re-captures under the same schedule
while the recorded target is a selector miss; any other classification or an
unavailable capture ends the wait. A target that never appears diverges as
before. The divergence capture takes no per-capture signal, so the wait
declares captureDeadline 'none'. ReplayDaemonDependencies gains an optional
clock the wait paces itself by.

REPLAY_DIVERGENCE now carries the cause's readiness detail at
error.details.readiness.

* docs(interaction): describe where readiness evidence appears

The client API page says the command performs the requested interaction after
the wait, and names the failures that carry readiness evidence and those that
do not. The replay page says how a missing target surfaces for annotated and
unannotated steps.

* test(layering): record AdReplayTargetObservation on the ad-replay root facade

* refactor(interaction): drop the readiness forceFresh plumbing; the press capture route has no cache

The readiness poll asked for a fresh capture from its second poll on, but no
press, click, or longpress capture can be served from the selector capture
cache, so the flag and the backend option that carried it are removed and
selector-runtime-backend.ts, backend.ts, and backend-snapshot-options.ts
return to main.

Trace:
- selector-readiness.ts pollSelectorReadinessOnce captured through
  attemptSelectorResolution, which calls captureInteractionSnapshot
  (interaction-snapshot-capture.ts:35) and so runtime.backend.captureSnapshot.
- The daemon press route builds that runtime with createInteractionRuntimeForRoute
  (src/daemon/interaction/internal/interaction-touch-press.ts:10, used for
  longPress/click/press at :177-187).
- Its backend is createInteractionBackend
  (src/daemon/interaction/internal/interaction-runtime.ts:126), whose
  captureSnapshot (:132-139) forwards interactiveOnly, preferredBackend,
  includeRects, and the signal, and nothing cache-related.
- That capture calls captureSnapshotForSession (src/daemon/interaction/index.ts:21),
  which calls the daemon captureSnapshot (src/daemon/snapshot-capture.ts:77).
- captureSnapshot runs captureSnapshotAttempt, which runs captureSnapshotData
  (snapshot-capture.ts:99) and captures live through the bound capture or
  captureSnapshotWithInteractor on every call.
- The 750 ms cache (SELECTOR_CAPTURE_CACHE_TTL_MS,
  src/daemon/selector-capture-runtime.ts:24, read at :253 and :280) exists only
  inside createSelectorBackend (src/daemon/selector-runtime-backend.ts:142).
- createSelectorRuntimeForDevice is reached from the selector read and wait
  routes (selector-runtime.ts, wait-runtime.ts:158), never from
  interaction-touch-press.ts.

The runtime-selector targetReadiness contract scenario keyed its late target on
the removed option; it now keys on the capture count.

* fix(replay): share one readiness budget between the target gate and the dispatch

A replayed step has one readiness budget. The pre-dispatch target gate and the
dispatch each spent the full budget, so a target that rendered late could
cost a step twice the budget. When the gate polled more than once, the
dispatch now receives what is left, max(0, budget - gate waitedMs), as its
readinessTimeoutMs; 0 makes the dispatch resolve its target once. A gate that
matched on its first capture leaves the dispatch the whole budget.

A gate that polled more than once emits the interaction_target_readiness
diagnostic the dispatched wait emits, with polls, waitedMs, end, and the
step's command. A selector-miss divergence after such a wait carries the same
evidence at error.details.readiness, where an unannotated step's
target-not-found failure already carries it.

* fix(command-registry): pin the readiness cap without importing the selector policy

The envelope widening imported the promotedTarget selector row to read its
2 s poll ceiling, which made every command-registry entry and src/cli.ts
evaluate the selector policy module. The cap is now READINESS_BUDGET_MAX_MS in
timeout-policy.ts, pinned by test to the row's maxTimeoutMs; selectors cannot
import command-registry, so the test is the tie.

The same eager-closure gate showed two more new edges from this branch:
- replay's native and test command entries evaluated observe-until and the
  selector policy through the target gate. Both now load inside the gate's
  readiness path, the only code that uses them.
- src/cli.ts evaluated the new target-readiness-grammar module. Its one
  function moves into interaction/metadata.ts, its one caller, and its tests
  into metadata.test.ts.

* refactor(selectors): one owner for the readiness schedule

The replay target gate and the dispatch each built the promotedTarget
readiness schedule from the row's poll budget. readinessScheduleFor now lives
next to SELECTOR_PIPELINE_POLICIES and both callers use it: the dispatch
directly, the replay gate through its existing dynamic import.

The two waits also counted the budget from different origins: the gate from
the start of its wait, the dispatch from the end of its first capture. The
schedule now carries budgetFrom: 'first-capture' for both, and the budget the
gate leaves the dispatch is the step budget minus the gate's time after its
first capture ended. The first capture is the one-attempt lookup a step pays
without a wait, in the gate as in the dispatch.

* refactor(capture-kit): observeUntil defaults to no per-capture deadline; cancel is an opt-in

captureDeadline is now optional and defaults to 'none'. A capture only ends the
loop stalled when its caller declares 'cancel', so an adapter whose capture
ignores the signal cannot claim cancellation by omission. The selector
readiness poll keeps its explicit 'cancel': its poll signal reaches the
platform as CaptureSnapshotInput.signal. The replay target gate drops its
explicit 'none'.

* fix(replay): keep a thrown error's typed details on the replay response

An error thrown out of the replay run reached the wire as code and message
only; the catch rebuilt it with errorResponse and dropped details, hint, and
diagnostic references. A request canceled during the pre-dispatch target gate
therefore lost its request_canceled reason. The catch now normalizes the error
with normalizeError and appends the run's artifactPaths to its details.

* test(replay): cancelling the readiness gate ends at the next poll

The target gate's divergence capture takes no signal, so the gate honors a
canceled request only at a poll boundary. A request canceled during the gate's
second capture ends the wait when that capture returns: two captures, the
request_canceled reason on the response, and no dispatch.

* docs(replay): state that the readiness clock is test injection like AgentDeviceRuntime.clock

* fix(capture-kit): a poll never starts after the signal aborted

observeUntil checked the caller's signal only at the top of each iteration,
before the sleep. A cancel that arrived during the sleep still paid one more
capture before the loop noticed. The loop now checks the signal after the
sleep and ends with the typed request-canceled error before starting a poll.
The host-kit sleep takes no signal, so the check after the sleep is the
boundary.

Author: thymikee

packages/ad-replay/src/index.ts

@@ -60,7 +60,7 @@
  * and the classification/guard/binding-evidence/verification-routing shapes
  * `verifyAndDispatchStep` exchanges with the daemon's `AdReplayStepRuntime`
  * implementation (`AdReplayVerificationEntry`, `AdReplayTargetClassification`,
- * `AdReplayTargetBindingEvidence`, `AdReplayDispatchGuard`,
+ * `AdReplayTargetObservation`, `AdReplayTargetBindingEvidence`, `AdReplayDispatchGuard`,
  * `AdReplayDispatchOutcome`) ARE named here: the
  * daemon builds/reads real values of these shapes directly now (routing in
  * `session-replay-target-verification.ts`, wire-narrowing in
@@ -93,6 +93,7 @@ export type {
   AdReplayStepRuntime,
   AdReplayTargetBindingEvidence,
   AdReplayTargetClassification,
+  AdReplayTargetObservation,
   AdReplayVarSources,
   AdReplayVerificationEntry,
 } from './internal/runtime-port-types.ts';

packages/ad-replay/src/internal/__tests__/step-loop.test.ts

@@ -63,11 +63,8 @@ function createFakeRuntime(params: { isRepairArmed?: () => boolean } = {}): {
   let armCount = 0;
   const runtime: AdReplayStepRuntime = {
     beginTargetVerification: () => ({ kind: 'inactive' }),
-    captureObservation: async () => {
-      throw new Error('captureObservation: not used by this fixture (no targetEvidence)');
-    },
-    classifyTarget: () => {
-      throw new Error('classifyTarget: not used by this fixture (no targetEvidence)');
+    observeTarget: async () => {
+      throw new Error('observeTarget: not used by this fixture (no targetEvidence)');
     },
     async dispatchStep(dispatchedAction, _resolvedAction, _index, artifactPaths) {
       dispatched.push(dispatchedAction.command);
@@ -175,13 +172,8 @@ test("a post-dispatch target-binding mismatch reports the pre-step artifact snap
     // called for it — routed to the #1349 deferred-landmark path, which
     // dispatches with a guard WITHOUT any capture/classify round trip.
     beginTargetVerification: () => ({ kind: 'post-resolution', isSelectorWait: true }),
-    captureObservation: async () => {
-      throw new Error(
-        'captureObservation: not used — deferred-landmark skips straight to dispatch',
-      );
-    },
-    classifyTarget: () => {
-      throw new Error('classifyTarget: not used — deferred-landmark skips straight to dispatch');
+    observeTarget: async () => {
+      throw new Error('observeTarget: not used — deferred-landmark skips straight to dispatch');
     },
     async dispatchStep(dispatchedAction, _resolvedAction, _index, artifactPaths, _guard) {
       if (dispatchedAction.command === 'open') {

packages/ad-replay/src/internal/runtime-port-types.ts

@@ -97,9 +97,9 @@ export type AdReplayVerifiedTargetGuard = Readonly<{
   matchCount: number;
 }>;
 
-/** `captureObservation`'s neutral result: nodes for classification, or why a capture was not available. */
-export type AdReplayObservation = Readonly<
-  | { readonly state: 'available'; readonly nodes: readonly SnapshotNode[] }
+/** `observeTarget`'s neutral result: the recorded target's classification, or why no capture was available. */
+export type AdReplayTargetObservation = Readonly<
+  | { readonly state: 'classified'; readonly classification: AdReplayTargetClassification }
   | { readonly state: 'unavailable'; readonly reason: string; readonly hint?: string }
 >;
 
@@ -120,7 +120,7 @@ export type AdReplayVerificationEntry = Readonly<
     }
 >;
 
-/** `classifyTarget`'s result: a verified guard, or the divergence evidence a target-binding failure reports. */
+/** The recorded target's classification: a verified guard, or the divergence evidence a target-binding failure reports. */
 export type AdReplayTargetClassification = Readonly<
   | { readonly verified: true; readonly guard: AdReplayVerifiedTargetGuard }
   | Readonly<{
@@ -220,26 +220,20 @@ export type AdReplayStepRuntime = Readonly<{
     targetRole?: 'source' | 'destination',
   ): AdReplayVerificationEntry;
   /**
-   * Captures a fresh snapshot for classification or for a divergence's
-   * `screen` — daemon authority (`SessionStore`, the capture pipeline, the
-   * #1385 launch-race retry).
+   * Captures a fresh snapshot (with the #1385 launch-race retry) and resolves
+   * the recorded target against it using the SAME lookup/matching a real
+   * dispatch would — daemon authority (`SessionStore`, the capture pipeline,
+   * tree helpers and the selectors package engine). A step whose dispatch
+   * would wait for its target under a readiness budget re-captures while the
+   * target does not match yet, under that same budget, so this gate never
+   * refuses a target the dispatch itself would have waited for. The last
+   * capture is the divergence's `screen`.
    */
-  captureObservation(
-    action: SessionAction,
-    index: number,
-    options: { retryLaunchRace: boolean },
-  ): Promise<AdReplayObservation>;
-  /**
-   * Resolves the recorded target against `nodes` using the SAME
-   * lookup/matching a real dispatch would — daemon authority (tree helpers and
-   * the selectors package engine).
-   */
-  classifyTarget(params: {
+  observeTarget(params: {
     action: SessionAction;
     index: number;
     token: string;
-    nodes: readonly SnapshotNode[];
-  }): AdReplayTargetClassification;
+  }): Promise<AdReplayTargetObservation>;
   /**
    * Dispatches the action, optionally carrying a pre-action identity guard,
    * and detects the guard-mismatch / wait-landmark-mismatch post-resolution
@@ -279,7 +273,7 @@ export type AdReplayStepRuntime = Readonly<{
   ): Promise<AdReplayStepFailure>;
   /**
    * Builds a target-binding divergence from `evidence`, reusing the LAST
-   * `captureObservation` result for its `screen` (the pre-dispatch capture
+   * `observeTarget` capture for its `screen` (the pre-dispatch capture
    * and classification/capture-failure evidence share one capture) —
    * daemon authority. `artifactPaths` is the pre-step snapshot, as above;
    * `scrubVars` as above.

packages/ad-replay/src/internal/verify-dispatch.ts

@@ -8,10 +8,10 @@ import {
 import type {
   AdReplayDispatchGuard,
   AdReplayDispatchOutcome,
-  AdReplayObservation,
   AdReplayScrubValue,
   AdReplayStepOutcome,
   AdReplayStepRuntime,
+  AdReplayTargetObservation,
   AdReplayVerifiedTargetGuard,
 } from './runtime-port-types.ts';
 
@@ -25,8 +25,8 @@ import type {
  * those four functions live here — engine-private, never re-exported by the
  * façade — and the daemon side is the narrow `AdReplayStepRuntime`
  * capabilities this function drives: routing (`beginTargetVerification`),
- * capture (`captureObservation`), classification (`classifyTarget`),
- * dispatch (`dispatchStep`), and wire-building the resulting divergence
+ * capture-and-classification (`observeTarget`), dispatch (`dispatchStep`),
+ * and wire-building the resulting divergence
  * (`buildRecordedUnverifiableFailure`, `buildTargetBindingFailure`,
  * `buildPostDispatchTargetBindingFailure`). `./step-loop.ts`'s `runAdReplay`
  * is this module's one caller.
@@ -121,11 +121,8 @@ export async function verifyAndDispatchStep(
   }
   const token = preDispatchPlan.token;
 
-  // #1385: this is the pre-dispatch gate a step right after `open --relaunch`
-  // can race — the app may still be launching/mounting when this capture
-  // lands. Bounded retry rides out that transition (`retryLaunchRace`).
-  const observation = await runtime.captureObservation(action, index, { retryLaunchRace: true });
-  if (observation.state !== 'available') {
+  const observation = await runtime.observeTarget({ action, index, token });
+  if (observation.state !== 'classified') {
     return {
       status: 'failed',
       failure: await runtime.buildTargetBindingFailure(
@@ -138,7 +135,7 @@ export async function verifyAndDispatchStep(
     };
   }
 
-  const classification = runtime.classifyTarget({ action, index, token, nodes: observation.nodes });
+  const { classification } = observation;
   if (classification.verified) {
     return dispatchWithGuard(runtime, scrubVars, action, resolvedAction, index, artifactPaths, {
       kind: 'target',
@@ -213,10 +210,12 @@ async function verifyAndDispatchMultiTargetStep(
       };
     }
 
-    const observation = await runtime.captureObservation(endpointAction, index, {
-      retryLaunchRace: true,
+    const observation = await runtime.observeTarget({
+      action: endpointAction,
+      index,
+      token: plan.token,
     });
-    if (observation.state !== 'available') {
+    if (observation.state !== 'classified') {
       return {
         status: 'failed',
         failure: await runtime.buildTargetBindingFailure(
@@ -229,12 +228,7 @@ async function verifyAndDispatchMultiTargetStep(
       };
     }
 
-    const classification = runtime.classifyTarget({
-      action: endpointAction,
-      index,
-      token: plan.token,
-      nodes: observation.nodes,
-    });
+    const { classification } = observation;
     if (!classification.verified) {
       return {
         status: 'failed',
@@ -268,7 +262,7 @@ async function verifyAndDispatchMultiTargetStep(
 }
 
 function captureUnavailableEvidence(
-  observation: Extract<AdReplayObservation, { state: 'unavailable' }>,
+  observation: Extract<AdReplayTargetObservation, { state: 'unavailable' }>,
 ) {
   return {
     kind: 'identity-unverifiable' as const,

packages/ad-script/src/internal/target-annotation-identity.ts

@@ -52,7 +52,7 @@ type IdentityTreeNode = Pick<RawSnapshotNode, 'type' | 'identifier' | 'label'>;
  * (`@agent-device/selectors/target-evidence`), replay-time verification
  * (`packages/replay-port/src/daemon-port/session-replay-target-verification.ts`), and the
  * dispatch-side post-resolution guard
- * (`src/commands/interaction/runtime/resolution.ts`), so all three compute
+ * (`src/commands/interaction/runtime/replay-target-guard.ts`), so all three compute
  * a node's identity with byte-identical semantics.
  */
 export function readNodeLocalIdentity(

packages/capture-kit/package.json

@@ -46,6 +46,10 @@
       "types": "./src/capture-admission/durable-capture-runtime-recovery.ts",
       "default": "./src/capture-admission/durable-capture-runtime-recovery.ts"
     },
+    "./observe-until": {
+      "types": "./src/observe-until.ts",
+      "default": "./src/observe-until.ts"
+    },
     "./perf-capture-admission-ledger": {
       "types": "./src/capture-admission/perf-capture-admission-ledger.ts",
       "default": "./src/capture-admission/perf-capture-admission-ledger.ts"