Closed-Loop Robot Agent Harness

[easydunker]

Summary

Add an opt-in agent harness that closes the loop between model decisions and observed robot outcomes while preserving dimOS's existing @skill and MCP tool model.

The proposed harness changes would be:

1. durably record standard tool calls and their results;
2. build and inject bounded world snapshots, then revalidate action assumptions against fresh state immediately before physical dispatch; and
3. distinguish acceptance of a physical command from completion of the robot action.

The goal is to make robot interactions observable and outcome-aware without replacing standard agent tools with a new action protocol.

observe world -> model decides -> standard tool call -> robot executes
      ^                                                  |
      +--------- verify outcome and decide again <-------+

Motivation

The current McpClient is a conventional LangGraph tool-using agent. It discovers @skill methods as MCP tools, keeps conversation history in process memory, and lets the model decide when to call observation tools.

This works well for general tool use, but physical execution has additional semantics. For example, navigate_with_text can call set_goal() and immediately return that navigation has started. The return confirms command acceptance, not that the robot reached its destination. Later controller feedback is not durably tied to the model tool call that started the motion.

The current runtime therefore cannot reliably reconstruct:

• what robot state the model saw before acting;
• which tool call initiated a movement;
• whether a command was rejected, accepted, running, or completed; and
• what evidence established the physical outcome.

Goals

• Keep standard model tool calls as the canonical invocation mechanism.
• Preserve existing @skill, MCP, RPC, module, blueprint, and typed-stream behavior.
• Record every tool call and result dispatched through the harness.
• Prevent a managed physical action from running when relevant world state has changed since the model decided to act.
• Correlate bounded physical actions with controller feedback and observed state.
• Never infer physical success from a successful RPC return alone.
• Give the model fresh robot context without requiring it to poll first.
• Support deterministic testing and replay without a live model or robot.

Non-goals

• Replacing MCP tools with a dimOS-specific action API.
• Reimplementing low-level control in the agent harness.
• Changing existing capability coordination or adding general resource arbitration.
• Defining approval or authorization policy.
• Cancellation or recovery of in-flight work after restart.
• Supporting continuous behaviors such as patrol and follow in the first version.
• Sending raw high-rate sensor streams into every model turn.

Emergency stop and low-level safety paths must remain independent of the model, journal, and harness.

Proposed design

robot state streams ------> WorldSnapshotBuilder -----+
                                                     |
                                                     v
human/salient event -> RobotAgentHarness -> model context -> model
                            |                              |
                            |                              v
                            |                    standard tool calls
                            |                              |
                            v                              v
                      MissionJournal <----- PreconditionValidator
                            ^                              |
                            |                              v
controller events ----> RobotActionTracker <--- RecordedToolDispatcher
                            |                              |
                            |                              v
                            +---- verified outcome + existing @skill

Standard tool recording

Every call passing through the harness is recorded, including observation, query, speech, computation, and physical tools. A call remains a standard tool call and receives exactly one standard result.

The durable record should include:

• mission, model-turn, and standard tool-call IDs;
• tool name and arguments, subject to redaction;
• dispatch and completion timestamps;
• returned content or structured error; and
• an optional robot-action ID when physical work was accepted.

Recording occurs at the dispatch boundary. Calls made directly by another MCP client are outside this guarantee unless that path is routed through the same recorder.

Harness-managed world snapshots

The harness captures a snapshot:

1. before every model turn;
2. immediately before dispatching a managed physical action;
3. after a bounded physical action reports a terminal condition and before verification; and
4. after a configured salient robot event.

A navigation-oriented snapshot initially contains:

• pose and velocity;
• navigation goal and controller state;
• active action and recent outcome; and
• source timestamp, age, and availability for each field.

Snapshot fields may be missing or stale, and that state must remain explicit. Large camera, point-cloud, and map values are stored as artifact references rather than inline data.

The model receives a size-bounded projection, not raw high-rate streams. It may still call tools such as observe() when it needs a current image, more detail, or a different viewpoint.

Pre-dispatch world validation

Each model turn is bound to the ID of the snapshot used to build its context. Before dispatching a managed physical action, the harness captures a second, fresh snapshot and compares the action-relevant state with the decision snapshot.

This is not a byte-for-byte comparison of the whole world. Each managed action declares the fields and tolerances that must still hold, for example:

• maximum age of required observations;
• maximum robot pose drift since the decision;
• unchanged controller and motion state;
• no newly active conflicting robot action;
• unchanged target identity or location within tolerance; and
• acceptable safety or battery state when relevant.

If the declared preconditions still hold, the harness records the successful check and dispatches the unchanged standard tool call. If they do not, the harness does not invoke the physical skill. It records the failed check, returns one standard tool result such as precondition_changed, and begins a new model turn using the fresh snapshot.

A failed precondition check does not create a robot action because physical execution never began.

The validator only determines whether declared assumptions remain valid. Route feasibility, collision checking, and low-level safety remain responsibilities of their existing planners and controllers.

Managed robot actions

dimOS already supports instant and background skill lifecycles and capability declarations through @skill. This proposal should extend those concepts rather than introduce a parallel lifecycle.

For a bounded background action, optional metadata needs to identify:

• the typed progress/completion source;
• how events correlate with the invocation;
• snapshot fields and tolerances required before dispatch;
• the verifier to run against fresh state;
• snapshot fields required by the verifier; and
• the execution timeout.

This metadata must not change the tool's MCP name, description, argument schema, or normal calling convention. Skills without it remain ordinary tools.

The harness tracks physical execution through:

REQUESTED -> ACCEPTED -> RUNNING -> VERIFYING -> SUCCEEDED
                                              -> FAILED
                         \--------------------> UNKNOWN

• REQUESTED: dispatch began and the tool call was recorded.
• ACCEPTED: the controller accepted the command.
• RUNNING: correlated feedback reports ongoing execution.
• VERIFYING: terminal feedback caused a fresh snapshot capture.
• SUCCEEDED: feedback and observed state establish the requested outcome.
• FAILED: execution failed or verification disproved success.
• UNKNOWN: execution timed out or the evidence is inconclusive.

The immediate standard tool result reports what the synchronous invocation established: rejection, acceptance, or true synchronous completion. If it reports acceptance, the later physical outcome is a correlated runtime event supplied to the next model turn, not a second result for the same tool call.

Closed-loop flow

For a bounded action such as navigation:

1. Capture and persist a world snapshot.
2. Inject its bounded projection into model context.
3. Invoke and record the model response.
4. Record the standard tool call and bind it to that decision snapshot.
5. Capture fresh pre-dispatch state and validate the action's declared preconditions.
6. If validation fails, do not invoke the physical skill; return the standard failure result and reconsider using the fresh snapshot.
7. If validation passes, dispatch the tool and record its immediate result.
8. Correlate subsequent controller events with the accepted action.
9. On terminal feedback or timeout, capture fresh state and verify the outcome.
10. Record the outcome and begin the next model turn with the new evidence.

Ordinary tools continue through the normal model/tool loop. The initial harness accepts at most one new managed physical action in a decision step. Continuous behaviors need different decision-boundary semantics and are deferred until the bounded-action design is validated.

Durable event model

The mission journal stores ordered, versioned events such as:

• MODEL_TURN_STARTED;
• WORLD_SNAPSHOT_CAPTURED;
• MODEL_RESPONSE_RECORDED;
• TOOL_CALL_RECORDED;
• ACTION_PRECONDITION_CHECKED;
• TOOL_RESULT_RECORDED;
• ROBOT_ACTION_STARTED;
• ROBOT_ACTION_PROGRESS; and
• ROBOT_ACTION_OUTCOME.

Each event includes a sequence number, timestamp, mission ID, and relevant tool-call or robot-action IDs. SQLite is sufficient for the first version. The journal is for causal reconstruction and evaluation, not a new source of conversational memory.

Failure behavior

• Tool exceptions produce one recorded standard error result.
• Changed or stale preconditions prevent physical dispatch and return one recorded standard failure result.
• Controller rejection produces a failed action outcome.
• Missing completion feedback produces UNKNOWN after timeout.
• Uncorrelated controller events cannot complete an action.
• Conflicting controller and snapshot evidence cannot produce success.
• Failure to persist a required record prevents new agent-initiated physical work.
• Independent emergency and low-level safety controls bypass the harness.

Compatibility

• Existing skills without action metadata remain ordinary tools.
• Existing MCP clients require no protocol changes.
• Existing capability declarations and coordination remain unchanged.
• Existing non-harness agentic blueprints remain available.
• The new harness is enabled through an opt-in blueprint during evaluation.

Risks and open questions

• Snapshot projection must be bounded to avoid context growth and duplication.
• Some controllers may need adapters before they can provide stable correlation IDs.
• Precondition comparison must be action-specific; comparing entire snapshots would reject valid actions because normal robot state is always changing.
• Verifiers must preserve uncertainty as UNKNOWN rather than creating false confidence.
• It remains open whether action metadata belongs directly on @skill, in an adapter registry, or supports both.
• The prototype should determine a backward-compatible structured result shape for command acceptance.
• The navigation vertical slice should identify the smallest reliable replay fixture for end-to-end testing.