Skip to content

harness: crash recovery leaves the session unusable — restored turn races worker boot, and the interrupted function_call is never closed #507

Description

@paulefde

Summary

First: thank you for MOT-3944 (harness-turn queue) — on harness 1.3.1 an
engine crash during a dispatched function call left the turn wedged in
running forever (stop/sweep/restart could not reap it). On 1.3.3 + engine
0.21.6 that wedge is gone: the restored queue job drives the turn to a
terminal state after restart, harness::send is accepted again, and the
interrupted call is not re-executed (at-most-once held in our test).

Two problems remain on that recovery path, and together they still cost the
session:

  1. Restored jobs race worker boot. The restored harness-turn job is
    delivered once harness::turn registers, but the turn step then calls
    context::assemble before context-manager has registered, so the turn
    fails terminally with
    harness/dependency: context::assemble: remote error (function_not_found)
    instead of resuming. The transient-resume mechanism did not cover this
    (transient_resumes: 0 with max_transient_resumes: 1).
  2. The interrupted call is never closed. The assistant entry with the
    dispatched function_call (our toolu_... id) stays in the session with
    no function_result ever appended — pending_function_calls still lists
    it on the failed turn. Every subsequent turn on that session then fails:
    the provider request carries a tool call with no result, and each attempt
    ends with provider anthropic stream ended without a terminal frame
    (one transient recovery retry, same outcome, turn failed). A fresh
    session on the same engine completes normally, so this is transcript
    poisoning, not a provider problem.

Reproduction (harness 1.3.3, engine 0.21.6, macOS arm64)

  1. Register a slow function (ours sleeps 10s, then appends to a log file).
  2. harness::send a turn that calls it (expose: "native", allow-listed).
  3. SIGKILL the engine while the function is executing (inside the 10s
    window). The worker process survives and completes its side effect; the
    result has nowhere to go.
  4. Restart the engine.
  5. Observe harness::status: turn failed with the context::assemble
    dependency error (problem 1), pending_function_calls still listing the
    interrupted call.
  6. Send any new message on the session: the new turn fails with
    provider anthropic stream ended without a terminal frame, repeatably
    (problem 2). A brand-new session works fine.

Expected

  • Restored jobs should wait for (or transiently retry on) the harness's own
    function dependencies, not just harness::turn registration, so a
    post-restart turn resumes instead of failing on a boot race.
  • On recovery, an interrupted function_call should be closed — e.g. a
    synthesized error function_result ("execution interrupted by engine
    restart") — so the session remains usable and the model can decide whether
    to retry. Bonus: that also gives the model an honest signal that the side
    effect may or may not have happened.

Why it matters

The queue work fixed the liveness gap, but from a product perspective the
outcome is nearly the same as before: one engine crash mid-call still
permanently kills that conversation — it just fails fast now instead of
hanging. Closing the dangling call is the missing piece that would make
crash recovery actually recover.

Metadata

Metadata

Labels

bugSomething isn't working

Type

Fields

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions