Summary
First: thank you for MOT-3944 (harness-turn queue) — on harness 1.3.1 an
engine crash during a dispatched function call left the turn wedged in
running forever (stop/sweep/restart could not reap it). On 1.3.3 + engine
0.21.6 that wedge is gone: the restored queue job drives the turn to a
terminal state after restart, harness::send is accepted again, and the
interrupted call is not re-executed (at-most-once held in our test).
Two problems remain on that recovery path, and together they still cost the
session:
- Restored jobs race worker boot. The restored
harness-turn job is
delivered once harness::turn registers, but the turn step then calls
context::assemble before context-manager has registered, so the turn
fails terminally with
harness/dependency: context::assemble: remote error (function_not_found)
instead of resuming. The transient-resume mechanism did not cover this
(transient_resumes: 0 with max_transient_resumes: 1).
- The interrupted call is never closed. The assistant entry with the
dispatched function_call (our toolu_... id) stays in the session with
no function_result ever appended — pending_function_calls still lists
it on the failed turn. Every subsequent turn on that session then fails:
the provider request carries a tool call with no result, and each attempt
ends with provider anthropic stream ended without a terminal frame
(one transient recovery retry, same outcome, turn failed). A fresh
session on the same engine completes normally, so this is transcript
poisoning, not a provider problem.
Reproduction (harness 1.3.3, engine 0.21.6, macOS arm64)
- Register a slow function (ours sleeps 10s, then appends to a log file).
harness::send a turn that calls it (expose: "native", allow-listed).
- SIGKILL the engine while the function is executing (inside the 10s
window). The worker process survives and completes its side effect; the
result has nowhere to go.
- Restart the engine.
- Observe
harness::status: turn failed with the context::assemble
dependency error (problem 1), pending_function_calls still listing the
interrupted call.
- Send any new message on the session: the new turn fails with
provider anthropic stream ended without a terminal frame, repeatably
(problem 2). A brand-new session works fine.
Expected
- Restored jobs should wait for (or transiently retry on) the harness's own
function dependencies, not just harness::turn registration, so a
post-restart turn resumes instead of failing on a boot race.
- On recovery, an interrupted
function_call should be closed — e.g. a
synthesized error function_result ("execution interrupted by engine
restart") — so the session remains usable and the model can decide whether
to retry. Bonus: that also gives the model an honest signal that the side
effect may or may not have happened.
Why it matters
The queue work fixed the liveness gap, but from a product perspective the
outcome is nearly the same as before: one engine crash mid-call still
permanently kills that conversation — it just fails fast now instead of
hanging. Closing the dangling call is the missing piece that would make
crash recovery actually recover.
Summary
First: thank you for MOT-3944 (harness-turn queue) — on harness 1.3.1 an
engine crash during a dispatched function call left the turn wedged in
runningforever (stop/sweep/restart could not reap it). On 1.3.3 + engine0.21.6 that wedge is gone: the restored queue job drives the turn to a
terminal state after restart,
harness::sendis accepted again, and theinterrupted call is not re-executed (at-most-once held in our test).
Two problems remain on that recovery path, and together they still cost the
session:
harness-turnjob isdelivered once
harness::turnregisters, but the turn step then callscontext::assemblebeforecontext-managerhas registered, so the turnfails terminally with
harness/dependency: context::assemble: remote error (function_not_found)instead of resuming. The transient-resume mechanism did not cover this
(
transient_resumes: 0withmax_transient_resumes: 1).dispatched
function_call(ourtoolu_...id) stays in the session withno
function_resultever appended —pending_function_callsstill listsit on the failed turn. Every subsequent turn on that session then fails:
the provider request carries a tool call with no result, and each attempt
ends with
provider anthropic stream ended without a terminal frame(one transient recovery retry, same outcome, turn
failed). A freshsession on the same engine completes normally, so this is transcript
poisoning, not a provider problem.
Reproduction (harness 1.3.3, engine 0.21.6, macOS arm64)
harness::senda turn that calls it (expose: "native", allow-listed).window). The worker process survives and completes its side effect; the
result has nowhere to go.
harness::status: turnfailedwith thecontext::assembledependency error (problem 1),
pending_function_callsstill listing theinterrupted call.
provider anthropic stream ended without a terminal frame, repeatably(problem 2). A brand-new session works fine.
Expected
function dependencies, not just
harness::turnregistration, so apost-restart turn resumes instead of failing on a boot race.
function_callshould be closed — e.g. asynthesized error
function_result("execution interrupted by enginerestart") — so the session remains usable and the model can decide whether
to retry. Bonus: that also gives the model an honest signal that the side
effect may or may not have happened.
Why it matters
The queue work fixed the liveness gap, but from a product perspective the
outcome is nearly the same as before: one engine crash mid-call still
permanently kills that conversation — it just fails fast now instead of
hanging. Closing the dangling call is the missing piece that would make
crash recovery actually recover.