fix(core): preserve queued messages after recall - #1569
Closed
AaronZ345 wants to merge 1 commit into
Closed
Conversation
AaronZ345
force-pushed
the
fix/recall-preserve-queued-messages-20260719
branch
from
July 19, 2026 14:41
1ef393a to
f1f8a9a
Compare
AaronZ345
force-pushed
the
fix/recall-preserve-queued-messages-20260719
branch
2 times, most recently
from
July 25, 2026 14:41
4292d91 to
e1c4e8f
Compare
AaronZ345
force-pushed
the
fix/recall-preserve-queued-messages-20260719
branch
2 times, most recently
from
August 7, 2026 14:42
a5f4183 to
e9771a4
Compare
AaronZ345
force-pushed
the
fix/recall-preserve-queued-messages-20260719
branch
2 times, most recently
from
August 14, 2026 14:43
cd4a767 to
037be5f
Compare
chenhg5
approved these changes
Aug 15, 2026
chenhg5
left a comment
Owner
There was a problem hiding this comment.
Conclusion: Approve
Overall assessment:
- This is a focused, well-scoped fix for the CUJ-A8 "messages queued after a recall are silently dropped" bug. The author splits the recovery into two complementary pieces —
resumePendingAfterStoppedTurn(new) recovers queued successors when the turn ends in aStoppedstate, and theisStopped()short-circuit inprocessInteractiveEventsprevents the engine from mis-treating that same intentional close as a crash and draining the queue a second time. Lock ownership is explicitly transferred to the nested processor viaunlocked = true, which mirrors the existing pattern in the file. Tests are strong: a CUJ-A8 scenario with three messages (active, recalled, survivor) plus a targeted engine test using acontrollableAgentwhose session can be swapped betweenoldSessionandreplacementSessionso we observe the replacement session receiving only the survivor.
Review scope:
- Reviewed
core/engine.go(the newresumePendingAfterStoppedTurn, thechannelClosedshort-circuit, and the two removedpendingMessages = nilblocks instopInteractiveSessionWithOptions) plus the new tests incore/cuj_test.goandcore/engine_test.go. - Focused on race safety, lock ownership handoff, FIFO ordering under the placeholder pattern, and backward compatibility of the silent-recall path.
✅ What looks good:
- FIFO preservation via the placeholder pattern.
resumePendingAfterStoppedTurntakes[P0..Pn]from the stopped state, movesPninto a placeholder entry, and starts the nested processor onP0. The freshly-created state ingetOrCreateInteractiveStateWiththen re-uses the existingadoptPendingFromPlaceholderto inherit the remainder. New messages that arrive between the unlock of the stopped state and the re-creation of the placeholder are appended afterPn, preserving order. - Lock handoff is explicit.
unlocked = trueis set immediately before the recursiveprocessInteractiveMessageWithcall; the deferred fallbackif !unlocked { session.Unlock() }then skips the unlock on the outer call, and the inner call's defer releases the lock exactly once. This matches the established handoff pattern indrainOrphanedQueue/drainPendingMessages. - Deduplication of "channel close" handling. Removing the
state.pendingMessages = nilblock from the non-notifyQueuedbranches ofstopInteractiveSessionWithOptionsmeans the silent-recall path now relies onresumePendingAfterStoppedTurnto recover the queue — and theisStopped()short-circuit atchannelClosed:correctly returns early beforenotifyDroppedQueuedMessages/cleanupInteractiveStatewould otherwise drain the queue. The two changes are independent but they compose: removing the wipe without the short-circuit would leak a queue; adding the short-circuit without removing the wipe would lose the queue. Together they work. - Tests cover both unit and CUJ angles.
TestCUJ_A8_RecallPreservesLaterQueuedMessageexercises the full Engine+CUJ path, andTestHandleMessageRecallPreservesLaterQueueWhenAgentChannelClosesexercises the fast path (channel close mid-recall). Together they lock in the FIFO invariant and the no-double-drain invariant. slog.Info("resuming queued messages after stopped turn", …)gives operators a breadcrumb to grep when triaging — the line we wanted in production is now there.
🚨/🔴 Must fix:
- None.
🟠 Should improve:
- Minor: log a parallel breadcrumb on the
channelClosed:short-circuit. Right now the only signal that "the channel close was a recall, not a crash" is the absence of the previous "agent process exited" warn line. Addingslog.Debug("agent channel closed after intentional stop; deferring queue recovery", "session_key", sessionKey)(orInfo, depending on volume) would make the connection between the two pieces of this PR obvious in logs. Pure observability nit. - Minor: consider using
state.markStopped()consistently. The newresumePendingAfterStoppedTurnonly readsstate.stopped; it does not callmarkStopped()itself. That's correct because the caller (stopInteractiveSessionWithOptions) already marks the state stopped and removes it from the map. But it's worth a one-line comment at the top of the new function pointing at the only call site, so future readers understand the invariant ("caller must have already calledmarkStopped()anddelete(e.interactiveStates, …)"). - Minor: When the placeholder is freshly allocated (
placeholder == nil || placeholder == state), it iseventsNeedResync: true. AfterprocessInteractiveMessageWithreplaces it with a real state viaadoptPendingFromPlaceholder, the new state'seventsNeedResyncis alsotrue(fromgetOrCreateInteractiveStateWith's constructor) — so the flag round-trip is correct. Worth a one-line comment so a future reader doesn't "optimize" one of them away.
🔵 Optional:
- A test that places a NEW message between
e.stopInteractiveSessionWithOptionsreturning andresumePendingAfterStoppedTurnacquiringinteractiveMuwould exercise the "messages appended to placeholder" branch explicitly. The current CUJ test only checks the case where the placeholder is freshly created with no pre-existing pending; the merge-with-prior-placeholder path is exercised by the implementation but not asserted on. Not blocking. - A test for the
eventsNeedResync: trueinvariant on the replacement session would close the small "did we forget to drain stale events from the old turn?" gap. Again, not blocking.
❓ Questions:
- Did the silent-recall path used to notify the user of dropped queued messages? Looking at the diff, the old code in the
elsebranch (whennotifyQueued == false) dropped without notification, and the new code preserves those messages instead. So this is a net-positive behavior change — but it would be good to confirm with PM that no caller was relying on the old "silently drop queue on recall" semantics.
Testing / Risk:
- Verified:
go test -race ./core/... -count=1 -timeout 180s(48.4s, all green); new tests pass; the full CUJ test suite (TestCUJ_A6_A7…,TestCUJ_A8…) is green. - Blast radius is
core/engine.goonly. No agent-runtime, no platform, no CLI touched. - Backward compatible for the recall target (the recalled message is still silently stopped); the only observable behavior change is "messages queued after the recall now get a response instead of being silently dropped." This is exactly the contract issue #1499 / #1498 callers have been asking for.
Next step:
- Merge. The two 🟠 minor comments are nice-to-have for the next maintainer; they are not required for correctness.
AaronZ345
force-pushed
the
fix/recall-preserve-queued-messages-20260719
branch
2 times, most recently
from
August 16, 2026 14:44
137a4ca to
f9837e8
Compare
AaronZ345
force-pushed
the
fix/recall-preserve-queued-messages-20260719
branch
from
August 17, 2026 14:41
f9837e8 to
1fc4a64
Compare
Contributor
Author
|
Superseded by #1706. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root cause
The silent recall path cleared the entire
pendingMessagesqueue when stopping the recalled turn. Messages sent after the recalled message were therefore accepted and acknowledged, then discarded without a response.Tests
go test -race ./core -run 'TestCUJ_A8_RecallPreservesLaterQueuedMessage|TestHandleMessageRecallPreservesLaterQueueWhenAgentChannelCloses' -count=1go test ./core -count=1go test -tags no_web ./... -count=1