Checks
SDK Language: TypeScript and Python
Strands Version: TypeScript main at 100429263; Python PR #4240 at 4b38b7443
Language Runtime Version: Not runtime-specific
Operating System: Not operating-system-specific
Installation Method: From source
Summary
Parallel graph nodes can save multi-agent checkpoints concurrently to the same snapshot_latest key. On an asynchronous storage backend, an older write can finish after a newer write and move the persisted checkpoint backward.
Both SDKs capture the complete orchestrator state and write it after each completed node when the node save strategy is active. Neither implementation serializes capture and write per orchestrator.
Steps to Reproduce
- Create a Graph with nodes that complete in parallel and enable the default per-node snapshot strategy.
- Use a storage implementation that can keep multiple writes in flight, such as S3, or a deterministic test storage that delays selected writes.
- Let node A capture an earlier state and begin a slow write.
- Let node B capture newer state and complete its write first.
- Allow node A's older write to finish last.
- Stop the process before the invocation-end save repairs the checkpoint.
- Restore the Graph and observe that work represented only by node B's checkpoint is missing and may execute again.
Expected Behavior
The latest snapshot must not move backward. A checkpoint written after more nodes complete should remain authoritative regardless of storage latency.
Actual Behavior
Concurrent writes race on the same key. The last write to finish wins even when it contains an older execution frontier. A restart can re-run completed, non-idempotent work.
Python's synchronous LocalFileStorage write path does not yield during replacement, but asynchronous remote storage exposes the race. TypeScript has the same uncoordinated save path.
Additional Context
Found while reviewing the Python TypeScript-parity port in #4240. The original discussion is #4240 (comment).
The final invocation save normally repairs the latest snapshot. The failure window is a crash or interruption after out-of-order node saves and before that final save completes.
Possible Solution
Serialize snapshot capture and storage write per orchestrator location. The lock or queue must cover both operations so every later capture observes at least the state represented by the prior completed save. The invocation-end save must use the same serialization mechanism.
Add a deterministic test that blocks the first write, allows a newer save request to queue, releases the first write, and verifies that storage ends with the newer snapshot. Implement the ordering guarantee in TypeScript first, then port it to Python.
Related Issues
Checks
mainat100429263d4f7ae114416b2670e7afd6a49a56b6and the Python parity implementation in feat(session-py): support snapshot session management for Graph and Swarm #4240snapshot_latestafter newer state is saved is not expected behaviorSDK Language: TypeScript and Python
Strands Version: TypeScript
mainat100429263; Python PR #4240 at4b38b7443Language Runtime Version: Not runtime-specific
Operating System: Not operating-system-specific
Installation Method: From source
Summary
Parallel graph nodes can save multi-agent checkpoints concurrently to the same
snapshot_latestkey. On an asynchronous storage backend, an older write can finish after a newer write and move the persisted checkpoint backward.Both SDKs capture the complete orchestrator state and write it after each completed node when the node save strategy is active. Neither implementation serializes capture and write per orchestrator.
Steps to Reproduce
Expected Behavior
The latest snapshot must not move backward. A checkpoint written after more nodes complete should remain authoritative regardless of storage latency.
Actual Behavior
Concurrent writes race on the same key. The last write to finish wins even when it contains an older execution frontier. A restart can re-run completed, non-idempotent work.
Python's synchronous
LocalFileStoragewrite path does not yield during replacement, but asynchronous remote storage exposes the race. TypeScript has the same uncoordinated save path.Additional Context
Found while reviewing the Python TypeScript-parity port in #4240. The original discussion is #4240 (comment).
The final invocation save normally repairs the latest snapshot. The failure window is a crash or interruption after out-of-order node saves and before that final save completes.
Possible Solution
Serialize snapshot capture and storage write per orchestrator location. The lock or queue must cover both operations so every later capture observes at least the state represented by the prior completed save. The invocation-end save must use the same serialization mechanism.
Add a deterministic test that blocks the first write, allows a newer save request to queue, releases the first write, and verifies that storage ends with the newer snapshot. Implement the ordering guarantee in TypeScript first, then port it to Python.
Related Issues