Retry the subject purge once on a Postgres deadlock - #389
Merged
Conversation
Main CI run 33638444055 (09-02) failed with `DELETE FROM memories` deadlocking in tests/integration/test_semantic.py: the purge's multi-row DELETEs can cross lock order with another multi-row writer on the same subject's rows (a still-draining compile batch, an embedding backfill). Postgres aborts exactly one side (SQLSTATE 40P01), which the endpoint surfaced as a raw 500. A single rollback-and-redo converges — the competing transaction has either finished or loses the rematch — so the purge now retries once on a deadlock-shaped DBAPIError (matched via the wrapped driver error) and logs `subject_delete_deadlock_retry`. Any other DB error, or a second deadlock, still raises immediately. Covered by unit tests: retry-then-succeed, give-up-after-second, no-retry-on-non-deadlock.
smaramwbc
force-pushed
the
fix/purge-deadlock-retry
branch
from
September 6, 2026 12:10
bbbbb1d to
8a36c35
Compare
The #384 concurrent-convergence test failed on this PR's CI run: the exact-match and cosine-merge paths appended linked_memory_ids via ORM read-modify-write, so two concurrent writers that both read the same committed array each wrote back their own copy and silently dropped the other's link. The unique index made concurrent writers converge on ONE row, but not on one array. Both paths now append through a single guarded SQL UPDATE (the same ANY()/array_append CASE the ON CONFLICT branch uses): the UPDATE takes the row lock and re-evaluates against the latest committed array, so concurrent appends serialize, and the row is refreshed so callers see the post-append state. Concurrency test now passes 12/12 consecutive local runs; it failed nondeterministically before.
Owner
Author
|
The CI failure on the first push was the #384 concurrent-convergence test catching a REAL second race, not a flake: the exact-match and cosine-merge paths appended |
This was referenced Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the main-CI failure on run 33638444055:
DELETE FROM memoriesdeadlocked against a concurrent multi-row writer on the same subject (integration flake — first occurrence of this race; the entity upsert from #384 never ran in that job, litellm being absent, so this is a surfaced pre-existing race, not a #384 regression). The purge now rolls back and redoes its transaction once on SQLSTATE 40P01 — Postgres aborts exactly one deadlock participant, so the redo converges. Non-deadlock errors and a second deadlock still raise immediately.3 new unit tests; 1052 unit + 288 integration passed; ruff clean. The failed run itself has been rerun to confirm flakiness.