Flaky concurrency tests on main: S3 conditional-write races no longer resolve deterministically

[orekav]

What's happening

Multiple pre-existing integration tests that assert strict "exactly one concurrent request wins, the rest get a specific conflict status" outcomes against S3 conditional writes are flaking intermittently in CI. Two different tests have failed so far, on two separate CI runs, each with a different symptom:

1. test_remove_sync_factor_etag_concurrency (tests/delete_factor_integration.rs)
   Failed on the CI run for PR feat: prove backup account key on backup creation #243's merge into main:
   https://github.com/worldcoin/backup-service/actions/runs/32370987568/job/96431263059

   thread 'test_remove_sync_factor_etag_concurrency' panicked at tests/delete_factor_integration.rs:539:5:
   assertion `left == right` failed: Expected exactly one operation to succeed
   

2. test_concurrent_sync_backup_prevention (tests/sync_backup_integration.rs)
   Failed on a rerun of PR test: cover Production enforcement of the backup account proof #245's CI:
   https://github.com/worldcoin/backup-service/actions/runs/32372620542/job/96439189750

   thread 'test_concurrent_sync_backup_prevention' panicked at tests/sync_backup_integration.rs:350:21:
   Unexpected status 412 Precondition Failed: {"allowRetry":false,"error":{"code":"manifest_hash_mismatch","message":"The manifest hash does not match the expected value."}}
   

Neither test's code path was touched by the PRs whose CI runs surfaced them (#243, #245).

Why this looks systemic rather than one flaky test

• Two different tests, in two different files, both in the same family (spawn several concurrent requests against the same backup, assert a specific single-winner/conflict outcome from S3's conditional-write semantics).
• chore: version bumps & RUSTSEC-2026-0258 #244 ("chore: version bumps & RUSTSEC-2026-0258", bumping the AWS SDK for S3/DynamoDB/KMS and h2) merged with a clean CI run on its own.
• The very next merge to main (feat: prove backup account key on backup creation #243, which had already merged main — including chore: version bumps & RUSTSEC-2026-0258 #244 — into its branch) failed on the first test above.
• A subsequent CI rerun on a branch descending from that same main failed on the second, different test.

This pattern is consistent with the AWS SDK bump in #244 changing some timing/retry behavior (e.g. default retry-on-transient-error) enough that concurrent conditional PUTs against local S3 (LocalStack) no longer reliably resolve to exactly one winner the way these tests assume — rather than either test being inherently flaky on its own.

Suggested next steps

• Try to reproduce locally by looping the two tests above a few dozen times against LocalStack.
• Diff the AWS SDK S3 client's default retry/backoff config between the pre- and post-chore: version bumps & RUSTSEC-2026-0258 #244 versions.
• If confirmed, either adjust the tests to tolerate the SDK's new retry behavior, or pin/configure the S3 client's retry config back to the previous behavior for these conditional-write paths.

Not blocking any specific feature PR — flagging so it doesn't keep silently costing CI reruns on unrelated PRs.

[orekav]

Bisected locally — confirmed and narrowed down

Built two worktrees at the commits right before/after the dependency bump in #244 (93da5d6 vs d2fbea3 — confirmed git diff on the test files and backup_storage.rs is empty between them, so it's a pure dependency-version comparison), and ran each flaky test 30x in a loop against local Redis + LocalStack:

Test	Pre-#244 (aws-sdk-s3 1.129.0)	Post-#244 (aws-sdk-s3 1.142.0)
test_remove_sync_factor_etag_concurrency	1/30 failed (~3%)	8/30 failed (~27%)
test_concurrent_sync_backup_prevention	1/30 failed (~3%)	0/30 failed

test_remove_sync_factor_etag_concurrency shows a clear, reproducible ~9x jump in failure rate tied directly to the dependency bump — not just CI noise. test_concurrent_sync_backup_prevention didn't reproduce locally in either sample; its earlier CI failure may be more runner-timing-dependent (GitHub Actions vs local machine) and needs more samples to characterize.

Mechanism

Captured a full failing run of test_remove_sync_factor_etag_concurrency:

assertion `left == right` failed: Expected exactly one operation to succeed
  left: 2
 right: 1

Two of the three concurrent PutObject calls with If-Match succeeded — not a client retry masking a failure (that would just delay/repeat a request), but LocalStack's S3 emulation actually accepting two conditional writes that should be mutually exclusive under If-Match. This points at a concurrency/atomicity gap in LocalStack's conditional-write handling being newly exposed, not at application logic in backup_storage.rs.

Likely trigger

Between the two commits, hyper is unchanged (0.14.32 / 1.6.0 in both), but aws-smithy-http-client jumps 1.1.12 → 1.3.0 and aws-smithy-runtime 1.10.3 → 1.13.1 (h2 only patch-bumps, 0.4.13 → 0.4.16). The smithy HTTP client sits between the S3 client and hyper and owns connection pooling/reuse — a behavior change there (e.g. different connection reuse or request pacing against localhost:4566) is the most likely reason LocalStack's backend now sees the 3 concurrent requests in a pattern its conditional-write check doesn't handle atomically.

Suggested next step

Try pinning aws-smithy-http-client back to 1.1.x alone (leaving the actual h2 security fix and AWS SDK versions bumped) and re-run the same loop — if the failure rate drops back to baseline, that isolates the exact crate. Otherwise this may need a LocalStack version bump or a report upstream to LocalStack, since real S3 guarantees atomic If-Match regardless of client transport.

[orekav]

Root cause confirmed: known LocalStack bug on versioned buckets

Found the exact match upstream: localstack/localstack#13294 — "S3 Puts sometimes don't respect IfMatch headers". Verbatim the same symptom: concurrent PutObject requests conditioned on the same IfMatch ETag sometimes both succeed on a versioned bucket, when exactly one should get 412 PreconditionFailed.

Our test bucket is versioned (tests/aws-seed.sh: put-bucket-versioning ... Status=Enabled), and remove_sync_factor/the other conditional writes in backup_storage.rs all use .if_match(e_tag) — an exact match to the reported bug's trigger condition.

The fix (localstack/localstack#13299) explains the mechanism precisely: LocalStack was reusing the per-object-version write lock to guard the conditional check, but on a versioned bucket two concurrent writes each create a new, distinct version — so they don't share a lock, and both slip through the IfMatch check before either commits. The fix adds a dedicated per-object-key lock specifically for versioned-bucket conditional writes.

The nuance

That fix merged 2025-10-28 and shipped in LocalStack v4.10.0 (released two days later, 2025-10-30). Our docker-compose.yml pins localstack/localstack:4.14 — released 2026-02-26, well after the fix. So the officially-merged fix is already in the image we use, and we're still seeing the race — just far less often (baseline ~3% in my local bisection) than with the newer AWS SDK client (~27%). My guess: the upstream fix was validated against a 2-concurrent-writer reproduction (see the linked Go snippet in the issue); our tests race 3 concurrent writers, which may still find a gap in the new per-key lock under tighter request timing. Haven't confirmed that specific hypothesis — flagging it as the next thing to check if this gets picked up.

Bottom line

This is a LocalStack test-infrastructure correctness gap, not an application bug: backup_storage.rs's use of if_match is exactly what real S3 documents and guarantees atomically. Production is not at risk — only local/CI test reliability is. Given that, the practical fix here is almost certainly on the LocalStack side (report the 3-way-race gap upstream, or find/wait for a further fix) rather than in this codebase.