feat(retention): enable prod modifications retention for real (POP-4134, DRY_RUN=false) - #2329
feat(retention): enable prod modifications retention for real (POP-4134, DRY_RUN=false)#2329carlomazzaferro wants to merge 2 commits into
Conversation
…34, DRY_RUN=false) Split from #2328 per review. Gated on the stage trim+bounce test: modifications have never had a nonzero delete anywhere, and genesis modification replay is watermark-forward (unbounded), so real-trim behavior gets validated on stage first. Inert until ~08-15 regardless (prod created_at backfill 07-16 + 30d window + newest-10k floor).
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 50778f2be9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| env: | ||
| RETENTION__ENABLED: "true" | ||
| RETENTION__DRY_RUN: "true" | ||
| RETENTION__DRY_RUN: "false" |
There was a problem hiding this comment.
When operators inspect these production values during rollout or an incident, the adjacent documentation still says DRY_RUN=true, “read-only COUNT,” and “no rows removed,” while this setting makes the reaper execute real DELETE batches. The same contradiction exists in common-values-iris-mpc-modifications-retention.yaml; update both comments to describe live deletion and the applicable rollback or kill-switch procedure so the destructive state is not misidentified as a preview.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
done — both comment blocks now describe live deletion, the newest-10k floor, and the kill switch (DRY_RUN=true → preview-only, suspend=true → no runs)
1 participant
Split from #2328 per review discussion (Ertugrul). Flips both prod modifications reapers (smpcv2 + ampc-hnsw) to real deletion.
Stage trim+bounce test: PASSED (2026-08-03, both fleets). One-off Jobs cloned from the reaper CronJobs with the window narrowed to 14d and the newest-rows floor lowered to 500 (kept above the 328-row startup-sync lookback):
Grouped modifications: 328→Modifications to update: [], to delete: [](no cross-party divergence) →Replaying last 328 modification results to SNS.Remaining consideration (non-blocking, separate PR if wanted): genesis modification replay is watermark-forward with no limit, so a graph checkpoint dormant >30d would silently skip trimmed rows — a loud-failure guard (
MIN(id) > last_indexed_modification_id→ abort) would convert that to a clean error.Inert until ~08-15 even once merged (prod created_at backfill 07-16 + 30d window + newest-10k floor).