Fix three cursor-expiry repair gaps (eviction history, overwrite-only gaps, restore memory)

ARCHITECTURE.md

@@ -234,8 +234,8 @@ The one exception: an update carrying the **unknown** version (an unparseable AC
 
 **Cursor-expiry repair.** <a id="cursor-expiry-repair"></a> The cursor expires at resume (`CursorExpired` from the `*_from` watch) or mid-watch (the live floor guard ends an All-scope watch with `CursorExpired` when retention overruns it). Both take one path. Correct means **every write minus every real delete** — not "whatever NATS still lists"; the two agree only when retention never evicts current values. The repair (`ExpiryRepair`, `src/repair.rs`) is chosen accordingly:
 
-- **Key-listing diff** (`Relist`; `Auto` on a bucket that keeps current values). The watch task lists the bucket's live keys, hands them to the main loop, and waits for an ack; the main loop diffs the fold's in-scope keys against the listing and runs a synthetic `KvUpdate::Delete` (unknown version — never advances the cursor) through `parse`/`apply`/store for each key that vanished; only then does the full-scope re-list (`watch_all` / `watch_prefix` / `watch_prefixes`) start. That ack ordering is the invariant: a synthetic delete always precedes the re-list put for the same key, so delete-then-recreate during the gap converges. **Refused** — the watch fails — when the watcher reports retention that evicts current values (`KvWatcher::retention`, read live from the stream config): there it deletes valid keys. `tests/eviction.rs` reproduced exactly that against a live server before the fix.
-- **Artifact restore** (`Restore`; `Auto` on a bucket that evicts current values or can't say). The watch task peeks the newest artifact's manifest and checks it before downloading: it must cover the watch scope (recorded in the manifest by the exporting `watch_applied`), and pass `protocol::restore_allowed` — strictly ahead of the local fold, cursor still inside retention. It then downloads and imports it into scratch as a temporary fold of the consumer's backend, re-checks what actually arrived, lists the bucket's live keys, and hands both to the main loop. The main loop re-checks "ahead" against its own applied cursor (a mid-watch expiry delivered past the resume cursor), diffs the in-scope fold against the artifact (keys only), and folds the difference through `parse`/`apply`/store in `max`-sized chunks, then advances the cursor to the artifact's in one final commit and replies. The diff takes an artifact entry only when it is newer than the local one, and deletes a key the artifact lacks only when the bucket does not list it live. An artifact is complete *together with the log after its cursor* — a NATS cursor C means every retained message ≤ C is applied, not "the truth at C": a key whose latest write is after C may be missing or older in an artifact exported mid catch-up — and in both skipped cases the key's latest write is after C, so the resume delivers it. The consumer therefore never sees a key move backward or a live key deleted, not even transiently. The watch resumes from that cursor (which can itself expire later and go round again, each time only for a strictly newer artifact). A failed check fails the watch and logs the reason at `error`: a stale artifact has no safe recovery.
+- **Key-listing diff** (`Relist`; `Auto` on a bucket that keeps current values). The watch task lists the bucket's live keys, hands them to the main loop, and waits for an ack; the main loop diffs the fold's in-scope keys against the listing and runs a synthetic `KvUpdate::Delete` (unknown version — never advances the cursor) through `parse`/`apply`/store for each key that vanished; only then does the full-scope re-list (`watch_all` / `watch_prefix` / `watch_prefixes`) start. That ack ordering is the invariant: a synthetic delete always precedes the re-list put for the same key, so delete-then-recreate during the gap converges. **Refused** — the watch fails — when the watcher reports retention that evicts current values (`KvWatcher::retention`, read live from the stream config), or the fold remembers seeing it do so (below): there it deletes valid keys. `tests/eviction.rs` reproduced exactly that against a live server before the fix.
+- **Artifact restore** (`Restore`; `Auto` on a bucket that evicts current values or can't say). The watch task peeks the newest artifact's manifest and checks it before downloading: it must cover the watch scope (recorded in the manifest by the exporting `watch_applied`), and pass `protocol::restore_allowed` — strictly ahead of the local fold, cursor still inside retention. It then downloads and imports it into scratch as a temporary fold of the consumer's backend, re-checks what actually arrived, and hands it to the main loop. The main loop re-checks "ahead" against its own applied cursor (a mid-watch expiry delivered past the resume cursor) and diffs the in-scope fold against the artifact in one key-ordered pass: keys the artifact's entry should replace are spilled to a scratch file next to the imported artifact (`RestoredFold::scratch_in`), and local keys the artifact lacks are collected as delete candidates. Only if there are candidates does it stream the bucket's live keys past them (`KvReader::for_each_key`) and then re-read retention: the listing vouches for writes after the artifact's cursor only while the log still holds all of them, so if retention passed that cursor meanwhile the restore is refused before anything is folded (`tests/model_repair.rs`, `NoListingRecheck`). It then folds the difference through `parse`/`apply`/store in `max`-sized chunks, read back from the spill, then advances the cursor to the artifact's in one final commit and replies. Memory is the delete candidates — the keys deleted between the two cursors — never the fold, the artifact, or the bucket's listing, so a billion-key fold (or a fresh node seeding from an artifact) restores in bounded memory. The diff takes an artifact entry only when it is newer than the local one, and deletes a key the artifact lacks only when the bucket does not list it live. An artifact is complete *together with the log after its cursor* — a NATS cursor C means every retained message ≤ C is applied, not "the truth at C": a key whose latest write is after C may be missing or older in an artifact exported mid catch-up — and in both skipped cases the key's latest write is after C, so the resume delivers it. The consumer therefore never sees a key move backward or a live key deleted, not even transiently. The watch resumes from that cursor (which can itself expire later and go round again, each time only for a strictly newer artifact). A failed check fails the watch and logs the reason at `error`: a stale artifact has no safe recovery.
 - **None** (`None`, or no store): the re-list alone, with a warning that deleted keys may persist.
 
 Whichever runs, the main loop first **drains** the watch channel: a mid-watch expiry leaves delivered updates buffered there, and the repair must fold them first — one applied after the repair would resurrect pre-repair state and could move the cursor backward. The watch task is parked on the ack/reply, so the drain takes exactly that backlog.
@@ -244,6 +244,10 @@ A cursor-less start is repaired before it watches in two cases. A fold that hold
 
 The listing counts as the truth — "not listed" means "deleted" — when retention never evicts current values, or when it can but has never evicted anything yet (first retained revision ≤ 1). Otherwise the key-listing diff is refused.
 
+**Retention can be edited, so the fold remembers what it saw.** A key that aged out while `max_age` or `discard: old` was in force is missing from the listing forever after, even once the operator turns eviction off or recreates the bucket — and the fold, the only place it still exists, would have it deleted by a diff against that listing. So the first time the watch task reads retention and finds it has evicted (eviction on and the log past its first revision), it remembers: the main loop commits a memo on the fold's cursor (`WatchCursor::to_bytes`: a 12- or 20-byte encoding, longer than any bare version; a cursor without the memo keeps the old encoding byte for byte), every later cursor carries it, and so does every artifact exported from the fold, so a node restored from one inherits it. A fold with the memo never treats the listing as the truth again (`listing_is_truth`'s `seen_evicting`; `Snapshot::stale_keys` reads it too). Retention is read at every watch start, at every expiry, and every 60 s during a live watch while a key-listing repair could still be planned (`Relist`/`Auto`) and the memo is unset. The axiom: a fold reads retention while eviction is in force. Eviction that starts, drops a key, and is turned off again entirely between two reads (within a minute, or while the node is down) is not seen. `tests/model_repair.rs` (`eviction_turned_off_repair_steps_are_correct`, mutation `NoEvictionMemo`) and `tests/eviction.rs::eviction_turned_off_never_trusts_the_listing_again` (live) pin it.
+
+**A supersession-only gap resumes** (`NatsKvWatcher::resume_point`). The resume check sees only the stream's first retained revision, so when every message after a node's cursor was superseded while it was down — a small hot bucket on a short restart — the cursor looks expired though nothing aged out, and on an evicting bucket the restore needs an artifact newer than the node. A cursor with the memo also keeps the server time its message was written. When the window check fails, the resume reads the stream's config and the server's clock (`ts` in the raw stream-info reply, NATS 2.10+); if age is the bucket's only way to evict a current value (`max_age`, no per-message TTLs, no `discard: old` limit) and the cursor's message is at most half of `max_age` old (`protocol::gap_holds_no_eviction`: half is headroom for clock steps and leader changes), every later message is younger still, none has aged out, and each missing one was superseded by a later write the resume delivers — so it resumes from the first retained revision instead. Assumes the config didn't switch from a `discard: old` limit to age-only while the node was down.
+
 Before diffing, both repairs flush until the store holds every delivered update: a batch re-queued by a transient store failure is invisible to a diff of the store, and a re-queued put of a key deleted during the gap would otherwise commit after the diff and resurrect it.
 
 **The key-listing repair's one axiom.** Between taking the listing and the full re-list delivering what the bucket holds, retention must not evict a delete marker (axiom 6 of `tests/model.rs`, narrowed to this window). On a bucket that keeps current values, the only such eviction is an admin purge of markers, and NATS's `purge_deletes` keeps markers younger than 30 minutes by default. `tests/model_repair.rs` drops the axiom and the checker reaches the purge-inside-the-window trace; the artifact restore resumes on a floor-guarded watch and holds without it.
@@ -358,9 +362,9 @@ artifact_revision > local_revision && resume_window_ok(artifact_revision, first_
 
 Machine-checked as: _"bootstrap never silently diverges from every write minus every real delete"_ on an evicting bucket (`tests/model.rs`), and _"a repair never moves the fold backward"_ (`tests/model_live_watch.rs`, where removing the guard is a mutation the checker catches).
 
-**`listing_is_truth(evicts_current_values, first_revision) → bool`** and **`plan_repair(mode, listing_truth) → RepairPlan`**
+**`listing_is_truth(evicts_current_values, first_revision, seen_evicting) → bool`**, **`listing_truth(retention, seen_evicting)`** and **`plan_repair(mode, listing_truth) → RepairPlan`**
 
-Whether "not listed" means "deleted": yes when retention never evicts current values, or when it can but has evicted nothing yet (first retained revision ≤ 1). The planner turns that (or `None`, a backend that can't say) and the armed `ExpiryRepair` into the repair: `Relist` where the listing isn't the truth is `RefuseRelist`; `Auto` re-lists only on `Some(true)` and restores otherwise.
+Whether "not listed" means "deleted": yes when retention never evicts current values, or when it can but has evicted nothing yet (first retained revision ≤ 1) — and never once the fold has seen the answer be no (`seen_evicting`, the cursor's memo). The planner turns that (or `None`, a backend that can't say) and the armed `ExpiryRepair` into the repair: `Relist` where the listing isn't the truth is `RefuseRelist`; `Auto` re-lists only on `Some(true)` and restores otherwise.
 
 **`cursorless_start_needs_repair(mode, fold_has_data, listing_truth) → bool`**
 
@@ -729,7 +733,8 @@ Production code and an exhaustive model checker running the same logic closes th
 | `CursorExpired`, bucket keeps current values | `watch_applied` lists live keys, diffs fold, applies synthetic deletes, then falls back to state-sync re-list | Automatic; raw callers use `stale_keys()` manually |
 | `CursorExpired`, bucket evicts current values | `watch_applied` restores the in-scope fold from the newest artifact and resumes from its cursor | Automatic with `ExpiryRepair::Restore`/`Auto` |
 | Fold holds data but no cursor (torn first checkpoint) | Repaired like an expiry at revision 0 before watching | Automatic with a repair armed |
-| Every message after a node's cursor SUPERSEDED (every key rewritten while it was down), evicting bucket | The conservative resume check sees the cursor as expired though nothing was evicted; the restore then needs an artifact ahead of the node, and without one the watch fail-stops (safe, unavailable) — pinned by `tests/eviction.rs::supersession_only_restart_fail_stops_without_a_newer_artifact` | Restarts once the next export round publishes; export more often on small hot buckets. Telling supersession from eviction needs the cursor's message timestamp against `max_age` (not stored today) |
+| Every message after a node's cursor SUPERSEDED (every key rewritten while it was down), evicting bucket | On an age-only (`max_age`) bucket within half of `max_age` of the cursor's message: resumes from the first retained revision (`tests/eviction.rs::supersession_only_restart_resumes_on_a_max_age_bucket`). Otherwise (a `discard: old` limit or per-message TTLs, an older cursor, or one without the memo) the cursor reads as expired, the restore needs an artifact ahead of the node, and without one the watch fail-stops (safe, unavailable) | Restarts once the next export round publishes; export more often on small hot buckets |
+| Eviction turned off (or the bucket recreated) after it dropped keys | The fold remembers it saw eviction (cursor memo) and never trusts the key listing again: `Auto` restores, `Relist` is refused | Automatic with `ExpiryRepair::Auto`; an artifact source is needed for good |
 | A write ages out before ANY node folds it | Every node that needed it fail-stops (no fold anywhere holds it; `tests/model_fleet.rs` proves this is the only way the fleet fail-stops) | Data loss by retention; widen retention or export more often |
 | Key-listing resync on a bucket that evicts current values | Refused: `KvError::WatchError` ("evicts current values"); fold untouched | Wire `ExpiryRepair::Restore`/`Auto` with an artifact source |
 | Restore: no artifact ahead, stale artifact, or one that doesn't cover the scope | `KvError::WatchError`, reason logged at `error`; fold untouched | Fix the export pipeline (run exports well inside `max_age` / `discard: old` turnover); restart |

Cargo.toml

@@ -34,6 +34,9 @@ serde_json = "1"
 tar = { version = "0.4.46", optional = true }
 tempfile = "3"
 thiserror = "2"
+# Parses the server time in a raw JetStream stream-info reply (`ts`), for the
+# supersession-only resume check. Already in the tree via async-nats.
+time = { version = "0.3", features = ["parsing"] }
 tokio = { version = "1", features = ["macros", "rt", "sync", "time"] }
 tracing = "0.1"
 url = "2"

README.md

@@ -374,6 +374,12 @@ A restore only accepts an artifact that is ahead of the local fold, covers the w
 
 A node with no cursor on a bucket that has already evicted current values seeds from the artifact too, since a re-list alone would miss whatever aged out.
 
+Turning eviction off later doesn't make the key list trustworthy again: keys that aged out while it was on stay missing from it. A node that ever saw its bucket evict remembers that in its saved cursor (and in every artifact it exports), so `Auto` keeps restoring and `Relist` keeps refusing.
+
+On a bucket that evicts only by `max_age`, a node restarted within half of `max_age` resumes even when every message after its cursor was overwritten while it was down: none of those can have aged out, so there is nothing to restore.
+
+A restore holds only the keys it will delete in memory; the keys it rewrites are spilled to the scratch directory, so it works at any fold size.
+
 `Some(reader)` from older code still compiles and means `Relist`. `None` (or no `store`) falls back to the re-list alone and warns that deleted keys may persist.
 
 ## NATS mapping