fix(replication): per-key follower cache invalidation after apply — a connected lobe no longer serves deleted memories (#869)

[likesjx]

Fixes #869. Taking you up on "it's yours" — implemented to the scope in your triage comment, traps included.

The fix, in your frame

The applier gains an invalidation callback and feeds every user key an applied entry touched through it, after Commit. The OpBatch path decodes the applied repr with pebble.ReadBatch; single-key ops pass entry.Key. The storage half maps keys back to cache entries and drops exactly those.

On the after-Commit ordering (not in the triage, but load-bearing): invalidating before the commit would let a racing read re-load the old value from Pebble and re-cache it just before the commit lands — resurrecting exactly the staleness this removes. After Commit, a racing read caches the new state, which is fine.

The four traps

1. No storage dependency — Applier.SetInvalidate(func(key []byte)), wired in cmd/muninn/server.go right next to storeCfg.RepLogAppend; the comment there names them as mirror images. It's a setter rather than a NewApplier parameter only because of construction order: server.go builds the applier with the coordinator before the store exists.
2. Per-key, never a full clear — only the touched keys drop. Foreign prefixes (assoc, index, counters, Hebbian/decay traffic) pass through unexamined; assocCache/revAssocCache have their own 2s TTL story and stay out of scope (storage: three assoc-mutation sites evict neither association cache (COG-31 blind spot, up to 2s) #818).
3. Both caches — 0x01 engram keys drop the vault-scoped L1 entry and the metaCache entry; 0x02 meta keys drop the metaCache entry. metaCache is keyed by ID alone (ULIDs are globally unique), so the vault prefix only gates the L1 delete.
4. The regression test fails without the fix — verified by actually reverting it, not by assertion:

invalidation_869_test.go:119: follower state after replicated soft-delete = active,
want StateSoftDeleted — the warmed cache entry survived the applied mutation (#869)

TestReplicatedSoftDelete_InvalidatesFollowerCache follows your spec: primary store ships real RepLogAppend batches, a follower applier applies them, a read warms the follower's cache, the replicated soft-delete is applied, and recall must observe it with no restart. It lives in internal/replication as an external test package — the seam crosses the replication ← cognitive ← storage import cycle, which is also exactly why production joins the two halves in server.go rather than by import.

A second test pins the invalidator's edges: nil/truncated/oversized/wrong-prefix keys neither panic nor evict unrelated entries — the applier feeds it every key of every batch, so it must be indifferent to shapes it doesn't own.

One behavior note

A decode failure in the applied repr is logged and abandoned rather than returned: the batch itself committed successfully, and failing the Apply would make the follower re-request an entry it already holds. The log line says which seq, and that caches may serve stale entries for that batch.

Verification

• go test ./internal/replication/ ./internal/storage/ — green (41.9s / 57.6s)
• go vet, gofmt — clean
• RED check as above

Next: deploying this build to one of my macOS observers and running the #869 repro live against it (write → recall-on-lobe to warm → forget on Cortex → recall-on-lobe again, no restart). I'll report the result on the issue either way.

[Copilot]

Pull request overview

This PR fixes replication correctness on follower/observer nodes by invalidating in-memory storage caches on a per-key basis after replication entries are applied to Pebble, ensuring replicated mutations (notably soft-deletes) are immediately visible without requiring a process restart.

Changes:

• Add a per-key invalidation callback to the replication applier and invoke it after commits for both batch and single-key replication ops.
• Implement PebbleStore.InvalidateReplicatedKey to evict only the affected engram/L1 and metadata cache entries based on replicated user keys.
• Add regression and edge-case tests verifying replicated soft-deletes penetrate a warmed follower cache and that foreign/invalid keys are safely ignored.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
internal/storage/replica_invalidation.go	Adds storage-side per-key cache eviction for replicated engram/meta keys.
internal/storage/replica_invalidation_test.go	Tests that invalidation ignores foreign/malformed keys and doesn’t evict unrelated entries.
internal/replication/invalidation_869_test.go	End-to-end regression test for replicated soft-delete invalidating a warmed follower cache.
internal/replication/applier.go	Adds invalidation callback plumbing and calls invalidation after apply/commit.
cmd/muninn/server.go	Wires the applier invalidation callback to the store during server construction.

Suppressed comments (1)

internal/replication/applier.go:154

• The OpBatch invalidation runs only after the marker batch commits. If reprBatch.Commit() succeeds but markerBatch.Commit() fails, the follower’s on-disk state has changed but caches will not be invalidated (and Apply returns an error, potentially stopping replication while caches keep serving stale data).

Move invalidateBatchKeys to immediately after reprBatch.Commit() so cache correctness tracks the actual data commit, not the bookkeeping marker.

		// #869: the repr is committed — tell the storage layer which keys
		// changed so its caches drop any entries the batch just mutated.
		a.invalidateBatchKeys(entry.Value, entry.Seq)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.