fix(checkpoint): repair stale session cursors after cleanup

[Arreboi06]

Related to #157. Complements #177.

Summary

This PR keeps #177 as the canonical aggregate checkpoint-counter recalibration fix, and narrows this branch to the additional behavior that #177 does not cover: repairing stale per-session cursors after cleanup/startup.

The extra failure mode is not just a wrong displayed count. After JSONL/DB cleanup, a session cursor can point past the newest retained record for that same session. When the scheduler restores from that stale checkpoint, incremental L1/L2 reads can skip retained or newly written data.

Relationship to #177

#177 should remain the reference fix for the aggregate counter drift in #157:

• recalibrate checkpoint counters against actual store counts;
• adjust memories_since_last_persona safely;
• run recalibration after cleanup and before scheduler startup.

This branch now follows that same safety model. In particular, memories_since_last_persona is adjusted only by the number of removed L1 memories. It is not recomputed from last_persona_at, because last_persona_at is a total_processed cursor, not an L1 memory counter.

Intentional Behavior Beyond #177

This PR adds session-aware cursor recovery:

• clamp runner_states[session].last_l1_cursor to that session's newest retained L0 recordedAt watermark;
• clamp runner_states[session].last_captured_timestamp only when retained L0 evidence proves it is stale;
• clamp pipeline_states[session].last_extraction_updated_time to that session's newest retained L1 updatedAt / updated_time watermark;
• reset a stale session cursor only when the scan can safely trust that the session has no retained records;
• restore the scheduler from the recalculated checkpoint so repaired pipeline cursors are used immediately.

The key point is that cursor correction is per-session, not global. A global newest timestamp can hide a stale cursor in one session when another session has newer retained data.

Implementation Notes

• recalibrate() handles explicit count reconciliation when callers already know actual counts.
• recalculate() scans retained JSONL shards, accepts optional VectorStore counts, updates aggregate counters, and repairs per-session cursors from retained watermarks.
• LocalMemoryCleaner.runOnce() invokes recalculation after JSONL/DB cleanup.
• Scheduler startup recalculates before restoring pipeline states.
• Correction stats are returned to callers but are not persisted into recall_checkpoint.json.

Verification

Latest verified commit: cd026e6

npm test -- src/utils/checkpoint.test.ts
Test Files  1 passed (1)
Tests       50 passed (50)

npm test
Test Files  5 passed (5)
Tests       117 passed (117)

npm run build
Build complete
scripts/migrate-sqlite-to-tcvdb: passed
scripts/export-tencent-vdb: passed
scripts/read-local-memory: passed

git diff --check
passed

High-Value Scenarios Covered

• manual JSONL pruning recalibrates aggregate counters;
• automatic cleaner recalibrates after deleting expired shards;
• memories_since_last_persona is reduced by removed L1 memories, not by mixing L1 counts with last_persona_at;
• stale runner cursors are clamped by the same session's retained L0 watermark;
• stale pipeline cursors are clamped by the same session's retained L1 watermark;
• sessions without retained data are reset only when the scan evidence is trustworthy;
• missing directories and malformed JSONL lines are handled without failing recalculation.

[Yuntong8888]

Hi @Arreboi06 👋,
Thank you for submitting this PR and participating in Tencent Rhino-bird Open-source Training Program!
We have successfully received your submission. The program is currently in full swing, and we will complete the Code Review for you as soon as possible. Please keep an eye on the status notifications for this PR so you can follow up promptly once the review feedback is provided.
Thanks again for your contribution and open-source spirit! 🚀

[YOMXXX]

Triage note: #177 has been locally verified and approved as the canonical #157 fix. It recalibrates checkpoint counters against actual store counts, adjusts memories_since_last_persona safely, and wires recalibration after cleanup plus scheduler startup. This PR appears to cover the same checkpoint-counter drift scope, so I recommend treating it as superseded unless it intentionally adds behavior beyond #177.

[Arreboi06]

Addressed the superseded-scope concern in the latest push (cd026e6).

I agree that #177 is the canonical fix for the aggregate checkpoint-counter drift in #157. This branch now keeps that same safety model: memories_since_last_persona is adjusted only by the number of removed L1 memories, not recomputed from last_persona_at, because last_persona_at tracks total_processed and is not an L1-memory counter.

The intentional behavior beyond #177 is narrower: session-aware cursor repair. Cleanup/startup can leave per-session cursors ahead of retained data:

• runner_states[session].last_l1_cursor can point past the newest retained L0 record, causing L1 incremental reads to skip retained/new records.
• pipeline_states[session].last_extraction_updated_time can point past the newest retained L1 record, causing L2 incremental reads to skip records.

recalculate() now keeps the aggregate counter recalibration semantics aligned with #177, and additionally clamps only stale per-session cursors using retained shard watermarks. It is wired after cleanup and before scheduler restore so the scheduler starts from repaired pipeline state.

Local verification after this update:

• npm test -- src/utils/checkpoint.test.ts - 50 passed

• npm test - 117 passed

• npm run build - passed

So I would frame this PR as: #177 fixes the canonical aggregate counter drift; this PR is only worth keeping if the maintainers want the extra session-cursor recovery behavior as a follow-up on top of that fix.