Skip to content

Fix/episodic capture timestamp and dedup guard#52

Open
joerybka wants to merge 2 commits into
CodeAbra:mainfrom
joerybka:fix/episodic-capture-timestamp-and-dedup-guard
Open

Fix/episodic capture timestamp and dedup guard#52
joerybka wants to merge 2 commits into
CodeAbra:mainfrom
joerybka:fix/episodic-capture-timestamp-and-dedup-guard

Conversation

@joerybka

@joerybka joerybka commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Fix _find_transcript_ts's content-hash fallback in migrate --rederive-timestamps so it no longer matches an internal transcript event (e.g. queue-operation) ahead of the real user/assistant turn when both carry identical text.
  • Add a regression test reproducing a real collapsed-timestamp group where this caused the migration to silently re-derive the same already-collapsed timestamp on every run.

Motivation

iai-mcp doctor's collapsed-timestamp check can report groups that migrate --rederive-timestamps never clears, even though the migration reportsrecords_updated > 0 each run. Root cause: the migration's content-hash matcher scans transcript lines top-to-bottom and returns on the first hash match, with no filter on line type. When an internal event (e.g. queue-operation) carries the same duplicated text as the real turn and appears earlier in the transcript file, the matcher locks onto it and returns its timestamp instead — which is the same collapsed value the record already has. The write is a no-op, but the function still counts it as updated.

The original capture path (capture_deferred_captures in capture.py) already filters transcript lines to type/role in {"user", "assistant"} before considering them; the migration's independent matcher never had that guard.

Changes

  • src/iai_mcp/migrate/_timestamps.py: filter candidate transcript lines to user/assistant type/role before content-hash comparison, mirroring the existing guard in capture.py.
  • tests/test_migrate.py: new test test_migrate_rederive_skips_internal_event_duplicate — builds a collapsed group of 3 duplicate episodic records with no source_uuid (forcing content-hash fallback) against a transcript containing an earlier queue-operation line and a later real user line with identical text, and asserts the migration resolves to the real turn's timestamp, not the collapsed one.

Test plan

  • pytest tests/test_migrate.py — 25 passed
  • Confirmed the new test fails against the pre-fix matcher and passes with the fix
  • ruff check / ruff format --check on changed files show only pre-existing findings already present on main, unrelated to this diff

No bench re-run included — this is a data-repair migration correctness fix, not a retrieval/capture/consolidation performance change.

Note

This doesn't fully resolve doctor's collapsed-timestamp warning for groups that are true duplicate records (same content+provenance) rather than distinct turns sharing a bad timestamp — those still need migrate --dedupe-episodic to collapse the duplicates themselves. This PR only fixes the matcher so rederive stops silently no-op'ing on genuinely distinct turns that happen to share text with an internal event.

joerybka and others added 2 commits July 5, 2026 14:10
_find_transcript_ts's content-hash fallback could lock onto an earlier
queue-operation line carrying duplicate text ahead of the real user/assistant
turn, silently re-deriving the same already-collapsed timestamp. Filter to
user/assistant lines, mirroring the guard capture.py already applies.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rive

Reproduces the collapsed-group case where an internal queue-operation
transcript line carries the same text as the real user turn and appears
first in the file, verifying the matcher now resolves to the real turn's
timestamp instead of the collapsed one.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant