Fix/episodic capture timestamp and dedup guard#52
Open
joerybka wants to merge 2 commits into
Open
Conversation
_find_transcript_ts's content-hash fallback could lock onto an earlier queue-operation line carrying duplicate text ahead of the real user/assistant turn, silently re-deriving the same already-collapsed timestamp. Filter to user/assistant lines, mirroring the guard capture.py already applies. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rive Reproduces the collapsed-group case where an internal queue-operation transcript line carries the same text as the real user turn and appears first in the file, verifying the matcher now resolves to the real turn's timestamp instead of the collapsed one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_find_transcript_ts's content-hash fallback inmigrate --rederive-timestampsso it no longer matches an internal transcript event (e.g.queue-operation) ahead of the realuser/assistantturn when both carry identical text.Motivation
iai-mcp doctor's collapsed-timestamp check can report groups thatmigrate --rederive-timestampsnever clears, even though the migration reportsrecords_updated> 0 each run. Root cause: the migration's content-hash matcher scans transcript lines top-to-bottom and returns on the first hash match, with no filter on line type. When an internal event (e.g.queue-operation) carries the same duplicated text as the real turn and appears earlier in the transcript file, the matcher locks onto it and returns its timestamp instead — which is the same collapsed value the record already has. The write is a no-op, but the function still counts it as updated.The original capture path (
capture_deferred_capturesincapture.py) already filters transcript lines totype/role in {"user", "assistant"}before considering them; the migration's independent matcher never had that guard.Changes
src/iai_mcp/migrate/_timestamps.py: filter candidate transcript lines touser/assistanttype/role before content-hash comparison, mirroring the existing guard incapture.py.tests/test_migrate.py: new testtest_migrate_rederive_skips_internal_event_duplicate— builds a collapsed group of 3 duplicate episodic records with nosource_uuid(forcing content-hash fallback) against a transcript containing an earlierqueue-operationline and a later realuserline with identical text, and asserts the migration resolves to the real turn's timestamp, not the collapsed one.Test plan
pytest tests/test_migrate.py— 25 passedruff check/ruff format --checkon changed files show only pre-existing findings already present onmain, unrelated to this diffNo bench re-run included — this is a data-repair migration correctness fix, not a retrieval/capture/consolidation performance change.
Note
This doesn't fully resolve
doctor's collapsed-timestamp warning for groups that are true duplicate records (same content+provenance) rather than distinct turns sharing a bad timestamp — those still needmigrate --dedupe-episodicto collapse the duplicates themselves. This PR only fixes the matcher so rederive stops silently no-op'ing on genuinely distinct turns that happen to share text with an internal event.