fix(memory-sync): reliable per-source sync indicators and counters#3308
Draft
graycyrus wants to merge 2 commits into
Draft
fix(memory-sync): reliable per-source sync indicators and counters#3308graycyrus wants to merge 2 commits into
graycyrus wants to merge 2 commits into
Conversation
Per-source "Syncing…" indicators and the live "N syncing / N synced" counters on the Intelligence memory sources panel were unreliable: only one source could show as syncing at a time, intermediate stages never lit up the per-source bar, and progress got stuck at a placeholder. - Track syncing sources as a Set<string> (syncingIds) instead of a single syncingId so concurrent syncs each render their own progress. - Add a new optional source_id field to MemorySyncStageChanged (distinct from connection_id, which still carries the ingest-pipeline document_id). MemorySyncStageBridge extracts the originating memory-source id from the mem_src:<source_id>:<item_id> chunk encoding (first-colon split, since item ids may be URLs) so stored/queued/ingesting stages reach the right row. Frontend matches on source_id ?? connection_id (backward compatible). - Make parseSyncProgress tolerant and add per-stage fallback percentages so the bar always advances; the 5s poll reconciles missed events to done. - Tests: Rust bridge/source_id-mapping unit tests + Vitest concurrent-sync and tolerant-parsing coverage. Composio same-toolkit counter collapse and remount state recovery are intentionally out of scope (tracked separately). Closes tinyhumansai#3295
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
CodeGhost21
reviewed
Jun 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the unreliable per-source "Syncing…" indicators and the live "N syncing / N synced" counters on the Intelligence memory sources panel (
MemorySourcesRegistry). Previously only one source could show as syncing at a time, intermediate stages never lit up the per-source bar, and progress got stuck at a placeholder.Closes #3295.
Root causes addressed
MemorySourcesRegistry.tsxtracked a singlesyncingId: string | null. Now asyncingIds: Set<string>so concurrent syncs each render their own progress (immutable Set updates).MemorySyncStageBridge(src/openhuman/memory/sync.rs) emittedstored/queuedwith noconnection_idandingestingwithconnection_id = document_id, so the frontend (matching byconnection_id === source.id) never lit up intermediate stages. Added a new optionalsource_id: Option<String>field toMemorySyncStageChanged(distinct fromconnection_id, which still carries the ingest-pipelinedocument_id). The bridge extracts the originating memory-source id from themem_src:<source_id>:<item_id>chunk encoding (first-colon split, since item ids may be URLs). Frontend matches onsource_id ?? connection_id(backward compatible).parseSyncProgressnow tolerates non-N/Mdetail strings and falls back to per-stage default percentages so the bar always advances.Out of scope (intentional, tracked separately)
memory_sources_active_syncsRPC; rely on the existing 5s poll to rehydrate after a remount.Event payload change
MemorySyncStageChangedgainssource_id: Option<String>(the memory-source row id;Nonefor channel-level syncs).connection_idsemantics are unchanged. Serialized into thememory-sync-stagesocket payload.Tests
src/openhuman/memory/sync.rs):extract_mem_src_idunit tests (incl. URL item ids with colons) +MemorySyncStageBridgetests assertingsource_idis populated formem_src:syncs andNoneotherwise, withconnection_idleft unchanged. 10 lib tests pass.app/src/components/intelligence/__tests__/MemorySourcesRegistry.test.tsx): concurrent syncs render both rows,completed/failedclears only the right row,source_idpreferred overconnection_id, and tolerantparseSyncProgresscases. 11 tests pass.Checks
pnpm typecheck,pnpm lint(0 errors),pnpm format:check,pnpm build,cargo check, the new Vitest suite, and thememory::syncRust lib tests all pass. The fullpnpm test:ruste2e suite andpnpm dev:appwere not run locally due to a disk-space limit in the dev environment (ld: errno=28); the library tests covering this change pass.Notes for reviewers
connection_idmust keep carryingdocument_idfor ingest-pipeline identity — this PR addssource_idrather than repurposingconnection_id. Please sanity-check no downstream consumer depends onconnection_idbeing the source id.