Skip to content

fix(memory-sync): reliable per-source sync indicators and counters#3308

Draft
graycyrus wants to merge 2 commits into
tinyhumansai:mainfrom
graycyrus:fix/memory-sync-indicators
Draft

fix(memory-sync): reliable per-source sync indicators and counters#3308
graycyrus wants to merge 2 commits into
tinyhumansai:mainfrom
graycyrus:fix/memory-sync-indicators

Conversation

@graycyrus
Copy link
Copy Markdown
Contributor

Summary

Fixes the unreliable per-source "Syncing…" indicators and the live "N syncing / N synced" counters on the Intelligence memory sources panel (MemorySourcesRegistry). Previously only one source could show as syncing at a time, intermediate stages never lit up the per-source bar, and progress got stuck at a placeholder.

Closes #3295.

Root causes addressed

  1. Only one source could show "syncing"MemorySourcesRegistry.tsx tracked a single syncingId: string | null. Now a syncingIds: Set<string> so concurrent syncs each render their own progress (immutable Set updates).
  2. Intermediate stages carried the wrong idMemorySyncStageBridge (src/openhuman/memory/sync.rs) emitted stored/queued with no connection_id and ingesting with connection_id = document_id, so the frontend (matching by connection_id === source.id) never lit up intermediate stages. Added a new optional source_id: Option<String> field to MemorySyncStageChanged (distinct from connection_id, which still carries the ingest-pipeline document_id). The bridge extracts the originating memory-source id from the mem_src:<source_id>:<item_id> chunk encoding (first-colon split, since item ids may be URLs). Frontend matches on source_id ?? connection_id (backward compatible).
  3. Brittle progress parsingparseSyncProgress now tolerates non-N/M detail strings and falls back to per-stage default percentages so the bar always advances.
  4. Event/poll races — the 5s poll reconciles missed events to a done state; documented the minor remount gap.

Out of scope (intentional, tracked separately)

  • Root cause Feat/landing revamp #3 (two accounts of the same Composio toolkit collapsing into one counter row) — requires reworking Composio chunk tagging with data-migration implications; deferred to a follow-up.
  • Root cause sync branches #5 full recovery — no new memory_sources_active_syncs RPC; rely on the existing 5s poll to rehydrate after a remount.

Event payload change

MemorySyncStageChanged gains source_id: Option<String> (the memory-source row id; None for channel-level syncs). connection_id semantics are unchanged. Serialized into the memory-sync-stage socket payload.

Tests

  • Rust (src/openhuman/memory/sync.rs): extract_mem_src_id unit tests (incl. URL item ids with colons) + MemorySyncStageBridge tests asserting source_id is populated for mem_src: syncs and None otherwise, with connection_id left unchanged. 10 lib tests pass.
  • Vitest (app/src/components/intelligence/__tests__/MemorySourcesRegistry.test.tsx): concurrent syncs render both rows, completed/failed clears only the right row, source_id preferred over connection_id, and tolerant parseSyncProgress cases. 11 tests pass.

Checks

pnpm typecheck, pnpm lint (0 errors), pnpm format:check, pnpm build, cargo check, the new Vitest suite, and the memory::sync Rust lib tests all pass. The full pnpm test:rust e2e suite and pnpm dev:app were not run locally due to a disk-space limit in the dev environment (ld: errno=28); the library tests covering this change pass.

Notes for reviewers

  • connection_id must keep carrying document_id for ingest-pipeline identity — this PR adds source_id rather than repurposing connection_id. Please sanity-check no downstream consumer depends on connection_id being the source id.

graycyrus added 2 commits June 3, 2026 19:24
Per-source "Syncing…" indicators and the live "N syncing / N synced"
counters on the Intelligence memory sources panel were unreliable: only
one source could show as syncing at a time, intermediate stages never lit
up the per-source bar, and progress got stuck at a placeholder.

- Track syncing sources as a Set<string> (syncingIds) instead of a single
  syncingId so concurrent syncs each render their own progress.
- Add a new optional source_id field to MemorySyncStageChanged (distinct
  from connection_id, which still carries the ingest-pipeline document_id).
  MemorySyncStageBridge extracts the originating memory-source id from the
  mem_src:<source_id>:<item_id> chunk encoding (first-colon split, since
  item ids may be URLs) so stored/queued/ingesting stages reach the right
  row. Frontend matches on source_id ?? connection_id (backward compatible).
- Make parseSyncProgress tolerant and add per-stage fallback percentages so
  the bar always advances; the 5s poll reconciles missed events to done.
- Tests: Rust bridge/source_id-mapping unit tests + Vitest concurrent-sync
  and tolerant-parsing coverage.

Composio same-toolkit counter collapse and remount state recovery are
intentionally out of scope (tracked separately).

Closes tinyhumansai#3295
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Jun 3, 2026

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 808b08ac-511e-4373-9193-002c13e12b78

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@CodeGhost21 CodeGhost21 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review withdrawn — PR is in draft. Will re-post once it's ready for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Per-source memory sync indicators and live syncing/synced counters are unreliable

2 participants