feat(export): paginate the non-session collections the frame guard's hint calls out - #1143
feat(export): paginate the non-session collections the frame guard's hint calls out#1143dmazhukov wants to merge 2 commits into
Conversation
|
@dmazhukov is attempting to deploy a commit to the rohitg00's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe export flow now supports collection selection and pagination, reports collection totals, and enforces a configurable serialized-size limit. REST and MCP responses return HTTP 413 for oversized exports. Proxy errors include truncated response details. ChangesExport protection
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant api_export as api::export
participant mem_export as mem::export
participant collections as Collection reads
Client->>api_export: Request export with collection selection and pagination
api_export->>mem_export: Forward validated export parameters
mem_export->>collections: Read selected collections and totals
collections-->>mem_export: Paginated data and totals
mem_export-->>api_export: ExportData or ExportTooLarge
api_export-->>Client: HTTP 200 or HTTP 413
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/mcp/server.ts (1)
365-381: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winForward
memory_exportpagination args tomem::export.
memory_exportalways callssdk.trigger({ function_id: "mem::export", payload: {} }), somem::exportignores pagination and never returnscollectionPagination. The REST export handler already reads and forwardsmaxSessions,offset,collectionLimit, andcollectionOffset, while the MCP tool schema still declares an emptyproperties. Add argument validation and pass the same fields into the payload.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/mcp/server.ts` around lines 365 - 381, Update the memory_export case and its MCP tool schema to validate and accept maxSessions, offset, collectionLimit, and collectionOffset, then forward those values in the payload of sdk.trigger for mem::export. Preserve the existing isExportTooLarge handling and response formatting while ensuring pagination arguments reach the export function.
🧹 Nitpick comments (1)
src/functions/export-import.ts (1)
176-191: 🚀 Performance & Scalability | 🔵 TrivialCollection pagination still requires a full
kv.list()read per collection.
sliceCollectionpagesmemories,graphNodes, and the other collections only afterkv.list()has already fetched every row for that scope. For very large collections (the issue mentions 8K+ memories, 34K observations per session),collectionLimit/collectionOffsetshrink the response size but do not reduce the KV read cost behind each collection. This matches the PR's stated goal (avoid oversized WebSocket responses), so it is not a blocking concern, but if a collection ever grows large enough that the full-list read itself becomes slow, pagination here will not help.Consider tracking this as a follow-up if
StateKV.listgains offset/limit support later.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/functions/export-import.ts` around lines 176 - 191, Track this as a follow-up rather than changing the current export flow: collection pagination in sliceCollection occurs after each kv.list call and cannot reduce the full KV read. If StateKV.list later supports offset/limit parameters, update the collection reads in the export function to pass collectionOffset and collectionLimit directly while preserving the existing collection mappings and response slicing behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@src/mcp/server.ts`:
- Around line 365-381: Update the memory_export case and its MCP tool schema to
validate and accept maxSessions, offset, collectionLimit, and collectionOffset,
then forward those values in the payload of sdk.trigger for mem::export.
Preserve the existing isExportTooLarge handling and response formatting while
ensuring pagination arguments reach the export function.
---
Nitpick comments:
In `@src/functions/export-import.ts`:
- Around line 176-191: Track this as a follow-up rather than changing the
current export flow: collection pagination in sliceCollection occurs after each
kv.list call and cannot reduce the full KV read. If StateKV.list later supports
offset/limit parameters, update the collection reads in the export function to
pass collectionOffset and collectionLimit directly while preserving the existing
collection mappings and response slicing behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 562fdced-a35f-4cde-bb2d-b2080bd80994
📒 Files selected for processing (7)
src/functions/export-import.tssrc/mcp/rest-proxy.tssrc/mcp/server.tssrc/triggers/api.tssrc/types.tstest/export-import.test.tstest/mcp-standalone-proxy.test.ts
|
Heads-up on the red CI here: both failures reproduce on a clean checkout of
An endpoint landed in #1132 / #1136 without the generated docs moving with it. I checked that this branch adds no endpoints: #1144 refreshes the four files that carry the count. Once it lands I'll rebase this branch so the matrix reflects the actual change. |
|
Good catch on The empty
🤖 Addressed by Claude Code |
|
Correction to my note above: I've folded the same 4-line refresh into this branch (3966cb8, identical to #1144's commit) rather than waiting.
#1144 still stands on its own if you'd rather take the housekeeping separately — it is the same commit, so whichever merges first makes the other a no-op. 🤖 Addressed by Claude Code |
c473c5b to
6bd1868
Compare
The frame guard added in v0.9.29 refuses an oversized export cleanly, and its hint says the non-session collections "are not yet paginated". They are now: ?collectionLimit / ?collectionOffset window memories, summaries and the sixteen top-level collections, and ?collections= narrows the payload to the ones a caller actually reads. Without this the unpaginatable floor grows with the store until it alone crosses the cap, at which point no parameter combination returns anything — the dead end rohitg00#890 describes for mesh/export. totals cover every collection even when deselected, since that is what clients read for corpus size; hasMore looks only at the selected ones so a client that asked for six of eighteen can stop on the flag instead of comparing against totals by hand. Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
The tool declared no properties and always triggered mem::export with an empty payload, so the paging the REST endpoint offers was unreachable from MCP — which is where an agent actually calls export. collections is forwarded as the raw string, empty value included: mem::export reads an empty selection as "no collections", and that only stays distinguishable from an absent argument if this layer does not helpfully drop it. Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
6bd1868 to
f1613dc
Compare
dmazhukov
changed the title
|
Rebased onto v0.9.29 and reduced. The size guard this PR originally carried is gone — Title and description rewritten to match. 1614 tests pass, 🤖 Updated by Claude Code |
1 participant
Summary
Follow-up to the frame guard that shipped in v0.9.29. Its hint says the non-session
collections "are not yet paginated" — this makes them paginated.
Originally this PR carried its own size guard for #1142. v0.9.29 landed
frame-guard.tsfirst, so that half is dropped and what remains builds on
checkPayloadFrameSizerather thancompeting with it.
What this changes
?collectionLimit=/?collectionOffset=windowmemories,summariesand the sixteentop-level collections, which previously came in full no matter what
?maxSessions=said. On astore I reproduced against,
?maxSessions=1still returned 5.50 MB, of which ~4.9 MB was thatfloor. It grows with the store, and once it alone crosses the cap the export returns nothing at
any parameter combination — the dead end #890 describes for
mesh/export.?collections=narrows the payload to the collections a caller actually reads, so an agentafter a lesson is not also pulling
graphEdges.sessions,observationsandprofilesareoutside the vocabulary on purpose: they are windowed by
maxSessions/offset, and profiles arederived from the session page rather than listed. Unknown names are dropped rather than refused,
so a client can name a collection an older build lacks and still get the ones it has.
collectionPaginationreports per-collectiontotalsand a combinedhasMore.totalscover every collection even when deselected, since that is what clients read for corpus size;
hasMorelooks only at the selected ones, so a client that asked for six of eighteen can stopon the flag instead of comparing against
totalsby hand.The MCP tool declared no properties and always triggered
mem::exportwith an emptypayload, so none of this was reachable from MCP — which is where an agent actually calls
export. It now declares and forwards the same arguments.
The frame guard's hint is updated to name the new parameters instead of describing the gap.
Behaviour
An unparameterised export returns exactly what it returns today. A test pins the full-corpus
response field by field so that cannot drift.
What this doesn't fix
collectionLimitdoes not slice them; they followthe session window. A single session whose observations alone exceed the cap still cannot be
exported.
api::exportreturns 200 for a refusal. The guard'sOversizedPayloadtravels as asuccessful HTTP response, so a client has to inspect the body to notice. Left alone here since
it is the guard's contract, not this change's — happy to send a 413 mapping separately if you
want one.
mesh/export(mesh/export has no pagination — backlogs past the worker WS message limit fail forever with "Invocation stopped", making mesh sync unrecoverable after an outage #890) has the same shape and is untouched.Testing
test/export-import.test.ts: full-corpus shape pinned;collectionLimitbounds everycollection while the unparameterised call still returns everything; a three-page walk ends on
hasMore: false;collectionsselects, drops unknown names, and treats an empty list as"none".
test/mcp-export-tool.test.ts: arguments forwarded, unusable bounds dropped, empty allowlistkept distinct from an absent one.
npx vitest run --exclude test/integration.test.ts→ 1614 passed, 0 failed.npx tsc --noEmit→ 25 errors, byte-identical tomain.npm run skills:checkandnpm run buildclean.