perf: bound review context hydration

[stainlu]

Summary

• Hydrate review context from bounded head/tail windows when GitHub count metadata already proves the collection is larger than the prompt limit.
• Preserve the existing prompt compaction shape while avoiding middle pages that cannot reach Codex.
• Render total, hydrated, and truncated counts in the review snapshot so the system can audit what was skipped.
• Add unit coverage for bounded windows and page-boundary tail planning.

Why

OpenClaw has thousands of open PRs and issues. For large PRs, the prompt keeps only a first/last window of files, commits, and review comments, but ClawSweeper was still paginating every middle page before compaction. That spends API calls, JSON parsing, memory, and wall-clock time on data that cannot be sent to the model.

Example: a 3,000-file PR keeps the same 80 prompt file entries, but avoids hydrating the middle file pages.

Behavior

• If GitHub total counts are missing, this falls back to the existing full pagination path.
• If a collection fits within the prompt limit, this keeps the full collection.
• If a collection exceeds the prompt limit, this hydrates the first window plus enough tail pages to preserve the same first/last prompt semantics.
• Related mention scanning follows the hydrated comment/review-comment window; body and timeline scanning remain intact, and the review snapshot now makes truncation visible.

Validation

• pnpm --ignore-workspace run build:all
• pnpm --ignore-workspace run test:unit
• pnpm --ignore-workspace run format:check
• git diff --check

Notes:

• pnpm --ignore-workspace run check currently reaches lint and fails on existing repo-wide oxlint findings unrelated to this PR.
• pnpm --ignore-workspace run test:repair also hits two existing notifier tests with Node 22 cancelledByParent; changed files are not in repair.

[ds4psb-ai]

Independent code review

Posted on behalf of @ds4psb-ai (read-only contributor). Maintainer judgment owns merge.

Summary verdict

LGTM after addressing one P1 (orchestration test gap) — compactMappedWindow, githubContextWindowPlan, and the counts struct extension all check out at every boundary I traced; the new behavior is correctly opt-out (null total → falls back to full pagination) and the compactMappedSlice aliasing is semantically identical to the pre-PR implementation.

Findings

P0 / blocker

• (none)

P1 / should fix before merge

• src/clawsweeper.ts:2143-2178 — ghPagedContextWindow is the load-bearing orchestration in this PR, and it has no direct unit test. The 4 added tests cover compactMappedWindow (2) and githubContextWindowPlan (2), but the tail-merge logic where tailFirstPageNumber === 1 reuses the already-fetched firstPage (line 167) is the trickiest part — a future refactor could easily double-fetch page 1 or, worse, drop the head when keepStart > 0 and tailFirstPageNumber === 1. Suggest one test that mocks ghPage (e.g. via dependency injection or by exporting ghPagedContextWindow for test) and asserts (a) page 1 is fetched exactly once when total=101 limit=80, and (b) the returned items are the correct [1..40, 62..101] shape. Without this, the boundary the PR specifically fixes (page 1 + page 2 share the tail) regresses silently.

P2 / nice to have / followup

• src/clawsweeper.ts:1717-1769 (collectRelatedMentions) — the PR body discloses that "Related mention scanning follows the hydrated comment/review-comment window". Confirmed: options.comments and options.pullReviewComments now contain only the head/tail slice, so #NNN references buried in the middle of a 3000-comment thread are silently dropped from the related-items sidebar of the review prompt. The trade-off is correct (the model couldn't see them anyway), but worth a short // Note: middle-window mentions are intentionally dropped — see ghPagedContextWindow comment at src/clawsweeper.ts:1748 so a future contributor doesn't "fix" this by re-paginating.
• src/clawsweeper.ts:1518-1556 (compactMappedWindow) — the items.length > boundedLimit && boundedTotal === items.length branch on line 1543 is a defensive case for callers who pass already-full hydration with total === items.length. It works, but the branch is non-obvious; one inline comment explaining "keep full retained set when caller already paginated everything" would help. Not blocking.
• src/clawsweeper.ts:2148-2150 (ghPagedContextWindow, total === null fallback) — the items.length self-report total = items.length collapses the distinction between "GitHub didn't tell us a total" and "there are exactly N items", which means commentsTruncated: false will be reported when the count is genuinely missing. This is fine for the prompt, but the rendered GitHub Snapshot block (src/clawsweeper.ts:6079-6082) won't ever say "truncated" in this branch — possibly worth surfacing the null-total case in the snapshot for diagnostic purposes.

Test coverage

4 added tests covering compactMappedWindow (head/tail-marker behavior at hydrated and unhydrated extremes) and githubContextWindowPlan (page-boundary tail and large-page-skip case). Cover the algorithmic core, but leave gaps:

• ghPagedContextWindow orchestration (P1 above).
• compactMappedWindow with keepStart === 0 && keepEnd > 0 — exercised by my static trace (it correctly returns [..., omitted, lastTail] with no head), but unreachable from production callers (smallest caller limit is 24). Worth a unit-test pin for future safety.
• The total === null fallback path of ghPagedContextWindow — locks in that when GitHub doesn't carry a total, full pagination still works.
• A pullRecord.review_comments === undefined case (older GitHub API responses or partial cache) — githubCount returns null, falls back to full pagination. Worth one test.

Risks I considered and dismissed

• compactMappedSlice ↔ compactMappedWindow aliasing: ran a full equivalence trace across 7 inputs (including limit=0, items=[], items.length === limit); the new compactMappedSlice = compactMappedWindow(items, items.length, limit, mapper) is a strict-output preserving rename. No callers of the old function need migration.
• tailFirstPageNumber math at boundaries: traced total ∈ {80, 81, 100, 101, 200, 3000} and limit ∈ {1, 2, 3, 80, 100}. Every case matches the PR's two unit-test assertions for tail offset and last page. The Math.floor((total - keepEnd) / perPage) + 1 formula is correct because GitHub pages are 1-indexed and the items-since-page-1-start is zero-indexed.
• counts struct extension breaking downstream: grep -n 'context\.counts' src/clawsweeper.ts shows only GitHub Snapshot markdown rendering uses these fields, and the PR updates that block (lines 6279-6306 in the diff) to consume the new optional fields via contextCountText with safe undefined handling. The relatedItemsContext passthrough block (lines 3401-3416) explicitly checks each new field for undefined before assigning. Nothing else reads these fields.
• PR#53 making existing notifier or repair-test failures worse: git diff --stat origin/main..HEAD confirms only src/clawsweeper.ts and test/clawsweeper.test.ts change; test/repair/ is untouched. The Node-22 cancelledByParent finding is preexisting and orthogonal.
• Mechanical preservation of full pagination for safety evidence: the four ghPagedContextWindow callers (limits 24, 40, 80, 40) cover comments, files, commits, review comments. They all correctly fall back to full ghPaged when GitHub doesn't expose a count. Reviews (ghPaged directly at src/clawsweeper.ts:3320) and timelines (ghPaged at src/clawsweeper.ts:3300) stay on full pagination — same pattern as the PR#49 split.

The math is right and the change is well-scoped. The orchestration test gap is the only thing I'd want closed before merge.

[stainlu]

Addressed the orchestration test gap.

What changed:

• Exported ghPagedContextWindow behind injectable page/paged fetchers for direct unit coverage.
• Added a regression for the total=101, limit=80 boundary where the tail starts on page 1 and must reuse the already-fetched first page instead of double-fetching or dropping head entries.
• Added coverage for the missing-total fallback path to full pagination.

Local proof:

• pnpm run build
• pnpm run lint:src
• pnpm run format:check
• git diff --check
• node --test test/clawsweeper.test.ts

Local note: this machine is on Node 22 while the repo declares Node >=24, so CI remains the authoritative full-suite check.