Skip to content

perf(graph): reuse bounded handles across interleaved pages - #4725

Draft
ScriptedAlchemy wants to merge 2 commits into
masterfrom
fix/4532-bounded-graph-segment-handles
Draft

ScriptedAlchemy wants to merge 2 commits into
masterfrom
fix/4532-bounded-graph-segment-handles

Conversation

@ScriptedAlchemy

Copy link
Copy Markdown
Owner

Root cause

Refs #4532. The current sealed graph reader already pins the last segment, avoiding a new open() for consecutive pages from the same file. However, it drops that handle when a different segment is encountered. An interleaved A, B, A, B page sequence therefore reopens a segment on every switch, which is expensive on the reported endpoint-scanned macOS machines.

Focused change

  • Extend that existing, pass-local pinned handle into an 8-entry least-recently-used handle window. Revisited segments retain the same opened identity; the oldest handle closes on eviction, and an evicted segment must pass the original open-time checks again.
  • Continue using the original exact range validation, opened-file size checks, full graph-page digest verification, and the per-read cancellation/deadline probe. The change does not retain handles between passes or introduce a process-wide cache.
  • No admission, store format, durable ownership, auth, deadline, or resource budget changes.

Regression coverage

  • A real two-segment filesystem fixture invokes the production with_verified_segments path, alternates six page reads, verifies returned content/digests, and requires 2 opens rather than 6.
  • A separate filesystem regression cycles through more than 8 distinct segment files, verifies the handle bound, refuses an out-of-bounds read on a cached segment, and requires a fresh validated open after eviction.
  • Existing sealed graph tests still cover mutation/unlink, corrupted payload, cancellation/deadline precedence and the ordinary contiguous range-reader path.

Validation

Reviewed the exact one-file GitHub diff against master at 65ddc060 (133 added, 11 removed) and checked source structure, whitespace and final newline. Rust compilation, rustfmt and the two new tests have not run: this execution environment has no supplied checkout or Bazel/Rust toolchain. Hosted CI must run the isolated branch through bazel test //crates/tracedecay-store-runtime:unit_test --test_output=errors and the graph publication journey before merge. No measured wall-time or CPU improvement is claimed.

Remaining acceptance

This is a focused remaining reopen case, not a substitute for the issue's required before/after graph-publication timing and file-open measurements on the reported 6.6k-file macOS endpoint-security setup. The existing contiguous page reader already opens one handle per segment. Keep #4532 open until the actual production journey is verified. This PR is independent of the graph rewarm PR #4709; PR repair/CI/merges remain owned by the separate PR automation.

Keep up to eight authenticated segment handles alive during a graph
publication pass. Alternating segment page reads no longer incur a
fresh file open, while evicted entries are re-opened and revalidated.

Exercise alternating reads, handle bounds, eviction, and refused
out-of-range reads. Refs #4532.
@changeset-bot

changeset-bot Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: e06fc29

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Apply the exact rustfmt line break required by the repository-gate
failure on the new LRU lookup; preserve its behavior and regressions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant