Skip to content

Coalesce cold-eye prefetch and bound foreground fetch latency - #2367

Merged
philsippl merged 1 commit into
codex/cpu-linear-scan-cold-eye-cachefrom
codex/cpu-linear-scan-cold-prefetch
Aug 28, 2026
Merged

Coalesce cold-eye prefetch and bound foreground fetch latency#2367
philsippl merged 1 commit into
codex/cpu-linear-scan-cold-eye-cachefrom
codex/cpu-linear-scan-cold-prefetch

Conversation

@philsippl

@philsippl philsippl commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes the cold-eye tail latency exposed in prod on 2026-08-26 (batch 201, 21:50:51Z): a broad-matching uniqueness request discovered ~3.7k second-eye candidates and took 22.8 s of compute, of which ~15.5 s were the two discovered-candidate stages — almost entirely un-metered foreground Postgres reads (two server-side slow-statement alerts bracket exactly those gaps in the log timeline).

Root-cause chain and fixes:

  1. Prefetch starvation. Discovery enqueues one tiny Fetch command per full-scan chunk; thousands of commands overflowed the 64-deep queue (queue_skips) and the surviving ones became 1–2-row round trips, so almost nothing was actually prefetched. The prefetch worker now drains and merges everything queued into one reservation pass and one batched read per database round trip, and the queue holds 4096 commands. Barrier ordering is preserved: a barrier drained mid-merge completes after the read that covers every fetch enqueued before it.
  2. Foreground fetch latency. Candidates that missed prefetch stalled the scan on a single id = ANY(...) statement over ~144 MiB of TOASTed rows. Large cold fetches — foreground and prefetch alike — are now split into 512-row sub-batches issued concurrently (parallelism 4) via a shared helper.
  3. Observability gap. The foreground read had no metric, which is why dashboards showed "DB time is short" while Postgres logged multi-second statements. Added linear_scan_cold_foreground_db_duration / _batch_size / _records_total, plus linear_scan_cold_prefetch_coalesced_records on the worker.

Additionally the cold-eye LFU capacity (default and stage/prod manifests) goes from 4096 to 12288 records (~460 MiB at ~38.4 KiB/record — well inside the deployment's memory headroom) so recurring broad-matcher candidates stay resident across requests instead of being refetched.

Validation

  • cargo clippy --all-targets --features db_dependent -- -D warnings
  • iris-mpc-cpu lib suite: 396 passed.
  • db-backed cold-eye tests incl. new cold_eye_coalesces_prefetch_commands_and_batches_large_fetches (1200 rows; 700 prefetched via ~234 small commands; single fetch across both sub-batch paths; byte-exact round-trip).
  • e2e_linear_scan_test, e2e_linear_scan_uniqueness_test (release).

Expected effect on the incident profile: the ~3.7k-candidate reads coalesce into a handful of concurrent batched statements that largely complete during the 5.6 s resident scan, and any residual foreground wait becomes visible on its own dashboard series.

Stack

Based on #2351 (cold-eye cache layer); #2348 should merge this branch after review.

@philsippl
philsippl requested a review from a team as a code owner August 27, 2026 13:42
A broad-matching query (prod batch 201: ~3.7k discovered candidates,
22.8s compute) exposed three compounding cold-eye weaknesses:

- Discovery enqueues one tiny prefetch command per full-scan chunk; with
  thousands of hits the 64-deep queue dropped most commands and the rest
  became single-row round trips, so almost nothing was prefetched. The
  worker now drains and merges everything queued into one reservation
  pass and one batched read per database round trip, and the queue holds
  4096 commands. Barriers drained mid-merge complete after that read,
  preserving their ordering contract.
- The candidates then stalled the scan on one giant foreground
  `id = ANY(...)` statement (~38 KiB TOASTed rows; Postgres logged
  multi-second slow statements). Large fetches, foreground and prefetch
  alike, are now split into 512-row sub-batches issued concurrently.
- That foreground read had no metric, which is why dashboards showed
  "DB time is short" while Postgres disagreed. It now records
  linear_scan_cold_foreground_db_duration/_batch_size/_records_total,
  and the worker records the coalesced batch size.

Also raise the cold-eye LFU capacity to 12288 records (~460 MiB, three
times the previous size) so recurring broad-matcher candidates stay
resident between requests instead of being refetched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAbAnQyfsSrWH6UGcRM1FF
@philsippl
philsippl force-pushed the codex/cpu-linear-scan-cold-prefetch branch from fb6349e to 4027552 Compare August 27, 2026 13:53
@philsippl
philsippl merged commit 7bdaac0 into codex/cpu-linear-scan-cold-eye-cache Aug 28, 2026
18 of 19 checks passed
@philsippl
philsippl deleted the codex/cpu-linear-scan-cold-prefetch branch August 28, 2026 07:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants