Skip to content

Optimize namespace searches and benchmark constrained burst traffic - #1750

Merged
rdimitrov merged 3 commits into
mainfrom
codex/read-throughput-benchmark
Oct 27, 2026
Merged

rdimitrov merged 3 commits into
mainfrom
codex/read-throughput-benchmark

Conversation

@rdimitrov

@rdimitrov rdimitrov commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Namespace-spanning substring searches can make the trigram index examine many candidates for very few results. Previously captured production plans for io.github.github/ examined about 133,000 candidates for one row under both custom and generic planning. A local burst test reproduced database CPU throttling, multi-second search latency, pool contention and delayed publishing.

This change adds an index on the reversed, lowercased namespace and an additional prefix predicate for searches spanning /. The existing database name-format constraint guarantees exactly one separator. The original complete ILIKE predicate remains, preserving case-insensitive literal substring matching, partial namespaces, filters and pagination. PostgreSQL's normal plan selection and pool/resource settings remain unchanged. A custom-plan-only experiment did not solve the overload and is not included.

The PR also introduces the previously prepared three-character minimum in both API versions and the UI. Compatibility: nonempty one/two-character or whitespace-only searches now return 422 before service/database work; empty searches still list normally.

Controlled evidence

The loopback-only runner uses 185,001 synthetic versions, two apps/caches, and a Linux PostgreSQL container with verified 1.5-CPU / 4-GiB limits. Three cyclic crawler waves vary API version, page size, query order and incremental watermark. Searches bypass caches, with each category balanced across both replicas. Every trial offers 30 full walks, 12,000 searches and 90 publishes; final order is A/B/B/A on a fresh database.

Measurement, range of two trials Baseline Candidate
Complete walks 18/30 30/30
Successful searches 7,514–9,075 / 12,000 12,000 / 12,000
Driver-dropped searches 2,914–4,483 0
Full-name search p95 2,111–2,932 ms 1.60–1.64 ms
Publish p95 452–591 ms 12.09–12.86 ms
Database CPU over observation 99.2–99.4 CPU-seconds 37.2–39.5 CPU-seconds

Both candidate trials completed every request without failures/drops. Each variant completed all 90 publishes. The candidate served more origin work while consuming less database CPU. Successful latency quantiles are not presented as equal-work comparisons when the baseline dropped work.

tools/readbench/CONSTRAINED_RESULTS.md and its JSON record final results, exploratory controls, the corrected generator routing, and limitations. Full response hashes match in 356 cases / 526 responses per variant across both API versions. Earlier statistics/cache/cyclic experiments remain documented separately.

Validation and rollout

Passed full database and API handler packages against isolated PostgreSQL; race-enabled search/benchmark tests, including category/replica balance; repository-wide golangci-lint with zero issues; go build ./...; make validate; Python and shell checks. The fresh runner exercised migration 016 and cleanup end to end. All owned services and the local VM were stopped.

Migration 016 adds the index (4.5 MiB on this fixture). For an existing busy database, prebuild it concurrently, verify the exact definition and validity/readiness, then analyze before deploying; deploy/README.md documents the procedure. No production/staging release, migration, configuration change or load test was performed.

The experiment establishes a local fix for the reproduced search-driven overload, not that every historical alert shares this cause or that production has a specific capacity multiplier. The synthetic mix, architecture/storage and application CPU differ. Late queues cleared in these tests, so a persistent post-burst stuck state remains outside this proof.

@rdimitrov rdimitrov changed the title Add local read-path benchmark with optimization evidence Require three-character searches and benchmark cyclic registry crawls Oct 27, 2026
@rdimitrov rdimitrov changed the title Require three-character searches and benchmark cyclic registry crawls Optimize namespace searches and benchmark constrained burst traffic Oct 27, 2026
@rdimitrov
rdimitrov marked this pull request as ready for review October 27, 2026 11:14
@rdimitrov
rdimitrov merged commit 3a891d2 into main Oct 27, 2026
8 checks passed
@rdimitrov
rdimitrov deleted the codex/read-throughput-benchmark branch October 27, 2026 11:15
rdimitrov added a commit that referenced this pull request Oct 27, 2026
Promote the search improvements from #1750 and #1752 as registry 1.8.6
and raise the PostgreSQL CPU quota from 1.5 to 3 on the existing
production nodes. Pool sizes, node count/type, memory and storage remain
unchanged.

Both migration 017 covering indexes were built with CREATE INDEX
CONCURRENTLY in production and verified valid, ready and structurally
correct; ANALYZE completed. The earlier migration 016 index is also
present and valid. A fresh pre-release database backup completed
successfully.

The combined application and CPU configuration passed two staging
repeats of 12,000 uncached searches, 30 full registry walks and 90
publishes with zero drops or request failures. Code alone at 1.5 CPUs
did not pass. See #1752 for full methodology and limitations.

Merge only after release artifacts/image are available and the full
production preview is verified. The singleton database will restart for
the CPU change. Monitor external probes, HTTP errors/latency, connection
queues, DB CPU/throttling and recovery throughout. If the new
application regresses, revert the image tag to 1.8.5 while retaining the
additional CPU budget and compatible indexes; reverting CPU would cause
another database restart.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant