Repository navigation
Optimize namespace searches and benchmark constrained burst traffic - #1750
Merged
Merged
Conversation
rdimitrov
marked this pull request as ready for review
October 27, 2026 11:14
rdimitrov
added a commit
that referenced
this pull request
Oct 27, 2026
Promote the search improvements from #1750 and #1752 as registry 1.8.6 and raise the PostgreSQL CPU quota from 1.5 to 3 on the existing production nodes. Pool sizes, node count/type, memory and storage remain unchanged. Both migration 017 covering indexes were built with CREATE INDEX CONCURRENTLY in production and verified valid, ready and structurally correct; ANALYZE completed. The earlier migration 016 index is also present and valid. A fresh pre-release database backup completed successfully. The combined application and CPU configuration passed two staging repeats of 12,000 uncached searches, 30 full registry walks and 90 publishes with zero drops or request failures. Code alone at 1.5 CPUs did not pass. See #1752 for full methodology and limitations. Merge only after release artifacts/image are available and the full production preview is verified. The singleton database will restart for the CPU change. Monitor external probes, HTTP errors/latency, connection queues, DB CPU/throttling and recovery throughout. If the new application regresses, revert the image tag to 1.8.5 while retaining the additional CPU budget and compatible indexes; reverting CPU would cause another database restart.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Namespace-spanning substring searches can make the trigram index examine many candidates for very few results. Previously captured production plans for
io.github.github/examined about 133,000 candidates for one row under both custom and generic planning. A local burst test reproduced database CPU throttling, multi-second search latency, pool contention and delayed publishing.This change adds an index on the reversed, lowercased namespace and an additional prefix predicate for searches spanning
/. The existing database name-format constraint guarantees exactly one separator. The original completeILIKEpredicate remains, preserving case-insensitive literal substring matching, partial namespaces, filters and pagination. PostgreSQL's normal plan selection and pool/resource settings remain unchanged. A custom-plan-only experiment did not solve the overload and is not included.The PR also introduces the previously prepared three-character minimum in both API versions and the UI. Compatibility: nonempty one/two-character or whitespace-only searches now return
422before service/database work; empty searches still list normally.Controlled evidence
The loopback-only runner uses 185,001 synthetic versions, two apps/caches, and a Linux PostgreSQL container with verified 1.5-CPU / 4-GiB limits. Three cyclic crawler waves vary API version, page size, query order and incremental watermark. Searches bypass caches, with each category balanced across both replicas. Every trial offers 30 full walks, 12,000 searches and 90 publishes; final order is A/B/B/A on a fresh database.
Both candidate trials completed every request without failures/drops. Each variant completed all 90 publishes. The candidate served more origin work while consuming less database CPU. Successful latency quantiles are not presented as equal-work comparisons when the baseline dropped work.
tools/readbench/CONSTRAINED_RESULTS.mdand its JSON record final results, exploratory controls, the corrected generator routing, and limitations. Full response hashes match in 356 cases / 526 responses per variant across both API versions. Earlier statistics/cache/cyclic experiments remain documented separately.Validation and rollout
Passed full database and API handler packages against isolated PostgreSQL; race-enabled search/benchmark tests, including category/replica balance; repository-wide golangci-lint with zero issues;
go build ./...;make validate; Python and shell checks. The fresh runner exercised migration 016 and cleanup end to end. All owned services and the local VM were stopped.Migration 016 adds the index (4.5 MiB on this fixture). For an existing busy database, prebuild it concurrently, verify the exact definition and validity/readiness, then analyze before deploying;
deploy/README.mddocuments the procedure. No production/staging release, migration, configuration change or load test was performed.The experiment establishes a local fix for the reproduced search-driven overload, not that every historical alert shares this cause or that production has a specific capacity multiplier. The synthetic mix, architecture/storage and application CPU differ. Late queues cleared in these tests, so a persistent post-burst stuck state remains outside this proof.