[History Server] Tune CPU, memory, and event logging for cold loads - #5095
Open
Future-Outlier wants to merge 8 commits into
Open
[History Server] Tune CPU, memory, and event logging for cold loads#5095Future-Outlier wants to merge 8 commits into
Future-Outlier wants to merge 8 commits into
Conversation
Loading a dead session is CPU-bound, but the sample manifest sets only `limits.cpu: "500m"`, so Kubernetes pins requests there too and the load saturates the quota for its entire duration. Measured on kind across 13 runs, the container sat at 0.42-0.50 cores every time. Raising the limit to 4 (requests stay at 500m, so scheduling cost is unchanged): | tasks in session | 500m | 4 cores | |---|---|---| | 50,000 | 97.9s | 30.5s | | 100,000 | 907.3s | 62.7s | This also removes what looked like superlinear degradation past 50k tasks: at 500m the per-task cost went from 1.96ms to 9.07ms between 50k and 100k, while at 4 cores it is flat (0.61ms and 0.63ms). The load uses ~1.2 cores on average and peaks at ~2.2, because Go's GC runs concurrently and needs cores of its own. The second change drops the per-event log line in storeEvent to Debug. It runs once per event, so a 100k-task session writes ~436,000 INFO lines per cold load, and the binary never calls logrus.SetLevel, so there is no way to turn it off. Worth ~13% of load time at 50k tasks (97.9s -> 85.3s).
Member
Author
|
The two red checks are pre-existing on master, not from this PR.
This PR only touches |
win5923
approved these changes
Aug 7, 2026
Signed-off-by: Future-Outlier <eric901201@gmail.com>
andrewsykim
reviewed
Aug 10, 2026
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Future-Outlier
changed the title
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Cold-loading large sessions is CPU- and memory-intensive. A
500mCPU limit throttles loads, while oneInfolog per event adds avoidable work and noise.CPU: keep the
500mrequest and leave the limit to operators. One 50k-task session, five randomized reads per setting: median 84s → 34s.xychart-beta title "Cold load — before vs after" x-axis ["Before: 500m limit", "After: no limit"] y-axis "Cold-load time (seconds)" 0 --> 100 bar [84, 34]Per-event logging: move
InfotoDebug. Separate 50k-task runs measured 97.9s → 85.3s (~13%); these runs were not paired, so this is supporting evidence rather than an isolated causal estimate.xychart-beta title "Per-event logging — observed cold-load time" x-axis ["Before: Info", "After: Debug"] y-axis "Cold-load time (seconds)" 0 --> 110 bar [97.9, 85.3]Memory: use a
2Girequest, an8Giencoded-cache soft bound, and a12Gilimit in all standalone examples. This leaves room for roughly one 100k-task decode; larger concurrent reads still need workload-specific tuning.Raw measurements and reproduction scripts.
Checks