Add GHA coordinator for performance evaluation task scatter/gather#791
Merged
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0d93c0080f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Shaptic
reviewed
Jul 8, 2026
Shaptic
previously approved these changes
Jul 9, 2026
Shaptic
left a comment
Shaptic
left a comment
Contributor
There was a problem hiding this comment.
Awesome! Final result is way more concise and easier to read, glad we iterated on this 👍
Shaptic
dismissed
their stale review
Gonna neg my approval until things pass 😄 since it'll imply more changes, but LGTM otherwise! I can just review the follow-up commits.
tamirms
reviewed
Jul 9, 2026
tamirms
approved these changes
Jul 10, 2026
marwen-abid
added a commit
to marwen-abid/stellar-rpc
that referenced
this pull request
Jul 12, 2026
…ison (stellar#775) Add an advisory "Full History Benchmarks" workflow that runs the bench-ingest/bench-query commands on every pull request against a defined dataset and flags statistically significant slowdowns before they land. Per the team's benchmarking research, the PR lane pairs rather than pins: the PR base and head builds are benchmarked interleaved on the same runner, so machine noise hits both sides equally, and no stored wall-time baseline is ever trusted across machines. Wall-time trend tracking (pinned box, GCS series, change-point detection) stays a follow-up; this lane is deliberately secret-free and runner-agnostic. - `bench-ingest fixture` generates the defined dataset: a synthetic chunk pack whose event-ledger transactions derive their source accounts from (seed, seq) — same seed, byte-identical pack. The test helper writeSourcePack now delegates to it, so the whole existing bench test suite validates the generator. - `bench-compare` judges repeated CSV runs of two builds per gated metric on three axes: an exact permutation test on the per-run means at the Bonferroni-corrected level alpha/m — fork calibration showed that without the correction a ~20-metric gate false-positives on most runs, exactly MongoDB's published experience — a --min-effect floor (default 5%) below which significant deltas are still too small to matter, and a --max-noise class (default 10% within-side spread, extremes trimmed) that reports a benchmark as unstable instead of gating on it. Regressions exit non-zero naming the metric; a Markdown table lands in the job summary. - scripts/bench-ci/run.sh owns the workload (fixture generation, cold + hot ingest, cold + hot query sweeps, one --out per run, sides alternating per run, 6 runs per side so the corrected significance level stays reachable) and runs locally with no network; scripts/bench-ci/metrics.csv lists the gated metrics — CPU-bound rows only, fsync-bound hot commit and source-wait rows excluded by design. - The workflow reuses .github/actions/setup-go (cached zstd/RocksDB) and rust-cache, self-compares when there is no PR base (a noise-floor check for manual/nightly runs), skips comparison when the base build predates the bench commands, and uploads all CSVs as artifacts. Its name stays outside the required_status_check.yml gate prefixes, so it is advisory, not merge-blocking. The EC2 perf-eval coordinator on main (stellar#791) is complementary: it drives load tests on ephemeral cloud boxes with AWS credentials; this lane needs no secrets and gates library-level regressions per PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
marwen-abid
added a commit
to marwen-abid/stellar-rpc
that referenced
this pull request
Jul 12, 2026
…ison (stellar#775) Add an advisory "Full History Benchmarks" workflow that runs the bench-ingest/bench-query commands on every pull request against a defined dataset and flags statistically significant slowdowns before they land. Per the team's benchmarking research, the PR lane pairs rather than pins: the PR base and head builds are benchmarked interleaved on the same runner, so machine noise hits both sides equally, and no stored wall-time baseline is ever trusted across machines. Wall-time trend tracking (pinned box, GCS series, change-point detection) stays a follow-up; this lane is deliberately secret-free and runner-agnostic. - `bench-ingest fixture` generates the defined dataset: a synthetic chunk pack whose event-ledger transactions derive their source accounts from (seed, seq) — same seed, byte-identical pack. The test helper writeSourcePack now delegates to it, so the whole existing bench test suite validates the generator. - `bench-compare` judges repeated CSV runs of two builds per gated metric on three axes: an exact permutation test on the per-run means at the Bonferroni-corrected level alpha/m — fork calibration showed that without the correction a ~20-metric gate false-positives on most runs, exactly MongoDB's published experience — a --min-effect floor (default 5%) below which significant deltas are still too small to matter, and a --max-noise class (default 10% within-side spread, extremes trimmed) that reports a benchmark as unstable instead of gating on it. Regressions exit non-zero naming the metric; a Markdown table lands in the job summary. - scripts/bench-ci/run.sh owns the workload (fixture generation, cold + hot ingest, cold + hot query sweeps, one --out per run, sides alternating per run, 6 runs per side so the corrected significance level stays reachable) and runs locally with no network; scripts/bench-ci/metrics.csv lists the gated metrics — CPU-bound rows only, fsync-bound hot commit and source-wait rows excluded by design. - The workflow reuses .github/actions/setup-go (cached zstd/RocksDB) and rust-cache, self-compares when there is no PR base (a noise-floor check for manual/nightly runs), skips comparison when the base build predates the bench commands, and uploads all CSVs as artifacts. Its name stays outside the required_status_check.yml gate prefixes, so it is advisory, not merge-blocking. The EC2 perf-eval coordinator on main (stellar#791) is complementary: it drives load tests on ephemeral cloud boxes with AWS credentials; this lane needs no secrets and gates library-level regressions per PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
marwen-abid
added a commit
to marwen-abid/stellar-rpc
that referenced
this pull request
Jul 14, 2026
…ison (stellar#775) Add an advisory "Full History Benchmarks" workflow that runs the bench-ingest/bench-query commands on every pull request against a defined dataset and flags statistically significant slowdowns before they land. Per the team's benchmarking research, the PR lane pairs rather than pins: the PR base and head builds are benchmarked interleaved on the same runner, so machine noise hits both sides equally, and no stored wall-time baseline is ever trusted across machines. Wall-time trend tracking (pinned box, GCS series, change-point detection) stays a follow-up; this lane is deliberately secret-free and runner-agnostic. - `bench-ingest fixture` generates the defined dataset: a synthetic chunk pack whose event-ledger transactions derive their source accounts from (seed, seq) — same seed, byte-identical pack. The test helper writeSourcePack now delegates to it, so the whole existing bench test suite validates the generator. - `bench-compare` judges repeated CSV runs of two builds per gated metric on three axes: an exact permutation test on the per-run means at the Bonferroni-corrected level alpha/m — fork calibration showed that without the correction a ~20-metric gate false-positives on most runs, exactly MongoDB's published experience — a --min-effect floor (default 5%) below which significant deltas are still too small to matter, and a --max-noise class (default 10% within-side spread, extremes trimmed) that reports a benchmark as unstable instead of gating on it. Regressions exit non-zero naming the metric; a Markdown table lands in the job summary. - scripts/bench-ci/run.sh owns the workload (fixture generation, cold + hot ingest, cold + hot query sweeps, one --out per run, sides alternating per run, 6 runs per side so the corrected significance level stays reachable) and runs locally with no network; scripts/bench-ci/metrics.csv lists the gated metrics — CPU-bound rows only, fsync-bound hot commit and source-wait rows excluded by design. - The workflow reuses .github/actions/setup-go (cached zstd/RocksDB) and rust-cache, self-compares when there is no PR base (a noise-floor check for manual/nightly runs), skips comparison when the base build predates the bench commands, and uploads all CSVs as artifacts. Its name stays outside the required_status_check.yml gate prefixes, so it is advisory, not merge-blocking. The EC2 perf-eval coordinator on main (stellar#791) is complementary: it drives load tests on ephemeral cloud boxes with AWS credentials; this lane needs no secrets and gates library-level regressions per PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
marwen-abid
added a commit
to marwen-abid/stellar-rpc
that referenced
this pull request
Jul 15, 2026
…ison (stellar#775) Add an advisory "Full History Benchmarks" workflow that runs the bench-ingest/bench-query commands on every pull request against a defined dataset and flags statistically significant slowdowns before they land. Per the team's benchmarking research, the PR lane pairs rather than pins: the PR base and head builds are benchmarked interleaved on the same runner, so machine noise hits both sides equally, and no stored wall-time baseline is ever trusted across machines. Wall-time trend tracking (pinned box, GCS series, change-point detection) stays a follow-up; this lane is deliberately secret-free and runner-agnostic. - `bench-ingest fixture` generates the defined dataset: a synthetic chunk pack whose event-ledger transactions derive their source accounts from (seed, seq) — same seed, byte-identical pack. The test helper writeSourcePack now delegates to it, so the whole existing bench test suite validates the generator. - `bench-compare` judges repeated CSV runs of two builds per gated metric on three axes: an exact permutation test on the per-run means at the Bonferroni-corrected level alpha/m — fork calibration showed that without the correction a ~20-metric gate false-positives on most runs, exactly MongoDB's published experience — a --min-effect floor (default 5%) below which significant deltas are still too small to matter, and a --max-noise class (default 10% within-side spread, extremes trimmed) that reports a benchmark as unstable instead of gating on it. Regressions exit non-zero naming the metric; a Markdown table lands in the job summary. - scripts/bench-ci/run.sh owns the workload (fixture generation, cold + hot ingest, cold + hot query sweeps, one --out per run, sides alternating per run, 6 runs per side so the corrected significance level stays reachable) and runs locally with no network; scripts/bench-ci/metrics.csv lists the gated metrics — CPU-bound rows only, fsync-bound hot commit and source-wait rows excluded by design. - The workflow reuses .github/actions/setup-go (cached zstd/RocksDB) and rust-cache, self-compares when there is no PR base (a noise-floor check for manual/nightly runs), skips comparison when the base build predates the bench commands, and uploads all CSVs as artifacts. Its name stays outside the required_status_check.yml gate prefixes, so it is advisory, not merge-blocking. The EC2 perf-eval coordinator on main (stellar#791) is complementary: it drives load tests on ephemeral cloud boxes with AWS credentials; this lane needs no secrets and gates library-level regressions per PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a GitHub Actions coordinator (
load-test-coordinator.yml) that, on a push to arelease/**branch, launches the release performance-evaluation leg(s) as callable workflows and posts their consolidated results as a sticky comment on the release PR. The apply-load ingest load test (#741) is adapted into a callable leg that reports back (outputs + an S3 result object) instead of commenting itself:Decomposes a great deal of the existing infrastructure into more modular components for future coordinator-driven load test. Of the diff, +870/-721 of it is purely refactoring existing code into a new
harnesspackage, so review of it does not necessarily have to be as thorough as coordinator-heavy sections. Here is the breakdown of that:runner/orchestrate.goandrunner/instantiate.gointo an importableharnesspackage, mechanical changes onlygather.go,s3.go,exec.go,harness.go, andharness_test.gorun-load-test.shand placed intobootstrap-common.shmostly as helper functionsrun-load-test.shis now extremely slim and only serves to actually run the ingestion load testload-test.ymland put inec2-leg.yml, which handles the EC2 lifecycle with no test-specific hardcodingResults of the load tests are commented on a sticky comment on the
releasePR. If >1 push is made/multiple runs are requested, previous run's results are folded into dropdown menus like this:🧪 Performance Evaluation Test #N...
[recent results for run N]Performance Evaluation Test #N-1
`[results for run N-1]`Performance Evaluation Test #...
`[results for run ...]`Performance Evaluation Test # 1
`[results for run 1]`Or, you can just look in the comments below to see how this looks!
The high-level flow of the coordinator is below:
Why
This is so that the remaining perf-eval tests slot in cleanly: each new test becomes another callable leg the coordinator launches and folds into the same report, with no per-leg reporting or permissions plumbing to duplicate.
Known limitations
N/A. Remaining work for the epic this task is a part of mainly includes finishing off the other perf eval tasks (only the ingest load test is complete at this point).