ci: self-hosted runners for benchmarks #29955
Open
+94
−23
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes. |
metamaskbot
added
the
team-tiger
HowardBraham
removed
the
team-tiger
Builds ready [9c55124]
Page Load Metrics (1730 ± 64 ms)
Bundle size diffs
|
1 similar comment
Builds ready [9c55124]
Page Load Metrics (1730 ± 64 ms)
Bundle size diffs
|
HowardBraham
force-pushed
the
self-hosted-benchmarks
branch
from
9c55124 to
ce07be7
Compare
HowardBraham
commented
Jan 29, 2025
| with: | ||
| name: ${{ inputs.name }} | ||
| path: test-artifacts/chrome/benchmark/ | ||
| retention-days: 5 |
There was a problem hiding this comment.
The number of retention-days was chosen very arbitrarily, and maybe someone has an opinion
| @@ -182,4 +182,4 @@ jobs: | |||
| env: | |||
| GITHUB_TOKEN: ${{ secrets.LAVAMOAT_UPDATE_TOKEN }} | |||
| PR_NUMBER: ${{ github.event.issue.number }} | |||
| ACTION_RUN_URL: "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" | |||
| ACTION_RUN_URL: '${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}' | |||
There was a problem hiding this comment.
This is just Prettier running on a file
HowardBraham
force-pushed
the
self-hosted-benchmarks
branch
from
ce07be7 to
e5a4333
Compare
Builds ready [e5a4333]
Page Load Metrics (2151 ± 76 ms)
Bundle size diffs
|
HowardBraham
force-pushed
the
self-hosted-benchmarks
branch
from
e5a4333 to
a6f5acb
Compare
Builds ready [a6f5acb]
Page Load Metrics (1708 ± 72 ms)
Bundle size diffs
|
|
Could you split this into multiple PRs? A lot of these changes seem to be distinct from each other |
HowardBraham
force-pushed
the
self-hosted-benchmarks
branch
from
a6f5acb to
c9f1e1e
Compare
HowardBraham
changed the title
HowardBraham
force-pushed
the
self-hosted-benchmarks
branch
from
c9f1e1e to
6266945
Compare
Builds ready [6266945]
Page Load Metrics (1958 ± 284 ms)
Bundle size diffs
|
|
The migration to GitHub Actions is a necessary step, but before committing to self-hosted runners, we should first validate the stability of measurements across different environments.
The suggested approach is to conduct benchmark runs on GitHub-hosted runners first (ubuntu-latest) and compare them against CircleCI and self-hosted runner results. If fluctuations remain high across all environments, this method may not be suitable for a quality gate. However, the migration to GitHub Actions remains beneficial regardless. Next steps:
Would be great to see some comparison data in the PR before merging. |
|
As requested by @itsyoboieltr, here are benchmarks running in 4 different configurations:
Probably the most important stat here is uiStartup (btw our old system highlights load). Two additional things I see:
I'm going to run this again to see if it varies over time, and also with some other options.
|
|
Second run of this, numbers are pretty close to the first run
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Major changes
runs-on: gha-mm-scale-set-ubuntu-22.04-amd64-medcontainer: image: cimg/node:22.13-browsersXvfbPrerequisites to merging this PR
benchmarksbranch ofgithub-tools: https://github.com/MetaMask/github-tools/blob/benchmarks/.github/actions/setup-environment/action.ymlsetup-environment@benchmarksback tosetup-environment@mainThis is just Part 1 of a larger 4-part task to make the startup time quality gate, but I think it's the hardest part
Related issues
Progresses: MetaMask/MetaMask-planning#3679