Repository navigation
[benchmark] Report an improvement only past the alarm's band - #1974
Conversation
A regression had to be past the error band, while any confirmed change for the better was listed as an improvement. With dozens of comparisons a run, chance alone listed a couple of improvements on unchanged code.
Janpot
added
the
scope: code-infra
Deploy previewBundle sizeTotal Size Change: 0B(0.00%) - Total Gzip Change: 0B(0.00%) Show details for 79 more bundles@mui/internal-docs-infra/abstractCreateDemo parsed: 0B(0.00%) gzip: 0B(0.00%) PerformanceNo regressions in 8 benchmarks · ran 1m 01s · details Check out the code infra dashboard for more information about this PR. |
severity and improved are now the same comparison on either side of the alarm's band. The dashboard colors an improvement on an alarmed metric only past the band, as the pull request comment lists it, and the run summary carries improved.
A regression alarms only once its interval is wholly past the alarm's
errorband, but any confirmed change for the better was listed as an improvement in the PR comment. With dozens of comparisons a run, each a 99% interval, chance alone listed a couple of improvements on code that didn't change (mui/base-ui#5974: Dialog paint −2% to −23%, ScrollArea render −1% to −22%, with library code identical to master).An improvement now has to be past the band too:
MetricComparison.improved, judged against the same edge asseverityand returned byisImprovement. The dashboard colors an improvement on an alarmed metric green only past the band, so the run details agree with the comment, and the JSON summary carriesimproved. Acompare()'s variants have no band and keep any confirmed improvement green.