Repository navigation
[benchmark] v2 - #1898
Merged
Merged
[benchmark] v2#1898
Conversation
Extracts the tachometer harness infrastructure from mui/base-ui-mosaic#428 (itself a port of mui/base-ui-charts#634) so a repo adopts it by adding case folders instead of copying ~1,100 lines of script. The two copies were already near-identical: packWorkspace.mts differed only in two doc comments. Ships three things: - `packWorkspace`, a reusable export that packs a ref's public workspace packages to tarballs with `pnpm pack`, cached by commit SHA. It lists packages through `getWorkspacePackages({ cwd })` in-process rather than shelling out to the CLI, so a ref's checkout no longer needs code-infra installed just to be enumerated. - A vite plugin. The harness keeps its own config and runs plain vite; the plugin contributes `root`, `appType: 'mpa'`, build entries derived from the `tachometer.json` files, and a dev-server index listing every case. It never resolves refs, so `vite build` needs no git history. - `code-infra tacho run`, which packs and builds one variant per ref, runs each case as its own tachometer invocation, and writes a combined JSON report. Three pieces the originals configured are now derived. The library under test and its version pins come from the harness's `workspace:*` dependencies rather than a hardcoded name and a hand-maintained map, which also fixes competitor libraries failing to resolve in a ref build. That in turn removes the reason TACHO_PAGES existed, so it and the page glob beside it are both gone -- the configs are now the only notion of which pages exist. The base branch is detected from origin/HEAD instead of being hardcoded to master.
Adds test/tachometer, a working harness for `code-infra tacho run`. This repository ships build tooling rather than a browser library, so there is nothing here whose render time is worth tracking; the cases run a deterministic CPU workload purely to give the pipeline something real to drive. Both pages import a workspace package and put its value on screen, so a break in either resolution path -- the workspace link for the working tree, a packed tarball for every other ref -- fails the build instead of quietly measuring nothing. Two cases, chosen to walk the parts most likely to break: `workload` is the ordinary auto-expanded regression shape, and `workload-large` owns no page of its own, parameterising its sibling's with `?size=`. Running it end to end found two bugs in the tooling: - The isolated install is its own workspace root, so it inherited none of the repository's build-script approvals and pnpm failed the install outright. It now copies the repository's allowBuilds policy across, minus strictDepBuilds -- an unapproved transitive script the pages do not need should not fail a run. - A plugin's returned config is merged *over* the inline config, so the plugin's default outDir silently overrode `vite build --outDir`, and every ref's pages landed in builds/manual. It now only fills outDir in when the caller left it unset. The harness's generated directories are listed in the root .gitignore rather than a nested one because ESLint is fed from the root file and never reads nested ones.
The extraction in this PR was taken from mui/base-ui-mosaic#428 at e62e7e7, which turned out to be stale: the branch had ten further commits that were never pushed, changing precisely the parts that were extracted. Those have since been pushed. This ports them, so the shared tooling encodes the current design rather than a superseded one. Every ref now goes through an isolated install, the working tree included. Building one side through a workspace link and the other from a tarball compared two different resolution paths -- source exports against an installed package's -- and that difference landed in the measurement rather than in the library. packWorkingTree names each tarball by a hash of its bytes, which is what lets the installs persist: an unchanged build keeps its path, so pnpm has nothing to do. The cache splits into .cache/packed (what CI should cache) and .cache/installs. Verdicts now require the millisecond interval and the percent interval to exclude zero on the same side. They are computed separately from the same samples, so near zero one can exclude it while the other does not, and a verdict resting on one of them flips depending which is read. Results are paired by (variant, measurement) rather than by position. Tachometer flattens variants x measurements into one list with a difference matrix over the whole thing, so the previous differences[index + 1] silently compared unrelated measurements as soon as a case declared more than one. Benchmarks run on Playwright's pinned Chrome for Testing instead of whatever Chrome the machine auto-updated to, and the chromedriver major is checked against it before anything is built -- a mismatch now fails in seconds naming both versions instead of after minutes of packing. The isolated install uses --ignore-scripts rather than copying the repository's allowBuilds policy: nothing installed there needs its scripts, since the pages are built from the packages exactly as packed.
Two things the integration into base-ui-mosaic#428 needs, found by reading its benchmarkTable.mts and root scripts rather than by running into them later. The renderer's TachometerReport requires `version`, `reportType` and `head.branch`; the writer emitted none of them, so the table would have failed to read a report the runner considered complete. Splitting the build command matters for speed, not output. Mosaic's `build` and `release:build` run the same per-package build, differing only in that the latter skips the nx cache. A ref is built in a throwaway checkout with a cold cache and wants the thorough one, but the working tree is rebuilt on every run with a warm cache, so forcing it through --skip-nx-cache would rebuild everything each time for nothing. --working-tree-build-cmd defaults to --build-cmd, so repositories where the two are the same are unaffected.
# Conflicts: # packages/code-infra/package.json # pnpm-lock.yaml
The throwaway package is its own workspace root, so it inherited none of the repository's `overrides`. That is a correctness gap, not just a convenience: a repository that pins a transitive dependency through an override would resolve a different graph in the sandbox than in its real install, quietly changing what is being benchmarked. Found by integrating into base-ui-mosaic#428, where it surfaced as a hard failure rather than a silent one: that repository sets blockExoticSubdeps, and the overrides keeping its dependencies within that policy were missing from the sandbox, so the install refused to proceed. The repository's overrides go in first and the packed workspace packages pin over them, since those are the whole point of the install.
The renderer lived in the consuming repository while the runner defined the report it renders, so `tacho run` could only print one-line verdicts and every consumer needed a wrapper script to get a table. Worse, the two halves could drift: a change to summarizeCase here would silently break a renderer over there. Moving it puts the report's shape and its presentation in one place. `tacho run` now renders from the object it already has, with no write-then-read hop, and a consumer's package.json script is just `code-infra tacho run`. It is also exported as `./tachometerReport` so a repository that multiplexes report types through its own CLI can still call it directly. Ported from base-ui-mosaic's scripts/benchmarkTable.mts, colours swapped to chalk. What stays there is the Vitest suite's own rendering, which measures something else entirely.
`tacho run` produced a table in the CI log and nothing else, so a regression was only found by someone reading the job output. It now uploads its report to the CI report store and asks the dashboard to refresh the PR comment, joining bundle size, deploy previews and the Vitest benchmarks in the same comment. Regressions are the point, so they are stated above the collapsed block and mark the section heading. GitHub renders <details> closed, so anything inside it is invisible until someone chooses to look -- a warning buried there would be no warning at all. Everything that did not regress collapses to one line, and `unsure` is never called a regression: it is the expected result for two equivalent builds, and treating it as a warning would make the marker meaningless within a week. Only comparisons across refs count as regressions. A cross-library case runs every variant from one build, so "slower than ag-grid" is true, worth knowing, and unrelated to what the pull request changed. Unlike the benchmark section this fetches one artifact and does no comparison: a tachometer report already contains both sides, sampled round-robin in a single session, so there is no base report to look up and no chance of pairing against a mismatched one. The dashboard mirrors the report types rather than importing them, matching what it does for @mui/internal-benchmark -- it renders artifacts written months ago by whatever version of the tooling a repository had. That is the opposite of the call made for the terminal renderer, which runs in-process at the same version. The details page is deliberately plain: every number, as tables, no charts. A confidence interval is two numbers and a verdict, and drawing it as a bar invites reading the picture as significance when the interval is what carries that. The upload transport is a small copy of the benchmark package's rather than a shared import: code-infra's dependency tree is large, and having the benchmark package depend on it would push all of that onto its consumers.
It is the repository that runs the job. Enabling it before the first upload is safe: with no artifact for the head commit the generator returns null and the comment simply has no tachometer section.
Containerised CI runs as root, and Chrome will not enter its sandbox there: it exits during launch, leaving the driver to report no more than "session not created: Chrome instance exited". Adding the flag only for root keeps the sandbox on for local runs, and a case's own arguments survive the merge — the flags that hold a measurement still would otherwise be dropped.
The harness only ever ran by hand, so nothing caught a break in the pipeline it exercises. Measuring the working tree against HEAD keeps both sides identical: the verdict is always "unsure", and what the job actually proves is that packing, installing, building, sampling and reporting still work end to end. Until this lands, production rejects the report type, so the upload points at this pull request's own dashboard preview.
A comparison holds the reference relative to the variant, which is the direction a regression is stated in. Both tables put that under a "vs reference" heading on the variant's row, where it says the opposite of what it means: the real report rendered ag-grid, at 324 ms against a 192 ms reference, as "faster -42.8%". The summary line beside it counted regressed measurements against a total in cases, and counted a case as "faster" for beating a competing library — neither of which the sentence claims. Both now read the cross-ref comparisons a regression is taken from, and count cases.
The page had one flat table per case, a row per measurement and variant, which is not how the run reads in a terminal. It now groups cases by their variant set the same way: two variants become one table across the group, a row per case and measurement with a column per variant and a Δ against each; more than two become a table per case, a row per variant, since a column per variant plus a Δ per pair would run off the page. Transfer size and the build labels move into the notes underneath, where the terminal keeps them. The grouping is a pure module of its own, so what decides a table's shape is testable without rendering anything.
`run`, `pathExists`, and the dashboard's name and number formatting had all been written out two or three times, and one report's two renderers could drift on precision without anything noticing. The upload and the comment sync each carried their own copy of the same authenticated POST. The transfer-size note in both renderers derived its cases from a set filled while printing, so the notes silently depended on the tables having rendered first. Along the way: packing caps its concurrency rather than starting a process per public package, tarballs are hashed as streams instead of read whole, a resolved git ref is remembered rather than re-resolved per case, and a ref's sandbox stops installing the browser and driver that only the runner resolves.
…path Discovery walked one level, so every case had to sit directly under `src/`, and the filter matched a folder name. Both now work on the path: cases are found at any depth and a filter is a case-insensitive substring of the path, so naming a folder selects everything beneath it. That is the whole of the partitioning story — a quick set for pull requests, everything for a release — without the filter having to mean anything. A case takes its name from its benchmark's own `name` rather than its directory, which were previously two declarations of one identity kept in sync by hand: the ` [<variant>]` prefix tachometer builds from that name is now strippable for display by construction, and moving a case between folders leaves its name in every report untouched. Two cases claiming the same name stop discovery, naming both files, since a duplicate would otherwise quietly overwrite the other's results.
The built pages, packed tarballs, isolated installs and reports each had their own directory in the harness, so a repository had three entries to ignore and had to learn which — mui-public's ignore file carried a comment explaining why they sat at the root, which is exactly the knowledge a second consumer has to rediscover. One directory means one entry, and deleting it is the whole reset. It also writes a `.gitignore` holding `*` inside, the way pytest marks its cache, so a harness that forgets the entry does not offer tarballs for commit. That does not replace the root entry here: ESLint is fed from the root ignore file and never reads nested ones. Tachometer serves the pages through koa-static, which refuses a dot-prefixed segment in a served path, so each case's config is now rooted at the output directory rather than at the harness — which keeps the folder out of the url.
…hometer code Node ships an ANSI stripper, so the regex built from a char code to dodge `no-control-regex` goes; globby is already a dependency and replaces the hand-rolled recursive walk that found case configs; the dashboard's own markdown table helper replaces hand-concatenated pipe rows and separator lines. The terminal renderer had drifted from the dashboard's version of the same tables: it accumulated a row's transfer size and sample count as side effects of rendering that row's cells, formatted milliseconds through `Intl` only to strip the unit back off with a regex, carried a parallel array of column alignments restating the header layout, and partitioned cases on two conditions that had to agree — leaving a case with an empty `measurements` array rendered nowhere and reported nowhere. All four now match how the dashboard does it. The output directory names its own `builds/` subpath, so the runner and the plugin no longer compose it separately while having to agree; the base branch is detected only when the `baseline` symbol is actually resolved.
…h its own tree Each ref used to be built in a copy of the harness — `src/`, one vite config and `.env*`, and nothing else. Every other file a harness owns was invisible to that build: `tsconfig.json`, which vite reads for its esbuild settings, and any postcss or tailwind config, and any file the vite config itself imports. The build still succeeded, just configured differently from the harness's own, and the difference landed in the measurement. Now the pages are built where they live, from the harness's own directory and config, and only *resolution* differs: a plugin sends every bare import into a small tree installed for that ref, where the library under test is that ref's packed build. Resolution is handed back to vite with the importer rewritten, so conditions and `exports` maps still apply — an alias to a directory would skip both. The tree carries the harness's own dependencies too, so the pages and the library share one copy of React. The tree no longer installs this package either. It is the harness's vite config that loads the plugin, and that config is read from the harness — which takes a ref's tree from 976 packages to 80.
`vite preview` runs as `serve`, so the plugin's early return skipped `build.outDir` and preview looked for `<root>/dist` — inside `src/` — failing with "The directory \"dist\" does not exist" instead of serving the pages that had just been built.
Four ways a legal `tachometer.json` was mishandled, none of which the smoke harness exercises:
- A case that declares no `browser` had `{ binary }` injected, which tachometer's schema rejects
outright — every browser variant requires `name` and forbids unknown keys — so the run died
before sampling. A case using the string shorthand ("chrome-headless") was spread into one key
per character, failing the same way.
- Browser defaults were applied only to the top-level benchmark. `expand` shallow-overrides, so a
variant declaring its own `browser` lost the binary and, as root, the flag Chrome needs to start.
- The `global` measurement shorthand expands to an expression measurement named by the expression,
so pairing on "global" never matched and the case was reported as failed.
- A benchmark that names neither itself nor its variants produced "undefined [current]", and a name
set partway down an `expand` tree was skipped in favour of the benchmark's.
`baseRef` labels `baseSha` in the header, so passing the run's own branch read "comparing against <this PR's branch> (<the merge base>)". The baseline ref's label describes that commit.
…er does Inheritance down `expand` was being re-derived per field: `url` and `name` threaded through the leaf walk, `browser` through a second recursive walk with an `isRoot` flag, and `measurementExpression` not at all. Each is a separate mirror of one rule — tachometer's `Object.assign` merge — and is only worth anything while it agrees with that rule exactly. Flattening once at discovery collapses all of it. The leaf walk and the browser walk both disappear, every benchmark handed to tachometer carries its own effective fields, and inheritance for fields nobody has needed yet comes for free. `expand` is dropped on the way out: tachometer leaves the parent's on the merged object where it is inert, but the config written here is parsed afresh and would otherwise expand a second time.
Vite needs a base directory and derives one from the importer's path, falling back to the project root unless that path exists on disk. That was satisfied by writing an empty `__resolve__.js` into every ref tree — inventing a file to make a path lookup succeed. The tree's `package.json` is already there and names the same directory, which is how `createRequire` is pointed at one. Also reuse `writePackageJson` for the generated manifest, and read this package's own name rather than spelling it out in `RUNNER_ONLY_DEPS` — a rename would otherwise quietly restore a multi-minute install per ref.
… script names The index listing every case was a dev-server behaviour, so a built directory could only be entered by typing page urls from memory — and a case that owns no page, reachable only through the query in its config, could not be entered at all. Emitting it as a real asset makes the build navigable under `preview`, under any static server, or opened from disk. Links are relative, which `base: './'` needs and which resolves the same on the dev server. The harness's `build:pages` becomes `build`, restoring vite's familiar dev/build/preview trio. Nothing swept it up: this repository's `release:build` is scoped to `./packages/*`.
The tree is its own workspace root and installs without the repository's lockfile, so copying a declared range let it float to whatever was newest at run time. base-ui-mosaic's CI hit the visible half of this: the harness declares `@base-ui/react: ^1.6.0`, the repository resolves 1.7.0, and the tree installed the freshly released 1.8.0 — which no longer exports the `./internals/*` paths the library imports, so every ref build failed. The quiet half is worse. A range that floats to a merely *slower* release measures that release and attributes the difference to our own code, which is the one thing a benchmark must not do. Direct dependencies are now pinned to what the harness actually has installed, read from its own node_modules.
# Conflicts: # package.json # packages/code-infra/package.json # pnpm-lock.yaml
`tacho run` prints its table and writes the JSON, but reading that JSON afterwards meant either sampling again or wiring a renderer into the consuming repository. The renderer already lives here, so the command that produced the report can print it: a report kept from an earlier run, or downloaded from a CI artifact, is now readable in place. Rejects a report from the other benchmark axis rather than rendering it — the envelope is the same but the statistics are not, and its numbers would be read as confidence intervals on a difference.
A case sends Input.synthesizeScrollGesture itself. A gesture API can come back once more than scrolling needs one.
Janpot
added
the
scope: code-infra
A case's registration only checks its options; the runner's merge drops a pick that changed nothing, since own options are already picked.
getSampling validates and keeps the options set, when a case registers and when a run starts; from there a plain spread over the defaults merges them. A bad --sample-size now fails before anything is built.
The benchmark CLI launches Chromium through it, so it is no longer optional.
Its launch error says how to install a missing Chromium; checking ahead of the builds only saved time on a fresh machine.
Completes the previous commit, which only deleted the file.
uploadReportFile resolves the report the way a run writes it, reads the CI metadata once, and says when there is no report rather than failing with ENOENT. compare() checks a case's own sampling by value.
Replaces the hand-rolled static server; every repository running the CLI already has sirv through Vitest.
ciReportUploadSchema moves to a browser-safe module, so the run report's upload extends it instead of repeating its fields.
Look metric names up as own keys, decide which side is worse once, drop a check severity already implies, and keep median with the other statistics.
Close the static server even when the browser fails to launch, take median from es-toolkit, orient a comparison's interval once instead of flipping signs in three places, share the own-or-base metric lookup, and keep commitTimestamp in the shared CI envelope.
Janpot
marked this pull request as ready for review
October 15, 2026 10:10
Janpot
added a commit
to Janpot/base-ui
that referenced
this pull request
Oct 15, 2026
Janpot
added a commit
to Janpot/material-ui
that referenced
this pull request
Oct 15, 2026
Janpot
added a commit
to Janpot/material-ui
that referenced
this pull request
Oct 15, 2026
Janpot
changed the title
The version 2 report from the benchmark CLI replaces the Vitest job's upload.
The benchmark CLI job takes over the test_benchmark name, which branch rules require.
hasdfa
temporarily deployed
to
janpot/benchmark-interleaved - code-infra-dashboard PR #1898
October 20, 2026 08:50 — with 
Render
Destroyed
Render
Destroyed
A timeout of 0 asks for exactly sampleSize rounds, so stopping there unsettled is not reported as timed out.
They check that the CLI works end to end rather than guard a library, so a few rounds are enough; the libs-sort comparison now samples the same way.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
benchmarkCLI, the successor to the Vitest harness. Benchmark files (*.bench.tsx) are written against@mui/internal-benchmark/page(benchmark(),reactBenchmark(),compare()).code-infra baseline, and the parent commit by default.sampleSizerounds, then keeps going until every alarmed difference is known to be past its threshold or short of it, or untiltimeout.--profilerecords DevTools traces instead of measuring.Against the Vitest harness, 10 CI runs each on a change that doesn't touch the library:
Repositories trying it from this pull request's pkg.pr.new build:
Vitest cleanup, once every repository has moved to the CLI:
.,./vitestand./reporterentry points and the code only they usevitestand@vitest/browser-playwrightpeer dependenciesbenchmarkreports in the dashboard