From f682de212f49083b812687c29b5fa7b481a32671 Mon Sep 17 00:00:00 2001 From: Mark Pearce Date: Thu, 24 Sep 2026 10:21:51 -0300 Subject: [PATCH 1/3] Add scenario benchmarks for real-project memory and re-validation `npm run benchmark:scenarios` runs pinned real projects in a fresh process each and records cold load/validate time, validate phases, retained heap, peak RSS, edit re-validation (shared vs leaf file, comment vs new function) and a diagnostics hash so behavior changes get caught. Can compare bsc versions, compare results across branches, capture cpu/heap/edit profiles and summarize them. Co-Authored-By: Claude Opus 5.5 --- .gitignore | 2 + benchmarks/results.json | 9 - benchmarks/scenarios/README.md | 55 +++ benchmarks/scenarios/ScenarioProjects.ts | 123 +++++++ benchmarks/scenarios/analyzeProfile.ts | 132 +++++++ benchmarks/scenarios/index.ts | 335 ++++++++++++++++++ benchmarks/scenarios/projects.json | 71 ++++ .../scenarios/projects.local.example.json | 15 + benchmarks/scenarios/runScenario.js | 246 +++++++++++++ package.json | 1 + 10 files changed, 980 insertions(+), 9 deletions(-) delete mode 100644 benchmarks/results.json create mode 100644 benchmarks/scenarios/README.md create mode 100644 benchmarks/scenarios/ScenarioProjects.ts create mode 100644 benchmarks/scenarios/analyzeProfile.ts create mode 100644 benchmarks/scenarios/index.ts create mode 100644 benchmarks/scenarios/projects.json create mode 100644 benchmarks/scenarios/projects.local.example.json create mode 100644 benchmarks/scenarios/runScenario.js diff --git a/.gitignore b/.gitignore index a11b07c92..a8af72b57 100644 --- a/.gitignore +++ b/.gitignore @@ -16,3 +16,5 @@ isolate-* v8*.log *.cpuprofile .cache.json +benchmarks/scenarios/results +benchmarks/scenarios/projects.local.json diff --git a/benchmarks/results.json b/benchmarks/results.json deleted file mode 100644 index 99dd812eb..000000000 --- a/benchmarks/results.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "lexer": { - "current": [ - 12645.727126865311, - 13072.590947406612, - 11663.650306847825 - ] - } -} \ No newline at end of file diff --git a/benchmarks/scenarios/README.md b/benchmarks/scenarios/README.md new file mode 100644 index 000000000..c21900cf6 --- /dev/null +++ b/benchmarks/scenarios/README.md @@ -0,0 +1,55 @@ +# Scenario benchmarks + +The ops/sec benchmarks in `benchmarks/targets` answer "is X faster than it was?". These answer "where does the time and memory go on a real project?" - and keep answering it the same way over time. + +Each run is a fresh node process per project/version, and records: +- cold load and cold validate time, plus time per validate phase +- retained heap after validation (after a forced GC) and peak RSS +- edit re-validation, like typing in the language server: a comment added to (`body`) or a function added to (`api`) the script file in the most scopes (`shared`) and the biggest script file in only one scope (`leaf`). Every edit is reverted at the end +- a hash of all diagnostics, so a perf change that changes behavior gets caught. `edits restored` checks the diagnostics are identical after all the edits are reverted + +Timings in the summary are the best (min) of `--runs`. + +## Usage + +```bash +npm run build +npm run benchmark:scenarios -- run +npm run benchmark:scenarios -- run --projects jellyfin-roku --runs 5 +``` + +Compare versions in one go (first `--bsc` is the baseline). Anything other than `local` or a path gets npm installed into `.tmp`: + +```bash +npm run benchmark:scenarios -- run --bsc 1.0.0-alpha.54 local +``` + +Compare runs from different branches: + +```bash +npm run benchmark:scenarios -- run --label before +# ...switch branches, npm run build... +npm run benchmark:scenarios -- run --label after +npm run benchmark:scenarios -- compare before.json after.json +``` + +Profiling (`cpu` = whole run, `edits` = just the edit re-validation, `heap` = sampled allocations still live at the end), then summarize: + +```bash +npm run benchmark:scenarios -- run --projects jellyfin-roku --runs 1 --profile edits +npm run benchmark:scenarios -- analyze scenarios/results/profiles/