Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove reportErrors check in relateVariances #55222

Draft
wants to merge 14 commits into
base: main
Choose a base branch
from
Draft

Remove reportErrors check in relateVariances #55222

wants to merge 14 commits into from

Conversation

jakebailey
Copy link
Member

@jakebailey jakebailey commented Jul 31, 2023 •
edited
Loading

Fixes #55217

As noted by @Andarist, this check is what causes things to differ between errors and non-error modes in relation, causing oddities when the same relation is observed in both ways but in different orders.

No doubt this is a bad idea but I wanted to run the test suite to see what happens.

Bunch of WIP code is left in the PR, I'll remove/refactor it if this code actually looks acceptable.

@typescript-bot typescript-bot added Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug labels Jul 31, 2023
@jakebailey
Copy link
Member Author

@typescript-bot test this
@typescript-bot test top100
@typescript-bot user test this
@typescript-bot user test tsserver
@typescript-bot test tsserver top100
@typescript-bot run dt
@typescript-bot perf test this
@typescript-bot pack this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 31, 2023 •
edited
Loading

Heya @jakebailey, I've started to run the diff-based user code test suite on this PR at df695d9. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 31, 2023 •
edited
Loading

Heya @jakebailey, I've started to run the diff-based user code test suite (tsserver) on this PR at df695d9. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 31, 2023 •
edited
Loading

Heya @jakebailey, I've started to run the tarball bundle task on this PR at df695d9. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 31, 2023 •
edited
Loading

Heya @jakebailey, I've started to run the extended test suite on this PR at df695d9. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 31, 2023 •
edited
Loading

Heya @jakebailey, I've started to run the parallelized Definitely Typed test suite on this PR at df695d9. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 31, 2023 •
edited
Loading

Heya @jakebailey, I've started to run the perf test suite on this PR at df695d9. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 31, 2023 •
edited
Loading

Heya @jakebailey, I've started to run the diff-based top-repos suite on this PR at df695d9. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 31, 2023 •
edited
Loading

Heya @jakebailey, I've started to run the diff-based top-repos suite (tsserver) on this PR at df695d9. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 31, 2023 •
edited
Loading

Hey @jakebailey, I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/156090/artifacts?artifactName=tgz&fileId=4E1314C8F2DBC21591658BB650878A546D6F6E3C21856E1045D90F437FA08F0902&fileName=/typescript-5.2.0-insiders.20230731.tgz"
    }
}

and then running npm install.

There is also a playground for this build and an npm module you can use via "typescript": "npm:@typescript-deploys/[email protected]".;

jakebailey
jakebailey commented Jul 31, 2023
View reviewed changes
src/compiler/checker.ts Outdated
Comment on lines 21289 to 21292
// if (entry !== undefined) {
// // If the previous entry and the result disagree, then something has gone wrong.
// Debug.assert(!!(entry & RelationComparisonResult.Succeeded) === (result !== Ternary.False), "Cached relationship does not match recalculated result");
// }
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably this is a good thing to leave in, once the actual fix is in, that way we can observe this kind of bug faster.

jakebailey
jakebailey commented Jul 31, 2023
View reviewed changes
src/compiler/checker.ts Outdated
@@ -21951,7 +21956,7 @@ export function createTypeChecker(host: TypeCheckerHost): TypeChecker {
// reveal the reason).
// We can switch on `reportErrors` here, since varianceCheckFailed guarantees we return `False`,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This mentions the code I deleted, so I'd have to delete this if this fix is right.

@typescript-bot
Copy link
Collaborator

@jakebailey Here are the results of running the user test suite comparing main and refs/pull/55222/merge:

Everything looks good!

@typescript-bot
Copy link
Collaborator

@jakebailey Here are the results of running the user test suite comparing main and refs/pull/55222/merge:

There were infrastructure failures potentially unrelated to your change:

  • 1 instance of "Unknown failure"
  • 1 instance of "Package install failed"

Otherwise...

Something interesting changed - please have a look.

Details

rxjs-src

/mnt/ts_downloads/rxjs-src/build.sh

  • [NEW] error TS2428: All declarations of 'WeakMap' must have identical type parameters.
    • /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.collection.d.ts(63,11)
    • /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.iterable.d.ts(162,11)
    • /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.symbol.wellknown.d.ts(140,11)
    • /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.collection.d.ts(63,11)
    • /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.iterable.d.ts(162,11)
    • /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.symbol.wellknown.d.ts(140,11)
    • /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.collection.d.ts(63,11)
    • /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.iterable.d.ts(162,11)
    • /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.symbol.wellknown.d.ts(140,11)
    • /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.collection.d.ts(63,11)
    • /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.iterable.d.ts(162,11)
    • /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.symbol.wellknown.d.ts(140,11)
    • /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.collection.d.ts(63,11)
    • /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.iterable.d.ts(162,11)
    • /home/vsts/work/1/s/typescript-55222/lib/lib.es2015.symbol.wellknown.d.ts(140,11)
  • [MISSING] error TS2428: All declarations of 'WeakMap' must have identical type parameters.
    • /home/vsts/work/1/s/typescript-main/lib/lib.es2015.collection.d.ts(63,11)
    • /home/vsts/work/1/s/typescript-main/lib/lib.es2015.iterable.d.ts(162,11)
    • /home/vsts/work/1/s/typescript-main/lib/lib.es2015.symbol.wellknown.d.ts(140,11)
    • /home/vsts/work/1/s/typescript-main/lib/lib.es2015.collection.d.ts(63,11)
    • /home/vsts/work/1/s/typescript-main/lib/lib.es2015.iterable.d.ts(162,11)
    • /home/vsts/work/1/s/typescript-main/lib/lib.es2015.symbol.wellknown.d.ts(140,11)
    • /home/vsts/work/1/s/typescript-main/lib/lib.es2015.collection.d.ts(63,11)
    • /home/vsts/work/1/s/typescript-main/lib/lib.es2015.iterable.d.ts(162,11)
    • /home/vsts/work/1/s/typescript-main/lib/lib.es2015.symbol.wellknown.d.ts(140,11)
    • /home/vsts/work/1/s/typescript-main/lib/lib.es2015.collection.d.ts(63,11)
    • /home/vsts/work/1/s/typescript-main/lib/lib.es2015.iterable.d.ts(162,11)
    • /home/vsts/work/1/s/typescript-main/lib/lib.es2015.symbol.wellknown.d.ts(140,11)
    • /home/vsts/work/1/s/typescript-main/lib/lib.es2015.collection.d.ts(63,11)
    • /home/vsts/work/1/s/typescript-main/lib/lib.es2015.iterable.d.ts(162,11)
    • /home/vsts/work/1/s/typescript-main/lib/lib.es2015.symbol.wellknown.d.ts(140,11)

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - main..55222
Metric main 55222 Delta Best Worst p-value
Angular - node (v18.10.0, x64)
Memory used 368,785k (± 0.01%) 368,794k (± 0.01%) ~ 368,756k 368,821k p=0.689 n=6
Parse Time 3.40s (± 0.55%) 3.40s (± 0.42%) ~ 3.38s 3.42s p=0.685 n=6
Bind Time 1.12s (± 0.92%) 1.12s (± 0.73%) ~ 1.11s 1.13s p=0.932 n=6
Check Time 8.95s (± 0.34%) 9.00s (± 0.44%) +0.05s (+ 0.60%) 8.93s 9.04s p=0.045 n=6
Emit Time 7.57s (± 1.03%) 7.58s (± 0.76%) ~ 7.50s 7.65s p=1.000 n=6
Total Time 21.05s (± 0.38%) 21.10s (± 0.49%) ~ 20.95s 21.22s p=0.421 n=6
Compiler-Unions - node (v18.10.0, x64)
Memory used 193,051k (± 1.18%) 194,922k (± 1.56%) ~ 192,120k 197,736k p=0.128 n=6
Parse Time 1.51s (± 0.65%) 1.50s (± 0.98%) ~ 1.48s 1.52s p=0.247 n=6
Bind Time 0.78s (± 0.66%) 0.77s (± 0.67%) ~ 0.77s 0.78s p=0.311 n=6
Check Time 9.47s (± 0.42%) 9.46s (± 0.30%) ~ 9.42s 9.50s p=1.000 n=6
Emit Time 2.76s (± 0.38%) 2.72s (± 1.02%) ~ 2.70s 2.77s p=0.052 n=6
Total Time 14.51s (± 0.35%) 14.46s (± 0.18%) ~ 14.42s 14.48s p=0.053 n=6
Monaco - node (v18.10.0, x64)
Memory used 347,755k (± 0.01%) 347,772k (± 0.01%) ~ 347,741k 347,813k p=0.471 n=6
Parse Time 2.62s (± 1.05%) 2.63s (± 1.19%) ~ 2.58s 2.66s p=0.418 n=6
Bind Time 1.01s (± 0.74%) 1.01s (± 0.51%) ~ 1.01s 1.02s p=0.784 n=6
Check Time 7.29s (± 0.38%) 7.29s (± 0.64%) ~ 7.21s 7.35s p=0.936 n=6
Emit Time 4.24s (± 0.90%) 4.27s (± 0.88%) ~ 4.22s 4.33s p=0.260 n=6
Total Time 15.16s (± 0.51%) 15.21s (± 0.65%) ~ 15.03s 15.31s p=0.261 n=6
TFS - node (v18.10.0, x64)
Memory used 301,794k (± 0.01%) 301,786k (± 0.01%) ~ 301,751k 301,818k p=0.574 n=6
Parse Time 2.08s (± 1.23%) 2.07s (± 1.77%) ~ 2.04s 2.13s p=0.871 n=6
Bind Time 1.13s (± 1.03%) 1.12s (± 0.36%) -0.01s (- 1.18%) 1.11s 1.12s p=0.025 n=6
Check Time 6.67s (± 0.53%) 6.66s (± 0.50%) ~ 6.61s 6.70s p=0.518 n=6
Emit Time 3.86s (± 0.95%) 3.89s (± 0.48%) ~ 3.86s 3.91s p=0.259 n=6
Total Time 13.75s (± 0.50%) 13.74s (± 0.44%) ~ 13.65s 13.84s p=1.000 n=6
material-ui - node (v18.10.0, x64)
Memory used 482,550k (± 0.01%) 482,585k (± 0.02%) ~ 482,533k 482,774k p=0.230 n=6
Parse Time 3.11s (± 1.18%) 3.09s (± 2.29%) ~ 2.95s 3.14s p=0.933 n=6
Bind Time 0.92s (± 2.44%) 0.94s (± 5.00%) ~ 0.91s 1.03s p=1.000 n=6
Check Time 17.32s (± 0.73%) 17.33s (± 0.78%) ~ 17.09s 17.49s p=0.470 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 21.36s (± 0.54%) 21.37s (± 0.63%) ~ 21.14s 21.55s p=0.334 n=6
xstate - node (v18.10.0, x64)
Memory used 563,842k (± 0.02%) 565,624k (± 0.01%) +1,782k (+ 0.32%) 565,514k 565,730k p=0.005 n=6
Parse Time 3.85s (± 0.61%) 3.85s (± 0.35%) ~ 3.83s 3.87s p=0.806 n=6
Bind Time 1.64s (± 1.24%) 1.64s (± 0.63%) ~ 1.62s 1.65s p=0.935 n=6
Check Time 2.81s (± 1.00%) 3.03s (± 0.61%) +0.22s (+ 7.64%) 3.01s 3.05s p=0.005 n=6
Emit Time 0.08s (± 0.00%) 0.08s (± 6.44%) ~ 0.08s 0.09s p=0.071 n=6
Total Time 8.39s (± 0.27%) 8.61s (± 0.24%) +0.22s (+ 2.64%) 8.58s 8.64s p=0.005 n=6
Angular - node (v16.17.1, x64)
Memory used 368,218k (± 0.00%) 368,222k (± 0.01%) ~ 368,193k 368,246k p=0.575 n=6
Parse Time 3.57s (± 0.39%) 3.55s (± 0.18%) ~ 3.54s 3.56s p=0.059 n=6
Bind Time 1.19s (± 0.63%) 1.19s (± 0.43%) ~ 1.18s 1.19s p=0.784 n=6
Check Time 9.76s (± 0.72%) 9.77s (± 0.54%) ~ 9.68s 9.83s p=1.000 n=6
Emit Time 8.04s (± 0.80%) 8.01s (± 0.39%) ~ 7.96s 8.04s p=0.228 n=6
Total Time 22.55s (± 0.58%) 22.51s (± 0.31%) ~ 22.37s 22.56s p=0.374 n=6
Compiler-Unions - node (v16.17.1, x64)
Memory used 193,823k (± 0.02%) 194,372k (± 0.67%) ~ 193,796k 197,022k p=0.261 n=6
Parse Time 1.59s (± 0.96%) 1.59s (± 0.62%) ~ 1.58s 1.61s p=0.557 n=6
Bind Time 0.82s (± 0.92%) 0.82s (± 0.50%) ~ 0.82s 0.83s p=1.000 n=6
Check Time 10.16s (± 0.48%) 10.17s (± 0.58%) ~ 10.09s 10.25s p=0.810 n=6
Emit Time 2.98s (± 0.87%) 3.01s (± 0.91%) ~ 2.97s 3.05s p=0.106 n=6
Total Time 15.56s (± 0.35%) 15.60s (± 0.24%) ~ 15.55s 15.64s p=0.149 n=6
Monaco - node (v16.17.1, x64)
Memory used 347,078k (± 0.01%) 347,085k (± 0.01%) ~ 347,028k 347,116k p=0.630 n=6
Parse Time 2.77s (± 0.44%) 2.76s (± 0.37%) ~ 2.75s 2.78s p=0.675 n=6
Bind Time 1.08s (± 0.48%) 1.08s (± 0.70%) ~ 1.07s 1.09s p=0.784 n=6
Check Time 8.04s (± 0.61%) 8.06s (± 0.33%) ~ 8.03s 8.10s p=0.295 n=6
Emit Time 4.48s (± 0.56%) 4.47s (± 0.60%) ~ 4.44s 4.51s p=0.227 n=6
Total Time 16.36s (± 0.36%) 16.37s (± 0.38%) ~ 16.30s 16.48s p=1.000 n=6
TFS - node (v16.17.1, x64)
Memory used 301,118k (± 0.01%) 301,109k (± 0.01%) ~ 301,082k 301,135k p=0.377 n=6
Parse Time 2.20s (± 0.38%) 2.19s (± 0.24%) ~ 2.18s 2.19s p=0.073 n=6
Bind Time 1.22s (± 1.70%) 1.21s (± 1.60%) ~ 1.18s 1.23s p=0.625 n=6
Check Time 7.34s (± 0.31%) 7.33s (± 0.71%) ~ 7.24s 7.38s p=1.000 n=6
Emit Time 4.30s (± 1.17%) 4.33s (± 0.97%) ~ 4.29s 4.39s p=0.146 n=6
Total Time 15.06s (± 0.25%) 15.06s (± 0.57%) ~ 14.95s 15.17s p=0.748 n=6
material-ui - node (v16.17.1, x64)
Memory used 481,882k (± 0.01%) 481,860k (± 0.01%) ~ 481,811k 481,954k p=0.297 n=6
Parse Time 3.26s (± 0.72%) 3.23s (± 0.26%) ~ 3.23s 3.25s p=0.122 n=6
Bind Time 0.96s (± 0.85%) 0.95s (± 0.57%) ~ 0.95s 0.96s p=0.859 n=6
Check Time 18.34s (± 0.39%) 18.34s (± 0.74%) ~ 18.21s 18.57s p=0.687 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 22.55s (± 0.36%) 22.53s (± 0.63%) ~ 22.41s 22.78s p=0.422 n=6
xstate - node (v16.17.1, x64)
Memory used 561,343k (± 0.01%) 563,061k (± 0.01%) +1,718k (+ 0.31%) 562,967k 563,164k p=0.005 n=6
Parse Time 4.02s (± 0.30%) 4.02s (± 0.43%) ~ 4.00s 4.05s p=0.871 n=6
Bind Time 1.66s (± 5.88%) 1.77s (± 1.62%) +0.11s (+ 6.82%) 1.73s 1.81s p=0.024 n=6
Check Time 3.09s (± 1.60%) 3.28s (± 1.26%) +0.19s (+ 6.09%) 3.23s 3.34s p=0.005 n=6
Emit Time 0.09s (± 5.53%) 0.09s (± 0.00%) ~ 0.09s 0.09s p=0.174 n=6
Total Time 8.87s (± 0.58%) 9.17s (± 0.82%) +0.29s (+ 3.33%) 9.06s 9.28s p=0.005 n=6
Angular - node (v14.21.3, x64)
Memory used 362,126k (± 0.01%) 362,093k (± 0.01%) ~ 362,054k 362,155k p=0.128 n=6
Parse Time 3.71s (± 0.14%) 3.73s (± 0.37%) +0.02s (+ 0.54%) 3.71s 3.74s p=0.020 n=6
Bind Time 1.22s (± 0.00%) 1.22s (± 0.62%) ~ 1.21s 1.23s p=0.598 n=6
Check Time 10.13s (± 0.37%) 10.15s (± 0.56%) ~ 10.06s 10.22s p=0.377 n=6
Emit Time 8.35s (± 0.56%) 8.36s (± 0.55%) ~ 8.30s 8.41s p=0.748 n=6
Total Time 23.41s (± 0.27%) 23.46s (± 0.32%) ~ 23.33s 23.53s p=0.336 n=6
Compiler-Unions - node (v14.21.3, x64)
Memory used 189,118k (± 0.01%) 189,094k (± 0.01%) ~ 189,060k 189,128k p=0.092 n=6
Parse Time 1.62s (± 0.72%) 1.61s (± 0.32%) ~ 1.61s 1.62s p=0.191 n=6
Bind Time 0.85s (± 0.61%) 0.85s (± 0.61%) ~ 0.84s 0.85s p=1.000 n=6
Check Time 10.32s (± 0.70%) 10.33s (± 0.44%) ~ 10.28s 10.39s p=0.574 n=6
Emit Time 3.11s (± 0.55%) 3.12s (± 0.33%) ~ 3.10s 3.13s p=0.741 n=6
Total Time 15.90s (± 0.49%) 15.91s (± 0.21%) ~ 15.86s 15.95s p=0.520 n=6
Monaco - node (v14.21.3, x64)
Memory used 342,072k (± 0.01%) 342,077k (± 0.01%) ~ 342,045k 342,108k p=0.936 n=6
Parse Time 2.80s (± 0.87%) 2.80s (± 0.76%) ~ 2.78s 2.84s p=0.684 n=6
Bind Time 1.09s (± 0.50%) 1.09s (± 0.37%) ~ 1.09s 1.10s p=0.282 n=6
Check Time 8.30s (± 0.26%) 8.31s (± 0.43%) ~ 8.26s 8.36s p=0.687 n=6
Emit Time 4.67s (± 0.87%) 4.66s (± 0.57%) ~ 4.63s 4.69s p=0.570 n=6
Total Time 16.86s (± 0.40%) 16.87s (± 0.23%) ~ 16.81s 16.91s p=0.809 n=6
TFS - node (v14.21.3, x64)
Memory used 296,199k (± 0.01%) 296,199k (± 0.01%) ~ 296,181k 296,222k p=0.936 n=6
Parse Time 2.42s (± 0.43%) 2.42s (± 0.80%) ~ 2.39s 2.44s p=1.000 n=6
Bind Time 1.08s (± 0.91%) 1.08s (± 0.59%) ~ 1.07s 1.09s p=0.733 n=6
Check Time 7.66s (± 0.42%) 7.67s (± 0.48%) ~ 7.61s 7.70s p=0.413 n=6
Emit Time 4.27s (± 0.69%) 4.28s (± 0.84%) ~ 4.23s 4.32s p=0.871 n=6
Total Time 15.44s (± 0.22%) 15.46s (± 0.40%) ~ 15.38s 15.54s p=0.629 n=6
material-ui - node (v14.21.3, x64)
Memory used 477,350k (± 0.00%) 477,350k (± 0.00%) ~ 477,319k 477,365k p=0.748 n=6
Parse Time 3.31s (± 0.31%) 3.29s (± 0.31%) ~ 3.28s 3.31s p=0.067 n=6
Bind Time 0.99s (± 0.41%) 0.99s (± 0.76%) ~ 0.98s 1.00s p=0.389 n=6
Check Time 19.15s (± 0.40%) 19.21s (± 0.56%) ~ 19.08s 19.34s p=0.471 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 23.45s (± 0.30%) 23.49s (± 0.48%) ~ 23.37s 23.64s p=0.630 n=6
xstate - node (v14.21.3, x64)
Memory used 550,168k (± 0.00%) 551,869k (± 0.01%) +1,701k (+ 0.31%) 551,833k 551,905k p=0.005 n=6
Parse Time 4.22s (± 0.61%) 4.20s (± 0.51%) ~ 4.18s 4.24s p=0.258 n=6
Bind Time 1.69s (± 0.30%) 1.68s (± 1.74%) ~ 1.62s 1.70s p=0.242 n=6
Check Time 3.13s (± 0.48%) 3.33s (± 0.84%) +0.21s (+ 6.56%) 3.29s 3.37s p=0.005 n=6
Emit Time 0.10s (± 5.76%) 0.09s (± 0.00%) ~ 0.09s 0.09s p=0.071 n=6
Total Time 9.13s (± 0.35%) 9.31s (± 0.62%) +0.18s (+ 1.97%) 9.20s 9.36s p=0.005 n=6
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-148-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v18.10.0, x64)
  • node (v16.17.1, x64)
  • node (v14.21.3, x64)
Scenarios
  • Angular - node (v18.10.0, x64)
  • Angular - node (v16.17.1, x64)
  • Angular - node (v14.21.3, x64)
  • Compiler-Unions - node (v18.10.0, x64)
  • Compiler-Unions - node (v16.17.1, x64)
  • Compiler-Unions - node (v14.21.3, x64)
  • Monaco - node (v18.10.0, x64)
  • Monaco - node (v16.17.1, x64)
  • Monaco - node (v14.21.3, x64)
  • TFS - node (v18.10.0, x64)
  • TFS - node (v16.17.1, x64)
  • TFS - node (v14.21.3, x64)
  • material-ui - node (v18.10.0, x64)
  • material-ui - node (v16.17.1, x64)
  • material-ui - node (v14.21.3, x64)
  • xstate - node (v18.10.0, x64)
  • xstate - node (v16.17.1, x64)
  • xstate - node (v14.21.3, x64)
Benchmark Name Iterations
Current 55222 6
Baseline main 6

TSServer

Comparison Report - main..55222
Metric main 55222 Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.10.0, x64)
Req 1 - updateOpen 2,525ms (± 1.15%) 2,537ms (± 0.76%) ~ 2,511ms 2,559ms p=0.630 n=6
Req 2 - geterr 5,384ms (± 0.47%) 5,403ms (± 0.56%) ~ 5,365ms 5,435ms p=0.336 n=6
Req 3 - references 342ms (± 0.86%) 340ms (± 1.19%) ~ 335ms 345ms p=0.226 n=6
Req 4 - navto 289ms (± 0.34%) 291ms (± 0.95%) ~ 287ms 295ms p=0.141 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 76ms (± 1.08%) 76ms (± 3.99%) ~ 74ms 82ms p=0.451 n=6
CompilerTSServer - node (v18.10.0, x64)
Req 1 - updateOpen 2,618ms (± 0.49%) 2,636ms (± 1.01%) ~ 2,604ms 2,670ms p=0.336 n=6
Req 2 - geterr 4,136ms (± 0.41%) 4,153ms (± 0.50%) ~ 4,118ms 4,177ms p=0.092 n=6
Req 3 - references 350ms (± 1.01%) 351ms (± 0.36%) ~ 349ms 353ms p=0.141 n=6
Req 4 - navto 289ms (± 0.36%) 289ms (± 0.48%) ~ 286ms 290ms p=0.867 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 79ms (± 6.58%) 78ms (± 1.32%) ~ 77ms 80ms p=1.000 n=6
xstateTSServer - node (v18.10.0, x64)
Req 1 - updateOpen 3,088ms (± 0.53%) 3,081ms (± 0.52%) ~ 3,066ms 3,107ms p=0.520 n=6
Req 2 - geterr 1,602ms (± 0.57%) 1,598ms (± 0.86%) ~ 1,579ms 1,614ms p=0.748 n=6
Req 3 - references 114ms (± 1.32%) 115ms (± 2.01%) ~ 112ms 118ms p=0.217 n=6
Req 4 - navto 369ms (± 0.37%) 368ms (± 0.34%) ~ 366ms 369ms p=0.096 n=6
Req 5 - completionInfo count 2,872 (± 0.00%) 2,872 (± 0.00%) ~ 2,872 2,872 p=1.000 n=6
Req 5 - completionInfo 383ms (± 0.98%) 375ms (± 1.01%) -8ms (- 1.96%) 372ms 380ms p=0.013 n=6
Compiler-UnionsTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 2,629ms (± 0.40%) 2,615ms (± 0.80%) ~ 2,591ms 2,645ms p=0.229 n=6
Req 2 - geterr 6,081ms (± 0.38%) 6,084ms (± 0.67%) ~ 6,004ms 6,116ms p=0.521 n=6
Req 3 - references 355ms (± 0.37%) 356ms (± 0.66%) ~ 353ms 360ms p=0.217 n=6
Req 4 - navto 284ms (± 0.41%) 284ms (± 0.84%) ~ 281ms 288ms p=0.934 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 83ms (± 7.49%) 81ms (± 1.22%) ~ 80ms 82ms p=0.673 n=6
CompilerTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 2,803ms (± 0.52%) 2,798ms (± 0.89%) ~ 2,760ms 2,824ms p=0.936 n=6
Req 2 - geterr 4,689ms (± 0.44%) 4,699ms (± 0.36%) ~ 4,673ms 4,723ms p=0.470 n=6
Req 3 - references 364ms (± 0.23%) 363ms (± 0.58%) ~ 361ms 366ms p=0.505 n=6
Req 4 - navto 281ms (± 0.95%) 281ms (± 0.84%) ~ 278ms 284ms p=1.000 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 77ms (± 1.09%) 76ms (± 1.75%) ~ 75ms 78ms p=0.342 n=6
xstateTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 3,230ms (± 0.30%) 3,221ms (± 0.24%) ~ 3,211ms 3,229ms p=0.261 n=6
Req 2 - geterr 1,733ms (± 1.20%) 1,757ms (± 1.34%) ~ 1,728ms 1,788ms p=0.093 n=6
Req 3 - references 130ms (± 7.76%) 126ms (± 6.65%) ~ 121ms 143ms p=0.216 n=6
Req 4 - navto 352ms (± 0.49%) 351ms (± 0.28%) ~ 350ms 352ms p=0.510 n=6
Req 5 - completionInfo count 2,872 (± 0.00%) 2,872 (± 0.00%) ~ 2,872 2,872 p=1.000 n=6
Req 5 - completionInfo 429ms (± 0.71%) 423ms (± 1.47%) ~ 414ms 431ms p=0.092 n=6
Compiler-UnionsTSServer - node (v14.21.3, x64)
Req 1 - updateOpen 2,771ms (± 0.36%) 2,767ms (± 0.33%) ~ 2,750ms 2,774ms p=0.688 n=6
Req 2 - geterr 6,220ms (± 0.51%) 6,188ms (± 0.65%) ~ 6,122ms 6,247ms p=0.298 n=6
Req 3 - references 361ms (± 0.51%) 364ms (± 0.51%) ~ 362ms 367ms p=0.061 n=6
Req 4 - navto 291ms (± 0.92%) 291ms (± 0.78%) ~ 289ms 295ms p=1.000 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 102ms (± 8.45%) 102ms (± 9.41%) ~ 91ms 111ms p=1.000 n=6
CompilerTSServer - node (v14.21.3, x64)
Req 1 - updateOpen 2,936ms (± 0.28%) 2,931ms (± 0.49%) ~ 2,906ms 2,943ms p=0.520 n=6
Req 2 - geterr 4,583ms (± 0.44%) 4,573ms (± 0.48%) ~ 4,535ms 4,594ms p=0.575 n=6
Req 3 - references 370ms (± 0.76%) 369ms (± 0.68%) ~ 366ms 372ms p=0.629 n=6
Req 4 - navto 299ms (± 0.46%) 298ms (± 0.25%) ~ 297ms 299ms p=0.111 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 86ms (± 8.01%) 87ms (± 7.63%) ~ 82ms 100ms p=0.366 n=6
xstateTSServer - node (v14.21.3, x64)
Req 1 - updateOpen 3,498ms (± 1.79%) 3,514ms (± 0.78%) ~ 3,480ms 3,538ms p=1.000 n=6
Req 2 - geterr 1,863ms (± 0.80%) 1,899ms (± 0.82%) +36ms (+ 1.91%) 1,876ms 1,919ms p=0.008 n=6
Req 3 - references 139ms (± 8.58%) 135ms (± 6.70%) ~ 130ms 153ms p=0.623 n=6
Req 4 - navto 391ms (± 0.77%) 386ms (± 0.16%) -5ms (- 1.15%) 385ms 387ms p=0.005 n=6
Req 5 - completionInfo count 2,872 (± 0.00%) 2,872 (± 0.00%) ~ 2,872 2,872 p=1.000 n=6
Req 5 - completionInfo 420ms (± 1.35%) 420ms (± 0.77%) ~ 415ms 424ms p=1.000 n=6
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-148-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v18.10.0, x64)
  • node (v16.17.1, x64)
  • node (v14.21.3, x64)
Scenarios
  • Compiler-UnionsTSServer - node (v18.10.0, x64)
  • Compiler-UnionsTSServer - node (v16.17.1, x64)
  • Compiler-UnionsTSServer - node (v14.21.3, x64)
  • CompilerTSServer - node (v18.10.0, x64)
  • CompilerTSServer - node (v16.17.1, x64)
  • CompilerTSServer - node (v14.21.3, x64)
  • xstateTSServer - node (v18.10.0, x64)
  • xstateTSServer - node (v16.17.1, x64)
  • xstateTSServer - node (v14.21.3, x64)
Benchmark Name Iterations
Current 55222 6
Baseline main 6

Startup

Comparison Report - main..55222
Metric main 55222 Delta Best Worst p-value
tsc-startup - node (v16.17.1, x64)
Execution time 143.04ms (± 0.25%) 142.52ms (± 0.18%) -0.52ms (- 0.37%) 141.86ms 145.01ms p=0.000 n=600
tsserver-startup - node (v16.17.1, x64)
Execution time 222.23ms (± 0.17%) 221.80ms (± 0.18%) -0.43ms (- 0.19%) 220.94ms 229.73ms p=0.000 n=600
tsserverlibrary-startup - node (v16.17.1, x64)
Execution time 223.71ms (± 0.17%) 223.26ms (± 0.17%) -0.45ms (- 0.20%) 222.31ms 228.31ms p=0.000 n=600
typescript-startup - node (v16.17.1, x64)
Execution time 206.38ms (± 0.24%) 205.27ms (± 0.16%) -1.10ms (- 0.53%) 204.37ms 209.04ms p=0.000 n=600
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-148-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v16.17.1, x64)
Scenarios
  • tsc-startup - node (v16.17.1, x64)
  • tsserver-startup - node (v16.17.1, x64)
  • tsserverlibrary-startup - node (v16.17.1, x64)
  • typescript-startup - node (v16.17.1, x64)
Benchmark Name Iterations
Current 55222 6
Baseline main 6

Developer Information:

Download Benchmark

@typescript-bot
Copy link
Collaborator

@jakebailey Here are the results of running the top-repos suite comparing main and refs/pull/55222/merge:

Everything looks good!

@typescript-bot
Copy link
Collaborator

Hey @jakebailey, it looks like the DT test run failed. Please check the log for more details.
You can check the log here.

@typescript-bot
Copy link
Collaborator

@jakebailey Here are the results of running the top-repos suite comparing main and refs/pull/55222/merge:

Everything looks good!

@jakebailey
Copy link
Member Author

7% check degradation in xstate, ouch

@typescript-bot perf test this faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented Aug 1, 2023 •
edited
Loading

Heya @jakebailey, I've started to run the abridged perf test suite on this PR at 5902018. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

Comparison Report - main..55222

Metric main 55222 Delta Best Worst p-value
Angular - node (v16.17.1, x64)
Memory used 368,218k (± 0.00%) 368,212k (± 0.00%) ~ 368,196k 368,238k p=0.298 n=6
Parse Time 3.57s (± 0.39%) 3.58s (± 0.63%) ~ 3.54s 3.60s p=0.370 n=6
Bind Time 1.19s (± 0.63%) 1.19s (± 0.34%) ~ 1.19s 1.20s p=0.389 n=6
Check Time 9.76s (± 0.72%) 9.73s (± 0.32%) ~ 9.68s 9.77s p=0.294 n=6
Emit Time 8.04s (± 0.80%) 7.99s (± 0.62%) ~ 7.92s 8.04s p=0.107 n=6
Total Time 22.55s (± 0.58%) 22.48s (± 0.36%) ~ 22.40s 22.58s p=0.520 n=6
Compiler-Unions - node (v16.17.1, x64)
Memory used 193,823k (± 0.02%) 193,817k (± 0.01%) ~ 193,777k 193,851k p=0.936 n=6
Parse Time 1.59s (± 0.96%) 1.59s (± 0.86%) ~ 1.56s 1.60s p=0.867 n=6
Bind Time 0.82s (± 0.92%) 0.82s (± 0.99%) ~ 0.82s 0.84s p=1.000 n=6
Check Time 10.16s (± 0.48%) 10.14s (± 0.76%) ~ 10.04s 10.23s p=0.628 n=6
Emit Time 2.98s (± 0.87%) 2.97s (± 0.45%) ~ 2.95s 2.98s p=0.565 n=6
Total Time 15.56s (± 0.35%) 15.52s (± 0.54%) ~ 15.39s 15.60s p=0.423 n=6
Monaco - node (v16.17.1, x64)
Memory used 347,078k (± 0.01%) 347,080k (± 0.01%) ~ 347,050k 347,102k p=0.688 n=6
Parse Time 2.77s (± 0.44%) 2.77s (± 0.63%) ~ 2.75s 2.79s p=0.369 n=6
Bind Time 1.08s (± 0.48%) 1.08s (± 0.59%) ~ 1.07s 1.09s p=0.386 n=6
Check Time 8.04s (± 0.61%) 8.01s (± 0.37%) ~ 7.96s 8.05s p=0.291 n=6
Emit Time 4.48s (± 0.56%) 4.45s (± 0.59%) ~ 4.42s 4.49s p=0.063 n=6
Total Time 16.36s (± 0.36%) 16.31s (± 0.38%) ~ 16.25s 16.42s p=0.128 n=6
TFS - node (v16.17.1, x64)
Memory used 301,118k (± 0.01%) 301,102k (± 0.01%) ~ 301,080k 301,137k p=0.199 n=6
Parse Time 2.20s (± 0.38%) 2.21s (± 0.62%) +0.02s (+ 0.84%) 2.20s 2.23s p=0.021 n=6
Bind Time 1.22s (± 1.70%) 1.22s (± 0.84%) ~ 1.21s 1.24s p=0.459 n=6
Check Time 7.34s (± 0.31%) 7.33s (± 0.22%) ~ 7.31s 7.35s p=0.252 n=6
Emit Time 4.30s (± 1.17%) 4.29s (± 0.41%) ~ 4.27s 4.32s p=0.745 n=6
Total Time 15.06s (± 0.25%) 15.06s (± 0.30%) ~ 15.00s 15.11s p=0.871 n=6
material-ui - node (v16.17.1, x64)
Memory used 481,882k (± 0.01%) 481,875k (± 0.01%) ~ 481,830k 481,901k p=0.748 n=6
Parse Time 3.26s (± 0.72%) 3.27s (± 0.45%) ~ 3.25s 3.29s p=0.287 n=6
Bind Time 0.96s (± 0.85%) 0.95s (± 0.54%) ~ 0.95s 0.96s p=0.523 n=6
Check Time 18.34s (± 0.39%) 18.35s (± 0.65%) ~ 18.20s 18.52s p=0.936 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 22.55s (± 0.36%) 22.58s (± 0.51%) ~ 22.45s 22.75s p=0.809 n=6
xstate - node (v16.17.1, x64)
Memory used 561,343k (± 0.01%) 563,176k (± 0.04%) +1,833k (+ 0.33%) 562,997k 563,455k p=0.005 n=6
Parse Time 4.02s (± 0.30%) 4.02s (± 0.30%) ~ 4.01s 4.04s p=1.000 n=6
Bind Time 1.66s (± 5.88%) 1.72s (± 5.38%) ~ 1.54s 1.80s p=0.370 n=6
Check Time 3.09s (± 1.60%) 3.31s (± 2.99%) +0.22s (+ 7.06%) 3.25s 3.51s p=0.005 n=6
Emit Time 0.09s (± 5.53%) 0.09s (± 0.00%) ~ 0.09s 0.09s p=0.174 n=6
Total Time 8.87s (± 0.58%) 9.15s (± 0.21%) +0.28s (+ 3.16%) 9.12s 9.17s p=0.005 n=6
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-148-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v16.17.1, x64)
Scenarios
  • Angular - node (v16.17.1, x64)
  • Compiler-Unions - node (v16.17.1, x64)
  • Monaco - node (v16.17.1, x64)
  • TFS - node (v16.17.1, x64)
  • material-ui - node (v16.17.1, x64)
  • xstate - node (v16.17.1, x64)
Benchmark Name Iterations
Current 55222 6
Baseline main 6

Developer Information:

Download Benchmark

@jakebailey
Copy link
Member Author

This PR failed RWC but the diffs aren't correctly being pushed. Need to figure that out.

@jakebailey
Copy link
Member Author

@typescript-bot perf test this faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented Aug 15, 2023 •
edited
Loading

Heya @jakebailey, I've started to run the tsc-only perf test suite on this PR at 5902018. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v16.17.1, x64)
Memory used 300,307k (± 0.01%) 300,336k (± 0.00%) +30k (+ 0.01%) 300,312k 300,349k p=0.020 n=6
Parse Time 3.02s (± 0.13%) 3.03s (± 0.13%) +0.01s (+ 0.22%) 3.02s 3.03s p=0.034 n=6
Bind Time 0.93s (± 0.00%) 0.93s (± 0.00%) ~ 0.93s 0.93s p=1.000 n=6
Check Time 9.46s (± 0.26%) 9.45s (± 0.20%) ~ 9.43s 9.47s p=0.188 n=6
Emit Time 7.62s (± 0.29%) 7.63s (± 0.41%) ~ 7.59s 7.68s p=1.000 n=6
Total Time 21.04s (± 0.12%) 21.03s (± 0.15%) ~ 21.00s 21.07s p=0.935 n=6
Compiler-Unions - node (v16.17.1, x64)
Memory used 193,822k (± 0.01%) 193,830k (± 0.01%) ~ 193,797k 193,859k p=0.810 n=6
Parse Time 1.57s (± 1.05%) 1.56s (± 1.05%) ~ 1.55s 1.59s p=0.441 n=6
Bind Time 0.79s (± 0.00%) 0.79s (± 0.00%) ~ 0.79s 0.79s p=1.000 n=6
Check Time 9.90s (± 0.21%) 9.86s (± 0.17%) -0.04s (- 0.37%) 9.84s 9.88s p=0.012 n=6
Emit Time 2.75s (± 0.19%) 2.74s (± 0.38%) ~ 2.73s 2.76s p=0.437 n=6
Total Time 15.01s (± 0.21%) 14.95s (± 0.18%) -0.06s (- 0.38%) 14.91s 14.99s p=0.013 n=6
Monaco - node (v16.17.1, x64)
Memory used 347,109k (± 0.01%) 347,102k (± 0.01%) ~ 347,081k 347,129k p=0.575 n=6
Parse Time 2.68s (± 0.30%) 2.69s (± 0.39%) ~ 2.67s 2.70s p=0.865 n=6
Bind Time 0.99s (± 0.00%) 0.99s (± 0.00%) ~ 0.99s 0.99s p=1.000 n=6
Check Time 7.90s (± 0.38%) 7.91s (± 0.13%) ~ 7.89s 7.92s p=0.806 n=6
Emit Time 4.25s (± 0.28%) 4.26s (± 0.38%) ~ 4.23s 4.28s p=0.549 n=6
Total Time 15.83s (± 0.19%) 15.83s (± 0.17%) ~ 15.80s 15.88s p=0.570 n=6
TFS - node (v16.17.1, x64)
Memory used 301,119k (± 0.01%) 301,119k (± 0.00%) ~ 301,101k 301,135k p=1.000 n=6
Parse Time 2.18s (± 0.61%) 2.18s (± 0.56%) ~ 2.16s 2.19s p=0.863 n=6
Bind Time 1.11s (± 1.51%) 1.11s (± 0.57%) ~ 1.10s 1.12s p=0.720 n=6
Check Time 7.22s (± 0.40%) 7.22s (± 0.32%) ~ 7.19s 7.26s p=1.000 n=6
Emit Time 3.97s (± 0.23%) 3.97s (± 0.36%) ~ 3.95s 3.99s p=1.000 n=6
Total Time 14.48s (± 0.33%) 14.48s (± 0.13%) ~ 14.46s 14.51s p=0.687 n=6
material-ui - node (v16.17.1, x64)
Memory used 479,389k (± 0.01%) 479,417k (± 0.01%) ~ 479,335k 479,503k p=0.575 n=6
Parse Time 3.15s (± 0.24%) 3.15s (± 0.13%) ~ 3.15s 3.16s p=0.389 n=6
Bind Time 0.91s (± 0.45%) 0.91s (± 0.00%) ~ 0.91s 0.91s p=0.405 n=6
Check Time 17.90s (± 0.25%) 17.92s (± 0.33%) ~ 17.85s 18.00s p=0.872 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 21.96s (± 0.21%) 21.98s (± 0.25%) ~ 21.91s 22.05s p=0.748 n=6
xstate - node (v16.17.1, x64)
Memory used 542,764k (± 0.01%) 544,461k (± 0.01%) +1,696k (+ 0.31%) 544,400k 544,557k p=0.005 n=6
Parse Time 3.70s (± 0.17%) 3.70s (± 0.17%) ~ 3.69s 3.71s p=1.000 n=6
Bind Time 1.38s (± 4.67%) 1.40s (± 4.45%) ~ 1.34s 1.46s p=1.000 n=6
Check Time 3.30s (± 2.68%) 3.41s (± 2.21%) ~ 3.33s 3.49s p=0.128 n=6
Emit Time 0.08s (± 0.00%) 0.08s (± 6.19%) ~ 0.08s 0.09s p=0.174 n=6
Total Time 8.46s (± 0.38%) 8.58s (± 0.23%) +0.13s (+ 1.48%) 8.55s 8.60s p=0.005 n=6
System info unknown
Hosts
  • node (v16.17.1, x64)
Scenarios
  • Angular - node (v16.17.1, x64)
  • Compiler-Unions - node (v16.17.1, x64)
  • Monaco - node (v16.17.1, x64)
  • TFS - node (v16.17.1, x64)
  • material-ui - node (v16.17.1, x64)
  • xstate - node (v16.17.1, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

@typescript-bot perf test this faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented Sep 7, 2023 •
edited
Loading

Heya @jakebailey, I've started to run the tsc-only perf test suite on this PR at 2ef27bc. You can monitor the build here.

Update: The results are in!

@jakebailey
Copy link
Member Author

@typescript-bot perf test this bun

@typescript-bot
Copy link
Collaborator

typescript-bot commented Sep 7, 2023 •
edited
Loading

Heya @jakebailey, I've started to run the bun perf test suite on this PR at 2ef27bc. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v16.17.1, x64)
Memory used 300,274k (± 0.01%) 300,272k (± 0.01%) ~ 300,253k 300,290k p=0.688 n=6
Parse Time 3.01s (± 0.18%) 3.01s (± 0.25%) ~ 3.00s 3.02s p=0.137 n=6
Bind Time 0.93s (± 0.00%) 0.93s (± 0.00%) ~ 0.93s 0.93s p=1.000 n=6
Check Time 9.33s (± 0.29%) 9.32s (± 0.22%) ~ 9.29s 9.35s p=0.571 n=6
Emit Time 7.65s (± 0.36%) 7.62s (± 0.25%) ~ 7.59s 7.64s p=0.051 n=6
Total Time 20.92s (± 0.25%) 20.88s (± 0.10%) ~ 20.85s 20.90s p=0.333 n=6
Compiler-Unions - node (v16.17.1, x64)
Memory used 194,329k (± 0.53%) 193,931k (± 0.02%) ~ 193,869k 193,985k p=0.936 n=6
Parse Time 1.58s (± 0.00%) 1.58s (± 0.26%) ~ 1.58s 1.59s p=0.405 n=6
Bind Time 0.79s (± 0.69%) 0.79s (± 0.65%) ~ 0.79s 0.80s p=0.640 n=6
Check Time 9.92s (± 0.58%) 9.93s (± 0.38%) ~ 9.88s 9.98s p=0.517 n=6
Emit Time 2.74s (± 0.15%) 2.74s (± 0.19%) ~ 2.73s 2.74s p=0.114 n=6
Total Time 15.03s (± 0.37%) 15.04s (± 0.27%) ~ 14.99s 15.10s p=0.808 n=6
Monaco - node (v16.17.1, x64)
Memory used 347,180k (± 0.01%) 347,170k (± 0.01%) ~ 347,144k 347,208k p=0.378 n=6
Parse Time 2.68s (± 0.30%) 2.69s (± 0.45%) ~ 2.68s 2.71s p=0.183 n=6
Bind Time 0.99s (± 0.52%) 0.99s (± 0.41%) ~ 0.99s 1.00s p=0.114 n=6
Check Time 7.93s (± 0.50%) 7.94s (± 0.32%) ~ 7.91s 7.97s p=0.936 n=6
Emit Time 4.26s (± 0.45%) 4.25s (± 0.38%) ~ 4.23s 4.27s p=0.417 n=6
Total Time 15.87s (± 0.32%) 15.88s (± 0.22%) ~ 15.83s 15.91s p=0.809 n=6
TFS - node (v16.17.1, x64)
Memory used 301,175k (± 0.00%) 301,170k (± 0.00%) ~ 301,151k 301,187k p=0.332 n=6
Parse Time 2.17s (± 0.54%) 2.18s (± 0.29%) ~ 2.17s 2.19s p=0.177 n=6
Bind Time 1.11s (± 0.46%) 1.12s (± 0.49%) ~ 1.11s 1.12s p=0.640 n=6
Check Time 7.22s (± 0.26%) 7.24s (± 0.31%) ~ 7.20s 7.26s p=0.219 n=6
Emit Time 3.99s (± 0.37%) 3.99s (± 0.20%) ~ 3.98s 4.00s p=1.000 n=6
Total Time 14.50s (± 0.21%) 14.51s (± 0.17%) ~ 14.47s 14.54s p=0.466 n=6
material-ui - node (v16.17.1, x64)
Memory used 479,468k (± 0.00%) 479,480k (± 0.00%) +13k (+ 0.00%) 479,467k 479,496k p=0.037 n=6
Parse Time 3.15s (± 0.17%) 3.15s (± 0.20%) ~ 3.14s 3.16s p=0.201 n=6
Bind Time 0.91s (± 0.00%) 0.91s (± 0.00%) ~ 0.91s 0.91s p=1.000 n=6
Check Time 17.78s (± 0.44%) 17.78s (± 0.44%) ~ 17.70s 17.92s p=0.873 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 21.83s (± 0.36%) 21.84s (± 0.38%) ~ 21.75s 21.98s p=0.936 n=6
xstate - node (v16.17.1, x64)
Memory used 542,825k (± 0.01%) 544,491k (± 0.01%) +1,666k (+ 0.31%) 544,408k 544,591k p=0.005 n=6
Parse Time 3.68s (± 0.28%) 3.69s (± 0.22%) ~ 3.68s 3.70s p=0.121 n=6
Bind Time 1.46s (± 5.21%) 1.43s (± 3.26%) ~ 1.34s 1.46s p=0.351 n=6
Check Time 3.19s (± 1.85%) 3.37s (± 1.85%) +0.18s (+ 5.48%) 3.32s 3.49s p=0.005 n=6
Emit Time 0.09s (± 5.95%) 0.08s (± 0.00%) 🟩-0.01s (- 7.69%) 0.08s 0.08s p=0.025 n=6
Total Time 8.42s (± 0.62%) 8.58s (± 0.25%) +0.16s (+ 1.90%) 8.55s 8.61s p=0.005 n=6
System info unknown
Hosts
  • node (v16.17.1, x64)
Scenarios
  • Angular - node (v16.17.1, x64)
  • Compiler-Unions - node (v16.17.1, x64)
  • Monaco - node (v16.17.1, x64)
  • TFS - node (v16.17.1, x64)
  • material-ui - node (v16.17.1, x64)
  • xstate - node (v16.17.1, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

Seems like quite a few percent in xstate. Very odd but I'll have to profile it I guess.

@jakebailey jakebailey marked this pull request as draft September 7, 2023 04:18
@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - bun (v0.7.3, x64)
Memory used 322,350k (± 0.13%) 322,024k (± 0.20%) ~ 320,828k 324,075k p=0.260 n=12
Parse Time 2.32s (± 0.68%) 2.31s (± 0.43%) ~ 2.29s 2.35s p=0.349 n=12
Bind Time 0.96s (± 2.77%) 0.94s (± 3.17%) ~ 0.87s 1.00s p=0.155 n=12
Check Time 7.95s (± 1.26%) 7.97s (± 1.18%) ~ 7.79s 8.15s p=0.885 n=12
Emit Time 6.42s (± 0.47%) 6.43s (± 0.41%) ~ 6.36s 6.49s p=0.977 n=12
Total Time 17.66s (± 0.56%) 17.65s (± 0.39%) ~ 17.47s 17.84s p=0.582 n=12
Compiler-Unions - bun (v0.7.3, x64)
Memory used 249,574k (± 2.27%) 247,784k (± 2.19%) ~ 239,219k 260,599k p=0.312 n=12
Parse Time 1.27s (± 1.31%) 1.26s (± 0.38%) ~ 1.25s 1.27s p=0.561 n=12
Bind Time 0.86s (± 0.66%) 0.84s (± 0.59%) -0.01s (- 1.27%) 0.84s 0.86s p=0.007 n=12
Check Time 8.94s (± 0.43%) 8.91s (± 0.33%) ~ 8.84s 9.00s p=0.338 n=12
Emit Time 2.77s (± 0.73%) 2.78s (± 0.95%) ~ 2.72s 2.85s p=0.503 n=12
Total Time 13.83s (± 0.35%) 13.80s (± 0.24%) ~ 13.69s 13.88s p=0.323 n=12
Monaco - bun (v0.7.3, x64)
Memory used 370,390k (± 0.22%) 369,843k (± 0.16%) ~ 368,961k 372,222k p=0.371 n=12
Parse Time 2.06s (± 0.20%) 2.06s (± 0.19%) ~ 2.05s 2.07s p=0.745 n=12
Bind Time 1.10s (± 1.54%) 1.10s (± 1.80%) ~ 1.06s 1.18s p=0.793 n=12
Check Time 7.17s (± 0.47%) 7.19s (± 0.45%) ~ 7.11s 7.29s p=0.183 n=12
Emit Time 3.76s (± 0.69%) 3.77s (± 0.87%) ~ 3.66s 3.85s p=0.505 n=12
Total Time 14.08s (± 0.22%) 14.12s (± 0.33%) ~ 14.00s 14.26s p=0.182 n=12
TFS - bun (v0.7.3, x64)
Memory used 317,171k (± 0.22%) 317,904k (± 0.17%) ~ 316,875k 319,836k p=0.061 n=12
Parse Time 1.80s (± 0.61%) 1.80s (± 0.31%) ~ 1.79s 1.82s p=0.558 n=12
Bind Time 0.99s (± 1.45%) 0.99s (± 1.16%) ~ 0.95s 1.02s p=0.953 n=12
Check Time 6.66s (± 0.99%) 6.64s (± 0.31%) ~ 6.59s 6.69s p=0.907 n=12
Emit Time 3.48s (± 0.74%) 3.46s (± 0.65%) ~ 3.42s 3.53s p=0.414 n=12
Total Time 12.93s (± 0.46%) 12.89s (± 0.30%) ~ 12.79s 13.00s p=0.384 n=12
material-ui - bun (v0.7.3, x64)
Memory used 526,052k (± 2.63%) 515,435k (± 3.32%) ~ 483,086k 538,997k p=0.214 n=12
Parse Time 2.48s (± 0.58%) 2.48s (± 0.47%) ~ 2.45s 2.50s p=0.906 n=12
Bind Time 0.72s (± 1.56%) 0.72s (± 1.45%) ~ 0.71s 0.76s p=0.430 n=12
Check Time 15.71s (± 0.32%) 15.69s (± 0.58%) ~ 15.51s 15.92s p=0.544 n=12
Emit Time 0.00s (±146.91%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=0.166 n=12
Total Time 18.92s (± 0.27%) 18.89s (± 0.50%) ~ 18.71s 19.11s p=0.435 n=12
xstate - bun (v0.7.3, x64)
Memory used 427,208k (± 5.78%) 490,279k (± 4.80%) +63,070k (+14.76%) 434,746k 518,961k p=0.000 n=12
Parse Time 3.58s (± 0.38%) 3.58s (± 0.38%) ~ 3.54s 3.60s p=0.839 n=12
Bind Time 1.39s (± 0.86%) 1.39s (± 1.34%) ~ 1.33s 1.43s p=0.930 n=12
Check Time 3.56s (± 0.48%) 3.65s (± 0.64%) +0.09s (+ 2.41%) 3.59s 3.69s p=0.000 n=12
Emit Time 0.22s (± 5.36%) 0.22s (± 5.62%) ~ 0.19s 0.26s p=0.190 n=12
Total Time 8.75s (± 0.31%) 8.83s (± 0.31%) +0.07s (+ 0.81%) 8.73s 8.88s p=0.001 n=12
System info unknown
Hosts
  • bun (v0.7.3, x64)
Scenarios
  • Angular - bun (v0.7.3, x64)
  • Compiler-Unions - bun (v0.7.3, x64)
  • Monaco - bun (v0.7.3, x64)
  • TFS - bun (v0.7.3, x64)
  • material-ui - bun (v0.7.3, x64)
  • xstate - bun (v0.7.3, x64)
Benchmark Name Iterations
Current pr 12
Baseline baseline 12

Startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - bun (v0.7.3, x64)
Execution time 426.05ms (± 0.08%) 425.47ms (± 0.06%) -0.58ms (- 0.14%) 424.15ms 434.03ms p=0.000 n=1200
tsserverlibrary-startup - bun (v0.7.3, x64)
Execution time 662.87ms (± 0.09%) 662.33ms (± 0.09%) -0.54ms (- 0.08%) 660.49ms 675.69ms p=0.000 n=1200
typescript-startup - bun (v0.7.3, x64)
Execution time 663.68ms (± 0.09%) 663.53ms (± 0.10%) -0.16ms (- 0.02%) 661.48ms 678.53ms p=0.000 n=1200
System info unknown
Hosts
  • bun (v0.7.3, x64)
Scenarios
  • tsc-startup - bun (v0.7.3, x64)
  • tsserverlibrary-startup - bun (v0.7.3, x64)
  • typescript-startup - bun (v0.7.3, x64)
Benchmark Name Iterations
Current pr 12
Baseline baseline 12

Developer Information:

Download Benchmarks

@weswigham
Copy link
Member

this check is what causes things to differ between errors and non-error modes in relation, causing oddities when the same relation is observed in both ways but in different orders.

It's been awhile, but I'll chime in, since I've been working on variance stuff so much: the variance result and non-variance result for a comparison are supposed to be the same. If they're not, many strange things happen. Like this. It's why in and out annotations need to be accurate (and why us allowing them to be "stricter" than the structural check still doesn't sit right with me). If we find cases where they're not the same, we really need to be tracking down and fixing those. This should be safe to do, and the fallthrough it enables is supposed to have the effect of giving structural errors even when it's a variance comparison that fails - a much more useful error, and the original intent of the check.

@jakebailey
Copy link
Member Author

Just to be clear, are you saying this PR is "good", in that it removes a place where we can produce differing variance results? Or am I misunderstanding?

(Generally I want to make progress on this PR though I have not figured out the perf problem yet...)

@jakebailey
Copy link
Member Author

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jan 19, 2024 •
edited
Loading

Heya @jakebailey, I've started to run the regular perf test suite on this PR at 26f1441. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,651k (± 0.00%) 295,675k (± 0.01%) ~ 295,650k 295,734k p=0.230 n=6
Parse Time 2.66s (± 0.34%) 2.67s (± 0.21%) ~ 2.66s 2.67s p=0.341 n=6
Bind Time 0.83s (± 0.99%) 0.83s (± 0.99%) ~ 0.82s 0.84s p=1.000 n=6
Check Time 8.20s (± 0.38%) 8.19s (± 0.49%) ~ 8.13s 8.24s p=0.871 n=6
Emit Time 7.10s (± 0.19%) 7.11s (± 0.43%) ~ 7.08s 7.16s p=0.935 n=6
Total Time 18.79s (± 0.23%) 18.80s (± 0.18%) ~ 18.74s 18.84s p=0.936 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 194,910k (± 1.48%) 193,494k (± 1.56%) ~ 191,517k 197,414k p=0.471 n=6
Parse Time 1.37s (± 1.26%) 1.37s (± 1.19%) ~ 1.34s 1.39s p=0.935 n=6
Bind Time 0.72s (± 0.00%) 0.72s (± 0.00%) ~ 0.72s 0.72s p=1.000 n=6
Check Time 9.38s (± 0.31%) 9.38s (± 0.30%) ~ 9.35s 9.42s p=1.000 n=6
Emit Time 2.62s (± 0.61%) 2.61s (± 0.79%) ~ 2.59s 2.64s p=0.685 n=6
Total Time 14.09s (± 0.26%) 14.08s (± 0.23%) ~ 14.04s 14.12s p=0.686 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,474k (± 0.00%) 347,463k (± 0.00%) ~ 347,447k 347,491k p=0.149 n=6
Parse Time 2.48s (± 0.90%) 2.49s (± 0.55%) ~ 2.46s 2.50s p=0.932 n=6
Bind Time 0.92s (± 0.59%) 0.93s (± 0.56%) ~ 0.92s 0.93s p=0.640 n=6
Check Time 6.91s (± 0.27%) 6.90s (± 0.60%) ~ 6.86s 6.97s p=0.685 n=6
Emit Time 4.06s (± 0.29%) 4.05s (± 0.30%) ~ 4.04s 4.07s p=0.323 n=6
Total Time 14.38s (± 0.25%) 14.37s (± 0.32%) ~ 14.31s 14.45s p=0.455 n=6
TFS - node (v18.15.0, x64)
Memory used 302,846k (± 0.01%) 302,825k (± 0.01%) ~ 302,805k 302,848k p=0.147 n=6
Parse Time 2.00s (± 0.93%) 2.02s (± 0.73%) ~ 2.00s 2.04s p=0.220 n=6
Bind Time 1.00s (± 1.60%) 1.01s (± 1.75%) ~ 0.99s 1.04s p=0.612 n=6
Check Time 6.33s (± 0.51%) 6.33s (± 0.52%) ~ 6.27s 6.37s p=1.000 n=6
Emit Time 3.60s (± 0.57%) 3.60s (± 0.45%) ~ 3.57s 3.62s p=0.805 n=6
Total Time 12.93s (± 0.44%) 12.95s (± 0.13%) ~ 12.92s 12.97s p=1.000 n=6
material-ui - node (v18.15.0, x64)
Memory used 511,314k (± 0.01%) 511,307k (± 0.01%) ~ 511,266k 511,376k p=0.689 n=6
Parse Time 2.66s (± 0.28%) 2.65s (± 0.50%) ~ 2.63s 2.66s p=0.230 n=6
Bind Time 1.00s (± 0.75%) 0.99s (± 0.55%) ~ 0.99s 1.00s p=0.137 n=6
Check Time 17.23s (± 0.50%) 17.26s (± 0.48%) ~ 17.11s 17.35s p=0.748 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 20.89s (± 0.43%) 20.90s (± 0.45%) ~ 20.73s 21.00s p=0.936 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,695,955k (± 0.00%) 1,696,883k (± 0.00%) +928k (+ 0.05%) 1,696,860k 1,696,931k p=0.005 n=6
Parse Time 6.53s (± 0.36%) 6.53s (± 0.41%) ~ 6.50s 6.58s p=1.000 n=6
Bind Time 2.35s (± 0.44%) 2.35s (± 0.32%) ~ 2.34s 2.36s p=0.931 n=6
Check Time 55.33s (± 0.43%) 55.52s (± 0.52%) ~ 55.12s 55.86s p=0.471 n=6
Emit Time 0.16s (± 0.00%) 0.16s (± 3.16%) ~ 0.16s 0.17s p=0.174 n=6
Total Time 64.37s (± 0.38%) 64.56s (± 0.47%) ~ 64.12s 64.91s p=0.378 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,413,013k (± 0.04%) 2,413,092k (± 0.01%) ~ 2,412,737k 2,413,622k p=1.000 n=6
Parse Time 4.94s (± 1.06%) 4.93s (± 1.24%) ~ 4.86s 5.01s p=0.748 n=6
Bind Time 1.87s (± 0.71%) 1.88s (± 1.10%) ~ 1.85s 1.91s p=0.744 n=6
Check Time 33.31s (± 0.25%) 33.19s (± 0.37%) ~ 33.06s 33.34s p=0.092 n=6
Emit Time 2.74s (± 1.54%) 2.69s (± 1.16%) ~ 2.67s 2.75s p=0.086 n=6
Total Time 42.89s (± 0.21%) 42.71s (± 0.35%) ~ 42.57s 42.90s p=0.065 n=6
self-compiler - node (v18.15.0, x64)
Memory used 419,758k (± 0.02%) 419,704k (± 0.00%) ~ 419,679k 419,736k p=0.378 n=6
Parse Time 2.73s (± 3.24%) 2.77s (± 2.56%) ~ 2.67s 2.83s p=0.466 n=6
Bind Time 1.14s (± 6.81%) 1.12s (± 6.38%) ~ 1.07s 1.22s p=0.864 n=6
Check Time 15.11s (± 0.35%) 15.12s (± 0.24%) ~ 15.05s 15.16s p=0.809 n=6
Emit Time 1.16s (± 1.05%) 1.16s (± 1.27%) ~ 1.14s 1.18s p=0.454 n=6
Total Time 20.14s (± 0.22%) 20.16s (± 0.25%) ~ 20.08s 20.23s p=0.630 n=6
vscode - node (v18.15.0, x64)
Memory used 2,807,295k (± 0.00%) 2,818,212k (± 0.00%) +10,918k (+ 0.39%) 2,818,179k 2,818,233k p=0.005 n=6
Parse Time 10.67s (± 0.53%) 10.60s (± 0.32%) ~ 10.56s 10.66s p=0.076 n=6
Bind Time 3.37s (± 0.24%) 3.39s (± 0.58%) ~ 3.37s 3.42s p=0.271 n=6
Check Time 59.67s (± 0.40%) 59.80s (± 0.45%) ~ 59.44s 60.20s p=0.378 n=6
Emit Time 16.18s (± 0.63%) 16.18s (± 0.45%) ~ 16.09s 16.28s p=1.000 n=6
Total Time 89.89s (± 0.32%) 89.97s (± 0.36%) ~ 89.59s 90.45s p=0.810 n=6
webpack - node (v18.15.0, x64)
Memory used 392,381k (± 0.02%) 392,410k (± 0.02%) ~ 392,326k 392,561k p=0.471 n=6
Parse Time 3.05s (± 0.72%) 3.05s (± 0.58%) ~ 3.03s 3.07s p=0.663 n=6
Bind Time 1.39s (± 0.60%) 1.40s (± 0.64%) ~ 1.39s 1.41s p=0.437 n=6
Check Time 13.98s (± 0.25%) 14.01s (± 0.49%) ~ 13.89s 14.06s p=0.226 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 18.43s (± 0.20%) 18.47s (± 0.47%) ~ 18.31s 18.55s p=0.229 n=6
xstate - node (v18.15.0, x64)
Memory used 513,431k (± 0.01%) 515,491k (± 0.02%) +2,060k (+ 0.40%) 515,378k 515,586k p=0.005 n=6
Parse Time 3.29s (± 0.16%) 3.28s (± 0.25%) -0.01s (- 0.30%) 3.27s 3.29s p=0.050 n=6
Bind Time 1.55s (± 0.49%) 1.54s (± 0.26%) ~ 1.54s 1.55s p=0.100 n=6
Check Time 2.86s (± 0.53%) 3.00s (± 0.65%) 🔻+0.14s (+ 4.96%) 2.98s 3.03s p=0.005 n=6
Emit Time 0.08s (± 0.00%) 0.08s (± 5.21%) ~ 0.07s 0.08s p=0.405 n=6
Total Time 7.77s (± 0.18%) 7.90s (± 0.25%) +0.13s (+ 1.67%) 7.88s 7.93s p=0.005 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,357ms (± 0.45%) 2,348ms (± 0.21%) ~ 2,343ms 2,356ms p=0.127 n=6
Req 2 - geterr 5,485ms (± 0.92%) 5,472ms (± 1.28%) ~ 5,436ms 5,614ms p=0.093 n=6
Req 3 - references 323ms (± 0.77%) 325ms (± 1.02%) ~ 321ms 330ms p=0.294 n=6
Req 4 - navto 277ms (± 1.03%) 277ms (± 1.09%) ~ 271ms 279ms p=1.000 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 85ms (± 5.07%) 85ms (± 5.43%) ~ 79ms 93ms p=0.720 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,478ms (± 0.85%) 2,484ms (± 0.94%) ~ 2,446ms 2,507ms p=0.630 n=6
Req 2 - geterr 4,153ms (± 1.38%) 4,138ms (± 1.52%) ~ 4,095ms 4,264ms p=0.378 n=6
Req 3 - references 332ms (± 1.56%) 340ms (± 1.49%) +8ms (+ 2.51%) 330ms 343ms p=0.032 n=6
Req 4 - navto 286ms (± 0.37%) 286ms (± 0.48%) ~ 283ms 287ms p=0.867 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 77ms (± 7.66%) 88ms (± 6.04%) 🔻+11ms (+13.82%) 77ms 90ms p=0.007 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,609ms (± 0.44%) 2,611ms (± 0.44%) ~ 2,590ms 2,623ms p=0.810 n=6
Req 2 - geterr 1,732ms (± 2.47%) 1,729ms (± 2.60%) ~ 1,677ms 1,786ms p=0.936 n=6
Req 3 - references 116ms (± 9.16%) 117ms (± 9.25%) ~ 106ms 127ms p=0.743 n=6
Req 4 - navto 371ms (± 0.49%) 372ms (± 0.50%) ~ 369ms 374ms p=0.566 n=6
Req 5 - completionInfo count 2,078 (± 0.00%) 2,078 (± 0.00%) ~ 2,078 2,078 p=1.000 n=6
Req 5 - completionInfo 310ms (± 1.46%) 308ms (± 2.32%) ~ 298ms 315ms p=1.000 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstateTSServer - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 154.11ms (± 0.21%) 153.96ms (± 0.22%) -0.15ms (- 0.09%) 152.88ms 158.70ms p=0.000 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 229.82ms (± 0.14%) 229.83ms (± 0.18%) ~ 228.58ms 237.12ms p=0.246 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 231.55ms (± 0.21%) 231.46ms (± 0.19%) ~ 229.79ms 237.69ms p=0.135 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 231.41ms (± 0.18%) 231.45ms (± 0.18%) ~ 229.84ms 235.95ms p=0.377 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 21, 2024 •
edited
Loading

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,628k (± 0.01%) 295,622k (± 0.01%) ~ 295,600k 295,669k p=0.689 n=6
Parse Time 2.67s (± 0.21%) 2.67s (± 0.28%) ~ 2.66s 2.68s p=0.476 n=6
Bind Time 0.83s (± 0.00%) 0.83s (± 1.24%) ~ 0.82s 0.85s p=0.599 n=6
Check Time 8.21s (± 0.35%) 8.21s (± 0.22%) ~ 8.18s 8.23s p=0.685 n=6
Emit Time 7.08s (± 0.74%) 7.04s (± 0.37%) ~ 7.01s 7.08s p=0.171 n=6
Total Time 18.79s (± 0.28%) 18.75s (± 0.28%) ~ 18.68s 18.83s p=0.518 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 193,063k (± 0.92%) 192,593k (± 0.77%) ~ 191,857k 195,606k p=0.936 n=6
Parse Time 1.37s (± 0.85%) 1.36s (± 0.30%) ~ 1.35s 1.36s p=0.102 n=6
Bind Time 0.72s (± 0.00%) 0.72s (± 0.57%) ~ 0.72s 0.73s p=0.405 n=6
Check Time 9.46s (± 0.36%) 9.50s (± 0.65%) ~ 9.43s 9.58s p=0.376 n=6
Emit Time 2.64s (± 0.46%) 2.64s (± 0.52%) ~ 2.62s 2.65s p=0.798 n=6
Total Time 14.18s (± 0.21%) 14.21s (± 0.42%) ~ 14.16s 14.30s p=0.469 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,396k (± 0.01%) 347,360k (± 0.00%) -37k (- 0.01%) 347,347k 347,374k p=0.013 n=6
Parse Time 2.48s (± 0.61%) 2.48s (± 0.56%) ~ 2.45s 2.49s p=0.243 n=6
Bind Time 0.93s (± 0.56%) 0.93s (± 0.44%) ~ 0.92s 0.93s p=0.595 n=6
Check Time 7.03s (± 0.49%) 7.02s (± 0.32%) ~ 6.98s 7.05s p=0.461 n=6
Emit Time 4.07s (± 0.53%) 4.07s (± 0.42%) ~ 4.05s 4.10s p=0.935 n=6
Total Time 14.51s (± 0.16%) 14.49s (± 0.14%) ~ 14.45s 14.51s p=0.460 n=6
TFS - node (v18.15.0, x64)
Memory used 302,713k (± 0.01%) 302,726k (± 0.01%) ~ 302,695k 302,758k p=0.336 n=6
Parse Time 2.02s (± 0.68%) 2.02s (± 0.68%) ~ 2.00s 2.03s p=0.241 n=6
Bind Time 1.00s (± 0.81%) 1.00s (± 0.75%) ~ 0.99s 1.01s p=0.340 n=6
Check Time 6.32s (± 0.49%) 6.32s (± 0.47%) ~ 6.28s 6.35s p=1.000 n=6
Emit Time 3.61s (± 0.65%) 3.62s (± 0.65%) ~ 3.59s 3.64s p=0.627 n=6
Total Time 12.95s (± 0.35%) 12.95s (± 0.36%) ~ 12.87s 13.01s p=0.810 n=6
material-ui - node (v18.15.0, x64)
Memory used 509,882k (± 0.00%) 509,903k (± 0.00%) ~ 509,881k 509,924k p=0.066 n=6
Parse Time 2.66s (± 0.24%) 2.66s (± 0.34%) ~ 2.65s 2.67s p=1.000 n=6
Bind Time 0.98s (± 0.85%) 0.98s (± 1.00%) ~ 0.96s 0.99s p=0.195 n=6
Check Time 17.25s (± 0.28%) 17.27s (± 0.37%) ~ 17.19s 17.38s p=0.744 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 20.90s (± 0.23%) 20.91s (± 0.32%) ~ 20.83s 21.03s p=0.808 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,719,658k (± 0.00%) 1,721,964k (± 0.00%) +2,306k (+ 0.13%) 1,721,919k 1,721,990k p=0.005 n=6
Parse Time 6.57s (± 0.47%) 6.60s (± 0.60%) ~ 6.55s 6.67s p=0.223 n=6
Bind Time 2.35s (± 0.22%) 2.36s (± 0.32%) ~ 2.35s 2.37s p=0.247 n=6
Check Time 56.15s (± 0.39%) 56.25s (± 0.31%) ~ 56.03s 56.53s p=0.378 n=6
Emit Time 0.13s (± 3.10%) 0.13s (± 0.00%) ~ 0.13s 0.13s p=0.405 n=6
Total Time 65.20s (± 0.31%) 65.34s (± 0.25%) ~ 65.18s 65.63s p=0.298 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,392,099k (± 0.03%) 2,391,754k (± 0.03%) ~ 2,391,156k 2,392,936k p=0.471 n=6
Parse Time 5.04s (± 1.56%) 5.01s (± 0.41%) ~ 4.98s 5.04s p=0.520 n=6
Bind Time 1.88s (± 0.64%) 1.89s (± 0.47%) ~ 1.88s 1.90s p=0.362 n=6
Check Time 33.66s (± 0.43%) 33.65s (± 0.21%) ~ 33.55s 33.72s p=0.936 n=6
Emit Time 2.65s (± 1.41%) 2.65s (± 1.00%) ~ 2.62s 2.69s p=0.810 n=6
Total Time 43.26s (± 0.32%) 43.22s (± 0.16%) ~ 43.11s 43.30s p=0.810 n=6
self-compiler - node (v18.15.0, x64)
Memory used 415,107k (± 0.01%) 415,100k (± 0.00%) ~ 415,077k 415,123k p=0.936 n=6
Parse Time 2.84s (± 0.73%) 2.81s (± 0.74%) -0.03s (- 1.00%) 2.79s 2.84s p=0.043 n=6
Bind Time 1.07s (± 0.76%) 1.07s (± 0.38%) ~ 1.07s 1.08s p=0.584 n=6
Check Time 15.26s (± 0.43%) 15.23s (± 0.31%) ~ 15.15s 15.29s p=1.000 n=6
Emit Time 1.11s (± 0.99%) 1.11s (± 1.20%) ~ 1.09s 1.13s p=0.869 n=6
Total Time 20.29s (± 0.37%) 20.23s (± 0.34%) ~ 20.12s 20.31s p=0.296 n=6
vscode - node (v18.15.0, x64)
Memory used 2,887,897k (± 0.00%) 2,898,897k (± 0.00%) +11,000k (+ 0.38%) 2,898,801k 2,898,996k p=0.005 n=6
Parse Time 10.84s (± 0.22%) 10.91s (± 0.89%) ~ 10.83s 11.10s p=0.064 n=6
Bind Time 3.46s (± 0.12%) 3.47s (± 1.09%) ~ 3.45s 3.55s p=0.673 n=6
Check Time 61.55s (± 0.46%) 61.63s (± 0.43%) ~ 61.39s 62.10s p=0.630 n=6
Emit Time 17.02s (± 8.70%) 16.54s (± 0.54%) ~ 16.42s 16.64s p=0.470 n=6
Total Time 92.87s (± 1.91%) 92.56s (± 0.32%) ~ 92.12s 92.97s p=0.173 n=6
webpack - node (v18.15.0, x64)
Memory used 408,076k (± 0.01%) 407,989k (± 0.01%) -87k (- 0.02%) 407,932k 408,046k p=0.013 n=6
Parse Time 3.23s (± 0.23%) 3.23s (± 0.32%) ~ 3.22s 3.25s p=0.931 n=6
Bind Time 1.37s (± 0.80%) 1.38s (± 0.46%) ~ 1.37s 1.39s p=0.081 n=6
Check Time 14.31s (± 0.31%) 14.28s (± 0.21%) ~ 14.25s 14.32s p=0.258 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 18.91s (± 0.26%) 18.89s (± 0.19%) ~ 18.85s 18.95s p=0.520 n=6
xstate - node (v18.15.0, x64)
Memory used 513,076k (± 0.02%) 515,171k (± 0.03%) +2,094k (+ 0.41%) 515,035k 515,425k p=0.005 n=6
Parse Time 3.28s (± 0.23%) 3.28s (± 0.16%) ~ 3.28s 3.29s p=0.241 n=6
Bind Time 1.54s (± 0.33%) 1.54s (± 0.33%) ~ 1.54s 1.55s p=1.000 n=6
Check Time 2.86s (± 0.61%) 3.00s (± 0.42%) 🔻+0.14s (+ 5.02%) 2.99s 3.02s p=0.004 n=6
Emit Time 0.07s (± 5.69%) 0.07s (± 5.69%) ~ 0.07s 0.08s p=1.000 n=6
Total Time 7.75s (± 0.32%) 7.90s (± 0.15%) +0.15s (+ 1.93%) 7.89s 7.92s p=0.005 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,374ms (± 0.69%) 2,384ms (± 0.82%) ~ 2,359ms 2,409ms p=0.575 n=6
Req 2 - geterr 5,586ms (± 0.26%) 5,586ms (± 0.50%) ~ 5,540ms 5,611ms p=0.810 n=6
Req 3 - references 318ms (± 0.28%) 319ms (± 0.62%) ~ 317ms 322ms p=0.803 n=6
Req 4 - navto 273ms (± 0.00%) 272ms (± 0.60%) ~ 269ms 273ms p=0.405 n=6
Req 5 - completionInfo count 1,357 (± 0.00%) 1,357 (± 0.00%) ~ 1,357 1,357 p=1.000 n=6
Req 5 - completionInfo 87ms (± 0.60%) 88ms (± 5.38%) ~ 86ms 98ms p=0.929 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,501ms (± 0.56%) 2,502ms (± 1.03%) ~ 2,465ms 2,531ms p=1.000 n=6
Req 2 - geterr 4,126ms (± 0.30%) 4,120ms (± 0.29%) ~ 4,106ms 4,132ms p=0.470 n=6
Req 3 - references 334ms (± 0.29%) 334ms (± 0.31%) ~ 332ms 335ms p=0.673 n=6
Req 4 - navto 295ms (± 0.28%) 295ms (± 0.33%) ~ 294ms 296ms p=0.588 n=6
Req 5 - completionInfo count 1,519 (± 0.00%) 1,519 (± 0.00%) ~ 1,519 1,519 p=1.000 n=6
Req 5 - completionInfo 73ms (± 0.70%) 74ms (± 0.74%) ~ 73ms 74ms p=0.640 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,607ms (± 0.48%) 2,609ms (± 0.20%) ~ 2,600ms 2,615ms p=0.575 n=6
Req 2 - geterr 1,729ms (± 2.42%) 1,727ms (± 2.22%) ~ 1,684ms 1,773ms p=1.000 n=6
Req 3 - references 117ms (± 9.48%) 112ms (±10.41%) ~ 104ms 127ms p=0.806 n=6
Req 4 - navto 371ms (± 0.75%) 370ms (± 1.87%) ~ 362ms 383ms p=0.466 n=6
Req 5 - completionInfo count 2,079 (± 0.00%) 2,079 (± 0.00%) ~ 2,079 2,079 p=1.000 n=6
Req 5 - completionInfo 308ms (± 1.82%) 307ms (± 1.54%) ~ 302ms 313ms p=1.000 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstateTSServer - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 153.63ms (± 0.15%) 153.60ms (± 0.19%) -0.03ms (- 0.02%) 152.44ms 157.21ms p=0.005 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 227.57ms (± 0.16%) 227.43ms (± 0.15%) -0.14ms (- 0.06%) 226.00ms 232.91ms p=0.001 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 222.40ms (± 0.14%) 222.24ms (± 0.15%) -0.16ms (- 0.07%) 220.87ms 227.81ms p=0.000 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 223.08ms (± 0.21%) 223.02ms (± 0.17%) ~ 221.38ms 228.95ms p=0.665 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

@typescript-bot perf test this faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 18, 2024 •
edited
Loading

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this faster ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 296,982k (± 0.01%) 297,002k (± 0.01%) ~ 296,972k 297,050k p=0.335 n=6
Parse Time 2.69s (± 0.39%) 2.69s (± 0.39%) ~ 2.68s 2.71s p=0.160 n=6
Bind Time 0.82s (± 0.50%) 0.82s (± 0.00%) ~ 0.82s 0.82s p=0.405 n=6
Check Time 8.32s (± 0.45%) 8.31s (± 0.31%) ~ 8.28s 8.35s p=0.517 n=6
Emit Time 7.07s (± 0.39%) 7.05s (± 0.17%) ~ 7.03s 7.06s p=0.079 n=6
Total Time 18.90s (± 0.30%) 18.87s (± 0.15%) ~ 18.83s 18.91s p=0.624 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 194,757k (± 0.71%) 193,094k (± 0.90%) ~ 191,837k 195,333k p=0.298 n=6
Parse Time 1.34s (± 1.12%) 1.35s (± 0.00%) ~ 1.35s 1.35s p=0.599 n=6
Bind Time 0.72s (± 0.00%) 0.72s (± 0.57%) ~ 0.72s 0.73s p=0.405 n=6
Check Time 9.54s (± 0.38%) 9.56s (± 0.22%) ~ 9.52s 9.58s p=0.374 n=6
Emit Time 2.62s (± 0.70%) 2.64s (± 0.54%) ~ 2.62s 2.66s p=0.062 n=6
Total Time 14.23s (± 0.28%) 14.28s (± 0.22%) ~ 14.22s 14.31s p=0.061 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,751,304k (± 0.00%) 1,753,663k (± 0.00%) +2,359k (+ 0.13%) 1,753,627k 1,753,710k p=0.005 n=6
Parse Time 6.81s (± 0.27%) 6.80s (± 0.22%) ~ 6.79s 6.82s p=0.406 n=6
Bind Time 2.28s (± 0.33%) 2.29s (± 0.36%) ~ 2.28s 2.30s p=0.383 n=6
Check Time 56.68s (± 0.38%) 56.79s (± 0.33%) ~ 56.57s 57.06s p=0.810 n=6
Emit Time 0.14s (± 2.95%) 0.14s (± 0.00%) ~ 0.14s 0.14s p=0.405 n=6
Total Time 65.91s (± 0.31%) 66.02s (± 0.27%) ~ 65.82s 66.28s p=0.689 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,307,152k (± 0.02%) 2,306,698k (± 0.02%) ~ 2,306,282k 2,307,782k p=0.066 n=6
Parse Time 4.97s (± 1.10%) 4.98s (± 0.98%) ~ 4.92s 5.05s p=0.872 n=6
Bind Time 1.86s (± 1.36%) 1.87s (± 1.14%) ~ 1.85s 1.90s p=0.462 n=6
Check Time 33.77s (± 0.28%) 33.84s (± 0.25%) ~ 33.74s 33.99s p=0.199 n=6
Emit Time 2.69s (± 1.09%) 2.67s (± 1.86%) ~ 2.58s 2.73s p=0.463 n=6
Total Time 43.31s (± 0.24%) 43.38s (± 0.23%) ~ 43.30s 43.57s p=0.575 n=6
self-build-src-public-api - node (v18.15.0, x64)
Memory used 2,381,280k (± 0.03%) 2,381,452k (± 0.05%) ~ 2,379,932k 2,382,789k p=0.689 n=6
Parse Time 6.20s (± 1.06%) 6.17s (± 0.80%) ~ 6.09s 6.23s p=0.378 n=6
Bind Time 2.04s (± 0.72%) 2.03s (± 1.24%) ~ 1.99s 2.06s p=0.870 n=6
Check Time 40.17s (± 0.23%) 40.12s (± 0.29%) ~ 39.97s 40.30s p=0.471 n=6
Emit Time 3.15s (± 2.18%) 3.17s (± 1.08%) ~ 3.11s 3.21s p=0.936 n=6
Total Time 51.58s (± 0.21%) 51.49s (± 0.19%) ~ 51.42s 51.68s p=0.173 n=6
self-compiler - node (v18.15.0, x64)
Memory used 419,442k (± 0.01%) 419,485k (± 0.01%) ~ 419,442k 419,526k p=0.065 n=6
Parse Time 4.20s (± 0.53%) 4.20s (± 0.15%) ~ 4.19s 4.21s p=0.720 n=6
Bind Time 1.58s (± 2.38%) 1.59s (± 2.37%) ~ 1.53s 1.64s p=0.747 n=6
Check Time 22.32s (± 0.45%) 22.36s (± 0.62%) ~ 22.15s 22.57s p=0.575 n=6
Emit Time 1.72s (± 1.07%) 1.74s (± 1.01%) ~ 1.71s 1.76s p=0.188 n=6
Total Time 29.82s (± 0.42%) 29.89s (± 0.44%) ~ 29.72s 30.12s p=0.520 n=6
ts-pre-modules - node (v18.15.0, x64)
Memory used 368,970k (± 0.02%) 368,972k (± 0.00%) ~ 368,950k 368,993k p=0.936 n=6
Parse Time 3.65s (± 0.62%) 3.66s (± 0.88%) ~ 3.64s 3.72s p=1.000 n=6
Bind Time 1.93s (± 2.12%) 1.99s (± 0.52%) +0.06s (+ 3.28%) 1.98s 2.01s p=0.035 n=6
Check Time 19.39s (± 0.31%) 19.42s (± 0.25%) ~ 19.35s 19.47s p=0.333 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 24.96s (± 0.28%) 25.07s (± 0.23%) +0.11s (+ 0.43%) 24.99s 25.13s p=0.024 n=6
vscode - node (v18.15.0, x64)
Memory used 2,915,645k (± 0.00%) 2,926,612k (± 0.01%) +10,968k (+ 0.38%) 2,926,408k 2,926,892k p=0.005 n=6
Parse Time 13.46s (± 0.25%) 13.45s (± 0.31%) ~ 13.37s 13.48s p=0.745 n=6
Bind Time 4.07s (± 0.29%) 4.06s (± 0.29%) ~ 4.04s 4.07s p=0.666 n=6
Check Time 72.17s (± 0.15%) 72.53s (± 0.40%) +0.36s (+ 0.51%) 72.32s 73.10s p=0.010 n=6
Emit Time 21.97s (± 8.95%) 22.14s (± 9.07%) ~ 19.51s 23.53s p=0.229 n=6
Total Time 111.66s (± 1.74%) 112.18s (± 1.70%) ~ 109.37s 113.50s p=0.128 n=6
webpack - node (v18.15.0, x64)
Memory used 409,506k (± 0.02%) 409,459k (± 0.02%) ~ 409,372k 409,568k p=0.471 n=6
Parse Time 4.81s (± 0.43%) 4.84s (± 0.80%) ~ 4.78s 4.90s p=0.134 n=6
Bind Time 2.02s (± 0.67%) 2.02s (± 0.68%) ~ 2.00s 2.04s p=0.452 n=6
Check Time 20.96s (± 0.52%) 21.06s (± 0.27%) ~ 20.98s 21.12s p=0.108 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 27.79s (± 0.41%) 27.91s (± 0.31%) ~ 27.81s 28.03s p=0.128 n=6
xstate-main - node (v18.15.0, x64)
Memory used 458,986k (± 0.01%) 462,792k (± 0.01%) +3,806k (+ 0.83%) 462,715k 462,895k p=0.005 n=6
Parse Time 3.99s (± 0.49%) 4.01s (± 0.69%) ~ 3.97s 4.04s p=0.257 n=6
Bind Time 1.48s (± 1.02%) 1.47s (± 1.17%) ~ 1.45s 1.50s p=0.514 n=6
Check Time 22.22s (± 0.44%) 22.43s (± 0.48%) +0.21s (+ 0.93%) 22.27s 22.58s p=0.020 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 27.69s (± 0.34%) 27.92s (± 0.25%) +0.22s (+ 0.81%) 27.82s 28.00s p=0.005 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-build-src-public-api - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • ts-pre-modules - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate-main - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

Surely if I keep retrying it, it will eventually not break perf, right?

@typescript-bot perf test this faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jul 24, 2024 •
edited
Loading

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this faster ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-Unions - node (v18.15.0, x64)
Errors 30 30 ~ ~ ~ p=1.000 n=6
Symbols 62,153 62,153 ~ ~ ~ p=1.000 n=6
Types 50,242 50,242 ~ ~ ~ p=1.000 n=6
Memory used 192,998k (± 0.78%) 194,107k (± 1.01%) ~ 192,286k 195,927k p=0.689 n=6
Parse Time 1.30s (± 0.90%) 1.30s (± 0.69%) ~ 1.29s 1.31s p=0.933 n=6
Bind Time 0.71s 0.71s ~ ~ ~ p=1.000 n=6
Check Time 9.56s (± 0.41%) 9.56s (± 0.50%) ~ 9.49s 9.62s p=0.936 n=6
Emit Time 2.73s (± 1.97%) 2.74s (± 0.63%) ~ 2.72s 2.76s p=0.565 n=6
Total Time 14.30s (± 0.50%) 14.31s (± 0.44%) ~ 14.22s 14.40s p=0.936 n=6
angular-1 - node (v18.15.0, x64)
Errors 7 7 ~ ~ ~ p=1.000 n=6
Symbols 945,532 945,679 +147 (+ 0.02%) ~ ~ p=0.001 n=6
Types 409,507 409,555 +48 (+ 0.01%) ~ ~ p=0.001 n=6
Memory used 1,221,125k (± 0.00%) 1,221,258k (± 0.00%) +133k (+ 0.01%) 1,221,189k 1,221,306k p=0.005 n=6
Parse Time 6.63s (± 0.37%) 6.63s (± 0.85%) ~ 6.55s 6.71s p=0.809 n=6
Bind Time 1.86s (± 0.66%) 1.86s (± 0.28%) ~ 1.86s 1.87s p=0.241 n=6
Check Time 31.10s (± 0.29%) 31.24s (± 0.52%) ~ 31.08s 31.50s p=0.092 n=6
Emit Time 14.99s (± 0.55%) 15.00s (± 0.60%) ~ 14.90s 15.11s p=1.000 n=6
Total Time 54.57s (± 0.30%) 54.74s (± 0.45%) ~ 54.47s 55.06s p=0.378 n=6
mui-docs - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,324,872 2,329,028 +4,156 (+ 0.18%) ~ ~ p=0.001 n=6
Types 949,556 951,073 +1,517 (+ 0.16%) ~ ~ p=0.001 n=6
Memory used 2,222,587k (± 0.00%) 2,225,049k (± 0.00%) +2,463k (+ 0.11%) 2,224,970k 2,225,130k p=0.005 n=6
Parse Time 6.63s (± 0.40%) 6.63s (± 0.31%) ~ 6.60s 6.65s p=0.685 n=6
Bind Time 2.32s (± 0.44%) 2.33s (± 0.52%) ~ 2.31s 2.34s p=0.675 n=6
Check Time 73.08s (± 0.45%) 72.96s (± 0.36%) ~ 72.62s 73.33s p=0.689 n=6
Emit Time 0.14s (± 3.77%) 0.14s (± 6.19%) ~ 0.13s 0.15s p=0.533 n=6
Total Time 82.17s (± 0.42%) 82.05s (± 0.32%) ~ 81.70s 82.40s p=0.630 n=6
self-build-src - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,230,120 1,230,140 +20 (+ 0.00%) ~ ~ p=0.001 n=6
Types 265,641 265,674 +33 (+ 0.01%) ~ ~ p=0.001 n=6
Memory used 2,347,505k (± 0.01%) 2,466,094k (±11.78%) ~ 2,346,788k 3,059,873k p=0.810 n=6
Parse Time 5.00s (± 0.58%) 5.00s (± 0.85%) ~ 4.93s 5.04s p=0.520 n=6
Bind Time 1.89s (± 0.64%) 1.90s (± 1.28%) ~ 1.85s 1.92s p=0.405 n=6
Check Time 34.73s (± 0.52%) 34.75s (± 0.22%) ~ 34.65s 34.86s p=0.936 n=6
Emit Time 3.28s (± 1.05%) 3.29s (± 1.05%) ~ 3.24s 3.33s p=0.872 n=6
Total Time 44.91s (± 0.46%) 44.96s (± 0.17%) ~ 44.86s 45.06s p=0.575 n=6
self-build-src-public-api - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,230,120 1,230,140 +20 (+ 0.00%) ~ ~ p=0.001 n=6
Types 265,641 265,674 +33 (+ 0.01%) ~ ~ p=0.001 n=6
Memory used 2,421,504k (± 0.01%) 2,421,649k (± 0.02%) ~ 2,421,151k 2,422,181k p=0.688 n=6
Parse Time 6.24s (± 0.51%) 6.27s (± 0.83%) ~ 6.22s 6.35s p=0.297 n=6
Bind Time 2.05s (± 1.29%) 2.05s (± 0.85%) ~ 2.02s 2.07s p=1.000 n=6
Check Time 41.33s (± 0.48%) 41.43s (± 0.14%) ~ 41.34s 41.52s p=0.066 n=6
Emit Time 4.06s (± 1.64%) 4.06s (± 0.77%) ~ 4.03s 4.11s p=0.686 n=6
Total Time 53.69s (± 0.51%) 53.83s (± 0.18%) ~ 53.74s 53.99s p=0.065 n=6
self-compiler - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 258,471 258,485 +14 (+ 0.01%) ~ ~ p=0.001 n=6
Types 105,534 105,561 +27 (+ 0.03%) ~ ~ p=0.001 n=6
Memory used 429,163k (± 0.05%) 429,270k (± 0.03%) ~ 429,106k 429,451k p=0.173 n=6
Parse Time 3.34s (± 0.77%) 3.34s (± 0.23%) ~ 3.33s 3.35s p=1.000 n=6
Bind Time 1.32s (± 1.64%) 1.31s (± 1.25%) ~ 1.30s 1.33s p=0.675 n=6
Check Time 17.96s (± 0.30%) 18.02s (± 0.32%) ~ 17.93s 18.10s p=0.108 n=6
Emit Time 1.63s (± 0.72%) 1.63s (± 1.21%) ~ 1.60s 1.66s p=0.605 n=6
Total Time 24.24s (± 0.27%) 24.29s (± 0.25%) ~ 24.23s 24.40s p=0.372 n=6
ts-pre-modules - node (v18.15.0, x64)
Errors 35 35 ~ ~ ~ p=1.000 n=6
Symbols 224,931 224,931 ~ ~ ~ p=1.000 n=6
Types 94,146 94,146 ~ ~ ~ p=1.000 n=6
Memory used 369,982k (± 0.02%) 370,106k (± 0.05%) ~ 369,888k 370,327k p=0.298 n=6
Parse Time 2.76s (± 1.13%) 2.77s (± 0.35%) ~ 2.76s 2.78s p=0.461 n=6
Bind Time 1.59s (± 1.43%) 1.59s (± 1.61%) ~ 1.56s 1.62s p=0.934 n=6
Check Time 15.65s (± 0.39%) 15.61s (± 0.45%) ~ 15.54s 15.73s p=0.378 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 20.00s (± 0.26%) 19.97s (± 0.42%) ~ 19.88s 20.10s p=0.423 n=6
vscode - node (v18.15.0, x64)
Errors 11 11 ~ ~ ~ p=1.000 n=6
Symbols 2,985,953 3,011,882 +25,929 (+ 0.87%) ~ ~ p=0.001 n=6
Types 1,027,137 1,030,528 +3,391 (+ 0.33%) ~ ~ p=0.001 n=6
Memory used 3,109,968k (± 0.00%) 3,120,841k (± 0.00%) +10,873k (+ 0.35%) 3,120,792k 3,120,914k p=0.005 n=6
Parse Time 17.12s (± 0.18%) 17.14s (± 0.17%) ~ 17.11s 17.19s p=0.293 n=6
Bind Time 5.21s (± 0.23%) 5.25s (± 1.82%) ~ 5.20s 5.45s p=0.438 n=6
Check Time 96.29s (± 0.38%) 96.30s (± 0.15%) ~ 96.09s 96.47s p=0.936 n=6
Emit Time 25.08s (± 0.47%) 25.09s (± 0.81%) ~ 24.90s 25.46s p=0.873 n=6
Total Time 143.71s (± 0.27%) 143.79s (± 0.23%) ~ 143.41s 144.25s p=0.748 n=6
webpack - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 267,560 267,560 ~ ~ ~ p=1.000 n=6
Types 109,076 109,076 ~ ~ ~ p=1.000 n=6
Memory used 412,464k (± 0.02%) 412,402k (± 0.01%) ~ 412,350k 412,479k p=0.128 n=6
Parse Time 3.83s (± 0.73%) 3.83s (± 0.65%) ~ 3.79s 3.86s p=0.520 n=6
Bind Time 1.71s (± 0.49%) 1.72s (± 0.30%) ~ 1.71s 1.72s p=0.533 n=6
Check Time 16.93s (± 0.66%) 16.89s (± 0.32%) ~ 16.79s 16.93s p=0.334 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 22.47s (± 0.54%) 22.44s (± 0.20%) ~ 22.37s 22.49s p=0.378 n=6
xstate-main - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 511,803 517,597 +5,794 (+ 1.13%) ~ ~ p=0.001 n=6
Types 162,088 163,963 +1,875 (+ 1.16%) ~ ~ p=0.001 n=6
Memory used 448,992k (± 0.09%) 452,469k (± 0.02%) +3,478k (+ 0.77%) 452,360k 452,659k p=0.005 n=6
Parse Time 3.90s (± 0.44%) 3.91s (± 0.56%) ~ 3.88s 3.94s p=0.408 n=6
Bind Time 1.44s (± 0.84%) 1.44s (± 0.68%) ~ 1.43s 1.46s p=0.226 n=6
Check Time 21.27s (± 0.29%) 21.15s (± 0.40%) -0.12s (- 0.56%) 21.02s 21.23s p=0.020 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 26.62s (± 0.29%) 26.51s (± 0.41%) ~ 26.34s 26.62s p=0.065 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Compiler-Unions - node (v18.15.0, x64)
  • angular-1 - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-build-src-public-api - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • ts-pre-modules - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate-main - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@weswigham weswigham Awaiting requested review from weswigham

At least 1 approving review is required to merge this pull request.

Assignees

@jakebailey jakebailey

Labels
Author: Team Experiment A fork with an experimental idea which might not make it into master For Milestone Bug PRs that fix a bug with a specific milestone
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Conditional type triggers "No error for last overload signature" exception
4 participants
@jakebailey @typescript-bot @weswigham @sandersn