Skip to content

Commit 82539be

Browse files
bashmishbashmish
committed
chore: make new benchmarks using tachometer
1 parent 067b39d commit 82539be

10 files changed

+847
-2627
lines changed

‎.eslintrc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
"import/no-extraneous-dependencies": [
99
"error",
1010
{
11-
"devDependencies": ["src/**/*.bench.js", "src/**/*.spec.js", "*.conf.js", "*.config.mjs"]
11+
"devDependencies": ["benchmarks/**/*.tachometer.js", "src/**/*.spec.js", "*.conf.js", "*.config.mjs"]
1212
}
1313
],
1414
"import/extensions": 0
@@ -21,7 +21,7 @@
2121
}
2222
},
2323
{
24-
"files": ["src/**/*.bench.js"],
24+
"files": ["benchmarks/**/*.tachometer.js"],
2525
"globals": {
2626
"suite": true,
2727
"benchmark": true

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
benchmarks/tachometer.json
12
coverage/
23
node_modules/
34
.vscode/

‎.npmignore

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
/.vscode/
2+
/benchmarks/
23
/coverage/
3-
*.bench.js
44
*.spec.js
55
.browserslistrc
66
.commitlintrc
@@ -10,7 +10,6 @@
1010
.lintstagedrc
1111
.prettierignore
1212
.prettierrc
13-
karma.bench.conf.js
1413
results.json
1514
wallaby.js
1615
web-test-runner.bs.config.mjs

‎README.md

Lines changed: 25 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -115,61 +115,31 @@ function fixture(litTemplate) {
115115
Runtime performance is not the key requirement for `carehtml` since the end goal is to compile the code and have static and still unique tag names in the production code.
116116
But some numbers might be interesting to show the impact of such solution on local development and the potential runtime usage in production for projects that want to stay compilation-free.
117117

118-
Test configuration:
119-
120-
- MacBook Pro (15-inch, 2016)
121-
- Mac OS X 10.14.6
122-
123-
First numbers show the difference in creating a template between using `lit-html` directly, doing same via `carehtml` wrapper, and most importantly, using Custom Element classes in place of tag names.
124-
125-
| create template for lit-html | ops/sec |
126-
| --------------------------------------- | -----------: |
127-
| Chrome 79.0.3945 | |
128-
| clean html\`\<el-name>\</el-name>\` | `149575145` |
129-
| care(html)\`\<el-name>\</el-name>\` | `30728712` |
130-
| care(html)\`<${ElClass}></${ElClass}>\` | `4654638` |
131-
| Firefox 71.0.0 | |
132-
| clean html\`\<el-name>\</el-name>\` | `1268411403` |
133-
| care(html)\`\<el-name>\</el-name>\` | `7112878` |
134-
| care(html)\`<${ElClass}></${ElClass}>\` | `1551298` |
135-
| Safari 13.0.3 | |
136-
| clean html\`\<el-name>\</el-name>\` | `29071424` |
137-
| care(html)\`\<el-name>\</el-name>\` | `7572036` |
138-
| care(html)\`<${ElClass}></${ElClass}>\` | `1800394` |
139-
140-
Next numbers show the rendering times of those from above.
141-
142-
| render template for lit-html | ops/sec |
143-
| --------------------------------------- | ---------: |
144-
| Chrome 79.0.3945 | |
145-
| clean html\`\<el-name>\</el-name>\` | `20270378` |
146-
| care(html)\`\<el-name>\</el-name>\` | `20915893` |
147-
| care(html)\`<${ElClass}></${ElClass}>\` | `20682304` |
148-
| Firefox 71.0.0 | |
149-
| clean html\`\<el-name>\</el-name>\` | `3252965` |
150-
| care(html)\`\<el-name>\</el-name>\` | `3418368` |
151-
| care(html)\`<${ElClass}></${ElClass}>\` | `3228236` |
152-
| Safari 13.0.3 | |
153-
| clean html\`\<el-name>\</el-name>\` | `10930284` |
154-
| care(html)\`\<el-name>\</el-name>\` | `10873434` |
155-
| care(html)\`<${ElClass}></${ElClass}>\` | `10822978` |
156-
157-
Next numbers show the combination of both.
158-
159-
| create and render template for lit-html | ops/sec |
160-
| --------------------------------------- | ---------: |
161-
| Chrome 79.0.3945 | |
162-
| clean html\`\<el-name>\</el-name>\` | `13208368` |
163-
| care(html)\`\<el-name>\</el-name>\` | `8984663` |
164-
| care(html)\`<${ElClass}></${ElClass}>\` | `559925` |
165-
| Firefox 71.0.0 | |
166-
| clean html\`\<el-name>\</el-name>\` | `3237455` |
167-
| care(html)\`\<el-name>\</el-name>\` | `2107983` |
168-
| care(html)\`<${ElClass}></${ElClass}>\` | `279335` |
169-
| Safari 13.0.3 | |
170-
| clean html\`\<el-name>\</el-name>\` | `8824180` |
171-
| care(html)\`\<el-name>\</el-name>\` | `5019988` |
172-
| care(html)\`<${ElClass}></${ElClass}>\` | `777846` |
118+
There are 2 things which `carehtml` can slow down and which can be measured: creating a template and rendering a template.
119+
Both can be measured together as well.
120+
121+
The original idea was that the benchmarks need to compare the most minimalistic template possible, e.g. `<my-element></my-element>` where `MyElement` does not render any internal template, otherwise the benchmarks will measure the DOM update caused by the internal template instead of the `carehtml` overhead.
122+
It turned out to be quite difficult to see the `carehtml` impact in such benchmarks, because it's insignificant as compared to even rendering such a minimalistic `<my-element></my-element>` template.
123+
You can play around with this by using `yarn bench:create-and-render:chrome` script and alike and modifying the `benchmarks/index.html` to your needs, e.g. by removing the constructors of the measured elements.
124+
125+
The only thing that makes sense to measure in this situation is the rerendering.
126+
The idea is to check if it does not rerender unnecessarily second time when the classes stay the same meaning that the actual template is also the same.
127+
That's what makes `lit-html` so fast after all and `carehml` should not break this essential optimisation.
128+
In such benchmarks the `<my-element></my-element>` should have an internal template which will take most of the time of each render, so that the rerendering (if it happens) is close to being 2 times slower due to that internal template being rendered again.
129+
The end setup has `MyElement` with a shadow root with 100000 divs containing some text.
130+
The script `yarn bench:create-and-render-twice` can be used to measure that.
131+
The goal is to have the same numbers when using `lit-html` directly or wrapped with `carehtml`.
132+
And it's important to measure this with `lit-html 2.x` which does not have a special template string cache originally introduced for legacy browsers where string template literals were buggy.
133+
134+
These are the results for Chrome which clearly show no overhead on rerendering when wrapping with `carehtml`:
135+
136+
| Benchmark | Avg time | vs direct | vs wrapped | vs wrapped with classes |
137+
| -------------------- | ------------------: | ---------------------------------------: | ---------------------------------------: | ---------------------------------------: |
138+
| direct | 108.63ms - 112.34ms | - | unsure<br>-1% - +3%<br>-1.57ms - +2.84ms | unsure<br>-2% - +2%<br>-2.42ms - +1.80ms |
139+
| wrapped | 108.66ms - 111.05ms | unsure<br>-3% - +1%<br>-2.84ms - +1.57ms | - | unsure<br>-2% - +1%<br>-2.51ms - +0.61ms |
140+
| wrapped with classes | 109.80ms - 111.80ms | unsure<br>-2% - +2%<br>-1.80ms - +2.42ms | unsure<br>-1% - +2%<br>-0.61ms - +2.51ms | - |
141+
142+
Measurements in other browsers are similar.
173143

174144
## Special Thanks
175145

‎benchmarks/generate-tachometer-config.js

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
const fs = require('fs');
2+
const templateJson = require('./template-tachometer.json');
3+
4+
const templateJsonString = JSON.stringify(templateJson, null, 2);
5+
6+
const args = process.argv.slice(2);
7+
const params = Object.fromEntries(args.map((arg) => arg.slice(1).split('=')));
8+
9+
const tachomaterJsonString = templateJsonString
10+
.replace(new RegExp('{{measurement}}', 'g'), params.m)
11+
.replace(new RegExp('{{browser}}', 'g'), params.b);
12+
13+
fs.writeFileSync(`${__dirname}/tachometer.json`, tachomaterJsonString);

‎benchmarks/index.html

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
<!DOCTYPE html>
2+
<script type="module">
3+
import { html as litHtml, render } from 'lit-html';
4+
import wrap from '../src/wrap.js';
5+
6+
const queryParams = new URL(document.location).searchParams;
7+
8+
class DefinedElement extends HTMLElement {
9+
constructor() {
10+
super();
11+
this.attachShadow({ mode: 'open' });
12+
for (let i = 0; i < 100000; i++) {
13+
const div = document.createElement('div');
14+
div.innerText = 'text';
15+
this.shadowRoot.appendChild(div);
16+
}
17+
}
18+
}
19+
window.customElements.define('defined-element', DefinedElement);
20+
21+
class UndefinedElement extends HTMLElement {
22+
constructor() {
23+
super();
24+
this.attachShadow({ mode: 'open' });
25+
for (let i = 0; i < 100000; i++) {
26+
const div = document.createElement('div');
27+
div.innerText = 'text';
28+
this.shadowRoot.appendChild(div);
29+
}
30+
}
31+
}
32+
33+
const careHtml = wrap(litHtml);
34+
35+
let getTemplate;
36+
37+
if (queryParams.has('direct')) {
38+
getTemplate = () => litHtml`<defined-element></defined-element>`;
39+
} else if (queryParams.has('wrapped')) {
40+
getTemplate = () => careHtml`<defined-element></defined-element>`;
41+
} else if (queryParams.has('wrapped-with-classes')) {
42+
getTemplate = () => careHtml`<${UndefinedElement}></${UndefinedElement}>`;
43+
}
44+
45+
performance.mark('create-and-render-twice-mark');
46+
47+
performance.mark('create-and-render-mark');
48+
performance.mark('create-mark');
49+
let template = getTemplate();
50+
performance.measure('create', 'create-mark');
51+
performance.mark('render-mark');
52+
render(template, document.body);
53+
performance.measure('render', 'render-mark');
54+
performance.measure('create-and-render', 'create-and-render-mark');
55+
56+
performance.mark('recreate-and-rerender-mark');
57+
performance.mark('recreate-mark');
58+
template = getTemplate();
59+
performance.measure('recreate', 'recreate-mark');
60+
performance.mark('rerender-mark');
61+
render(template, document.body);
62+
performance.measure('rerender', 'rerender-mark');
63+
performance.measure('recreate-and-rerender', 'recreate-and-rerender-mark');
64+
65+
performance.measure('create-and-render-twice', 'create-and-render-twice-mark');
66+
</script>

‎benchmarks/template-tachometer.json

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
{
2+
"$schema": "https://raw.githubusercontent.com/Polymer/tachometer/v0.5.8/config.schema.json",
3+
"sampleSize": 30,
4+
"timeout": 0,
5+
"benchmarks": [
6+
{
7+
"name": "direct",
8+
"url": "benchmarks/index.html?direct",
9+
"browser": "{{browser}}",
10+
"measurement": [
11+
{
12+
"mode": "performance",
13+
"entryName": "{{measurement}}"
14+
}
15+
]
16+
},
17+
{
18+
"name": "wrapped",
19+
"url": "benchmarks/index.html?wrapped",
20+
"browser": "{{browser}}",
21+
"measurement": [
22+
{
23+
"mode": "performance",
24+
"entryName": "{{measurement}}"
25+
}
26+
]
27+
},
28+
{
29+
"name": "wrapped with classes",
30+
"url": "benchmarks/index.html?wrapped-with-classes",
31+
"browser": "{{browser}}",
32+
"measurement": [
33+
{
34+
"mode": "performance",
35+
"entryName": "{{measurement}}"
36+
}
37+
]
38+
}
39+
]
40+
}

‎package.json

Lines changed: 33 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -12,13 +12,11 @@
1212
"@bundled-es-modules/chai": "^4.2.2",
1313
"@commitlint/cli": "^8.2.0",
1414
"@commitlint/config-conventional": "^8.2.0",
15-
"@open-wc/testing-karma": "^3.1.18",
1615
"@web/dev-server-legacy": "^0.1.7",
1716
"@web/test-runner": "^0.12.17",
1817
"@web/test-runner-browserstack": "^0.4.2",
1918
"@web/test-runner-playwright": "^0.8.4",
2019
"babel-eslint": "^10.0.3",
21-
"benchmark": "^2.1.4",
2220
"browserslist": "^4.16.3",
2321
"browserslist-browserstack": "^3.1.1",
2422
"eclint": "^2.8.1",
@@ -28,31 +26,53 @@
2826
"eslint-plugin-import": "^2.19.1",
2927
"htm": "^2.2.1",
3028
"husky": "^3.1.0",
31-
"karma": "^4.4.1",
32-
"karma-benchmark": "^1.0.4",
33-
"karma-benchmark-json-reporter": "^1.0.1",
34-
"karma-benchmarkjs-reporter": "^1.0.0",
35-
"karma-chrome-launcher": "^3.1.0",
36-
"karma-detect-browsers": "^2.3.3",
37-
"karma-firefox-launcher": "^1.2.0",
38-
"karma-safari-applescript-launcher": "^0.1.0",
3929
"lint-staged": "^9.5.0",
40-
"lit-html": "^1.1.2",
30+
"lit-html": "2.0.0-pre.7",
4131
"preact": "^10.1.0",
4232
"prettier": "^2.2.1",
33+
"tachometer": "^0.5.8",
4334
"wallaby-webpack": "^3.9.15",
4435
"webpack": "^4.41.2"
4536
},
4637
"scripts": {
47-
"bench": "karma start karma.bench.conf.js",
48-
"bench:debug": "karma start karma.bench.conf.js --no-single-run",
38+
"bench": "yarn bench:create && yarn bench:render && yarn bench:create-and-render && yarn bench:update",
39+
"bench:create": "yarn bench:create:chrome && yarn bench:create:firefox && yarn bench:create:safari",
40+
"bench:create:chrome": "yarn tachometer:generate-config -m=create -b=chrome-headless && yarn tachometer:run",
41+
"bench:create:firefox": "yarn tachometer:generate-config -m=create -b=firefox-headless && yarn tachometer:run",
42+
"bench:create:safari": "yarn tachometer:generate-config -m=create -b=safari && yarn tachometer:run",
43+
"bench:render": "yarn bench:render:chrome && yarn bench:render:firefox && yarn bench:render:safari",
44+
"bench:render:chrome": "yarn tachometer:generate-config -m=render -b=chrome-headless && yarn tachometer:run",
45+
"bench:render:firefox": "yarn tachometer:generate-config -m=render -b=firefox-headless && yarn tachometer:run",
46+
"bench:render:safari": "yarn tachometer:generate-config -m=render -b=safari && yarn tachometer:run",
47+
"bench:create-and-render": "yarn bench:create-and-render:chrome && yarn bench:create-and-render:firefox && yarn bench:create-and-render:safari",
48+
"bench:create-and-render:chrome": "yarn tachometer:generate-config -m=create-and-render -b=chrome-headless && yarn tachometer:run",
49+
"bench:create-and-render:firefox": "yarn tachometer:generate-config -m=create-and-render -b=firefox-headless && yarn tachometer:run",
50+
"bench:create-and-render:safari": "yarn tachometer:generate-config -m=create-and-render -b=safari && yarn tachometer:run",
51+
"bench:recreate": "yarn bench:recreate:chrome && yarn bench:recreate:firefox && yarn bench:recreate:safari",
52+
"bench:recreate:chrome": "yarn tachometer:generate-config -m=recreate -b=chrome-headless && yarn tachometer:run",
53+
"bench:recreate:firefox": "yarn tachometer:generate-config -m=recreate -b=firefox-headless && yarn tachometer:run",
54+
"bench:recreate:safari": "yarn tachometer:generate-config -m=recreate -b=safari && yarn tachometer:run",
55+
"bench:rerender": "yarn bench:rerender:chrome && yarn bench:rerender:firefox && yarn bench:rerender:safari",
56+
"bench:rerender:chrome": "yarn tachometer:generate-config -m=rerender -b=chrome-headless && yarn tachometer:run",
57+
"bench:rerender:firefox": "yarn tachometer:generate-config -m=rerender -b=firefox-headless && yarn tachometer:run",
58+
"bench:rerender:safari": "yarn tachometer:generate-config -m=rerender -b=safari && yarn tachometer:run",
59+
"bench:recreate-and-rerender": "yarn bench:recreate-and-rerender:chrome && yarn bench:recreate-and-rerender:firefox && yarn bench:recreate-and-rerender:safari",
60+
"bench:recreate-and-rerender:chrome": "yarn tachometer:generate-config -m=recreate-and-rerender -b=chrome-headless && yarn tachometer:run",
61+
"bench:recreate-and-rerender:firefox": "yarn tachometer:generate-config -m=recreate-and-rerender -b=firefox-headless && yarn tachometer:run",
62+
"bench:recreate-and-rerender:safari": "yarn tachometer:generate-config -m=recreate-and-rerender -b=safari && yarn tachometer:run",
63+
"bench:create-and-render-twice": "yarn bench:create-and-render:chrome && yarn bench:create-and-render-twice:firefox && yarn bench:create-and-render:safari",
64+
"bench:create-and-render-twice:chrome": "yarn tachometer:generate-config -m=create-and-render-twice -b=chrome-headless && yarn tachometer:run",
65+
"bench:create-and-render-twice:firefox": "yarn tachometer:generate-config -m=create-and-render-twice -b=firefox-headless && yarn tachometer:run",
66+
"bench:create-and-render-twice:safari": "yarn tachometer:generate-config -m=create-and-render-twice -b=safari && yarn tachometer:run",
4967
"format": "yarn format:eclint && yarn format:eslint && yarn format:prettier",
5068
"format:eclint": "eclint fix $(git ls-files)",
5169
"format:eslint": "eslint '**/*.js' --fix",
5270
"format:prettier": "prettier '**/*.{js,md}' --write",
5371
"lint": "yarn lint:eclint && yarn lint:eslint",
5472
"lint:eclint": "eclint check $(git ls-files)",
5573
"lint:eslint": "eslint '**/*.js'",
74+
"tachometer:generate-config": "node benchmarks/generate-tachometer-config.js",
75+
"tachometer:run": "tach --config benchmarks/tachometer.json",
5676
"test": "web-test-runner",
5777
"test:bs": "web-test-runner --config=web-test-runner.bs.config.mjs"
5878
}

‎src/wrap.spec.js

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,13 +29,12 @@ describe('wrap', () => {
2929
});
3030

3131
it('integrates with lit-html', async () => {
32-
const { html: litHtml, TemplateResult, render } = await import('lit-html');
32+
const { html: litHtml, render } = await import('lit-html');
3333

3434
const html = wrap(litHtml);
3535
class MyAzalea extends HTMLElement {}
3636

3737
const template = html`<${MyAzalea} id="${'my-id'}">${'my text'}</${MyAzalea}>`;
38-
expect(template).to.be.instanceof(TemplateResult);
3938

4039
const fixture = document.createElement('div');
4140
document.body.appendChild(fixture);

0 commit comments

Comments
 (0)