Skip to content

Commit 6c24d92

Browse files
committed
docs(bench): the first three-engine table, including where mcpp loses
15 ok / 0 failed / 零告警 —— 三条臂第一次同时产出可运行二进制,所以这是第一份 真实的三方数据。此前 cmake 与 xmake 两列一直是 `failed`,表只能是 mcpp 对 mcpp。 | 场景 | mcpp | cmake | xmake | |---|---|---|---| | cold | 92.49s · 1.29x | 119.46s | 105.02s · 1.14x | | noop | 0.74s · **0.49x** | 0.36s | 0.40s · 0.90x | | touch-hub | **1.79s · 54.96x** | 98.16s | 98.16s | | edit-body | 88.38s · 1.11x | 98.43s | 98.00s | | edit-comment | 93.81s · 1.04x | 97.98s | 97.97s | **两处是 mcpp 输,都写进了正文而不是脚注:** * `noop` 0.49x —— 什么都不做时 mcpp 是三者里最慢的(0.74s vs cmake 0.36s)。 这是每次调用的固定开销,也是用户在「改一行、构建一次」循环里唯一每次都感受 得到的数字。 * `edit-comment` 只有 1.04x,不是 mcpp 自身负载上的 200x。注释插进了 xlings 保留 在接口单元里的内联函数体,BMI 真的变了,**级联是欠的**。格子的 note 记录当次 是哪种形态,不要读成优化失效。 `touch-hub` 54.96x 才是真结果,而且 cmake 与 xmake 相差 **0.00s** —— 两个时间戳 驱动的引擎本就该长这样。 原始报告存进 bench/results/xlings-3way-20260814/;中英文两版同步。旧的 old→new 两风格对照表保留,并标明它两列都是 mcpp、不受外部臂当时未完成的影响。
1 parent 2a9da6b commit 6c24d92

3 files changed

Lines changed: 306 additions & 8 deletions

File tree

bench/README.md

Lines changed: 37 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -194,14 +194,43 @@ Four things this says, and the fixture can say none of them:
194194
are the same project either side of one refactor. Ratios against the released
195195
mcpp.
196196

197-
> **These numbers predate the foreign arms working.** For a long time this said
198-
> the cmake and xmake arms "stop at the link", and that was recorded as a known
199-
> gap — it was not a gap, it was unfinished work, and calling it a gap let it
200-
> sit. The cmake and bazel arms now build both trees into a running binary
201-
> (`xlings --version` reports the tree's own version on each); xmake is in
202-
> progress. The table below is still mcpp-against-mcpp because it has not been
203-
> re-measured since. Until it is, do not read the absence of cmake and xmake
204-
> columns here as a statement about those engines.
197+
#### The three engines, on the combined tree
198+
199+
First measurement in which all three arms produce a running binary — the cmake
200+
and xmake columns below were `failed` cells until the arms were finished, and the
201+
table that stood here was mcpp-against-mcpp for that reason.
202+
203+
| scenario | **mcpp** | cmake | xmake |
204+
|---|---|---|---|
205+
| `cold` | **92.49s** · 1.29x | 119.46s · 1.00x | 105.02s · 1.14x |
206+
| `noop` | **0.74s** · 0.49x | 0.36s · 1.00x | 0.40s · 0.90x |
207+
| `touch-hub` | **1.79s** · 54.96x | 98.16s · 1.00x | 98.16s · 1.00x |
208+
| `edit-body` | **88.38s** · 1.11x | 98.43s · 1.00x | 98.00s · 1.00x |
209+
| `edit-comment` | **93.81s** · 1.04x | 97.98s · 1.00x | 97.97s · 1.00x |
210+
211+
<sub>xlings `2026.8.11.2`, gcc 16.1.0 payload, Linux x86_64 · i9-13900K · n=1 ·
212+
`--baseline cmake`. Raw report: `bench/results/xlings-3way-20260814/`.</sub>
213+
214+
Three things in that table are worth reading carefully, because two of them are
215+
mcpp LOSING:
216+
217+
* **`noop` is 0.49x — mcpp is the slowest of the three at doing nothing.** 0.74s
218+
against cmake's 0.36s. It is a fixed cost on every invocation, and it is the
219+
one number here that a user feels on every keystroke-to-build cycle.
220+
* **`edit-comment` is 1.04x, not the 200x the mcpp workload shows.** The comment
221+
lands INSIDE an inline function body that xlings keeps in its interface unit,
222+
so the BMI genuinely changes and the cascade is owed. The cell's note records
223+
which form ran; see §3, and do not read this as the optimisation failing.
224+
* **`touch-hub` is the real result: 54.96x.** Content unchanged, so mcpp compares
225+
the BMI it just produced against the previous one and skips 45 importers.
226+
cmake and xmake decide by timestamp and rebuild all of them — to within 0.00s
227+
of each other, which is what two timestamp-driven engines should look like.
228+
229+
#### The two code styles, mcpp against mcpp
230+
231+
Older run, kept because it is the only side-by-side of the two pinned
232+
styles. Both columns are mcpp, so the foreign arms being unfinished at the
233+
time does not affect it.
205234

206235
| scenario | combined `2026.8.11.2` old → new | split `2026.8.13.1` old → new | what the split buys |
207236
|---|---|---|---|

bench/README.zh-CN.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -159,6 +159,34 @@ job 一个测量都没有,这个状态持续了好几周。
159159

160160
---
161161

162+
## 4b. 三个引擎在 combined 树上的实测
163+
164+
**第一次三条臂都能产出可运行二进制的测量** —— 在此之前 cmake 与 xmake 两列一直是
165+
`failed`,所以这里原本只有 mcpp 对 mcpp。
166+
167+
| 场景 | **mcpp** | cmake | xmake |
168+
|---|---|---|---|
169+
| `cold` | **92.49s** · 1.29x | 119.46s · 1.00x | 105.02s · 1.14x |
170+
| `noop` | **0.74s** · 0.49x | 0.36s · 1.00x | 0.40s · 0.90x |
171+
| `touch-hub` | **1.79s** · 54.96x | 98.16s · 1.00x | 98.16s · 1.00x |
172+
| `edit-body` | **88.38s** · 1.11x | 98.43s · 1.00x | 98.00s · 1.00x |
173+
| `edit-comment` | **93.81s** · 1.04x | 97.98s · 1.00x | 97.97s · 1.00x |
174+
175+
<sub>xlings `2026.8.11.2`,gcc 16.1.0 载荷,Linux x86_64 · i9-13900K · n=1 ·
176+
`--baseline cmake`。原始报告:`bench/results/xlings-3way-20260814/`。</sub>
177+
178+
这张表里有三处值得细读,其中**两处是 mcpp 输**:
179+
180+
* **`noop` 是 0.49x —— 什么都不做时 mcpp 是三者中最慢的。** 0.74s 对 cmake 的
181+
0.36s。这是每次调用的固定开销,也是用户在「改一行、构建一次」的循环里唯一
182+
每次都感受得到的数字。
183+
* **`edit-comment` 是 1.04x,不是 mcpp 自身工作负载上的 200x。** 注释插进了
184+
xlings 保留在接口单元里的内联函数体,BMI 真的变了,级联是**欠的**。格子的
185+
note 会记录当次是哪种形态(见 §3),不要把它读成优化失效。
186+
* **`touch-hub` 才是真结果:54.96x。** 内容没变,mcpp 拿编译器刚产出的 BMI 和
187+
上一份比,跳过 45 个导入者;cmake 与 xmake 按时间戳判断,把它们全部重建 ——
188+
两者相差 0.00s,这正是两个时间戳驱动的引擎该有的样子。
189+
162190
## 5. 可观测性:跑的时候看得见
163191

164192
harness 会把进度实时打到 **stderr**(逐行 flush),stdout 留给报告:
Lines changed: 241 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
{
2+
"protocol_version": 1,
3+
"started_at": "2026-08-13T17:43:09Z",
4+
"host": {
5+
"os": "linux",
6+
"arch": "x86_64",
7+
"cpu_model": "13th Gen Intel(R) Core(TM) i9-13900K",
8+
"logical_cores": 32,
9+
"physical_cores": 24,
10+
"heterogeneous": true,
11+
"ram_bytes": 67147722752,
12+
"toolchain": "/home/speak/.mcpp/registry/data/xpkgs/xim-x-gcc/16.1.0/bin/g++"
13+
},
14+
"cells": [
15+
{
16+
"engine": "mcpp@2026.8.13.1",
17+
"compiler": "gcc",
18+
"profile": "release",
19+
"scenario": "cold",
20+
"fixture": "xlings-2026.8.11.2",
21+
"variant": "modules",
22+
"status": "ok",
23+
"note": "mcpp 2026.8.13.1",
24+
"runs": 1,
25+
"median_s": 92.493,
26+
"min_s": 92.493,
27+
"max_s": 92.493,
28+
"samples": [92.493]
29+
},
30+
{
31+
"engine": "mcpp@2026.8.13.1",
32+
"compiler": "gcc",
33+
"profile": "release",
34+
"scenario": "noop",
35+
"fixture": "xlings-2026.8.11.2",
36+
"variant": "modules",
37+
"status": "ok",
38+
"note": "mcpp 2026.8.13.1",
39+
"runs": 1,
40+
"median_s": 0.744,
41+
"min_s": 0.744,
42+
"max_s": 0.744,
43+
"samples": [0.744]
44+
},
45+
{
46+
"engine": "mcpp@2026.8.13.1",
47+
"compiler": "gcc",
48+
"profile": "release",
49+
"scenario": "touch-hub",
50+
"fixture": "xlings-2026.8.11.2",
51+
"variant": "modules",
52+
"status": "ok",
53+
"note": "mcpp 2026.8.13.1",
54+
"runs": 1,
55+
"median_s": 1.786,
56+
"min_s": 1.786,
57+
"max_s": 1.786,
58+
"samples": [1.786]
59+
},
60+
{
61+
"engine": "mcpp@2026.8.13.1",
62+
"compiler": "gcc",
63+
"profile": "release",
64+
"scenario": "edit-body",
65+
"fixture": "xlings-2026.8.11.2",
66+
"variant": "modules",
67+
"status": "ok",
68+
"note": "mcpp 2026.8.13.1 · perturbation: in-body",
69+
"runs": 1,
70+
"median_s": 88.379,
71+
"min_s": 88.379,
72+
"max_s": 88.379,
73+
"samples": [88.379]
74+
},
75+
{
76+
"engine": "mcpp@2026.8.13.1",
77+
"compiler": "gcc",
78+
"profile": "release",
79+
"scenario": "edit-comment",
80+
"fixture": "xlings-2026.8.11.2",
81+
"variant": "modules",
82+
"status": "ok",
83+
"note": "mcpp 2026.8.13.1 · perturbation: in-body",
84+
"runs": 1,
85+
"median_s": 93.813,
86+
"min_s": 93.813,
87+
"max_s": 93.813,
88+
"samples": [93.813]
89+
},
90+
{
91+
"engine": "cmake",
92+
"compiler": "gcc",
93+
"profile": "release",
94+
"scenario": "cold",
95+
"fixture": "xlings-2026.8.11.2",
96+
"variant": "modules",
97+
"status": "ok",
98+
"note": "cmake version 4.0.2 + ninja",
99+
"runs": 1,
100+
"median_s": 119.458,
101+
"min_s": 119.458,
102+
"max_s": 119.458,
103+
"samples": [119.458]
104+
},
105+
{
106+
"engine": "cmake",
107+
"compiler": "gcc",
108+
"profile": "release",
109+
"scenario": "noop",
110+
"fixture": "xlings-2026.8.11.2",
111+
"variant": "modules",
112+
"status": "ok",
113+
"note": "cmake version 4.0.2 + ninja",
114+
"runs": 1,
115+
"median_s": 0.361,
116+
"min_s": 0.361,
117+
"max_s": 0.361,
118+
"samples": [0.361]
119+
},
120+
{
121+
"engine": "cmake",
122+
"compiler": "gcc",
123+
"profile": "release",
124+
"scenario": "touch-hub",
125+
"fixture": "xlings-2026.8.11.2",
126+
"variant": "modules",
127+
"status": "ok",
128+
"note": "cmake version 4.0.2 + ninja",
129+
"runs": 1,
130+
"median_s": 98.161,
131+
"min_s": 98.161,
132+
"max_s": 98.161,
133+
"samples": [98.161]
134+
},
135+
{
136+
"engine": "cmake",
137+
"compiler": "gcc",
138+
"profile": "release",
139+
"scenario": "edit-body",
140+
"fixture": "xlings-2026.8.11.2",
141+
"variant": "modules",
142+
"status": "ok",
143+
"note": "cmake version 4.0.2 + ninja · perturbation: in-body",
144+
"runs": 1,
145+
"median_s": 98.434,
146+
"min_s": 98.434,
147+
"max_s": 98.434,
148+
"samples": [98.434]
149+
},
150+
{
151+
"engine": "cmake",
152+
"compiler": "gcc",
153+
"profile": "release",
154+
"scenario": "edit-comment",
155+
"fixture": "xlings-2026.8.11.2",
156+
"variant": "modules",
157+
"status": "ok",
158+
"note": "cmake version 4.0.2 + ninja · perturbation: in-body",
159+
"runs": 1,
160+
"median_s": 97.979,
161+
"min_s": 97.979,
162+
"max_s": 97.979,
163+
"samples": [97.979]
164+
},
165+
{
166+
"engine": "xmake",
167+
"compiler": "gcc",
168+
"profile": "release",
169+
"scenario": "cold",
170+
"fixture": "xlings-2026.8.11.2",
171+
"variant": "modules",
172+
"status": "ok",
173+
"note": "xmake v3.1.0+HEAD.96ad28e, A cross-platform build utility based on Lua",
174+
"runs": 1,
175+
"median_s": 105.024,
176+
"min_s": 105.024,
177+
"max_s": 105.024,
178+
"samples": [105.024]
179+
},
180+
{
181+
"engine": "xmake",
182+
"compiler": "gcc",
183+
"profile": "release",
184+
"scenario": "noop",
185+
"fixture": "xlings-2026.8.11.2",
186+
"variant": "modules",
187+
"status": "ok",
188+
"note": "xmake v3.1.0+HEAD.96ad28e, A cross-platform build utility based on Lua",
189+
"runs": 1,
190+
"median_s": 0.401,
191+
"min_s": 0.401,
192+
"max_s": 0.401,
193+
"samples": [0.401]
194+
},
195+
{
196+
"engine": "xmake",
197+
"compiler": "gcc",
198+
"profile": "release",
199+
"scenario": "touch-hub",
200+
"fixture": "xlings-2026.8.11.2",
201+
"variant": "modules",
202+
"status": "ok",
203+
"note": "xmake v3.1.0+HEAD.96ad28e, A cross-platform build utility based on Lua",
204+
"runs": 1,
205+
"median_s": 98.157,
206+
"min_s": 98.157,
207+
"max_s": 98.157,
208+
"samples": [98.157]
209+
},
210+
{
211+
"engine": "xmake",
212+
"compiler": "gcc",
213+
"profile": "release",
214+
"scenario": "edit-body",
215+
"fixture": "xlings-2026.8.11.2",
216+
"variant": "modules",
217+
"status": "ok",
218+
"note": "xmake v3.1.0+HEAD.96ad28e, A cross-platform build utility based on Lua · perturbation: in-body",
219+
"runs": 1,
220+
"median_s": 97.999,
221+
"min_s": 97.999,
222+
"max_s": 97.999,
223+
"samples": [97.999]
224+
},
225+
{
226+
"engine": "xmake",
227+
"compiler": "gcc",
228+
"profile": "release",
229+
"scenario": "edit-comment",
230+
"fixture": "xlings-2026.8.11.2",
231+
"variant": "modules",
232+
"status": "ok",
233+
"note": "xmake v3.1.0+HEAD.96ad28e, A cross-platform build utility based on Lua · perturbation: in-body",
234+
"runs": 1,
235+
"median_s": 97.974,
236+
"min_s": 97.974,
237+
"max_s": 97.974,
238+
"samples": [97.974]
239+
}
240+
]
241+
}

0 commit comments

Comments
 (0)