fix(inference): align unofficial-run overlay x-axis + roofline with official chart / 对齐非官方运行叠加层的 x 轴与 roofline#596
Merged
Conversation
…-axis The `?unofficialrun=` overlay path (`processOverlayChartData`) never applied the selected latency percentile to the x-axis field, so on agentic charts it plotted overlay points against the raw `median_intvty` / `median_e2el` column while the official run's points sat on the percentile-adjusted column (`p90_intvty` etc.) selected via the percentile picker. Because `p90_intvty = 1/p90_itl < median_intvty = 1/median_itl`, an overlay of the *same* run rendered shifted to the right of its own official points on the "P90 Interactivity" chart (identical throughput + config labels, mismatched x). Mirror `useChartData`'s percentile handling in `processOverlayChartData`: the natural x is now `withPercentile(chartDef.x, isAgentic ? selectedPercentile : 'median')`, plus the final agentic rewrite that carries the percentile onto the input-metric override. Non-agentic (fixed-seq) stays forced to `median` (those rows carry no p90_/p99_ columns). Thread `selectedPercentile` through the `ChartDisplay` caller and its memo deps. The parallel helper `buildReplayTimeline.resolveXAxisField` has the same shape of divergence but is inert: replay only runs for fixed-seq (history is keyed by isl/osl, disabled for agentic), so its natural x is always `median`. Left unchanged; documented so it isn't mistaken for a live bug. 中文：修复非官方运行叠加层（`?unofficialrun=`）在 agentic 图表上 x 轴未应用所选延迟 百分位（percentile）的问题。此前叠加层路径 `processOverlayChartData` 始终使用原始的 `median_intvty` / `median_e2el` 列，而官方运行的数据点使用百分位选择器选定的 `p90_intvty` 等列。由于 `p90_intvty = 1/p90_itl < median_intvty = 1/median_itl`， 同一次运行的叠加层会相对其自身官方数据点向右偏移（吞吐量与配置标签相同、x 值不一致）， 即用户报告的 GLM5.2 B300 "P90 Interactivity" 图表错位。现让 `processOverlayChartData` 与 `useChartData` 的百分位逻辑保持一致：自然 x 轴取 `withPercentile(chartDef.x, isAgentic ? selectedPercentile : 'median')`，并在 agentic 下对输入指标（input metric） 覆盖项再套用百分位；非 agentic（固定序列）仍强制 `median`（这些行没有 p90_/p99_ 列）。 同时将 `selectedPercentile` 传入 `ChartDisplay` 调用点及其 memo 依赖。并行的 `buildReplayTimeline.resolveXAxisField` 存在同类差异但无实际影响：replay 仅对固定序列 运行（历史数据按 isl/osl 键控，agentic 时禁用），其自然 x 轴恒为 `median`，故保持不变并加注说明。
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…o winners On the agentic interactivity chart the official roofline is restricted to configs that ALSO win on end-to-end latency (anti-benchmark-hacking): useChartData stamps `isOnE2eFrontier` and ScatterGraph's `rooflines` narrows to those winners before paretoing on the interactivity axis. The overlay path did neither — `overlayRooflines` computed a fresh frontier directly on the (interactivity, y) plane — so an `?unofficialrun=` overlay of the same run drew a line that rode ABOVE the official green line wherever an interactivity-optimal config was not e2e-optimal. - Extract the e2e-frontier seed into a shared helper `e2eFrontierWinners` (components/inference/utils/e2eFrontier.ts). `useChartData.e2eParetoIds` now derives its persisted-id set from it, so official and overlay seed identically. - `processOverlayChartData` stamps `isOnE2eFrontier` on agentic interactivity overlay points, seeded per unofficial run (matching overlayRooflines' per-run grouping) so runs don't cross-dominate. The e2e chart and fixed-seq leave the flag unset (no restriction), as on the official path. - `overlayRooflines` now honors the flag with the same narrow-to-winners logic as the official `rooflines`. Depends on the selectedPercentile plumbing from the previous commit (the e2e-latency seed field is percentile-prefixed, e.g. p90_e2el). 中文：修复非官方运行叠加层（`?unofficialrun=`）的 roofline 未套用 E2E 帕累托（Pareto） 前沿约束的问题。在 agentic 的 interactivity 图表上，官方 roofline 会限制为同时在端到端 延迟（end-to-end latency）上取胜的配置（反 benchmark 作弊）：`useChartData` 为数据点打上 `isOnE2eFrontier` 标记，`ScatterGraph` 的 `rooflines` 在按 interactivity 轴求前沿前先收窄到 这些优胜点。叠加层路径两者皆无——`overlayRooflines` 直接在 (interactivity, y) 平面上重新 计算前沿，导致同一次运行的叠加层在“interactivity 最优但非 E2E 最优”的区段画出的线高于官方 绿线。改动：将 E2E 前沿种子逻辑抽取为共享函数 `e2eFrontierWinners` （components/inference/utils/e2eFrontier.ts），`useChartData.e2eParetoIds` 由其派生持久化 id 集合，使官方与叠加层种子一致；`processOverlayChartData` 为 agentic interactivity 叠加层 数据点打上 `isOnE2eFrontier`，并按每个非官方运行分别求种子（与 `overlayRooflines` 的按运行 分组一致），避免不同运行相互支配；E2E 图表与固定序列不设该标记（不做约束），与官方路径一致； `overlayRooflines` 现按与官方 `rooflines` 相同的收窄逻辑处理该标记。依赖上一提交引入的 selectedPercentile 传递（E2E 延迟种子字段带百分位前缀，如 p90_e2el）。
Oseltamivir
changed the title
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 3f8197a. Configure here.
…-mode The official path stamps `isOnE2eFrontier` for EVERY non-e2e x-axis mode (`selectedXAxisMode !== 'e2e'`), which includes the 'ttft' mode — the e2e chartType rendered with a *_ttft x override. The overlay stamping added in the previous commit only covered the interactivity chartType, so the TTFT chart's overlay roofline still drew a fresh TTFT-plane frontier instead of the e2e-restricted one. Stamp whenever the agentic chart's resolved x-axis is not the natural e2e latency (interactivity chartType, or e2e chartType with the TTFT override), matching the official condition mode-for-mode. Verified end-to-end in a real browser (Cypress against the running app, real run-29682242847 values served as both official and overlay data): official and overlay interactivity rooflines both draw 4 vertices (C=48, C=8, C=2, C=1), excluding the e2e-dominated TP8 C=4. 中文：将叠加层的 E2E 前沿约束扩展到 ttft x 轴模式。官方路径对所有非 e2e 的 x 轴模式 （`selectedXAxisMode !== 'e2e'`）都会打上 `isOnE2eFrontier` 标记，其中包括 'ttft' 模式（以 *_ttft 覆盖 x 轴渲染的 e2e chartType）。上一提交新增的叠加层打标仅覆盖了 interactivity chartType，导致 TTFT 图表的叠加层 roofline 仍在 TTFT 平面上重新计算前沿， 而非使用 E2E 约束后的前沿。现在只要 agentic 图表解析出的 x 轴不是自然 e2e 延迟 （interactivity chartType，或带 TTFT 覆盖的 e2e chartType）即打标，与官方条件逐模式对齐。 已在真实浏览器中端到端验证（Cypress 驱动运行中的应用，将 run 29682242847 的真实数据同时 作为官方与叠加层数据）：官方与叠加层的 interactivity roofline 均绘制 4 个顶点 （C=48、C=8、C=2、C=1），排除了在 E2E 上被支配的 TP8 C=4。
Overlay (unofficial run) X markers rendered every point unconditionally, while official points are hidden by `isPointVisible` when Optimal Only is on — and Optimal Only is ON BY DEFAULT (`i_optimal !== '0'`). In the default agentic interactivity view this made an e2e-dominated overlay config read as a pareto point: the official TP8 C=4 dot was hidden, but the overlay C=4 X stayed visible sitting on the dashed roofline (the monotone spline between C=8 and C=2 passes within ~0.5% of it at the same x). On the e2e chart the same X sits far below the curve, so the artifact only showed on non-e2e axes. Mirror the official visibility rule on the overlay path: - `overlayOptimalPoints`: identity set of every point on any overlay run's drawn roofline (already e2e-restricted for agentic non-e2e modes). Frontier arrays hold the same refs as `processedOverlayData` items, so identity membership is exact and can't collide across runs. - `isOverlayPointVisible` = `!hideNonOptimal || overlayOptimalPoints.has(d)`, exposed through `interactionRef`; applied in the overlay layer render and in the decoration effect so a toggle flip restyles existing DOM without a chart rebuild (same Effect-4 path official dots use). - Scale domains (`visiblePoints`) and the legend points table exclude hidden overlay points, matching the official branch's filters. New e2e spec drives the real pipeline with the run-29682242847 values: in the default Optimal Only view exactly 4 of 5 overlay X's are visible (C=4 hidden, matching its hidden official twin), toggling off shows all 5, re-enabling hides C=4 again. Existing overlay specs (ttft-x-axis-toggle, watermark, inference-chart) still pass. 中文：修复非官方运行叠加层数据点不响应 "Optimal Only"（仅显示最优）开关的问题。 叠加层的 X 标记此前无条件渲染所有数据点，而官方数据点在开关开启时会通过 `isPointVisible` 隐藏非帕累托（Pareto）点——且该开关默认开启（`i_optimal !== '0'`）。 在默认的 agentic interactivity 视图中，这使得在 E2E 上被支配的叠加层配置看起来像 帕累托点：官方 TP8 C=4 圆点被隐藏，但叠加层的 C=4 X 标记仍然可见，且恰好落在虚线 roofline 上（C=8 与 C=2 之间的单调样条曲线在同一 x 处与其相差不到 0.5%）。在 E2E 图表上该 X 明显低于曲线，因此该现象只出现在非 E2E 轴上。现将官方可见性规则镜像到 叠加层路径：新增 `overlayOptimalPoints`（任一叠加层运行已绘制 roofline 上数据点的 恒等集合，agentic 非 E2E 模式下已受 E2E 约束）；`isOverlayPointVisible` 通过 `interactionRef` 暴露，在叠加层渲染与装饰 effect 中应用（与官方数据点相同的 Effect-4 路径，开关切换无需重建图表）；坐标轴域（`visiblePoints`）与图例数据点 表格同样排除被隐藏的叠加层点。新增 e2e 测试以 run 29682242847 的真实数值驱动完整 渲染管线：默认视图下 5 个叠加层 X 中恰好 4 个可见（C=4 隐藏，与其官方对应点一致）， 关闭开关显示全部 5 个，重新开启后 C=4 再次隐藏。既有叠加层测试全部通过。
…t across official/overlay paths
Simplification pass over the overlay-parity fixes; also fixes the Bugbot
finding that direction-less metrics cleared the overlay roofline.
- `e2eFrontierWinners` now returns `Set | null` — null means the y-metric
declares no e2e roofline direction, i.e. no restriction applies. Previously
it returned an empty set for that case and `processOverlayChartData` stamped
`isOnE2eFrontier = false` on every point, seeding an EMPTY overlay frontier
for the measured-power metrics (official leaves the flag unset and draws
unrestricted). `e2eParetoIds` loses its duplicated chartDef/direction guard.
Frontier framing now projects `{x, y, orig}` instead of spread-copying every
~70-property point and back-mapping through a Map — the pareto fns only read
x/y and return input refs.
- New shared `resolveXAxisField` (utils/resolveXAxisField.ts): the x-axis
branch ladder (natural percentile-adjust → input-metric overrides → e2e ttft
override → agentic percentile rewrite) previously lived twice, in
`useChartData.stableChartDefinitions` and `processOverlayChartData`, held in
sync by "must mirror" comments — the drift pattern behind three of this
branch's four bugs. Both now call the resolver; labels/headings stay in
useChartData keyed off the resolver's branch discriminant.
- The e2e-frontier stamping gate is now caller-computed: ChartDisplay passes
`restrictToE2eFrontier = isAgentic && selectedXAxisMode !== 'e2e'` — the
exact official gate — instead of processOverlayChartData re-deriving it from
chartType + field suffix (the reconstruction that missed the ttft mode).
- New `e2eRestrictedSeed` in e2eFrontier.ts is the single interpreter of the
`isOnE2eFrontier` tri-state; both `rooflines` and `overlayRooflines` call it
instead of carrying verbatim copies of the narrowing dance.
- `isOverlayPointVisible` hoisted above `pointsTable`/`visiblePoints` so all
Optimal Only consumers call the one predicate; overlay layer render folds
its two per-point predicate calls into one `.each`.
Verified: 453 inference unit tests (new regression test for the direction-less
metric case), 27 overlay-related Cypress e2e tests, full unit suite at the
pre-existing baseline, typecheck/lint/fmt clean.
中文：对叠加层一致性修复做简化重构，并修复 Bugbot 发现的问题（无 roofline 方向的
指标会清空叠加层前沿）。`e2eFrontierWinners` 现返回 `Set | null`——null 表示所选
y 指标未声明 e2e roofline 方向，即不适用任何约束；此前该情形返回空集合，导致
`processOverlayChartData` 给所有点打上 `isOnE2eFrontier = false`，使实测功耗类指标
的叠加层前沿为空（官方路径不设标记、绘制不受限的 roofline）。前沿计算改用
`{x, y, orig}` 精简投影，不再整体展开复制约 70 个属性的点对象。新增共享的
`resolveXAxisField`：x 轴字段解析逻辑此前在 `useChartData` 与
`processOverlayChartData` 中各有一份、仅靠"必须镜像"注释保持同步（本分支四个 bug
中有三个源于此类漂移），现两条路径调用同一解析器。E2E 前沿打标条件改由调用方
（ChartDisplay）按官方口径计算并传入（`isAgentic && selectedXAxisMode !== 'e2e'`）。
新增 `e2eRestrictedSeed` 作为 `isOnE2eFrontier` 三态标记的唯一解释器，官方与叠加层
roofline memo 均调用它。`isOverlayPointVisible` 上移，所有 Optimal Only 消费方调用
同一谓词。验证：453 个单元测试（新增无方向指标回归测试）、27 个叠加层 e2e 测试、
完整单元套件与既有基线一致，typecheck/lint/fmt 全部通过。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Aligns the
?unofficialrun=overlay with the official chart on agentic charts. Three divergences, found in sequence (each unmasked the next):1. Overlay x-axis ignored the selected percentile
Official plots agentic charts at the percentile-adjusted x (
p90_intvtyetc.); the overlay used rawmedian_intvty, shifting the same run's points right. Fix:processOverlayChartDatamirrorsuseChartData's percentile logic (withPercentile(chartDef.x, isAgentic ? selectedPercentile : 'median')+ agentic override rewrite);selectedPercentilethreaded throughChartDisplay.2. Overlay roofline wasn't restricted to E2E-pareto winners
Official restricts the non-e2e-mode roofline to configs that also win on end-to-end latency (
isOnE2eFrontierviauseChartData→ScatterGraph.rooflines). The overlay computed a fresh frontier on the displayed axis. Fix: extracted the seed into a sharede2eFrontierWinnershelper used by both paths;processOverlayChartDatastamps the flag per unofficial run for every agentic non-e2e axis (interactivity chartType + the e2e chartType under the TTFT override);overlayRooflinesnarrows identically to the official path.3. Overlay points ignored the Optimal Only toggle — the user-visible symptom
Optimal Only is ON by default (
i_optimal !== '0'). Official non-pareto points are hidden byisPointVisible; overlay X markers rendered every point unconditionally. In the default agentic interactivity view the e2e-dominated TP8 C=4 (GLM5.2 B300 hicache run) kept its X visible sitting on the dashed roofline — the monotone spline between C=8 and C=2 passes within ~0.5% of it — while its official twin was hidden, reading exactly as "C=4 is on the overlay pareto (but not on E2E, where the X sits far below the curve)". Fix: mirrored the official visibility rule:overlayOptimalPoints(identity set of points on any overlay run's drawn roofline, already e2e-restricted),isOverlayPointVisibleapplied in the overlay layer render, the decoration (toggle) effect, scale domains, and the legend points table.Verification
overlay-optimal-only.cy.ts, real run-29682242847 values): default view shows exactly 4 of 5 overlay X's (C=4 hidden, matching official); toggle off → 5; re-enable → 4. Spec failed before the fix, passes after.Overlay-path support
This PR is entirely fixes to the overlay path (AGENTS.md mandatory overlay support). Verified against
?unofficialrun=29682242847data end-to-end.中文说明
将
?unofficialrun=叠加层与官方图表在 agentic 图表上的行为对齐。共修复三处不一致（逐层排查，每一处修复后暴露下一处）：1. 叠加层 x 轴未应用所选百分位（percentile）：官方按百分位调整后的 x 列（如
p90_intvty）绘制，叠加层却使用原始median_intvty，导致同一运行的点右移。修复：processOverlayChartData镜像useChartData的百分位逻辑，并将selectedPercentile传入ChartDisplay。2. 叠加层 roofline 未受 E2E 帕累托（Pareto）约束：官方将非 E2E 模式的 roofline 限制为同时在端到端延迟上取胜的配置（反 benchmark 作弊），叠加层却在显示轴上重新计算前沿。修复：抽取共享的
e2eFrontierWinners，两条路径使用同一种子；叠加层按运行分别打isOnE2eFrontier标记（覆盖 interactivity 与 TTFT 覆盖下的 e2e chartType）；overlayRooflines采用与官方相同的收窄逻辑。3. 叠加层数据点不响应 "Optimal Only" 开关——用户可见的症状：该开关默认开启（
i_optimal !== '0'）。官方非帕累托点会被isPointVisible隐藏，叠加层 X 标记却无条件全部渲染。在默认 agentic interactivity 视图中，被 E2E 支配的 TP8 C=4（GLM5.2 B300 hicache 运行）的 X 标记恰好落在虚线 roofline 上（C=8 与 C=2 之间的单调样条曲线与其相差不到 0.5%），而其官方对应点已被隐藏——看起来正是"C=4 在叠加层帕累托上（但不在 E2E 上，因为该 X 明显低于 E2E 曲线）"。修复：镜像官方可见性规则（overlayOptimalPoints恒等集合 +isOverlayPointVisible，应用于叠加层渲染、开关装饰 effect、坐标轴域与图例数据点表格）。验证：新增端到端测试
overlay-optimal-only.cy.ts使用 run 29682242847 的真实数值驱动完整渲染管线：默认视图 5 个叠加层 X 中恰好 4 个可见（C=4 隐藏，与官方一致），关闭开关显示 5 个，重新开启后为 4 个；该测试在修复前失败、修复后通过。模块级校验（真实产物 + 已入库行）显示两条路径产出完全相同的前沿（C=48 → C=8 → C=2 → C=1，C=4 被 C=8 支配而排除）。452 个推理单元测试（新增 12 个）、24 个既有叠加层 e2e 测试、typecheck/lint/fmt 全部通过。Note
Medium Risk
Touches core scatter chart rendering (D3 visibility, roofline seeding, axis domains) for both official and overlay paths; risk is mitigated by shared helpers and new e2e/unit coverage rather than auth or data writes.
Overview
Aligns
?unofficialrun=overlay behavior with the official inference charts on agentic views so unofficial X markers land on the same columns, draw the same rooflines, and respect Optimal Only (on by default).Shared plumbing: New
resolveXAxisFieldande2eFrontierhelpers (e2eFrontierWinners,e2eRestrictedSeed) replace duplicated logic inuseChartDataandprocessOverlayChartData. Overlays now use the selected percentile for x (e.g.p90_intvty), stampisOnE2eFrontierper unofficial run when agentic and not in e2e x-mode, and narrow overlay rooflines the same way as official points.User-visible fix: Overlay X markers no longer ignore Optimal Only—
isOverlayPointVisiblehides points not on the drawn overlay roofline (render, decoration effect, scale domains, legend table). This fixes e2e-dominated configs looking pareto when their official twin was hidden.Tests: Cypress
overlay-optimal-only.cy.ts(real run-29682242847 fixtures) plus 12 unit tests onprocessOverlayChartData.Reviewed by Cursor Bugbot for commit ea81b2f. Bugbot is set up for automated code reviews on this repo. Configure here.