diff --git a/docs/state-ownership.md b/docs/state-ownership.md index 0c3eb993c..07e5a2bcf 100644 --- a/docs/state-ownership.md +++ b/docs/state-ownership.md @@ -169,6 +169,8 @@ Does NOT consume `GlobalFilterProvider`. Fully standalone — reliability data h **TCO Calculator** (`calculator` tab): All state is local `useState` inside `ThroughputCalculatorDisplay`. It reads `effectiveSequence` and `effectivePrecisions` from `useGlobalFilters()` for the initial GPU list but does not share state back. +It reads `unofficialBenchmarkRows` / `unofficialRunInfos` / `runIndexByUrl` from `useUnofficialRun()` to render unofficial-run overlay bars, but it neither reads nor writes the shared `activeOverlayHwTypes` that the inference and evaluation tabs use: the calculator's local `visibleHwKeys` governs its official and overlay bars alike. Two visibility sets behind one legend can only drift — see [TCO Calculator › Unofficial-Run Overlays](./tco-calculator.md#unofficial-run-overlays-unofficialrun). + **Historical Trends** (`historical` tab): Rendered inside `InferenceProvider` (shares the `inference` + `historical` `isActive` gate). It reads `useInference()` directly — no additional provider. Uses InferenceContext's model/sequence/precision/date state. **GPU Specs**: Static data, no provider. diff --git a/docs/tco-calculator.md b/docs/tco-calculator.md index 12c5d3124..09fca7df2 100644 --- a/docs/tco-calculator.md +++ b/docs/tco-calculator.md @@ -62,3 +62,44 @@ Adding another context provider to the nesting hierarchy would increase re-rende ## Bar Selection & Comparison Click-to-compare uses `resultKey` (not hwKey) because multi-precision mode produces multiple bars per GPU. Comparison ratios use the lower value as denominator (ratio >= 1.0). Both metric and token type are reflected in the comparison text to avoid ambiguity. + +## Unofficial-Run Overlays (`?unofficialrun=`) + +A loaded unofficial run contributes an extra bar per (hardware × run) to the bar chart, in the run's palette color (`overlayRunColor`) and labeled `B300 (✕ my-branch)`. The label keeps the branch inside the same paren group as the precision so the `twoRowYAxisLabels({ split: 'parens' })` y-axis customizer still splits it into two rows. + +**Overlay results are interpolated separately from official ones.** `useThroughputData` builds two group maps — `gpuDataByGroupKey` (official) and `overlayGpuDataByGroupKey` (per-run, keyed `hwKey[__precision]__run`) — and runs `interpolateForGPU` over each independently. Folding overlay points into the official Pareto front would silently move the official numbers, and you'd lose the before/after delta that makes the overlay useful in the first place. + +Both paths share one row → `GPUDataPoint` mapper, `buildGpuGroups`, so an overlay bar and its official twin can never differ because of a mapping drift. Group identity is carried in `gpuGroupMeta` / `overlayGroupMeta` rather than re-parsed out of the key string; `FleetPlanner` still splits keys itself, which is safe because official keys are unchanged. + +Overlay rows arrive unfiltered by model (the unofficial-run API returns every model in the run, while `/api/v1/benchmarks` is already model-scoped), so the hook filters them with `DB_MODEL_TO_DISPLAY`. + +**Only the bar chart and its legend show overlay data.** The table view, CSV export, and fleet planner deliberately stay official-only — an exported sheet or an MW projection that silently blends in numbers from an unmerged branch is worse than one that omits them. This is why `barResults` (official + overlay) exists separately from `results` (official) and only reaches `ThroughputBarChart`. + +Legend behavior: + +- One entry per run that contributes bars, same shape as the inference/evaluation overlay legends (`✕ `, palette swatch, workflow link). The entry is a label, not a series: per-run removal happens in the banner, so it sets `isRemovable: false` (a default-true opt-out on `CommonLegendItemProps`). Without it those always-active entries inflate `ChartLegend`'s `activeCount`, which is the guard that stops the hide control emptying the chart — and their own hide control would call `removeGpu` with an `overlay-run-*` key and do nothing. +- Hardware entries merge official hardware with hardware only the run has data for (`legendHwKeys`), otherwise an overlay-only bar would be unhideable. +- `visibleHwKeys` is the **single source of truth** for both series: one legend entry governs a GPU's official and overlay bars together. + +That last point is deliberate and worth not "fixing" back. The obvious-looking alternative — read/write the provider's shared `activeOverlayHwTypes` for the overlay series — gives the one legend two backing sets, and every way they drift renders a legend entry that contradicts the bar beside it: + +- the reset effect reseeds `visibleHwKeys` when the available hardware changes but has no business reseeding a set two other tabs share, so a GPU hidden before a model/sequence switch comes back as "active" in the legend with its overlay bar still hidden; +- the inference or evaluation tab re-enabling a GPU resurrects its calculator overlay bar while this tab's legend still marks it inactive. + +Per-tab hardware visibility is already how the calculator treats official data — `visibleHwKeys` has never been shared with the inference tab — so the overlay series just follows the same rule. AGENTS.md's "respect `activeOverlayHwTypes`" exists so overlay points can't ignore a user's hide action; here the calculator's own legend _is_ that hide action. `calculator-overlay.cy.ts` pins the first scenario ("brings hidden overlay bars back when the available hardware changes"). + +### Seeding the legend selection + +Two effects, and the split matters: + +- **Reset** keys on the **official** hardware list (`availableHwKeys`) and reseeds `visibleHwKeys` to the merged list. A run is fetched separately from the benchmarks and usually lands later, so keying the reset on the merged list let a late overlay arrival — or a run dismissal — wipe GPU filters the user had already set. +- **Overlay arrival/departure** is applied **additively**: newly available overlay GPUs start visible, departed ones stop being tracked, everything else keeps whatever the user set. It falls back to all official hardware if the result would be empty, so dismissing a run while an overlay-only GPU was soloed can't leave a blank chart. + +The reset's early-out guards on the **merged** list, not the official one. An empty official list is a real state — the "model/sequence exists only in the run" case this feature is for — and bailing on it would leave the previous selection's official keys in `visibleHwKeys`. `toggleGpuVisibility` therefore also counts visible keys against `legendHwKeys` rather than comparing raw set size, so a stale entry can never skew solo/show-all. + +### Honesty in the tooltip + +- **Clamped values.** `interpolateForGPU` clamps the target into each series' measured range and always returns a value, so a bar can be showing its nearest edge point rather than an interpolation. This is pre-existing across GPUs with different ranges, but widening the slider to cover overlay operating points makes it reachable for every official bar at once — which would turn a side-by-side overlay delta into a real-vs-clamped comparison. Results carry a `clamped` flag and the tooltip says so. (Narrowing the slider back is not the fix: it only moves the clamping onto the overlay bars, and an overlay-only model loses its bounds entirely.) +- **Escaping.** The tooltip is a hand-built HTML string injected with `.html()`, and branch names and run URLs come from the GitHub API for whatever run id the user pasted. Everything untrusted goes through `escapeHtml` (`lib/utils`). The y-axis tick labels render the same branch but go through d3 `.text()`, and the legend entry is React — both already safe. + +Note the calculator only supports fixed-sequence data (`sequenceToIslOsl`: 1k/1k, 1k/8k, 8k/1k). Agentic-traces rows carry null isl/osl and are invisible here for official and unofficial data alike. E2E fixtures therefore use `singleTurnRows` from `cypress/support/overlay-fixtures.ts`, not the agentic `b300Rows` the inference specs use. diff --git a/packages/app/cypress/e2e/calculator-overlay.cy.ts b/packages/app/cypress/e2e/calculator-overlay.cy.ts new file mode 100644 index 000000000..6c7c0c0f6 --- /dev/null +++ b/packages/app/cypress/e2e/calculator-overlay.cy.ts @@ -0,0 +1,283 @@ +/** + * Unofficial-run overlays in the TCO calculator. + * + * A run loaded via `?unofficialrun=` contributes an extra bar per hardware + * config, interpolated separately from the official data so official bars keep + * their own Pareto frontier. The table view, CSV export, and fleet planner stay + * official-only by design — mixing unmerged-branch numbers into an exported + * sheet or a fleet projection would be silently misleading. + * + * Fixtures are fixed-sequence (1k/1k): the calculator resolves the selected + * sequence through `sequenceToIslOsl` and filters rows on isl/osl, so the + * agentic overlay fixtures used by the inference specs never reach it. + */ +import { + ALT_SEQUENCE_LABEL, + interceptCalculatorMultiRunOverlay, + interceptCalculatorOverlayRun, + OVERLAY_ONLY_HARDWARE, + OVERLAY_RUN_BRANCH, + OVERLAY_ONLY_SEQUENCE_LABEL, + OVERLAY_RUN_ID, + SECOND_OFFICIAL_HARDWARE, + SECOND_OVERLAY_RUN_BRANCH, + SECOND_OVERLAY_RUN_ID, +} from '../support/overlay-fixtures'; + +/** Official data covers B300 + B200; the run adds a B300 bar and an MI355X bar. */ +const TOTAL_BARS = 4; +const OVERLAY_BARS = 2; + +const SEQUENCE = '1k/1k'; +const SEQUENCE_LABEL = '1K / 1K'; +const BARS = '[data-testid="calculator-bar-chart"] svg .bar'; +const Y_TICKS = '[data-testid="calculator-bar-chart"] svg .y-axis .tick text'; + +const selectSequence = (label: string) => { + cy.get('[data-testid="calc-sequence-selector"]').click(); + cy.get('[role="option"]').contains(label).click(); +}; + +/** + * `i_seq` is pinned because the global default sequence is 8k/1k, which the + * fixtures also cover (with different hardware) — without pinning, the default + * view would be the alt sequence rather than the overlay-carrying 1k/1k one. + */ +const visitCalculatorWithOverlay = () => { + interceptCalculatorOverlayRun(); + cy.visit(`/calculator?unofficialrun=${OVERLAY_RUN_ID}&i_seq=${encodeURIComponent(SEQUENCE)}`, { + onBeforeLoad(win) { + win.localStorage.setItem('inferencex-star-modal-dismissed', String(Date.now())); + }, + }); + cy.wait('@unofficialRun'); + cy.get(BARS).should('have.length.at.least', 1); +}; + +describe('TCO calculator — unofficial run overlay', () => { + describe('rendering', () => { + before(visitCalculatorWithOverlay); + + it('renders overlay bars alongside the official bar', () => { + cy.get(BARS).should('have.length', TOTAL_BARS); + cy.get(Y_TICKS).should('contain.text', OVERLAY_RUN_BRANCH); + }); + + it('labels the overlay bar with ✕ and the branch, leaving the official bar unmarked', () => { + cy.get(Y_TICKS).then(($ticks) => { + const labels = [...$ticks].map((el) => el.textContent ?? ''); + expect(labels.filter((l) => l.includes('✕'))).to.have.length(OVERLAY_BARS); + expect(labels.filter((l) => !l.includes('✕'))).to.have.length(TOTAL_BARS - OVERLAY_BARS); + }); + }); + + it('paints the overlay bar with the run palette color, not the hardware color', () => { + cy.get(BARS).then(($bars) => { + const fills = [...$bars].map((el) => el.getAttribute('fill') ?? ''); + expect(fills.filter((f) => f.includes('overlay-run-0'))).to.have.length(OVERLAY_BARS); + expect(fills.filter((f) => !f.includes('overlay-run-'))).to.have.length( + TOTAL_BARS - OVERLAY_BARS, + ); + }); + }); + + it('drops overlay rows belonging to a model the calculator is not showing', () => { + // The run payload also carries glm5 rows at 5x throughput. If model + // filtering regressed they'd render as extra bars far off the scale. + cy.get(BARS).should('have.length', TOTAL_BARS); + }); + + it('shows the run in the legend with its palette swatch', () => { + cy.get('.sidebar-legend').should('contain.text', OVERLAY_RUN_BRANCH); + }); + }); + + describe('hardware visibility', () => { + beforeEach(visitCalculatorWithOverlay); + + it('lists overlay-only hardware in the legend', () => { + // MI355X exists only in the run — without the legend merge there'd be no + // way to hide its bar. + cy.get('.sidebar-legend').should('contain.text', OVERLAY_ONLY_HARDWARE.toUpperCase()); + }); + + it('hides a GPU official and overlay bar together when another GPU is soloed', () => { + // Clicking one entry while all are visible solos it. + cy.get('.sidebar-legend label').contains(OVERLAY_ONLY_HARDWARE.toUpperCase()).click(); + // Only the MI355X overlay bar survives — both B300 bars (official AND + // overlay) are gone, proving one legend entry governs both series. + cy.get(BARS).should('have.length', 1); + cy.get(Y_TICKS).should('not.contain.text', 'B300'); + }); + + it('brings hidden overlay bars back when the available hardware changes', () => { + // Regression: overlay visibility used to live in a second, provider-shared + // set that the legend reset did not reseed. Hiding a GPU, then changing + // the selection, left the legend showing it as active while its overlay + // bar stayed hidden by the earlier filter. + cy.get('.sidebar-legend label').contains(OVERLAY_ONLY_HARDWARE.toUpperCase()).click(); + cy.get(BARS).should('have.length', 1); + + // 8k/1k covers different hardware, so switching there and back reseeds + // the legend's available set. + selectSequence(ALT_SEQUENCE_LABEL); + cy.get(BARS).should('have.length', 1); // H100 only, no overlay data + selectSequence(SEQUENCE_LABEL); + + cy.get(BARS).should('have.length', TOTAL_BARS); + }); + + it('does not offer a hide control on the run legend entry', () => { + // Run entries are labels, not series: they must not render the hide "×" + // (it would call removeGpu with an overlay-run-* key and do nothing) nor + // count toward the guard that stops the user emptying the chart. + // Sanity-check the selector against a real GPU entry, so a markup change + // can't turn the assertion below into a no-op. + cy.get(`[aria-label="Hide B300 (SGLang)"]`).should('exist'); + cy.get(`[aria-label="Hide ✕ ${OVERLAY_RUN_BRANCH}"]`).should('not.exist'); + }); + + it('restores every bar via reset filter', () => { + cy.get('.sidebar-legend label').contains(OVERLAY_ONLY_HARDWARE.toUpperCase()).click(); + cy.get(BARS).should('have.length', 1); + cy.contains('button', 'Reset filter').click(); + cy.get(BARS).should('have.length', TOTAL_BARS); + }); + }); + + describe('late overlay arrival', () => { + it('keeps a GPU filter the user set before the run landed', () => { + // Regression: the reset effect keyed on the MERGED official+overlay + // hardware list. The unofficial run is fetched separately and usually + // resolves after the benchmarks, so when it landed and added + // overlay-only hardware the legend reseeded and wiped filters the user + // had already set. Reseeding on a user-driven model/sequence change is + // intentional; reseeding on async overlay arrival is not. + interceptCalculatorOverlayRun({ runDelayMs: 2000 }); + cy.visit( + `/calculator?unofficialrun=${OVERLAY_RUN_ID}&i_seq=${encodeURIComponent(SEQUENCE)}`, + { + onBeforeLoad(win) { + win.localStorage.setItem('inferencex-star-modal-dismissed', String(Date.now())); + }, + }, + ); + + // Official data only, for now: B300 + B200. + cy.get(BARS).should('have.length', 2); + cy.get('.sidebar-legend label').contains('B300').click(); // solo B300 + cy.get(BARS).should('have.length', 1); + + cy.wait('@unofficialRun'); + // The run adds its own B300 bar and the overlay-only MI355X bar, but the + // hidden B200 must stay hidden. + cy.get(BARS).should('have.length', 3); + cy.get(Y_TICKS).should('not.contain.text', SECOND_OFFICIAL_HARDWARE.toUpperCase()); + }); + }); + + describe('sequence covered only by the run', () => { + beforeEach(visitCalculatorWithOverlay); + + it('drops stale official hardware from the legend selection', () => { + // Regression: the reset effect bailed out when the official hardware list + // was empty, treating "no official data for this selection" as "still + // loading". The previous selection's official keys stayed in + // `visibleHwKeys`, so the solo/show-all arithmetic in the toggle counted + // hardware that is not on the chart. + selectSequence(OVERLAY_ONLY_SEQUENCE_LABEL); + cy.get(BARS).should('have.length', 2); // both overlay-only, no official data + cy.get(Y_TICKS).should('not.contain.text', SECOND_OFFICIAL_HARDWARE.toUpperCase()); + + // With a clean selection this solos B300. With stale official keys still + // counted, `allVisible` is false and the same click REMOVES B300 instead, + // leaving MI355X — one bar either way, but the wrong one. + cy.get('.sidebar-legend label').contains('B300').click(); + cy.get(BARS).should('have.length', 1); + cy.get(Y_TICKS).should('contain.text', 'B300'); + cy.get(Y_TICKS).should('not.contain.text', OVERLAY_ONLY_HARDWARE.toUpperCase()); + }); + }); + + describe('dismissing one of several runs on an overlay-only sequence', () => { + it('keeps the surviving run visible instead of blanking the chart', () => { + // Regression: when the additive overlay effect cleared the last visible + // key it fell back to the OFFICIAL hardware list — which is empty on an + // overlay-only sequence, so the chart went blank even though the other + // run still had data. The official-list reset cannot recover it either: + // that list stays empty across the change, so it never reseeds. + interceptCalculatorMultiRunOverlay(); + cy.visit( + `/calculator?unofficialruns=${OVERLAY_RUN_ID},${SECOND_OVERLAY_RUN_ID}` + + `&i_seq=${encodeURIComponent('1k/8k')}`, + { + onBeforeLoad(win) { + win.localStorage.setItem('inferencex-star-modal-dismissed', String(Date.now())); + }, + }, + ); + cy.wait('@unofficialRun'); + + // One bar per run, no official data on this sequence. + cy.get(BARS).should('have.length', 2); + + // Solo the GPU that belongs to the first run. + cy.get('.sidebar-legend label').contains('B300').click(); + cy.get(BARS).should('have.length', 1); + + // Dismissing that run empties the visible set — the fallback has to reach + // for the surviving run's hardware, not the (empty) official list. + cy.get(`[aria-label="Dismiss ${OVERLAY_RUN_BRANCH}"]`).click(); + cy.get(BARS).should('have.length', 1); + cy.get(Y_TICKS).should('contain.text', SECOND_OVERLAY_RUN_BRANCH); + }); + }); + + describe('official-only surfaces', () => { + before(visitCalculatorWithOverlay); + + it('excludes overlay rows from the table view', () => { + cy.get('[data-testid="calculator-table-view-btn"]').click(); + cy.get('[data-testid="calculator-bar-chart"]').should('not.exist'); + cy.get('table').should('not.contain.text', '✕'); + cy.get('table').should('not.contain.text', OVERLAY_RUN_BRANCH); + // MI355X has no official data, so it must not appear in the table either. + cy.get('table').should('not.contain.text', OVERLAY_ONLY_HARDWARE.toUpperCase()); + }); + }); + + describe('Chinese page', () => { + before(() => { + interceptCalculatorOverlayRun(); + cy.visit( + `/zh/calculator?unofficialrun=${OVERLAY_RUN_ID}&i_seq=${encodeURIComponent(SEQUENCE)}`, + { + onBeforeLoad(win) { + win.localStorage.setItem('inferencex-star-modal-dismissed', String(Date.now())); + }, + }, + ); + cy.wait('@unofficialRun'); + cy.get(BARS).should('have.length', TOTAL_BARS); + }); + + it('renders the overlay legend strings in Chinese', () => { + // The branch name itself stays English (it is an identifier); the + // surrounding chrome must not. + cy.get('.sidebar-legend').should('contain.text', OVERLAY_RUN_BRANCH); + cy.get('.sidebar-legend').should('not.contain.text', 'UNOFFICIAL RUN'); + }); + }); + + describe('dismissal', () => { + before(visitCalculatorWithOverlay); + + it('removes the overlay bar when the run is dismissed from the banner', () => { + cy.get(BARS).should('have.length', TOTAL_BARS); + cy.get(`[aria-label="Dismiss ${OVERLAY_RUN_BRANCH}"]`).click(); + cy.get(BARS).should('have.length', TOTAL_BARS - OVERLAY_BARS); + cy.get(Y_TICKS).should('not.contain.text', OVERLAY_RUN_BRANCH); + cy.url().should('not.include', 'unofficialrun'); + }); + }); +}); diff --git a/packages/app/cypress/support/mock-data.ts b/packages/app/cypress/support/mock-data.ts index bcad303fe..43b6c1771 100644 --- a/packages/app/cypress/support/mock-data.ts +++ b/packages/app/cypress/support/mock-data.ts @@ -466,6 +466,7 @@ export function createMockUnofficialRunContext( unofficialRunInfos: [], runIndexByUrl: {}, unofficialChartData: null, + unofficialBenchmarkRows: null, unofficialEvalRows: null, loading: false, error: null, diff --git a/packages/app/cypress/support/overlay-fixtures.ts b/packages/app/cypress/support/overlay-fixtures.ts index ccac55337..14047287f 100644 --- a/packages/app/cypress/support/overlay-fixtures.ts +++ b/packages/app/cypress/support/overlay-fixtures.ts @@ -106,3 +106,263 @@ export const interceptOverlayRun = () => { export const countVisible = ($els: JQuery): number => [...$els].filter((el) => getComputedStyle(el).opacity !== '0').length; + +// --------------------------------------------------------------------------- +// Single-turn (fixed-sequence) overlay fixtures — for the TCO calculator +// --------------------------------------------------------------------------- +// +// The agentic rows above are invisible to the calculator: it resolves the +// selected sequence through `sequenceToIslOsl`, which only maps 1k/1k, 1k/8k +// and 8k/1k, then filters rows on `row.isl`/`row.osl`. Agentic rows carry +// null isl/osl, so they never match. Calculator specs need fixed-sequence rows. + +export const SINGLE_TURN_DATE = '2026-07-19'; +export const SINGLE_TURN_ISL = 1024; +export const SINGLE_TURN_OSL = 1024; +/** A second sequence, covering different hardware — switching to it and back + * changes the calculator's available-hardware set, which reseeds the legend. */ +export const ALT_SEQUENCE_ISL = 8192; +export const ALT_SEQUENCE_OSL = 1024; +export const ALT_SEQUENCE_LABEL = '8K / 1K'; +export const ALT_SEQUENCE_HARDWARE = 'h100'; +/** A model the calculator is NOT showing — used to prove overlay model filtering. */ +export const OTHER_MODEL_DB_KEY = 'glm5'; +/** Hardware present only in the unofficial run, never in the official rows. */ +export const OVERLAY_ONLY_HARDWARE = 'mi355x'; +/** A second official GPU, so a hardware filter set before the run lands is observable. */ +export const SECOND_OFFICIAL_HARDWARE = 'b200'; +/** + * A sequence the unofficial run covers but the DB does not — the "this + * model/sequence exists only in the run" case the overlay feature exists for. + * Selecting it leaves the calculator with zero official hardware. + */ +export const OVERLAY_ONLY_ISL = 1024; +export const OVERLAY_ONLY_OSL = 8192; +export const OVERLAY_ONLY_SEQUENCE_LABEL = '1K / 8K'; + +/** conc, interactivity (tok/s/user), tput_per_gpu */ +export const SINGLE_TURN_CONFIGS: [number, number, number][] = [ + [48, 10.6, 17199], + [8, 68.5, 12874], + [2, 111.1, 5018], + [1, 130.2, 2600], +]; + +export const singleTurnMetrics = (intvty: number, tput: number): Record => ({ + median_intvty: intvty, + median_itl: 1 / intvty, + median_e2el: 30, + median_ttft: 0.5, + tput_per_gpu: tput, + output_tput_per_gpu: tput * 0.3, + input_tput_per_gpu: tput * 0.7, +}); + +let singleTurnIdCursor = 800000; + +/** + * Fixed-sequence (1k/1k) rows for one hardware config. + * + * `tputScale` lets an overlay run report a different throughput than the + * official rows, so the two bars are distinguishable in assertions. + */ +export const singleTurnRows = ( + runUrl: string | null, + { + hardware = 'b300', + model = DEFAULT_MODEL_DB_KEY, + tputScale = 1, + isl = SINGLE_TURN_ISL, + osl = SINGLE_TURN_OSL, + }: { + hardware?: string; + model?: string; + tputScale?: number; + isl?: number; + osl?: number; + } = {}, +) => + SINGLE_TURN_CONFIGS.map(([conc, intvty, tput]) => ({ + id: runUrl ? 0 : singleTurnIdCursor++, + hardware, + framework: 'sglang', + model, + precision: 'fp4', + spec_method: 'none', + disagg: false, + is_multinode: false, + prefill_tp: 8, + decode_tp: 8, + num_prefill_gpu: 8, + num_decode_gpu: 8, + isl, + osl, + conc, + offload_mode: 'off', + benchmark_type: 'single_turn', + image: 'sglang:test', + metrics: singleTurnMetrics(intvty, tput * tputScale), + workers: null, + date: SINGLE_TURN_DATE, + run_url: runUrl, + })); + +export const singleTurnAvailability = [ + { + model: DEFAULT_MODEL_DB_KEY, + isl: SINGLE_TURN_ISL, + osl: SINGLE_TURN_OSL, + precision: 'fp4', + hardware: 'b300', + framework: 'sglang', + spec_method: 'none', + disagg: false, + benchmark_type: 'single_turn', + date: SINGLE_TURN_DATE, + }, + { + model: DEFAULT_MODEL_DB_KEY, + isl: SINGLE_TURN_ISL, + osl: SINGLE_TURN_OSL, + precision: 'fp4', + hardware: SECOND_OFFICIAL_HARDWARE, + framework: 'sglang', + spec_method: 'none', + disagg: false, + benchmark_type: 'single_turn', + date: SINGLE_TURN_DATE, + }, + { + model: DEFAULT_MODEL_DB_KEY, + isl: ALT_SEQUENCE_ISL, + osl: ALT_SEQUENCE_OSL, + precision: 'fp4', + hardware: ALT_SEQUENCE_HARDWARE, + framework: 'sglang', + spec_method: 'none', + disagg: false, + benchmark_type: 'single_turn', + date: SINGLE_TURN_DATE, + }, +]; + +/** + * Intercept availability + benchmarks + unofficial-run with fixed-sequence + * rows the calculator can actually read. + * + * The overlay payload deliberately carries two extra kinds of rows: + * - `OTHER_MODEL_DB_KEY` rows, which must be filtered out — the unofficial-run + * API returns every model in the run, the calculator shows only the selected one. + * - `OVERLAY_ONLY_HARDWARE` rows, hardware with no official data at all, which + * must still get a legend entry so its overlay bar can be hidden. + */ +export const interceptCalculatorOverlayRun = ({ runDelayMs }: { runDelayMs?: number } = {}) => { + cy.intercept('GET', '/api/v1/availability', { body: singleTurnAvailability }).as('availability'); + cy.intercept('GET', '/api/v1/benchmarks*', { + // The route is intercepted regardless of query params; the hook filters by + // isl/osl client-side, so both sequences ship in one payload. + body: [ + ...singleTurnRows(null), + ...singleTurnRows(null, { hardware: SECOND_OFFICIAL_HARDWARE, tputScale: 0.6 }), + ...singleTurnRows(null, { + hardware: ALT_SEQUENCE_HARDWARE, + isl: ALT_SEQUENCE_ISL, + osl: ALT_SEQUENCE_OSL, + }), + ], + }).as('benchmarks'); + cy.intercept('GET', '/api/unofficial-run*', { + // A delay lets a spec set a GPU filter in the window after the official + // benchmarks land but before the run does — the real-world ordering. + delay: runDelayMs, + body: { + runInfos: [ + { + id: OVERLAY_RUN_ID, + name: OVERLAY_RUN_BRANCH, + branch: OVERLAY_RUN_BRANCH, + sha: 'abc000', + createdAt: `${SINGLE_TURN_DATE}T00:00:00Z`, + url: OVERLAY_RUN_URL, + conclusion: 'success', + status: 'completed', + isNonMainBranch: true, + }, + ], + benchmarks: [ + ...singleTurnRows(OVERLAY_RUN_URL, { tputScale: 1.3 }), + ...singleTurnRows(OVERLAY_RUN_URL, { hardware: OVERLAY_ONLY_HARDWARE, tputScale: 0.8 }), + ...singleTurnRows(OVERLAY_RUN_URL, { model: OTHER_MODEL_DB_KEY, tputScale: 5 }), + // 1k/8k: covered by the run only, so selecting it leaves zero official + // hardware. Same two GPUs as 1k/1k, which keeps `overlayAvailableHwKeys` + // unchanged across the switch — the adversarial case, since only the + // official-list reset can clear the now-stale official keys. + ...singleTurnRows(OVERLAY_RUN_URL, { + tputScale: 1.1, + isl: OVERLAY_ONLY_ISL, + osl: OVERLAY_ONLY_OSL, + }), + ...singleTurnRows(OVERLAY_RUN_URL, { + hardware: OVERLAY_ONLY_HARDWARE, + tputScale: 0.9, + isl: OVERLAY_ONLY_ISL, + osl: OVERLAY_ONLY_OSL, + }), + ], + evaluations: [], + }, + }).as('unofficialRun'); +}; + +/** Second run id, for multi-run (`?unofficialruns=a,b`) specs. */ +export const SECOND_OVERLAY_RUN_ID = '29682242848'; +export const SECOND_OVERLAY_RUN_BRANCH = 'perf/second-branch'; +export const SECOND_OVERLAY_RUN_URL = `https://github.com/SemiAnalysisAI/InferenceX/actions/runs/${SECOND_OVERLAY_RUN_ID}`; + +const runInfoFor = (id: string, branch: string, url: string) => ({ + id, + name: branch, + branch, + sha: 'abc000', + createdAt: `${SINGLE_TURN_DATE}T00:00:00Z`, + url, + conclusion: 'success', + status: 'completed', + isNonMainBranch: true, +}); + +/** + * Two runs, each contributing ONE hardware config on the 1k/8k sequence — a + * sequence the DB does not cover, so the calculator has zero official hardware + * there. + * + * That combination is what exercises the "don't strand an empty chart" fallback + * in the additive overlay effect: soloing the GPU from run A and then dismissing + * run A empties the visible set while run B's bar still has data, and the + * official hardware list (empty here) is not something the fallback can use. + */ +export const interceptCalculatorMultiRunOverlay = () => { + cy.intercept('GET', '/api/v1/availability', { body: singleTurnAvailability }).as('availability'); + cy.intercept('GET', '/api/v1/benchmarks*', { body: singleTurnRows(null) }).as('benchmarks'); + cy.intercept('GET', '/api/unofficial-run*', { + body: { + runInfos: [ + runInfoFor(OVERLAY_RUN_ID, OVERLAY_RUN_BRANCH, OVERLAY_RUN_URL), + runInfoFor(SECOND_OVERLAY_RUN_ID, SECOND_OVERLAY_RUN_BRANCH, SECOND_OVERLAY_RUN_URL), + ], + benchmarks: [ + ...singleTurnRows(OVERLAY_RUN_URL, { + isl: OVERLAY_ONLY_ISL, + osl: OVERLAY_ONLY_OSL, + }), + ...singleTurnRows(SECOND_OVERLAY_RUN_URL, { + hardware: OVERLAY_ONLY_HARDWARE, + isl: OVERLAY_ONLY_ISL, + osl: OVERLAY_ONLY_OSL, + tputScale: 0.7, + }), + ], + evaluations: [], + }, + }).as('unofficialRun'); +}; diff --git a/packages/app/src/components/calculator/ThroughputBarChart.test.ts b/packages/app/src/components/calculator/ThroughputBarChart.test.ts index 7fae21d6a..664775fee 100644 --- a/packages/app/src/components/calculator/ThroughputBarChart.test.ts +++ b/packages/app/src/components/calculator/ThroughputBarChart.test.ts @@ -1,7 +1,11 @@ import { describe, expect, it } from 'vitest'; +import { splitLabel } from '@/lib/d3-chart/axis-labels'; + import type { InterpolatedResult } from './types'; import { + generateTooltipHTML, + getResultLabel, getCostForType, getCostProviderLabel, getCostTypeLabel, @@ -494,3 +498,127 @@ describe('getSortedResults', () => { expect(sorted[0].hwKey).toBe('x'); }); }); + +// ========================================================================= +// getResultLabel() — official + unofficial-run overlay bars +// ========================================================================= + +describe('getResultLabel', () => { + // Empty config → the helper falls back to the global getHardwareConfig + // lookup, which resolves 'b300' to its registry display label. + const noConfig = {}; + + it('returns the bare hardware name for a single-precision official bar', () => { + expect(getResultLabel(makeResult({ hwKey: 'b300' }), noConfig)).toBe('B300'); + }); + + it('appends the precision when multiple precisions are selected', () => { + expect(getResultLabel(makeResult({ hwKey: 'b300', precision: 'fp4' }), noConfig)).toBe( + 'B300 (FP4)', + ); + }); + + it('marks an overlay bar with ✕ and the branch name', () => { + const label = getResultLabel( + makeResult({ hwKey: 'b300', isOverlay: true, runIndex: 0, runLabel: 'feat/my-branch' }), + noConfig, + ); + expect(label).toBe('B300 (✕ feat/my-branch)'); + }); + + it('combines precision and branch inside a single paren group', () => { + const label = getResultLabel( + makeResult({ + hwKey: 'b300', + precision: 'fp8', + isOverlay: true, + runIndex: 1, + runLabel: 'feat/my-branch', + }), + noConfig, + ); + // One paren group only — twoRowYAxisLabels({split:'parens'}) splits on the + // last "(...)", so a second group would break the two-row y-axis label. + expect(label).toBe('B300 (FP8 · ✕ feat/my-branch)'); + expect(splitLabel(label, 'parens')).toEqual(['B300', '(FP8 · ✕ feat/my-branch)']); + }); + + it('falls back to "unofficial" when the run has no branch name', () => { + expect(getResultLabel(makeResult({ hwKey: 'b300', isOverlay: true }), noConfig)).toBe( + 'B300 (✕ unofficial)', + ); + }); +}); + +// ========================================================================= +// generateTooltipHTML() — overlay treatment +// ========================================================================= + +describe('generateTooltipHTML overlay treatment', () => { + const officialRunUrl = 'https://github.com/org/repo/actions/runs/999'; + const overlayRunUrl = 'https://github.com/org/repo/actions/runs/111'; + + it('omits the unofficial header for official bars', () => { + const html = generateTooltipHTML( + makeResult({ hwKey: 'b300' }), + {}, + 'interactivity_to_throughput', + 'throughput', + 'total', + officialRunUrl, + ); + expect(html).not.toContain('UNOFFICIAL RUN'); + expect(html).toContain(officialRunUrl); + }); + + it('adds the unofficial header, branch, and run link for overlay bars', () => { + const html = generateTooltipHTML( + makeResult({ + hwKey: 'b300', + isOverlay: true, + runIndex: 0, + runLabel: 'feat/my-branch', + runUrl: overlayRunUrl, + }), + {}, + 'interactivity_to_throughput', + 'throughput', + 'total', + officialRunUrl, + ); + expect(html).toContain('UNOFFICIAL RUN'); + expect(html).toContain('feat/my-branch'); + // Links to the overlay's own workflow run, not the official one behind the + // DB data — the two are unrelated. + expect(html).toContain(overlayRunUrl); + expect(html).not.toContain(officialRunUrl); + }); + + it('localizes the overlay strings when Chinese strings are supplied', () => { + const html = generateTooltipHTML( + makeResult({ + hwKey: 'b300', + isOverlay: true, + runIndex: 0, + runLabel: 'feat/my-branch', + runUrl: overlayRunUrl, + }), + {}, + 'interactivity_to_throughput', + 'throughput', + 'total', + undefined, + false, + { + unofficialRun: '非官方运行', + branch: '分支', + viewRun: '查看工作流运行', + clamped: '超出实测范围——显示最接近的数据点', + }, + ); + expect(html).toContain('非官方运行'); + expect(html).toContain('分支'); + expect(html).toContain('查看工作流运行'); + expect(html).not.toContain('UNOFFICIAL RUN'); + }); +}); diff --git a/packages/app/src/components/calculator/ThroughputBarChart.tsx b/packages/app/src/components/calculator/ThroughputBarChart.tsx index de3c145ba..0310a7f82 100644 --- a/packages/app/src/components/calculator/ThroughputBarChart.tsx +++ b/packages/app/src/components/calculator/ThroughputBarChart.tsx @@ -8,6 +8,7 @@ import { useLocale } from '@/lib/use-locale'; import type { HardwareConfig } from '@/components/inference/types'; import { getHardwareConfig } from '@/lib/constants'; import { getChartWatermark } from '@/lib/data-mappings'; +import { overlayRunColor } from '@/lib/overlay-run-style'; import { contrastColors } from '@/lib/d3-chart/contrast-colors'; import { computeLeftMargin, measureTextWidth } from '@/lib/d3-chart/dynamic-margins'; import { twoRowYAxisLabels } from '@/lib/d3-chart/axis-labels'; @@ -19,7 +20,7 @@ import type { RenderContext, } from '@/lib/d3-chart/D3Chart/types'; import type { ContinuousScale } from '@/lib/d3-chart/types'; -import { getDisplayLabel } from '@/lib/utils'; +import { escapeHtml, getDisplayLabel } from '@/lib/utils'; import type { BarMetric, @@ -29,6 +30,27 @@ import type { InterpolatedResult, } from './types'; +/** + * Overlay-only tooltip strings. The rest of the tooltip is English-only today; + * these are new user-visible strings, so they ship with a Chinese version. + */ +const OVERLAY_STRINGS = { + en: { + unofficialRun: 'UNOFFICIAL RUN', + branch: 'Branch', + viewRun: 'View workflow run', + clamped: 'Outside measured range — showing nearest data point', + }, + zh: { + unofficialRun: '非官方运行', + branch: '分支', + viewRun: '查看工作流运行', + clamped: '超出实测范围——显示最接近的数据点', + }, +} as const; + +export type OverlayTooltipStrings = (typeof OVERLAY_STRINGS)[keyof typeof OVERLAY_STRINGS]; + interface ThroughputBarChartProps { results: InterpolatedResult[]; hardwareConfig: HardwareConfig; @@ -201,6 +223,23 @@ export function getCostTypeLabel(costType: CostType): string { return '/M tok'; } +/** + * Display label for a result: `B300`, `B300 (FP4)` when multiple precisions are + * selected, and `B300 (✕ my-branch)` / `B300 (FP4 · ✕ my-branch)` for an + * unofficial-run overlay bar. + * + * The suffix always stays inside one pair of parens so the y-axis customizer + * `twoRowYAxisLabels({ split: 'parens' })` keeps splitting it into two rows. + */ +export function getResultLabel(d: InterpolatedResult, hardwareConfig: HardwareConfig): string { + const config = hardwareConfig[d.hwKey] || getHardwareConfig(d.hwKey); + const baseName = config ? getDisplayLabel(config) : d.hwKey; + const parts: string[] = []; + if (d.precision) parts.push(d.precision.toUpperCase()); + if (d.isOverlay) parts.push(`✕ ${d.runLabel ?? 'unofficial'}`); + return parts.length > 0 ? `${baseName} (${parts.join(' · ')})` : baseName; +} + export function generateTooltipHTML( d: InterpolatedResult, hardwareConfig: HardwareConfig, @@ -209,10 +248,12 @@ export function generateTooltipHTML( costType: CostType, runUrl?: string, isPinned?: boolean, + overlayStrings: OverlayTooltipStrings = OVERLAY_STRINGS.en, ): string { - const config = hardwareConfig[d.hwKey] || getHardwareConfig(d.hwKey); - const baseName = config ? getDisplayLabel(config) : d.hwKey; - const label = d.precision ? `${baseName} (${d.precision.toUpperCase()})` : baseName; + // Everything interpolated below is escaped if it can carry text this codebase + // did not author — today that is the overlay branch name and run URL, which + // come from the GitHub API for whatever run id the user pasted. + const label = escapeHtml(getResultLabel(d, hardwareConfig)); const costLabel = getCostTypeLabel(costType); const costValue = getCostForType(d, costType); @@ -257,16 +298,38 @@ export function generateTooltipHTML( ? `$${metricValue.toFixed(3)}${metricUnit}` : `${metricValue.toFixed(barMetric === 'power' ? 0 : 1)} ${metricUnit}`; - const runLinkHtml = runUrl - ? `` + // Overlay bars link to their own workflow run, not the official run behind + // the DB data — the two are unrelated. + const effectiveRunUrl = d.isOverlay ? d.runUrl : runUrl; + const runLinkLabel = d.isOverlay ? overlayStrings.viewRun : 'View raw result on GitHub'; + const runLinkHtml = effectiveRunUrl + ? `` + : ''; + + // The target can sit outside a given series' measured range — series ranges + // differ, and a loaded unofficial run widens the slider to cover its own + // operating points. Say so rather than letting a clamped edge value read as a + // measurement at the current target. + const clampedHtml = d.clamped + ? `
${overlayStrings.clamped}
` + : ''; + + const overlayBranchHtml = + d.isOverlay && d.runLabel + ? `
${overlayStrings.branch}: ${escapeHtml(d.runLabel)}
` + : ''; + const overlayHeaderHtml = d.isOverlay + ? `
${overlayStrings.unofficialRun}
${overlayBranchHtml}` : ''; return `
${isPinned ? '
Click elsewhere to dismiss
' : ''} + ${overlayHeaderHtml}
${label}
+ ${clampedHtml}
${metricName}: ${metricDisplay}
@@ -279,7 +342,7 @@ export function generateTooltipHTML(
Concurrency: ~${d.concurrency}
- ${precision ? `
Precision: ${precision.toUpperCase()}
` : ''} + ${precision ? `
Precision: ${escapeHtml(precision.toUpperCase())}
` : ''} ${parallelismHtml} ${disagg ? '
Disaggregated: Yes
' : ''} ${runLinkHtml} @@ -289,12 +352,7 @@ export function generateTooltipHTML( // ── Helpers at module scope for use in memos and layers ── -function getLabel(d: InterpolatedResult, hardwareConfig: HardwareConfig): string { - const config = hardwareConfig[d.hwKey] || getHardwareConfig(d.hwKey); - const baseName = config ? getDisplayLabel(config) : d.hwKey; - if (d.precision) return `${baseName} (${d.precision.toUpperCase()})`; - return baseName; -} +const getLabel = getResultLabel; function getColor(): string { return 'var(--foreground)'; @@ -357,9 +415,17 @@ export default function ThroughputBarChart({ colorResolver, }: ThroughputBarChartProps) { const chartRef = useRef(null); + const locale = useLocale(); - // Color resolution: prefer dynamic colorResolver, fall back to static config - const resolveBarColor = (hwKey: string) => (colorResolver ? colorResolver(hwKey) : getColor()); + // Color resolution: unofficial-run overlay bars take the run's palette color + // (so they match the banner + legend swatch — see lib/overlay-run-style.ts); + // official bars prefer the dynamic colorResolver, falling back to static config. + const resolveBarColor = (d: InterpolatedResult) => + d.isOverlay + ? overlayRunColor(d.runIndex ?? 0) + : colorResolver + ? colorResolver(d.hwKey) + : getColor(); // Stable refs to avoid re-running the D3 effect const hoveredBarXRef = useRef(0); @@ -414,7 +480,7 @@ export default function ThroughputBarChart({ config: { getY: (d) => d.resultKey, getX: (d) => getMetricValue(d, barMetric, costType), - getColor: (d) => resolveBarColor(d.hwKey), + getColor: (d) => resolveBarColor(d), rx: 4, opacity: 0.85, keyFn: (d) => d.resultKey, @@ -466,12 +532,12 @@ export default function ThroughputBarChart({ }); // Position both labels together using the longer text width - const barColor = (d: InterpolatedResult) => resolveBarColor(d.hwKey); + const barColor = (d: InterpolatedResult) => resolveBarColor(d); positionLabelPairs(zoomGroup, xScale, ctx.width, barMetric, costType, barColor); }, onZoom: (zoomGroup, ctx) => { const newXScale = ctx.newXScale as d3.ScaleLinear; - const barColor = (d: InterpolatedResult) => resolveBarColor(d.hwKey); + const barColor = (d: InterpolatedResult) => resolveBarColor(d); positionLabelPairs(zoomGroup, newXScale, ctx.width, barMetric, costType, barColor); }, }; @@ -485,7 +551,16 @@ export default function ThroughputBarChart({ () => ({ rulerType: 'vertical' as const, content: (d: InterpolatedResult, isPinned: boolean) => - generateTooltipHTML(d, hardwareConfig, mode, barMetric, costType, runUrl, isPinned), + generateTooltipHTML( + d, + hardwareConfig, + mode, + barMetric, + costType, + runUrl, + isPinned, + OVERLAY_STRINGS[locale], + ), getRulerX: () => hoveredBarXRef.current, onHoverStart: (sel: d3.Selection) => { hoveredBarXRef.current = parseFloat(sel.attr('width') || '0'); @@ -507,7 +582,7 @@ export default function ThroughputBarChart({ }, attachToLayer: 0, }), - [hardwareConfig, mode, barMetric, costType, runUrl], + [hardwareConfig, mode, barMetric, costType, runUrl, locale], ); // ── Y axis customize: map resultKey → display label, then split into two-line GPU labels ── @@ -555,8 +630,6 @@ export default function ThroughputBarChart({ applySelectionOpacities(svg as any, selectedBars); }, [selectedBars]); - const locale = useLocale(); - if (results.length === 0) { return (
opt); }, [locale, viewModeOptions]); + // Unofficial-run overlay (`?unofficialrun=…`). Overlay bars are interpolated + // separately from official ones and only ever reach the bar chart — the + // table, CSV export, and fleet planner stay official-only. + const { isUnofficialRun, unofficialBenchmarkRows, unofficialRunInfos, runIndexByUrl } = + useUnofficialRun(); + + const overlayInput = useMemo( + () => ({ rows: unofficialBenchmarkRows, runIndexByUrl }), + [unofficialBenchmarkRows, runIndexByUrl], + ); + const { gpuDataByGroupKey, hardwareConfig, ranges, getResults, + getOverlayResults, loading, error, hasData, + hasOverlayData, availableHwKeys, - } = useThroughputData(selectedModel, selectedSequence, selectedPrecisions, selectedRunDate); + overlayAvailableHwKeys, + } = useThroughputData( + selectedModel, + selectedSequence, + selectedPrecisions, + selectedRunDate, + overlayInput, + ); + + /** + * Hardware listed in the legend: official hardware, plus hardware that only + * the loaded unofficial run has data for (otherwise there'd be no way to hide + * an overlay-only bar). + * + * `visibleHwKeys` — seeded from this list — is the SINGLE source of truth for + * what the calculator draws, official bars and overlay bars alike. It is + * deliberately not cross-wired to the provider's shared `activeOverlayHwTypes` + * (which the inference and evaluation tabs read/write): two visibility sets + * for one legend can only drift, and every way they drift renders a legend + * entry whose active state contradicts the bar next to it — e.g. a selection + * change reseeds the local set but not the shared one, or another tab + * re-enables a GPU this tab has hidden. + * + * Per-tab hardware visibility is already how the calculator treats official + * data (it has never shared `visibleHwKeys` with the inference tab), so the + * overlay series simply follows the same rule. AGENTS.md's "respect + * `activeOverlayHwTypes`" exists so overlay points can't ignore the user's + * hide action; here the calculator's own legend IS that hide action, and it + * is respected. + */ + const legendHwKeys = useMemo(() => { + if (!isUnofficialRun || overlayAvailableHwKeys.length === 0) return availableHwKeys; + return [...new Set([...availableHwKeys, ...overlayAvailableHwKeys])]; + }, [isUnofficialRun, availableHwKeys, overlayAvailableHwKeys]); // Dynamic vendor-aware colors for visible GPUs const visibleKeysArray = useMemo(() => [...visibleHwKeys], [visibleHwKeys]); @@ -306,33 +361,112 @@ function ThroughputCalculatorInner() { // Track previous available keys to detect when the GPU set changes const prevAvailableKeyRef = useRef(''); + const prevOverlayKeyRef = useRef(''); - // Reset visible GPUs when the available set changes (model/sequence/precision change or customer filter toggle) + // Reset visible GPUs on a user-driven selection change. The key is the + // selection itself PLUS the official hardware list — the selection so an + // overlay-only model/sequence (where the official list is empty and stays + // empty) still reseeds, the official list so anything else that changes which + // GPUs have data still reseeds. Deliberately NOT keyed on the merged list: an + // unofficial run is fetched separately and usually lands after the + // benchmarks, so a late arrival — or a run dismissal — would otherwise wipe + // GPU filters the user had already set. + const selectionKey = `${selectedModel}|${selectedSequence}|${[...selectedPrecisions] + .toSorted() + .join(',')}|${selectedRunDate}|${[...availableHwKeys].toSorted().join(',')}`; useEffect(() => { - if (availableHwKeys.length === 0) return; - const key = [...availableHwKeys].toSorted().join(','); - if (key !== prevAvailableKeyRef.current) { - prevAvailableKeyRef.current = key; - setVisibleHwKeys(new Set(availableHwKeys)); + // Nothing to seed from yet (first load, before either source has resolved). + // Guards on the MERGED list: an empty official list is a real state, not + // just a loading one, and bailing on it would leave stale official keys in + // `visibleHwKeys` and throw off the solo/show-all arithmetic below. + if (legendHwKeys.length === 0) return; + if (selectionKey !== prevAvailableKeyRef.current) { + prevAvailableKeyRef.current = selectionKey; + setVisibleHwKeys(new Set(legendHwKeys)); } - }, [availableHwKeys]); + }, [selectionKey, legendHwKeys]); + + // Overlay hardware arriving or leaving is additive: newly available overlay + // GPUs start visible, ones that are gone stop being tracked, and every other + // entry keeps whatever the user set. + useEffect(() => { + const key = overlayAvailableHwKeys.join(','); + if (key === prevOverlayKeyRef.current) return; + const prev = prevOverlayKeyRef.current ? prevOverlayKeyRef.current.split(',') : []; + prevOverlayKeyRef.current = key; + + const added = overlayAvailableHwKeys.filter((k) => !prev.includes(k)); + // Only drop hardware that has no official data either — otherwise dismissing + // a run would hide a GPU whose official bar is still on the chart. + const removed = prev.filter( + (k) => !overlayAvailableHwKeys.includes(k) && !availableHwKeys.includes(k), + ); + if (added.length === 0 && removed.length === 0) return; + + setVisibleHwKeys((cur) => { + const next = new Set(cur); + added.forEach((k) => next.add(k)); + removed.forEach((k) => next.delete(k)); + // Never strand the user with an empty chart. Falls back to everything + // that still has data, official AND overlay — on an overlay-only + // selection the official list is empty, so falling back to it would blank + // the chart while overlay bars were still available. + if (next.size === 0) return new Set([...availableHwKeys, ...overlayAvailableHwKeys]); + return next; + }); + }, [overlayAvailableHwKeys, availableHwKeys]); + + const hasAnyData = hasData || hasOverlayData; // Clamp target into range when data changes useEffect(() => { - if (!hasData) return; + if (!hasAnyData) return; const { min, max } = ranges.interactivity; if (targetValue < min || targetValue > max) { const clamped = Math.max(min, Math.min(max, targetValue)); setTargetValue(clamped); setInputValue(String(clamped)); } - }, [hasData, ranges]); + }, [hasAnyData, ranges]); const results: InterpolatedResult[] = useMemo(() => { if (!hasData) return []; return getResults(targetValue, mode, costProvider, visibleHwKeys); }, [hasData, targetValue, mode, costProvider, getResults, visibleHwKeys]); + /** Branch + URL per run index, stamped onto overlay results for labels/tooltips. */ + const runInfoByIndex = useMemo(() => { + const map: Record = {}; + unofficialRunInfos.forEach((info, idx) => { + map[idx] = { branch: info.branch || `run ${info.id}`, url: info.url }; + }); + return map; + }, [unofficialRunInfos]); + + const overlayResults: InterpolatedResult[] = useMemo(() => { + if (!hasOverlayData) return []; + return getOverlayResults(targetValue, mode, costProvider, visibleHwKeys, runInfoByIndex); + }, [ + hasOverlayData, + targetValue, + mode, + costProvider, + getOverlayResults, + visibleHwKeys, + runInfoByIndex, + ]); + + /** + * Bars drawn in the chart: official + overlay. Deliberately NOT used by the + * table, the CSV export, or the fleet planner — those stay official-only, so + * an exported sheet or a fleet projection never silently mixes in numbers + * from an unmerged branch. + */ + const barResults = useMemo( + () => (overlayResults.length > 0 ? [...results, ...overlayResults] : results), + [results, overlayResults], + ); + const currentRange = useMemo(() => ranges.interactivity, [ranges]); const handleSliderChange = useCallback((e: React.ChangeEvent) => { @@ -404,16 +538,19 @@ function ThroughputCalculatorInner() { const toggleGpuVisibility = useCallback( (hwKey: string) => { setVisibleHwKeys((prev) => { - const allVisible = prev.size === availableHwKeys.length; + // Count against the legend rather than the raw set size, so an entry + // that is no longer in the legend can never skew solo/show-all. + const visibleLegendKeys = legendHwKeys.filter((k) => prev.has(k)); + const allVisible = visibleLegendKeys.length === legendHwKeys.length; const isVisible = prev.has(hwKey); if (isVisible) { if (allVisible) { // If all visible and clicking one, solo it return new Set([hwKey]); - } else if (prev.size === 1) { + } else if (visibleLegendKeys.length === 1) { // If only one visible and clicking it, show all - return new Set(availableHwKeys); + return new Set(legendHwKeys); } // Remove it const next = new Set(prev); @@ -426,7 +563,7 @@ function ThroughputCalculatorInner() { }); track('calculator_gpu_toggled', { gpu: hwKey }); }, - [availableHwKeys], + [legendHwKeys], ); const removeGpu = useCallback((hwKey: string) => { @@ -451,9 +588,9 @@ function ThroughputCalculatorInner() { }, []); const handleResetGpus = useCallback(() => { - setVisibleHwKeys(new Set(availableHwKeys)); - track('calculator_gpu_reset', { gpuCount: availableHwKeys.length }); - }, [availableHwKeys]); + setVisibleHwKeys(new Set(legendHwKeys)); + track('calculator_gpu_reset', { gpuCount: legendHwKeys.length }); + }, [legendHwKeys]); // Derive runUrl from workflowInfo for the selected sequence const runUrl = useMemo(() => { @@ -480,21 +617,17 @@ function ThroughputCalculatorInner() { // Clear bar selection when results change (data/filter changes) useEffect(() => { setSelectedBars(new Set()); - }, [results]); + }, [barResults]); - // Generate comparison text when 2+ bars are selected + // Generate comparison text when 2+ bars are selected. Overlay bars are + // selectable too, so this reads the combined chart list. const comparisonText = useMemo(() => { if (selectedBars.size < 2) return null; - const selectedResults = results.filter((r) => selectedBars.has(r.resultKey)); + const selectedResults = barResults.filter((r) => selectedBars.has(r.resultKey)); if (selectedResults.length < 2) return null; - const getLabel = (r: InterpolatedResult) => { - const config = hardwareConfig[r.hwKey] || getHardwareConfig(r.hwKey); - const baseName = config ? getDisplayLabel(config) : r.hwKey; - if (r.precision) return `${baseName} (${r.precision.toUpperCase()})`; - return baseName; - }; + const getLabel = (r: InterpolatedResult) => getResultLabel(r, hardwareConfig); const metricName = barMetric === 'power' @@ -551,24 +684,76 @@ function ThroughputCalculatorInner() { } return comparisons; - }, [selectedBars, results, hardwareConfig, barMetric, costType, mode, locale, t]); + }, [selectedBars, barResults, hardwareConfig, barMetric, costType, mode, locale, t]); + + /** + * Overlay legend: one entry per loaded unofficial run that contributes bars + * to the chart, in the same palette color as its bars. Same shape as the + * inference scatter and evaluation bar chart legends. + */ + const overlayLegendItems = useMemo(() => { + if (overlayResults.length === 0) return []; + return unofficialRunInfos + .map((info, idx) => { + if (!overlayResults.some((r) => r.runIndex === idx)) return null; + const branch = info.branch || `run ${info.id}`; + return { + name: `✕ unofficial-run-${info.id}`, + label: `✕ ${branch}`, + color: overlayRunColor(idx), + title: `${t.unofficialRun}: ${branch}`, + isHighlighted: true, + hw: `overlay-run-${info.id}`, + isActive: true, + // A label, not a series: dismissing a run happens in the banner, and + // counting it as removable would let the hide control empty the chart + // of real GPUs. + isRemovable: false, + onClick: () => {}, + tooltip: ( +
+
{t.unofficialRun}
+
+ {t.branch}: {branch} +
+ {info.url && ( + + {t.viewRun} + + )} +
+ ), + }; + }) + .filter((item): item is NonNullable => item !== null); + }, [overlayResults, unofficialRunInfos, t]); // Build legend items for ChartLegend sidebar, sorted by MODEL_ORDER (same as Inference Performance tab) const legendItems = useMemo(() => { - const availableSet = new Set(availableHwKeys); - return Object.entries(hardwareConfig) - .filter(([key]) => availableSet.has(key)) - .toSorted(([a], [b]) => getModelSortIndex(a) - getModelSortIndex(b) || a.localeCompare(b)) - .map(([key, config]) => ({ - name: config.name, - label: getDisplayLabel(config), - color: resolveColor(key), - title: config.gpu, - hw: key, - isActive: visibleHwKeys.has(key), - onClick: () => toggleGpuVisibility(key), - })); - }, [availableHwKeys, hardwareConfig, visibleHwKeys, toggleGpuVisibility, resolveColor]); + const availableSet = new Set(legendHwKeys); + return [ + ...overlayLegendItems, + ...Object.entries(hardwareConfig) + .filter(([key]) => availableSet.has(key)) + .toSorted(([a], [b]) => getModelSortIndex(a) - getModelSortIndex(b) || a.localeCompare(b)) + .map(([key, config]) => ({ + name: config.name, + label: getDisplayLabel(config), + color: resolveColor(key), + title: config.gpu, + hw: key, + isActive: visibleHwKeys.has(key), + onClick: () => toggleGpuVisibility(key), + })), + ]; + }, [ + legendHwKeys, + overlayLegendItems, + hardwareConfig, + visibleHwKeys, + toggleGpuVisibility, + resolveColor, + ]); if (!loading && error) { console.error(error); @@ -719,7 +904,7 @@ function ThroughputCalculatorInner() {
{/* Target value slider + input */} - {!loading && hasData && ( + {!loading && hasAnyData && (
)}

- {barMetric === 'power' && results.length > 0 && ( + {barMetric === 'power' && barResults.length > 0 && ( <>

)} - {barMetric === 'cost' && results.length > 0 && ( + {barMetric === 'cost' && barResults.length > 0 && ( <>

0 ? ( + legendHwKeys.length > 0 ? ( {(() => { const resultKey = [...selectedBars][0]; - const r = results.find((res) => res.resultKey === resultKey); + const r = barResults.find((res) => res.resultKey === resultKey); if (!r) return resultKey; - const config = hardwareConfig[r.hwKey] || getHardwareConfig(r.hwKey); - const baseName = config ? getDisplayLabel(config) : r.hwKey; - return r.precision ? `${baseName} (${r.precision.toUpperCase()})` : baseName; + return getResultLabel(r, hardwareConfig); })()}{' '} {t.clickToCompare}

diff --git a/packages/app/src/components/calculator/interpolation.ts b/packages/app/src/components/calculator/interpolation.ts index 74533bd18..950ca0158 100644 --- a/packages/app/src/components/calculator/interpolation.ts +++ b/packages/app/src/components/calculator/interpolation.ts @@ -265,6 +265,12 @@ export function interpolateForGPU( // Clamp target value to the data range to avoid null returns and prevent extrapolation const clampedTarget = Math.max(minInput, Math.min(maxInput, targetValue)); + // Surfaced on the result so callers can tell the user this series was NOT + // measured at the requested target — it is showing its nearest edge point. + // Series can have different ranges (and an unofficial run can widen the + // slider past every official point), so a clamped bar sitting next to an + // unclamped one is a comparison the user needs to be able to see. + const clamped = targetValue < minInput || targetValue > maxInput; if (sorted.length === 1) { return { @@ -281,6 +287,7 @@ export function interpolateForGPU( outputTpPerMw: sorted[0].outputTpPerMw, concurrency: sorted[0].concurrency, nearestPoints: [sorted[0]], + clamped, }; } @@ -332,5 +339,6 @@ export function interpolateForGPU( outputTpPerMw, concurrency, nearestPoints: [sorted[lowerIdx], sorted[upperIdx]], + clamped, }; } diff --git a/packages/app/src/components/calculator/types.ts b/packages/app/src/components/calculator/types.ts index d9c7a2bdb..c47c48311 100644 --- a/packages/app/src/components/calculator/types.ts +++ b/packages/app/src/components/calculator/types.ts @@ -47,4 +47,20 @@ export interface InterpolatedResult { outputTpPerMw: number; // output throughput per megawatt at that operating point concurrency: number; // concurrency at that operating point nearestPoints: GPUDataPoint[]; // the data points used for interpolation + /** + * True when the requested target fell outside this series' measured range and + * the value is its nearest edge point rather than an interpolation. Shown in + * the tooltip so a clamped bar isn't read as measured at the current target. + */ + clamped?: boolean; + /** + * True when this result was interpolated from an unofficial-run overlay + * (`?unofficialrun=…`) rather than official DB data. Overlay results are + * rendered in the run's palette color and never mixed into the official + * Pareto frontier. + */ + isOverlay?: boolean; + runIndex?: number; // position of the run in the loaded set — drives the palette color + runLabel?: string; // branch name (or `run ` fallback) shown in labels/tooltips + runUrl?: string; // GitHub Actions run URL, linked from the overlay tooltip } diff --git a/packages/app/src/components/calculator/useThroughputData.test.ts b/packages/app/src/components/calculator/useThroughputData.test.ts index efb5ca4a5..5a53b5930 100644 --- a/packages/app/src/components/calculator/useThroughputData.test.ts +++ b/packages/app/src/components/calculator/useThroughputData.test.ts @@ -1,7 +1,11 @@ import { describe, expect, it } from 'vitest'; +import type { BenchmarkRow } from '@/lib/api'; +import { overlayRunIndex } from '@/lib/overlay-run-style'; + import type { GPUDataPoint } from './types'; import { + buildGpuGroups, getCostField, hermiteInterpolate, interpolateForGPU, @@ -971,3 +975,206 @@ describe('maxInteractivityAtCost', () => { expect(maxInteractivityAtCost(withDominated, 0.1, 'costh', 'total')).toBeNull(); }); }); + +// ========================================================================= +// buildGpuGroups() — shared by the official and unofficial-run overlay paths +// ========================================================================= + +function makeRow(overrides: Partial = {}): BenchmarkRow { + return { + id: 1, + hardware: 'b300', + framework: 'sglang', + model: 'dsv4', + precision: 'fp4', + spec_method: 'none', + disagg: false, + is_multinode: false, + prefill_tp: 8, + prefill_ep: 8, + prefill_dp_attention: false, + prefill_num_workers: 1, + decode_tp: 8, + decode_ep: 8, + decode_dp_attention: false, + decode_num_workers: 1, + num_prefill_gpu: 8, + num_decode_gpu: 8, + benchmark_type: 'single_turn', + isl: 1024, + osl: 1024, + conc: 8, + offload_mode: 'off', + image: 'sglang:test', + metrics: { + median_intvty: 50, + tput_per_gpu: 900, + output_tput_per_gpu: 300, + input_tput_per_gpu: 600, + }, + date: '2026-07-19', + run_url: null, + ...overrides, + }; +} + +/** The official path's classifier: one group per hwKey (per precision when multi). */ +const singlePrecisionClassify = (hwKey: string) => ({ key: hwKey, meta: { hwKey } }); + +describe('buildGpuGroups', () => { + const shared = { isl: 1024, osl: 1024, precisions: ['fp4'] }; + + it('groups rows by the caller-supplied key and derives cost + power metrics', () => { + const { grouped, groupMeta, hwConfigMap } = buildGpuGroups( + [ + makeRow({ conc: 8 }), + makeRow({ conc: 16, metrics: { median_intvty: 30, tput_per_gpu: 1500 } }), + ], + { ...shared, classify: singlePrecisionClassify }, + ); + + const keys = Object.keys(grouped); + expect(keys).toHaveLength(1); + const [hwKey] = keys; + expect(grouped[hwKey]).toHaveLength(2); + expect(groupMeta[hwKey]).toEqual({ hwKey }); + expect(hwConfigMap[hwKey]).toBeDefined(); + + const [first] = grouped[hwKey]; + expect(first.interactivity).toBe(50); + expect(first.throughput).toBe(900); + expect(first.concurrency).toBe(8); + // Cost per million tokens is derived, not passed through. + expect(first.costh).toBeGreaterThan(0); + expect(first.tpPerMw).toBeGreaterThan(0); + }); + + it('drops rows whose isl/osl do not match the selected sequence', () => { + const { grouped } = buildGpuGroups( + [makeRow({ isl: 8192, osl: 1024 }), makeRow({ isl: null, osl: null })], + { ...shared, classify: singlePrecisionClassify }, + ); + expect(Object.keys(grouped)).toHaveLength(0); + }); + + it('drops rows whose precision is not selected', () => { + const { grouped } = buildGpuGroups([makeRow({ precision: 'fp8' })], { + ...shared, + classify: singlePrecisionClassify, + }); + expect(Object.keys(grouped)).toHaveLength(0); + }); + + it('drops rows the caller classifies as null', () => { + const { grouped } = buildGpuGroups([makeRow()], { + ...shared, + classify: () => null, + }); + expect(Object.keys(grouped)).toHaveLength(0); + }); + + it('splits into one group per precision when multiple precisions are selected', () => { + const { grouped, groupMeta } = buildGpuGroups( + [makeRow({ precision: 'fp4' }), makeRow({ precision: 'fp8' })], + { + isl: 1024, + osl: 1024, + precisions: ['fp4', 'fp8'], + classify: (hwKey, row) => ({ + key: `${hwKey}__${row.precision}`, + meta: { hwKey, precision: row.precision }, + }), + }, + ); + + const keys = Object.keys(grouped).toSorted(); + expect(keys).toHaveLength(2); + expect(keys.every((k) => k.includes('__fp4') || k.includes('__fp8'))).toBe(true); + for (const key of keys) { + expect(groupMeta[key].precision).toBe(key.endsWith('fp4') ? 'fp4' : 'fp8'); + } + }); + + it('keys overlay rows per run so two runs never share a group', () => { + const runA = 'https://github.com/org/repo/actions/runs/111'; + const runB = 'https://github.com/org/repo/actions/runs/222'; + const runIndexByUrl = { [runA]: 0, [runB]: 1 }; + + const { grouped, groupMeta } = buildGpuGroups( + [makeRow({ run_url: runA }), makeRow({ run_url: runB, conc: 16 })], + { + ...shared, + classify: (hwKey, row) => { + const runIndex = overlayRunIndex(row.run_url, runIndexByUrl); + return { key: `${hwKey}__run${runIndex}`, meta: { hwKey, runIndex } }; + }, + }, + ); + + const keys = Object.keys(grouped).toSorted(); + expect(keys).toHaveLength(2); + expect(keys.map((k) => groupMeta[k].runIndex).toSorted()).toEqual([0, 1]); + // Each run keeps its own points — no cross-run mixing into one frontier. + expect(grouped[keys[0]]).toHaveLength(1); + expect(grouped[keys[1]]).toHaveLength(1); + }); + + it('shares the same hwKey between an official row and its overlay twin', () => { + const official = buildGpuGroups([makeRow()], { + ...shared, + classify: singlePrecisionClassify, + }); + const overlay = buildGpuGroups( + [makeRow({ run_url: 'https://github.com/org/repo/actions/runs/111' })], + { + ...shared, + classify: (hwKey) => ({ key: `${hwKey}__run0`, meta: { hwKey, runIndex: 0 } }), + }, + ); + + const officialHw = Object.values(official.groupMeta)[0].hwKey; + const overlayHw = Object.values(overlay.groupMeta)[0].hwKey; + // Legend visibility is keyed on hwKey, so the two must agree. + expect(overlayHw).toBe(officialHw); + }); +}); + +// ========================================================================= +// interpolateForGPU() — clamped reporting +// ========================================================================= + +describe('interpolateForGPU clamped flag', () => { + const points = [ + makePoint({ interactivity: 20, throughput: 900 }), + makePoint({ interactivity: 50, throughput: 600 }), + makePoint({ interactivity: 80, throughput: 300 }), + ]; + + it('is falsy for a target inside the measured range', () => { + const result = interpolateForGPU(points, 50, 'interactivity_to_throughput', 'costh'); + expect(result?.clamped).toBeFalsy(); + }); + + it('is set for a target above every measured point', () => { + const result = interpolateForGPU(points, 200, 'interactivity_to_throughput', 'costh'); + // Still returns a value (the calculator never drops a bar), but the caller + // can now tell the user it is the nearest edge point, not a measurement. + expect(result?.value).toBeGreaterThan(0); + expect(result?.clamped).toBe(true); + }); + + it('is set for a target below every measured point', () => { + const result = interpolateForGPU(points, 1, 'interactivity_to_throughput', 'costh'); + expect(result?.clamped).toBe(true); + }); + + it('is set on the single-point path when the target misses that point', () => { + const single = [makePoint({ interactivity: 40, throughput: 700 })]; + expect( + interpolateForGPU(single, 40, 'interactivity_to_throughput', 'costh')?.clamped, + ).toBeFalsy(); + expect(interpolateForGPU(single, 90, 'interactivity_to_throughput', 'costh')?.clamped).toBe( + true, + ); + }); +}); diff --git a/packages/app/src/components/calculator/useThroughputData.ts b/packages/app/src/components/calculator/useThroughputData.ts index cf1a9fb29..d35c728d3 100644 --- a/packages/app/src/components/calculator/useThroughputData.ts +++ b/packages/app/src/components/calculator/useThroughputData.ts @@ -2,14 +2,16 @@ import { useCallback, useMemo } from 'react'; -import { sequenceToIslOsl } from '@semianalysisai/inferencex-constants'; +import { DB_MODEL_TO_DISPLAY, sequenceToIslOsl } from '@semianalysisai/inferencex-constants'; import type { HardwareConfig } from '@/components/inference/types'; import { useBenchmarks } from '@/hooks/api/use-benchmarks'; +import type { BenchmarkRow } from '@/lib/api'; import { rowToAggDataEntry } from '@/lib/benchmark-transform'; import { getHardwareKey } from '@/lib/chart-utils'; import { getModelSortIndex, getHardwareConfig, getGpuSpecs } from '@/lib/constants'; import type { Model, Sequence } from '@/lib/data-mappings'; +import { overlayRunIndex } from '@/lib/overlay-run-style'; import { getCostField, @@ -37,11 +39,118 @@ export { const computeGpuCost = (costPerHour: number, tps: number) => costPerHour && tps > 0 ? costPerHour / ((tps * 3600) / 1_000_000) : 0; +/** Metadata describing what a group key stands for. */ +export interface GroupMeta { + hwKey: string; + /** Set only when multiple precisions are selected, matching the group key. */ + precision?: string; +} + +/** Group metadata for an unofficial-run overlay group. */ +export interface OverlayGroupMeta extends GroupMeta { + /** Index of the run in the loaded set — drives the overlay palette color. */ + runIndex: number; +} + +/** + * Build `GPUDataPoint` groups from raw benchmark rows. + * + * Shared by the official and the unofficial-run overlay paths so both are + * derived by identical logic — the only difference is how rows are keyed into + * groups, which the caller controls via `classify`. + */ +export function buildGpuGroups( + rows: BenchmarkRow[], + options: { + isl: number; + osl: number; + precisions: string[]; + /** Derive a row's group key + metadata. Return null to drop the row. */ + classify: (hwKey: string, row: BenchmarkRow) => { key: string; meta: M } | null; + }, +): { + grouped: Record; + groupMeta: Record; + hwConfigMap: HardwareConfig; +} { + const { isl, osl, precisions, classify } = options; + const grouped: Record = {}; + const groupMeta: Record = {}; + const hwConfigMap: HardwareConfig = {}; + + for (const row of rows) { + if (row.isl !== isl || row.osl !== osl) continue; + if (!precisions.includes(row.precision)) continue; + + const entry = rowToAggDataEntry(row); + const hwKey = getHardwareKey(entry); + const hwConfig = getHardwareConfig(hwKey, entry.model); + if (!hwConfig) continue; + + const classified = classify(hwKey, row); + if (!classified) continue; + const { key: groupKey, meta } = classified; + + if (!hwConfigMap[hwKey]) hwConfigMap[hwKey] = { ...hwConfig, name: hwKey }; + + const m = row.metrics; + const tput = m.tput_per_gpu ?? 0; + const outputTput = m.output_tput_per_gpu ?? tput; + const inputTput = m.input_tput_per_gpu ?? 0; + const specs = getGpuSpecs(hwKey); + const power = specs.power; + + if (!grouped[groupKey]) grouped[groupKey] = []; + groupMeta[groupKey] = meta; + + grouped[groupKey].push({ + hwKey, + interactivity: m.median_intvty ?? 0, + throughput: tput, + outputThroughput: outputTput, + inputThroughput: inputTput, + concurrency: row.conc, + tp: row.decode_tp, + precision: row.precision, + ep: row.decode_ep, + dp_attention: row.decode_dp_attention, + disagg: row.disagg, + costh: computeGpuCost(specs.costh, tput), + costn: computeGpuCost(specs.costn, tput), + costr: computeGpuCost(specs.costr, tput), + costhi: computeGpuCost(specs.costh, inputTput), + costni: computeGpuCost(specs.costn, inputTput), + costri: computeGpuCost(specs.costr, inputTput), + costhOutput: computeGpuCost(specs.costh, outputTput), + costnOutput: computeGpuCost(specs.costn, outputTput), + costrOutput: computeGpuCost(specs.costr, outputTput), + tpPerMw: power && power > 0 ? (tput * 1000) / power : 0, + inputTpPerMw: power && power > 0 ? (inputTput * 1000) / power : 0, + outputTpPerMw: power && power > 0 ? (outputTput * 1000) / power : 0, + }); + } + + return { grouped, groupMeta, hwConfigMap }; +} + +/** + * Optional unofficial-run overlay inputs. When a run is loaded via + * `?unofficialrun=…`, its raw rows are interpolated into a *separate* set of + * results so official bars keep their own Pareto frontier untouched. + */ +export interface OverlayInput { + /** Raw rows from the unofficial-run API — every model, unfiltered. */ + rows: BenchmarkRow[] | null; + /** `run.url`/id → position in the loaded set, from the provider. */ + runIndexByUrl: Record; +} + export function useThroughputData( selectedModel: Model, selectedSequence: Sequence, selectedPrecisions: string[], selectedRunDate: string, + overlay?: OverlayInput, ) { // Reuse the same API + React Query cache as the inference charts const { @@ -55,110 +164,119 @@ export function useThroughputData( // Build GPUDataPoints directly from raw rows, skipping transformBenchmarkRows. // This avoids the expensive roofline/chart-data pipeline that isn't needed for interpolation. - const { gpuDataByGroupKey, hardwareConfig, hasData } = useMemo(() => { - if (!allRows) - return { - gpuDataByGroupKey: {} as Record, - hardwareConfig: {} as HardwareConfig, - hasData: false, - }; + const overlayRows = overlay?.rows ?? null; + const runIndexByUrl = overlay?.runIndexByUrl; + + const { + gpuDataByGroupKey, + gpuGroupMeta, + overlayGpuDataByGroupKey, + overlayGroupMeta, + hardwareConfig, + hasData, + hasOverlayData, + } = useMemo(() => { + const empty = { + gpuDataByGroupKey: {} as Record, + gpuGroupMeta: {} as Record, + overlayGpuDataByGroupKey: {} as Record, + overlayGroupMeta: {} as Record, + hardwareConfig: {} as HardwareConfig, + hasData: false, + hasOverlayData: false, + }; + if (!allRows) return empty; const seqIslOsl = sequenceToIslOsl(selectedSequence); - if (!seqIslOsl) - return { - gpuDataByGroupKey: {} as Record, - hardwareConfig: {} as HardwareConfig, - hasData: false, - }; + if (!seqIslOsl) return empty; const multiPrecision = selectedPrecisions.length > 1; - const grouped: Record = {}; - const hwConfigMap: HardwareConfig = {}; - - for (const row of allRows) { - if (row.isl !== seqIslOsl.isl || row.osl !== seqIslOsl.osl) continue; - if (!selectedPrecisions.includes(row.precision)) continue; - - const entry = rowToAggDataEntry(row); - const hwKey = getHardwareKey(entry); - const hwConfig = getHardwareConfig(hwKey, entry.model); - if (!hwConfig) continue; - - if (!hwConfigMap[hwKey]) hwConfigMap[hwKey] = { ...hwConfig, name: hwKey }; - - const m = row.metrics; - const tput = m.tput_per_gpu ?? 0; - const outputTput = m.output_tput_per_gpu ?? tput; - const inputTput = m.input_tput_per_gpu ?? 0; - const specs = getGpuSpecs(hwKey); - const power = specs.power; - - const groupKey = multiPrecision ? `${hwKey}__${row.precision}` : hwKey; - if (!grouped[groupKey]) grouped[groupKey] = []; - - grouped[groupKey].push({ - hwKey, - interactivity: m.median_intvty ?? 0, - throughput: tput, - outputThroughput: outputTput, - inputThroughput: inputTput, - concurrency: row.conc, - tp: row.decode_tp, - precision: row.precision, - ep: row.decode_ep, - dp_attention: row.decode_dp_attention, - disagg: row.disagg, - costh: computeGpuCost(specs.costh, tput), - costn: computeGpuCost(specs.costn, tput), - costr: computeGpuCost(specs.costr, tput), - costhi: computeGpuCost(specs.costh, inputTput), - costni: computeGpuCost(specs.costn, inputTput), - costri: computeGpuCost(specs.costr, inputTput), - costhOutput: computeGpuCost(specs.costh, outputTput), - costnOutput: computeGpuCost(specs.costn, outputTput), - costrOutput: computeGpuCost(specs.costr, outputTput), - tpPerMw: power && power > 0 ? (tput * 1000) / power : 0, - inputTpPerMw: power && power > 0 ? (inputTput * 1000) / power : 0, - outputTpPerMw: power && power > 0 ? (outputTput * 1000) / power : 0, - }); - } + const shared = { + isl: seqIslOsl.isl, + osl: seqIslOsl.osl, + precisions: selectedPrecisions, + }; + + const official = buildGpuGroups(allRows, { + ...shared, + classify: (hwKey, row) => ({ + key: multiPrecision ? `${hwKey}__${row.precision}` : hwKey, + meta: { hwKey, precision: multiPrecision ? row.precision : undefined }, + }), + }); - // Sort hardware config - const sortedKeys = Object.keys(hwConfigMap).toSorted( + // Overlay rows arrive unfiltered by model (the official path gets model + // filtering server-side from /api/v1/benchmarks), so scope them here. + const overlayForModel = (overlayRows ?? []).filter( + (row) => (DB_MODEL_TO_DISPLAY[row.model] ?? row.model) === selectedModel, + ); + const overlayGroups = buildGpuGroups(overlayForModel, { + ...shared, + classify: (hwKey, row) => { + const runIndex = overlayRunIndex(row.run_url, runIndexByUrl ?? {}); + const precision = multiPrecision ? row.precision : undefined; + return { + key: `${hwKey}${precision ? `__${precision}` : ''}__run${runIndex}`, + meta: { hwKey, precision, runIndex }, + }; + }, + }); + + // Sort hardware config. Overlay-only hardware is merged in so its bars and + // legend entries can resolve a display label. + const mergedConfig = { ...overlayGroups.hwConfigMap, ...official.hwConfigMap }; + const sortedKeys = Object.keys(mergedConfig).toSorted( (a, b) => getModelSortIndex(a) - getModelSortIndex(b) || a.localeCompare(b), ); const config: HardwareConfig = {}; sortedKeys.forEach((key) => { - config[key] = hwConfigMap[key]; + config[key] = mergedConfig[key]; }); return { - gpuDataByGroupKey: grouped, + gpuDataByGroupKey: official.grouped, + gpuGroupMeta: official.groupMeta, + overlayGpuDataByGroupKey: overlayGroups.grouped, + overlayGroupMeta: overlayGroups.groupMeta, hardwareConfig: config, - hasData: Object.keys(grouped).length > 0, + hasData: Object.keys(official.grouped).length > 0, + hasOverlayData: Object.keys(overlayGroups.grouped).length > 0, }; - }, [allRows, selectedSequence, selectedPrecisions]); + }, [allRows, selectedModel, selectedSequence, selectedPrecisions, overlayRows, runIndexByUrl]); // All available GPU hardware keys from data, ordered by hardwareConfig (HARDWARE_CONFIG order) // This returns unique GPU-level hwKeys (not composite keys) for the legend - const availableHwKeys = useMemo(() => { - // Extract unique hwKeys from group keys (strip __precision suffix if present) - const dataHwKeys = new Set(); - for (const groupKey of Object.keys(gpuDataByGroupKey)) { - const hwKey = groupKey.includes('__') ? groupKey.split('__')[0] : groupKey; - dataHwKeys.add(hwKey); - } - // Use hardwareConfig key order (already sorted by HARDWARE_CONFIG), then append any extras - const ordered = Object.keys(hardwareConfig).filter((k) => dataHwKeys.has(k)); - // Add any keys in data but not in hardwareConfig at the end - for (const k of dataHwKeys) { - if (!hardwareConfig[k]) ordered.push(k); - } - return ordered; - }, [gpuDataByGroupKey, hardwareConfig]); + const orderHwKeys = useCallback( + (dataHwKeys: Set) => { + // Use hardwareConfig key order (already sorted by HARDWARE_CONFIG), then append any extras + const ordered = Object.keys(hardwareConfig).filter((k) => dataHwKeys.has(k)); + // Add any keys in data but not in hardwareConfig at the end + for (const k of dataHwKeys) { + if (!hardwareConfig[k]) ordered.push(k); + } + return ordered; + }, + [hardwareConfig], + ); + + const availableHwKeys = useMemo( + () => orderHwKeys(new Set(Object.values(gpuGroupMeta).map((meta) => meta.hwKey))), + [gpuGroupMeta, orderHwKeys], + ); - // Compute global ranges from GPUDataPoints + /** Hardware present in the loaded unofficial run(s) for the current selection. */ + const overlayAvailableHwKeys = useMemo( + () => orderHwKeys(new Set(Object.values(overlayGroupMeta).map((meta) => meta.hwKey))), + [overlayGroupMeta, orderHwKeys], + ); + + // Compute global ranges from GPUDataPoints. Overlay points are included so + // the target-interactivity slider can reach operating points that only an + // unofficial run covers. const ranges = useMemo(() => { - const allPoints = Object.values(gpuDataByGroupKey).flat(); + const allPoints = [ + ...Object.values(gpuDataByGroupKey).flat(), + ...Object.values(overlayGpuDataByGroupKey).flat(), + ]; if (allPoints.length === 0) { return { interactivity: { min: 0, max: 100 }, @@ -187,7 +305,7 @@ export function useThroughputData( max: Math.ceil(maxTput), }, }; - }, [gpuDataByGroupKey]); + }, [gpuDataByGroupKey, overlayGpuDataByGroupKey]); // Interpolate results for all GPUs at a given target value const getResults = useCallback( @@ -200,9 +318,7 @@ export function useThroughputData( const results: InterpolatedResult[] = []; for (const [groupKey, points] of Object.entries(gpuDataByGroupKey)) { - // Extract the base hwKey for visibility check and config lookup - const hwKey = groupKey.includes('__') ? groupKey.split('__')[0] : groupKey; - const precision = groupKey.includes('__') ? groupKey.split('__')[1] : undefined; + const { hwKey, precision } = gpuGroupMeta[groupKey] ?? { hwKey: groupKey }; // Skip GPUs that are not visible (legend filters by hwKey) if (visibleHwKeys && !visibleHwKeys.has(hwKey)) continue; @@ -223,17 +339,68 @@ export function useThroughputData( return results; }, - [gpuDataByGroupKey], + [gpuDataByGroupKey, gpuGroupMeta], + ); + + /** + * Interpolate the unofficial-run overlay groups at the same target value. + * Kept separate from `getResults` so official bars keep their own Pareto + * frontier — overlay points never enter the official interpolation. + * + * `visibleHwKeys` is the same legend selection `getResults` is filtered by, + * so one legend entry governs a GPU's official and overlay bars together. + */ + const getOverlayResults = useCallback( + ( + targetValue: number, + mode: 'interactivity_to_throughput' | 'throughput_to_interactivity', + costProvider: CostProvider, + visibleHwKeys?: Set, + runInfoByIndex?: Record, + ): InterpolatedResult[] => { + const results: InterpolatedResult[] = []; + + for (const [groupKey, points] of Object.entries(overlayGpuDataByGroupKey)) { + const meta = overlayGroupMeta[groupKey]; + if (!meta) continue; + if (visibleHwKeys && !visibleHwKeys.has(meta.hwKey)) continue; + + const result = interpolateForGPU(points, targetValue, mode, costProvider); + if (result && result.value > 0) { + results.push({ + ...result, + hwKey: meta.hwKey, + resultKey: groupKey, + precision: meta.precision, + isOverlay: true, + runIndex: meta.runIndex, + runLabel: runInfoByIndex?.[meta.runIndex]?.branch, + runUrl: runInfoByIndex?.[meta.runIndex]?.url, + }); + } + } + + results.sort((a, b) => b.value - a.value); + + return results; + }, + [overlayGpuDataByGroupKey, overlayGroupMeta], ); return { gpuDataByGroupKey, + gpuGroupMeta, + overlayGpuDataByGroupKey, + overlayGroupMeta, hardwareConfig, ranges, getResults, + getOverlayResults, loading, error, hasData, + hasOverlayData, availableHwKeys, + overlayAvailableHwKeys, }; } diff --git a/packages/app/src/components/ui/chart-legend-item.tsx b/packages/app/src/components/ui/chart-legend-item.tsx index 6b7e9d6d7..ee2f0241e 100644 --- a/packages/app/src/components/ui/chart-legend-item.tsx +++ b/packages/app/src/components/ui/chart-legend-item.tsx @@ -33,6 +33,14 @@ export interface CommonLegendItemProps { isLegendExpanded?: boolean; // Whether the legend is expanded to show full text sidebarMode?: boolean; // Use sidebar-style visual feedback (line-through + faded dot) onRemove?: (name: string) => void; + /** + * Set false for entries that are labels rather than toggleable series — e.g. + * the unofficial-run entries, which are always `isActive` and have nothing to + * remove (a run is dismissed from the banner). Such entries render no hide + * control and are excluded from the "keep at least one series" active count, + * which they would otherwise inflate. Defaults to true. + */ + isRemovable?: boolean; /** * When provided, renders a small table icon that opens a per-series points * table (all data points for this hardware/framework series). Only the diff --git a/packages/app/src/components/ui/chart-legend.tsx b/packages/app/src/components/ui/chart-legend.tsx index 9c14c49f7..eba4ead9b 100644 --- a/packages/app/src/components/ui/chart-legend.tsx +++ b/packages/app/src/components/ui/chart-legend.tsx @@ -133,11 +133,16 @@ export default function ChartLegend({ const advancedControlsId = useId(); const effectiveExpanded = isLegendExpanded; + // Counts only removable series: the guard below exists to stop the user + // emptying the chart, and label-only entries (unofficial runs) are not + // something removing leaves you without. const activeCount = useMemo( - () => legendItems.filter((item) => item.isActive).length, + () => legendItems.filter((item) => item.isActive && item.isRemovable !== false).length, [legendItems], ); const effectiveRemove = onItemRemove && activeCount > 1 ? onItemRemove : undefined; + const removeFor = (item: CommonLegendItemProps) => + item.isRemovable === false ? undefined : effectiveRemove; useLayoutEffect(() => { setHasLongText(legendItems.some((item) => item.label && item.label.length > 8)); @@ -453,7 +458,7 @@ export default function ChartLegend({ onClick={item.onClick} onHover={onItemHover} onHoverEnd={onItemHoverEnd} - onRemove={effectiveRemove} + onRemove={removeFor(item)} onShowPoints={item.onShowPoints} asFragment isLegendExpanded={effectiveExpanded} @@ -549,7 +554,7 @@ export default function ChartLegend({ onClick={item.onClick} onHover={onItemHover} onHoverEnd={onItemHoverEnd} - onRemove={effectiveRemove} + onRemove={removeFor(item)} onShowPoints={item.onShowPoints} sidebarMode={isSidebar} asFragment diff --git a/packages/app/src/components/unofficial-run-provider.tsx b/packages/app/src/components/unofficial-run-provider.tsx index 54b470ff7..2ee7138c8 100644 --- a/packages/app/src/components/unofficial-run-provider.tsx +++ b/packages/app/src/components/unofficial-run-provider.tsx @@ -62,6 +62,14 @@ export interface UnofficialRunContextType { */ runIndexByUrl: Record; unofficialChartData: UnofficialChartData | null; + /** + * Raw benchmark rows as returned by the unofficial-run API, before any chart + * transform. Kept alongside `unofficialChartData` so consumers that build + * their own derived shapes from raw rows — today the TCO calculator's + * `useThroughputData` — can run overlay rows through the exact same mapping + * as official DB rows instead of re-deriving them from chart data. + */ + unofficialBenchmarkRows: BenchmarkRow[] | null; unofficialEvalRows: EvalRow[] | null; loading: boolean; error: string | null; @@ -169,6 +177,9 @@ export function UnofficialRunProvider({ children }: { children: ReactNode }) { const [unofficialRunInfos, setUnofficialRunInfos] = useState([]); const unofficialRunInfo = unofficialRunInfos[0] ?? null; const [unofficialChartData, setUnofficialChartData] = useState(null); + const [unofficialBenchmarkRows, setUnofficialBenchmarkRows] = useState( + null, + ); const [unofficialEvalRows, setUnofficialEvalRows] = useState(null); const [loading, setLoading] = useState(false); const [error, setError] = useState(null); @@ -231,6 +242,7 @@ export function UnofficialRunProvider({ children }: { children: ReactNode }) { const clearUnofficialRun = useCallback(() => { setUnofficialRunInfos([]); setUnofficialChartData(null); + setUnofficialBenchmarkRows(null); setUnofficialEvalRows(null); setError(null); setAvailableModelsAndSequences([]); @@ -268,6 +280,7 @@ export function UnofficialRunProvider({ children }: { children: ReactNode }) { if (remaining.length === 0) { setUnofficialRunInfos([]); setUnofficialChartData(null); + setUnofficialBenchmarkRows(null); setUnofficialEvalRows(null); setError(null); setAvailableModelsAndSequences([]); @@ -313,6 +326,10 @@ export function UnofficialRunProvider({ children }: { children: ReactNode }) { // the dismissed run. setAvailableModelsAndSequences(parseAvailableModelsAndSequences(nextChartData)); + setUnofficialBenchmarkRows((prev) => + prev ? prev.filter((row) => !belongsToDismissed(row.run_url)) : prev, + ); + setUnofficialEvalRows((prev) => prev ? prev.filter((row) => !belongsToDismissed(row.run_url)) : prev, ); @@ -357,6 +374,7 @@ export function UnofficialRunProvider({ children }: { children: ReactNode }) { if (!unofficialRunIdParam) { setUnofficialRunInfos([]); setUnofficialChartData(null); + setUnofficialBenchmarkRows(null); setUnofficialEvalRows(null); setError(null); setAvailableModelsAndSequences([]); @@ -374,8 +392,10 @@ export function UnofficialRunProvider({ children }: { children: ReactNode }) { if (!response.ok) throw new Error(data.error || 'Failed to fetch unofficial run'); setUnofficialRunInfos(Array.isArray(data.runInfos) ? data.runInfos : []); - const chartData = buildChartData(data.benchmarks ?? []); + const benchmarkRows: BenchmarkRow[] = data.benchmarks ?? []; + const chartData = buildChartData(benchmarkRows); setUnofficialChartData(chartData); + setUnofficialBenchmarkRows(benchmarkRows); setUnofficialEvalRows(data.evaluations ?? []); setAvailableModelsAndSequences(parseAvailableModelsAndSequences(chartData)); }) @@ -383,6 +403,7 @@ export function UnofficialRunProvider({ children }: { children: ReactNode }) { setError(caughtError instanceof Error ? caughtError.message : 'Unknown error'); setUnofficialRunInfos([]); setUnofficialChartData(null); + setUnofficialBenchmarkRows(null); setUnofficialEvalRows(null); setAvailableModelsAndSequences([]); }) @@ -402,6 +423,7 @@ export function UnofficialRunProvider({ children }: { children: ReactNode }) { unofficialRunInfos, runIndexByUrl, unofficialChartData, + unofficialBenchmarkRows, unofficialEvalRows, loading, error, diff --git a/packages/app/src/lib/utils.ts b/packages/app/src/lib/utils.ts index 1321eb770..7a2472137 100644 --- a/packages/app/src/lib/utils.ts +++ b/packages/app/src/lib/utils.ts @@ -34,6 +34,27 @@ export function updateRepoUrl(url: string): string { ); } +/** + * Escape a string for interpolation into an HTML string. + * + * D3 tooltips are built as HTML strings and injected with `.html()`, so any + * value that did not originate in this codebase must be escaped on the way in. + * The motivating case is unofficial-run branch names: they come from the GitHub + * API for whatever run id the user pastes into `?unofficialrun=`, and git + * permits `<` and `>` in a branch name. + * + * @param value - Untrusted text to interpolate into markup + * @returns The text with HTML-significant characters escaped + */ +export function escapeHtml(value: string): string { + return value + .replaceAll('&', '&') + .replaceAll('<', '<') + .replaceAll('>', '>') + .replaceAll('"', '"') + .replaceAll("'", '''); +} + /** * Formats a number for display - returns plain string for numbers < 10000, * and formatted number with commas for larger numbers.