Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,7 @@ Authoritative total / active parameter counts for every model in the dashboard.
| Qwen3.5-397B-A17B | 397B | 17B | `Qwen/Qwen3.5-397B-A17B` | HF model card |
| GLM-5 | 744B | 40B | `zai-org/GLM-5` | HF model card |
| GLM-5.1 | 744B | 40B | `zai-org/GLM-5.1-FP8` | HF model card (same base as GLM-5) |
| GLM-5.2 | 743B | 39B | `zai-org/GLM-5.2` | HF config.json (see trap below) |
| MiniMax-M2.5 | 230B | 10B | `MiniMaxAI/MiniMax-M2.5` | HF model card |
| MiniMax-M2.7 | 230B | 10B | `MiniMaxAI/MiniMax-M2.7` | NVIDIA M2.7 blog |
| gpt-oss-120b | 120B | 5.1B | `openai/gpt-oss-120b` | HF model card |
Expand All @@ -184,6 +185,7 @@ Authoritative total / active parameter counts for every model in the dashboard.
**Common mislabel traps** (have all bitten this repo at least once — do not repeat):

- **GLM-5 ≠ 355B.** 355B is GLM-4.5. GLM-5 jumped to 744B / 40B active (256-expert MoE with DSA).
- **GLM-5.2 is NOT a GLM-5 point release, and ≠ 753B.** Unlike 5.0 → 5.1, GLM-5.2 is a new architecture bucket (`glm_moe_dsa` with IndexShare sparse attention, 78 layers) — it maps to its own `GLM-5.2` display model; never group `glm5.2` with `glm5`/`glm5.1`. Its HF metadata shows 753B because the bundled MTP head (`num_nextn_predict_layers: 1`) adds ~10B; the core MoE is 743B / 39B active.
- **MiniMax-M2.5/M2.7 ≠ 456B.** 456B is the older MiniMax-Text-01 / M1 (32 large experts). The M2 series is a different architecture: 230B / 10B active, 256 small experts.
- **DeepSeek-R1 is 671B, not 685B.** HF metadata shows 685B because the bundled MTP head adds ~14B; the core MoE is 671B / 37B active.
- **Kimi K2.5, K2.6, and K2.7-Code are post-training refinements**, not new pre-trained sizes. Same 1T / 32B / 384-expert backbone as the original K2. K2.7-Code is a coding-focused refinement of the same backbone.
Expand Down
1 change: 1 addition & 0 deletions packages/app/cypress/e2e/dropdown-switching.cy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ describe('Dropdown one-click switching', () => {
cy.contains('Maintenance Mode').scrollIntoView().should('be.visible');
cy.contains('[role="option"]', 'DeepSeek R1 0528 671B').scrollIntoView().should('be.visible');
cy.contains('Deprecated').scrollIntoView().should('be.visible');
cy.contains('[role="option"]', 'GLM5/5.1 744B').scrollIntoView().should('be.visible');
cy.contains('[role="option"]', 'gpt-oss 120B').scrollIntoView().should('be.visible');
cy.contains('[role="option"]', 'Llama 3.3 70B Instruct').scrollIntoView().should('be.visible');
});
Expand Down
2 changes: 1 addition & 1 deletion packages/app/src/app/compare-per-dollar/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import { bucketComparePairsByVendor, formatModelList } from '@/lib/compare-ssr';

export const dynamic = 'force-dynamic';

const DESCRIPTION = `Which GPU delivers more inference performance per dollar? InferenceX is the independent, open-source benchmark from SemiAnalysis, with verified, reproducible results. ${SUPPORTERS_LINE} Compare cost per million tokens, normalized by hyperscaler TCO, across DeepSeek V4 Pro, DeepSeek R1, Kimi K2, MiniMax M3, GLM 5, Qwen 3.5 & more.`;
const DESCRIPTION = `Which GPU delivers more inference performance per dollar? InferenceX is the independent, open-source benchmark from SemiAnalysis, with verified, reproducible results. ${SUPPORTERS_LINE} Compare cost per million tokens, normalized by hyperscaler TCO, across DeepSeek V4 Pro, DeepSeek R1, Kimi K2, MiniMax M3, GLM 5.2, Qwen 3.5 & more.`;

export const metadata: Metadata = {
title: 'GPU Performance per Dollar',
Expand Down
2 changes: 1 addition & 1 deletion packages/app/src/app/compare/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import { bucketComparePairsByVendor, formatModelList } from '@/lib/compare-ssr';

export const dynamic = 'force-dynamic';

const DESCRIPTION = `InferenceX is the independent, open-source GPU inference benchmark from SemiAnalysis, with verified, reproducible nightly results. ${SUPPORTERS_LINE} Compare latency, throughput & cost head-to-head across DeepSeek V4 Pro, DeepSeek R1, Kimi K2, MiniMax M3, GLM 5, Qwen 3.5 & more.`;
const DESCRIPTION = `InferenceX is the independent, open-source GPU inference benchmark from SemiAnalysis, with verified, reproducible nightly results. ${SUPPORTERS_LINE} Compare latency, throughput & cost head-to-head across DeepSeek V4 Pro, DeepSeek R1, Kimi K2, MiniMax M3, GLM 5.2, Qwen 3.5 & more.`;

export const metadata: Metadata = {
title: 'GPU Comparisons',
Expand Down
2 changes: 1 addition & 1 deletion packages/app/src/app/zh/compare-per-dollar/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ import { ZH_OG_LOCALE, zhAlternates } from '@/lib/i18n';

export const dynamic = 'force-dynamic';

const DESCRIPTION = `哪款 GPU 每美元推理性能最高?InferenceX 是 SemiAnalysis 推出的独立开源基准测试平台,提供经过验证的、可复现的测试结果。${SUPPORTERS_LINE_ZH}横向对比 DeepSeek V4 Pro、DeepSeek R1、Kimi K2、MiniMax M3、GLM 5、Qwen 3.5 等模型基于云服务商 TCO 归一化的每百万 token 成本。`;
const DESCRIPTION = `哪款 GPU 每美元推理性能最高?InferenceX 是 SemiAnalysis 推出的独立开源基准测试平台,提供经过验证的、可复现的测试结果。${SUPPORTERS_LINE_ZH}横向对比 DeepSeek V4 Pro、DeepSeek R1、Kimi K2、MiniMax M3、GLM 5.2、Qwen 3.5 等模型基于云服务商 TCO 归一化的每百万 token 成本。`;

export const metadata: Metadata = {
title: 'GPU 每美元性能',
Expand Down
2 changes: 1 addition & 1 deletion packages/app/src/app/zh/compare/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ import { ZH_OG_LOCALE, zhAlternates } from '@/lib/i18n';

export const dynamic = 'force-dynamic';

const DESCRIPTION = `InferenceX 是 SemiAnalysis 推出的独立开源 GPU 推理基准测试平台,提供经过验证的、可复现的每夜测试结果。${SUPPORTERS_LINE_ZH}横向对比 DeepSeek V4 Pro、DeepSeek R1、Kimi K2、MiniMax M3、GLM 5、Qwen 3.5 等模型的延迟、吞吐量与成本。`;
const DESCRIPTION = `InferenceX 是 SemiAnalysis 推出的独立开源 GPU 推理基准测试平台,提供经过验证的、可复现的每夜测试结果。${SUPPORTERS_LINE_ZH}横向对比 DeepSeek V4 Pro、DeepSeek R1、Kimi K2、MiniMax M3、GLM 5.2、Qwen 3.5 等模型的延迟、吞吐量与成本。`;

export const metadata: Metadata = {
title: 'GPU 对比',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ const MODEL_LABELS: Record<string, string> = {
dsv4: 'DeepSeek V4 Pro',
glm5: 'GLM-5',
'glm5.1': 'GLM-5.1',
'glm5.2': 'GLM-5.2',
gptoss120b: 'gpt-oss 120B',
kimik2: 'Kimi K2',
'kimik2.5': 'Kimi K2.5',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -300,6 +300,7 @@ describe('buildInferenceCompareUrl', () => {
const cases: [string, string][] = [
['gptoss120b', 'gpt-oss-120b'],
['glm5.1', 'GLM-5'],
['glm5.2', 'GLM-5.2'],
['llama70b', 'Llama-3.3-70B-Instruct-FP8'],
['kimik2.6', 'Kimi-K2.5'],
['kimik2.7-code', 'Kimi-K2.5'],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ export function buildInferenceCompareUrl(
previousRow: SubmissionSummaryRow,
): string | null {
// DB_MODEL_TO_DISPLAY covers every DB prefix incl. point-release aliases
// (gptoss120b, glm5.1, kimik2.6, kimik2.7-code, minimaxm2.7, llama70b).
// (gptoss120b, glm5.1, glm5.2, kimik2.6, kimik2.7-code, minimaxm2.7, llama70b).
// MODEL_PREFIX_MAPPING only has the single canonical prefix per Model enum
// and misses those rows.
const displayModel = DB_MODEL_TO_DISPLAY[currentRow.model];
Expand Down
9 changes: 8 additions & 1 deletion packages/app/src/lib/compare-slug.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,13 @@ describe('parseCompareSlug — new model-prefixed form', () => {
expect(parsed?.b).toBe('gb200');
});

it('parses the glm-5-2 slug as its own model, distinct from glm-5-1', () => {
const parsed = parseCompareSlug('glm-5-2-h100-vs-h200');
expect(parsed?.model.slug).toBe('glm-5-2');
expect(parsed?.model.dbKeys).toEqual(['glm5.2']);
expect(parsed?.isAliasModel).toBe(false);
});

it('parses the minimax-m3 slug as its own model, distinct from minimax-m27', () => {
const parsed = parseCompareSlug('minimax-m3-h100-vs-h200');
expect(parsed?.model.slug).toBe('minimax-m3');
Expand Down Expand Up @@ -252,7 +259,7 @@ describe('compareModelDisplayLabel', () => {
expect(compareModelDisplayLabel(KIMI_K26, 'gb200', 'mi355x')).toBe(
'Kimi K2.5/K2.6/K2.7-Code 1T — GB200 NVL72 vs MI355X',
);
expect(compareModelDisplayLabel(GLM_51, 'h100', 'h200')).toBe('GLM 5/5.1/5.2 — H100 vs H200');
expect(compareModelDisplayLabel(GLM_51, 'h100', 'h200')).toBe('GLM 5/5.1 — H100 vs H200');
});
});

Expand Down
22 changes: 16 additions & 6 deletions packages/app/src/lib/compare-slug.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,9 @@ export interface CompareModelSlug {
// gpt-oss → Llama 70B. Per product spec — flagship Chinese-developed models
// first, smaller open US-developed models at the bottom. Qwen sits between
// MiniMax and gpt-oss to keep the Chinese-lab cluster contiguous before the
// US transition. The two MiniMax entries stay adjacent with the newer M3
// flagship leading the older M2 series.
// US transition. The two GLM entries and the two MiniMax entries each stay
// adjacent with the newer flagship (GLM 5.2, MiniMax M3) leading the older
// series.
export const COMPARE_MODEL_SLUGS: CompareModelSlug[] = [
{
slug: 'deepseek-v4',
Expand All @@ -63,13 +64,22 @@ export const COMPARE_MODEL_SLUGS: CompareModelSlug[] = [
// count appended so the label conveys model scale alongside the version.
label: 'Kimi K2.5/K2.6/K2.7-Code 1T',
},
{
slug: 'glm-5-2',
displayName: 'GLM-5.2',
// GLM-5.2 is a new 743B architecture (IndexShare sparse attention), not a
// point release of the GLM-5/5.1 family, so it gets its own slug and dbKey
// rather than joining the glm-5-1 group.
dbKeys: ['glm5.2'],
label: 'GLM 5.2 743B',
},
{
slug: 'glm-5-1',
displayName: 'GLM-5',
// GLM-5 point releases share the same base architecture. Keep the existing
// canonical slug for URL stability while querying every DB bucket.
dbKeys: ['glm5.2', 'glm5.1', 'glm5'],
label: 'GLM 5/5.1/5.2',
// GLM-5.0 and GLM-5.1 share an architecture per the model card; the slug
// uses the newer version name but the data pull covers both DB buckets.
dbKeys: ['glm5.1', 'glm5'],
label: 'GLM 5/5.1',
},
{
slug: 'minimax-m3',
Expand Down
1 change: 1 addition & 0 deletions packages/app/src/lib/compare-ssr.ts
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ export const KNOWN_MODELS = new Set([
'MiniMax-M2.5',
'MiniMax-M3',
'GLM-5',
'GLM-5.2',
'DeepSeek-V4-Pro',
]);
export const KNOWN_SEQUENCES = new Set(['1k/1k', '1k/8k', '8k/1k']);
Expand Down
30 changes: 29 additions & 1 deletion packages/app/src/lib/data-mappings.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,16 @@ describe('getModelAndSequence', () => {
expect(result).toEqual({ model: Model.Kimi_K2_5, sequence: Sequence.OneK_OneK });
});

it('parses glm5.2 artifact names as GLM_5_2, not GLM_5 (substring collision)', () => {
const result = getModelAndSequence('results_glm5.2_1k1k');
expect(result).toEqual({ model: Model.GLM_5_2, sequence: Sequence.OneK_OneK });
});

it('still parses plain glm5 artifact names as GLM_5', () => {
const result = getModelAndSequence('results_glm5_8k1k');
expect(result).toEqual({ model: Model.GLM_5, sequence: Sequence.EightK_OneK });
});

it('returns undefined for unrecognized model prefix', () => {
expect(getModelAndSequence('results_unknown_1k1k')).toBeUndefined();
});
Expand Down Expand Up @@ -127,6 +137,15 @@ describe('getModelAndSequenceFromArtifact', () => {
expect(result).toEqual({ model: Model.Kimi_K2_5, sequence: Sequence.EightK_OneK });
});

it('parses structured artifact with glm5.2 prefix as GLM_5_2', () => {
const result = getModelAndSequenceFromArtifact({
infmax_model_prefix: 'glm5.2',
isl: 1024,
osl: 1024,
});
expect(result).toEqual({ model: Model.GLM_5_2, sequence: Sequence.OneK_OneK });
});

it('returns undefined for unknown model prefix', () => {
const result = getModelAndSequenceFromArtifact({
infmax_model_prefix: 'unknown',
Expand Down Expand Up @@ -162,9 +181,17 @@ describe('isModelDeprecated', () => {
expect(isModelDeprecated(Model.GptOss)).toBe(true);
});

it('returns true for deprecated model GLM_5 (superseded by GLM-5.2)', () => {
expect(isModelDeprecated(Model.GLM_5)).toBe(true);
});

it('returns false for non-deprecated model DeepSeek_R1', () => {
expect(isModelDeprecated(Model.DeepSeek_R1)).toBe(false);
});

it('returns false for non-deprecated model GLM_5_2', () => {
expect(isModelDeprecated(Model.GLM_5_2)).toBe(false);
});
});

// ===========================================================================
Expand Down Expand Up @@ -224,7 +251,8 @@ describe('getModelLabel', () => {
expect(getModelLabel(Model.GptOss)).toBe('gpt-oss 120B');
expect(getModelLabel(Model.Qwen3_5)).toBe('Qwen3.5 397B');
expect(getModelLabel(Model.Kimi_K2_5)).toBe('Kimi K2.5/2.6/2.7-Code 1T');
expect(getModelLabel(Model.GLM_5)).toBe('GLM5/5.1/5.2 744B');
expect(getModelLabel(Model.GLM_5)).toBe('GLM5/5.1 744B');
expect(getModelLabel(Model.GLM_5_2)).toBe('GLM5.2 743B');
expect(getModelLabel(Model.MiniMax_M2_5)).toBe('MiniMax M2.5/2.7 230B');
});

Expand Down
20 changes: 19 additions & 1 deletion packages/app/src/lib/data-mappings.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ export enum Model {
MiniMax_M2_5 = 'MiniMax-M2.5',
MiniMax_M3 = 'MiniMax-M3',
GLM_5 = 'GLM-5',
GLM_5_2 = 'GLM-5.2',
DeepSeek_V4_Pro = 'DeepSeek-V4-Pro',
}

Expand Down Expand Up @@ -111,8 +112,25 @@ const MODEL_CONFIG: Record<Model, ModelConfig> = {
prefix: 'dsr1',
category: 'maintenance',
},
[Model.GLM_5]: { label: 'GLM5/5.1/5.2 744B', prefix: 'glm5', category: 'default' },
[Model.GLM_5_2]: {
// GLM-5.2 is a new architecture bucket (743B core MoE / 39B active with
// IndexShare DSA; HF metadata shows 753B because of the bundled MTP head),
// NOT a point release of GLM-5/5.1 — so it gets its own entry instead of
// joining the GLM_5 grouping. Keep this entry above GLM_5:
// MODEL_PREFIX_MAPPING matches artifact names by substring in insertion
// order, and 'glm5.2' artifact names also contain 'glm5'.
label: 'GLM5.2 743B',
prefix: 'glm5.2',
category: 'default',
},
[Model.Qwen3_5]: { label: 'Qwen3.5 397B', prefix: 'qwen3.5', category: 'default' },
[Model.GLM_5]: {
// Superseded by GLM-5.2 (a different architecture bucket, split into its
// own entry above), so it's deprecated (no longer actively benchmarked).
label: 'GLM5/5.1 744B',
prefix: 'glm5',
category: 'deprecated',
},
[Model.GptOss]: { label: 'gpt-oss 120B', prefix: 'gptoss', category: 'deprecated' },
[Model.MiniMax_M2_5]: {
// M2.5 and M2.7 share an architecture — same GLM5/5.1 pattern as Kimi.
Expand Down
9 changes: 6 additions & 3 deletions packages/app/src/lib/models-mapping.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,12 @@ describe('DISPLAY_MODEL_TO_DB', () => {
});

it('groups point-release DB keys under one display', () => {
expect(DISPLAY_MODEL_TO_DB['GLM-5']).toEqual(
expect.arrayContaining(['glm5', 'glm5.1', 'glm5.2']),
);
expect(DISPLAY_MODEL_TO_DB['GLM-5']).toEqual(expect.arrayContaining(['glm5', 'glm5.1']));
});

it('keeps glm5.2 in its own GLM-5.2 display bucket (new architecture)', () => {
expect(DISPLAY_MODEL_TO_DB['GLM-5.2']).toEqual(['glm5.2']);
expect(DISPLAY_MODEL_TO_DB['GLM-5']).not.toContain('glm5.2');
});
});

Expand Down
9 changes: 6 additions & 3 deletions packages/constants/src/models.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,7 @@ describe('DB_MODEL_TO_DISPLAY / DISPLAY_MODEL_TO_DB consistency', () => {
});

it('groups point-release DB keys under the same display name', () => {
expect(DISPLAY_MODEL_TO_DB['GLM-5']).toEqual(
expect.arrayContaining(['glm5', 'glm5.1', 'glm5.2']),
);
expect(DISPLAY_MODEL_TO_DB['GLM-5']).toEqual(expect.arrayContaining(['glm5', 'glm5.1']));
expect(DISPLAY_MODEL_TO_DB['Kimi-K2.5']).toEqual(
expect.arrayContaining(['kimik2.5', 'kimik2.6', 'kimik2.7-code']),
);
Expand All @@ -30,6 +28,11 @@ describe('DB_MODEL_TO_DISPLAY / DISPLAY_MODEL_TO_DB consistency', () => {
it('maps minimaxm3 to its own MiniMax-M3 display name', () => {
expect(DISPLAY_MODEL_TO_DB['MiniMax-M3']).toEqual(['minimaxm3']);
});

it('maps glm5.2 to its own GLM-5.2 display name (new architecture, not a GLM-5 point release)', () => {
expect(DISPLAY_MODEL_TO_DB['GLM-5.2']).toEqual(['glm5.2']);
expect(DISPLAY_MODEL_TO_DB['GLM-5']).not.toContain('glm5.2');
});
});

describe('sequenceToIslOsl', () => {
Expand Down
5 changes: 4 additions & 1 deletion packages/constants/src/models.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,10 @@ export const DB_MODEL_TO_DISPLAY: Record<string, string> = {
minimaxm3: 'MiniMax-M3',
glm5: 'GLM-5',
'glm5.1': 'GLM-5',
'glm5.2': 'GLM-5',
// GLM-5.2 is a new architecture bucket (743B core MoE with IndexShare DSA),
// not a point release of GLM-5/5.1 (744B) — it renders as its own display
// model instead of joining the GLM-5 grouping.
'glm5.2': 'GLM-5.2',
dsv4: 'DeepSeek-V4-Pro',
};

Expand Down
3 changes: 2 additions & 1 deletion packages/db/src/etl/normalizers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -98,9 +98,10 @@ export const MODEL_TO_KEY: Record<string, string> = {
'MiniMaxAI/MiniMax-M2.5': 'minimaxm2.5',
// MiniMax-M3 (428B, distinct architecture from the M2 series)
'MiniMaxAI/MiniMax-M3': 'minimaxm3',
// GLM-5
// GLM-5 / 5.1 (same architecture, grouped under one display)
'zai-org/GLM-5-FP8': 'glm5',
'amd/GLM-5.1-MXFP4': 'glm5.1',
// GLM-5.2 (new architecture — its own display bucket, not a GLM-5 point release)
'zai-org/GLM-5.2-FP8': 'glm5.2',
// DeepSeek-V4-Pro
'deepseek-ai/DeepSeek-V4-Pro': 'dsv4',
Expand Down
Loading