Skip to content

Commit 1d4d2d3

Browse files
felixweinbergerfelixweinberger
committed
feat: add specVersion classification to conformance scenarios
Add a SpecVersion type to Scenario and ClientScenario interfaces, enabling SDK maintainers and the tier-check scoring logic to filter tests by target spec version. - Add SpecVersion union type ('2025-03-26' | '2025-06-18' | '2025-11-25' | 'draft' | 'extension') - Tag all 32 server scenarios and 23 client scenarios with their specVersion - Add helper functions: listScenariosForSpec, listScenariosUpToSpec, etc. - Add --spec-version CLI filter to client, server, and list commands - Date versions are cumulative (2025-11-25 includes all prior) - draft/extension are exact-match only - Show [specVersion] tags in list output - Attach specVersion to tier-check detail entries with per-version breakdown - Update tier-audit skill docs to reflect new spec version metadata
1 parent 3b4a92b commit 1d4d2d3

31 files changed

Lines changed: 338 additions & 44 deletions

‎.claude/skills/mcp-sdk-tier-audit/SKILL.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,9 @@ npm run --silent tier-check -- \
6666

6767
If no client-cmd was detected, omit the `--client-cmd` flag (client conformance will be skipped).
6868

69-
The CLI output includes server conformance pass rate, client conformance pass rate, issue triage compliance, P0 resolution times, label taxonomy, stable release status, policy signal files, and spec tracking gap. Parse the JSON output to feed into Step 4.
69+
The CLI output includes server conformance pass rate, client conformance pass rate (with per-spec-version breakdown), issue triage compliance, P0 resolution times, label taxonomy, stable release status, policy signal files, and spec tracking gap. Parse the JSON output to feed into Step 4.
70+
71+
The conformance results now include a `specVersion` field on each detail entry, enabling per-version pass rate analysis. The `list` command also shows spec version tags: `node dist/index.js list` shows `[2025-06-18]`, `[2025-11-25]`, `[draft]`, or `[extension]` next to each scenario.
7072

7173
### Conformance Baseline Check
7274

@@ -200,9 +202,12 @@ After the subagents finish, output a short executive summary directly to the use
200202
| Check | Value | T2 | T1 |
201203
|-------|-------|----|----|
202204
| Server Conformance | <passed>/<total> (<rate>%) | ✓/✗ | ✓/✗ |
205+
| — 2025-06-18 | <pass>/<total> (<rate>%) | — | — |
206+
| — 2025-11-25 | <pass>/<total> (<rate>%) | — | — |
203207
| Client Conformance (full) | <passed>/<total> (<rate>%) | ✓/✗ | ✓/✗ |
204208
| — Core scenarios | <core_pass>/<core_total> (<rate>%) | — | — |
205209
| — Auth scenarios | <auth_pass>/<auth_total> (<rate>%) | — | — |
210+
| — draft / extension | <count> (informational) | — | — |
206211
| Issue Triage | <rate>% (<triaged>/<total>) | ✓/✗ | ✓/✗ |
207212
| Labels | <present>/<required> | ✓/✗ | ✓/✗ |
208213
| P0 Resolution | <count> open | ✓/✗ | ✓/✗ |

‎.claude/skills/mcp-sdk-tier-audit/references/tier-requirements.md‎

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -32,12 +32,23 @@ Source: `modelcontextprotocol/docs/community/sdk-tiers.mdx` in the spec reposito
3232

3333
## Conformance Score Calculation
3434

35-
Conformance scores are calculated against **applicable required tests** only:
35+
Every scenario in the conformance suite now has a `specVersion` field indicating which spec version it targets:
3636

37-
- Tests for the specification version the SDK targets
38-
- Excluding tests marked as pending or skipped
39-
- Excluding tests for experimental features
40-
- Excluding legacy backward-compatibility tests (unless the SDK claims legacy support)
37+
| specVersion | Meaning | Count toward tier score? |
38+
| ------------ | ------------------------------------------------------ | ------------------------ |
39+
| `2025-03-26` | Legacy spec (March 2025) | Yes |
40+
| `2025-06-18` | Base spec — Tier 2 baseline | Yes |
41+
| `2025-11-25` | Current spec — Tier 1 baseline | Yes |
42+
| `draft` | Unreleased draft spec features | Listed separately |
43+
| `extension` | Optional protocol extensions (e.g. ext-auth, SEP-1046) | Listed separately |
44+
45+
The `--spec-version` CLI flag filters scenarios cumulatively for date versions:
46+
47+
- `--spec-version 2025-06-18` includes `2025-03-26` + `2025-06-18` scenarios
48+
- `--spec-version 2025-11-25` includes all three dated versions
49+
- `--spec-version draft` or `--spec-version extension` returns only exact matches
50+
51+
The tier-check CLI output includes a per-version breakdown alongside the aggregate pass rate, and groups `draft`/`extension` results separately as informational.
4152

4253
## Tier Relegation Rules
4354

‎src/index.ts‎

Lines changed: 103 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,14 @@ import {
1919
listMetadataScenarios,
2020
listCoreScenarios,
2121
listExtensionScenarios,
22-
listBackcompatScenarios
22+
listBackcompatScenarios,
23+
listScenariosUpToSpec,
24+
listScenariosForSpec,
25+
listClientScenariosUpToSpec,
26+
listClientScenariosForSpec,
27+
getScenarioSpecVersion
2328
} from './scenarios';
29+
import type { SpecVersion } from './scenarios';
2430
import { ConformanceCheck } from './types';
2531
import { ClientOptionsSchema, ServerOptionsSchema } from './schemas';
2632
import {
@@ -31,6 +37,40 @@ import {
3137
import { createTierCheckCommand } from './tier-check';
3238
import packageJson from '../package.json';
3339

40+
const VALID_SPEC_VERSIONS: SpecVersion[] = [
41+
'2025-03-26',
42+
'2025-06-18',
43+
'2025-11-25',
44+
'draft',
45+
'extension'
46+
];
47+
48+
function resolveSpecVersion(value: string): SpecVersion {
49+
if (VALID_SPEC_VERSIONS.includes(value as SpecVersion)) {
50+
return value as SpecVersion;
51+
}
52+
console.error(`Unknown spec version: ${value}`);
53+
console.error(`Valid versions: ${VALID_SPEC_VERSIONS.join(', ')}`);
54+
process.exit(1);
55+
}
56+
57+
function filterScenariosBySpecVersion(
58+
allScenarios: string[],
59+
version: SpecVersion,
60+
kind: 'client' | 'server'
61+
): string[] {
62+
const isDateVersion = version !== 'draft' && version !== 'extension';
63+
const versionScenarios = isDateVersion
64+
? kind === 'client'
65+
? listScenariosUpToSpec(version)
66+
: listClientScenariosUpToSpec(version)
67+
: kind === 'client'
68+
? listScenariosForSpec(version)
69+
: listClientScenariosForSpec(version);
70+
const allowed = new Set(versionScenarios);
71+
return allScenarios.filter((s) => allowed.has(s));
72+
}
73+
3474
const program = new Command();
3575

3676
program
@@ -53,12 +93,19 @@ program
5393
'Path to YAML file listing expected failures (baseline)'
5494
)
5595
.option('-o, --output-dir <path>', 'Save results to this directory')
96+
.option(
97+
'--spec-version <version>',
98+
'Filter scenarios by spec version (cumulative for date versions)'
99+
)
56100
.option('--verbose', 'Show verbose output')
57101
.action(async (options) => {
58102
try {
59103
const timeout = parseInt(options.timeout, 10);
60104
const verbose = options.verbose ?? false;
61105
const outputDir = options.outputDir;
106+
const specVersionFilter = options.specVersion
107+
? resolveSpecVersion(options.specVersion)
108+
: undefined;
62109

63110
// Handle suite mode
64111
if (options.suite) {
@@ -85,7 +132,14 @@ program
85132
process.exit(1);
86133
}
87134

88-
const scenarios = suites[suiteName]();
135+
let scenarios = suites[suiteName]();
136+
if (specVersionFilter) {
137+
scenarios = filterScenariosBySpecVersion(
138+
scenarios,
139+
specVersionFilter,
140+
'client'
141+
);
142+
}
89143
console.log(
90144
`Running ${suiteName} suite (${scenarios.length} scenarios) in parallel...\n`
91145
);
@@ -262,6 +316,10 @@ program
262316
'Path to YAML file listing expected failures (baseline)'
263317
)
264318
.option('-o, --output-dir <path>', 'Save results to this directory')
319+
.option(
320+
'--spec-version <version>',
321+
'Filter scenarios by spec version (cumulative for date versions)'
322+
)
265323
.option('--verbose', 'Show verbose output (JSON instead of pretty print)')
266324
.action(async (options) => {
267325
try {
@@ -270,6 +328,9 @@ program
270328

271329
const verbose = options.verbose ?? false;
272330
const outputDir = options.outputDir;
331+
const specVersionFilter = options.specVersion
332+
? resolveSpecVersion(options.specVersion)
333+
: undefined;
273334

274335
// If a single scenario is specified, run just that one
275336
if (validated.scenario) {
@@ -317,6 +378,14 @@ program
317378
process.exit(1);
318379
}
319380

381+
if (specVersionFilter) {
382+
scenarios = filterScenariosBySpecVersion(
383+
scenarios,
384+
specVersionFilter,
385+
'server'
386+
);
387+
}
388+
320389
console.log(
321390
`Running ${suite} suite (${scenarios.length} scenarios) against ${validated.url}\n`
322391
);
@@ -393,20 +462,48 @@ program
393462
.description('List available test scenarios')
394463
.option('--client', 'List client scenarios')
395464
.option('--server', 'List server scenarios')
465+
.option(
466+
'--spec-version <version>',
467+
'Filter scenarios by spec version (cumulative for date versions)'
468+
)
396469
.action((options) => {
470+
const specVersionFilter = options.specVersion
471+
? resolveSpecVersion(options.specVersion)
472+
: undefined;
473+
397474
if (options.server || (!options.client && !options.server)) {
398475
console.log('Server scenarios (test against a server):');
399-
const serverScenarios = listClientScenarios();
400-
serverScenarios.forEach((s) => console.log(` - ${s}`));
476+
let serverScenarios = listClientScenarios();
477+
if (specVersionFilter) {
478+
serverScenarios = filterScenariosBySpecVersion(
479+
serverScenarios,
480+
specVersionFilter,
481+
'server'
482+
);
483+
}
484+
serverScenarios.forEach((s) => {
485+
const v = getScenarioSpecVersion(s);
486+
console.log(` - ${s}${v ? ` [${v}]` : ''}`);
487+
});
401488
}
402489

403490
if (options.client || (!options.client && !options.server)) {
404491
if (options.server || (!options.client && !options.server)) {
405492
console.log('');
406493
}
407494
console.log('Client scenarios (test against a client):');
408-
const clientScenarios = listScenarios();
409-
clientScenarios.forEach((s) => console.log(` - ${s}`));
495+
let clientScenarioNames = listScenarios();
496+
if (specVersionFilter) {
497+
clientScenarioNames = filterScenariosBySpecVersion(
498+
clientScenarioNames,
499+
specVersionFilter,
500+
'client'
501+
);
502+
}
503+
clientScenarioNames.forEach((s) => {
504+
const v = getScenarioSpecVersion(s);
505+
console.log(` - ${s}${v ? ` [${v}]` : ''}`);
506+
});
410507
}
411508
});
412509

‎src/scenarios/client/auth/basic-cimd.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import type { Scenario, ConformanceCheck } from '../../../types';
2-
import { ScenarioUrls } from '../../../types';
2+
import { ScenarioUrls, SpecVersion } from '../../../types';
33
import { createAuthServer } from './helpers/createAuthServer';
44
import { createServer } from './helpers/createServer';
55
import { ServerLifecycle } from './helpers/serverLifecycle';
@@ -22,6 +22,7 @@ export const CIMD_CLIENT_METADATA_URL =
2222
*/
2323
export class AuthBasicCIMDScenario implements Scenario {
2424
name = 'auth/basic-cimd';
25+
specVersion: SpecVersion = '2025-11-25';
2526
description =
2627
'Tests OAuth flow with Client ID Metadata Documents (SEP-991/URL-based client IDs). Server advertises client_id_metadata_document_supported=true and client should use URL as client_id instead of DCR.';
2728
private authServer = new ServerLifecycle();

‎src/scenarios/client/auth/client-credentials.ts‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,11 @@
11
import * as jose from 'jose';
22
import type { CryptoKey } from 'jose';
3-
import type { Scenario, ConformanceCheck, ScenarioUrls } from '../../../types';
3+
import type {
4+
Scenario,
5+
ConformanceCheck,
6+
ScenarioUrls,
7+
SpecVersion
8+
} from '../../../types';
49
import { createAuthServer } from './helpers/createAuthServer';
510
import { createServer } from './helpers/createServer';
611
import { ServerLifecycle } from './helpers/serverLifecycle';
@@ -32,6 +37,7 @@ async function generateTestKeypair(): Promise<{
3237
*/
3338
export class ClientCredentialsJwtScenario implements Scenario {
3439
name = 'auth/client-credentials-jwt';
40+
specVersion: SpecVersion = 'extension';
3541
description =
3642
'Tests OAuth client_credentials flow with private_key_jwt authentication (SEP-1046)';
3743

@@ -250,6 +256,7 @@ export class ClientCredentialsJwtScenario implements Scenario {
250256
*/
251257
export class ClientCredentialsBasicScenario implements Scenario {
252258
name = 'auth/client-credentials-basic';
259+
specVersion: SpecVersion = 'extension';
253260
description =
254261
'Tests OAuth client_credentials flow with client_secret_basic authentication';
255262

‎src/scenarios/client/auth/discovery-metadata.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ function createMetadataScenario(config: MetadataScenarioConfig): Scenario {
8787

8888
return {
8989
name: `auth/${config.name}`,
90+
specVersion: '2025-11-25',
9091
description: `Tests Basic OAuth metadata discovery flow.
9192
9293
**PRM:** ${config.prmLocation}${config.inWwwAuth ? '' : ' (not in WWW-Authenticate)'}

‎src/scenarios/client/auth/march-spec-backcompat.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import type { Scenario, ConformanceCheck } from '../../../types';
2-
import { ScenarioUrls } from '../../../types';
2+
import { ScenarioUrls, SpecVersion } from '../../../types';
33
import { createAuthServer } from './helpers/createAuthServer';
44
import { createServer } from './helpers/createServer';
55
import { ServerLifecycle } from './helpers/serverLifecycle';
@@ -8,6 +8,7 @@ import { SpecReferences } from './spec-references';
88

99
export class Auth20250326OAuthMetadataBackcompatScenario implements Scenario {
1010
name = 'auth/2025-03-26-oauth-metadata-backcompat';
11+
specVersion: SpecVersion = '2025-03-26';
1112
description =
1213
'Tests 2025-03-26 spec OAuth flow: no PRM (Protected Resource Metadata), OAuth metadata at root location';
1314
private server = new ServerLifecycle();
@@ -68,6 +69,7 @@ export class Auth20250326OAuthMetadataBackcompatScenario implements Scenario {
6869

6970
export class Auth20250326OEndpointFallbackScenario implements Scenario {
7071
name = 'auth/2025-03-26-oauth-endpoint-fallback';
72+
specVersion: SpecVersion = '2025-03-26';
7173
description =
7274
'Tests OAuth flow with no metadata endpoints, relying on fallback to standard OAuth endpoints at server root (2025-03-26 spec behavior)';
7375
private server = new ServerLifecycle();

‎src/scenarios/client/auth/pre-registration.ts‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
1-
import type { Scenario, ConformanceCheck, ScenarioUrls } from '../../../types';
1+
import type {
2+
Scenario,
3+
ConformanceCheck,
4+
ScenarioUrls,
5+
SpecVersion
6+
} from '../../../types';
27
import { createAuthServer } from './helpers/createAuthServer';
38
import { createServer } from './helpers/createServer';
49
import { ServerLifecycle } from './helpers/serverLifecycle';
@@ -19,6 +24,7 @@ const PRE_REGISTERED_CLIENT_SECRET = 'pre-registered-secret';
1924
*/
2025
export class PreRegistrationScenario implements Scenario {
2126
name = 'auth/pre-registration';
27+
specVersion: SpecVersion = '2025-11-25';
2228
description =
2329
'Tests OAuth flow with pre-registered client credentials. Server does not support DCR.';
2430

‎src/scenarios/client/auth/resource-mismatch.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import type { Scenario, ConformanceCheck } from '../../../types.js';
2-
import { ScenarioUrls } from '../../../types.js';
2+
import { ScenarioUrls, SpecVersion } from '../../../types.js';
33
import { createAuthServer } from './helpers/createAuthServer.js';
44
import { createServer } from './helpers/createServer.js';
55
import { ServerLifecycle } from './helpers/serverLifecycle.js';
@@ -27,6 +27,7 @@ import { MockTokenVerifier } from './helpers/mockTokenVerifier.js';
2727
*/
2828
export class ResourceMismatchScenario implements Scenario {
2929
name = 'auth/resource-mismatch';
30+
specVersion: SpecVersion = 'draft';
3031
description =
3132
'Tests that client rejects when PRM resource does not match server URL';
3233
allowClientError = true;

‎src/scenarios/client/auth/scope-handling.ts‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import type { Scenario, ConformanceCheck } from '../../../types';
2-
import { ScenarioUrls } from '../../../types';
2+
import { ScenarioUrls, SpecVersion } from '../../../types';
33
import { createAuthServer } from './helpers/createAuthServer';
44
import { createServer } from './helpers/createServer';
55
import { ServerLifecycle } from './helpers/serverLifecycle';
@@ -15,6 +15,7 @@ import type { Request, Response, NextFunction } from 'express';
1515
*/
1616
export class ScopeFromWwwAuthenticateScenario implements Scenario {
1717
name = 'auth/scope-from-www-authenticate';
18+
specVersion: SpecVersion = '2025-11-25';
1819
description =
1920
'Tests that client uses scope parameter from WWW-Authenticate header when provided';
2021
private authServer = new ServerLifecycle();
@@ -100,6 +101,7 @@ export class ScopeFromWwwAuthenticateScenario implements Scenario {
100101
*/
101102
export class ScopeFromScopesSupportedScenario implements Scenario {
102103
name = 'auth/scope-from-scopes-supported';
104+
specVersion: SpecVersion = '2025-11-25';
103105
description =
104106
'Tests that client uses all scopes from scopes_supported when scope not in WWW-Authenticate header';
105107
private authServer = new ServerLifecycle();
@@ -195,6 +197,7 @@ export class ScopeFromScopesSupportedScenario implements Scenario {
195197
*/
196198
export class ScopeOmittedWhenUndefinedScenario implements Scenario {
197199
name = 'auth/scope-omitted-when-undefined';
200+
specVersion: SpecVersion = '2025-11-25';
198201
description =
199202
'Tests that client omits scope parameter when scopes_supported is undefined';
200203
private authServer = new ServerLifecycle();
@@ -281,6 +284,7 @@ export class ScopeOmittedWhenUndefinedScenario implements Scenario {
281284
*/
282285
export class ScopeStepUpAuthScenario implements Scenario {
283286
name = 'auth/scope-step-up';
287+
specVersion: SpecVersion = '2025-11-25';
284288
description =
285289
'Tests that client handles step-up authentication with different scope requirements per operation';
286290
private authServer = new ServerLifecycle();
@@ -477,6 +481,7 @@ export class ScopeStepUpAuthScenario implements Scenario {
477481
*/
478482
export class ScopeRetryLimitScenario implements Scenario {
479483
name = 'auth/scope-retry-limit';
484+
specVersion: SpecVersion = '2025-11-25';
480485
description =
481486
'Tests that client implements retry limits to prevent infinite authorization loops on repeated 403 responses';
482487
allowClientError = true;

0 commit comments

Comments
 (0)