test: derive the command-surface enumerations that restated a declaration (#2713)

* test(coverage): recount the published summaries instead of pinning them

Each platform coverage manifest already derives its classification summary from its own rows, and
the smoke test that owns the manifest then compared that derived value to a hand-incremented
literal. Adding one public command therefore cost five files whose only edit was `54` becoming
`55`, while the invariant those files actually protect — every public command is classified, and
the published rollup is the manifest's own — was asserted nowhere.

`assertCoverageClassificationSummaryDerivedFromManifest` now recounts the manifest independently of
the builder, ties `total` to the public catalog's size, and checks the buckets still sum. Parity was
verified against the deleted literals before removal, as ADR 0008 requires: macOS 22/15/18/55, web
42/1/12/55, Linux 29/9/17/55, tvOS 17/38/0/55, Android 14/0/41/55, all equal to the derived rollup
at this head. The web lane's `liveCommands.length === 12` goes the same way: the loop beneath it
already proves each live claim is executed by the smoke scenario, so a count proved only that the
list was not empty.

A planted mis-bucketed summary and a planted denominator drift both fail, naming the platform.

* test(daemon): read the generic route from the descriptor that declares it

The test enumerated eleven commands to prove that the daemon's fallthrough route is reached on
purpose. Ten of those commands already declare the decision as `daemon.route: 'generic'`, which is
where `getDaemonCommandRoute` reads it from, so the list was a second source of truth that every new
generic command had to be added to.

The expected set is now that trait, projected. What remains hand-written is the case the list was
actually guarding — a catalog command that declares no daemon facet at all and lands on the
`?? 'generic'` fallback — and it is asserted alone, with a message that names the declaration to
add instead. Removing `viewport`'s facet locally fails with `['viewport', 'install-from-source']`
against that one expected member.

* test(runtime-facts): fill the fail-closed test facts from the owner's helper

`createUnavailableRuntimeFactsForTest` restated all twenty-seven fail-closed cells, plus the nine
lifecycle cells, to say what `createFullyUnavailablePlatformRuntimeFacts` already builds: every cell
unavailable for one reason. The two forms are equal because the input's `network` cell carries the
same sentinel the omitted optional cells would have inherited.

A new runtime cell therefore costs nothing here, which is the point of the fail-closed default:
a test that overrides a cell states its own exception, and one that does not inherits the refusal
without anyone remembering to write it down.

* test(coverage): name the summary guard for what it guards, and close the web live bucket

`buildCoverageClassificationSummary` and the recount in the assertion read the same manifest rows, so
a row classified under the wrong `level` moves both sides and stays consistent. The helper therefore
guards wiring and denominator, never per-row classification, and it now says so by name
(`assertCoverageClassificationSummaryWiredToManifest`) and in its doc comment, which also names where
row membership is actually gated: the Linux lane against its parsed replay script, the Android lane
against each scenario's own command declaration, the web lane against the commands its smoke scenario
invokes, and macOS one way only, since its provider scenarios invoke commands they deliberately do not
claim live.

The web lane gains that third gate, replacing both the old `liveCommands.length === 12` and the weaker
`> 0` this PR first introduced. It parses the smoke scenario's argv heads and compares them to the
manifest's live set in both directions, so a command the scenario runs without a live claim, or a live
claim the scenario stopped honouring, fails by name. The parsed set equals the claimed set exactly
today, so the count the pin carried survives without being hand-incremented. Appending a line
containing `['hover',` to the scenario makes the gate fail on `hover`.

`localCliCommands()` is dropped from the daemon route test: `CATALOG_COMMANDS` is the public and
internal groups, and a `catalog.group: 'local-cli'` descriptor is never projected into either, so the
clause could not fire. The reasoning is left where the clause was.

* test(coverage): close the macOS live bucket from its scenario sources too

The commit before this one recorded macOS as gated one way only, on the finding that its provider
scenarios execute `open`, `appstate`, `snapshot`, `screenshot`, `wait` and `click` without claiming
them live. That reading was wrong: those commands are claimed live, attributed to the replay scenario,
so the mismatch is scenario attribution and not a missing claim. Compared at platform level, the
commands the live scenarios execute and the commands the manifest calls live are the same 18.

`assertLiveCoverageMatchesEvidence` states that tie for the two platforms whose evidence enumerates its
own commands — macOS through `parseReplayScriptDetailed` and the provider call shapes, web through the
argv heads its smoke scenario passes — and Linux and Android already read the same kind of enumeration
directly. Appending `callCommand('hover', …)` to a macOS live scenario fails the gate with the platform
named, and the rollup assertion's doc comment now points here for membership instead of implying it
covers classification.

Author: thymikee

src/__tests__/test-utils/runtime-operation-facts.ts

@@ -8,7 +8,10 @@ import type {
   RuntimeOperationUnavailability,
   RuntimeOwnerRef,
 } from '@agent-device/contracts/platform-runtime';
-import { createUnavailablePlatformRuntimeFacts } from '@agent-device/contracts/platform-runtime-unavailable';
+import {
+  createFullyUnavailablePlatformRuntimeFacts,
+  createUnavailablePlatformRuntimeFacts,
+} from '@agent-device/contracts/platform-runtime-unavailable';
 import type { DeviceInfo } from '@agent-device/kernel/device';
 import { screenshotRuntimeOperationFacts } from '@agent-device/contracts/screenshot-runtime';
 import { scrollRuntimeOperationFacts } from '@agent-device/contracts/scroll-runtime';
@@ -104,43 +107,9 @@ export function createUnavailableRuntimeFactsForTest(
   owner: RuntimeOwnerRef,
   fact: RuntimeOperationUnavailability = unavailable,
 ) {
-  return createUnavailablePlatformRuntimeFacts(device, owner, {
-    appLog: fact,
-    network: fact,
-    screenshot: fact,
-    viewport: fact,
-    focus: fact,
-    gesture: fact,
-    scroll: fact,
-    typeText: fact,
-    touch: fact,
-    elementText: fact,
-    back: fact,
-    home: fact,
-    orientation: fact,
-    tvRemote: fact,
-    keyboard: fact,
-    clipboard: fact,
-    appSwitcher: fact,
-    actionButton: fact,
-    triggerAppEvent: fact,
-    setSetting: fact,
-    readAlert: fact,
-    awaitAlert: fact,
-    acceptAlert: fact,
-    dismissAlert: fact,
-    audioProbeCapture: fact,
-    audioProbeQuery: fact,
-    lifecycle: applicationLifecycleOperationFacts({
-      resolveOpenTarget: fact,
-      prepareApplicationOpen: fact,
-      openApplication: fact,
-      applyRuntimeHints: fact,
-      clearRuntimeHints: fact,
-      closeApplication: fact,
-      finalizeApplicationClose: fact,
-      prepareAppleRunner: fact,
-      configureProviderPortReverse: fact,
-    }),
-  });
+  return createUnavailablePlatformRuntimeFacts(
+    device,
+    owner,
+    createFullyUnavailablePlatformRuntimeFacts(fact),
+  );
 }

src/daemon/__tests__/request-handler-catalog.test.ts

@@ -4,6 +4,7 @@ import path from 'node:path';
 import { test } from 'vitest';
 import { withTestDeviceInventoryProvider as withTargetDeviceResolutionScope } from '../../__tests__/test-utils/device-inventory-gateways.ts';
 import { INTERNAL_COMMANDS, PUBLIC_COMMANDS } from '@agent-device/command-registry/catalog';
+import { commandDescriptors } from '@agent-device/command-registry/registry';
 import { isRequestCanceledError, type AppError } from '@agent-device/kernel/errors';
 import { makeSessionStore } from '../../__tests__/test-utils/store-factory.ts';
 import { getDaemonCommandRoute, type DaemonCommandRoute } from '../daemon-command-registry.ts';
@@ -52,26 +53,43 @@ test('specialized daemon routes are claimed by their handler chain', async () =>
   }
 });
 
+// The generic route is the router's fallthrough, so it has two ways in: a descriptor that declares
+// `daemon.route: 'generic'`, and a catalog command that declares no daemon facet at all and lands
+// on the `?? 'generic'` fallback. The first is a decision read from the declaration; the second is
+// what this test exists to keep rare, so it is enumerated alone and by name.
+const CATALOG_COMMANDS = [...Object.values(PUBLIC_COMMANDS), ...Object.values(INTERNAL_COMMANDS)];
+
+function declaredDaemonTraitCommands(): string[] {
+  return commandDescriptors
+    .filter((descriptor) => 'daemon' in descriptor && descriptor.daemon !== undefined)
+    .map((descriptor) => descriptor.name);
+}
+
+function genericByTraitCommands(): string[] {
+  return commandDescriptors
+    .filter((descriptor) => 'daemon' in descriptor && descriptor.daemon?.route === 'generic')
+    .map((descriptor) => descriptor.name);
+}
+
 test('catalog commands use generic routing only when intentionally passthrough or projected', () => {
+  // Local-CLI commands are excluded by construction: `CATALOG_COMMANDS` holds the public and
+  // internal groups, which a `catalog.group: 'local-cli'` descriptor is never projected into.
+  const genericByAbsence = CATALOG_COMMANDS.filter(
+    (command) => !declaredDaemonTraitCommands().includes(command),
+  );
+  assert.deepEqual(
+    genericByAbsence,
+    [PUBLIC_COMMANDS.installFromSource],
+    'a catalog command reaches the generic route by declaring no daemon facet; declare daemon.route instead',
+  );
+
   const intentionalGenericCatalogCommands = [
-    PUBLIC_COMMANDS.actionButton,
-    PUBLIC_COMMANDS.appSwitcher,
-    PUBLIC_COMMANDS.back,
-    PUBLIC_COMMANDS.focus,
-    PUBLIC_COMMANDS.home,
-    PUBLIC_COMMANDS.installFromSource,
-    PUBLIC_COMMANDS.orientation,
-    PUBLIC_COMMANDS.screenshot,
-    PUBLIC_COMMANDS.scroll,
-    PUBLIC_COMMANDS.tvRemote,
-    PUBLIC_COMMANDS.viewport,
+    ...genericByTraitCommands(),
+    ...genericByAbsence,
   ].sort();
-  const genericCatalogCommands = [
-    ...Object.values(PUBLIC_COMMANDS),
-    ...Object.values(INTERNAL_COMMANDS),
-  ]
-    .filter((command) => getDaemonCommandRoute(command) === 'generic')
-    .sort();
+  const genericCatalogCommands = CATALOG_COMMANDS.filter(
+    (command) => getDaemonCommandRoute(command) === 'generic',
+  ).sort();
 
   assert.deepEqual(genericCatalogCommands, intentionalGenericCatalogCommands);
 });

test/integration/smoke-android-emulator-coverage.test.ts

@@ -4,6 +4,7 @@ import path from 'node:path';
 import test from 'node:test';
 
 import { PUBLIC_COMMANDS } from '@agent-device/command-registry/catalog';
+import { assertCoverageClassificationSummaryWiredToManifest } from './support/coverage-classification.ts';
 import { ANDROID_EMULATOR_BEHAVIOR_COVERAGE } from './android-emulator-e2e/behavior-coverage.ts';
 import { ANDROID_PERMISSION_PROMPT_COMMAND } from './android-emulator-e2e/live-lifecycle-scenario.ts';
 import {
@@ -19,8 +20,9 @@ import {
   selectAndroidEmulatorScenarios,
 } from './android-emulator-e2e/scenarios.ts';
 
+const publicCommands = Object.values(PUBLIC_COMMANDS).sort();
+
 test('Android emulator coverage exhaustively classifies the public catalog', () => {
-  const publicCommands = Object.values(PUBLIC_COMMANDS).sort();
   assert.deepEqual(Object.keys(ANDROID_EMULATOR_E2E_COVERAGE).sort(), publicCommands);
   for (const command of publicCommands) {
     const entry = ANDROID_EMULATOR_E2E_COVERAGE[command];
@@ -38,14 +40,12 @@ test('Android emulator coverage exhaustively classifies the public catalog', ()
 });
 
 test('Android coverage report summary accounts for every manifest classification', () => {
-  const summary = ANDROID_EMULATOR_COVERAGE_CLASSIFICATION_SUMMARY;
-  assert.deepEqual(summary, {
-    contract: 14,
-    gap: 0,
-    live: 41,
-    total: 55,
-  });
-  assert.equal(summary.live + summary.contract + summary.gap, summary.total);
+  assertCoverageClassificationSummaryWiredToManifest(
+    'Android emulator',
+    ANDROID_EMULATOR_E2E_COVERAGE,
+    ANDROID_EMULATOR_COVERAGE_CLASSIFICATION_SUMMARY,
+    publicCommands,
+  );
 });
 
 test('Android live command ownership is structural and exhaustive', () => {

test/integration/smoke-linux-coverage.test.ts

@@ -5,6 +5,7 @@ import test from 'node:test';
 
 import { parseReplayScriptDetailed } from '@agent-device/ad-script';
 import { PUBLIC_COMMANDS } from '@agent-device/command-registry/catalog';
+import { assertCoverageClassificationSummaryWiredToManifest } from './support/coverage-classification.ts';
 import {
   LINUX_COVERAGE_GAP_ISSUE,
   LINUX_COMMAND_EVIDENCE,
@@ -42,20 +43,15 @@ test('Linux coverage exhaustively classifies the public catalog', () => {
 });
 
 test('Linux coverage report has the expected classification counts', () => {
-  assert.deepEqual(LINUX_PLATFORM_COVERAGE_CLASSIFICATION_SUMMARY, {
-    // focus (#1925), click, and type remain live via the existing replay. The separate
-    // command-evidence lane adds nine generic-command rows without changing that replay. Artifact
-    // inventory remains a gap because local Linux screenshot paths are not daemon-downloadable.
-    // Keyboard, orientation and tv-remote were already fact-owned command-contract rows rather
-    // than catalog denials; R56 moves app-switcher the same way, for the same reason.
-    contract: 29,
-    gap: 9,
-    live: 17,
-    total: 55,
-  });
-
-  const { contract, gap, live, total } = LINUX_PLATFORM_COVERAGE_CLASSIFICATION_SUMMARY;
-  assert.equal(contract + gap + live, total);
+  // Focus (#1925), click, and type are live through the replay; the separate command-evidence
+  // lane adds generic-command rows without changing it. Artifact inventory stays a gap because
+  // local Linux screenshot paths are not daemon-downloadable.
+  assertCoverageClassificationSummaryWiredToManifest(
+    'Linux',
+    LINUX_PLATFORM_COVERAGE,
+    LINUX_PLATFORM_COVERAGE_CLASSIFICATION_SUMMARY,
+    publicCommands,
+  );
 });
 
 test('Linux live claims reference commands in the existing smoke replay', () => {

test/integration/smoke-macos-coverage.test.ts

@@ -1,10 +1,16 @@
 import assert from 'node:assert/strict';
+import { parseReplayScriptDetailed } from '@agent-device/ad-script';
 import fs from 'node:fs';
 import path from 'node:path';
 import test from 'node:test';
 
 import { mkdtempForTest } from '../../src/__tests__/test-utils/tmp-dir.ts';
 import { PUBLIC_COMMANDS } from '@agent-device/command-registry/catalog';
+import {
+  assertCoverageClassificationSummaryWiredToManifest,
+  assertLiveCoverageMatchesEvidence,
+} from './support/coverage-classification.ts';
+import type { MacOsLiveScenario } from './macos-e2e/live-scenarios.ts';
 import {
   MACOS_COVERAGE_GAP_ISSUE,
   MACOS_LIVE_SCENARIOS,
@@ -14,8 +20,9 @@ import {
 } from './macos-e2e/coverage.ts';
 import { writeCoverageReport } from './macos-e2e/coverage-report.ts';
 
+const publicCommands = Object.values(PUBLIC_COMMANDS).sort();
+
 test('macOS coverage exhaustively classifies the public catalog', () => {
-  const publicCommands = Object.values(PUBLIC_COMMANDS).sort();
   assert.deepEqual(Object.keys(MACOS_PLATFORM_COVERAGE).sort(), publicCommands);
 
   for (const command of publicCommands) {
@@ -38,17 +45,22 @@ test('macOS coverage exhaustively classifies the public catalog', () => {
 });
 
 test('macOS coverage report counts every manifest classification', () => {
-  assert.deepEqual(MACOS_PLATFORM_COVERAGE_CLASSIFICATION_SUMMARY, {
-    contract: 22,
-    gap: 15,
-    live: 18,
-    total: 55,
-  });
-  assert.equal(
-    MACOS_PLATFORM_COVERAGE_CLASSIFICATION_SUMMARY.live +
-      MACOS_PLATFORM_COVERAGE_CLASSIFICATION_SUMMARY.contract +
-      MACOS_PLATFORM_COVERAGE_CLASSIFICATION_SUMMARY.gap,
-    MACOS_PLATFORM_COVERAGE_CLASSIFICATION_SUMMARY.total,
+  assertCoverageClassificationSummaryWiredToManifest(
+    'macOS',
+    MACOS_PLATFORM_COVERAGE,
+    MACOS_PLATFORM_COVERAGE_CLASSIFICATION_SUMMARY,
+    publicCommands,
+  );
+});
+
+// The scenario sources say which commands macOS actually runs, independent of what the manifest
+// labels them. Provider scenarios run commands whose live claim is attributed to the replay
+// scenario, so this closes the platform's live bucket rather than each scenario's partition of it.
+test('macOS live claims are exactly the commands its live scenarios execute', () => {
+  assertLiveCoverageMatchesEvidence(
+    'macOS',
+    MACOS_PLATFORM_COVERAGE,
+    commandsExecutedByMacOsLiveScenarios(),
   );
 });
 
@@ -123,6 +135,40 @@ test('macOS coverage report persists the manifest rollup and live command list',
   );
 });
 
+function commandsExecutedByMacOsLiveScenarios(): string[] {
+  const catalogCommands: ReadonlySet<string> = new Set(publicCommands);
+  const executed = new Set<string>();
+  for (const scenario of MACOS_LIVE_SCENARIOS) {
+    for (const command of commandsExecutedInScenarioSource(
+      readEvidence(scenario.owner),
+      scenario.syntax,
+    )) {
+      if (catalogCommands.has(command)) {
+        executed.add(command);
+      }
+    }
+  }
+  return [...executed];
+}
+
+function commandsExecutedInScenarioSource(
+  source: string,
+  syntax: MacOsLiveScenario['syntax'],
+): string[] {
+  const group = (pattern: RegExp): string[] =>
+    [...source.matchAll(pattern)]
+      .map((match) => match[1])
+      .filter((command): command is string => command !== undefined);
+  switch (syntax) {
+    case 'replay':
+      return parseReplayScriptDetailed(source).actions.map((action) => action.command);
+    case 'provider-command':
+      return group(/command:\s*'([a-z][a-z-]*)'/g);
+    case 'provider-call-command':
+      return group(/callCommand\(\s*'([a-z][a-z-]*)'/g);
+  }
+}
+
 function readEvidence(owner: { path: string; test: string }): string {
   const evidencePath = path.resolve(owner.path);
   assert.equal(fs.existsSync(evidencePath), true, `${owner.path} does not exist`);

test/integration/smoke-tvos-platform-coverage.test.ts

@@ -5,6 +5,7 @@ import test from 'node:test';
 
 import { TVOS_SIMULATOR } from '../../src/__tests__/test-utils/device-fixtures.ts';
 import { PUBLIC_COMMANDS } from '@agent-device/command-registry/catalog';
+import { assertCoverageClassificationSummaryWiredToManifest } from './support/coverage-classification.ts';
 import { createPlatformRuntimeGateway } from '../../src/platform-runtime.ts';
 import { gestureRefusalMessage } from '@agent-device/contracts/gesture-admission';
 import {
@@ -39,12 +40,12 @@ test('tvOS coverage exhaustively classifies the public catalog', () => {
 });
 
 test('tvOS coverage report has the expected classification counts', () => {
-  assert.deepEqual(TVOS_PLATFORM_COVERAGE_CLASSIFICATION_SUMMARY, {
-    contract: 17,
-    gap: 38,
-    live: 0,
-    total: 55,
-  });
+  assertCoverageClassificationSummaryWiredToManifest(
+    'tvOS',
+    TVOS_PLATFORM_COVERAGE,
+    TVOS_PLATFORM_COVERAGE_CLASSIFICATION_SUMMARY,
+    publicCommands,
+  );
 });
 
 test('tvOS contract claims name existing executable evidence', () => {