Skip to content

Commit 937ccdc

Browse files
feat(ios-runner): key runner startup failures on typed reasons (#2688)
* refactor(ios-runner): give the host DevToolsSecurity probe its own module runner-session.ts is 1,181 lines, so the next behavior owed a split first. The host developer-tools security probe is a self-contained preflight: it reads a macOS setting for this machine, not the iPhone's Developer Mode toggle, and today nothing in its name or its home says which machine it speaks for. Pure move. The startup step key is renamed to say what it actually verifies, and the existing coverage in runner-session.test.ts carries unchanged because the probe consumes the same runner host port. * feat(ios-runner): key runner startup failures on typed reasons A `build-for-testing` failure was the one Apple failure shape with no typed reason: `runner-artifact.ts` threw COMMAND_FAILED with a hint picked by substring-matching the lowercased message plus a JSON dump of the details, and nothing downstream could switch on which signing problem it was. `AGENTS.md` keys behavior on typed reasons and details, never error text, so the build path was the exception to the rule. `classifyRunnerStartupFailure` is now the one classifier reachable from that catch, and it reads `RUNNER_ERROR_RULES` rather than adding a second table: a row either carries recovery verdicts, a `buildFailure` reason and hint, or both. The reason and the hint beside it therefore cannot disagree, and an unproven cause is never claimed -- `build_failed_unclassified` keeps the cache-recovery hint it already gave. The host `DevToolsSecurity` refusal, which had a hint and no reason at all, now publishes its own reason keyed on the typed status it read, and that reason says host so it can never be read as the device's Developer Mode toggle. `resolveRunnerBuildFailureHint` is gone, not kept as a pass-through. Reasons are recorded as fixtures carrying the tool output, the command, the Xcode, and how the line reached the file; each one is driven through the real build catch and asserted on the normalized envelope, because every case is COMMAND_FAILED and the reason is the assertion. * refactor(ios-runner): reach the host dev-tools probe only when a device needs it A static edge from `runner-session.ts` to the new module grew three Apple facade closures by one module each (app-lifecycle, doctor, runner-operations), which the eager-import-closure ratchet refuses: the runner subtree is eagerly evaluated to answer a simulator request, and this preflight only ever runs for a physical iOS device. Function-scoped import keeps the preflight where it belongs without paying for it on the paths that can never use it. * fix(ios-runner): classify a build failure only from what xcodebuild published The startup rules read the whole `details` bag as JSON, and `execFailureDetails` puts `cmd`/`args` in that bag. A caller who pinned a profile therefore handed the profile rule its trigger: an unrelated Swift compile error classified as `signing_provisioning_profile_missing` and lost the cache-recovery hint. The same bag holds the `reason` and `hint` this classifier publishes, so a re-wrapped failure would match its own verdict. `runnerToolText` now builds the one haystack a startup rule may read — our message plus the tool's stdout and stderr — and the argv case has a fixture that proves a pinned profile is not evidence (#2680). Reading only `details` also dropped the message on the floor, so a failure the exec layer raised as a plain `Error` (which the catch wraps with `String(err)`) became unclassified while the same sentence in `stdout` classified. The message is in the haystack now, and a `message-only` fixture keeps both carriers at parity. `signing_style_conflict` is removed rather than kept as a guess: the hint named a `CODE_SIGN_STYLE` env lever that does not exist and claimed a cause nothing captured. The conflicting-settings line keeps a row so the profile row below cannot answer it with missing-profile advice, but it publishes `build_failed_unclassified` until a capture shows which setting disagrees and which lever clears it. Versioned help and the installation docs no longer list the withdrawn reason, and `.device-evidence/CHECKLIST.md` gained the capture that would let a follow-up claim it. Provenance got honest vocabulary. `OBSERVED_ON` stamped invented sentences as observed on Xcode 26.2, and one invocation was recorded as the producer of seven configurations that cannot coexist; nothing here was captured, so entries now say `shipped-sniff-trigger` (the matched substrings shipped before #2680, sentence reconstructed) or `invented-shape`, carry `xcodeVersion: 'unobserved'`, and omit `command` until a run records one. * fix(apple): take the exec result type from host-kit and thin the fixture assertion body #2689 derived the runner host port from the modules it fronts, so `ExecResult` is read from `@agent-device/host-kit/command` now rather than restated in `runner/host.ts`. The per-fixture assertion body had grown past the complexity the Fallow audit allows a changed file: the envelope checks move into `assertFailureEnvelope` and the tool-output check into `assertToolOutputReachable`, so each recorded shape is still asserted through the same path and the test body reads as one call. * fix(apple): make a named provisioning profile earn its reason One bare `provisioning profile` substring was the shipped sniffer's trigger, and it is a phrase a failing build prints while talking about something else: the codesign invocation, a settings dump, a note about the profile it used. Each row now requires the profile plus the complaint Xcode attaches to it — its `IDEProvisioningErrorDomain` diagnostic, "doesn't include", "has expired" — and a mention that says nothing keeps the cache-recovery advice it already had. * fix(apple): require a profile and its complaint in one line A whole-log AND proves two phrases exist, not that one qualifies the other: a note about the profile the build used, three lines above an unrelated expired-certificate warning, published `signing_provisioning_profile_missing` for a profile that was fine. The profile rows now read one line at a time and the expiry row asks for Xcode's own `has expired` phrase. The rules table grows a `toolTextLineIncludesAll` axis beside `toolTextIncludesAll`, and the bundle-identifier row that also paired two phrases moves onto it so no startup row is left reading a whole log; its cross-line negative rides along. `signing_no_profiles_for_bundle_id` shares the `PROFILE_UNUSABLE` advice instead of spelling it out a third time. `.device-evidence/CHECKLIST.md` belongs to the merged #2682 lane, so this stack's capture sheet lives beside it as `CHECKLIST-runner-failures.md`. * test(ios-runner): record the captured profile row and the host gating the rest need Co-Authored-By: Apex <noreply@callstack.com> --------- Co-authored-by: Apex <noreply@callstack.com>
1 parent 42ad0bc commit 937ccdc

11 files changed

Lines changed: 1115 additions & 53 deletions

File tree

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
# Runner-failure evidence checklist (#2680, #2683)
2+
3+
Live evidence the coordinator runs serially on the connected iPhone. Each item names the exact
4+
command, the environment it needs, and the rendered error that proves the change. Do not paraphrase
5+
the JSON: paste it. Record the commit SHA the build under test was made from.
6+
7+
## #2680 — typed build-failure reasons
8+
9+
Build the CLI first, then stop any warm daemon so the run is on this commit:
10+
11+
```sh
12+
pnpm build && pnpm clean:daemon
13+
node --experimental-strip-types src/bin.ts daemon stop --all || true
14+
```
15+
16+
### 1. Signing with no team configured -> `signing_no_development_team`
17+
18+
```sh
19+
env -u AGENT_DEVICE_IOS_TEAM_ID -u AGENT_DEVICE_IOS_PROVISIONING_PROFILE \
20+
node --experimental-strip-types src/bin.ts --json \
21+
prepare ios-runner --platform ios --device "<physical iPhone name>"
22+
```
23+
24+
Expected: exit non-zero, one error object with
25+
26+
```json
27+
{
28+
"code": "COMMAND_FAILED",
29+
"message": "xcodebuild build-for-testing failed",
30+
"hint": "Configure signing in Xcode or set AGENT_DEVICE_IOS_TEAM_ID for physical-device runs.",
31+
"details": { "reason": "signing_no_development_team" }
32+
}
33+
```
34+
35+
`hint`, `logPath` and `diagnosticId` are top-level, never inside `details`. Also record the Xcode
36+
version (`xcodebuild -version`) so the `signing_no_development_team` fixture in
37+
`packages/platform-apple/src/runner/__tests__/runner-startup-failure-fixtures.ts` can move from
38+
`shipped-sniff-trigger` to `captured` and its `xcodeVersion` from `unobserved` to that version.
39+
Paste the whole error so the fixture's `output` can become the capture and its `command` can be
40+
recorded.
41+
42+
### 2. A bundle identifier somebody else already owns -> `bundle_identifier_already_registered`
43+
44+
```sh
45+
env AGENT_DEVICE_IOS_TEAM_ID="<your team id>" \
46+
AGENT_DEVICE_IOS_BUNDLE_ID="com.apple.TestFlight" \
47+
node --experimental-strip-types src/bin.ts --json \
48+
prepare ios-runner --platform ios --device "<physical iPhone name>"
49+
```
50+
51+
Expected: same envelope shape with
52+
`details.reason: "bundle_identifier_already_registered"` and a hint naming
53+
`AGENT_DEVICE_IOS_BUNDLE_ID`. A registered-but-foreign identifier may surface the `Failed registering
54+
bundle identifier` line or the `App Identifier ... is not available` line; both rules produce this
55+
one reason, so record which line xcodebuild printed.
56+
57+
### 3. Reasons with no device exposure (host-side or configuration-only)
58+
59+
These do not need the iPhone, but do need a real xcodebuild run; record output with the Xcode
60+
version so the matching fixture's `provenance` can be upgraded:
61+
62+
```sh
63+
# devtools_security_developer_mode_disabled (macOS admin state, no device work)
64+
DevToolsSecurity -status
65+
66+
# signing_provisioning_profile_missing: point the runner build at a profile that is not installed,
67+
# then read the reason off the same prepare command as above.
68+
env AGENT_DEVICE_IOS_TEAM_ID="<your team id>" \
69+
AGENT_DEVICE_IOS_PROVISIONING_PROFILE="no-such-profile-installed" \
70+
node --experimental-strip-types src/bin.ts --json \
71+
prepare ios-runner --platform ios --device "<physical iPhone name>"
72+
```
73+
74+
Expected: `details.reason` is `signing_provisioning_profile_missing`, and the profile rows only fire
75+
when xcodebuild says what is wrong with the profile **on the same line as the profile**: its
76+
`IDEProvisioningErrorDomain` diagnostic naming the profile, `doesn't include ...`, or `has expired`
77+
(#2688 review). Paste the whole error and keep the line breaks — which line carried which phrase is
78+
what promotes the `profile-xcode-signing-error`, `profile-does-not-cover-app-id` and `profile-expired`
79+
fixtures from `invented-shape` to `captured`, and a capture that splits the two phrases across lines
80+
belongs to `profile-note-above-an-expired-certificate` instead. A different reason is worth recording
81+
just as much: say which one and treat the fixtures as unconfirmed rather than editing the rules to
82+
fit.
83+
84+
### 4. The line that claims no reason yet -> `build_failed_unclassified`
85+
86+
`xcodebuild` reports a settings mismatch with a line that names a profile ("has conflicting
87+
provisioning settings"). #2680 deliberately publishes `build_failed_unclassified` for it, because no
88+
capture has proved which lever clears it. To reach it, pin a profile while leaving automatic signing
89+
on:
90+
91+
```sh
92+
env AGENT_DEVICE_IOS_TEAM_ID="<your team id>" \
93+
AGENT_DEVICE_IOS_PROVISIONING_PROFILE="match-development" \
94+
node --experimental-strip-types src/bin.ts --json \
95+
prepare ios-runner --platform ios --device "<physical iPhone name>"
96+
```
97+
98+
Expected: either `signing_provisioning_profile_missing` (xcodebuild complained about the profile and
99+
said what was wrong with it) or `build_failed_unclassified` — which is also what a run that merely
100+
mentions the profile it used gets, since a name is not a complaint (#2688 review). Paste the error and the `xcodebuild -version` either way: a
101+
capture of the conflicting-settings line is what would let a follow-up name the cause, and the
102+
capture must show which build setting disagrees before any hint naming a lever is written.
103+
104+
## Results — coordinator run, 2026-09-20
105+
106+
Built from `15808ae228` on `thymikee-iphone` (iPhone 17 Pro, iOS 27.0, build 24A437), cabled.
107+
108+
```
109+
$ xcodebuild -version
110+
Xcode 26.2
111+
Build version 17C52
112+
```
113+
114+
Section 3 is captured. A device build pointed at a team with no certificate, on a fresh derived
115+
path so no cached artifact short-circuits it, reaches signing and fails with one long `error:` line
116+
per target:
117+
118+
```
119+
.../AgentDeviceRunner.xcodeproj: error: No Accounts: Add a new account in Accounts settings. (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')
120+
.../AgentDeviceRunner.xcodeproj: error: No profiles for 'com.callstack.agentdevice.runner' were found: Xcode couldn't find any iOS App Development provisioning profiles matching 'com.callstack.agentdevice.runner'. (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')
121+
```
122+
123+
`details.reason` is `signing_provisioning_profile_missing` with the profile hint, and the fixture
124+
`no-profiles-for-bundle-id` is now `captured` with this transcript verbatim. This is the run that
125+
answers the wrapping question the rows were held on: the matched phrase arrives inside one `error:`
126+
line, so the sibling rows in the same provisioning family do not split the way a wrapped line would.
127+
128+
Sections 1 and 2 are blocked on this account, and the mechanism is worth recording because it is the
129+
same for all of them: against a signed-in account with a valid identity, `xcodebuild` is invoked with
130+
`-allowProvisioningUpdates`, so the build either signs successfully or dies earlier than the
131+
diagnostic a row keys on.
132+
133+
- Unsetting `AGENT_DEVICE_IOS_TEAM_ID` **succeeds** — automatic signing resolves the team from the
134+
installed identity and reuses an installed team profile. So section 1's `signing_no_development_team`
135+
cannot be induced here; it needs an account signed in with no development team.
136+
- `AGENT_DEVICE_IOS_BUNDLE_ID=com.apple.TestFlight` **succeeds** for the same reason, and a bogus
137+
`AGENT_DEVICE_IOS_PROVISIONING_PROFILE` is repaired rather than honoured. So section 2's
138+
`bundle_identifier_already_registered` needs an app id owned by a different team that automatic
139+
signing cannot register.
140+
- The same gating applies to `bundle_identifier_unavailable` (`App Identifier` + `not available`),
141+
`profile-does-not-cover-app-id` (`Provisioning profile` + `doesn't include`) and `profile-expired`
142+
(`Provisioning profile` + `has expired`): each needs a profile or app id already claimed elsewhere,
143+
which this account will not produce. Recorded beside the fixtures in
144+
`runner-startup-failure-fixtures.ts` so the rows read as host-gated, not unexamined.
145+
146+
Section 4 needs a build that fails for an unrelated reason while naming no signing fact; the
147+
classifier's behaviour there is pinned by `runner-startup-failure-reasons.test.ts` and needs no
148+
device claim to hold.

‎packages/platform-apple/src/runner/__tests__/runner-client.test.ts‎

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,6 @@ import type { DeviceInfo } from '@agent-device/kernel/device';
3434
import { isReadOnlyRunnerCommand } from '../runner-command-traits.ts';
3535
import {
3636
isRetryableRunnerError,
37-
resolveRunnerBuildFailureHint,
3837
resolveRunnerEarlyExitHint,
3938
shouldRetryRunnerConnectError,
4039
withRunnerCommandId,
@@ -437,15 +436,6 @@ test('resolveRunnerEarlyExitHint falls back to runner connect timeout hint', ()
437436
assert.match(hint, /pnpm clean:xcuitest/i);
438437
});
439438

440-
test('resolveRunnerBuildFailureHint suggests cache cleanup for non-signing failures', () => {
441-
const hint = resolveRunnerBuildFailureHint(
442-
new AppError('COMMAND_FAILED', 'xcodebuild build-for-testing failed'),
443-
);
444-
445-
assert.match(hint, /pnpm clean:xcuitest/i);
446-
assert.match(hint, /~\/\.agent-device\/apple-runner\/derived/i);
447-
});
448-
449439
test('shouldRetryRunnerConnectError does not retry xcodebuild early-exit errors', () => {
450440
const err = new AppError(
451441
'COMMAND_FAILED',
Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
import assert from 'node:assert/strict';
2+
import { beforeEach, test, vi } from 'vitest';
3+
import { AppError, normalizeError } from '@agent-device/kernel/errors';
4+
import { appleRunnerTestHost } from '../test-host.ts';
5+
import { assertDevToolsSecurityForIosRunner } from '../runner-dev-tools-security.ts';
6+
import { IOS_DEVICE, IOS_SIMULATOR } from './device-fixtures.ts';
7+
import { RUNNER_STARTUP_FAILURE_FIXTURES } from './runner-startup-failure-fixtures.ts';
8+
9+
/**
10+
* `DevToolsSecurity -status` answers for the Mac, not for the iPhone (#2680). The refusal this probe
11+
* threw used to carry a hint and no reason, so a caller could only match its wording — and that
12+
* wording is nearly the same as the device's own Developer Mode state, which is a different fact on
13+
* a different machine. These cases pin that the host refusal publishes the host's own reason, keyed
14+
* on the status it read rather than on the sentence it printed.
15+
*/
16+
17+
const HOST_REFUSAL_FIXTURE = RUNNER_STARTUP_FAILURE_FIXTURES.find(
18+
(fixture) => fixture.reason === 'devtools_security_developer_mode_disabled',
19+
);
20+
21+
const runAppleToolCommand = vi.fn();
22+
23+
beforeEach(() => {
24+
runAppleToolCommand.mockReset().mockResolvedValue({ exitCode: 0, stdout: '', stderr: '' });
25+
appleRunnerTestHost.update({ runAppleToolCommand });
26+
});
27+
28+
test('the host DevToolsSecurity refusal publishes its own typed reason', async () => {
29+
assert.ok(HOST_REFUSAL_FIXTURE);
30+
mockDevToolsSecurityOutput(HOST_REFUSAL_FIXTURE.output);
31+
const expectedStatus = HOST_REFUSAL_FIXTURE.output.trim();
32+
33+
await assert.rejects(
34+
() => assertDevToolsSecurityForIosRunner(IOS_DEVICE),
35+
(error: unknown) => {
36+
assert.ok(error instanceof AppError);
37+
assert.equal(error.code, 'COMMAND_FAILED');
38+
assert.match(error.message, /Developer mode is disabled/);
39+
assert.equal(error.details?.reason, 'devtools_security_developer_mode_disabled');
40+
assert.equal(error.details?.devToolsSecurityStatus, expectedStatus);
41+
42+
// What the caller renders: hint at top level, reason in details.
43+
const envelope = normalizeError(error, { diagnosticId: 'diag-devtools-1' });
44+
assert.match(String(envelope.hint), /DevToolsSecurity -enable/);
45+
assert.equal(envelope.diagnosticId, 'diag-devtools-1');
46+
assert.equal(envelope.details?.hint, undefined);
47+
assert.equal(envelope.details?.diagnosticId, undefined);
48+
assert.equal(envelope.details?.reason, 'devtools_security_developer_mode_disabled');
49+
return true;
50+
},
51+
);
52+
});
53+
54+
test('an enabled host developer mode is not a failure', async () => {
55+
mockDevToolsSecurityOutput('Developer mode is currently enabled for development tools.\n');
56+
57+
await assert.doesNotReject(() => assertDevToolsSecurityForIosRunner(IOS_DEVICE));
58+
});
59+
60+
test('a simulator never takes the host probe', async () => {
61+
mockDevToolsSecurityOutput('Developer mode is currently disabled.\n');
62+
63+
await assert.doesNotReject(() => assertDevToolsSecurityForIosRunner(IOS_SIMULATOR));
64+
65+
assert.equal(
66+
runAppleToolCommand.mock.calls.some((call) => call[0] === 'DevToolsSecurity'),
67+
false,
68+
);
69+
});
70+
71+
function mockDevToolsSecurityOutput(stdout: string): void {
72+
runAppleToolCommand.mockImplementation(async (cmd: string) => ({
73+
exitCode: 0,
74+
stdout: cmd === 'DevToolsSecurity' ? stdout : '',
75+
stderr: '',
76+
}));
77+
}

0 commit comments

Comments
 (0)