fix(flow): let a flow launch an app that can never be instrumented - #677
fix(flow): let a flow launch an app that can never be instrumented#677filip131311 wants to merge 1 commit into
Conversation
A flow's launch step waits for native devtools on iOS, and an Apple system app
is a platform binary with library validation, so that connection can never
happen. The step burned the full 8s timeout, failed, and took the rest of the
flow with it — even when nothing in the flow needed the view hierarchy. The
advice it gave ("re-run", "restart the argent server") could never work, which
is the retry loop #453 set out to remove.
The signal was already there: isInjectableBundleId, used by
native-devtools-status and by describe, whose own docs call injectable:false a
terminal state. The gate never consulted it. It does now, and the launch passes
with a warning saying what does and does not work for such an app.
Removing the wait exposes something the wait was accidentally hiding, so the
same change has to close it. Selector steps reach resolveNativeTargetApp, which
auto-targets whatever app is connected and accepts one reporting "inactive" as
foreground-like. An app the user was driving reports exactly that while it
backgrounds. Today the first tree read lands ~9.5s after launch, past that
window; without the wait it lands at ~1.5s, inside it — so a selector could
resolve against a different app's tree, and a hidden assertion against it would
pass. The run now records that the launched app cannot be instrumented and
fails tree reads immediately with a terminal reason.
The guard sits on settleTree and waitForCondition rather than on step kinds, so
it covers tap, long-press, await, assert, scroll-to and snapshot at once, and
leaves selector-less pinch and rotate working — those degrade to a default
aspect and never read a tree.
Reporting a pass rather than a failure is deliberate: the launch did succeed,
and the control case in the issue proves the rest of such a flow runs fine. The
warning field was already rendered by both the CLI and MCP as legacy
wire-compat; it is now produced, so those comments no longer describe it as
something only an old server sends.
The recording walkthrough in the create-flow skill used com.apple.Preferences
with selector steps, which could never have been captured or replayed. It now
uses a third-party app.
|
This is likely a duplicate of #560, although the very same bug seems fixed using a different approach. |
latekvo
left a comment
latekvo
left a comment
There was a problem hiding this comment.
[Diplomat]: Reviewed against bce41535, comparing behaviour with origin/main (3fff176a). Evidence is one identical test file run on both, driven through the real createRunFlowTool: the native-devtools service is faked (the pattern in native-target-app.test.ts), not fetchFlowTree, so the whole chain runs — settleTree → fetchFlowTree → queryFullHierarchyTree → resolveNativeTargetApp → queryViewHierarchy → adaptFullHierarchy.
main: 16/16 passed. This branch: 4 failed | 12 passed.
Two things I want to say plainly before the findings. The launch-step fix is right — a flow that launches a system app should not have its launch step errored, and the control in #623 proves the rest of such a flow runs. And the race this PR raises in its description is real: I reproduced it, and on a build with the gate skip kept but the two guards removed, launch of a non-connecting system app followed by assert hidden does flip to a false pass. That case is genuine and the guard does close it.
The findings below are about the guard's derivation, not about whether the problem is worth solving.
| } | ||
|
|
||
| export async function settleTree(env: ActionEnv): Promise<DescribeNode | undefined> { | ||
| if (env.nonInjectableApp) { |
There was a problem hiding this comment.
[Diplomat]: This derives a terminal verdict about the view hierarchy from bundleId.startsWith("com.apple.") rather than from whether the hierarchy read succeeds, and where those two disagree it turns a passing flow into a failing one.
With com.apple.Preferences in listConnectedBundleIds(), isConnected() returning true, applicationState: "active" and a getFullHierarchy payload containing Settings, the step assert: { visible: "Settings" } after launch: com.apple.Preferences:
main @3fff176a -> launch:pass, assert:pass ok=true
this branch -> launch:pass, assert:fail ok=false
with the reason "... can never be injected into it and selector-based steps cannot resolve. This is terminal — relaunching or restarting the argent server will not change it." — emitted while the read that message describes as impossible had already returned a tree containing Settings. Same for tap: { text: Search }.
That state is not hypothetical: it is what was measured on an iOS 18.5 simulator during the review of #560 — lsof showing both libArgentInjectionBootstrap.dylib and libNativeDevtoolsIos.dylib mapped into the running process, the process holding a live unix peer of the tool-server's own /tmp/argent-nd-<udid>.sock, native-devtools-status returning "connected": true and "injectable": false in the same response, and a selector fragment passing 4/4. #453 recorded connected: false on iOS 26.5 and #623 was filed from a 26.5 matrix, so the evidence splits by runtime; the message states one runtime's reading as a universal.
The clearest form of it is a pair that differs by one line of YAML. Same app, same connection, same three steps, same tree:
no `launch` step -> 3/3 pass on BOTH branches (the guard never arms)
`launch` prefixed -> assert:fail, tap:skip, assert:skip on this branch only
A second consequence: for a system app that genuinely never connects, this branch emits byte-identical output to the connected case above, so the two are no longer distinguishable from the report. On the guards-removed build the unreadable case still fails, but says "could not read the UI tree: No native-devtools-connected apps are available for auto-targeting."
On isolating it — deleting only the two env.nonInjectableApp guards, keeping the flow-run.ts:298 gate skip and the launch warning, turns the whole file green at 16/16. The gate skip is not implicated.
On the 8 s the description cites as the cost being removed: waitForNativeDevtools tests api.isConnected(bundleId) at the top of its loop, before the first sleep, and isConnected is a synchronous Map.has. A connected app therefore already returns in ~0 ms on main. Measured, launching a connected system app takes 1507 ms on this branch and 1508 ms on main — the wait being removed never fires for the case that regresses. The full 8 s elapses only when the app never connects (9522 ms), and there main errored the launch, so no selector step ran.
| // wait is the one condition that would otherwise resolve TRUE off an | ||
| // unreadable screen. | ||
| if (env.nonInjectableApp) { | ||
| return { ok: false, reason: nonInjectableTreeReason(env.nonInjectableApp) }; |
There was a problem hiding this comment.
[Diplomat]: This early return omits indeterminate, so a when: guard reads an unreadable tree as a plainly-false condition and green-skips the block.
This one does not depend on the injectability question at all — it fires when the premise is entirely correct, on an app that genuinely never connects.
probeWhenCondition's own docstring states the contract: "indeterminate distinguishes an unreadable tree (the caller errors — unknown is not false) from a plainly unmet condition (the caller skips)." execWhenStep branches on exactly that flag, and its docstring adds that "silently skipping would let a broken tree source turn every guarded dismissal into a green no-op." Every other unreadable-tree exit in this same function sets it (the lastTrustedReadAt === undefined arm, and both !lastReadTrusted arms).
Driving launch: com.apple.Preferences then when: { hidden: "Onboarding" } wrapping an echo, against a registry whose resolveService throws:
this branch -> launch:pass, when:skip, echo:skip ok=TRUE
when reason: 'condition not met (hidden text="Onboarding") — block skipped (1 step)'
main -> launch:error, when:skip ok=false
The run reports green and the report asserts the element was not hidden, which nothing observed. The sibling await: { hidden: ... } on the identical state correctly reports fail, so the same condition yields two different verdicts depending only on which directive asked. That is the shape of #519, reached through the when: door.
| if (!(await sleepOrAbort(POST_LAUNCH_SETTLE_MS, signal))) return ABORTED_OUTCOME; | ||
| // Recorded on every launch, so a later injectable launch clears it. | ||
| state.nonInjectableApp = | ||
| device.platform === "ios" && !isInjectableBundleId(bundleId) ? bundleId : undefined; |
There was a problem hiding this comment.
[Diplomat]: The guard is keyed on the launched bundle id, but the read it guards resolves a different quantity: flow-ios-tree.ts:287 calls resolveNativeTargetApp(nativeApi, undefined), which auto-targets from the connected list and never consults state.nonInjectableApp. Two consequences fall out of the mismatch.
It is bypassable. tool: { name: launch-app, args: { bundleId: com.apple.Preferences } } followed by the same assert never sets the flag:
tool:pass, assert:pass on BOTH branches
Same app, same launch, same tree — the guard is silent because the launch did not go through the launch: directive. So the terminal claim holds or not depending on which spelling of "launch this app" the flow used.
It fires for an app that was never the read target. Launching com.apple.Preferences while a connected, active, injectable com.example.myapp is what auto-target would resolve: this branch blocks that read with "com.apple.Preferences ... can never be injected", naming an app that is not the one the read would have used.
| warning: | ||
| `${bundleId} is an Apple system app: it is a platform binary with library validation, so ` + | ||
| `argent's view-hierarchy instrumentation can never be injected into it. The app launched — ` + | ||
| `coordinate steps (\`tap: { x, y }\`), \`wait\` and \`snapshot\` work; selector-based steps ` + |
There was a problem hiding this comment.
[Diplomat]: This warning tells the flow author that snapshot works for such an app, and the guard added in this PR contradicts it.
snapshot: { cropOn } routes through waitForFrame into settleTree, which is the flow-actions.ts:342 throw. On launch: com.apple.Preferences + snapshot: { name: row, cropOn: { text: Settings } }:
"kind": "snapshot", "status": "error",
"reason": "`com.apple.Preferences` is an Apple system app, ... This is terminal ..."
On main the same step gets past the tree read. A plain snapshot with no cropOn does work, because flow-visual.ts swallows that throw — so the sentence is true for one form of the directive and false for the other. The PR description's design-choices paragraph lists snapshot as covered by the guard, which is the opposite of what this line tells the author.
2 participants
Fixes #623.
Reproduced
Control — the identical flow without the launch step passes (
ok=true). So the runner was fine; the gate alone made system apps undriveable, and it gave advice that can never work for them.The signal already existed:
isInjectableBundleIdis used bynative-devtools-status(whose docs callinjectable: falseterminal — do NOT restart/retry) and bydescribe. The flow gate never consulted it, so it burned the full 8 sNATIVE_READY_TIMEOUT_MSwaiting for a connection that cannot occur.After:
Removing the wait exposes a race the wait was hiding — so this closes it too
This is the part I'd want a reviewer to look at hardest.
Selector steps reach
resolveNativeTargetApp, which auto-targets whatever app is connected.chooseFrontmostConnectedApphas a weak tier acceptingapplicationState === "inactive"— exactly what an app reports while it backgrounds. Today the first tree read lands ~9.5 s after launch (8 s gate + 1.5 s settle), comfortably past that window. Without the wait it lands at ~1.5 s, inside it.So a selector step could resolve against a different app's tree. Most conditions then fail noisily, but
hidden-shaped ones (assert: { hidden: X },await: { hidden: X }, awhen: { hidden: X }guard reporting a green skip) would pass. The fix records that the launched app cannot be instrumented and fails tree reads immediately with a terminal reason, which closes it.Without that guard the later path was also worse than the gate it replaced:
NATIVE_TARGET_NO_CONNECTED_APPSsays "Launch or restart the app first", after a 3 s settle.Design choices
Guard on tree reads, not step kinds —
settleTree+waitForCondition. That coverstap/long-press/await/assert/scroll-to/snapshotin two places, and leaves selector-lesspinch/rotateworking (they degrade to a default aspect and never read a tree). Guarding by step kind would have missedscroll-to, which callssettleTreedirectly.No pre-flight "does this flow use selectors" scan.
launchcan appear anywhere, including inside a nested fragment, andrun:fragments are read lazily at execution time — so "does this flow contain selectors" is ill-posed (which steps? after this launch? under awhen:that may not be entered?).Pass, not fail. The launch genuinely succeeded — the issue's own control proves the rest of such a flow runs. Failing it would re-introduce the bug for coordinate flows.
No 5th status. The CLI maps status with no fallback, so an unknown value renders
undefined. Used the existingwarningfield instead: ⚠ replaces the pass glyph, the text prints under the step, and it's counted in the summary. Both renderers already handle it as legacy wire-compat, so an old CLI renders it correctly — those comments are updated, since it's now actually produced.Also fixed
The create-flow skill's recording walkthrough used
com.apple.Preferenceswith selector steps (tap: { text: General }). That could never have been captured (selector capture reads the same tree) or replayed. Switched to a third-party app.Not fixed, deliberately
A fragment with no
launchstep, run while a system app is frontmost, never sets the flag and still gets the raw message. That's the issue's own control case.Checks
flow-composition.test.ts:213, which pins the retry-worded failure for a genuinely injectable app.