Skip to content

feat(flow): record the screen identity gate from a fingerprint read #729

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
hubgan wants to merge 3 commits into from
Closed

feat(flow): record the screen identity gate from a fingerprint read #729

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9779bd4
feat(tools): add screen-fingerprint, a stable answer to "which screen…
hubgan Aug 4, 2026
88670cc
feat(flow): add screen identity and readiness conditions to flows
hubgan Aug 4, 2026
3715422
feat(flow): record the screen identity gate from a fingerprint read
hubgan Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions packages/skills/rules/argent.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ Decision order:
If the user started Metro separately, ask whether to call `stop-metro` (specify the port if not 8081).
- If tools provided by mcp-server are not sufficient and action can be done using `xcrun`, `adb`, or other commands, use the command. Examples: changing device options, performing a device action such as lock, shake, etc.
- When waiting for an action, do not call `screenshot` repeatedly without a proper wait mechanism. Use the `await-ui-element` tool to block until the UI settles (e.g. wait for an element to become `visible`/`hidden`, or to contain expected `text`) instead of polling.
- To confirm which screen a React Native app is on, use `screen-fingerprint` — it reads the app's focused navigation route. It answers only "which screen": it does not see a modal or system alert on top, and the route commits before the transition finishes animating.
</general_rules>

<react_native_detection>
Expand Down
68 changes: 48 additions & 20 deletions packages/skills/skills/argent-device-interact/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,25 +56,27 @@ Common schemes: `messages://`, `settings://`, `maps://?q=<query>`, `tel://<numbe

## 4. Choosing the Right Tool

| Action | Tool | Notes |
| ----------------- | ------------------ | ---------------------------------------------------------------- |
| Multiple actions | `run-sequence` | Batch steps in one call (no intermediate screenshots) |
| Open an app | `launch-app` | **Always — never tap home-screen icons** |
| Restart an app | `restart-app` | Terminate and relaunch by bundle ID |
| Open URL/scheme | `open-url` | Web pages, deep links, URL schemes |
| Single tap | `gesture-tap` | Buttons, links, checkboxes |
| Scroll/swipe | `gesture-swipe` | Straight-line scroll or swipe |
| Scroll (Chromium) | `gesture-scroll` | Wheel-based; deltas are window fractions, positive deltaY = down |
| Drag (Chromium) | `gesture-drag` | Sliders, drag-and-drop, text selection |
| Long press | `gesture-custom` | Context menus, drag start |
| Drag & drop | `gesture-custom` | Complex drag interactions |
| Pinch/zoom | `gesture-pinch` | Two-finger pinch with auto-interpolation |
| Rotation | `gesture-rotate` | Two-finger rotation with auto-interpolation |
| Custom gesture | `gesture-custom` | Arbitrary touch sequences, optional interpolation |
| Hardware key | `button` | Home, back, power, volume, appSwitch, actionButton |
| Type text | `keyboard` | iOS+Android. Supports Enter, Escape, arrows |
| Rotate device | `rotate` | Orientation changes |
| Wait for UI | `await-ui-element` | Block until an element is visible/hidden/exists/contains text |
| Action | Tool | Notes |
| ----------------- | -------------------- | ---------------------------------------------------------------- |
| Multiple actions | `run-sequence` | Batch steps in one call (no intermediate screenshots) |
| Open an app | `launch-app` | **Always — never tap home-screen icons** |
| Restart an app | `restart-app` | Terminate and relaunch by bundle ID |
| Open URL/scheme | `open-url` | Web pages, deep links, URL schemes |
| Single tap | `gesture-tap` | Buttons, links, checkboxes |
| Scroll/swipe | `gesture-swipe` | Straight-line scroll or swipe |
| Scroll (Chromium) | `gesture-scroll` | Wheel-based; deltas are window fractions, positive deltaY = down |
| Drag (Chromium) | `gesture-drag` | Sliders, drag-and-drop, text selection |
| Long press | `gesture-custom` | Context menus, drag start |
| Drag & drop | `gesture-custom` | Complex drag interactions |
| Pinch/zoom | `gesture-pinch` | Two-finger pinch with auto-interpolation |
| Rotation | `gesture-rotate` | Two-finger rotation with auto-interpolation |
| Custom gesture | `gesture-custom` | Arbitrary touch sequences, optional interpolation |
| Hardware key | `button` | Home, back, power, volume, appSwitch, actionButton |
| Type text | `keyboard` | Every platform. Supports Enter, Escape, arrows (not on TV) |
| Rotate device | `rotate` | Orientation changes |
| Wait for UI | `await-ui-element` | Block until an element is visible/hidden/exists/contains text |
| Wait for idle | `await-screen-idle` | Block until a non-empty screen tree stops changing |
| Identify a screen | `screen-fingerprint` | Read which screen a React Native app is on, as its route path |

## 5. Finding Tap Targets

Expand Down Expand Up @@ -203,11 +205,37 @@ Instead of polling `screenshot`/`describe` in a loop, use `await-ui-element` to
- `selector`: `{ text?, identifier?, role? }` — every provided field must match. `text` matches the element's label or value and `role` its element role (e.g. `AXButton`, `button`, `TextView`, `StaticText`), both as case-insensitive substrings; `identifier` matches its accessibility id / resource-id / testID **exactly** (case-insensitive), also accepting the unqualified Android resource-id name (`submit` matches `com.example.app:id/submit`). The synthetic `ROOT` container `describe` prints is never matched, so a `role` like `AXGroup`/`html` won't trivially "match the screen".
- Prefer a **specific** selector. A loose substring can match several elements, and the tool may then key off one you didn't mean: `text` reads the first **visible** match in **reading order** (top-to-bottom, left-to-right — the same order `describe` lists them, so it's the one you saw first; when no match is visible, the first match overall), while `visible`/`exists` are satisfied by **any** match. Disambiguate with a longer or more exact string, an `identifier`, or a `role` (e.g. pin to a text role like `StaticText` to skip a same-named button). On a `text` timeout the `note` quotes the matched element's text, so you can see which one it landed on.
- `text` condition also needs `expectedText` (substring the matched element must contain).
- `hidden` treats a selector that matches **nothing** as already-hidden, so a typo'd selector returns an instant (false) success. Double-check the selector for `hidden` waits — the result `note` flags when the selector never matched any element. (On iOS, if the accessibility backend is down the tree comes back empty; the tool will **not** report `hidden` success off such a degraded read and the `note` surfaces the boot hint instead.)
- `hidden` treats a selector that matches **nothing** as already-hidden, so a typo'd selector returns an instant (false) success. The result `note` flags when the selector never matched any element; treat that note as a failed check, not a pass, and find the real selector before continuing. `flow-add-step` refuses to record such a wait, because a gate that cannot fail proves nothing on replay. (On iOS, if the accessibility backend is down the tree comes back empty; the tool will **not** report `hidden` success off such a degraded read and the `note` surfaces the boot hint instead.)
- Optional `timeoutMs` (default 5000) and `pollIntervalMs` (default 400).

Returns `{ success, elapsed }`; on a timeout `success` is `false` and a `note` explains what was seen.

### await-screen-idle — Block until the screen tree settles

Use this after launch/navigation and before a raw tap when an early-painted element may still be moving or non-interactive:

```json
{ "udid": "<UDID>", "timeoutMs": 3000, "minStableMs": 250 }
```

The tool supports local iOS, Android, and Chromium. It polls the same accessibility/DOM tree as `describe` until a non-empty tree remains unchanged. Continue only when it returns `settled: true`; `settled: false` is a soft timeout result, not a thrown error. Pair it with `await-ui-element` for a destination-specific landmark because an idle wrong screen is still wrong, and verify the action's outcome because tree stability does not directly test hit-testing. Call it directly for live diagnosis: do not add it with `flow-add-step` or persist it as a raw flow `tool:` step, where a soft `settled: false` result would not hard-fail replay. Do not place it inside `run-sequence` (it is not an allowed nested tool). Flow selector actions already resolve their target from a settled flow tree during replay.

Within a flow, the persistable equivalent is `await: { idle: true }`, which hard-fails on timeout.

### screen-fingerprint — Which screen is the app on

For a React Native app served by Metro, this reads the focused React Navigation route path and returns it as `"HomeTab>Profile"`:

```json
{ "app_id": "com.acme.notes" }
```

It answers "which screen" and nothing else — pair it with `await-screen-idle` for readiness, and an element check for an overlay above the screen.

- `available: false` — this app has no reader (release build, fully native app, Chromium, Metro down). Recognize the screen by a destination-only element instead.
- `route: null` with `available: true` — no focused route at this instant: a native screen, or a transition still in flight. Let it settle and probe again.
- Optional `platform`, `device`, and `metro_port` (default 8081).

---

## 7. Screenshots
Expand Down
Loading