Skip to content

feat(tools): add screen-fingerprint, a stable answer to "which screen is this" #727

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
hubgan wants to merge 1 commit into from
Closed

feat(tools): add screen-fingerprint, a stable answer to "which screen is this" #727

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions packages/skills/rules/argent.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ Decision order:
If the user started Metro separately, ask whether to call `stop-metro` (specify the port if not 8081).
- If tools provided by mcp-server are not sufficient and action can be done using `xcrun`, `adb`, or other commands, use the command. Examples: changing device options, performing a device action such as lock, shake, etc.
- When waiting for an action, do not call `screenshot` repeatedly without a proper wait mechanism. Use the `await-ui-element` tool to block until the UI settles (e.g. wait for an element to become `visible`/`hidden`, or to contain expected `text`) instead of polling.
- To confirm which screen a React Native app is on, use `screen-fingerprint` — it reads the app's focused navigation route. It answers only "which screen": it does not see a modal or system alert on top, and the route commits before the transition finishes animating.
</general_rules>

<react_native_detection>
Expand Down
56 changes: 36 additions & 20 deletions packages/skills/skills/argent-device-interact/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,25 +56,27 @@ Common schemes: `messages://`, `settings://`, `maps://?q=<query>`, `tel://<numbe

## 4. Choosing the Right Tool

| Action | Tool | Notes |
| ----------------- | ------------------ | ---------------------------------------------------------------- |
| Multiple actions | `run-sequence` | Batch steps in one call (no intermediate screenshots) |
| Open an app | `launch-app` | **Always — never tap home-screen icons** |
| Restart an app | `restart-app` | Terminate and relaunch by bundle ID |
| Open URL/scheme | `open-url` | Web pages, deep links, URL schemes |
| Single tap | `gesture-tap` | Buttons, links, checkboxes |
| Scroll/swipe | `gesture-swipe` | Straight-line scroll or swipe |
| Scroll (Chromium) | `gesture-scroll` | Wheel-based; deltas are window fractions, positive deltaY = down |
| Drag (Chromium) | `gesture-drag` | Sliders, drag-and-drop, text selection |
| Long press | `gesture-custom` | Context menus, drag start |
| Drag & drop | `gesture-custom` | Complex drag interactions |
| Pinch/zoom | `gesture-pinch` | Two-finger pinch with auto-interpolation |
| Rotation | `gesture-rotate` | Two-finger rotation with auto-interpolation |
| Custom gesture | `gesture-custom` | Arbitrary touch sequences, optional interpolation |
| Hardware key | `button` | Home, back, power, volume, appSwitch, actionButton |
| Type text | `keyboard` | iOS+Android. Supports Enter, Escape, arrows |
| Rotate device | `rotate` | Orientation changes |
| Wait for UI | `await-ui-element` | Block until an element is visible/hidden/exists/contains text |
| Action | Tool | Notes |
| ----------------- | -------------------- | ---------------------------------------------------------------- |
| Multiple actions | `run-sequence` | Batch steps in one call (no intermediate screenshots) |
| Open an app | `launch-app` | **Always — never tap home-screen icons** |
| Restart an app | `restart-app` | Terminate and relaunch by bundle ID |
| Open URL/scheme | `open-url` | Web pages, deep links, URL schemes |
| Single tap | `gesture-tap` | Buttons, links, checkboxes |
| Scroll/swipe | `gesture-swipe` | Straight-line scroll or swipe |
| Scroll (Chromium) | `gesture-scroll` | Wheel-based; deltas are window fractions, positive deltaY = down |
| Drag (Chromium) | `gesture-drag` | Sliders, drag-and-drop, text selection |
| Long press | `gesture-custom` | Context menus, drag start |
| Drag & drop | `gesture-custom` | Complex drag interactions |
| Pinch/zoom | `gesture-pinch` | Two-finger pinch with auto-interpolation |
| Rotation | `gesture-rotate` | Two-finger rotation with auto-interpolation |
| Custom gesture | `gesture-custom` | Arbitrary touch sequences, optional interpolation |
| Hardware key | `button` | Home, back, power, volume, appSwitch, actionButton |
| Type text | `keyboard` | Every platform. Supports Enter, Escape, arrows (not on TV) |
| Rotate device | `rotate` | Orientation changes |
| Wait for UI | `await-ui-element` | Block until an element is visible/hidden/exists/contains text |
| Wait for idle | `await-screen-idle` | Block until a non-empty screen tree stops changing |
| Identify a screen | `screen-fingerprint` | Read which screen a React Native app is on, as its route path |

## 5. Finding Tap Targets

Expand Down Expand Up @@ -203,11 +205,25 @@ Instead of polling `screenshot`/`describe` in a loop, use `await-ui-element` to
- `selector`: `{ text?, identifier?, role? }` — every provided field must match. `text` matches the element's label or value and `role` its element role (e.g. `AXButton`, `button`, `TextView`, `StaticText`), both as case-insensitive substrings; `identifier` matches its accessibility id / resource-id / testID **exactly** (case-insensitive), also accepting the unqualified Android resource-id name (`submit` matches `com.example.app:id/submit`). The synthetic `ROOT` container `describe` prints is never matched, so a `role` like `AXGroup`/`html` won't trivially "match the screen".
- Prefer a **specific** selector. A loose substring can match several elements, and the tool may then key off one you didn't mean: `text` reads the first **visible** match in **reading order** (top-to-bottom, left-to-right — the same order `describe` lists them, so it's the one you saw first; when no match is visible, the first match overall), while `visible`/`exists` are satisfied by **any** match. Disambiguate with a longer or more exact string, an `identifier`, or a `role` (e.g. pin to a text role like `StaticText` to skip a same-named button). On a `text` timeout the `note` quotes the matched element's text, so you can see which one it landed on.
- `text` condition also needs `expectedText` (substring the matched element must contain).
- `hidden` treats a selector that matches **nothing** as already-hidden, so a typo'd selector returns an instant (false) success. Double-check the selector for `hidden` waits — the result `note` flags when the selector never matched any element. (On iOS, if the accessibility backend is down the tree comes back empty; the tool will **not** report `hidden` success off such a degraded read and the `note` surfaces the boot hint instead.)
- `hidden` treats a selector that matches **nothing** as already-hidden, so a typo'd selector returns an instant (false) success. The result `note` flags when the selector never matched any element; treat that note as a failed check, not a pass, and find the real selector before continuing. `flow-add-step` refuses to record such a wait, because a gate that cannot fail proves nothing on replay. (On iOS, if the accessibility backend is down the tree comes back empty; the tool will **not** report `hidden` success off such a degraded read and the `note` surfaces the boot hint instead.)
- Optional `timeoutMs` (default 5000) and `pollIntervalMs` (default 400).

Returns `{ success, elapsed }`; on a timeout `success` is `false` and a `note` explains what was seen.

### screen-fingerprint — Which screen is the app on

For a React Native app served by Metro, this reads the focused React Navigation route path and returns it as `"HomeTab>Profile"`:

```json
{ "app_id": "com.acme.notes" }
```

It answers "which screen" and nothing else — pair it with `await-screen-idle` for readiness, and an element check for an overlay above the screen.

- `available: false` — this app has no reader (release build, fully native app, Chromium, Metro down). Recognize the screen by a destination-only element instead.
- `route: null` with `available: true` — no focused route at this instant: a native screen, or a transition still in flight. Let it settle and probe again.
- Optional `platform`, `device`, and `metro_port` (default 8081).

---

## 7. Screenshots
Expand Down
183 changes: 183 additions & 0 deletions packages/tool-server/src/tools/screen-fingerprint/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,183 @@
import { z } from "zod";
import type { Registry, ToolContext, ToolDefinition } from "@argent/registry";
import { resolveFlowDevice } from "../flows/flow-device";
import { connectRouteReader, probeRoute, routeFingerprint } from "../../utils/route-identity";
import { metroServerRunning } from "../../utils/debugger/discovery";

/**
* Whether a Metro dev server is answering on `port` at all. Never throws.
*
* Asks only whether the SERVER is up — see {@link metroServerRunning}. Target
* discovery cannot answer it: a Metro serving one app reports an empty target
* list for the several seconds after that app relaunches, which is exactly
* when this tool is called, and reading that as "Metro is down" produced the
* one message the branch below exists to avoid.
*/
async function metroReachable(port: number): Promise<boolean> {
return metroServerRunning(port);
}

/**
* Read the current screen's route fingerprint from the running app — the
* recorder's source for a flow's `await: { screen: … }` gate. A thin,
* device-only probe: nothing is tapped and no file is touched.
*/

export const SCREEN_FINGERPRINT_TOOL_ID = "screen-fingerprint";

// chromium is deliberately absent: an Electron app has no React Navigation
// state to read, so there is nothing this tool could return for it.
const FINGERPRINT_PLATFORMS = ["ios", "android", "vega"] as const;

const zodSchema = z.object({
app_id: z
.string()
.min(1)
.describe(
"Bundle id / package name of the app under test — guards against reading a foreign " +
"app's Metro on the same port."
),
platform: z
.enum(FINGERPRINT_PLATFORMS)
.optional()
.describe(
"Platform of the device under test. Only needed to disambiguate when several platforms " +
"have a booted device."
),
device: z
.string()
.optional()
.describe(
"Device id to probe (iOS UDID, Android/Vega serial). Auto-detected from the single " +
"booted device when omitted."
),
metro_port: z
.number()
.int()
.min(1)
.max(65535)
.optional()
.describe("Metro dev-server port the app was launched from (default 8081)."),
});

type Params = z.infer<typeof zodSchema>;

export interface ScreenFingerprintResult {
/** false = no reader for this app (not a Metro-served debuggable RN app). */
available: boolean;
/** The fingerprint to gate on, or null when none could be read. */
route: string | null;
/** Focused route names outermost→innermost (what `route` joins). */
path?: string[];
/**
* The leaf route's params — evidence the screen is parameterized (one route
* serves every instance). Data about the screen, never its identity.
*/
params?: Record<string, unknown> | null;
reason?: string;
hint?: string;
}

export function createScreenFingerprintTool(
registry: Registry
): ToolDefinition<Params, ScreenFingerprintResult> {
return {
id: SCREEN_FINGERPRINT_TOOL_ID,
interaction: {
startedMsg: ({ params }) => `Reading screen identity of ${params.app_id}`,
// A read that returns no route still succeeds, so the message has to say
// which of the two happened — "read the screen" would imply a fingerprint
// that the caller may not have got.
completedMsg: ({ params, result }) =>
result.route === null
? `Read no focused route for ${params.app_id}`
: `Read screen ${result.route}`,
failedMsg: ({ params, failureSignal }) =>
`Failed to read screen identity of ${params.app_id}: ${failureSignal.error_code}`,
},
description: `Read which screen the app is on, as its focused React Navigation route path ("HomeTab>Profile"), via the RN debugger over Metro.
This is screen IDENTITY, and it is stronger than any element check: it comes from the app's own navigation state, so it does not depend on which tree source rendered \`describe\`, on content, on count, or on locale, and every instance of a parameterized screen (one profile vs another) shares one route. Gate a flow on it with \`await: { screen: "HomeTab>Profile" }\`, or record that step by passing this command to \`flow-add-step\`.
It answers "which screen" and nothing else. It does NOT prove the screen finished animating (navigation state commits before the transition ends) and it does NOT see a native overlay above the screen (a permission alert, a share sheet, an RN <Modal> leave the route unchanged) — pair it with \`await: { idle: true }\` for readiness and an element check for the overlay.
Only Metro-served debuggable RN apps have routes: \`available: false\` means this app has no reader (release build, fully native, chromium) — gate on elements instead. \`route: null\` with \`available: true\` means no focused route right now (a native screen, or a transition mid-flight) — let the screen settle and probe again.`,
searchHint:
"screen identity fingerprint which screen route react navigation current screen prove navigation",
zodSchema,
services: () => ({}),
async execute(_services, params, ctx?: ToolContext) {
const device = await resolveFlowDevice(registry, ctx, {
...(params.device !== undefined ? { device: params.device } : {}),
...(params.platform !== undefined ? { platform: params.platform } : {}),
});
const metroPort = params.metro_port ?? 8081;
if (device.platform === "chromium") {
return {
available: false,
route: null,
reason:
"chromium apps have no React Navigation route identity — gate on a destination-only " +
"element instead.",
};
}
// "ios-remote" is an iOS simulator over a remote bridge — same runtime.
const platform = device.platform === "ios-remote" ? "ios" : device.platform;
const reader = await connectRouteReader(registry, ctx, {
udid: device.id,
bundleId: params.app_id,
metroPort,
platform,
});
if (reader === undefined) {
// Which cause holds is checkable, so check it. Asserting "Metro is
// down" while Metro is demonstrably up — the routine case, because an
// app re-registers a few seconds AFTER it relaunches and the recorder
// probes immediately — sent authors to repair a working dev server,
// and the conclusion the old wording drew for them ("screens of this
// app can only be recognized by element checks") deleted the identity
// proof for the whole flow on the strength of a transient miss.
const metroUp = await metroReachable(metroPort);
return {
available: false,
route: null,
reason: metroUp
? `Metro is running on port ${metroPort}, but no debuggable target for ` +
`${params.app_id} is registered there. If the app was just launched or restarted, ` +
`it re-registers a few seconds later — wait for the screen to settle and probe ` +
`again. If it stays this way, the app is not a debuggable RN build, or this port ` +
`serves a different app.`
: `No Metro dev server is answering on port ${metroPort}, so ${params.app_id} has ` +
`no route reader. Start Metro (or pass the right \`metro_port\`); if this app is ` +
`not a debuggable RN build at all, its screens can only be recognized by element ` +
`checks.`,
};
}
const route = await probeRoute(reader, {
...(ctx?.signal !== undefined ? { signal: ctx.signal } : {}),
});
if (route === null) {
return {
available: true,
route: null,
reason:
"The app exposes no focused React Navigation route right now — a fully native " +
"screen, or a transition mid-flight. Let the screen settle and probe again; if it " +
"stays null, this screen has no route identity (gate on an element instead).",
};
}
const fingerprint = routeFingerprint(route);
return {
available: true,
route: fingerprint,
path: route.path,
params: route.params,
hint:
`Gate on it with: - await: { screen: "${fingerprint}" }, PAIRED with a readiness ` +
`check — navigation state commits before the screen renders, so this route is already ` +
`reported while the app is still blank. If the screen you just navigated FROM reports ` +
`this same route, the two are one route and gating on it would prove nothing: use a ` +
`destination-only element instead. ` +
`Non-null params mean the screen is parameterized — the route still identifies it, ` +
`but do not also gate on the specific instance's content.`,
};
},
};
}
24 changes: 24 additions & 0 deletions packages/tool-server/src/utils/debugger/discovery.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ export interface CDPTarget {
description: string;
webSocketDebuggerUrl: string;
deviceName?: string;
/** Bundle id / package name of the app that registered the target (modern RN). */
appId?: string;
/** Legacy inspector-proxy only. Its synthetic reload page reports "don't use". */
vm?: string;
reactNative?: {
Expand Down Expand Up @@ -34,6 +36,28 @@ export interface MetroInfo {
*/
const DECOY_VM = "don't use";

/**
* Is a Metro dev server LISTENING on `port` — regardless of whether any app is
* currently attached to it? Probes `/status` only. Never throws.
*
* Deliberately not `discoverMetro`, which also requires at least one CDP
* target and throws `DEBUGGER_METRO_NO_TARGETS` otherwise. That distinction is
* the whole point here: a Metro serving ONE app has an empty target list for
* several seconds after that app is relaunched, so "no targets" is the normal
* post-launch state, not a down server. Callers that used discovery for this
* question therefore concluded "Metro is down" exactly when an app was coming
* back up — telling the author to start a server that was already running, and
* skipping the extended connect budget that window exists to cover.
*/
export async function metroServerRunning(port: number): Promise<boolean> {
try {
const res = await fetch(`http://localhost:${port}/status`);
return (await res.text()).includes("packager-status:running");
} catch {
return false;
}
}

export async function discoverMetro(port: number): Promise<MetroInfo> {
let statusRes: Response;
try {
Expand Down
Loading