Skip to content

fix(tv): give Apple TV a working wait primitive - #681

Open
filip131311 wants to merge 1 commit into
mainfrom
filip/tv-wait-primitives
Open

fix(tv): give Apple TV a working wait primitive#681
filip131311 wants to merge 1 commit into
mainfrom
filip/tv-wait-primitives

Conversation

@filip131311

Copy link
Copy Markdown
Collaborator

Fixes #620.

Reproduced

Apple TV 4K simulator, tvOS 26.5, com.apple.TVSettings, on a provably static screen:

await-screen-idle  → { settled: false, waitedMs: 6002, polls: 31 }      ← and no note at all
await-ui-element   → { success: false, elapsed: 4001, note: "…tvOS is not supported…" }
describe           → 12 focusables, source "tv-focus"                    ← works fine

Root cause — one line

Both wait tools fetch through describeIos, which short-circuits every tvOS read (describe/platforms/ios/index.ts:93-96) to emptyTree() + a hint. await-screen-idle resets whenever the tree is empty (await-screen-idle/index.ts:164-169), so on tvOS that condition is permanently true — the tool is structurally incapable of returning settled: true there, and polls to the timeout to find out. It also reads only data.tree and never data.hint, which is why the good explanatory note its sibling emits was silently dropped.

describe works on the same screens because it routes TV targets to the focus view before the iOS dispatch (describe/index.ts:107,126) — a branch the wait tools never got. They now poll that same source.

Design decisions worth reviewing

Real frames, not synthesized ones. My first read of the payload (and the issue's doc nit) said TvElement has no frame. That's true of the TypeScript interface and false of the data — I probed the daemon socket directly:

{"label":"Appearance","frame":{"y":0.249,"x":0.552,"width":0.406,"height":0.061},
 "value":"Dark","traits":["button","_focusGuide"],"tapPoint":{"x":0.755,"y":0.280}}

They survive into the tool because tv-control.ts:459-465 passes the parsed objects through by reference; only the type omitted them, and describe's rendering drops them. So the adapter uses them, and isVisible / reading order mean here what they mean everywhere else. An index-derived band remains as a fallback for backends that genuinely report no bounds (Android TV) and for zero-size elements.

The hint on an empty read is load-bearing, not decoration. await-ui-element treats an empty tree as an untrustworthy read only when a hint (or a prior match) says so. An adapter that returned an empty focus view without one would make condition: "hidden" report success on the first poll of a still-launching app — releasing an interaction that was deliberately gated. That is strictly worse than the bug being fixed, so it has two tests.

The cursor is marked with a synthetic focused trait, not only the focused field. That buys two things the field cannot: {role:"focused"} becomes selectable, so "wait until focus lands on X" is expressible — the wait a TV run-sequence actually needs between tv-remote and select — and the cursor enters the idle fingerprint, so a screen whose focus is still moving doesn't read as settled. Safe as a role token: role matching is a case-insensitive substring and no real trait on either backend contains "focused" (_focusGuide, _tvFocusable do not).

Android TV deliberately keeps its full uiautomator tree. An empty focus set is steady state there — react-native-tvos screens drive focus with RN's own engine, invisible to the OS accessibility tree — whereas on Apple TV it's a transition. Routing Android TV onto the focus source "for consistency" would import this very bug onto a platform that works. There's an explicit test guarding it.

The wait path never repairs the read path. describeTv retries with sleeps and can respawn the tvOS ax daemon (recycleAx). Inside a 200 ms poll that is redundant at best and destructive at worst — respawning drops the very state being waited on. The wait takes one bare read per poll and points at describe for the repair; a test asserts recycleAx is never called and that api.describe call count equals polls.

A tax this removes that the issue doesn't mention

The MCP layer waits on await-screen-idle before every auto-screenshot and handles failure by sleeping the full budget (mcp-server.ts:342-344). Since settled could never become true on tvOS, every launch-app/restart-app cost 3000 ms and every run-sequence 15000 ms before the screenshot. It now returns as soon as the screen holds.

That also rules out the issue's suggested "fail fast": returning immediately would make the auto-screenshot capture splash screens, and returning by throwing is the slowest possible outcome there.

note is not tvOS-specific

await-screen-idle gained a note for any unsettled wait on an empty read. I found this matters beyond TV: on a device whose screen is simply off, describe returns an empty root with no hint and the tool stalls the whole budget saying only settled: false. Now it says the screen reported no content. (Neither tool detects "screen is off" as such — worth its own issue, since mWakefulness makes it a one-line check.)

Verified live

Apple TV 4K / tvOS 26.5, branch tool-server:

await-screen-idle                                   → { settled: true, waitedMs: 765, polls: 3 }
await-ui-element {text:"About", role:"focused"}     → { success: true, elapsed: 5 }
await-ui-element text "Region" == "Poland"          → { success: true, elapsed: 4 }

Previously: never settled, and every condition failed at full timeout.

Tests

tv-focus-tree.test.ts (11) covers the adapter: real frames preserved, ordered non-degenerate fallback, bundle id on the root but not matchable, labels/values/traits matchable, cursor marked once and not duplicated, orphan cursor still surfaced, and the empty-read hint. await-tv-focus.test.ts (12) covers tool behaviour: settles on a static view, doesn't settle while the cursor moves or the set grows, explains an empty view, never calls recycleAx, all four conditions on tvOS, hidden refusing to resolve on an empty view, and the Android-TV routing guard.

Mutation-verified: removing the tvOS routing fails 11 of 12 (the 12th is the Android-TV guard, correctly unaffected).

Full tool-server suite green — 3109 passed / 299 files.

Docs

The skill's "focusable elements with labels and normalized frames" is corrected: the frames exist in the source but the focus view doesn't print them, and a TV is navigated by D-pad rather than coordinate. The navigation step that told the agent to "count rows/columns from the frames" now points at the order of the focusable list. Only packages/skills/skills/… is edited — the other copies are generated.

Not in scope

  • Flows on tvOS are still broken. utils/ui-tree-match.ts fetchTree routes tvOS into describeIos with no isTvOs option, so flow directives and the recorder still see an empty tree. describeTvFocus drops in there verbatim; I left it out to keep this reviewable and will file it separately.
  • Neither wait tool is picked up by the tool-description extractor (both use const-reference ids), so their descriptions aren't scored by the quality gate. Real gap, worth its own issue; deliberately not "fixed" here since enrolling two long descriptions into the scored average is a separate risk.
  • A physical Apple TV still falls into the ordinary iOS path (isTvOsSimulator is simulator-list based). Pre-existing.

Both wait tools fetched their tree through `describeIos`, which short-circuits
every tvOS read to an empty tree — the iOS accessibility service cannot drive an
Apple TV. `await-screen-idle` resets whenever the tree is empty, so on tvOS that
condition was permanently true: it was structurally incapable of ever returning
settled, and polled to the timeout to discover it. It also read only the tree and
never the hint, so it said nothing at all about why. No selector could match
either. The TV surface had no wait primitive, leaving `run-sequence` there
timing-tuned and never robust.

`describe` works on the same screens because it routes TV targets to the focus
view before the iOS dispatch. The wait tools now poll that same source.

The adapter turns a focus read into an ordinary describe tree, so the polling,
fingerprinting and selector matching engines are untouched. Two things it must
get right:

- It uses the daemon's REAL frames. They were always in the payload — undeclared
  on `TvElement` and dropped by describe's rendering — so `isVisible` and reading
  order mean here what they mean everywhere else. An index-derived band is only a
  fallback for backends that genuinely report no bounds.
- An empty read carries a hint. That is load-bearing, not decoration:
  `await-ui-element` treats an empty tree as untrustworthy only when something
  says so, and without it `condition: "hidden"` would report success on the first
  poll of a still-launching app — releasing an interaction that was deliberately
  gated. Strictly worse than the bug being fixed.

The cursor is marked with a synthetic `focused` trait rather than only the
`focused` field, which buys two things a field cannot: `{role:"focused"}` becomes
selectable, so "wait until focus lands on X" is expressible — the wait a TV
sequence actually needs between `tv-remote` and `select` — and the cursor enters
the idle fingerprint, so a screen whose focus is still moving does not read as
settled.

Android TV deliberately keeps its full uiautomator tree. An empty focus set is
steady state there, not a transition, for react-native-tvos screens whose focus
engine the OS accessibility tree cannot see; routing it onto the focus source for
consistency would import this very bug onto a platform that works. There is a
test guarding that.

The wait path never repairs the read path: `describeTv` retries and can respawn
the tvOS ax daemon, which inside a poll loop is redundant at best and destructive
at worst — it would drop the state being waited on. It takes one bare read per
poll and points at `describe` for the repair.

`await-screen-idle` also gains a `note`, which is not tvOS-specific. Any unsettled
wait on an empty read now explains itself, so a device whose screen is simply off
no longer produces a silent full-budget stall on any platform.

Beyond the reported symptom, this removes a real tax: the MCP layer waits on this
tool before every auto-screenshot and treats a failure by sleeping the whole
budget, so on tvOS every launch-app cost 3s and every run-sequence 15s. It now
returns as soon as the screen holds.

The skill's claim that describe returns "normalized frames" is corrected — the
frames exist, but the focus view does not print them — along with the navigation
step that told the agent to count rows from them.

Verified on an Apple TV 4K simulator (tvOS 26.5, TVSettings): idle settles in
~780ms where it previously never settled, cursor and value waits resolve in
single-digit ms where every condition previously failed at full timeout.

Fixes #620
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tvOS has no working wait primitive: await-screen-idle never settles (silently), await-ui-element is unsupported

1 participant