fix(tv): give Apple TV a working wait primitive - #681
Open
filip131311 wants to merge 1 commit into
Open
Conversation
Both wait tools fetched their tree through `describeIos`, which short-circuits
every tvOS read to an empty tree — the iOS accessibility service cannot drive an
Apple TV. `await-screen-idle` resets whenever the tree is empty, so on tvOS that
condition was permanently true: it was structurally incapable of ever returning
settled, and polled to the timeout to discover it. It also read only the tree and
never the hint, so it said nothing at all about why. No selector could match
either. The TV surface had no wait primitive, leaving `run-sequence` there
timing-tuned and never robust.
`describe` works on the same screens because it routes TV targets to the focus
view before the iOS dispatch. The wait tools now poll that same source.
The adapter turns a focus read into an ordinary describe tree, so the polling,
fingerprinting and selector matching engines are untouched. Two things it must
get right:
- It uses the daemon's REAL frames. They were always in the payload — undeclared
on `TvElement` and dropped by describe's rendering — so `isVisible` and reading
order mean here what they mean everywhere else. An index-derived band is only a
fallback for backends that genuinely report no bounds.
- An empty read carries a hint. That is load-bearing, not decoration:
`await-ui-element` treats an empty tree as untrustworthy only when something
says so, and without it `condition: "hidden"` would report success on the first
poll of a still-launching app — releasing an interaction that was deliberately
gated. Strictly worse than the bug being fixed.
The cursor is marked with a synthetic `focused` trait rather than only the
`focused` field, which buys two things a field cannot: `{role:"focused"}` becomes
selectable, so "wait until focus lands on X" is expressible — the wait a TV
sequence actually needs between `tv-remote` and `select` — and the cursor enters
the idle fingerprint, so a screen whose focus is still moving does not read as
settled.
Android TV deliberately keeps its full uiautomator tree. An empty focus set is
steady state there, not a transition, for react-native-tvos screens whose focus
engine the OS accessibility tree cannot see; routing it onto the focus source for
consistency would import this very bug onto a platform that works. There is a
test guarding that.
The wait path never repairs the read path: `describeTv` retries and can respawn
the tvOS ax daemon, which inside a poll loop is redundant at best and destructive
at worst — it would drop the state being waited on. It takes one bare read per
poll and points at `describe` for the repair.
`await-screen-idle` also gains a `note`, which is not tvOS-specific. Any unsettled
wait on an empty read now explains itself, so a device whose screen is simply off
no longer produces a silent full-budget stall on any platform.
Beyond the reported symptom, this removes a real tax: the MCP layer waits on this
tool before every auto-screenshot and treats a failure by sleeping the whole
budget, so on tvOS every launch-app cost 3s and every run-sequence 15s. It now
returns as soon as the screen holds.
The skill's claim that describe returns "normalized frames" is corrected — the
frames exist, but the focus view does not print them — along with the navigation
step that told the agent to count rows from them.
Verified on an Apple TV 4K simulator (tvOS 26.5, TVSettings): idle settles in
~780ms where it previously never settled, cursor and value waits resolve in
single-digit ms where every condition previously failed at full timeout.
Fixes #620
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #620.
Reproduced
Apple TV 4K simulator, tvOS 26.5,
com.apple.TVSettings, on a provably static screen:Root cause — one line
Both wait tools fetch through
describeIos, which short-circuits every tvOS read (describe/platforms/ios/index.ts:93-96) toemptyTree()+ a hint.await-screen-idleresets whenever the tree is empty (await-screen-idle/index.ts:164-169), so on tvOS that condition is permanently true — the tool is structurally incapable of returningsettled: truethere, and polls to the timeout to find out. It also reads onlydata.treeand neverdata.hint, which is why the good explanatory note its sibling emits was silently dropped.describeworks on the same screens because it routes TV targets to the focus view before the iOS dispatch (describe/index.ts:107,126) — a branch the wait tools never got. They now poll that same source.Design decisions worth reviewing
Real frames, not synthesized ones. My first read of the payload (and the issue's doc nit) said
TvElementhas no frame. That's true of the TypeScript interface and false of the data — I probed the daemon socket directly:{"label":"Appearance","frame":{"y":0.249,"x":0.552,"width":0.406,"height":0.061}, "value":"Dark","traits":["button","_focusGuide"],"tapPoint":{"x":0.755,"y":0.280}}They survive into the tool because
tv-control.ts:459-465passes the parsed objects through by reference; only the type omitted them, and describe's rendering drops them. So the adapter uses them, andisVisible/ reading order mean here what they mean everywhere else. An index-derived band remains as a fallback for backends that genuinely report no bounds (Android TV) and for zero-size elements.The hint on an empty read is load-bearing, not decoration.
await-ui-elementtreats an empty tree as an untrustworthy read only when a hint (or a prior match) says so. An adapter that returned an empty focus view without one would makecondition: "hidden"report success on the first poll of a still-launching app — releasing an interaction that was deliberately gated. That is strictly worse than the bug being fixed, so it has two tests.The cursor is marked with a synthetic
focusedtrait, not only thefocusedfield. That buys two things the field cannot:{role:"focused"}becomes selectable, so "wait until focus lands on X" is expressible — the wait a TVrun-sequenceactually needs betweentv-remoteandselect— and the cursor enters the idle fingerprint, so a screen whose focus is still moving doesn't read as settled. Safe as aroletoken: role matching is a case-insensitive substring and no real trait on either backend contains "focused" (_focusGuide,_tvFocusabledo not).Android TV deliberately keeps its full uiautomator tree. An empty focus set is steady state there — react-native-tvos screens drive focus with RN's own engine, invisible to the OS accessibility tree — whereas on Apple TV it's a transition. Routing Android TV onto the focus source "for consistency" would import this very bug onto a platform that works. There's an explicit test guarding it.
The wait path never repairs the read path.
describeTvretries with sleeps and can respawn the tvOS ax daemon (recycleAx). Inside a 200 ms poll that is redundant at best and destructive at worst — respawning drops the very state being waited on. The wait takes one bare read per poll and points atdescribefor the repair; a test assertsrecycleAxis never called and thatapi.describecall count equalspolls.A tax this removes that the issue doesn't mention
The MCP layer waits on
await-screen-idlebefore every auto-screenshot and handles failure by sleeping the full budget (mcp-server.ts:342-344). Sincesettledcould never become true on tvOS, everylaunch-app/restart-appcost 3000 ms and everyrun-sequence15000 ms before the screenshot. It now returns as soon as the screen holds.That also rules out the issue's suggested "fail fast": returning immediately would make the auto-screenshot capture splash screens, and returning by throwing is the slowest possible outcome there.
noteis not tvOS-specificawait-screen-idlegained anotefor any unsettled wait on an empty read. I found this matters beyond TV: on a device whose screen is simply off,describereturns an empty root with no hint and the tool stalls the whole budget saying onlysettled: false. Now it says the screen reported no content. (Neither tool detects "screen is off" as such — worth its own issue, sincemWakefulnessmakes it a one-line check.)Verified live
Apple TV 4K / tvOS 26.5, branch tool-server:
Previously: never settled, and every condition failed at full timeout.
Tests
tv-focus-tree.test.ts(11) covers the adapter: real frames preserved, ordered non-degenerate fallback, bundle id on the root but not matchable, labels/values/traits matchable, cursor marked once and not duplicated, orphan cursor still surfaced, and the empty-read hint.await-tv-focus.test.ts(12) covers tool behaviour: settles on a static view, doesn't settle while the cursor moves or the set grows, explains an empty view, never callsrecycleAx, all four conditions on tvOS,hiddenrefusing to resolve on an empty view, and the Android-TV routing guard.Mutation-verified: removing the tvOS routing fails 11 of 12 (the 12th is the Android-TV guard, correctly unaffected).
Full tool-server suite green — 3109 passed / 299 files.
Docs
The skill's "focusable elements with labels and normalized frames" is corrected: the frames exist in the source but the focus view doesn't print them, and a TV is navigated by D-pad rather than coordinate. The navigation step that told the agent to "count rows/columns from the frames" now points at the order of the focusable list. Only
packages/skills/skills/…is edited — the other copies are generated.Not in scope
utils/ui-tree-match.tsfetchTreeroutes tvOS intodescribeIoswith noisTvOsoption, so flow directives and the recorder still see an empty tree.describeTvFocusdrops in there verbatim; I left it out to keep this reviewable and will file it separately.isTvOsSimulatoris simulator-list based). Pre-existing.