fix(ios): say when described coordinates are not the ones taps use - #694
Open
filip131311 wants to merge 1 commit into
Open
fix(ios): say when described coordinates are not the ones taps use#694filip131311 wants to merge 1 commit into
filip131311 wants to merge 1 commit into
Conversation
On a rotated iOS simulator argent reports coordinates in two different spaces depending on which path served the tree, and says nothing about it. Measured on a landscape iPad Pro 13-inch, the same button reads (0.238, 0.342) from the ax-service tree and (0.342, 0.762) from native-devtools — related by ax.x = 1 - native.y, ax.y = native.x, a 90 degree rotation. Touch input is in the unrotated space, so the ax-service numbers hit and the native-devtools ones miss. `native-describe-screen` documents its normalized fields as tap coordinates, and on a rotated device they are not. The same is true of `describe` when the accessibility tree comes back empty and the native fallback serves the tree instead. Nothing is transformed. The only orientation signal available on this path is the screen's aspect, which gives the axis but not the sense — it cannot separate LandscapeLeft from LandscapeRight, and those are 180 degrees apart. Rotating on a guess would replace a visible mismatch with taps that land somewhere plausible but wrong, which is much harder to notice. Until the orientation itself is on the wire, saying so is the honest fix. `rotate` now carries the same warning on iOS, and deliberately does not stop at "pass rotation to screenshot": that yields a readable image whose coordinates silently disagree with `describe` and with touch, which is the trap the issue reports rather than a workaround for it. Refs #609
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The iOS half of #609. Companion to #693, which fixes the Android half; the two are independent.
What was wrong
On a rotated iOS simulator argent reports coordinates in two different spaces depending on which path served the tree, and says nothing about it.
Measured on a landscape iPad Pro 13-inch, the same button:
describevia ax-service(0.238, 0.342)describevia native-devtools fallback(0.342, 0.762)native-describe-screen(0.342, 0.762)The two are related by
ax.x = 1 - native.y,ax.y = native.x— a 90° rotation.native-describe-screenreportsscreenFrame: {width: 1376, height: 1032}— landscape — because it measures against the app's own screen, which follows rotation. Touch input does not.So
native-describe-screendocumentsnormalizedFrame/normalizedTapPointas tap coordinates, and on a rotated device they are not.describehas the same problem whenever the accessibility tree comes back empty and the native fallback serves the tree instead.Why this warns instead of transforming
The only orientation signal on this path is the screen's aspect, which gives the axis but not the sense. It cannot separate LandscapeLeft from LandscapeRight, and those differ by 180°.
Rotating on a guess would replace a visible mismatch (coordinates that obviously miss) with taps that land somewhere plausible but wrong — much harder to notice, and a worse failure. Until the orientation itself is on the wire, saying so is the honest fix. The numbers are left exactly as they were, and there is a test asserting that.
rotategets the same warning on iOS, and deliberately does not stop at "passrotationtoscreenshot". That advice produces a readable image whose coordinates silently disagree withdescribeand with touch — which is the trap the issue reports, not a workaround for it. Theargent-device-interactskill said exactly that and is corrected here.Verified live on a rotated iPad Pro 13-inch
Full tool-server suite 3096 passing; lint, prettier,
typecheck:testsand the tool-description gate all clean.Not in scope
Making iOS coordinates consistent. That needs the real orientation — including its sense — from the native side (
windowScene.interfaceOrientation), which means an argent-private change plus a prebuilt-binary release. Worth doing, but it is a different PR, and the advisories here are useful whether or not it happens.