fix(flow): refuse a recorded tap that would replay somewhere else - #722
Draft
hubgan wants to merge 5 commits into
Draft
fix(flow): refuse a recorded tap that would replay somewhere else#722hubgan wants to merge 5 commits into
hubgan wants to merge 5 commits into
Conversation
hubgan
force-pushed
the
fix/recorder-tap-replay-fidelity
branch
from
August 4, 2026 07:23
fd7f1b1 to
2d8157d
Compare
hubgan
force-pushed
the
fix/recorder-tap-replay-fidelity
branch
from
August 5, 2026 10:15
6205ea7 to
c67e984
Compare
hubgan
force-pushed
the
fix/recorder-tap-replay-fidelity
branch
from
August 5, 2026 10:38
c67e984 to
2236770
Compare
hubgan
force-pushed
the
fix/recorder-tap-replay-fidelity
branch
from
August 6, 2026 09:56
2236770 to
0d47eb4
Compare
hubgan
force-pushed
the
fix/recorder-tap-replay-fidelity
branch
from
August 9, 2026 20:47
0d47eb4 to
4d288ce
Compare
Replay taps a selector's CENTRE, not the point that was recorded. The recorder only checked that the resolved frame contained the point, which lets two silent misfires through. A tap on blank space inside a drawer resolved to the drawer's whole scroll area (0.72 of the screen). It contains the point, so it recorded — and replay tapped the centre, hit the "Chat" item, and reported PASS while navigating somewhere the walkthrough never went. A frame that covers most of the screen is a container, not a control, and is now refused with that reason. The size test is deliberately not "does the centre resolve back to the same node". That was tried and is wrong on a FLATTENED tree: a control's own label is a sibling rect sitting on its centre, so a like button, a search field, a full-width row and every grid cell were refused while replaying perfectly. When the derived selector matches the tapped element AND something else and ranks the other first, the recorder now narrows it — the node's own role, then its identifier — before giving up. Ambiguity is not "this element cannot be addressed", and answering it with coordinates threw away a perfectly good target. A `within` scope is deliberately NOT derived: the flow tree is flattened and carries no z-order, so the geometric ancestor of a tap inside a modal can be a background element, and one recorded that way failed on any screen whose content differed. `deriveSelector` also stops deriving POSITIONAL ids (`…-selector-2`). The number is the element's index among its siblings, so the id names a slot rather than a thing. It matters most where the recorder is least reliable: a tap on an edit-profile sheet derived the profile pager's "Media" tab from a view BEHIND the modal. An oversized or ambiguous background match is caught by the checks above, but a positional id passes all of them and records silently. Every kept-coordinate warning now names the failure AND the retarget, and the two are different: an element nothing can address needs a real target found in the tree; one that several things address equally needs disambiguating. Saying "no selector could be derived" for the second sent authors to re-discover a selector they already had. Raw-coordinate steps with no capture attempt at all (`delayMs` set, `gesture-custom`, taps buried in a `run-sequence`) say why the rewrite did not run and how to record it so that it does.
…fused
narrowedSelectors added the tapped node's identifier to disambiguate a
retargeting selector, but that branch is reachable only when deriveSelector
declined the id — i.e. only when the id is positional. So it could re-record
`{ text, identifier: profilePager-selector-2 }` silently, smuggling the slot
id past the very guard deriveSelector applies, in the Bluesky edit-profile
modal case the refusal exists for. Guard the branch with POSITIONAL_ID so an
ambiguous positional-only target keeps coordinates with its warning instead.
Also make the oversized-target path honest: at >60% area the tree cannot tell
a container from a full-bleed control, so the warning no longer asserts "it is
a container, not a control", and the kept-coordinate remedy no longer tells the
author an element that HAS an id has "no id or label" — it points at the
smaller control under the point. Correct the POSITIONAL_ID doc (no
argent-create-flow "audit" exists) and the argent-create-flow skill's
tap-capture description to match the new refusals.
The identifier branch in narrowedSelectors was unreachable: deriveSelector already makes any stable, non-positional id the base selector, so `base` lacks an identifier only when the id is absent or positional — the exact negation of the branch's own guard. Remove it, and with it the sole cross-module use of POSITIONAL_ID (now un-exported). Correct the two messages the dead branch made false: the ambiguity warning no longer claims it tried an identifier (only role is ever added), and the generic coordinate remedy no longer asserts "no id or label" for the no-element / capture-failed cases. Pin the container-area threshold on both sides and cover the unaddressable remedy branch, which no test exercised.
hubgan
force-pushed
the
fix/recorder-tap-replay-fidelity
branch
from
August 10, 2026 08:55
4d288ce to
7cc3469
Compare
…r-tap-replay-fidelity # Conflicts: # packages/skills/skills/argent-create-flow/SKILL.md
…r-tap-replay-fidelity
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The core mismatch
Replay taps a selector's centre, not the point that was recorded. The recorder only checked that the resolved frame contained the point — which lets two silent misfires through.
1. Container-sized targets
A tap on blank space inside a drawer resolved to the drawer's whole scroll area — 0.72 of the screen. It contains the point, so it recorded. Replay then tapped its centre, hit the "Chat" item, and reported pass while navigating somewhere the walkthrough never went.
A frame covering most of the screen is a container, not a control, and is now refused with that reason. Narrowing cannot help here: the problem is the element, not the selector.
The threshold is a judgement between two real failures. Too permissive records containers (above); too strict makes ordinary widgets unrecordable — a feed post is half the screen and tapping it is a perfectly normal QA step.
Why not "does the centre resolve back to the same node"? That test was tried and is wrong on a flattened tree: a control's own label is a sibling rect sitting on its centre, so a like button, a search field, a full-width row and every grid cell were refused — while replaying perfectly, because the touch is still inside the control. Node identity cannot tell a label from an independent control; size can tell a control from a container.
2. Ambiguity is answered by narrowing, not coordinates
When the derived selector matches the tapped element and something else and ranks the other first, the recorder now tries narrower forms — the node's own role, then its identifier — before giving up. The runner resolves either form, so answering ambiguity with coordinates was throwing away a perfectly good target.
A
withinscope is deliberately not derived, even though it would separate one feed row's button from another's. The flow tree is flattened and carries no z-order, so a container can only be found geometrically — and with a modal open, the foreground modal is a perfectly good geometric ancestor of a background element. A tap on a composer's text input recorded as a feed post "inside" the composer, then failed on any screen whose feed content differed. The scopes that survive are the ones an author writes knowingly at polish.3. Positional ids are no longer derived
profilePager-selector-2encodes the element's index among its siblings, so it names a slot rather than a thing: it survives no re-order and silently addresses a different control once one is inserted before it.It matters most where the recorder is least reliable. Observed on an edit-profile sheet: a tap on the display-name field derived the profile pager's "Media" tab — a view behind the modal. An oversized or ambiguous background match is caught by the two checks above; a positional id passes all of them and records silently.
4. Every kept-coordinate warning names the retarget
A coordinate tap replays fine today and breaks on the first layout change, so this warning is the author's only signal — and it is read at the one moment the screen is still there to retarget against.
The two failures need opposite advice: an element nothing can address needs a real target found in the tree; one that several things address equally needs disambiguating. Saying "no selector could be derived" for the second sent authors off to re-discover a selector they already had.
Raw-coordinate steps where capture never ran at all —
delayMsset,gesture-custom, taps buried inside arun-sequence— now say why the rewrite was skipped and how to record it so that it happens.