Skip to content

fix(flow): refuse a recorded tap that would replay somewhere else - #722

Draft
hubgan wants to merge 5 commits into
fix/ios-flow-tree-depth-and-targetingfrom
fix/recorder-tap-replay-fidelity
Draft

fix(flow): refuse a recorded tap that would replay somewhere else#722
hubgan wants to merge 5 commits into
fix/ios-flow-tree-depth-and-targetingfrom
fix/recorder-tap-replay-fidelity

Conversation

@hubgan

@hubgan hubgan commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #721.

The core mismatch

Replay taps a selector's centre, not the point that was recorded. The recorder only checked that the resolved frame contained the point — which lets two silent misfires through.

1. Container-sized targets

A tap on blank space inside a drawer resolved to the drawer's whole scroll area — 0.72 of the screen. It contains the point, so it recorded. Replay then tapped its centre, hit the "Chat" item, and reported pass while navigating somewhere the walkthrough never went.

A frame covering most of the screen is a container, not a control, and is now refused with that reason. Narrowing cannot help here: the problem is the element, not the selector.

The threshold is a judgement between two real failures. Too permissive records containers (above); too strict makes ordinary widgets unrecordable — a feed post is half the screen and tapping it is a perfectly normal QA step.

Why not "does the centre resolve back to the same node"? That test was tried and is wrong on a flattened tree: a control's own label is a sibling rect sitting on its centre, so a like button, a search field, a full-width row and every grid cell were refused — while replaying perfectly, because the touch is still inside the control. Node identity cannot tell a label from an independent control; size can tell a control from a container.

2. Ambiguity is answered by narrowing, not coordinates

When the derived selector matches the tapped element and something else and ranks the other first, the recorder now tries narrower forms — the node's own role, then its identifier — before giving up. The runner resolves either form, so answering ambiguity with coordinates was throwing away a perfectly good target.

A within scope is deliberately not derived, even though it would separate one feed row's button from another's. The flow tree is flattened and carries no z-order, so a container can only be found geometrically — and with a modal open, the foreground modal is a perfectly good geometric ancestor of a background element. A tap on a composer's text input recorded as a feed post "inside" the composer, then failed on any screen whose feed content differed. The scopes that survive are the ones an author writes knowingly at polish.

3. Positional ids are no longer derived

profilePager-selector-2 encodes the element's index among its siblings, so it names a slot rather than a thing: it survives no re-order and silently addresses a different control once one is inserted before it.

It matters most where the recorder is least reliable. Observed on an edit-profile sheet: a tap on the display-name field derived the profile pager's "Media" tab — a view behind the modal. An oversized or ambiguous background match is caught by the two checks above; a positional id passes all of them and records silently.

4. Every kept-coordinate warning names the retarget

A coordinate tap replays fine today and breaks on the first layout change, so this warning is the author's only signal — and it is read at the one moment the screen is still there to retarget against.

The two failures need opposite advice: an element nothing can address needs a real target found in the tree; one that several things address equally needs disambiguating. Saying "no selector could be derived" for the second sent authors off to re-discover a selector they already had.

Raw-coordinate steps where capture never ran at all — delayMs set, gesture-custom, taps buried inside a run-sequence — now say why the rewrite was skipped and how to record it so that it happens.

@hubgan
hubgan force-pushed the fix/recorder-tap-replay-fidelity branch from fd7f1b1 to 2d8157d Compare August 4, 2026 07:23
@hubgan
hubgan force-pushed the fix/recorder-tap-replay-fidelity branch from 6205ea7 to c67e984 Compare August 5, 2026 10:15
@hubgan
hubgan force-pushed the fix/recorder-tap-replay-fidelity branch from c67e984 to 2236770 Compare August 5, 2026 10:38
@hubgan
hubgan force-pushed the fix/recorder-tap-replay-fidelity branch from 2236770 to 0d47eb4 Compare August 6, 2026 09:56
@hubgan
hubgan force-pushed the fix/recorder-tap-replay-fidelity branch from 0d47eb4 to 4d288ce Compare August 9, 2026 20:47
hubgan added 3 commits August 10, 2026 10:55
Replay taps a selector's CENTRE, not the point that was recorded. The recorder
only checked that the resolved frame contained the point, which lets two silent
misfires through.

A tap on blank space inside a drawer resolved to the drawer's whole scroll area
(0.72 of the screen). It contains the point, so it recorded — and replay tapped
the centre, hit the "Chat" item, and reported PASS while navigating somewhere
the walkthrough never went. A frame that covers most of the screen is a
container, not a control, and is now refused with that reason.

The size test is deliberately not "does the centre resolve back to the same
node". That was tried and is wrong on a FLATTENED tree: a control's own label is
a sibling rect sitting on its centre, so a like button, a search field, a
full-width row and every grid cell were refused while replaying perfectly.

When the derived selector matches the tapped element AND something else and
ranks the other first, the recorder now narrows it — the node's own role, then
its identifier — before giving up. Ambiguity is not "this element cannot be
addressed", and answering it with coordinates threw away a perfectly good
target. A `within` scope is deliberately NOT derived: the flow tree is flattened
and carries no z-order, so the geometric ancestor of a tap inside a modal can be
a background element, and one recorded that way failed on any screen whose
content differed.

`deriveSelector` also stops deriving POSITIONAL ids (`…-selector-2`). The number
is the element's index among its siblings, so the id names a slot rather than a
thing. It matters most where the recorder is least reliable: a tap on an
edit-profile sheet derived the profile pager's "Media" tab from a view BEHIND
the modal. An oversized or ambiguous background match is caught by the checks
above, but a positional id passes all of them and records silently.

Every kept-coordinate warning now names the failure AND the retarget, and the
two are different: an element nothing can address needs a real target found in
the tree; one that several things address equally needs disambiguating. Saying
"no selector could be derived" for the second sent authors to re-discover a
selector they already had. Raw-coordinate steps with no capture attempt at all
(`delayMs` set, `gesture-custom`, taps buried in a `run-sequence`) say why the
rewrite did not run and how to record it so that it does.
…fused

narrowedSelectors added the tapped node's identifier to disambiguate a
retargeting selector, but that branch is reachable only when deriveSelector
declined the id — i.e. only when the id is positional. So it could re-record
`{ text, identifier: profilePager-selector-2 }` silently, smuggling the slot
id past the very guard deriveSelector applies, in the Bluesky edit-profile
modal case the refusal exists for. Guard the branch with POSITIONAL_ID so an
ambiguous positional-only target keeps coordinates with its warning instead.

Also make the oversized-target path honest: at >60% area the tree cannot tell
a container from a full-bleed control, so the warning no longer asserts "it is
a container, not a control", and the kept-coordinate remedy no longer tells the
author an element that HAS an id has "no id or label" — it points at the
smaller control under the point. Correct the POSITIONAL_ID doc (no
argent-create-flow "audit" exists) and the argent-create-flow skill's
tap-capture description to match the new refusals.
The identifier branch in narrowedSelectors was unreachable: deriveSelector
already makes any stable, non-positional id the base selector, so `base`
lacks an identifier only when the id is absent or positional — the exact
negation of the branch's own guard. Remove it, and with it the sole
cross-module use of POSITIONAL_ID (now un-exported).

Correct the two messages the dead branch made false: the ambiguity warning
no longer claims it tried an identifier (only role is ever added), and the
generic coordinate remedy no longer asserts "no id or label" for the
no-element / capture-failed cases.

Pin the container-area threshold on both sides and cover the unaddressable
remedy branch, which no test exercised.
@hubgan
hubgan force-pushed the fix/recorder-tap-replay-fidelity branch from 4d288ce to 7cc3469 Compare August 10, 2026 08:55
hubgan added 2 commits August 11, 2026 11:54
…r-tap-replay-fidelity

# Conflicts:
#	packages/skills/skills/argent-create-flow/SKILL.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant