fix(android): carry the checked state of a checkable control on the snapshot node (#2913)

* fix(android): carry the checked state of a checkable control on the snapshot node

The snapshot helper never serialized `checked` or `checkable`, and the host reads only the helper's XML, so no later layer could recover it: a switch, checkbox, or radio button looked the same on or off, `get attrs` had no `checked` field, and snapshot text showed the control as plain. The helper now writes `checked` on every node Android reports as checkable, with both answers, so an unchecked switch reads `false`, a node that cannot be checked reads nothing, and so does a helper older than the attribute. The parser, the Android hierarchy node, and the published snapshot node carry it to `get attrs`, the unchanged-snapshot comparison, the selector digest, and the `[checked]` and `[unchecked]` markers in snapshot text. Both answers render because the diff compares the state and a checkable control rendered as plain would hide that it toggles.

* fix(daemon): the interaction outcome weighs the states stateMarkers prints, and a scroll's container check matches content by identity

Review follow-up. The interaction surface key folded enabled and selected but not checked, so a tap whose only effect was a toggle read as a no-op on the outcome lane while the diff called it a change, and a no-change retry would tap the switch straight back. The key now spreads stateMarkers, so the outcome lane, the unchanged-snapshot comparison, and the diff weigh one list.

With checked in the key, a flip inside a scroll container beside an unrelated change elsewhere (the status clock) read as content moving within the container. discriminatingSurfaceChangedWithinRect now matches entries on the flip-tolerant identity where one exists, told apart by document order when repeated, so a state flip at the same rect is not movement.

Tests: a checked-only flip is a change on the outcome lane; a flip at the same rect is not movement and a moved row is; the scroll claim is withheld for a flipped toggle inside the container; checked in both answers invalidates snapshot reuse and survives the selector digest.

* test(daemon): the capture-retry fixtures spell the interaction key with state markers

The two hand-written pre-signatures carried the literal enabled and unselected segments the key no longer writes for a plain node, so the post-tap capture read as changed and the retry never fired.

* fix(daemon): a surface entry carries a state-free content key, so an anonymous toggle is not movement

Review follow-up. The within-container movement check fell back to `key` for an entry without an identity, and `key` carries the checked state, so an unlabelled switch a swipe brushed still read as content moving. Every entry now carries `content`: the identity where the node has one, else its type and role; the check compares on that alone.

* fix(maestro): the snapshot signature weighs checked, so a toggle tap is not a no-op to retap

maestroSnapshotSignature hashed label, value, enabled, selected, focused
and bounds; a switch, checkbox or radio whose only observable effect is
its checked state produced the same signature before and after the tap,
so the settle and the no-change retry read the tap as a no-op and could
tap it back. checked joins the hashed object the way the interaction
outcome key already weighs it.

Author: okwasniewski

android/snapshot-helper/src/main/java/com/callstack/agentdevice/snapshothelper/AccessibilityTreeXml.java

@@ -69,6 +69,12 @@ static void appendNode(
     // unselected control reports selected=false and a helper older than this attribute reports
     // nothing at all.
     appendAttribute(xml, "selected", Boolean.toString(node.isSelected()));
+    // Present only on a checkable control, with both answers: an unchecked switch reports
+    // checked=false, while a node that cannot be checked reports nothing, like a helper older than
+    // this attribute.
+    if (node.isCheckable()) {
+      appendAttribute(xml, "checked", Boolean.toString(node.isChecked()));
+    }
     boolean scrollable = node.isScrollable();
     if (scrollable) {
       appendAttribute(xml, "scrollable", "true");
@@ -164,9 +170,8 @@ private static boolean isHeading(AccessibilityNodeInfo node) {
     return Build.VERSION.SDK_INT >= Build.VERSION_CODES.P && node.isHeading();
   }
 
-  // Declared residue (agent-device #1832): checked / checkable / long-clickable are not serialized,
-  // so toggle state is invisible to agents. Adding them is a helper protocol change (new attributes
-  // + host parser + fields on the wire node), tracked there.
+  // Declared residue (agent-device #1832): long-clickable is not serialized. Adding it is a helper
+  // protocol change (new attribute + host parser + field on the wire node).
   private static void appendDrawingOrderAttribute(StringBuilder xml, AccessibilityNodeInfo node) {
     if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) {
       appendAttribute(xml, "drawing-order", Integer.toString(node.getDrawingOrder()));

packages/capture-kit/src/snapshot/__tests__/snapshot-diff-checked.test.ts

@@ -0,0 +1,49 @@
+import assert from 'node:assert/strict';
+import { test } from 'vitest';
+import type { SnapshotNode } from '@agent-device/kernel/snapshot';
+import { buildSnapshotDiff } from '../snapshot-diff.ts';
+
+function toggle(checked?: boolean): SnapshotNode {
+  return {
+    ref: 'e12',
+    index: 0,
+    depth: 0,
+    type: 'android.widget.Switch',
+    label: 'Wi-Fi switch',
+    enabled: true,
+    hittable: true,
+    ...(checked === undefined ? {} : { checked }),
+  };
+}
+
+test('a checked-only flip diffs as changed lines that read differently', () => {
+  const diff = buildSnapshotDiff([toggle(false)], [toggle(true)]);
+  const changed = diff.lines.filter((line) => line.kind !== 'unchanged');
+
+  assert.equal(diff.summary.unchanged, 0);
+  assert.equal(changed.length > 0, true);
+  // The comparable key carries the checked state, so the rendered line has to as well, and both
+  // answers render: the pair reads `[unchecked]` -> `[checked]`, not two identical lines.
+  assert.match(changed[0]!.text, /\[unchecked\]/);
+  assert.match(changed.at(-1)!.text, /\[checked\]/);
+});
+
+test.each([
+  ['both unchecked', [toggle(false)], [toggle(false)]],
+  ['both unreported', [toggle()], [toggle()]],
+])('a still toggle with %s diffs as unchanged', (_label, previous, current) => {
+  const diff = buildSnapshotDiff(previous, current);
+
+  assert.equal(
+    diff.lines.every((line) => line.kind === 'unchanged'),
+    true,
+  );
+});
+
+test('a node that cannot be checked never reads as unchecked', () => {
+  const diff = buildSnapshotDiff([toggle()], [toggle()]);
+  assert.equal(
+    diff.lines.every((line) => !line.text.includes('checked')),
+    true,
+  );
+});

packages/capture-kit/src/snapshot/snapshot-diff.ts

@@ -5,6 +5,7 @@ import {
   displayLabel,
   formatRole,
   formatSnapshotLine,
+  stateMarkers,
 } from './snapshot-lines.ts';
 
 export type SnapshotDiffResult = {
@@ -27,18 +28,17 @@ type SnapshotComparableLine = {
 function snapshotNodeToComparableLine(node: SnapshotNode, depthOverride?: number): string {
   const role = formatRole(node.type ?? 'Element');
   const textPart = displayLabel(node, role);
-  const enabledPart = node.enabled === false ? 'disabled' : 'enabled';
-  const selectedPart = node.selected === true ? 'selected' : 'unselected';
+  // The states the rendered line prints are the states the key compares, from one function, so a
+  // fact the diff weighs is always visible in the line it prints.
+  const statePart = stateMarkers(node).join(',');
   const hittablePart = node.hittable === true ? 'hittable' : 'not-hittable';
   const depthPart = String(depthOverride ?? node.depth ?? 0);
   // The rendered line carries the actions list, so the comparable key has to as
   // well: otherwise an action-only change diffs as 'unchanged' while its text
   // silently differs from the baseline's. JSON-encoded because the names are
   // app-authored and may contain the field separator.
   const actionsPart = node.actions ? JSON.stringify(node.actions) : '';
-  return [depthPart, role, textPart, enabledPart, selectedPart, hittablePart, actionsPart].join(
-    '|',
-  );
+  return [depthPart, role, textPart, statePart, hittablePart, actionsPart].join('|');
 }
 
 export function buildSnapshotDiff(

packages/capture-kit/src/snapshot/snapshot-freshness/android.ts

@@ -28,11 +28,12 @@ export function isNavigationSensitiveAction(command: string): boolean {
 
 /**
  * Route signature of an Android snapshot, from the fields the Android backend actually carries.
- * The helper serializes no `role`, `checked` or `long-clickable` (declared residue,
- * #1832), so a signature keying on them would compare constants and claim discrimination it does
- * not have. `selected` is left out as a judgement call rather than an inability: a tab-bar tap
- * flips it on two nodes, which the 90%-identical threshold below absorbs at every tree size this
- * check runs on, so keying on it would only add capture retries.
+ * The helper serializes no `role` or `long-clickable` (declared residue, #1832), so a signature
+ * keying on them would compare constants and claim discrimination it does not have. `selected`
+ * and `checked` are left out as a judgement call rather than an inability: a tab-bar tap flips
+ * selection on two nodes and a toggle tap flips one checked state, which the 90%-identical
+ * threshold below absorbs at every tree size this check runs on, so keying on them would only add
+ * capture retries.
  */
 export function buildSnapshotSignatures(nodes: SnapshotState['nodes']): string[] {
   return nodes.map((node) =>

packages/capture-kit/src/snapshot/snapshot-lines.ts

@@ -195,6 +195,21 @@ export function formatRole(type: string): string {
   return lookupRoleLabel(normalized) || normalized || 'element';
 }
 
+/**
+ * The state markers every rendering path prints, and the states a snapshot diff compares: the diff
+ * renders its lines without text-surface summarizing, and a fact it weighs has to be visible in the
+ * line it prints, or a flip reads as a changed pair whose two lines look identical. Both checked
+ * answers render, since a checkable control shown plain would hide that it toggles; a node that
+ * cannot be checked carries neither.
+ */
+export function stateMarkers(node: SnapshotNode): string[] {
+  const markers: string[] = [];
+  if (node.enabled === false) markers.push('disabled');
+  if (node.selected === true) markers.push('selected');
+  if (node.checked !== undefined) markers.push(node.checked ? 'checked' : 'unchecked');
+  return markers;
+}
+
 function lookupRoleLabel(normalized: string): string | undefined {
   return Object.prototype.hasOwnProperty.call(ROLE_LABELS, normalized)
     ? ROLE_LABELS[normalized]
@@ -238,13 +253,7 @@ function buildLineMetadata(
   options: SnapshotLineFormatOptions,
   textSurface: { text: string; isLargeSurface: boolean; shouldSummarize: boolean },
 ): string[] {
-  const metadata: string[] = [];
-  if (node.enabled === false) metadata.push('disabled');
-  // Selection is a state a snapshot diff can report as changed, and the diff renders its lines
-  // without text-surface summarizing. A fact the diff compares has to be visible in the line it
-  // prints, or a selection flip reads as a changed pair whose two lines look identical.
-  if (node.selected === true) metadata.push('selected');
-  metadata.push(...(node.presentationHints ?? []));
+  const metadata = [...stateMarkers(node), ...(node.presentationHints ?? [])];
   if (!options.summarizeTextSurfaces) {
     return uniqueMetadata(metadata);
   }

packages/kernel/src/snapshot.ts

@@ -251,6 +251,8 @@ export type RawSnapshotNode = {
   rect?: Rect;
   enabled?: boolean;
   selected?: boolean;
+  /** Checked state of a checkable control (switch, checkbox, radio); absent means not checkable or unavailable. */
+  checked?: boolean;
   focused?: boolean;
   /** Accessibility heading flag an app set on the node; absent means not a heading or unavailable. */
   heading?: boolean;

packages/maestro/src/daemon-port/__tests__/daemon-runtime-port-observation.test.ts

@@ -359,12 +359,26 @@ test('normalizes absent attributes like Maestro iOS hierarchy mapping', () => {
       enabled: false,
       selected: false,
       focused: false,
+      checked: false,
     },
   ]);
 
   expect(maestroSnapshotSignature(first)).toBe(maestroSnapshotSignature(second));
 });
 
+test('a checked-only flip changes the snapshot signature', () => {
+  // An Android switch tapped on: nothing about it moves or renames, only `checked` flips. The
+  // same signature before and after would read the tap as a no-op and retap it off again.
+  const off = makeSnapshot([
+    { index: 0, type: 'android.widget.Switch', label: 'Wi-Fi', checked: false },
+  ]);
+  const on = makeSnapshot([
+    { index: 0, type: 'android.widget.Switch', label: 'Wi-Fi', checked: true },
+  ]);
+
+  expect(maestroSnapshotSignature(off)).not.toBe(maestroSnapshotSignature(on));
+});
+
 test('excludes agent-device presentation metadata from Maestro hierarchy signatures', () => {
   const first = makeSnapshot([
     {

packages/maestro/src/daemon-port/daemon-runtime-port-observation.ts

@@ -408,6 +408,7 @@ export function maestroSnapshotSignature(snapshot: SnapshotState): string {
             enabled: node.enabled ?? false,
             selected: node.selected ?? false,
             focused: node.focused ?? false,
+            checked: node.checked ?? false,
             bounds: maestroSnapshotBounds(node.rect),
           })),
       ),

packages/platform-android/src/__tests__/ui-hierarchy-checked.test.ts

@@ -0,0 +1,70 @@
+import { expect, test } from 'vitest';
+import { buildUiHierarchySnapshot, parseUiHierarchyTree } from '../ui-hierarchy.ts';
+
+// A settings row: a switch the helper reports as checkable beside a label that is not. The helper
+// writes `checked` only on the checkable node, with both answers.
+function togglesXml(wifiChecked: boolean): string {
+  return `<hierarchy>
+    <node class="android.widget.LinearLayout" resource-id="toggles" bounds="[0,0][400,200]"
+      window-index="0" window-type="1" window-layer="1" window-active="true" window-focused="true"
+      window-bounds="[0,0][400,800]" visible-to-user="true" enabled="true">
+      <node class="android.widget.TextView" resource-id="wifi-label" text="Wi-Fi"
+        bounds="[0,0][200,100]" enabled="true" visible-to-user="true" />
+      <node class="android.widget.Switch" resource-id="wifi-switch" content-desc="Wi-Fi switch"
+        bounds="[200,0][400,100]" clickable="true" enabled="true" visible-to-user="true"
+        checked="${wifiChecked}" />
+      <node class="android.widget.RadioButton" resource-id="size-small" text="Small"
+        bounds="[0,100][400,200]" clickable="true" enabled="true" visible-to-user="true"
+        checked="false" />
+    </node>
+  </hierarchy>`;
+}
+
+// A helper older than the `checked` attribute, or a switch it did not report as checkable.
+const UNREPORTED_CHECKED_XML =
+  '<hierarchy><node class="android.widget.Switch" resource-id="legacy-switch" content-desc="Wi-Fi"' +
+  ' bounds="[200,0][400,100]" clickable="true" enabled="true" visible-to-user="true" /></hierarchy>';
+
+function toggleNodes(raw: boolean, interactiveOnly = false, wifiChecked = true) {
+  const { nodes } = buildUiHierarchySnapshot(
+    parseUiHierarchyTree(togglesXml(wifiChecked)),
+    undefined,
+    { raw, interactiveOnly },
+  );
+  const byId = (identifier: string) => nodes.find((node) => node.identifier === identifier);
+  return { label: byId('wifi-label'), wifi: byId('wifi-switch'), small: byId('size-small') };
+}
+
+test.each([
+  { raw: false, interactiveOnly: false },
+  { raw: false, interactiveOnly: true },
+  { raw: true, interactiveOnly: false },
+  { raw: true, interactiveOnly: true },
+])(
+  'checked state reaches snapshot nodes in every projection (raw=$raw, -i=$interactiveOnly)',
+  ({ raw, interactiveOnly }) => {
+    const { wifi, small } = toggleNodes(raw, interactiveOnly);
+    expect(wifi?.checked).toBe(true);
+    expect(small?.checked).toBe(false);
+  },
+);
+
+test.each([true, false])('the switch answers the state the helper observed (%s)', (wifiChecked) => {
+  expect(toggleNodes(false, false, wifiChecked).wifi?.checked).toBe(wifiChecked);
+});
+
+test('attrs answer explicit false where a node that cannot be checked answers nothing', () => {
+  // Serialized, because that is the answer an agent reads: an unavailable fact drops the key
+  // while JSON encodes an observed `false`.
+  const { label, small } = toggleNodes(false);
+  expect(JSON.parse(JSON.stringify(small)).checked).toBe(false);
+  expect(JSON.parse(JSON.stringify(label))).not.toHaveProperty('checked');
+});
+
+test('an unreported checked state stays unknown instead of becoming false', () => {
+  const node = buildUiHierarchySnapshot(parseUiHierarchyTree(UNREPORTED_CHECKED_XML), undefined, {
+    raw: false,
+  }).nodes.find((node) => node.identifier === 'legacy-switch');
+  expect(node).toBeDefined();
+  expect(node?.checked).toBeUndefined();
+});

packages/platform-android/src/ui-hierarchy-builder.ts

@@ -356,6 +356,7 @@ function createAndroidRawSnapshotNode(
     selected: node.selected,
     heading: node.heading,
     roleDescription: node.roleDescription,
+    checked: node.checked,
     editable: node.editable,
     password: node.password,
     hintShowing: node.hintShowing,