Feature request: emit structured evidence bundles for actions and failures (text + json content)

[KazuCocoa]

(Wrote this doc with LLM's help)

I'd like to find better evidence for each action. So, current response is text only basis. If we gives more better usage in CI, probably it makes sense to add more context for each action such as what kind of locator the tool actually used. This would help for the future CI usage etc.

This assumption could be wrong if current info was sufficient though.

Body

Problem

Appium MCP is positioned as an MCP server for mobile automation and recent releases show active development, but public-facing descriptions emphasize actions, session control, and generation more than evidence-oriented execution artifacts. For automation at scale, actions are not enough; every run needs replayable and diagnosable output.

Today, when a tool call fails (e.g., tap_element, type_text, or future wait_for_element), users often see only a high-level error. It’s difficult to answer questions like:

• Which locator was actually used?
• Which element did we resolve (by WebDriver/Appium element ID)?
• What platform/app/context/screen was active at the time?
• What did the UI look like before and after the action?
• Did the app navigate, crash, or stay on the same screen?
• Was it a timeout, a locator issue, or something else?

This makes failures hard to debug (especially in CI) and keeps Appium MCP closer to an “interactive operator surface” than an auditable automation platform.

---

Proposal

For every “meaningful” tool call (starting with core action tools and later waits/asserts), Appium MCP should return a structured evidence bundle alongside a compact summary.

To keep token usage under control while still providing rich evidence, I’m proposing a dual-content pattern:

• A short text summary for the LLM
• A structured json object for tools/CI/logs

EvidenceBundle shape (example)

type EvidenceBundle = {
  id: string;
  toolName: string;
  args: any;

  locator?: {
    strategy: string;  // e.g., "accessibilityId", "id", "xpath"
    value: string;     // e.g., "login_button"
  };

  element?: {
    webdriverId?: string; // Appium/WebDriver element id
    logicalId?: string;   // optional: test-oriented ID like "LoginScreen.LoginButton"
  };

  context?: {
    platform: "android" | "ios";
    appPackageOrBundle?: string;
    activityOrScreen?: string;
    contextName?: string; // e.g., "NATIVE_APP", "WEBVIEW_com.example"
  };

  screenshots?: {
    beforePath?: string; // file paths, not inline base64
    afterPath?: string;
  };

  hierarchy?: {
    pageSourcePath?: string;     // optional: UI hierarchy snapshot
    accessibilityPath?: string;  // optional: accessibility tree snapshot
  };

  timing: {
    startedAt: string;   // ISO timestamp
    finishedAt: string;  // ISO timestamp
    durationMs: number;
  };

  error?: {
    code: string;    // e.g., "ELEMENT_NOT_FOUND", "TIMEOUT", "CONTEXT_NOT_AVAILABLE", "ACTION_FAILED"
    message: string; // human-readable summary
  };

  evidenceRef?: string; // path or ID for the persisted bundle
};

This adds element-level information (both WebDriver element ID and an optional logical/test ID), which makes it easier to correlate with driver logs and test models.

Return format (text + json)

Instead of stuffing everything into text, the server would return:

return {
  content: [
    {
      type: "json",
      mimeType: "application/vnd.appium.evidence+json",
      data: {
        id: evidence.id,
        toolName: "tap_element",
        locator: evidence.locator,
        element: evidence.element,
        context: evidence.context,
        timing: evidence.timing,
        error: evidence.error,
        evidenceRef: evidenceJsonPath // e.g., runs/.../evidence.json
      }
    },
    {
      type: "text",
      text: !evidence.error
        ? `tap_element succeeded (locator=${evidence.locator?.strategy}:${evidence.locator?.value}, evidenceId=${evidence.id})`
        : `tap_element failed (code=${evidence.error.code}, locator=${evidence.locator?.strategy}:${evidence.locator?.value}, evidenceId=${evidence.id})`
    }
  ]
};

This keeps the LLM-facing text small, while providing a structured json blob for tools and CI systems to consume.

---

Token-efficiency considerations

To avoid unnecessary token usage:

• The text summary should remain compact:
  • Short sentence + error code + evidence ID
• Heavy data should not be embedded in text:
  • Screenshots saved as files (before.png, after.png)
  • Full hierarchy snapshots saved as files
• The json evidence should reference heavy artifacts by path/ID, not inline base64 blobs.

Example for tap_element:

const summary = evidence.error
  ? `tap_element failed (code=${evidence.error.code}, evidenceId=${evidence.id})`
  : `tap_element succeeded (evidenceId=${evidence.id})`;

return {
  content: [
    {
      type: "json",
      mimeType: "application/vnd.appium.evidence+json",
      data: {
        id: evidence.id,
        locator: evidence.locator,
        element: evidence.element,
        context: evidence.context,
        timing: evidence.timing,
        error: evidence.error,
        evidenceRef: evidenceJsonPath
      }
    },
    {
      type: "text",
      text: summary
    }
  ]
};

---

Example flow (narrative)

For a tap_element tool call:

1. Before:
   • Capture context (platform/app/contextName)
   • Capture optional before.png (saved to disk)
2. Action:
   • Resolve locator (strategy + selector)
   • Call findElement and record the WebDriver element ID in evidence.element.webdriverId
   • Optionally map to a logical/test ID (e.g., LoginScreen.LoginButton)
   • Attempt click
   • Normalize any failure into { code, message }
3. After:
   • Capture optional after.png
   • Record timing
   • Persist evidence.json to disk (if configured)
4. Return:
   • json evidence content (lightweight subset + evidenceRef)
   • text summary with code + evidence ID

---

Expected benefit

This change would:

• Make Appium MCP much easier to debug:
  • Every action has a reproducible, structured trace, including which element was actually targeted.
• Provide a clear CI story:
  • Failures come with artifacts (screenshots, element IDs, context, timing) as a matter of course.
• Improve agent behavior:
  • Agents can check error codes, element IDs, and evidence references instead of guessing.
• Preserve token efficiency:
  • LLMs see a small summary; heavy evidence stays in separate files or structured json data.
• Move Appium MCP closer to a full-fledged test runner for agents, not just a thin tool bridge.

---

You can paste this as your Issue 2, or tweak field names if you want to align with your internal naming conventions.

[Rul1an]

Really like this direction. For Appium MCP, a structured evidence surface feels like the difference between "an agent can operate the device" and "CI can review what actually happened."

One framing worth keeping clear early: the object returned from a single tool call is an action evidence record, while the durable CI artifact is a run-level evidence bundle that references many of those records plus screenshots, hierarchy dumps, logs, and summaries.

That split keeps the tool response small, but still gives CI something complete to inspect later.

A few choices worth making early:

• add schema_version and producer/version fields
• avoid making raw args the stable CI contract; for sensitive actions, prefer an args summary, args digest, or explicit redaction marker
• reference screenshots and page sources by path/URI plus digest and size, not inline bodies
• make outcome explicit: status, normalized error code, and failure stage when known
• give the run-level bundle a manifest or digest so reviewers know exactly which artifacts were reviewed

The text response can stay tiny: success/failure, error code, evidence id. The JSON response can then be a stable consumer contract for agents, CI, and report generators.

The main thing I would avoid is letting EvidenceBundle become only "whatever this tool call returned." If the goal is CI/debug/audit, the more durable shape is probably:

action record -> referenced artifacts -> run manifest -> bundle digest

That gives Appium MCP room to start small per action without painting itself into a corner when people want to compare runs, persist artifacts, or verify later that a failure report still points at the same screenshots and hierarchy snapshots.

One prior-art example is Assay's evidence bundles, which made similar choices around content addressing, redaction markers, and references-over-bodies. Not suggesting Appium MCP needs that shape wholesale, but those are the tradeoffs I'd look at before the schema settles.

[abhinvv1]

@KazuCocoa
I'd like to take this. Can you assign this to me?
One thing worth flagging: the example { type: "json", mimeType }, but that's not a valid MCP content type. { type: 'resource', resource: { uri: evidence://${id}, mimeType: 'application/json', text: JSON.stringify(evidence) } } would be more suitable.

Planning to create:

1. Evidence types + a buildEvidenceResult() helper + a classifyError() mapper (ELEMENT_NOT_FOUND / TIMEOUT / STALE_ELEMENT / CONTEXT_NOT_AVAILABLE / INVALID_SELECTOR / ACTION_FAILED), behind an opt-in flag (APPIUM_MCP_EVIDENCE=1), should there be an opt-in flag, or default-on?
2. Populate locator/element/context for appium_find_element and appium_gesture first.

[Rul1an]

Good catch on the MCP content type. I would avoid type: "json" as well.

For the first implementation, I think the clean shape is:

• tiny text summary for the model
• embedded resource for the lightweight action evidence record, with mimeType: "application/vnd.appium.evidence+json"
• optional persisted resource_link / evidenceRef for the heavier run artifacts

Something like this feels closer to the protocol shape:

{
  type: "resource",
  resource: {
    uri: `evidence://${evidence.id}`,
    mimeType: "application/vnd.appium.evidence+json",
    text: JSON.stringify(evidence),
  },
}

For rollout, I would make it opt-in at first with APPIUM_MCP_EVIDENCE=1, then consider default-on once the schema settles. The small action record can probably become default-on later, but screenshots, page source, accessibility dumps, and run manifests should stay separately configurable because they affect storage, privacy, and CI artifact volume.

Starting with appium_find_element and appium_gesture sounds like the right cut. I would keep the first schema narrow and stable:

• schema_version
• producer / Appium MCP version
• evidence_id
• tool_name
• status and stage
• locator
• element.webdriver_id
• context
• timing
• normalized error
• artifact refs with digest and size when available

The one thing I would keep explicit in the helper is that this is an action evidence record. A later run-level bundle can then reference many action records plus screenshots, page sources, logs, and a manifest digest without changing the per-tool response contract.

[KazuCocoa]

@abhinvv1
Let's disable this by default to start from small. Potentially we'll modify the structure so I assume disabling this first, and test out a bit, and enable it if they are confident makes sense to me

[Rul1an]

@abhinvv1 the run-level half of this still looks open, and the issue is still yours. Are you planning to carry on with it after #386? Happy to leave it with you, and equally happy to take a slice if that helps.

If a slice would help, the smallest one that stays inside disable-by-default is a per-run directory holding the action records that were emitted, plus a manifest.json listing each file with size and sha256, along the lines sketched up-thread. It needs no driver calls and no new capture timing, and it gives the heavier artifacts something to hang off later.

One thing I have not raised before: whichever artifact classes get added, it helps if the manifest records the setting even when a class is off. In CI "nothing was captured" and "capture was off" currently read the same, and that difference is usually the diagnosis.

Worth flagging on screenshots before anyone starts. An after-capture on failure is cheap since the handler already has the result, but a before-capture cannot be conditional on failure, so it costs a driver round trip on every successful action too. There is also the existing SCREENSHOTS_DIR behaviour to reconcile with. That is why the manifest looks like the safer first cut.

@KazuCocoa happy to follow whichever split you and @abhinvv1 prefer.