Stream devices live, add agent companion mode, finish unified trails

.trailblaze-sync

@@ -1 +1 @@
-7c171455076b64daf369218b3b72b9b5267404e9
+232c993d0a2648f77ea9c822f7de1f589de183dd

docs/CLI.md

@@ -1115,6 +1115,7 @@ trailblaze config reset
 | `screenshot-format` | Image format used for screenshots sent to the LLM and shown in the timeline | png, jpeg, webp, or 'unset' to use the framework default (webp) |
 | `screenshot-max-dimensions` | Max screenshot dimensions as <longer>x<shorter> (e.g. 1536x768, 2048x1024) | WIDTHxHEIGHT (positive ints), or 'unset' to use the framework default (1536x768) |
 | `screenshot-quality` | Compression quality 0.05..1.0 for lossy formats (jpeg, webp); ignored for png | 0.05..1.0, or 'unset' to use the framework default (0.80) |
+| `android-stream-screenshots` | Experimental: serve Android agent-loop screenshots from the live device stream (default: off) | true, false, or 'unset' to inherit the default (off) |
 
 **Examples:**
 

docs/showcase-trails.yml

@@ -26,12 +26,12 @@
 
 ios:
   slug: ios-contacts
-  recording: trails/ios-contacts/test-create-then-delete/ios-iphone.trail.yaml
+  recording: trails/ios-contacts/test-create-then-delete/trail.yaml
 
 android:
   slug: clock
   recording: trails/clock/set-alarm-730am/trail.yaml
 
 web:
   slug: wikipedia
-  recording: trails/wikipedia/test-article-shakespeare/web.trail.yaml
+  recording: trails/wikipedia/test-article-shakespeare/trail.yaml

examples/dependencies/debugAndroidTestRuntimeClasspath.txt

@@ -3,6 +3,7 @@
 :trailblaze-android
 :trailblaze-common
 :trailblaze-models
+:trailblaze-ondevice-rpc-proto
 :trailblaze-quickjs-tools
 :trailblaze-tracing
 ai.koog:agents-core-android:1.0.0
@@ -102,6 +103,8 @@ com.squareup.okhttp3:okhttp-android:5.3.2
 com.squareup.okhttp3:okhttp:5.3.2
 com.squareup.okio:okio-jvm:3.17.0
 com.squareup.okio:okio:3.17.0
+com.squareup.wire:wire-runtime-jvm:5.3.3
+com.squareup.wire:wire-runtime:5.3.3
 com.typesafe:config:1.4.8
 dev.mobile:maestro-client:2.6.1
 dev.mobile:maestro-orchestra-models:2.6.1
@@ -119,6 +122,8 @@ io.ktor:ktor-client-logging-jvm:3.5.0
 io.ktor:ktor-client-logging:3.5.0
 io.ktor:ktor-client-okhttp-jvm:3.5.0
 io.ktor:ktor-client-okhttp:3.5.0
+io.ktor:ktor-client-websockets-jvm:3.5.0
+io.ktor:ktor-client-websockets:3.5.0
 io.ktor:ktor-events-jvm:3.5.0
 io.ktor:ktor-events:3.5.0
 io.ktor:ktor-http-cio-jvm:3.5.0

examples/wikipedia/README.md

@@ -7,7 +7,7 @@ Native driver. The trailmap ships:
 - **9 scripted tools** (in TypeScript) covering search, language switching,
   random article, main-page section verification, banner dismissal, article
   structure assertions, plus a composition example.
-- **28 trails** under `trails/wikipedia/` exercising those tools
+- **30 trails** under `trails/wikipedia/` exercising those tools
   + the built-in `web_*` toolset, with `tags` for selective runs and one
   `skip:` example for opting out gracefully.
 - **A target-scoped system prompt** that teaches the LLM when to reach for
@@ -253,20 +253,23 @@ watching which tool the agent chose for each step.
 
 ## What a trail actually looks like
 
-A trail directory's `blaze.yaml` is the source of truth — a natural-language
+A trail directory's `trail.yaml` is the source of truth — a natural-language
 step the agent resolves against the live page. Here's
-[`test-search-einstein/blaze.yaml`](../../trails/wikipedia/test-search-einstein/blaze.yaml)
+[`test-search-einstein/trail.yaml`](../../trails/wikipedia/test-search-einstein/trail.yaml)
 verbatim:
 
 ```yaml
-- config:
-    title: "Wikipedia: Search for Albert Einstein"
-    platform: web
-    driver: PLAYWRIGHT_NATIVE
-    target: wikipedia
-    tags: [smoke, search]
-- prompts:
-    - step: Search Wikipedia for "Albert Einstein" and verify the resulting article page shows the heading "Albert Einstein".
+config:
+  target: wikipedia
+  devices:
+    web: PLAYWRIGHT_NATIVE
+  tags:
+  - smoke
+  - search
+  title: 'Wikipedia: Search for Albert Einstein'
+
+trail:
+  - step: Search Wikipedia for "Albert Einstein" and verify the resulting article page shows the heading "Albert Einstein".
 ```
 
 The agent sees the trailmap's `target.tools:` (which includes the scripted
@@ -278,22 +281,23 @@ tools when a step matches their task patterns. Run it:
 trailblaze run trails/wikipedia/test-search-einstein --device web/playwright-native
 ```
 
-After a passing run, the CLI auto-saves a fresh `<device>.trail.yaml`
-alongside the `blaze.yaml` — that's the recording artifact you commit for
-deterministic replay. **One caveat for this example specifically:**
-scripted-tool calls (`wikipedia_web_*`) currently don't dispatch from a
-saved `web.trail.yaml` recording — the Playwright agent rejects them at
-replay (tracked under [Known issues](#known-issues)). 5/28 trails here
-carry recordings — the ones that exercise pure `web_*` built-ins; the
-rest stay NL.
+After a passing run, the CLI folds a fresh recording into the trail's
+`trail.yaml` — a `recording:` block nested under each step, keyed by device
+classifier (`web`) — that's the artifact you commit for deterministic replay.
+**One caveat for this example specifically:** scripted-tool calls
+(`wikipedia_web_*`) currently don't dispatch from a saved `web` recording —
+the Playwright agent rejects them at replay (tracked under
+[Known issues](#known-issues)). 5/30 trails here carry recordings — the ones
+that exercise pure `web_*` built-ins; the rest stay NL.
 
 ## Recording vs natural-language: when to use which
 
-Each trail directory has a `blaze.yaml` (the source of truth). If a
-`web.trail.yaml` is also present, the CLI replays it deterministically
-instead of going through the LLM. **Use the decision tree:**
+Each trail directory has a `trail.yaml` (the source of truth). If a step
+carries an embedded `recording:` block for the device under test, the CLI
+replays those tools deterministically instead of going through the LLM.
+**Use the decision tree:**
 
-| Property of the trail                                  | NL (`blaze.yaml` only) | Recorded (`+ web.trail.yaml`) |
+| Property of the trail                                  | NL (no recording)      | Recorded (embedded `recording:`) |
 |--------------------------------------------------------|------------------------|-------------------------------|
 | Should pass on any reasonable Wikipedia state          | ✅ preferred           | risky — recording fixes one DOM |
 | You want zero LLM cost on every run                    | ❌ ~$0.02-0.05/run     | ✅ free after recording      |
@@ -305,9 +309,9 @@ The 5 recorded trails in this repo are the ones that hit structural
 anchors stable across days (`test-article-shakespeare`,
 `test-language-switch-spanish`, the 3 main-page section trails).
 
-**To re-record a trail**: delete its `web.trail.yaml` and re-run the
-`blaze.yaml`. The CLI auto-saves a fresh recording next to the source on
-a passing run.
+**To re-record a trail**: delete the step's `recording:` block(s) from its
+`trail.yaml` and re-run it. The CLI folds a fresh recording back into the
+trail file on a passing run.
 
 ---
 
@@ -341,22 +345,28 @@ Conventions used in this example:
 
 ### Skipping a trail with a written reason
 
-A trail's `config:` block can carry `skip: "reason..."` to opt out of every
-run until the reason is removed. Better than commenting trails out or
-maintaining a separate exclusion list — the reason is committed alongside
-the trail and shows up in `--verbose` output.
+A trail's `config:` block can carry a `skip:` map — keyed by device classifier
+— to opt out until the reason is removed. Listing every device the trail
+targets skips it entirely; keying a single device skips just that one while the
+trail still runs on the others. Better than commenting trails out or
+maintaining a separate exclusion list — the reason is committed alongside the
+trail and shows up in `--verbose` output.
 
 ```yaml
-- config:
-    title: "Wikipedia: Autocomplete suggestions visible while typing search"
-    tags: [search, flaky]
-    skip: "Autocomplete popup is timing-sensitive; remove `skip:` once the suggestion-list probe lands."
-- prompts:
-    - step: …
+config:
+  title: "Wikipedia: Autocomplete suggestions visible while typing search"
+  tags: [search, flaky]
+  devices:
+    web: PLAYWRIGHT_NATIVE
+  skip:
+    web: "Autocomplete popup is timing-sensitive; remove `skip:` once the suggestion-list probe lands."
+
+trail:
+  - step: …
 ```
 
-To run a skipped trail anyway, delete its `skip:` line (or set it to an
-empty string). The CLI treats blank values as "not skipped".
+To run a skipped trail anyway, delete the device's entry from `skip:` (or set
+its reason to an empty string). The CLI treats blank values as "not skipped".
 
 ---
 
@@ -372,10 +382,10 @@ when you're authoring new tools or trails for your own target:
 | Direct scripted-tool dispatch | `test-custom-search` | Trail prompts name the tool explicitly + pass typed args. Maximum determinism short of a recording. |
 | Composition (tool calls tool) | `test-search-multi-topic` | Exercises `wikipedia_web_searchAndVerify`, which internally calls two other scripted tools. |
 | Data-driven shape | `test-search-multi-topic` | Same workflow across multiple queries — copy-paste the prompt block + change the data. |
-| Recorded deterministic replay | `test-article-shakespeare` | `blaze.yaml` + `web.trail.yaml` — replay path skips the LLM entirely. |
+| Recorded deterministic replay | `test-article-shakespeare` | `trail.yaml` with an embedded `recording:` block — replay path skips the LLM entirely. |
 | Conditional UI (banner present?) | `test-main-page-featured-article` | Exercises `dismissBannerIfPresent`, which no-ops cleanly when no banner is shown. |
 | Branch coverage on a feature flag | `test-article-short-no-refs` + `test-article-references-section` | Pair covers both branches of `verifyArticleStructure`'s `requireReferences` flag. |
-| Graceful skip | `test-search-autocomplete` | Shows the `skip: "reason..."` config field. |
+| Graceful skip | `test-search-autocomplete` | Shows the per-device `skip:` config map. |
 
 ---
 
@@ -468,7 +478,7 @@ daemon writes the SDK + `trailblaze-client.d.ts` on every restart.
 flake on long agent rounds. See **Known issues** below.
 
 **Tool calls that worked live now fail on replay** — `wikipedia_web_*`
-scripted tools currently don't dispatch from a `web.trail.yaml` recording.
+scripted tools currently don't dispatch from an embedded `web` recording.
 See **Known issues** below.
 
 ---
@@ -478,11 +488,11 @@ See **Known issues** below.
 These are tracked framework gaps. The example works around them today; the
 canonical shape will improve when these land.
 
-- **Scripted-tool replay-dispatch gap.** When a `web.trail.yaml` recording
+- **Scripted-tool replay-dispatch gap.** When an embedded `web` recording
   captures a call to `wikipedia_web_*`, the Playwright agent rejects it at
   replay as `OtherTrailblazeTool` (the dispatcher only handles `web_*`
   built-ins). Until fixed, only trails that exercise pure `web_*` tools
-  get useful recordings. That's why 23/28 trails here stay NL-only.
+  get useful recordings. That's why 25/30 trails here stay NL-only.
 - **CLI poll-timeout on long LLM rounds.** `trailblaze run …` can
   return `FAILED: Daemon unreachable after 30 consecutive poll failures`
   while the daemon is still healthy and the session is making progress.

scripts/dev-jar-cache.sh

@@ -127,12 +127,67 @@ dev_ensure_jar() {
   fi
 
   if [ "$need_build" = true ]; then
-    # Kill the daemon BEFORE building — it has stale code and must not survive
-    # into the new JAR. It auto-starts on the next command, so this is safe.
-    # We confirm the port is free before proceeding to the build.
+    # Stop the daemon BEFORE building — it has stale code and must not survive into the
+    # new JAR. It auto-starts on the next command, so this is safe. EXCEPTION: a daemon
+    # with in-flight runs is left running (see the busy-daemon guard below); it picks up
+    # the new JAR at its next restart. We confirm the port is free before building only
+    # when we actually stopped the daemon.
     local http_port="${TRAILBLAZE_PORT:-52525}"
     local pids
     pids=$(lsof -ti "tcp:$http_port" 2>/dev/null || true)
+    # NEVER stop a daemon with in-flight runs (unless TRAILBLAZE_FORCE_DAEMON_STOP is set).
+    # The daemon on this port may belong to a DIFFERENT checkout/worktree (jar staleness is
+    # per-checkout, the port is machine-global), so "stale from here" can mean "mid-run for
+    # someone else" — killing it severs that run (truncated /agentlog uploads, then 'Daemon
+    # unreachable' for the victim CLI). A busy daemon is left alone; it picks up the new JAR
+    # at its next restart. This staleness-stop only exists in the dev launcher — installed
+    # CLIs never rebuild JARs, so none of this applies outside a source checkout.
+    if [ -n "$pids" ]; then
+      local status_json curl_exit keep_reason=""
+      # --max-time bounds the WHOLE transfer, not just the connect (--connect-timeout): a
+      # wedged daemon can accept the TCP connection and then hang its /cli/status handler
+      # forever, and without --max-time this probe would block dev_ensure_jar indefinitely.
+      status_json=$(curl -s --connect-timeout 2 --max-time 5 "http://localhost:$http_port/cli/status" 2>/dev/null)
+      curl_exit=$?
+      if [ "$curl_exit" -ne 0 ]; then
+        # Something is listening (lsof found the pid) but /cli/status didn't answer within the
+        # timeout. The most likely cause is a daemon busy running a trail (saturated event
+        # loop) — precisely the daemon we must NOT kill. We can't confirm it's idle, so fail
+        # CLOSED (keep it) rather than open. A truly wedged daemon is handled by the FORCE
+        # override below. NOTE: an old daemon with no /cli/status route answers fast with 404
+        # (curl_exit 0), so it still falls through to the stop — only a genuine no-answer keeps.
+        keep_reason="it is listening but /cli/status did not respond within the timeout (it may be busy running a trail)"
+      else
+        local active_runs
+        active_runs=$(printf '%s' "$status_json" | sed -n 's/.*"activeRuns"[[:space:]]*:[[:space:]]*\([0-9][0-9]*\).*/\1/p')
+        # Empty/absent activeRuns (older daemon, or a genuinely idle one) → no keep_reason →
+        # fall through to the stop, same as before.
+        if [ -n "$active_runs" ] && [ "$active_runs" -gt 0 ]; then
+          keep_reason="stopping it would kill $active_runs in-flight run(s)"
+        fi
+      fi
+      if [ -n "$keep_reason" ]; then
+        # tr instead of ${var,,}: macOS ships bash 3.2, which lacks case conversion.
+        case "$(printf '%s' "${TRAILBLAZE_FORCE_DAEMON_STOP:-}" | tr '[:upper:]' '[:lower:]')" in
+          1|true)
+            echo "Daemon on port $http_port: $keep_reason, but TRAILBLAZE_FORCE_DAEMON_STOP is set — stopping it anyway (in-flight runs will fail)." >&2
+            ;;
+          *)
+            echo "NOT stopping the daemon on port $http_port: its code is stale, but $keep_reason." >&2
+            # Only parse run details when we actually got a status body (curl_exit 0).
+            if [ "$curl_exit" -eq 0 ] && command -v jq >/dev/null 2>&1; then
+              printf '%s' "$status_json" | jq -r '(.activeRunSummaries // [])[] | "  - " + .' >&2 2>/dev/null || true
+              local daemon_workspace
+              daemon_workspace=$(printf '%s' "$status_json" | jq -r '.workspaceAnchor // empty' 2>/dev/null || true)
+              [ -n "$daemon_workspace" ] && echo "  daemon workspace: $daemon_workspace" >&2
+            fi
+            echo "The daemon keeps serving and picks up the newly built JAR at its next (re)start." >&2
+            echo "To stop it now anyway: 'trailblaze stop', or re-run with TRAILBLAZE_FORCE_DAEMON_STOP=1." >&2
+            pids=""
+            ;;
+        esac
+      fi
+    fi
     if [ -n "$pids" ]; then
       echo "Stopping daemon (stale code)..." >&2
       # Path must match CliEndpoints.SHUTDOWN ("/cli/shutdown"). Posting to the

scripts/test_ipc_replay.sh

@@ -4,17 +4,16 @@
 # (scripts/trailblaze). The fix it guards:
 #
 #   `$(jq -r .stdout)` and `$(jq -r .stderr)` strip ALL trailing newlines
-#   from the captured bytes. Without restoring one with `printf '%s\n'`,
-#   stdout (e.g. a snapshot UI tree) visually smashes into the stderr
-#   replay that follows ("Connecting to …", "Connected: …"). The smashed
-#   form looked like:
+#   from the captured bytes. The shim now decodes all fields in one jq process
+#   with NUL delimiters, so bash preserves the exact stream endings. Without
+#   that preservation, stdout (e.g. a snapshot UI tree) visually smashed into
+#   the stderr replay that followed. The smashed form looked like:
 #
 #     [n635] "Options"Connecting to Android device (emulator-5556)...
 #
 # This test mirrors the exact JSON-unmarshal + replay codepath against a
-# synthetic CliExecResponse so a future revert of `printf '%s\n'` back to
-# `printf '%s'` (or a refactor that loses the trailing newline) trips a
-# loud, scriptable failure rather than silently re-introducing the bug.
+# synthetic CliExecResponse so a refactor that loses the trailing newline
+# trips a loud, scriptable failure rather than silently re-introducing the bug.
 #
 # Run:
 #   bash scripts/test_ipc_replay.sh
@@ -33,23 +32,40 @@ fi
 # (Console.error -> stderr) and emitted a UI tree (Console.info -> stdout).
 RESPONSE='{"stdout":"### Screen\nApp: com.android.camera2\n[i209] ImageView \"Shutter\"\n[n635] \"Options\"\n","stderr":"Connecting to Android device (emulator-5556)...\nConnected: android/emulator-5556\nEnded previous session.\n","exitCode":0,"forwarded":true}'
 
-# Mirror the shim: jq-decode into bash vars (which strips trailing \n via $()),
-# then run the EXACT replay lines from `ipc_try_forward`.
-stdout=$(printf '%s' "$RESPONSE" | jq -r '.stdout // ""')
-stderr=$(printf '%s' "$RESPONSE" | jq -r '.stderr // ""')
+# Mirror the shim: one jq decode with NUL delimiters preserves trailing newlines.
+fields=()
+while IFS= read -r -d '' field; do
+  fields+=("$field")
+done < <(printf '%s' "$RESPONSE" | jq -j '
+  if (.forwarded // false) != true then
+    error("not forwarded")
+  else
+    (.stdout // ""), "\u0000",
+    (.stderr // ""), "\u0000",
+    (
+      if (.exitCode | type) == "number" then
+        (.exitCode | floor | if . < 0 then 256 + (. % 256) elif . > 255 then . % 256 else . end)
+      else 1 end
+      | tostring
+    ), "\u0000"
+  end
+')
+[ "${#fields[@]}" -eq 3 ] || { printf 'FAIL: response did not decode into three fields\n' >&2; exit 1; }
+stdout="${fields[0]}"
+stderr="${fields[1]}"
 
 # Capture rendered output to inspect.
 combined=$(
   {
-    [ -n "$stdout" ] && printf '%s\n' "$stdout"
-    [ -n "$stderr" ] && printf '%s\n' "$stderr" >&2
+    [ -n "$stdout" ] && printf '%s' "$stdout"
+    [ -n "$stderr" ] && printf '%s' "$stderr" >&2
   } 2>&1
 )
 
 # Smashed form: the bug allowed "...Options\"Connecting" to appear on a
 # single rendered line. Assert the smashed sequence is gone.
 if printf '%s' "$combined" | grep -qE '"Options"Connecting'; then
-  printf 'FAIL: stderr replay smashes into stdout last line — printf %%s\\n regression\n' >&2
+  printf 'FAIL: stderr replay smashes into stdout last line — trailing newline regression\n' >&2
   printf 'Got (truncated):\n' >&2
   printf '%s' "$combined" | head -10 >&2
   exit 1
@@ -65,12 +81,16 @@ fi
 
 # Empty-output regression: no extra blank lines when both streams are empty.
 EMPTY='{"stdout":"","stderr":"","exitCode":0,"forwarded":true}'
-estdout=$(printf '%s' "$EMPTY" | jq -r '.stdout // ""')
-estderr=$(printf '%s' "$EMPTY" | jq -r '.stderr // ""')
+empty_fields=()
+while IFS= read -r -d '' field; do
+  empty_fields+=("$field")
+done < <(printf '%s' "$EMPTY" | jq -j '(.stdout // ""), "\u0000", (.stderr // ""), "\u0000", "0", "\u0000"')
+estdout="${empty_fields[0]}"
+estderr="${empty_fields[1]}"
 empty_out=$(
   {
-    [ -n "$estdout" ] && printf '%s\n' "$estdout"
-    [ -n "$estderr" ] && printf '%s\n' "$estderr" >&2
+    [ -n "$estdout" ] && printf '%s' "$estdout"
+    [ -n "$estderr" ] && printf '%s' "$estderr" >&2
   } 2>&1
 )
 if [ -n "$empty_out" ]; then