kane-cli runs tests in two modes: an interactive TUI for development and exploration, and a headless CLI for scripts and CI. Both modes execute the same agent against the same browser; only the surface differs.
An objective is the natural-language instruction the agent follows. The clearer the objective, the more reliably the agent completes it.
What makes a good objective:
- Specific. Name the site, the action, and the field values where they matter.
- Action-oriented. Lead with a verb: search, open, fill, click, verify.
- Ends in a check of the result. State what "done" looks like as a claim about the page, so the agent knows when to stop and a replay has something to verify.
Two habits make an objective reliable enough to save and replay.
End every flow in a check of the result. Close with an assertion about the resulting page state — verify the cart shows 1 item — not a bare submit or confirm the dialog. Without a closing check, "done" only means the agent believed it finished; with one, the run passes or fails on the real end state. Name the outcome, never the control that triggered it: verify the Submit button is visible fails exactly when the action worked, because the button disappears on success. If an objective has several phases (log in, then search, then check out), each phase ends in its own check. Make the closing check something the agent can see on the screen the last action leaves it on, with no further click or navigation. If the proof lives on another page, get there with an explicit action first: place the order, open Order History, then verify the newest order shows "Processing", not verify the order succeeded by checking Order History. A check that has to find its own way to the evidence is an action hiding inside an assertion, and the route it improvises can differ from run to run.
Say the goal, not the clicks. Phrase actions as goals — Log in as {{tester}}, Complete checkout with the saved card — so a run absorbs a reordered form or a new popup instead of breaking on it. Keep exact values such as credentials, prices, and URLs literal or in {{variables}}. When a test needs fresh data on every run — a new sign-up email, a unique name, today's date — use a smart variable such as {{smart.random_email}} or {{smart.current_date}} instead of a fixed value that the second run would collide with. For a step that only sometimes appears, such as a cookie banner, write it as a condition (if a cookie banner appears, dismiss it) rather than assuming it away.
Search for "noise-cancelling headphones" on amazon.com and confirm at least
one result appears with a price under $200.
Fill the registration form at example.com/signup with name "Test User",
email {{smart.random_email}}, and password {{smart.random_string_8}}. Submit
and verify the confirmation page loads.
Open https://app.example.com, log in with the credentials in {{tester}},
navigate to Settings > Billing, and verify the current plan is "Pro".
Open https://booking.example.com, search for a room with check-in
{{smart.next_day}}, and verify at least one room is listed for that date.
Open https://app.example.com, set a cookie named session with value
{{auth_token}}, reload, and verify the dashboard loads. Then click the
"Copy invite link" button and verify the clipboard contains "/invite/".
Objectives can also set, delete, and clear cookies, localStorage, and the test clipboard directly — see Browser State Actions.
Objectives can also make API calls directly — POST/GET a URL or paste a curl,
save the response, and assert on it — see API Calls.
| Objective | |
|---|---|
| Bad | Test the login page. |
| Better | Open https://app.example.com/login, log in as {{tester}}, and verify the dashboard URL contains /home. |
The bad version leaves "test" undefined and gives the agent no end state. The better version names the URL, the credentials, and the assertion that closes the loop.
Launch the TUI by running kane-cli --tui. The TUI is the right surface when you are exploring objectives, debugging failures, or working through a multi-run flow that should share browser state.
On launch, kane-cli runs a short boot sequence (auth check, environment resolution, mascot animation), then drops into the main menu. The top-level entries are:
| Entry | Purpose |
|---|---|
| Run | Start a run or adjust per-run options. |
| Auth | Login, logout, switch profile, view identity, check credit balance. |
| Config | View and change settings (mode, project, folder, Chrome profile, window size). |
| Exit | Graceful shutdown (uploads the session if applicable). |
Use the arrow keys to navigate, Enter to select, and Esc to back out of a submenu.
Selecting Run > Start Run switches the TUI into chat mode. Type your objective at the prompt and press Enter. The agent begins streaming steps into the scrollback as it works: each step shows the action taken, a short rationale, and a status icon. When the run finishes, a result summary block appears.
Subsequent runs in the same TUI session reuse the same browser, so you can iterate on objectives without re-logging in or re-navigating.
Typing / in chat mode opens an autocomplete palette. Continue typing to filter, use the arrow keys to select, and press Enter to insert the command. The full command list:
| Command | Args | Description |
|---|---|---|
/run |
"objective" |
Execute a test run. |
/login |
[--profile name] |
OAuth login. |
/logout |
[--profile name] |
Logout and revoke tokens. |
/whoami |
[--profile name] |
Show profile info. |
/balance |
Show credit balance. | |
/profiles |
list|switch|delete |
Manage profiles. |
/config |
show|set-window|set-url|set-mode|chrome-profile|project|folder |
Manage configuration. |
/mobile |
Switch the session to a device: an emulator, a simulator, or a real Android/iOS device. | |
/desktop |
Switch the session back to Desktop (Chrome). | |
/doctor |
Check mobile tooling & devices. | |
/new |
Start a fresh session (uploads the current session and seals its evidence pack first). | |
/summary |
[index] |
View detailed run summaries. |
/cancel |
Abort the current run. | |
/help |
Show the command reference. | |
/clear |
Clear chat history. | |
/exit |
Quit kane-cli. |
You can also send a bare line of text without a leading /. It is treated as the objective for /run.
Press Ctrl+R in the input prompt to open reverse history search across past inputs in this and previous sessions. Type to filter, use the arrow keys to move between matches, Enter to accept, and Esc to dismiss.
The same prompt also offers ghost-text completion: if your current input is a prefix of a recent entry or a slash command, the rest is shown dimmed and Tab accepts it.
A two-row status bar sits at the bottom of the TUI. It shows:
| Indicator | Meaning |
|---|---|
| Model | The model in use (default v16-alpha). |
| Session | Last six characters of the current session ID. |
| Auth dot | Green when authenticated, red when not logged in. |
| Profile | Active profile name (or no profile). |
| Environment | prod (green) or a yellow stage warning. |
| Runs | Number of runs completed in this session. |
| Hint line | Context-aware shortcuts (chat: ctrl+c cancel/exit; menu: ↑↓ navigate ↵ select esc back). |
| Action | Shortcut |
|---|---|
| Cancel the current run | /cancel, or Ctrl+C once during a run. |
| Exit the TUI | /exit, or Ctrl+C twice in quick succession. |
| Force exit during shutdown upload | Ctrl+C twice while exit is in progress. |
A graceful /exit runs the upload pipeline (if applicable), seals the session's evidence pack, offers to open it in the browser viewer, and prints any final links to your terminal scrollback before the process ends.
Every run launched from the same TUI invocation shares one Chrome instance and one session directory. Cookies, login state, and tabs persist across runs in that session, so an early run can log in and a later run can land mid-application without re-authenticating. Starting a fresh session from inside the TUI is done with /new, which uploads the current session and then resets state.
If the agent needs information mid-run (for example, a one-time code or a clarifying choice), it pauses and asks at the input prompt. Type your answer and press Enter; the agent resumes from where it left off. Use Ctrl+C to cancel the run instead of answering.
kane-cli run is the headless surface. It is the right choice for CI jobs, scripted invocations, and any environment that has no interactive TTY. Progress streams to stderr; the final JSON result is written to stdout.
A minimal run:
kane-cli run "Search for 'noise-cancelling headphones' on amazon.com"A longer invocation with options:
kane-cli run "Log in as {{tester}} and verify the dashboard loads" \
--headless \
--max-steps 40 \
--timeout 300 \
--variables-file ./vars.jsonThe customer-facing flags accepted by kane-cli run:
| Flag | Description | Default |
|---|---|---|
--headless |
Run Chrome in headless mode. | Off |
--max-steps <n> |
Maximum agent steps. | 50 |
--timeout <seconds> |
Kill the run after N seconds. | None |
--url <url> |
Start URL for the run. Overrides the configured default_url; bare domains are normalized to https://. See Default start URL. |
Config default_url |
--allow-missing-url |
Non-TTY only: proceed from the browser's current page instead of failing when no start URL resolves (a provided --url is still used). |
Off |
--cdp-endpoint <url> |
Connect to an existing Chrome via CDP. | None |
--ws-endpoint <url> |
Connect to a Playwright WebSocket endpoint (e.g. TestmuAI wss://). The run still executes on this machine and drives that browser; to run a whole suite on the grid instead, see testrun run --remote. |
None |
--global-context <file> |
Override the global context Markdown file. | ~/.testmuai/kaneai/global-memory.md |
--local-context <file> |
Override the local context Markdown file. | <cwd>/.testmuai/context.md |
--variables <json> |
Inline variables JSON. | None |
--variables-file <path> |
Load variables from a JSON file. | None |
--session-context <json> |
Prior runs context JSON. | None |
--username <user> |
Basic auth username (skip OAuth). | None |
--access-key <key> |
Basic auth access key (skip OAuth). | None |
--mode <name> |
Run mode: action (strict) or testing (lenient). |
Config value, otherwise testing |
--bug-detection <mode> |
Detect product bugs while authoring: off, stop (halt the run on a confirmed bug), or continue (record it and keep going). Overrides config set-bug-detection. See Configuration. |
Config value, otherwise off |
--agent |
Plain NDJSON output, no colors or UI. | Off |
--code-export |
Generate code export after upload. | config (true by default) |
--code-language <lang> |
Code export language (currently python). |
python |
--skip-code-validation |
Skip post-codegen worker-side validation. | On |
--no-skip-code-validation |
Force post-codegen worker-side validation. | Off |
kane-cli checks every {{name}} in the objective before the run starts. A name with no value is a warning: the run goes ahead and types the name as written, unless a step sets it first. On a TTY the warning is the receipt shown in Variables and context; with --agent it is a single typed event before the first progress frame:
{"type":"warning","code":"unresolved_variables","message":"2 variable(s) have no value — typed as written unless a step sets them first","suggested_file":".testmuai/variables/variables.json","variables":[{"name":"checkout_url","reason":"not_declared","used_by":[{"file":"objective","step":1}]},{"name":"login_password","reason":"value_missing","file":".testmuai/variables/variables.json","used_by":[{"file":"objective","step":1}]}]}reason is value_missing (the key exists in file, with no value), not_declared (the key is in no file; suggested_file is where to add it), or not_a_dataset_column (a ${x} that is no column of the dataset row the run was given). The warning does not change the exit code. An explicit {{global.*}} reference (resolved from Test Manager at run time), names an earlier step stores, and the {{smart.*}} / {{environment.*}} / {{secrets.*}} / {{totp.*}} namespaces are never checked.
For variables and context file behavior, see ./variables-and-context.md. For code export and the run mode toggle, see ./configuration.md.
By default a run targets the desktop browser (Chrome), so every example above is unchanged. From any machine you can instead point a run at a real device in the TestMu AI cloud: android or ios. On macOS Apple Silicon you can also use a virtual mobile device on this machine: an emulator (a virtual Android device) or a simulator (a virtual iOS device). Every mobile run needs an app under test. (From any other machine, run a saved emulator/simulator suite on the cloud grid with testrun run --remote.)
# desktop (default): nothing changes for web runs
kane-cli run "Search for 'noise-cancelling headphones' on amazon.com"
# emulator (Android): install an .apk build and run against it
kane-cli run "Add the first item to the cart" --target emulator --app ./builds/app-debug.apk
# simulator (iOS): install a .zip build and run against it
kane-cli run "Sign in and open the account tab" --target simulator --app ./builds/MyApp.zip
# real device (any machine): the hub picks a free device unless you name one
kane-cli run "Sign in and open the account tab" --target ios --app ./builds/MyApp.ipa
kane-cli run "Add the first item to the cart" --target android --device-name "Pixel.*" --app ./builds/app-release.apkThe mobile run flags:
--target desktop|emulator|simulator|android|ios: which target to run against. Defaults to the saved session target, otherwisedesktop.--device-name <name>+--os-version <version>: pick a device askane-cli devices list --target emulator|simulatorprints it (on an emulator or simulator a name needs a version; a version alone matches any device on it; on a real device either one alone is enough, a name can be a pattern, and with neither the hub picks — list real devices withkane-cli devices list --target android|ios --remote). In the TUI/TTY, omitting them opens a one-time picker and the choice is saved; in non-interactive runs a device must already be set (via the flags orkane-cli config set-device-name/set-os-version) or the run exits with the fix spelled out.--app <path|APPid>: the app under test, required for every mobile run. Pass a build (emulator:.apk, simulator:.zip, android:.apkor.aab, ios:.ipa) or an uploaded app id (APPfollowed by six or more digits;kane-cli apps list --target <kind>lists yours). On thedesktoptarget, the device flags and--appare ignored.--no-stream: on a real device, don't open its live view for this run (it opens in its own Chrome window by default at a terminal; never for agents).
In the interactive TUI, a first run offers a chooser of the targets this machine can run (a machine that can't run emulators or simulators sees Desktop / Android / iOS), and you can switch targets with /mobile and /desktop before the first dispatch; afterwards use /new before switching. Run /doctor to check mobile tooling and devices.
For setup (Xcode or Android Studio, kane-cli login, and kane-cli doctor --target emulator|simulator --install) and the app formats each target accepts, see Mobile testing.
| Stream | Contents |
|---|---|
| stderr | Live progress (banner, step tree, result box, links, upload progress, feedback prompt). |
| stdout | The final JSON run_end payload, including the share URL when an upload succeeds. |
This separation lets you capture each independently:
kane-cli run "..." > result.json 2> progress.logIn CI, redirect stdout to a file your job can parse and let stderr stream to the build log so engineers can read the run in real time.
When stdin is not a TTY, kane-cli automatically switches to plain NDJSON mode (the same as --agent). Each line on stdout is one JSON event terminated by a newline.
| Code | Meaning |
|---|---|
0 |
Run passed. |
1 |
Run failed (agent reached an unrecoverable failure or upload failed). |
2 |
Error (auth, TMS credential exchange, configuration, Chrome, unknown subcommand, or unhandled exception). |
3 |
Run was cancelled or hit the --timeout. |
Use these codes to gate downstream CI steps.
When a run finishes, kane-cli prints a result summary to the terminal:
| Field | Meaning |
|---|---|
| Status | PASSED (green check) or FAILED (red cross). |
| Steps | Total step count, with a (N passed, M failed) breakdown when there were failures. |
| Duration | Wall-clock time in seconds (or minutes and seconds for longer runs). |
| Credits | Credits consumed, when reported. |
| Summary | Bullet-point summary of what the agent did. |
| Reason | Failure reason (failed runs only). |
Below the summary, kane-cli prints any of the following links that apply to the run:
| Label | Points to |
|---|---|
ShareLink |
A shareable session URL on TestmuAI TMS. |
TestCase |
The test case detail page in TestmuAI TMS. |
CodeExport |
The local directory containing the generated code (when code export is enabled). |
Modern terminals render these as clickable hyperlinks. For details on what each link leads to, see ./test-manager-integration.md.
The run is also captured as a sealed evidence pack — screenshots, per-step console/network logs, and failure records. In a terminal, kane-cli offers to open it in the browser viewer; in agent or non-interactive runs it prints a one-line evidence: view locally with … hint to stderr instead.
After the result and links print, kane-cli prompts you to rate the session with thumbs up or thumbs down. Use the left and right arrow keys to choose, Enter to submit, or Esc to skip. The rating is sent to TestmuAI TMS for the active test case; see ./test-manager-integration.md for what is recorded.
The run_end parsing strategy applies to one-shot run only. For testmd run, collect test_md_done.overall_status, duration_s, session_id, and optional share_url; embedded run_end events can finish individual steps. Local suites emit testrun_done; dispatched remote suites then emit remote_done (retain status, exit, sessions_path). generate emits generate_done. Assurance conversational agent streams end in done; review/read verbs have their own contracts. Always check process exit too: early refusal, invalid plan or dry-run can exit without the normal completion event.
Progress is for live display: count only done/failed completions, retaining child and execution context when step indices repeat.
In the interactive TUI, /mobile and /desktop can switch targets before the first dispatch. The target locks after dispatch; use /new to start a new session before switching.
run and testmd run support --assertion-mode dom|visual (default dom with vision fallback) and --final-validation on|off (default off). Persist with config set-assertion-mode and config set-final-validation. Final validation controls the synthesized cp_final checkpoint independently of action/testing mode; keep explicit terminal assertions in objectives.
run --analyzer-only --condition "<condition>" checks the current desktop browser page without an objective, action steps or saved test. Repeat --condition for multiple checks. Use --agent or non-TTY input. Results contain condition_results: boolean[]; exit 0 means all conditions were judged, not that all are true. Exit 1 means a result was missing, and 3 means cancelled. This mode rejects mobile target/app/device options and code-export/name options.
Experimental run --network-ws and run --network-sse enable WebSocket and SSE capture; both default off. Persist with config set-network-ws on|off and config set-network-sse on|off. SSE capture is Chromium-only. Do not copy these run-only flags onto testmd or testrun commands.
Code export defaults to enabled, subject to saved configuration, and supports python (default) or javascript. run/testmd run use --code-language; testmd export uses --language. Upload eligibility is independent of action/testing mode.