e2e

End-to-end tests against real applications

Each subdirectory here runs a real application's own shell-integration test suite inside a container, with the shell under test swapped in for bash. This catches interactive-shell regressions (bindings, bind -x, READLINE_*, completion, history) that unit tests can't.

cargo build
cargo xtask test e2e fzf                     # one adapter against brush
cargo xtask test e2e                         # default adapters (excludes blesh, bash-tests, bash-completion)
cargo xtask test e2e --with bash-completion  # default adapters plus an opt-in one
cargo xtask test e2e blesh                   # opt in to the slow ble.sh suite
cargo xtask test e2e --baseline fzf          # baseline against Bash
cargo xtask test e2e --shell /path/to/sh fzf # any other binary
cargo xtask test e2e fzf -- -n /ctrl_r/      # adapter-specific arguments
cargo xtask test --release e2e fzf           # use target/release/brush
cargo xtask test e2e --verbose fzf           # stream the runner's own output
cargo xtask test e2e --timeout 600 fzf      # bound each adapter (default 1800s)
cargo xtask test e2e --summary-output e2e.md # also write the run as a Markdown dashboard

A run reports one line per adapter and a summary, read back from the JUnit each runner writes:

    Starting 7 e2e adapter(s) against brush
     Results target/e2e/run-ABC123
        PASS [    1.2s] starship  3 tests
        FAIL [   13.3s] atuin     5 tests, 1 failed
        PASS [  542.2s] blesh     9 tests, 9 xfailed
------------
     Summary [ 11m35.0s] 7 adapter(s): 6 passed, 1 failed

  atuin     failed: ctrl-r search runs the selected command
            -> target/e2e/run-ABC123/atuin/log.txt

Results land in target/e2e/run-<id>/<app>/: log.txt (full runner output), junit/*.xml (JUnit, for CI upload), and any adapter-specific diagnostic logs (such as PTY transcripts). Each invocation creates and prints a fresh run directory, retaining previous runs and leaving existing files untouched, including when using --results-dir. Missing, malformed, or empty JUnit reports fail the adapter, as does an adapter that runs longer than --timeout (default 1800s): its container is killed, so a hung shell cannot hang the run. The runner's own output goes to log.txt rather than the terminal; --verbose streams it instead. Multi-adapter runs continue after failures and report them together at the end. On a terminal, statuses are green for clean passes, yellow for passes with expected failures, and red for failures. Redirected output, TERM=dumb, and a nonempty NO_COLOR disable color.

They need Docker and a built shell binary. CI runs the default selection on every PR (the e2e-tests job in ci.yaml, against the release binary the build job produced) and posts the --summary-output dashboard to the PR comment, alongside the coverage and benchmark reports, with each failing adapter's log.txt in the job log and every run directory in the e2e-logs artifact. Because the xfail-list.txt files are the committed baseline, a PR that fixes or breaks a test shows it in its own diff; no run against main is needed. blesh is excluded from the default selection because compatibility failures and timeouts make it take several minutes against brush. Run it explicitly with cargo xtask test e2e blesh, or select one file with cargo xtask test e2e blesh -- util. Its per-file timeout remains 180 seconds. bash-tests is likewise opt-in, until it has shown itself reliable. Neither runs in CI. bash-completion is opt-in because its ~2000 tests take several minutes; CI adds it with --with bash-completion, and it is report-only (below), so it cannot fail the job on test results.

Adapter contract

An adapter is a directory e2e/<app>/ containing a Dockerfile (build context is e2e/, so it can COPY shim /e2e/bin) whose entrypoint:

1. Receives the shell via $SHELL_UNDER_TEST, an absolute path to a binary bind-mounted into the container (/shell/<name>). It's built on the host, so the image's glibc must be at least the host's. Most apps hard-code bash, so the shared shim/bash exec's $SHELL_UNDER_TEST; put /e2e/bin first on PATH. When the variable is unset the shim runs the real bash, which gives the baseline run. If the suite has its own knob for which shell to test (atuin's ATUIN_TEST_BASH), set that instead and skip the shim; a shim on PATH also captures test tooling written in bash (tmux, say), which must not run under the shell under test. A suite that takes a shell path has nowhere to put a command-line flag; turn the behavior on through brush's config file ($HOME/.config/brush/config.toml, and HOME is /tmp in the container) instead, which bash ignores and so leaves the baseline run alone.
2. Writes results to /results: log.txt plus JUnit XML under junit/. Write whatever else helps debugging there too.
3. Exits non-zero when any test fails. Entrypoints pipe the runner through tee, so exit ${PIPESTATUS[0]} rather than $?, which is tee's status. Known failures are not the adapter's problem: the runner reads them from xfail-list.txt (below) and decides the result from the JUnit, so an adapter that exits non-zero purely because of expected failures still passes.

The container runs as the host user with HOME=/tmp, so chmod anything the tests write into.

Reproducing a failure

The dashboard names each failing test with the command that runs it alone, built from the adapter's select-args.txt: the adapter arguments that select one test, with {test} standing for its name (fzf: -n /{test}/). Adapters without the file are assumed to take pytest's -k {test}, since that is what every adapter's own tests use.

Tracking known failures

Every dashboard row shows the pass rate: passed tests over those that ran (skips excluded, xfails included).

An adapter containing a report-only file (a comment saying why) is tracked by that rate alone: its failing tests are listed, collapsed, and its row is marked 📊, but they do not fail the adapter; nor does the non-zero exit they cause. A run that produced no results still fails it: an image that does not build, a timeout, or missing JUnit. Such a suite needs no xfail-list.txt; the runner's own expected failures (@pytest.mark.xfail) are counted as xfailed as usual. Everything below applies to the other adapters.

Two lists per adapter, both optional, both #-commented, one test name per line.

xfail-list.txt is the default home for anything that fails today. The test still runs; the runner matches it against the JUnit and counts it as xfailed instead of a failure. Crucially it also fails the run when such a test passes ("passed unexpectedly") or when an entry matches no test that ran ("stale xfail entry" -- renamed, removed, or moved to skip-list.txt), so a gap that closes is reported rather than absorbed, and the list cannot rot. Stale detection needs the whole suite, so it is skipped when adapter arguments select a subset. Because the matching happens over JUnit, one mechanism covers every runner. Name a test by the name its JUnit carries, or as classname::name. These expectations apply only to brush runs; --baseline and --shell runs use no brush expectations and fail on any reported test failure.

skip-list.txt is for tests that must not run: they hang, or they cannot exercise the shell at all (nvm has one with a /bin/zsh shebang). This is the exception -- prefer xfail-list.txt, because a skipped test reports nothing when it starts working -- and listing one in both files fails the run as a stale xfail entry. The runner reports skipped tests by name, so a suite that starts skipping something itself does not pass unnoticed.

Applying the list is mostly handled for you: lib/run_pytest.sh exports the file as $E2E_SKIP_LIST and loads lib/e2e_skip.py, a pytest plugin that skips any test named in it. Adapters that must decide before collection (blesh) or at image build time (nvm, which takes the execute bit off) parse $E2E_SKIP_LIST themselves; fzf turns it into a minitest --exclude regex, since its runner is upstream's.

Every entry in either list needs a comment saying why, with an issue link where one exists.

Shared adapter code

Every adapter's own tests are pytest, so lib/ (copied to /e2e/lib, on PYTHONPATH) carries what they would otherwise each reinvent:

run_pytest.sh	the pytest half of the contract: JUnit, log, skip list, exit status. Source it and call e2e_run_pytest <report name>
rcfile.py	write_rc(): a known prompt and no HISTFILE, then the app's own integration lines
pty_shell.py	drives a shell over a raw PTY (pexpect) and matches its output stream. The default
tmux_shell.py	drives a shell inside tmux (via libtmux) and matches the rendered screen. For full-screen TUIs, whose redraws a byte stream cannot answer questions about
e2e_skip.py	the skip-list pytest plugin, plus parse() for adapters that must skip before collection

Rules

1. Never copy upstream files into this tree. Clone the application at build time, pinned to a commit (not a branch).
2. The Dockerfile is our own environment recipe. Deriving its package list from upstream's Dockerfile or CI config is fine; say so in a comment, with the source and license.
3. The entrypoint adapts upstream's runner to the results contract. If the runner can't emit JUnit itself, add a reporter there (fzf: minitest-ci).
4. Not every app has a suite to borrow. When upstream only tests its own code (atuin, say), the adapter brings its own tests: a small terminal-driven suite that installs the app's hooks into the shell under test and checks the observable behavior. The contract is the same; only the author differs.

Adapters

app	notes
fzf	upstream test/test_shell_integration.rb, TestBash only; minitest + tmux, JUnit via minitest-ci
atuin	our own suite in atuin/tests/ (pytest + tmux), written to be upstreamable. atuin's bash integration is bash-preexec, so the adapter turns on brush's zsh-hooks through a config file to exercise the native hooks rather than the DEBUG-trap emulation
bash-tests	bash's own regression suite (tests/run-*), from a bash-5.3 build that also provides its helper binaries and the --baseline shell; one pytest case per run-* script, which is sourced verbatim, on a terminal, with diff replaced by a function that captures the output for byte-exact comparison against the .right file. PLATFORM_DIFFS in tests/test_bash.py excuses the few differences bash itself shows on Linux. Failures report how many expected lines match, and log.txt ends with the total across the run. Args go to pytest: -- -k alias, -- --subset minimal for the scripts run-minimal runs, -- --script-timeout 60 (default 30s per script)
bash-completion	upstream's test/t pytest suite (tag 2.18.0), in upstream's own rolling fedoradev test image, which has the commands under test installed; built with make as upstream's CI does, and run under xvfb-run. Report-only: upstream's xfail markers are its only expectations. Args go to pytest: -- -k test_ls
blesh	upstream ble.sh --test, one pytest case per test section with a process-group timeout (BLESH_TEST_TIMEOUT, default 180s); sections are read from ble.sh's own build output, so a section added upstream runs rather than being missed; full output and leftover per-section artifacts are retained; args select sections: cargo xtask test e2e blesh -- util. Every section is currently expected to fail: ble.sh gets far enough to emit no section summary at all
mise	selected upstream bash activation tests (pytest), run directly against the shell under test
nvm	upstream test/fast/Listing versions suite (urchin) plus interactive regression coverage for #1173
starship	our own suite in starship/tests/ (pytest + direct PTY)
zoxide	our own suite in zoxide/tests/ (pytest + direct PTY)