Skip to content

feat: update e2e tests to properly work with async mode#840

Open
svarlamov wants to merge 3 commits intomainfrom
devin/1774639930-e2e-async-mode
Open

feat: update e2e tests to properly work with async mode#840
svarlamov wants to merge 3 commits intomainfrom
devin/1774639930-e2e-async-mode

Conversation

@svarlamov
Copy link
Copy Markdown
Member

@svarlamov svarlamov commented Mar 27, 2026

Summary

Enables async mode (GIT_AI_ASYNC_MODE: "true") across all three CI workflows by starting a per-test/per-job daemon and configuring the wrapper to poll for authorship notes.

BATS e2e tests (tests/e2e/user-scenarios.bats):

  • setup() creates an isolated TEST_DAEMON_HOME (overriding $HOME) with async-mode config, starts the daemon via git-ai bg run, and polls for Unix domain sockets to appear
  • git() and git-ai() shell wrappers now pass daemon socket paths, trace2 config, force-TTY, and a 30s post-commit timeout
  • teardown() sends bg shutdown, waits up to 2s, then kill -9 as fallback, restores $HOME, and cleans up both temp dirs
  • New wait_for_note() helper polls for authorship notes (up to 20s) after non-commit operations like rebase where the wrapper doesn't auto-poll
  • init.defaultBranch main set in global gitconfig before git init (required because isolated HOME loses the system default)

Reusable daemon helper scripts (scripts/nightly/):

  • start-async-daemon.sh: creates isolated daemon home with async-mode config, probes /usr/bin/git and /usr/local/bin/git to avoid accidentally using the git-ai proxy symlink, starts daemon in background, polls for sockets (10s), exports all env vars to GITHUB_ENV for subsequent workflow steps
  • stop-async-daemon.sh: graceful bg shutdown via control socket, 2s wait, force-kill fallback, daemon home cleanup

CI workflow changes:

  • e2e-tests.yml: GIT_AI_ASYNC_MODE flipped to "true"
  • install-scripts-local.yml: GIT_AI_ASYNC_MODE flipped to "true" with Start/Stop daemon steps around Unix test steps; Windows job overrides to "false" (no Unix-domain socket support)
  • nightly-agent-integration.yml: GIT_AI_ASYNC_MODE flipped to "true" with Start/Stop daemon steps in both Tier 1 (hook wiring + synthetic checkpoint) and Tier 2 (live agent + attribution verification) jobs

Closes #839

Review & Testing Checklist for Human

  • Daemon process survival across GitHub Actions steps: The daemon is started as a background process via source start-async-daemon.sh in one step and must stay alive for subsequent steps. The PID is persisted via GITHUB_ENV for the stop step. Verify background processes are not killed between steps on Ubuntu and macOS runners.
  • Env var propagation to nightly test scripts: The existing scripts (test-synthetic-checkpoint.sh, verify-synthetic-attribution.sh, etc.) are not modified — they depend on GIT_AI_TEST_FORCE_TTY and GIT_AI_POST_COMMIT_TIMEOUT_MS being set in the environment via GITHUB_ENV. Confirm these are correctly inherited by the bash subprocesses that run the scripts.
  • Real git path detection on macOS: start-async-daemon.sh probes /usr/bin/git then /usr/local/bin/git to avoid the proxy symlink. Verify this finds the correct git binary on macos-latest runners (Xcode CLT vs Homebrew).
  • git_hooks_enabled: false in daemon config doesn't prevent authorship note generation — the daemon should use trace2 events rather than git hooks, but verify this is the correct code path in async mode.
  • Recommended test plan: Trigger the install-scripts-local and nightly-agent-integration workflows manually on this branch (apply integration label to the PR) and verify all jobs pass. For install-scripts-local, check both Ubuntu and macOS Unix jobs succeed and Windows jobs still pass with sync mode.

Notes

  • No Rust source changes — only BATS test infrastructure, CI workflows, and new shell helper scripts
  • task lint and task format:check pass
  • Windows jobs in install-scripts-local.yml remain on sync mode (GIT_AI_ASYNC_MODE: "false") because Unix-domain sockets are unavailable
  • Commit operations rely on the wrapper's built-in post-commit polling (GIT_AI_TEST_FORCE_TTY=1 + GIT_AI_POST_COMMIT_TIMEOUT_MS=30000); wait_for_note() is only needed after rebase/squash operations in the BATS tests
  • CI timeouts: daemon start 10s, post-commit polling 30s, wait_for_note 20s — confirm these are sufficient under load

Link to Devin session: https://app.devin.ai/sessions/85ba6a1442ec430b807e35f9d66c23dc
Requested by: @svarlamov


Open with Devin

- Modified setup() to start a per-test daemon with isolated HOME,
  config, and Unix domain sockets for async processing
- Modified teardown() to gracefully shut down daemon and clean up
- Wrapped git() and git-ai() shell helpers to pass daemon env vars
  (socket paths, trace2, force-TTY, timeout)
- Set init.defaultBranch=main in global gitconfig before git init
- Added wait_for_note() helper for polling authorship notes after
  non-commit operations (rebase, cherry-pick, merge --squash)
- Inserted wait_for_note calls after all rebase operations in tests
- Updated CI workflows to use GIT_AI_ASYNC_MODE=true

Closes #839

Co-Authored-By: Sasha Varlamov <sasha@sashavarlamov.com>
@devin-ai-integration
Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@CLAassistant
Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration bot and others added 2 commits March 27, 2026 19:54
These workflows run shell scripts that don't start a daemon or set
force-TTY/timeout env vars, so async mode would race. Only the BATS
e2e tests (e2e-tests.yml) have proper daemon infrastructure.

Co-Authored-By: Sasha Varlamov <sasha@sashavarlamov.com>
Add reusable daemon helper scripts (start-async-daemon.sh,
stop-async-daemon.sh) that handle daemon lifecycle in CI:
- Create isolated daemon home with async_mode config
- Probe /usr/bin/git to avoid pointing at the proxy symlink
- Export env vars (GIT_AI_TEST_FORCE_TTY, POST_COMMIT_TIMEOUT_MS,
  socket paths) to GITHUB_ENV for subsequent steps
- Graceful shutdown with force-kill fallback

Update install-scripts-local.yml:
- Flip GIT_AI_ASYNC_MODE to true at workflow level
- Add Start/Stop async daemon steps around Unix test steps
- Override GIT_AI_ASYNC_MODE to false for Windows job (no
  Unix-domain socket support)

Update nightly-agent-integration.yml:
- Flip GIT_AI_ASYNC_MODE to true at workflow level
- Add Start/Stop async daemon steps in both Tier 1 and Tier 2 jobs

Co-Authored-By: Sasha Varlamov <sasha@sashavarlamov.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrate nightly CI jobs to use async mode instead of wrapper

2 participants