fix(test): stop the gemini-cli live smoke flaking on the default timeout - #318
Merged
Conversation
The bundled gemini-cli smoke test failed roughly 2 runs in 3 on the full suite, while the CLI itself was fine. Timed directly, the spawn costs 3.3-4.2s: bun has to parse and run a large JS bundle before --version prints. That fits inside bun's 5s default only on an idle machine, so under the suite's parallel load it tipped over and reported a timeout that looked like a broken provider. Gives the test an explicit 30s ceiling, matching the convention already used for slow tests in local-mode-server.test.ts and store-lock.test.ts. It stays a real smoke test — it still spawns the actual binary and asserts exit 0 plus a semver — it just no longer races the default. Verified: the CLI returns 0.50.0 in 3.3-4.2s standalone, and the suite now passes this test across repeated full runs. Note for anyone chasing suite flakes: a SECOND, unrelated flake remains and is NOT addressed here. About 1 run in 4, bun drops src/tests/sanitize-terminal.test.ts entirely with "Cannot call describe() after the test run has completed" — the pass count falls by exactly its 11 tests. Confirmed pre-existing by re-running with this change reverted. It is a runner-level module-load race rather than a product bug, but it silently skips the terminal-escape sanitisation tests, so it deserves its own issue. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
akhiljavelin
approved these changes
Sep 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
provider-acp.test.ts→ "live smoke: the bundled gemini-cli runs under the daemon runtime" failed ~2 runs in 3 on the full suite, always as a 5s timeout — which reads like a broken provider.The provider is fine. Timed directly:
Cold-starting the bundled CLI costs 3.3–4.2s — bun parses and runs a large JS bundle before
--versionprints. That fits inside bun's 5s default only on an idle machine; under the suite's parallel load it tips over.Fix
An explicit
}, 30_000)ceiling, matching the convention already used for slow tests inlocal-mode-server.test.ts:263andstore-lock.test.ts:129.It remains a real smoke test — still spawns the actual binary, still asserts exit 0 and a semver. It just no longer races the default timeout.
Verification
Full suite run repeatedly before and after. This test no longer fails.
Worth a separate issue. About 1 run in 4, bun drops
src/tests/sanitize-terminal.test.tsentirely:The pass count falls from 2429 to 2418 — exactly the 11 tests in that file.
I initially suspected this change caused it (2418 only appeared afterwards), so I reverted this fix and re-ran: the baseline produced 2418 on its own. Pre-existing, not introduced here.
It looks like a runner-level module-load race rather than a product bug, but the consequence is that the terminal-escape sanitisation tests silently do not run — the ones covering OSC 52 clipboard writes and cursor/screen control in untrusted model output. That is a bad thing to lose quietly.
Options, none taken here: file and live with it; try a bun upgrade (currently 1.3.11); or dig into the race.
🤖 Generated with Claude Code