Skip to content

Simplify test suites without weakening coverage - #1143

Merged
robinebers merged 1 commit into
mainfrom
codex/simplify-test-suites
Aug 24, 2026
Merged

Simplify test suites without weakening coverage#1143
robinebers merged 1 commit into
mainfrom
codex/simplify-test-suites

Conversation

@robinebers

@robinebers robinebers commented Aug 24, 2026

Copy link
Copy Markdown
Owner

TL;DR

Audit all 139 Swift test/support files and simplify 53 test files without weakening meaningful regression coverage. Reduce repository test/support code from 27,885 to 25,825 lines (2,060 fewer; 7.4%).

What was happening

  • Provider, layout, widget, pricing, cache, and notification suites repeated equivalent cases and assertions across multiple files.
  • Large JSON and filesystem fixtures obscured the behavior under test, while catalog snapshots and private implementation checks added maintenance cost.
  • Coverage was sometimes duplicated in provider-specific tests even though shared mappers already exercised the same observable behavior.

What this changes

  • Parameterize equivalent auth, error, pricing, threshold, reset, accessibility, and rendering scenarios while preserving genuinely distinct failure modes.
  • Minimize provider payloads, scanner fixtures, persistence setup, and notification scaffolding across 53 test files.
  • Preserve account ownership, credential permissions, HTTP 401/403/5xx classification, cache freshness and concurrency, cleanup and cancellation, alias canonicalization, and scanner-to-widget integration.
  • Keep production code and documentation unchanged.

Heads-up

  • Three existing live/parity tests remain skipped unless their opt-in environment variables and credentials are available.

Tests

  • Affected suites: 685 XCTest cases across 70 classes, 0 failures, 3 expected skips.
  • swift test: 1,190 app XCTest cases, 4 CLI XCTest cases, and 3 Swift Testing cases; 0 failures, 3 expected skips.
  • swift build: passed.
  • git diff --check: passed.

Note

Low Risk
Tests-only refactor: no app behavior, auth, or data-handling changes. Residual risk is slightly less isolated cases if a merged loop misses a previously separate assertion.

Overview
Shrinks and consolidates XCTest suites (~2k fewer test lines) with no production-code changes. Equivalent auth, mapping, scanner, layout, cache, and formatter cases are folded into table-driven loops; shared helpers (configuredAuthStore, makePersistence, makeSync, limitResource) replace copy-pasted setup.

Fixtures and assertions are tightened (smaller JSON/CSV payloads, map equality instead of per-field checks). A few overlapping tests are merged (e.g. unknown vs synthetic Claude models, Copilot extra-usage/org-seat variants, Codex reset-credit fallbacks) while keeping distinct failure modes. Some concurrency fixtures use fewer files; pin-cap tests use two providers instead of four.

Reviewed by Cursor Bugbot for commit ecffbc6. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions github-actions Bot added the tests label Aug 24, 2026
@robinebers robinebers added the gate-passed Passed the automated PR gatekeeper checks (gatekeeper skips it on later runs) label Aug 24, 2026 — with Cursor
@robinebers
robinebers merged commit b053901 into main Aug 24, 2026
3 checks passed
@robinebers
robinebers deleted the codex/simplify-test-suites branch August 24, 2026 09:47
robinebers added a commit that referenced this pull request Aug 25, 2026
Second simplification pass after #1143, covering the suites that pass
did not touch: consolidates duplicated fixtures (Cursor JWT/SQLite
fakes, OpenCode stub, BlockingParser, sequence runtimes), parameterizes
near-identical test clusters, and removes tests whose behavior is
verified elsewhere or that restated the implementation.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gate-passed Passed the automated PR gatekeeper checks (gatekeeper skips it on later runs) tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant