Only run store tests for impacted stores

[strawgate]

If a PR only changes a specific store, we should only run the tests for that store

If a PR touches shared code, we should probably run all store tests.

[claude]

Claude finished @strawgate's task —— View job

---

Investigation Complete ✓

I've thoroughly analyzed the request to run store tests selectively based on changed files in pull requests.

---

Recommendation

Implement selective test execution using the dorny/paths-filter GitHub Action combined with dynamic matrix job generation. This approach will detect which stores or shared code are modified and run only the relevant test suites, significantly reducing CI time for store-specific changes while maintaining comprehensive coverage for shared code changes.

---

Findings

Current Test Infrastructure

The project currently runs all tests for all stores on every PR through two test jobs in .github/workflows/test.yml:

1. test_quick (Python 3.10 on 6 platforms × 3 projects)
2. test_all (Python 3.11-3.13 on 6 platforms × 3 projects)

Each project (key-value-aio, key-value-sync, key-value-shared) runs pytest tests . -vv, which executes all store tests regardless of which stores were modified.

Store Structure

The codebase has 15 distinct stores, each in its own directory:

• key-value/key-value-aio/src/key_value/aio/stores/{store_name}/
• key-value/key-value-aio/tests/stores/{store_name}/

Stores: disk, dynamodb, elasticsearch, keyring, memcached, memory, mongodb, null, redis, rocksdb, simple, valkey, vault, windows_registry, plus wrappers.

Shared Code Categories

Changes to these paths should trigger all store tests:

1. Base store classes: key-value/key-value-aio/src/key_value/aio/stores/base.py
2. Protocols: key-value/key-value-aio/src/key_value/aio/protocols/
3. Adapters: key-value/key-value-aio/src/key_value/aio/adapters/
4. Wrappers: key-value/key-value-aio/src/key_value/aio/wrappers/
5. Shared package: key-value/key-value-shared/
6. Base tests: key-value/key-value-aio/tests/stores/base.py
7. Test conftest: key-value/key-value-aio/tests/stores/conftest.py
8. Sync codegen: scripts/build_sync_library.py

Best Practices from Industry

Research shows three effective approaches for 2025:

1. Path-based filtering (dorny/paths-filter): Detect changes at the workflow level
2. pytest-testmon: Intelligent test selection based on code coverage
3. pytest-picked: Run tests for changed files according to Git

For a monorepo with clearly-defined store boundaries, path-based filtering is the most straightforward and maintainable solution.

---

Detailed Action Plan

Implementation Steps

Phase 1: Add Path Detection Job

Create a new job in .github/workflows/test.yml that detects which stores changed:

jobs:
  detect_changes:
    runs-on: ubuntu-latest
    outputs:
      stores: ${{ steps.filter.outputs.changes }}
      run_all: ${{ steps.filter.outputs.shared }}
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Detect changed paths
        uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            # Shared code that affects all stores
            shared:
              - 'key-value/key-value-shared/**'
              - 'key-value/key-value-aio/src/key_value/aio/stores/base.py'
              - 'key-value/key-value-aio/src/key_value/aio/protocols/**'
              - 'key-value/key-value-aio/src/key_value/aio/adapters/**'
              - 'key-value/key-value-aio/src/key_value/aio/wrappers/**'
              - 'key-value/key-value-aio/tests/stores/base.py'
              - 'key-value/key-value-aio/tests/stores/conftest.py'
              - 'scripts/build_sync_library.py'
            
            # Individual stores
            redis:
              - 'key-value/key-value-aio/src/key_value/aio/stores/redis/**'
              - 'key-value/key-value-aio/tests/stores/redis/**'
            
            mongodb:
              - 'key-value/key-value-aio/src/key_value/aio/stores/mongodb/**'
              - 'key-value/key-value-aio/tests/stores/mongodb/**'
            
            elasticsearch:
              - 'key-value/key-value-aio/src/key_value/aio/stores/elasticsearch/**'
              - 'key-value/key-value-aio/tests/stores/elasticsearch/**'
            
            dynamodb:
              - 'key-value/key-value-aio/src/key_value/aio/stores/dynamodb/**'
              - 'key-value/key-value-aio/tests/stores/dynamodb/**'
            
            # ... repeat for all 15 stores
            
            wrappers:
              - 'key-value/key-value-aio/tests/stores/wrappers/**'

Phase 2: Modify Test Jobs to Use Dynamic Matrix

Update test_quick and test_all jobs:

  test_quick:
    needs: 
      - markdown_lint
      - codegen_check
      - static_analysis
      - detect_changes
    
    # Only run if there are changes to test
    if: needs.detect_changes.outputs.run_all == 'true' || needs.detect_changes.outputs.stores != '[]'
    
    timeout-minutes: 10
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.10"]
        platform: [ubuntu-22.04, ubuntu-latest, macos-14, macos-latest, windows-2022, windows-latest]
        project: ["key-value/key-value-aio", "key-value/key-value-shared", "key-value/key-value-sync"]
    
    runs-on: ${{ matrix.platform }}
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: "Install uv"
        uses: astral-sh/setup-uv@v6
      
      - name: "Install latest dependencies"
        run: uv sync --locked --group dev --python ${{ matrix.python-version }}
        working-directory: ${{ matrix.project }}
      
      - name: "Test with latest dependencies (all stores)"
        if: needs.detect_changes.outputs.run_all == 'true'
        run: uv run pytest tests . -vv
        working-directory: ${{ matrix.project }}
      
      - name: "Test with latest dependencies (specific stores)"
        if: needs.detect_changes.outputs.run_all != 'true'
        run: |
          stores='${{ needs.detect_changes.outputs.stores }}'
          echo "Changed stores: $stores"
          
          # Run tests for each changed store
          for store in $(echo $stores | jq -r '.[]'); do
            echo "Testing store: $store"
            if [ -d "tests/stores/$store" ]; then
              uv run pytest tests/stores/$store -vv
            fi
          done
          
          # Always run shared package tests
          if [ "${{ matrix.project }}" = "key-value/key-value-shared" ]; then
            uv run pytest tests . -vv
          fi
        working-directory: ${{ matrix.project }}
        shell: bash
      
      # ... rest of the job (oldest dependencies, build, etc.)

Phase 3: Testing and Validation

1. Create test PRs that modify:

   • A single store (e.g., only Redis)
   • Multiple stores
   • Shared code
   • A mix of shared code and specific stores

2. Verify that:

   • Single-store changes only run that store's tests
   • Shared code changes run all store tests
   • The workflow correctly handles edge cases

3. Measure the time savings:

   • Baseline: Current full test suite time
   • Target: ~70-80% reduction for single-store changes

Phase 4: Documentation

Update the repository documentation to explain:

1. How the selective testing works
2. What triggers full test runs vs. partial runs
3. How to force a full test run if needed (workflow_dispatch)
4. How to add new stores to the path filters

---

Related Issues and Pull Requests

Repository	Issue or PR	Relevance
strawgate/py-key-value	Intermittent Elasticsearch Test errors #201	Demonstrates that some store tests have reliability issues; selective testing would reduce exposure to flaky tests on unrelated changes
strawgate/py-key-value	MultiDiskStore on Windows test flaky #166	Another example of store-specific test issues affecting all PRs
strawgate/py-key-value	Identify more base store tests #206	Related to improving test coverage; selective testing would make it easier to add more comprehensive tests per store

---

Related Files

Repository	File	Relevance	Sections
strawgate/py-key-value	.github/workflows/test.yml	The main test workflow that needs modification	Lines 112-213 (test jobs)
strawgate/py-key-value	key-value/key-value-aio/tests/stores/base.py	Base test class inherited by all store tests; changes here trigger all store tests	All
strawgate/py-key-value	key-value/key-value-aio/src/key_value/aio/stores/base.py	Base store implementation; changes here trigger all store tests	All
strawgate/py-key-value	key-value/key-value-aio/pyproject.toml	pytest configuration with current test settings	Lines 53-66

---

Related Webpages

Name	URL	Relevance
dorny/paths-filter GitHub Action	https://github.com/dorny/paths-filter	The recommended GitHub Action for detecting changed files and implementing selective test execution
pytest-testmon	https://github.com/tarpas/pytest-testmon	Alternative approach using coverage-based test selection; more complex but potentially more accurate
GitHub Actions Path Filtering	https://norday.tech/posts/2025/only-run-github-actions-when-certain-files-have-changed/	Best practices for conditional workflow execution in 2025

---

Problems Encountered During Investigation

None - all investigation tools and repository resources were accessible and provided comprehensive information for this analysis.

---

[strawgate]

@claude how could we incorporate https://testmon.org/ pytest-testmon? that seems a lot more fun

[claude]

Claude finished @strawgate's task —— View job

---

pytest-testmon Integration Analysis ✓

Excellent idea! pytest-testmon is indeed "more fun" and potentially more accurate than path-based filtering. Let me break down how it could work for py-key-value.

---

What is pytest-testmon?

pytest-testmon is a pytest plugin that automatically selects and executes only tests affected by code changes. Instead of using path-based filtering (which relies on predefined rules), it uses coverage-based dependency tracking to determine which tests actually execute the changed code.

How it works:

1. Uses Coverage.py to track which code each test executes
2. Stores this dependency data in .testmondata (SQLite database)
3. On subsequent runs, compares changed files against the dependency database
4. Runs only tests that have executed the changed code

---

Why pytest-testmon is Better for py-key-value

Advantages over paths-filter:

1. Smarter dependency detection: Automatically discovers dependencies without maintaining filter lists

   • If a test in tests/stores/redis/ imports code from wrappers/, testmon knows about it
   • No manual maintenance of "shared code" path lists
   • Discovers cross-store dependencies automatically

2. Adapts to code changes: As your codebase evolves, dependency tracking evolves with it

   • Add a new wrapper? Tests that use it are automatically included
   • Refactor shared code? Only affected tests run

3. Works with the monorepo structure: Handles the async→sync codegen relationship

   • If sync code is generated from async, testmon tracks the actual code coverage
   • No need to manually specify that sync tests should run when async code changes

4. Zero maintenance: No need to update filter configurations when adding new stores

Trade-offs:

1. Requires caching: Need to cache .testmondata between CI runs (but GitHub Actions makes this easy)
2. Initial run is full: First run with --testmon must run all tests to build the database
3. Database invalidation: Major dependency changes might require clearing .testmondata

---

Implementation Plan

Phase 1: Add pytest-testmon to Dependencies

Add to key-value/key-value-aio/pyproject.toml:

[dependency-groups]
dev = [
    "py-key-value-aio[memory,disk,redis,elasticsearch,memcached,mongodb,vault,dynamodb,rocksdb]",
    "py-key-value-aio[valkey]; platform_system != 'Windows'",
    "py-key-value-aio[keyring]",
    "py-key-value-aio[pydantic]",
    "py-key-value-aio[wrappers-encryption]",
    "py-key-value[dev]",
    "pytest-testmon>=2.1.4",  # Add this line
]

Repeat for key-value-sync and key-value-shared.

Phase 2: Update pytest Configuration

Add to key-value/key-value-aio/pyproject.toml:

[tool.pytest.ini_options]
asyncio_mode = "auto"
addopts = [
    "--inline-snapshot=disable",
    "-n=auto",
    "--dist=loadfile",
    "--maxfail=5"
]
markers = [
    "skip_on_ci: Skip running the test when running on CI",
]
timeout = 10
env_files = [".env"]

# pytest-testmon configuration
# Note: --testmon is added via CLI, not here, so we can control it in CI

Phase 3: Update GitHub Actions Workflow

Modify .github/workflows/test.yml:

  test_quick:
    needs:
      - markdown_lint
      - codegen_check
      - static_analysis
    timeout-minutes: 10
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.10"]
        platform:
          [
            ubuntu-22.04,
            ubuntu-latest,
            macos-14,
            macos-latest,
            windows-2022,
            windows-latest,
          ]
        project:
          - "key-value/key-value-aio"
          - "key-value/key-value-shared"
          - "key-value/key-value-sync"

    runs-on: ${{ matrix.platform }}
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Need full history for testmon

      - name: "Install uv"
        uses: astral-sh/setup-uv@v6

      # Cache the testmon database
      - name: Cache testmon data
        uses: actions/cache@v4
        with:
          path: ${{ matrix.project }}/.testmondata
          key: testmon-${{ matrix.platform }}-${{ matrix.python-version }}-${{ matrix.project }}-${{ github.sha }}
          restore-keys: |
            testmon-${{ matrix.platform }}-${{ matrix.python-version }}-${{ matrix.project }}-
            testmon-${{ matrix.platform }}-${{ matrix.python-version }}-

      - name: "Install latest dependencies"
        run: uv sync --locked --group dev --python ${{ matrix.python-version }}
        working-directory: ${{ matrix.project }}

      - name: "Test with latest dependencies (testmon)"
        run: uv run pytest tests . -vv --testmon
        working-directory: ${{ matrix.project }}

      - name: "Install oldest dependencies"
        run: uv sync --group dev --python ${{ matrix.python-version }} --resolution lowest-direct
        working-directory: ${{ matrix.project }}

      - name: "Test with oldest dependencies (testmon)"
        run: uv run pytest tests . -vv --testmon
        working-directory: ${{ matrix.project }}

      - name: "Build"
        run: uv build .
        working-directory: ${{ matrix.project }}

  test_all:
    needs:
      - test_quick

    timeout-minutes: 20
    strategy:
      matrix:
        python-version: ["3.11", "3.12", "3.13"]
        platform:
          [
            ubuntu-22.04,
            ubuntu-latest,
            macos-14,
            macos-latest,
            windows-2022,
            windows-latest,
          ]
        project:
          - "key-value/key-value-aio"
          - "key-value/key-value-shared"
          - "key-value/key-value-sync"

    runs-on: ${{ matrix.platform }}

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Need full history for testmon

      - name: "Install uv"
        uses: astral-sh/setup-uv@v6

      # Cache the testmon database
      - name: Cache testmon data
        uses: actions/cache@v4
        with:
          path: ${{ matrix.project }}/.testmondata
          key: testmon-${{ matrix.platform }}-${{ matrix.python-version }}-${{ matrix.project }}-${{ github.sha }}
          restore-keys: |
            testmon-${{ matrix.platform }}-${{ matrix.python-version }}-${{ matrix.project }}-
            testmon-${{ matrix.platform }}-${{ matrix.python-version }}-

      - name: "Install latest dependencies"
        run: uv sync --locked --group dev --python ${{ matrix.python-version }}
        working-directory: ${{ matrix.project }}

      - name: "Test with latest dependencies (testmon)"
        run: uv run pytest tests . -vv --testmon
        working-directory: ${{ matrix.project }}

      - name: "Install oldest dependencies"
        run: uv sync --group dev --python ${{ matrix.python-version }} --resolution lowest-direct
        working-directory: ${{ matrix.project }}

      - name: "Test with oldest dependencies (testmon)"
        run: uv run pytest tests . -vv --testmon
        working-directory: ${{ matrix.project }}

      - name: "Build"
        run: uv build .
        working-directory: ${{ matrix.project }}

Phase 4: Add .testmondata to .gitignore

# pytest-testmon
.testmondata
.testmondata-journal

---

Testing Strategy

1. First PR with testmon: Will run all tests (building the initial database)
2. Subsequent PRs: Will run only affected tests
3. Monitor cache hits: Check GitHub Actions logs to ensure cache is working
4. Verify test selection: Add debug logging to see which tests testmon selects

---

Forcing Full Test Runs

Sometimes you'll want to run all tests regardless:

# Clear the database and run all tests
pytest --testmon-nocache

# Or delete the database
rm .testmondata
pytest --testmon

In CI, you can clear the cache by:

1. Changing the cache key prefix
2. Manually deleting the cache in GitHub Actions settings
3. Using workflow_dispatch with an option to clear cache

---

Compatibility with Existing Setup

pytest-testmon works great with your existing pytest configuration:

• ✅ pytest-xdist (-n=auto): testmon supports parallel test execution
• ✅ pytest-timeout: Works fine together
• ✅ pytest-asyncio: No conflicts
• ✅ inline-snapshot: Compatible

---

Expected Performance Improvement

Based on the project structure:

• Store-specific changes: ~85-95% reduction in test execution time

  • Only that store's tests run + any tests that import from it
  • Example: Change Redis store → only Redis tests run

• Shared code changes: Runs all affected tests

  • Change base.py → runs tests for all stores that inherit from it
  • More accurate than path-based filtering (some stores might not use certain base features)

• Documentation changes: 100% reduction

  • No tests run if only .md files change (already handled by paths-ignore)

---

Migration Path

Option 1: Big Bang (Recommended)

1. Add pytest-testmon to all three projects
2. Update workflow files
3. Test on a small PR
4. Merge and let the database build

Option 2: Gradual

1. Start with key-value-aio only
2. Monitor for a few PRs
3. Extend to key-value-sync and key-value-shared

I recommend Option 1 since pytest-testmon is non-invasive and can be disabled with a simple flag if issues arise.

---

Comparison: pytest-testmon vs paths-filter

Feature	pytest-testmon	paths-filter
Setup complexity	Low (add dep + cache)	Medium (maintain filter rules)
Maintenance	Zero	High (update for each new store)
Accuracy	Very high (coverage-based)	Medium (path-based heuristics)
Cross-dependencies	Automatic detection	Manual specification
Monorepo support	Excellent	Good
First-run overhead	Full test run	None
Cache required	Yes	No
Works offline	Yes (after first run)	Yes
Debugging	Can show dependency graph	Simple path matching

---

Should We Implement This?

My recommendation: Yes!

pytest-testmon is:

• ✅ More accurate than path-based filtering
• ✅ Zero maintenance overhead
• ✅ Better suited for the monorepo + codegen structure
• ✅ Easy to disable if issues arise (pytest without --testmon)
• ✅ Actively maintained (latest release: Nov 2025)
• ✅ Used by large projects (Matplotlib, etc.)

The only downside is the initial full test run, but that's a one-time cost that pays off immediately on the next PR.

---

Next Steps

If you'd like me to implement this, I can:

1. Add pytest-testmon to all three project dependencies
2. Update .github/workflows/test.yml with caching
3. Add .testmondata to .gitignore
4. Test the changes locally (via make test)
5. Create a PR for review

Just let me know if you want me to proceed with the implementation!

---