Flaky Elasticsearch Tests

[strawgate]

FAILED tests/stores/elasticsearch/test_elasticsearch.py::TestElasticsearchStore::test_empty_get[9.2.0-with_ctx_manager] - Failed: Timeout (>10.0s) from pytest-timeout.
FAILED tests/stores/elasticsearch/test_elasticsearch.py::TestElasticsearchStore::test_empty_get[9.2.0-no_ctx_manager] - Failed: Timeout (>10.0s) from pytest-timeout.
FAILED tests/stores/elasticsearch/test_elasticsearch.py::TestElasticsearchStore::test_empty_put[9.2.0-with_ctx_manager] - key_value.shared.errors.store.StoreSetupError: Failed to setup collection: BadRequestError(400, 'resource_already_exists_exception', 'index [kv-store-e2e-test-test/4G1uWN3mRaa5TBGbcGdiAw] already exists'): (collection: test)
FAILED tests/stores/elasticsearch/test_elasticsearch.py::TestElasticsearchStore::test_empty_put[9.2.0-no_ctx_manager] - Failed: Timeout (>10.0s) from pytest-timeout.
FAILED tests/stores/elasticsearch/test_elasticsearch.py::TestElasticsearchStore::test_empty_ttl[9.2.0-with_ctx_manager] - elasticsearch.ApiError: ApiError(503, 'no_shard_available_action_exception', 'No shard available for [get [kv-store-e2e-test-test][test]: routing [null]]')

[claude]

Claude finished @strawgate's task —— View job

---

Investigation Complete ✓

I've thoroughly analyzed the flaky Elasticsearch tests and identified the root causes and actionable solutions.

---

Recommendation

Implement a two-pronged fix to address race conditions and ensure proper index cleanup:

1. Fix the cleanup_elasticsearch_indices function to properly handle errors and ensure indices are actually deleted
2. Add a wait-for-consistency step after cleanup to ensure Elasticsearch has processed deletions before tests begin

The current test infrastructure has timing-dependent race conditions that manifest as three types of failures:

• resource_already_exists_exception - indices not properly cleaned up between tests
• Timeouts - Elasticsearch operations hanging due to inconsistent state
• no_shard_available_action_exception - trying to access documents before shards are ready

---

Findings

Error Pattern Analysis

The issue reports five distinct test failures, all with Elasticsearch 9.2.0:

1. Timeout errors (3 occurrences):

   • test_empty_get[9.2.0-with_ctx_manager] - timeout >10s
   • test_empty_get[9.2.0-no_ctx_manager] - timeout >10s
   • test_empty_put[9.2.0-no_ctx_manager] - timeout >10s

2. Index already exists (1 occurrence):

   • test_empty_put[9.2.0-with_ctx_manager] - resource_already_exists_exception: index [kv-store-e2e-test-test/4G1uWN3mRaa5TBGbcGdiAw] already exists

3. No shard available (1 occurrence):

   • test_empty_ttl[9.2.0-with_ctx_manager] - no_shard_available_action_exception: No shard available for [get [kv-store-e2e-test-test][test]: routing [null]]

These errors share a common theme: Elasticsearch eventual consistency and timing issues.

Root Cause: Inadequate Cleanup and Missing Wait-for-Consistency

Problem 1: Silent Cleanup Failures

File: test_elasticsearch.py:47-50

async def cleanup_elasticsearch_indices(elasticsearch_client: AsyncElasticsearch):
    indices = await elasticsearch_client.options(ignore_status=404).indices.get(index="kv-store-e2e-test-*")
    for index in indices:
        _ = await elasticsearch_client.options(ignore_status=404).indices.delete(index=index)

Issues:

1. The ignore_status=404 suppresses errors, meaning deletion failures go unnoticed
2. No verification that indices were actually deleted
3. No error handling for scenarios where indices.get() returns unexpected data
4. Iterating directly over the response object (for index in indices) doesn't properly extract index names

Problem 2: Missing Wait-for-Consistency

After deleting indices, the tests immediately start creating new ones. However, Elasticsearch index deletion is eventually consistent - the cluster may still think indices exist even after successful deletion, especially under load.

This causes:

• resource_already_exists_exception when trying to recreate indices too quickly
• no_shard_available_action_exception when trying to access documents before shards are allocated
• Timeouts when Elasticsearch operations hang waiting for cluster state to settle

Problem 3: Fixture Scope and Cleanup Timing

File: test_elasticsearch.py:85-100

@pytest.fixture(autouse=True, scope="session", params=ELASTICSEARCH_VERSIONS_TO_TEST)
async def setup_elasticsearch(self, request: pytest.FixtureRequest) -> AsyncGenerator[None, None]:
    version = request.param
    # ... creates container for ENTIRE test session

File: test_elasticsearch.py:124-128

@pytest.fixture(autouse=True)
async def cleanup_elasticsearch_indices(self, es_client: AsyncElasticsearch):
    await cleanup_elasticsearch_indices(elasticsearch_client=es_client)  # ← Before each test
    yield
    await cleanup_elasticsearch_indices(elasticsearch_client=es_client)  # ← After each test

The mismatch between session-scoped container and function-scoped cleanup creates edge cases:

• Container persists across all tests
• Cleanup runs before/after each test
• If cleanup fails, the next test inherits a polluted state

Problem 4: Store Already Handles Race Conditions (Partially)

File: store.py:262-273

async def _setup_collection(self, *, collection: str) -> None:
    index_name = self._get_index_name(collection=collection)
    
    if await self._client.options(ignore_status=404).indices.exists(index=index_name):
        return
    
    try:
        _ = await self._client.options(ignore_status=404).indices.create(...)
    except BadRequestError as e:
        if "index_already_exists_exception" in str(e).lower():
            return  # ← Already handles this!
        raise

The store code already handles resource_already_exists_exception, but this doesn't help with:

• Cleanup failures leaving stale indices
• Shard availability issues
• Timeouts from Elasticsearch being in inconsistent state

Why This is Intermittent

The failures are intermittent because they depend on:

1. Elasticsearch internal timing: How quickly the cluster processes index deletions and shard allocations
2. System load: Higher load increases the window for race conditions
3. Test execution order: Some test orderings expose the race more than others
4. Elasticsearch version differences: Version 9.2.0 may have different timing characteristics than 9.0.0

The fact that issue #218 only shows failures with version 9.2.0 (not 9.0.0) suggests version-specific behavior differences in index lifecycle management.

---

Detailed Action Plan

Fix 1: Improve Index Cleanup Function

File: test_elasticsearch.py:47-50

Replace the cleanup function with robust error handling:

async def cleanup_elasticsearch_indices(elasticsearch_client: AsyncElasticsearch):
    """Clean up all test indices and wait for deletion to complete."""
    try:
        # Get all matching indices
        response = await elasticsearch_client.indices.get(index="kv-store-e2e-test-*")
        
        if not response or not response.body:
            logger.info("No indices to clean up")
            return
        
        # Extract index names from response
        index_names = list(response.body.keys()) if hasattr(response.body, 'keys') else []
        
        if not index_names:
            logger.info("No matching indices found")
            return
        
        logger.info(f"Deleting indices: {index_names}")
        
        # Delete each index explicitly
        for index_name in index_names:
            try:
                await elasticsearch_client.indices.delete(index=index_name)
                logger.info(f"Deleted index: {index_name}")
            except Exception as e:
                # Log but don't fail - index might already be gone
                logger.warning(f"Failed to delete index {index_name}: {e}")
        
        # CRITICAL: Wait for Elasticsearch to finish processing deletions
        # Elasticsearch index deletion is eventually consistent
        await async_wait_for_true(
            bool_fn=lambda: check_indices_gone(elasticsearch_client, index_names),
            tries=10,
            wait_time=0.5,
        )
        
    except Exception as e:
        # No indices exist or other error - log and continue
        logger.debug(f"Cleanup check failed (likely no indices): {e}")


async def check_indices_gone(client: AsyncElasticsearch, index_names: list[str]) -> bool:
    """Verify that all specified indices are actually deleted."""
    try:
        response = await client.options(ignore_status=404).indices.get(index="kv-store-e2e-test-*")
        if not response or not response.body:
            return True
        
        existing = set(response.body.keys()) if hasattr(response.body, 'keys') else set()
        return not any(name in existing for name in index_names)
    except Exception:
        # Error likely means no indices exist
        return True

Fix 2: Add Refresh After Index Creation (Already Implemented)

The store already has refresh logic in _put_managed_entry (line 370), but it's controlled by _should_refresh_on_put which returns False for serverless. For testing, we want immediate consistency.

Verify that refresh=self._should_refresh_on_put evaluates to True in the test environment (non-serverless Elasticsearch).

Fix 3: Add Wait-for-Green-Health Check

File: test_elasticsearch.py:124-128

Add a health check after cleanup to ensure cluster is in good state:

@pytest.fixture(autouse=True)
async def cleanup_elasticsearch_indices(self, es_client: AsyncElasticsearch):
    await cleanup_elasticsearch_indices(elasticsearch_client=es_client)
    
    # Wait for cluster to be ready
    await es_client.cluster.health(wait_for_status="yellow", timeout="10s")
    
    yield
    
    await cleanup_elasticsearch_indices(elasticsearch_client=es_client)
    await es_client.cluster.health(wait_for_status="yellow", timeout="10s")

This ensures:

• Cluster is in a stable state before tests run
• All shards are allocated properly
• Index operations won't encounter "no shard available" errors

Fix 4: Increase pytest Timeout for Elasticsearch Tests (Optional)

File: pyproject.toml:64

The global timeout is 10 seconds, which may be too aggressive for Elasticsearch operations including container startup, index operations, and cleanup.

Consider adding a test-specific timeout override:

@pytest.mark.timeout(20)  # Double the timeout for Elasticsearch tests
class TestElasticsearchStore(ContextManagerStoreTestMixin, BaseStoreTests):
    # ...

Or update the global timeout in pyproject.toml:

[tool.pytest.ini_options]
timeout = 20  # Increase from 10 to 20 seconds

---

Related Items

Related Issues and Pull Requests

Repository	Issue or PR	Relevance
strawgate/py-key-value	Issue #201 - Intermittent Elasticsearch Test errors	Directly related - reports identical resource_already_exists_exception and timeout errors. Previous investigation identified fixture scoping issues and cleanup problems. The store was patched to handle race conditions, but test infrastructure issues remain.
strawgate/py-key-value	Issue #166 - MultiDiskStore on Windows test flaky	Similar flaky test pattern resolved by improving test isolation
strawgate/py-key-value	Issue #129 - Ubuntu 22.04 tests hanging	Test timeout issue resolved by fixing pytest-xdist race condition

Related Files

Repository	File	Relevance	Sections
strawgate/py-key-value	test_elasticsearch.py	Contains the failing tests and problematic cleanup fixture	47-50 (cleanup), 85-100 (session fixture), 124-128 (cleanup fixture)
strawgate/py-key-value	store.py	ElasticsearchStore implementation with partial race condition handling	262-273 (_setup_collection already handles resource_already_exists_exception)
strawgate/py-key-value	conftest.py	Docker container management and test utilities	148-174 (docker_container context manager)
strawgate/py-key-value	pyproject.toml	pytest configuration including 10s timeout	53-64 (pytest config with 10s timeout)

Related Webpages

Name	URL	Relevance
Elasticsearch Index Lifecycle	https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-delete-index.html	Documents index deletion and eventual consistency behavior
Elasticsearch Cluster Health API	https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html	API for waiting for cluster to reach stable state before operations
Elasticsearch Shard Allocation	https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-shards.html	Explains shard allocation and "no shard available" errors
pytest-timeout documentation	https://pypi.org/project/pytest-timeout/	Documentation for the timeout plugin used in the test suite

---

Problems Encountered During Investigation

None - all investigation tools were accessible. The codebase is well-structured with clear test patterns, making it straightforward to identify the timing and consistency issues with Elasticsearch index lifecycle management.

---