Review additional store helpers

[strawgate]

I'm wondering if we could improve code re-use by adding basic client methods as hidden helpers on the stores. For example, get raw value at key, put raw value at key, etc?

Side benefit that then in the tests we aren't using raw clients to interact with the backend?

[claude]

Claude finished @strawgate's task —— View job

---

Investigation Complete ✓

I've thoroughly investigated the request to add basic client methods as hidden helpers on stores.

---

Recommendation

Introduce optional public helper methods on BaseStore for accessing raw backend values, but proceed cautiously. While this could improve code reuse and testing patterns, there are significant trade-offs to consider. I recommend implementing a minimal, opt-in approach rather than a comprehensive solution across all stores.

Key Considerations

Pros:

• Better Testing: Tests could use store helper methods instead of accessing private _client attributes with # pyright: ignore[reportPrivateUsage]
• Code Reuse: Eliminates duplicate raw client interaction code across test files
• Cleaner API: Provides a sanctioned way to access raw backend data during testing

Cons:

• Additional Maintenance Burden: Adds more methods to maintain across 18+ store implementations
• Inconsistent Across Stores: Each backend has different raw APIs (DynamoDB uses get_item(), MongoDB uses find_one(), Elasticsearch uses get(), etc.)
• Not Framework Purpose: py-key-value is designed to abstract away backend differences - exposing raw client methods works against this goal
• Breaking Abstraction: If users rely on these helpers, they become tightly coupled to specific backends
• Testing Alternative Exists: Tests could use dedicated test fixture helpers instead of store methods

---

Findings

Current Test Patterns

Tests currently access raw backends in two ways:

1. Direct Private Client Access (Most Common)

# DynamoDB - tests access _connected_client
client = store._connected_client  # pyright: ignore[reportPrivateUsage]
response = await client.get_item(TableName=..., Key=...)

2. Helper Functions in Test Files

# tests/stores/dynamodb/test_dynamodb.py
def get_dynamo_client_from_store(store: DynamoDBStore) -> DynamoDBClient:
    return store._connected_client  # pyright: ignore[reportPrivateUsage]

3. Internal Store Methods (Elasticsearch)

# Elasticsearch has _get_index_name() and _get_document_id() helpers
index_name = store._get_index_name(collection="test")  # pyright: ignore[reportPrivateUsage]
doc_id = store._get_document_id(key="test_key")  # pyright: ignore[reportPrivateUsage]
response = await es_client.get(index=index_name, id=doc_id)

Where Raw Client Access is Used

Grepped the test codebase and found ~30 instances of # pyright: ignore[reportPrivateUsage] where tests access:

• store._client (Redis, Memcached, MongoDB)
• store._connected_client (DynamoDB, Valkey)
• store._db (RocksDB, MongoDB)
• store._cache (DiskStore)
• store._collections_by_name (MongoDB)
• Internal helper methods like _get_index_name(), _sanitize_collection()

Primary use cases:

1. Document Format Tests (Issue Add tests for raw document format #180): Verify raw document structure stored in backends
2. Test Cleanup: Flush databases between tests
3. Direct Backend Verification: Confirm values stored correctly without going through the store abstraction

Architectural Context

BaseStore Pattern:

• All stores inherit from BaseStore which implements AsyncKeyValueProtocol
• Stores implement abstract methods: _get_managed_entry(), _put_managed_entry(), _delete_managed_entry()
• Values are always wrapped in ManagedEntry objects with metadata (TTL, timestamps)
• Each backend has unique serialization adapters (e.g., ElasticsearchSerializationAdapter, MongoDBSerializationAdapter)

Current Helper Methods:

• Elasticsearch has _get_index_name(), _get_document_id(), _get_destination() - but these are private
• MongoDB has _sanitize_collection()
• No stores currently expose public raw client helpers

---

Detailed Action Plan

If you decide to proceed, here's how to implement this feature:

Option A: Minimal Approach (Recommended)

Add optional public methods to specific stores where testing is most complex:

Step 1: Add public helpers to ElasticsearchStore (replace private methods used in tests)

# key-value/key-value-aio/src/key_value/aio/stores/elasticsearch/store.py
def get_index_name(self, collection: str) -> str:
    """Get the Elasticsearch index name for a collection (for testing/debugging)."""
    return self._get_index_name(collection=collection)

def get_document_id(self, key: str) -> str:
    """Get the Elasticsearch document ID for a key (for testing/debugging)."""
    return self._get_document_id(key=key)

Step 2: Add similar helpers to MongoDBStore

def get_raw_collection(self, collection: str) -> AsyncCollection:
    """Get the raw PyMongo collection object (for testing/debugging)."""
    await self._setup_collection(collection=collection)
    sanitized = self._sanitize_collection(collection=collection)
    return self._collections_by_name[sanitized]

Step 3: Update tests to use public methods instead of private ones

# Before:
index_name = store._get_index_name(collection="test")  # pyright: ignore[reportPrivateUsage]

# After:
index_name = store.get_index_name(collection="test")

Step 4: Run codegen to sync changes

make codegen

Step 5: Update tests and verify all pass

make test

Effort: Low (2-3 files modified per store, ~5-10 stores)

Option B: Comprehensive Approach (Not Recommended)

Add a standardized RawBackendProtocol to BaseStore:

Step 1: Define protocol in key-value/key-value-aio/src/key_value/aio/protocols/raw_backend.py

class AsyncRawBackendProtocol(Protocol):
    """Protocol for accessing raw backend clients (for testing)."""
    
    async def get_raw_value(self, *, collection: str, key: str) -> Any:
        """Get raw value from backend without ManagedEntry unwrapping."""
        ...
    
    async def put_raw_value(self, *, collection: str, key: str, value: Any) -> None:
        """Put raw value to backend without ManagedEntry wrapping."""
        ...

Step 2: Implement in each store (18+ implementations)

Step 3: Handle inconsistencies across backends (DynamoDB returns different structure than MongoDB, etc.)

Effort: High (18+ files modified, complex to standardize)

Why Not Recommended:

• High maintenance cost
• Violates abstraction principle
• Inconsistent backend APIs make standardization difficult

Option C: Test-Only Helpers (Alternative)

Instead of modifying stores, create dedicated test helper utilities:

Step 1: Create key-value/key-value-aio/tests/stores/helpers.py

# Centralized test helpers for raw backend access
from typing import Any, Protocol

class StoreWithClient(Protocol):
    _client: Any
    _connected_client: Any

async def get_raw_elasticsearch_doc(store, collection: str, key: str):
    """Get raw Elasticsearch document."""
    index = store._get_index_name(collection=collection)
    doc_id = store._get_document_id(key=key)
    return await store._client.get(index=index, id=doc_id)

# Similar helpers for other stores...

Step 2: Use in tests

from tests.stores.helpers import get_raw_elasticsearch_doc

async def test_value_stored(store):
    await store.put(...)
    raw_doc = await get_raw_elasticsearch_doc(store, "test", "key")
    assert raw_doc["_source"]["value"] == ...

Effort: Low (single file, tests updated to import)

Why Consider:

• Keeps store implementations clean
• Centralizes test-specific logic
• No changes to production code

---

Related Items

Related Issues and Pull Requests

Repository	Issue or PR	Relevance
strawgate/py-key-value	#180 - Add tests for raw document format	Directly related - these tests need raw client access to verify document format
strawgate/py-key-value	#205 - Identify more base store shared utilities	Discusses sharing more code across stores, including potential helper utilities
strawgate/py-key-value	#232 - Review Client Ownership	Related to how stores manage their underlying clients
strawgate/py-key-value	#206 - Identify more base store tests	Discusses test infrastructure improvements, which would benefit from better raw access patterns

Related Files

Repository	File	Relevance	Sections
strawgate/py-key-value	key-value-aio/src/key_value/aio/stores/base.py	Base class for all stores where helper methods would be defined	All
strawgate/py-key-value	key-value-aio/src/key_value/aio/stores/elasticsearch/store.py	Already has private helper methods that could be made public	275-285
strawgate/py-key-value	key-value-aio/tests/stores/elasticsearch/test_elasticsearch.py	Uses private helpers to access raw documents	178-179
strawgate/py-key-value	key-value-aio/tests/stores/mongodb/test_mongodb.py	Accesses private _collections_by_name and _setup_collection	159-162
strawgate/py-key-value	key-value-aio/tests/stores/dynamodb/test_dynamodb.py	Uses helper function to access _connected_client	62-63

---

Problems Encountered During Investigation

None - all investigation tools and codebase resources were accessible.

---