Warn on concurrent partial shard writes to prevent silent data corruption

[kkollsga]

Summary

• Add detection and warning for concurrent partial shard writes
• Only warn when actual concurrent access is detected (not on sequential writes)
• Add config option sharding.warn_on_partial_write (default: True)

Problem

When multiple tasks write to different regions of the same shard concurrently, each task:

1. Reads the full shard from storage
2. Modifies its portion in memory
3. Writes the entire shard back

This read-modify-write pattern causes race conditions where earlier writes get silently overwritten, resulting in 30-50% data loss in parallel write scenarios (e.g., dask to_zarr with misaligned chunks).

Related: pydata/xarray#10831, #3682

Solution

Track in-progress partial shard writes using a thread-safe counter per shard. When concurrent access is detected, emit a warning:

ZarrUserWarning: Concurrent partial shard writes detected. Writing to different 
regions of the same shard concurrently may result in data corruption due to 
read-modify-write race conditions. Consider aligning your write chunks with 
shard boundaries, or use a lock to coordinate writes.

Key design decisions

1. Warn, don't error - Non-breaking change that alerts users without stopping execution
2. Only warn on actual concurrency - Sequential partial writes (safe, intended usage) don't trigger warnings
3. Configurable - Users can disable via zarr.config.set({'sharding.warn_on_partial_write': False})
4. Negligible overhead - ~0.001% (lock acquisition per partial write)

Test plan

• Sequential partial writes: No warning (safe pattern)
• Concurrent partial writes: Warning fires (corruption risk)
• Full shard writes: No warning (no read-modify-write)
• All 124 sharding tests pass
• Performance benchmark confirms negligible overhead

🤖 Generated with Claude Code