feat: Prepare Dift for Distributed Comparison Architecture

[ReginaldErzoah]

Summary

Prepare Dift internally for future distributed dataset comparison workflows.

---

Motivation

Distributed comparison will be important for future enterprise-scale workflows, especially when datasets are too large for a single local process.

This issue focuses on architectural preparation, not full distributed execution.

Future distributed workflows may support:

• partitioned comparison
• distributed workers
• remote execution
• warehouse-native compute
• Spark or Ray-based execution

---

Proposed Improvements

• Identify comparison stages that can be partitioned
• Define distributed execution boundaries
• Prepare interfaces for future distributed engines
• Avoid tightly coupling comparison logic to local-only execution
• Document architectural direction

---

Suggested Files

Potential implementation areas:

dift/core/comparator.py
dift/core/row_diff.py
dift/core/stats_diff.py
dift/io/
dift/developer docs

---

Suggested Tasks

• Review comparison pipeline for partitionable stages
• Add internal abstractions where useful
• Avoid implementation that assumes single-process execution only
• Add architecture notes for future distributed support
• Add tests ensuring existing workflows remain stable
• Update developer documentation

---

How to Test

Run:

pytest
ruff check .

Run targeted tests:

pytest tests/test_comparator.py
pytest tests/test_large_dataset_workflows.py

Manual validation:

dift examples/old.csv examples/new.csv --key customer_id

Verify:

• existing local comparisons still work
• architecture remains backward compatible
• no report behavior changes unexpectedly
• developer docs explain the distributed preparation clearly

---

Documentation Impact

Update:

docs/developer/architecture.md
docs/developer/plugin-preparation.md
docs/performance.md
docs/roadmap.md

Documentation should include:

• future distributed comparison direction
• partitioning considerations
• current limitations
• contributor guidance for distributed-safe changes

---

Acceptance Criteria

• Distributed comparison preparation is documented
• Comparison architecture is better prepared for partitioned execution
• Existing workflows remain stable
• Tests pass
• Documentation updated