feat: Add Dynamic Severity Classification

[ReginaldErzoah]

Summary

Add dynamic severity classification for drift and validation findings.

---

Motivation

Current severity classification may rely on fixed thresholds.

Dynamic severity classification would allow Dift to adjust severity based on context such as:

• dataset size
• magnitude of change
• historical drift patterns
• column-level sensitivity
• configured thresholds

This makes Dift’s warnings more accurate and useful.

---

Proposed Improvements

• Add context-aware severity classification
• Improve severity labels for numeric drift, categorical drift, and outliers
• Preserve existing low/medium/high severity model
• Include severity explanation metadata in reports

---

Suggested Files

Potential implementation areas:

dift/core/stats_diff.py
dift/core/risk.py
dift/thresholds.py
dift/reports/models.py
dift/reports/console_report.py
dift/reports/html_report.py

---

Suggested Tasks

• Review current severity classification rules
• Add dynamic severity utility
• Add severity explanation fields
• Add tests for low, medium, and high severity cases
• Update reports
• Update documentation

---

How to Test

Run:

pytest
ruff check .

Run targeted tests:

pytest tests/test_stats_diff.py
pytest tests/test_risk.py
pytest tests/test_reports.py

Manual validation:

dift examples/old_drift.csv examples/new_drift.csv --key id

Verify:

• severity levels are assigned correctly
• severity explanations appear where appropriate
• reports remain valid
• existing workflows remain stable

---

Documentation Impact

Update:

docs/statistical-analysis.md
docs/thresholds.md
docs/reports.md

Documentation should include:

• how severity is classified
• what low, medium, and high mean
• how dynamic classification works
• examples of severity interpretation

---

Acceptance Criteria

• Dynamic severity classification is implemented
• Severity labels remain understandable
• Reports include severity information
• Tests pass
• Documentation updated