streaming-lakehouse-platform

Reference implementation of a medallion (bronze/silver/gold) lakehouse over the platform's event streams using PySpark + Delta Lake. Ingests ledger/CDC events from Kafka, applies data quality + dedup, then materializes the CQRS balance read model and an AI feature store that feeds agentic-rag-engine and ML models.

Part of the Enterprise Platform Reference Architecture. Models the enterprise data platform domain. See docs/INDUSTRY-APPLICABILITY.md.

Design principle: testable transforms, thin Spark glue

Business logic lives in pure-Python functions (cdc, read_model, features, quality) that are unit-tested without a cluster. The Spark/Delta layer (spark_jobs.py) is a thin adapter that applies the same semantics at scale. This keeps the interesting logic fast to test and correct independent of Spark.

Architecture (medallion)

flowchart LR
  kafka[(Kafka: ledger + legacy CDC)] --> bronze["BRONZE: raw events (Delta)"]
  bronze --> silver["SILVER: parsed, quality-checked, deduped (Delta)"]
  silver --> goldBal["GOLD: CQRS balance read model"]
  silver --> goldFeat["GOLD: AI feature store"]
  goldFeat --> ml["agentic-rag-engine / ML models"]
  goldBal --> bi["BI / reconciliation"]

Loading

Run

Tests + local medallion run (no Spark, no infra)

python -m venv .venv && source .venv/bin/activate
pip install pytest
pytest -q tests/test_transforms.py
python scripts/run_local.py     # prints quality report, CQRS balances, feature store

Real Spark + Delta

pip install "pyspark>=3.5,<3.6" "delta-spark>=3.2,<3.3"
pytest -q tests/test_spark_smoke.py
# Batch jobs:
python -m lakehouse.spark_jobs silver <bronze_path> <silver_path>
python -m lakehouse.spark_jobs gold   <silver_path> <gold_path>

Modules

Module	Layer	Responsibility
cdc.py	bronze->silver	ACL: normalize legacy CDC, dedupe by version
quality.py	bronze->silver	Data quality expectations + quarantine
read_model.py	gold	CQRS balance projection (matches the Java projector)
features.py	gold	Per-account feature engineering
spark_jobs.py	all	PySpark + Delta medallion jobs (Kafka stream + batch)

Documentation

• System design + SLOs + capacity
• Industry applicability
• Business & governance: BRD - SOP - NFR - Cost savings
• ADRs: docs/adr/

Tech

Python 3.10+, PySpark 3.5, Delta Lake 3.2, Structured Streaming (Kafka), pytest. Pure-Python transform core.