Oil Data Back has two parts: a read API (app/) and a replayable telemetry pipeline (pipeline/). The visual map on the overview page is the short version; this page explains what each boundary owns. The full design lives in design/specs.
generator → Kafka telemetry.raw
→ Spark raw stream → Bronze Parquet (append-only, fixed schema)
→ Spark materializer → versioned snapshot: Silver, rejects, conflicts, hourly + daily marts
→ active.json manifest → verify command
UTF-8 JSON keyed by well_id. Schema version 1 has event_id, source_id, well_id, event_time, emitted_at and four measurements; version 2 adds the optional pump_current_a. Malformed, unsupported or out-of-range events are never dropped: they are rejected with a reason and the raw bytes are kept.
Spark Structured Streaming copies each Kafka record (value, topic, partition, offset, Kafka timestamp, ingestion time) into Parquet partitioned by ingestion date, so an old event_time is still accepted. A missing offset fails the query instead of being skipped.
A pure function from Bronze rows to datasets. One Silver row per event_id; identical repeats collapse, differing repeats are excluded and recorded as conflicts. Hourly and daily marts group by UTC event_time and well_id. Flow rate is averaged, never summed.
Every micro-batch rebuilds all datasets from all committed Bronze files, so a late event corrects its original hour and day. The result is validated in a staging directory, renamed to snapshots/<id>, and only then exposed by atomically replacing active.json. Snapshot ids derive from the Spark batch_id, which makes a retried foreachBatch idempotent. A file lock serializes the materializer and replay.
Reads only Parquet and active.json — no transformation code — and exits nonzero if counts or aggregates differ from the expectation.
app/ serves the marts over HTTP: /api/v1/snapshot, /wells and /marts/{hourly|daily}. It resolves active.json per request, reads only Parquet (no Spark, no pipeline imports) and retries once if a publish prunes the snapshot it just resolved. Responses are tagged with their snapshot_id; before the first publish it answers 503. Blocking reads run in FastAPI's threadpool. The async SQLAlchemy foundation stays unused until something needs a relational store.
docs/ is a standalone static site published through GitHub Pages; it does not call the API yet.
Materialization rereads all of Bronze each time. That is deliberate: it favors clear correction and replay semantics for a bounded local fixture. Partition-scoped rebuilds are the next optimization if the data grows. Kafka retention (7 days locally) must exceed any planned processor outage; events that expire before raw ingestion cannot be reconstructed.
- Keep the API contract versioned from the beginning.
- Never silently drop data: reject with a reason or fail visibly.
- Make correction and replay deterministic.
- Be honest about what is simulated and what is wired.