dbx_ingestion_monitoring

[cbotev-databricks]

This directory contains common code and DABs to deploy observability ETL and dashboards for Databricks ingestion projects. The goal is to provide an example and a starting point for building ingestion observability across pipelines and datasets.

In particular, the package provides:

• Tools to ETL observability data from a variety of sources such as SDP event log, Auto Loader cloud_file_states, system tables and other.
• Tag-based pipeline discovery: Specify pipelines to monitor using flexible tag expressions with OR-of-ANDs logic (e.g., "tier:T0;team:data,tier:T1") instead of maintaining lists of pipeline IDs
• Build a collection of observability tables on top of the above data using the medallion architecture.
• Provide out-of-the-box AI/BI Dashboards based on the above observability tables
• Code and examples to integrate the observability tables with third-party monitoring providers such as Datadog, New Relic, Azure Monitor, Splunk

Currently Generic SDP pipelines and Lakeflow CDC Connector pipelines are supported.