A curated list of awesome tools, frameworks, and resources for the Modern Data Stack (MDS).
The Modern Data Stack refers to a collection of cloud-native tools and technologies that work together to help organizations collect, store, transform, and analyze data. This list aims to help data professionals navigate the ever-evolving data ecosystem.
- Data Integration & Ingestion
- Data Warehouses & Lakehouses
- Data Lakes & Storage
- Data Transformation
- Data Orchestration
- Data Quality & Testing
- Data Observability
- Data Catalog & Discovery
- Data Governance
- Business Intelligence & Analytics
- Metrics Layer & Semantic Layer
- Reverse ETL
- Data Contracts
- Streaming & Real-Time
- Query Engines
- Data Notebooks & Exploration
- Feature Stores
- ML Platforms & MLOps
- Vector Databases
- Data Privacy & Security
- Data Sharing
- DataOps & Version Control
- Learning Resources
- Community
- querybear.com - Ask your data anything – with persistent memory, schema learning, and more.
Tools for extracting data from various sources and loading it into data warehouses or lakes.
- Airbyte - Open-source data integration platform with 300+ connectors. ELT-first approach.
- dlt (data load tool) - Python library for data loading with automatic schema inference.
- Singer - Open-source standard for writing scripts that move data (taps and targets).
- Meltano - Open-source DataOps platform built on Singer. CLI-first, version-controlled pipelines.
- Ingestr - CLI tool to copy data between databases with a single command.
- Boomi - Modern data integration and agent management platform.
- Fivetran - Automated data integration with 500+ connectors. Industry leader in managed ELT.
- Stitch - Simple, extensible ETL built for developers. Part of Talend.
- Hevo Data - No-code data pipeline platform.
- Informatica - Enterprise data integration and management platform.
- Talend - Enterprise data integration suite.
Cloud data warehouses and lakehouse platforms for storing and querying analytical data.
- Snowflake - Cloud-native data warehouse with separation of storage and compute.
- Google BigQuery - Serverless, highly scalable data warehouse by Google.
- Amazon Redshift - Fast, scalable data warehouse by AWS.
- Azure Synapse Analytics - Limitless analytics service by Microsoft.
- Databricks SQL - Serverless SQL analytics on the Lakehouse.
- Firebolt - Cloud data warehouse built for high-performance analytics.
- ClickHouse Cloud - Managed ClickHouse for real-time analytics.
- StarRocks - High-performance analytical database.
- MotherDuck - Serverless analytics powered by DuckDB.
- Databricks - Unified analytics platform combining data lake and warehouse.
- Apache Iceberg - Open table format for huge analytic datasets.
- Delta Lake - Open-source storage layer with ACID transactions on data lakes.
- Apache Hudi - Data lake platform for incremental data processing.
- Dremio - Lakehouse platform with Apache Iceberg support.
- Onehouse - Managed lakehouse platform built on Apache Hudi.
- Tabular - Managed Apache Iceberg service by its creators.
Object storage and data lake solutions for storing raw and processed data.
- Amazon S3 - Industry-standard object storage by AWS.
- Google Cloud Storage - Object storage by Google Cloud.
- Azure Blob Storage - Object storage by Microsoft Azure.
- MinIO - High-performance, S3-compatible object storage.
- Cloudflare R2 - S3-compatible storage with zero egress fees.
- Apache Parquet - Columnar storage file format.
- Apache ORC - Columnar storage format for Hadoop.
- Apache Avro - Row-based data serialization format.
- Lance - Modern columnar format for ML datasets.
Tools for transforming, modeling, and preparing data for analysis.
- dbt (data build tool) - Industry-standard SQL-first transformation tool. Version-controlled, tested, documented.
- SQLMesh - Data transformation framework with built-in data quality and CI/CD.
- SDF (Semantic Data Fabric) - SQL compiler for data transformation with static analysis.
- Coalesce - Data transformation platform purpose-built for Snowflake.
- Dataform - SQL-based data transformation (acquired by Google).
- Pandas - Data manipulation and analysis library for Python.
- Polars - Lightning-fast DataFrame library written in Rust.
- DuckDB - In-process SQL OLAP database. Perfect for local data transformation.
- PySpark - Python API for Apache Spark.
- Vaex - Out-of-core DataFrames for large datasets.
- Modin - Parallel pandas using Ray or Dask.
- Ibis - Python DataFrame API that compiles to SQL.
- Fugue - Unified interface for distributed computing.
- Hamilton - Micro-framework for dataframe generation.
- Apache Spark - Unified analytics engine for large-scale data processing.
- Dask - Flexible parallel computing library for analytics.
- Ray - Unified framework for scaling AI and Python applications.
- Apache Flink - Stream and batch processing framework.
- Apache Beam - Unified programming model for batch and streaming.
- Asgarde - Java library for simplified error handling in Beam pipelines.
Tools for scheduling, monitoring, and managing data pipelines and workflows.
- Apache Airflow - Platform to programmatically author, schedule, and monitor workflows.
- Dagster - Cloud-native orchestration platform with software-defined assets.
- Prefect - Modern workflow orchestration with Python-native approach.
- Mage - Open-source data pipeline tool with notebook-style interface.
- Kestra - Event-driven orchestration platform with declarative YAML.
- Luigi - Python module for building complex pipelines (by Spotify).
- Argo Workflows - Kubernetes-native workflow engine.
- Astronomer - Managed Airflow platform.
- Dagster Cloud - Managed Dagster platform.
- Prefect Cloud - Managed Prefect platform.
- Google Cloud Composer - Managed Apache Airflow by Google.
- Amazon MWAA - Managed Airflow by AWS.
- Azure Data Factory - Cloud-based data integration service.
- Orchestra - Unified data orchestration platform.
Tools for ensuring data accuracy, completeness, and reliability.
- Great Expectations - Python library for data validation and documentation.
- dbt Tests - Built-in testing framework in dbt.
- Elementary - Open-source data observability for dbt.
- Provero - Vendor-neutral, declarative data quality engine with YAML-based checks.
- Soda - Data quality platform with SodaCL language.
- Monte Carlo - Data observability platform with ML-powered anomaly detection.
Tools for monitoring, alerting, and understanding data pipeline health.
- Datadog - Monitoring and security platform for developers.
- Elementary - Open-source dbt-native data observability.
- Datafold - Data reliability platform with data diff and regression testing.
- Sifflet - Full-stack data observability platform.
Tools for discovering, understanding, and documenting data assets.
- DataHub - Open-source metadata platform by LinkedIn.
- Amundsen - Open-source data discovery platform by Lyft.
- OpenMetadata - Open-source metadata platform with discovery and governance.
- Atlan - Active metadata platform with collaboration features.
- Alation - Enterprise data intelligence platform.
- Collibra - Data intelligence platform with governance and catalog.
Tools for managing data policies, access, compliance, and security.
- Collibra - Comprehensive data governance platform.
- Alation - Data catalog with governance capabilities.
Tools for data visualization, reporting, and self-service analytics.
- Apache Superset - Modern data exploration and visualization platform.
- Metabase - Open-source business intelligence with SQL and visual query builder.
- Redash - Connect and query data sources, build dashboards.
- Lightdash - Open-source BI for dbt users.
- Evidence - Code-based BI with Markdown and SQL.
- Grafana - Open-source analytics and monitoring platform.
- stratif.io - Open-source warehouse-native product analytics for DuckDB, Postgres, Snowflake, and ClickHouse.
- Looker - Enterprise BI with semantic modeling (Google Cloud).
- Tableau - Visual analytics platform (Salesforce).
- Power BI - Business analytics by Microsoft.
- Qlik - Data analytics and business intelligence platform.
- Preset - Managed Apache Superset.
- Omni - Shared-model BI platform.
- Holistics - Self-service BI platform with data modeling.
Tools for defining consistent business metrics and semantic models.
- dbt Semantic Layer - Define metrics in dbt, query from any tool.
- Cube - Headless BI and semantic layer with caching.
- MetricFlow - Semantic layer engine (now part of dbt).
- Looker - LookML semantic modeling language.
- Transform (Acquired by dbt Labs) - Metrics store platform.
- Minerva (Airbnb) - Airbnb's internal metrics platform.
Tools for syncing data from warehouses back to operational tools.
- Census - Operational analytics platform for syncing data to business tools.
- Hightouch - Data activation platform for reverse ETL.
- Polytomic - Sync data bidirectionally between databases and SaaS.
Tools and frameworks for defining and enforcing data contracts between producers and consumers.
- Soda Data Contracts - Define and verify data contracts.
- Bitol - Open-source data contract specification.
- DataContract CLI - CLI for managing data contracts.
- Great Expectations - Can be used for contract-like validations.
- JSON Schema - Schema specification for JSON data.
- Protobuf - Protocol Buffers for schema definition.
- Apache Avro - Data serialization with schema evolution.
Tools for real-time data processing and event streaming.
- Apache Kafka - Distributed event streaming platform.
- Confluent - Enterprise Kafka platform with managed cloud.
- Amazon Kinesis - Real-time streaming data service by AWS.
- Google Pub/Sub - Messaging service by Google Cloud.
- Azure Event Hubs - Big data streaming platform by Azure.
- RabbitMQ - Open-source message broker.
- Apache Flink - Stateful stream processing framework.
- Apache Kafka Streams - Client library for stream processing.
- ksqlDB - Database for stream processing on Kafka.
- Apache Spark Streaming - Spark module for stream processing.
- Boomi Data Integration - Automated CDC for operational databases and apps.
- Debezium - Open-source distributed CDC platform.
- Airbyte CDC - CDC connectors in Airbyte.
- Fivetran CDC - Managed CDC by Fivetran.
Distributed SQL query engines for data lakes and federated queries.
- PrestoDB - Distributed SQL query engine by Meta.
- Apache Drill - Schema-free SQL query engine.
- Apache Impala - Massively parallel SQL query engine.
- DuckDB - In-process analytical database.
- ClickHouse - Column-oriented DBMS for OLAP.
- Apache Druid - Real-time analytics database.
Interactive environments for data exploration and analysis.
- Jupyter - Web-based interactive computing platform.
- JupyterLab - Next-generation Jupyter interface.
- Zeppelin - Web-based notebook for data analytics.
- Marimo - Reactive Python notebook with reproducibility.
- Streamlit - Python framework for data apps.
- Google Colab - Free Jupyter notebooks by Google.
- Amazon SageMaker Studio - ML IDE by AWS.
- Databricks Notebooks - Collaborative notebooks on Databricks.
- Hex - Collaborative data workspace with notebooks and apps.
- Deepnote - Collaborative data notebook for teams.
Platforms for managing, storing, and serving ML features.
- Vertex AI Feature Store - Google Cloud feature store.
- Amazon SageMaker Feature Store - AWS managed feature store.
- Databricks Feature Store - Feature store in Databricks.
- Feast - Open-source feature store for ML.
- Hopsworks - Platform with feature store and MLOps.
End-to-end platforms for building, deploying, and managing ML models.
- MLflow - Open-source platform for ML lifecycle management.
- Kubeflow - ML toolkit for Kubernetes.
- Metaflow - Framework for real-life data science (by Netflix).
- ZenML - MLOps framework for reproducible pipelines.
- Google Vertex AI - Unified ML platform by Google Cloud.
- Databricks MLflow - Managed MLflow on Databricks.
- Amazon SageMaker - Fully managed ML service by AWS.
- Azure Machine Learning - ML platform by Microsoft.
- Weights & Biases - ML experiment tracking and model management.
Databases optimized for storing and querying vector embeddings for AI/ML applications.
- MongoDB - Managed vector store alognside your operational data in MongoDB Atlas.
- Pinecone - Managed vector database for similarity search.
- Weaviate - Open-source vector search engine.
- Qdrant - Open-source vector similarity search engine.
- Chroma - Open-source embedding database.
- pgvector - Vector similarity search for PostgreSQL.
- Elasticsearch - Search engine with vector search capabilities.
Platforms for securely sharing data across organizations.
- Google Analytics Hub - Data exchange by Google Cloud.
- Snowflake Data Marketplace - Data sharing and marketplace by Snowflake.
- Databricks Delta Sharing - Open protocol for secure data sharing.
- AWS Data Exchange - Data marketplace by AWS.
Tools for DataOps practices, version control, and CI/CD for data.
- Fundamentals of Data Engineering - by Joe Reis & Matt Housley
- The Data Warehouse Toolkit - by Ralph Kimball
- Designing Data-Intensive Applications - by Martin Kleppmann
- Data Mesh - by Zhamak Dehghani
- Building a Scalable Data Warehouse with Data Vault 2.0 - by Dan Linstedt
- dbt Learn - Free dbt fundamentals course.
- DataCamp - Data science and engineering courses.
- DataTalks.Club - Free data engineering zoomcamp.
- Coursera Data Engineering - Google Cloud Data Engineering certificate.
- Snowflake Training - Snowflake certifications.
- Databricks Training - Databricks certifications.
- Blef.fr - The hub to explore Data News links.
- Joe Reis's Substack - Insights on data engineering.
- dbt Blog - Articles on analytics engineering.
- Airbyte Blog - Data integration and engineering content.
- DataStackGuide - Independent reviews of B2B data stack tools (CRMs, enrichment, BI, sales engagement) backed by analysis of 23,000+ job postings.
- DataGen - Interviews with French data practitioners & leaders.
- The Data Engineering Podcast - Interviews with data engineering practitioners.
- Data Engineering Show - Discussions on data engineering topics.
- dbt Community Slack - 50k+ members discussing analytics engineering.
- Airbyte Slack - Data integration community.
- Apache Airflow Slack - Workflow orchestration.
- Coalesce - dbt's annual conference.
- Snowflake Summit - Snowflake user conference.
Comming soon...
- r/dataengineering - Reddit community.
- Data Engineering Wiki - Community-maintained wiki.
- Awesome Data Engineering - Another curated list.
- Modern Data Stack Glossary - Data terminology.
Contributions are welcome! Please read the contribution guidelines first.
To the extent possible under law, the contributors have waived all copyright and related or neighboring rights to this work.
This list is maintained by the community. Special thanks to all contributors who help keep it up-to-date.
If you find this resource helpful, please give it a star and share it with others!