[CONTENT] Reshaping: Converge into the layout of crate-clients-tools

docs/domain/index.md

@@ -0,0 +1,15 @@
+(domain)=
+(domains)=
+
+# Application Domains
+
+This section of the documentation includes tutorials and guidelines about
+how to use CrateDB optimally, related to different topic domains.
+
+
+```{toctree}
+:maxdepth: 3
+
+timeseries/index
+../integrate/ml/index
+```

docs/index.md

@@ -3,9 +3,9 @@
 (howtos)=
 (tutorials)=
 
-# CrateDB Guides and Tutorials
+# The CrateDB Guide
 
-About how to use CrateDB and CrateDB Cloud in practice.
+Guides and tutorials about how to use CrateDB and CrateDB Cloud in practice.
 
 CrateDB is a distributed and scalable SQL database for storing and analyzing
 massive amounts of data in near real-time, even with complex queries. It is
@@ -89,34 +89,71 @@ use-cases.
 
 
 
-
 ## Application Domains
 
-Native features of CrateDB paired with auxiliary software components provide
-optimal coverage of different application domains.
+Learn how to apply CrateDB's features to optimally cover use-cases in different
+application and topic domains.
 
 ::::{grid} 3
 :padding: 0
 
+:::{grid-item-card} Time Series Data
+:link: timeseries
+:link-type: ref
+:link-alt: Managing Time Series Data with CrateDB
+:padding: 3
+:text-align: center
+:class-card: sd-pt-3
+:class-body: sd-fs-1
+:class-title: sd-fs-5
+
+{material-outlined}`query_stats;1.3em`
+:::
 
-:::{grid-item-card} Analysis and Visualization
-:link: analysis
+
+:::{grid-item-card} Machine Learning
+:link: machine-learning
 :link-type: ref
-:link-alt: Data Analysis and Visualization with CrateDB
+:link-alt: Machine Learning with CrateDB
 :padding: 3
 :text-align: center
 :class-card: sd-pt-3
 :class-body: sd-fs-1
 :class-title: sd-fs-5
 
-{material-outlined}`analytics;1.3em`
+{material-outlined}`model_training;1.3em`
 :::
 
+::::
 
-:::{grid-item-card} Time Series Data
-:link: timeseries
+
+## Integrations
+
+Learn how to use CrateDB with 3rd-party software applications, libraries, and
+frameworks.
+
+::::{grid} 3
+:padding: 0
+
+
+:::{grid-item-card} ETL
+:link: etl
 :link-type: ref
-:link-alt: Managing Time Series Data with CrateDB
+:link-alt: Load and export data into/from CrateDB
+:padding: 3
+:text-align: center
+:class-card: sd-pt-3
+:class-body: sd-fs-1
+:class-title: sd-fs-5
+
+{material-outlined}`transform;1.3em`
+:::
+
+
+:::{grid-item-card} Metrics
+:link: metrics
+:link-type: ref
+:link-alt: CrateDB with metrics collection agents, brokers, and stores
 :padding: 3
 :text-align: center
 :class-card: sd-pt-3
@@ -127,6 +164,34 @@ optimal coverage of different application domains.
 :::
 
 
+:::{grid-item-card} Data Visualization
+:link: analysis
+:link-type: ref
+:link-alt: Data visualization with CrateDB
+:padding: 3
+:text-align: center
+:class-card: sd-pt-3
+:class-body: sd-fs-1
+:class-title: sd-fs-5
+
+{material-outlined}`bar_chart;1.3em`
+:::
+
+
+:::{grid-item-card} Business Intelligence
+:link: bi
+:link-type: ref
+:link-alt: Analyse information with CrateDB
+:padding: 3
+:text-align: center
+:class-card: sd-pt-3
+:class-body: sd-fs-1
+:class-title: sd-fs-5
+
+{material-outlined}`analytics;1.3em`
+:::
+
+
 :::{grid-item-card} Machine Learning
 :link: machine-learning
 :link-type: ref
@@ -141,23 +206,35 @@ optimal coverage of different application domains.
 :::
 
 
+:::{grid-item-card} Software Testing
+:link: testing
+:link-type: ref
+:link-alt: Software testing with CrateDB
+:padding: 3
+:text-align: center
+:class-card: sd-pt-3
+:class-body: sd-fs-1
+:class-title: sd-fs-5
+
+{material-outlined}`integration_instructions;1.3em`
+:::
+
+
 ::::
 
 
 
+
 ```{toctree}
 :hidden:
 
 install/index
 getting-started/index
 admin/index
-integrate/index
-
 reference-architectures/index
-topic/analysis/index
-topic/timeseries/index
-topic/ml/index
-topic/testing
+
+domain/index
+integrate/index
 ```
 
 
@@ -166,8 +243,13 @@ Please also visit the [Overview of CrateDB integration tutorials].
 :::
 
 :::{seealso}
-CrateDB and its documentation are open source projects. We host the source code and
-issue tracker on [GitHub].
+CrateDB and its documentation are open source projects.
+Contributions to the pages in this section and subsections are much appreciated.
+If you can spot a flaw, or would like to contribute additional content, you
+are most welcome.
+
+You will find corresponding links within the topmost right navigation element
+on each page, linking to the relevant page where this project is hosted on [GitHub].
 :::
 
 

docs/integrate/bi/index.md

@@ -0,0 +1,17 @@
+(analysis)=
+(bi)=
+
+# Business Intelligence
+
+Integrations of CrateDB with other tools, specifically related to business
+analytics and intelligence software.
+
+
+## PowerBI
+
+```{toctree}
+:maxdepth: 1
+
+powerbi-desktop
+powerbi-gateway
+```

docs/topic/analysis/powerbi-desktop.rst → docs/integrate/bi/powerbi-desktop.rst

@@ -1,4 +1,4 @@
-.. _cratedb-powerbi-desktop:
+.. _powerbi-desktop:
 
 =========================================
 Reports with CrateDB and Power BI Desktop

docs/topic/analysis/powerbi-gateway.rst → docs/integrate/bi/powerbi-gateway.rst

@@ -1,4 +1,4 @@
-.. _cratedb-powerbi-service:
+.. _powerbi-service:
 
 ===========================================
 Real Time Reports with CrateDB and Power BI

docs/integrate/etl/index.md

@@ -0,0 +1,149 @@
+(etl)=
+
+# Load and Export
+
+You have a variety of options to connect and integrate with 3rd-party
+ETL applications, mostly using [CrateDB's PostgreSQL interface].
+
+This documentation section lists corresponding ETL applications and
+frameworks which can be used together with CrateDB, and outlines how
+to use them optimally.
+
+
+## Apache Airflow / Astronomer
+
+A set of starter tutorials.
+
+- [Automating the import of Parquet files with Apache Airflow]
+- [Updating stock market data automatically with CrateDB and Apache Airflow]
+- [Automating stock data collection and storage with CrateDB and Apache Airflow]
+
+A set of elaborated tutorials, including blueprint implementations.
+
+- [Automating export of CrateDB data to S3 using Apache Airflow]
+- [Implementing a data retention policy in CrateDB using Apache Airflow]
+- [CrateDB and Apache Airflow: Building a data ingestion pipeline]
+- [Building a hot and cold storage data retention policy in CrateDB with Apache Airflow]
+
+Tutorials and resources about configuring the managed variants, Astro and CrateDB Cloud.
+
+- [ETL with Astro and CrateDB Cloud in 30min - fully up in the cloud]
+- [ETL pipeline using Apache Airflow with CrateDB (Source)]
+- [Run an ETL pipeline with CrateDB and data quality checks]
+
+
+## Apache Flink
+
+- {ref}`kafka-connect`
+- [Build a data ingestion pipeline using Kafka, Flink, and CrateDB]
+- [Community Day: Stream processing with Apache Flink and CrateDB]
+- [Executable stack: Apache Kafka, Apache Flink, and CrateDB]
+
+
+## Apache Kafka
+
+- [Data Ingestion using Kafka and Kafka Connect]
+- [Executable stack: Apache Kafka, Apache Flink, and CrateDB]
+- [Tutorial: Replicating data to CrateDB with Debezium and Kafka]
+
+
+## Azure Functions
+
+- {ref}`azure-functions`
+
+
+## dbt
+
+- [Using dbt with CrateDB]
+
+
+## Debezium
+
+- [Tutorial: Replicating data to CrateDB with Debezium and Kafka]
+- [Webinar: How to replicate data from other databases to CrateDB with Debezium and Kafka]
+
+
+## Kestra
+
+- [Setting up data pipelines with CrateDB and Kestra]
+
+
+## MongoDB
+
+- {ref}`integrate-mongodb`
+
+
+## MySQL
+
+- {ref}`integrate-mysql`
+
+
+## Node-RED
+
+- [Ingesting MQTT messages into CrateDB using Node-RED]
+- [Automating recurrent CrateDB queries using Node-RED]
+
+
+## Singer / Meltano
+
+- [meltano-target-cratedb]
+- [meltano-tap-cratedb]
+- [Examples about working with CrateDB and Meltano]
+
+🚧 _Please note these adapters are a work in progress._ 🚧
+
+
+## SQL Server Integration Services
+
+A demo project which uses SSIS and ODBC to read and write data from CrateDB:
+
+- [Using SQL Server Integration Services with CrateDB]
+
+
+## StreamSets
+
+- {ref}`streamsets`
+
+
+[Automating recurrent CrateDB queries using Node-RED]: https://community.cratedb.com/t/automating-recurrent-cratedb-queries/788
+[Automating export of CrateDB data to S3 using Apache Airflow]: https://community.cratedb.com/t/cratedb-and-apache-airflow-automating-data-export-to-s3/901
+[Automating stock data collection and storage with CrateDB and Apache Airflow]: https://community.cratedb.com/t/automating-stock-data-collection-and-storage-with-cratedb-and-apache-airflow/990
+[Automating the import of Parquet files with Apache Airflow]: https://community.cratedb.com/t/automating-the-import-of-parquet-files-with-apache-airflow/1247
+[Build a data ingestion pipeline using Kafka, Flink, and CrateDB]: https://dev.to/crate/build-a-data-ingestion-pipeline-using-kafka-flink-and-cratedb-1h5o
+[Building a hot and cold storage data retention policy in CrateDB with Apache Airflow]: https://community.cratedb.com/t/cratedb-and-apache-airflow-building-a-hot-cold-storage-data-retention-policy/934
+[Community Day: Stream processing with Apache Flink and CrateDB]: https://cratedb.com/blog/cratedb-community-day-2nd-edition-summary-and-highlights
+[CrateDB and Apache Airflow: Building a data ingestion pipeline]: https://community.cratedb.com/t/cratedb-and-apache-airflow-building-a-data-ingestion-pipeline/926 
+[CrateDB's PostgreSQL interface]: inv:crate-reference#interface-postgresql
+[Data Ingestion using Kafka and Kafka Connect]: https://cratedb.com/docs/crate/howtos/en/latest/integrations/kafka-connect.html
+[ETL pipeline using Apache Airflow with CrateDB (Source)]: https://github.com/astronomer/astro-cratedb-blogpost
+[ETL with Astro and CrateDB Cloud in 30min - fully up in the cloud]: https://www.astronomer.io/blog/run-etlelt-with-airflow-and-cratedb/
+[Examples about working with CrateDB and Meltano]: https://github.com/crate/cratedb-examples/tree/amo/meltano/framework/singer-meltano
+[Executable stack: Apache Kafka, Apache Flink, and CrateDB]: https://github.com/crate/cratedb-examples/tree/main/application/apache-kafka-flink
+[Implementing a data retention policy in CrateDB using Apache Airflow]: https://community.cratedb.com/t/implementing-a-data-retention-policy-in-cratedb-using-apache-airflow/913 
+[Ingesting MQTT messages into CrateDB using Node-RED]: https://community.cratedb.com/t/ingesting-mqtt-messages-into-cratedb-using-node-red/803
+[meltano-tap-cratedb]: https://github.com/crate-workbench/meltano-tap-cratedb
+[meltano-target-cratedb]: https://github.com/crate-workbench/meltano-target-cratedb
+[Run an ETL pipeline with CrateDB and data quality checks]: https://registry.astronomer.io/dags/etl_pipeline/
+[Setting up data pipelines with CrateDB and Kestra]: https://community.cratedb.com/t/setting-up-data-pipelines-with-cratedb-and-kestra-io/1400
+[Tutorial: Replicating data to CrateDB with Debezium and Kafka]: https://community.cratedb.com/t/replicating-data-to-cratedb-with-debezium-and-kafka/1388
+[Updating stock market data automatically with CrateDB and Apache Airflow]: https://community.cratedb.com/t/updating-stock-market-data-automatically-with-cratedb-and-apache-airflow/1304
+[Using dbt with CrateDB]: https://community.cratedb.com/t/using-dbt-with-cratedb/1566
+[Using SQL Server Integration Services with CrateDB]: https://github.com/crate/cratedb-examples/tree/main/application/microsoft-ssis
+[Webinar: How to replicate data from other databases to CrateDB with Debezium and Kafka]: https://cratedb.com/resources/webinars/lp-wb-debezium-kafka
+
+
+
+```{toctree}
+:hidden:
+
+mongodb
+mysql
+```
+
+```{toctree}
+:hidden:
+
+kafka-connect
+azure-functions
+streamsets
+```