Some docs

Author: sbernauer

docs/modules/airflow/pages/getting_started/first_steps.adoc

@@ -10,9 +10,10 @@ With the external dependencies required by Airflow (Postgresql and Redis) instal
 
 Supported versions for PostgreSQL and Redis can be found in the https://airflow.apache.org/docs/apache-airflow/stable/installation/prerequisites.html#prerequisites[Airflow documentation].
 
-== Secret with Airflow credentials
+== Airflow secrets
 
-Create a Secret with the necessary credentials, this entails database connection credentials as well as an admin account for Airflow itself.
+Secrets are required for the mandatory metadata database connection and the Airflow admin user.
+When using celery executors it's also required to provide information for the celery database and broker.
 Create a file called `airflow-credentials.yaml`:
 
 [source,yaml]
@@ -23,15 +24,14 @@ And apply it:
 [source,bash]
 include::example$getting_started/code/getting_started.sh[tag=apply-airflow-credentials]
 
-`connections.sqlalchemyDatabaseUri` must contain the connection string to the SQL database storing the Airflow metadata.
+`airflow-postgresql-credentials` contains credentials for the SQL database storing the Airflow metadata.
+In this example we will use the same database for both the Airflow job metadata as well as the Celery broker metadata.
 
-`connections.celeryResultBackend` must contain the connection string to the SQL database storing the job metadata (the example above uses the same PostgreSQL database for both).
+`airflow-redis-credentials` contains credentials for the the Redis instance used for queuing the jobs submitted to the Airflow executor(s).
 
-`connections.celeryBrokerUrl` must contain the connection string to the Redis instance used for queuing the jobs submitted to the airflow executor(s).
+`airflow-admin-credentials`: the `adminUser.*` fields are used to create the initial admin user.
 
-The `adminUser` fields are used to create an admin user.
-
-NOTE: The admin user is disabled if you use a non-default authentication mechanism like LDAP.
+NOTE: The admin user is disabled if you use a non-default authentication mechanism like LDAP or OIDC.
 
 == Airflow
 
@@ -60,21 +60,27 @@ include::example$getting_started/code/getting_started.sh[tag=install-airflow]
 
 Where:
 
-* `metadata.name` contains the name of the Airflow cluster.
+* `metadata.name`: contains the name of the Airflow cluster.
+* `spec.clusterConfig.metadataDatabase`: specifies one of the supported database types (in this case, `postgresql`) along with references to the host, database and the secret containing the connection credentials.
 * the product version of the Docker image provided by Stackable must be set in `spec.image.productVersion`.
 * `spec.celeryExecutors`: deploy executors managed by Airflow's Celery engine.
 Alternatively you can use `kuberenetesExectors` that use Airflow's Kubernetes engine for executor management.
 For more information see https://airflow.apache.org/docs/apache-airflow/stable/executor/index.html#executor-types).
-* the `spec.clusterConfig.loadExamples` key is optional and defaults to `false`.
+* `spec.celeryExecutors.celeryResultBackend`: specifies one of the supported database types (in this case, `postgresql`) along with references to the host, database and the secret containing the connection credentials.
+* `spec.celeryExecutors.celeryBroker`: specifies one of the supported queue/broker types (in this case, `redis`) along with references to the host and the secret containing the connection credentials.
+* `spec.clusterConfig.loadExamples`: this key is optional and defaults to `false`.
 It is set to `true` here as the example DAGs are used when verifying the installation.
-* the `spec.clusterConfig.exposeConfig` key is optional and defaults to `false`. It is set to `true` only as an aid to verify the configuration and should never be used as such in anything other than test or demo clusters.
-* the previously created secret must be referenced in `spec.clusterConfig.credentialsSecret`.
+* `spec.clusterConfig.exposeConfig`: this key is optional and defaults to `false`.
+It is set to `true` only as an aid to verify the configuration and should never be used as such in anything other than test or demo clusters.
+* `spec.clusterConfig.credentialsSecret`: specifies the secret containing the Airflow admin user information.
 
 NOTE: The version you need to specify for `spec.image.productVersion` is the desired version of Apache Airflow.
 You can optionally specify the `spec.image.stackableVersion` to a certain release like `23.11.0` but it is recommended to leave it out and use the default provided by the operator.
 Check our https://oci.stackable.tech/[image registry,window=_blank] for a list of available versions. Information on how to browse the registry can be found xref:contributor:project-overview.adoc#docker-images[here,window=_blank].
 It should generally be safe to simply use the latest version that is available.
 
+NOTE: Refer to xref:usage-guide/database-connections.adoc[] for more information about database/broker connections.
+
 This creates the actual Airflow cluster.
 
 After a while, all the Pods in the StatefulSets should be ready:

docs/modules/airflow/pages/usage-guide/database-connections.adoc

@@ -0,0 +1,73 @@
+= Database connections
+:description: Configure Airflow Database connectivity.
+
+Airflow requires a metadata database for storing e.g. DAG, task and job data.
+The actual connection string is calculated by the operator so that the user does not need to remember the exact structure.
+
+== Typed connections
+
+[source,yaml]
+----
+spec:
+  clusterConfig:
+    metadataDatabase:
+      postgresql: # <1>
+        host: airflow-postgresql
+        database: airflow
+        credentialsSecret: airflow-postgresql-credentials # <2>
+----
+<1> A reference to one of the supported database backends (e.g. `postgresql`).
+<2> A reference to a Secret which must contain the two fields `username` and `password`.
+
+The queue/broker metadata and URL is only needed when running the celery executor.
+The `celeryResultBackend` definition uses the same structure as `metadataDatabase` shown above.
+The `celeryBroker` definition is similar but does not require a `databaseName`.
+
+[source,yaml]
+----
+spec:
+  celeryExecutors:
+    celeryResultBackend:
+      postgresql: # <1>
+        host: airflow-postgresql
+        database: airflow
+        credentialsSecret: airflow-postgresql-credentials # <2>
+    celeryBroker:
+      redis: # <3>
+        host: airflow-redis-master
+        credentialsSecret: airflow-redis-credentials # <2>
+----
+<1> A reference to one of the supported database backends (e.g. `postgresql`).
+<2> A reference to a secret which must contain the two fields `username` and `password`.
+<3> A reference to one of the supported queue brokers (e.g. `redis`).
+
+== Generic connections
+
+Alternatively, these connections can also be defined in full in a referenced secret:
+
+[source,yaml]
+----
+spec:
+  clusterConfig:
+    metadataDatabase:
+      generic:
+        uriSecret: postgresql-metadata # <1>
+----
+
+[source,yaml]
+----
+spec:
+  celeryResultBackend:
+      generic:
+        uriSecret: postgresql-celery # <2>
+    celeryBroker:
+      generic:
+        uriSecret: redis-celery # <3>
+----
+
+<1> A reference to a secret which must contain the single fields `uri` e.g.
+`uri: postgresql+psycopg2://airflow:airflow@airflow-postgresql/airflow`
+<2> A reference to a secret which must contain the single fields `uri` e.g.
+`uri: db+postgresql://airflow:airflow@airflow-postgresql/airflow`
+<3> A reference to a secret which must contain the single fields `uri` e.g.
+`uri: redis://:redis@airflow-redis-master:6379/0`

docs/modules/airflow/partials/nav.adoc

@@ -4,6 +4,7 @@
 * xref:airflow:required-external-components.adoc[]
 * xref:airflow:usage-guide/index.adoc[]
 ** xref:airflow:usage-guide/db-init.adoc[]
+** xref:airflow:usage-guide/database-connections.adoc[]
 ** xref:airflow:usage-guide/mounting-dags.adoc[]
 ** xref:airflow:usage-guide/applying-custom-resources.adoc[]
 ** xref:airflow:usage-guide/listenerclass.adoc[]