Airflow Demo: Extend to show 3.x new features

[adwk67]

Description

Airflow 3.x brings with it significant changes to the architecture (UI, services, auth backends etc.) as well as new features. These (new features) do not figure in our demo and it would be good to showcase some of them e.g.

• run a deferred DAG using the triggerer service
• illustrate using a DAG bundle
• incorporate Opa authorisation

Motivation

• tests things not covered by integration tests
• used as a basis for answering support questions
• motivate users to migrate to 3.x+

PRs

• feat(airflow): Add apache-kafka and update queue/messaging libs docker-images#1313
• feat: Airflow with Event-scheduling, Triggerer, Opa #320
• docs: Add troubleshooting items airflow-operator#701

[lfrancke]

During refinement decide the scope of the demo and the new features to demonstrate.

[adwk67]

Suggested Scope

See e.g. the 3.0.0 release notes for major changes.
Use the latest product image (3.1.0) for the demo.

1. Run a deferred DAG using the triggerer service

There is an integration test that uses the triggerer, and then can used as a basis for mounting a DAG that makes use of TimeDeltaTrigger (or similar), illustrating a) delayed DAG runs, and maybe b) DAG runs that persist across a re-start.

• mount a deferrable DAG and show triggerer use (S)

2. DAG bundles

The documentation is a little thin regarding DAG bundles, but they are basically intended to extend previous the DAGS_FOLDER (allowing DAGs to be fetched from an S3 bucket or a git repo). We already have gitsync integration (via gitsync in our images) so it does not make much sense to use another git mechanism.

• write a custom DAG bundle and show how its versions are displayed in the UI (S)
• mount a DAG to S3 and load it from there (M)

3. Validate DAGs with ruff prior to migration

• show one or two examples of checking DAGs, similar to here (S)

4. OPA authorizer

We also have an integration test and documentation for this, so the demo would not need to do anything "new" but present existing functionality in a more step-by-step manner (also mentioning how the plugin works).

• use the existing docs to display the same in a demo (S)

5. Event-driven DAG runs

It is now possible for DAGs to react to events external to Airflow. There is an out-of-the-box example available with 3.0, but it would be nice to Stackable-ize this and show Airflow reacting to e.g. a message being written from Nifi to Kafka.

• (Nifi -->) Kafka --> start a DAG (M/L)

6. (Optional): HITL workflow

See https://airflow.apache.org/blog/airflow-3.1.0/#-human-in-the-loop-hitl-when-automation-meets-human-judgment. Can't really think of a stackable-specific example for this... (M/L)

[lfrancke]

@adwk67 and I looked at the items together and decided on this plan.

• Do (in order) items 5, 1 and 4 (integrate them into a demo)
• Create a troubleshooting article for 3
• Leave out/skip 2 and 6 (at least for now)

Maintaining demos is a lot of work so we've tried focusing on things that highlight our platform and not just demonstrate Airflow features.