Fix logging in Airflow 3.1.6

[siegfriedweber]

Airflow 3.1.6 does not log into log files anymore. Logging was switched from stdlib logging to structlog in Airflow 3.1.0 (apache/airflow#52651). Now it is not configurable anymore the same way as before.

The documentation of the log configuration is outdated and not valid anymore.

The doc is out of date, [...].

In Airflow 3 the mechanism to configure logging has changed, and things should be implemented as a structlog processor, and user control of the full logging config is not possible anymore. Sorry.

Yes this is a problem that needs addressing, [...]

apache/airflow#60384 (comment)

It seems that a patch for Airflow is required to get the logs working again. Changes could be necessary at the following places:

• https://github.com/apache/airflow/blob/3.1.6/airflow-core/src/airflow/logging_config.py#L86
• https://github.com/apache/airflow/blob/3.1.6/shared/logging/src/airflow_shared/logging/structlog.py#L379

Tasks

• fix: Allow overriding the logging configuration in Airflow 3.1 #757
• fix(airflow): Allow overriding the logging configuration in Airflow 3.1 docker-images#1445

[siegfriedweber]

We decided in the daily meeting to postpone this issue for a bit and hope for a solution upstream.

[soenkeliebau]

Possible relevant upstream issue: apache/airflow#61175

Also: apache/airflow#61257