Airflow 3: Get S3 logging to work

[sbernauer]

Description / Context

Remote logging (e.g. S3) does not work in Airflow 3.0.1.

Problem

The following pages illustrate resources where the external logging does not work:

• https://docs.stackable.tech/home/stable/airflow/usage-guide/using-kubernetes-executors/#_logging
• https://docs.stackable.tech/home/stable/airflow/usage-guide/using-kubernetes-executors/#_store_xcoms_objects_in_object_storage_e_g_s3

Customers need to be given a work-around or at least some indication when this feature will work in 3.x.

This was raised in

• https://github.com/apache/airflow/issues/52501 and
• https://github.com/apache/airflow/issues/50583
• and has been fixed in 3.0.4 via https://github.com/apache/airflow/pull/53719
• see also https://github.com/apache/airflow/issues/54603 and https://github.com/apache/airflow/pull/54679

However, the fix introduced a different bug that causes problems when using a custom logging config

• https://github.com/apache/airflow/issues/54414
• and fixed in https://github.com/apache/airflow/pull/54555
• this will probably be included in 3.0.6

Tested with 3.0.4 without our custom logging:

[lfrancke]

If 3.0.6 does not come out in time: Can we back port the patches using Patchable?

[adwk67]

If 3.0.6 does not come out in time: Can we back port the patches using Patchable?

We would need to extend Patchable to either patch in-place (after uv pip install), or so that patchable checkout covers all that the current command (uv pip install --no-cache-dir apache-airflow[${AIRFLOW_EXTRAS}]==${PRODUCT_VERSION} --constraint /tmp/constraints.txt --build-constraints /tmp/build-constraints.txt) is doing.

[adwk67]

Having looked at the code it looks as though there is a difference in behaviour between Airflow 2.x and 3.x:

• Airflow 2.x

  • REMOTE_TASK_LOG does not exist in the 2.x codebase and so remote logging is handled on the basis of remote logging env-vars

• Airflow 3.x

  • REMOTE_TASK_LOG has to be defined in custom logging, but if it is defined as None, this value is apparently not overridden later
  • REMOTE_TASK_LOG must thus be set to a valid remote logging instance in custom logging, which is difficult as remote logging can be defined outside of the operator
    • either reproduce internal Airflow code to instantiate all possible types of remote logging handling, based on the prefix of the remote logging base
    • or document that custom logging either has to be deactivated when using remote logging, or override the default custom logging with a separate config map in the resource yaml

[adwk67]

N.B. As a workaround, custom logging can be turned off by adding this env-var to the Airflow resource envOverrides:
AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS: ""

[adwk67]

It looks like this can be fixed by resolving REMOTE_TASK_LOG in our custom logging with this:

from airflow.config_templates.airflow_local_settings import REMOTE_TASK_LOG as RESOLVED_REMOTE_TASK_LOG
REMOTE_TASK_LOG = deepcopy(RESOLVED_REMOTE_TASK_LOG)

I'll open a PR and test it.

[adwk67]

Release notes snippet

Remote logging can now be used with Airflow 3.x in combination with the operator's custom logging. Previously (SDP 25.7) that would have been possible if custom logging is deactivated, but this is no longer necessary.