Skip to content

Commit 007bca0

Browse files
committed
added note on api workers
1 parent 138f333 commit 007bca0

1 file changed

Lines changed: 26 additions & 0 deletions

File tree

  • docs/modules/airflow/pages/troubleshooting

docs/modules/airflow/pages/troubleshooting/index.adoc

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,29 @@ webservers: # Also add to other roles!
2525
----
2626

2727
See e.g. https://github.com/minio/minio/issues/20845[this MinIO issue] for details.
28+
29+
== Setting API Workers
30+
31+
In Airflow the webserver (called the API Server in Airflow 3.x+) can use multiple workers.
32+
This is determined by the environment variable `+AIRFLOW__API__WORKERS+` and is set by default to `4` in Airflow 2.x and `1` in Airflow 3.x+.
33+
The reason for this difference is that Airflow uses a backend library to manage child processes and in 3.x+ this library can cause child processes to be killed if a hard-coded startup timeout is exceeded.
34+
For most cases a default of `1` should be sufficient, but if you run into performance issues and would like to add more workers, you can either change the environment variable using `envOverrides`:
35+
36+
[source,yaml]
37+
----
38+
webservers:
39+
envOverrides:
40+
AIRFLOW__API__WORKERS: 2 # something other than the default of 1
41+
----
42+
43+
or modulate multiple worker processes at the level of webserver, keeping the default of a single worker per webserver:
44+
45+
[source,yaml]
46+
----
47+
webservers:
48+
roleGroups:
49+
default:
50+
replicas: 2 # add a replica (with a single worker)
51+
----
52+
53+
TIP: Our recommendation is to increase the webserver replicas, with each webserver running a single worker, as this removes the risk of running into timeouts or memory issues.

0 commit comments

Comments
 (0)