You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/modules/airflow/pages/troubleshooting/index.adoc
+26Lines changed: 26 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -25,3 +25,29 @@ webservers: # Also add to other roles!
25
25
----
26
26
27
27
See e.g. https://github.com/minio/minio/issues/20845[this MinIO issue] for details.
28
+
29
+
== Setting API Workers
30
+
31
+
In Airflow the webserver (called the API Server in Airflow 3.x+) can use multiple workers.
32
+
This is determined by the environment variable `+AIRFLOW__API__WORKERS+` and is set by default to `4` in Airflow 2.x and `1` in Airflow 3.x+.
33
+
The reason for this difference is that Airflow uses a backend library to manage child processes and in 3.x+ this library can cause child processes to be killed if a hard-coded startup timeout is exceeded.
34
+
For most cases a default of `1` should be sufficient, but if you run into performance issues and would like to add more workers, you can either change the environment variable using `envOverrides`:
35
+
36
+
[source,yaml]
37
+
----
38
+
webservers:
39
+
envOverrides:
40
+
AIRFLOW__API__WORKERS: 2 # something other than the default of 1
41
+
----
42
+
43
+
or modulate multiple worker processes at the level of webserver, keeping the default of a single worker per webserver:
44
+
45
+
[source,yaml]
46
+
----
47
+
webservers:
48
+
roleGroups:
49
+
default:
50
+
replicas: 2 # add a replica (with a single worker)
51
+
----
52
+
53
+
TIP: Our recommendation is to increase the webserver replicas, with each webserver running a single worker, as this removes the risk of running into timeouts or memory issues.
0 commit comments