airflow 2.8.3 The scheduler does not appear to be running.

[yeangPan]

I installed airflow 2.8.3 at local machine, I changed the airflow.cfg located at ~/airflow/airflow.cfg. after successfully initializing db and creating admin user for it, I run the following command:

airflow webserver -D
airflow scheduler -D

both cmds successfully executed. below is the content of the airflow-scheduler.err

[2024-04-17 13:26:52 +0800] [393586] [INFO] Starting gunicorn 21.2.0
[2024-04-17 13:26:52 +0800] [393586] [INFO] Listening at: http://[::]:8793 (393586)
[2024-04-17 13:26:52 +0800] [393586] [INFO] Using worker: sync
[2024-04-17 13:26:52 +0800] [393587] [INFO] Booting worker with pid: 393587
[2024-04-17 13:26:52 +0800] [393588] [INFO] Booting worker with pid: 393588

when I go to the web UI, warning:

The scheduler does not appear to be running.

The DAGs list may not update, and new tasks will not be scheduled.

I started an example dag named example_bash_operator, the dag could't start.

ps -ef | grep scheduler
yangp     393585    1597  0 13:26 ?        00:00:00 /home/yangp/apps/anaconda3/envs/air/bin/python /home/yangp/apps/anaconda3/envs/air/bin/airflow scheduler -D
yangp     427298  123334  0 13:42 pts/19   00:00:00 grep --color=auto scheduler

netstat -tulpu | grep 8793
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6       0      0 [::]:8793               [::]:*                  LISTEN      393586/gunicorn: ma 

I tried to kill the scheduler with cmd, the scheduler process can't be killed:

(air) yangp@yangpan:~/airflow$ cat airflow-scheduler.pid 
393585
(air) yangp@yangpan:~/airflow$ ps -ef | grep scheduler
yangp     393585    1597  0 13:26 ?        00:00:00 /home/yangp/apps/anaconda3/envs/air/bin/python /home/yangp/apps/anaconda3/envs/air/bin/airflow scheduler -D
yangp     435225  123334  0 13:46 pts/19   00:00:00 grep --color=auto scheduler
(air) yangp@yangpan:~/airflow$ kill 393585
(air) yangp@yangpan:~/airflow$ ps -ef | grep scheduler
yangp     393585    1597  0 13:26 ?        00:00:00 /home/yangp/apps/anaconda3/envs/air/bin/python /home/yangp/apps/anaconda3/envs/air/bin/airflow scheduler -D
yangp     435499  123334  0 13:46 pts/19   00:00:00 grep --color=auto scheduler

after the I killed the process that use port 393586, the scheduler was really shut down.

(air) yangp@yangpan:~/airflow$ kill 393586
(air) yangp@yangpan:~/airflow$ ll
total 116
drwxrwxr-x  3 yangp yangp  4096  4月 17 13:48 ./
drwxr-x--x 52 yangp yangp  4096  4月 17 13:44 ../
-rw-------  1 yangp yangp 76167  4月 17 11:25 airflow.cfg
-rw-rw-r--  1 yangp yangp  1131  4月 17 13:48 airflow-scheduler.err
-rw-rw-r--  1 yangp yangp   125  4月 17 13:26 airflow-scheduler.log
-rw-rw-r--  1 yangp yangp   244  4月 17 13:26 airflow-scheduler.out
-rw-rw-r--  1 yangp yangp     0  4月 17 13:05 airflow-webserver.err
-rw-rw-r--  1 yangp yangp     0  4月 17 13:02 airflow-webserver.log
-rw-------  1 yangp yangp     7  4月 17 13:05 airflow-webserver-monitor.pid
-rw-rw-r--  1 yangp yangp     0  4月 17 13:05 airflow-webserver.out
-rw-r--r--  1 yangp yangp     7  4月 17 13:05 airflow-webserver.pid
drwxrwxr-x  3 yangp yangp  4096  4月 17 11:24 logs/
-rw-rw-r--  1 yangp yangp  4761  4月 17 11:24 webserver_config.py
(air) yangp@yangpan:~/airflow$ ll airflow-scheduler.pid
ls: cannot access 'airflow-scheduler.pid': No such file or directory
(air) yangp@yangpan:~/airflow$ ps -ef | grep scheduler
yangp     440594  123334  0 13:48 pts/19   00:00:00 grep --color=auto scheduler

the following is the info of installed airflow:

(air) yangp@yangpan:~/airflow$ airflow info

Apache Airflow
version                | 2.8.3                                                   
executor               | LocalExecutor                                           
task_logging_handler   | airflow.utils.log.file_task_handler.FileTaskHandler     
sql_alchemy_conn       | postgresql+psycopg2://airflow:wy123456@localhost/airflow
dags_folder            | /home/yangp/airflow/dags                                
plugins_folder         | /home/yangp/airflow/plugins                             
base_log_folder        | /home/yangp/airflow/logs                                
remote_base_log_folder |                                                         
                                                                                 

System info
OS              | Linux                                                                                                                                                          
architecture    | x86_64                                                                                                                                                         
uname           | uname_result(system='Linux', node='yangpan', release='6.5.0-1019-oem', version='#20-Ubuntu SMP PREEMPT_DYNAMIC Mon Mar 18 17:38:55 UTC 2024', machine='x86_64')
locale          | ('en_US', 'UTF-8')                                                                                                                                             
python_version  | 3.11.7 (main, Dec 15 2023, 18:12:31) [GCC 11.2.0]                                                                                                              
python_location | /home/yangp/apps/anaconda3/envs/air/bin/python                                                                                                                 
                                                                                                                                                                                 

Tools info
git             | git version 2.34.1                                                                         
ssh             | OpenSSH_8.9p1 Ubuntu-3ubuntu0.7, OpenSSL 3.0.2 15 Mar 2022                                 
kubectl         | NOT AVAILABLE                                                                              
gcloud          | NOT AVAILABLE                                                                              
cloud_sql_proxy | NOT AVAILABLE                                                                              
mysql           | mysql  Ver 8.0.36-0ubuntu0.22.04.1 for Linux on x86_64 ((Ubuntu))                          
sqlite3         | 3.41.2 2023-03-22 11:56:21 0d1fc92f94cb6b76bffe3ec34d69cffde2924203304e8ffc4155597af0c191da
psql            | psql (PostgreSQL) 16.2 (Ubuntu 16.2-1.pgdg22.04+1)                                         
                                                                                                             

Paths info
airflow_home    | /home/yangp/airflow                                                                                                                                                              
system_path     | /home/yangp/.local/bin:/home/yangp/bin:/home/yangp/.pyenv/bin:/home/yangp/.local/bin:/home/yangp/apps/anaconda3/envs/air/bin:/home/yangp/apps/anaconda3/condabin:/usr/lib/jvm/jdk
                | -21-oracle-x64/bin:/home/yangp/.local/bin:/home/yangp/bin:/usr/lib/jvm/jdk-21-oracle-x64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/g
                | ames:/snap/bin:/snap/bin                                                                                                                                                         
python_path     | /home/yangp/apps/anaconda3/envs/air/bin:/home/yangp/apps/anaconda3/envs/air/lib/python311.zip:/home/yangp/apps/anaconda3/envs/air/lib/python3.11:/home/yangp/apps/anaconda3/envs/
                | air/lib/python3.11/lib-dynload:/home/yangp/apps/anaconda3/envs/air/lib/python3.11/site-packages:/home/yangp/airflow/dags:/home/yangp/airflow/config:/home/yangp/airflow/plugins  
airflow_on_path | True                                                                                                                                                                             
                                                                                                                                                                                                   

Providers info
apache-airflow-providers-common-io  | 1.3.1 
apache-airflow-providers-common-sql | 1.12.0
apache-airflow-providers-ftp        | 3.8.0 
apache-airflow-providers-http       | 4.10.1
apache-airflow-providers-imap       | 3.5.0 
apache-airflow-providers-smtp       | 1.6.1 
apache-airflow-providers-sqlite     | 3.7.1 
apache-airflow-providers-ssh        | 3.10.1

when I start scheduler with non-daemon mode, the scheduler was started successfully, the web UI was normally started.

[4andy]

@yeangPan was this issue ever resolved? If so how. We are seeing the same issue at 2.9.3 after an upgrade. Thanks.