Workers crash when stopping distributed test

[cooperbenson-qz]

Prerequisites

• I am using the latest version of Locust
• I am reporting a bug, not asking a question

Description

When stopping a load test in distributed mode the workers emit two exceptions and then after 60s die because they haven't received a heartbeat from the master. This causes the UI and master to believe the test hasn't stopped and everything breaks.

Worker log

[2025-03-03 20:00:33,905] locust-worker-c955f5bfb-h6b7v/INFO/locust.main: Starting Locust 2.33.0
[2025-03-03 20:38:13,263] locust-worker-c955f5bfb-h6b7v/ERROR/locust.user.task: expected string or bytes-like object, got 'NoneType'
Traceback (most recent call last):
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1125, in _rollback_impl
    self.engine.dialect.do_rollback(self.connection)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/default.py", line 700, in do_rollback
    dbapi_connection.rollback()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/pymysql/connections.py", line 493, in rollback
    self._read_ok_packet()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/pymysql/connections.py", line 453, in _read_ok_packet
    pkt = self._read_packet()
  File "/app/.venv/lib/python3.13/site-packages/pymysql/connections.py", line 744, in _read_packet
    packet_header = self._read_bytes(4)
  File "/app/.venv/lib/python3.13/site-packages/pymysql/connections.py", line 782, in _read_bytes
    data = self._rfile.read(num_bytes)
  File "/usr/local/lib/python3.13/socket.py", line 719, in readinto
    return self._sock.recv_into(b)
           ~~~~~~~~~~~~~~~~~~~~^^^
  File "/app/.venv/lib/python3.13/site-packages/gevent/ssl.py", line 635, in recv_into
    return self.read(nbytes, buffer)
           ~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/gevent/ssl.py", line 443, in read
    self._wait(self._read_event, timeout_exc=_SSLErrorReadTimeout)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 304, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
greenlet.GreenletExit

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/.venv/lib/python3.13/site-packages/locust/user/task.py", line 340, in run
    self.execute_next_task()
    ~~~~~~~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/locust/user/task.py", line 373, in execute_next_task
    self.execute_task(self._task_queue.popleft())
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/locust/user/task.py", line 490, in execute_task
    task(self.user)
    ~~~~^^^^^^^^^^^
  File "/app/locustfiles/common.py", line 103, in non_index_update
    with self.engine.connect() as conn:
         ~~~~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 235, in __exit__
    self.close()
    ~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1240, in close
    self._transaction.close()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 2586, in close
    self._do_close()
    ~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 2724, in _do_close
    self._close_impl()
    ~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 2710, in _close_impl
    self._connection_rollback_impl()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 2702, in _connection_rollback_impl
    self.connection._rollback_impl()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1127, in _rollback_impl
    self._handle_dbapi_exception(e, None, None, None, None)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 2349, in _handle_dbapi_exception
    raise newraise.with_traceback(exc_info[2]) from e
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1125, in _rollback_impl
    self.engine.dialect.do_rollback(self.connection)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/default.py", line 700, in do_rollback
    dbapi_connection.rollback()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/pymysql/connections.py", line 493, in rollback
    self._read_ok_packet()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/pymysql/connections.py", line 453, in _read_ok_packet
    pkt = self._read_packet()
  File "/app/.venv/lib/python3.13/site-packages/pymysql/connections.py", line 744, in _read_packet
    packet_header = self._read_bytes(4)
  File "/app/.venv/lib/python3.13/site-packages/pymysql/connections.py", line 782, in _read_bytes
    data = self._rfile.read(num_bytes)
  File "/usr/local/lib/python3.13/socket.py", line 719, in readinto
    return self._sock.recv_into(b)
           ~~~~~~~~~~~~~~~~~~~~^^^
  File "/app/.venv/lib/python3.13/site-packages/gevent/ssl.py", line 635, in recv_into
    return self.read(nbytes, buffer)
           ~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/gevent/ssl.py", line 443, in read
    self._wait(self._read_event, timeout_exc=_SSLErrorReadTimeout)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 304, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
TypeError: expected string or bytes-like object, got 'NoneType'

[2025-03-03 20:39:13,004] locust-worker-c955f5bfb-h6b7v/ERROR/locust.runners: Didn't get heartbeat from master in over 60s
[2025-03-03 20:39:13,005] locust-worker-c955f5bfb-h6b7v/INFO/locust.main: Shutting down (exit code 0)
❯ k logs -n db-load-tester locust-worker-57bf97498-cczvb -f
[2025-03-03 20:42:31,365] locust-worker-57bf97498-cczvb/INFO/locust.main: Starting Locust 2.33.0
[2025-03-03 20:43:28,560] locust-worker-57bf97498-cczvb/ERROR/locust.user.task: expected string or bytes-like object, got 'NoneType'
Traceback (most recent call last):
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1125, in _rollback_impl
    self.engine.dialect.do_rollback(self.connection)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/default.py", line 700, in do_rollback
    dbapi_connection.rollback()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/pymysql/connections.py", line 493, in rollback
    self._read_ok_packet()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/pymysql/connections.py", line 453, in _read_ok_packet
    pkt = self._read_packet()
  File "/app/.venv/lib/python3.13/site-packages/pymysql/connections.py", line 744, in _read_packet
    packet_header = self._read_bytes(4)
  File "/app/.venv/lib/python3.13/site-packages/pymysql/connections.py", line 782, in _read_bytes
    data = self._rfile.read(num_bytes)
  File "/usr/local/lib/python3.13/socket.py", line 719, in readinto
    return self._sock.recv_into(b)
           ~~~~~~~~~~~~~~~~~~~~^^^
  File "/app/.venv/lib/python3.13/site-packages/gevent/ssl.py", line 635, in recv_into
    return self.read(nbytes, buffer)
           ~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/gevent/ssl.py", line 443, in read
    self._wait(self._read_event, timeout_exc=_SSLErrorReadTimeout)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 304, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
greenlet.GreenletExit

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/.venv/lib/python3.13/site-packages/locust/user/task.py", line 340, in run
    self.execute_next_task()
    ~~~~~~~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/locust/user/task.py", line 373, in execute_next_task
    self.execute_task(self._task_queue.popleft())
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/locust/user/task.py", line 490, in execute_task
    task(self.user)
    ~~~~^^^^^^^^^^^
  File "/app/locustfiles/simple.py", line 16, in ping
    with self.engine.connect() as conn:
         ~~~~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 235, in __exit__
    self.close()
    ~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1240, in close
    self._transaction.close()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 2586, in close
    self._do_close()
    ~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 2724, in _do_close
    self._close_impl()
    ~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 2710, in _close_impl
    self._connection_rollback_impl()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 2702, in _connection_rollback_impl
    self.connection._rollback_impl()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1127, in _rollback_impl
    self._handle_dbapi_exception(e, None, None, None, None)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 2349, in _handle_dbapi_exception
    raise newraise.with_traceback(exc_info[2]) from e
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1125, in _rollback_impl
    self.engine.dialect.do_rollback(self.connection)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/sqlalchemy/engine/default.py", line 700, in do_rollback
    dbapi_connection.rollback()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/pymysql/connections.py", line 493, in rollback
    self._read_ok_packet()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/pymysql/connections.py", line 453, in _read_ok_packet
    pkt = self._read_packet()
  File "/app/.venv/lib/python3.13/site-packages/pymysql/connections.py", line 744, in _read_packet
    packet_header = self._read_bytes(4)
  File "/app/.venv/lib/python3.13/site-packages/pymysql/connections.py", line 782, in _read_bytes
    data = self._rfile.read(num_bytes)
  File "/usr/local/lib/python3.13/socket.py", line 719, in readinto
    return self._sock.recv_into(b)
           ~~~~~~~~~~~~~~~~~~~~^^^
  File "/app/.venv/lib/python3.13/site-packages/gevent/ssl.py", line 635, in recv_into
    return self.read(nbytes, buffer)
           ~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/gevent/ssl.py", line 443, in read
    self._wait(self._read_event, timeout_exc=_SSLErrorReadTimeout)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 304, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
TypeError: expected string or bytes-like object, got 'NoneType'

[2025-03-03 20:44:28,467] locust-worker-57bf97498-cczvb/ERROR/locust.runners: Didn't get heartbeat from master in over 60s
[2025-03-03 20:44:28,468] locust-worker-57bf97498-cczvb/INFO/locust.main: Shutting down (exit code 0)

Command line

locust --master --locustfile=locustfiles/simple.py --loglevel=INFO

locust --worker --master-host=locust-master --locustfile=locustfiles/simple.py --processes=1

Locustfile contents

from environs import Env
from locust import task
from sqlalchemy import select

from utils.engine import SQLAlchemyUser, make_url_from_environ

env = Env()
env.read_env()


class SimpleUser(SQLAlchemyUser):
    url = make_url_from_environ(env)

    @task
    def ping(self):
        with self.engine.connect() as conn:
            conn.execute(select(1))

Python version

3.13.2

Locust version

2.33.0

Operating system

Linux (Debian)

[k-kdn-copilot01]

It also crashes when not in worker mode.
Logs

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/xxx/.local/bin/locust", line 8, in <module>
    sys.exit(main())
  File "/home/xxx/.local/lib/python3.10/site-packages/locust/main.py", line 671, in main
    save_html_report()
  File "/home/xxx/.local/lib/python3.10/site-packages/locust/main.py", line 653, in save_html_report
    process_html_filename(options)
  File "/home/xxx/.local/lib/python3.10/site-packages/locust/html.py", line 32, in process_html_filename
    html_filename = html_filename.replace(option_term, str(int(option_value)))
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'
make: *** [Makefile:21: report] Error 1

[cyberw]

What underlying library does sqlalchemy use - is it gevent-friendly? If not, then a single request can block the entire Locust process (which will impact performance when run standalone, but be even more visible in master-worker mode because it will also block heartbeats)

What is SQLAlchemyUser?