What happened
TestRayServiceIncrementalUpgradeWithLocust and TestRayServiceIncrementalUpgradeRollbackMatrixWithLocust intentionally stop the Locust load generator early by sending it SIGINT (pkill -SIGINT -f "locust --headless"). Locust sometimes exits with code 2 in response, and the test's cleanup treats that as a hard failure.
CI failures such as:
show
2026-08-08T03:02:38Z <Greenlet at 0x7fd4d9abb4c0: <bound method StatsCSVFileWriter.stats_writer of <locust.stats.StatsCSVFileWriter object at 0x7fd4d998b4c0>>> failed with KeyboardInterrupt
[2026-08-07 20:02:38,442] locust-cluster-head-56wxv/INFO/root: Exiting due to CTRL+C interruption
[2026-08-07 20:02:38,442] locust-cluster-head-56wxv/INFO/locust.main: --run-time limit reached, shutting down
[2026-08-07 20:02:38,444] locust-cluster-head-56wxv/CRITICAL/locust.main: Unhandled exception in greenlet: <Greenlet at 0x7fd4d9abb4c0: <bound method StatsCSVFileWriter.stats_writer of <locust.stats.StatsCSVFileWriter object at 0x7fd4d998b4c0>>>
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 912, in gevent._gevent_cgreenlet.Greenlet.run
File "/home/ray/anaconda3/lib/python3.10/site-packages/locust/stats.py", line 1111, in stats_writer
self._stats_history_data_rows(self.stats_history_csv_writer, now)
File "/home/ray/anaconda3/lib/python3.10/site-packages/locust/stats.py", line 1156, in _stats_history_data_rows
self._percentile_fields(stats_entry, use_current=self.full_history),
File "/home/ray/anaconda3/lib/python3.10/site-packages/locust/stats.py", line 982, in _percentile_fields
return [int(stats_entry.get_response_time_percentile(x) or 0) for x in self.percentiles_to_report]
File "/home/ray/anaconda3/lib/python3.10/site-packages/locust/stats.py", line 982, in <listcomp>
return [int(stats_entry.get_response_time_percentile(x) or 0) for x in self.percentiles_to_report]
File "/home/ray/anaconda3/lib/python3.10/site-packages/locust/stats.py", line 588, in get_response_time_percentile
return calculate_response_time_percentile(self.response_times, self.num_requests, percent)
File "/home/ray/anaconda3/lib/python3.10/site-packages/locust/stats.py", line 148, in calculate_response_time_percentile
for response_time in sorted(response_times.keys(), reverse=True):
KeyboardInterrupt
[2026-08-07 20:02:38,664] locust-cluster-head-56wxv/INFO/locust.runners: Worker 'locust-cluster-head-56wxv_682c89d1b57140e993ed900249adee7f' (index 0) quit. 0 workers ready.
[2026-08-07 20:02:38,664] locust-cluster-head-56wxv/INFO/locust.runners: The last worker quit, stopping test.
[2026-08-07 20:02:39,057] locust-cluster-head-56wxv/INFO/locust.main: writing html report to file: locust_results/2026-08-07-PM-20-00-22-results.html
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
######--|###########################################################################-|######-|############-|######-|######-|######-|######-|######--|#########--
GET /test 56828 0(0.00%) | 21 3 281 17 | 451.40 0.00
######--|###########################################################################-|######-|############-|######-|######-|######-|######-|######--|#########--
Aggregated 56828 0(0.00%) | 21 3 281 17 | 451.40 0.00
[2026-08-07 20:02:39,560] locust-cluster-head-56wxv/INFO/locust.main: Shutting down (exit code 2)
returncode: 2
Root Cause Analysis
When a subtest (t.Run("BlueGreen", ...)) is about to return, the deferred cleanup fires and sends pkill to the Locust process. Locust normally catches this and shuts down cleanly (exit 0), but due to a timing race in how gevent delivers the signal, it sometimes crashes a background greenlet and exits with code 2 instead, even though the load test itself succeeded. The crash isn't in shutdown-specific code; it's whichever greenlet happens to be executing CPU-bound work at the instant the signal arrives. locust_runner.py propagates this exit code even when every assertion passed, and eg.Wait() receives that error and fails the test.
Also, why does most of the time, the master ends up with exit code 0? tracing the code
https://github.com/locustio/locust/blob/6272527fbc4c692a00ee9851878ccbe085b31735/locust/main.py#L681-L695
try:
main_greenlet.join()
...
except KeyboardInterrupt:
logging.info("Exiting due to CTRL+C interruption")Most of the master process's life, nothing is actively executing Python bytecode, every simulated-user greenlet is blocked waiting on a socket, and the program's "main thread of control" is effectively parked at this .join() call, waiting for the test to end. When SIGINT arrives at some random instant, the overwhelming majority of the time that's where the interpreter actually is sitting at this top-level wait, so that's where the pending KeyboardInterrupt gets raised, straight into the except clause that's specifically there to catch it. Then it falls through to shutdown() normally: runner.quit(), JSON printed, sys.exit(0).
The failure mode only happens on the rare occasion the random SIGINT instant instead lands during one of stats_writer's brief periodic CPU-bound bursts, a unguarded piece of code that happens to be executing at that exact moment instead of the .join() call.
Further detail: neither the worker (sending load) nor the master (coordinating and aggregating stats) ever terminates on its own as there's no --run-time limit, and StagesShape's single stage never completes within the test's actual runtime. pkill is the only thing that ever ends either process, meaning proc.communicate() in locust_runner.py stays blocked on the master until that signal triggers its exit. So this isn't a rare edge case triggered by unusual timing, every single subtest run depends on this same signal-driven termination to end at all, which is exactly why the race matters on every run, not just some.
Two more things worth noting:
Proposed fix
a.
Trust assert num_failures == 0 instead of sys.exit(proc.returncode), since that's the real correctness signal, it is also computed and asserted before shutdown even begins. This means that it is unaffected by which process or which greenlet the race happens to hit.
This will be a one line fix. Worth noting that the reference script this is based on (https://raw.githubusercontent.com/ray-project/serve_workloads/main/microbenchmarks/locust_runner.py) doesn't propagate proc.returncode either. Verified in db7b78d & 8acb8c2. 0/50 failure happened.
or
b. Alternative considered and rejected from the experimnet
use pkill -SIGTERM instead of -SIGINT
reason: locust supports sigterm better
https://github.com/locustio/locust/blob/6272527fbc4c692a00ee9851878ccbe085b31735/locust/main.py#L667-L670
https://github.com/locustio/locust/blob/6272527fbc4c692a00ee9851878ccbe085b31735/locust/main.py#L679
basically saying they take care of the SIGTERM -> graceful shutdown. Comparing to sigint, we have to handle the exception ourself
https://github.com/locustio/locust/blob/6272527fbc4c692a00ee9851878ccbe085b31735/locust/main.py#L232-L248
In practice this made things worse. Experiment runs 593ab58 & 7907857 show the worker always shuts down gracefully, but the master consistently does not (returncode: -15, i.e. killed by SIGTERM via the OS default disposition, not gevent's handler). This left stdout empty and json.loads() failing deterministically — 16/16 runs, worse than the original intermittent race. Root cause of why the master doesn't handle SIGTERM gracefully here is still unconfirmed.
This would require runtime tracing (e.g. py-spy/strace) to pin down further, which I'm uncertain is worth the effort.
Are you willing to submit a PR?
What happened
TestRayServiceIncrementalUpgradeWithLocust and TestRayServiceIncrementalUpgradeRollbackMatrixWithLocust intentionally stop the Locust load generator early by sending it SIGINT (pkill -SIGINT -f "locust --headless"). Locust sometimes exits with code 2 in response, and the test's cleanup treats that as a hard failure.
CI failures such as:
show
Root Cause Analysis
When a subtest (t.Run("BlueGreen", ...)) is about to return, the deferred cleanup fires and sends pkill to the Locust process. Locust normally catches this and shuts down cleanly (exit 0), but due to a timing race in how gevent delivers the signal, it sometimes crashes a background greenlet and exits with code 2 instead, even though the load test itself succeeded. The crash isn't in shutdown-specific code; it's whichever greenlet happens to be executing CPU-bound work at the instant the signal arrives. locust_runner.py propagates this exit code even when every assertion passed, and eg.Wait() receives that error and fails the test.
Also, why does most of the time, the master ends up with exit code 0? tracing the code
https://github.com/locustio/locust/blob/6272527fbc4c692a00ee9851878ccbe085b31735/locust/main.py#L681-L695
Most of the master process's life, nothing is actively executing Python bytecode, every simulated-user greenlet is blocked waiting on a socket, and the program's "main thread of control" is effectively parked at this .join() call, waiting for the test to end. When SIGINT arrives at some random instant, the overwhelming majority of the time that's where the interpreter actually is sitting at this top-level wait, so that's where the pending KeyboardInterrupt gets raised, straight into the except clause that's specifically there to catch it. Then it falls through to shutdown() normally: runner.quit(), JSON printed, sys.exit(0).
The failure mode only happens on the rare occasion the random SIGINT instant instead lands during one of stats_writer's brief periodic CPU-bound bursts, a unguarded piece of code that happens to be executing at that exact moment instead of the .join() call.
Further detail: neither the worker (sending load) nor the master (coordinating and aggregating stats) ever terminates on its own as there's no --run-time limit, and StagesShape's single stage never completes within the test's actual runtime. pkill is the only thing that ever ends either process, meaning proc.communicate() in locust_runner.py stays blocked on the master until that signal triggers its exit. So this isn't a rare edge case triggered by unusual timing, every single subtest run depends on this same signal-driven termination to end at all, which is exactly why the race matters on every run, not just some.
Two more things worth noting:
kuberay/ray-operator/test/support/locust_runner.py
Lines 54 to 60 in 55ffdeb
which is what spawns stats_writer in the first place. The periodic greenlet that infinitely writing CSV stats:
https://github.com/locustio/locust/blob/6272527fbc4c692a00ee9851878ccbe085b31735/locust/stats.py#L1171-L1175
Proposed fix
a.
Trust assert num_failures == 0 instead of sys.exit(proc.returncode), since that's the real correctness signal, it is also computed and asserted before shutdown even begins. This means that it is unaffected by which process or which greenlet the race happens to hit.
This will be a one line fix. Worth noting that the reference script this is based on (https://raw.githubusercontent.com/ray-project/serve_workloads/main/microbenchmarks/locust_runner.py) doesn't propagate proc.returncode either. Verified in db7b78d & 8acb8c2. 0/50 failure happened.
or
b. Alternative considered and rejected from the experimnet
use pkill -SIGTERM instead of -SIGINT
reason: locust supports sigterm better
https://github.com/locustio/locust/blob/6272527fbc4c692a00ee9851878ccbe085b31735/locust/main.py#L667-L670
https://github.com/locustio/locust/blob/6272527fbc4c692a00ee9851878ccbe085b31735/locust/main.py#L679
basically saying they take care of the SIGTERM -> graceful shutdown. Comparing to sigint, we have to handle the exception ourself
https://github.com/locustio/locust/blob/6272527fbc4c692a00ee9851878ccbe085b31735/locust/main.py#L232-L248
In practice this made things worse. Experiment runs 593ab58 & 7907857 show the worker always shuts down gracefully, but the master consistently does not (returncode: -15, i.e. killed by SIGTERM via the OS default disposition, not gevent's handler). This left stdout empty and json.loads() failing deterministically — 16/16 runs, worse than the original intermittent race. Root cause of why the master doesn't handle SIGTERM gracefully here is still unconfirmed.
This would require runtime tracing (e.g. py-spy/strace) to pin down further, which I'm uncertain is worth the effort.
Are you willing to submit a PR?