Skip to content

Commit 51a9076

Browse files
sl0thentr0pyclaude
andcommitted
test(celery): faulthandler dump child stacks on beat hang
The pidfile-race fix did not unhang py3.7 celery in CI. pytest-timeout only dumps the parent pytest process's threads — the parent is stuck in pytest-forked's waitpid, so the actual hang is somewhere inside the forked child, invisible. Schedule faulthandler.dump_traceback_later(45) in run_beat (which runs inside the forked child) and cancel it on successful beat shutdown. If beat hangs in CI again, the child's full thread dump lands in the log and tells us where to look. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent f30916f commit 51a9076

1 file changed

Lines changed: 11 additions & 1 deletion

File tree

tests/integrations/celery/integration_tests/__init__.py

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import faulthandler
12
import os
23
import signal
34
import sys
@@ -65,6 +66,12 @@ def run_beat(celery_app, runtime_seconds=1, loglevel="warning", quiet=True):
6566
Run Celery Beat that immediately starts tasks.
6667
The Celery Beat instance is automatically terminated after `runtime_seconds`.
6768
"""
69+
# If anything below wedges this (forked) child process, dump all of its
70+
# thread stacks to stderr and exit, so a future hang surfaces a diagnosis
71+
# in the CI log instead of just stalling pytest-forked's waitpid in the
72+
# parent. Generous budget: kill_beat itself can wait up to ~30s for the
73+
# pidfile, plus runtime_seconds, plus celery beat shutdown.
74+
faulthandler.dump_traceback_later(45, exit=True)
6875
logger.info("Starting Celery Beat...")
6976
pid_file = os.path.join(tempfile.mkdtemp(), f"celery-beat-{os.getpid()}.pid")
7077

@@ -80,4 +87,7 @@ def run_beat(celery_app, runtime_seconds=1, loglevel="warning", quiet=True):
8087
quiet=quiet,
8188
pidfile=pid_file,
8289
)
83-
beat_instance.run()
90+
try:
91+
beat_instance.run()
92+
finally:
93+
faulthandler.cancel_dump_traceback_later()

0 commit comments

Comments
 (0)