Flaky: test_mip_incumbent_stream hangs in PROCESSING after the preceding cancel test kills the gRPC worker

[ramakrishnap-nv]

Summary

TestGrpcClient::test_mip_incumbent_stream flakes in PR CI: the submitted MIP job never leaves PROCESSING, the client's 120 s poll expires, and the test fails with a misleading assertion. It passed on re-run of the same commit, so the failure is not related to the PR's changes.

Observed on run 31686594110, attempt 1 (conda-python-tests / 13.3.0, 3.14, arm64, ubuntu26.04, l4), PR #1700:

    terminal = _poll_until_complete(client, job_id, _MIP_NAMES)
>   assert terminal == JobStatus.COMPLETED
E   assert <JobStatus.PROCESSING: 1> == <JobStatus.COMPLETED: 2>
tests/linear_programming/test_grpc_client.py:276

Everything else in the job passed (1 failed, 126 passed, 9 skipped). Attempt 2 of the same run was fully green.

What the failure actually means

The job reached PROCESSING, which in check_job_status() (cpp/src/grpc/server/grpc_job_management.cpp:191) only happens once a worker has CAS-claimed the queue slot. So a worker took the job and then produced no result for 120 s — even though the test sets time_limit = 30. The solver time limit only bounds solve_mip(); it does not bound anything before the solve starts (worker CUDA/RMM init, problem transfer over the worker pipe), and nothing on the server side ever times a claimed job out. A wedged or very slow worker leaves the job in PROCESSING indefinitely.

Two properties of the test environment make that window much more likely on this runner:

1. The immediately preceding test SIGKILLs the worker. test_cancel_job (line 221) cancels a mid-solve swath1.mps job; cancel_job() sends SIGKILL to the worker PID (grpc_job_management.cpp:283). The worker-monitor thread then forks a replacement (grpc_server_threads.cpp:12-80), which must redo cudaSetDevice + allocate a fresh RMM pool (grpc_worker.cpp:43-83) while the killed process's GPU memory is still being reclaimed by the driver. test_mip_incumbent_stream submits into exactly that window.
2. Four xdist workers share one GPU. PR CI runs pytest -n 4 --dist loadgroup (ci/run_cuopt_pytests.sh:41), so three other GPU test processes are competing with the respawning worker for the L4.

Also worth noting: if init_worker_rmm_pool() throws (rmm::bad_alloc under contention), nothing catches it — init_worker_cuda_environment() and worker_process() have no try/catch, so the worker aborts and the monitor respawns it in a 100 ms loop with no backoff.

None of this is diagnosable from CI today, because the fixture starts the server with stdout=DEVNULL, stderr=DEVNULL (python/cuopt/cuopt/tests/fixtures/grpc_server_fixtures.py, spawn_server). We have no server or worker log for the failing attempt.

Finally, PR CI does not rerun: --reruns 2 --reruns-delay 5 is applied only when RAPIDS_BUILD_TYPE=nightly (ci/run_cuopt_pytests.sh:36-41). So one flake blocks the PR and the classifier reports it as "1 genuine test failure".

Proposed fixes

Observability (do first — otherwise the next occurrence is equally opaque)

• spawn_server() should tee server stdout/stderr to a file under the test's tmp dir instead of DEVNULL, and the grpc_server fixture should dump the tail of that file when a test in the class fails.
• Make _poll_until_complete raise an explicit timeout error (elapsed time + last observed status + incumbent count) rather than returning a non-terminal status that surfaces as PROCESSING != COMPLETED.

Test-level flake removal

• Don't let a worker-killing test run immediately before a solve test on the same server: either move test_cancel_job to the end of TestGrpcClient, or give it its own server instance (its own grpc_port_offset).
• After a cancel, wait for worker readiness before the next submit — needs a server-side health signal (below), or, as a stopgap, a fixture-level wait that submits a trivial probe job and waits for it to complete.
• Reduce poll pressure in _poll_until_complete: it issues status + result every 50 ms (~40 RPC/s), all taking tracker_mutex, which the result-retrieval thread also needs to mark the job COMPLETED. 200-250 ms is plenty.
• Consider --reruns 2 for PR CI too, at least for the grpc_server xdist group.

Server-side (the real fix)

• Add a job watchdog: if a claimed job exceeds its requested time_limit plus a margin, transition it to FAILED with a clear message instead of leaving it PROCESSING forever. Today a wedged worker is invisible to the client.
• Expose worker readiness (active_workers in SharedMemoryControl is already tracked) via the health/status RPC so clients can distinguish "queued behind a restart" from "running".
• Wrap init_worker_cuda_environment() / init_worker_rmm_pool() in a try/catch that logs and exits cleanly, and add backoff to the monitor's respawn path so a persistently failing worker doesn't fork-storm.

[ramakrishnap-nv]

Checked the attempt-1 log for a crash: no SIGSEGV/SIGABRT, no core dump, no terminate called, no CUDA error, no xdist node-crash message. pytest exited 1 (a plain assertion failure) and finished the session normally — ci/run_cuopt_pytests.sh only treats rc > 128 as a crash, and no synthetic crash marker was written. GPU was idle (0 MiB / 23034 MiB used) at the start of the step.

That is only weak evidence about the server, though: cuopt_grpc_server and its forked worker run with stdout/stderr=DEVNULL, so a worker segfault would leave no trace in the CI log at all. What argues against a crash is the observed status, not the log — when the monitor thread reaps a dead worker it calls mark_worker_jobs_failed(), which transitions the job to FAILED/CANCELLED. We saw PROCESSING, so either the worker was alive and slow/stuck, or it died in a window where the monitor could not attribute the job to it.

There is exactly one such window, and it is worth closing regardless of whether it fired here: claim_job_slot() (cpp/src/grpc/server/grpc_worker.cpp:265-270) CAS-sets claimed = true and only then writes worker_pid = getpid(). mark_worker_jobs_failed() matches on ready && claimed && worker_pid == dead_pid, so a worker that dies between those two stores orphans the slot: it stays claimed with worker_pid == 0 forever, no worker will re-claim it, and the job sits in PROCESSING with no result — exactly the observed symptom. The window is a few instructions wide, so this is a long shot as the cause here, but the fix is cheap: write worker_pid before the CAS (or have mark_worker_jobs_failed also sweep slots that are claimed with worker_pid == 0).

Tee-ing the server output to a file (first checklist item) is what would let us distinguish "slow worker" from "silently dead worker" the next time this happens.

[ramakrishnap-nv]

History check — where this has and has not been seen.

Nightly: no occurrences, but nightly cannot detect this. Sampled 12 conda-python-tests jobs across the last 5 nightly runs (2026-08-09 → 08-13), covering arm64 L4, arm64 A100, GB300 and amd64. All green (127 passed, 9 skipped, same 136 collected items), and zero RERUN lines — so the test did not even flake and self-heal. That is expected: ci/run_cuopt_pytests.sh runs nightly serially (pytest -s --cache-clear --reruns 2) while PR CI runs -n 4 --dist loadgroup. Nightly never exercises the 4-way GPU contention this failure depends on, and it has reruns on top. Nightly is not a useful signal for this flake.

PR CI: swept attempt-1 logs of every failed python test job (conda-python-tests + wheel-tests-cuopt) across the 60 most recent pr workflow runs. This test failed in exactly two runs:

1. run 31686594110 — PR Root heuristics #1700, one job (arm64 L4). The occurrence in this report.
2. run 31218812786 — PR Add cooperative cancel for LP/MIP solves #1693, five jobs with the identical assert <JobStatus.PROCESSING: 1> == <JobStatus.COMPLETED: 2>, spanning amd64 H100, arm64 L4 and arm64 rockylinux8, in both conda and wheel test jobs.

The second one is not an independent flake on main: PR #1693 ("Add cooperative cancel for LP/MIP solves", still open) modifies cpp/src/grpc/server/grpc_job_management.cpp, cpp/src/grpc/server/grpc_worker.cpp, grpc_server_fixtures.py and test_grpc_client.py itself, and adds grpc_lifecycle_stress.py — i.e. it is reworking exactly the cancel/worker-lifecycle path. Its failures are PR-specific.

That said, it is useful corroboration of the causal chain here: when the cancel/worker-restart path is perturbed, this specific test is the one that reproducibly stalls in PROCESSING across many configs. The dependency between test_cancel_job killing the worker and test_mip_incumbent_stream submitting into the restart window is real and load-bearing, not incidental.

@NVIDIA/cuopt — whoever owns #1693 may want to fold the server-side items from this issue (job watchdog, worker-readiness signal, the claim_job_slot / worker_pid ordering) into that work rather than duplicating it.