Repository navigation
Flaky test: IngestionSmokeTest.test_streamLogs_ofCancelledTask (ConsulDiscoveryPlainDockerTest) #20435
Description
Activity
I'd like to pick this up — will move the log read into the
waitForResultpoll loop so it waits for the expected log line (Running task[%s] for [%d] millis) rather than just stream presence, per the suggested fix above. Will open a PR shortly.@rishi-rana I think this fixes the test. But I believe there may be an underlying bug here.
The Overlord's HttpRemoteTaskRunner.streamTaskLog passes the worker's response through without checking the HTTP status. A 404, which happens once the worker has dropped a finished task, becomes a present-but-empty log stream.
SwitchingTaskLogStreamer then never falls back to deep storage, where the log actually is.
6. @rishi-rana I think this fixes the test. But I believe there may be an underlying bug here.
The Overlord's HttpRemoteTaskRunner.streamTaskLog passes the worker's response through without checking the HTTP status. A 404, which happens once the worker has dropped a finished task, becomes a present-but-empty log stream.
SwitchingTaskLogStreamer then never falls back to deep storage, where the log actually is.
https://github.com/apache/druid/blob/98bbcb4c7f78/indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java#L936-L940Nvm ignore me, looking at logs, the test flake happened before this bug would've opened up
- added a parent issue
on Oct 10, 2026 - added a commit that references this issue
on Oct 11, 2026
This issue was generated automatically by Claude Code (Anthropic's AI coding agent) running a scheduled CI-triage routine on behalf of @FrankChen021. Analysis and suggested fixes are AI-produced; please verify before acting on them.
Status: Fix PR #20549 open
Subject:
IngestionSmokeTest.test_streamLogs_ofCancelledTask, seen viaConsulDiscoveryPlainDockerTest(embedded-tests,docker-tests)Failures: 1 · First seen: 2026-09-25 · Last seen: 2026-09-25
Root cause
Race: the test polls
TaskLogStreamer.streamTaskLog(taskId, 0)only until theOptionalis present. In the Docker setup, the peon's log is pushed after the process exits, so the stream can be present but empty, which failsassertFalse(logs.isEmpty())atIngestionSmokeTest.java:398. The TLS and MTLS variants passed in the same job.Suggested fix
Move the read into the
waitForResultsupplier, and wait until the log contains"Running task[%s] for [%d] millis"instead of justOptional::isPresent. #20549 implements this.Occurrences
Failed push-triggered master jobs only. The daily triage routine adds one row per new failed job.
docker-tests