Repository navigation
[FLINK-40503][core] Document effective idleness detection timing - #29039
MartijnVisser wants to merge 1 commit into
Conversation
|
LGTM, also kudos for nice testing |
|
Small one on the lower bound. The countdown cannot start until a periodic probe sees silence, so detection is never within one interval of the timeout either. Driving Your upper bound matches exactly. "no earlier than idleTimeout" is correct but a full interval loose, "no earlier than idleTimeout plus one watermark interval" held everywhere I measured. Harness
import org.apache.flink.api.common.eventtime.*;
import org.apache.flink.util.clock.RelativeClock;
import java.time.Duration;
/** Measures when WatermarksWithIdleness actually calls markIdle() after the last event. */
public class IdlenessTiming {
static long detect(long timeout, long interval, long firstProbeOffset) {
long[] now = {0};
RelativeClock clock = new RelativeClock() {
public long relativeTimeNanos() { return now[0]; }
public long relativeTimeMillis() { return now[0] / 1_000_000L; }
};
boolean[] idle = {false};
WatermarkOutput out = new WatermarkOutput() {
public void emitWatermark(Watermark w) {}
public void markIdle() { idle[0] = true; }
public void markActive() { idle[0] = false; }
};
WatermarksWithIdleness<Long> g = new WatermarksWithIdleness<>(
new AscendingTimestampsWatermarks<>(), Duration.ofNanos(timeout), clock);
long ts = 0;
for (int i = 0; i < 3; i++) { // events flowing, probes in between
g.onEvent(1L, ++ts, out);
now[0] += interval;
g.onPeriodicEmit(out);
}
g.onEvent(1L, ++ts, out); // last event, at t = 0
long base = now[0];
for (long t = base + firstProbeOffset; ; t += interval) {
now[0] = t;
g.onPeriodicEmit(out);
if (idle[0]) return t - base; // delay after the last event
}
}
public static void main(String[] args) {
final long I = 1000; // watermark interval
double min = Double.MAX_VALUE, max = -1;
long tMin = 0, tMax = 0;
for (long timeout = 200; timeout <= 12000; timeout++) {
for (long phase = 1; phase <= I; phase++) {
double over = (detect(timeout, I, phase) - timeout) / (double) I;
if (over < min) { min = over; tMin = timeout; }
if (over > max) { max = over; tMax = timeout; }
}
}
System.out.printf("earliest markIdle: idleTimeout + %.2f x interval (timeout=%d)%n", min, tMin);
System.out.printf("latest markIdle: idleTimeout + %.2f x interval (timeout=%d)%n", max, tMax);
}
} |
1996fanrui
left a comment
There was a problem hiding this comment.
Thanks @MartijnVisser for this polish!
nit: Time spent backpressured or paused for alignment doesn't count towards the timeout (FLINK-35886), so the bound doesn't hold in wall-clock time then. Could we add a sentence about that in WatermarkStrategy#withIdleness and the docs?
The withIdleness Javadoc implied idleness at idleTimeout after the last record. Idleness is only checked on watermark emissions, so it takes more than idleTimeout, and with on-schedule emissions more than idleTimeout plus one interval and at most plus three. Document this and pin it in a test. Generated-by: Claude Code (Fable 5, Opus 5.5)
ca757a0 to
bb7cbe3
Compare
Thanks, you're right. An extra emission on a backlog event can land in between, so the text now says plus one interval when emissions run on schedule. |
Good point, added it to both. A busy task delays the emissions as well, so the upper bound now only claims to hold when they run on schedule. |
What is the purpose of the change
Resolves FLINK-40503. Documents when
withIdlenesstakes effect: idleness is only checked on watermark emissions, so an input goes idle more than idleTimeout after its last record and, with on-schedule emissions, more than idleTimeout plus one interval and at most plus three. The timing is unchanged.Brief change log
WatermarkStrategy#withIdlenessJavadoc and theWatermarksWithIdlenessclass doc.WatermarksWithIdlenessTimeoutTest, which pins the timing with a manual clock.Verifying this change
This change added tests and can be verified as follows:
WatermarksWithIdlenessTimeoutTestpins the documented timing. Changing>to>=inIdlenessTimer, or starting the timeout one probe earlier, fails this test whileWatermarksWithIdlenessTestandWatermarkStrategyTeststay green.WatermarksWithIdlenessTestandWatermarkStrategyTestpass unchanged.Does this pull request potentially affect one of the following parts:
@Public(Evolving): yes, Javadoc onlyDocumentation
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Fable 5, Opus 5.5)