Skip to content

[FLINK-40503][core] Document effective idleness detection timing - #29039

Open
MartijnVisser wants to merge 1 commit into
apache:masterfrom
MartijnVisser:FLINK-40503
Open

MartijnVisser wants to merge 1 commit into
apache:masterfrom
MartijnVisser:FLINK-40503

Conversation

@MartijnVisser

@MartijnVisser MartijnVisser commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

What is the purpose of the change

Resolves FLINK-40503. Documents when withIdleness takes effect: idleness is only checked on watermark emissions, so an input goes idle more than idleTimeout after its last record and, with on-schedule emissions, more than idleTimeout plus one interval and at most plus three. The timing is unchanged.

Brief change log

  • Clarified the WatermarkStrategy#withIdleness Javadoc and the WatermarksWithIdleness class doc.
  • Added the same paragraph to "Dealing With Idle Sources" in the event-time docs, English and Chinese.
  • Added WatermarksWithIdlenessTimeoutTest, which pins the timing with a manual clock.

Verifying this change

This change added tests and can be verified as follows:

  • New WatermarksWithIdlenessTimeoutTest pins the documented timing. Changing > to >= in IdlenessTimer, or starting the timeout one probe earlier, fails this test while WatermarksWithIdlenessTest and WatermarkStrategyTest stay green.
  • Existing WatermarksWithIdlenessTest and WatermarkStrategyTest pass unchanged.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): yes, Javadoc only
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Fable 5, Opus 5.5)

@flinkbot

flinkbot commented Aug 28, 2026 •

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@uros-b

uros-b commented Aug 29, 2026

Copy link
Copy Markdown
Member

LGTM, also kudos for nice testing

@github-actions github-actions Bot added the community-reviewed PR has been reviewed by the community. label Aug 30, 2026
@SEPURI-SAI-KRISHNA

Copy link
Copy Markdown
Contributor

Small one on the lower bound.

The countdown cannot start until a periodic probe sees silence, so detection is never within one interval of the timeout either. Driving WatermarksWithIdleness with a manual clock and recording when markIdle() fires, across timeout/interval ratios from 0.2 to 12 and every probe phase:

earliest markIdle: idleTimeout + 1.00 x interval (timeout=999)
latest   markIdle: idleTimeout + 3.00 x interval (timeout=1000)

Your upper bound matches exactly. "no earlier than idleTimeout" is correct but a full interval loose, "no earlier than idleTimeout plus one watermark interval" held everywhere I measured.

Harness

javac -cp flink-core/target/classes IdlenessTiming.java && java -cp .:flink-core/target/classes IdlenessTiming

import org.apache.flink.api.common.eventtime.*;
import org.apache.flink.util.clock.RelativeClock;
import java.time.Duration;

/** Measures when WatermarksWithIdleness actually calls markIdle() after the last event. */
public class IdlenessTiming {

    static long detect(long timeout, long interval, long firstProbeOffset) {
        long[] now = {0};
        RelativeClock clock = new RelativeClock() {
            public long relativeTimeNanos() { return now[0]; }
            public long relativeTimeMillis() { return now[0] / 1_000_000L; }
        };
        boolean[] idle = {false};
        WatermarkOutput out = new WatermarkOutput() {
            public void emitWatermark(Watermark w) {}
            public void markIdle() { idle[0] = true; }
            public void markActive() { idle[0] = false; }
        };
        WatermarksWithIdleness<Long> g = new WatermarksWithIdleness<>(
                new AscendingTimestampsWatermarks<>(), Duration.ofNanos(timeout), clock);

        long ts = 0;
        for (int i = 0; i < 3; i++) {            // events flowing, probes in between
            g.onEvent(1L, ++ts, out);
            now[0] += interval;
            g.onPeriodicEmit(out);
        }
        g.onEvent(1L, ++ts, out);                // last event, at t = 0
        long base = now[0];
        for (long t = base + firstProbeOffset; ; t += interval) {
            now[0] = t;
            g.onPeriodicEmit(out);
            if (idle[0]) return t - base;        // delay after the last event
        }
    }

    public static void main(String[] args) {
        final long I = 1000;                     // watermark interval
        double min = Double.MAX_VALUE, max = -1;
        long tMin = 0, tMax = 0;
        for (long timeout = 200; timeout <= 12000; timeout++) {
            for (long phase = 1; phase <= I; phase++) {
                double over = (detect(timeout, I, phase) - timeout) / (double) I;
                if (over < min) { min = over; tMin = timeout; }
                if (over > max) { max = over; tMax = timeout; }
            }
        }
        System.out.printf("earliest markIdle: idleTimeout + %.2f x interval (timeout=%d)%n", min, tMin);
        System.out.printf("latest   markIdle: idleTimeout + %.2f x interval (timeout=%d)%n", max, tMax);
    }
}

@1996fanrui 1996fanrui self-assigned this Sep 23, 2026

@1996fanrui 1996fanrui left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @MartijnVisser for this polish!

nit: Time spent backpressured or paused for alignment doesn't count towards the timeout (FLINK-35886), so the bound doesn't hold in wall-clock time then. Could we add a sentence about that in WatermarkStrategy#withIdleness and the docs?

The withIdleness Javadoc implied idleness at idleTimeout after the last record. Idleness is only
checked on watermark emissions, so it takes more than idleTimeout, and with on-schedule emissions
more than idleTimeout plus one interval and at most plus three. Document this and pin it in a test.

Generated-by: Claude Code (Fable 5, Opus 5.5)
@MartijnVisser

MartijnVisser commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

"no earlier than idleTimeout plus one watermark interval" held everywhere I measured.

Thanks, you're right. An extra emission on a backlog event can land in between, so the text now says plus one interval when emissions run on schedule.

@MartijnVisser

MartijnVisser commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Time spent backpressured or paused for alignment doesn't count towards the timeout (FLINK-35886)

Good point, added it to both. A busy task delays the emissions as well, so the upper bound now only claims to hold when they run on schedule.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-reviewed PR has been reviewed by the community.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants