[Python] Bound and harden the Watch transform - #1
Closed
Eliaaazzz wants to merge 1 commit into
Closed
Conversation
Follow-up to the experimental Watch transform that documents the event-time contract, makes late data observable, and adds an opt-in bound on the per-input dedup state. Watermark/event-time: - Document the event-time/watermark contract on the module and PollResult: explicit watermark, else the earliest event time of the round's new outputs, held when a round is empty, released to MAX on complete(). - Warn (throttled) when an output is emitted behind the current watermark, so out-of-order late data is observable rather than silently dropped, and point to PollResult.with_watermark for out-of-order sources. Scalability: - Add opt-in Watch.with_dedup_horizon(Duration): drop and collect outputs more than the horizon behind the watermark, so the dedup set tracks a sliding event-time window instead of the full history. A re-listed aged output is suppressed rather than re-emitted: the suppression cutoff equals the prior round's eviction cutoff, so there is no resurrection gap, and a full-relisting poll stays exactly-once with an advancing watermark. Off by default; horizon validated non-negative; suppression reported via a throttled warning. - Reuse the parent dedup map on idle rounds to drop the O(N) per-round copy. Tests cover the watermark contract, the out-of-order late warning, dedup horizon eviction/retention/boundary/order-independence/suppression and re-emission, idle-round reuse, and an end-to-end relisting + suppression case. 30 tests and 5 subtests pass.
Eliaaazzz
marked this pull request as draft
June 24, 2026 12:45
Owner
Author
|
Validated on Dataflow Runner v2 (us-east1, streaming). The poll re-lists a perpetual old output every round; with All 3 seeds: |
Owner
Author
|
Superseded by #2, which solves the unbounded-state problem with a timestamp high-water-mark cursor (O(1) per-input state) instead of a hash set with time-based GC (O(window)). Closing in favor of that design. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on apache#39023, the Watch transform MVP. The base of this PR is the
python-watch-transformbranch, so it shows only the single follow-up commit.It documents the event-time contract, makes late data observable, and adds an
opt-in bound on the per-input dedup state. The default emission and watermark
behavior are unchanged.
Watermark and event-time
PollResult: an explicit watermark if the poll supplies one, otherwise theearliest event time of the round's new outputs, held when a round is empty,
and released to
MAX_TIMESTAMPoncomplete().watermark, so out-of-order late data is observable instead of being silently
dropped downstream. Out-of-order sources should supply
PollResult.with_watermark.Scalability
The per-input dedup set keeps one hash per distinct output and is unbounded by
default, so it grows with the number of distinct outputs an input produces.
Watch.with_dedup_horizon. Outputs more than the horizon behindthe watermark are dropped and their dedup state collected, so the set tracks a
sliding event-time window. A re-listed aged output is suppressed rather than
re-emitted, because the suppression cutoff equals the prior round's eviction
cutoff and leaves no resurrection gap, so a full-relisting poll stays
exactly-once with an advancing watermark. The trade-off is that a genuinely
new output arriving more than the horizon behind the watermark is dropped as
late and reported through a throttled warning. Off by default. The horizon is
validated non-negative.
Testing
30 tests and 5 subtests pass on the in-memory DirectRunner, covering the
watermark contract, the out-of-order late warning, dedup horizon eviction,
retention, the at-cutoff boundary, order-independence, suppression, re-emission,
idle-round reuse, and an end-to-end relisting plus suppression case. yapf
0.43.0, isort 7.0.0, and pylint are clean.