[FLINK-40170][table-planner] Infer update-producing changelog mode for early-fire interval join - #3
Draft
weiqingy wants to merge 33 commits into
Draft
[FLINK-40170][table-planner] Infer update-producing changelog mode for early-fire interval join#3weiqingy wants to merge 33 commits into
weiqingy wants to merge 33 commits into
Conversation
weiqingy
force-pushed
the
FLINK-36953-pr3-changelog
branch
from
July 18, 2026 22:12
f6163da to
16ae7d2
Compare
weiqingy
force-pushed
the
FLINK-36953-pr2-target
branch
from
July 18, 2026 22:12
0b76a4b to
dc1e210
Compare
weiqingy
changed the title
weiqingy
force-pushed
the
FLINK-36953-pr3-changelog
branch
from
July 21, 2026 02:17
16ae7d2 to
17d607b
Compare
weiqingy
force-pushed
the
FLINK-36953-pr2-target
branch
2 times, most recently
from
July 21, 2026 04:43
2d8840b to
8000ad1
Compare
weiqingy
force-pushed
the
FLINK-36953-pr3-changelog
branch
from
July 21, 2026 04:43
17d607b to
9dbe90d
Compare
weiqingy
force-pushed
the
FLINK-36953-pr2-target
branch
2 times, most recently
from
August 2, 2026 22:18
0ef163a to
3b14d56
Compare
weiqingy
force-pushed
the
FLINK-36953-pr3-changelog
branch
from
August 2, 2026 22:42
9dbe90d to
8f7448f
Compare
…tract to `MiscSemanticTests`
…to make executing while CI/local tests
…ration reference ConfigurationOptionLocator discovers ConfigOptions from a hard-coded list of packages and reads each with Files.newDirectoryStream, which does not recurse into sub-packages. Options outside that list are dropped from the generated configuration reference without any error, and ConfigOptionsDocsCompletenessITCase cannot detect it because it derives its expectations from the same list. All seven state.backend.rocksdb.manual-compaction.* options were affected: they carry @Documentation.Section(EXPERT_ROCKSDB) but live in org.apache.flink.state.rocksdb.sstmerge, a sub-package of a searched package, so the feature shipped in 1.20 had no documented configuration. Add a location for the sub-package and regenerate the affected tables. RocksDBManualCompactionOptions needs a stability annotation because becoming discoverable also subjects it to ConfigOptionsDocGenerator#verifyClassAnnotation; @PublicEvolving matches RocksDBOptions and RocksDBConfigurableOptions in the same module. Two option descriptions were missing a space between concatenated sentences, which is now user-visible, so fix those too. Add ConfigurationOptionLocatorTest to prevent recurrence: it scans the source tree and fails when a @Documentation.Section option sits in a package the locator does not search. Repo-wide it needs no exclusions. Generated-by: Claude Code (claude-opus-5)
Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…th a leading None (apache#28819) _infer_type inferred a list's element type from obj[0] rather than the first non-None element the loop scans for, so a leading None collapsed the array element type to NULL. Infer from the scanned element v instead, matching the dict branch above.
…ng name requirements This closes apache#28887.
…n ON CONFLICT clause This closes apache#28918
`BinaryVariantUtil` decoded string values and object field names with `new String(byte[], int, int)`, which uses the JVM default charset, while `BinaryVariantInternalBuilder` writes both as UTF-8. The two only agree on Java 18+, where JEP 400 made UTF-8 the default charset. On Java 11 and 17 a non-UTF-8 platform charset corrupts any non-ASCII text. Corrupted field names are the worse half of this. `getField(name)` silently returns null, and `getFieldNames()` and `toJson()` return mangled keys. Both call sites now pass `StandardCharsets.UTF_8` explicitly, matching Spark's `VariantUtil`.
…utable via MDC propagation and thread names Source split-fetcher threads previously carried no job identity, so on a shared TaskManager their logs and thread dumps could not be traced back to the job that owned them. This adds the job id into each fetcher pool thread's MDC and appends a truncated job-name/job-id suffix to the fetcher thread name, making both logs and thread dumps attributable per job.
ThreadMXBean.dumpAllThreads(true, true) enters a single JVM-wide
safepoint to collect monitor/synchronizer state; on busy JVMs the pause
can exceed heartbeat.timeout and cause unnecessary TaskManager failover.
- Introduce ThreadDumpMode {LITE, FULL}: LITE = dumpAllThreads(false,
false), FULL preserves today's (true, true) behavior. Exposed via an
optional query parameter `?mode=lite|full` on the JM/TM thread-dump
endpoints.
- Add config cluster.thread-dump.default-mode (default FULL to preserve
upgrade behavior; LITE recommended for large clusters).
- Add a Lite/Full toggle to both Web UI thread-dump pages; selecting a
mode does not auto-fetch, the download link tracks the selection.
The decrement in acquire() was generalized from a constant 1 to numberOfEvents as part of FLIP-535, but the gate was left closing only at capacityLeft <= 0, so a cycle can admit more events than capacityPerCycle. The gate is now closed unless the capacity left covers the whole request.
…of ForSt transfer threads
…ilteringHandler onRecoveredStateBuffer() takes over the buffer before anything can fail, so the compensating loop in the catch block must skip the buffer it failed on.
Generated-by: Claude Code (claude-opus-4-8)
…n Batch Mode (apache#28500) Co-authored-by: Yuepeng Pan <hipanyuepeng@gmail.com>
…fire interval join With the EARLY_FIRE hint, an outer interval join speculatively emits a padded unmatched row after the delay and corrects it when a match later arrives, so it no longer produces insert-only changes. Teach FlinkChangelogModeInferenceProgram to reflect this. Split StreamPhysicalIntervalJoin into its own ModifyKindSet arm: its children still consume insert-only, but the node provides INSERT and, when the hint makes it update-producing, UPDATE. A new produceEarlyFireUpdates accessor gates that on the hint being set, the join being outer, and a non-negative window span, so the hint stays inert for inner joins and negative-window joins (which only ever emit inserts). The interval join keeps its place in the UpdateKind and DeleteKind arms. When such a join feeds an insert-only downstream, planning fails with a tailored error that names the hint, rather than the generic "doesn't support consuming update changes" message. Runtime behavior is unchanged; the operator still ignores the hint. The compiled-plan round-trip test's sink now declares sink-insert-only=false: its early-fire outer join produces updates under the new inference, so the previously insert-only sink no longer accepts the pipeline.
weiqingy
force-pushed
the
FLINK-36953-pr3-changelog
branch
from
August 8, 2026 04:27
8f7448f to
4f6ef24
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the FLIP-497 stack under umbrella FLINK-36953. Stacked on PR-2. Landing order: PR-1a → PR-1b → PR-2 → PR-3 (this) → PR-4 → PR-5 → PR-6 → PR-7.
What is the purpose of the change
When
EARLY_FIREis set on an outer interval join, the join emits a speculative null-padded row and later corrects it — an updating result. This PR makes the planner infer that update-producing changelog mode and reject an insert-only downstream with a tailored error. Inner joins and negative-window joins remain append-only.Brief change log
FlinkChangelogModeInferenceProgram(SatisfyModifyKindSetTraitVisitor): early-fire + outer join + non-negative window ⇒INSERT,UPDATE; otherwiseINSERT_ONLY.TableExceptionwhen such an updating early-fire interval join feeds an insert-only sink.StreamPhysicalIntervalJoinforwards the trait.Verifying this change
This change added tests and can be verified as follows:
IntervalJoinTestplan/changelog cases: early-fire outer join is update-producing; insert-only-downstream throws the tailored error; negative-window + early-fire and inner + early-fire stayINSERT_ONLY.Does this pull request potentially affect one of the following parts:
@Public(Evolving): noDocumentation
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Anthropic)