Repository navigation
[Improve] Consume any ready child in CollectOperator - #18813
Open
JackieTien97 wants to merge 1 commit into
Open
JackieTien97 wants to merge 1 commit into
JackieTien97 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
CollectOperatorcurrently stalls on its current child even when another child can produce data. Consume whichever child becomes unblocked, retaining the selected index on the driver thread soMappingCollectOperatorapplies the correct column mapping. This followsCollectNode's existing allowance to consume other children while one is blocked.Cache each child's blocking future and register at most one listener per pending child future. Reuse the aggregate future within a wait round, and retarget existing listeners between rounds with a completion recheck to prevent missed wakeups. Already-ready children need no listener registration or aggregate-future allocation. Child failures and cancellations are propagated before consumption.
Track remaining children independently of their indexes, close all remaining children, and account for intermediate state retained by other children when estimating memory.
Validation
Both clean reactor builds passed, including dependencies and all 15 new unit tests:
Coverage includes 2,000 wait rounds with one long-blocked child, 1,000 concurrent-completion rounds, retargeting races and delayed callbacks, cancellation/failure handling, out-of-order completion and cleanup, column mapping, and retained-memory estimates. Spotless formatting and
git diff --checkalso passed.