[FLINK-40270][connector-base][runtime] Make source threads job-attributable via MDC propagation and thread names - #28857
Open
Savonitar wants to merge 1 commit into
Open
Conversation
Collaborator
…utable via MDC propagation and thread names On a shared/multi-tenant TaskManager, source fetcher and coordinator threads carried no job identity: their logs emitted with an empty flink-job-id and their thread names could not be attributed to a job. Building on the JobInfo API from FLINK-39776, the fetcher thread factory now seeds the job id into each pool thread's MDC and appends a job suffix (truncated job name + hex job id) to fetcher and coordinator thread names, and the SourceCoordinatorContext worker executor is wrapped with MdcUtils.scopeToJob so callAsync callables carry the job id as well. All existing @PublicEvolving constructors keep their exact behavior; connectors opt in via a new trailing nullable JobInfo constructor overload on SplitFetcherManager / SingleThreadFetcherManager. Generated-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Savonitar
force-pushed
the
source-job-attribution
branch
from
July 31, 2026 14:54
d1afc33 to
53d54c1
Compare
Savonitar
marked this pull request as ready for review
July 31, 2026 15:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
On a session / multi-tenant TaskManager, source data-plane and coordinator threads carry no job identity, so their logs and thread dumps cannot be attributed to a job (source operator names collide across jobs). Building on the
JobInfoAPI from FLINK-39776, this PR closes three gaps:SplitFetcherManager): the fetcher pool did not inherit the task thread's SLF4J MDC, so every connectorSplitReaderlogged with an emptyflink-job-id, and the "Source Data Fetcher for ..." thread name carried no job identity.SourceCoordinatorContext):callAsyncwork (e.g. periodic file-split scans) still logged withoutflink-job-id.SourceCoordinatorProvider):SourceCoordinator-<operatorName>is identical for two jobs using the same source operator name. The thread-name suffix (truncated job name + full hex job id) deliberately matches theflink-job-idMDC value, so thread dumps correlate with log output.Brief change log
MdcUtils: newjobThreadNameSuffix(JobInfo)helper (job name capped at 32 chars, the length of the hexJobIDthat follows it).SplitFetcherManager/SingleThreadFetcherManager: new constructor overload with a trailing@Nullable JobInfo, when provided, the thread factory seeds the job id into each pool thread's MDC once and appends the job suffix to the thread name. All existing constructors delegate withnulland keep their exact previous behavior.SourceCoordinatorContext: worker executor wrapped withMdcUtils.scopeToJob.SourceCoordinatorProvider: job suffix appended to the coordinator thread name (the derived-workerpool inherits it).Verifying this change
This change added tests:
MdcUtilsTest: suffix format, exact truncation boundary (at cap / cap + 1), empty and null job names.SplitFetcherManagerTest: on a real fetcher thread, the MDC carries the job id and the thread name carries the exact suffix; the no-JobInfoconstructors preserve the historical thread name and empty MDC.SourceCoordinatorContextTest:callAsynccallables run with the job id in the MDC.SourceCoordinatorProviderTest: the lazily created coordinator thread's name contains the job name and id.Does this pull request potentially affect one of the following parts:
@Public(Evolving): (yes / no)Documentation
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Opus 4.8 (1M context) noreply@anthropic.com