Skip to content

MINOR: Batch the produce in Connect integration tests - #23028

Open
mingyen066 wants to merge 1 commit into
apache:trunkfrom
mingyen066:MINOR-batch-connect-integration-test-produce
Open

MINOR: Batch the produce in Connect integration tests#23028
mingyen066 wants to merge 1 commit into
apache:trunkfrom
mingyen066:MINOR-batch-connect-integration-test-produce

Conversation

@mingyen066

@mingyen066 mingyen066 commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

EmbeddedKafkaCluster#produce only accepts one record and blocks on
send().get(), so callers that need many records have to loop and pay a
round trip per record. Each of those round trips also waits out
linger.ms for a batch that can never fill, because the caller is blocked
in get(). Six call sites across four integration test classes produce
~11,000 records this way.

Add a produce(List<ProducerRecord<byte[], byte[]>>) overload that sends
the whole list before awaiting any acknowledgement, and have the
single-record overloads delegate to it. Blocking behaviour is unchanged.

Measured with -PmaxParallelForks=1, two runs per side:

before   after
TransformationIntegrationTest#testFilterOnTopicNameWithSinkConnector
34.9s   14.6s
ExampleConnectIntegrationTest#testSinkConnector
29.1s   14.8s
TransformationIntegrationTest#testFilterOnTombstonesWithSinkConnector
24.4s   14.4s
ErrorHandlingIntegrationTest#testErrantRecordReporter
13.5s    7.2s
ErrorHandlingIntegrationTest#testSkipRetryAndDLQWithHeaders
12.2s    7.4s
MonitorableSinkIntegrationTest#testMonitorableSinkConnectorAndTask
9.7s    3.7s

Total for these four classes: 149.2s -> 87.4s.

@github-actions github-actions Bot added triage PRs from the community connect small Small PRs labels Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

connect small Small PRs triage PRs from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant