[improve][test] PIP-473: extensive v5 transaction tests on scalable topics#25958
Merged
Merged
Conversation
…opics Motivation: P6 hardening for v5 (scalable-topic) transactions. Broaden coverage across single-broker, broker-restart recovery, and multi-broker docker-cluster scenarios, and re-enable the split/merge transaction tests that the earlier P5.4 fixes already made pass. Modifications: - V5TransactionScalableTest (single-broker unit): multi-segment commit; many concurrent transactions with mixed commit/abort; abort across split and across merge; large transaction; transactional-ack lifecycle (abort -> redeliver, then commit -> stick); per-subscription isolation of transactional acks; read-committed pinning while a transaction is open. - V5TransactionRecoveryTest (single restartable broker): the mock BookKeeper and metadata stores are reused across restartBroker(), so a segment topic reloads cold and exercises MetadataTransactionBuffer / MetadataPendingAckStore recovery — committed data stays visible, aborted data stays filtered, and committed transactional acks are not redelivered. - V5ScalableTopicTransactionTest (docker, 2-broker cluster): produce/consume commit+abort, multi-topic atomic commit, consume-transform-produce, broker failover mid-transaction, and commit spanning a split / merge. Registered in pulsar-transaction.xml. - Re-enabled V5TransactionTest.testCommitSpansSplit / testCommitSpansMerge and removed their stale "disabled" comments (the gap they described was specific to the old v4 RPC commit model; v5 endTransaction is metadata-event driven). Assisted-by: Claude Code
…against Oxia Motivation: The new V5 scalable-topic transaction integration tests run against a ZooKeeper-backed cluster, which exercises the metadata layout via the scan-and-filter fallback. Oxia is the backend PIP-473 targets natively (scanByIndex / subscribeSequence / partition-key / sequence-delta primitives back the /txn layout directly), so it should also be covered end to end. Modifications: - V5ScalableTopicTransactionOnOxiaTest: extends V5ScalableTopicTransactionTest and reruns all six scenarios (produce/consume commit+abort, multi-topic atomic commit, consume-transform-produce, broker failover mid-transaction, commit-spans-split, commit-spans-merge) on a cluster whose metadata store is a containerized Oxia (enableOxia=true). - TcMetadataDiscoveryTestBase: extract the initialize-transaction-coordinator- metadata configuration-store connection string into an overridable configurationStoreConnectionString() (defaults to the ZooKeeper container; the Oxia subclass returns the oxia:// URL). The existing ZK-backed TcMetadataDiscoveryTest is unaffected. - Registered the new class in pulsar-transaction.xml. Assisted-by: Claude Code
lhotari
approved these changes
Jun 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
P6 (test coverage) hardening for PIP-473 v5 transactions on scalable (
topic://) topics. This broadens automated coverage across three layers — single-broker, broker-restart recovery, and a multi-broker docker cluster — and re-enables the split/merge transaction tests that the earlier P5.4 fixes already made pass.Modifications
V5TransactionScalableTest(single-broker unit): multi-segment commit; many concurrent transactions with mixed commit/abort; abort across split and across merge; large transaction; transactional-ack lifecycle (abort → redeliver, then commit → stick); per-subscription isolation of transactional acks; read-committed pinning while a transaction is open.V5TransactionRecoveryTest(single restartable broker): the mock BookKeeper and metadata stores are reused acrossrestartBroker(), so a segment topic reloads cold and exercisesMetadataTransactionBuffer/MetadataPendingAckStorerecovery — committed data stays visible, aborted data stays filtered, and committed transactional acks are not redelivered.V5ScalableTopicTransactionTest(docker, 2-broker cluster): produce/consume commit+abort, multi-topic atomic commit, consume-transform-produce, broker failover mid-transaction, and commit spanning a split / merge. Registered inpulsar-transaction.xml.V5TransactionTest.testCommitSpansSplit/testCommitSpansMergeand removed their stale "disabled" comments (the gap they described was specific to the old v4 RPC commit model; v5endTransactionis metadata-event driven).Verifying this change
This change is a test-only addition and is covered by the new tests. Locally:
V5TransactionScalableTest(8 tests) — stable across repeated runsV5TransactionRecoveryTest(3 tests) — stable across repeated runsV5ScalableTopicTransactionTest(6 docker tests) — stable across repeated runs (6/6, incl. failover and split/merge)V5Transactiontest— 7/7 with the split/merge tests re-enabledDoes this pull request potentially affect one of the following parts: