[improve][test] Add Debezium SQL Server source integration test#60
[improve][test] Add Debezium SQL Server source integration test#60david-streamlio wants to merge 1 commit into
Conversation
Fixes apache#43 The debezium/mssql module had no tests. Add a Testcontainers-based integration test following the existing DebeziumMysqlSourceTest pattern: start an MSSQLServerContainer (SQL Server 2022) with the SQL Server Agent enabled (required for CDC capture jobs), enable CDC on the test database and table, start a PulsarContainer for schema history, and assert the initial snapshot produces CDC records via source.read(). The SQL Server connector runs in multi-partition mode and requires a task.id; since DebeziumSource starts the connector task directly (bypassing SqlServerConnector.taskConfigs which would normally assign it), the test sets task.id explicitly. Adds a testcontainers-mssqlserver entry to the version catalog (version resolved via the existing Testcontainers BOM).
|
Note for reviewers: the explicit |
There was a problem hiding this comment.
Pull request overview
Adds CI coverage for the Debezium SQL Server source connector by introducing a Testcontainers-based integration test, aligning debezium/mssql with existing Debezium connector modules that already have similar coverage.
Changes:
- Added a new MSSQL CDC integration test for
DebeziumMsSqlSourceusing Testcontainers (SQL Server + Pulsar) and Awaitility-style assertions. - Added
testcontainers-mssqlserverto the version catalog and wired new test dependencies into thedebezium/mssqlmodule. - Ensured connector startup succeeds in the test by explicitly setting
task.id.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| gradle/libs.versions.toml | Adds a version-catalog alias for org.testcontainers:mssqlserver to support MSSQL Testcontainers usage. |
| debezium/mssql/build.gradle.kts | Adds the necessary testImplementation dependencies for the new MSSQL + Pulsar integration test. |
| debezium/mssql/src/test/java/org/apache/pulsar/io/debezium/mssql/DebeziumMsSqlSourceTest.java | New integration test that provisions SQL Server CDC and asserts snapshot/CDC records are emitted by the source. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| private static final String PULSAR_IMAGE = | ||
| System.getenv().getOrDefault("PULSAR_TEST_IMAGE", "apachepulsar/pulsar:4.1.3"); | ||
|
|
||
| private static final int MSSQL_PORT = 1433; | ||
|
|
||
| private MSSQLServerContainer<?> mssqlContainer; | ||
| private PulsarContainer pulsarContainer; | ||
| private PulsarClient pulsarClient; | ||
| private DebeziumMsSqlSource source; | ||
|
|
||
| @BeforeMethod | ||
| public void setup() throws Exception { | ||
| mssqlContainer = new MSSQLServerContainer<>( | ||
| DockerImageName.parse("mcr.microsoft.com/mssql/server:2022-latest")) | ||
| .acceptLicense() |
| Awaitility.await() | ||
| .atMost(60, TimeUnit.SECONDS) | ||
| .pollInterval(500, TimeUnit.MILLISECONDS) | ||
| .untilAsserted(() -> { | ||
| int recordCount = 0; | ||
| Record<KeyValue<byte[], byte[]>> record; | ||
| while ((record = source.read()) != null) { | ||
| assertNotNull(record.getValue()); | ||
| log.info("Received CDC record: key={}", record.getKey().orElse(null)); | ||
| recordCount++; | ||
| record.ack(); | ||
| if (recordCount >= 2) { | ||
| break; | ||
| } | ||
| } | ||
| assertTrue(recordCount >= 2, | ||
| "Expected at least 2 CDC records from initial snapshot, got " + recordCount); | ||
| }); |
|
@lhotari flagging this for your review because of the
Two things worth your judgement: whether the right fix in #61 is to have |
Fixes #43
Motivation
The
debezium/mssqlmodule has zero tests —DebeziumMsSqlSourceis not exercised at all in CI. This adds a Testcontainers-based integration test following the existing pattern indebezium/mysql(DebeziumMysqlSourceTest).Modifications
gradle/libs.versions.toml: add atestcontainers-mssqlservercatalog entry (no version needed — it resolves via the Testcontainers BOM imported by the dependencies platform).debezium/mssql/build.gradle.kts: addtestImplementationdependencies (testcontainers-mssqlserver,testcontainers-pulsar,pulsar-client).debezium/mssql/src/test/java/org/apache/pulsar/io/debezium/mssql/DebeziumMsSqlSourceTest.java: new integration test thatMSSQLServerContainer(mcr.microsoft.com/mssql/server:2022-latest) withMSSQL_AGENT_ENABLED=true— the SQL Server Agent is required to run the CDC capture jobs,testdb, enables CDC on the database (sys.sp_cdc_enable_db) and on aproductstable (sys.sp_cdc_enable_table), and inserts two rows over plain JDBC assa,PulsarContainerfor the Debezium schema history topic,DebeziumMsSqlSourceand asserts (via Awaitility) that the initial snapshot yields at least 2 CDC records fromsource.read().One SQL Server-specific detail: the test sets
task.id=0explicitly. The SQL Server connector runs in multi-partition mode and its metrics require a task id, butDebeziumSourcestarts the connector task directly (it does not set the adapter'skafkaConnectorSourceClassconfig), soSqlServerConnector.taskConfigs()— which would normally assign the task id — never runs. Without it,source.open()fails with an NPE in Debezium's JMX metric-name construction.Verifying this change
Ran locally with Docker (macOS, Docker Desktop):
./gradlew :debezium:pulsar-io-debezium-mssql:compileTestJava— BUILD SUCCESSFUL./gradlew :debezium:pulsar-io-debezium-mssql:test—DebeziumMsSqlSourceTest > testMsSqlCdcEvents PASSED, BUILD SUCCESSFUL in 48s (with images already pulled)./gradlew :debezium:pulsar-io-debezium-mssql:check -x test— BUILD SUCCESSFULDocumentation
doc-not-neededTest-only change; no user-facing behavior is modified.