[improve][test] Add Debezium Oracle source integration test#62
[improve][test] Add Debezium Oracle source integration test#62david-streamlio wants to merge 1 commit into
Conversation
Adds a Testcontainers-based integration test for the Debezium Oracle source connector using gvenzl/oracle-free:23-slim-faststart. A SYSDBA init script enables archive log mode and supplemental logging, creates the c##dbzuser LogMiner user with the grant set from the Debezium documentation, and seeds a DEBEZIUM.PRODUCTS table in FREEPDB1. The test opens DebeziumOracleSource against the container and asserts that the initial snapshot emits records for the two seeded rows. Fixes apache#44
There was a problem hiding this comment.
Pull request overview
Adds an end-to-end Testcontainers integration test for the debezium/oracle connector module (DebeziumOracleSource) by standing up a real Oracle Free database plus a Pulsar broker for Pulsar-backed schema history, then asserting the initial Debezium snapshot produces records.
Changes:
- Added
DebeziumOracleSourceTestthat boots Oracle Free + Pulsar, configures the Oracle Debezium connector, and asserts snapshot records are emitted. - Added an Oracle container init SQL script to enable LogMiner prerequisites (archive log mode, supplemental logging), create required users/tablespaces, and seed a test table.
- Added Testcontainers + Pulsar client test dependencies to the Oracle Debezium module.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| debezium/oracle/src/test/resources/debezium-oracle-setup.sql | Initializes Oracle Free container for LogMiner CDC and seeds a test table used by the integration test. |
| debezium/oracle/src/test/java/org/apache/pulsar/io/debezium/oracle/DebeziumOracleSourceTest.java | New Testcontainers-based integration test exercising DebeziumOracleSource against a real Oracle DB and Pulsar schema history. |
| debezium/oracle/build.gradle.kts | Adds required test dependencies for Testcontainers and Pulsar client usage in the new test. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| int recordCount = 0; | ||
| Record<KeyValue<byte[], byte[]>> record; | ||
| while ((record = source.read()) != null) { | ||
| assertNotNull(record.getValue()); | ||
| log.info("Received CDC record: key={}", record.getKey().orElse(null)); | ||
| recordCount++; | ||
| record.ack(); | ||
| if (recordCount >= 2) { | ||
| break; | ||
| } | ||
| } |
|
@lhotari requesting your eyes on this one specifically — it's the least conventional of the integration tests in flight and touches shared build config, so an independent review seems warranted:
|
Fixes #44
Motivation
The
debezium/oraclemodule (DebeziumOracleSource) had no tests. This PR adds a Testcontainers integration test that exercises the connector end-to-end against a real Oracle database, mirroring the shape of the existingDebeziumMysqlSourceTest.Oracle is the heaviest Debezium variant to test: LogMiner-based CDC requires archive log mode, supplemental logging, a common (CDB-wide) user with an extensive grant set, and PDB-aware connector configuration. All of that is handled inside the test via a container init script.
Modifications
DebeziumOracleSourceTestindebezium/oracle:gvenzl/oracle-free:23-slim-faststart(the image family Debezium's own test infrastructure uses) plus aPulsarContainerfor the schema-history topic.src/test/resources/debezium-oracle-setup.sql, copied to/container-entrypoint-initdb.d/) enables archive log mode (SHUTDOWN IMMEDIATE/STARTUP MOUNT/ALTER DATABASE ARCHIVELOG), enables minimal supplemental logging, createsLOGMINER_TBStablespaces in CDB and PDB, createsc##dbzuserwith the LogMiner grant set from the Debezium 3.x documentation, and seeds aDEBEZIUM.PRODUCTStable (2 rows) inFREEPDB1withSUPPLEMENTAL LOG DATA (ALL) COLUMNS. The gvenzl entrypoint runs init scripts before printingDATABASE IS READY TO USE!, so the container wait strategy also covers the DB restart.DebeziumOracleSourcewithdatabase.dbname=FREE,database.pdb.name=FREEPDB1,table.include.list=DEBEZIUM.PRODUCTS,log.mining.strategy=online_catalog, and a Pulsar-backed schema history, then asserts the initial snapshot emits at least 2 records.debezium/oracle/build.gradle.kts: addstestcontainers,testcontainers-pulsar, andpulsar-clienttest dependencies.Notes on two dependency findings:
io.debezium:debezium-connector-oracle:3.4.2.Finalpullscom.oracle.database.jdbc:ojdbc11:21.15.0.0(plusorai18n/xdb/xmlparserv2) transitively onto the runtime classpath, so the driver also ends up in the NAR. No runtime-driver gap in this module as built today.GenericContainerinstead oforg.testcontainers.oracle.OracleContainer:debezium-bom:3.4.2.Finalconstrains Testcontainers core to 2.0.2, while theorg.testcontainers:oracle-freemodule only exists up to 1.21.4 and references shaded classes removed in core 2.x (NoClassDefFoundError: org.testcontainers.shaded.org.apache.commons.lang3.StringUtilsatOracleContainer.withPassword). UsingGenericContaineragainst core 2.0.2 avoids the mixed-version breakage and matches the approach already used byDebeziumMysqlSourceTest.Verifying this change
Run locally (Docker required):
Local result (macOS/arm64, Docker Desktop): PASSED in ~37s once the image was pulled; the test received snapshot records with keys
{"ID":1}and{"ID":2}. The init script was additionally verified standalone:V$DATABASE.LOG_MODE=ARCHIVELOG,SUPPLEMENTAL_LOG_DATA_MIN=YES,c##dbzusercan connect toFREEPDB1and sees the 2 seeded rows, and the container logs contain noORA-/SP2-errors.:debezium:pulsar-io-debezium-oracle:check(spotless/checkstyle) passes.CI-runner risk worth calling out explicitly: the
gvenzl/oracle-free:23-slim-faststartimage is large (~1.6 GB) and Oracle Free wants ~2 GB of memory; the init script also restarts the database once to enable archive log mode. Startup on a standard GitHub-hosted runner will be minutes, not seconds — the test uses a 6-minute container startup timeout, a 5-minute Awaitility window for the snapshot, and a 10-minute overall test timeout. If the runner is memory-constrained alongside other container tests, this test will be the first to feel it.