feat: Add MERGE (UPSERT) support to Oracle Sink #636
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit enhances the Oracle Sink plugin to support `MERGE` operations, providing an idempotent way to handle data writes, especially in scenarios involving Spark task retries. New Features: - **UPSERT Operation:** A new `operation` property in the sink configuration allows users to specify "upsert". - **Merge Keys:** A `mergeKeys` property is introduced to define the column(s) for the `MERGE` statement's `ON` clause. - **`OracleETLDBOutputFormat`:** A new class extending `ETLDBOutputFormat` to construct the Oracle-specific `MERGE` SQL statement using a `USING DUAL` approach. - **Conditional Logic:** The sink now conditionally uses `OracleETLDBOutputFormat` when the "upsert" operation is selected. Test Improvements: - Updated `docker-compose.yml` to use `container-registry.oracle.com/database/free:latest` with healthcheck. - README.md updated to reflect new Oracle image and Podman instructions. - Increased test timeout in `DatabasePluginTestBase`. - Added `testDBSinkUniqueConstraintViolation` to `OracleSinkTestRun` to demonstrate ORA-00001 on duplicate INSERTs. - Added `testDBSinkMerge` to `OracleSinkTestRun` to validate the new UPSERT functionality. - Commented out `apache-rat-plugin` in the root `pom.xml` to potentially speed up local builds. This change addresses issues where task retries could lead to unique constraint violations (ORA-00001) by providing a true upsert mechanism.
Author
|
Internal notes: go/sf/64273740 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit enhances the Oracle Sink plugin to support
MERGEoperations, providing an idempotent way to handle data writes, especially in scenarios involving Spark task retries.New Features:
operationproperty in the sink configuration allows users to specify "upsert".mergeKeysproperty is introduced to define the column(s) for theMERGEstatement'sONclause.OracleETLDBOutputFormat: A new class extendingETLDBOutputFormatto construct the Oracle-specificMERGESQL statement using aUSING DUALapproach.OracleETLDBOutputFormatwhen the "upsert" operation is selected.Test Improvements:
docker-compose.ymlto usecontainer-registry.oracle.com/database/free:latestwith healthcheck.DatabasePluginTestBase.testDBSinkUniqueConstraintViolationtoOracleSinkTestRunto demonstrate ORA-00001 on duplicate INSERTs.testDBSinkMergetoOracleSinkTestRunto validate the new UPSERT functionality.apache-rat-pluginin the rootpom.xmlto potentially speed up local builds.This change addresses issues where task retries could lead to unique constraint violations (ORA-00001) by providing a true upsert mechanism.