Modified the Incompatible data type section for the Feature page #162
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
merlynm20
requested changes
Sep 24, 2025
| ### Handling Incompatible Data Type Changes | ||
|
|
||
| For type changes not supported by Iceberg v2 (like STRING to INT), OLake offers two options: | ||
| For data type changes not supported by Iceberg v2: |
There was a problem hiding this comment.
not supported or supported ?
docs/features/index.mdx
Outdated
| 2. Narrowing Type Conversions like: | ||
| - BIGINT to INT | ||
| - DOUBLE to FLOAT | ||
| - TIMESTAMP to DATE |
There was a problem hiding this comment.
i dont think timestamp to date is possible. Once check with tech
docs/features/index.mdx
Outdated
| - BIGINT to INT | ||
| - DOUBLE to FLOAT | ||
| - TIMESTAMP to DATE | ||
| - DECIMAL(P, S) to DECIMAL(P', S) where P' < P |
There was a problem hiding this comment.
once confirm this as well
| - DECIMAL(P, S) to DECIMAL(P', S) where P' < P | ||
|
|
||
| 1. **Schema Data Type Changes Enabled with DLQ** ([coming soon](https://github.com/datazip-inc/olake/issues/265)): | ||
| When a source value’s data type has a smaller range or precision than the destination column’s type, OLake treats this as a narrowing conversion and handles it seamlessly. For example, if the destination column is defined as BIGINT but the incoming values are INT, OLake recognizes that every INT value falls within BIGINT’s range and simply stores the values without error. This validation step ensures that compatible, smaller-range types are accepted even when Iceberg v2 would flag a mismatch. |
There was a problem hiding this comment.
not sure if iceberg v2 will flag for a mismatch during this.
docs/features/index.mdx
Outdated
| - Test schema changes in non-production environments first | ||
| - Document your schema and track changes over time | ||
| :::tip | ||
| All other incompatible type changes will be captured by OLake using a Dead Letter Queue column (DLQ) (feature coming soon). |
There was a problem hiding this comment.
Any other data type change willbe handled in a dead letter queue column
docs/features/index.mdx
Outdated
|
|
||
| ### Scenario 6: Incompatible Type Change | ||
|
|
||
| When a column changes from STRING to INT type:, incompatible with Iceberg v2, sync fails. |
There was a problem hiding this comment.
why are we mentioning that it is incompatible with v2 and sync is failing. This is a doc where we are explaining how olake handles evolution and data type changes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.