Skip to content

feat: add delta writer for row-level changes#2219

Open
DAlperin wants to merge 1 commit intoapache:mainfrom
MaterializeInc:upstream/delta-writer
Open

feat: add delta writer for row-level changes#2219
DAlperin wants to merge 1 commit intoapache:mainfrom
MaterializeInc:upstream/delta-writer

Conversation

@DAlperin
Copy link

@DAlperin DAlperin commented Mar 6, 2026

Which issue does this PR close?

Closes #2218

What changes are included in this PR?

This PR adds support for row-level changes (inserts, updates, and deletes) to Iceberg tables through a new DeltaWriter implementation. This enables CDC (Change Data Capture), upsert operations, and efficient row-level deletions.

Key Components:

  1. RowDeltaAction Transaction (could be supplanted by feat(transaction): add RowDelta transaction action for row-level modi… #2203

    • New transaction type for applying row-level changes atomically
    • Supports both data files and delete files in a single transaction
    • Located in transaction/row_delta.rs
  2. Position Delete Writer

    • Writes position delete files for efficient row-level deletions
    • Deletes rows by file path and row position
    • Located in writer/base_writer/position_delete_writer.rs
  3. Delta Writer (Combined Writer)

    • Orchestrates data file, position delete, and equality delete writers
    • Intelligently routes operations based on row tracking
    • Memory-bounded row tracking with configurable limits
    • Falls back to equality deletes for older/evicted rows
    • Located in writer/combined_writer/delta_writer.rs

Input Format:
The DeltaWriter expects RecordBatch with an operations column:

  • Value 1 = Insert/Update (write to data file)
  • Value -1 = Delete (write to delete file)

Memory Management:

  • Tracks recently written rows for efficient position deletes
  • Configurable max_seen_rows limit (default: 100,000)
  • FIFO eviction when limit is reached
  • Can be disabled (set to 0) to use only equality deletes

Are these changes tested?

Unit tests for each new writer are added.

This commit adds support for row-level changes (inserts, updates, and deletes) 
to Iceberg tables through a new DeltaWriter implementation. This enables CDC 
(Change Data Capture), upsert operations, and efficient row-level deletions.

Key Components:

1. RowDeltaAction Transaction
   - New transaction type for applying row-level changes atomically
   - Supports both data files and delete files in a single transaction
   - Located in transaction/row_delta.rs

2. Position Delete Writer
   - Writes position delete files for efficient row-level deletions
   - Deletes rows by file path and row position
   - Located in writer/base_writer/position_delete_writer.rs

3. Delta Writer (Combined Writer)
   - Orchestrates data file, position delete, and equality delete writers
   - Intelligently routes operations based on row tracking
   - Memory-bounded row tracking with configurable limits
   - Falls back to equality deletes for older/evicted rows
   - Located in writer/combined_writer/delta_writer.rs

Input Format:
The DeltaWriter expects RecordBatch with an operations column:
- Value 1 = Insert/Update (write to data file)
- Value -1 = Delete (write to delete file)

Memory Management:
- Tracks recently written rows for efficient position deletes
- Configurable max_seen_rows limit (default: 100,000)
- FIFO eviction when limit is reached
- Can be disabled (set to 0) to use only equality deletes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support for a DeltaWriter

1 participant