feat!: Support compression codecs for Avro Files (including manifest and manifest lists) by emkornfield · Pull Request #1851 · apache/iceberg-rust

emkornfield · 2025-11-13T00:53:58Z

Which issue does this PR close?

Partially fixes [Bug] iceberg-rust does not respect compression settings for metadata & avro #1731

What changes are included in this PR?

Previously these properties where not honored on tabel properties.

Adds table properties for these values.
Plumbs them through for writers.

Are these changes tested?

Added unit tests

BREAKING CHANGE: Adds codec parameter to some public functions. By default start compressing manifests and manifest lists.

Previously these properties where not honored on tabel properties. - Adds table properties for these values. - Plumbs them through for writers.

kevinjqliu · 2025-11-13T03:11:13Z

looks like a clippy issue in CI

error: this function has too many arguments (8/7)
   --> crates/iceberg/src/spec/manifest_list.rs:243:5
    |
243 | /     fn new(
244 | |         format_version: FormatVersion,
245 | |         output_file: OutputFile,
246 | |         metadata: HashMap<String, String>,
...   |
251 | |         compression_level: u8,
252 | |     ) -> Self {
    | |_____________^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#too_many_arguments
    = note: `-D clippy::too-many-arguments` implied by `-D warnings`
    = help: to override `-D warnings` add `#[allow(clippy::too_many_arguments)]`

emkornfield · 2025-11-13T06:45:26Z

looks like a clippy issue in CI

Yeah working on a fix.

emkornfield · 2025-11-13T18:09:36Z

@liurenjie1024 @kevinjqliu would you have time to review?

kevinjqliu

Generally LGTM, I left a few comments around using the same string as the java implementation

crates/iceberg/src/spec/table_properties.rs

crates/iceberg/src/spec/avro_util.rs

crates/iceberg/src/spec/manifest_list.rs

crates/iceberg/src/transaction/snapshot.rs

crates/iceberg/src/spec/table_metadata.rs

emkornfield · 2025-11-16T00:44:54Z

crates/iceberg/src/spec/avro_util.rs

+///
+/// # Compression Levels
+///
+/// The compression level mapping is based on miniz_oxide's CompressionLevel enum:


clarify this is for gzip.

nit: maybe merge this comment with the one above (L61-L66), the compression levels are explained twice

crates/iceberg/src/spec/manifest/writer.rs

kevinjqliu

this looks great, thanks!
adding the "breaking" label since we're modifying a few public functions

crates/iceberg/src/spec/table_properties.rs

crates/iceberg/src/spec/table_metadata.rs

crates/iceberg/src/transaction/snapshot.rs

crates/iceberg/src/spec/manifest/writer.rs

crates/iceberg/src/spec/avro_util.rs

emkornfield · 2025-11-17T22:02:22Z

@kevinjqliu thanks for the reviews, I also added more e2e tests for manifest and manifest list writers to confirm compression.

emkornfield · 2025-11-18T00:40:05Z

adding the "breaking" label since we're modifying a few public functions

I'm not sure that I can access labels as a contributor, I put a breaking change note in the PR description.

liurenjie1024

Thanks @emkornfield for this pr. In general, I think this is a useful feature implementation. But we should not mix two things in one pr, I would suggest to split them into two prs: one for table metadata, and one for manifests.

liurenjie1024 · 2025-11-18T10:04:11Z

crates/iceberg/src/spec/table_properties.rs

+fn parse_optional_property<T: std::str::FromStr>(
+    properties: &HashMap<String, String>,
+    key: &str,
+) -> Result<Option<T>, anyhow::Error>


We should use this crate's error.

Do we really need this method? I think passing a None to default value in parse_property would be enough?

I'm new to rust but would that work? Doesn't parse_property, always return an extension of String instead of an Option? It is useful to know if the value was actually configured (and thus the option return)?

I think in iceberg all table properties are optional, if they have default values, then we should return a default value, otherwise we should return an error.

I'll see if I can make this work with empty string.

I think in iceberg all table properties are optional, if they have default values, then we should return a default value, otherwise we should return an error.

The problem here is that the default value is dependent on the value for compression, it seems baking this into the parsing layer is coupling two concerns (business logic for parsing) and the actual values. To eliminate this method I've used a sentinel value for the old method. Let me know if that is seems OK.

crates/iceberg/src/spec/table_metadata.rs

emkornfield · 2025-11-19T21:57:27Z

crates/iceberg/src/spec/table_properties.rs


 // Helper function to parse a property from a HashMap
 // If the property is not found, use the default value
-fn parse_property<T: std::str::FromStr>(


this is cleanup because it sounds like the crate's errors are preferred I can revert this.

emkornfield · 2025-11-19T22:04:07Z

@liurenjie1024 split of this PR to just be about Avro. Also based on feedback, I cleaned up some additional imports/Errors that weren't using the Crate's error before.

emkornfield · 2025-12-16T23:13:43Z

crates/iceberg/src/spec/manifest_list.rs

+
+    #[tokio::test]
+    async fn test_manifest_list_writer_with_compression() {
+        use std::fs;


move these to top level.

Resolved conflict in crates/iceberg/src/spec/manifest/writer.rs by keeping both tests: - test_manifest_writer_with_compression (from fix_compression) - test_v3_delete_manifest_delete_file_roundtrip (from databricks/main) Also added missing Codec::Null parameter to ManifestWriterBuilder::new call in the incoming test.

emkornfield · 2026-01-05T20:31:36Z

crates/iceberg/src/spec/table_properties.rs

+                let level_raw = parse_property(
+                    props,
+                    TableProperties::PROPERTY_AVRO_COMPRESSION_LEVEL,
+                    255u8,


maybe make a constant?

emkornfield · 2026-02-02T04:26:59Z

Moving to draft until we finalize #1876

github-actions · 2026-03-06T00:27:49Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

emkornfield · 2026-03-06T00:30:08Z

Still waiting on Moving to draft until we finalize #1876, I'll resolve conflicts.

## Which issue does this PR close? Split off from #1851 - Partially fixes #1731. ## What changes are included in this PR? This change honors the compression setting for metadata.json file (`write.metadata.compression-codec`). ## Are these changes tested? Add unit test to verify files are gzipped when the flag is enabled. BREAKING CHANGE: Make `write_to` take `MetadataLocation` --------- Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io>

## Which issue does this PR close? This is an intermediate PR for #1731 I'm splitting out changes from #1851 to the compression codec to make it easier to review. Once we decide on approach here and merge it I'll update #1851 accordingly. ## What changes are included in this PR? - Add optional compression level to gzip and zstd (needed for when avro compression usage). - Add Snappy as a compression codec (also will be used for Avro) - Manually code up some previously auto-generated methods as a result. AI helped with an initial version of this PR. ## Are these changes tested? Additional unit tests

) ## Which issue does this PR close? Split off from apache#1851 - Partially fixes apache#1731. ## What changes are included in this PR? This change honors the compression setting for metadata.json file (`write.metadata.compression-codec`). ## Are these changes tested? Add unit test to verify files are gzipped when the flag is enabled. BREAKING CHANGE: Make `write_to` take `MetadataLocation` --------- Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io>

emkornfield added 2 commits November 13, 2025 00:45

feat!: Support compression codecs for JSON metadata and Avro

665b970

Previously these properties where not honored on tabel properties. - Adds table properties for these values. - Plumbs them through for writers.

fmt

091d3bc

emkornfield added 2 commits November 13, 2025 06:54

fix clippy

41a8c1c

clippy again

51e781e

kevinjqliu reviewed Nov 15, 2025

View reviewed changes

emkornfield added 2 commits November 16, 2025 00:27

wip

253bf59

address comments

8bdb52d

emkornfield commented Nov 16, 2025

View reviewed changes

crates/iceberg/src/spec/manifest/writer.rs Outdated Show resolved Hide resolved

emkornfield added 2 commits November 16, 2025 00:52

no clone needed

9d27116

test compression works

d1ee0b2

kevinjqliu reviewed Nov 17, 2025

View reviewed changes

emkornfield added 2 commits November 17, 2025 21:51

comments

393622f

update tests

46fdb8e

emkornfield requested a review from kevinjqliu November 17, 2025 22:02

liurenjie1024 suggested changes Nov 18, 2025

View reviewed changes

emkornfield added 2 commits November 19, 2025 21:29

address comments

d50cb7d

remove parse optional property

5370f77

emkornfield commented Nov 19, 2025

View reviewed changes

emkornfield changed the title ~~feat!: Support compression codecs for JSON metadata and Avro~~ feat!: Support compression codecs for Avro Files (inlcuding manifest and manifest lists) Nov 19, 2025

fmt

6b3d8ed

emkornfield requested a review from liurenjie1024 November 19, 2025 22:03

emkornfield mentioned this pull request Nov 19, 2025

feat: Honor compression settings for metadata.json on write #1876

Merged

kevinjqliu added the breaking label Nov 21, 2025

emkornfield added 4 commits December 16, 2025 22:22

update cargo lock

ef41b4d

add missing args

d9603ff

address clippy

e43e0cc

Fmt

0ec5784

emkornfield commented Dec 16, 2025

View reviewed changes

emkornfield added 3 commits December 16, 2025 23:30

move use statements to top level

ed71f0e

remove duplicate imports

edb6886

emkornfield commented Jan 5, 2026

View reviewed changes

emkornfield requested a review from liurenjie1024 January 5, 2026 20:31

Merge remote-tracking branch 'databricks/main' into fix_compression

1a5e8fc

emkornfield marked this pull request as draft February 2, 2026 04:27

github-actions bot added the stale label Mar 6, 2026

github-actions bot removed the stale label Mar 7, 2026

Merge remote-tracking branch 'databricks/main' into fix_compression

1ca553a

blackmwk added the not-stale label Mar 12, 2026

emkornfield added 3 commits March 19, 2026 17:39

Merge remote-tracking branch 'databricks/main' into fix_compression

6dcd177

update to use compression codec

26f6d45

fmt

50bfe44

emkornfield mentioned this pull request Mar 25, 2026

feat!: Enhance compression codec enum. #2288

Merged

emkornfield added 4 commits March 31, 2026 18:05

Merge remote-tracking branch 'databricks/main' into fix_compression

175e61f

fmt

722cb55

simplify test

05db1d7

fixes

da34d53

Conversation

emkornfield commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

Uh oh!

kevinjqliu commented Nov 13, 2025

Uh oh!

emkornfield commented Nov 13, 2025

Uh oh!

emkornfield commented Nov 13, 2025

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emkornfield commented Nov 17, 2025

Uh oh!

emkornfield commented Nov 18, 2025

Uh oh!

liurenjie1024 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emkornfield commented Nov 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emkornfield commented Feb 2, 2026

Uh oh!

github-actions bot commented Mar 6, 2026

Uh oh!

emkornfield commented Mar 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

emkornfield commented Nov 13, 2025 •

edited

Loading