This branch is a research prototype built on Apache Iceberg 1.12.0 (tag apache-iceberg-1.12.0). It adds compaction maps: a compact data structure that records the position transformations applied by a compaction so that concurrent transactions writing position deletes (or deletion vectors) can be rebased onto the new layout instead of restarted.
Compactions and concurrent updates logically commute — compaction does not change table contents — but in current table formats they conflict on direct file references. A compaction map captures, per run of rows, the move from a source file to one or more target files. Either side of a conflict can use the map to rewrite its position-delete references and commit, with no global coordination beyond Iceberg's existing snapshot pointer swap.
Chris Douglas and Joseph M. Hellerstein. Commutative Compaction. 1st International Workshop on Data FORMATS for Modern Architectures and Workloads (FORMATS '26), May 31–June 5, 2026, Bengaluru, India. ACM. https://doi.org/10.1145/3802514.3809174
The paper introduces compaction maps, the rebase operation, and the remapping policy, and evaluates an Apache Iceberg prototype on Apache Spark 3.5 and 4.0 with both v2 position delete files and v3 deletion vectors. That prototype is this branch's earlier version on Iceberg 1.10.1, kept on the cmpmap-1.10 branch.
End-to-end remapping was measured against object storage in three clouds (AWS S3 us-west-2, Azure ADLSv2 westus2, GCS uswest1) on commodity VMs (4 vCPU, 16 GiB), varying the number of runs in the compaction map (10 to 10K) and the size of the position delete file or deletion vector (1K to 1M deletes). These results are from the Iceberg 1.10.1 version:
- Repairing a 1M-delete commit against a 10K-run compaction map completes in under one second in every cloud, including all I/O — 0.34–0.45 s for deletion vectors and 1.8–2.3 s for position delete files.
- At 10K deletes (typical commit size) latency never exceeds half a second in any cloud, regardless of run count.
- Deletion vectors outperform position delete files across the board; Parquet encode dominates the PD cost while the DV roaring-bitmap region is only a few KiB.
- The compaction map itself is small: 10 runs occupies 2.6 KiB and 10K runs occupies 8.9 KiB on disk.
Cost is negligible relative to the compaction it commutes with — compactions typically run for minutes to hours.
See docs/docs/compaction_maps_bench.md for the full benchmark methodology and the JMH microbenchmark suite that drives the remapping-algorithm policy.
Implementation is on the cmpmap branch:
- Core data structure, builder, Avro storage, and manifest-list reference.
- Conflict detection and SERIALIZABLE-isolation integration in
BaseRowDelta/MergingSnapshotProducer. - Automatic remapping in
RewriteDataFilesCommitManagerfor Spark 3.5, 4.0 and 4.2, for both position delete files (v2) and deletion vectors (v3). Spark 4.1 is not supported. - Empirically-tuned remapping policy (IntervalTree / RangeQuery / StreamJoin) selected at runtime from the shape of the inputs.
Detailed documentation lives in docs/docs/:
- compaction_maps.md — user guide, table properties, conflict-resolution workflow.
- compaction_maps_impl.md — implementation walkthrough, schema, integration points.
- compaction_maps_impl_pseudocode.md — pseudocode for the remapping strategies.
- compaction_maps_bench.md — benchmark suites, methodology, and how to reproduce.
- compaction_maps_errata.md — design scope and known limitations (e.g. sort/z-order rewrites write no map, and maps are written in plaintext).
This branch builds with the standard Iceberg toolchain (Gradle, Java 17 or 21):
./gradlew :iceberg-core:compileJava
./gradlew :iceberg-core:test --tests "*CompactionMap*"
./gradlew :iceberg-core:test --tests "*Remapping*"
./gradlew spotlessApplySpark 4.0 and 4.2 build with Scala 2.13 (-DscalaVersion=2.13). Flink builds with the default Scala
2.12, so the two need separate Gradle invocations.
For general Iceberg build, engine-compatibility, and contribution information, see the upstream project at https://iceberg.apache.org.