Performance Issues with Compaction

[nagraham]

Summary

We observe very slow compaction times across a number of different tables, and we'd like to fix it.

Desired Result

It should take a 1-3 seconds to compact 22MB, and yet I consistently see at least 60 seconds.

Examples

We ran compaction on a table where it only needed to compact 5 files totaling 315KB of data. According to the metrics from the library, the batch fetch duration was 85,514ms. The total job time (which we measure ourselves) was 93,092ms.

Here is a summary of 10 other tables compacted over the course of an hour.

Datetime	Input Files	Input Size (bytes)	Fetch Time	Total Time
2026-01-09T18:18:43	8	153,606	12,777ms	37,641ms
2026-01-09T18:16:31	231	11,704,639,665	1,324,772ms	536,109ms
2026-01-09T18:13:56	21	255,614,268	95,795ms	110,706ms
2026-01-09T18:10:59	8	11,359,652	74,555ms	95,978ms
2026-01-09T18:07:26	8	144,834	32,370ms	50,014ms
2026-01-09T17:49:55	5	2,408,452	684,052ms	692,934ms
2026-01-09T17:48:06	5	1,781,963	241,175ms	251,488ms
2026-01-09T17:34:13	5	17,838,726	140,258ms	152,979ms
2026-01-09T17:33:56	9	199,839	91,041ms	102,836ms
2026-01-09T17:21:05	5	1,465,254	416,035ms	421,645ms

Reproduction

I can reproduce long times from my laptop on my home network (600Mbps download, 38Mbps upload). I edited the rest catalog example in the iceberg-compaction library to run against my table in R2 Data Catalog. I generated fake data tables with N number of files and M number of rows, and loaded them into test tables. I tried adding updates/deletes. I also tried with greater number of columns. I ran with similar configurations as our production system. I hard coded a timer in the datafusion/mod.rs::rewrite_files that times the reads, and could confirm read time dominates the total time locally.

Test	Columns	Files	Rows per file	Size per file	Deletes?	Time to Read	Time to compact
Test 1	4	5	6000	68KB	No	9s	17-18s
Test 2	8	5	6000	138KB	No	12s	21-22s
Test 3	8	10	100,000	2.3MB	No	34-35s	60s
Test 4	8	5	6000	23MB / 138MB	Yes	26s	47s
Test 5	8	10	100,000	2.3MB	Yes	43s	75s
Test 6	32	5	1000	125KB	No	35s	44s

Root Cause

Iceberg-rust's ArrowReader only reads one range of bytes at a time. Each read requires an HTTP request to the same object. It reads a range for each column in the table. So as the number of columns in a table increases, so does the number of http requests to read the whole file.

Algorithmically, the number of http requests is O(N x M), where N is the number of columns, and M is the number of files to read. Prefetching reduces the number of http requests to O(M).

Proof: Static analysis of code

• ArrowFileReader only implements get_bytes on the ArrowAsyncReader trait. iceberg-rust/arrow/reader.rs.
• ArrowAsyncReader::get_byte_ranges will fall back to sequentially calling get_bytes (arrow-rs/parquet/arrow/async_reader.rs

Proof: The data from local testing (see previous section).

Solutions

I prototyped two solutions which can work. Both eliminated re-reading columns. I'll outline a summary of both approaches. I recommend Option A for now because it's cheaper to implement and it has better performance for compaction. Option B can be submitted as a general purpose improvement.

• (Recommended) Option A: Pre-fetch files in iceberg-compaction
  • (+) Aligns with compaction use case, where we know we always want to download 100% of the file.
  • (+) 2x faster than Option B in our tests
  • (+) Less effort than committing a fix to iceberg-rust
  • (-) Read more data into memory at once (although, we were already expecting to load the full file into memory anyway).
• Option B: Implement get_file_ranges on ArrowFileReader to actually read multiple ranges at once
  • (+) This is a general purpose improvement that can speed up queries as well
  • (-) Twice as slow a Option A in our tests (probably because it reads once for metadata and once again for the data).
  • (-) More effort to implement and to iceberg-rust, and then to add here.

How to implement Option A: Pre-fetch Files in Iceberg-compaction

I added the optimization in iceberg_file_task_scan.rs. The ArrowReaderBuilder does not have a way of piping in bytes or files directly. The only interface to providing files is through the iceberg-rust FileIO object (link to code), which you get from the table. So what I did is use the FileIO instance passed into the function to fetch the file. Then I created a new FileIO which uses a Storage::Memory implementation, and I loaded the file into the memory storage.

This might not be the original intent of the Storage::Memory enum variant, but it actually fits the purpose of what we are trying to do. Instead of having the ArrowReader trigger calls to a remote object store using a FileIO with an S3::Storage, we want to give it bytes that are held in memory. We don't want it to call the object store at all. Thus, the Storage::Memory is a perfect fit.

Performance Tests

Note that I skipped Test 1 when evaluating the improvement

Test	Columns	Files	Rows per file	Size per file	Deletes?	Old Read	Old Total	Improved Read	Improved Total	Read Speed Up
Test 1	4	5	6000	68KB	No	9s	17-18s	n/a	n/a	n/a
Test 2	8	5	6000	138KB	No	12s	21-22s	2.3s	12s	5x
Test 3	8	10	100,000	2.3MB	No	34-35s	60s	9s	29s	4x
Test 4	8	5	6000	23MB / 138MB	Yes	26s	47s	3s	24s	8x
Test 5	8	10	100,000	2.3MB	Yes	43s	75s	9.5s	42s	4x
Test 6	32	5	1000	125KB	No	35s	44s	2.2s	12s	16x

[chenzl25]

Hi @nagraham Personally, I would like the Option B. Since this issue only happens when compacting small files, so the gaps among these ranges should be small get_byte_ranges(&mut self, ranges: Vec<Range<u64>>). I think we can merge these small IO to a large IO and later split the single large byte back to multiple ranges.

[Li0k]

hi @nagraham . Thank you very much for your proposal and testing. Regarding the small file I/O issue, in our test scenarios, we mitigate the I/O problem by increasing the input parallelism. This is because the root cause is that the I/O of small files cannot be pipelined. I hope you can share two things with us:

1. The compaction configuration you adopted
2. The branch of your optimized code

[nagraham]

Thank you @Li0k and @chenzl25 for your responses! It helped call out gaps in the explanation and my tests, so I hope now I'll be able to clear things up.

This affects all files -- not just small ones

Since this issue only happens when compacting small files

This is not what I said, but sorry if perhaps what I wrote above led you to assume this. This performance issue impacts large files as well as small ones. It doesn't have the same impact, but it's still a nice improvement. To demonstrate this, I ran two new tests with bigger files (50-60MB), and I compacted them into 256MB files. I'll put the configuration code snippet below.

• Test 7: I uploaded 10 files from the year 2024 in the NYC Taxi public dataset. I partitioned by year, so it all grouped into 2024. There were 10 files which were roughly 60MB each. There are 19 columns in that table.
• Test 8: I generated a test table with 32 columns. I created 10 files with 500,000 rows each. Each file is roughly 53MB.

Test	Columns	Files	Rows per file	Size per file	Deletes?	Old Read	Old Total	Improved Read	Improved Total	Read Speed Up
Test 7	19	10	???	60MB	No	50s	269s	30s	247s	1.6x
Test 8	32	10	500,000	53MB	No	60s	222s	17s	176s	3.5x

Two points about these tests:

• The long total time is due to writes: my home internet upload speed is very slow. Writing a 256MB compacted file takes forever. So focus on the read values.
  • Uploading 256MB in 32MB chunks @ 38Mpbs ~= 55-60s per file
• These tables were big enough to have multiple concurrent file groups executing at once. The read times here are the total read time for the worst performing file group.

This is not an IO / parallelism problem

Regarding the small file I/O issue, in our test scenarios, we mitigate the I/O problem by increasing the input parallelism.

This is not an I/O parallelism problem. We have tried increasing the max_parallelism parameter multiple times to no avail -- that is why I started this deep dive. Please review the "Root Cause" section in the description of the issue, where I have outlined the code in iceberg-rust that is the bottleneck. The default implementation of get_byte_ranges is a for loop which sequentially calls get_bytes. This for-loop is serial -- it is not run concurrently.

// https://github.com/apache/arrow-rs/blob/main/parquet/src/arrow/async_reader/mod.rs#L79

fn get_byte_ranges(&mut self, ranges: Vec<Range<u64>>) -> BoxFuture<'_, Result<Vec<Bytes>>> {
    async move {
        let mut result = Vec::with_capacity(ranges.len());

        // 👇 this for loop is the problem
        for range in ranges.into_iter() {
            let data = self.get_bytes(range).await?;
            result.push(data);
        }

        Ok(result)
    }
    .boxed()
}

That means this currently affects singled-threaded reading of files. It occurs regardless of the value of max_parallelism. I hypothesize that fixing this bottleneck could improve read performance in your system as well.

Option A or B?

Personally, I would like the Option B. Since this issue only happens when compacting small files, so the gaps among these ranges should be small ...

I have proven it's not just a small file problem, so are there other good reasons Option B would be superior? I think the reasons for Option A (better performance, easier to implement/deploy, fits with the compaction use case) are still more compelling. And I'd like to get this improvement implemented quickly so we can evaluate it's impact in production.

Also, an implementation of Option A would include a file_prefetch_enabled parameter in the configuration so this can be controlled by the user. I'd default it to false since it's still experimental.

Still Disagree? Happy to chat!

If you still disagree with the conclusions I've reached I am more than happy to hear more concerns. I'd love to see some counter tests / data points, or perhaps something we're missing in the configuration / code.

Example of Option A: Prefetching files

Here is an implementation of Option A, which I pushed to my fork: nagraham/improve-read-performance

Compaction Configuration Used in Tests

    let planning_config = SmallFilesConfigBuilder::default()
        .target_file_size_bytes(1_000_000_000_u64)
        .small_file_threshold_bytes(150_000_000_u64)
        .max_parallelism(4_usize)
        .grouping_strategy(GroupingStrategy::BinPack(BinPackConfig::new(
            256_000_000_u64,
        )))
        .group_filters(
            GroupFiltersBuilder::default()
                .min_group_file_count(2_usize)
                .build()
                .unwrap(),
        )
        .build()
        .unwrap();

    let execution_config = CompactionExecutionConfigBuilder::default()
        .target_file_size_bytes(256_000_000_u64)
        .enable_normalized_column_identifiers(false)
        .enable_dynamic_size_estimation(true) // Enable to learn compression ratio
        .size_estimation_smoothing_factor(0.3)
        .max_concurrent_closes(1) // Sequential closes for immediate ratio learning
        .max_record_batch_rows(1024)
        .max_concurrent_compaction_plans(4)
        .build()
        .unwrap();

    let compaction_config = CompactionConfigBuilder::default()
        .planning(CompactionPlanningConfig::SmallFiles(planning_config))
        .execution(execution_config)
        .build()
        .unwrap();

    let compaction = CompactionBuilder::new(catalog.clone(), table_ident.clone())
        .with_config(Arc::new(compaction_config))
        .with_catalog_name("rest_catalog".to_owned())
        .build();

[chenzl25]

Here is an implementation of Option A, which I pushed to my fork: nagraham/improve-read-performance

Cool. I got the whole idea of the Option A now. You first download the whole parquet file from S3 into memory first, so we don't need to fetch each ranges step by step and it is a good fit to the compaction use case, since compaction always requires all columns. So it looks good to me. As for the memory consumption, the max memory consumption would be execution parallelism x parquet file size.

Also, an implementation of Option A would include a file_prefetch_enabled parameter in the configuration so this can be controlled by the user. I'd default it to false since it's still experimental.

A new parameter file_prefetch_enabled sounds good.

[Li0k]

@nagraham I completely endorse the file_prefetch_enabled approach, but it is highly dependent on users understanding their own workloads and strictly limiting task parallelism.

I would like to emphasize this point again: even if you set a larger max_parallelism, it does not mean that your tasks will use this maximum value. This is calculated by an algorithm and depends on the file-count and file-size included in the tasks. Therefore, if you want to increase the parallelism for small tasks, you need to adjust the relevant configurations accordingly. I believe that increasing the input-parallelism will effectively improve the compaction performance of small files, and I recommend you to give it a try. (like max_file_count_per_partition and min_size_per_partition).

If you are calling the underlying interface, you can manually adjust the parallelism of the file-group.

[nagraham]

highly dependent on users understanding their own workloads

I think that's a fair callout. I'll also call out that users with complex use cases will need to have a sharp understanding of their workloads, and will need to be able to configure advanced optimizations. That's where we are at the moment.

strictly limiting task parallelism.

Task parallelism isn't our bottleneck right now. It's network I/O. So that's what we must address first, even at the cost of task parallelism.

I'll also call out that we need to be able to run this on relatively small nodes (2-4 CPU, and 4-8 GB). We will probably need to have a lower bound of parallelism (~4 or ~8). With async I/O, we don't need to restrict reads to a 1:1 relationship with task runners. I think there may be further room to download more files in a buffered stream. That's a hypothesis: we don't yet know if I/O will still be our bottleneck after this optimization is complete.

I believe that increasing the input-parallelism will effectively improve the compaction performance of small files, and I recommend you to give it a try. (like max_file_count_per_partition and min_size_per_partition).

One of our primary requirements is to hit the target output file size, even for partitioned files. If customers set 128MB or 256MB for the file size, then that's the target. We don't want to compromise this size for performance. So min_size_per_partition must equal target_file_size in our case. We're not going to lower it -- and thus lower the target output size -- in an attempt to get more parallelism.

That said, I did try two configurations and saw a nice improvement for our small files test.

• max_file_count_per_partition: I set this to 8, down from 32
• max_record_batch_rows: i set this to 4096, up from 1024

I only used 1 test scenario (10 files, 2.2MB per file, 8 columns), just to see the effect. I re-ran the test and got slightly different timings than the previous run displayed in the top-level summary.

Name	Total Read Time	Max File Read	Total Compaction Time
Baseline: no prefetch, no config update	30s	3.5s	53s
Prefetch, no config update	6.5s	1.0s	27s
No Prefetch, Config Update	16s	3.0s	37s
Prefetch, Config Update	2.3s	0.6s	23s

This looks promising! It provides a big boost on top of prefetching as well. I'm going to run some more tests and see whether we still see improvements in other scenarios.

So now I'll make another suggestion: could we perhaps include performance tuning documentation that helps users make wise decisions on what settings to use on the configuration settings given their workload?