Proposed Change
Motivation
IncrementalFileCleanup.cleanFiles (core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java) determines which manifests/files to delete by scanning manifests across all currently-valid snapshots (afterExpiration.snapshots(), line ~90) in addition to the expired snapshots, to detect whether a file deleted by an expiring snapshot is still referenced by any valid snapshot.
This is correct in general, tables can have branches, tags, or cherry-picked/WAP commits where a valid ref's ancestry isn't a simple straight line back through the expired snapshots but for the common case of a single, purely linear history with no branches/tags/cherry-picks, this full scan does more work than necessary and scales with total valid-snapshot count rather than with the number of snapshots actually being expired.
Proposed optimization
Add a new opt-in flag, passed through the expireSnapshots call (e.g. via ExpireSnapshots builder API down into IncrementalFileCleanup), that when enabled, restricts the "still referenced?" scan in cleanFiles to only the first valid snapshot after the expiry cutoff, instead of all valid snapshots. On a linear history, if a manifest entry is marked DELETED in an expiring snapshot, the next valid snapshot after the cutoff is the earliest point where that file could still be referenced (e.g. re-added, or carried forward in a reused manifest); if it isn't referenced there, no later snapshot in a linear chain can reference it either.
This is scoped to IncrementalFileCleanup only (not ReachableFileCleanup), and must be strictly opt-in via the flag, the caller is responsible for guaranteeing the table has no branches/tags/cherry-picked snapshots when enabling it, since IncrementalFileCleanup already has non-trivial handling for ancestor tracking, cherry-picks (pickedAncestorSnapshotIds), and reverted manifests (manifestsToRevert) that assumes full-history awareness.
Proposal document
No response
Specifications
Proposed Change
Motivation
IncrementalFileCleanup.cleanFiles (core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java) determines which manifests/files to delete by scanning manifests across all currently-valid snapshots (afterExpiration.snapshots(), line ~90) in addition to the expired snapshots, to detect whether a file deleted by an expiring snapshot is still referenced by any valid snapshot.
This is correct in general, tables can have branches, tags, or cherry-picked/WAP commits where a valid ref's ancestry isn't a simple straight line back through the expired snapshots but for the common case of a single, purely linear history with no branches/tags/cherry-picks, this full scan does more work than necessary and scales with total valid-snapshot count rather than with the number of snapshots actually being expired.
Proposed optimization
Add a new opt-in flag, passed through the expireSnapshots call (e.g. via ExpireSnapshots builder API down into IncrementalFileCleanup), that when enabled, restricts the "still referenced?" scan in cleanFiles to only the first valid snapshot after the expiry cutoff, instead of all valid snapshots. On a linear history, if a manifest entry is marked DELETED in an expiring snapshot, the next valid snapshot after the cutoff is the earliest point where that file could still be referenced (e.g. re-added, or carried forward in a reused manifest); if it isn't referenced there, no later snapshot in a linear chain can reference it either.
This is scoped to IncrementalFileCleanup only (not ReachableFileCleanup), and must be strictly opt-in via the flag, the caller is responsible for guaranteeing the table has no branches/tags/cherry-picked snapshots when enabling it, since IncrementalFileCleanup already has non-trivial handling for ancestor tracking, cherry-picks (pickedAncestorSnapshotIds), and reverted manifests (manifestsToRevert) that assumes full-history awareness.
Proposal document
No response
Specifications