Repository navigation
Core: Preserve concurrent snapshots in replace transactions - #16943
wombatu-kun wants to merge 4 commits into
Conversation
|
@nastra @amogh-jahagirdar this one has been sitting since June 24 with no review at all. CI is green (56/56) and it still merges cleanly on current It fixes silent snapshot loss: when two Tagging you two because you handled the closest prior work in this path (#11671). A look whenever you have time would be much appreciated. |
49f6d44 to
c995464
Compare
c995464 to
e339dd4
Compare
wombatu-kun
changed the title
|
@pvary you merged #15389 in |
|
@wombatu-kun: I likely will not have time to dig into the details in the near future 😢 |
16dd446 to
8037fda
Compare
A replace transaction committed the replacement it built from the table as it began, so it dropped whatever other writers committed in between: a commit to main or to another branch, a tag, a property such as gc.enabled=false, a format upgrade (the replace downgraded the table), and it brought back snapshots expired since, without their manifest lists. A replace replaces the table's data, schema and partitioning, but is meant to keep its history. As in apache#16943, the catalog's table builder passes the transaction a function that builds the replacement from a table, and when the table changed, the transaction rebuilds the replacement on the refreshed table and applies its updates to it again, so the replace keeps what the other writers committed, as it would had it begun after them. Unlike apache#16943, which then commits the replacement as it began, a replace whose rebuilt replacement would have another schema, partition spec, sort order or format version, which the files it wrote depend on, or whose updates don't apply again (a schema update commits once), fails with CommitFailedException: committing it would drop the other writers' changes. Two racing replaces with the same definition still end as the second built it, now keeping the first's snapshot in history. Two whose schemas or sort orders differ make the second fail, so two CatalogTests cases expect that except over REST, whose server applies a replace's changes to the table as it is. A transaction built without the function commits as before, still without snapshots the table no longer has. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-by: Claude Code (claude-opus-5-5)
Generated-by: Claude Code (claude-opus-5-5) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
8037fda to
8c89ec2
Compare
|
@findepi you merged #18095 in this same cleanup code - could you take a look at this one? On metastore catalogs a replace transaction commits metadata built before a concurrent commit, so the other writer's snapshot drops out of history and an expired snapshot can come back (#16942, part of #16232, which RussellSpitzer called "a pretty serious issue"). The replacement is now rebuilt on the refreshed metadata, with a fallback to today's behavior when that isn't safe. It's rebased onto #18095. |
Closes #16942. Part of #16232.
A replace transaction commits metadata built when it started, so on
BaseMetastoreCatalogcatalogs (Hive, Glue, JDBC, Nessie, Hadoop) changes committed in between are lost: another writer's snapshot drops out of history, and a snapshot thatexpire_snapshotsremoved comes back. When the table changed, the replacement is now rebuilt on the refreshed metadata and the transaction's updates are replayed, which also keeps concurrent property changes, as REST does. If the rebuilt schema, spec, sort order or format version would differ from the transaction's (IDs included), or an update cannot be replayed (for example a merge-schemaCREATE OR REPLACE), the commit falls back to today's behavior. REST keeps merging server-side; path-basedHadoopTablesis a follow-up. Builds on #16975, #15092 and #15904.Test plan: each
CatalogTestssuite checks that a replace keeps a concurrent snapshot, does not bring back an expired snapshot, and, when a no-op schema update blocks the replay, keeps its own data and every manifest list;TestReplaceTransactionchecks the history and that the replace's files keep their sort order, on format versions 1 to 4. The history tests fail onmainfor metastore catalogs, the replay test fails without the fallback or the cleanup filter, and the sort-order test fails without the sort-order check.AI Disclosure
createOrReplacedrops concurrent writers' snapshots on commit retry #16942 (part of Core: REPLACE TABLE transaction silently overwrites concurrent committed changes #16232) by rebuilding a replace transaction on refreshed metadata when the table changed concurrently. The fix, its tests and this description are generated.