Skip to content

Conversation

@teskje
Copy link
Contributor

@teskje teskje commented Nov 19, 2025

This is a poc implementing an idea for resolving the current CatalogItemId/GlobalId mess.

Roughly, the mess is twofold:

  • Implementation complexity: Everywhere in adapter we have to deal with both ID types. Sometimes we need to translate them into one another, sometimes we have to pass around two IDs instead of one. Also, sometimes we use the logically wrong ID type (GlobalId instead of CatalogItemId) for backwards compat reasons. What's more, the CatalogItemId type is supposed to be a SQL-level concept, but it leaks into the storage and compute crates as well.
  • Interface complexity: If we keep both ID types, users will have to learn about the various layers of our implementation. Or, we attempt to paper over the split by providing views translating everything to the same ID type, at the cost of more noise in the system catalog, and some runtime cost for joins (and more implementation complexity).

What this poc tries is to remove CatalogItemId again. Here we simply make it an alias of GlobalId, to avoid a huge amount of typing, which is why all of the implementation complexity remains (though it could be removed by a huge amount of typing). Instead of treating item IDs as a namespace separate from the GlobalId namespace, we declare the ID of a catalog item to be the GlobalId of its most recent underlying collection. A necessary assumption for this is that each catalog item only has a single underlying collection (with some handwaving for MVs that have two collections that share a GlobalId) at any point in time.

Fixing the item ID to the latest GlobalId means that the item ID can change over time. This is a major departure from how things worked previously. The thinking is that nothing relies on item IDs being stable, but the point of this poc is to figure out if this is actually true.

If it is true, the approach has the benefit that it brings us mostly back into a world where we only have to worry about a single ID for each object. We don't quite get there for two reasons:

  • Collections depending on previous versions of an evolved object still need a way to reference that object. For this purpose, every evolvable object keeps a versions list with its previous GlobalIds, and the catalog provides mechanisms to resolve GlobalIds for older versions through these lists. (This exists already but can be cleaned up.)
  • Users might want to connect metrics collected for previous versions of a catalog item back to the item. For this purpose, we can provide an mz_object_versions system relations, that's mostly equivalent to the current mz_object_global_ids but models the versioning explicitly and is presumably easier to explain to users.

Motivation

Tips for reviewer

Checklist

  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

@teskje teskje force-pushed the poc-no-catalogitemid branch 2 times, most recently from b4c49bb to ac65cef Compare November 19, 2025 15:25
Make it an alias of `GlobalId`.
@teskje teskje force-pushed the poc-no-catalogitemid branch from ac65cef to 0c9451c Compare November 19, 2025 17:10
When running ALTER TABLE, move the table's ID to the new GlobalId,
instead of leaving it at its initial value.
@teskje teskje force-pushed the poc-no-catalogitemid branch from 0c9451c to 6cd3bf5 Compare November 19, 2025 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant