feat: ddl statements + catalog

[clintropolis]

Description

This PR adds DDL statement support to Druid so that CREATE and ALTER statements can be used to manage the catalog contents. This functionality requires WRITE permissions to the datasource (same as the backing catalog APIs), and is gated behind a new runtime property druid.sql.planner.enableCatalogDdl which is false by default. This PR includes support for defining the logical schema (columns list in catalog), aggregate projections (projections property in catalog), base table projection (baseTable property in catalog), clustering (clusterKeys in catalog), time partitioning (segmentGranularity in catalog), and any other properties (sealed, targetSegmentRows) with generic property setting syntax.

These operations are purely metadata operations, and so DROP TABLE has been omitted since in my mind it has the most room for confusion (a DROP TABLE only cleared metadata and didn't do anything else sounds kind of confusing to me). Dropping columns and projections from a table felt more easily explainable that these do not modify any existing data and instead stage the schema for subsequent ingestion (and future work should wire this catalog stuff better into compaction/reindexing so that we can also frame it such that compaction/reindexing will begin to eventually converge on the updated schema).

These all go through the regular query path, and return 0 rows on successful operation. Some follow-up work is needed to improve web-console syntax highlighting and behaviors, and since there is no drop table statement it would still probably be nice to eventually add a catalog management ui, but the statements at least already work as-is through the current query interface.

Some examples:

create table:

CREATE TABLE "wikipedia" (
  "channel" VARCHAR,
  "__time" TIMESTAMP,
  "page" VARCHAR,
  "namespace" VARCHAR,
  "user" VARCHAR,
  "comment" VARCHAR,
  "added" BIGINT,
  "delta" BIGINT
)
PARTITIONED BY DAY
CLUSTERED BY "channel"

alter table to add column:

ALTER TABLE "wikipedia" ADD COLUMN "deleted" BIGINT

alter table to add projection:

ALTER TABLE "wikipedia" ADD PROJECTION "channel_sums" AS (
  SELECT 
    TIME_FLOOR("__time", 'PT1H'),
    "channel",
    SUM("added") as "sum_added",
    SUM("delta") as "sum_delta",
    SUM("deleted") as "sum_deleted"
  GROUP BY 1,2
)

alter table set property:

ALTER TABLE "wikipedia" SET PROPERTIES (sealed = true)

alter table set 'base table' projection to create clustered segments:

ALTER TABLE "wikipedia" ADD PROJECTION __base AS (
  SELECT
    "channel",
    "__time",
    "page",
    "namespace,"
    "user",
    "comment",
    "added",
    "delta",
    "deleted"
  CLUSTERED BY "channel"
)

create table with base table and projection definitions:

CREATE TABLE "wikipedia" SEALED (
  "channel" VARCHAR,
  "__time" TIMESTAMP,
  "page" VARCHAR,
  "namespace" VARCHAR,
  "user" VARCHAR,
  "comment" VARCHAR,
  "added" BIGINT,
  "delta" BIGINT,
  "deleted" BIGINT,
  PROJECTION __base AS (
    SELECT
      "channel",
      "__time",
      "page",
      "namespace",
      "user",
      "comment",
      "added",
      "delta",
      "deleted"
    CLUSTERED BY "channel"
  ),
  PROJECTION "channel_sums" AS (
    SELECT 
      TIME_FLOOR("__time", 'PT1H'),
      "channel",
      SUM("added") as "sum_added",
      SUM("delta") as "sum_delta",
      SUM("deleted") as "sum_deleted"
    GROUP BY 1,2
  ),
  PROJECTION "channel_page_max" AS (
    SELECT 
      TIME_FLOOR("__time", 'PT1H'),
      "channel",
      "page",
      MAX("added") as "sum_added",
      MAX("delta") as "sum_delta",
      MAX("deleted") as "sum_deleted"
    GROUP BY 1,2,3
  )
)
PARTITIONED BY DAY
CLUSTERED BY channel

[FrankChen021]

Severity	Findings
P0	0
P1	2
P2	3
P3	0
Total	5

Reviewed 42 of 42 changed files.

---

This is an automated review by Codex GPT-5.6-Sol

[FrankChen021]

[P1] Revalidate the complete table after property edits

This uses the property-only edit endpoint, whose transaction loads and validates properties without columns, so DatasourceDefn.validate(ResolvedTable) and its cross-field checks never run. For example, after defining a DAY projection, SET PROPERTIES can change segmentGranularity to PT1H even though full validation rejects a projection coarser than its segments; it can likewise clear sealed while __base remains. The catalog then contains an invalid specification and subsequent ingestion fails. Load and validate the complete revised TableSpec inside the Coordinator transaction.

[FrankChen021]

[P1] Enforce column operation predicates atomically

UpdateColumns appends a column when its name is absent, so ALTER TABLE t ALTER COLUMN typo SET DATA TYPE BIGINT silently adds typo instead of rejecting the nonexistent target. ADD COLUMN has the inverse predicate checked by a separate Broker read, allowing concurrent ADDs to both pass and the later merge to overwrite the first type. Add/alter existence semantics need dedicated checks inside the Coordinator's column-update transaction.

[FrankChen021]

[P2] Preserve statement context when planning projections

The nested planner receives only the hard-coded CONTEXT and discards the enclosing statement context, even though SET clauses are explicitly supported before DDL. For example, SET sqlTimeZone = 'America/Los_Angeles' followed by a projection using TIME_FLOOR stores a UTC expression, so the equivalent query under the same context plans differently and cannot match the projection. Merge relevant outer PlannerContext values before applying the deterministic overrides.

[FrankChen021]

[P2] Handle direct aliases in base projections

ScanQuery deduplicates its column list, while outputNames retains every SELECT item. A valid declared layout [id, copy] with SELECT id, id AS copy therefore has two outputs but only one selected entry, and the second iteration throws IndexOutOfBoundsException; a direct alias also has no virtual column to materialize copy. Preserve the select-to-source mapping or reject this form with a user-facing validation error.

[FrankChen021]

[P2] Do not unparse omitted columns as empty parentheses

The grammar permits CREATE TABLE tbl PARTITIONED BY DAY with no parenthesized element list, but unparse always emits CREATE TABLE tbl () PARTITIONED BY DAY. Empty parentheses cannot be parsed because AddDruidTableElement is mandatory once '(' is present, so a valid AST does not round-trip. Omit the frame when both lists are empty or teach the grammar to accept ().

[jtuglu1]

cc @maytasm

[FrankChen021]

Severity	Findings
P0	0
P1	1
P2	1
P3	0
Total	2

Reviewed 44 of 44 changed files. Found two current-head correctness issues around projection type compatibility and atomic __base updates.

---

This is an automated review by Codex GPT-5.6-Luna(max)

[FrankChen021]

[P1] ALTER COLUMN can leave projections type-incompatible

Projection validation checks referenced-column presence but does not reconcile projection dimension and aggregator types with current column definitions. A table with x BIGINT and SUM(x) can accept ALTER COLUMN x SET DATA TYPE VARCHAR, leaving an incompatible projection that later fails during initialization and can break ingestion. Validate compatibility or reject incompatible column edits before commit.

[FrankChen021]

[P2] Base projection existence check is not atomic

The __base branch performs the IF NOT EXISTS or duplicate check from a prior table read, then calls updateProperties without repeating that predicate inside the Coordinator transaction. Concurrent ADD PROJECTION __base statements can both observe absence and commit, with the later update overwriting the earlier projection; the non-IF form can also succeed instead of reporting a duplicate. Move the check into the atomic update.

[FrankChen021]

Severity	Findings
P0	0
P1	1
P2	1
P3	0
Total	2

Reviewed 44 of 44 changed files.

---

This is an automated review by Codex GPT-5.6-Luna(max)

[FrankChen021]

[P1] Base-table edits are not actually serialized

updateProperties performs an unlocked read followed by an update of the properties blob, with no row lock or version predicate. Concurrent ADD [IF NOT EXISTS] PROJECTION __base calls can both observe no layout and let the latter overwrite the former; a concurrent column edit can also leave __base referencing removed or retyped columns. Use row locking or compare-and-set updates and validate against the committed version.

[FrankChen021]

[P2] IF NOT EXISTS still plans an existing base layout

The handler translates and plans the supplied base-table query before the Coordinator evaluates ifNotExists. If __base already exists, ALTER TABLE ... ADD IF NOT EXISTS PROJECTION __base AS (...) should be a no-op, but an invalid or missing column in the supplied body can still make it fail. Short-circuit known existing layouts before translation while retaining the server-side race check.

[FrankChen021]

Severity	Findings
P0	0
P1	2
P2	0
P3	0
Total	2

Reviewed 45 of 45 changed files.

---

This is an automated review by Codex GPT-5.6-Luna(max)

[FrankChen021]

[P1] CAS version can repeat

The new compare-and-set update uses System.currentTimeMillis() as the replacement version, while the predicate compares the row against existing.updateTime(). Two edits in the same millisecond can therefore reuse the same version: both transactions may pass the old-version predicate, overwrite one another, and publish indistinguishable cache versions. Generate a strictly increasing version atomically (or use an equivalent database-side mechanism) before relying on this CAS.

[FrankChen021]

[P1] Retries can replay catalog mutations

This new mutating POST goes through a client configured with StandardRetryPolicy.maxAttempts(6). If the Coordinator commits the DDL but the response is lost, a retryable channel error or 5xx can replay the non-idempotent edit (and the analogous create request), producing conflicts or duplicate effects while the SQL caller sees failure. Disable retries for writes or add request idempotency/deduplication.

[clintropolis]

producing conflicts or duplicate effects
I don't think this can really happen with the compare and set stuff, so mainly the risk is a confusing error message? Re: idempotency, the SQL supports IF NOT EXISTS/IF EXISTS, so i don't think we want it to be implicitly idempotent.

Also this retrying isn't really new in this PR, createTable was already going through it prior to this PR.. maybe in a follow-up we could consider tightening up retry on write operations, since I think all it really gives us is a more accurate error message?

[FrankChen021]

I agree that compare-and-set prevents a retry from overwriting a concurrent update, and that the retry policy predates this PR for createTable. The remaining issue is the new editTable POST: ADD COLUMN, ADD PROJECTION without IF NOT EXISTS, and base-table edits without existence flags all use the same retrying CatalogClient. If the Coordinator commits the first request but its response is lost, the retry reads the new version and runs the guard again, returning an error such as already exists even though the DDL committed. CAS prevents stale concurrent writes; it does not deduplicate post-commit replays. Please track this explicitly as a write-retry/idempotency follow-up, or disable retries for mutating calls. Reviewed 11 of 45 changed files.

[FrankChen021]

I have reviewed the code for correctness, edge cases, concurrency, and integration risks; no new issues found.

Reviewed 45 of 45 changed files.

Validation: git diff --check on the current 45-file PR diff; tests and builds not run.

---

This is an automated review by Codex GPT-5.6-Luna(max)

[capistrant]

This is a very cool addition to Druid, thank you for all the work and iteration that has gone into the PR. I reviewed test coverage and docs extensively to understand the features. I support the overall design and impl decisions spec'd out. I appreciate the detailed doc and the coverage that spans UTs and Embedded tests to provide confidence in the fact that the feature does what is spec'd out in the PR description and docs.

The many rounds with @FrankChen021 seem to have also worked through a lot of potential gotchas and have contributed to confidence in bugs being smoked out in this process versus in live testing after merge.

I'm approving with some nits I found reading through all the test files and javadocs. I also took into consideration that this is an experimental feature defaulted to off that is part of an experimental extension. release notes should stress this rawness and solicit bug reports/feedback from any early adopters.

[capistrant]

nit: don't interleave these in the tests

[capistrant]

nit: another interleaved private that could be moved out of test area

[capistrant]

nice doc, ty for adding it up front. greatly helped set stage for review

[capistrant]

nit: don't interleave with tests here. sorry to be repeating so many of these 😅

[capistrant]

The end reads confusing to me. are you meaning "regardless of what was declared" or something in that direction

[FrankChen021]

I have reviewed the code for correctness, edge cases, concurrency, and integration risks; no issues found.

Reviewed 45 of 45 changed files.

---

This is an automated review by Codex GPT-5.6-Luna(max)

[gianm]

Reviewed only the documentation for now.

[gianm]

never create, modify, or delete segments

Well, they don't immediately do these things synchronously, but they may trigger these things to happen in the background as part of compaction. This is worth mentioning.

(Or do they? What's the current state of catalog-based compaction?)

[clintropolis]

catalog based compaction doesn't currently handle baseTable spec, but once in place it would apply all of the other things like projections and clustering and stuff. I'm going to omit catalog stuff from the docs for now and address this stuff in a follow-up branch I have in progress that wires that up so that it is consistent with the rest of the catalog and (also make catalog compaction config work with MSQ compaction since that didn't work either).

[gianm]

Can you make these require both READ and WRITE? This is a change I'm proposing to #20075 for INSERT and REPLACE, on the grounds that users without READ can't see a table and therefore can't write to it.

[gianm]

This one line may be clearer if formatted on multiple lines.

[clintropolis]

split out into a separate 'element' declaration to try to be a bit easier on the eyes

[gianm]

Feels like this should go closer to the column list. Maybe this?

CREATE TABLE tbl SEALED ( columns ... )

[gianm]

It would be great to have SHOW CREATE TABLE or DESCRIBE TABLE too, otherwise people will find these APIs hard to use without referring back to the JSON. This can be done in a follow up.

[clintropolis]

cool, yea that seems nice, can look into that in a follow-up

[gianm]

This doc point about AVG(x) doesn't really make sense unless you know that Druid internally converts AVG to SUM / COUNT. Not all databases do this, so a better example would be an explicit expression like SUM(x) / COUNT(x). There is also a list of aggregation functions that can be used at ingest time at docs/multi-stage-query/concepts.md#rollup, which you can link to.

[gianm]

But will it trigger compaction?

[clintropolis]

not yet, will address this in follow-up that fixes up catalog compaction to honor baseTable spec and stuff

[gianm]

"computed column is written by the expression, not by the ingestion query" to me is unclear wording. The example helps though. Maybe try:

Computed columns, like bucket in the example above, are computed based on inputs provided by INSERT or REPLACE. In the example above, the INSERT or REPLACE command should provide user_id, not bucket.

[gianm]

Re: "defining it makes the table a clustered table": Is it an error to provide __base without CLUSTERED BY?

[clintropolis]

it is right now, since only the clustered base table spec exists. I do plan on adding regular table and rollup-table specs eventually so that all of the v10 base table metadata is represented and everything can be fully expressed, but this will be done in follow-up PR(s) which can expand on the docs here.

[gianm]

Hopefully these will be syntax errors with nice error messages. Do we have tests asserting what the error messages are if you do various incorrect things?

[clintropolis]

there is test coverage for this, the error messages aren't super specific, they are like parser's generic syntax errors instead of 'projections cannot use limit' or whatever (we would have to allow them in the grammar and then reject them later in the translator for that to work)

[gianm]

This could probably use pattern matching to be cooler.

[gianm]

I don't think there's really a need to justify the concept of validating the entire table as a unit.

[gianm]

It's sketchy to execute the operation in plan(). It should happen only when PlannerResult#run is called. Please also include a test that when the DDL operation is unauthorized, no methods are called on the catalog client.

[clintropolis]

good catch, fixed

[gianm]

Is this just needed so the Broker can immediately see its own writes? I guess it should eventually update its cache even without this code.

[clintropolis]

this is reworked a bit from the resync method added to address other problems using update

[gianm]

I think doing every change as an UPDATE is going to confuse the CachedMetadataCatalog. It will cause computeUpdate to run, which expects to see a pre-existing object and does a LOG.error if it doesn't see one. In some cases it won't see one, like CREATE TABLE.

[clintropolis]

reworked this a bit into to use resync to be cool for this usage

[gianm]

Please include a unit test for this, if it does not already exist. (I haven't checked yet.)

[gianm]

I think structuring it this way will cause DDL to be accepted by all engines. It makes me wonder what we want. I think what we want is for the Task engine, which is meant to be async/offline, to reject DDL. I think we want the interactive ones (native, Dart) to accept it. Consider adding EngineFeature#CAN_DDL (and maybe consolidate CAN_INSERT and CAN_REPLACE into CAN_DML while you're at it) so we can do this.

[clintropolis]

done, though didn't refactor CAN_DML, can do that in a follow-up

[gianm]

Should become a helper function that gets applied to all the UPDATE_TIME_COL usages. There's a few others.

[clintropolis]

done, also drive creates through here too instead of using current time inline

[gianm]

I'm not sure this is actually better than moving the extension's edit types into core. With the approach currently in the PR, the edit types don't need to be here, but we have a 1-1 coupling anyway where there needs to be one method here for each edit type. Seems like a judgment call though. Thoughts?

[clintropolis]

left this for now

[FrankChen021]

Severity	Findings
P0	0
P1	0
P2	2
P3	0
Total	2

Severity	Findings
P0	0
P1	0
P2	2
P3	0
Total	2

Reviewed 54 of 54 changed files.

---

This is an automated review by Codex GPT-5.6-Luna(max)

[FrankChen021]

[P2] Avoid misclassifying projection-named complex columns

CREATE TABLE t (projection TYPE('COMPLEX<json>')) is a valid column declaration: PROJECTION is intentionally non-reserved and TYPE('...') is the supported Druid-native type escape hatch. With LOOKAHEAD(3), the token sequence PROJECTION TYPE ( selects DruidProjectionDefinition (TYPE is also a non-reserved Calcite keyword), which then expects SELECT and rejects the declaration. This makes valid schema syntax depend on the column name; distinguish the complete projection production or special-case TYPE after a projection identifier.

[FrankChen021]

[P2] Keep CAS edits restricted to active rows

The new compare-and-set update for updateProperties (and the analogous UPDATE_COLUMNS_STMT) checks only schema/name/updateTime; it omits the state = 'A' predicate used by REPLACE_SPEC_STMT. markDeleting changes state to D without a version predicate and assigns nextVersion(0), so if deletion and an edit occur in the same millisecond, the deleting row can retain the version the edit read and the edit still rewrites the DELETING row and emits an update event. TableMetadata.forUpdate reports that event as ACTIVE, violating the existing "can't update when deleting" contract and letting caches observe metadata after deletion. Add AND state = 'A' to both incremental update statements or make deletion participate in the same version guard.

[FrankChen021]

No new issues found in the changes since the previous MergeLens review; existing inline findings remain tracked separately.

Reviewed 54 of 54 changed files.

---

This is an automated review by Codex GPT-5.6-Luna(max)

[gianm]

I don't think ABS(user_id) % 128 is valid Druid SQL. (Should use the MOD function.)

[gianm]

Please change it to ADD PROJECTION [IF NOT EXISTS] <name>, as this is more commonly the way such syntaxes are done in SQL systems. It also better matches DROP PROJECTION [IF EXISTS].

[gianm]

Isn't it bad that time filters are retained as filters? Typically when a real SQL query is actually issued, __time filters will be moved to intervals, so they will not actually show up in the filter. So, does that mean they won't be able to match the stored projection? Or, is there something that causes this to be handled well?

[clintropolis]

Good eye, there is not currently logic that can handle matching __time filters in projections against query intervals, so this is currently modeling building an unreachable projection.

Claude came up with this, I left it because i think retaining them as filters makes more sense than trying to give a standalone interval to projections, so i view at as the right form and since this test is just covering projection construction and not query time matching it seemed harmless to leave. That is, assuming I add support for special handling projections with __time filters that are comparable to intervals as a follow-up, which I think it would be pretty straightforward to add, so I have it on my list.

[gianm]

This test class has a lot of assertNotNull that really should be asserting equality to the expected thing. Please tighten them up.

[gianm]

Should be table name, not type, in this position.

[clintropolis]

table name isn't actually available here but i think that is probably ok since I think there could be at most 1 table involved in these statements

[gianm]

Use Projections.BASE_TABLE_PROJECTION_NAME?

[gianm]

Any reason to not allow non-complex types here? Seems like it would be fine to allow arrays and primitives.

[clintropolis]

not sure why it was like this (this was migrated from Externals to here), I don't see any problem allowing any native type in that function, but I have not changed this in this PR yet, I think i'd rather save for a follow-up if we want to do that.

[gianm]

Should check that the complex type is valid, or else this will cause failures later when we attempt to apply the table spec, leading to a broken table.

[clintropolis]

Initially I was trying to avoid really changing anything here (this logic was basically ported from Externals.java), I would assume the validation wasn't here in part to handle cases where like the coordinator doesn't have some extension loaded that provides the type but presumably whatever is actually using the catalog definition to execute does. However, this PR made it so that in some cases at least we already do this validation from any projection stuff used since that kicks in a validation of those specs which are computed from these definitions, so I went ahead and changed things to just do the validation for datasource columns too. I updated the catalog docs to mention that the coordinator needs all extensions loaded for complex types defined in a catalog schema.