Part of #22715.
Problem
GroupValuesColumn (the column-wise multi-column group-by path) is all-or-nothing: if a single group-by column has no type-specialized GroupColumn, supported_schema returns false and the entire key falls back to the row-wise GroupValuesRows — even when every other column would have qualified for the column-wise fast path. Nested types (Struct, List, FixedSizeList, ...) currently trigger this, so a mixed key like GROUP BY int_col, struct_col loses the fast path (and its lower memory footprint) entirely because of that one column.
Proposal
(Suggested by @alamb in #23128 (comment).)
Add a single generic GroupColumn backed by arrow's row format (a one-field RowConverter) as a fallback. Then GroupValuesColumn stays usable for any row-encodable type, and row-encoding is confined to just the columns that lack a native builder — instead of dropping the whole key onto GroupValuesRows. Hashing is unchanged (done on the raw input columns, which already supports nested types).
Scope (first step)
The generic RowsGroupColumn framework + routing for the nested types RowConverter can encode (Struct / List / LargeList / FixedSizeList, recursively). A quick local prototype suggests a mixed native + nested key uses roughly half the memory of the all-rows fallback, with grouping identical to the GroupValuesRows reference.
Follow-ups
- Cover types
RowConverter cannot encode (e.g. Map, which needs key-order normalization) so supported_schema can always be true.
- With full coverage, retire
GroupValuesRows entirely.
- Batch-scoped cache for the scalar
equal_to / append_val path (needs a GroupColumn batch hook).
Part of #22715.
Problem
GroupValuesColumn(the column-wise multi-column group-by path) is all-or-nothing: if a single group-by column has no type-specializedGroupColumn,supported_schemareturnsfalseand the entire key falls back to the row-wiseGroupValuesRows— even when every other column would have qualified for the column-wise fast path. Nested types (Struct,List,FixedSizeList, ...) currently trigger this, so a mixed key likeGROUP BY int_col, struct_colloses the fast path (and its lower memory footprint) entirely because of that one column.Proposal
(Suggested by @alamb in #23128 (comment).)
Add a single generic
GroupColumnbacked by arrow's row format (a one-fieldRowConverter) as a fallback. ThenGroupValuesColumnstays usable for any row-encodable type, and row-encoding is confined to just the columns that lack a native builder — instead of dropping the whole key ontoGroupValuesRows. Hashing is unchanged (done on the raw input columns, which already supports nested types).Scope (first step)
The generic
RowsGroupColumnframework + routing for the nested typesRowConvertercan encode (Struct/List/LargeList/FixedSizeList, recursively). A quick local prototype suggests a mixed native + nested key uses roughly half the memory of the all-rows fallback, with grouping identical to theGroupValuesRowsreference.Follow-ups
RowConvertercannot encode (e.g.Map, which needs key-order normalization) sosupported_schemacan always betrue.GroupValuesRowsentirely.equal_to/append_valpath (needs aGroupColumnbatch hook).