feat: enable Decimal64 handling in fuse table deserialization #19008
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
Fuse-table parquet deserialization now loads decimal columns with precision ≤18 as Decimal64 instead of Decimal128.
Parquet already picks the physical type based on precision during serialization, so only the read path needed to change. This remains fully backward compatible and safe for rolling upgrades.
Fuse-table native serialization and deserialization now store Decimal64 columns as true 64-bit values per row instead of widening them to 128 bits.
This change is not backward compatible: tables produced before this PR cannot be read by a build that includes this PR and will fail during Decimal64 decoding.
But given the clear performance gain and the native format still being in the research phase, accepting this compatibility trade-off seems reasonable.
To keep rolling upgrades viable, we leave the table-column metadata protobuf in meta untouched even though it still lacks Decimal64; changing it would break older nodes in mixed-version deployments.
Technically we could push part of the Decimal conversion logic into
schema_from_to_protobuf_impl.rs, but that would force edits to existingproto_convunit tests which would break that module’s append-only testing style and feels like a poor trade-off.Performance
select l_quantity from lineitem ignore_resultselect l_extendedprice from lineitem ignore_resultThree hot runs per query/version (all values in seconds).
Tests
Type of change
This change is