20099: Add Decimal support for floor preimage#224
20099: Add Decimal support for floor preimage#224martin-augment wants to merge 4 commits intomainfrom
Conversation
Summary of ChangesHello @martin-augment, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request enhances the Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
WalkthroughThis change adds support for decimal types (Decimal32, Decimal64, Decimal128, Decimal256) to the floor preimage optimization logic in DataFusion. A new helper function ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Code Review: Add Decimal support for floor preimageSummaryThis PR adds decimal type support (Decimal32, Decimal64, Decimal128, Decimal256) to the floor function's preimage optimization. The implementation looks solid overall with comprehensive test coverage. Here are my findings: ✅ Strengths
🔍 Code Quality & Best Practices1. Import Organization (datafusion/functions/src/math/floor.rs:22-27)The imports are clean, but consider alphabetical ordering within groups: use arrow::compute::{DecimalCast, rescale_decimal};
use arrow::datatypes::{
ArrowNativeTypeOp, DataType, Decimal32Type, Decimal64Type, Decimal128Type,
Decimal256Type, DecimalType, Float32Type, Float64Type,
};2. Potential for Code Deduplication (datafusion/functions/src/math/floor.rs:262-301)The four decimal type cases are nearly identical. While the current approach is clear and explicit, consider if a macro or generic helper could reduce repetition. However, the current explicit approach is acceptable for maintainability. 🐛 Potential Issues1. Fractional Part Check Logic (datafusion/functions/src/math/floor.rs:378-379)if scale > 0 && value % one_scaled != D::Native::ZERO {
return None;
}Concern: This check only applies when Questions:
Recommendation: Either:
2. Error Handling in rescale_decimal (datafusion/functions/src/math/floor.rs:368-373)let one_scaled: D::Native = rescale_decimal::<D, D>(
D::Native::ONE,
1,
0,
precision,
scale,
)?;Observation:
Recommendation: Add a comment explaining failure conditions for future maintainers. ⚡ Performance Considerations
🔒 Security ConcernsNo security issues identified. The implementation:
🧪 Test CoverageExcellent coverage with the following test categories:
Potential additions:
📝 Documentation & CommentsGood:
Could improve:
🎯 Specific Recommendations
✅ Final VerdictThis is a high-quality PR that correctly implements decimal support for floor preimage optimization. The code is well-tested, follows existing patterns, and handles edge cases properly. Recommended Action: Approve with minor suggestions for documentation improvements and clarification on negative scale handling. Nice work! 🎉 Minor Nits
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds support for Decimal types to the floor function's preimage optimization, which is a great enhancement. The implementation looks correct, and the added unit tests and sqllogictest are comprehensive.
I have a few suggestions to improve code clarity and maintainability, mainly around reducing code duplication in the preimage function and improving the clarity of some test cases by using more idiomatic arithmetic operators. Overall, this is a solid contribution.
| // Decimal types | ||
| ScalarValue::Decimal32(Some(n), precision, scale) => { | ||
| decimal_preimage_bounds::<Decimal32Type>(*n, *precision, *scale).map( | ||
| |(lo, hi)| { | ||
| ( | ||
| ScalarValue::Decimal32(Some(lo), *precision, *scale), | ||
| ScalarValue::Decimal32(Some(hi), *precision, *scale), | ||
| ) | ||
| }, | ||
| ) | ||
| } | ||
| ScalarValue::Decimal64(Some(n), precision, scale) => { | ||
| decimal_preimage_bounds::<Decimal64Type>(*n, *precision, *scale).map( | ||
| |(lo, hi)| { | ||
| ( | ||
| ScalarValue::Decimal64(Some(lo), *precision, *scale), | ||
| ScalarValue::Decimal64(Some(hi), *precision, *scale), | ||
| ) | ||
| }, | ||
| ) | ||
| } | ||
| ScalarValue::Decimal128(Some(n), precision, scale) => { | ||
| decimal_preimage_bounds::<Decimal128Type>(*n, *precision, *scale).map( | ||
| |(lo, hi)| { | ||
| ( | ||
| ScalarValue::Decimal128(Some(lo), *precision, *scale), | ||
| ScalarValue::Decimal128(Some(hi), *precision, *scale), | ||
| ) | ||
| }, | ||
| ) | ||
| } | ||
| ScalarValue::Decimal256(Some(n), precision, scale) => { | ||
| decimal_preimage_bounds::<Decimal256Type>(*n, *precision, *scale).map( | ||
| |(lo, hi)| { | ||
| ( | ||
| ScalarValue::Decimal256(Some(lo), *precision, *scale), | ||
| ScalarValue::Decimal256(Some(hi), *precision, *scale), | ||
| ) | ||
| }, | ||
| ) | ||
| } |
There was a problem hiding this comment.
These match arms for Decimal32, Decimal64, Decimal128, and Decimal256 are very repetitive. This code duplication could be reduced to improve maintainability.
Consider defining a local macro within the preimage function to handle the repetitive logic. For example:
macro_rules! handle_decimal {
($T:ty, $n:expr, $p:expr, $s:expr, $scalar_variant:path) => {
decimal_preimage_bounds::<$T>(*$n, *$p, *$s).map(|(lo, hi)| {
($scalar_variant(Some(lo), *$p, *$s), $scalar_variant(Some(hi), *$p, *$s))
})
};
}Then each match arm could be simplified to a single line, e.g.:
ScalarValue::Decimal32(Some(n), precision, scale) => handle_decimal!(Decimal32Type, n, precision, scale, ScalarValue::Decimal32)
Since adding the macro definition is outside the changed lines, I'm not providing a direct code suggestion, but this refactoring would make the code more concise.
There was a problem hiding this comment.
value:good-to-have; category:bug; feedback: The Gemini AI reviewer is correct! The match arms are identical and differ only in the type of the handled Decimal type. Prevents code duplication and double maintenance.
|
|
||
| // Decimal256: i256::MAX | ||
| assert_preimage_none(ScalarValue::Decimal256( | ||
| Some(i256::MAX.wrapping_sub(i256::from(50))), |
There was a problem hiding this comment.
Using wrapping_sub here can be misleading as the subtraction is not expected to wrap. Using the standard subtraction operator - would be clearer and more accurately reflect the test's intent, as i256 implements Sub.
| Some(i256::MAX.wrapping_sub(i256::from(50))), | |
| Some(i256::MAX - i256::from(50)), |
There was a problem hiding this comment.
value:annoying; category:bug; feedback: The Gemini AI reviewer is not correct! There is no implementation of std::ops::Sub for i256 in Apache Arrow
| assert_preimage_range( | ||
| ScalarValue::Decimal256(Some(large_256), 76, 2), | ||
| ScalarValue::Decimal256(Some(large_256), 76, 2), | ||
| ScalarValue::Decimal256(Some(large_256.wrapping_add(i256::from(100))), 76, 2), |
There was a problem hiding this comment.
For clarity and consistency with decimal_preimage_bounds which uses add_checked, it's better to use the + operator here instead of wrapping_add. Since this test case is for non-overflowing values, + better reflects the intent.
| ScalarValue::Decimal256(Some(large_256.wrapping_add(i256::from(100))), 76, 2), | |
| ScalarValue::Decimal256(Some(large_256 + i256::from(100)), 76, 2), |
There was a problem hiding this comment.
value:annoying; category:bug; feedback: The Gemini AI reviewer is not correct! There is no implementation of std::ops::Add for i256 in Apache Arrow
| assert_preimage_range( | ||
| ScalarValue::Decimal256(Some(neg_256), 76, 2), | ||
| ScalarValue::Decimal256(Some(neg_256), 76, 2), | ||
| ScalarValue::Decimal256(Some(neg_256.wrapping_add(i256::from(100))), 76, 2), |
There was a problem hiding this comment.
Similar to the previous case, using + instead of wrapping_add would be clearer here, as this is a non-overflowing test case.
| ScalarValue::Decimal256(Some(neg_256.wrapping_add(i256::from(100))), 76, 2), | |
| ScalarValue::Decimal256(Some(neg_256 + i256::from(100)), 76, 2), |
There was a problem hiding this comment.
value:annoying; category:bug; feedback: The Gemini AI reviewer is not correct! There is no implementation of std::ops::Add for i256 in Apache Arrow
🤖 Augment PR SummarySummary: Extends the Changes:
Technical Notes: Decimal preimage optimization is only applied when the compared literal is an integer value at its declared scale and 🤖 Was this summary useful? React with 👍 or 👎 |
![augmentcode[bot]](https://avatars.githubusercontent.com/in/1027498?s=60&v=4){kind=small}
|
|
||
| // Decimal32: i32::MAX | ||
| // For scale=2, we add 100, so i32::MAX - 50 would overflow | ||
| assert_preimage_none(ScalarValue::Decimal32(Some(i32::MAX - 50), 9, 2)); |
There was a problem hiding this comment.
These scale=2 “overflow” cases (e.g. i32::MAX - 50) aren’t aligned to the scale factor (100), so preimage returns None due to the “non-integer decimal” check rather than actually exercising the add_checked overflow path.
Other Locations
datafusion/functions/src/math/floor.rs:684datafusion/functions/src/math/floor.rs:688datafusion/functions/src/math/floor.rs:692
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
There was a problem hiding this comment.
value:useful; category:bug; feedback: The Augment AI reviewer is correct! i32::MAX - 50 gives 2147483597, which has 10 digits and does not fit into precision=9 at all. So, the decimal number is overflown by construction and there is no call to floor() at all
value:useful; category:bug; feedback: The Claude AI reviewer is correct! Apache Arrow supports negative scales for decimals, so this check won't detect such decimals with fractional part. Prevents a bug with decimals with negative scales. Also shows that there are no unit/slt tests for such. |
1 participant
20099: To review by AI