Feat: Allow pow with negative & non-integer exponent on decimals #19369

Yuvraj-cyborg · 2025-12-17T10:20:33Z

Which issue does this PR close?

Closes #19348

Rationale for this change

Previously, pow() on decimal types would error for negative exponents and non-integer exponents with messages like:

Arrow error: Arithmetic overflow: Unsupported exp value: -5
Compute error: Cannot use non-integer exp
This was a regression from when decimals were cast to float before pow(). The efficient integer-based algorithm for computing power on scaled integers cannot handle these cases.

What changes are included in this PR?

Modified pow_decimal_int to fallback to pow_decimal_float for negative exponents
Modified pow_decimal_float to use an efficient integer path for non-negative integer exponents, otherwise fallback to f64 computation

Added pow_decimal_float_fallback function that:

Converts the decimal to f64
Computes powf(exp)
Converts back to the original decimal type with proper scaling
Added decimal_from_i128 helper to convert i128 results back to generic decimal types (needed for Decimal256 support)
Updated sqllogictests to expect success for negative/non-integer exponents

Are these changes tested?

Yes:

Unit tests for pow_decimal_float_fallback covering negative exponents, fractional exponents, cube roots
Updated SQL logic tests in decimal.slt

Are there any user-facing changes?

Yes. The following queries now work instead of returning errors:

-- Negative exponent
SELECT power(4::decimal(38, 5), -1);  -- Returns 0.25

-- Non-integer exponent  
SELECT power(2.5, 4.2);  -- Returns 46.9

-- Square root via power
SELECT power(4::decimal, 0.5);  -- Returns 2

Yuvraj-cyborg · 2025-12-17T10:21:12Z

@Jefffrey could you review this !
And let me know what changes are needed if any

datafusion/functions/src/math/power.rs

martin-g · 2025-12-17T13:41:06Z

datafusion/functions/src/math/power.rs

+    decimal_from_i128::<T>(result_rounded as i128)
+}
+
+fn decimal_from_i128<T>(value: i128) -> Result<T, ArrowError>


Is this intentionally not supporting Decimal256 (i.e. i256) ?

practically the f64 computation already loses precision past ~10^15 significant digits , isn't it ??
That's why I thought it isn't needed..

If you say then I can add that too
like add a Decimal256 path.

Jefffrey

I do wonder, with how many different paths we're introducing here to account for all edge cases, how other systems handle doing power for numeric/decimals 🤔

Do they just cast to float anyway or they have similar complexity of checks

datafusion/functions/src/math/power.rs

Jefffrey · 2025-12-19T03:06:21Z

datafusion/functions/src/math/power.rs

+    if value >= i32::MIN as i128 && value <= i32::MAX as i128 {
+        return Ok(T::from(value as i32));
+    }
+
+    let is_negative = value < 0;
+    let abs_value = value.unsigned_abs();
+
+    let billion = 1_000_000_000u128;
+    let mut result = T::from(0);
+    let mut multiplier = T::from(1);
+    let billion_t = T::from(1_000_000_000);
+
+    let mut remaining = abs_value;
+    while remaining > 0 {
+        let chunk = (remaining % billion) as i32;
+        remaining /= billion;
+
+        let chunk_value = T::from(chunk).mul_checked(multiplier).map_err(|_| {
+            ArrowError::ArithmeticOverflow(format!(
+                "Overflow while converting {value} to decimal type"
+            ))
+        })?;
+


I'm really not quite sure what is happening here 🤔

converting an i128 result back to the generic type T (which could be i32, i64, i128, or i256). Since we can't directly convert i128 → T for all types, so we split the i128 into chunks of 10^9 (a billion), which fits in i32. Then we reconstruct T by: chunk[n] * (10^9)^n + chunk[n-1] * (10^9)^(n-1) + and so on.

I would prefer a simplified implementation that has no need for this manual conversion from i128 back to i32/i64

@Jefffrey I used NumCast, review this one but... i256 doesn't implement NumCast... So I did that manually.

Let me know what you think on this

cc: @martin-g

Yuvraj-cyborg · 2025-12-19T06:40:33Z

I do wonder, with how many different paths we're introducing here to account for all edge cases, how other systems handle doing power for numeric/decimals 🤔

Do they just cast to float anyway or they have similar complexity of checks

In PostgreSQL they cast decimal to float64, compute pow(), cast back. They accept the precision loss.

Also in MySQL same approach - converts DECIMAL to DOUBLE for POW().

Yuvraj-cyborg · 2025-12-23T11:54:08Z

Hey @Jefffrey could you review this !!

Jefffrey

Seeing all this logic introduced, I'm beginning to question whether there is actual benefit to having a native log implementation 🤔

Perhaps we should just revert to casting it to float and accept the accuracy loss

Thoughts @theirix ?

Jefffrey · 2026-01-03T06:10:06Z

datafusion/functions/src/math/power.rs

-                    "Cannot make unscale factor for {scale} and {exp}"
-                ))
-            })
+            <T as From<i32>>::from(10)


This looks a little ugly, I recommend checking the existing traits to see if they have what we already need.

See:

https://docs.rs/arrow/latest/arrow/array/trait.ArrowNativeTypeOp.html

https://docs.rs/arrow/latest/arrow/datatypes/trait.ArrowNativeType.html

theirix · 2026-01-03T10:03:48Z

Seeing all this logic introduced, I'm beginning to question whether there is actual benefit to having a native log implementation 🤔

Perhaps we should just revert to casting it to float and accept the accuracy loss

Thoughts @theirix ?

Fair enough, the logic becomes more convoluted.

The original idea was to introduce common decimal operations. Scale-preserving operations like abs, round, gcd, etc., are easy to implement and support. Some other operations with a natural mapping to decimals (like log10, pow10) adjust scales and do not have a natural analogue in the arrow buffer, leading to more complex logic. These operations are typical for data analytics, and applications could benefit from them. So ten-based operations can be calculated precisely, while for the rest and for more complicated operations, of course, it is fine to lose precision using a native float implementation.

First, we should reuse the arrow's foundational primitives as much as possible. If there is an OP_checked, it's better to piggyback on it. A few num traits were recently added to decimals in arrow-buffer, making it easier for us.

Second, I believe more logic should be isolated in calculate_binary_decimal_math, especially for handling different scales, to shift responsibility from UDF implementers (like pow) to middleware. It is in progress, and I'll submit it shortly.

Jefffrey · 2026-01-03T10:10:21Z

Seeing all this logic introduced, I'm beginning to question whether there is actual benefit to having a native log implementation 🤔
Perhaps we should just revert to casting it to float and accept the accuracy loss
Thoughts @theirix ?

Fair enough, the logic becomes more convoluted.

The original idea was to introduce common decimal operations. Scale-preserving operations like abs, round, gcd, etc., are easy to implement and support. Some other operations with a natural mapping to decimals (like log10, pow10) adjust scales and do not have a natural analogue in the arrow buffer, leading to more complex logic. These operations are typical for data analytics, and applications could benefit from them. So ten-based operations can be calculated precisely, while for the rest and for more complicated operations, of course, it is fine to lose precision using a native float implementation.

First, we should reuse the arrow's foundational primitives as much as possible. If there is an OP_checked, it's better to piggyback on it. A few num traits were recently added to decimals in arrow-buffer, making it easier for us.

Second, I believe more logic should be isolated in calculate_binary_decimal_math, especially for handling different scales, to shift responsibility from UDF implementers (like pow) to middleware. It is in progress, and I'll submit it shortly.

That makes sense. I guess what we could also do to alleviate this complexity (and ensure less performance impact) would be:

At invoke time of function, only use native decimal operations when we have a scalar exponent
Otherwise fall back to casting to float

This can be done in followup PRs of course but at least sets a roadmap for us.

theirix · 2026-01-03T10:13:09Z

Seeing all this logic introduced, I'm beginning to question whether there is actual benefit to having a native log implementation 🤔
Perhaps we should just revert to casting it to float and accept the accuracy loss
Thoughts @theirix ?

Fair enough, the logic becomes more convoluted.
The original idea was to introduce common decimal operations. Scale-preserving operations like abs, round, gcd, etc., are easy to implement and support. Some other operations with a natural mapping to decimals (like log10, pow10) adjust scales and do not have a natural analogue in the arrow buffer, leading to more complex logic. These operations are typical for data analytics, and applications could benefit from them. So ten-based operations can be calculated precisely, while for the rest and for more complicated operations, of course, it is fine to lose precision using a native float implementation.
First, we should reuse the arrow's foundational primitives as much as possible. If there is an OP_checked, it's better to piggyback on it. A few num traits were recently added to decimals in arrow-buffer, making it easier for us.
Second, I believe more logic should be isolated in calculate_binary_decimal_math, especially for handling different scales, to shift responsibility from UDF implementers (like pow) to middleware. It is in progress, and I'll submit it shortly.

That makes sense. I guess what we could also do to alleviate this complexity (and ensure less performance impact) would be:
* At invoke time of function, only use native decimal operations when we have a scalar exponent

* Otherwise fall back to casting to float

Sounds like a plan. The routing should be better made based on the type signature, rather than at eval time.

This can be done in followup PRs of course but at least sets a roadmap for us.

…pow with array exponents

Yuvraj-cyborg · 2026-01-03T21:40:33Z

cc: @Jefffrey @theirix

github-actions bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Dec 17, 2025

Yuvraj-cyborg force-pushed the pow-with-negative branch from 1ccabc0 to 029e2bd Compare December 17, 2025 10:23

martin-g reviewed Dec 17, 2025

View reviewed changes

Yuvraj-cyborg force-pushed the pow-with-negative branch from 029e2bd to d0193a1 Compare December 17, 2025 14:16

Jefffrey reviewed Dec 19, 2025

View reviewed changes

Yuvraj-cyborg force-pushed the pow-with-negative branch 5 times, most recently from 4dda5d8 to 9f79025 Compare December 23, 2025 11:49

Yuvraj-cyborg force-pushed the pow-with-negative branch from 9f79025 to d1dde6d Compare December 24, 2025 19:43

Jefffrey reviewed Jan 3, 2026

View reviewed changes

Yuvraj-cyborg added 2 commits January 3, 2026 21:04

feat: Allow pow with negative & non-integer exponent on decimals

b3e1fb0

fix: used ArrowType

5015797

Yuvraj-cyborg force-pushed the pow-with-negative branch from d1dde6d to 5015797 Compare January 3, 2026 15:57

Yuvraj-cyborg added 2 commits January 3, 2026 21:40

refactor: use Arrow native traits and add float fallback for decimal …

b628796

…pow with array exponents

chore: formatting

8326e40

Feat: Allow pow with negative & non-integer exponent on decimals #19369

Are you sure you want to change the base?

Feat: Allow pow with negative & non-integer exponent on decimals #19369

Conversation

Yuvraj-cyborg commented Dec 17, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Yuvraj-cyborg commented Dec 17, 2025

Uh oh!

Uh oh!

martin-g Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Yuvraj-cyborg Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Jefffrey Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Yuvraj-cyborg Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Jefffrey Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

Yuvraj-cyborg Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

Yuvraj-cyborg commented Dec 19, 2025

Uh oh!

Yuvraj-cyborg commented Dec 23, 2025

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

Jefffrey Jan 3, 2026

Choose a reason for hiding this comment

Uh oh!

theirix commented Jan 3, 2026

Uh oh!

Jefffrey commented Jan 3, 2026

Uh oh!

theirix commented Jan 3, 2026

Uh oh!

Yuvraj-cyborg commented Jan 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants