Skip to content

Commit 2845654

Browse files
vuanhphungIsaacpeco-engineer-bot[bot]
authored
fix: keep DECIMAL parameter scale on the Thrift backend (#585)
* fix: keep DECIMAL parameter scale on the Thrift backend A DBSQLParameter typed DECIMAL was sent to the Thrift server as the bare type "DECIMAL", which the server binds as a scale-0 decimal, so "123.45" came back as 123. toSparkParameter now qualifies it as DECIMAL(p,s) derived from the value, so both backends share the derivation the kernel path already did. Exponent forms such as 1e-7 are rewritten as plain numerals, and non-numeric or >38-digit values throw a ParameterError. Fixes #474 Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> * engineer-bot: address @mention on PR #585 Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> --------- Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com> Co-authored-by: Isaac <no-reply@databricks.com> Co-authored-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
1 parent 9ec2e36 commit 2845654

7 files changed

Lines changed: 267 additions & 56 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
- Upgrade `thrift` to 0.25, which depends on the CommonJS-compatible `uuid@11` instead of the ESM-only `uuid@13`, so `require('@databricks/sql')` works again in Jest and other environments without `require(esm)`. (#461)
1212
- **Behavior change for direct users of the exported `thrift.TCLIService`** (`DBSQLClient` itself uses the callback form and is unaffected): `TCLIService.Client` calls made without a callback now return native Promises instead of Q promises, and `TCLIService.Processor` runs promise-returning handlers through native Promises. Q-only methods such as `.fail()`, `.fin()`, `.done()` and `.spread()` no longer exist; use `.catch()`, `.finally()` and `.then()`. An unhandled rejection is now a real Node unhandled rejection, which crashes the process under Node's default `--unhandled-rejections=throw`. Q only emitted the `unhandledRejection` event manually, so it never crashed. Handle every promise a callback-less client call returns.
1313
- `oauthScopes` is now honored on the Thrift backend: OAuth client-secret M2M sends the configured scopes instead of always `all-apis`, so scoped service-principal secrets can authenticate, and U2M uses them in place of `sql offline_access`. On both backends, entries are split on whitespace and blank values fall back to those defaults. A token read from `persistence` that records different scopes from the ones the client would request is no longer reused or refreshed; the client acquires a new one (for U2M, through a new browser login). Tokens that don't record scopes are still reused. (#540)
14+
- Fix `DECIMAL` parameters losing their fractional digits on the Thrift backend: `"123.45"` was bound as `decimal(3,0)` and came back as `123`. Both backends now send `DECIMAL(p,s)` derived from the value (also for the `DEC` / `NUMERIC` synonyms), stringify `Int64` values beyond 2^53 exactly instead of as `Infinity`, accept exponent forms such as `1e-7` (rewritten as a plain numeral), and throw a `ParameterError` for non-numeric values or values that need more than 38 digits. (#474)
1415

1516
## 3.0.0
1617

‎lib/DBSQLParameter.ts‎

Lines changed: 86 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import Int64 from 'node-int64';
22
import { TSparkParameter, TSparkParameterValue } from '../thrift/TCLIService_types';
3+
import ParameterError from './errors/ParameterError';
34

45
export type DBSQLParameterValue = undefined | null | boolean | number | bigint | Int64 | Date | string;
56

@@ -60,6 +61,77 @@ function inferNumberType(value: number): DBSQLParameterType {
6061
return DBSQLParameterType.DOUBLE;
6162
}
6263

64+
const DECIMAL_MAX_PRECISION = 38;
65+
66+
/**
67+
* Qualify a bare `DECIMAL` (or its `DEC` / `NUMERIC` synonyms) with the
68+
* precision and scale of its value. Both backends otherwise lose the fraction:
69+
* the Thrift server binds a scale-0 decimal, truncating `"123.45"` to `123`,
70+
* and the kernel's raw-param path forwards the type verbatim, so SEA binds it
71+
* as `DECIMAL(10,0)` and rounds.
72+
*
73+
* Precision is significant integer digits plus scale (Spark's decimal-literal
74+
* rule), so `"99.99"` ⇒ `DECIMAL(4,2)` and `"0.00001"` ⇒ `DECIMAL(5,5)`. Plain
75+
* numerals are sent unchanged; exponent forms (`1e-7`, which `Number#toString`
76+
* produces) are rewritten as the equivalent plain numeral. A non-numeric value
77+
* or one needing precision > 38 throws instead of being truncated server-side.
78+
*/
79+
function qualifyDecimal(stringValue: string): { type: string; value: string } {
80+
const m = /^([+-]?)(\d*)(?:\.(\d*))?(?:[eE]([+-]?\d+))?$/.exec(stringValue.trim());
81+
const [, sign = '', intPart = '', fracPart, exponent] = m ?? [];
82+
if (!m || (intPart === '' && !fracPart)) {
83+
throw new ParameterError(
84+
`DECIMAL parameter value "${stringValue}" is not a decimal number (expected e.g. "123.45" or "1.5e-7").`,
85+
);
86+
}
87+
88+
const digits = intPart + (fracPart ?? '');
89+
const exp = exponent === undefined ? 0 : Number(exponent);
90+
// Position of the decimal point within `digits` once the exponent is applied.
91+
const pointPos = intPart.length + exp;
92+
const leadingZeros = digits.length - digits.replace(/^0+/, '').length;
93+
const isZero = leadingZeros === digits.length;
94+
const scale = Math.max(digits.length - pointPos, 0);
95+
const significantIntDigits = isZero ? 0 : Math.max(pointPos - leadingZeros, 0);
96+
const precision = Math.max(significantIntDigits + scale, 1);
97+
if (precision > DECIMAL_MAX_PRECISION) {
98+
throw new ParameterError(
99+
`DECIMAL parameter value "${stringValue}" needs precision ${precision}, exceeding the Databricks maximum of ` +
100+
`${DECIMAL_MAX_PRECISION}. Round/scale the value or bind it as a STRING.`,
101+
);
102+
}
103+
104+
const type = `DECIMAL(${precision},${scale})`;
105+
if (exponent === undefined && fracPart !== '') {
106+
return { type, value: stringValue };
107+
}
108+
// Rebuild as a plain numeral; both padding lengths are bounded by `precision`.
109+
let plain: string;
110+
if (isZero) {
111+
plain = scale > 0 ? `0.${'0'.repeat(scale)}` : '0';
112+
} else if (pointPos >= digits.length) {
113+
plain = digits + '0'.repeat(pointPos - digits.length);
114+
} else if (pointPos <= 0) {
115+
plain = `0.${'0'.repeat(-pointPos)}${digits}`;
116+
} else {
117+
plain = `${digits.slice(0, pointPos)}.${digits.slice(pointPos)}`;
118+
}
119+
plain = plain.replace(/^0+(?=\d)/, '');
120+
return { type, value: sign === '-' ? `-${plain}` : plain };
121+
}
122+
123+
// Unqualified decimal type names; a qualified `DECIMAL(p,s)` is left as given.
124+
const BARE_DECIMAL_TYPE = /^(DECIMAL|DEC|NUMERIC)$/i;
125+
126+
/**
127+
* Exact decimal string for an `Int64`. `Int64#toString` goes through a JS
128+
* number and returns `"Infinity"` for magnitudes ≥ 2^53, so read the
129+
* big-endian two's-complement bytes as a `bigint` instead.
130+
*/
131+
function int64ToString(value: Int64): string {
132+
return value.toBuffer().readBigInt64BE(0).toString();
133+
}
134+
63135
interface DBSQLParameterOptions {
64136
type?: DBSQLParameterType;
65137
value: DBSQLParameterValue;
@@ -79,7 +151,19 @@ export class DBSQLParameter {
79151
this.value = value;
80152
}
81153

82-
public toSparkParameter({ name }: ToSparkParameterOptions = {}): TSparkParameter {
154+
public toSparkParameter(options: ToSparkParameterOptions = {}): TSparkParameter {
155+
const param = this.toUnqualifiedSparkParameter(options);
156+
const stringValue = param.value?.stringValue;
157+
// Only an explicit type can be DECIMAL; inferred types never are.
158+
if (this.type !== undefined && BARE_DECIMAL_TYPE.test(this.type.trim()) && typeof stringValue === 'string') {
159+
const decimal = qualifyDecimal(stringValue);
160+
param.type = decimal.type;
161+
param.value = new TSparkParameterValue({ stringValue: decimal.value });
162+
}
163+
return param;
164+
}
165+
166+
private toUnqualifiedSparkParameter({ name }: ToSparkParameterOptions): TSparkParameter {
83167
// If VOID type was set explicitly - ignore value
84168
if (this.type === DBSQLParameterType.VOID) {
85169
return new TSparkParameter({ name }); // for NULL neither `type` nor `value` should be set
@@ -121,7 +205,7 @@ export class DBSQLParameter {
121205
name,
122206
type: wireType ?? DBSQLParameterType.BIGINT,
123207
value: new TSparkParameterValue({
124-
stringValue: this.value.toString(),
208+
stringValue: this.value instanceof Int64 ? int64ToString(this.value) : this.value.toString(),
125209
}),
126210
});
127211
}

‎lib/kernel/KernelPositionalParams.ts‎

Lines changed: 3 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -13,57 +13,15 @@
1313
// limitations under the License.
1414

1515
import { DBSQLParameter, DBSQLParameterValue } from '../DBSQLParameter';
16-
import ParameterError from '../errors/ParameterError';
1716
import { KernelNativeRawParameterInput } from './KernelNativeLoader';
1817
import assertBindableValue from '../utils/assertBindableValue';
1918

20-
/**
21-
* Derive `(precision,scale)` from a decimal value string for the kernel
22-
* `DECIMAL(p,s)` type name — the kernel param codec requires the parenthesised
23-
* form (plain `"DECIMAL"` is rejected) so it preserves the caller's fractional
24-
* digits. `"99.99"` ⇒ `"4,2"`; `"-123"` ⇒ `"3,0"`; `"0.00001"` ⇒ `"5,5"`.
25-
*
26-
* Precision is significant integer digits (insignificant leading zeros
27-
* stripped, matching Spark's decimal-literal rule) plus the fractional scale.
28-
* The value is NOT clamped: a non-numeric value, or one that needs precision >
29-
* the Databricks maximum of 38, throws `ParameterError` at bind time rather
30-
* than emitting an inconsistent `DECIMAL(38,…)` type alongside an
31-
* unrepresentable value (which the kernel would reject or silently truncate).
32-
*/
33-
function decimalPrecisionScale(v: string): string {
34-
// Accept an optional sign + plain decimal numeral only. Exponential form
35-
// ("1e+21"), empty, and non-numeric ("abc") are rejected with a clear error
36-
// instead of mis-deriving (p,s) and surfacing an opaque kernel failure.
37-
const m = /^[+-]?(\d*)(?:\.(\d+))?$/.exec(v.trim());
38-
const intPart = m?.[1] ?? '';
39-
const fracPart = m?.[2] ?? '';
40-
if (!m || (intPart === '' && fracPart === '')) {
41-
throw new ParameterError(
42-
`DECIMAL parameter value "${v}" is not a plain decimal numeral (expected [+-]?digits[.digits]).`,
43-
);
44-
}
45-
const scale = fracPart.length;
46-
// Significant integer digits: drop insignificant leading zeros ("007" → 1,
47-
// "0" / "" → 0) so e.g. 0.00001 is DECIMAL(5,5), not DECIMAL(6,5).
48-
const significantIntDigits = intPart.replace(/^0+/, '').length;
49-
const precision = Math.max(significantIntDigits + scale, 1);
50-
if (precision > 38) {
51-
throw new ParameterError(
52-
`DECIMAL parameter value "${v}" needs precision ${precision}, exceeding the Databricks maximum of 38. ` +
53-
'Round/scale the value or bind it as a STRING.',
54-
);
55-
}
56-
return `${precision},${scale}`;
57-
}
58-
5919
/**
6020
* Reduce a `DBSQLParameter | DBSQLParameterValue` to the napi
6121
* `RawParameterInput` (`{ name?, sqlType, value? }`) accepted by the kernel's
6222
* raw-parameter path. Reuses `DBSQLParameter.toSparkParameter` — the same
63-
* type-inference + value-stringification the Thrift backend uses — then adapts
64-
* the type name where required:
65-
* - DECIMAL → `DECIMAL(p,s)` (parenthesised form required)
66-
* - a missing value ⇒ SQL NULL.
23+
* type-inference + value-stringification (including DECIMAL → `DECIMAL(p,s)`)
24+
* the Thrift backend uses; a missing value ⇒ SQL NULL.
6725
*/
6826
function toRawParameterInput(value: DBSQLParameter | DBSQLParameterValue): KernelNativeRawParameterInput {
6927
const param = value instanceof DBSQLParameter ? value : new DBSQLParameter({ value });
@@ -76,12 +34,7 @@ function toRawParameterInput(value: DBSQLParameter | DBSQLParameterValue): Kerne
7634
return { sqlType: 'VOID' };
7735
}
7836

79-
let sqlType = spark.type ?? 'STRING';
80-
const upper = sqlType.toUpperCase();
81-
if (upper === 'DECIMAL') {
82-
sqlType = `DECIMAL(${decimalPrecisionScale(stringValue)})`;
83-
}
84-
return { sqlType, value: stringValue };
37+
return { sqlType: spark.type ?? 'STRING', value: stringValue };
8538
}
8639

8740
/**

‎tests/e2e/kernel/execution-e2e.test.ts‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,34 @@ describe('kernel execution end-to-end', function e2eSuite() {
147147
}
148148
});
149149

150+
it('binds DECIMAL parameters with value-derived precision and scale', async () => {
151+
const client = new DBSQLClient();
152+
153+
await client.connect({
154+
host: hostName as string,
155+
path: httpPath as string,
156+
token: token as string,
157+
useKernel: true,
158+
} as ConnectionOptions & InternalConnectionOptions);
159+
160+
const session = await client.openSession({ initialCatalog: 'main' });
161+
let operation;
162+
try {
163+
const decimal = (value: string | number) => new DBSQLParameter({ type: DBSQLParameterType.DECIMAL, value });
164+
operation = await session.executeStatement('SELECT ? AS a, typeof(?) AS a_type, ? AS b, typeof(?) AS b_type', {
165+
// `1e-7` exercises the exponent form `Number#toString` produces.
166+
ordinalParameters: [decimal('123.45'), decimal('123.45'), decimal(1e-7), decimal(1e-7)],
167+
});
168+
expect(await operation.fetchAll()).to.deep.equal([
169+
{ a: 123.45, a_type: 'decimal(5,2)', b: 1e-7, b_type: 'decimal(7,7)' },
170+
]);
171+
} finally {
172+
await operation?.close();
173+
await session.close();
174+
await client.close();
175+
}
176+
});
177+
150178
it('binds named NULL and empty string through rawParams on the async path', async () => {
151179
const client = new DBSQLClient();
152180

‎tests/e2e/query_parameters.test.ts‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -232,6 +232,48 @@ describe('Query parameters', () => {
232232
}
233233
});
234234

235+
// https://github.com/databricks/databricks-sql-nodejs/issues/474
236+
it('should keep the precision and scale of DECIMAL parameters', async () => {
237+
const session = await openSession();
238+
try {
239+
const operation = await session.executeStatement(
240+
'SELECT :a AS a, typeof(:a) AS a_type, :b AS b, typeof(:b) AS b_type, :c AS c, typeof(:c) AS c_type',
241+
{
242+
namedParameters: {
243+
a: new DBSQLParameter({ type: DBSQLParameterType.DECIMAL, value: '123.45' }),
244+
// `Number#toString` renders this as "-1.5e-7".
245+
b: new DBSQLParameter({ type: DBSQLParameterType.DECIMAL, value: -1.5e-7 }),
246+
// More digits than the scale-0 DECIMAL the server used to apply could hold.
247+
c: new DBSQLParameter({ type: DBSQLParameterType.DECIMAL, value: BigInt('12345678901234') }),
248+
},
249+
},
250+
);
251+
const result = await operation.fetchAll();
252+
await operation.close();
253+
expect(result).to.deep.equal([
254+
{
255+
a: 123.45,
256+
a_type: 'decimal(5,2)',
257+
b: -1.5e-7,
258+
b_type: 'decimal(8,8)',
259+
c: 12345678901234,
260+
c_type: 'decimal(14,0)',
261+
},
262+
]);
263+
264+
const ordinal = await session.executeStatement('SELECT ? AS v, typeof(?) AS t', {
265+
ordinalParameters: [
266+
new DBSQLParameter({ type: DBSQLParameterType.DECIMAL, value: '123.45' }),
267+
new DBSQLParameter({ type: DBSQLParameterType.DECIMAL, value: '123.45' }),
268+
],
269+
});
270+
expect(await ordinal.fetchAll()).to.deep.equal([{ v: 123.45, t: 'decimal(5,2)' }]);
271+
await ordinal.close();
272+
} finally {
273+
await session.close();
274+
}
275+
});
276+
235277
// https://github.com/databricks/databricks-sql-nodejs/issues/261
236278
describe('non-scalar parameter values (issue #261)', () => {
237279
// Non-scalar values are rejected client-side (before any RPC is sent), so executeStatement

0 commit comments

Comments
 (0)