Skip to content

Commit a5086c2

Browse files
vuanhphungIsaac
andcommitted
fix: keep DECIMAL parameter scale on the Thrift backend
A DBSQLParameter typed DECIMAL was sent to the Thrift server as the bare type "DECIMAL", which the server binds as a scale-0 decimal, so "123.45" came back as 123. toSparkParameter now qualifies it as DECIMAL(p,s) derived from the value, so both backends share the derivation the kernel path already did. Exponent forms such as 1e-7 are rewritten as plain numerals, and non-numeric or >38-digit values throw a ParameterError. Fixes #474 Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
1 parent 7691f2f commit a5086c2

7 files changed

Lines changed: 229 additions & 55 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
- Reject non-scalar query parameter values at bind time with a `ParameterError` on both the Thrift and kernel backends, instead of stringifying them (which could leave a Thrift operation stuck before FINISHED). Newly rejected on the Thrift backend: arrays, plain objects, boxed primitives (e.g. `new String('x')`, `new Number(1)`), functions, symbols, and a `DBSQLParameter` that wraps another `DBSQLParameter`. Primitives, `Date`, `Int64`, and `DBSQLParameter` are still accepted.
88
- `DBSQLLogger`: a `filepath` naming an existing directory now writes to `databricks-sql-nodejs.log` inside it, instead of silently producing no logs. (#565)
99
- `DBSQLLogger`: if the log file can't be opened (including when its directory can't be created, which used to throw from the constructor), the logger now logs to the console only, leaves `transports.file` undefined, and emits a process warning with code `DBSQL_LOG_FILE_UNAVAILABLE`. Previously an unwritable file could silently stall all logging, including the console. (#565)
10+
- Fix `DECIMAL` parameters losing their fractional digits on the Thrift backend: `"123.45"` was bound as `decimal(3,0)` and came back as `123`. Both backends now send `DECIMAL(p,s)` derived from the value, accept exponent forms such as `1e-7` (rewritten as a plain numeral), and throw a `ParameterError` for non-numeric values or values that need more than 38 digits. (#474)
1011

1112
## 3.0.0
1213

‎lib/DBSQLParameter.ts‎

Lines changed: 71 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import Int64 from 'node-int64';
22
import { TSparkParameter, TSparkParameterValue } from '../thrift/TCLIService_types';
3+
import ParameterError from './errors/ParameterError';
34

45
export type DBSQLParameterValue = undefined | null | boolean | number | bigint | Int64 | Date | string;
56

@@ -60,6 +61,63 @@ function inferNumberType(value: number): DBSQLParameterType {
6061
return DBSQLParameterType.DOUBLE;
6162
}
6263

64+
const DECIMAL_MAX_PRECISION = 38;
65+
66+
/**
67+
* Qualify a bare `DECIMAL` with the precision and scale of its value. A bare
68+
* `DECIMAL` makes the Thrift server bind a scale-0 decimal, silently truncating
69+
* `"123.45"` to `123`, and the kernel codec rejects it outright.
70+
*
71+
* Precision is significant integer digits plus scale (Spark's decimal-literal
72+
* rule), so `"99.99"` ⇒ `DECIMAL(4,2)` and `"0.00001"` ⇒ `DECIMAL(5,5)`. Plain
73+
* numerals are sent unchanged; exponent forms (`1e-7`, which `Number#toString`
74+
* produces) are rewritten as the equivalent plain numeral. A non-numeric value
75+
* or one needing precision > 38 throws instead of being truncated server-side.
76+
*/
77+
function qualifyDecimal(stringValue: string): { type: string; value: string } {
78+
const m = /^([+-]?)(\d*)(?:\.(\d*))?(?:[eE]([+-]?\d+))?$/.exec(stringValue.trim());
79+
const [, sign = '', intPart = '', fracPart, exponent] = m ?? [];
80+
if (!m || (intPart === '' && !fracPart)) {
81+
throw new ParameterError(
82+
`DECIMAL parameter value "${stringValue}" is not a decimal number (expected e.g. "123.45" or "1.5e-7").`,
83+
);
84+
}
85+
86+
const digits = intPart + (fracPart ?? '');
87+
const exp = exponent === undefined ? 0 : Number(exponent);
88+
// Position of the decimal point within `digits` once the exponent is applied.
89+
const pointPos = intPart.length + exp;
90+
const leadingZeros = digits.length - digits.replace(/^0+/, '').length;
91+
const isZero = leadingZeros === digits.length;
92+
const scale = Math.max(digits.length - pointPos, 0);
93+
const significantIntDigits = isZero ? 0 : Math.max(pointPos - leadingZeros, 0);
94+
const precision = Math.max(significantIntDigits + scale, 1);
95+
if (precision > DECIMAL_MAX_PRECISION) {
96+
throw new ParameterError(
97+
`DECIMAL parameter value "${stringValue}" needs precision ${precision}, exceeding the Databricks maximum of ` +
98+
`${DECIMAL_MAX_PRECISION}. Round/scale the value or bind it as a STRING.`,
99+
);
100+
}
101+
102+
const type = `DECIMAL(${precision},${scale})`;
103+
if (exponent === undefined && fracPart !== '') {
104+
return { type, value: stringValue };
105+
}
106+
// Rebuild as a plain numeral; both padding lengths are bounded by `precision`.
107+
let plain: string;
108+
if (isZero) {
109+
plain = scale > 0 ? `0.${'0'.repeat(scale)}` : '0';
110+
} else if (pointPos >= digits.length) {
111+
plain = digits + '0'.repeat(pointPos - digits.length);
112+
} else if (pointPos <= 0) {
113+
plain = `0.${'0'.repeat(-pointPos)}${digits}`;
114+
} else {
115+
plain = `${digits.slice(0, pointPos)}.${digits.slice(pointPos)}`;
116+
}
117+
plain = plain.replace(/^0+(?=\d)/, '');
118+
return { type, value: sign === '-' ? `-${plain}` : plain };
119+
}
120+
63121
interface DBSQLParameterOptions {
64122
type?: DBSQLParameterType;
65123
value: DBSQLParameterValue;
@@ -79,7 +137,19 @@ export class DBSQLParameter {
79137
this.value = value;
80138
}
81139

82-
public toSparkParameter({ name }: ToSparkParameterOptions = {}): TSparkParameter {
140+
public toSparkParameter(options: ToSparkParameterOptions = {}): TSparkParameter {
141+
const param = this.toUnqualifiedSparkParameter(options);
142+
const stringValue = param.value?.stringValue;
143+
// Only an explicit type can be DECIMAL; inferred types never are.
144+
if (this.type?.toUpperCase() === DBSQLParameterType.DECIMAL && typeof stringValue === 'string') {
145+
const decimal = qualifyDecimal(stringValue);
146+
param.type = decimal.type;
147+
param.value = new TSparkParameterValue({ stringValue: decimal.value });
148+
}
149+
return param;
150+
}
151+
152+
private toUnqualifiedSparkParameter({ name }: ToSparkParameterOptions): TSparkParameter {
83153
// If VOID type was set explicitly - ignore value
84154
if (this.type === DBSQLParameterType.VOID) {
85155
return new TSparkParameter({ name }); // for NULL neither `type` nor `value` should be set

‎lib/kernel/KernelPositionalParams.ts‎

Lines changed: 3 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -13,57 +13,15 @@
1313
// limitations under the License.
1414

1515
import { DBSQLParameter, DBSQLParameterValue } from '../DBSQLParameter';
16-
import ParameterError from '../errors/ParameterError';
1716
import { KernelNativeRawParameterInput } from './KernelNativeLoader';
1817
import assertBindableValue from '../utils/assertBindableValue';
1918

20-
/**
21-
* Derive `(precision,scale)` from a decimal value string for the kernel
22-
* `DECIMAL(p,s)` type name — the kernel param codec requires the parenthesised
23-
* form (plain `"DECIMAL"` is rejected) so it preserves the caller's fractional
24-
* digits. `"99.99"` ⇒ `"4,2"`; `"-123"` ⇒ `"3,0"`; `"0.00001"` ⇒ `"5,5"`.
25-
*
26-
* Precision is significant integer digits (insignificant leading zeros
27-
* stripped, matching Spark's decimal-literal rule) plus the fractional scale.
28-
* The value is NOT clamped: a non-numeric value, or one that needs precision >
29-
* the Databricks maximum of 38, throws `ParameterError` at bind time rather
30-
* than emitting an inconsistent `DECIMAL(38,…)` type alongside an
31-
* unrepresentable value (which the kernel would reject or silently truncate).
32-
*/
33-
function decimalPrecisionScale(v: string): string {
34-
// Accept an optional sign + plain decimal numeral only. Exponential form
35-
// ("1e+21"), empty, and non-numeric ("abc") are rejected with a clear error
36-
// instead of mis-deriving (p,s) and surfacing an opaque kernel failure.
37-
const m = /^[+-]?(\d*)(?:\.(\d+))?$/.exec(v.trim());
38-
const intPart = m?.[1] ?? '';
39-
const fracPart = m?.[2] ?? '';
40-
if (!m || (intPart === '' && fracPart === '')) {
41-
throw new ParameterError(
42-
`DECIMAL parameter value "${v}" is not a plain decimal numeral (expected [+-]?digits[.digits]).`,
43-
);
44-
}
45-
const scale = fracPart.length;
46-
// Significant integer digits: drop insignificant leading zeros ("007" → 1,
47-
// "0" / "" → 0) so e.g. 0.00001 is DECIMAL(5,5), not DECIMAL(6,5).
48-
const significantIntDigits = intPart.replace(/^0+/, '').length;
49-
const precision = Math.max(significantIntDigits + scale, 1);
50-
if (precision > 38) {
51-
throw new ParameterError(
52-
`DECIMAL parameter value "${v}" needs precision ${precision}, exceeding the Databricks maximum of 38. ` +
53-
'Round/scale the value or bind it as a STRING.',
54-
);
55-
}
56-
return `${precision},${scale}`;
57-
}
58-
5919
/**
6020
* Reduce a `DBSQLParameter | DBSQLParameterValue` to the napi
6121
* `RawParameterInput` (`{ name?, sqlType, value? }`) accepted by the kernel's
6222
* raw-parameter path. Reuses `DBSQLParameter.toSparkParameter` — the same
63-
* type-inference + value-stringification the Thrift backend uses — then adapts
64-
* the type name where required:
65-
* - DECIMAL → `DECIMAL(p,s)` (parenthesised form required)
66-
* - a missing value ⇒ SQL NULL.
23+
* type-inference + value-stringification (including DECIMAL → `DECIMAL(p,s)`)
24+
* the Thrift backend uses; a missing value ⇒ SQL NULL.
6725
*/
6826
function toRawParameterInput(value: DBSQLParameter | DBSQLParameterValue): KernelNativeRawParameterInput {
6927
const param = value instanceof DBSQLParameter ? value : new DBSQLParameter({ value });
@@ -76,12 +34,7 @@ function toRawParameterInput(value: DBSQLParameter | DBSQLParameterValue): Kerne
7634
return { sqlType: 'VOID' };
7735
}
7836

79-
let sqlType = spark.type ?? 'STRING';
80-
const upper = sqlType.toUpperCase();
81-
if (upper === 'DECIMAL') {
82-
sqlType = `DECIMAL(${decimalPrecisionScale(stringValue)})`;
83-
}
84-
return { sqlType, value: stringValue };
37+
return { sqlType: spark.type ?? 'STRING', value: stringValue };
8538
}
8639

8740
/**

‎tests/e2e/kernel/execution-e2e.test.ts‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,34 @@ describe('kernel execution end-to-end', function e2eSuite() {
147147
}
148148
});
149149

150+
it('binds DECIMAL parameters with value-derived precision and scale', async () => {
151+
const client = new DBSQLClient();
152+
153+
await client.connect({
154+
host: hostName as string,
155+
path: httpPath as string,
156+
token: token as string,
157+
useKernel: true,
158+
} as ConnectionOptions & InternalConnectionOptions);
159+
160+
const session = await client.openSession({ initialCatalog: 'main' });
161+
let operation;
162+
try {
163+
const decimal = (value: string | number) => new DBSQLParameter({ type: DBSQLParameterType.DECIMAL, value });
164+
operation = await session.executeStatement('SELECT ? AS a, typeof(?) AS a_type, ? AS b, typeof(?) AS b_type', {
165+
// `1e-7` exercises the exponent form `Number#toString` produces.
166+
ordinalParameters: [decimal('123.45'), decimal('123.45'), decimal(1e-7), decimal(1e-7)],
167+
});
168+
expect(await operation.fetchAll()).to.deep.equal([
169+
{ a: 123.45, a_type: 'decimal(5,2)', b: 1e-7, b_type: 'decimal(7,7)' },
170+
]);
171+
} finally {
172+
await operation?.close();
173+
await session.close();
174+
await client.close();
175+
}
176+
});
177+
150178
it('binds named NULL and empty string through rawParams on the async path', async () => {
151179
const client = new DBSQLClient();
152180

‎tests/e2e/query_parameters.test.ts‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -232,6 +232,48 @@ describe('Query parameters', () => {
232232
}
233233
});
234234

235+
// https://github.com/databricks/databricks-sql-nodejs/issues/474
236+
it('should keep the precision and scale of DECIMAL parameters', async () => {
237+
const session = await openSession();
238+
try {
239+
const operation = await session.executeStatement(
240+
'SELECT :a AS a, typeof(:a) AS a_type, :b AS b, typeof(:b) AS b_type, :c AS c, typeof(:c) AS c_type',
241+
{
242+
namedParameters: {
243+
a: new DBSQLParameter({ type: DBSQLParameterType.DECIMAL, value: '123.45' }),
244+
// `Number#toString` renders this as "-1.5e-7".
245+
b: new DBSQLParameter({ type: DBSQLParameterType.DECIMAL, value: -1.5e-7 }),
246+
// More digits than the scale-0 DECIMAL the server used to apply could hold.
247+
c: new DBSQLParameter({ type: DBSQLParameterType.DECIMAL, value: BigInt('12345678901234') }),
248+
},
249+
},
250+
);
251+
const result = await operation.fetchAll();
252+
await operation.close();
253+
expect(result).to.deep.equal([
254+
{
255+
a: 123.45,
256+
a_type: 'decimal(5,2)',
257+
b: -1.5e-7,
258+
b_type: 'decimal(8,8)',
259+
c: 12345678901234,
260+
c_type: 'decimal(14,0)',
261+
},
262+
]);
263+
264+
const ordinal = await session.executeStatement('SELECT ? AS v, typeof(?) AS t', {
265+
ordinalParameters: [
266+
new DBSQLParameter({ type: DBSQLParameterType.DECIMAL, value: '123.45' }),
267+
new DBSQLParameter({ type: DBSQLParameterType.DECIMAL, value: '123.45' }),
268+
],
269+
});
270+
expect(await ordinal.fetchAll()).to.deep.equal([{ v: 123.45, t: 'decimal(5,2)' }]);
271+
await ordinal.close();
272+
} finally {
273+
await session.close();
274+
}
275+
});
276+
235277
// https://github.com/databricks/databricks-sql-nodejs/issues/261
236278
describe('non-scalar parameter values (issue #261)', () => {
237279
// Non-scalar values are rejected client-side (before any RPC is sent), so executeStatement

‎tests/unit/DBSQLParameter.test.ts‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ import { expect } from 'chai';
22
import Int64 from 'node-int64';
33
import { TSparkParameterValue, TSparkParameter } from '../../thrift/TCLIService_types';
44
import { DBSQLParameter, DBSQLParameterType, DBSQLParameterValue } from '../../lib/DBSQLParameter';
5+
import ParameterError from '../../lib/errors/ParameterError';
56

67
describe('DBSQLParameter', () => {
78
it('should infer types correctly', () => {
@@ -169,6 +170,81 @@ describe('DBSQLParameter', () => {
169170
);
170171
});
171172

173+
describe('DECIMAL', () => {
174+
const decimal = (value: DBSQLParameterValue, type: string = DBSQLParameterType.DECIMAL) =>
175+
new DBSQLParameter({ type: type as DBSQLParameterType, value }).toSparkParameter();
176+
const expected = (type: string, stringValue: string) =>
177+
new TSparkParameter({ type, value: new TSparkParameterValue({ stringValue }) });
178+
179+
it('derives precision and scale from the value instead of sending a scale-0 DECIMAL', () => {
180+
// A bare `DECIMAL` makes the Thrift server bind decimal(p,0): "123.45" → 123.
181+
expect(decimal('123.45')).to.deep.equal(expected('DECIMAL(5,2)', '123.45'));
182+
expect(decimal('-123.45')).to.deep.equal(expected('DECIMAL(5,2)', '-123.45'));
183+
expect(decimal('-123')).to.deep.equal(expected('DECIMAL(3,0)', '-123'));
184+
expect(decimal('1.500')).to.deep.equal(expected('DECIMAL(4,3)', '1.500'));
185+
expect(decimal('99.99', 'decimal')).to.deep.equal(expected('DECIMAL(4,2)', '99.99'));
186+
});
187+
188+
it('excludes insignificant leading zeros from precision', () => {
189+
expect(decimal('0.00001')).to.deep.equal(expected('DECIMAL(5,5)', '0.00001'));
190+
expect(decimal('007.50')).to.deep.equal(expected('DECIMAL(3,2)', '007.50'));
191+
expect(decimal('.5')).to.deep.equal(expected('DECIMAL(1,1)', '.5'));
192+
expect(decimal('0')).to.deep.equal(expected('DECIMAL(1,0)', '0'));
193+
expect(decimal('000.000')).to.deep.equal(expected('DECIMAL(3,3)', '000.000'));
194+
});
195+
196+
it('derives precision and scale for number, bigint and Int64 values', () => {
197+
expect(decimal(123.45)).to.deep.equal(expected('DECIMAL(5,2)', '123.45'));
198+
expect(decimal(-7)).to.deep.equal(expected('DECIMAL(1,0)', '-7'));
199+
expect(decimal(BigInt('12345678901234567890'))).to.deep.equal(expected('DECIMAL(20,0)', '12345678901234567890'));
200+
expect(decimal(new Int64(1234))).to.deep.equal(expected('DECIMAL(4,0)', '1234'));
201+
});
202+
203+
it('rewrites exponent forms as plain numerals', () => {
204+
// `Number#toString` uses exponent notation below 1e-6 and from 1e21.
205+
expect(decimal(1e-7)).to.deep.equal(expected('DECIMAL(7,7)', '0.0000001'));
206+
expect(decimal(-1.5e-7)).to.deep.equal(expected('DECIMAL(8,8)', '-0.00000015'));
207+
expect(decimal(1e21)).to.deep.equal(expected('DECIMAL(22,0)', '1000000000000000000000'));
208+
expect(decimal('1.5e3')).to.deep.equal(expected('DECIMAL(4,0)', '1500'));
209+
expect(decimal('1.50E1')).to.deep.equal(expected('DECIMAL(3,1)', '15.0'));
210+
expect(decimal('+12.5e-1')).to.deep.equal(expected('DECIMAL(3,2)', '1.25'));
211+
expect(decimal('0.0e5')).to.deep.equal(expected('DECIMAL(1,0)', '0'));
212+
expect(decimal('0e999999999999')).to.deep.equal(expected('DECIMAL(1,0)', '0'));
213+
expect(decimal('123.')).to.deep.equal(expected('DECIMAL(3,0)', '123'));
214+
});
215+
216+
it('accepts the full 38-digit precision', () => {
217+
const value = '12345678901234567890.123456789012345678';
218+
expect(decimal(value)).to.deep.equal(expected('DECIMAL(38,18)', value));
219+
expect(decimal('-0.12345678901234567890123456789012345678')).to.deep.equal(
220+
expected('DECIMAL(38,38)', '-0.12345678901234567890123456789012345678'),
221+
);
222+
});
223+
224+
it('rejects a value that needs precision > 38 instead of truncating it', () => {
225+
const tooBig = () => decimal('123456789012345678901234567890123456789');
226+
expect(tooBig).to.throw(ParameterError, /needs precision 39, exceeding the Databricks maximum of 38/);
227+
expect(() => decimal('0.000000000000000000000000000000000000001')).to.throw(ParameterError, /precision 39/);
228+
expect(() => decimal('1e38')).to.throw(ParameterError, /precision 39/);
229+
expect(() => decimal('1e-39')).to.throw(ParameterError, /precision 39/);
230+
expect(() => decimal(`1e${'9'.repeat(400)}`)).to.throw(ParameterError, /exceeding the Databricks maximum/);
231+
});
232+
233+
it('rejects non-numeric values', () => {
234+
for (const value of ['abc', '', ' ', '.', 'e5', '1,000', '1e', '0x10', 'NaN', NaN, Infinity, true]) {
235+
expect(() => decimal(value), String(value)).to.throw(ParameterError, /is not a decimal number/);
236+
}
237+
});
238+
239+
it('leaves an explicitly qualified DECIMAL(p,s) type unchanged', () => {
240+
expect(decimal('123.45', 'DECIMAL(10,2)')).to.deep.equal(expected('DECIMAL(10,2)', '123.45'));
241+
});
242+
243+
it('still binds a NULL DECIMAL without a type', () => {
244+
expect(decimal(null)).to.deep.equal(new TSparkParameter({}));
245+
});
246+
});
247+
172248
it('binds a typed DATE from local calendar accessors, never via toISOString (no UTC off-by-one)', () => {
173249
// A `Date` constructed from local components must bind the wall-calendar
174250
// date the user intended, regardless of the process timezone.

‎tests/unit/kernel/positionalParams.test.ts‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -57,10 +57,14 @@ describe('KernelPositionalParams.buildKernelPositionalParams', () => {
5757
);
5858
});
5959

60-
it('rejects a non-numeric / exponential DECIMAL value', () => {
61-
expect(decimal('1e+21')).to.throw(ParameterError, /not a plain decimal numeral/);
62-
expect(decimal('abc')).to.throw(ParameterError, /not a plain decimal numeral/);
63-
expect(decimal('')).to.throw(ParameterError, /not a plain decimal numeral/);
60+
it('rewrites an exponent-form DECIMAL value as a plain numeral', () => {
61+
expect(decimal('1e+21')()).to.deep.equal([{ sqlType: 'DECIMAL(22,0)', value: '1000000000000000000000' }]);
62+
expect(decimal('1e-7')()).to.deep.equal([{ sqlType: 'DECIMAL(7,7)', value: '0.0000001' }]);
63+
});
64+
65+
it('rejects a non-numeric DECIMAL value', () => {
66+
expect(decimal('abc')).to.throw(ParameterError, /is not a decimal number/);
67+
expect(decimal('')).to.throw(ParameterError, /is not a decimal number/);
6468
});
6569

6670
it('preserves qualified INTERVAL types for the kernel raw binder', () => {

0 commit comments

Comments
 (0)