Skip to content

Commit 91a64ff

Browse files
vuanhphungIsaac
andauthored
fix(kernel): decode a failed async status poll into the driver error (#608)
* fix(kernel): decode a failed async status poll into the driver error On a runAsync: true statement, waitUntilReady() and status() awaited the napi AsyncStatement.status() without decodeNapiKernelError, so a failed poll (for example, retries exhausted on HTTP 500) reached the caller as a raw Error whose message is the __databricks_error__:{...} envelope. Decode it like the other napi calls. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> * test(kernel): assert the decoded poll error keeps its metadata; widen changelog scope The unit test now also checks kernelMetadata.httpStatus and the bounded cause. The changelog notes that a default execution that outlives the server's inline wait polls the same status() and was affected too, and adds the PR number. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> * fix(kernel): decode a failed batch fetch in hasMore() too hasMore() prefetches the next batch through the napi fetchNextBatch(), so when hasMoreRows() runs before the first fetch, a failed fetch reached the caller as the raw __databricks_error__:{...} envelope. Handle it like fetchChunk(): best-effort close the wedged statement, then throw the decoded error. Both paths now share one helper. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> --------- Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
1 parent 2037efd commit 91a64ff

3 files changed

Lines changed: 100 additions & 31 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
## Unreleased
44

5+
- Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608)
56
- Reduce result chunking and `fetchAll()` overhead for large query results.
67
- Thrift CloudFetch downloads are now bounded end to end by `retriesTimeout` (default 15 minutes), including every attempt and retry backoff. Previously a stalled download, where the response headers or body never arrived, blocked the fetch forever. A download still running when the deadline expires is aborted and fails with `RetryError` (`TIMEOUT_EXCEEDED`, message `CloudFetch result download failed: Retry timeout exceeded`). (#520)
78
- Thrift connections made through `proxy` now apply `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey`. Previously the proxied path ignored them and verified against Node's process-wide trust store, so a certificate outside the configured trust set could be accepted. **Compatibility:** proxied connections now verify unless `checkServerCertificate: false` is set; as on direct connections, `NODE_TLS_REJECT_UNAUTHORIZED=0` no longer turns verification off. For an `https` proxy these options also apply to the proxy's own certificate. (#485)

‎lib/kernel/KernelOperationBackend.ts‎

Lines changed: 38 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -330,34 +330,19 @@ export default class KernelOperationBackend implements IOperationBackend {
330330
}
331331
return await slicer.fetchNext({ limit, disableBuffering });
332332
} catch (err) {
333-
// The napi fetch contract leaves the stream in an unspecified state on
334-
// error ("call close() and discard"). Close the statement so the server
335-
// reclaims it promptly — best-effort, so a close failure never masks the
336-
// original fetch error — then surface a typed kernel error.
337-
//
338-
// If close() ALSO fails, kernelClose has reset isClosed back to false and
339-
// the kernel-side statement handle is now leaked (the stream is already
340-
// wedged, so nothing downstream forces another close). We still don't
341-
// mask the original fetch error, but log the close failure at warn so
342-
// the leak is diagnosable rather than completely invisible.
343-
await kernelClose(this.lifecycle, this.lifecycleHandle, this.context, this._id).catch((closeErr) => {
344-
const cause = closeErr instanceof Error ? closeErr.message : String(closeErr);
345-
this.context
346-
.getLogger()
347-
.log(
348-
LogLevel.warn,
349-
`kernel fetch-error cleanup: close() failed for operation ${this._id}; the server-side ` +
350-
`statement may leak until the session is closed. Cause: ${cause}`,
351-
);
352-
});
353-
throw decodeNapiKernelError(err);
333+
throw await this.fetchFailure(err);
354334
}
355335
}
356336

357337
public async hasMore(): Promise<boolean> {
358338
failIfNotActive(this.lifecycle);
359-
const slicer = await this.getResultSlicer();
360-
return slicer.hasMore();
339+
try {
340+
// `hasMore()` prefetches the next batch, so it fails like `fetchChunk()`.
341+
const slicer = await this.getResultSlicer();
342+
return await slicer.hasMore();
343+
} catch (err) {
344+
throw await this.fetchFailure(err);
345+
}
361346
}
362347

363348
public async getResultMetadata(): Promise<ResultMetadata> {
@@ -426,7 +411,7 @@ export default class KernelOperationBackend implements IOperationBackend {
426411
// `numModifiedRows` from the inline count on the terminal response), so
427412
// surface them — mirroring the sync path. Reading them does not fetch the
428413
// result.
429-
const state = statusStringToOperationState(await this.asyncStatement.status());
414+
const state = await this.pollAsyncStatus();
430415
if (state === OperationState.Succeeded) {
431416
this.asyncSucceeded = true;
432417
return { state, hasResultSet: true, ...(await this.readRichStatusFields()) };
@@ -624,7 +609,7 @@ export default class KernelOperationBackend implements IOperationBackend {
624609
failIfNotActive(this.lifecycle);
625610

626611
// eslint-disable-next-line no-await-in-loop
627-
const state = statusStringToOperationState(await this.asyncStatement!.status());
612+
const state = await this.pollAsyncStatus();
628613
// Record success before the progress callback runs, so a callback that
629614
// throws on the Succeeded tick still leaves `isFinished()` true.
630615
if (state === OperationState.Succeeded) {
@@ -721,6 +706,19 @@ export default class KernelOperationBackend implements IOperationBackend {
721706
}
722707
}
723708

709+
/**
710+
* One `AsyncStatement.status()` poll. A failed poll (e.g. retries exhausted on
711+
* a 5xx) rejects with the binding's error envelope, which is decoded here like
712+
* every other napi call.
713+
*/
714+
private async pollAsyncStatus(): Promise<OperationState> {
715+
try {
716+
return statusStringToOperationState(await this.asyncStatement!.status());
717+
} catch (err) {
718+
throw decodeNapiKernelError(err);
719+
}
720+
}
721+
724722
/**
725723
* Drive `awaitResult()` on a Failed statement to surface the kernel's typed
726724
* SQL-error envelope. Falls back to a generic error if `awaitResult()`
@@ -735,21 +733,30 @@ export default class KernelOperationBackend implements IOperationBackend {
735733
throw new HiveDriverError(`kernel operation ${this._id} reported Failed but produced a result.`);
736734
}
737735

736+
/**
737+
* A failed fetch leaves the napi stream in an unspecified state ("call close()
738+
* and discard"), so close the statement, then return the decoded error to throw.
739+
*/
740+
private async fetchFailure(err: unknown): Promise<Error> {
741+
await this.bestEffortClose('fetch-error');
742+
return decodeNapiKernelError(err);
743+
}
744+
738745
/**
739746
* Best-effort close of the kernel statement when the poll loop ends on a
740-
* server-driven terminal error (Failed/Cancelled/Closed/Unknown/Timeout).
741-
* Without it the kernel-side statement handle leaks until session close (the
742-
* poll loop, unlike `fetchChunk`, otherwise just throws). Never masks the
743-
* original error; warn-logs a close failure so the leak is diagnosable.
747+
* server-driven terminal error (Failed/Cancelled/Closed/Unknown/Timeout) or a
748+
* fetch fails. Without it the kernel-side statement handle leaks until session
749+
* close. Never masks the original error; warn-logs a close failure so the leak
750+
* is diagnosable.
744751
*/
745-
private async bestEffortClose(): Promise<void> {
752+
private async bestEffortClose(cleanup: 'poll-loop' | 'fetch-error' = 'poll-loop'): Promise<void> {
746753
await kernelClose(this.lifecycle, this.lifecycleHandle, this.context, this._id).catch((closeErr) => {
747754
const cause = closeErr instanceof Error ? closeErr.message : String(closeErr);
748755
this.context
749756
.getLogger()
750757
.log(
751758
LogLevel.warn,
752-
`kernel poll-loop cleanup: close() failed for operation ${this._id}; the server-side ` +
759+
`kernel ${cleanup} cleanup: close() failed for operation ${this._id}; the server-side ` +
753760
`statement may leak until the session is closed. Cause: ${cause}`,
754761
);
755762
});

‎tests/unit/kernel/execution.test.ts‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
import { expect } from 'chai';
1616
import sinon from 'sinon';
1717
import Int64 from 'node-int64';
18+
import { tableFromArrays, tableToIPC } from 'apache-arrow';
1819
import expectNativeConnectionOptions from './_helpers/nativeOptions';
1920
import KernelBackend from '../../../lib/kernel/KernelBackend';
2021
import KernelSessionBackend from '../../../lib/kernel/KernelSessionBackend';
@@ -122,6 +123,8 @@ class FakeAsyncStatement {
122123

123124
public awaitResultError: Error | null = null;
124125

126+
public statusError: Error | null = null;
127+
125128
// Successive status() returns drain this queue; the last value sticks.
126129
private readonly states: string[];
127130

@@ -136,6 +139,9 @@ class FakeAsyncStatement {
136139

137140
public async status(): Promise<string> {
138141
this.statusCalls += 1;
142+
if (this.statusError) {
143+
throw this.statusError;
144+
}
139145
return this.states.length > 1 ? (this.states.shift() as string) : this.states[0];
140146
}
141147

@@ -1465,6 +1471,61 @@ describe('KernelOperationBackend — async (submitStatement) path', () => {
14651471
expect((thrown as Error).message).to.match(/TABLE_OR_VIEW_NOT_FOUND/);
14661472
});
14671473

1474+
it('waitUntilReady() and status() decode a failed status poll instead of leaking the envelope', async () => {
1475+
const envelope = `__databricks_error__:${JSON.stringify({
1476+
code: 'Internal',
1477+
httpStatus: 500,
1478+
message: 'HTTP 500: Internal Server Error',
1479+
})}`;
1480+
for (const call of ['waitUntilReady', 'status'] as const) {
1481+
const stmt = new FakeAsyncStatement('Running');
1482+
stmt.statusError = new Error(envelope);
1483+
const op = makeAsyncOp(stmt);
1484+
let thrown: unknown;
1485+
try {
1486+
// eslint-disable-next-line no-await-in-loop
1487+
await (call === 'waitUntilReady' ? op.waitUntilReady() : op.status(false));
1488+
} catch (err) {
1489+
thrown = err;
1490+
}
1491+
// eslint-disable-next-line @typescript-eslint/no-explicit-any
1492+
const err = thrown as any;
1493+
expect(err, call).to.be.instanceOf(HiveDriverError);
1494+
expect(err.message, call).to.equal('HTTP 500: Internal Server Error');
1495+
expect(err.kernelMetadata?.httpStatus, call).to.equal(500);
1496+
expect(err.cause?.message, call).to.equal('Internal: HTTP 500: Internal Server Error');
1497+
}
1498+
});
1499+
1500+
// `hasMoreRows()` before the first fetch is what prefetches the first batch.
1501+
it('hasMoreRows() decodes a failed batch fetch and closes the statement', async () => {
1502+
const envelope = `__databricks_error__:${JSON.stringify({
1503+
code: 'Internal',
1504+
httpStatus: 500,
1505+
message: 'HTTP 500: Internal Server Error',
1506+
})}`;
1507+
const ipcBytes = Buffer.from(tableToIPC(tableFromArrays({ x: Int32Array.from([1, 2]) }), 'stream'));
1508+
const handle = new FakeNativeStatement();
1509+
handle.schema = () => ({ ipcBytes });
1510+
handle.fetchNextBatch = async () => {
1511+
throw new Error(envelope);
1512+
};
1513+
const stmt = new FakeAsyncStatement('Succeeded', handle);
1514+
const op = new DBSQLOperation({ backend: makeAsyncOp(stmt), context: makeContext() });
1515+
let thrown: unknown;
1516+
try {
1517+
await op.hasMoreRows();
1518+
} catch (err) {
1519+
thrown = err;
1520+
}
1521+
// eslint-disable-next-line @typescript-eslint/no-explicit-any
1522+
const err = thrown as any;
1523+
expect(err).to.be.instanceOf(HiveDriverError);
1524+
expect(err.message).to.equal('HTTP 500: Internal Server Error');
1525+
expect(err.kernelMetadata?.httpStatus).to.equal(500);
1526+
expect(stmt.closed, 'the wedged statement is closed').to.equal(true);
1527+
});
1528+
14681529
// A server-driven terminal state MUST throw OperationStateError (not a plain
14691530
// HiveDriverError) so the DBSQLOperation facade — which only mirrors its
14701531
// cancelled/closed flags when `err instanceof OperationStateError` — stays in

0 commit comments

Comments
 (0)