Skip to content

Commit 843637e

Browse files
Preserve the kernel CloudFetch default window of 16
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
1 parent 26f6699 commit 843637e

9 files changed

Lines changed: 52 additions & 10 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
- The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611)
66
- Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608)
7-
- Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 10 (previously the kernel default of 16) and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open.
7+
- Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open.
88
- Reduce result chunking and `fetchAll()` overhead for large query results.
99
- Thrift CloudFetch downloads are now bounded end to end by `retriesTimeout` (default 15 minutes), including every attempt and retry backoff. Previously a stalled download, where the response headers or body never arrived, blocked the fetch forever. A download still running when the deadline expires is aborted and fails with `RetryError` (`TIMEOUT_EXCEEDED`, message `CloudFetch result download failed: Retry timeout exceeded`). (#520)
1010
- Thrift connections made through `proxy` now apply `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey`. Previously the proxied path ignored them and verified against Node's process-wide trust store, so a certificate outside the configured trust set could be accepted. **Compatibility:** proxied connections now verify unless `checkServerCertificate: false` is set; as on direct connections, `NODE_TLS_REJECT_UNAUTHORIZED=0` no longer turns verification off. For an `https` proxy these options also apply to the proxy's own certificate. (#485)

‎CONNECTION_PARAMETERS.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -133,11 +133,14 @@ the presigned URLs returned by the server on both backends.
133133
## CloudFetch download window
134134

135135
Set `client.getConfig().cloudFetchConcurrentDownloads` before `openSession()`
136-
to tune downloads on either backend (default 10). Kernel sessions snapshot this
136+
to tune downloads on either backend (default 16 on the kernel, 10 on Thrift). Kernel sessions snapshot this
137137
value at open; changes apply to subsequent sessions. Thrift reads it on each
138138
fetch. Kernel mode requires a positive safe integer and rejects invalid values
139139
at session open. Values above 256 are clamped by the kernel.
140140

141+
Before connecting, an unset value reads as 10; it reads as 16 while connected
142+
through the kernel. Explicit assignments, including 10 before connecting, are retained.
143+
141144
The kernel bounds downloading and queued chunks ahead of consumption, excluding
142145
consumer buffers. This is not a byte limit. An explicit
143146
`cloudfetch_max_chunks_in_memory` in `openSession({ configuration: ... })` wins

‎lib/DBSQLClient.ts‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -195,6 +195,19 @@ export default class DBSQLClient extends EventEmitter implements IDBSQLClient, I
195195
constructor(options?: ClientOptions) {
196196
super();
197197
this.config = DBSQLClient.getDefaultConfig();
198+
// Explicit 10 must override the kernel default of 16.
199+
let cloudFetchConcurrentDownloads: number | undefined;
200+
Object.defineProperty(this.config, 'cloudFetchConcurrentDownloads', {
201+
enumerable: true,
202+
configurable: true,
203+
get: () => {
204+
if (cloudFetchConcurrentDownloads !== undefined) return cloudFetchConcurrentDownloads;
205+
return this.backend instanceof KernelBackend ? 16 : 10;
206+
},
207+
set: (value: number) => {
208+
cloudFetchConcurrentDownloads = value;
209+
},
210+
});
198211
this.logger = options?.logger ?? DBSQLClient.getDefaultLogger();
199212
this.logger.log(LogLevel.info, 'Created DBSQLClient');
200213
}

‎lib/contracts/IClientContext.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ export interface ClientConfig {
2020
retryDelayMax: number; // in milliseconds
2121

2222
useCloudFetch: boolean;
23-
/** Download window (default 10). Kernel sessions snapshot it at openSession;
23+
/** Download window (Thrift 10, connected kernel 16). Kernel sessions snapshot it at openSession;
2424
* it bounds downloading and queued chunks ahead of consumption, excluding
2525
* consumer buffers. It is not a byte limit. */
2626
cloudFetchConcurrentDownloads: number;

‎lib/kernel/KernelBackend.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,7 @@ export default class KernelBackend implements IBackend {
153153
sessionOptions.sessionConf = { ...request.configuration };
154154
}
155155
const configuredWindow = this.context.getConfig().cloudFetchConcurrentDownloads;
156-
const downloadWindow = configuredWindow === undefined ? 10 : configuredWindow;
156+
const downloadWindow = configuredWindow === undefined ? 16 : configuredWindow;
157157
if (!Number.isSafeInteger(downloadWindow) || downloadWindow <= 0) {
158158
throw new HiveDriverError('cloudFetchConcurrentDownloads must be a positive safe integer');
159159
}

‎tests/unit/DBSQLClient.test.ts‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -308,6 +308,30 @@ describe('DBSQLClient.connect', () => {
308308
logSpy.restore();
309309
});
310310

311+
for (const [useKernel, configuredWindow, expectedWindow] of [
312+
[false, undefined, 10],
313+
[true, undefined, 16],
314+
[true, 10, 10],
315+
] as const) {
316+
it(`resolves the download window for kernel=${useKernel}, setting=${configuredWindow}`, async () => {
317+
const client = new DBSQLClient();
318+
expect(client.getConfig().cloudFetchConcurrentDownloads).to.equal(10);
319+
if (configuredWindow !== undefined) {
320+
client.getConfig().cloudFetchConcurrentDownloads = configuredWindow;
321+
}
322+
sinon.stub(KernelNativeLoader, 'getKernelNative').returns({} as any);
323+
sinon.stub(KernelBackend.prototype, 'connect').resolves();
324+
sinon.stub(KernelBackend.prototype, 'close').resolves();
325+
try {
326+
await client.connect({ ...connectOptions, telemetryEnabled: false, useKernel } as any);
327+
expect(client.getConfig().cloudFetchConcurrentDownloads).to.equal(expectedWindow);
328+
} finally {
329+
await client.close();
330+
}
331+
expect(client.getConfig().cloudFetchConcurrentDownloads).to.equal(10);
332+
});
333+
}
334+
311335
it('useKernel: true routes to KernelBackend and leaves `backend` unset when connect() throws', async () => {
312336
const client = new DBSQLClient();
313337

@@ -333,6 +357,7 @@ describe('DBSQLClient.connect', () => {
333357
// failed connect, so the next openSession surfaces "not connected" rather
334358
// than the KernelBackend's own connect/auth error.
335359
expect((client as any).backend).to.equal(undefined);
360+
expect(client.getConfig().cloudFetchConcurrentDownloads).to.equal(10);
336361

337362
try {
338363
await client.openSession();

‎tests/unit/kernel/auth-m2m.test.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -381,7 +381,7 @@ describe('KernelAuth + KernelBackend — OAuth M2M auth flow', () => {
381381
expect(calls).to.have.lengthOf(1);
382382
expect(calls[0].method).to.equal('openSession');
383383
expectNativeConnectionOptions(calls[0].args[0], {
384-
sessionConf: { cloudfetch_max_chunks_in_memory: '10' },
384+
sessionConf: { cloudfetch_max_chunks_in_memory: '16' },
385385
hostName: 'example.cloud.databricks.com',
386386
httpPath: '/sql/1.0/warehouses/abc',
387387
intervalsAsString: true,

‎tests/unit/kernel/auth-u2m.test.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -223,7 +223,7 @@ describe('KernelAuth + KernelBackend — OAuth U2M auth flow', () => {
223223
expect(calls).to.have.lengthOf(1);
224224
expect(calls[0].method).to.equal('openSession');
225225
expectNativeConnectionOptions(calls[0].args[0], {
226-
sessionConf: { cloudfetch_max_chunks_in_memory: '10' },
226+
sessionConf: { cloudfetch_max_chunks_in_memory: '16' },
227227
hostName: 'example.cloud.databricks.com',
228228
httpPath: '/sql/1.0/warehouses/abc',
229229
intervalsAsString: true,

‎tests/unit/kernel/execution.test.ts‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -557,7 +557,7 @@ describe('KernelBackend', () => {
557557
expectNativeConnectionOptions(args, {
558558
hostName: 'workspace.example',
559559
httpPath: '/sql/1.0/warehouses/xyz',
560-
sessionConf: { cloudfetch_max_chunks_in_memory: '10' },
560+
sessionConf: { cloudfetch_max_chunks_in_memory: '16' },
561561
authMode: 'Pat',
562562
token: 'dapi-token',
563563
intervalsAsString: true,
@@ -754,7 +754,7 @@ describe('KernelBackend', () => {
754754
await backend.openSession({ configuration });
755755
const conf = (binding.openSessionStub.firstCall.args[0] as { sessionConf?: Record<string, string> }).sessionConf;
756756
expect(conf?.cloudfetch_enabled).to.equal(useCloudFetch === false ? 'false' : undefined);
757-
expect(conf?.cloudfetch_max_chunks_in_memory).to.equal('10');
757+
expect(conf?.cloudfetch_max_chunks_in_memory).to.equal('16');
758758
expect(conf?.can_cloud_download).to.equal(undefined);
759759
expect(conf?.TIMEZONE).to.equal('UTC');
760760
expect(configuration).to.deep.equal({ TIMEZONE: 'UTC' });
@@ -764,6 +764,7 @@ describe('KernelBackend', () => {
764764
for (const [window, override] of [
765765
[1, undefined],
766766
[4, undefined],
767+
[10, undefined],
767768
[1000, undefined],
768769
[4, 'cloudfetch_max_chunks_in_memory'],
769770
[4, 'CLOUDFETCH_MAX_CHUNKS_IN_MEMORY'],
@@ -839,7 +840,7 @@ describe('KernelBackend', () => {
839840
[key]: 'true',
840841
TIMEZONE: 'UTC',
841842
cloudfetch_enabled: 'false',
842-
cloudfetch_max_chunks_in_memory: '10',
843+
cloudfetch_max_chunks_in_memory: '16',
843844
});
844845
expect(configuration).to.deep.equal({ [key]: 'true', TIMEZONE: 'UTC' });
845846
});
@@ -917,7 +918,7 @@ describe('KernelBackend', () => {
917918
token: 't',
918919
catalog: 'main',
919920
schema: 'default',
920-
sessionConf: { 'spark.sql.execution.arrow.enabled': 'true', cloudfetch_max_chunks_in_memory: '10' },
921+
sessionConf: { 'spark.sql.execution.arrow.enabled': 'true', cloudfetch_max_chunks_in_memory: '16' },
921922
});
922923

923924
// And the SQL still threads through executeStatement (now with no

0 commit comments

Comments
 (0)