Skip to content

Commit b358888

Browse files
feat(cloudfetch): apply download settings to kernel sessions (#616)
* Honor standard CloudFetch download settings with the kernel Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com> * Document the kernel download window default change Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com> * Reject null CloudFetch download windows in kernel sessions Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com> * Document download setting validation at session open Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com> * Preserve the kernel CloudFetch default window of 16 Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com> --------- Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
1 parent afc7291 commit b358888

9 files changed

Lines changed: 142 additions & 2 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
- The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611)
66
- Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608)
7+
- Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open.
78
- Reduce result chunking and `fetchAll()` overhead for large query results.
89
- Thrift CloudFetch downloads are now bounded end to end by `retriesTimeout` (default 15 minutes), including every attempt and retry backoff. Previously a stalled download, where the response headers or body never arrived, blocked the fetch forever. A download still running when the deadline expires is aborted and fails with `RetryError` (`TIMEOUT_EXCEEDED`, message `CloudFetch result download failed: Retry timeout exceeded`). (#520)
910
- Thrift connections made through `proxy` now apply `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey`. Previously the proxied path ignored them and verified against Node's process-wide trust store, so a certificate outside the configured trust set could be accepted. **Compatibility:** proxied connections now verify unless `checkServerCertificate: false` is set; as on direct connections, `NODE_TLS_REJECT_UNAUTHORIZED=0` no longer turns verification off. For an `https` proxy these options also apply to the proxy's own certificate. (#485)

‎CONNECTION_PARAMETERS.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,22 @@ still use HTTPS to `host`. Kernel uses `http://host[:port]` for all of its
130130
requests, including its own OAuth and telemetry calls. CloudFetch downloads use
131131
the presigned URLs returned by the server on both backends.
132132

133+
## CloudFetch download window
134+
135+
Set `client.getConfig().cloudFetchConcurrentDownloads` before `openSession()`
136+
to tune downloads on either backend (default 16 on the kernel, 10 on Thrift). Kernel sessions snapshot this
137+
value at open; changes apply to subsequent sessions. Thrift reads it on each
138+
fetch. Kernel mode requires a positive safe integer and rejects invalid values
139+
at session open. Values above 256 are clamped by the kernel.
140+
141+
Before connecting, an unset value reads as 10; it reads as 16 while connected
142+
through the kernel. Explicit assignments, including 10 before connecting, are retained.
143+
144+
The kernel bounds downloading and queued chunks ahead of consumption, excluding
145+
consumer buffers. This is not a byte limit. An explicit
146+
`cloudfetch_max_chunks_in_memory` in `openSession({ configuration: ... })` wins
147+
case-insensitively. Invalid raw overrides warn and retain the kernel default 16.
148+
133149
## Results and type rendering
134150

135151
| Option | Type | Thrift | Kernel | Default Value | Note |

‎lib/DBSQLClient.ts‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -195,6 +195,19 @@ export default class DBSQLClient extends EventEmitter implements IDBSQLClient, I
195195
constructor(options?: ClientOptions) {
196196
super();
197197
this.config = DBSQLClient.getDefaultConfig();
198+
// Explicit 10 must override the kernel default of 16.
199+
let cloudFetchConcurrentDownloads: number | undefined;
200+
Object.defineProperty(this.config, 'cloudFetchConcurrentDownloads', {
201+
enumerable: true,
202+
configurable: true,
203+
get: () => {
204+
if (cloudFetchConcurrentDownloads !== undefined) return cloudFetchConcurrentDownloads;
205+
return this.backend instanceof KernelBackend ? 16 : 10;
206+
},
207+
set: (value: number) => {
208+
cloudFetchConcurrentDownloads = value;
209+
},
210+
});
198211
this.logger = options?.logger ?? DBSQLClient.getDefaultLogger();
199212
this.logger.log(LogLevel.info, 'Created DBSQLClient');
200213
}

‎lib/contracts/IClientContext.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,9 @@ export interface ClientConfig {
2020
retryDelayMax: number; // in milliseconds
2121

2222
useCloudFetch: boolean;
23+
/** Download window (Thrift 10, connected kernel 16). Kernel sessions snapshot it at openSession;
24+
* it bounds downloading and queued chunks ahead of consumption, excluding
25+
* consumer buffers. It is not a byte limit. */
2326
cloudFetchConcurrentDownloads: number;
2427
cloudFetchSpeedThresholdMBps: number;
2528

‎lib/kernel/KernelBackend.ts‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,15 @@ export default class KernelBackend implements IBackend {
152152
if (request.configuration !== undefined) {
153153
sessionOptions.sessionConf = { ...request.configuration };
154154
}
155+
const configuredWindow = this.context.getConfig().cloudFetchConcurrentDownloads;
156+
const downloadWindow = configuredWindow === undefined ? 16 : configuredWindow;
157+
if (!Number.isSafeInteger(downloadWindow) || downloadWindow <= 0) {
158+
throw new HiveDriverError('cloudFetchConcurrentDownloads must be a positive safe integer');
159+
}
160+
const chunkKey = 'cloudfetch_max_chunks_in_memory';
161+
if (!Object.keys(sessionOptions.sessionConf ?? {}).some((key) => key.toLowerCase() === chunkKey)) {
162+
sessionOptions.sessionConf = { ...sessionOptions.sessionConf, [chunkKey]: String(downloadWindow) };
163+
}
155164
if (this.context.getConfig().useCloudFetch === false) {
156165
sessionOptions.sessionConf = { ...sessionOptions.sessionConf, cloudfetch_enabled: 'false' };
157166
}

‎tests/unit/DBSQLClient.test.ts‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -308,6 +308,30 @@ describe('DBSQLClient.connect', () => {
308308
logSpy.restore();
309309
});
310310

311+
for (const [useKernel, configuredWindow, expectedWindow] of [
312+
[false, undefined, 10],
313+
[true, undefined, 16],
314+
[true, 10, 10],
315+
] as const) {
316+
it(`resolves the download window for kernel=${useKernel}, setting=${configuredWindow}`, async () => {
317+
const client = new DBSQLClient();
318+
expect(client.getConfig().cloudFetchConcurrentDownloads).to.equal(10);
319+
if (configuredWindow !== undefined) {
320+
client.getConfig().cloudFetchConcurrentDownloads = configuredWindow;
321+
}
322+
sinon.stub(KernelNativeLoader, 'getKernelNative').returns({} as any);
323+
sinon.stub(KernelBackend.prototype, 'connect').resolves();
324+
sinon.stub(KernelBackend.prototype, 'close').resolves();
325+
try {
326+
await client.connect({ ...connectOptions, telemetryEnabled: false, useKernel } as any);
327+
expect(client.getConfig().cloudFetchConcurrentDownloads).to.equal(expectedWindow);
328+
} finally {
329+
await client.close();
330+
}
331+
expect(client.getConfig().cloudFetchConcurrentDownloads).to.equal(10);
332+
});
333+
}
334+
311335
it('useKernel: true routes to KernelBackend and leaves `backend` unset when connect() throws', async () => {
312336
const client = new DBSQLClient();
313337

@@ -333,6 +357,7 @@ describe('DBSQLClient.connect', () => {
333357
// failed connect, so the next openSession surfaces "not connected" rather
334358
// than the KernelBackend's own connect/auth error.
335359
expect((client as any).backend).to.equal(undefined);
360+
expect(client.getConfig().cloudFetchConcurrentDownloads).to.equal(10);
336361

337362
try {
338363
await client.openSession();

‎tests/unit/kernel/auth-m2m.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -381,6 +381,7 @@ describe('KernelAuth + KernelBackend — OAuth M2M auth flow', () => {
381381
expect(calls).to.have.lengthOf(1);
382382
expect(calls[0].method).to.equal('openSession');
383383
expectNativeConnectionOptions(calls[0].args[0], {
384+
sessionConf: { cloudfetch_max_chunks_in_memory: '16' },
384385
hostName: 'example.cloud.databricks.com',
385386
httpPath: '/sql/1.0/warehouses/abc',
386387
intervalsAsString: true,

‎tests/unit/kernel/auth-u2m.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -223,6 +223,7 @@ describe('KernelAuth + KernelBackend — OAuth U2M auth flow', () => {
223223
expect(calls).to.have.lengthOf(1);
224224
expect(calls[0].method).to.equal('openSession');
225225
expectNativeConnectionOptions(calls[0].args[0], {
226+
sessionConf: { cloudfetch_max_chunks_in_memory: '16' },
226227
hostName: 'example.cloud.databricks.com',
227228
httpPath: '/sql/1.0/warehouses/abc',
228229
intervalsAsString: true,

‎tests/unit/kernel/execution.test.ts‎

Lines changed: 73 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -557,6 +557,7 @@ describe('KernelBackend', () => {
557557
expectNativeConnectionOptions(args, {
558558
hostName: 'workspace.example',
559559
httpPath: '/sql/1.0/warehouses/xyz',
560+
sessionConf: { cloudfetch_max_chunks_in_memory: '16' },
560561
authMode: 'Pat',
561562
token: 'dapi-token',
562563
intervalsAsString: true,
@@ -753,12 +754,77 @@ describe('KernelBackend', () => {
753754
await backend.openSession({ configuration });
754755
const conf = (binding.openSessionStub.firstCall.args[0] as { sessionConf?: Record<string, string> }).sessionConf;
755756
expect(conf?.cloudfetch_enabled).to.equal(useCloudFetch === false ? 'false' : undefined);
757+
expect(conf?.cloudfetch_max_chunks_in_memory).to.equal('16');
756758
expect(conf?.can_cloud_download).to.equal(undefined);
757759
expect(conf?.TIMEZONE).to.equal('UTC');
758760
expect(configuration).to.deep.equal({ TIMEZONE: 'UTC' });
759761
});
760762
});
761763

764+
for (const [window, override] of [
765+
[1, undefined],
766+
[4, undefined],
767+
[10, undefined],
768+
[1000, undefined],
769+
[4, 'cloudfetch_max_chunks_in_memory'],
770+
[4, 'CLOUDFETCH_MAX_CHUNKS_IN_MEMORY'],
771+
] as const) {
772+
it(`openSession() maps download window ${window}, override=${override}`, async () => {
773+
const binding = makeBinding(new FakeNativeConnection());
774+
const context = makeContext(undefined, { cloudFetchConcurrentDownloads: window });
775+
const backend = new KernelBackend({ context, nativeBinding: binding });
776+
await backend.connect({ host: 'h', path: '/p', token: 't' } as ConnectionOptions);
777+
const configuration = override ? { [override]: '2' } : {};
778+
await backend.openSession({ configuration });
779+
const options = binding.openSessionStub.firstCall.args[0] as { sessionConf: Record<string, string> };
780+
expect(options.sessionConf).to.deep.equal(
781+
override
782+
? { [override]: '2' }
783+
: {
784+
cloudfetch_max_chunks_in_memory: String(window),
785+
},
786+
);
787+
expect(configuration).to.deep.equal(override ? { [override]: '2' } : {});
788+
});
789+
}
790+
791+
for (const window of [0, -1, 1.5, NaN, Infinity, Number.MAX_SAFE_INTEGER + 1, null as unknown as number]) {
792+
it(`openSession() rejects invalid download window ${window}`, async () => {
793+
const binding = makeBinding(new FakeNativeConnection());
794+
const backend = new KernelBackend({
795+
context: makeContext(undefined, { cloudFetchConcurrentDownloads: window }),
796+
nativeBinding: binding,
797+
});
798+
await backend.connect({ host: 'h', path: '/p', token: 't' } as ConnectionOptions);
799+
try {
800+
await backend.openSession({});
801+
expect.fail('invalid window accepted');
802+
} catch (error) {
803+
expect(error).to.be.instanceOf(HiveDriverError);
804+
expect((error as Error).message).to.include('positive safe integer');
805+
}
806+
expect(binding.openSessionStub.called).to.equal(false);
807+
});
808+
}
809+
810+
it('snapshots the download window separately for each opened session', async () => {
811+
const binding = makeBinding(new FakeNativeConnection());
812+
const context = makeContext(undefined, { cloudFetchConcurrentDownloads: 1 });
813+
const backend = new KernelBackend({ context, nativeBinding: binding });
814+
await backend.connect({ host: 'h', path: '/p', token: 't' } as ConnectionOptions);
815+
await backend.openSession({});
816+
context.getConfig().cloudFetchConcurrentDownloads = 4;
817+
await backend.openSession({});
818+
expect(binding.openSessionStub.firstCall.args[0]).to.have.nested.property(
819+
'sessionConf.cloudfetch_max_chunks_in_memory',
820+
'1',
821+
);
822+
expect(binding.openSessionStub.secondCall.args[0]).to.have.nested.property(
823+
'sessionConf.cloudfetch_max_chunks_in_memory',
824+
'4',
825+
);
826+
});
827+
762828
['can_cloud_download', 'CLOUDFETCH_ENABLED', 'cloudfetch_enabled'].forEach((key) => {
763829
it(`openSession() forwards the client disable with ${key}`, async () => {
764830
const binding = makeBinding(new FakeNativeConnection());
@@ -770,7 +836,12 @@ describe('KernelBackend', () => {
770836
const configuration = { [key]: 'true', TIMEZONE: 'UTC' };
771837
await backend.openSession({ configuration });
772838
const conf = (binding.openSessionStub.firstCall.args[0] as { sessionConf?: Record<string, string> }).sessionConf;
773-
expect(conf).to.deep.equal({ [key]: 'true', TIMEZONE: 'UTC', cloudfetch_enabled: 'false' });
839+
expect(conf).to.deep.equal({
840+
[key]: 'true',
841+
TIMEZONE: 'UTC',
842+
cloudfetch_enabled: 'false',
843+
cloudfetch_max_chunks_in_memory: '16',
844+
});
774845
expect(configuration).to.deep.equal({ [key]: 'true', TIMEZONE: 'UTC' });
775846
});
776847
});
@@ -847,7 +918,7 @@ describe('KernelBackend', () => {
847918
token: 't',
848919
catalog: 'main',
849920
schema: 'default',
850-
sessionConf: { 'spark.sql.execution.arrow.enabled': 'true' },
921+
sessionConf: { 'spark.sql.execution.arrow.enabled': 'true', cloudfetch_max_chunks_in_memory: '16' },
851922
});
852923

853924
// And the SQL still threads through executeStatement (now with no

0 commit comments

Comments
 (0)