Skip to content

Commit a9a6052

Browse files
vuanhphungIsaac
andcommitted
fix: retry Thrift OpenSession and a 503-with-Retry-After ExecuteStatement
Thrift OpenSession was never retried, so a single 503, 429, 502, 504 or 408 failed the connection, and ExecuteStatement failed on a 503 with Retry-After, which the server returns without running the statement. The kernel backend retries both. OpenSession joins the idempotent methods, HttpRetryPolicy retries 408, and ExecuteStatement is also retried on a 503 with Retry-After. So that a wrong host or an untrusted certificate on OpenSession isn't retried for the whole budget and doesn't lose its cause, the shared classifier no longer retries certificate verification failures or ENOTFOUND, as in the kernel. After other network errors the retry-exhaustion message names the error code, and a CloudFetch error thrown without retrying no longer carries the presigned URL. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
1 parent f436057 commit a9a6052

11 files changed

Lines changed: 253 additions & 33 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
## Unreleased
44

5+
- Thrift `OpenSession` is now retried like other idempotent requests: on transient HTTP errors (408, 429, 5xx except 501) and transient network errors such as a refused or reset connection. HTTP 408 is now retried on every idempotent request. Thrift `ExecuteStatement` is now also retried on HTTP 503 with a `Retry-After` header, which the server returns without running the statement; other HTTP errors on `ExecuteStatement` are still not retried. Server certificate verification failures and a host name that doesn't resolve (`ENOTFOUND`) are no longer retried by idempotent Thrift requests, CloudFetch downloads or the feature-flag fetch: the original error is thrown at once (for CloudFetch as `CloudFetch result download failed: network error <code>`, without the presigned URL). These retries use the existing `retryMaxAttempts` / `retriesTimeout` budget. **Behavior change:** when it runs out, these calls fail with `HiveDriverError` (`Hive driver: <status or network error code> when connecting to resource. Max retry count exceeded.` or `... Retry timeout exceeded.`) instead of the `THTTPException` or network error, so `statusCode` and the server's error detail are no longer on the error.
56
- The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611)
67
- Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608)
78
- Reduce result chunking and `fetchAll()` overhead for large query results.

‎lib/connection/connections/HttpRetryPolicy.ts‎

Lines changed: 35 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,10 +36,38 @@ const RETRYABLE_NETWORK_ERROR_CODES = new Set([
3636
'EHOSTUNREACH',
3737
'ENETUNREACH',
3838
'EPIPE',
39-
'ENOTFOUND',
4039
'EAI_AGAIN',
4140
]);
4241

42+
// Retrying can't fix these, so they are thrown at once, as in the kernel: the client's certificate
43+
// verification (OpenSSL's X509 codes as Node names them, and its hostname check) and a host name that
44+
// doesn't resolve (`ENOTFOUND`; a temporary DNS failure is `EAI_AGAIN`).
45+
const PERMANENT_NETWORK_ERROR_CODES = new Set([
46+
'CERT_CHAIN_TOO_LONG',
47+
'CERT_HAS_EXPIRED',
48+
'CERT_NOT_YET_VALID',
49+
'CERT_REJECTED',
50+
'CERT_REVOKED',
51+
'CERT_SIGNATURE_FAILURE',
52+
'CERT_UNTRUSTED',
53+
'DEPTH_ZERO_SELF_SIGNED_CERT',
54+
'ERROR_IN_CERT_NOT_AFTER_FIELD',
55+
'ERROR_IN_CERT_NOT_BEFORE_FIELD',
56+
'ERR_TLS_CERT_ALTNAME_FORMAT',
57+
'ERR_TLS_CERT_ALTNAME_INVALID',
58+
'HOSTNAME_MISMATCH',
59+
'INVALID_CA',
60+
'INVALID_PURPOSE',
61+
'PATH_LENGTH_EXCEEDED',
62+
'SELF_SIGNED_CERT_IN_CHAIN',
63+
'UNABLE_TO_DECODE_ISSUER_PUBLIC_KEY',
64+
'UNABLE_TO_DECRYPT_CERT_SIGNATURE',
65+
'UNABLE_TO_GET_ISSUER_CERT',
66+
'UNABLE_TO_GET_ISSUER_CERT_LOCALLY',
67+
'UNABLE_TO_VERIFY_LEAF_SIGNATURE',
68+
'ENOTFOUND',
69+
]);
70+
4371
// Fallback message patterns for errors that don't carry an errno. node-fetch surfaces
4472
// "socket hang up" as a generic FetchError, and "Premature close" when the response
4573
// body stream closes before all data is received — both occur regularly when a
@@ -172,6 +200,10 @@ export default class HttpRetryPolicy implements IRetryPolicy<HttpTransactionDeta
172200
}
173201
const candidate = error as { code?: string; type?: string; message?: string };
174202

203+
if (typeof candidate.code === 'string' && PERMANENT_NETWORK_ERROR_CODES.has(candidate.code)) {
204+
return false;
205+
}
206+
175207
// node-fetch FetchError surfaces low-level network failures with `type: 'system'`
176208
// and a body-stream timeout with `type: 'body-timeout'`. Both should be retried;
177209
// `request-timeout` is converted to a Thrift TApplicationException upstream so
@@ -197,7 +229,8 @@ export default class HttpRetryPolicy implements IRetryPolicy<HttpTransactionDeta
197229
const result =
198230
// Retry on all codes below 100
199231
statusCode < 100 ||
200-
// ...and on `429 Too Many Requests`
232+
// ...and on `408 Request Timeout` and `429 Too Many Requests`
233+
statusCode === 408 ||
201234
statusCode === 429 ||
202235
// ...and on all `5xx` codes except for `501 Not Implemented`
203236
(statusCode >= 500 && statusCode !== 501);

‎lib/connection/connections/ThriftHttpConnection.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,8 @@ type ThriftClient = {
7676
};
7777

7878
const retryableThriftMethods = new Set([
79+
// A repeated OpenSession leaves at most an unused session, which the server expires
80+
'OpenSession',
7981
'GetOperationStatus',
8082
'CancelOperation',
8183
'CloseOperation',
@@ -93,7 +95,8 @@ const retryableThriftMethods = new Set([
9395
'GetCrossReference',
9496
]);
9597

96-
// Not safe to repeat once the server may have the request, so retried only if it was never sent
98+
// Not safe to repeat once the server may have run the request, so retried only if it was never sent or
99+
// the server rejected it unrun (see UnsentRequestRetryPolicy)
97100
const unsentRequestRetryableThriftMethods = new Set(['ExecuteStatement']);
98101

99102
export default class ThriftHttpConnection extends EventEmitter {

‎lib/connection/connections/UnsentRequestRetryPolicy.ts‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ import RetryError from '../../errors/RetryError';
77
// socket (ECONNRESET, EPIPE, timeouts, ...) are excluded: the server may already have the request.
88
const UNSENT_REQUEST_ERROR_CODES = new Set(['ECONNREFUSED', 'ENOTFOUND', 'EAI_AGAIN']);
99

10-
// For operations that are not safe to repeat: retries only requests that never left the client.
10+
// For operations that are not safe to repeat: retries only requests that never left the client, and a
11+
// 503 with `Retry-After`, where the server asks for a retry without running the request (as in the kernel).
1112
export default class UnsentRequestRetryPolicy extends HttpRetryPolicy {
1213
public async invokeWithRetry(operation: RetryableOperation<HttpTransactionDetails>): Promise<HttpTransactionDetails> {
1314
try {
@@ -21,8 +22,8 @@ export default class UnsentRequestRetryPolicy extends HttpRetryPolicy {
2122
}
2223
}
2324

24-
protected isRetryable(): boolean {
25-
return false;
25+
protected isRetryable({ response }: HttpTransactionDetails): boolean {
26+
return response.status === 503 && response.headers.has('Retry-After');
2627
}
2728

2829
protected isRetryableNetworkError(error: unknown): boolean {

‎lib/hive/Commands/BaseCommand.ts‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -27,16 +27,15 @@ export default abstract class BaseCommand<ClientType> {
2727
) {
2828
statusCode = error.payload.response.status;
2929
}
30+
// After network errors, name the last one's code (e.g. ECONNREFUSED) instead of a status
31+
const code = (error.payload as { code?: unknown } | null | undefined)?.code;
32+
const cause = statusCode ?? (typeof code === 'string' && code ? code : 'Error');
3033

3134
switch (error.errorCode) {
3235
case RetryErrorCode.AttemptsExceeded:
33-
throw new HiveDriverError(
34-
`Hive driver: ${statusCode ?? 'Error'} when connecting to resource. Max retry count exceeded.`,
35-
);
36+
throw new HiveDriverError(`Hive driver: ${cause} when connecting to resource. Max retry count exceeded.`);
3637
case RetryErrorCode.TimeoutExceeded:
37-
throw new HiveDriverError(
38-
`Hive driver: ${statusCode ?? 'Error'} when connecting to resource. Retry timeout exceeded.`,
39-
);
38+
throw new HiveDriverError(`Hive driver: ${cause} when connecting to resource. Retry timeout exceeded.`);
4039
// no default
4140
}
4241
}

‎lib/result/CloudFetchResultHandler.ts‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import fetch, { RequestInfo, RequestInit, Request, Response } from 'node-fetch';
1+
import fetch, { FetchError, RequestInfo, RequestInit, Request, Response } from 'node-fetch';
22
import { TGetResultSetMetadataResp, TRowSet, TSparkArrowResultLink } from '../../thrift/TCLIService_types';
33
import HiveDriverError from '../errors/HiveDriverError';
44
import RetryError, { RetryErrorCode } from '../errors/RetryError';
@@ -244,10 +244,12 @@ export default class CloudFetchResultHandler implements IResultsProvider<ArrowBa
244244
* only the error code / type. Raw network-error messages are deliberately NOT copied, because
245245
* node-fetch embeds the presigned (credential-bearing) request URL in them.
246246
* The same error instance, class, error code and payload (including the node-fetch
247-
* `Request` / `Response` objects) are kept. Errors other than `RetryError` are returned unchanged.
247+
* `Request` / `Response` objects) are kept. A node-fetch error thrown without retrying (e.g. an
248+
* untrusted certificate) gets the same prefix, with its code in place of its message. Other errors
249+
* are returned unchanged.
248250
*/
249251
private static describeDownloadError(error: unknown): unknown {
250-
if (!(error instanceof RetryError)) {
252+
if (!(error instanceof RetryError) && !(error instanceof FetchError)) {
251253
return error;
252254
}
253255

@@ -259,7 +261,7 @@ export default class CloudFetchResultHandler implements IResultsProvider<ArrowBa
259261
}
260262

261263
let cause: string | undefined;
262-
const { payload } = error;
264+
const payload = error instanceof RetryError ? error.payload : error;
263265
if (payload instanceof Error) {
264266
const { code, type } = payload as Error & { code?: unknown; type?: unknown };
265267
if (typeof code === 'string' && code) {
@@ -281,7 +283,10 @@ export default class CloudFetchResultHandler implements IResultsProvider<ArrowBa
281283
// where the retry policy gave up. V8 bakes the message into the `stack` header at construction
282284
// time, so rewrite that header too — otherwise loggers that print `err.stack` miss the context.
283285
const originalHeader = `${error.name}: ${error.message}`;
284-
const message = `${prefix}${error.message}${cause ? ` (last error: ${cause})` : ''}`;
286+
const message =
287+
error instanceof RetryError
288+
? `${prefix}${error.message}${cause ? ` (last error: ${cause})` : ''}`
289+
: `${prefix}${cause}`;
285290
error.message = message;
286291
if (typeof error.stack === 'string' && error.stack.startsWith(originalHeader)) {
287292
error.stack = `${error.name}: ${message}${error.stack.slice(originalHeader.length)}`;

‎tests/unit/connection/connections/HttpRetryPolicy.test.ts‎

Lines changed: 63 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,9 +64,9 @@ describe('HttpRetryPolicy', () => {
6464
}
6565

6666
// Status codes between 100 (including) and 500 (excluding) should not be retried
67-
// The only exception is 429 (Too many requests)
67+
// The only exceptions are 408 (Request Timeout) and 429 (Too many requests)
6868
for (let status = 100; status < 500; status += 1) {
69-
const expectedResult = status === 429 ? true : false;
69+
const expectedResult = status === 408 || status === 429;
7070
expect(policy['isRetryable'](createStub(status))).to.equal(expectedResult);
7171
}
7272

@@ -401,6 +401,67 @@ describe('HttpRetryPolicy', () => {
401401
}
402402
});
403403

404+
describe('network error classification', () => {
405+
function systemError(code: string): FetchError {
406+
return new FetchError(`request to https://example.com failed, reason: ${code}`, 'system', {
407+
code,
408+
} as unknown as NodeJS.ErrnoException);
409+
}
410+
411+
function createPolicy() {
412+
return new HttpRetryPolicy(new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 5 }));
413+
}
414+
415+
// Certificate verification failures and a host name that doesn't resolve
416+
for (const code of [
417+
'DEPTH_ZERO_SELF_SIGNED_CERT',
418+
'UNABLE_TO_VERIFY_LEAF_SIGNATURE',
419+
'SELF_SIGNED_CERT_IN_CHAIN',
420+
'CERT_HAS_EXPIRED',
421+
'ERR_TLS_CERT_ALTNAME_INVALID',
422+
'ENOTFOUND',
423+
]) {
424+
it(`throws ${code} at once, without retrying`, async () => {
425+
const error = systemError(code);
426+
const operation = sinon.stub().rejects(error);
427+
428+
try {
429+
await createPolicy().invokeWithRetry(operation);
430+
expect.fail('It should throw the error');
431+
} catch (e) {
432+
if (e instanceof AssertionError) {
433+
throw e;
434+
}
435+
expect(e).to.equal(error);
436+
}
437+
expect(operation.callCount).to.equal(1);
438+
});
439+
}
440+
441+
// Transient connect/DNS failures, a TLS alert sent by the server, and unavailable revocation data
442+
for (const code of [
443+
'ECONNREFUSED',
444+
'ECONNRESET',
445+
'EAI_AGAIN',
446+
'ERR_SSL_TLSV13_ALERT_CERTIFICATE_REQUIRED',
447+
'UNABLE_TO_GET_CRL',
448+
]) {
449+
it(`retries ${code}`, async () => {
450+
const operation = sinon.stub();
451+
operation.onCall(0).rejects(systemError(code));
452+
operation.onCall(1).resolves({
453+
request: new Request('http://localhost'),
454+
response: new Response(undefined, { status: 200 }),
455+
});
456+
457+
const result = await createPolicy().invokeWithRetry(operation);
458+
459+
expect(result.response.status).to.equal(200);
460+
expect(operation.callCount).to.equal(2);
461+
});
462+
}
463+
});
464+
404465
it('should cancel a pending backoff sleep when the signal is aborted', async () => {
405466
const context = new ClientContextStub({
406467
retryDelayMin: 10000,

‎tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts‎

Lines changed: 50 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,13 @@ function thriftMessage(name: string, type: number, seqId: number): Buffer {
2020
const CALL = 1;
2121
const REPLY = 2;
2222

23-
type ServerAction = 'reply' | 'close-after-request' | 'status-503';
23+
type ServerAction =
24+
| 'reply'
25+
| 'close-after-request'
26+
| 'status-408'
27+
| 'status-429'
28+
| 'status-503'
29+
| 'status-503-retry-after';
2430

2531
interface TestServer {
2632
port: number;
@@ -39,8 +45,11 @@ async function startServer(actions: ServerAction[]): Promise<TestServer> {
3945
if (action === 'close-after-request') {
4046
// The full request has been received; drop the connection without a response
4147
req.socket.destroy();
42-
} else if (action === 'status-503') {
43-
res.writeHead(503);
48+
} else if (action === 'status-503-retry-after') {
49+
res.writeHead(503, { 'Retry-After': '1' });
50+
res.end();
51+
} else if (action.startsWith('status-')) {
52+
res.writeHead(Number(action.slice('status-'.length)));
4453
res.end();
4554
} else {
4655
const body = Buffer.concat(chunks);
@@ -179,20 +188,52 @@ describe('ThriftHttpConnection retries', () => {
179188
expect(server.requests).to.equal(2);
180189
});
181190

182-
it('does not retry ExecuteStatement on an HTTP error status', async () => {
183-
const { server, connection } = await setup(['status-503'], 0);
191+
for (const status of ['status-503', 'status-429', 'status-408'] as const) {
192+
it(`does not retry ExecuteStatement on an HTTP error status (${status})`, async () => {
193+
const { server, connection } = await setup([status], 0);
184194

185-
const error = await expectRejection(send(connection, 'ExecuteStatement', 1));
195+
const error = await expectRejection(send(connection, 'ExecuteStatement', 1));
196+
197+
expect(error).to.be.instanceOf(THTTPException);
198+
expect(error.statusCode).to.equal(Number(status.slice('status-'.length)));
199+
expect(server.requests).to.equal(1);
200+
});
201+
}
202+
203+
it('retries ExecuteStatement on a 503 with Retry-After, which the server did not run', async () => {
204+
const { server, connection } = await setup(['status-503-retry-after', 'reply'], 0);
186205

187-
expect(error).to.be.instanceOf(THTTPException);
188-
expect(error.statusCode).to.equal(503);
206+
const result = await send(connection, 'ExecuteStatement', 1);
207+
208+
expect(result).to.equal('ExecuteStatement result');
209+
expect(server.requests).to.equal(2);
210+
});
211+
212+
for (const status of ['status-503', 'status-429', 'status-408'] as const) {
213+
it(`retries OpenSession on a transient HTTP error (${status})`, async () => {
214+
const { server, connection } = await setup([status, 'reply'], 0);
215+
216+
const result = await send(connection, 'OpenSession', 1);
217+
218+
expect(result).to.equal('OpenSession result');
219+
expect(server.requests).to.equal(2);
220+
});
221+
}
222+
223+
it('retries OpenSession when the connection is refused', async () => {
224+
const { server, agent, connection } = await setup(['reply'], 1);
225+
226+
const result = await send(connection, 'OpenSession', 1);
227+
228+
expect(result).to.equal('OpenSession result');
229+
expect(agent.connectAttempts).to.equal(2);
189230
expect(server.requests).to.equal(1);
190231
});
191232

192233
it('does not retry other non-idempotent operations when the connection is refused', async () => {
193234
const { server, agent, connection } = await setup(['reply'], 1);
194235

195-
const error = await expectRejection(send(connection, 'OpenSession', 1));
236+
const error = await expectRejection(send(connection, 'FetchResults', 1));
196237

197238
expect(error).to.be.instanceOf(FetchError);
198239
expect(error.code).to.equal('ECONNREFUSED');

‎tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts‎

Lines changed: 35 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -64,9 +64,20 @@ describe('UnsentRequestRetryPolicy', () => {
6464
});
6565
}
6666

67-
for (const status of [429, 500, 503]) {
68-
it(`does not retry an HTTP ${status} response`, async () => {
69-
const details = { request: new Request('http://localhost'), response: new Response(undefined, { status }) };
67+
function statusResponse(status: number, headers: Record<string, string> = {}) {
68+
return { request: new Request('http://localhost'), response: new Response(undefined, { status, headers }) };
69+
}
70+
71+
for (const [status, headers] of [
72+
[429, {}],
73+
[500, {}],
74+
[503, {}],
75+
[500, { 'Retry-After': '1' }],
76+
] as const) {
77+
it(`does not retry an HTTP ${status} response ${
78+
Object.keys(headers).length ? 'with' : 'without'
79+
} Retry-After`, async () => {
80+
const details = statusResponse(status, headers);
7081
const policy = createPolicy();
7182
const operation = sinon.stub().resolves(details);
7283

@@ -76,6 +87,27 @@ describe('UnsentRequestRetryPolicy', () => {
7687
});
7788
}
7889

90+
it('retries an HTTP 503 with Retry-After, which the server did not run', async () => {
91+
const operation = sinon.stub();
92+
operation.onCall(0).resolves(statusResponse(503, { 'Retry-After': '1' }));
93+
operation.onCall(1).resolves(okResponse());
94+
95+
const result = await createPolicy().invokeWithRetry(operation);
96+
97+
expect(result.response.status).to.equal(200);
98+
expect(operation.callCount).to.equal(2);
99+
});
100+
101+
it('throws a RetryError once a persistent 503 with Retry-After runs out of attempts', async () => {
102+
const operation = sinon.stub().resolves(statusResponse(503, { 'Retry-After': '1' }));
103+
104+
const error = await expectRejection(createPolicy(3).invokeWithRetry(operation));
105+
106+
expect(error).to.be.instanceOf(RetryError);
107+
expect((error as RetryError).errorCode).to.equal(RetryErrorCode.AttemptsExceeded);
108+
expect(operation.callCount).to.equal(3);
109+
});
110+
79111
it('rethrows the original error once max attempts is reached', async () => {
80112
const error = systemError('ECONNREFUSED');
81113
const operation = sinon.stub().rejects(error);

0 commit comments

Comments
 (0)