Skip to content

Commit 79ae23e

Browse files
vuanhphungIsaac
andcommitted
fix: throw OpenSession network errors at once instead of retrying them
OpenSession went through HttpRetryPolicy, which retries every node-fetch system error. A wrong host, a refused port or an untrusted certificate then took the whole retry budget (30 s by default) and surfaced as "Hive driver: Error when connecting to resource. Max retry count exceeded.", without the cause. That broke the TLS certificate checks (driver-test AUTH-017/018) on Thrift. OpenSession now has its own policy: transient HTTP statuses are retried as before, network errors are thrown at once, as they were before this PR. Also say that only the status rule of NonIdempotentRetryPolicy matches the kernel. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
1 parent 0b0f974 commit 79ae23e

7 files changed

Lines changed: 126 additions & 4 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
## Unreleased
44

5-
- Thrift `OpenSession` is now retried on transient failures (HTTP 408, 429, 5xx except 501, and network errors), like other idempotent requests. Thrift `ExecuteStatement` is now also retried on HTTP 429 and on HTTP 503 with a `Retry-After` header, which the server returns without running the statement; other HTTP errors on `ExecuteStatement` are still not retried, so a statement can't run twice. HTTP 408 is now retried on every idempotent request. These retries use the existing `retryMaxAttempts` / `retriesTimeout` budget, matching the kernel backend. **Behavior change:** when that budget runs out, these calls now fail like other retried requests, with a `HiveDriverError` (`Hive driver: <status> when connecting to resource. Max retry count exceeded.` or `... Retry timeout exceeded.`) instead of a `THTTPException` with `statusCode`.
5+
- Thrift `OpenSession` is now retried on transient HTTP errors (408, 429, 5xx except 501), like other idempotent requests. Network errors on `OpenSession` (for example a wrong host or port, or an untrusted certificate) are still thrown at once. Thrift `ExecuteStatement` is now also retried on HTTP 429 and on HTTP 503 with a `Retry-After` header, which the server returns without running the statement; other HTTP errors on `ExecuteStatement` are still not retried, so a statement can't run twice. HTTP 408 is now retried on every idempotent request. These retries use the existing `retryMaxAttempts` / `retriesTimeout` budget, matching the kernel backend. **Behavior change:** when that budget runs out, these calls now fail like other retried requests, with a `HiveDriverError` (`Hive driver: <status> when connecting to resource. Max retry count exceeded.` or `... Retry timeout exceeded.`) instead of a `THTTPException` with `statusCode`.
66
- Reduce result chunking and `fetchAll()` overhead for large query results.
77
- Thrift CloudFetch downloads are now bounded end to end by `retriesTimeout` (default 15 minutes), including every attempt and retry backoff. Previously a stalled download, where the response headers or body never arrived, blocked the fetch forever. A download still running when the deadline expires is aborted and fails with `RetryError` (`TIMEOUT_EXCEEDED`, message `CloudFetch result download failed: Retry timeout exceeded`). (#520)
88
- Thrift connections made through `proxy` now apply `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey`. Previously the proxied path ignored them and verified against Node's process-wide trust store, so a certificate outside the configured trust set could be accepted. **Compatibility:** proxied connections now verify unless `checkServerCertificate: false` is set; as on direct connections, `NODE_TLS_REJECT_UNAUTHORIZED=0` no longer turns verification off. For an `https` proxy these options also apply to the proxy's own certificate. (#485)

‎lib/connection/connections/HttpConnection.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ import ThriftHttpConnection from './ThriftHttpConnection';
1414
import IRetryPolicy from '../contracts/IRetryPolicy';
1515
import HttpRetryPolicy from './HttpRetryPolicy';
1616
import NonIdempotentRetryPolicy from './NonIdempotentRetryPolicy';
17+
import OpenSessionRetryPolicy from './OpenSessionRetryPolicy';
1718

1819
// Set by the driver, auth provider or Thrift transport. node-fetch merges
1920
// same-named headers case-insensitively into one value, so a caller copy
@@ -214,6 +215,7 @@ export default class HttpConnection implements IConnectionProvider {
214215
transport: thrift.TBufferedTransport,
215216
protocol: thrift.TBinaryProtocol,
216217
getRetryPolicy: () => this.getRetryPolicy(),
218+
getOpenSessionRetryPolicy: async () => new OpenSessionRetryPolicy(this.context),
217219
getNonIdempotentRetryPolicy: async () => new NonIdempotentRetryPolicy(this.context),
218220
},
219221
{

‎lib/connection/connections/NonIdempotentRetryPolicy.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ const UNSENT_REQUEST_ERROR_CODES = new Set(['ECONNREFUSED', 'ENOTFOUND', 'EAI_AG
99

1010
// For operations that are not safe to repeat: retries only requests the server did not run, i.e.
1111
// requests that never left the client, a 429, or a 503 with `Retry-After` (the server asks for a
12-
// retry). Other failures are ambiguous and are not retried. Same rules as the kernel backend.
12+
// retry). Other failures are ambiguous and are not retried. The status rule matches the kernel backend.
1313
export default class NonIdempotentRetryPolicy extends HttpRetryPolicy {
1414
public async invokeWithRetry(operation: RetryableOperation<HttpTransactionDetails>): Promise<HttpTransactionDetails> {
1515
try {
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
import HttpRetryPolicy from './HttpRetryPolicy';
2+
3+
// For OpenSession: retried on transient HTTP statuses like other idempotent requests (a repeated open
4+
// leaves at most an unused session, which the server expires). Network errors are thrown at once: on
5+
// the first request they usually mean a wrong host or port or an untrusted certificate, and retrying
6+
// would only delay that error and replace it with a generic retry-exhaustion one.
7+
export default class OpenSessionRetryPolicy extends HttpRetryPolicy {
8+
protected isRetryableNetworkError(): boolean {
9+
return false;
10+
}
11+
}

‎lib/connection/connections/ThriftHttpConnection.ts‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@ interface ThriftHttpConnectionOptions {
6060
transport?: TTransportType;
6161
protocol?: TProtocolConstructor;
6262
getRetryPolicy(): Promise<IRetryPolicy<HttpTransactionDetails>>;
63+
getOpenSessionRetryPolicy?(): Promise<IRetryPolicy<HttpTransactionDetails>>;
6364
getNonIdempotentRetryPolicy?(): Promise<IRetryPolicy<HttpTransactionDetails>>;
6465
}
6566

@@ -76,8 +77,6 @@ type ThriftClient = {
7677
};
7778

7879
const retryableThriftMethods = new Set([
79-
// A repeated OpenSession leaves at most an unused session, which the server expires
80-
'OpenSession',
8180
'GetOperationStatus',
8281
'CancelOperation',
8382
'CloseOperation',
@@ -128,6 +127,9 @@ export default class ThriftHttpConnection extends EventEmitter {
128127
if (thriftMethodName && retryableThriftMethods.has(thriftMethodName)) {
129128
return this.options.getRetryPolicy();
130129
}
130+
if (thriftMethodName === 'OpenSession' && this.options.getOpenSessionRetryPolicy) {
131+
return this.options.getOpenSessionRetryPolicy();
132+
}
131133
if (
132134
thriftMethodName &&
133135
nonIdempotentThriftMethods.has(thriftMethodName) &&
Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
import { expect, AssertionError } from 'chai';
2+
import sinon from 'sinon';
3+
import { Request, Response, FetchError } from 'node-fetch';
4+
import OpenSessionRetryPolicy from '../../../../lib/connection/connections/OpenSessionRetryPolicy';
5+
import RetryError, { RetryErrorCode } from '../../../../lib/errors/RetryError';
6+
7+
import ClientContextStub from '../../.stubs/ClientContextStub';
8+
9+
function systemError(code: string, reason = code): FetchError {
10+
return new FetchError(`request to https://example.com failed, reason: ${reason}`, 'system', {
11+
code,
12+
} as unknown as NodeJS.ErrnoException);
13+
}
14+
15+
function statusResponse(status: number) {
16+
return { request: new Request('http://localhost'), response: new Response(undefined, { status }) };
17+
}
18+
19+
function createPolicy(retryMaxAttempts = 5) {
20+
return new OpenSessionRetryPolicy(new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts }));
21+
}
22+
23+
async function expectRejection(promise: Promise<unknown>): Promise<unknown> {
24+
try {
25+
await promise;
26+
} catch (error) {
27+
if (error instanceof AssertionError) {
28+
throw error;
29+
}
30+
return error;
31+
}
32+
return expect.fail('It should throw an error');
33+
}
34+
35+
describe('OpenSessionRetryPolicy', () => {
36+
for (const status of [408, 429, 500, 502, 503, 504]) {
37+
it(`retries an HTTP ${status} response`, async () => {
38+
const operation = sinon.stub();
39+
operation.onCall(0).resolves(statusResponse(status));
40+
operation.onCall(1).resolves(statusResponse(200));
41+
42+
const result = await createPolicy().invokeWithRetry(operation);
43+
44+
expect(result.response.status).to.equal(200);
45+
expect(operation.callCount).to.equal(2);
46+
});
47+
}
48+
49+
for (const status of [400, 401, 403, 501]) {
50+
it(`does not retry an HTTP ${status} response`, async () => {
51+
const details = statusResponse(status);
52+
const operation = sinon.stub().resolves(details);
53+
54+
expect(await createPolicy().invokeWithRetry(operation)).to.equal(details);
55+
expect(operation.callCount).to.equal(1);
56+
});
57+
}
58+
59+
it('throws a RetryError once a persistent 503 runs out of attempts', async () => {
60+
const operation = sinon.stub().resolves(statusResponse(503));
61+
62+
const error = await expectRejection(createPolicy(3).invokeWithRetry(operation));
63+
64+
expect(error).to.be.instanceOf(RetryError);
65+
expect((error as RetryError).errorCode).to.equal(RetryErrorCode.AttemptsExceeded);
66+
expect(operation.callCount).to.equal(3);
67+
});
68+
69+
const networkErrors: Array<[string, Error]> = [
70+
['an untrusted certificate', systemError('DEPTH_ZERO_SELF_SIGNED_CERT', 'self-signed certificate')],
71+
['a hostname mismatch', systemError('ERR_TLS_CERT_ALTNAME_INVALID')],
72+
['an unknown host', systemError('ENOTFOUND')],
73+
['a refused connection', systemError('ECONNREFUSED')],
74+
['a connection reset', systemError('ECONNRESET', 'socket hang up')],
75+
['a premature close', new Error('Premature close')],
76+
];
77+
78+
for (const [label, error] of networkErrors) {
79+
it(`throws ${label} at once, without retrying`, async () => {
80+
const operation = sinon.stub().rejects(error);
81+
82+
expect(await expectRejection(createPolicy().invokeWithRetry(operation))).to.equal(error);
83+
expect(operation.callCount).to.equal(1);
84+
});
85+
}
86+
});

‎tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -222,6 +222,27 @@ describe('ThriftHttpConnection retries', () => {
222222
});
223223
}
224224

225+
it('throws a refused OpenSession connection at once, without retrying', async () => {
226+
const { server, agent, connection } = await setup(['reply'], 1);
227+
228+
const error = await expectRejection(send(connection, 'OpenSession', 1));
229+
230+
expect(error).to.be.instanceOf(FetchError);
231+
expect(error.code).to.equal('ECONNREFUSED');
232+
expect(agent.connectAttempts).to.equal(1);
233+
expect(server.requests).to.equal(0);
234+
});
235+
236+
it('does not retry OpenSession when the connection closes after the request was sent', async () => {
237+
const { server, connection } = await setup(['close-after-request', 'reply'], 0);
238+
239+
const error = await expectRejection(send(connection, 'OpenSession', 1));
240+
241+
expect(error).to.be.instanceOf(FetchError);
242+
expect(error.code).to.equal('ECONNRESET');
243+
expect(server.requests).to.equal(1);
244+
});
245+
225246
it('does not retry other non-idempotent operations when the connection is refused', async () => {
226247
const { server, agent, connection } = await setup(['reply'], 1);
227248

0 commit comments

Comments
 (0)