Skip to content

Commit 9301a2b

Browse files
authored
fix(client): retry ExecuteStatement when the connection was never opened (#555)
On the Thrift backend, `RetryPolicy` never retried a transport error on `ExecuteStatement`, not even a refused connection, because the RPC isn't idempotent. Every other RPC retries those errors, as do the kernel and Node.js on their ExecuteStatement paths. Now `ExecuteStatement` retries, using the existing `WithRetries` budget and backoff, only when two things hold. First, the error is a dial-stage `*net.OpError`: connection refused, unreachable, connect timeout, transient DNS, or the dial to a proxy inside `proxyconnect`. Second, the attempt didn't follow a redirect. A dial error means the failing hop never had a connection. A followed redirect means an earlier hop already delivered the statement, so the transport wrapper records redirect hops per attempt and those attempts aren't retried. DNS not-found, cancellation and the existing certificate carve-outs are excluded. TLS handshake, read and write failures still fail after one attempt, so a statement can't run twice. A stale pooled connection where nothing was written already recovers inside `net/http`, since Thrift requests carry `GetBody`. Verification: - **Table-driven classifier tests:** cover 30 error shapes. - **Fault tests:** run the real Thrift client, retryablehttp, `http.Transport` and a real Thrift processor over local sockets. - Refused for the first two dials gives 3 dials and exactly 1 execution. Persistent refused, direct or through a proxy, gives 5 attempts. - These cases each give exactly 1 attempt: reset after the server executed the statement (1 execution), reset mid-body (16 MiB statement), reset after headers, reset after the first bytes over HTTP and HTTPS, untrusted certificate, caller cancellation during the dial, and a client timeout during the dial. - A 303, 307 or 308 redirect to a refused host after the server executed gives exactly 1 execution (5 before the redirect fix). - A redirect to a reachable server still succeeds. A refused first hop after an earlier redirected attempt is still retried. - The tests are stable across 100 runs, half of them under `-race`. - **Live (Thrift):** the four `TestE2E*` tests pass. A temporary probe refused two dials at ExecuteStatement time against the warehouse. On the branch the query returned its row after 3 dials (287 ms); on `main` it failed with `request error after 1 attempt(s) … connection refused`. - **Checks:** `make test` (1352 tests) and `make lint` pass, and kernel-tagged `go vet` passes. - **Performance (benchstat, `main` vs branch, 8 alternating fresh processes):** `Transport.RoundTrip` 254.4 ns vs 260.9 ns (p=0.28, no significant change); loopback ExecuteStatement 91.5 µs vs 90.6 µs (p=0.80). The branch adds 2 allocations per ExecuteStatement. The driver-test #895 scenario is deliberately not satisfied. Its proxy decodes the full ExecuteStatement body, then kills the connection, and it requires at least 2 ExecuteStatement calls at the proxy. Only re-sending a statement the server already received can meet that, which could run an INSERT or MERGE twice. The kernel and Node.js refuse that retry for the same reason. ### Cross-driver state | Driver | State | Notes | |---|---|---| | Kernel (SEA) | Supported, never-sent only | NonIdempotent retries only `is_connect()`, minus cert and permanent-DNS failures (`src/client/retry.rs:252,261-262` @db23442) | | Node.js (Thrift) | Supported, never-sent only | `ECONNREFUSED`/`ENOTFOUND`/`EAI_AGAIN` only (`UnsentRequestRetryPolicy.ts:6-8`, `ThriftHttpConnection.ts:97` @93a9112) | | JDBC | Not supported | Transport `IOException` maps to status 0 and isn't retried (`DatabricksHttpRetryHandler.java:130-135,251-256` @0cb8c24) | | Python (Thrift) | Supported, includes post-send | urllib3 read retries with POST allowed (`auth/retry.py:132` @5a1a7ec) | | C# ADBC | Supported, includes post-send | Any transport exception, including a mid-transfer `IOException` (`RetryHttpHandler.cs:331-347` @be5ea31) | | Go (Thrift) | Not supported → Supported, never-sent only (this PR) | Follows the kernel/Node.js rule; DNS not-found excluded as in the kernel | | Go (kernel backend) | Supported | Inherits the kernel row | Adversarial review: converged in 2 rounds. A high finding (a dial failure on a redirect hop could replay an already-delivered statement) is fixed with per-attempt redirect tracking and regression tests; a medium (retrying SOCKS target refusals) is out of scope, pushback accepted. Addresses #423 This pull request and its description were written by Isaac.
2 parents 980941b + ec41d56 commit 9301a2b

3 files changed

Lines changed: 504 additions & 1 deletion

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
- **Breaking change:** Upgrade Apache Arrow Go from v12 to v18.8.0 (`github.com/apache/arrow-go/v18`). The public `rows.ArrowBatchIterator` now uses v18 `arrow.RecordBatch` and `*arrow.Schema` types; callers using Arrow batches must migrate their Arrow imports and dependent code. This requires a major driver release (v2.0.0), including a `/v2` driver module path before publication.
66
- Fix CloudFetch reads failing with `row number 0 is not contained in any arrow batch: EOF` when `WithMaxDownloadThreads` / `maxDownloadThreads` is 0 or negative. Such values now log a warning and use the default of 10 (databricks/databricks-sql-go#439).
7+
- **Behavior change:** On the Thrift backend, `ExecuteStatement` is now retried (per `WithRetries`) when the connection to the server or proxy can't be opened, for example connection refused or a connect timeout, because the statement was never sent. Failures after the connection exists are still not retried, so a statement never runs twice. With default retry settings, a refused connection now fails after about 15 seconds instead of immediately (databricks/databricks-sql-go#423).
78
- Honor `WithCloudFetch(false)` / `useCloudFetch=false` on the kernel backend to fetch complete results without cloud-storage downloads. The option overrides conflicting session configuration and remains effective after automatic backend recovery. Servers without paged inline support return a configuration error; the driver does not fall back to storage downloads.
89
- **Behavior change:** After `Close`, a raw `driver.Conn` now fails `Prepare`, `PrepareContext`, `ExecContext` and `QueryContext` immediately with a `driver.ErrBadConn` ("connection is closed") and sends no request. Previously `Prepare` returned a statement and the failure came from the server or kernel at execute. `database/sql` users are unaffected (databricks/databricks-sql-go#521).
910
- Report `statement_type` (`SQL`, as on the kernel backend) and the result delivery format (`INLINE_ARROW`, `EXTERNAL_LINKS` or `COLUMNAR_INLINE`) in Thrift statement telemetry. The format is reported when the driver already has the result metadata (databricks/databricks-sql-go#454).

‎internal/client/client.go‎

Lines changed: 48 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ import (
1717
"regexp"
1818
"strconv"
1919
"strings"
20+
"sync/atomic"
2021
"time"
2122

2223
dbsqlerr "github.com/databricks/databricks-sql-go/errors"
@@ -57,8 +58,23 @@ const (
5758
// outcome surfaced individually (e.g. the telemetry exporter, whose
5859
// circuit breaker is the sole arbiter of when to back off).
5960
SkipTransientRetries
61+
attemptStateKey
6062
)
6163

64+
// attemptState tracks the current HTTP attempt of an ExecuteStatement call.
65+
type attemptState struct {
66+
// redirected is set once the attempt follows a redirect: an earlier hop
67+
// then received the request, so a later dial error doesn't prove it unsent.
68+
redirected atomic.Bool
69+
}
70+
71+
// sentByEarlierHop reports whether the current attempt may already have
72+
// delivered the request. Without attempt tracking it assumes so.
73+
func sentByEarlierHop(ctx context.Context) bool {
74+
s, ok := ctx.Value(attemptStateKey).(*attemptState)
75+
return !ok || s.redirected.Load()
76+
}
77+
6278
// WithSkipTransientRetries returns a context that disables retryablehttp's
6379
// retries for the transient-server-response codes (429 + 503). The caller
6480
// takes responsibility for handling these responses (e.g. via a circuit
@@ -195,6 +211,7 @@ func (tsc *ThriftServiceClient) GetResultSetMetadata(ctx context.Context, req *c
195211
// If RecordResults is true, the results will be marshalled to JSON format and written to ExecuteStatement<index>.json
196212
func (tsc *ThriftServiceClient) ExecuteStatement(ctx context.Context, req *cli_service.TExecuteStatementReq) (*cli_service.TExecuteStatementResp, error) {
197213
ctx = startClientMethod(ctx, clientMethodExecuteStatement)
214+
ctx = context.WithValue(ctx, attemptStateKey, &attemptState{})
198215
var log *logger.DBSQLLogger
199216
log, ctx = LoggerAndContext(ctx, req)
200217
msg, start := log.Track("ExecuteStatement")
@@ -544,6 +561,12 @@ type Transport struct {
544561
}
545562

546563
func (t *Transport) RoundTrip(req *http.Request) (*http.Response, error) {
564+
// http.Client sets Response only on the requests it creates to follow a
565+
// redirect, and each retry attempt starts with the original request.
566+
if s, ok := req.Context().Value(attemptStateKey).(*attemptState); ok {
567+
s.redirected.Store(req.Response != nil)
568+
}
569+
547570
if t.trace {
548571
trace := &httptrace.ClientTrace{
549572
GotConn: func(info httptrace.GotConnInfo) { log.Printf("conn was reused: %t", info.Reused) },
@@ -738,7 +761,9 @@ func RetryPolicy(ctx context.Context, resp *http.Response, err error) (bool, err
738761
_, nonRetryableClientMethod := nonRetryableClientMethods[caller]
739762

740763
if err != nil {
741-
if isRetryableError(err) && !nonRetryableClientMethod {
764+
// ExecuteStatement is not idempotent, so it retries only requests that were never sent.
765+
if isRetryableError(err) && (!nonRetryableClientMethod ||
766+
caller == clientMethodExecuteStatement && isDialError(err) && !sentByEarlierHop(ctx)) {
742767
return true, nil
743768
}
744769

@@ -806,6 +831,28 @@ func isRetryableError(err error) bool {
806831
return true
807832
}
808833

834+
// isDialError reports whether err came from dialing the server or proxy, so
835+
// no request bytes were sent and a non-idempotent request can be retried.
836+
// TLS handshake, read and write failures happen after the dial and never match.
837+
func isDialError(err error) bool {
838+
if errors.Is(err, context.Canceled) {
839+
return false
840+
}
841+
var dnsErr *net.DNSError
842+
if errors.As(err, &dnsErr) && dnsErr.IsNotFound {
843+
return false
844+
}
845+
var opErr *net.OpError
846+
if !errors.As(err, &opErr) {
847+
return false
848+
}
849+
// net/http reports a failed proxy dial as proxyconnect wrapping the dial error.
850+
if opErr.Op == "proxyconnect" && !errors.As(opErr.Err, &opErr) {
851+
return false
852+
}
853+
return opErr.Op == "dial"
854+
}
855+
809856
func backoff(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration {
810857
// honour the Retry-After header
811858
if resp != nil && resp.Header != nil {

0 commit comments

Comments
 (0)