Repository navigation
Commit 9301a2b
authored
fix(client): retry ExecuteStatement when the connection was never opened (#555)
On the Thrift backend, `RetryPolicy` never retried a transport error on
`ExecuteStatement`, not even a refused connection, because the RPC isn't
idempotent. Every other RPC retries those errors, as do the kernel and
Node.js on their ExecuteStatement paths.
Now `ExecuteStatement` retries, using the existing `WithRetries` budget
and backoff, only when two things hold. First, the error is a dial-stage
`*net.OpError`: connection refused, unreachable, connect timeout,
transient DNS, or the dial to a proxy inside `proxyconnect`. Second, the
attempt didn't follow a redirect. A dial error means the failing hop
never had a connection. A followed redirect means an earlier hop already
delivered the statement, so the transport wrapper records redirect hops
per attempt and those attempts aren't retried. DNS not-found,
cancellation and the existing certificate carve-outs are excluded. TLS
handshake, read and write failures still fail after one attempt, so a
statement can't run twice. A stale pooled connection where nothing was
written already recovers inside `net/http`, since Thrift requests carry
`GetBody`.
Verification:
- **Table-driven classifier tests:** cover 30 error shapes.
- **Fault tests:** run the real Thrift client, retryablehttp,
`http.Transport` and a real Thrift processor over local sockets.
- Refused for the first two dials gives 3 dials and exactly 1 execution.
Persistent refused, direct or through a proxy, gives 5 attempts.
- These cases each give exactly 1 attempt: reset after the server
executed the statement (1 execution), reset mid-body (16 MiB statement),
reset after headers, reset after the first bytes over HTTP and HTTPS,
untrusted certificate, caller cancellation during the dial, and a client
timeout during the dial.
- A 303, 307 or 308 redirect to a refused host after the server executed
gives exactly 1 execution (5 before the redirect fix).
- A redirect to a reachable server still succeeds. A refused first hop
after an earlier redirected attempt is still retried.
- The tests are stable across 100 runs, half of them under `-race`.
- **Live (Thrift):** the four `TestE2E*` tests pass. A temporary probe
refused two dials at ExecuteStatement time against the warehouse. On the
branch the query returned its row after 3 dials (287 ms); on `main` it
failed with `request error after 1 attempt(s) … connection refused`.
- **Checks:** `make test` (1352 tests) and `make lint` pass, and
kernel-tagged `go vet` passes.
- **Performance (benchstat, `main` vs branch, 8 alternating fresh
processes):** `Transport.RoundTrip` 254.4 ns vs 260.9 ns (p=0.28, no
significant change); loopback ExecuteStatement 91.5 µs vs 90.6 µs
(p=0.80). The branch adds 2 allocations per ExecuteStatement.
The driver-test #895 scenario is deliberately not satisfied. Its proxy
decodes the full ExecuteStatement body, then kills the connection, and
it requires at least 2 ExecuteStatement calls at the proxy. Only
re-sending a statement the server already received can meet that, which
could run an INSERT or MERGE twice. The kernel and Node.js refuse that
retry for the same reason.
### Cross-driver state
| Driver | State | Notes |
|---|---|---|
| Kernel (SEA) | Supported, never-sent only | NonIdempotent retries only
`is_connect()`, minus cert and permanent-DNS failures
(`src/client/retry.rs:252,261-262` @db23442) |
| Node.js (Thrift) | Supported, never-sent only |
`ECONNREFUSED`/`ENOTFOUND`/`EAI_AGAIN` only
(`UnsentRequestRetryPolicy.ts:6-8`, `ThriftHttpConnection.ts:97`
@93a9112) |
| JDBC | Not supported | Transport `IOException` maps to status 0 and
isn't retried (`DatabricksHttpRetryHandler.java:130-135,251-256`
@0cb8c24) |
| Python (Thrift) | Supported, includes post-send | urllib3 read retries
with POST allowed (`auth/retry.py:132` @5a1a7ec) |
| C# ADBC | Supported, includes post-send | Any transport exception,
including a mid-transfer `IOException` (`RetryHttpHandler.cs:331-347`
@be5ea31) |
| Go (Thrift) | Not supported → Supported, never-sent only (this PR) |
Follows the kernel/Node.js rule; DNS not-found excluded as in the kernel
|
| Go (kernel backend) | Supported | Inherits the kernel row |
Adversarial review: converged in 2 rounds. A high finding (a dial
failure on a redirect hop could replay an already-delivered statement)
is fixed with per-attempt redirect tracking and regression tests; a
medium (retrying SOCKS target refusals) is out of scope, pushback
accepted.
Addresses #423
This pull request and its description were written by Isaac.3 files changed
Lines changed: 504 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
| |||
57 | 58 | | |
58 | 59 | | |
59 | 60 | | |
| 61 | + | |
60 | 62 | | |
61 | 63 | | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
62 | 78 | | |
63 | 79 | | |
64 | 80 | | |
| |||
195 | 211 | | |
196 | 212 | | |
197 | 213 | | |
| 214 | + | |
198 | 215 | | |
199 | 216 | | |
200 | 217 | | |
| |||
544 | 561 | | |
545 | 562 | | |
546 | 563 | | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
547 | 570 | | |
548 | 571 | | |
549 | 572 | | |
| |||
738 | 761 | | |
739 | 762 | | |
740 | 763 | | |
741 | | - | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
742 | 767 | | |
743 | 768 | | |
744 | 769 | | |
| |||
806 | 831 | | |
807 | 832 | | |
808 | 833 | | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
809 | 856 | | |
810 | 857 | | |
811 | 858 | | |
| |||
0 commit comments