Skip to content

Commit 9983f08

Browse files
fix(kernel): honor the CloudFetch connection option (#536)
With the kernel backend, `WithCloudFetch(false)` and the DSN option `useCloudFetch=false` are ignored, so large results still require direct access to cloud storage. This PR forwards the existing option when opening the native session, allowing complete results to be fetched through the SQL connection. The connector uses the kernel's existing `cloudfetch_enabled` client setting. The kernel handles option precedence and result delivery. This also preserves the disable option when existing automatic recovery selects the kernel backend. **Dependency:** Requires [databricks-sql-kernel #436](databricks/databricks-sql-kernel#436), which fixes large-result delivery with CloudFetch disabled. `KERNEL_REV` pins that implementation for source builds. Before merging, publish native Go bindings containing the fix and update the platform modules in `go.mod` and `go.sum` to require them. The release version has not yet been assigned. **Validation:** The default and kernel-tagged test suites pass. A probe linked against the locally built native library returns all 250,000 rows and 32 MB of payload with the expected hash. Disabled delivery succeeds through a proxy that blocks storage access; enabled delivery attempts storage access. Query errors, reuse, and close also pass. **Performance follow-up:** Five measured runs per mode, after warmup, give medians of 2.658 s for Thrift and 3.241 s for kernel (+21.9%). The 10% performance criterion remains unmet. [Kernel issue #441](databricks/databricks-sql-kernel#441) tracks result-transfer costs and evaluation of response compression separately.
2 parents 95d97b3 + b08ba0c commit 9983f08

11 files changed

Lines changed: 282 additions & 13 deletions

‎CHANGELOG.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
11
# Release History
22

3+
## Unreleased
4+
- Honor `WithCloudFetch(false)` / `useCloudFetch=false` on the kernel backend to fetch complete results without cloud-storage downloads. The option overrides conflicting session configuration and remains effective after automatic backend recovery. Servers without paged inline support return a configuration error; the driver does not fall back to storage downloads.
5+
36
## v1.19.0 (2026-11-04)
47
- Upgrade the SQL kernel bindings to v1.4.0.
58
- Upgrade go-oidc to v3.21.0 (go-jose v4.1.6) and x/oauth2 to v0.36.0 to address go-jose/v3 security findings (databricks/databricks-sql-go#526).

‎CONNECTION_PARAMETERS.md‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -113,11 +113,11 @@ Notes:
113113

114114
- Boolean-valued keys should use the exact strings `"true"` / `"false"` — the kernel does
115115
not pre-validate values and forwards them as-is.
116-
- **`CAN_CLOUD_DOWNLOAD` is not a valid SEA session conf**: it may be accepted at
117-
CreateSession and rejected by the first statement with `CONFIG_NOT_AVAILABLE`.
118-
To disable Cloud Fetch on the kernel path, set the client-result conf
119-
`cloudfetch_enabled=false`; `WithCloudFetch(false)` is a Thrift option and is inert
120-
on the kernel path.
116+
- `WithCloudFetch(false)` fetches complete results through the SQL connection
117+
without cloud-storage downloads. On the kernel backend, this option takes
118+
precedence over conflicting session parameters. Servers without paged inline
119+
support return a configuration error at session creation or statement execution;
120+
the driver does not fall back to storage downloads.
121121

122122
### Kernel client-result confs
123123

@@ -129,7 +129,7 @@ Thrift options where available.
129129

130130
| DSN / session-conf key | Dedicated connector option | Thrift | Kernel | Default | Notes |
131131
|---|---|:---:|:---:|---|---|
132-
| `cloudfetch_enabled` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `true` | Kernel Cloud Fetch toggle; `false` forces inline results. On Thrift use `WithCloudFetch`. |
132+
| `cloudfetch_enabled` | `WithCloudFetch` | ⚠️ | ✅ | `true` | Set `false` to fetch complete results without cloud-storage downloads. Use `WithCloudFetch` for the standard option on either backend. |
133133
| `cloudfetch_link_prefetch_window` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `10` | Number of presigned Cloud Fetch links prefetched ahead of the consumer. |
134134
| `cloudfetch_max_chunks_in_memory` | `WithKernelMaxChunksInMemory` | ⚠️ | ✅ | `16` | Maximum decompressed Cloud Fetch chunks held in memory. Values above 256 are clamped with a warning, not rejected. |
135135
| `inline_max_chunks_in_memory` | *(none; use `WithSessionParams`)* | ⚠️ | ✅ | `4` | Inline-Arrow prefetch window; `1` disables prefetching. |
@@ -174,7 +174,7 @@ an internal setting) and non-Arrow values also retain legacy session-location pa
174174

175175
| DSN parameter | Connector option | Thrift | Kernel | Default | Notes |
176176
|---|---|:---:|:---:|---|---|
177-
| `useCloudFetch` | `WithCloudFetch` | ✅ | ⚠️ | `true` | Enable Cloud Fetch on Thrift. This option is inert on the kernel path; use the `cloudfetch_enabled` client-result conf there. |
177+
| `useCloudFetch` | `WithCloudFetch` | ✅ | ✅ | `true` | Set false to fetch complete results through the SQL connection without direct cloud-storage access. |
178178
| `maxDownloadThreads` | `WithMaxDownloadThreads` | ✅ | ⚠️ | `10` | Concurrent download goroutines (Thrift). Inert on the kernel path. |
179179
| | `WithKernelMaxChunksInMemory(n)` | ❌ | ✅ | kernel default (16) | Bounds decompressed Cloud Fetch chunks held in memory — trades large-result throughput for peak memory. |
180180

‎KERNEL_REV‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
6f4b6d607a61bdde61dcda2cf82bc8e80fd32cb9
1+
0b5d0edfed32c0745d5abc2468fc3070e2eb2a17

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -327,7 +327,7 @@ parallel via cloud storage
327327

328328
| DSN parameter | Connector option | Protocol | Default | Description |
329329
|---|---|---|---|---|
330-
| `useCloudFetch` | `WithCloudFetch` | Thrift only (inert on kernel) | `true` | Enable Cloud Fetch. On the kernel path Cloud Fetch is always managed internally, so the flag is inert. |
330+
| `useCloudFetch` | `WithCloudFetch` | Both | `true` | Enable Cloud Fetch. Set to false to fetch results through the SQL connection without direct cloud-storage access. |
331331
| `maxDownloadThreads` | `WithMaxDownloadThreads` | Thrift only (inert on kernel) | `10` | Concurrent download goroutines (Thrift). Inert on the kernel path. |
332332
| | `WithKernelMaxChunksInMemory(n)` | SEA only | kernel default (16) | Bounds decompressed Cloud Fetch chunks held in memory — trades large-result throughput for peak memory. |
333333

‎connector_reyden_test.go‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import (
88

99
dbsqlerr "github.com/databricks/databricks-sql-go/errors"
1010
"github.com/databricks/databricks-sql-go/internal/backend"
11+
"github.com/databricks/databricks-sql-go/internal/backend/kernel"
1112
"github.com/databricks/databricks-sql-go/internal/backend/thrift"
1213
"github.com/databricks/databricks-sql-go/internal/cli_service"
1314
"github.com/databricks/databricks-sql-go/internal/client"
@@ -193,10 +194,15 @@ func TestReydenReactiveRecovery(t *testing.T) {
193194
return fakeThrift, nil
194195
},
195196
func(ctx context.Context, cfg *config.Config) (backend.Backend, error) {
197+
require.False(t, cfg.UseCloudFetch)
198+
kc := buildKernelConfig(cfg, kernel.Auth{Mode: kernel.AuthPAT, Token: "test-token"})
199+
require.NotNil(t, kc.CloudFetchEnabled)
200+
require.False(t, *kc.CloudFetchEnabled)
196201
return kernelBackend, nil
197202
},
198203
)
199204

205+
conn.cfg.UseCloudFetch = false
200206
be, _, err := conn.openSessionWithReydenFallback(context.Background())
201207
require.NoError(t, err)
202208

‎internal/backend/kernel/backend.go‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@ import (
6767
"context"
6868
"errors"
6969
"fmt"
70+
"strconv"
7071
"strings"
7172
"sync/atomic"
7273
"time"
@@ -297,12 +298,29 @@ func (k *KernelBackend) OpenSession(ctx context.Context) error {
297298
}
298299
}
299300

301+
if k.cfg.CloudFetchEnabled != nil {
302+
key := newCStr("cloudfetch_enabled")
303+
value := newCStr(strconv.FormatBool(*k.cfg.CloudFetchEnabled))
304+
errSet := call(func() C.KernelStatusCode {
305+
return C.kernel_session_config_set_client_conf(cfg, key.c, value.c)
306+
})
307+
key.free()
308+
value.free()
309+
if errSet != nil {
310+
return fmt.Errorf("kernel: set_client_conf[cloudfetch_enabled]: %w", toConnError(errSet))
311+
}
312+
}
313+
300314
// Log the resolved retry policy + CloudFetch chunk cap before connecting, so if a
301315
// customer reports a hung connect or a large-result OOM, on-call can see from the
302316
// debug log what was actually applied (these are otherwise silent — forwarded to
303317
// the kernel with no observable trace). Kept at Debug and cheap to format.
304-
klogCtx(ctx, "OpenSession resolved: retry=%s cloudfetchMaxChunksInMemory=%s",
305-
describeRetry(k.cfg.Retry), k.cfg.SessionConf[kernelMaxChunksInMemoryConfKey])
318+
cloudFetchOption := "unset"
319+
if k.cfg.CloudFetchEnabled != nil {
320+
cloudFetchOption = strconv.FormatBool(*k.cfg.CloudFetchEnabled)
321+
}
322+
klogCtx(ctx, "OpenSession resolved: retry=%s cloudfetchMaxChunksInMemory=%s cloudFetchOption=%s",
323+
describeRetry(k.cfg.Retry), k.cfg.SessionConf[kernelMaxChunksInMemoryConfKey], cloudFetchOption)
306324

307325
// kernel_session_open takes ownership of cfg here. Its documented C-ABI
308326
// contract (databricks_kernel.h: "CONSUMES config on both success and failure

‎internal/backend/kernel/config.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,9 @@ type Config struct {
4444
// Thrift backend forwards (STATEMENT_TIMEOUT, QUERY_TAGS, TIMEZONE, …).
4545
SessionConf map[string]string
4646

47+
// Nil preserves the kernel default and any raw session configuration.
48+
CloudFetchEnabled *bool
49+
4750
// TLSSkipVerify accepts any server cert (maps the driver's
4851
// WithSkipTLSHostVerify / TLSConfig.InsecureSkipVerify). crypto/tls's
4952
// InsecureSkipVerify disables both chain validation and the hostname check,

‎kernel_cloudfetch_e2e_test.go‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
//go:build cgo && databricks_kernel
2+
3+
package dbsql
4+
5+
import (
6+
"context"
7+
"crypto/sha256"
8+
"crypto/sha512"
9+
"database/sql"
10+
"fmt"
11+
"io"
12+
"net"
13+
"net/http"
14+
"net/http/httptest"
15+
"strconv"
16+
"strings"
17+
"sync/atomic"
18+
"testing"
19+
"time"
20+
21+
"github.com/stretchr/testify/require"
22+
)
23+
24+
func TestE2EKernelCloudFetchDisabledLargeResults(t *testing.T) {
25+
if testing.Short() {
26+
t.Skip("skipping large inline result in -short mode")
27+
}
28+
host, httpPath, token := pecoTestingCreds(t)
29+
_, workspaceHost := parseHostName(host)
30+
workspaceAuthority := net.JoinHostPort(workspaceHost, "443")
31+
var workspaceConnections, blockedConnections atomic.Int64
32+
proxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
33+
if r.Method != http.MethodConnect || !strings.EqualFold(r.Host, workspaceAuthority) {
34+
blockedConnections.Add(1)
35+
http.Error(w, "Only the SQL workspace is reachable", http.StatusForbidden)
36+
return
37+
}
38+
upstream, err := net.DialTimeout("tcp", r.Host, 10*time.Second)
39+
if err != nil {
40+
http.Error(w, "Workspace connection failed", http.StatusBadGateway)
41+
return
42+
}
43+
defer upstream.Close() //nolint:errcheck
44+
hijacker, ok := w.(http.Hijacker)
45+
if !ok {
46+
http.Error(w, "Proxy tunneling is unavailable", http.StatusInternalServerError)
47+
return
48+
}
49+
client, buffered, err := hijacker.Hijack()
50+
if err != nil {
51+
return
52+
}
53+
defer client.Close() //nolint:errcheck
54+
workspaceConnections.Add(1)
55+
_, _ = buffered.WriteString("HTTP/1.1 200 Connection Established\r\n\r\n")
56+
if err := buffered.Flush(); err != nil {
57+
return
58+
}
59+
go func() { _, _ = io.Copy(upstream, buffered) }()
60+
_, _ = io.Copy(client, upstream)
61+
}))
62+
defer proxy.Close()
63+
connector, err := NewConnector(
64+
WithServerHostname(host), WithHTTPPath(httpPath), WithAccessToken(token),
65+
WithUseKernel(true), WithCloudFetch(false),
66+
WithKernelProxy(KernelProxy{URL: proxy.URL}),
67+
WithRetries(-1, time.Millisecond, time.Millisecond),
68+
)
69+
require.NoError(t, err)
70+
db := sql.OpenDB(connector)
71+
defer db.Close() //nolint:errcheck
72+
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
73+
defer cancel()
74+
rows, err := db.QueryContext(ctx, "SELECT id, sha2(CAST(id AS STRING), 512) AS payload FROM range(250000) ORDER BY id")
75+
require.NoError(t, err)
76+
defer rows.Close() //nolint:errcheck
77+
var count int64
78+
var payloadBytes int
79+
digest := sha256.New()
80+
for rows.Next() {
81+
var id int64
82+
var payload string
83+
require.NoError(t, rows.Scan(&id, &payload))
84+
require.Equal(t, count, id)
85+
require.Equal(t, fmt.Sprintf("%x", sha512.Sum512([]byte(strconv.FormatInt(id, 10)))), payload)
86+
_, _ = fmt.Fprintf(digest, "%d\t%s\n", id, payload)
87+
payloadBytes += len(payload)
88+
count++
89+
}
90+
require.NoError(t, rows.Err())
91+
require.Positive(t, workspaceConnections.Load(), "SQL traffic must use the restricted proxy")
92+
require.Zero(t, blockedConnections.Load(), "no non-workspace connections are allowed")
93+
require.Equal(t, int64(250000), count)
94+
require.Equal(t, 32000000, payloadBytes)
95+
require.Equal(t, "a25ada4c6c476b53e97eb6e8a945e9ce4c14cc432c7ad672dc6fb607d660c86a", fmt.Sprintf("%x", digest.Sum(nil)))
96+
}

‎kernel_cloudfetch_test.go‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
//go:build cgo && databricks_kernel
2+
3+
package dbsql
4+
5+
import (
6+
"context"
7+
"database/sql"
8+
"encoding/json"
9+
"fmt"
10+
"io"
11+
"net/http"
12+
"net/http/httptest"
13+
"strings"
14+
"sync/atomic"
15+
"testing"
16+
"time"
17+
18+
dbsqlerr "github.com/databricks/databricks-sql-go/errors"
19+
"github.com/databricks/databricks-sql-go/internal/backend"
20+
"github.com/databricks/databricks-sql-go/internal/cli_service"
21+
"github.com/databricks/databricks-sql-go/internal/client"
22+
"github.com/databricks/databricks-sql-go/internal/config"
23+
"github.com/databricks/databricks-sql-go/internal/warehouse_cache"
24+
"github.com/stretchr/testify/require"
25+
)
26+
27+
func TestKernelCloudFetchDisabledPreservesUnsupportedServerError(t *testing.T) {
28+
for _, automaticRecovery := range []bool{false, true} {
29+
for _, rejectSession := range []bool{false, true} {
30+
t.Run(fmt.Sprintf("recovery=%t/session_rejected=%t", automaticRecovery, rejectSession), func(t *testing.T) {
31+
testKernelCloudFetchDisabledUnsupportedServer(t, automaticRecovery, rejectSession)
32+
})
33+
}
34+
}
35+
}
36+
37+
func testKernelCloudFetchDisabledUnsupportedServer(t *testing.T, automaticRecovery, rejectSession bool) {
38+
t.Helper()
39+
defer warehouse_cache.ClearCache()
40+
var sessions atomic.Int64
41+
var executions atomic.Int64
42+
server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
43+
w.Header().Set("Content-Type", "application/json")
44+
switch {
45+
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/sessions"):
46+
sessions.Add(1)
47+
var body struct {
48+
SessionConfs map[string]string `json:"session_confs"`
49+
}
50+
if err := json.NewDecoder(r.Body).Decode(&body); err != nil || body.SessionConfs["can_cloud_download"] != "false" {
51+
http.Error(w, "missing disabled download capability", http.StatusBadRequest)
52+
return
53+
}
54+
for key := range body.SessionConfs {
55+
if strings.EqualFold(key, "cloudfetch_enabled") || (strings.EqualFold(key, "can_cloud_download") && key != "can_cloud_download") {
56+
http.Error(w, "conflicting download parameter", http.StatusBadRequest)
57+
return
58+
}
59+
}
60+
if rejectSession {
61+
w.WriteHeader(http.StatusBadRequest)
62+
_, _ = io.WriteString(w, `{"error_code":"CONFIG_NOT_AVAILABLE","message":"Configuration can_cloud_download is not available. SQLSTATE: 42K0I"}`)
63+
return
64+
}
65+
_, _ = io.WriteString(w, `{"session_id":"test-session"}`)
66+
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/statements"):
67+
executions.Add(1)
68+
_, _ = io.WriteString(w, `{"statement_id":"unsupported-inline","status":{"state":"FAILED","sql_state":"42K0I","error":{"error_code":"CONFIG_NOT_AVAILABLE","message":"Configuration can_cloud_download is not available"}}}`)
69+
default:
70+
_, _ = io.WriteString(w, `{}`)
71+
}
72+
}))
73+
defer server.Close()
74+
dbConnector, err := NewConnector(
75+
WithServerHostname(server.URL), WithHTTPPath("/sql/1.0/warehouses/test"),
76+
WithAccessToken("test-token"), WithUseKernel(true), WithCloudFetch(false),
77+
WithSessionParams(map[string]string{"CAN_CLOUD_DOWNLOAD": "true"}),
78+
WithSkipTLSHostVerify(), WithRetries(-1, time.Millisecond, time.Millisecond),
79+
)
80+
require.NoError(t, err)
81+
var thriftAttempts atomic.Int64
82+
if automaticRecovery {
83+
c := dbConnector.(*connector)
84+
c.cfg.UseKernel = false
85+
c.thriftBackendFactory = func(context.Context, *config.Config, *http.Client) (backend.Backend, error) {
86+
thriftAttempts.Add(1)
87+
sqlState := "KP001"
88+
return &fakeThriftBackend{openSessionErr: client.CheckOpenSessionStatus(&cli_service.TOpenSessionResp{
89+
Status: &cli_service.TStatus{StatusCode: cli_service.TStatusCode_ERROR_STATUS, SqlState: &sqlState},
90+
})}, nil
91+
}
92+
}
93+
db := sql.OpenDB(dbConnector)
94+
defer db.Close() //nolint:errcheck
95+
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
96+
defer cancel()
97+
connection, err := db.Conn(ctx)
98+
if rejectSession {
99+
require.Error(t, err)
100+
require.Contains(t, err.Error(), "CONFIG_NOT_AVAILABLE")
101+
require.Contains(t, err.Error(), "42K0I")
102+
require.Equal(t, int64(0), executions.Load())
103+
} else {
104+
require.NoError(t, err)
105+
defer connection.Close() //nolint:errcheck
106+
_, err = connection.ExecContext(ctx, "SELECT 1")
107+
require.Error(t, err)
108+
var executionError dbsqlerr.DBExecutionError
109+
require.ErrorAs(t, err, &executionError)
110+
require.Equal(t, "42K0I", executionError.SqlState())
111+
require.Contains(t, err.Error(), "CONFIG_NOT_AVAILABLE")
112+
require.Equal(t, int64(1), executions.Load(), "unsupported delivery must not fall back")
113+
}
114+
require.Equal(t, int64(1), sessions.Load(), "unsupported delivery must not reopen the session")
115+
if automaticRecovery {
116+
require.Equal(t, int64(1), thriftAttempts.Load(), "backend recovery must not repeat")
117+
}
118+
}

‎kernel_config.go‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -180,6 +180,10 @@ func buildKernelConfig(cfg *config.Config, kauth kernel.Auth) kernel.Config {
180180
// An explicit true below opts into native geospatial values.
181181
EnableGeoSpatialSupport: false,
182182
}
183+
if !cfg.UseCloudFetch {
184+
enabled := false
185+
kc.CloudFetchEnabled = &enabled
186+
}
183187
if cfg.ClientQueryTimeout != nil {
184188
timeout := *cfg.ClientQueryTimeout
185189
kc.ClientQueryTimeout = &timeout

0 commit comments

Comments
 (0)