Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

## Unreleased (v2.0.0 prototype)

- Honor `WithPort` / DSN port on the kernel backend, including the automatic Reyden recovery onto the kernel: a non-default port is folded into the host (`host:port`) instead of being rejected. An IPv6 literal may be bare (`::1`) or bracketed (`[::1]`), and both fold to `[::1]:8443`. A port embedded in the host wins over a differing `WithPort` (a warning is logged), a zero port means the default, and a trailing `/` on the host is ignored. An out-of-range port, a malformed or empty port in the host, or a non-default port with a host that has a path, query, fragment or credentials, is rejected with `ErrInvalidKernelConfig`. A host with a password (`user:pw@h`) is rejected at every port, except that the default port keeps a host with a path, query or fragment unchanged. Error messages show the host with its credentials replaced by `<redacted>`. With OAuth and a non-default port, the kernel uses the same `host:port` for OAuth; Thrift keeps OAuth on the bare host (documented, not guarded).
- Fix OAuth M2M token requests hanging indefinitely when the token endpoint stops responding. Each request now uses a copy of `http.DefaultClient` with a timeout of at most 60s; `WithRetries` repeats failed fetches (databricks/databricks-sql-go#475).
- Render non-finite FLOAT / DOUBLE values inside ARRAY / MAP / STRUCT results as the JSON strings `"NaN"`, `"Infinity"`, and `"-Infinity"` on both backends instead of failing row iteration. SQL NULL remains `null`; top-level floats remain numeric (databricks/databricks-sql-go#573).
- **Behavior change:** Non-finite floating-point MAP keys use the same spellings as values on both backends. On the kernel backend, keys previously rendered as `"+Inf"` and `"-Inf"` now render as `"Infinity"` and `"-Infinity"`; `"NaN"` is unchanged. On Thrift, these keys previously failed row iteration (databricks/databricks-sql-go#573).
Expand Down
2 changes: 1 addition & 1 deletion CONNECTION_PARAMETERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ wire. They are listed under [Kernel client-result confs](#kernel-client-result-c
|---|---|:---:|:---:|---|---|
| *(host)* | `WithServerHostname` | ✅ | ✅ | *(required)* | Workspace hostname. |
| *(path)* | `WithHTTPPath` | ✅ | ✅ | required on Thrift except localhost; required on kernel unless `warehouseId` is set | Warehouse/endpoint HTTP path. On the kernel path, a bare warehouse id can be used instead. |
| *(port)* | `WithPort` | ✅ | ⚠️ | `443` for connector options; required in a DSN | Kernel accepts **443 only** and rejects any other port. |
| *(port)* | `WithPort` | ✅ | ✅ | `443` for connector options; required in a DSN | Kernel folds a non-default port into the host it connects to. A port embedded in the host wins over a differing `WithPort` (a warning is logged and the embedded port is used); an out-of-range port, a malformed port in the host, or a non-default port with a host that has a path, query, fragment or credentials, is rejected at connect. A host with a password (`user:pw@h`) is rejected at every port, except that the default port keeps a host with a path, query or fragment unchanged. Errors show the host with its credentials redacted. An IPv6 literal may be bare (`::1`) or bracketed (`[::1]`). With OAuth the kernel uses the same `host:port` for its OAuth endpoints; Thrift keeps OAuth on the bare host. |
| *(scheme)* | *(via `WithServerHostname`)* | ✅ | ⚠️ | `https` | Kernel accepts `https` only and rejects `http`. A bare `localhost` passed to `WithServerHostname` selects `http`; a scheme-less DSN, including localhost, is prefixed with `https`. |
| `warehouseId` | `WithWarehouseID` | ⚠️ | ✅ | | Bare warehouse id. The kernel normally prefers it over the HTTP path; a canonical path carrying `?o=` takes precedence so workspace-org routing is preserved. The Thrift backend **silently ignores** it. |
| `catalog` | `WithInitialNamespace` | ✅ | ✅ | | Initial catalog. Kernel applies it post-connect via `USE CATALOG`. |
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,7 @@ session parameter on both backends.
|---|---|---|---|---|
| *(host)* | `WithServerHostname` | Both | *(required)* | Workspace hostname. |
| *(path)* | `WithHTTPPath` | Both | *(required)* | Warehouse/endpoint HTTP path. |
| *(port)* | `WithPort` | Thrift only | `443` | Kernel connects on **443 only** and rejects any other port. |
| *(port)* | `WithPort` | Both | `443` | Kernel folds a non-default port into the host it connects to. A port embedded in the host wins over a differing `WithPort` (a warning is logged); an out-of-range port, a malformed port in the host, or a non-default port with a host that has a path, query, fragment or credentials, is rejected at connect. A host with a password (`user:pw@h`) is rejected at every port, except that the default port keeps a host with a path, query or fragment unchanged. Errors show the host with its credentials redacted. An IPv6 literal may be bare (`::1`) or bracketed (`[::1]`). With OAuth the kernel uses the same `host:port` for its OAuth endpoints; Thrift keeps OAuth on the bare host. |
| `warehouseId` | `WithWarehouseID` | SEA only | | Bare warehouse id; the kernel routes by it (preferred over the HTTP path). **The Thrift backend ignores it.** |
| `catalog` | `WithInitialNamespace` | Both | | Initial catalog. Kernel applies it post-connect via `USE CATALOG`. |
| `schema` | `WithInitialNamespace` | Both | | Initial schema. Kernel applies it post-connect via `USE SCHEMA`. |
Expand Down
23 changes: 23 additions & 0 deletions connector_reyden_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,29 @@ func TestReydenReactiveRecovery(t *testing.T) {
assert.True(t, warehouse_cache.IsKnownReyden(tt.host, tt.warehouseID),
"warehouse should be marked Reyden after the rejection")
})

t.Run("recovery onto the kernel keeps a custom WithPort", func(t *testing.T) {
defer warehouse_cache.ClearCache()

tt := NewTestReydenFallback()
var kernelHost string
conn := tt.makeConnector(
func(ctx context.Context, cfg *config.Config, client *http.Client) (backend.Backend, error) {
return &fakeThriftBackend{openSessionErr: dbsqlerrint.NewReydenThriftUnsupportedError(
"Lakehouse/RT is not supported for Thrift protocol")}, nil
},
func(ctx context.Context, cfg *config.Config) (backend.Backend, error) {
require.NoError(t, validateKernelHostPort(cfg.Host, cfg.Port))
kernelHost = buildKernelConfig(cfg, kernel.Auth{Mode: kernel.AuthPAT, Token: "test-token"}).Host
return &fakeKernelBackend{}, nil
},
)
WithPort(8443)(conn.cfg)

_, _, err := conn.openSessionWithReydenFallback(context.Background())
require.NoError(t, err)
assert.Equal(t, "reyden.example.com:8443", kernelHost)
})
}

func TestReydenPreCheck(t *testing.T) {
Expand Down
32 changes: 32 additions & 0 deletions internal/config/thrift_endpoint_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
package config

import "testing"

// TestThriftEndpointUnchanged pins the Thrift endpoint for the port inputs the kernel custom-port
// change decides on. Thrift is not changed by that work, so these strings are what it builds today.
// An embedded host port plus the default 443 yields an unparseable URL on Thrift; that is the baseline.
func TestThriftEndpointUnchanged(t *testing.T) {
cases := []struct {
name string
host string
port int
want string
}{
{"bare host, default port", "h.io", 443, "https://h.io:443/sql/1.0/warehouses/x"},
{"bare host, WithPort", "h.io", 8443, "https://h.io:8443/sql/1.0/warehouses/x"},
{"embedded port, default port", "h.io:8443", 443, "https://h.io:8443:443/sql/1.0/warehouses/x"},
{"embedded port, WithPort", "h.io:8443", 8443, "https://h.io:8443:8443/sql/1.0/warehouses/x"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
c := &Config{Protocol: "https", Host: tc.host, Port: tc.port, HTTPPath: "/sql/1.0/warehouses/x"}
got, err := c.ToEndpointURL()
if err != nil {
t.Fatalf("unexpected err %v", err)
}
if got != tc.want {
t.Errorf("ToEndpointURL() = %q, want %q", got, tc.want)
}
})
}
}
154 changes: 145 additions & 9 deletions kernel_config.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"context"
"errors"
"fmt"
"net"
"net/url"
"strconv"
"strings"
Expand All @@ -15,6 +16,7 @@ import (
"github.com/databricks/databricks-sql-go/internal/backend/kernel"
"github.com/databricks/databricks-sql-go/internal/client"
"github.com/databricks/databricks-sql-go/internal/config"
"github.com/databricks/databricks-sql-go/logger"
"github.com/databricks/databricks-sql-go/telemetry"
)

Expand Down Expand Up @@ -73,18 +75,15 @@ func validateKernelConfigContext(ctx context.Context, cfg *config.Config) (kerne
// folds its server conf (spark.sql.thriftserver.metadata.metricview.enabled=true)
// into SessionConf backend-neutrally, so the kernel path sends the identical conf
// the Thrift path does. No per-backend handling needed here.
// Port / Protocol: the kernel C ABI takes only a bare host and connects over
// https:443; it has no port or scheme setter. The Thrift path honors a custom
// port/scheme via ToEndpointURL, so a non-default value here would be silently
// ignored on the kernel path (it would just hit 443) — reject it instead, per
// the "nothing silently ignored" contract. Defaults (https/443) are fine.
// Protocol: the kernel always connects over https, so a non-https value would be ignored.
if cfg.Protocol != "" && cfg.Protocol != "https" {
return kernel.Auth{}, fmt.Errorf("databricks: a non-https protocol is %w "+
"(it connects over https); use the default (Thrift) backend", dbsqlerr.ErrNotSupportedByKernel)
}
if cfg.Port != 0 && cfg.Port != 443 {
return kernel.Auth{}, fmt.Errorf("databricks: a non-default port (WithPort) is %w "+
"(it connects on 443); omit it or use the default (Thrift) backend", dbsqlerr.ErrNotSupportedByKernel)
// Port: the kernel folds host:port (see kernelHostWithPort); reject a bad port or host here. An
// embedded host port wins over a differing WithPort, with a warning rather than an error.
if err := validateKernelHostPort(cfg.Host, cfg.Port); err != nil {
return kernel.Auth{}, err
}
// Transport (WithTransport, a custom http.RoundTripper carrying a custom CA
// bundle / mTLS / proxy): the kernel uses its own Rust HTTP stack below the C
Expand Down Expand Up @@ -155,7 +154,7 @@ func validateKernelConfigContext(ctx context.Context, cfg *config.Config) (kerne
// with newKernelBackend's kernel.Config assembly.
func buildKernelConfig(cfg *config.Config, kauth kernel.Auth) kernel.Config {
kc := kernel.Config{
Host: cfg.Host,
Host: kernelHostWithPort(cfg.Host, cfg.Port),
Comment thread
peco-review-bot[bot] marked this conversation as resolved.
Resolved
HTTPPath: cfg.HTTPPath,
WarehouseID: cfg.WarehouseID,
Auth: kauth,
Expand Down Expand Up @@ -239,6 +238,143 @@ func buildKernelConfig(cfg *config.Config, kauth kernel.Auth) kernel.Config {
return kc
}

// kernelHost is cfg.Host split by net/url: the parts the kernel port logic needs.
type kernelHost struct {
raw string // cfg.Host with a trailing "/" trimmed and a bare IPv6 literal bracketed; the kernel gets it unchanged when no port is folded
redacted string // raw with any user[:password] replaced by "<redacted>"; the only form of the host that goes into an error
name string // host name without IPv6 brackets (url.URL.Hostname)
port string // digits after the authority's ":"; "" when there is none
hasPort bool // the authority has a ":", even with no digits after it ("h:")
hasPath bool // a path, query or fragment follows the host, or a bare "#" (url.URL drops an empty fragment)
userinfo bool // a user@ prefix is present
password bool // the user@ prefix carries a password
}

// splitAuthority splits a host string at its first "/", "?" or "#": the authority, and the rest.
func splitAuthority(raw string) (authority, rest string) {
if i := strings.IndexAny(raw, "/?#"); i >= 0 {
return raw[:i], raw[i:]
}
return raw, ""
}

// bracketBareIPv6 wraps a bare IPv6 literal (after any user@ prefix) in brackets. net/url parses
// "https://[::1]" the same way on every Go version, but "https://::1" differs: a newer net/url rejects it,
// and an older one reads the last group as a port. So a DSN host (ParseDSN strips the brackets) and a
// bare WithServerHostname("::1") both reach the kernel path in the bracketed form.
func bracketBareIPv6(raw string) string {
authority, rest := splitAuthority(raw)
start := strings.LastIndex(authority, "@") + 1
if host := authority[start:]; strings.Contains(host, ":") && net.ParseIP(host) != nil {
return authority[:start] + "[" + host + "]" + rest
}
return raw
}

// redactUserinfo replaces the user[:password] of a host with "<redacted>", keeping the host itself.
func redactUserinfo(raw string) string {
authority, rest := splitAuthority(raw)
at := strings.LastIndex(authority, "@")
if at < 0 {
return raw
}
return "<redacted>@" + authority[at+1:] + rest
}

// parseKernelHost parses cfg.Host (trailing "/" trimmed, bare IPv6 bracketed) as the authority of an https
// URL, which is the form the kernel connects to. The fields are set only when the error is nil. The error
// never carries the host or url.Parse's text: url.Parse quotes the whole URL, and an invalid escape quotes
// the password itself, so a host with user info gets a generic reason.
func parseKernelHost(host string) (kernelHost, error) {
raw := bracketBareIPv6(strings.TrimRight(host, "/"))
h := kernelHost{raw: raw, redacted: redactUserinfo(raw)}
u, err := url.Parse("https://" + raw)
if err != nil {
authority, _ := splitAuthority(raw)
reason := "it does not parse as a host"
var ue *url.Error
if !strings.Contains(authority, "@") && errors.As(err, &ue) {
reason = ue.Err.Error()
}
return h, errors.New(reason)
}
h.name = u.Hostname()
h.port = u.Port()
h.hasPort = h.port != "" || strings.HasSuffix(u.Host, ":")
Comment thread
cathleeny marked this conversation as resolved.
Resolved
h.hasPath = u.Path != "" || u.RawQuery != "" || u.ForceQuery || u.Fragment != "" || strings.Contains(h.raw, "#")
h.userinfo = u.User != nil
_, h.password = u.User.Password()
return h, nil
}

// validateKernelHostPort rejects an out-of-range WithPort, a host that net/url cannot parse, and a
// non-default WithPort with a host that has a path, query, fragment or credentials. A password is
// rejected at every port, except that the default port keeps a host with a path, query or fragment
// unchanged (checked before the password, as before). It also rejects unbalanced brackets and an empty, out-of-range or malformed
// port embedded in the host. An IPv6 literal may be bare or bracketed. WithDefaults fills 443 when no port
// is set. A port embedded in the host wins over a differing WithPort, with a warning rather than an error.
// Every error names the host with its credentials redacted.
func validateKernelHostPort(host string, port int) error {
if port < 0 || port > 65535 {
return fmt.Errorf("databricks: WithPort %d is out of range (want 1..65535, or 0 for the default): %w", port, dbsqlerr.ErrInvalidKernelConfig)
}
h, err := parseKernelHost(host)
if err != nil {
return fmt.Errorf("databricks: host %q is not a valid host (want a host name, an IPv4 address or an IPv6 literal): %v: %w", h.redacted, err, dbsqlerr.ErrInvalidKernelConfig)
}
nonDefault := port != 0 && port != 443
// A path, query or fragment is checked before the embedded-port parse. With the default port the
// host goes to the kernel unchanged, as before (the kernel appends its API paths after it). A
// non-default port cannot be folded into such a host, so reject it.
if h.hasPath {
if nonDefault {
return fmt.Errorf("databricks: host %q has a path, query or fragment, so WithPort %d cannot be applied: %w",
h.redacted, port, dbsqlerr.ErrInvalidKernelConfig)
}
return nil
}
// Credentials are kept only at the default port: a non-default port would have to be folded in without them.
if h.userinfo && nonDefault {
return fmt.Errorf("databricks: host %q carries credentials, so WithPort %d cannot be applied: %w",
h.redacted, port, dbsqlerr.ErrInvalidKernelConfig)
}
if h.password {
Comment thread
cathleeny marked this conversation as resolved.
Resolved
return fmt.Errorf("databricks: host %q carries a password: %w", h.redacted, dbsqlerr.ErrInvalidKernelConfig)
}
// url.Parse accepts a stray "]" in a plain name, so brackets are checked on the parsed host name
// (IPv6 brackets are already removed from it).
if strings.ContainsAny(h.name, "[]") {
return fmt.Errorf("databricks: host %q has unbalanced IPv6 brackets: %w", h.redacted, dbsqlerr.ErrInvalidKernelConfig)
}
if !h.hasPort {
return nil // no embedded port
}
embedded, err := strconv.Atoi(h.port)
if err != nil || embedded < 1 || embedded > 65535 {
return fmt.Errorf("databricks: host %q carries an invalid port %q (want 1..65535): %w", h.redacted, h.port, dbsqlerr.ErrInvalidKernelConfig)
}
// The embedded port is used, as on the kernel path before WithPort was honored. An explicit 443 is
// the default and is not warned about.
if nonDefault && embedded != port {
logger.Warn().Msgf("databricks: the host carries port %d, which differs from WithPort %d; connecting to port %d from the host",
embedded, port, embedded)
}
return nil
}

// kernelHostWithPort returns the Host the kernel connects to. The kernel has no port setter but keeps a
// `host:port` authority, so a non-default port is folded in with net.JoinHostPort, which brackets an IPv6
// literal. A trailing "/" is trimmed first, and a bare IPv6 literal is bracketed. 0 and 443 add nothing. A host that already carries a port, a
// path, query or fragment, or credentials is returned unchanged; validateKernelHostPort rejects the last
// two with a non-default port before the kernel is built.
func kernelHostWithPort(host string, port int) string {
h, err := parseKernelHost(host)
if err != nil || port == 0 || port == 443 || h.hasPort || h.hasPath || h.userinfo {
return h.raw
}
return net.JoinHostPort(h.name, strconv.Itoa(port))
}

func kernelTelemetryConfig(cfg *config.Config) *kernel.TelemetryConfig {
var enabled *bool
if val, isSet := cfg.EnableTelemetry.Get(); isSet {
Expand Down
Loading
Loading