Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .claude/knowledge/journal/2026-11-21.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Learning journal 2026-11-21

### 2026-11-21: HTTP client — Custom retry RoundTrippers must drain and close every non-returned response
**Context:** PR #584 replaced go-retryablehttp with an internal retryTransport. The old path leaked connections on failed or retry-exhausted requests, which could exhaust a connector's pool. The fix drains up to 4 KiB and closes the body before each retry, closes the final response before returning the wrapped error, and calls CloseIdleConnections.
**Rule:** In any retrying http.RoundTripper, drain (bounded) and close the body of every response you will not return, including the last one on retry exhaustion, and return a nil response with the error, because net/http discards a response that comes back together with an error.

### 2026-11-21: Go harness — Detect connection leaks with a body-counting transport wrapper
**Context:** PR #584 proved the leak fix by wrapping the base transport in a RoundTripper that counts responses and open bodies, with a sync.Once-guarded Close. Unit tests (retry exhaustion at -1/0/3 retries) and an E2E with an invalid token (repeated 403s) asserted zero open bodies.
**Rule:** To test HTTP resource leaks, inject a RoundTripper that wraps each response body with a once-guarded close counter, then assert open==0 after error, retry-exhaustion and rejected-auth paths, not just on success.

### 2026-11-21: Reviewer — Check reimplemented-library review findings against main's real behavior
**Context:** In PR #584 a reviewer asked that an empty buffered body with a positive declared ContentLength be rejected. The author ran the same real httptest on origin/main and on the PR head. Both sent length 0: retryablehttp already used http.NoBody, and net/http ignores ContentLength for NoBody. The finding was withdrawn.
**Rule:** When a PR replaces a dependency with in-house code, run a real httptest on main and on the PR head before acting on behavior-changing review suggestions, and keep the existing behavior unless a regression is shown.

### 2026-11-21: Arrow results and metadata — Take reported type names and value-decoder selection from different metadata keys
**Context:** In PR #583, GEOMETRY/GEOGRAPHY names now come from Spark:DataType:SqlName, which both Thrift and the kernel populate (e.g. GEOMETRY(0), GEOMETRY(ANY)). Binary-geo value decoding still uses the physical STRUCT type plus databricks.type_name. A reviewer repeatedly flagged a mismatch that only contradictory server metadata could cause, and it was deferred as out of scope.
**Rule:** Report DatabaseTypeName from Spark:DataType:SqlName so both backends agree, keep decoder selection on physical Arrow type plus databricks.type_name, and treat parameterized names (e.g. SRIDs) as a behavior change whose consumers must match the family before "(".
15 changes: 12 additions & 3 deletions .claude/knowledge/lessons.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Lessons

<!-- distilled-through: 2026-11-18 -->
<!-- distilled-through: 2026-11-21 -->

## Configuration and authentication
- L-0001 (1×, #446): Verify kernel session parameters and proxies separately from Thrift: kernel allowlists configuration keys and supports HTTP(S) proxies, while Thrift may accept additional keys/schemes.
Expand All @@ -25,6 +25,7 @@
- L-0020 (1×, #556): Check integer duration values against MinInt64/unit and MaxInt64/unit before multiplication; report overflow with the parameter name.
- L-0060 [phase=review] (1×, #566): Kernel applies case variants of client-only session-conf keys in sorted byte order, so the canonical lowercase key wins; check pinned session.rs before flagging map-order nondeterminism.
- L-0061 (1×, #566): Forwarding a formerly inert option to a backend is a behavior change: label it so in CHANGELOG, explain the effect of existing tuned values, and test set values, not only unset.
- L-0067 (1×, #557): Bound x/oauth2 token fetches with a copied *http.Client with capped Timeout in the context under oauth2.HTTPClient, passing that context to TokenSource; never mutate DefaultClient or the caller's client. AuthStyle auto-detect may double first-fetch time.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — L-0066, L-0067 and L-0070, plus the L-0041 (#577) and L-0050 (#574) updates, are distilled from a 2026-11-20 journal that isn't in the checkout, so they have no source entry and the journal is no longer the record they were built from. Restore .claude/knowledge/journal/2026-11-20.md in this PR, or drop those lessons and updates until it lands.

🔀 Found by Claude Opus 5.5 · the judge couldn't confirm it


## Native boundaries and lifecycle
- L-0021 (1×, #450): Never place fabricated Go pointers or cgo.Handle values into C pointer slots. Use NULL with a global sink or a real C allocation.
Expand All @@ -49,7 +50,9 @@
- L-0038 (1×, #509): HasNext may fetch remotely. Drain local batches and check finished state first; preserve boolean short-circuit order and assert early Close sends no FetchResults RPC.
- L-0039 (1×, log:2026-11-12): Duplicate-name support can differ between Arrow schemas, struct/cdata imports and IPC. Test both kernel and Thrift boundaries against the current Arrow version.
- L-0040 (1×, #550 #535): New Arrow code uses github.com/apache/arrow-go/v18 RecordBatch APIs; avoid copying legacy arrow/go/v12 imports.
- L-0041 (1×, #563 #567): Derive logical result types from field metadata with ColumnTypeInfoForField, including empty SEA string-backed schemas; test zero-row, NULL-only and populated results on both backends.
- L-0041 (2×, #577 #567 #563): Derive logical result types from field metadata with ColumnTypeInfoForField, including empty SEA string-backed schemas. E2E must assert the wire metadata tag for inline, CloudFetch, zero-row and NULL-only results on both backends.
- L-0065 (1×, #583): Report DatabaseTypeName from Spark:DataType:SqlName so backends agree; select value decoders by physical Arrow type plus databricks.type_name. Parameterized names (e.g. SRIDs) are a behavior change; match the family before "(".
- L-0066 (1×, #574): Shared formatters must spell special values (NaN/Inf) identically as map keys and values on both backends; drop backend-only fallbacks and return the error as the other backend does.

## Verification and dependencies
- L-0042 (3×, #458 #510 #567): Compile/vet relevant tagged tests explicitly, including databricks_kernel,e2e where needed, and register E2E names in nightly filters. Default unit tests omit tagged coverage.
Expand All @@ -60,15 +63,21 @@
- L-0047 (2×, #463 #440 #469): Confirm which archive the build links. Default kernel builds use published bindings; local source builds need explicit go.work/replace routing, and KERNEL_REV alone does not ship native behavior.
- L-0048 (1×, #516): Linter fixes must preserve production pointer/value error types; use interface-typed test variables or narrowly justified suppressions.
- L-0049 (1×, #516): Fork CI without registry credentials must install tools from prebuilt releases rather than go install; keep fork-behavior comments accurate.
- L-0050 (1×, #525): Allocation/performance changes need AllocsPerRun assertions or benchmarks; output parity alone cannot detect a performance regression.
- L-0050 (2×, #574 #525): Allocation/performance changes need AllocsPerRun assertions or benchmarks; output parity alone cannot detect a regression. Comparative benchmarks must exclude inputs one side rejects so both measure successful work.
- L-0051 (2×, #526 #534 #464): Make targeted go get updates and preserve deliberate indirect security pins/hashes and require-block layout. Revalidate suppressions on tagged/untagged graphs and document security versions/advisories.
- L-0052 (1×, #527): Fast paths need byte/error parity against the original over exhaustive/random inputs and proof the fast path runs; keep expensive sweeps out of short tests.
- L-0053 (1×, #536): Prove no-storage-egress behavior through a workspace-only CONNECT proxy, asserting complete results and zero blocked attempts.
- L-0054 (2×, #559 #558 #555): Join test goroutines with bounded cleanup and context cancellation. Synchronize with channels and report callback failures without FailNow/Fatal from foreign goroutines.
- L-0055 (1×, #561): Root Go files must remain package dbsql/dbsql_test even behind tags; declare developer tools with the go.mod tool directive.
- L-0070 (1×, #577): In tests using SetMaxOpenConns(1), fully read or explicitly Close each result set and check rows.Err() before the next query; a deferred Close holds the only connection.
- L-0071 [phase=review,followup] (1×, #584): When in-house code replaces a dependency, run a real httptest on main and the PR head before acting on behavior-changing review suggestions; keep existing behavior unless a regression is shown.

## Bot workflow
- L-0056 (1×, #442): Use concrete serving-endpoints/model/invocations MODEL_ENDPOINT URLs; anthropic/invocations produces an invalid translated CLI route.
- L-0057 (1×, #442): Schedule plus workflow_dispatch workflows should declare numeric knobs as strings and parse them downstream.
- L-0058 (2×, #442): Omit unused prompt-file config keys instead of referencing missing files; explicitly unignore and seed any required tracked file under ignored directories.
- L-0059 (4×, log:2026-10-21): Reuse completed inspections, report unavailable native/E2E environments promptly, and distinguish skipped coverage or compile failures from assertion-based reproduction. Add runnable unit coverage where possible.

## HTTP client
- L-0068 (1×, #584): Retrying RoundTrippers must bounded-drain and close every response they do not return, including the last on retry exhaustion, and return a nil response with the error.
- L-0069 (1×, #584): Test connection leaks with a RoundTripper wrapping bodies in once-guarded close counters; assert open==0 after error, retry-exhaustion and rejected-auth paths, not just success.
Loading