From bcc24c8abdf4991e9673ae327c0b0bbd1951c3af Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Sat, 21 Nov 2026 17:26:27 +0000 Subject: [PATCH] learn: retrospective learnings Signed-off-by: peco-engineer-bot[bot] --- .claude/knowledge/journal/2026-11-21.md | 17 +++++++++++++++++ .claude/knowledge/lessons.md | 15 ++++++++++++--- 2 files changed, 29 insertions(+), 3 deletions(-) create mode 100644 .claude/knowledge/journal/2026-11-21.md diff --git a/.claude/knowledge/journal/2026-11-21.md b/.claude/knowledge/journal/2026-11-21.md new file mode 100644 index 00000000..5b9a0587 --- /dev/null +++ b/.claude/knowledge/journal/2026-11-21.md @@ -0,0 +1,17 @@ +# Learning journal 2026-11-21 + +### 2026-11-21: HTTP client — Custom retry RoundTrippers must drain and close every non-returned response +**Context:** PR #584 replaced go-retryablehttp with an internal retryTransport. The old path leaked connections on failed or retry-exhausted requests, which could exhaust a connector's pool. The fix drains up to 4 KiB and closes the body before each retry, closes the final response before returning the wrapped error, and calls CloseIdleConnections. +**Rule:** In any retrying http.RoundTripper, drain (bounded) and close the body of every response you will not return, including the last one on retry exhaustion, and return a nil response with the error, because net/http discards a response that comes back together with an error. + +### 2026-11-21: Go harness — Detect connection leaks with a body-counting transport wrapper +**Context:** PR #584 proved the leak fix by wrapping the base transport in a RoundTripper that counts responses and open bodies, with a sync.Once-guarded Close. Unit tests (retry exhaustion at -1/0/3 retries) and an E2E with an invalid token (repeated 403s) asserted zero open bodies. +**Rule:** To test HTTP resource leaks, inject a RoundTripper that wraps each response body with a once-guarded close counter, then assert open==0 after error, retry-exhaustion and rejected-auth paths, not just on success. + +### 2026-11-21: Reviewer — Check reimplemented-library review findings against main's real behavior +**Context:** In PR #584 a reviewer asked that an empty buffered body with a positive declared ContentLength be rejected. The author ran the same real httptest on origin/main and on the PR head. Both sent length 0: retryablehttp already used http.NoBody, and net/http ignores ContentLength for NoBody. The finding was withdrawn. +**Rule:** When a PR replaces a dependency with in-house code, run a real httptest on main and on the PR head before acting on behavior-changing review suggestions, and keep the existing behavior unless a regression is shown. + +### 2026-11-21: Arrow results and metadata — Take reported type names and value-decoder selection from different metadata keys +**Context:** In PR #583, GEOMETRY/GEOGRAPHY names now come from Spark:DataType:SqlName, which both Thrift and the kernel populate (e.g. GEOMETRY(0), GEOMETRY(ANY)). Binary-geo value decoding still uses the physical STRUCT type plus databricks.type_name. A reviewer repeatedly flagged a mismatch that only contradictory server metadata could cause, and it was deferred as out of scope. +**Rule:** Report DatabaseTypeName from Spark:DataType:SqlName so both backends agree, keep decoder selection on physical Arrow type plus databricks.type_name, and treat parameterized names (e.g. SRIDs) as a behavior change whose consumers must match the family before "(". diff --git a/.claude/knowledge/lessons.md b/.claude/knowledge/lessons.md index db104f10..8019b244 100644 --- a/.claude/knowledge/lessons.md +++ b/.claude/knowledge/lessons.md @@ -1,6 +1,6 @@ # Lessons - + ## Configuration and authentication - L-0001 (1×, #446): Verify kernel session parameters and proxies separately from Thrift: kernel allowlists configuration keys and supports HTTP(S) proxies, while Thrift may accept additional keys/schemes. @@ -25,6 +25,7 @@ - L-0020 (1×, #556): Check integer duration values against MinInt64/unit and MaxInt64/unit before multiplication; report overflow with the parameter name. - L-0060 [phase=review] (1×, #566): Kernel applies case variants of client-only session-conf keys in sorted byte order, so the canonical lowercase key wins; check pinned session.rs before flagging map-order nondeterminism. - L-0061 (1×, #566): Forwarding a formerly inert option to a backend is a behavior change: label it so in CHANGELOG, explain the effect of existing tuned values, and test set values, not only unset. +- L-0067 (1×, #557): Bound x/oauth2 token fetches with a copied *http.Client with capped Timeout in the context under oauth2.HTTPClient, passing that context to TokenSource; never mutate DefaultClient or the caller's client. AuthStyle auto-detect may double first-fetch time. ## Native boundaries and lifecycle - L-0021 (1×, #450): Never place fabricated Go pointers or cgo.Handle values into C pointer slots. Use NULL with a global sink or a real C allocation. @@ -49,7 +50,9 @@ - L-0038 (1×, #509): HasNext may fetch remotely. Drain local batches and check finished state first; preserve boolean short-circuit order and assert early Close sends no FetchResults RPC. - L-0039 (1×, log:2026-11-12): Duplicate-name support can differ between Arrow schemas, struct/cdata imports and IPC. Test both kernel and Thrift boundaries against the current Arrow version. - L-0040 (1×, #550 #535): New Arrow code uses github.com/apache/arrow-go/v18 RecordBatch APIs; avoid copying legacy arrow/go/v12 imports. -- L-0041 (1×, #563 #567): Derive logical result types from field metadata with ColumnTypeInfoForField, including empty SEA string-backed schemas; test zero-row, NULL-only and populated results on both backends. +- L-0041 (2×, #577 #567 #563): Derive logical result types from field metadata with ColumnTypeInfoForField, including empty SEA string-backed schemas. E2E must assert the wire metadata tag for inline, CloudFetch, zero-row and NULL-only results on both backends. +- L-0065 (1×, #583): Report DatabaseTypeName from Spark:DataType:SqlName so backends agree; select value decoders by physical Arrow type plus databricks.type_name. Parameterized names (e.g. SRIDs) are a behavior change; match the family before "(". +- L-0066 (1×, #574): Shared formatters must spell special values (NaN/Inf) identically as map keys and values on both backends; drop backend-only fallbacks and return the error as the other backend does. ## Verification and dependencies - L-0042 (3×, #458 #510 #567): Compile/vet relevant tagged tests explicitly, including databricks_kernel,e2e where needed, and register E2E names in nightly filters. Default unit tests omit tagged coverage. @@ -60,15 +63,21 @@ - L-0047 (2×, #463 #440 #469): Confirm which archive the build links. Default kernel builds use published bindings; local source builds need explicit go.work/replace routing, and KERNEL_REV alone does not ship native behavior. - L-0048 (1×, #516): Linter fixes must preserve production pointer/value error types; use interface-typed test variables or narrowly justified suppressions. - L-0049 (1×, #516): Fork CI without registry credentials must install tools from prebuilt releases rather than go install; keep fork-behavior comments accurate. -- L-0050 (1×, #525): Allocation/performance changes need AllocsPerRun assertions or benchmarks; output parity alone cannot detect a performance regression. +- L-0050 (2×, #574 #525): Allocation/performance changes need AllocsPerRun assertions or benchmarks; output parity alone cannot detect a regression. Comparative benchmarks must exclude inputs one side rejects so both measure successful work. - L-0051 (2×, #526 #534 #464): Make targeted go get updates and preserve deliberate indirect security pins/hashes and require-block layout. Revalidate suppressions on tagged/untagged graphs and document security versions/advisories. - L-0052 (1×, #527): Fast paths need byte/error parity against the original over exhaustive/random inputs and proof the fast path runs; keep expensive sweeps out of short tests. - L-0053 (1×, #536): Prove no-storage-egress behavior through a workspace-only CONNECT proxy, asserting complete results and zero blocked attempts. - L-0054 (2×, #559 #558 #555): Join test goroutines with bounded cleanup and context cancellation. Synchronize with channels and report callback failures without FailNow/Fatal from foreign goroutines. - L-0055 (1×, #561): Root Go files must remain package dbsql/dbsql_test even behind tags; declare developer tools with the go.mod tool directive. +- L-0070 (1×, #577): In tests using SetMaxOpenConns(1), fully read or explicitly Close each result set and check rows.Err() before the next query; a deferred Close holds the only connection. +- L-0071 [phase=review,followup] (1×, #584): When in-house code replaces a dependency, run a real httptest on main and the PR head before acting on behavior-changing review suggestions; keep existing behavior unless a regression is shown. ## Bot workflow - L-0056 (1×, #442): Use concrete serving-endpoints/model/invocations MODEL_ENDPOINT URLs; anthropic/invocations produces an invalid translated CLI route. - L-0057 (1×, #442): Schedule plus workflow_dispatch workflows should declare numeric knobs as strings and parse them downstream. - L-0058 (2×, #442): Omit unused prompt-file config keys instead of referencing missing files; explicitly unignore and seed any required tracked file under ignored directories. - L-0059 (4×, log:2026-10-21): Reuse completed inspections, report unavailable native/E2E environments promptly, and distinguish skipped coverage or compile failures from assertion-based reproduction. Add runnable unit coverage where possible. + +## HTTP client +- L-0068 (1×, #584): Retrying RoundTrippers must bounded-drain and close every response they do not return, including the last on retry exhaustion, and return a nil response with the error. +- L-0069 (1×, #584): Test connection leaks with a RoundTripper wrapping bodies in once-guarded close counters; assert open==0 after error, retry-exhaustion and rejected-auth paths, not just success.