Repository navigation
Complete native Volume transfers for ExecContext - #560
aakash-saravanan-db wants to merge 38 commits into
Conversation
|
Go integration tests triggered ( |
There was a problem hiding this comment.
Verdict: 1 Low · 1 Nit
Mostly sound, with two low-severity items. The native staging dispatch, cancellation watcher, and defer teardown order look correct. REMOVE not-found=success matches Thrift's 404 handling. The kernel-side behavior depends on the pinned kernel revision, which isn't in this checkout, so I couldn't check it here. Nit: the error strings "Volume transfer: ..." and "Volume operation requires..." start with a capital letter, which is against Go convention (staticcheck ST1005).
Other findings
- ⚪ Nit —
nativeStagingStubhas anerrfield, but no table case ever sets it. The dispatch path that wraps aCompleteStagingfailure inNewDriverError(..., "error performing staging operation", err)is never tested. No test checks that the returned error is non-nil in that case, or that the cause can still be reached witherrors.Is/errors.As(e.g.context.Canceled,*os.PathError). That cause chain is the point of the newKernelError.Unwrapand the ctx-wrapping inCompleteStaging. A case witherr: context.Canceledthat assertserrors.Is(err, context.Canceled)would cover it.
|
Integration test approval reset. New commits were pushed to this PR. Label(s) A maintainer must re-review and re-add a label to preview tests again. (The real gate runs in the merge queue.) Latest commit: 26f142a |
There was a problem hiding this comment.
Verdict: 1 Medium · 2 Low
The dispatch is mostly sound, with one medium lint concern in the new kernel CompleteStaging path. ExecContext now routes kernel Volume operations to native completion, and Thrift keeps its existing row-based path. The rest are low-severity issues: missing session-fatal eviction and thin test coverage.
|
Go integration tests triggered ( |
|
Integration test approval reset. New commits were pushed to this PR. Label(s) A maintainer must re-review and re-add a label to preview tests again. (The real gate runs in the merge queue.) Latest commit: f0c6f41 |
There was a problem hiding this comment.
Verdict: 1 Medium · 1 Nit
Mostly sound. One medium issue: when native staging completion fails with a session-fatal error, the session is not marked dead, unlike every other kernel result path. Kernel-side test coverage for the new paths is also thin. Nit: the new CHANGELOG entry is followed by a stray blank line inside the bullet list.
Other findings
- ⚪ Nit — None of the new kernel-side behaviour has unit coverage. Untested paths:
kernelOp.IsStagingwith a nil exec.- The
CompleteStagingoutcome mapping (outcome != 2returns an error). - Wrapping of the cancelled-context error.
- The new
KernelError.Unwrap/causepath, whereerrors.Is(err, fs.ErrNotExist)should work through a*KernelErrorcarrying anos.PathError.
The only new test (TestConnNativeStagingDispatch) uses a Go stub at the conn layer. The removed TestExecuteRejectsStaging has no replacement, and driver_e2e_test.go has no kernel staging test. Unwrap could be pinned in the non-cgo errors_classify_test.go. kernel_test.go could cover IsStaging with a nil exec and the outcome mapping.
|
Go integration tests triggered ( |
|
Integration test approval reset. New commits were pushed to this PR. Label(s) A maintainer must re-review and re-add a label to preview tests again. (The real gate runs in the merge queue.) Latest commit: 3629843 |
There was a problem hiding this comment.
Verdict: 1 Medium · 2 Low
The PR looks correct overall. There is one medium concern: CompleteStaging skips evicting a dead session when a session-fatal error races context cancellation, which execute() deliberately avoids. There are also two low gaps: no eviction on stream-acquisition failure, and no test for the cancellation path.
There was a problem hiding this comment.
Verdict: 2 Low
The dispatch and cancellation design look sound, with two low-severity concerns: two early-failure paths in CompleteStaging skip session-fatal eviction, and the native path drops two telemetry tags. As the PR description already says, the pinned KERNEL_REV and header point at an unmerged kernel commit and need exports that the published binding modules don't have, so databricks_kernel-tagged builds against published bindings will fail to link until those modules and the module/sum updates ship.
There was a problem hiding this comment.
Verdict: 1 Medium · 1 Low · 1 Nit
Mostly solid, with one medium gap: the session-fatal eviction this PR adds is skipped on the early get_result_stream failure path in CompleteStaging. Also worth noting for release: KERNEL_REV pins an unmerged kernel PR commit, and the new C symbols need a kernel release later than 1.4.0 before this can ship. The cancellation test relies on the exact number of ctx.Err() calls (errCalls == 2), which is fragile (nit).
Other findings
- 🔵 Low — Removing the
isStagingStatementpre-check means staging detection now relies only on the server manifest'sis_volume_operationflag, reported throughkernel_executed_statement_is_volume_operation. If a warehouse or result shape leaves that flag out (or the kernel misses it),IsStagingreturns false, andExecContexton a PUT/GET reports success with 0 rows while no file moves. That is the silent-data-loss case the removed guard existed to block. Consider keeping the SQL-text detector as a fallback: if the text looks like PUT/GET/REMOVE but the operation is not marked, return an error instead of success. Alternatively, record that the server is guaranteed to always set the flag. - ⚪ Nit — The native branch returns before the
telemetryUpdatecallback is wired, so kernel Volume transfers never setTagChunkCount/TagBytesDownloaded. The Thrift staging path does set them. This is a minor telemetry gap compared with Thrift. If the C ABI can't report byte counts, a short comment noting the omission would help.
There was a problem hiding this comment.
Verdict: 1 Low · 2 Nit
Mostly sound, with 3 low concerns: the session-eviction policy is applied inconsistently on the early error paths of CompleteStaging, one error message is misleading for status 0, and the new cancellation watcher has no test. Follow-up outside this PR: per the description, the published 1.4.0 binding archives lack the new kernel_get_last_error_* and volume exports, so a -tags databricks_kernel build against the published bindings will fail to link until a matching bindings release ships.
Other findings
- ⚪ Nit — The
outcome != 2branch handles two different cases: status 0 ("ordinary result": the stream turned out not to be a Volume operation) and status 1 ("input bytes required"). Both get the same error,volume operation requires a local file instruction, and that message only makes sense for status 1. A status-0 result afterIsStagingreturned true points to a kernel or server inconsistency, and this message would send whoever debugs it in the wrong direction. Neither case is tested.staging_kernel_test.goonly covers error returns from completion, andTestConnNativeStagingDispatchstubs outCompleteStaging. Consider using a distinct message for each status (including the numeric status helps) and adding a test for it. - ⚪ Nit — The
*Cancelledsubtests do not exercise the cancellation path this PR adds.cancelOnCompletionErrorContextcancels on the secondErr()call, which comes afterkernel_result_stream_complete_volume_operationhas already returned an error. So the watcher goroutine never callskernel_result_stream_canceller_cancelwhile a transfer is running. These cases only check how the error is wrapped after the fact. They also depend on how many timesCompleteStaginghappens to callctx.Err(), so adding or removing one check would silently change what they test. A test that cancels while the storage request is in flight (for example, a handler that blocks until the ctx is cancelled) would cover the documented promise that "context cancellation" interrupts the transfer.
There was a problem hiding this comment.
Verdict: 1 Low · 1 Nit
Looks good overall, with 2 low concerns: some kernel CompleteStaging setup failures skip the session-eviction policy, and nothing tests that cancellation actually interrupts an in-flight transfer. Follow-up (not caused by this PR): the native path doesn't emit the chunk-count/bytes telemetry tags that the Thrift staging path does.
Other findings
- ⚪ Nit — Nothing tests the result-stream canceller watcher (
case <-ctx.Done(): C.kernel_result_stream_canceller_cancel(canceller)). The two*Cancelledsubtests instaging_kernel_test.gocancel the context from insideErr()only after native completion has already returned. They exercise the error-wrapping branch, not an interrupt of an in-flight transfer. If the watcher were dropped or never fired, a cancelled ExecContext would block until the storage transfer finished and every test would still pass. Consider adding a case where the/objecthandler blocks until the request context is cancelled, then assert thatExecContextreturnscontext.Canceledpromptly.
There was a problem hiding this comment.
Verdict: 1 Low
Looks good overall, with one low concern. Native Volume completion now goes through CompleteStaging. The canceller and deferred cleanup run in the correct LIFO order, and storage failures leave the session valid. When the instruction stream fails to open, CompleteStaging doesn't apply the session-eviction policy that Results applies to the same call. Nit: the VOLUME_STORAGE_* error-code strings are inline literals and could be named constants.
There was a problem hiding this comment.
Verdict: 1 Medium
Looks mostly correct, with 1 medium issue: the new CompleteStaging doesn't evict a dead session when getting the result stream or the canceller fails. Otherwise the dispatch, cancellation wiring, defer ordering, and the storage-vs-instruction eviction split look right. Follow-up: the native path no longer emits the chunk-count and bytes-downloaded telemetry tags that the Thrift staging path records.
There was a problem hiding this comment.
Verdict: 1 Low
The change looks sound, with one low-severity concern: the kernel example still describes staging as rejected on the kernel backend. CompleteStaging cleans up in a safe order (stops the watcher goroutine, then frees the canceller, then closes the stream). It only evicts the session for non-storage failures, and it never returns ErrBadConn. Nits: the cancellation test depends on the exact number of ctx.Err() calls (errCalls==2), so a harmless refactor could break it. The native completion path also skips the chunk-count/bytes telemetry tags that the Thrift staging path records.
Other findings
- 🔵 Low — This PR removes the up-front staging rejection from
Execute, butexamples/kernel/main.gostill says staging is "rejected at execute". It also runs a realPUT '/tmp/local.csv' INTO '/Volumes/main/default/vol/f.csv' OVERWRITEand expectsErrNotSupportedByKernel.
After this change, that statement goes to the server. With no staging context set, it fails with "staging ctx must be provided." instead of the documented rejection. If someone copies the example and adds staging roots, the PUT really runs and overwrites the Volume file. The example and its comment should be updated or the staging block removed, along with the doc.go/CHANGELOG edits.
|
The obsolete staging-rejection example is removed in 6218f29. The example retains backend selection, connection handling and SELECT; it no longer submits the hard-coded PUT. The cancellation fault-injection test's dependence on the |
There was a problem hiding this comment.
Verdict: 1 Low
Looks mostly good, with one low concern. Native completion goes through ExecContext cleanly, and the storage-vs-instruction eviction split is tested against the real native library, including races with cancellation. Possible follow-up: the native staging path no longer records chunk/byte telemetry tags the way the Results-based path did.
There was a problem hiding this comment.
Verdict: 1 Medium
Looks good overall, with one medium gap. On two early setup failures in CompleteStaging, the session-fatal eviction that Results() applies is skipped. Not blocking but worth a follow-up: the native path drops the chunk/bytes telemetry tags that the Thrift staging path sets, and no tagged test covers a successful native completion or the outcome != 2 branch.
There was a problem hiding this comment.
Verdict: 1 Low
Looks good overall. I found one low-severity gap: the early error paths in CompleteStaging skip the session-eviction policy. The dispatch, cancellation teardown order, ErrBadConn avoidance and REMOVE-404 parity with Thrift all check out. A possible follow-up: the native path records no chunk/bytes telemetry tags, while the Thrift staging path does.
There was a problem hiding this comment.
Verdict: 1 Low
Looks good overall, with one low concern: CompleteStaging doesn't apply the fatal-session eviction when it fails to get the result stream or create the canceller. The dispatch in execStagingOperation, the cancellation wrapping and the storage-vs-instruction eviction split look correct. The new error-code values (VOLUME_STORAGE_HTTP_ERROR and the others) are typed in as string literals; named constants would be a nice touch (nit).
There was a problem hiding this comment.
Verdict: 1 Low
Looks good overall, with one low-severity gap. When opening the result stream or its canceller fails inside CompleteStaging, the session is not evicted, unlike in Results. As a nit, the native path no longer adds the chunk-count/bytes telemetry tags that the Thrift staging path adds, which could be follow-up work.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
fbbfbf5 to
72e9816
Compare
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 2 Low
Needs work: 1 high, 1 low. The ExecContext dispatch, the cancellation watcher's lifetime and the eviction classification look correct. The module still pins kernel-bindings v1.4.0, which lacks the new native symbols.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.
| var outcome C.KernelVolumeCompletion | ||
| err := call(func() C.KernelStatusCode { | ||
| // Go Thrift REMOVE treats a missing object as success. | ||
| return C.kernel_result_stream_complete_volume_operation(stream, paths.c, true, &outcome) |
There was a problem hiding this comment.
🟠 High — The new code calls ABI symbols such as kernel_result_stream_complete_volume_operation. KERNEL_REV changed, but go.mod and go.sum still pin all seven bindings modules at v1.4.0, so installed-driver builds may not link. Bump the bindings versions and sums to a release that exports these symbols, then check a clean kernel build.
🔀 Found by both reviewers · confirmed by the judge
There was a problem hiding this comment.
Release gate. The kernel ships as 1.5.0 first; this PR then moves the seven bindings modules in go.mod and go.sum to that release, with the same module paths, before merge. Keeping this thread open until then.
There was a problem hiding this comment.
Still open at 82bf2e4: go.mod:65-71 still pin all seven bindings modules at v1.4.0; this needs the 1.5.0 bump and go.sum before merge.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 1 Low
Looks good, no blocking defects found. The header declares new ABI symbols, but the bindings pins in go.mod/go.sum aren't bumped yet, so kernel-tagged builds won't link until the coordinated release lands (the PR lists this as a merge prerequisite). Nit: the cancelOnCompletionErrorContext test relies on counting Err() calls, which is brittle.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.
Other findings
- 🟠 High — The new ABI calls require a coordinated native release, but installed kernel builds still select the unchanged v1.4.0 bindings rather than the source revision in KERNEL_REV, leaving the required archive compatibility unresolved. Update every platform module and checksum to the compatible release and validate a clean tagged installation without local workspace replacements before merging. (Found by GPT-6.1 Sol · the judge couldn't confirm it)
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 1 Low
Looks good, no blocking defects in the Go changes. Dispatch, watcher join and defer order, storage-error eviction exemption, and the errno-backed PathError chain look correct. Before merge, still needed (as the description says): released bindings pins in go.mod/go.sum for the new ABI symbols, and native-success E2E coverage.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 1 Low
Looks good, no findings. Dispatch, cancellation, session-eviction logic and error chaining are consistent. Before merge: go.mod/go.sum still pin bindings without the new Volume ABI symbols. Tagged builds against published bindings will fail to link until the coordinated release is pinned, which the PR already lists as a prerequisite.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 1 Low
Looks good, with 1 low concern. The dispatch, the shutdown order between the cancel watcher and the canceller, and the session-eviction split all hold up. Still needed before merge, as the PR description says: the new ABI symbols need released bindings pinned in go.mod/go.sum, because KERNEL_REV alone won't link.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 1 Low
Mostly sound. There is one medium item, a merge prerequisite (the bindings pin), and one low test-hygiene issue. Dispatch, cancellation, error-chain preservation and storage-versus-session eviction look correct. As follow-up, the new storage-read-timeout setter is not wired up, so a stalled transfer is bounded only by the caller's context.
| server.CloseClientConnections() | ||
| go connection.Close() | ||
| }() | ||
| var execResult sql.Result |
There was a problem hiding this comment.
🔵 Low — The deferred cleanup runs go connection.Close() without joining it, so it can outlive the subtest and race with db.Close() and server.Close(). Close synchronously after cancelling, or wait on a bounded channel.
🔀 Found by Claude Opus 5.5 · confirmed by the judge
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High
Mostly sound: one medium concern, and it blocks merge. The native dispatch, cancellation watcher join, error-chain Unwrap and storage-vs-session eviction logic look correct and are tested. The kernel bindings pin still has to be updated before merge.
| if e.query_id != nil { | ||
| ke.QueryID = C.GoString(e.query_id) | ||
| } | ||
| ke.ErrorCode = C.GoString(C.kernel_get_last_error_code()) |
There was a problem hiding this comment.
🟠 High — The PR adds cgo calls to new kernel symbols (e.g. kernel_result_stream_complete_volume_operation, kernel_get_last_error_code), but go.mod/go.sum still pin the v1.4.0 per-platform binding archives. Only KERNEL_REV and the Makefile source-build path get the new kernel, so a consumer's databricks_kernel build can fail to link. Bump the bindings pins and sums to a release that exports these symbols, and check a clean install.
🔀 Found by both reviewers · confirmed by the judge
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 1 Nit
Looks good, with one low concern. The dispatch, cancellation join and storage-vs-session eviction logic look correct. Before merge, the published native bindings pin in go.mod/go.sum must include the new ABI symbols; the PR description already lists this as a prerequisite, and KERNEL_REV alone does not change what ships.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.
Other findings
- 🟠 High — The new ABI calls still link against the unchanged v1.4.0 bindings, while the PR states that the compatible native release has not been selected; changing KERNEL_REV does not update archives used by installed consumers. Pin compatible released bindings and checksums for all supported platforms, then validate a clean tagged installation without local workspace overrides before merging. (Found by GPT-6.1 Sol · the judge couldn't confirm it)
- ⚪ Nit — The UTF-8 root check runs after the result stream, canceller and watcher goroutine are created, and it lives in a cgo-only file. This wastes native setup on invalid input and keeps the check out of CGO_ENABLED=0 tests. Move the validation and JSON encoding into an untagged helper and call it first. (Found by Claude Opus 5.5 · confirmed by the judge)
Feature or gap
ExecContext needs to complete native Volume operations using the staging paths in its context.
What this PR addresses
Delegate marked file PUT, GET and REMOVE to the native core, propagate context cancellation and preserve filesystem error chains. Distinguish instruction/session failures from storage failures when deciding whether to evict a connection. Use the typed native completion outcome and synchronize the C header.
Dependencies
Requires databricks/databricks-sql-kernel#465 and a compatible coordinated native release. Module versions, sums and clean installed-driver validation remain merge prerequisites.
Kernel version
The compatible kernel release has not been selected. When it is available, update the native module versions and sums in this PR and validate a clean installation before merge.
KERNEL_REVselects source for local and CI builds only.Tests and validation
Focused dispatch/error checks and three comparator packages pass. The native provenance and cancellation cases run through the tagged kernel suite; compatible package installation and scheduled native-success coverage remain delivery gates.