Skip to content

fix(server-core): rebuild the cached driver when its configuration changes - #11453

Draft
MikeNitsenko wants to merge 1 commit into
masterfrom
mikhail/cub-3599-rebuild-driver-on-config-change
Draft

fix(server-core): rebuild the cached driver when its configuration changes#11453
MikeNitsenko wants to merge 1 commit into
masterfrom
mikhail/cub-3599-rebuild-driver-on-config-change

Conversation

@MikeNitsenko

Copy link
Copy Markdown
Contributor

The stickiness half of CUB-3599. The console-server side fixes what triggers a bad credential; this fixes why a bad credential then survives until redeploy.

The problem

getOrchestratorApi builds its driver factory as a closure over the context of the request that created the orchestrator, and the driver that factory resolves is cached for the life of the process. Nothing ever revisits either.

So when driverFactory derives the connection from the security context — a per-user OAuth token, as our own per-user OAuth recipe instructs — the driver stays pinned to the token it was first built with. The JDBC pool drops to zero connections after ~30s idle, so the next query opens a fresh session with a token that expired an hour ago and gets 403/401 from the warehouse. Only a redeploy clears it.

The fix

Track the latest request context per cached orchestrator, and when it changes, check whether the factory now resolves a different configuration. If it does, build a new driver and release the old one.

The check is layered so deployments that cannot be affected never leave the fast path, and no user-supplied function is called more often than it has to be:

  1. No custom driverFactory, or one returning a constructed driver rather than a config — nothing context-derived to compare. Reuse, exactly as before.
  2. Security context byte-identical to what the cached driver was built from — reuse without calling the factory at all. This is the common case: requestId changes per request, credentials do not.
  3. Context changed — ask the factory. Most ignore it and return an identical config, so reuse, and remember the new context so step 2 short-circuits next time.
  4. Config genuinely changed — rebuild.

Why this is safe

  • Configurations are compared by a truncated SHA-256 of a key-sorted serialisation, never kept verbatim — the values include database passwords and OAuth tokens, and a raw copy would live for the process lifetime and land in any heap dump.
  • Anything that cannot be fingerprinted (a circular structure, a constructed driver) yields null, and every caller reads null as "assume unchanged". Failure degrades to today's resolve-once behaviour, never to churn.
  • The replaced driver is released off the request path. release drains the pool, so queries already running on it finish before its connections close.
  • Step 4 only fires where the deployment is already broken: today that config difference is silently discarded and the first request's credential is used for everyone.
  • This follows the documented contract of contextToOrchestratorId — that it is the cache key for database connections. Two contexts resolving to different connections while sharing an orchestrator id remain a misconfiguration; they were already sharing one user's connection before this change.

resolveDriver is split so a caller that has already invoked the factory (to compare) builds from that same result instead of invoking user code twice. Its public signature and behaviour are unchanged.

Testing

  • test/unit/driver-config-fingerprint.test.ts (9 tests): property-order stability, credential-only changes, undefined vs absent, arrays/dates/bigints, no plaintext in the digest, null on circular input.
  • test/unit/driver-cache-invalidation.test.ts (7 tests): rebuild on rotated credential, release of the replaced driver, reuse on unchanged context (and that the factory is not re-invoked), reuse when the factory ignores context, never rebuilding for a constructed driver, the @pre_agg alias following the rebuild, and no poisoned cache entry after a failed rebuild.
  • Full cubejs-server-core unit suite: 95 passing. The 10 RefreshScheduler failures in my checkout are pre-existing — they reproduce identically on a clean master (Unsupported env variable: "nativeSqlPlannerPreAggregations", a schema-compiler/shared version mismatch) and are unrelated to this diff.

Not verified against a live warehouse; the tests stub driver construction so no database is required.

Also here

The per-user OAuth recipe is updated in the same PR — its context_to_orchestrator_id returned the username alone, which cannot survive a token rotation, and it gated on the credential status flag in a way that fell back to the service account whenever a background refresh had failed. Both samples now derive the credential and the orchestrator id from one shared helper so they cannot drift.

Follow-up, deliberately not here

OrchestratorStorage's LRU has no dispose, so evicted orchestrators are never released. Adding one would close connections under queries still running on an evicted orchestrator, which needs its own thought — filing separately rather than bundling it into a fix that needs to ship.

🤖 Generated with Claude Code

…anges

`getOrchestratorApi` builds its driver factory as a closure over the context of
the request that created the orchestrator, and the driver that factory resolves
is then cached for the life of the process. Nothing ever revisits either. When
`driverFactory` derives the connection from the security context — a per-user
OAuth token, say — the driver stays pinned to the token it was first built
with, and every new connection it opens after that token rotates fails to
authenticate. Only a redeploy clears it.

Track the latest request context per cached orchestrator and, when it changes,
check whether the factory now resolves a different configuration. If it does,
build a new driver and release the old one.

The check is layered so deployments that cannot be affected never leave the
fast path, and no user-supplied function is called more than it has to be:

  1. No custom driverFactory, or one returning a constructed driver rather than
     a config — nothing context-derived to compare, so reuse, exactly as before.
  2. Security context byte-identical to what the cached driver was built from —
     reuse without calling the factory at all. This is the common case;
     requestId changes per request, credentials do not.
  3. Context changed, so ask the factory. Most ignore it and return an identical
     config — reuse, and remember the new context so step 2 short-circuits next
     time.
  4. Config genuinely changed — rebuild.

Configurations are compared by a truncated SHA-256 of a key-sorted
serialisation rather than kept verbatim, since the values include database
passwords and OAuth tokens. Anything that cannot be fingerprinted (a circular
structure, a constructed driver) yields null, which every caller reads as
"assume unchanged" so behaviour degrades to the previous resolve-once
semantics rather than to churn.

The replaced driver is released off the request path; `release` drains the
pool, so queries already running on it finish before its connections close.

This follows the documented contract of `contextToOrchestratorId` — that it is
the cache key for database connections. Two contexts resolving to different
connections while sharing an orchestrator id remain a misconfiguration; they
were already sharing one user's connection before this change.

Also updates the per-user OAuth recipe, whose `context_to_orchestrator_id`
returned the username alone and so could not survive a token rotation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mintlify

mintlify Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
cubed3 🟢 Ready View Preview Aug 2, 2026, 7:00 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@github-actions github-actions Bot added javascript Pull requests that update Javascript code pr:community Contribution from Cube.js community members. labels Aug 2, 2026
@codecov

codecov Bot commented Aug 2, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.75000% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.14%. Comparing base (a433127) to head (d15c69c).
⚠️ Report is 6 commits behind head on master.

Files with missing lines Patch % Lines
packages/cubejs-server-core/src/core/server.ts 94.11% 2 Missing and 1 partial ⚠️
...-server-core/src/core/driver-config-fingerprint.ts 93.10% 1 Missing and 1 partial ⚠️

❗ There is a different number of reports uploaded between BASE (a433127) and HEAD (d15c69c). Click for more details.

HEAD has 1 upload less than BASE
Flag BASE (a433127) HEAD (d15c69c)
cubesql 1 0
Additional details and impacted files
@@             Coverage Diff             @@
##           master   #11453       +/-   ##
===========================================
- Coverage   83.95%   59.14%   -24.81%     
===========================================
  Files         257      224       -33     
  Lines       80887    17965    -62922     
  Branches        0     3657     +3657     
===========================================
- Hits        67908    10626    -57282     
+ Misses      12979     6819     -6160     
- Partials        0      520      +520     
Flag Coverage Δ
cube-backend 59.14% <93.75%> (?)
cubesql ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

javascript Pull requests that update Javascript code pr:community Contribution from Cube.js community members.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant