feat(registry): configurable driver registries via TOML config by zeroshade · Pull Request #360 · columnar-tech/dbc

zeroshade · 2026-04-17T16:16:47Z

Summary

Adds support for configuring driver registries via TOML config files, allowing users to add custom registries at both the global (~/.config/columnar/dbc/config.toml) and project (dbc.toml) level.

What's New

Global registry config (~/.config/columnar/dbc/config.toml): loaded on every command via ConfigureRegistries() called in main()
Project registry config (dbc.toml): [[registries]] section respected by sync and add commands
Priority order: project → global → built-in defaults (first-match-wins, URL-deduplicated)
replace_defaults = true: opt-in flag to suppress built-in registries entirely
DBC_BASE_URL env var: still overrides everything (unchanged behavior)
Backward compatible: existing dbc.toml files without [[registries]] work unchanged

Config Format

Global (~/.config/columnar/dbc/config.toml):

[[registries]]
url = "https://my-registry.example.com"
name = "my-registry"

# replace_defaults = true  # omit built-in registries

Project (dbc.toml):

[[registries]]
url = "https://project-registry.example.com"
name = "project"

[drivers]
[drivers.my-driver]
version = ">=1.0.0"

Implementation Notes

URL validation requires http/https scheme and non-empty host
defaultRegistries is snapshotted eagerly in init() (after drivers.go init runs); env-sensitivity documented in comments
add.go decodes dbc.toml once (single file open) for both registry config and driver list
remove intentionally omits registry wiring — it never queries a registry

amoeba

I did a quick code review over the new pieces and left some comments. Even if we don't have an immediate use case for this, I think having it in dbc is useful.

Some other more general thoughts:

Would we want CLI integration for this? I think it would be useful for dbc to be able to do all the work to show/add/remove registries from the project and user config.
I didn't see any tests for how dbc install will work with this. Presumably dbc install will respect the new fields in the user config.
Do we need to define a default priority order for registries to handle name conflicts? We have a lot of the same concerns as conda does with channels. I'm wishing for more tests in this PR for the interaction between the default, user, and project registries around name conflicts.

Introduce RegistryEntry, GlobalConfig, LoadGlobalConfig, WithGlobalConfig, and WithProjectRegistries. NewClient now merges project, global, and default registries (project wins, then global, then defaults) with deduplication by scheme+host+path. WithBaseURL still overrides all other registry sources. Validation (empty URL, missing host, non-http scheme, replace_defaults=true with no entries) runs at NewClient time so malformed configuration surfaces immediately rather than at first request.

Extend DriversList with omitempty Registries and ReplaceDefaults fields so projects can pin or override driver registries inline in dbc.toml. Existing dbc.toml files without those keys continue to decode unchanged. applyProjectRegistries rebuilds the process-wide dbc client with the merged registry list; callers (add, sync) invoke it after decoding the list.

Load ~/.config/dbc/config.toml once at startup and thread it into the default client via WithGlobalConfig. Project commands (add, sync) rebuild the client with WithProjectRegistries after parsing dbc.toml so driver lookups use the merged registry list. DBC_BASE_URL continues to short circuit all registry merging. Tests cover project-registry wiring, invalid project URLs, and backwards-compatibility with dbc.toml files that omit the new fields.

Two issues from roborev review: - DriversList.ReplaceDefaults is now *bool so a project can explicitly set replace_defaults = false to override a global config's replace_defaults = true (previously collapsed into unset vs false). - NewClient now validates GlobalConfig.Registries URLs with the same path used for project registries, and rejects any configuration that produces an empty registry list (e.g. global replace_defaults = true with zero entries), so malformed config surfaces at NewClient time instead of at first request with silent zero-registry lookups. - LoadGlobalConfig applies the same no-entries check when decoding config.toml so CLI users see the error at startup.

…ries The existing TestAdd/TestSync project-registry tests route through testBaseModel, whose getDriverRegistry stub bypasses dbcClient entirely. Add direct coverage that swaps in a lookup routed to the real dbcClient against an httptest server, proving applyProjectRegistries genuinely rewires driver resolution to the merged registry list and that: - a project [[registries]] entry routes Search() to that server; - an explicit project replace_defaults=false restores built-in defaults even when the global config declared replace_defaults=true; - DBC_BASE_URL continues to short-circuit project registry overrides.

LoadGlobalConfig rejecting global replace_defaults=true with zero entries made the CLI diverge from NewClient: the CLI dropped the global config with only a warning, silently re-enabling the built-in defaults even when the project dbc.toml supplied the registries the user actually wanted. Remove that premature check; the post-merge empty-list guard in NewClient already catches the genuinely broken config (global replace_defaults=true with no entries AND no project entries), while letting the valid combination through. Adds both a library test and a CLI wiring test for the project-supplies-entries case.

Eager initDBCClient() in main() failed startup for the valid 'global replace_defaults=true + project supplies registries' case — NewClient was called before the project dbc.toml could merge in. Remove the eager call and rely on lazy init via getDriverRegistry / initDBCClient, so project commands get a chance to rebuild the client via applyProjectRegistries first. Non-project commands (search, info, install) still fail fast on the broken case (global replace_defaults=true with no entries and no project override) because NewClient rejects the empty merge — now surfaced at first use rather than startup. auth.go's requestDeviceCode now ensures init before touching dbcClient.HTTPClient(). dbcClientOnce is a pointer so tests can simulate fresh startup. Adds TestAddCmdEndToEndThroughRealClient which runs AddCmd.GetModel() against an httptest registry — proves the production getDriverRegistry closure routes through the rebuilt dbcClient. Adds TestStartupDeferred* tests for the main() ordering invariants. Also replaces 'switch { case err == ... }' with 'switch err' in main().

…in GetDriverList Two findings from roborev review of the branch: MEDIUM: OAuth device-code login ran through initDBCClient just to get an HTTPClient. A misconfigured global registry (e.g. replace_defaults=true with no entries) broke 'dbc auth login' — the very recovery path the user would run to fix the config. Add authHTTPClient() that builds a bare HTTP client without touching registry validation, and route device-code calls through it. LOW: GetDriverList decoded the new [[registries]] / replace_defaults fields but never called applyProjectRegistries, so library consumers using that helper silently ignored project registry overrides. Call it immediately after decoding. Regression tests pin both behaviors: - TestAuthHTTPClientDoesNotRequireRegistryConfig - TestGetDriverListHonorsProjectRegistries

Three findings from review of the branch: MEDIUM (1632): GetDriverList mutated the process-wide dbcClient via applyProjectRegistries, which leaked registry state across calls — a later invocation on a dbc.toml without [[registries]] would silently reuse the previous call's client. Switch GetDriverList to build a per-call dbc.Client scoped to the decoded file, so each call sees a fresh registry merge. Test exercises two sequential calls against two different servers to prove no leakage. MEDIUM (1630): NewClient's pre-merge guard rejected project replace_defaults=true with no project entries, even when the global config supplied registries. Remove the pre-merge guard; the post-merge empty-list check is the correct gate and lets 'drop defaults, keep globals' through. Test updated and new test pins the global-entries+project-replace_defaults case. LOW (1629): startup tests didn't exercise the real main() loading path. Extract loadStartupRegistryConfig() and add TestStartupEndToEndGlobalReplaceDefaultsWithProjectEntries that writes a real config.toml + dbc.toml, runs the full startup sequence, and drives AddCmd.GetModel() against an httptest server.

Addresses review 1633: MEDIUM: extract parseStartupArgs() and have main() use it, so the same config-load + argv-parse helper runs under test. The startup regression test now asserts at each step that dbcClient remains nil — if anyone reintroduces eager initDBCClient() between config load and subcommand dispatch the test fails at whichever step broke. LOW: update WithProjectRegistries doc comment to match the actual semantics: an error only when the MERGED registry list is empty. The old text claimed replace_defaults=true with no entries is always rejected, which contradicts the post-merge-only validation.

Addresses review 1634: the previous parseStartupArgs extraction only covered the parser bit. Anyone reintroducing an eager initDBCClient() between p.Parse() and GetModel() would still not have failed the test. Extract a single runStartup() that covers the full path main() runs: config load, argv parse, subcommand selection, and GetModel() for model commands. main() now calls runStartup() and switches on the returned startupKind for its exit cases. Tests drive runStartup() end-to-end, asserting dbcClient remains nil throughout. No functional change to main()'s behavior — same exit codes, same help and version paths, same subcommand dispatch.

Addresses review 1635: after the runStartup refactor, main() always called runStartup(configDir, ...) even when internal.GetUserConfigPath failed, leaving configDir = "". runStartup then forwarded that to LoadGlobalConfig(""), which reads ./config.toml from the current working directory. That changed registry resolution based on cwd — a regression against the pre-refactor behavior that simply skipped the load on this error path. Guard runStartup to skip the load when configDir is empty. Document the sentinel in the helper's doc comment. Add TestRunStartupSkipsLoadWhenConfigDirEmpty that plants a hostile ./config.toml and asserts runStartup leaves globalRegistryConfig nil.

Addresses review 1636: runStartup("", ...) skipped the load but did not clear an already-primed globalRegistryConfig, so a second in-process call would silently reuse the previous invocation's global registries. Set globalRegistryConfig = nil explicitly in the empty-configDir branch and add TestRunStartupClearsStaleGlobalConfigWhenConfigDirEmpty, which primes the global before calling runStartup("") and asserts the state is cleared.

zeroshade · 2026-05-20T18:09:19Z

Would we want CLI integration for this? I think it would be useful for dbc to be able to do all the work to show/add/remove registries from the project and user config.

Give the size of this PR, we can add CLI integration as a follow-up PR.

I didn't see any tests for how dbc install will work with this. Presumably dbc install will respect the new fields in the user config.

Good point, i'll add tests for this

Do we need to define a default priority order for registries to handle name conflicts? We have a lot of the same concerns as conda does with channels. I'm wishing for more tests in this PR for the interaction between the default, user, and project registries around name conflicts.

dbc search would show all the driver matches along with their [registry] name tags in the output. The priority order would be:

Project (dbc.toml)
Global (~/.config/columnar/dbc/config.toml)
built-in defaults

Having replace_defaults = true at either the Project or Global config level would drop the defaults, but doesn't change the priority that registries at the project level would win over the global config level.

That said, there's no tests currently for that behavior so I'll add some tests for it.

…erge The merge of main brought in PR #374 which threaded context.Context through Client request methods. Two call sites added by this branch were missed by the merge: - cmd/dbc/driver_list.go:155 in GetDriverList — now passes context.Background() (matches the pattern used by drivers.GetDriverList and cmd/dbc/main.getDriverRegistry). - cmd/dbc/registry_wiring_test.go test stubs that mimic the production getDriverRegistry closure — match the production context.Background() call so the stubs faithfully reproduce the closure they replace. Build & Integrate (ubuntu/macos/windows), snapshot, and the per-platform build matrix were all failing on the same compile error.

Adds end-to-end tests pinning that 'dbc install' honors the new configurable registry surface. install is user-scope (it never reads dbc.toml), so the global config plus built-in defaults are its only configurable registry sources — these tests drive InstallCmd.GetModel() through the real dbcClient.Search path and assert the merged registry list and the resolved package URL. - TestInstallCmdHonorsGlobalConfigRegistry: a [[registries]] entry in the global config routes install's driver lookup through that registry, and the resolved PkgInfo's Driver.Registry / package URL come from it. - TestInstallCmdGlobalReplaceDefaultsLimitsToConfiguredRegistry: replace_defaults=true in the global config drops the built-in defaults from install's view; the merged client holds exactly one registry. - TestInstallCmdRegistryPriorityFirstDeclaredWins: when two registries publish a driver with the same path, the first-declared wins — Client.Search returns both copies, findDriver picks the first match, and the resolved package URL comes from the first registry's index. - TestInstallCmdDBCBaseURLOverridesGlobalConfig: DBC_BASE_URL takes precedence over the global config; install queries only the override and the global-config registry is never hit. Each test stubs downloadPkg with a sentinel error so the install ends right after the registry lookup — capturedPkg records what install would have downloaded, exposing both the resolving registry and the package URL for assertions. Adds runInstallModelToCompletion driver helper because install's Init() returns tea.Batch (spinner + lookup), which the existing runTeaCmdToCompletion (used by add tests, single tea.Cmd Init) cannot dispatch.

…efault Pins behavior at the boundary where the project, global (user), and built-in default registry tiers interact. Covers both URL conflicts (merge-level dedup) and driver-name conflicts (Search-level priority). Library tests (registry_config_test.go): - Extend TestMergeRegistries with cross-tier URL collision cases: - project URL collides with default URL — project wins, default URL dedup'd, project's Name kept; - global URL collides with default URL — global wins; - all three tiers share a URL — project wins, others dedup'd; - project URL matches second default — only that one default URL is dropped, the rest of the merge is preserved. - TestSearchPriorityAcrossRegistryTiers — Client.Search returns drivers in registry-priority order across three httptest servers acting as project/global/default tiers. Confirms first-match-wins in cmd/dbc.findDriver and Client.Install resolves cross-tier name conflicts to the highest-priority tier. Constructs the Client directly so the test isn't sensitive to whether the real built-in default URLs are reachable from CI. CLI tests (cmd/dbc/registry_wiring_test.go): - TestApplyProjectRegistriesPreservesAllThreeTiersInOrder — when no replace_defaults is set, dbcClient.Registries() exposes all three tiers in [project, global, defaults...] order. Asserts the tail is exactly the built-in defaults (project/global URLs are not duplicated into it). - TestApplyProjectRegistriesProjectReplaceDefaultsKeepsGlobal — pins the asymmetric semantics of project replace_defaults=true: it drops only the built-in defaults, NOT the global tier. A project opting out of defaults still inherits the user's global registries. - TestProjectRegistryShadowsGlobalOnDriverNameConflict — full CLI wiring (applyProjectRegistries -> dbcClient.Search -> findDriver): when project and global both publish a driver with the same path, project wins and the resolved package URL comes from the project tier's index. replace_defaults=true is set on the project so the built-in defaults stay out of the merge — the all-three-tiers case is covered by the ordering test above and by the library TestSearchPriorityAcrossRegistryTiers.

zeroshade · 2026-05-20T19:53:40Z

@amoeba I addressed the comments, updated this, and added the tests that were mentioned. Let me know what you think

Resolves conflict in cmd/dbc/main.go: ports the initVTProcessing() call added on main (#383) into the new main() that calls runStartup(), instead of the pre-refactor main() body that no longer exists on this branch.

zeroshade · 2026-05-21T17:12:41Z

the failing snapshot CI is an upstream snapd problem

amoeba · 2026-05-26T20:21:29Z

I pushed a tiny safeguard in 7cb0e41 since we now have user-injectable registry labels. Feel free to tweak.

I think the last thing I'm not sure I like is that dbc search isn't aware of custom registries defined in dbc.toml which I think would lead to confusion. If a user has a custom registry set in dbc.toml,

$ cat dbc.toml
replace_defaults = true

[[registries]]
url = 'http://localhost:8080'
name = 'local-fake'

and they try to dbc add it,

$ go run ./cmd/dbc add f
Error: driver `f` not found in driver registry index; try: `dbc search` to list available drivers
exit status 1

we suggest 'dbc search' but dbc search won't list drivers in the local-fake repo because it's only been set in dbc.toml and not in the global config. I also note wonder if replace_defaults in dbc.toml should effect dbc search and not just add/sync.

Is there any downside to simplifying the logic so the behavior is more consistent between the project-level and global dbc config?

zeroshade · 2026-05-27T19:53:47Z

Can't think of a downside offhand, I agree that the behavior should be more consistent. I'll fix that

Previously only `dbc add` and `dbc sync` applied the [[registries]] and replace_defaults configured in a project's dbc.toml; the read-only discovery commands (search, info, docs) queried only the global config and built-in defaults. This was confusing: when a driver isn't found, `dbc add` suggests running `dbc search`, but search couldn't see a driver served only by a project-configured registry, and replace_defaults in dbc.toml had no effect on search. search, info, and docs now load ./dbc.toml (when present) and apply its registries before querying, via a shared applyProjectRegistriesFromCWD helper. A missing dbc.toml is a no-op so these commands still work outside a project; a malformed one is a hard error naming the file. dbc install intentionally remains user-scope and does not read a project dbc.toml, since it installs into a user- or system-level config rather than the project.

applyProjectRegistriesFromCWD opened and decoded ./dbc.toml before newDBCClient could apply the DBC_BASE_URL short-circuit, so a malformed project dbc.toml made dbc search/info/docs fail even when the user had set DBC_BASE_URL to bypass registry configuration -- blocking the documented escape hatch in exactly the scenario where registry config may be broken. Return early (no-op) when DBC_BASE_URL is set, before touching dbc.toml. Adds a regression test covering DBC_BASE_URL plus a malformed dbc.toml.

zeroshade · 2026-06-10T20:43:14Z

@amoeba good call — addressed in 59a3514 (+ a follow-up in 2171cdf).

dbc search, dbc info, and dbc docs now load ./dbc.toml and apply its [[registries]] + replace_defaults before querying, so they resolve drivers against the same registry set that dbc add/dbc sync use in the same project. Your example now works: with a project dbc.toml setting replace_defaults = true and a local-fake registry, dbc search lists that registry's drivers, and the dbc add "not found → try dbc search" suggestion is coherent. replace_defaults in dbc.toml affects these commands too now.

A missing dbc.toml is a no-op (these commands still work outside a project), and a malformed one is a hard error that names the file.

On scope: I deliberately left dbc install reading only the global + built-in default registries — it installs into a user/system config level rather than the project, so making it silently cwd-sensitive felt like a surprising, larger change than this. Happy to revisit if you'd prefer install to be project-aware too.

One nuance worth flagging: DBC_BASE_URL still overrides everything and short-circuits dbc.toml entirely, so a broken project file can't block that escape hatch (2171cdf).

Docs updated in docs/concepts/driver_registry.md and docs/reference/driver_list.md.

amoeba

I just checked the final thing I noted was addressed. Can we back out the docs changes before merge and do the docs updates in a separate pr?

Move the driver-registry documentation to a separate PR (#393) so the docs can be reviewed and merged independently, per review feedback.

zeroshade · 2026-06-12T18:25:54Z

Backed out the docs changes here (docs/concepts/driver_registry.md and docs/reference/driver_list.md) in 6012dbd and moved them to a separate docs PR: #393. This PR now contains only the implementation + tests.

zeroshade temporarily deployed to snapshot April 17, 2026 16:16 — with GitHub Actions Inactive

zeroshade requested a review from amoeba April 17, 2026 16:17

zeroshade force-pushed the feat/configurable-registries branch from 5a8bf36 to 5ac1ddd Compare April 17, 2026 16:19

zeroshade temporarily deployed to snapshot April 17, 2026 16:19 — with GitHub Actions Inactive

zeroshade requested review from eitsupi, esadek and ianmcook April 17, 2026 18:34

zeroshade force-pushed the feat/configurable-registries branch 2 times, most recently from 0813c82 to 79e6f5c Compare April 22, 2026 21:33

zeroshade temporarily deployed to snapshot April 22, 2026 21:33 — with GitHub Actions Inactive

zeroshade force-pushed the feat/configurable-registries branch from 79e6f5c to 40e5ad2 Compare April 22, 2026 21:39

zeroshade temporarily deployed to snapshot April 22, 2026 21:39 — with GitHub Actions Inactive

zeroshade requested a review from kentkwu April 26, 2026 18:21

amoeba reviewed Apr 27, 2026

View reviewed changes

zeroshade temporarily deployed to snapshot April 28, 2026 17:51 — with GitHub Actions Inactive

zeroshade added 3 commits May 1, 2026 13:53

zeroshade force-pushed the feat/configurable-registries branch from a41a2ad to 5f8660a Compare May 1, 2026 17:54

zeroshade had a problem deploying to snapshot May 1, 2026 17:54 — with GitHub Actions Failure

zeroshade added 10 commits May 1, 2026 14:06

Merge branch 'main' into feat/configurable-registries

8c55e51

zeroshade had a problem deploying to snapshot May 20, 2026 17:46 — with GitHub Actions Failure

zeroshade temporarily deployed to snapshot May 20, 2026 18:16 — with GitHub Actions Inactive

zeroshade temporarily deployed to snapshot May 20, 2026 18:37 — with GitHub Actions Inactive

zeroshade temporarily deployed to snapshot May 20, 2026 19:14 — with GitHub Actions Inactive

zeroshade requested a review from amoeba May 20, 2026 19:48

Merge branch 'main' into feat/configurable-registries

51a138c

Resolves conflict in cmd/dbc/main.go: ports the initVTProcessing() call added on main (#383) into the new main() that calls runStartup(), instead of the pre-refactor main() body that no longer exists on this branch.

zeroshade had a problem deploying to snapshot May 21, 2026 17:02 — with GitHub Actions Failure

zeroshade had a problem deploying to snapshot May 21, 2026 17:08 — with GitHub Actions Failure

elide long registry names

7cb0e41

amoeba temporarily deployed to snapshot May 26, 2026 20:10 — with GitHub Actions Inactive

zeroshade added 2 commits June 10, 2026 15:56

zeroshade temporarily deployed to snapshot June 10, 2026 20:04 — with GitHub Actions Inactive

amoeba approved these changes Jun 11, 2026

View reviewed changes

zeroshade mentioned this pull request Jun 12, 2026

docs(registry): document configurable driver registries #393

Open

docs: back out registry docs changes

6012dbd

Move the driver-registry documentation to a separate PR (#393) so the docs can be reviewed and merged independently, per review feedback.

zeroshade temporarily deployed to snapshot June 12, 2026 18:25 — with GitHub Actions Inactive

zeroshade merged commit 1590b22 into main Jun 12, 2026
12 checks passed

zeroshade deleted the feat/configurable-registries branch June 12, 2026 18:28

Conversation

zeroshade commented Apr 17, 2026

Summary

What's New

Config Format

Implementation Notes

Uh oh!

amoeba left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zeroshade commented May 20, 2026

Uh oh!

zeroshade commented May 20, 2026

Uh oh!

zeroshade commented May 21, 2026

Uh oh!

amoeba commented May 26, 2026

Uh oh!

zeroshade commented May 27, 2026

Uh oh!

zeroshade commented Jun 10, 2026

Uh oh!

amoeba left a comment

Choose a reason for hiding this comment

Uh oh!

zeroshade commented Jun 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants