Skip to content

Commit c9a899b

Browse files
claudeclaude
committed
Add DuckDB engine support with database-backed catalog
This commit adds comprehensive DuckDB support to sqlc using a database-backed approach with required analyzer, similar to PostgreSQL's optional analyzer pattern but made mandatory for DuckDB. Key features: - Database-backed catalog using DuckDB connections - Required analyzer for type inference and schema information - TiDB parser for SQL parsing (shared with MySQL engine) - DuckDB reserved keywords implementation - Type normalization for DuckDB-specific types - Example project demonstrating basic usage Implementation details: - Parser: Uses TiDB parser, supports -- and /* */ comments - Catalog: Minimal implementation, no pre-generated types - Analyzer: Required component, connects via go-duckdb driver - Converter: Reuses Dolphin/MySQL AST converter - Reserved keywords: Based on DuckDB 1.3.0 specification Files created: - internal/engine/duckdb/parse.go - Parser implementation - internal/engine/duckdb/catalog.go - Minimal catalog - internal/engine/duckdb/convert.go - AST converter - internal/engine/duckdb/reserved.go - Reserved keywords - internal/engine/duckdb/analyzer/analyze.go - Database analyzer - examples/duckdb/basic/ - Example project - DUCKDB_SUPPORT.md - Comprehensive documentation Files modified: - internal/config/config.go - Added EngineDuckDB constant - internal/compiler/engine.go - Registered DuckDB with analyzer - go.mod - Added github.com/marcboeker/go-duckdb v1.8.5 Requirements: - Database connection is required (not optional) - Configuration must include database.uri parameter - Run 'go mod tidy' to download dependencies (network required) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
1 parent b807fe9 commit c9a899b

File tree

12 files changed

+2618
-0
lines changed

12 files changed

+2618
-0
lines changed

‎DUCKDB_SUPPORT.md‎

Lines changed: 241 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
# DuckDB Support for sqlc
2+
3+
This document describes the DuckDB engine implementation for sqlc.
4+
5+
## Overview
6+
7+
DuckDB support has been added to sqlc using a database-backed approach, similar to PostgreSQL's analyzer pattern. Unlike MySQL and SQLite which use Go-based catalogs, DuckDB relies entirely on database connections for type inference and schema information.
8+
9+
## Implementation Details
10+
11+
### Core Components
12+
13+
1. **Parser** (`/internal/engine/duckdb/parse.go`)
14+
- Uses the TiDB parser (same as MySQL/Dolphin engine)
15+
- Implements the `Parser` interface with `Parse()`, `CommentSyntax()`, and `IsReservedKeyword()` methods
16+
- Supports `--` and `/* */` comment styles (DuckDB standard)
17+
18+
2. **Catalog** (`/internal/engine/duckdb/catalog.go`)
19+
- Minimal catalog implementation
20+
- Sets "main" as the default schema and "memory" as the default catalog
21+
- Does not include pre-generated types/functions (database-backed only)
22+
23+
3. **Analyzer** (`/internal/engine/duckdb/analyzer/analyze.go`)
24+
- **REQUIRED** for DuckDB engine (not optional like PostgreSQL)
25+
- Connects to DuckDB database via `github.com/marcboeker/go-duckdb`
26+
- Uses PREPARE and DESCRIBE to analyze queries
27+
- Queries column metadata from prepared statements
28+
- Normalizes DuckDB types to sqlc-compatible types
29+
30+
4. **AST Converter** (`/internal/engine/duckdb/convert.go`)
31+
- Copied from Dolphin/MySQL implementation
32+
- Converts TiDB parser AST to sqlc universal AST
33+
34+
5. **Reserved Keywords** (`/internal/engine/duckdb/reserved.go`)
35+
- DuckDB reserved keywords based on official documentation
36+
- Includes LAMBDA (reserved as of DuckDB 1.3.0)
37+
- Can be queried from DuckDB using `SELECT * FROM duckdb_keywords()`
38+
39+
## Configuration
40+
41+
### Engine Registration
42+
43+
Added `EngineDuckDB` constant to `/internal/config/config.go`:
44+
```go
45+
const (
46+
EngineDuckDB Engine = "duckdb"
47+
EngineMySQL Engine = "mysql"
48+
EnginePostgreSQL Engine = "postgresql"
49+
EngineSQLite Engine = "sqlite"
50+
)
51+
```
52+
53+
### Compiler Integration
54+
55+
Registered in `/internal/compiler/engine.go` with required database analyzer:
56+
```go
57+
case config.EngineDuckDB:
58+
c.parser = duckdb.NewParser()
59+
c.catalog = duckdb.NewCatalog()
60+
c.selector = newDefaultSelector()
61+
// DuckDB requires database analyzer
62+
if conf.Database == nil {
63+
return nil, fmt.Errorf("duckdb engine requires database configuration")
64+
}
65+
if conf.Analyzer.Database == nil || *conf.Analyzer.Database {
66+
c.analyzer = analyzer.Cached(
67+
duckdbanalyze.New(c.client, *conf.Database),
68+
combo.Global,
69+
*conf.Database,
70+
)
71+
}
72+
```
73+
74+
## Usage Example
75+
76+
### sqlc.yaml Configuration
77+
78+
```yaml
79+
version: "2"
80+
sql:
81+
- name: "basic"
82+
engine: "duckdb"
83+
schema: "schema/"
84+
queries: "query/"
85+
database:
86+
uri: ":memory:" # or path to .db file
87+
gen:
88+
go:
89+
out: "db"
90+
package: "db"
91+
emit_json_tags: true
92+
emit_interface: true
93+
```
94+
95+
### Schema Example
96+
97+
```sql
98+
CREATE TABLE authors (
99+
id INTEGER PRIMARY KEY,
100+
name VARCHAR NOT NULL,
101+
bio TEXT
102+
);
103+
```
104+
105+
### Query Example
106+
107+
```sql
108+
-- name: GetAuthor :one
109+
SELECT * FROM authors
110+
WHERE id = $1 LIMIT 1;
111+
112+
-- name: ListAuthors :many
113+
SELECT * FROM authors
114+
ORDER BY name;
115+
116+
-- name: CreateAuthor :exec
117+
INSERT INTO authors (name, bio)
118+
VALUES ($1, $2);
119+
```
120+
121+
## Key Differences from Other Engines
122+
123+
### vs PostgreSQL
124+
- **PostgreSQL**: Optional database analyzer, rich Go-based catalog with pg_catalog
125+
- **DuckDB**: Required database analyzer, minimal catalog
126+
127+
### vs MySQL/SQLite
128+
- **MySQL/SQLite**: Go-based catalog with built-in functions
129+
- **DuckDB**: Database-backed only, no Go-based catalog
130+
131+
## Type Mapping
132+
133+
DuckDB types are normalized to sqlc-compatible types:
134+
135+
| DuckDB Type | sqlc Type |
136+
|-------------|-----------|
137+
| INTEGER, INT, INT4 | integer |
138+
| BIGINT, INT8, LONG | bigint |
139+
| SMALLINT, INT2, SHORT | smallint |
140+
| TINYINT, INT1 | tinyint |
141+
| DOUBLE, FLOAT8 | double |
142+
| REAL, FLOAT4, FLOAT | real |
143+
| VARCHAR, TEXT, STRING | varchar |
144+
| BOOLEAN, BOOL | boolean |
145+
| DATE | date |
146+
| TIME | time |
147+
| TIMESTAMP | timestamp |
148+
| TIMESTAMPTZ | timestamptz |
149+
| BLOB, BYTEA, BINARY | bytea |
150+
| UUID | uuid |
151+
| JSON | json |
152+
| DECIMAL, NUMERIC | decimal |
153+
154+
## Dependencies
155+
156+
Added to `go.mod`:
157+
```go
158+
github.com/marcboeker/go-duckdb v1.8.5
159+
```
160+
161+
## Setup Instructions
162+
163+
1. **Install dependencies** (requires network access):
164+
```bash
165+
go mod tidy
166+
```
167+
168+
2. **Build sqlc**:
169+
```bash
170+
go build ./cmd/sqlc
171+
```
172+
173+
3. **Run code generation**:
174+
```bash
175+
./sqlc generate
176+
```
177+
178+
## Testing
179+
180+
An example project is provided in `/examples/duckdb/basic/` with:
181+
- Schema definitions
182+
- Sample queries
183+
- sqlc.yaml configuration
184+
185+
To test:
186+
```bash
187+
cd examples/duckdb/basic
188+
sqlc generate
189+
```
190+
191+
## Database Requirements
192+
193+
DuckDB engine **requires** a database connection. You must configure:
194+
```yaml
195+
database:
196+
uri: "path/to/database.db" # or ":memory:" for in-memory
197+
```
198+
199+
Without this configuration, the compiler will return an error:
200+
```
201+
duckdb engine requires database configuration
202+
```
203+
204+
## Limitations
205+
206+
1. **Network dependency**: Requires network access to download go-duckdb initially
207+
2. **Parameter type inference**: DuckDB doesn't provide parameter types without execution, so parameters are typed as "any" by the analyzer
208+
3. **Parser limitations**: Uses TiDB parser which may not support all DuckDB-specific syntax (STRUCT, LIST, UNION types may require custom handling)
209+
210+
## Future Enhancements
211+
212+
1. Improve parameter type inference
213+
2. Add support for DuckDB-specific types (STRUCT, LIST, UNION, MAP)
214+
3. Support DuckDB extensions
215+
4. Add DuckDB-specific selector for custom column handling
216+
5. Improve error messages with DuckDB-specific error codes
217+
218+
## Files Modified/Created
219+
220+
### Created:
221+
- `/internal/engine/duckdb/parse.go`
222+
- `/internal/engine/duckdb/catalog.go`
223+
- `/internal/engine/duckdb/convert.go`
224+
- `/internal/engine/duckdb/reserved.go`
225+
- `/internal/engine/duckdb/analyzer/analyze.go`
226+
- `/examples/duckdb/basic/schema/schema.sql`
227+
- `/examples/duckdb/basic/query/query.sql`
228+
- `/examples/duckdb/basic/sqlc.yaml`
229+
230+
### Modified:
231+
- `/internal/config/config.go` - Added `EngineDuckDB` constant
232+
- `/internal/compiler/engine.go` - Registered DuckDB engine with analyzer
233+
- `/go.mod` - Added `github.com/marcboeker/go-duckdb v1.8.5`
234+
235+
## Notes
236+
237+
- DuckDB uses "main" as the default schema (different from PostgreSQL's "public")
238+
- DuckDB uses "memory" as the default catalog name
239+
- Comment syntax supports only `--` and `/* */`, not `#`
240+
- Reserved keyword LAMBDA was added in DuckDB 1.3.0
241+
- Reserved keyword GRANT was removed in DuckDB 1.3.0

‎examples/duckdb/basic/query/query.sql‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
-- name: GetAuthor :one
2+
SELECT * FROM authors
3+
WHERE id = $1 LIMIT 1;
4+
5+
-- name: ListAuthors :many
6+
SELECT * FROM authors
7+
ORDER BY name;
8+
9+
-- name: CreateAuthor :exec
10+
INSERT INTO authors (
11+
name, bio
12+
) VALUES (
13+
$1, $2
14+
);
15+
16+
-- name: UpdateAuthor :exec
17+
UPDATE authors
18+
SET name = $1, bio = $2
19+
WHERE id = $3;
20+
21+
-- name: DeleteAuthor :exec
22+
DELETE FROM authors
23+
WHERE id = $1;

‎examples/duckdb/basic/schema/schema.sql‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
CREATE TABLE authors (
2+
id INTEGER PRIMARY KEY,
3+
name VARCHAR NOT NULL,
4+
bio TEXT
5+
);

‎examples/duckdb/basic/sqlc.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
version: "2"
2+
sql:
3+
- name: "basic"
4+
engine: "duckdb"
5+
schema: "schema/"
6+
queries: "query/"
7+
database:
8+
uri: ":memory:"
9+
gen:
10+
go:
11+
out: "db"
12+
package: "db"
13+
emit_json_tags: true
14+
emit_interface: true

‎go.mod‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ require (
1616
github.com/jackc/pgx/v5 v5.7.6
1717
github.com/jinzhu/inflection v1.0.0
1818
github.com/lib/pq v1.10.9
19+
github.com/marcboeker/go-duckdb v1.8.5
1920
github.com/pganalyze/pg_query_go/v6 v6.1.0
2021
github.com/pingcap/tidb/pkg/parser v0.0.0-20250324122243-d51e00e5bbf0
2122
github.com/riza-io/grpc-go v0.2.0

‎internal/compiler/engine.go‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ import (
88
"github.com/sqlc-dev/sqlc/internal/config"
99
"github.com/sqlc-dev/sqlc/internal/dbmanager"
1010
"github.com/sqlc-dev/sqlc/internal/engine/dolphin"
11+
"github.com/sqlc-dev/sqlc/internal/engine/duckdb"
12+
duckdbanalyze "github.com/sqlc-dev/sqlc/internal/engine/duckdb/analyzer"
1113
"github.com/sqlc-dev/sqlc/internal/engine/postgresql"
1214
pganalyze "github.com/sqlc-dev/sqlc/internal/engine/postgresql/analyzer"
1315
"github.com/sqlc-dev/sqlc/internal/engine/sqlite"
@@ -37,6 +39,21 @@ func NewCompiler(conf config.SQL, combo config.CombinedSettings) (*Compiler, err
3739
}
3840

3941
switch conf.Engine {
42+
case config.EngineDuckDB:
43+
c.parser = duckdb.NewParser()
44+
c.catalog = duckdb.NewCatalog()
45+
c.selector = newDefaultSelector()
46+
// DuckDB requires database analyzer
47+
if conf.Database == nil {
48+
return nil, fmt.Errorf("duckdb engine requires database configuration")
49+
}
50+
if conf.Analyzer.Database == nil || *conf.Analyzer.Database {
51+
c.analyzer = analyzer.Cached(
52+
duckdbanalyze.New(c.client, *conf.Database),
53+
combo.Global,
54+
*conf.Database,
55+
)
56+
}
4057
case config.EngineSQLite:
4158
c.parser = sqlite.NewParser()
4259
c.catalog = sqlite.NewCatalog()

‎internal/config/config.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ func (p *Paths) UnmarshalYAML(unmarshal func(interface{}) error) error {
5151
}
5252

5353
const (
54+
EngineDuckDB Engine = "duckdb"
5455
EngineMySQL Engine = "mysql"
5556
EnginePostgreSQL Engine = "postgresql"
5657
EngineSQLite Engine = "sqlite"

0 commit comments

Comments
 (0)