Proposal: Standalone DuckDB Extension for zvec

[akuligowski9]

Hi all -- this is my first time opening a Discussion, so I appreciate any grace as I learn the process!

I recently opened PR #136, which implemented a DuckDB extension using a collection-bridge approach (zvec owns storage, DuckDB functions act as a SQL bridge). Based on the great feedback from the maintainers, I'm closing that PR and moving the work to a standalone repository instead.

I'd love to use this Discussion to align on design direction before building further. Here's where my thinking is based on the feedback:

Current state

The initial implementation (from PR #136) includes:

• 4 SQL functions: zvec_create, zvec_insert, zvec_search, zvec_fetch
• Thread-safe CollectionRegistry, zvec-to-DuckDB type mapping, JSON schema parser
• Collection-bridge pattern where zvec owns storage

Open questions (based on maintainer feedback)

1. Core-based approach: The suggestion was to use zvec's core as an indexing-only layer and let DuckDB own storage (similar to duckdb-vss). Since the core C++ SDK is targeting v0.3.0, should I wait for that to stabilize, or is there enough in src/include/core to start building against now?

2. MVP scope: What should the MVP include? My current thinking:

   • In-memory storage backend + disk persistence
   • Basic HNSW index support
   • Core vector search functions

3. Arrow integration: The maintainers mentioned Arrow as a future transport layer for tighter DuckDB integration (e.g., SELECT * FROM zvec_collection). Is this a post-MVP goal, or should the MVP architecture account for it?

4. Standalone repo structure: I'll be setting up a separate repository for this. Any preferences on naming, build system, or how it should reference/depend on zvec?

Happy to hear any thoughts on scope, design, or priorities. Thanks!