Skip to content

Local-first: store data in a SQLite vault instead of Supabase - #7

Open
myaschmitz wants to merge 2 commits into
mainfrom
myaschmitz-local-first-design-doc
Open

Local-first: store data in a SQLite vault instead of Supabase#7
myaschmitz wants to merge 2 commits into
mainfrom
myaschmitz-local-first-design-doc

Conversation

@myaschmitz

Copy link
Copy Markdown
Owner

Makes Koku run entirely on the user's own machine: clone, npm install, npm run dev. No account, no Supabase project, no network.

This is phases 0 and 1 of docs/local-first-design.md, which is added in the first commit and is the best place to start reviewing. Phase 2 (conflict-free multi-device sync) is deliberately not here — see the caveat at the bottom.

Two commits, reviewable independently

1. refactor: introduce storage-agnostic repository layer — no behavior change. Pages and components stop talking to Supabase directly and go through a repository interface in src/lib/db, with the existing Supabase client behind a single adapter. This is the de-risking step: it makes the backend swap an adapter change rather than a rewrite.

2. feat: store data in a local SQLite vault instead of Supabase — adds the local backend, makes it the default, and removes auth.

How it fits together

UI ──> lib/db (repository interface)
         ├─ local adapter ──/api/local/rpc──> server repo ──> ~/.koku/vault
         └─ supabase adapter  (NEXT_PUBLIC_KOKU_BACKEND=supabase)

Data lives in a vault directory, KOKU_VAULT_DIR or ~/.koku/vault by default:

vault/
  koku.sqlite   decks, cards, review history, settings
  blobs/        card images, addressed by content hash

Decisions worth a look

  • node:sqlite (built into Node 22+) rather than better-sqlite3, so there's no native module to compile. Cost: engines.node moves to >=22.
  • One RPC endpoint + a Proxy client adapter, so the repository's method list lives only in the interface instead of being restated per backend. Only own methods on allow-listed repositories are reachable; prototype keys like constructor and __proto__ are rejected.
  • Schema as a TS module, not a .sql file read at runtime, so it survives bundling.
  • Content-addressed images, which means duplicating a card reuses the same bytes and duplicateCardImages becomes a no-op. Blob reads validate the key shape before touching the filesystem.
  • Booleans are coerced on read — SQLite returns 0/1 and the UI relies on truthiness. This was the most likely source of subtle bugs in the port, so it's covered by tests.
  • end_vacation_mode moves from a Postgres RPC to a local transaction.

Removed

Auth in full: login, the auth callback/confirm routes, the account-delete endpoint, the middleware, and the Supabase server client — plus user_id on every domain type and the 19 auth.getUser() calls that existed only to build a user_id filter. Also deleted image-upload.tsx and image-grid.tsx, which were unreferenced.

Privacy and terms copy is updated to describe local storage. That's user-facing legal text and deserves a careful read.

Incidental fixes

  • Undo looked up the newest review log by created_at, a column review_logs doesn't have. Now orders by reviewed_at.
  • A pre-existing type error in the import/export test, surfaced by the @types/node bump.

Verification

npm test (95 passing, 19 new), npm run lint, npx tsc --noEmit, and npm run build are all clean. The build no longer needs any Supabase env vars.

Beyond that I ran the app against a live vault and exercised deck/card CRUD, due-card filtering and ordering, duplicate detection, review logging, settings round-trips, vacation-mode date shifting, image upload/serve/dedup, and the RPC and path-traversal guards.

Known limitation

Multi-device sync via Google Drive or Dropbox is not safe yet. The vault is still a single SQLite file, which is exactly the hazard section 3 of the design doc warns about: file-sync services replicate whole files plus -wal/-shm sidecars, and two devices editing offline resolve as last-writer-wins. The README says this plainly. Phase 2 replaces the single file with per-device append-only op logs, which is what actually makes syncing safe.

Supabase remains available via NEXT_PUBLIC_KOKU_BACKEND=supabase — whether to keep that adapter long-term is still an open question in the design doc.

myaschmitz and others added 2 commits August 30, 2026 12:48
Phase 0 of the local-first migration described in docs/local-first-design.md.

Pages and components no longer talk to Supabase directly. They go through a
repository interface in src/lib/db, with the existing Supabase client behind a
single adapter. Behavior is unchanged; this exists so a local SQLite/vault
backend can be swapped in as one implementation rather than a rewrite.

Notable points:

- The repository scopes every query to the current user internally, so the 19
  auth.getUser() calls that existed only to build a user_id filter are gone.
- updated_at is now set by the adapter, not by each call site.
- Settings reads always resolve to a full object, removing the default-settings
  literal that was duplicated across three pages.
- Deck count aggregation moved into a shared helper.
- upload-image.ts is replaced by the blob repository; resolveImageUrl moves to
  lib/image-url so it stays dependency-free for render paths.
- The offline queue and its sync hook now share one applyOp helper and write
  through the repository, keeping the stale-write guard as an explicit
  ifUnmodifiedSince option.
- Deleted image-upload.tsx and image-grid.tsx, which were unreferenced.

Fixes a latent bug: undo looked up the newest review log by created_at, a
column review_logs does not have. It now orders by reviewed_at.

Tests move to mocking the repository seam instead of the Supabase client.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Phase 1 of the local-first migration. Koku now runs with no account, no
Supabase project, and no network: clone, npm install, npm run dev.

Data lives in a vault directory, KOKU_VAULT_DIR or ~/.koku/vault by default:

  vault/
    koku.sqlite   decks, cards, review history, settings
    blobs/        card images, addressed by content hash

Point KOKU_VAULT_DIR at a Drive/Dropbox/Syncthing folder to move a collection
between machines. Concurrent multi-device editing is not safe yet; the vault is
still a single SQLite file. The append-only op log that fixes that is phase 2.

How it fits together:

- schema.ts ports the Postgres migrations: no user_id or RLS, TIMESTAMPTZ as
  ISO text, booleans as INTEGER, user_settings pinned to a single row. It is a
  TS module rather than a .sql file so it survives bundling.
- server-repo.ts implements the repository over node:sqlite (built in, so no
  native dependency). Booleans are coerced back on read, since SQLite returns
  0/1 and the UI relies on truthiness.
- A single /api/local/rpc endpoint dispatches repository calls, and the client
  adapter is a Proxy over it, so the method list is not restated per backend.
  Only own methods on allow-listed repositories are reachable.
- Images are content-addressed under blobs/, so duplicating a card reuses the
  same bytes and copy() is a no-op. Reads validate the key shape before
  touching the filesystem.
- end_vacation_mode moves from a Postgres RPC to a local transaction.

Supabase is still available via NEXT_PUBLIC_KOKU_BACKEND=supabase.

Auth is gone: login, the auth callback/confirm routes, the account-delete
endpoint, the middleware, and the server client are all deleted, along with
user_id on every domain type. Landing page now redirects to /decks, and the
privacy/terms copy is updated to match local storage.

Requires Node 22+ for node:sqlite; engines and @types/node bumped to match.

Adds 19 tests covering boolean coercion, due-card filtering, the vacation-mode
shift, LIKE-wildcard escaping in duplicate detection, and the stale-write
guard. Also fixes a pre-existing type error in the import/export test surfaced
by the @types/node bump.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@vercel

vercel Bot commented Aug 30, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
koku Ready Ready Preview Aug 30, 2026 9:41pm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant