feat: add local mode for pull, plan, apply without a server deployment

README.md

@@ -11,6 +11,32 @@ SchemaBot provides a fully interactive CLI for planning, applying, and monitorin
 
 ![SchemaBot CLI Demo](./docs/assets/cli-demo.gif)
 
+### Quick start
+
+Use the CLI to manage schema changes on any MySQL database you can connect to. The `--dsn` flag accepts any MySQL connection string — local, remote, RDS, etc. Schema changes are applied online using [Spirit](https://github.com/block/spirit) — reads and writes continue uninterrupted during the change.
+
+```bash
+# Install
+go install github.com/block/schemabot/pkg/cmd@latest
+
+# Start local mysql on port 3306 if not running
+brew install mysql && mysql.server start
+
+# Ensure $GOPATH/bin is on your PATH
+export PATH="$PATH:$(go env GOPATH)/bin"
+
+# Pull your existing schema into a declarative schema directory
+schemabot pull --dsn "root@tcp(localhost:3306)/mydb" -e staging -o ./schema
+
+# Edit a .sql file, then plan and apply
+schemabot plan -s ./schema -e staging
+schemabot apply -s ./schema -e staging
+```
+
+A background server auto-starts on first use and stores state in `_schemabot` on your local MySQL.
+
+This quickstart runs entirely on your machine. For GitHub PR integration, long-running schema changes, or managing lots of databases at scale, deploy SchemaBot as a server — see [deploy/](./deploy/) for sample code and [GitHub App setup](./docs/github-app-setup.md) (docs are still a work in progress).
+
 ## PR Demo
 
 _Coming soon_
@@ -53,7 +79,9 @@ SchemaBot handles the full lifecycle:
 
 Simple changes (e.g., adding a column) use instant DDL and complete in milliseconds. Operations that require a row copy (e.g., adding an index) run online without blocking reads or writes.
 
-## Quick Start
+## Docker demo
+
+Create a Docker environment that runs schemabot server and test databases
 
 ```bash
 make demo    # Start services, apply schema, seed data

docs/configuration.md

@@ -1,10 +1,113 @@
 # Configuration
 
-SchemaBot loads config from the `SCHEMABOT_CONFIG_FILE` environment variable.
+- [Config Files](#config-files)
+- [Local](#local)
+- [Server Deployment](#server-deployment)
+- [Profiles](#profiles)
+- [Secret Resolution](#secret-resolution)
 
-## Local Mode
+## Config Files
 
-The SchemaBot process runs the engine directly. This is best for most users and recommended to start with.
+SchemaBot uses different config files depending on how you run it:
+
+| File | Location | Purpose | Used by |
+|------|----------|---------|---------|
+| `schemabot.yaml` | In your schema directory | Declares the database name and type | CLI (plan, apply, pull) |
+| `~/.schemabot/config.yaml` | Home directory | CLI profiles and local database connections | CLI (all commands) |
+| Server config (`SCHEMABOT_CONFIG_FILE`) | Anywhere | Storage DSN, database configs, Tern endpoints | `schemabot serve` |
+
+`schemabot.yaml` — declares which database your schema files belong to:
+
+```yaml
+database: mydb
+type: mysql
+```
+
+`~/.schemabot/config.yaml` — tells the CLI how to connect:
+
+```yaml
+default_profile: local
+
+profiles:
+  staging:
+    endpoint: http://localhost:8080
+
+local:
+  mydb:
+    type: mysql
+    environments:
+      staging:
+        dsn: "root@tcp(localhost:3306)/mydb"
+```
+
+Server config (`SCHEMABOT_CONFIG_FILE`) — only used when deploying SchemaBot as a server:
+
+```yaml
+storage:
+  dsn: "env:SCHEMABOT_DSN"
+databases:
+  mydb:
+    type: mysql
+    environments:
+      staging:
+        dsn: "env:STAGING_DSN"
+```
+
+The first two are for CLI usage. The server config is separate — see [Server Deployment](#server-deployment) for details.
+
+## Local
+
+When using SchemaBot without a server deployment, the CLI auto-starts a lightweight background server on your machine. Database connection details are stored in the `local` section of the config. The `pull` command sets this up automatically.
+
+```yaml
+# ~/.schemabot/config.yaml
+local:
+  mydb:
+    type: mysql
+    environments:
+      staging:
+        dsn: "root@tcp(localhost:3306)/mydb"
+      production:
+        dsn: "env:PRODUCTION_DSN"
+  mypsdb:
+    type: vitess
+    environments:
+      production:
+        organization: myorg
+        token: "env:PLANETSCALE_SERVICE_TOKEN"
+```
+
+### Adding a database
+
+```bash
+# Pull creates the local config entry automatically
+schemabot pull --dsn "root@tcp(localhost:3306)/mydb" -e staging -o ./schema
+```
+
+This writes the schema files and adds `mydb/staging` to `~/.schemabot/config.yaml`.
+
+### Managing the background server
+
+```bash
+schemabot local status   # show server state, PID, port
+schemabot local stop     # stop the background server
+schemabot local reset    # stop server and drop _schemabot database
+```
+
+## Server Deployment
+
+For GitHub PR integration, team coordination, or managing many databases, deploy SchemaBot as a server.
+
+The server loads config from a YAML file. Set `SCHEMABOT_CONFIG_FILE` to the path:
+
+```bash
+export SCHEMABOT_CONFIG_FILE=/etc/schemabot/config.yaml
+schemabot serve
+```
+
+### Single-process (recommended for most deployments)
+
+The server runs the schema change engine directly:
 
 ```yaml
 storage:
@@ -20,9 +123,9 @@ databases:
         dsn: "file:/run/secrets/prod-dsn"
 ```
 
-## gRPC Mode
+### Distributed (gRPC)
 
-SchemaBot delegates to remote services that implement the Tern proto. This is useful for distributed deployments where schema changes need to run in separate isolated environments.
+For deployments where schema changes need to run in separate isolated environments, SchemaBot delegates to remote services that implement the Tern proto:
 
 ```yaml
 storage:
@@ -36,9 +139,36 @@ tern_deployments:
     production: "tern1-production:9090"
 ```
 
+## Profiles
+
+CLI profiles let you switch between local mode and server deployments. Use `--profile` to select one, or set a default.
+
+```yaml
+# ~/.schemabot/config.yaml
+default_profile: local
+
+profiles:
+  staging:
+    endpoint: http://localhost:8080
+  production:
+    endpoint: https://schemabot.example.com
+```
+
+```bash
+schemabot plan -s ./schema -e staging                    # uses default profile
+schemabot plan -s ./schema -e staging --profile staging   # explicit server profile
+schemabot plan -s ./schema -e staging --profile local     # force local mode
+```
+
+`local` is a reserved profile name that triggers local mode. Set `default_profile: local` to make it the default.
+
+```bash
+schemabot configure   # interactive profile setup
+```
+
 ## Secret Resolution
 
-DSN values support secret resolution prefixes:
+DSN and credential values support secret resolution prefixes:
 
 | Prefix | Example | Description |
 |---|---|---|

integration/local_mode_test.go

@@ -0,0 +1,178 @@
+//go:build integration
+
+package integration
+
+import (
+	"bytes"
+	"database/sql"
+	"fmt"
+	"net"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	_ "github.com/go-sql-driver/mysql"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/block/schemabot/pkg/e2eutil"
+)
+
+var cliFinder e2eutil.CLIFinder
+
+// TestLocalMode_PullPlanApply exercises the full local mode UX:
+// pull schema from a live database, edit a file, plan, and apply —
+// with the background daemon auto-starting just like a real user.
+func TestLocalMode_PullPlanApply(t *testing.T) {
+	ctx := t.Context()
+
+	// Build a storage DSN from the target MySQL container.
+	// The testcontainer uses root:testpassword, so we need the full DSN.
+	storageDSN := strings.Replace(targetDSN, "/target_test", "/_schemabot", 1)
+
+	// Build the CLI binary (cached across tests in this package)
+	binPath := cliFinder.FindOrBuild(t, "..", "./pkg/cmd", "../bin/schemabot")
+
+	// Create a test database with a table
+	db, err := sql.Open("mysql", targetDSN+"&multiStatements=true")
+	require.NoError(t, err)
+	defer func() { _ = db.Close() }()
+
+	dbName := fmt.Sprintf("local_mode_%d", time.Now().UnixNano()%100000)
+	_, err = db.ExecContext(ctx, "CREATE DATABASE IF NOT EXISTS `"+dbName+"`")
+	require.NoError(t, err)
+	t.Cleanup(func() {
+		_, _ = db.ExecContext(ctx, "DROP DATABASE IF EXISTS `"+dbName+"`")
+		// Clean up _schemabot storage database created by the daemon
+		_, _ = db.ExecContext(ctx, "DROP DATABASE IF EXISTS `_schemabot`")
+	})
+
+	appDSN := strings.Replace(targetDSN, "/target_test", "/"+dbName, 1)
+	appDB, err := sql.Open("mysql", appDSN+"&multiStatements=true")
+	require.NoError(t, err)
+	defer func() { _ = appDB.Close() }()
+
+	_, err = appDB.ExecContext(ctx, `CREATE TABLE users (
+  id bigint unsigned NOT NULL AUTO_INCREMENT,
+  name varchar(255) NOT NULL,
+  email varchar(255) NOT NULL,
+  PRIMARY KEY (id),
+  UNIQUE KEY email (email)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci`)
+	require.NoError(t, err)
+
+	// Set up isolated HOME and schema directory
+	tmpHome := t.TempDir()
+	schemaDir := t.TempDir()
+
+	// Find a free port for the daemon
+	daemonPort := freePort(t)
+
+	// Environment for all CLI commands — isolated HOME, custom storage DSN/port
+	env := append(os.Environ(),
+		"HOME="+tmpHome,
+		"SCHEMABOT_LOCAL_STORAGE_DSN="+storageDSN,
+		"SCHEMABOT_LOCAL_PORT="+fmt.Sprintf("%d", daemonPort),
+	)
+
+	// Helper to run CLI commands with the test environment
+	run := func(args ...string) string {
+		t.Helper()
+		cmd := exec.CommandContext(ctx, binPath, args...)
+		cmd.Env = env
+		var stdout, stderr bytes.Buffer
+		cmd.Stdout = &stdout
+		cmd.Stderr = &stderr
+		err := cmd.Run()
+		output := stdout.String() + stderr.String()
+		require.NoErrorf(t, err, "CLI %v failed:\n%s", args, output)
+		return output
+	}
+
+	runInDir := func(dir string, args ...string) string {
+		t.Helper()
+		cmd := exec.CommandContext(ctx, binPath, args...)
+		cmd.Dir = dir
+		cmd.Env = env
+		var stdout, stderr bytes.Buffer
+		cmd.Stdout = &stdout
+		cmd.Stderr = &stderr
+		err := cmd.Run()
+		output := stdout.String() + stderr.String()
+		require.NoErrorf(t, err, "CLI %v failed:\n%s", args, output)
+		return output
+	}
+
+	// Step 1: Pull schema from the live database
+	pullOutput := run("pull", "--dsn", appDSN, "-o", schemaDir, "-e", "staging", "-d", dbName)
+	t.Logf("Pull output:\n%s", pullOutput)
+	assert.Contains(t, pullOutput, "Pulled 1 table")
+	assert.Contains(t, pullOutput, "users.sql")
+
+	// Verify schema files
+	usersSQL, err := os.ReadFile(filepath.Join(schemaDir, "users.sql"))
+	require.NoError(t, err)
+	assert.Contains(t, string(usersSQL), "CREATE TABLE `users`")
+
+	configData, err := os.ReadFile(filepath.Join(schemaDir, "schemabot.yaml"))
+	require.NoError(t, err)
+	assert.Contains(t, string(configData), "database: "+dbName)
+
+	// Verify local config was created
+	localConfig, err := os.ReadFile(filepath.Join(tmpHome, ".schemabot", "config.yaml"))
+	require.NoError(t, err)
+	assert.Contains(t, string(localConfig), dbName)
+	assert.Contains(t, string(localConfig), "staging")
+
+	// Step 2: Edit a .sql file — add a column
+	modifiedSQL := strings.Replace(string(usersSQL),
+		"PRIMARY KEY (`id`)",
+		"`phone` varchar(20) DEFAULT NULL,\n  PRIMARY KEY (`id`)",
+		1)
+	require.NoError(t, os.WriteFile(filepath.Join(schemaDir, "users.sql"), []byte(modifiedSQL), 0644))
+
+	// Step 3: Plan — this triggers daemon auto-start
+	planOutput := runInDir(schemaDir, "plan", "-s", schemaDir, "-e", "staging", "--json")
+	t.Logf("Plan output:\n%s", planOutput)
+	assert.Contains(t, planOutput, "phone")
+	assert.Contains(t, planOutput, "plan_id")
+
+	// Verify daemon is running
+	t.Cleanup(func() {
+		// Stop the daemon after the test
+		cmd := exec.CommandContext(ctx, binPath, "local", "stop")
+		cmd.Env = env
+		_ = cmd.Run()
+	})
+
+	// Step 4: Apply — the daemon is already running
+	applyOutput := runInDir(schemaDir, "apply", "-s", schemaDir, "-e", "staging", "-y", "-w", "-o", "log")
+	t.Logf("Apply output:\n%s", applyOutput)
+
+	// Step 5: Verify the column was added to the target database
+	var columnExists bool
+	err = appDB.QueryRowContext(ctx,
+		"SELECT COUNT(*) > 0 FROM information_schema.COLUMNS WHERE TABLE_SCHEMA = ? AND TABLE_NAME = 'users' AND COLUMN_NAME = 'phone'",
+		dbName,
+	).Scan(&columnExists)
+	require.NoError(t, err)
+	assert.True(t, columnExists, "phone column should exist after apply")
+	t.Log("Verified: phone column exists in target database")
+
+	// Step 6: Verify local server status
+	statusOutput := run("local", "status")
+	assert.Contains(t, statusOutput, "running")
+}
+
+// freePort finds a free TCP port by binding to :0 and returning the assigned port.
+func freePort(t *testing.T) int {
+	t.Helper()
+	l, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", "127.0.0.1:0")
+	require.NoError(t, err)
+	port := l.Addr().(*net.TCPAddr).Port
+	require.NoError(t, l.Close())
+	return port
+}