diff --git a/CLAUDE.md b/CLAUDE.md index b76a777..3cf8156 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -63,7 +63,7 @@ Capability directories (`.filter`, `.order`, `.first`, `.last`, `.export`, `.col ### Synth Apps -Tables whose names start with `_` (e.g., `_blog`, `_docs`) can be exposed as synthesized views -- markdown files, task lists, or plain text snippets. The synth layer (`fs/synth/`) maps filesystem operations to table rows using format-specific conventions. See `docs/markdown-app.md` and `docs/tasks-app.md`. +Tables whose names start with `_` (e.g., `_blog`, `_docs`) can be exposed as synthesized views -- markdown files, task lists, or plain text snippets. The synth layer (`fs/synth/`) maps filesystem operations to table rows using format-specific conventions. See `docs/file-first.md`. ### SQL Identifier Quoting @@ -109,6 +109,7 @@ Multiple TigerFS mounts may connect to the same database concurrently. This mean - **Cache metadata only:** Sizes, permissions, directory entries, column names/types, primary keys, table lists. These change rarely and have short TTLs. - **Stat cache keys must be unique per path:** Different pipeline paths (e.g., `.export/json` vs `.filter/active/true/.export/json`) must not share cache entries, even on the same table. - **Pattern:** `Stat` may use caches. `ReadFile` must always hit the DB. +- **Audit invalidation on every new write path:** any write must call `statCache.invalidate(schema, table)` and `pathCache.invalidate(schema, table)` with the *user* schema (not `synth.TigerFSSchema`); a schema-key mismatch silently leaves stale entries cached for up to 2 seconds and surfaces as cross-mount read inconsistencies. ## Logging @@ -154,6 +155,10 @@ For each implementation task: Integration tests use testcontainers-go for PostgreSQL. See `test/integration/` for examples. +### Stress-test monotonicity warnings + +Stress runs (`bin/tigerfs-stress start`) may emit `[warn iter ...] readLatestLogID regressed after ...` lines. These are **expected** — they indicate the macOS NFS layer (go-nfs library or kernel client, ruled out at every TigerFS layer) returned a stale `.log/.last/N/.export/json` snapshot after a heavy commit. The runner's monotonic helper retries up to 2.5s and falls back to the prior known-good `lastLogID`; the run continues correctly. The end-of-run "Monotonicity Warnings" section summarizes rate, recovery distribution, and op kinds preceding regressions. Do not treat these as bugs to fix in TigerFS; see the iter-107 investigation arc on `feat/undo` (commits `85a2495` → `f59cfcf`) for context. + ### Test Naming Convention Integration tests that mount the filesystem work with **both** FUSE (Linux) and NFS (macOS) — the mount method is auto-detected. Name tests by what they test, not the mount method: diff --git a/README.md b/README.md index 86f1f3a..e505713 100644 --- a/README.md +++ b/README.md @@ -2,78 +2,78 @@ A filesystem backed by PostgreSQL, and a filesystem interface to PostgreSQL. -Every file is a real PostgreSQL row. Directories are tables. File contents are columns. Multiple agents and humans can read and write the same files concurrently with full ACID guarantees. No sync protocols. No coordination layer. **The filesystem is the API.** +Every file is a real PostgreSQL row. Directories are tables. File contents are columns. Multiple agents and humans can read and write the same files concurrently with full ACID guarantees. Every change is versioned and reversible (file-first with history). No sync protocols. No coordination layer. **The filesystem is the API.** You can use TigerFS in two ways: -* **File-first**: Write markdown with frontmatter or other file types, organize into directories. Writes are atomic, and everything is auto-versioned. Any tool that works with files -- Claude Code, Cursor, grep, vim -- just works. Build lightweight apps via the filesystem: multi-agent task coordination is just `mv`'ing files between todo/doing/done directories. +* **File-first**: Write markdown with frontmatter or other file types, organize into directories. Writes are atomic, changes are versioned, and everything is reversible. Any tool that works with files -- Claude Code, Cursor, grep, vim -- just works. Build lightweight workspaces via the filesystem: multi-agent task coordination is just `mv`'ing files between todo/doing/done directories. * **Data-first**: Mount any Postgres database and explore it with `ls`, `cat`, `grep`, and other unix tools. For large databases, chain filters into paths that push down to SQL: `.by/customer_id/123/.order/created_at/.last/10/.export/json`. No database client or SQL needed, and ships with agent skills. -Both modes are backed by the same transactional database. You get real transactions, true concurrent access, and a SQL escape hatch when you need it. +Both modes are backed by the same transactional database. You get real transactions, true concurrent access, and a SQL escape hatch when you need it. TigerFS mounts via FUSE on Linux and NFS on macOS, no extra dependencies needed. -TigerFS mounts via FUSE on Linux and NFS on macOS, no extra dependencies needed. +### Agent Skills -## Quick Start +TigerFS ships with agent skills for Claude Code, Gemini CLI, Codex, and others, automatically installed at mount time. You don't need to learn the filesystem interface. Just ask: -```bash -# Install (macOS requires no dependencies; Linux needs fuse3) -curl -fsSL https://install.tigerfs.io | sh -``` +- "Create a workspace for my notes" +- "What changed since the savepoint?" +- "Undo agent-7's changes" +- "Show me the last 10 orders by customer 123" -| Mode | You have... | You want to... | -|------|------------|----------------| -| File-first | A new project or workflow | Store markdown or other files, and build simple apps via the file system (e.g., task queues, agent workspaces, collaborative docs). | -| Data-first | An existing Postgres database | Explore and operate on it with `ls`, `cat`, `grep` instead of SQL. | +The skills teach your agent the filesystem paths, diff commands, and undo workflows. For details on what's underneath, read on. -### File-first +### Install ```bash -# Mount a database and create a markdown app -tigerfs mount postgres://localhost/mydb /mnt/db -echo "markdown,history" > /mnt/db/.build/notes +curl -fsSL https://install.tigerfs.io | sh +``` -# Write a file — frontmatter becomes columns, body becomes text -cat > /mnt/db/notes/hello.md << 'EOF' ---- -title: Hello World -author: alice ---- -# Hello World -EOF +**New project?** Start file-first. **Existing database?** Start data-first. -# Search, explore, unmount -grep -l "author: alice" /mnt/db/notes/*.md -ls /mnt/db/notes/ -tigerfs unmount /mnt/db -``` +### The Filesystem is the API -### Data-first +Your data lives in regular files and directories. Metadata, queries, and operations live in dot-directories: invisible by default, always available. ```bash -# Mount an existing database -tigerfs mount postgres://localhost/mydb /mnt/db +$ ls /mnt/db/notes/ +hello.md tutorials/ + +$ ls -a /mnt/db/notes/ +. .. .history/ .log/ .savepoint/ .undo/ hello.md tutorials/ +``` -ls /mnt/db/ # list tables -ls /mnt/db/users/ # list rows -cat /mnt/db/users/1.json # read a row -cat /mnt/db/users/.by/email/alice@co.com.json # index lookup +Dot-directories are the control surface. Navigate them to browse history, filter data, undo changes, and manage schemas, all through the same filesystem interface. -# Chain filters, ordering, pagination — pushed down as one SQL query -cat /mnt/db/orders/.by/customer_id/1/.order/created_at/.last/5/.export/json +``` +┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ +│ Unix Tools │────▶│ Filesystem │────▶│ TigerFS │────▶│ PostgreSQL │ +│ ls, cat, │ │ Backend │ │ Daemon │ │ Database │ +│ echo, rm │◀────│ (FUSE/NFS) │◀────│ │◀────│ │ +└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘ ``` +| File-first | Data-first | +|------------|------------| +| `.history/` past versions | `.info/` table metadata | +| `.log/` operation log with diffs | `.by/` index lookups | +| `.savepoint/` bookmarks for undo | `.filter/` column filtering | +| `.undo/` preview and apply undo | `.order/`, `.first/`, `.last/` sort and paginate | +| `.build/` create workspaces | `.export/`, `.import/` bulk I/O | +| | `.create/`, `.modify/`, `.delete/` schema management | + ## File-First: Transactional Workspace -### Apps +### Workspaces -Apps tell TigerFS how to present a table as a native file format. Write "markdown" to `.build/` and the table becomes a directory of .md files with YAML frontmatter: +Workspaces tell TigerFS how to present a table as a native file format. Write "markdown" to `.build/` and the table becomes a directory of .md files with YAML frontmatter: ```bash -# Create a markdown app -echo "markdown" > /mnt/db/.build/blog +# Mount a database and create a workspace with history +tigerfs mount postgres://localhost/mydb /mnt/db +echo "markdown,history" > /mnt/db/.build/blog # Write a post. Frontmatter becomes columns, body becomes text cat > /mnt/db/blog/hello-world.md << 'EOF' @@ -99,35 +99,26 @@ mkdir /mnt/db/blog/tutorials mv /mnt/db/blog/hello-world.md /mnt/db/blog/tutorials/ ``` -See [docs/markdown-app.md](docs/markdown-app.md) for column mapping, frontmatter handling, and use cases. +See [docs/file-first.md](docs/file-first.md) for column mapping, frontmatter handling, and use cases. -### Version History +### History, Savepoints, and Undo -Any app can opt into automatic versioning. Every edit and delete is captured as a timestamped snapshot under a read-only `.history/` directory. - -To enable automatic versioning, write "history" to `.build/` when creating the app: +Any workspace can opt into automatic versioning. Every edit and delete is captured as a timestamped snapshot. Create savepoints before risky work, preview what changed, and undo if needed. ```bash -# Create an app with history enabled -echo "markdown,history" > /mnt/db/.build/notes - -# Browse past versions of a file -ls /mnt/db/notes/.history/hello.md/ -# .id 2026-02-24T150000Z 2026-02-12T013000Z - -# Read a past version -cat /mnt/db/notes/.history/hello.md/2026-02-12T013000Z +echo '{"description":"Before refactoring"}' > /mnt/db/notes/.savepoint/checkpoint.json +# ... Perform refactoring ... +diff -u /mnt/db/notes/.log//before /mnt/db/notes/.log//current # Compare changes to single file +diff -ru /mnt/db/notes/.undo/to-savepoint/checkpoint /mnt/db/notes/ -x '.*' # Review all changes +touch /mnt/db/notes/.undo/to-savepoint/checkpoint/.apply # Undo all changes since savepoint ``` -History tracks files across renames via stable row UUIDs and uses TimescaleDB hypertables for compressed storage. - -See [docs/history.md](docs/history.md) for cross-rename tracking, subdirectory scoping, and recovery workflows. +Per-user undo, single-file undo, and full version history are also available. See [docs/history.md](docs/history.md) for the full guide. ### Use Cases **Shared agent workspace.** -Multiple agents and humans operate on the same knowledge base concurrently. -Every edit is automatically versioned, so if one agent overwrites another's work, recover it from `.history/`. +Multiple agents and humans operate on the same knowledge base concurrently. Changes are visible instantly. Every edit is automatically versioned, so if one agent overwrites another's work, browse the full edit trail in `.history/` and recover it. ```bash # Agent A writes research findings @@ -140,6 +131,9 @@ EOF # Agent B reads it immediately, no sync, no pull cat /mnt/db/kb/auth-analysis.md + +# Browse the full edit trail +ls /mnt/db/kb/.history/auth-analysis.md/ ``` **Multi-agent task queue.** Three directories (`todo/`, `doing/`, `done/`) and `mv` is your only API. Moves are atomic database operations, so two agents can't claim the same task. @@ -160,38 +154,37 @@ ls /mnt/db/tasks/doing/ grep "author:" /mnt/db/tasks/doing/*.md ``` -**Collaborative docs.** A human writes a draft, an agent reviews and edits it, another agent summarizes it. All in the same directory, all visible immediately, no pull/push/merge. History shows who changed what and when. +**Safe exploration.** An agent creates a savepoint, investigates a bug, makes changes across multiple files. If the approach doesn't work, undo atomically to the savepoint. Every file reverts in one step. ```bash -# Human writes a draft -cat > /mnt/db/docs/proposal.md << 'EOF' ---- -title: Q2 Proposal -status: draft ---- -We should invest in... -EOF +# Agent creates savepoint before investigating +echo '{"description":"Before investigating auth bug"}' > /mnt/db/notes/.savepoint/pre-investigation.json -# Agent reads, edits, and updates the status -cat /mnt/db/docs/proposal.md -cat > /mnt/db/docs/proposal.md << 'EOF' ---- -title: Q2 Proposal -status: reviewed -reviewer: agent-b ---- -We should invest in... (with agent edits) -EOF - -# Human sees changes instantly. Browse the full edit trail -ls /mnt/db/docs/.history/proposal.md/ -cat /mnt/db/docs/.history/proposal.md/2026-02-25T100000Z # see previous version +# Agent explores, edits multiple files... +# User reviews: "that's not right, roll it back" +touch /mnt/db/notes/.undo/to-savepoint/pre-investigation/.apply +# All files restored to pre-investigation state ``` ## Data-First: Database as Filesystem Mount any Postgres database and explore it with `ls`, `cat`, `grep`. Every path resolves to optimized SQL pushed down to the database. +``` + Filesystem Database + ────────── ──────── + /mnt/db/ → tables (default schema) + /mnt/db/users/ → rows (by PK) + /mnt/db/users/123/ → columns as files + /mnt/db/.schemas/ → all schemas (including default) +``` + +```bash +$ ls -a /mnt/db/users/ +. .. .by/ .filter/ .order/ .first/ .last/ .info/ .export/ .import/ +1/ 2/ 3/ 4/ 5/ ... +``` + **Explore an unfamiliar database.** Point an agent at a mounted database and it understands the schema immediately using `ls` and `cat`. No SQL, no database client, no connection strings to pass around. **Quick data fixes.** Update a customer's email, toggle a feature flag, delete a test record. One shell command instead of opening a SQL client, remembering the table schema, and writing a `WHERE` clause. @@ -201,6 +194,9 @@ Mount any Postgres database and explore it with `ls`, `cat`, `grep`. Every path ### Explore ```bash +# Mount an existing database +tigerfs mount postgres://localhost/mydb /mnt/db + ls /mnt/db/ # List tables ls /mnt/db/users/ # List rows (by primary key) cat /mnt/db/users/123.json # Row as JSON @@ -212,7 +208,7 @@ cat /mnt/db/users/.by/email/foo@example.com.json # Index lookup ```bash echo 'new@example.com' > /mnt/db/users/123/email.txt # Update column -echo '{"email":"a@b.com","name":"A"}' > /mnt/db/users/ 123.json # Update via JSON (PATCH) +echo '{"email":"a@b.com","name":"A"}' > /mnt/db/users/123.json # Update via JSON (PATCH) mkdir /mnt/db/users/456 # Create row rm -r /mnt/db/users/456/ # Delete row ``` @@ -248,7 +244,7 @@ mkdir /mnt/db/.create/orders && echo "CREATE TABLE orders (...)" > /mnt/db/.crea touch /mnt/db/.create/orders/.commit ``` -See [docs/native-tables.md](docs/native-tables.md) for the full reference: row formats, index navigation, pipeline query chaining, schema management workflows, and configuration. +See [docs/data-first.md](docs/data-first.md) for the full reference: row formats, index navigation, pipeline query chaining, schema management workflows, and configuration. ## Why TigerFS @@ -309,37 +305,12 @@ tigerfs info /mnt/db tigerfs info --json /mnt/db # JSON output for scripting ``` -## Architecture - -``` -┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ -│ Unix Tools │────▶│ Filesystem │────▶│ TigerFS │────▶│ PostgreSQL │ -│ ls, cat, │ │ Backend │ │ Daemon │ │ Database │ -│ echo, rm │◀────│ (FUSE/NFS) │◀────│ │◀────│ │ -└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘ -``` - -TigerFS replaces "application-level coordination" with database transactions. **The filesystem becomes the API.** - -TigerFS maps filesystem paths to database queries: - -``` - Filesystem Database - ────────── ──────── - /mnt/db/ → schemas - /mnt/db/public/ → tables - /mnt/db/public/users/ → rows (by PK) - /mnt/db/public/users/123/ → columns as files -``` - -FUSE on Linux, NFS on macOS. No external dependencies on either platform. - ## Design Principles - **Keep the interface familiar.** If you can `ls`, you can explore a database. - **Make concurrency safe.** Multiple writers without corruption or conflicts. - **Push logic down.** Every path resolves to optimized SQL. -- **Preserve history.** Every change is recoverable. +- **Make changes reversible.** Savepoints, undo, and version history mean you can always go back. - **Remove coordination code.** The database handles it. ## Try the Demo @@ -359,9 +330,9 @@ Config file: `~/.config/tigerfs/config.yaml`. Run `tigerfs config show` to see a | Guide | Description | |-------|-------------| -| [docs/markdown-app.md](docs/markdown-app.md) | Markdown app: column mapping, frontmatter, directories | -| [docs/history.md](docs/history.md) | Version history: snapshots, cross-rename tracking, recovery | -| [docs/native-tables.md](docs/native-tables.md) | Native table access: row formats, indexes, pipeline queries, schema management | +| [docs/file-first.md](docs/file-first.md) | File-first mode: workspaces, column mapping, frontmatter, directories | +| [docs/history.md](docs/history.md) | History, savepoints, and undo: versioned snapshots, safe exploration, atomic rollback | +| [docs/data-first.md](docs/data-first.md) | Data-first mode: row formats, indexes, pipeline queries, schema management | | [docs/quickstart.md](docs/quickstart.md) | Guided scenarios with sample data | ## Development @@ -379,20 +350,28 @@ For development guidelines, architecture details, and the full specification, se TigerFS is early, but the core idea is stable: transactional, concurrent files as the foundation for human-agent collaboration. -**v0.5.0.** Performance and observability — dramatically fewer SQL queries, flexible logging, and column projection. +**v0.7.0.** Undo and recovery: savepoints, operation log, and atomic undo for safe exploration. + +**v0.6.0.** Dedicated tigerfs schema, security hardening, and unified demo. **Highlights:** -- Markdown apps with YAML frontmatter, directory hierarchies, and automatic version history +- Markdown and plaintext workspaces with YAML frontmatter, directory hierarchies, and version history +- Savepoints, undo, and operation log: create checkpoints, preview changes, roll back atomically +- Per-user undo: multiple agents with separate identities, selectively undo one agent's work +- Auto-savepoints: detect session boundaries on inactivity gaps +- Relational directory model with parent-pointer hierarchy and UUIDv7 identifiers +- Dedicated tigerfs schema with migration framework (`tigerfs migrate`) +- TLS enforcement, SQL injection hardening, and credential sanitization +- Agent skill auto-install for Claude Code, Gemini CLI, and Codex - Cloud backends: mount, create, and fork Tiger Cloud and Ghost databases by service ID - Pipeline queries with full database pushdown (`.by/`, `.filter/`, `.order/`, `.columns/`, chained pagination, `.export/`) - DDL staging for tables, indexes, views, and schemas (`.create/`, `.modify/`, `.delete/`) - Full CRUD with multiple formats (TSV, CSV, JSON, YAML), index navigation, and PATCH semantics -- Binary distribution via GoReleaser with install script (`curl -fsSL https://install.tigerfs.io | sh`) +- Binary distribution via GoReleaser with install script - Multi-tier stat caching and query reduction for fast operations over remote databases **Planned:** - Tables without primary keys (read-only via ctid) -- TimescaleDB hypertables (time-based navigation) - Windows support ## Contributing diff --git a/docs/adr/005-capability-directory-taxonomy.md b/docs/adr/005-capability-directory-taxonomy.md index 038a86b..9df5e03 100644 --- a/docs/adr/005-capability-directory-taxonomy.md +++ b/docs/adr/005-capability-directory-taxonomy.md @@ -170,12 +170,9 @@ Considered alternatives: ├── .columns/ # capability: column projection (pipeline query) │ └── id,name,price/ │ └── .export/ -├── .first/ # capability: first N rows -│ └── 100/ -├── .last/ # capability: last N rows -│ └── 100/ -├── .sample/ # capability: random sample -│ └── 100/ +├── .first/N/ # capability: first N rows (navigate directly, e.g. .first/100/) +├── .last/N/ # capability: last N rows (navigate directly, e.g. .last/100/) +├── .sample/N/ # capability: random sample (navigate directly, e.g. .sample/100/) ├── .order/ # capability: specify row ordering (see ADR-006) │ └── price/ # (dynamic: any column name) ├── .export/ # capability: bulk read (see ADR-006) @@ -190,7 +187,7 @@ Considered alternatives: │ │ └── csv, tsv, json, yaml │ └── .append/ │ └── csv, tsv, json, yaml -├── .all/ # capability: all rows iterator +├── .all/ # capability: all rows (hidden from ls to prevent infinite recursion) ├── .indexes/ # capability: index DDL management │ └── .create/ ├── .delete/ # capability: table delete staging diff --git a/docs/adr/006-bulk-export-capability.md b/docs/adr/006-bulk-export-capability.md index e60dba7..8ac1fd9 100644 --- a/docs/adr/006-bulk-export-capability.md +++ b/docs/adr/006-bulk-export-capability.md @@ -189,7 +189,7 @@ email: bob@example.com ### Read Semantics (Export) **Size limits:** -- Limited to `DirListingLimit` rows (default 10,000) +- Limited to `DirListingLimit` rows (default 1,000) - For larger exports, use explicit pagination: `.first/10000/.export/csv` - Error message suggests pagination if limit exceeded diff --git a/docs/adr/011-directory-hierarchies.md b/docs/adr/011-directory-hierarchies.md index cfaac3e..dd710c9 100644 --- a/docs/adr/011-directory-hierarchies.md +++ b/docs/adr/011-directory-hierarchies.md @@ -1,6 +1,6 @@ # ADR-011: Directory Hierarchies in Synthesized Apps -**Status:** Accepted +**Status:** Superseded by [ADR-017](017-relational-directory-structure.md) **Date:** 2026-02-12 **Author:** Mike Freedman diff --git a/docs/adr/016-undo-and-recovery.md b/docs/adr/016-undo-and-recovery.md new file mode 100644 index 0000000..fcdebf0 --- /dev/null +++ b/docs/adr/016-undo-and-recovery.md @@ -0,0 +1,1184 @@ +# ADR-016: Undo and Recovery + +**Status:** Accepted +**Date:** 2026-04-07 +**Author:** Mike Freedman + +## Context + +TigerFS users and agents make changes to files backed by PostgreSQL. When mistakes happen -- a user edits the wrong file, or an agent makes a series of exploratory changes during debugging -- there's no way to revert. Today, recovery depends on either the agent manually undoing its changes or having a clean git commit to fall back on. + +TigerFS already has a **history** system for synth apps: a PostgreSQL BEFORE trigger captures the old row state into a companion hypertable on every UPDATE/DELETE. This design builds on that foundation to add structured undo operations, savepoints, and an operation log -- all exposed through the filesystem. + +**Scope:** This feature applies only to synth app tables with history enabled. Native/data-first tables and synth apps without history are not affected. + +--- + +## 1. Operation Log + +### 1.1 Purpose + +The operation log records every data change (create, edit, rename, delete, undo) made to a synth app table. It provides: + +- An audit trail of what changed, when, and by whom +- The ordering information needed to undo operations +- Pointers to the history table for retrieving before-states + +### 1.2 Schema + +One log table per synth app with history enabled, stored in the `tigerfs` schema (hidden from top-level `ls`): + +```sql +CREATE TABLE tigerfs._log ( + log_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY, + file_id UUID NOT NULL, + type TEXT NOT NULL CHECK (type IN ('create', 'edit', 'rename', 'delete', 'undo')), + user_id TEXT, + filename TEXT NOT NULL, + version_id UUID, + description TEXT +) WITH ( + tsdb.hypertable, + tsdb.partition_column = 'log_id', + tsdb.chunk_interval = '7 days', + tsdb.segmentby = 'file_id', + tsdb.orderby = 'log_id ASC' +); +``` + +| Column | Description | +|--------|-------------| +| `log_id` | UUIDv7 primary key. Time-ordered, used as hypertable partition key. Timestamp extractable via TimescaleDB's `uuid_v7_to_timestamptz()` (works on PG17; PG18+ has built-in `uuid_extract_timestamp()`). | +| `file_id` | Stable UUID of the affected row (the `id` column in the synth app table). Persists across renames and moves. | +| `type` | Operation type. Filesystem-centric names (ADR-017): `create` (new file/directory), `edit` (content change), `rename` (name or parent_id change), `delete` (removal); `undo` for undo operations. | +| `user_id` | Identity of the user/agent making the change. NULL in single-user/anonymous mode. Set from `.info/user` at the mount root. | +| `filename` | Denormalized full path at the time of the operation (e.g., `projects/web/todo.md`). Computed from the parent-pointer chain at log-write time. Historically correct -- records what the file was called when the operation happened, even if later renamed or moved. Note: the log's `filename` stores the full path; the source table's `filename` stores only the leaf name (ADR-017). | +| `version_id` | Pointer to the history table row (`version_id`) containing the before-state. NULL for `create` operations (no before-state exists). For `edit`/`rename`/`delete`/`undo`, this points to the row captured by the BEFORE trigger. | +| `description` | Optional human-readable note about the operation. | + +### 1.3 Hypertable Configuration + +The log table uses the modern `CREATE TABLE WITH` syntax (see schema above) which combines hypertable creation, chunk interval, segmentby, and orderby in a single DDL statement. TimescaleDB automatically creates a columnstore compression policy matching the chunk interval. + +### 1.4 Indexes + +A composite index on `(file_id, log_id)` enables SkipScan for the undo query pattern: + +```sql +CREATE INDEX ON tigerfs._log (file_id, log_id ASC); +``` + +The `file_id` leading column enables SkipScan for the `DISTINCT ON (file_id) ... ORDER BY file_id, log_id ASC` undo query pattern. + +### 1.5 Compression + +Compression is configured inline via the `CREATE TABLE WITH` syntax: `tsdb.segmentby = 'file_id'` and `tsdb.orderby = 'log_id ASC'`. TimescaleDB automatically creates a compression policy using the chunk interval. + +**Design decisions:** + +- **`segmentby = 'file_id'`**: Each file gets its own compressed segment. This aligns with SkipScan on compressed hypertables -- SkipScan jumps between segments by `file_id`, reading only the first matching entry per segment. The trade-off is slightly lower compression ratio for tables with many files (many small segments), but log rows are small (UUIDs and short text), so this is acceptable. + +- **`orderby = 'log_id ASC'`**: Matches the `ORDER BY file_id, log_id ASC` in the undo query, allowing SkipScan to work on compressed chunks without reordering. + +### 1.6 How Log Entries Are Created + +Log entries are created by TigerFS's write path, not by database triggers. When `WriteFile`, `Delete`, or `Rename` executes a DML operation on a history-enabled synth app table, it also inserts a log entry in the same database round-trip (or transaction, for undo operations). + +The log entry captures: +- The `file_id` from the row being modified +- The `filename` as the denormalized full path (computed from the parent-pointer chain) +- The `version_id` from the history entry created by the BEFORE trigger (for edit/rename/delete). For `create`, `version_id` is NULL because there is no before-state. + +**Determining `version_id` for edit/rename/delete:** The BEFORE trigger inserts a row into the history table with a new UUIDv7 `version_id`. To capture this in the log, the write operation queries the most recent history entry for the file immediately after the DML. Since the trigger fires synchronously before the DML completes, the history row exists by the time the log insert runs. + +### 1.7 Relationship Between Log and History + +The log and history tables serve complementary purposes: + +| | History Table | Log Table | +|---|---|---| +| **What it stores** | Full row state (before-state of every change) | Operation metadata (what happened, who, when, pointer to before-state) | +| **Created by** | PostgreSQL BEFORE trigger (automatic) | TigerFS write path (application-level) | +| **Used for** | Reading old file content, restoring state | Ordering operations, undo sequencing, audit trail | +| **Schema** | Matches the synth app's columns + `version_id`, `operation` | Fixed schema (log_id, file_id, type, user_id, filename, version_id, description) | + +The log does not duplicate the file content. It stores a `version_id` pointer to the history row that contains the full before-state. This keeps the log table small (just UUIDs and metadata) while leveraging the history table's existing compressed storage for file content. + +### 1.8 SkipScan Optimization + +TimescaleDB's SkipScan is a custom executor node that transforms `DISTINCT ON` queries from O(N) to O(K x log N), where K is the number of distinct values and N is total rows. It works by hopping through the B-tree index from one distinct value to the next, rather than scanning every row. + +The undo execution query uses `DISTINCT ON (file_id)` to find the first log entry per affected file: + +```sql +SELECT DISTINCT ON (file_id) file_id, type, version_id +FROM tigerfs._log +WHERE log_id > $1 +ORDER BY file_id, log_id ASC +``` + +With the `(file_id, log_id ASC)` index and `segmentby = 'file_id'`, SkipScan activates on both uncompressed and compressed chunks. + +The same `DISTINCT ON` query is used for both the undo execution and the preview summary (Section 4.3.8). One query, one code path, SkipScan throughout. + +**Requirements:** +- TimescaleDB >= 2.20.0 (for compressed-chunk SkipScan) +- PostgreSQL >= 16 +- The `(file_id, log_id ASC)` index with `file_id` as the leading column +- `segmentby = 'file_id'` for compressed chunks + +**Verification:** Run `EXPLAIN ANALYZE` on the undo query and look for `Custom Scan (SkipScan)` nodes. + +### 1.9 Why No `after_id` / `after_state` + +The log stores only a `version_id` pointing to the before-state. There is no `after_id` pointing to the after-state. This is a deliberate design choice: + +1. **The after-state doesn't exist as a history entry at log-write time.** When an UPDATE executes, the BEFORE trigger fires and creates a history entry (the before-state) -- we know its `version_id`. But the after-state is just the live row in the current table. It won't become a history entry until the *next* operation's trigger fires. So `after_id` can't be populated at the time the log entry is created without either backfilling it retroactively (fragile) or creating an extra history entry (wasteful). + +2. **The before-state is sufficient for undo.** Undo restores the before-state. The after-state is never needed for the restore operation itself. + +3. **The after-state is derivable from the chain.** For any log entry L, the after-state is either the current row (if L is the latest operation) or the next log entry's `version_id` (the next operation's before-state IS the previous operation's after-state). + +--- + +## 2. Savepoints + +### 2.1 Purpose + +Savepoints are named bookmarks in time. They let users mark a point ("before-agent-run") and later say "undo everything after this point." Savepoints are cheap to create and carry no operational overhead. + +### 2.2 Schema + +Savepoints are stored in a **separate table** from the log (not as log entries with `type = 'savepoint'`). One savepoint table per synth app with history enabled: + +```sql +CREATE TABLE tigerfs._savepoint ( + name TEXT NOT NULL PRIMARY KEY, + savepoint_id UUID NOT NULL DEFAULT uuidv7() UNIQUE, + user_id TEXT, + description TEXT +); +``` + +`name` is the PK so standard filesystem operations (list, read, write, delete) work without savepoint-specific code. `savepoint_id` (UUIDv7) is retained with a UNIQUE constraint for undo time-ordering (`log_id > savepoint_id`). + +This is a regular PostgreSQL table, not a hypertable. Savepoints are small (tens to hundreds, not millions) and don't need time-series features. + +**Why separate from the log:** + +1. **Clean schemas.** The log has `file_id`, `version_id`, `filename` (operation-specific). Savepoints have `name`, `description` (bookmark-specific). Combining them means every row wastes half its columns as NULLs. + +2. **Natural constraints.** `name UNIQUE` is straightforward on a dedicated table. On the log table, it would require a partial unique index (`WHERE type = 'savepoint'`). + +3. **No compression concerns.** The savepoint table is tiny and doesn't need hypertable features. Mixing savepoints into a compressed hypertable segmented by `file_id` is awkward because savepoints have no `file_id`. + +4. **Simpler queries.** `SELECT * FROM notes_savepoint WHERE name = 'foo'` vs `SELECT * FROM notes_log WHERE type = 'savepoint' AND name = 'foo'`. + +**UUIDv7 ordering is preserved across tables.** Both the log and savepoint tables use `uuidv7()` defaults generated from the same clock. A savepoint created at time T has a `savepoint_id` less than any log entry created after T. So `WHERE log_id > savepoint_id` correctly identifies "operations after this savepoint." + +### 2.3 Filesystem Interface + +Savepoints are exposed as a data-first directory under each synth app table. The `.savepoint/` path is an alias for the `tigerfs._savepoint` table, using existing pipeline machinery. + +**Creating a savepoint:** +```bash +echo -e "description\nStarting agent exploration" > notes/.savepoint/before-exploration.tsv +# Inserts: name='before-exploration', savepoint_id=uuidv7(), description='Starting agent exploration' + +echo '{}' > notes/.savepoint/quick-mark.json +# Inserts: name='quick-mark', savepoint_id=uuidv7(), description=NULL +``` + +**Listing savepoints:** +```bash +ls notes/.savepoint/ +# before-exploration/ +# quick-mark/ +``` + +**Reading savepoint data (rows-as-directories):** +```bash +cat notes/.savepoint/before-exploration/description +# Starting agent exploration + +cat notes/.savepoint/before-exploration/savepoint_id +# 019590a0-1234-7000-8000-000000000001 + +cat notes/.savepoint/before-exploration.json +# {"savepoint_id":"019590a0-...","name":"before-exploration","description":"Starting agent exploration","user_id":null} +``` + +**Updating:** +```bash +echo "Updated description" > notes/.savepoint/before-exploration/description +# Updates the description column + +mv notes/.savepoint/before-exploration notes/.savepoint/before-foo +# UPDATE notes_savepoint SET name='before-foo' WHERE name='before-exploration' +# The savepoint_id is unchanged, so undo-to-savepoint references still work +``` + +**Deleting:** +```bash +rm notes/.savepoint/before-exploration +# DELETE FROM notes_savepoint WHERE name='before-exploration' +# Only removes the bookmark. All log entries remain intact. +# Undo is still possible by log_id, just not by this savepoint name. +``` + +**Pipeline queries (reusing data-first infrastructure):** +```bash +ls notes/.savepoint/.last/5 # 5 most recent +ls notes/.savepoint/.by/user_id/agent-7 # by user +cat notes/.savepoint/.export/json # all as JSON +cat notes/.savepoint/*.json # all as JSON (format suffix) +``` + +### 2.4 Implementation: Reusing Data-First Mode + +When the path parser sees `.savepoint/`, it creates an `FSContext` targeting `tigerfs._savepoint` with `name` as the display filename. All pipeline operations (`.by/`, `.last/`, `.first/`, `.order/`, `.export/`, `.columns/`, format suffixes) are handled by existing data-first pipeline code. + +The only special behaviors: +- The directory uses `name` as the row identifier (filename), not the PK (`savepoint_id`) +- Write operations (create, update description, rename, delete) map to INSERT/UPDATE/DELETE on the savepoint table + +--- + +## 3. Undo Operations + +### 3.1 Semantics: Rollback, Not Revert + +Undo operations use **rollback semantics** (like `git checkout`), not **revert semantics** (like `git revert`). + +- **Rollback:** Restore a file (or all files) to a specific prior state. Discards all changes after that point for the affected files. +- **Revert:** Surgically reverse one specific change while preserving later changes. Requires column-level diffing and three-way merge logic. + +Rollback is the right model for TigerFS because: +1. History stores full row state, not column-level diffs. Merging partial changes would require complex diff/merge logic. +2. The primary use cases ("undo a mistake," "restore after agent exploration") are rollback operations. +3. Surgical revert can be done at the tool/agent level if needed -- read the history versions, compute the diff, apply a selective edit. + +### 3.2 Undo Variants + +| Variant | Path | What it does | +|---------|------|-------------| +| Single operation | `.undo/id//` | Undo one operation on one file | +| To a log entry | `.undo/to-id//` | Undo all operations after this log entry | +| To a savepoint | `.undo/to-savepoint//` | Undo all operations after this savepoint | +| To a savepoint by user | `.undo/to-savepoint//.by/user_id//` | Undo only this user's operations after the savepoint | +| To a log entry by user | `.undo/to-id//.by/user_id//` | Undo only this user's operations after this log entry | + +### 3.3 How Undo Works + +**Single operation undo (`.undo/id//`):** + +1. Look up the log entry by `log_id` +2. Based on `type`: + - `create`: DELETE the row (it didn't exist before) + - `edit`: Fetch the before-state from history (`version_id`), UPDATE the row to that state + - `rename`: Fetch the before-state from history, UPDATE the row to restore old name/parent + - `delete`: Fetch the before-state from history, INSERT the row back + - `undo`: Same as edit/delete -- fetch history state, restore it +3. The BEFORE trigger fires on the restore operation, capturing the current state into history +4. Insert a new log entry with `type = 'undo'` + +**Multi-file undo (to savepoint or to log_id):** + +The execution query finds the first log entry per affected file after the target point, using `DISTINCT ON` to leverage SkipScan (see Section 1.8): + +```sql +SELECT DISTINCT ON (file_id) file_id, type, version_id +FROM tigerfs._log +WHERE log_id > $1 +ORDER BY file_id, log_id ASC +``` + +Where `$1` is either the `savepoint_id` (for undo-to-savepoint) or the target `log_id` (for undo-to-ID). + +**Key insight:** The first log entry's `version_id` for each file IS the state at the target point, by definition. The BEFORE trigger captured the row state just before the first post-target operation. + +For each affected file: + +| First operation type | Current row state | Action | +|---------------------|-------------------|--------| +| `create` | Exists | DELETE (row didn't exist at target point) | +| `create` | Doesn't exist (deleted later) | No-op (correct -- row didn't exist at target) | +| `edit`/`rename` | Exists | UPSERT from history state | +| `edit`/`rename` | Doesn't exist (deleted later) | INSERT from history state | +| `delete` | Doesn't exist | INSERT from history state | + +The UPSERT pattern handles the ambiguity of "does the row currently exist?" for `edit`/`rename` entries: + +```sql +-- Delete rows inserted after the target point +DELETE FROM public. +WHERE id IN (SELECT file_id FROM affected WHERE first_type = 'create'); + +-- Restore rows that were edited, renamed/moved, or deleted after the target point. +-- UPSERT must restore parent_id (ADR-017) to reverse moves. +INSERT INTO public. (id, parent_id, filename, title, author, body, ...) +SELECT h.file_id, h.parent_id, h.filename, h.title, h.author, h.body, ... +FROM tigerfs._history h +JOIN affected a ON a.first_version_id = h.version_id +WHERE a.first_type IN ('edit', 'rename', 'delete') +ON CONFLICT (id) DO UPDATE SET + parent_id = EXCLUDED.parent_id, + filename = EXCLUDED.filename, + title = EXCLUDED.title, + author = EXCLUDED.author, + body = EXCLUDED.body, + ...; +``` + +All executed in a **single PostgreSQL transaction**. The BEFORE triggers fire for each restore operation, creating new history entries. New log entries with `type = 'undo'` are also inserted within the transaction. + +**By-user variant:** Same query with an additional `WHERE user_id = $2` filter. Only operations by the specified user are undone; other users' changes are preserved. + +### 3.4 Undo of Undo + +Undo operations are themselves logged (with `type = 'undo'`). Each undo operation fires the BEFORE trigger, which captures the current state into history. The undo log entry's `version_id` points to that captured state. + +To undo an undo: look up the undo log entry, fetch its `version_id` from history, restore to that state. This is identical to undoing any other operation -- no special case needed. + +**Traced example:** + +1. User updates `hello.md` to "v2". Trigger captures "v1" as H1. Log: L1 (type=edit, version_id=H1). +2. User undoes L1. TigerFS restores "v1" from H1. Trigger captures "v2" as H2. Log: L2 (type=undo, version_id=H2). +3. User undoes L2 (undo-of-undo). TigerFS restores "v2" from H2. Trigger captures "v1" as H3. Log: L3 (type=undo, version_id=H3). + +Result: `hello.md` is back to "v2". The chain is self-consistent because every write (including undo writes) fires the same trigger. + +### 3.5 Idempotent Undo-to-Savepoint + +Undoing to the same savepoint twice produces the same data state. On the second undo: + +- The query finds all operations after the savepoint, including the undo entries from the first undo +- For each file, the first entry after the savepoint is still the original operation (same `log_id`) +- That original operation's `version_id` still points to the state at the savepoint +- The restore produces the same data (though it creates new log/history entries) + +This is correct and expected -- "undo to savepoint S" always means "make the data look like it did at S." + +### 3.6 Transaction Safety and Crash Recovery + +The entire undo operation (all DELETEs, UPSERTs, and log INSERTs) runs in a single PostgreSQL transaction. If TigerFS crashes mid-undo, PostgreSQL rolls back the transaction automatically. The database stays in the pre-undo state. The user can retry. + +For typical agent sessions (tens to hundreds of operations on tens of files), the transaction is small. For very large undo operations (thousands of files), the transaction may hold many row locks and use significant memory, but this is an unusual case. + +### 3.7 Concurrency During Undo + +If an undo transaction and a concurrent write both modify the same row, PostgreSQL serializes at the row level. The second writer blocks until the first commits. + +**Risk:** The undo can overwrite a concurrent change, or a concurrent change can overwrite the restored state. This is inherent in rollback semantics -- the same as any two concurrent writes to the same row. + +**Mitigation:** Agents should use separate mounts (see Section 5) so their writes are independently tracked. "Undo to savepoint by user" only reverses one user's operations, preserving others. True conflict detection (optimistic locking) could be added later if needed. + +--- + +## 4. Filesystem Interfaces + +### 4.1 `.log/` -- Operation Log + +The `.log/` directory exposes the `tigerfs._log` table through data-first mode. When the path parser sees `.log/`, it creates an `FSContext` targeting the log table and delegates to existing pipeline code. + +```bash +# Recent operations (UUIDv7 PKs, so .last gives most recent) +ls notes/.log/.last/10 +cat notes/.log/.last/10.json + +# Filter by user +ls notes/.log/.by/user_id/agent-7/.last/20 + +# Filter by file +ls notes/.log/.by/file_id/ + +# Filter by operation type +ls notes/.log/.by/type/delete + +# Single entry as directory (column access) +cat notes/.log//type +cat notes/.log//filename +cat notes/.log//version_id + +# Bulk export +cat notes/.log/.export/json +``` + +All pipeline operations (`.by/`, `.last/`, `.first/`, `.order/`, `.export/`, `.columns/`, format suffixes) work through existing data-first pipeline code. No new functionality needed. + +#### 4.1.1 Diff Symlinks on Log Entries + +Each log entry (as a row-as-directory) exposes three symlinks alongside its data columns: + +``` +notes/.log// +├── log_id # column +├── type # column +├── filename # column +├── file_id # column +├── version_id # column +├── description # column +├── before -> ../../.history/docs/hello.md/2026-04-07T143000.123Z-zzz0063hd8e5r42 # symlink +├── after -> ../../.history/docs/hello.md/2026-04-07T150000.456Z-1230deadbeef1z0 # symlink +└── current -> ../../docs/hello.md # symlink +``` + +**Resolution rules:** + +| Symlink | Points to | +|---------|-----------| +| `before` | `.history//` derived from `version_id`. If `version_id` is NULL (`create`): `/dev/null`. | +| `after` | Query next log entry for this `file_id` after this `log_id`. If found and its `version_id` is non-NULL: `.history//` from that entry. If found and its `version_id` is NULL (next op was a `create`): current file path. If no next entry and file exists: current file path. If no next entry and file deleted: `/dev/null`. | +| `current` | The live file path `../../`. If the file has been deleted: `/dev/null`. | + +The `after` lookup uses the existing `(file_id, log_id ASC)` index -- a single index seek, sub-millisecond even on compressed hypertables. + +**Why `/dev/null`:** When a file doesn't exist (before a `create`, after a `delete`), the symlink points to `/dev/null`. This makes `diff` produce the right output with zero special cases: + +```bash +# create: shows entire file as added +diff -u --color notes/.log//before notes/.log//after + +# delete: shows entire file as removed +diff -u --color notes/.log//before notes/.log//after + +# edit/rename: shows the actual changes +diff -u --color notes/.log//before notes/.log//after +``` + +The script is always the same command. No case statements, no error handling for missing files. + +**Scripting examples:** + +```bash +# Diff a specific operation +diff -u --color notes/.log//before notes/.log//after + +# Diff all recent changes +for id in $(ls notes/.log/.last/10); do + echo "=== $id ===" + diff -u --color notes/.log/$id/before notes/.log/$id/after +done + +# What changed since this operation vs now? +diff -u --color notes/.log//after notes/.log//current +``` + +**Implementation note:** Symlink support was added in Task 12.2. The `Entry` struct has a `Target` field and `IsSymlink()` method. Both NFS (`Readlink()`) and FUSE (`OpsNode.Readlink()`) adapters delegate to `Operations.Readlink()`. Symlinks appear in two contexts: `.log//` directories (before, after, current diff symlinks) and `.undo/` preview directories (`/dev/null` symlinks for deleted files). + +**Full state matrix:** + +| Log entry type | `before` | `after` | `current` | +|---|---|---|---| +| `create` | `/dev/null` | history or current file | current file or `/dev/null` | +| `edit` | `.history/` version | history or current file | current file or `/dev/null` | +| `rename` | `.history/` version | history or current file | current file or `/dev/null` | +| `delete` | `.history/` version | `/dev/null` | `/dev/null` | +| `undo` | depends on `version_id` | depends on next entry | current file or `/dev/null` | + +**Simplified resolution:** The `type` column is not needed for symlink resolution. The two rules are: +- **`before`**: `version_id` is NULL → `/dev/null`. Otherwise → `.history//`. +- **`after`**: Find next log entry for same `file_id`. Use its `version_id` to point to `.history/`, or fall back to current file path, or `/dev/null` if file deleted. +- **`current`**: File exists in source table → live path. Doesn't exist → `/dev/null`. + +### 4.2 `.savepoint/` -- Savepoint Management + +See Section 2.3 for the full interface specification. + +### 4.3 `.undo/` -- Undo Operations + +The `.undo/` directory provides a **preview-then-apply** interface for undo operations. This approach was chosen over two alternatives: + +- **DDL-style staging** (`.test/.commit/.abort`): Rejected because DDL staging allows editing the SQL before committing, which is meaningless for undo -- the path fully specifies the operation, there is nothing to edit. +- **Direct write** (one-step trigger): Rejected because multi-file undo is destructive, and a single `echo` with no preview is too easy to execute accidentally. + +Each undo path is a virtual directory containing: + +- **`.info/summary`**: TSV listing of affected files with actions (supports format suffixes: `summary.json`, `summary.csv`, etc.) +- **Affected files**: Directory tree mirroring the synth app structure, containing only files that would change. `cat` on each file returns the restored content (the before-state from history). +- **`.apply`**: Touch or write to this file to execute the undo + +#### 4.3.1 Top-Level Structure + +```bash +ls notes/.undo/ +id/ +to-id/ +to-savepoint/ +``` + +Three visible routing directories, each representing a different undo mode: + +| Directory | Purpose | Lists | +|---|---|---| +| `id/` | Undo a single operation | Log entries (default 100 most recent) | +| `to-id/` | Undo all operations after a log entry | Log entries (default 100 most recent) | +| `to-savepoint/` | Undo all operations after a savepoint | Savepoints (default 100 most recent) | + +The default listing limit (100) is configurable via `--undo-list-limit` or `TIGERFS_UNDO_LIST_LIMIT`. + +#### 4.3.2 Pipeline Capabilities + +Each sub-directory supports the full data-first pipeline. `id/` and `to-id/` query the log table; `to-savepoint/` queries the savepoint table. + +| Capability | Example | Purpose | +|---|---|---| +| `.all/` | `.undo/id/.all/` | Full listing (no limit) | +| `.last/N/` | `.undo/id/.last/20/` | Last N entries | +| `.first/N/` | `.undo/id/.first/20/` | First N entries (oldest) | +| `.sample/N/` | `.undo/to-savepoint/.sample/10/` | Random N entries | +| `.by///` | `.undo/id/.by/user_id/agent-7/` | Filter by column | +| `.filter///` | `.undo/id/.filter/type/delete/` | Filter by any column | +| `.order//` | `.undo/to-savepoint/.order/name/` | Sort by column | +| `.columns//` | `.undo/id/.columns/log_id,filename,type/` | Project columns | +| `.export/` | `.undo/to-savepoint/.export/json` | Export (json, csv, tsv, yaml) | + +#### 4.3.3 Single Operation Undo (`id/`) + +```bash +ls notes/.undo/id/ # default 100 most recent log entries +2026-04-08T143015.234Z-i9j0k1l2m3n4b/ +2026-04-08T143012.001Z-g7h8i9j0k1l2a/ +... +``` + +Each entry contains summary and apply (no preview tree -- single operations affect one file, use `.log//before` and `.log//after` for diffs): + +``` +notes/.undo/id// +├── .info/ +│ └── summary +└── .apply +``` + +```bash +cat notes/.undo/id//.info/summary +# target: 2026-04-08T14:30:15Z +# affected: 1 file +# type filename user timestamp +edit docs/hello.md agent-7 2026-04-08T14:30:15Z + +# Diff using .log/ symlinks (not .undo/ preview tree) +diff -u --color notes/.log//before notes/.log//after + +touch notes/.undo/id//.apply +``` + +#### 4.3.4 Undo to Savepoint (`to-savepoint/`) + +```bash +ls notes/.undo/to-savepoint/ # default 100 most recent savepoints +before-exploration/ +sprint-start/ +auto-agent-7-2026-04-08T143000Z/ +... +``` + +Each entry is a preview directory: + +``` +notes/.undo/to-savepoint/before-exploration/ +├── .info/ +│ └── summary +├── docs/ +│ └── hello.md +├── bar/ +│ └── baz/ +│ └── readme.md +├── scratch/ +│ └── temp.md -> /dev/null # file will be deleted +└── .apply +``` + +```bash +cat notes/.undo/to-savepoint/before-exploration/.info/summary +restore docs/hello.md +restore bar/baz/readme.md +delete scratch/temp.md + +touch notes/.undo/to-savepoint/before-exploration/.apply +``` + +#### 4.3.5 Undo to Log ID (`to-id/`) + +Same interface as `to-savepoint/`, but browsing log entries instead of savepoint names: + +```bash +ls notes/.undo/to-id/ # default 100 most recent log entries +2026-04-08T143015.234Z-i9j0k1l2m3n4b/ +... + +cat notes/.undo/to-id//.info/summary +touch notes/.undo/to-id//.apply +``` + +#### 4.3.6 Undo by User + +Adds a `/.by/user_id//` suffix to `to-savepoint/` or `to-id/` to filter operations: + +```bash +cat notes/.undo/to-savepoint/before-exploration/.by/user_id/agent-7/.info/summary +touch notes/.undo/to-savepoint/before-exploration/.by/user_id/agent-7/.apply + +cat notes/.undo/to-id//.by/user_id/agent-7/.info/summary +touch notes/.undo/to-id//.by/user_id/agent-7/.apply +``` + +Only the specified user's operations are undone; other users' changes are preserved. + +**Caveat:** If two users interleave edits on the same file, per-user undo restores the file to the state before the specified user's first edit -- which also reverts the other user's interleaved edits on that file. This is inherent in rollback semantics. + +#### 4.3.7 Pipeline Filter Composition with `.apply` + +Pipeline capabilities within each undo sub-directory narrow which operations are in scope. **What you see in the preview is what gets undone.** + +`.apply` is available with all pipeline capabilities **except `.sample/`** (random selection of operations to undo is nonsensical). + +| Capability | Affects undo scope? | `.apply` available? | +|---|---|---| +| `.all/` | No -- removes default limit | Yes | +| `.last/N/` | Yes -- limits to last N ops | Yes | +| `.first/N/` | Yes -- limits to first N ops | Yes | +| `.by///` | Yes -- narrows by indexed column | Yes | +| `.filter///` | Yes -- narrows by any column | Yes | +| `.order//` | No -- display order only | Yes (ignores order) | +| `.columns//` | No -- column projection only | Yes (ignores projection) | +| `.export/` | No -- output format only | Yes (ignores format) | +| `.sample/N/` | Yes -- random selection | **No** | + +Filters compose with each other and with `.apply`, just as they do in data-first pipeline queries: + +```bash +# Undo only delete operations after a savepoint ("bring back deleted files") +cat notes/.undo/to-savepoint/before-refactor/.filter/type/delete/.info/summary +touch notes/.undo/to-savepoint/before-refactor/.filter/type/delete/.apply + +# Undo only changes to a specific file +cat notes/.undo/to-savepoint/before-refactor/.filter/filename/hello.md/.info/summary +touch notes/.undo/to-savepoint/before-refactor/.filter/filename/hello.md/.apply + +# Undo the last 5 operations after a savepoint +cat notes/.undo/to-savepoint/before-refactor/.last/5/.info/summary +touch notes/.undo/to-savepoint/before-refactor/.last/5/.apply + +# Undo only agent-7's delete operations after a savepoint +cat notes/.undo/to-savepoint/before-refactor/.by/user_id/agent-7/.filter/type/delete/.info/summary +touch notes/.undo/to-savepoint/before-refactor/.by/user_id/agent-7/.filter/type/delete/.apply +``` + +**Algorithm for filtered undo:** For any filtered set of operations, group by `file_id`. For each file, the earliest entry in the filtered set provides the `version_id` (before-state). Apply the same DELETE/UPSERT logic as unfiltered undo, scoped to only the affected files. + +#### 4.3.8 Summary Format + +The `.info/summary` file is TSV with two columns: action and filename. Format suffixes are supported (`summary.json`, `summary.csv`, etc.): + +``` +restore docs/hello.md +restore bar/baz/readme.md +delete scratch/temp.md +``` + +**Actions:** +- `restore`: File will be restored to its state at the target point (exists in the preview directory tree) +- `delete`: File will be deleted (inserted after the target point; appears as `/dev/null` symlink in preview directory tree) + +The summary uses the same `DISTINCT ON` query as the undo execution (Section 1.8), so both benefit from SkipScan. No separate `GROUP BY` query is needed. If a user wants per-file operation counts, they can query `.log/.by/file_id/`. + +#### 4.3.9 Preview Directory Structure + +The preview directory tree contains only affected files (not the entire synth app). Only directories that contain affected files appear. Intermediate directories are virtual path components derived from affected filenames. + +**Files being restored** (`restore` action): Appear in the directory tree. The **filename comes from the history entry** (the target state), not the log entry. If a file was renamed after the savepoint, the preview shows its name at the savepoint. `cat` returns the restored content -- the before-state from history, rendered through the synth format layer (markdown with frontmatter, plain text, etc.). + +**Files being deleted** (`delete` action): Appear in the directory tree as **symlinks to `/dev/null`**. The **filename comes from the log entry** (the current/most-recent name). This enables uniform diffing without conditional logic (see Section 4.3.12). + +**Files being re-inserted** (were deleted after the target point): DO appear in the directory tree. The **filename comes from the history entry** (the state being restored). `cat` returns their restored content. + +#### 4.3.10 Computing the Preview + +The preview is computed lazily -- only materialized when `ls` or `cat` is called on a specific path. + +**For `ls` (ReadDir):** Run the undo query to get the list of affected files with their actions. Include all actions (restore and delete). Parse filenames to determine directory structure at the requested depth. Delete entries appear as symlinks to /dev/null. + +**For `cat` (ReadFile):** Check if the requested file is in the affected set. If so, fetch the before-state from history via `version_id` and render through the synth format layer. If not affected, return an error (file doesn't exist in the preview). + +**For `.info/summary`:** Run the same `DISTINCT ON` undo query (Section 1.8) to get all affected files with their actions. Format as TSV (or JSON/CSV via format suffix). + +**Performance:** The undo query (with SkipScan) returns one entry per affected file. For `ls`, no history content is fetched -- just filenames. For `cat`, one history lookup per file. Both are efficient. + +#### 4.3.11 Error Handling + +- **Invalid log_id in `.undo/id//`:** Returns ENOENT (no such file or directory). +- **Invalid savepoint name in `.undo/to-savepoint//`:** Returns ENOENT. +- **No operations to undo (e.g., savepoint is the most recent event):** `.info/summary` is empty. `.apply` is a no-op (or returns an error). +- **`.apply` on an already-applied undo:** The undo is idempotent (see Section 3.5), so re-applying produces the same data state with additional log entries. + +#### 4.3.12 Combined Diff + +Because deleted files appear as `/dev/null` symlinks in the preview tree, diffing all changes is a uniform one-liner with no conditional logic: + +```bash +while IFS=$'\t' read action path; do + diff -u --color "notes/$path" "notes/.undo/to-savepoint/x/$path" +done < notes/.undo/to-savepoint/x/.info/summary +``` + +Diff direction is **current first, preview second** ("what changes to go from current to undone state"): +- **restore**: `diff current-file restored-preview` -- shows the changes +- **delete**: `diff current-file /dev/null` -- shows the entire file as removed + +**Limitation:** For re-inserted files (files that were deleted after the savepoint and will be restored), `notes/$path` does not currently exist on disk, so `diff` would fail. For these entries, use the preview file directly: `cat notes/.undo/to-savepoint/x/$path` to see what will be re-created. The summary action column distinguishes `restore` (file exists currently) from `delete` (file will be removed). Re-inserted files appear with `restore` action in the summary, and the diff script works because the current file path resolves to a non-existent path that `diff` reports as missing. + +#### 4.3.13 Finding a Log Entry by Time + +With the timestamp+base36 display format, log entries sort chronologically. To find the first entry at or after a specific time: + +```bash +LOG_ID=$(cat notes/.log/.export/json | grep -m1 '"2026-04-08T1400' | jq -r '.log_id') +cat notes/.undo/to-id/$LOG_ID/.info/summary +``` + +#### 4.3.14 Undo Apply Feedback + +`touch .apply` returns errno 0 on success or an error code on failure. Detailed feedback is logged at Info level on the TigerFS process: + +```go +logging.Info("undo applied", + zap.String("savepoint", "before-agent"), + zap.Int("files_restored", 3), + zap.Int("files_deleted", 1)) +``` + +Agents can verify the undo was applied by re-reading the preview for specific files and confirming the current state matches the target state, or by comparing `.info/summary` output before and after (the actions should remain the same, confirming idempotent application). + +### 4.4 Auto-Savepoints + +#### 4.4.1 Purpose + +The most common failure mode is forgetting to create a savepoint before an agent starts working. Auto-savepoints eliminate this by detecting "session" boundaries based on write inactivity gaps. + +#### 4.4.2 Mechanism + +When a write occurs on a history-enabled synth app table, TigerFS checks the most recent log entry's timestamp. If the gap exceeds a configurable threshold (default: 30 minutes), an auto-savepoint is created before the write is logged. + +```sql +SELECT log_id FROM tigerfs._log ORDER BY log_id DESC LIMIT 1 +``` + +Single index lookup on the log's PK -- sub-millisecond. + +#### 4.4.3 Naming + +``` +auto-agent-7-2026-04-08T143000Z +auto-agent-7-2026-04-08T160000Z +auto-2026-04-08T143000Z # anonymous (no user_id) +``` + +Uses the same timestamp format as UUIDv7 display names (without milliseconds). Greppable by agent: + +```bash +ls notes/.savepoint/ | grep "^auto-agent-7" +``` + +#### 4.4.4 Configuration + +- Config: `auto_savepoint_interval: 30m` +- Env: `TIGERFS_AUTO_SAVEPOINT_INTERVAL=30m` +- Flag: `--auto-savepoint-interval 30m` +- Set to `0` to disable + +--- + +## 5. User Identity + +### 5.1 Current State + +TigerFS has no identity concept today. No `user_id`, no `agent_id`, no way to know who made a change. + +### 5.2 Identity Model + +Identity is a single string (`user_id`) stored in-memory at the mount level. It is set at mount time and can be modified at runtime. + +**Precedence (high to low):** +1. `--user-id` CLI flag +2. `TIGERFS_USER_ID` environment variable +3. NULL (anonymous) + +Both the flag and env var set the initial value of `.info/user` at the mount root. The value can be changed at runtime: + +```bash +# Set at mount time +tigerfs mount --user-id agent-7 postgres://... + +# Read current identity +cat /mnt/db/.info/user +# agent-7 + +# Change mid-session +echo "agent-9" > /mnt/db/.info/user + +# All subsequent logged operations use "agent-9" +``` + +### 5.3 Storage + +The identity is stored in-memory on the `Operations` struct (one per mount). It is ephemeral -- lost on remount. If persistent identity is needed, use `--user-id` at mount time or set `TIGERFS_USER_ID` in the environment. + +### 5.4 Why Not Per-Process Identity + +Reading the calling process's environment (e.g., `CLAUDE_USER_ID`) from within TigerFS is not feasible across platforms. On Linux FUSE, the caller's PID is available and `/proc//environ` can be read (hacky, requires permissions). On macOS NFS, the server only sees UID/GID from RPC credentials -- no PID, no environment. Since TigerFS uses NFS on macOS, per-process identity cannot be reliably implemented. The mount-level `.info/user` approach works on all platforms. + +### 5.5 Multi-Agent Architecture + +When multiple agents need to work on the same database: + +**Use separate mounts, not a shared mountpoint.** + +```bash +# Agent 7 +TIGERFS_USER_ID=agent-7 tigerfs mount /mnt/db-agent7 postgres://... + +# Agent 9 +TIGERFS_USER_ID=agent-9 tigerfs mount /mnt/db-agent9 postgres://... +``` + +This is the design TigerFS was built for. The consistency model explicitly supports multiple mounts to the same database: "A write on one mount must be immediately visible to reads on any other mount." PostgreSQL handles concurrency through MVCC and row-level locking. + +Benefits: +- Clean identity -- each mount has its own `user_id`, no race conditions +- Clean isolation -- no shared in-process state +- Independent undo -- each agent's operations are tagged, "undo by user" works correctly + +### 5.6 Root-Level `.info/` + +This design introduces a root-level `.info/` directory (the mount root, not table-level). Initially it contains only `user`: + +```bash +ls /mnt/db/.info/ +# user + +cat /mnt/db/.info/user +# agent-7 +``` + +The root-level `.info/` can be expanded later with other mount-level metadata. + +--- + +## 6. DDL Limitations + +The undo system tracks DML operations only (create, edit, rename, delete). It cannot reverse DDL changes: + +| DDL Change | Risk | +|------------|------| +| Column added after savepoint | History entries lack the new column. Restore sets it to NULL. | +| Column removed after savepoint | History entries reference a non-existent column. Restore fails. | +| Table dropped | Log and history tables are orphaned. Undo is impossible. | +| Table renamed | Log references may not resolve. | + +**Why this is acceptable:** +- Synth app schemas are managed by TigerFS (`.build/`). Schema changes are rare and intentional. +- DDL operations are already staged with `.test/.commit` -- users understand they're structural changes. +- The undo feature targets data changes (agent edited files), not schema changes (someone altered the table). + +**Future mitigation (not in Phase 12):** Record a schema fingerprint in the savepoint table. On undo, compare fingerprints and warn if the schema has changed since the savepoint was created. + +--- + +## 7. Log Entry Creation: Integration with Write Path + +### 7.1 Where Log Entries Are Created + +Every write operation in TigerFS that modifies a history-enabled synth app table must also insert a log entry. The affected code paths: + +| Operation | Write Path | Log Entry Type | +|-----------|-----------|---------------| +| Create a file | `writeSynthFile()` / `db.InsertRow()` | `create` | +| Edit a file | `writeSynthFile()` / `db.UpdateRow()` | `edit` | +| Delete a file | `deleteSynthFile()` / `db.DeleteRow()` | `delete` | +| Rename a file/dir | `renameSynthFile()` / `db.UpdateRow()` | `rename` | +| Move a file/dir | `renameSynthFile()` / `db.UpdateRow()` | `rename` | +| Undo operation | New undo handler | `undo` | + +### 7.2 Determining `version_id` + +For edit, rename, and delete operations, the BEFORE trigger creates a history entry synchronously. To capture the `version_id` for the log: + +**Option A:** Query the history table for the most recent entry matching the `file_id` immediately after the DML. + +**Option B:** Modify the DML to use a CTE or RETURNING clause that captures the trigger's output. + +**Option C:** Use a PostgreSQL function that performs the DML and returns the generated `version_id`. + +The exact mechanism is an implementation detail to be determined during development. All options are correct; they differ in complexity and round-trip count. + +--- + +## 8. Table Setup: DDL for Log and Savepoint Tables + +When a synth app with history is created (via `.build/`), the setup must additionally create: + +1. The log hypertable (`tigerfs._log`) +2. The composite index on `(file_id, log_id ASC)` +3. The compression policy +4. The savepoint table (`tigerfs._savepoint`) + +This extends the existing DDL generation in `synth/build.go` which already creates the backing table, view, history table, history trigger, and hypertable/compression configuration. + +--- + +## 9. Performance Characteristics + +### 9.1 Write Overhead + +Each write to a history-enabled table now requires one additional insert (the log entry) in addition to the existing DML + trigger. This is a small overhead -- one extra row insert into a hypertable. + +### 9.2 Undo-to-Savepoint Query Performance + +| Scenario | Log entries after savepoint | Unique files | Expected time | +|----------|---------------------------|-------------|---------------| +| Agent did 20 edits, recent savepoint | ~20 | 5-15 | < 100ms | +| Agent did 500 edits, recent savepoint | ~500 | 50-200 | < 1s | +| Old savepoint, months of history | 10,000+ | 1,000+ | seconds (decompression) | + +The common case (recent savepoint, agent cleanup) is fast because: +1. **Chunk exclusion:** `WHERE log_id > savepoint_id` prunes compressed chunks before the savepoint +2. **SkipScan:** Hops across `file_id` values instead of scanning every entry +3. **History lookups:** One PK lookup per affected file (not per log entry) + +### 9.3 Preview Computation + +The preview (`.info/summary` and affected file listing) uses the same query as the undo itself. No additional cost beyond the undo query + history lookups for file content (only when `cat` is called on a specific file). + +--- + +## 10. Naming Conventions + +### 10.1 Why "Undo" (Not "Rollback") + +- **"Undo"** maps to the Ctrl+Z mental model -- "take back what I just did." This matches the primary use cases. +- **"Rollback"** in PostgreSQL means "discard uncommitted work inside a transaction." Here, changes are already committed and we're creating new operations to restore old state. "Rollback" borrows the word but changes the semantics. +- One word, one concept, one directory (`.undo/`). The path structure communicates scope: `.undo/id/` (one operation) vs `.undo/to-id/` (everything after this entry) vs `.undo/to-savepoint/` (everything after a checkpoint). + +### 10.2 Directory Summary + +| Path | Purpose | Implementation | +|------|---------|---------------| +| `/.log/` | Operation log (read-only) | Data-first pipeline on `tigerfs._log` | +| `/.savepoint/` | Savepoint management (read/write) | Data-first pipeline on `tigerfs._savepoint` | +| `/.undo/` | Undo operations (preview + apply) | Custom handler with preview directory tree | +| `/.info/user` | User identity (read/write) | In-memory on Operations struct | + +--- + +## 11. UUIDv7 Display Format + +### 11.1 Problem + +UUIDv7 values displayed as standard hex (`019590a0-1234-7fff-8000-a1b2c3d4e5f6`) are opaque. A directory listing of log entries or history versions tells you nothing about timing without reading each entry. Since UUIDv7 embeds a millisecond timestamp, the display format should surface it. + +### 11.2 Format + +Display UUIDv7 values as `-`: + +``` +2026-04-07T143000.123Z-zzz0063hd8e5r42 +├── timestamp (ms) ────┘ └── entropy (base36) +``` + +**Timestamp portion:** Millisecond-precision, UTC, filesystem-safe (no colons). Format: `2006-01-02T150405.000Z`. Extracted from UUIDv7 bits 0-47. + +**Entropy portion:** The 74 non-timestamp, non-fixed bits (12 bits rand_a + 62 bits rand_b), encoded as base36 (0-9a-z). ~15 characters. + +**Total length:** ~40 characters (vs 36 for standard UUID hex). + +### 11.3 Why Base36 + +- **Case-insensitive safe.** Only 0-9a-z. No collisions on macOS APFS (which defaults to case-insensitive). Base62 and base64url use mixed case and are NOT safe. +- **Familiar charset.** Lowercase alphanumeric -- nothing surprising. +- **Compact.** 15 chars for 74 bits (vs 19 for hex, 13 for base62). +- **Trivial to implement.** `big.Int.Text(36)` encodes, `big.Int.SetString(s, 36)` decodes. No libraries needed. + +### 11.4 Encoding and Decoding + +```go +func UUIDv7ToDisplayName(id uuid.UUID) string { + // Extract ms timestamp from bits 0-47 + b := id[:] + msec := int64(binary.BigEndian.Uint16(b[0:2]))<<32 | + int64(binary.BigEndian.Uint32(b[2:6])) + ts := time.UnixMilli(msec).UTC() + + // Extract 74 entropy bits: rand_a (bits 52-63) + rand_b (bits 66-127) + // Pack into 10 bytes, encode as base36 + entropy := packEntropy(id) + var n big.Int + n.SetBytes(entropy) + + return fmt.Sprintf("%s-%s", ts.Format("2006-01-02T150405.000Z"), n.Text(36)) +} + +func DisplayNameToUUIDv7(name string) (uuid.UUID, error) { + // Split on last '-' that separates timestamp from entropy + // Parse timestamp → reconstruct bits 0-47 + // Parse base36 entropy → reconstruct bits 52-63, 66-127 + // Set version bits (48-51 = 0111) and variant bits (64-65 = 10) + // → Full UUID reconstructed, 1:1 reversible, no lookup needed +} +``` + +**Fully reversible.** The display name encodes all 122 meaningful bits of the UUIDv7. The 4 version bits and 2 variant bits are fixed constants and reconstructed on decode. + +### 11.5 Scope of Change + +The display format applies **globally** wherever UUIDv7 values are used as filenames in directory listings. UUIDv7 is detected by checking the version bits (bits 48-51 = `0111`), which is reliable per RFC 9562. Non-v7 UUIDs (v4, v1, etc.) continue to display as standard hex. + +| Context | Format | Why | +|---|---|---| +| Directory entry name (PK as filename) | Display format for v7, hex for others | Filenames should be human-readable | +| Column value in file content (`cat .../col`) | Always hex | Standard UUID text representation in data formats | +| `.by//` input path | Accept both formats | User might type either | +| `.export/json` output | Always hex | JSON data should use standard UUID format | + +**Implementation point:** The conversion is applied in `scanAndEncodePK()` (`db/query.go`), where PK values become filenames. It detects UUIDv7 and applies the display format. The generic `ConvertValueToText()` in `format/convert.go` is unchanged (it handles content display, not filenames). + +**Affected contexts:** + +| Context | Current format | New format | +|---|---|---| +| `.history//` | `2026-04-07T143000Z` (second precision, lossy) | `2026-04-07T143000.123Z-zzz0063hd8e5r42` (lossless) | +| `.log/` | hex UUID (new code) | timestamp+base36 | +| `.undo/id/`, `.undo/to-id/` paths | hex UUID (new code) | timestamp+base36 | +| Data-first tables with UUIDv7 PK | hex UUID | timestamp+base36 (auto-detected) | +| Data-first tables with UUIDv4 PK | hex UUID | hex UUID (unchanged) | + +**No database migration needed.** The display format is a pure presentation-layer conversion -- version IDs are computed on-the-fly from the UUIDv7 bytes stored in the `version_id` column, never stored as strings. Changing the conversion function immediately produces the new format for all existing data. The only breakage is external scripts that cached old-format paths (e.g., `2026-04-07T143000Z`), which is acceptable. + +### 11.6 Visual Comparison + +```bash +# Current .history/ listing +ls notes/.history/hello.md/ +2026-04-07T100000Z +2026-04-07T143000Z +2026-04-07T150000Z + +# New .history/ listing +ls notes/.history/hello.md/ +2026-04-07T100000.000Z-a230b1c2d3e4f5x +2026-04-07T143000.123Z-zzz0063hd8e5r42 +2026-04-07T150000.456Z-1230deadbeef1z0 + +# New .log/ listing +ls notes/.log/.last/3 +2026-04-07T143000.123Z-zzz0063hd8e5r42 +2026-04-07T143001.456Z-a230b1c2d3e4f5x +2026-04-07T143005.789Z-1230deadbeef1z0 +``` + +--- + +## 12. Skills and Documentation Updates + +### 12.1 Key Behavioral Guidance (SKILL.md) + +The top-level skill must establish savepoints as a core agent workflow, not an optional feature: + +- **Before making multiple or risky edits, always create a savepoint.** This is cheap and provides a clean revert path. +- **Never manually reverse edits across files to "undo."** Use `touch .undo/to-savepoint/name/.apply` instead. TigerFS tracks exact before-states; manual reversal is error-prone and may miss files or introduce inconsistencies. +- **When to savepoint:** Before investigating/debugging, before refactoring, before multi-file operations, before anything uncertain. +- **When to undo:** User asks to revert, agent realizes approach isn't working, changes were made to wrong files. + +Add to the anti-patterns table: "Manually reverse edits to undo" -> "Create savepoints, use `.undo/` to revert atomically." + +Add to the "What you can build" section: pointer to undo workflow when asked to revert or roll back. + +### 12.2 File-First Reference (files.md) + +Add three new sections after the existing "Versioned History" section: + +1. **Operation Log (`.log/`):** Browsing recent operations, filtering by user/file/type, diff symlinks (before/after/current). +2. **Savepoints (`.savepoint/`):** Creating, listing, renaming, deleting, auto-savepoints. +3. **Undo (`.undo/`):** Single operation undo, undo to savepoint, undo by user, preview + apply workflow, combined diff one-liner. + +Update the existing history section to reflect the new UUIDv7 display format. + +### 12.3 Recipes (recipes.md) + +Add two new recipes: + +- **Recipe 5: Safe Agent Exploration.** Auto-savepoints, manual savepoints, agent self-revert pattern. +- **Recipe 6: Selective Undo in Multi-Agent Workflows.** Separate mounts per agent, per-user undo, preserving other agents' work. + +### 12.4 Operations Reference (ops.md) + +Add `--user-id` flag, `TIGERFS_USER_ID` env var, and `--auto-savepoint-interval` flag. + +### 12.5 Quick Reference Updates (SKILL.md) + +Add to the quick reference table: + +| Goal | Tool Call | +|------|-----------| +| Create savepoint | `Bash "echo '{}' > mount/app/.savepoint/name.json"` | +| Preview undo | `Read "mount/app/.undo/to-savepoint/name/.info/summary"` | +| Apply undo | `Bash "touch mount/app/.undo/to-savepoint/name/.apply"` | +| View log | `Read "mount/app/.log/.last/10/.export/json"` | +| Diff a change | `Bash "diff -u --color mount/app/.log//before mount/app/.log//after"` | + +--- + +## 13. Open Design Questions (Deferred) + + +### 13.1 Time-Travel Snapshots + +The undo preview shows only affected files. A more powerful feature would be full time-travel snapshots -- browse the entire synth app as it existed at any point in time. This could live under `.snapshot/` or as an extension to `.history/`. Deferred to a future phase. + +### 13.2 Schema Fingerprinting for Savepoints + +Record the table schema at savepoint creation time. Warn on undo if the schema has changed. Deferred -- synth app schema changes are rare. + +### 13.3 Optimistic Concurrency for Undo + +Detect when a concurrent write conflicts with an undo operation and abort rather than silently overwriting. Deferred -- the current "last writer wins" behavior matches standard PostgreSQL semantics. + +### 13.4 Cross-Table Undo + +Undo operations across multiple tables in a single operation. Deferred -- the common case is single-table undo (one synth app = one mini-filesystem). Per-table undo is sufficient for Phase 12. + +### 13.5 CLI Commands (`tigerfs undo`, `tigerfs savepoint`) + +Thin CLI wrappers around the filesystem operations for human ergonomics (`tigerfs undo --to-savepoint x --preview`, `tigerfs savepoint create x`). Would resolve mount points from the mount registry. Deferred -- the filesystem interface is complete and agents use it directly. + +### 13.6 Bulk Operations in the Log + +Should `.import/` operations create one log entry per row, or one log entry for the entire import? Per-row is correct but could create thousands of log entries. Deferred to implementation. + +--- + +## 14. Summary of Design Decisions + +| Decision | Choice | Rationale | +|----------|--------|-----------| +| Log scope | Per-table | Matches history tables. Simpler queries. No cross-table transaction complexity. | +| Log stores | Pointer to history (`version_id`), not copies | No data duplication. History already has full row state. Log stays small. | +| Savepoint storage | Separate table | Clean schema. Natural UNIQUE on name. No compression concerns. | +| Undo semantics | Rollback (git checkout), not revert (git revert) | Full row state in history. No column-level diff/merge needed. | +| Undo interface | Preview + apply (stateless) | Preview directory tree shows affected files. `touch .apply` triggers execution. No session management. | +| `after_id` in log | Not stored | After-state doesn't exist as a history entry at log-write time. Before-state is sufficient for undo. After-state is derivable from the chain. | +| `filename` in log | Denormalized | Avoids JOINs in display queries. Preserves historically-correct filename. Keeps hypertable optimizations. | +| Type column | TEXT with CHECK constraint | Matches existing `operation` in history. Easy to extend without ALTER TYPE migration. | +| User identity | Mount-level `.info/user` | Set at mount time via --user-id or TIGERFS_USER_ID. Modifiable at runtime. In-memory storage. | +| Multi-agent | Separate mounts | One mount per agent. Concurrency through PostgreSQL. Clean identity and isolation. | +| Naming | "Undo" not "rollback" | Ctrl+Z mental model. "Rollback" implies uncommitted transaction. One concept, one directory. | +| Cross-table undo | Deferred | Single-table covers the common case. Cross-table adds significant transaction complexity. | +| Compression segmentby | `file_id` | Enables SkipScan on compressed chunks. Matches undo query pattern. | +| Compression orderby | `log_id ASC` | Aligns with undo query ORDER BY. Enables SkipScan within compressed segments. | +| UUIDv7 display format | Timestamp+base36, global | `2026-04-07T143000.123Z-zzz0063hd8e5r42`. Case-insensitive safe, lossless, 40 chars. Applied globally for UUIDv7 PKs via `scanAndEncodePK()`. | +| Diff symlinks | before/after/current on log entries | Symlinks to .history/ versions or /dev/null. Enables `diff` with no special cases. | +| Deleted files in preview | /dev/null symlinks | Appear in preview tree as symlinks to /dev/null. Enables uniform one-liner diff across all action types. | +| Auto-savepoints | Session-based, on inactivity gap | Auto-create savepoint when gap > threshold (default 30m). Named `auto--`. Eliminates "forgot to checkpoint" failure mode. | +| Apply trigger | `touch .apply` (not `.commit`) | "Apply this undo" reads naturally. Distinct from DDL's `.commit`. Triggered via Create or Write (both `touch` and `echo` work). | +| Undo feedback | `logging.Info` + verify state | Log at Info level on TigerFS process. Agents verify by checking files match target state. | diff --git a/docs/adr/017-relational-directory-structure.md b/docs/adr/017-relational-directory-structure.md new file mode 100644 index 0000000..e8664b8 --- /dev/null +++ b/docs/adr/017-relational-directory-structure.md @@ -0,0 +1,512 @@ +# ADR-017: Relational Directory Structure for Synth Apps + +**Status:** Accepted +**Date:** 2026-04-10 +**Author:** Mike Freedman + +## Context + +Synth apps encode directory structure in the `filename` column using slashes (`projects/web/todo.md`). Directory renames use `RenameByPrefix` -- a single SQL UPDATE that modifies N rows' filename columns. This creates a fundamental problem for the undo log (ADR-016): + +- The undo log is per-file (one `file_id` per entry with a `version_id` pointing to the before-state) +- A directory rename is a batch operation affecting N files +- Multiple approaches were evaluated (single rename-dir entry, JSONB arrays, batch_id grouping, N independent entries). All have correctness gaps: partial undo risk, DISTINCT ON incompatibility, ordering dependencies, or filter blind spots. + +The relational parent-pointer model eliminates the problem: renaming a directory is a single-row update (change the directory row's `filename` column), which naturally fits the per-file undo log model. + +Beyond undo, the relational model also improves ReadDir performance (O(children) vs O(all_rows)) and eliminates prefix-collision risks in LIKE queries. + +## Decision + +Replace path-encoded filenames with a parent-pointer model: `filename` stores only the leaf name, and `parent_id` references the parent directory row. + +### Source table schema + +```sql +CREATE TABLE tigerfs. ( + id UUID PRIMARY KEY DEFAULT uuidv7(), + parent_id UUID REFERENCES tigerfs.(id) DEFERRABLE INITIALLY IMMEDIATE, + filename TEXT NOT NULL, + filetype TEXT NOT NULL DEFAULT 'file' CHECK (filetype IN ('file', 'directory')), + title TEXT, -- markdown only + author TEXT, -- markdown only + headers JSONB DEFAULT '{}'::jsonb, -- markdown only + body TEXT, + encoding TEXT NOT NULL DEFAULT 'utf8' CHECK (encoding IN ('utf8', 'base64')), + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + modified_at TIMESTAMPTZ NOT NULL DEFAULT now(), + UNIQUE NULLS NOT DISTINCT (parent_id, filename, filetype) DEFERRABLE INITIALLY IMMEDIATE +) +``` + +Changes from ADR-011: +- `filename`: now stores leaf name only (e.g., `todo.md`), not full path (`projects/web/todo.md`). No slashes. +- `parent_id UUID`: self-referencing FK (DEFERRABLE INITIALLY IMMEDIATE) to parent directory row. NULL for root-level entries. `DEFERRABLE` is needed so undo transactions can `SET CONSTRAINTS ALL DEFERRED` to DELETE/INSERT rows in any order. `INITIALLY IMMEDIATE` (not DEFERRED) because PostgreSQL's ON CONFLICT clause does not support deferrable constraints as arbiters -- `InsertIfNotExists` for parent directory creation needs immediate constraint checking for normal operations. +- `UNIQUE NULLS NOT DISTINCT (parent_id, filename, filetype) DEFERRABLE INITIALLY IMMEDIATE`: uniqueness is per-directory, not global. `NULLS NOT DISTINCT` (PostgreSQL 15+) ensures uniqueness at the root level where parent_id is NULL. `DEFERRABLE` allows undo transactions to defer checking via `SET CONSTRAINTS ALL DEFERRED`. `INITIALLY IMMEDIATE` for the same ON CONFLICT reason as above; note that `InsertIfNotExists` uses plain INSERT with unique-violation error handling (SQLSTATE 23505) rather than ON CONFLICT, because even `INITIALLY IMMEDIATE` deferrable constraints are not valid ON CONFLICT arbiters in PostgreSQL. +- Self-referencing FK uses ON DELETE RESTRICT (default) -- cannot delete a directory that has children. + +Additional DDL: + +```sql +-- Index for ReadDir and path resolution +CREATE INDEX idx__parent ON tigerfs.(parent_id, filename); +``` + +### History table schema + +```sql +CREATE TABLE tigerfs._history ( + file_id UUID, + parent_id UUID, + filename TEXT NOT NULL, + filetype TEXT CHECK (filetype IN ('file', 'directory')), + title TEXT, + author TEXT, + headers JSONB, + body TEXT, + encoding TEXT CHECK (encoding IN ('utf8', 'base64')), + created_at TIMESTAMPTZ, + modified_at TIMESTAMPTZ, + version_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY, + operation TEXT NOT NULL CHECK (operation IN ('edit', 'rename', 'delete')) +) WITH ( + tsdb.hypertable, + tsdb.partition_column = 'version_id', + tsdb.chunk_interval = '7 days', + tsdb.segmentby = 'file_id', + tsdb.orderby = 'version_id DESC' +) +``` + +Changes from previous: +- `id` renamed to `file_id` (references source table's `id`; clearer in context) +- `filename` stores leaf name (matching source table) +- `parent_id` added (captures the file's directory at the time of the operation) +- `_history_id` renamed to `version_id` (more descriptive; no underscore prefix) +- `_operation` renamed to `operation` (drop underscore convention) +- CHECK constraints on `filetype`, `encoding`, and `operation` +- Uses modern `CREATE TABLE WITH` syntax for hypertable + columnstore +- `segmentby = 'file_id'` (was `'filename'`; better for relational model where leaf names can collide across directories) +- `orderby = 'version_id DESC'` (was `'_history_id DESC'`) + +### Log table schema + +```sql +CREATE TABLE tigerfs._log ( + log_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY, + file_id UUID NOT NULL, + type TEXT NOT NULL CHECK (type IN ('create', 'edit', 'rename', 'delete', 'undo')), + user_id TEXT, + filename TEXT NOT NULL, + version_id UUID, + description TEXT +) WITH ( + tsdb.hypertable, + tsdb.partition_column = 'log_id', + tsdb.chunk_interval = '7 days', + tsdb.segmentby = 'file_id', + tsdb.orderby = 'log_id ASC' +) +``` + +Changes from ADR-016 Task 12.3: +- `history_id` renamed to `version_id` (matches history table) +- Operation types renamed from DB-centric to filesystem-centric: + - `insert` -> `create` + - `update` (content) -> `edit` + - `update` (rename/move) -> `rename` + - `delete` -> `delete` + - `undo` -> `undo` +- Log table retains the `(file_id, log_id ASC)` composite index for SkipScan on undo queries: + `CREATE INDEX idx__log_by_file ON tigerfs._log (file_id, log_id ASC);` +- `filename` column stores the **denormalized full path** (e.g., `projects/web/todo.md`), computed at log-write time by walking the parent chain. This is for human-readable log display; the authoritative structure is in the source table's `parent_id` references. Note: the log's `filename` has different semantics from the source table's `filename` (leaf name only). The log stores the full path for display; the source table stores the leaf name for structure. + +### Path resolution + +A PL/pgSQL function resolves path segments to a row ID, with an optional starting parent for cache integration. Returns a table of `(depth, id, filename)` so the Go layer can populate the path cache for all intermediate segments. + +```sql +CREATE FUNCTION tigerfs.resolve_path(tbl REGCLASS, start_parent UUID, segments TEXT[]) +RETURNS TABLE(depth INT, resolved_id UUID, resolved_name TEXT) AS $$ +DECLARE + current_id UUID := start_parent; + i INT := 0; + seg TEXT; +BEGIN + FOREACH seg IN ARRAY segments LOOP + i := i + 1; + -- EXECUTE required because PL/pgSQL doesn't support variables as table names. + -- format('%s', tbl) produces the properly schema-qualified name from REGCLASS. + EXECUTE format('SELECT id FROM %s WHERE filename = $1 AND parent_id IS NOT DISTINCT FROM $2', tbl) + INTO current_id + USING seg, current_id; + IF current_id IS NULL THEN RETURN; END IF; + depth := i; + resolved_id := current_id; + resolved_name := seg; + RETURN NEXT; + END LOOP; +END; +$$ LANGUAGE plpgsql; +``` + +Usage: `SELECT * FROM tigerfs.resolve_path('tigerfs.app', NULL, ARRAY['projects','web','todo.md'])` + +Returns 3 rows: `(1, uuid-1, 'projects')`, `(2, uuid-2, 'web')`, `(3, uuid-3, 'todo.md')`. The Go layer caches each `(parent_id, name) → id` pair. If any segment doesn't resolve, the function returns fewer rows than segments (indicating a missing path component). + +With a cached start_parent: `SELECT * FROM tigerfs.resolve_path('tigerfs.app', uuid-1, ARRAY['web','todo.md'])` -- skips the already-cached root resolution. + +Performance: One network round-trip (~10ms). N index lookups server-side (~0.01ms each). For typical depth (2-5 levels), server-side cost is negligible vs network latency. + +### Path cache (Go-level) + +A Go-level cache maps `(parent_id, filename)` -> `id` with the same 2-second TTL as the existing stat cache. This avoids calling resolve_path for repeated access to the same directory subtree. + +Cache invalidation: directory renames/moves invalidate entries for the renamed directory. The 2-second TTL provides a bounded staleness window matching the existing consistency model ("Stat may use caches"). + +For the multi-agent task board use case (file moves between fixed directories), the path cache is never stale because directory names don't change. + +### Key operations + +| Operation | Current (path-encoded) | New (parent-pointer) | +|---|---|---| +| Rename directory | `RenameByPrefix` (N rows) | `UPDATE SET filename='new' WHERE id=dir_id` (1 row) | +| Move directory | `RenameByPrefix` with new prefix (N rows) | `UPDATE SET parent_id=new_parent WHERE id=dir_id` (1 row) | +| Rename file | `UpdateColumnCAS` on filename (full path) | `UPDATE SET filename='new' WHERE id=file_id` (1 row) | +| Move file | `UpdateColumnCAS` on filename (full path) | `UPDATE SET parent_id=new_dir, filename='new' WHERE id=file_id` (1 row) | +| ReadDir | `GetAllRows` + in-memory prefix filter O(all_rows) | `SELECT * WHERE parent_id = dir_id` O(children) | +| Stat/ReadFile | `WHERE filename = 'full/path'` O(1) | Path cache hit: O(0) network + 1 query for leaf. Cache miss: `resolve_path` O(1 round-trip) | +| mkdir | `INSERT (filename='full/path', filetype='directory')` | `INSERT (filename='dirname', parent_id=X, filetype='directory')` | +| Ensure parents | Split path, InsertIfNotExists per ancestor | Same but chain parent_id values | +| Delete check | `LIKE prefix/%` | `WHERE parent_id = dir_id` | +| Undo of rename | Broken (batch problem) | Standard single-row UPSERT from history | + +### ReadFile / Stat with resolve_path and cache + +The Go-level path cache short-circuits already-resolved ancestor segments. Only unresolved segments are sent to the DB via resolve_path. The resolve_path function returns intermediate results so the cache can be populated in one call. + +**Go-level resolution flow for `projects/web/todo.md`:** + +1. Walk path segments, checking cache at each level: + - `(NULL, "projects")` → cache hit → uuid-1 + - `(uuid-1, "web")` → cache miss → stop +2. Remaining segments: `["web", "todo.md"]` +3. One DB call: `resolve_path(tbl, uuid-1, ARRAY['web','todo.md'])` → returns rows: `(1, uuid-2, "web"), (2, uuid-3, "todo.md")` +4. One network round-trip (~10ms). Server does 2 index lookups. +5. Cache populated from returned rows: `(uuid-1,"web")→uuid-2`, `(uuid-2,"todo.md")→uuid-3` + +**Next access to `projects/web/notes.md`:** + +1. Cache: `(NULL,"projects")→uuid-1` hit, `(uuid-1,"web")→uuid-2` hit +2. Cache: `(uuid-2,"notes.md")` → miss +3. Remaining: `["notes.md"]` with start_parent=uuid-2 +4. For ReadFile, combine the last resolution step with the row fetch: + `SELECT * FROM t WHERE parent_id = uuid-2 AND filename = 'notes.md'` +5. One round-trip (~10ms). Returns the full row (content + metadata). + +After the first access to any file in a directory, the path prefix is fully cached. Subsequent accesses to siblings only query the leaf. + +**For ReadFile** (consistency model: "ReadFile must always hit the DB"): +- Parent path resolution uses the cache (parent_id mappings are structural, not content) +- The leaf row is fetched with a combined resolve + fetch query: + `SELECT * FROM t WHERE parent_id = $cached_parent AND filename = $leaf_name` +- One round-trip (~10ms) -- same as the current model's `WHERE filename = 'full/path'` + +**For Stat** (consistency model: "Stat may use caches"): +- Both parent resolution AND leaf stat can use the cache +- Potentially zero DB round-trips if fully cached + +**Cold path (first access, nothing cached):** +- `resolve_path(NULL, ARRAY['projects','web','todo.md'])` → one round-trip (~10ms), returns all intermediate IDs +- Then fetch full row: `SELECT * FROM t WHERE id = uuid-3` → one round-trip (~10ms) +- Total: ~20ms. Subsequent accesses: ~10ms (ancestors cached, only leaf fetch needed) + +**Path cache TTL:** 2 seconds, matching the existing stat cache. Directory structure changes (renames/moves) are visible to other mounts within 2 seconds. + +### .history/ navigation + +`.history/` follows the same parent_id structure as the live table: + +- `ls .history/`: `SELECT DISTINCT file_id, filename FROM history WHERE parent_id IS NULL` +- `ls .history/projects/`: Resolve `projects` to its `id`, then `SELECT DISTINCT file_id, filename FROM history WHERE parent_id = ` +- `ls .history/projects/web/todo.md/`: Resolve full path to `file_id`, then `SELECT version_id FROM history WHERE file_id = ORDER BY version_id DESC` + +File renames are handled correctly: `.history//` resolves by file_id. All versions of that file (under any past name or directory) are accessible via `.history/.by//`. + +### Rename and move operations + +**FUSE `Rename(oldParent, oldName, newParent, newName)`** maps to: +- **Same parent (rename):** `UPDATE SET filename = newName WHERE id = fileID` (1 row) +- **Different parent (move):** `UPDATE SET parent_id = newParentID, filename = newName WHERE id = fileID` (1 row) +- **Directory rename/move:** Same as above -- the directory row is updated, children are unaffected + +Both cases are single-row updates. The BEFORE trigger fires once, creating one history entry. One log entry is created with `type = 'rename'`. + +The current `UpdateColumnCAS` pattern (compare-and-swap on filename) is replaced by a simpler `UPDATE ... WHERE id = X`. Concurrent renames of the same file are serialized by PostgreSQL's row-level locking. + +### Delete behavior + +The self-referencing FK uses `ON DELETE RESTRICT` (PostgreSQL default). The Go layer checks for children before attempting delete: + +```go +// Check: SELECT EXISTS(SELECT 1 FROM t WHERE parent_id = dir_id) +// If children exist → return ENOTEMPTY +// If no children → DELETE FROM t WHERE id = dir_id +``` + +The FK constraint is a safety net -- if the Go check races with a concurrent insert, the DELETE fails at the DB level rather than orphaning children. + +### PostgreSQL version requirement + +This ADR requires PostgreSQL 15+ for `UNIQUE NULLS NOT DISTINCT`. Combined with ADR-016's requirement of PostgreSQL 16+ for SkipScan, the effective minimum is **PostgreSQL 16**. + +### BEFORE trigger + +The archive trigger copies the OLD row (including `parent_id`) to the history table: + +```sql +CREATE OR REPLACE FUNCTION tigerfs.archive__history() RETURNS TRIGGER AS $$ +BEGIN + INSERT INTO tigerfs._history + (file_id, parent_id, filename, filetype, title, author, headers, + body, encoding, created_at, modified_at, + version_id, operation) + VALUES + (OLD.id, OLD.parent_id, OLD.filename, OLD.filetype, OLD.title, OLD.author, OLD.headers, + OLD.body, OLD.encoding, OLD.created_at, OLD.modified_at, + uuidv7(), + CASE TG_OP + WHEN 'DELETE' THEN 'delete' + WHEN 'UPDATE' THEN + CASE WHEN OLD.filename != NEW.filename + OR OLD.parent_id IS DISTINCT FROM NEW.parent_id + THEN 'rename' + ELSE 'edit' + END + END); + IF TG_OP = 'DELETE' THEN + RETURN OLD; + END IF; + RETURN NEW; +END; +$$ LANGUAGE plpgsql; +``` + +The trigger fires BEFORE UPDATE OR DELETE, capturing the complete row state including `parent_id`. When a directory is renamed (filename changes) or moved (parent_id changes), the history entry records the old values. + +**Note:** The trigger SQL above shows markdown-specific columns (title, author, headers). Plain text tables omit these columns. The DDL generator in `build.go` produces format-specific triggers, same as the current implementation. + +### Impact on undo + +With the relational model, every operation (including directory rename and move) is a single-row change: + +| Operation | Log type | What changes | Undo mechanism | +|---|---|---|---| +| Create file | `create` | New row | DELETE the row | +| Edit file | `edit` | body/title/etc columns | UPSERT from version (history entry) | +| Rename file/dir | `rename` | filename column | UPSERT from version | +| Move file/dir | `rename` | parent_id column | UPSERT from version | +| Delete file/dir | `delete` | Row removed | INSERT from version | + +All use the standard DISTINCT ON + UPSERT undo pattern. No special cases for directory operations. + +**UPSERT must restore `parent_id`.** The undo UPSERT (ADR-016 Section 3.3) must include `parent_id` in the restored columns, along with `filename`, `body`, `title`, etc. This ensures that file moves and directory moves are correctly reversed. + +**Filtered undo FK failure.** Any filtered undo (`.by/user_id/`, `.filter/`, `.last/N/`) can fail at commit if the restored `parent_id` references a directory that was deleted by an operation OUTSIDE the filter scope. The undo transaction rolls back entirely (PostgreSQL ACID guarantee) and reports a descriptive error. The database is unchanged. Specific scenarios: + +- **Undo of directory creation:** Agent-1 creates a directory, agent-2 adds files. Per-user undo of agent-1 tries to DELETE the directory. FK fails because agent-2's files reference it. Same semantics as `rmdir` on non-empty directory (ENOTEMPTY). + +- **Undo of file move:** Agent-1 moves a file from `docs/` to `archive/`. Agent-2 deletes `docs/`. Per-user undo of agent-1 tries to restore `parent_id=uuid-docs`. FK fails because `uuid-docs` no longer exists. + +- **Undo of file deletion:** A file was deleted, and its parent directory was later deleted by someone else. Undo tries to INSERT the file with its old `parent_id`. FK fails. + +**Unfiltered undo-to-savepoint always succeeds** because ALL affected rows (including deleted parent directories) are restored within the same transaction. The DEFERRABLE FK allows intermediate states; all constraints are satisfied at COMMIT. + +**UNIQUE constraint at commit.** Both the FK and UNIQUE constraints are `DEFERRABLE INITIALLY IMMEDIATE`. In normal operations, they check immediately. Undo transactions explicitly call `SET CONSTRAINTS ALL DEFERRED` at the start of the transaction, deferring all constraint checks to COMMIT. This allows intermediate violations within the undo transaction (e.g., restoring a filename before deleting the row that currently holds it). At COMMIT, PostgreSQL checks all deferred constraints. If the final state has a UNIQUE violation (e.g., two rows with the same name in the same directory that weren't both handled by the undo), the transaction rolls back. This is a genuine conflict requiring manual resolution, but it can only happen with filtered undos -- unfiltered undo-to-savepoint restores the savepoint state exactly, which was valid. + +### Deprecated code (kept for backward compatibility) + +The following are superseded by the parent-pointer model but retained behind `info.Roles.ParentID == ""` guards for pre-migration databases: + +- `RenameByPrefix` query and callers (replaced by single-row `UPDATE WHERE id`) +- `HasChildrenWithPrefix` query (replaced by `WHERE parent_id = X`) +- `filterHierarchicalChildren` (replaced by `GetRowsByParent` queries) +- Path-based prefix matching logic in synth_ops.go + +These code paths are exercised only when a synth app lacks the `parent_id` column (old schema). After running `tigerfs migrate`, all apps use the parent-pointer model and these paths become dead code. They can be removed once backward compatibility with pre-ADR-017 databases is no longer needed. + +### Files requiring changes + +| File | Changes | +|---|---| +| `synth/build.go` | Source table schema: add parent_id, change UNIQUE constraint. History table: file_id, parent_id, version_id, operation. Create resolve_path function. | +| `synth/cache.go` | ColumnRoles: add ParentID role | +| `synth/markdown.go` | GetMarkdownFilename uses leaf filename | +| `synth/plaintext.go` | GetPlainTextFilename uses leaf filename | +| `fs/synth_ops.go` | All write/read/stat/rename/delete operations use parent_id model. Remove filterHierarchicalChildren. ReadDir queries by parent_id. Path resolution via resolve_path. | +| `fs/history.go` | History queries use file_id + parent_id. version_id column name. | +| `db/query.go` | Remove RenameByPrefix, HasChildrenWithPrefix. Add GetRowsByParent, resolve_path caller. | +| `db/interfaces.go` | Update HierarchyWriter interface | +| All synth tests | Rewrite for new model | + +### Migration + +The `tigerfs migrate` command includes a `relational-directories` migration that handles existing databases. TigerFS creates new apps with the new schema automatically; migration is only needed for databases created before ADR-017. Run `tigerfs migrate --describe` to check for pending migrations, or `tigerfs migrate ` to execute. + +The `relational-directories` migration in `cmd/migrate.go` performs these steps per app (in a single transaction): + +1. **Add parent_id column** to source table +2. **Populate parent_id** via PL/pgSQL DO block: processes rows with "/" in filename, shallowest first. For each row, looks up the parent directory by its old full-path filename (which still exists at this point) and sets parent_id to that directory's UUID +3. **Strip filenames** to leaf names (`split_part` on last "/") +4. **Add FK constraint** (DEFERRABLE INITIALLY IMMEDIATE) +5. **Replace UNIQUE constraint** with `UNIQUE NULLS NOT DISTINCT (parent_id, filename, filetype) DEFERRABLE INITIALLY IMMEDIATE` +6. **Create parent index** on `(parent_id, filename)` +7. **Recreate view** -- PostgreSQL views with `SELECT *` snapshot columns at creation time; `ALTER TABLE ADD COLUMN` does NOT update existing views. The migration drops and recreates the view, preserving the tigerfs comment +8. **Migrate history table** (if exists): add parent_id, rename columns (`id` -> `file_id`, `_history_id` -> `version_id`, `_operation` -> `operation`), populate parent_id from source table, strip filenames, recreate BEFORE trigger with new column names +9. **Migrate log table** (if exists): rename `history_id` -> `version_id`, rename type values (`insert` -> `create`, `update` -> `edit`), update CHECK constraint. Order matters: drop old CHECK, rename values, then add new CHECK +10. **Create resolve_path function** (idempotent, shared across all apps) + +**Note:** The history parent_id migration uses the source table's CURRENT parent_id for each file_id. This is correct for files that haven't moved, but for files that were moved, earlier history entries will have the current parent_id rather than the historical one. This is an acceptable trade-off since the full path was also not preserved in the old history model. + +## Sequencing + +Implement ADR-017 as **Phase 13** before continuing Phase 12 undo tasks (12.5+). + +### Phase 13: Relational Directory Structure + +**13.1 Schema and DDL changes** +- Update `synth/build.go`: source table with `parent_id`, new UNIQUE constraint, `resolve_path` function +- Update history table DDL: `file_id`, `parent_id`, `version_id`, `operation` columns +- Update log table DDL: `version_id`, filesystem-centric type names +- Update BEFORE trigger to copy `parent_id` and use new column names +- Unit tests for new DDL generation + +**13.2 Path resolution** +- Create `resolve_path` PL/pgSQL function in `synth/build.go` DDL +- Add Go wrapper in `db/query.go`: `ResolvePath(ctx, schema, table, segments []string) (string, error)` +- Add Go-level path cache: `pathCache` with 2-second TTL, keyed on `(parent_id, filename)` +- Unit tests for path resolution and cache behavior + +**13.3 ReadDir by parent_id** +- Replace `GetAllRows` + `filterHierarchicalChildren` with `SELECT * WHERE parent_id = X` +- Add `GetRowsByParent(ctx, schema, table, parentID)` to DB layer +- Remove `filterHierarchicalChildren` +- Update `readDirSynthView` and `readDirSynthHierarchical` +- Unit and integration tests + +**13.4 Write path updates** +- `writeSynthFile`: resolve parent_id from path segments, insert/update with parent_id +- `ensureSynthParentDirs`: create parent chain with parent_id linking +- `mkdirSynth`: insert with parent_id +- Unit and integration tests + +**13.5 Rename and move** +- `renameSynthFile`: directory rename is `UPDATE SET filename='new' WHERE id=dir_id` (1 row) +- File rename: `UPDATE SET filename='new' WHERE id=file_id` +- File/directory move: `UPDATE SET parent_id=new_parent WHERE id=X` +- Remove `RenameByPrefix`, `HasChildrenWithPrefix` +- Log entries use filesystem-centric types: `rename` for rename/move +- Unit and integration tests + +**13.6 Delete path updates** +- `deleteSynthFile`: check children via `WHERE parent_id = dir_id` instead of LIKE prefix +- Remove `HasChildrenWithPrefix` usage +- Unit and integration tests + +**13.7 History and .history/ updates** +- Update history queries for new column names (`file_id`, `version_id`, `operation`) +- `.history/` navigation uses parent_id traversal (same as live table) +- Update `historyIDToVersionID` for `version_id` column name +- Unit and integration tests + +**13.8 Log entry updates** +- Rename operation types: create/edit/rename/delete/undo +- `logSynthOp`: compute denormalized full path by walking parent chain at log-write time +- Update `InsertLogEntry` for `version_id` column name +- Unit tests + +**13.9 Migration** +- `relational-directories` migration in `tigerfs migrate` framework (`cmd/migrate.go`) +- Test with existing demo data + +**13.10 Update existing tests** +- Rewrite all synth hierarchy tests for parent_id model +- Update integration tests +- Run full test suite + +**13.11 Update ADR-016 and Phase 12 tasks** +- Rewrite ADR-016 sections affected by schema changes: log table schema (version_id, filesystem-centric types), history references, column naming +- Update Phase 12 tasks 12.5-12.12 in implementation-tasks.md to reference the new schema (version_id, type names, parent_id model) +- Verify consistency between ADR-016, ADR-017, and implementation tasks + +**13.12 Documentation** +- Write ADR-017 (this document) to `docs/adr/` +- Update `docs/spec.md` with new schema and directory model +- Update skills if needed + +After Phase 13 is complete, resume Phase 12 from task 12.5. + +## Verification + +### Unit and integration test coverage + +All tests should cover both markdown and plain text formats. + +**Basic operations (unit + integration):** +1. Create file at root level +2. Create file in a nested directory (auto-create parents) +3. Edit file content +4. Rename file (same directory) +5. Move file between directories (change parent_id) +6. Delete file +7. Create directory (mkdir) +8. Rename directory +9. Move directory between parents +10. Delete empty directory +11. Delete non-empty directory → ENOTEMPTY + +**Path resolution (unit + integration):** +12. resolve_path for root-level file +13. resolve_path for deeply nested file (5 levels) +14. resolve_path for nonexistent path → returns NULL +15. resolve_path with cached start_parent (partial cache hit) +16. Path cache TTL expiry +17. Path cache invalidation on directory rename + +**ReadDir (integration):** +18. ReadDir at root level → only immediate children +19. ReadDir in subdirectory → only that directory's children +20. ReadDir on empty directory → empty list +21. ReadDir performance: fewer DB rows than GetAllRows approach + +**Complex undo scenarios (integration -- these are critical):** +22. Savepoint → edit file → undo to savepoint → file restored +23. Savepoint → create file → undo → file deleted +24. Savepoint → delete file → undo → file re-created +25. Savepoint → rename file → undo → old name restored +26. Savepoint → move file → undo → file back in original directory +27. Savepoint → rename directory → undo → directory and all children restored to old path +28. Savepoint → move directory → undo → directory back in original parent +29. Savepoint → edit + rename dir + edit + move dir + delete + create → undo all → exact savepoint state restored +30. Undo-of-undo: savepoint → edit → undo → undo-of-undo → post-edit state restored +31. Undo of delete then undo-of-that-undo (re-delete) +32. Idempotent: undo to same savepoint twice → same result +33. Filtered undo: per-user undo leaves other users' changes intact +34. Filtered undo FK failure: agent-1 creates dir, agent-2 adds file, undo agent-1 → error, DB unchanged +35. Filtered undo FK failure: agent-1 moves file to dir, agent-2 deletes dir, undo agent-1 → error, DB unchanged +36. UNIQUE conflict: rename A→B, create new A, undo → handled by DEFERRABLE constraint + +**History navigation (integration):** +37. `.history/file.md/` shows versions by file_id +38. `.history/` after rename → shows under new name, history under old name via .by/uuid/ +39. `.history/` with directory structure mirrors live parent_id hierarchy +40. File moved between directories → history entries accessible from both old and new locations via .by/uuid/ + +**Multi-agent (integration):** +41. Two agents: file moves between fixed directories (task board pattern) → no cache staleness +42. Two agents: one renames directory, other sees updated paths within 2 seconds + +**Demo verification:** +43. Run demo seed script → log tables populated for all operations +44. Rename a directory in the demo → one log entry created +45. Undo to savepoint in demo → all files restored correctly diff --git a/docs/adr/018-stress-test.md b/docs/adr/018-stress-test.md new file mode 100644 index 0000000..38fec33 --- /dev/null +++ b/docs/adr/018-stress-test.md @@ -0,0 +1,114 @@ +# ADR-018: Stress Test for File-First Workspaces + +**Status:** Accepted +**Date:** 2026-04-18 +**Author:** Mike Freedman + +## Context + +TigerFS file-first workspaces support a rich set of operations -- file creation, editing, renaming, moving, deletion, directory management -- plus undo capabilities via savepoints and operation logs. These features interact in complex ways: undo-of-undo chains, rename-as-replace with history tracking, savepoint rollback across many files, concurrent state in the log/history tables. + +Integration tests cover individual operations and specific edge cases, but do not exercise long sequences of mixed operations with interleaved undos. Real-world usage involves unpredictable operation sequences that may expose ordering bugs, state tracking errors, or data corruption that targeted tests miss. + +We need a stress test that: +- Exercises all operation types in randomized but reproducible sequences +- Verifies correctness after every operation (not just at the end) +- Tests undo at multiple granularities (single op, multi-op to log_id, to savepoint) +- Scales to large files and dense directories +- Is self-contained (no external infrastructure dependencies beyond Docker) + +## Decision + +Build `tigerfs-stress`, a standalone Go binary in `test/stress/` that: + +1. **Manages its own infrastructure**: spins up Docker PostgreSQL (TimescaleDB), builds and mounts TigerFS, creates a workspace, tears down on exit +2. **Runs deterministic randomized operations**: seeded PRNG drives all operation selection, target selection, and content generation -- same seed always produces identical run +3. **Tracks expected state in-memory**: maintains a `WorkspaceState` (path-to-hash map) updated after each operation, with a push/pop stack for undo rollback +4. **Validates against the real filesystem**: reads files back from the NFS mount, computes md5 hashes, compares to expected state. Detects missing files, unexpected files, and content mismatches +5. **Operates as a pure filesystem client**: all operations use `os.WriteFile`, `os.Rename`, `os.Remove`, etc. against the mounted filesystem. No tigerfs internal imports. Undo is triggered via `os.WriteFile` to `.undo/id//.apply` + +### Architecture + +``` + NFS mount + tigerfs-stress ──── os.WriteFile ────────────> TigerFS ──> PostgreSQL + │ │ + │ in-memory expected state │ actual data + │ (path -> md5 hash) │ (rows in DB) + │ │ + └── ValidateWorkspace() ── os.ReadFile ──────┘ + + md5 compare +``` + +The test loop: +1. Select random operation (weighted by type, respecting preconditions) +2. Push current expected state onto stack +3. Execute operation on real filesystem +4. Update expected state +5. Validate: walk filesystem, hash files, compare to expected +6. Repeat + +For undo operations: +1. Read log entries from `.log/.last/N/.export/json` +2. Apply undo via `.undo/id//.apply` or `.undo/to-savepoint//.apply` +3. Restore expected state from stack (undo_single pops one; undo_to_savepoint restores to saved index) +4. Validate + +### Operations tested + +| Operation | Description | +|-----------|-------------| +| create_file | New markdown file with PRNG-generated content | +| edit_file | Modify existing file body | +| rename_file | Rename within same directory | +| move_file | Move to different directory | +| delete_file | Remove existing file | +| create_dir | New subdirectory | +| rename_dir | Rename directory | +| create_savepoint | Named savepoint with snapshot | +| undo_single | Undo most recent operation | +| undo_to_id | Undo all operations after a log entry | +| undo_to_savepoint | Undo all operations after a savepoint | + +### Scale modes + +- **Default**: files up to 100KB, max 10 files per directory +- **`--large-files`**: files up to 10MB, log-normal size distribution +- **`--many-files`**: up to 1000 files per directory + +File sizes follow a log-normal distribution to model real-world file size patterns (many small files, few large ones). + +### Determinism + +All randomness flows through a single `math/rand.Rand` instance initialized with the provided seed. No goroutines, no time-dependent operations in the test loop. The seed is printed at startup; any failing run can be exactly reproduced with `--seed N`. + +### State tracking + +Only md5 hashes are stored in memory, not file content. Actual data lives in PostgreSQL via TigerFS. This keeps memory bounded even with `--large-files` mode -- the stress tester's heap is proportional to file count, not file size. + +### Not in scope + +- Concurrent multi-client testing (single sequential client) +- Data-first (table) mode testing (file-first only) +- Performance benchmarking (correctness only) +- NFS/FUSE adapter testing (goes through the real mount but doesn't test adapter-specific behavior) + +## Location + +`test/stress/` -- part of the root Go module but not in `cmd/`, so it is not packaged with releases. Built on demand: + +```bash +go build -o bin/tigerfs-stress ./test/stress +``` + +## Consequences + +- Every filesystem operation and undo mode has automated coverage via randomized sequences +- Regressions in undo, rename-as-replace, history tracking, or state management are caught by hash mismatches +- Seed-based reproducibility means any failure can be debugged deterministically +- The test is self-contained -- no shared infrastructure, no flaky external dependencies +- Unit tests for the stress tester itself (state tracking, validation, content generation) provide confidence in the test harness + +## Implementation + +See Phase 14 in `docs/implementation/implementation-tasks.md` (Tasks 14.1-14.5). diff --git a/docs/native-tables.md b/docs/data-first.md similarity index 99% rename from docs/native-tables.md rename to docs/data-first.md index 7f3bd60..9cba6d6 100644 --- a/docs/native-tables.md +++ b/docs/data-first.md @@ -1,4 +1,4 @@ -# Native Table Access +# Data-First Mode Access PostgreSQL tables as directories of files — read rows, write columns, navigate indexes, chain pipeline queries, and manage schema, all from the command line. @@ -7,7 +7,7 @@ Access PostgreSQL tables as directories of files — read rows, write columns, n TigerFS maps every table to a directory. Each row appears as a file (in your choice of format) or as a subdirectory of column files. Primary keys become filenames, columns become file contents, and standard Unix tools replace SQL. ``` -/mnt/db/public/users/ +/mnt/db/users/ ├── 1.json # Row as JSON file ├── 1.tsv # Row as TSV file ├── 1/ # Row as directory @@ -463,7 +463,7 @@ touch /mnt/db/.views/.create/active_users/.commit ```yaml # ~/.config/tigerfs/config.yaml filesystem: - dir_listing_limit: 10000 # Max rows in directory listing + dir_listing_limit: 1000 # Max rows in directory listing dir_filter_limit: 100000 # Skip value listing for tables larger than this query_timeout: 30s # Maximum query execution time no_filename_extensions: false # Set true to disable .txt/.json/.bin extensions diff --git a/docs/file-first.md b/docs/file-first.md new file mode 100644 index 0000000..58f7f21 --- /dev/null +++ b/docs/file-first.md @@ -0,0 +1,208 @@ +# File-First Mode + +File-first mode presents database tables as directories of files. Markdown files have YAML frontmatter mapped to columns. Plain text files are body-only. Multiple users and agents access the same files concurrently with transactional writes. With history enabled, every change is versioned and reversible -- create savepoints, preview changes, and undo when needed. + +## Creating Workspaces + +Workspaces tell TigerFS how to present a table as files. Write a format to `.build/` to create a new workspace: + +```bash +echo "markdown,history" > /mnt/db/.build/notes # Markdown with history (recommended) +echo "markdown" > /mnt/db/.build/notes # Markdown without history +echo "plaintext" > /mnt/db/.build/snippets # Plain text, no frontmatter +echo "history" > /mnt/db/.build/notes # Add history to existing workspace +``` + +Each workspace creates a directory backed by a table in the `tigerfs` schema: + +``` +/mnt/db/notes/ +├── hello.md # Your files +├── tutorials/ +│ └── getting-started.md +├── .history/ # Past versions (with history) +├── .log/ # Operation log (with history) +├── .savepoint/ # Bookmarks for undo (with history) +├── .undo/ # Preview and apply undo (with history) +└── .info/ # Workspace metadata +``` + +Access the backing table via `/mnt/db/.tables/notes/`. + +To add file-first access to an existing data-first table: + +```bash +echo "markdown" > /mnt/db/posts/.format/markdown +# Creates posts_md/ view (appends _md to avoid collision) +``` + +## File Formats + +### Markdown + +Each `.md` file has YAML frontmatter (from columns) and a body (from the body column): + +```markdown +--- +title: Getting Started +author: alice +tags: + - tutorial + - intro +draft: false +--- + +# Getting Started + +This is the body content stored in the text column... +``` + +### Plain Text + +Plain text files have body content only, no frontmatter: + +``` +This is the entire file content. +No YAML frontmatter is parsed or generated. +``` + +### How Frontmatter Works + +Frontmatter fields map to database columns. The write behavior depends on the column type: + +- **Known columns** (e.g., `title`, `author`): Omitting a key from frontmatter **keeps the old value**. To clear a field, set it explicitly: `title: ""` +- **Headers JSONB** (e.g., `tags`, `draft` -- keys with no dedicated column): **Full-replace on each write**. Omitting a key removes it from the database. +- **Body**: Always replaced with what you write. +- **Timestamps** (`created_at`, `modified_at`): File times only -- they don't appear in frontmatter and can't be set via writes. + +To see which columns a table has: `cat /mnt/db/.tables/notes/.info/columns` + +## Reading and Writing + +Standard file operations work as expected: + +```bash +# List all files +ls /mnt/db/notes/ + +# Read a file +cat /mnt/db/notes/hello-world.md + +# Search across all files +grep -r "TODO" /mnt/db/notes/ + +# Create with frontmatter +cat > /mnt/db/notes/new-post.md << 'EOF' +--- +title: New Post +author: bob +tags: [update] +--- + +Content goes here... +EOF + +# Edit with any editor +vim /mnt/db/notes/hello-world.md + +# Rename (updates the filename column) +mv /mnt/db/notes/old-name.md /mnt/db/notes/new-name.md + +# Delete +rm /mnt/db/notes/unwanted.md +``` + +## Directories + +Workspaces support subdirectories. Create directories with `mkdir` and organize files into them: + +```bash +mkdir /mnt/db/notes/tutorials + +cat > /mnt/db/notes/tutorials/getting-started.md << 'EOF' +--- +title: Getting Started +author: alice +--- + +Follow these steps... +EOF + +ls /mnt/db/notes/tutorials/ +# getting-started.md +``` + +**Auto-creation:** Writing a file with a path automatically creates parent directories. Writing to `notes/a/b/c.md` auto-creates `a/` and `a/b/`. + +**Atomic rename:** Renaming a directory atomically renames all files within it: + +```bash +mv /mnt/db/notes/tutorials /mnt/db/notes/guides +# All files under tutorials/ are now under guides/ +``` + +## Column Mapping + +### Automatic Detection + +TigerFS automatically detects column roles by naming convention (first match wins): + +| Role | Detected From (priority order) | Required | +|------|-------------------------------|----------| +| Filename | `filename`, `name`, `title`, `slug` | Yes | +| Body | `body`, `content`, `description`, `text` | Yes | +| Timestamps | `modified_at`, `updated_at` (modification time); `created_at` (creation time) | No | +| Extra Headers | `headers` (JSONB, merged into frontmatter) | No | +| Frontmatter | All remaining columns (excluding primary key) | -- | + +Timestamp columns set file modification/creation times (visible in `ls -l`) but are **not** rendered as frontmatter. + +### Explicit Mapping (Planned) + +Currently, column roles are always auto-detected by naming convention. A future release will allow explicit mapping for tables whose column names don't match the conventions. + +### Custom Frontmatter (Extra Headers) + +Tables created with `.build/` include a `headers JSONB` column for storing arbitrary frontmatter keys beyond the fixed schema columns. + +- On **read**, entries from `headers` are merged into YAML frontmatter after known columns, sorted alphabetically. +- On **write**, frontmatter keys that don't match a known column are collected into `headers`. +- **Overwrite semantics** -- the entire `headers` value is replaced on each write. Omitting a key removes it. + +Example: `title` and `author` are stored in their own columns. `tags` and `draft` (no dedicated columns) are stored together in `headers` JSONB. + +## Backing Table Access + +The underlying table is accessible for data-first operations: + +```bash +# .build/ workspaces +ls /mnt/db/.tables/notes/ # Data-first access to backing table +ls /mnt/db/notes/ # File-first access + +# .format/ views +ls /mnt/db/posts/ # Data-first (original table) +ls /mnt/db/posts_md/ # File-first (synthesized view) +``` + +See [data-first.md](data-first.md) for the full data-first reference. + +## History and Undo + +Workspaces created with `history` get automatic versioning, an operation log, savepoints, and undo. The dot-directories (`.history/`, `.log/`, `.savepoint/`, `.undo/`) are shown in the directory structure above. + +```bash +# Create savepoint, work, review, undo if needed +echo '{"description":"Before refactoring"}' > /mnt/db/notes/.savepoint/checkpoint.json +# ... make changes ... +diff -ru /mnt/db/notes/.undo/to-savepoint/checkpoint /mnt/db/notes/ -x '.*' +touch /mnt/db/notes/.undo/to-savepoint/checkpoint/.apply +``` + +See [History](history.md) for the full guide on version browsing, savepoints, undo, and recovery. + +## Further Reading + +- [History](history.md) -- Version browsing, savepoints, undo, and recovery +- [Data-First](data-first.md) -- Direct table access via row-as-file and row-as-directory +- [Recipes](../skills/tigerfs/recipes.md) -- Blog, knowledge base, task boards, and other patterns diff --git a/docs/history.md b/docs/history.md index 5821624..b074ce9 100644 --- a/docs/history.md +++ b/docs/history.md @@ -1,25 +1,61 @@ -# History +# History, Savepoints, and Undo -Automatic versioning for synthesized apps — every edit and delete is captured as a timestamped snapshot you can browse and recover. +Automatic versioning for file-first workspaces -- every edit and delete is captured as a timestamped snapshot you can browse, diff, and undo. ## What It Does -When history is enabled on a synthesized app, a PostgreSQL BEFORE trigger automatically copies the old row into a companion history table on every UPDATE and DELETE. Past versions appear as read-only files under a `.history/` directory alongside your current files. +When history is enabled on a workspace, every change is automatically tracked. The filesystem exposes four directories for browsing and managing your workspace's history: -- **Automatic** — no manual save or commit; every change is captured -- **Read-only** — `.history/` cannot be written to (returns EACCES) -- **Composable** — works with any synth format (markdown, text, future formats) -- **Add anytime** — enable at creation or add to an existing app later +``` +notes/ +├── hello.md # Your current files +├── tutorials/ +│ └── getting-started.md +├── .history/ # Browse past versions of any file +│ ├── hello.md/ +│ │ ├── .id +│ │ └── 2026-02-24T150000.123Z-abc123def +│ └── .by/ # Look up history by row UUID +├── .log/ # Operation log: what changed, when, by whom +│ └── / +│ ├── before → .history/... # Diff symlinks +│ ├── after → .history/... +│ └── current → hello.md +├── .savepoint/ # Named bookmarks for undo +│ ├── before-refactor/ +│ └── auto-agent-7-20260408T143000Z/ +└── .undo/ # Preview and apply undo + ├── id/ # Undo a single operation + ├── to-id/ # Undo to a log entry + └── to-savepoint/ # Undo to a savepoint +``` -## Quick Start +Key properties: +- **Automatic** -- no manual save or commit; every change is captured +- **Reversible** -- undo any change or roll back to any savepoint +- **Attributable** -- each operation records who made it (via `--user-id`) +- **Composable** -- works with any workspace format (markdown, text) +- **Add anytime** -- enable at creation or add to an existing workspace -Enable history when creating a new app: +## Quick Start ```bash +# Create a workspace with history echo "markdown,history" > /mnt/db/.build/notes + +# Create a savepoint before risky work +echo '{"description":"Before refactoring"}' > /mnt/db/notes/.savepoint/before-refactor.json + +# Work, explore, make changes... + +# Review what changed +diff -ru /mnt/db/notes/.undo/to-savepoint/before-refactor /mnt/db/notes/ -x '.*' + +# Undo if needed (atomic, all files at once) +touch /mnt/db/notes/.undo/to-savepoint/before-refactor/.apply ``` -Or add history to an existing app: +Or add history to an existing workspace: ```bash echo "history" > /mnt/db/.build/notes @@ -27,7 +63,7 @@ echo "history" > /mnt/db/.build/notes Both paths store the feature flag in the view comment (`tigerfs:md,history`). -## Browsing History +## Browsing History (.history/) ### List files that have history @@ -42,7 +78,7 @@ Each entry is a directory containing past versions of that file. ```bash ls /mnt/db/notes/.history/hello.md/ -# .id 2026-02-24T150000Z 2026-02-12T021500Z 2026-02-12T013000Z +# .id 2026-02-24T150000.123Z-abc123def 2026-02-12T021500.456Z-xyz789ghi 2026-02-12T013000.789Z-jkl012mno ``` Returns `.id` (the row UUID) followed by version timestamps, newest first. @@ -50,16 +86,14 @@ Returns `.id` (the row UUID) followed by version timestamps, newest first. ### Read a past version ```bash -cat /mnt/db/notes/.history/hello.md/2026-02-12T013000Z +cat /mnt/db/notes/.history/hello.md/2026-02-12T013000.789Z-jkl012mno ``` Returns the full file content (frontmatter + body) as it existed at that point. ## Version IDs -Version timestamps are extracted from the history row's UUIDv7, formatted as `2006-01-02T150405Z` — filesystem-safe with no colons. Versions are listed newest-first. - -Since UUIDv7 encodes millisecond-precision timestamps but the filesystem path uses second precision, sub-second edits may share the same version ID. +Version IDs use the UUIDv7 display format: `2026-02-24T150000.123Z-abc123def` (UTC timestamp with millisecond precision + base36 entropy suffix). This format is filesystem-safe, case-insensitive, lossless, and sorts chronologically. Versions are listed newest-first. ## Cross-Rename Tracking @@ -77,14 +111,9 @@ cat /mnt/db/notes/.history/hello.md/.id The `.by/` directory lets you look up history by row UUID, which works even after renames: ```bash -# List all row UUIDs with history ls /mnt/db/notes/.history/.by/ - -# List all versions for a specific UUID ls /mnt/db/notes/.history/.by/a1b2c3d4-e5f6-7890-abcd-ef1234567890/ - -# Read a past version by UUID -cat /mnt/db/notes/.history/.by/a1b2c3d4-e5f6-7890-abcd-ef1234567890/2026-02-12T013000Z +cat /mnt/db/notes/.history/.by/a1b2c3d4-e5f6-7890-abcd-ef1234567890/2026-02-12T013000.789Z-jkl012mno ``` `.by/` is only available at the root `.history/` level, not in subdirectory `.history/` directories. @@ -94,64 +123,161 @@ cat /mnt/db/notes/.history/.by/a1b2c3d4-e5f6-7890-abcd-ef1234567890/2026-02-12T0 Each directory has its own `.history/` scoped to files in that directory: ```bash -# History for files in the tutorials/ directory only ls /mnt/db/notes/tutorials/.history/ - -# Versions of a specific file in that directory ls /mnt/db/notes/tutorials/.history/intro.md/ ``` Subdirectory `.history/` does not include `.by/` (UUID browsing is root-level only). -## Recovering a Past Version +## Operation Log (.log/) + +Every create, edit, rename, and delete is recorded in the `.log/` directory. Each entry has a log_id (UUIDv7), the operation type, affected file, and the user who performed it. + +```bash +# Recent operations +ls /mnt/db/notes/.log/.last/10/ +cat /mnt/db/notes/.log/.last/10/.export/json + +# Filter by user or type +ls /mnt/db/notes/.log/.by/user_id/agent-7/.last/5/ +ls /mnt/db/notes/.log/.by/type/edit/.last/10/ +``` + +### Diff Symlinks + +Each log entry directory contains `before`, `after`, and `current` symlinks for diffing: + +```bash +# What did this edit change? +diff -u --color /mnt/db/notes/.log//before /mnt/db/notes/.log//after + +# How has the file drifted since this edit? +diff -u --color /mnt/db/notes/.log//before /mnt/db/notes/.log//current +``` + +- **before**: the file content before this operation (from `.history/`). `/dev/null` for creates. +- **after**: the file content after this operation. `/dev/null` for deletes. +- **current**: the live file. `/dev/null` if deleted. + +History symlink paths are per-directory: a file at `tutorials/getting-started.md` links to `tutorials/.history/getting-started.md/`. + +### Relationship to .history/ + +The log records *what happened* (operations). History stores *what it looked like* (content snapshots). Each log entry's `version_id` points to the history row that captured the file's state just before that operation. -Read the old version from `.history/`, then write it back to the current path: +## Savepoints (.savepoint/) + +Named bookmarks in the operation log timeline. Create one before risky work so you can undo back to that point. ```bash -# Read the version you want to restore -cat /mnt/db/notes/.history/hello.md/2026-02-12T013000Z > /tmp/restore.md +# Create a savepoint +echo '{"description":"Before investigating auth bug"}' > /mnt/db/notes/.savepoint/pre-investigation.json + +# List savepoints +ls /mnt/db/notes/.savepoint/ -# Write it back to the current file -cp /tmp/restore.md /mnt/db/notes/hello.md +# Read savepoint details +cat /mnt/db/notes/.savepoint/pre-investigation/description + +# Delete a savepoint (does not affect history or log) +rm /mnt/db/notes/.savepoint/pre-investigation ``` -The restored content becomes the current version, and the overwritten content is captured as a new history entry. +Savepoint creation requires a format suffix (`.json`, `.tsv`, `.csv`, `.yaml`). If `--user-id` is set, the user identity is auto-injected. + +### Auto-Savepoints + +TigerFS automatically creates savepoints when it detects an inactivity gap (default 30 minutes). Named `auto--` or `auto-` for anonymous mounts. + +Configure via `--auto-savepoint-interval` (set to `0` to disable). + +## Undo (.undo/) -## Use Cases +The `.undo/` directory provides a preview-then-apply interface for reversing operations. -### Shared Agent Memory +### Three Modes -Multiple AI agents read and write to the same directory. If one agent accidentally overwrites another's work, browse `.history/` to see what changed and recover the lost content. +| Mode | Purpose | +|------|---------| +| `.undo/id//` | Undo a single operation | +| `.undo/to-id//` | Undo all operations after a log entry | +| `.undo/to-savepoint//` | Undo all operations after a savepoint | -### Blog Audit Trail +### Preview and Apply + +```bash +# What would undo do? +cat /mnt/db/notes/.undo/to-savepoint/pre-investigation/.info/summary + +# Diff all affected files since savepoint +diff -ru /mnt/db/notes/.undo/to-savepoint/pre-investigation /mnt/db/notes/ -x '.*' + +# Apply undo (atomic, all files at once) +touch /mnt/db/notes/.undo/to-savepoint/pre-investigation/.apply + +# Undo only a specific agent's changes +touch /mnt/db/notes/.undo/to-savepoint/pre-investigation/.by/user_id/agent-7/.apply + +# Diff and undo a single file change +diff -u --color /mnt/db/notes/.log//before /mnt/db/notes/.log//current +touch /mnt/db/notes/.undo/id//.apply +``` -Track every edit to published content. Compare past versions to see what was added, removed, or reworded. +### Undo of Undo -### Knowledge Base Versioning +Undo operations are themselves logged (with type='undo'). You can undo an undo. Create a savepoint before a major undo for extra safety. -Maintain a living knowledge base where articles are frequently updated. History provides a full audit trail without manual version management. +## Recovering Past Versions + +**Single file via undo (preferred):** Find the change in the log, then undo it: + +```bash +cat /mnt/db/notes/.log/.by/filename/hello.md/.last/5/.export/json +touch /mnt/db/notes/.undo/id//.apply +``` + +**Multi-file rollback:** Undo all changes since a savepoint: + +```bash +touch /mnt/db/notes/.undo/to-savepoint/before-investigation/.apply +``` + +## Limitations + +- **Requires TimescaleDB:** history, log, and savepoint tables use TimescaleDB hypertables for compressed storage. Will not work on vanilla PostgreSQL. +- **File-first only:** data-first tables don't get history. Writing directly to the backing table via `.tables/` bypasses the history trigger. +- **Per-user undo caveat:** if two users interleave edits on the same file, undoing one user's changes also reverts the other user's interleaved edits on that file. +- **Storage cost:** every edit creates a history row. TimescaleDB compression mitigates this (7-day chunks, automatic compression). ## How It Works -- **Companion table** -- Each history-enabled app gets a `tigerfs._history` table (e.g., `tigerfs.notes_history`) that mirrors the source table's columns plus history metadata (`_history_id`, `_operation`) -- **Trigger** — A PostgreSQL BEFORE UPDATE/DELETE trigger copies the OLD row into the history table on every change -- **TimescaleDB hypertable** — The history table is partitioned by `_history_id` (UUIDv7) with 1-month chunks, compressed after 1 day, using `segment_by='filename'` and `order_by='_history_id DESC'` -- **Detection** -- TigerFS detects history via the view comment (`tigerfs:md,history`) or by checking for a companion `tigerfs._history` table +Each history-enabled workspace is backed by three companion tables in the `tigerfs` schema, alongside the main backing table. + +The **history table** stores a snapshot of every file version. A PostgreSQL BEFORE trigger fires on every update and delete, copying the old row into history with a UUIDv7 version_id. The trigger detects whether the operation was an edit, rename, or delete by comparing the old and new rows. + +The **log table** records each operation (create, edit, rename, delete, undo) with who did it and when. Log entries reference history entries via version_id, connecting "what happened" to "what it looked like." + +The **savepoint table** stores named bookmarks. Each savepoint's UUIDv7 timestamp enables efficient "find all operations after this point" queries. -## Requirements +All three tables use TimescaleDB hypertables for time-partitioned storage and automatic compression. The log table is indexed on (file_id, log_id) for SkipScan-optimized undo queries. -History requires **TimescaleDB** — it will not work on vanilla PostgreSQL. The companion history table uses TimescaleDB hypertables for efficient time-partitioned storage and automatic compression. +TigerFS detects history via the view comment (`tigerfs:md,history`) or by checking for a companion history table. ## Quick Reference | Goal | Path | |------|------| -| List files with history | `ls mount/app/.history/` | -| List versions of a file | `ls mount/app/.history/file.md/` | -| Read a past version | `cat mount/app/.history/file.md/` | -| Get row UUID | `cat mount/app/.history/file.md/.id` | -| List all row UUIDs | `ls mount/app/.history/.by/` | -| Versions by UUID | `ls mount/app/.history/.by//` | -| Read version by UUID | `cat mount/app/.history/.by//` | -| Subdirectory history | `ls mount/app/subdir/.history/` | -| Restore old version | Read from `.history/`, write to current path | +| List files with history | `ls workspace/.history/` | +| List versions of a file | `ls workspace/.history/file.md/` | +| Read a past version | `cat workspace/.history/file.md/` | +| Get row UUID | `cat workspace/.history/file.md/.id` | +| Versions by UUID | `ls workspace/.history/.by//` | +| Recent log entries | `cat workspace/.log/.last/10/.export/json` | +| Diff a specific change | `diff -u workspace/.log//before workspace/.log//after` | +| Create savepoint | `echo '{"description":"..."}' > workspace/.savepoint/name.json` | +| List savepoints | `ls workspace/.savepoint/` | +| Preview undo to savepoint | `cat workspace/.undo/to-savepoint/name/.info/summary` | +| Diff all since savepoint | `diff -ru workspace/.undo/to-savepoint/name workspace/ -x '.*'` | +| Undo to savepoint | `touch workspace/.undo/to-savepoint/name/.apply` | +| Undo single change | `touch workspace/.undo/id//.apply` | +| Per-user undo | `touch workspace/.undo/to-savepoint/name/.by/user_id/X/.apply` | diff --git a/docs/implementation/implementation-tasks-checklist.md b/docs/implementation/implementation-tasks-checklist.md index 743423e..7988bb9 100644 --- a/docs/implementation/implementation-tasks-checklist.md +++ b/docs/implementation/implementation-tasks-checklist.md @@ -261,6 +261,58 @@ Quick reference for task status. Line numbers reference `implementation-tasks.md --- +## Phase 12: Undo and Recovery + +| Status | Task | Description | Line | +|--------|------|-------------|------| +| ✅ | 12.1 | UUIDv7 Display Format (timestamp+base36) | ~8680 | +| ✅ | 12.2 | Symlink Support (Adapter Plumbing) | ~8730 | +| ✅ | 12.3 | Log and Savepoint Table DDL | ~8780 | +| ✅ | 12.4 | Log Entry Creation in Write Path | ~8820 | +| ✅ | 12.5 | User Identity (.info/user) | ~8860 | +| ✅ | 12.6 | Path Parsing for .log/, .savepoint/, .undo/ | ~8900 | +| ✅ | 12.7 | .log/ Interface (Data-First on Log Table + Diff Symlinks) | ~8930 | +| ✅ | 12.8 | .savepoint/ Interface (Data-First + Write Support) | ~8970 | +| ✅ | 12.9 | Auto-Savepoints (Session-Based) | ~9000 | +| ✅ | 12.10 | Undo Execution Logic (Core Engine) | ~9030 | +| ✅ | 12.11 | .undo/ Interface (Preview + Apply) | ~9080 | +| ✅ | 12.12 | Short-Lived Caching (Undo, Log, Savepoint) | ~9100 | +| ✅ | 12.13 | Skills | ~9150 | +| ✅ | 12.14 | Documentation | ~9180 | + +--- + +## Phase 13: Relational Directory Structure + +| Status | Task | Description | Line | +|--------|------|-------------|------| +| ✅ | 13.1 | Schema and DDL Changes (parent_id, version_id, resolve_path) | ~9200 | +| ✅ | 13.2 | Path Resolution (resolve_path + Go cache) | ~9230 | +| ✅ | 13.3 | ReadDir by parent_id | ~9260 | +| ✅ | 13.4 | Write Path Updates (parent_id) | ~9280 | +| ✅ | 13.5 | Rename and Move (single-row operations) | ~9300 | +| ✅ | 13.6 | Delete Path Updates | ~9330 | +| ✅ | 13.7 | History and .history/ Updates | ~9350 | +| ✅ | 13.8 | Log Entry Updates (FS-centric types, version_id) | ~9370 | +| ✅ | 13.9 | Migration Script | ~9390 | +| ✅ | 13.10 | Update Existing Tests (45 verification scenarios) | ~9410 | +| ✅ | 13.11 | Update ADR-016 and Phase 12 Tasks | ~9430 | +| ✅ | 13.12 | Documentation (ADR-017, spec, skills) | ~9450 | + +--- + +## Phase 14: Stress Test (`tigerfs-stress`) + +| Status | Task | Description | Line | +|--------|------|-------------|------| +| ✅ | 14.1 | Scaffolding + Infrastructure (CLI, Docker, mount, teardown) | ~9470 | +| ✅ | 14.2 | State Tracking + Validation (WorkspaceState, stack, ValidateWorkspace) | ~9500 | +| ✅ | 14.3 | Operations + Content Generation (all FS ops, size distribution, pools) | ~9530 | +| ✅ | 14.4 | Runner + Undo + End-to-End (test loop, undo ops, wiring) | ~9560 | +| ✅ | 14.5 | README Finalization (architecture docs, usage guide) | ~9590 | + +--- + ## Summary | Phase | Complete | Total | Progress | @@ -280,4 +332,7 @@ Quick reference for task status. Line numbers reference `implementation-tasks.md | Phase 9: Shared Core Library | 17 | 17 | 100% | | Phase 10: Performance | 0 | 4 | 0% | | Phase 11: Skills | 1 | 1 | 100% | -| **Total** | **106** | **123** | **86%** | +| Phase 12: Undo and Recovery | 14 | 14 | 100% | +| Phase 13: Relational Directory Structure | 12 | 12 | 100% | +| Phase 14: Stress Test | 5 | 5 | 100% | +| **Total** | **133** | **154** | **86%** | diff --git a/docs/implementation/implementation-tasks.md b/docs/implementation/implementation-tasks.md index baff4e0..1d53eb8 100644 --- a/docs/implementation/implementation-tasks.md +++ b/docs/implementation/implementation-tasks.md @@ -744,7 +744,7 @@ cat /tmp/testmount/users/.schema go run ./cmd/tigerfs postgres://... /tmp/testmount # Test TSV update -echo -e "1\tnew@example.com" > /tmp/testmount/users/1 +echo -e "id\temail\n1\tnew@example.com" > /tmp/testmount/users/1.tsv cat /tmp/testmount/users/1/email # Should show: new@example.com @@ -8663,3 +8663,622 @@ goreleaser release --snapshot --clean - **Ask questions when blocked** - Don't guess or skip ahead **Ready to begin! Start with Task 1.1: Evaluate and Select FUSE Library** + + +--- + +## Phase 12: Undo and Recovery + +Design spec: `docs/adr/016-undo-and-recovery.md` + +Tasks ordered: infrastructure first (UUIDv7 display format, symlink support), then undo features incrementally. Demo data grows with each step so every feature can be manually tested as it lands. + +### Task 12.1: UUIDv7 Display Format +Implement timestamp+base36 globally before any new undo code exists. All new code uses the new format from day one. + +**Depends on:** Nothing +**Files:** `internal/tigerfs/fs/synth/format.go`, `internal/tigerfs/db/query.go`, `internal/tigerfs/db/pk_match.go`, `internal/tigerfs/fs/history.go` +**Tasks:** +1. Add `UUIDv7ToDisplayName()` and `DisplayNameToUUIDv7()` in synth/format.go +2. Add `isUUIDv7()` helper (check version bits 48-51 = 0111) +3. Update `scanAndEncodePK()`: detect UUIDv7, use display format for filenames +4. Update PK path resolution (pk_match.go): accept display format, convert back to UUID +5. Update `.history/` version ID generation (replaces `UUIDv7ToVersionID`) +6. Update `.history/` version ID parsing to accept new format +7. Accept both hex UUID and display format in `.by//` paths +8. Update ALL existing tests referencing hex UUIDs or old version ID format +9. Unit tests: encode/decode round-trip, isUUIDv7 detection, mixed v4/v7 tables +10. Integration tests: data-first table with UUIDv7 PK, .history/ display +**Manual test:** `ls mount/notes/.history/hello.md/` shows timestamp+base36 entries + +### Task 12.2: Symlink Support (Adapter Plumbing) +Add full symlink lifecycle to FUSE + NFS adapters: create, stat, readlink, read-through, write-through, delete. + +**Depends on:** Nothing +**Files:** `internal/tigerfs/fs/operations.go`, `internal/tigerfs/fs/types.go`, `internal/tigerfs/nfs/ops_filesystem.go`, `internal/tigerfs/fuse/ops_node.go`, `internal/tigerfs/fuse/adapter.go` +**Tasks:** +1. Add `Readlink(ctx, path) (string, *FSError)` to fs.Operations (stub returning ErrNotSupported for non-symlink paths) +2. Add `os.ModeSymlink` support to Entry.Mode in types.go +3. NFS adapter: implement `Readlink()` -> delegate to ops; update `Lstat()` to return symlink info without following; update `opsFileInfo.Mode()` for S_IFLNK; ensure `Stat()` follows symlinks (returns target's info) +4. FUSE adapter: add `Readlink(ctx) ([]byte, syscall.Errno)` on OpsNode; update `EntryToAttr` for S_IFLNK +5. Ensure kernel/client-level symlink following works: `cat ` should resolve through to the target file and return its content; `echo > ` should write to the target +6. Unit tests for symlink Entry mode handling, adapter dispatch, Lstat vs Stat behavior +7. Integration tests: + - Create symlink at table root level, verify: `readlink` returns target path, `stat` shows S_IFLNK mode, `cat` returns target file content + - Create symlink within a subdirectory, verify same behaviors + - Write through symlink, verify target file is modified + - `stat` on symlink shows S_IFLNK; `stat` following symlink shows target's type/size + - Delete symlink, verify gone; verify target file still exists and unchanged + - Symlink to `/dev/null`: verify `cat` returns empty, `stat` works +8. **Cross-platform verification:** Run symlink integration tests on both NFS (macOS) and FUSE (Linux). Both adapters must handle all symlink operations identically. + +### Task 12.3: Log and Savepoint Table DDL +Extend `synth/build.go` to create the log hypertable and savepoint table. **Only for history-enabled synth apps** -- apps without history do not get these tables. + +**Depends on:** Nothing +**Files:** `internal/tigerfs/fs/synth/build.go` +**Tasks:** +1. Add `CREATE TABLE tigerfs._log` with all columns from ADR Section 1.2 (log_id, file_id, type, user_id, filename, version_id, description) +2. Add `create_hypertable()` with `chunk_time_interval => INTERVAL '1 month'` (Section 1.3) +3. Add composite index `(file_id, log_id ASC)` (Section 1.4) +4. Add compression policy: `segmentby=file_id`, `orderby=log_id ASC`, compress after 1 day (Section 1.5) +5. Add `CREATE TABLE tigerfs._savepoint` with all columns (Section 2.2) +6. Conditional: only create log/savepoint tables when history feature is enabled. Synth apps without history skip these. +7. Unit tests for DDL generation (verify SQL includes all columns, hypertable, index, compression) +8. Integration tests: + - Create synth app WITH history: verify log and savepoint tables exist with correct schema + - Create synth app WITHOUT history: verify log and savepoint tables do NOT exist +**Prerequisites note:** Requires TimescaleDB >= 2.20.0 and PostgreSQL >= 16 (for compressed-chunk SkipScan) +**Manual test:** Connect to DB, confirm tables, indexes, and compression policies + +### Task 12.4: Log Entry Creation in Write Path +Insert log entries on writes. Every file operation now produces a log record. + +**Depends on:** 12.3 (log table exists) +**Files:** `internal/tigerfs/fs/write.go`, `internal/tigerfs/fs/synth_ops.go`, `internal/tigerfs/db/query.go` +**Tasks:** +1. Add `InsertLogEntry()` to DB layer +2. Determine `version_id` capture mechanism (ADR Section 7.2) +3. Integrate into `writeSynthFile()` (insert/update), `deleteSynthFile()`, `renameSynthFile()` +4. user_id is NULL for now (identity not yet wired) +5. Unit tests for log entry creation +6. Integration tests: create/edit/delete/rename files, verify log entries with correct type, file_id, filename, version_id (NULL for create, non-NULL for edit/rename/delete) +**Demo data:** Create `_demo` synth app with history. Write several files across subdirectories (`docs/intro.md`, `docs/api.md`, `guides/setup.md`). Edit some files. Delete one. Rename one. Query the DB directly to confirm log entries exist with correct data. + +### Task 12.5: User Identity (.info/user) +Add mount-level user identity. Wire into log entries so operations are tagged with who made them. Identity is in-memory only -- does NOT persist across unmount/remount. This is by design: identity is a session-level setting. For persistent identity, use `--user-id` at mount time or `TIGERFS_USER_ID` env var (both are set before mount, so they're "persistent" in that sense). + +**Depends on:** 12.4 (log entries exist, now add user_id to them) +**Files:** `internal/tigerfs/fs/operations.go`, `internal/tigerfs/cmd/mount.go`, `internal/tigerfs/config/config.go` +**Tasks:** +1. Add `UserID` field to Config, bind `--user-id` flag and `TIGERFS_USER_ID` env +2. Add `userID` field to Operations struct, initialized from config at mount time +3. Add root-level `.info/` directory with `user` file (read/write) +4. Wire `userID` into `InsertLogEntry()` calls from 12.4 +5. Unit tests for identity resolution (flag > env > NULL) +6. Integration tests: + - Mount with --user-id, `cat .info/user` returns it + - `echo "new-user" > .info/user`, subsequent log entries have new user_id + - Unmount/remount without --user-id: `.info/user` is empty/NULL (not persisted) + - Remount with --user-id: `.info/user` has the flag value again +**Demo step:** Set `.info/user` to `demo-user`. Make more edits. Query DB to confirm log entries now have user_id set. Change user to `agent-explorer`, make more edits, confirm different user_id in log. + +### Task 12.6: Path Parsing for .log/, .savepoint/, .undo/ +Add new path types and parsing. Key design: `.log/` and `.savepoint/` delegate to existing data-first pipeline parsing (reuse `processSegments` for `.by/`, `.filter/`, `.last/`, `.first/`, `.order/`, `.export/`, `.columns/`, `.sample/`, `.all/`). No new pipeline parsing code needed -- just new entry points that set the target table and delegate. `.undo/` has additional structure (`id/`, `to-id/`, `to-savepoint/` routing + `.apply` trigger + `.info/summary`) that requires new parsing. + +**Depends on:** Nothing (pure parsing logic) +**Files:** `internal/tigerfs/fs/path.go`, `internal/tigerfs/fs/constants.go` +**Tasks:** +1. Add constants: `DirLog`, `DirSavepoint`, `DirUndo` +2. Add to `capabilityDirectories` map +3. Add `PathLog`, `PathSavepoint`, `PathUndo` path types with parsed fields +4. `.log/` and `.savepoint/` paths: after recognizing the dot-directory, delegate remaining segments to existing pipeline parsing (same code that handles `.by/`, `.filter/`, etc. for data-first tables) +5. `.undo/` paths: parse `id/`, `to-id/`, `to-savepoint/` routing, then target identifiers, then pipeline segments (reuse), then `.info/summary` or `.apply` leaf +6. Unit tests for all path variants: + - `.log/.last/10`, `.log/.by/user_id/agent-7/.export/json`, `.log//before`, `.log//after`, `.log//current` + - `.savepoint/name/description`, `.savepoint/.by/user_id/agent-7/.last/5` + - `.undo/id//.apply`, `.undo/id//.info/summary` + - `.undo/to-id//.apply`, `.undo/to-id//.by/user_id//.apply` + - `.undo/to-savepoint//.apply`, `.undo/to-savepoint//.by/user_id//.info/summary` + - `.undo/to-savepoint//.filter/type/delete/.apply` + - `.undo/to-savepoint//.last/5/.info/summary` + - `.undo/to-savepoint//.sample/5/` -- path parses but `.apply` not available under `.sample/` + +### Task 12.7: .log/ Interface (Data-First on Log Table) +Expose the log through the filesystem using existing data-first pipeline code (no new pipeline logic -- just create FSContext targeting the log table and delegate). Add diff symlinks on log entries. + +**Depends on:** 12.2 (symlinks), 12.4 (log entries exist), 12.6 (path parsing) +**Files:** `internal/tigerfs/fs/operations.go` +**Tasks:** +1. Add `readDirLog()`, `statLog()`, `readFileLog()` -- create FSContext targeting `tigerfs._log`, delegate to existing data-first pipeline handlers +2. Add `readlinkLog()` -- resolve before/after/current symlinks (ADR Section 4.1.1) +3. Wire into dispatch switch statements +4. Unit tests for symlink resolution (all state matrix cases from ADR: INSERT/UPDATE/DELETE/UNDO, /dev/null for missing states) +5. Integration tests -- pipeline operations: + - `ls .log/.last/10` -- most recent entries + - `ls .log/.first/5` -- oldest entries + - `cat .log//type` -- single column access + - `cat .log/.json` -- row as JSON + - `ls .log/.by/user_id/agent-7` -- filter by user + - `ls .log/.by/type/delete` -- filter by type + - `ls .log/.by/filename/hello.md` -- filter by filename + - `cat .log/.by/user_id/agent-7/.last/5/.export/json` -- chained pipeline + - `cat .log/.export/tsv` -- bulk export + - `ls .log/.order/filename` -- sorted +6. Integration tests -- diff symlinks: + - `readlink .log//before` -- points to .history/ path + - `readlink .log//after` -- points to .history/ path or current file + - `readlink .log//current` -- points to current file or /dev/null + - `readlink .log//before` -- points to /dev/null + - `readlink .log//after` -- points to /dev/null + - `diff -u --color .log//before .log//after` -- produces correct diff output + - `cat .log//before` -- read through symlink returns file content +7. **SkipScan verification:** Run `EXPLAIN ANALYZE` on the pipeline query used for `.log/.last/N` or `.log/.by/file_id/` listings. Verify the `(file_id, log_id ASC)` index is used and SkipScan activates for `DISTINCT ON` queries when applicable. +**Demo step:** With the demo data: +- `ls mount/demo/.log/.last/10` -- see recent operations with timestamp+base36 IDs +- `cat mount/demo/.log/.last/5/.export/json` -- JSON output +- `diff -u --color mount/demo/.log//before mount/demo/.log//after` -- see a colored diff +- `ls mount/demo/.log/.by/user_id/demo-user` vs `.by/user_id/agent-explorer` -- filtered views + +### Task 12.8: .savepoint/ Interface (Data-First on Savepoint Table) +Expose savepoints using existing data-first pipeline code (FSContext targets savepoint table with `name` as display filename). Add write support for create/update/rename/delete. + +**Depends on:** 12.3 (savepoint table), 12.6 (path parsing) +**Files:** `internal/tigerfs/fs/operations.go`, `internal/tigerfs/fs/write.go` +**Tasks:** +1. Add `readDirSavepoint()`, `statSavepoint()`, `readFileSavepoint()` -- delegate to pipeline with `name` as display filename +2. With `name` as PK, savepoints use standard `writeRowFile` (no custom CRUD functions). A thin `writeSavepoint` wrapper injects `user_id` from mount identity. Format suffix required for creation (`.tsv`, `.json`, `.csv`) to avoid macOS NFS inode cache conflicts. +3. Wire into dispatch switch statements +4. Unit tests +5. Integration tests -- CRUD operations: + - `echo '{}' > .savepoint/my-checkpoint.json` -- creates with name only + - `echo -e "description\nmy desc" > .savepoint/my-checkpoint.tsv` -- creates with description + - `echo '{"description":"my desc","user_id":"agent-7"}' > .savepoint/my-checkpoint.json` -- creates with all fields + - `cat .savepoint/my-checkpoint/description` -- read individual field + - `cat .savepoint/my-checkpoint/savepoint_id` -- read auto-generated UUIDv7 + - `cat .savepoint/my-checkpoint.json` -- read full row as JSON + - `echo "new desc" > .savepoint/my-checkpoint/description` -- update description + - `rm .savepoint/my-checkpoint` -- delete + - Verify delete only removes bookmark, not log entries +6. Integration tests -- pipeline operations: + - `ls .savepoint/.last/5` -- most recent + - `ls .savepoint/.by/user_id/demo-user` -- filter by user + - `cat .savepoint/.export/json` -- bulk export + - `ls .savepoint/.order/name` -- sorted + - `ls .savepoint/.all/` -- full listing +**Demo step:** With demo data: +- `echo -e "description\nInitial data loaded" > mount/demo/.savepoint/initial-data.tsv` +- Make more edits (update files, delete one, create new one) +- `echo -e "description\nAfter edits" > mount/demo/.savepoint/after-edits.tsv` +- `ls mount/demo/.savepoint/` -- see both savepoints listed by name +- `cat mount/demo/.savepoint/initial-data.json` -- see details +- `rm mount/demo/.savepoint/initial-data` -- delete + +### Task 12.9: Auto-Savepoints +Implement session-based auto-savepoint creation. + +**Depends on:** 12.4 (log entries for gap detection), 12.8 (savepoint write) +**Files:** `internal/tigerfs/fs/operations.go`, `internal/tigerfs/config/config.go` +**Tasks:** +1. Add `AutoSavepointInterval` to Config (default 30m), bind flag and env +2. Check last log entry timestamp before each write; create auto-savepoint if gap > threshold +3. Auto-savepoint naming: `auto--` or `auto-` +4. Unit tests for gap detection and naming (use injected clock, not real sleeps, to avoid flaky tests). Test that interval=0 disables auto-savepoints entirely. +5. Integration test: use a very short interval (100ms), perform two writes separated by a `time.Sleep(150ms)`, verify auto-savepoint created between them. Use generous timing margins to avoid flakiness. Alternatively, use a mock/injectable clock in the gap detection logic so tests don't depend on wall-clock timing at all. +**Demo step:** Set interval to 1s for testing. Write, wait, write. `ls mount/demo/.savepoint/auto-*` + +### Task 12.10: Undo Execution Logic +Implement core undo operations. Tested via direct function calls. + +**Depends on:** 12.4 (log entries), 12.8 (savepoints for to-savepoint lookups) +**Files:** `internal/tigerfs/db/query.go`, `internal/tigerfs/fs/undo.go` (new) +**Tasks:** +1. Add `QueryUndoAffectedFiles()` -- DISTINCT ON query (ADR Section 1.8). Also add `QueryNextLogEntry(file_id, log_id)` for `after` symlink resolution using the `(file_id, log_id ASC)` index. +2. Add `ExecuteUndo()` -- single transaction: DELETE inserts, UPSERT updates/deletes, AND insert `type='undo'` log entries for each restored file within the same transaction (ADR Section 3.3). Set `description` on undo log entries (e.g., "Undo to savepoint before-exploration"). +3. Add `ExecuteUndoSingle()` -- single operation undo, also inserts `type='undo'` log entry. +4. Handle pipeline filter composition (`.by/user_id/`, `.filter/type/`, `.filter/filename/`, `.last/N/`, `.first/N/`). Reject `.sample/` (return error if `.apply` attempted after `.sample/`). +5. Handle DDL limitation gracefully: if undo SQL fails due to schema mismatch (column removed after savepoint), return a descriptive error rather than crashing. +6. Unit tests for query generation with filters +7. Integration tests (direct function calls, not filesystem): + - Undo single INSERT/UPDATE/DELETE + - Undo to savepoint (multiple files, mixed ops) + - Undo to log_id + - Undo by user + - Undo with `.filter/type/delete/` + - Undo-of-undo + - Idempotent double undo to same savepoint +8. **SkipScan verification:** Run `EXPLAIN ANALYZE` on the `QueryUndoAffectedFiles()` DISTINCT ON query against a test table with log entries. Verify the plan shows `Custom Scan (SkipScan)` nodes, confirming the `(file_id, log_id ASC)` index is being used efficiently. + +### Task 12.11: .undo/ Interface (Preview + Apply) +Expose undo preview and apply through the filesystem. Full end-to-end demo. The default listing limit for undo sub-directories is configurable (not hardcoded). + +**Depends on:** 12.2 (symlinks for /dev/null), 12.6 (path parsing), 12.10 (undo execution) +**Files:** `internal/tigerfs/fs/operations.go`, `internal/tigerfs/fs/write.go`, `internal/tigerfs/fs/undo.go`, `internal/tigerfs/config/config.go` +**Tasks:** +1. Add `UndoListLimit` to Config (default 100), bind `--undo-list-limit` flag and `TIGERFS_UNDO_LIST_LIMIT` env. Store in standard config location alongside other limit configs. +2. Add `readDirUndo()` -- top-level `id/`, `to-id/`, `to-savepoint/`; sub-dirs use `UndoListLimit` as default + full pipeline; leaf entries as preview directory trees with /dev/null symlinks for deletes +3. Add `statUndo()`, `readFileUndo()` -- .info/summary (TSV + format suffixes), affected file content from history +4. Add `writeUndoApply()` -- trigger on both Create (`touch`) and Write (`echo`), following the DDL Chtimes adapter pattern. Log at Info level with structured fields: `zap.String("target", ...)`, `zap.Int("files_restored", ...)`, `zap.Int("files_deleted", ...)` +5. Wire into all dispatch switch statements including Readlink +6. Unit tests for preview generation, summary formatting (TSV and JSON), /dev/null symlinks in preview tree +7. Integration tests -- navigation and preview: + - `ls .undo/` shows `id/`, `to-id/`, `to-savepoint/` + - `ls .undo/id/` shows entries up to UndoListLimit + - `ls .undo/id/.all/` shows all entries + - `ls .undo/to-savepoint/` shows savepoints + - `cat .undo/to-savepoint//.info/summary` -- correct TSV + - `cat .undo/to-savepoint//.info/summary.json` -- correct JSON + - `cat .undo/to-savepoint//docs/hello.md` -- restored content + - `ls .undo/to-savepoint//` -- shows affected file directory tree +8. Integration tests -- apply via `id/`: + - Undo single INSERT: file is deleted + - Undo single UPDATE: file restored to before-state + - Undo single DELETE: file re-inserted from history + - Undo an UNDO operation: original state returns (undo-of-undo) + - Verify log entries created with type=undo for each apply +9. Integration tests -- apply via `to-savepoint/`: + - Create savepoint, make mixed ops (insert + update + delete), undo to savepoint + - Verify all files restored to savepoint state, new file deleted, deleted file re-inserted + - Idempotent: undo to same savepoint again, verify same data state +10. Integration tests -- apply via `to-id/`: + - Undo to a specific log entry, verify ops after that entry are reversed +11. Integration tests -- filtered undo: + - `.undo/to-savepoint//.by/user_id//.apply` -- only that user's ops undone + - `.undo/to-savepoint//.filter/type/delete/.apply` -- only deletes undone + - `.undo/to-savepoint//.filter/filename/hello.md/.apply` -- only one file undone + - `.undo/to-savepoint//.last/3/.apply` -- only last 3 ops undone + - Combined: `.undo/to-savepoint//.by/user_id//.filter/type/delete/.apply` +12. Integration tests -- pipeline on undo sub-directories: + - `ls .undo/id/.by/user_id/agent-7` -- filtered listing + - `ls .undo/id/.by/type/update/.last/5` -- chained + - `cat .undo/to-savepoint/.export/json` -- export savepoints +13. Integration tests -- edge cases: + - Undo with no operations to undo (empty summary, .apply is no-op) + - Undo on a file that was inserted then deleted after savepoint (no-op, correct) + - Undo on a file that was updated then deleted (re-insert from history) + - Preview then apply: verify preview content matches what actually gets restored + - Invalid log_id in `.undo/id//`: returns ENOENT + - Invalid savepoint name in `.undo/to-savepoint//`: returns ENOENT + - `.undo/to-savepoint//.sample/5/.apply`: returns error (`.sample/` restriction) + - `.undo/to-savepoint//.order/filename/.apply`: succeeds (`.order/` ignored by apply) + - `.undo/to-savepoint//.columns/log_id,filename/.apply`: succeeds (`.columns/` ignored) + - `echo "confirm" > .undo/to-savepoint//.apply`: works same as `touch` (both trigger mechanisms) +14. Integration tests -- filename resolution in preview: + - Create file A, savepoint, rename A to B, preview undo to savepoint: verify preview shows A (from history entry), not B (from current/log) + - Create file, savepoint, delete file, preview undo: verify deleted file appears as /dev/null symlink in preview tree +15. Integration tests -- `to-id/` with `.by/user_id/`: + - Set user to agent-7, make edits; set user to agent-9, make edits; undo to-id with .by/user_id/agent-7: verify only agent-7's ops reversed +**Demo step:** Full manual walkthrough with demo data: +- `cat mount/demo/.undo/to-savepoint/initial-data/.info/summary` +- `diff -u --color mount/demo/docs/intro.md mount/demo/.undo/to-savepoint/initial-data/docs/intro.md` +- Combined diff one-liner from ADR Section 4.3.12 +- `touch mount/demo/.undo/to-savepoint/after-edits/.apply` -- undo to savepoint +- Verify files restored +- Undo the undo: `touch mount/demo/.undo/id//.apply` +- Per-user undo: `touch mount/demo/.undo/to-savepoint/initial-data/.by/user_id/agent-explorer/.apply` + +### Task 12.12: Short-Lived Caching for Undo, Log, and Savepoint Queries +Add short-lived caching to reduce redundant DB queries across undo preview, log symlink resolution, and savepoint lookups. NFS/FUSE make multiple RPCs (GETATTR, LOOKUP, READ) per user operation, and each RPC independently runs the full query chain. A single `cat` on a preview file generates ~20 queries; target is ~5. + +**Depends on:** 12.11 (.undo/ interface) +**Files:** `internal/tigerfs/fs/undo.go`, `internal/tigerfs/fs/operations.go` + +**Caches to implement (2-second TTL, matching statCache/pathCache patterns):** + +1. **queryUndoAffected cache** (highest impact): Cache the DISTINCT ON query result by (undoMode, target, filters). The same affected-files list is queried 3-6 times per preview file access: stat computes size by rendering content, readFile renders content again, and NFS repeats GETATTR multiple times. This is a read-only query on the append-only log table. Stale data means the preview might miss an operation written by another mount during the 2s window, but apply re-queries fresh, so execution is always correct. **Risk: Very low.** + +2. **Savepoint name-to-ID cache**: Cache the savepoint row (including savepoint_id) by (schema, table, name). The same savepoint is looked up 4+ times per preview navigation: validateUndoTarget, queryUndoAffected (to resolve savepoint_id), statUndo, and readFileUndo each independently call GetRow. Savepoint rows are rarely modified -- the savepoint_id is immutable once created. The only risk is a savepoint deleted during the 2s window, which would fail at apply time with a clear error. **Risk: None.** + +3. **Log entry cache**: Cache QueryLogEntry result by (schema, logTable, logID). The same log entry is fetched 3-4 times per `id/` operation: validateUndoTarget, queryUndoAffected (for single-op mode), readUndoPreviewFile, and during log diff symlink resolution. Log entries are append-only and immutable -- once written, file_id, type, version_id, and filename never change. **Risk: None.** + +**Rejected: File existence cache (QueryFileExists).** Evaluated and rejected due to concurrent write risk. QueryFileExists is used in undo execution (to decide DELETE vs skip for create-type entries) and log diff symlink resolution (current symlink). Caching could cause stale data under concurrent multi-mount writes: undo might skip deleting a file it thinks doesn't exist, or try to delete an already-deleted file. The cost of not caching is low -- undo apply runs the check once per file (not repeated), and symlink resolution adds only 2-3 extra queries per log entry listing. + +**Implementation:** +1. Follow established patterns: mutex-protected map, per-table namespace, 2-second TTL, explicit invalidation on undo execution and writes. +2. Unit tests for cache hit/miss behavior +3. Verify query reduction: `cat .undo/to-savepoint/X/file.md` should go from ~20 queries to ~5 + +### Task 12.13: Skills +Update agent skills for undo and recovery features. + +**Depends on:** 12.1-12.12 (all features complete) +**Files:** `skills/tigerfs/SKILL.md`, `skills/tigerfs/files.md`, `skills/tigerfs/recipes.md`, `skills/tigerfs/data.md`, `skills/tigerfs/ops.md` +**Tasks:** +1. SKILL.md: "Safe Editing with Savepoints" section; undo in "What you can build"; Quick Reference; anti-pattern +2. files.md: Operation Log, Savepoints, Undo sections; update history for UUIDv7 format +3. recipes.md: Recipe 5 (Safe Agent Exploration), Recipe 6 (Multi-Agent Selective Undo) +4. data.md: `.info/user`, UUIDv7 display format +5. ops.md: `--user-id`, `TIGERFS_USER_ID`, `--auto-savepoint-interval`, `--undo-list-limit` + +### Task 12.14: Documentation +Update spec and implementation docs for undo and recovery. + +**Depends on:** 12.1-12.12 (all features complete) +**Files:** `docs/spec.md`, `docs/implementation/` +**Tasks:** +1. spec.md: `.log/`, `.savepoint/`, `.undo/`, user identity, DDL limitations (undo doesn't cross schema changes) +2. implementation-tasks.md and checklist with Phase 12 + + +--- + +## Verification (Phase 12) + +1. `go fmt ./... && go vet ./... && go test ./...` passes after every task +2. Each task's integration tests pass before starting the next +3. Demo data grows incrementally -- each new feature can be manually explored as it lands +4. Full end-to-end lifecycle tested in 12.11 (15 categories of integration tests) + +--- + +## Notes for Claude Code + +- **Each task is designed to be self-contained** - You can complete it with the information provided +- **Verification commands are provided** - Run them to confirm success +- **Tests are integrated throughout** - Not deferred to the end +- **Dependencies are explicit** - Tasks build on previous work +- **Commit after each task** - Keep git history clean and logical +- **Ask questions when blocked** - Don't guess or skip ahead + +**Ready to begin! Start with Task 1.1: Evaluate and Select FUSE Library** +--- + +## Phase 13: Relational Directory Structure + +Design spec: `docs/adr/017-relational-directory-structure.md` + +Replaces path-encoded filenames with a parent-pointer model. Must be completed before resuming Phase 12 undo tasks (12.5+). + +### Task 13.1: Schema and DDL Changes + +**Objective:** Update synth/build.go to generate the new source table, history table, log table, and BEFORE trigger with parent_id and renamed columns. + +**Depends on:** Nothing +**Files:** `internal/tigerfs/fs/synth/build.go` +**Tasks:** +1. Source table: add `parent_id UUID REFERENCES ... DEFERRABLE INITIALLY DEFERRED`, change `filename` semantics to leaf name, `UNIQUE NULLS NOT DISTINCT (parent_id, filename, filetype) DEFERRABLE INITIALLY DEFERRED`, index on `(parent_id, filename)` +2. History table: rename `id` → `file_id`, `_history_id` → `version_id`, `_operation` → `operation`, add `parent_id`, CHECK constraints, modern CREATE TABLE WITH syntax +3. Log table: rename `history_id` → `version_id`, update CHECK constraint to `create/edit/rename/delete/undo` +4. BEFORE trigger: copy `parent_id`, use new column names (`file_id`, `version_id`, `operation`) +5. Create `resolve_path` PL/pgSQL function +6. Unit tests for all DDL generation + +### Task 13.2: Path Resolution + +**Objective:** Implement resolve_path Go wrapper and path cache. + +**Depends on:** 13.1 +**Files:** `internal/tigerfs/db/query.go`, `internal/tigerfs/fs/operations.go` +**Tasks:** +1. Add `ResolvePath(ctx, schema, table, startParent, segments)` to DB layer -- calls resolve_path function, returns intermediate IDs +2. Add Go-level `pathCache` with 2-second TTL, keyed on `(parent_id, filename)` → `id` +3. Integration: walk path segments checking cache, call ResolvePath for remaining segments, populate cache from results +4. Unit tests for cache behavior (hits, misses, TTL, partial hits) +5. Integration test: resolve_path against real DB + +### Task 13.3: ReadDir by parent_id + +**Objective:** Replace GetAllRows + in-memory filtering with direct parent_id queries. + +**Depends on:** 13.1, 13.2 +**Files:** `internal/tigerfs/fs/synth_ops.go`, `internal/tigerfs/db/query.go` +**Tasks:** +1. Add `GetRowsByParent(ctx, schema, table, parentID)` to DB layer +2. Remove `filterHierarchicalChildren` +3. Update `readDirSynthView` (root level: `WHERE parent_id IS NULL`) +4. Update `readDirSynthHierarchical` (subdirectory: `WHERE parent_id = X`) +5. Unit and integration tests + +### Task 13.4: Write Path Updates + +**Objective:** Update file creation and editing to use parent_id. + +**Depends on:** 13.2, 13.3 +**Files:** `internal/tigerfs/fs/synth_ops.go`, `internal/tigerfs/fs/write.go` +**Tasks:** +1. `writeSynthFile`: resolve parent_id from path, insert/update with parent_id +2. `ensureSynthParentDirs`: create parent chain with parent_id linking +3. `mkdirSynth`: insert with parent_id +4. Unit and integration tests + +### Task 13.5: Rename and Move + +**Objective:** Replace RenameByPrefix with single-row updates. + +**Depends on:** 13.4 +**Files:** `internal/tigerfs/fs/synth_ops.go`, `internal/tigerfs/db/query.go`, `internal/tigerfs/db/interfaces.go` +**Tasks:** +1. Directory rename: `UPDATE SET filename='new' WHERE id=dir_id` (1 row) +2. File rename: `UPDATE SET filename='new' WHERE id=file_id` +3. File/directory move: `UPDATE SET parent_id=new_parent WHERE id=X` +4. Remove `RenameByPrefix`, `HasChildrenWithPrefix` from DB layer and interfaces +5. Log entries use filesystem-centric types: `rename` for rename/move +6. Unit and integration tests including: rename dir with children, move dir between parents, concurrent rename + +### Task 13.6: Delete Path Updates + +**Objective:** Update delete operations to use parent_id for child checks. + +**Depends on:** 13.3 +**Files:** `internal/tigerfs/fs/synth_ops.go` +**Tasks:** +1. Check children: `SELECT EXISTS(... WHERE parent_id = dir_id)` instead of LIKE prefix +2. Delete file: standard DELETE, no FK issues (leaf node) +3. Delete empty directory: DELETE after child check +4. Delete non-empty directory: return ENOTEMPTY +5. Unit and integration tests + +### Task 13.7: History and .history/ Updates + +**Objective:** Update history queries for new column names and parent_id navigation. + +**Depends on:** 13.1 +**Files:** `internal/tigerfs/fs/history.go`, `internal/tigerfs/db/query.go` +**Tasks:** +1. Update all history queries: `file_id` (was `id`), `version_id` (was `_history_id`), `operation` (was `_operation`) +2. `.history/` navigation uses parent_id traversal +3. `.history/.by//` unchanged (queries by file_id) +4. Unit and integration tests + +### Task 13.8: Log Entry Updates + +**Objective:** Update log entry creation for new column names and filesystem-centric types. + +**Depends on:** 13.5 +**Files:** `internal/tigerfs/fs/synth_ops.go`, `internal/tigerfs/db/query.go` +**Tasks:** +1. Rename types: `insert`→`create`, `update`→`edit`/`rename`, `delete`→`delete`, `undo`→`undo` +2. `logSynthOp`: compute denormalized full path by walking parent chain at log-write time +3. Update `InsertLogEntry` for `version_id` column name +4. Update `QueryLatestHistoryID` for `version_id` column name +5. Unit and integration tests + +### Task 13.9: Migration + +**Objective:** Add `relational-directories` migration to `tigerfs migrate` framework. + +**Depends on:** 13.1-13.8 +**Files:** `internal/tigerfs/cmd/migrate.go`, `test/integration/migrate_test.go` +**Tasks:** +1. Add `relational-directories` migration to the `migrations` slice in `cmd/migrate.go` +2. Detect: find synth apps in tigerfs schema without parent_id column +3. Plan: add parent_id, populate from path parsing, strip to leaf names, replace constraints, recreate view, rename history/log columns, update log type values +4. Integration test: create old-schema tables, run migration, verify parent_id chain + leaf filenames + column renames + TigerFS operations work on migrated data + idempotency + +### Task 13.10: Update Existing Tests + +**Objective:** Rewrite all synth hierarchy tests for parent_id model. + +**Depends on:** 13.1-13.8 +**Tasks:** +1. Rewrite synth_ops_test.go hierarchy tests +2. Rewrite integration tests for hierarchy operations +3. Add complex undo scenarios from ADR-017 verification section (45 test cases) +4. Run full test suite: `go fmt ./... && go vet ./... && go test ./...` + +### Task 13.11: Update ADR-016 and Phase 12 Tasks + +**Objective:** Ensure consistency across ADRs and implementation tasks. + +**Depends on:** 13.1-13.10 +**Tasks:** +1. Rewrite ADR-016 sections: log schema (version_id, filesystem-centric types), history references, UPSERT SQL (includes parent_id) +2. Update Phase 12 tasks 12.5-12.12 in implementation-tasks.md for new schema +3. Verify consistency between ADR-016, ADR-017, and implementation tasks + +### Task 13.12: Documentation + +**Objective:** Finalize documentation. + +**Depends on:** 13.1-13.11 +**Tasks:** +1. Save ADR-017 to `docs/adr/017-relational-directory-structure.md` +2. Update `docs/spec.md` with new schema and directory model +3. Update skills if needed +4. Update implementation-tasks-checklist.md + +--- + +## Verification (Phase 13) + +1. `go fmt ./... && go vet ./... && go test ./...` passes after every task +2. All 45 verification scenarios from ADR-017 covered by tests +3. Demo: directory rename produces one log entry, undo restores correctly +4. `tigerfs migrate` relational-directories migration tested with integration test + +--- + +## Phase 14: Stress Test (`tigerfs-stress`) + +A comprehensive, repeatable stress test for file-first workspaces. Exercises all filesystem operations (create, edit, rename, move, delete files and directories) and all undo operations (single undo by log_id, multi-op undo to log_id, undo to savepoint). Deterministic via PRNG seeding, self-verifying via content hashing, self-contained (spins up own Docker PostgreSQL + TigerFS). All operations go through the real mounted filesystem -- the stress tester is a pure filesystem client with no tigerfs internal imports. + +**Location:** `test/stress/` +**Binary:** `go build -o bin/tigerfs-stress ./test/stress` (not packaged with releases) + +### Task 14.1: Scaffolding + Infrastructure + +**Objective:** Standalone binary that spins up Docker PostgreSQL, builds and mounts TigerFS, creates a workspace, and tears down cleanly on exit or via `stop` command. + +**Files:** `test/stress/main.go`, `test/stress/infra.go`, `test/stress/docker-compose.yml`, `test/stress/README.md` +**Tasks:** +1. CLI parsing with `flag` package: `start` and `stop` subcommands, all flags (--seed, --iterations, --debug, --keep, --workspace, --large-files, --many-files, --validate-every) +2. Infrastructure lifecycle: Docker compose up (timescale/timescaledb-ha:pg18, port 5433), pg_isready wait loop, tigerfs build, mount to `/tmp/tigerfs-stress-`, workspace creation via `.build/` +3. Teardown: kill tigerfs (SIGTERM then SIGKILL), unmount (diskutil/umount), remove mountpoint, docker compose down -v, remove PID/info files +4. Signal handling: trap SIGINT/SIGTERM for clean teardown when Ctrl-C'd +5. `stop` command: read `/tmp/tigerfs-stress.info`, execute teardown from another terminal +6. Initial README with build instructions and flag reference + +### Task 14.2: State Tracking + Validation + +**Objective:** In-memory expected state tracking with push/pop stack for undo rollback, and a standalone validation function that compares expected state against the actual mounted filesystem. + +**Depends on:** 14.1 +**Files:** `test/stress/state.go`, `test/stress/state_test.go`, `test/stress/validate_test.go` +**Tasks:** +1. `WorkspaceState` struct: `Files map[string]string` (path to md5 hash), `Dirs map[string]bool` +2. Deep copy function for WorkspaceState +3. `StateStack`: push before every operation, pop on undo_single, restore to savepoint index on undo_to_savepoint, restore to iteration on undo_to_id. `savepoints map[string]int` for savepoint-to-stack-index mapping +4. `ValidateWorkspace(wsPath, expected)`: walk filesystem skipping dotfiles/virtual dirs, compute md5 per file, compare to expected, detect missing/unexpected/mismatched files +5. `SnapshotHash(wsPath)`: deterministic hash of sorted "relpath:filehash" lines +6. Unit tests: deep copy correctness, push/pop, savepoint restore, stack trim, ValidateWorkspace against real temp dirs (passing, unexpected file, missing file, hash mismatch, dotfile skipping) + +### Task 14.3: Operations + Content Generation + +**Objective:** All filesystem operations with deterministic PRNG-driven content generation, log-normal file size distribution, and directory density control. + +**Depends on:** 14.2 +**Files:** `test/stress/operations.go`, `test/stress/operations_test.go` +**Tasks:** +1. `SizeConfig` struct with default (max 100KB, typical ~5KB) and large (max 10MB, typical ~22KB) presets +2. `generateFileSize(rng, cfg)`: log-normal distribution clamped to [64, MaxBytes] +3. `generateContent(rng, title, targetSize)`: deterministic markdown filling target size +4. `randomName(rng)`, `randomWords(rng, n)`: deterministic name/content generation +5. File/dir pool management: `[]string` slices, capacity tracking per directory +6. All filesystem operations: create_file, edit_file, rename_file, move_file, delete_file, create_dir, rename_dir, create_savepoint -- each takes `rng`, pools, state, wsPath; performs the real filesystem operation; updates expected state +7. Directory density: default max 10 files/3 subdirs per dir, --many-files max 1000/20 +8. Unit tests: size bounds, content determinism with fixed seed, valid filenames + +### Task 14.4: Runner + Undo + End-to-End + +**Objective:** Main test loop with weighted operation selection, undo operations through the real filesystem, and full end-to-end wiring. + +**Depends on:** 14.3 +**Files:** `test/stress/runner.go`, `test/stress/undo.go`, `test/stress/runner_test.go` +**Tasks:** +1. Weighted operation selection: weighted random via `rng` with precondition checks. Re-roll on invalid preconditions, force create_file when nothing else possible +2. Operation table with weights: create_file(25), edit_file(25), rename_file(10), move_file(10), delete_file(10), create_dir(5), rename_dir(5), create_savepoint(5), undo_single(3), undo_to_id(2), undo_to_savepoint(2) +3. `undo_single`: read `.log/.last/1/.export/json`, parse LogEntry JSON, apply `.undo/id//.apply`, pop state stack +4. `undo_to_id`: read `.log/.last/N/.export/json`, pick entry, apply `.undo/to-id//.apply`, restore state to stack index +5. `undo_to_savepoint`: apply `.undo/to-savepoint//.apply`, restore state from savepoint snapshot +6. `--validate-every N` logic: validate every N operations, always validate after undo +7. Step logging to stdout, error reporting to stderr with seed for replay +8. Wire runner into main.go start command: after infrastructure ready, run iterations, then teardown +9. Unit tests: fixed seed produces identical operation sequence, empty pool re-roll, precondition checks + +### Task 14.5: README Finalization + +**Objective:** Complete documentation with architecture description and usage guide. + +**Depends on:** 14.4 +**Files:** `test/stress/README.md` +**Tasks:** +1. Expanded "What it is" section: stress test architecture, infrastructure lifecycle, PRNG-seeded operation loop, state tracking model (hash-based verification with push/pop stack), undo rollback testing approach +2. Prerequisites (Docker, Go, macOS or Linux) +3. Full flag reference table +4. Examples: default, reproducible, large-scale, debug, keep-for-inspection +5. What it tests: full operation list +6. Stopping a run: `stop` command and Ctrl-C +7. Interpreting output: stdout progress, stderr errors, tigerfs.log +8. Replaying failures: copy seed, re-run with --seed +9. Running unit tests: `go test ./test/stress/...` + +--- + +## Verification (Phase 14) + +1. `go test ./test/stress/...` passes (unit tests for state, operations, validation, runner) +2. `bin/tigerfs-stress start --seed 42 --iterations 20` completes with exit 0 +3. Same seed produces identical output across runs +4. `bin/tigerfs-stress stop` cleanly tears down from another terminal +5. Ctrl-C during a run triggers clean teardown +6. `--large-files --many-files --iterations 50` completes without OOM or timeouts diff --git a/docs/markdown-app.md b/docs/markdown-app.md deleted file mode 100644 index dcc2a8d..0000000 --- a/docs/markdown-app.md +++ /dev/null @@ -1,391 +0,0 @@ -# Markdown App - -Store markdown files in PostgreSQL — edit them as plain files, share them across tools, and get transactional writes for free. - -## What It Does - -Write and organize `.md` files the way you normally would — with any text editor, shell script, or AI agent. The Markdown App stores each file as a database row with YAML frontmatter mapped to columns, so your content is: - -- **Shareable** — multiple users, editors, and agents access the same files simultaneously -- **Transactionally safe** — every write is an atomic database operation; no partial saves or corrupted files -- **Searchable** — use `grep` across all files, or query metadata via SQL on the underlying table -- **Metadata-rich** — frontmatter fields (author, tags, etc.) are real columns you can index and query - -Under the hood, each markdown file is stored as a database row — frontmatter fields map to columns and the body maps to a text column: - -**`hello-world.md`:** -```markdown ---- -title: Hello -author: alice -draft: false ---- - -# Hello - -Welcome... -``` - -**Stored as:** -``` -id: 1, filename: "hello-world.md", title: "Hello", author: "alice", headers: {"draft": false}, body: "# Hello\n\nWelcome..." -``` - -## Quick Start - -### Option 1: Create New App - -Start fresh with a pre-configured table: - -```bash -# Create a new markdown app called "notes" -echo "markdown" > /mnt/db/.build/notes - -# Start writing -echo "--- -title: Shopping List ---- - -- Milk -- Eggs -- Bread" > /mnt/db/notes/shopping.md -``` - -### Option 2: Add to Existing Table - -If you already have a table with content: - -```bash -# Create a markdown view on your posts table -echo "markdown" > /mnt/db/posts/.format/markdown - -# Your posts are now available as .md files -ls /mnt/db/posts_md/ -# hello-world.md my-first-post.md announcement.md - -cat /mnt/db/posts_md/hello-world.md -``` - -## Naming Conventions - -The two creation methods use different naming conventions: - -| Method | Synthesized View | Native Table | Example | -|--------|------------------|--------------|---------| -| `.build/notes` | `notes/` | `tigerfs.notes` | `/notes/hello.md` and `/.tables/notes/1/body` | -| `posts/.format/markdown` | `posts_md/` | `posts/` | `/posts_md/hello.md` and `/posts/1/body` | - -**`.build/` (new app):** View gets the clean name in the user's schema, backing table lives in the `tigerfs` schema. This is the primary method. - -**`.format/` (existing table):** View gets `_md` suffix to avoid collision with the existing table name. - -## Usage - -These examples use a `.build/` app called `notes`. The same operations work with `.format/` views (just use `posts_md/` instead of `notes/`). - -### Reading Files - -```bash -# List all files -ls /mnt/db/notes/ - -# Read a file -cat /mnt/db/notes/hello-world.md - -# Search across all files -grep -r "TODO" /mnt/db/notes/ -``` - -### Creating Files - -```bash -# Create with frontmatter -cat > /mnt/db/notes/new-post.md << 'EOF' ---- -author: bob -tags: [tutorial, getting-started] ---- - -# Getting Started - -This is my new post... -EOF -``` - -### Editing Files - -```bash -# Edit with any editor -vim /mnt/db/notes/hello-world.md - -# Or append content -echo "\n## Update\n\nMore content here." >> /mnt/db/notes/hello-world.md -``` - -### Renaming Files - -```bash -# Rename updates the filename column in the database -mv /mnt/db/notes/old-name.md /mnt/db/notes/new-name.md -``` - -### Deleting Files - -```bash -rm /mnt/db/notes/unwanted-post.md -``` - -### Organizing with Directories - -Synthesized apps support subdirectories. Create directories with `mkdir` and organize files into them: - -```bash -# Create a directory -mkdir /mnt/db/notes/tutorials - -# Create a file in the directory -cat > /mnt/db/notes/tutorials/getting-started.md << 'EOF' ---- -title: Getting Started -author: alice ---- - -# Getting Started - -Follow these steps... -EOF - -# List files in the directory -ls /mnt/db/notes/tutorials/ -# getting-started.md -``` - -**Auto-creation:** Writing a file with a path automatically creates parent directories. Writing to `notes/a/b/c.md` auto-creates the `a/` and `a/b/` directories. - -**Directory rename:** Renaming a directory atomically renames all files within it: - -```bash -mv /mnt/db/notes/tutorials /mnt/db/notes/guides -# All files under tutorials/ are now under guides/ -``` - -**`.format/` views:** Directory support also works with `.format/` views, as long as the underlying table has a `filetype` column. - -## Column Mapping - -### Automatic Detection - -TigerFS automatically detects column roles by naming convention (first match wins): - -| Role | Detected From (priority order) | Required | -|------|-------------------------------|----------| -| Filename | `filename`, `name`, `title`, `slug` | Yes | -| Body | `body`, `content`, `description`, `text` | Yes | -| Timestamps | `modified_at`, `updated_at` (modification time); `created_at` (creation time) | No | -| Extra Headers | `headers` (JSONB, merged into frontmatter) | No | -| Frontmatter | All remaining columns (excluding primary key) | — | - -Timestamp columns are used for file modification/creation times (visible in `ls -l`), but are **not** rendered as frontmatter fields. - -### Explicit Mapping (Planned) - -Currently, column roles are always auto-detected by naming convention. A future release will allow explicit mapping for tables whose column names don't match the conventions: - -```bash -# Planned — not yet implemented -echo '{filename:post_slug,body:post_content}' > /mnt/db/posts/.format/markdown -``` - -### Checking Configuration (Planned) - -A future release will allow reading the current column mapping by reading the `.format/markdown` control file: - -```bash -# Planned — not yet implemented -cat /mnt/db/posts/.format/markdown -``` - -## Custom Frontmatter (Extra Headers) - -Tables created with `.build/` include a `headers JSONB` column for storing arbitrary frontmatter keys beyond the fixed schema columns. - -**How it works:** - -- On **read**, entries from the `headers` column are merged into YAML frontmatter after the known columns, sorted alphabetically by key. -- On **write**, any frontmatter keys that don't match a known column are collected into the `headers` JSONB value. -- **Overwrite semantics** — the entire `headers` value is replaced on each write. If you remove a key from the frontmatter, it's removed from the database. - -**Example:** - -```bash -cat > /mnt/db/blog/welcome.md << 'EOF' ---- -title: Welcome to My Blog -author: alice -tags: [intro, welcome] -draft: false ---- - -# Welcome - -Thanks for visiting... -EOF -``` - -Here `title` and `author` are stored in their own columns. `tags` and `draft` — which don't have dedicated columns — are stored together in the `headers` JSONB column. - -Reading the file back: - -```markdown ---- -title: Welcome to My Blog -author: alice -draft: false -tags: - - intro - - welcome ---- - -# Welcome - -Thanks for visiting... -``` - -Known columns (`title`, `author`) appear first in schema order, then extra headers (`draft`, `tags`) appear alphabetically. - -## Use Cases - -### Blog or Documentation - -```bash -# Set up blog -echo "markdown" > /mnt/db/.build/blog - -# Write posts -cat > /mnt/db/blog/welcome.md << 'EOF' ---- -title: Welcome to My Blog -author: alice -date: 2024-01-15 -tags: [intro, welcome] -draft: false ---- - -# Welcome - -Thanks for visiting... -EOF - -# Find all drafts (draft and tags are stored in the headers JSONB column) -grep -l "draft: true" /mnt/db/blog/*.md -``` - -### Knowledge Base - -```bash -# Create knowledge base -echo "markdown" > /mnt/db/.build/kb - -# Organize into directories -mkdir /mnt/db/kb/getting-started -mkdir /mnt/db/kb/reference - -cat > /mnt/db/kb/getting-started/setup-guide.md << 'EOF' ---- -title: Setup Guide -last_updated: 2024-01-20 ---- - -# Setup Guide - -Follow these steps... -EOF - -cat > /mnt/db/kb/reference/api.md << 'EOF' ---- -title: API Reference ---- - -# API Reference - -Endpoints and usage... -EOF - -# List a section -ls /mnt/db/kb/getting-started/ -# setup-guide.md - -# Search across all sections -grep -r "TODO" /mnt/db/kb/ -``` - -### Meeting Notes - -```bash -# Create meeting notes app -echo "markdown" > /mnt/db/.build/meetings - -# Record a meeting -cat > /mnt/db/meetings/2024-01-15-standup.md << 'EOF' ---- -date: 2024-01-15 -attendees: [alice, bob, charlie] -type: standup ---- - -# Daily Standup - Jan 15 - -## Updates -- Alice: Finished API work -- Bob: Working on frontend -- Charlie: Reviewing PRs - -## Action Items -- [ ] Alice to deploy API -- [ ] Bob to demo on Friday -EOF -``` - -### Agent Workflows - -AI agents can read and write content naturally: - -```bash -# Agent reads all posts -for f in /mnt/db/blog/*.md; do - echo "=== $f ===" - cat "$f" -done - -# Agent creates content -cat > /mnt/db/blog/ai-generated.md << 'EOF' ---- -title: AI-Generated Summary -author: assistant -generated: true ---- - -Based on my analysis... -EOF -``` - -## Native Table Access - -The underlying table is still accessible for SQL operations: - -```bash -# For .format/ (existing table) -ls /mnt/db/posts/ # Native row-as-directory -ls /mnt/db/posts_md/ # Synthesized markdown - -# For .build/ (new app) -ls /mnt/db/.tables/notes/ # Native backing table -ls /mnt/db/notes/ # Synthesized markdown -``` - -## Tips - -1. **Frontmatter is automatic** — All columns except filename, body, timestamps, and primary key become frontmatter -2. **Extra headers** — Add a `headers JSONB` column to store arbitrary frontmatter keys beyond the fixed schema -3. **Timestamps are file times** — `modified_at` and `created_at` set file mtime/ctime (visible in `ls -l`), not rendered as frontmatter -4. **Arrays supported** — `tags: [a, b, c]` works with PostgreSQL array columns diff --git a/docs/spec.md b/docs/spec.md index 959db73..ae94141 100644 --- a/docs/spec.md +++ b/docs/spec.md @@ -207,18 +207,9 @@ │ │ ├── test.log # Validation result (read-only) │ │ ├── .commit # Touch to execute │ │ └── .abort # Touch to cancel -│ ├── .sample/ # Random samples -│ │ └── 100/ # 100 random rows -│ │ ├── 47392/ # Row directory -│ │ └── 103847/ # Row directory -│ ├── .first/ # First N rows (ascending) -│ │ └── 50/ # First 50 rows by PK -│ │ ├── 1/ # Row directory -│ │ └── 2/ # Row directory -│ ├── .last/ # Last N rows (descending) -│ │ └── 50/ # Last 50 rows by PK -│ │ ├── 99999/ # Row directory -│ │ └── 99998/ # Row directory +│ ├── .sample/N/ # Random N rows (navigate directly, e.g. .sample/100/) +│ ├── .first/N/ # First N rows by PK (navigate directly, e.g. .first/50/) +│ ├── .last/N/ # Last N rows by PK (navigate directly, e.g. .last/50/) │ └── 123/ # Row directory - shown in ls │ ├── email.txt # Individual column file │ ├── name.txt # Individual column file @@ -322,6 +313,14 @@ Synthesized apps, DDL operations, and pipeline queries add additional dotfile di - `.stats` - Performance monitoring - `.views/` - Database views and view creation +**User Dotfiles:** + +Files and directories starting with `.` that are not in the reserved list above are treated as regular files/directories. For example, `.gitignore`, `.env`, `.git/`, `.vscode/` can be created and used normally in file-first workspaces. They are stored in the database like any other filename. + +**Reserved Name Enforcement:** + +The dotfile names listed above are reserved for TigerFS virtual directories. Attempts to create files, create directories, or rename to these names return `EACCES` (permission denied). If hard links, symlinks, or mknod are added in the future, they must also enforce this restriction. + **Rationale:** - Standard Unix convention (dotfiles hidden from `ls`) - `ls -a` reveals metadata and special paths @@ -341,7 +340,7 @@ Every row accessible in **two ways**: **Directory Listing Behavior:** ```bash $ ls /mount/users/ -.all/ .by/ .columns/ .export/ .filter/ .first/ .import/ .info/ .last/ .order/ .sample/ +.by/ .columns/ .export/ .filter/ .first/ .import/ .info/ .last/ .order/ .sample/ 1/ 2/ 3/ ... $ cd /mount/users/1 # Enter row directory @@ -577,12 +576,14 @@ SELECT id FROM users ORDER BY id; ``` **Output:** -- Capability directories: `.all/`, `.by/`, `.columns/`, `.export/`, `.filter/`, `.first/`, `.import/`, `.info/`, `.last/`, `.order/`, `.sample/` +- Capability directories: `.by/`, `.columns/`, `.export/`, `.filter/`, `.first/`, `.import/`, `.info/`, `.last/`, `.order/`, `.sample/` - Row directories: `1/`, `2/`, `3/`, ... (primary key values) -**Note:** Row files (`1.json`, `1.csv`, etc.) are accessible via direct path but not shown in listings. +**Note:** Row files (`1.json`, `1.csv`, etc.) are accessible via direct path but not shown in listings. Similarly, `.all/` is accessible but not listed (it is equivalent to the table listing itself). Pagination directories (`.first/`, `.last/`, `.sample/`) appear in listings but show empty contents -- navigate directly with a number, e.g., `.first/50/`. These are hidden from `ls` to prevent recursive scanners (`rm -rf`, `find`, AI agents) from triggering expensive or infinite directory traversals. + +**Virtual directories in file-first workspaces:** `.history/`, `.log/`, `.savepoint/`, `.undo/` only appear at the workspace root level, not inside subdirectories. Pipeline capabilities (`.by/`, `.filter/`, `.order/`, `.export/`) inside `.log/` and `.savepoint/` are accessible by explicit path but hidden from `ls` listings. A configurable `max_pipeline_depth` (default: 10) provides defense-in-depth by suppressing capability listings after N chained pipeline operations. -**Constraint:** Tables > `dir_listing_limit` (default: 10,000) return EIO with helpful log message directing users to `.first/` or `.sample/`. +**Constraint:** Tables > `dir_listing_limit` (default: 1,000) return EIO with helpful log message directing users to `.first/` or `.sample/`. #### Indexed Lookups @@ -921,7 +922,7 @@ Tables with millions of rows make `ls /mount/users/` unusable: **Configuration:** ```yaml filesystem: - dir_listing_limit: 10000 # Default threshold + dir_listing_limit: 1000 # Default threshold ``` **Behavior:** @@ -1209,7 +1210,7 @@ ls /mnt/db/events/.first/200/.last/100/ cat /mnt/db/orders/.by/status/pending/.filter/priority/high/.order/amount/.last/20/.export/json ``` -For complete documentation, see [docs/native-tables.md](../docs/native-tables.md). +For complete documentation, see [docs/data-first.md](../docs/data-first.md). --- @@ -1221,7 +1222,7 @@ Synthesized apps present database rows as domain-specific files — markdown doc This is the primary user interface for content-oriented workflows: blog posts, knowledge bases, meeting notes, and agent-generated documents are all managed as plain files while being stored transactionally in PostgreSQL. -For the complete user guide with examples, see [docs/markdown-app.md](../docs/markdown-app.md). +For the complete user guide with examples, see [docs/file-first.md](../docs/file-first.md). ### Creation: `.build/` and `.format/` @@ -1298,20 +1299,27 @@ The `.build/` command creates a table with this schema (for markdown apps): ```sql CREATE TABLE tigerfs.notes ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + id UUID PRIMARY KEY DEFAULT uuidv7(), + parent_id UUID REFERENCES tigerfs.notes(id) DEFERRABLE INITIALLY IMMEDIATE, filename TEXT NOT NULL, filetype TEXT NOT NULL DEFAULT 'file' CHECK (filetype IN ('file', 'directory')), title TEXT, author TEXT, headers JSONB DEFAULT '{}'::jsonb, body TEXT, + encoding TEXT NOT NULL DEFAULT 'utf8' CHECK (encoding IN ('utf8', 'base64')), created_at TIMESTAMPTZ NOT NULL DEFAULT now(), modified_at TIMESTAMPTZ NOT NULL DEFAULT now(), - UNIQUE(filename, filetype) + UNIQUE NULLS NOT DISTINCT (parent_id, filename, filetype) DEFERRABLE INITIALLY IMMEDIATE ); ``` -A `BEFORE UPDATE` trigger automatically sets `modified_at = now()` on every update. +The `parent_id` column implements a parent-pointer directory model (ADR-017): `filename` stores only the leaf name (e.g., `todo.md`), and `parent_id` references the parent directory row. NULL for root-level entries. Directory renames and moves are single-row updates. The `DEFERRABLE` constraints support undo transactions. + +Two triggers maintain timestamps: + +1. **`BEFORE UPDATE` trigger** (`trg__modified_at`): Sets `modified_at = now()` on every row update. +2. **`AFTER INSERT OR DELETE OR UPDATE OF parent_id, filename` trigger** (`trg__parent_mtime`): Bumps the parent directory's `modified_at` when children are added, removed, or moved. Uses column-level filtering so content-only edits (body, title, headers) never fire it. This gives directories POSIX-correct mtime semantics. ### View Architecture @@ -1359,34 +1367,42 @@ Each history-enabled app gets a companion table named `tigerfs._history`: ```sql CREATE TABLE tigerfs.notes_history ( -- Mirrors all source table columns -- - id UUID, + file_id UUID, + parent_id UUID, filename TEXT NOT NULL, - filetype TEXT, + filetype TEXT CHECK (filetype IN ('file', 'directory')), title TEXT, author TEXT, headers JSONB, body TEXT, + encoding TEXT CHECK (encoding IN ('utf8', 'base64')), created_at TIMESTAMPTZ, modified_at TIMESTAMPTZ, -- History metadata -- - _history_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY, - _operation TEXT NOT NULL -- 'UPDATE' or 'DELETE' + version_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY, + operation TEXT NOT NULL CHECK (operation IN ('edit', 'rename', 'delete')) +) WITH ( + tsdb.hypertable, + tsdb.partition_column = 'version_id', + tsdb.chunk_interval = '7 days', + tsdb.segmentby = 'file_id', + tsdb.orderby = 'version_id DESC' ); ``` -Indexes are created on `(filename, _history_id DESC)` and `(id, _history_id DESC)` for efficient lookups by filename or row UUID. +Indexes are created on `(filename, version_id DESC)` and `(file_id, version_id DESC)` for efficient lookups by filename or row UUID. ### Trigger Mechanism -A `BEFORE UPDATE OR DELETE` trigger on the source table copies the `OLD` row into the history table with a UUIDv7 `_history_id` (encoding the current timestamp) and the operation type (`UPDATE` or `DELETE`). +A `BEFORE UPDATE OR DELETE` trigger on the source table copies the `OLD` row (including `parent_id`) into the history table with a UUIDv7 `version_id` (encoding the current timestamp) and the operation type (`UPDATE` or `DELETE`). ### TimescaleDB Integration -The history table is converted to a TimescaleDB hypertable for efficient time-partitioned storage: +The history table uses the modern `CREATE TABLE WITH` syntax to configure the hypertable and columnstore inline: -- **Chunk interval:** 1 month (partitioned by `_history_id`) -- **Compression:** `segment_by='filename'`, `order_by='_history_id DESC'` -- **Compression policy:** After 1 day +- **Chunk interval:** 7 days (partitioned by `version_id`) +- **Compression:** `segmentby='file_id'`, `orderby='version_id DESC'` +- **Compression policy:** Automatic (matches chunk interval) History requires **TimescaleDB** — it will not work on vanilla PostgreSQL. @@ -1405,7 +1421,7 @@ The `.history/` directory appears inside each synthesized app: | `app/.history/.by//` | Read a past version by UUID | | `app/subdir/.history/` | Per-directory history (scoped to that directory) | -Version timestamps are extracted from the UUIDv7 `_history_id`, formatted as `2006-01-02T150405Z` (filesystem-safe, no colons). Versions are listed newest-first. +Version IDs are derived from the UUIDv7 `version_id`, displayed as a timestamp+base36 string (e.g., `2026-04-10T173000.123Z-abc123`). This display name format is lossless and case-insensitive safe (ADR-016 Section 11). Versions are listed newest-first. **Cross-rename tracking:** Every row has a stable UUID that persists across renames. The `.by/` directory enables lookups by UUID even after a file is renamed. `.by/` is only available at the root `.history/` level. @@ -1419,6 +1435,180 @@ TigerFS detects history support via: --- +## Operation Log, Savepoints, and Undo + +History-enabled synth apps (those created with `markdown,history` or `plaintext,history`) automatically gain three additional capabilities: an operation log, savepoints, and undo. These are described in detail in [ADR-016](adr/016-undo-and-recovery.md). + +### User Identity + +Each mount has an optional user identity for tracking who made changes: + +```bash +tigerfs mount --user-id agent-7 postgres://... /mnt/db # Set at mount time +export TIGERFS_USER_ID=agent-7 # Or via environment +echo "agent-7" > /mnt/db/.info/user # Or change at runtime +cat /mnt/db/.info/user # Read current identity +``` + +Precedence: flag > environment > empty (anonymous). The identity is stored in-memory per-mount and used for log entries, savepoint auto-injection, and per-user undo filtering. + +### Operation Log (.log/) + +Every create, edit, rename, and delete is recorded in the `.log/` directory: + +| Path | Description | +|------|-------------| +| `app/.log/` | List log entries (default: last 100) | +| `app/.log/.last/N/` | Last N entries | +| `app/.log/.by/user_id/agent-7/` | Filter by user | +| `app/.log/.by/type/edit/` | Filter by operation type | +| `app/.log//` | Single entry as directory (column files + diff symlinks) | +| `app/.log/.export/json` | Export entries as JSON | + +The log table schema: + +```sql +CREATE TABLE tigerfs._log ( + log_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY, + file_id UUID NOT NULL, + type TEXT NOT NULL CHECK (type IN ('create', 'edit', 'rename', 'delete', 'undo')), + user_id TEXT, + filename TEXT NOT NULL, + version_id UUID, + description TEXT +) WITH ( + tsdb.hypertable, + tsdb.partition_column = 'log_id', + tsdb.chunk_interval = '7 days', + tsdb.segmentby = 'file_id', + tsdb.orderby = 'log_id ASC' +); +``` + +Indexed on `(file_id, log_id ASC)` for efficient per-file lookups and SkipScan-optimized DISTINCT ON queries. + +#### Diff Symlinks + +Each log entry directory contains three symlinks for diffing: + +| Symlink | Target | +|---------|--------| +| `before` | History version before this operation (or `/dev/null` for creates) | +| `after` | History version after this operation (or current file, or `/dev/null`) | +| `current` | Live file path (or `/dev/null` if deleted) | + +History symlink paths are per-directory: a file at `tutorials/getting-started.md` links to `tutorials/.history/getting-started.md/`, not `.history/tutorials/getting-started.md/`. + +```bash +diff -u --color app/.log//before app/.log//after # What this edit changed +diff -u --color app/.log//before app/.log//current # Drift since this edit +``` + +Log entry IDs use UUIDv7 display format (e.g., `2026-04-07T143000.123Z-abc123`). + +### Savepoints (.savepoint/) + +Named bookmarks for undo-to-savepoint operations: + +| Path | Description | +|------|-------------| +| `app/.savepoint/` | List savepoints | +| `app/.savepoint/name.json` | Create savepoint (write JSON with description) | +| `app/.savepoint/name/description` | Read savepoint description | +| `app/.savepoint/name/savepoint_id` | Read auto-generated UUIDv7 | + +Schema: + +```sql +CREATE TABLE tigerfs._savepoint ( + name TEXT NOT NULL PRIMARY KEY, + savepoint_id UUID NOT NULL DEFAULT uuidv7() UNIQUE, + user_id TEXT, + description TEXT +); +``` + +Creation requires a format suffix (`.json`, `.tsv`, `.csv`, `.yaml`). This avoids a macOS NFS client inode cache conflict where bare-path writes create FILE handles that conflict with DIRECTORY handles in READDIRPLUS. If `--user-id` is set, `user_id` is auto-injected into the savepoint. + +#### Auto-Savepoints + +TigerFS creates savepoints automatically when the gap since the last write exceeds a configurable threshold: + +- **Config:** `auto_savepoint_interval` (default: 30 minutes) +- **Flag:** `--auto-savepoint-interval` (e.g., `30m`, `1h`, `0` to disable) +- **Env:** `TIGERFS_AUTO_SAVEPOINT_INTERVAL` +- **Naming:** `auto--` or `auto-` (anonymous) +- **Detection:** In-memory per-table timestamp, no DB query + +### Undo (.undo/) + +The `.undo/` directory provides a preview-then-apply interface for reversing operations: + +| Path | Description | +|------|-------------| +| `app/.undo/` | List modes: `id/`, `to-id/`, `to-savepoint/` | +| `app/.undo/id//` | Single operation undo (summary + apply only) | +| `app/.undo/to-id//` | Multi-file undo to a log entry (preview tree) | +| `app/.undo/to-savepoint//` | Multi-file undo to a savepoint (preview tree) | + +#### Preview + +For `to-id/` and `to-savepoint/` modes, the target directory contains: +- `.info/summary` -- TSV with metadata headers and affected files (also `.json`, `.csv`, `.yaml`) +- `.apply` -- touch or write to execute the undo +- Affected files rendered from history (restore) or as `/dev/null` symlinks (delete) + +The `.info/summary` TSV format includes comment headers with metadata: + +``` +# savepoint: before-edits +# created: 2026-04-13T14:28:15Z +# user: agent-7 +# description: All posts created, before any edits +# affected: 2 files +# type filename user timestamp +edit hello-world.md agent-7 2026-04-13T14:28:17Z +edit tutorials/getting-started-with-sql.md agent-7 2026-04-13T14:28:17Z +``` + +For `id/` mode (single operation), only `.info/summary` and `.apply` are present -- no preview tree. Use `.log//before` and `.log//after` for diffs. + +#### Applying Undo + +```bash +touch app/.undo/to-savepoint/before-edits/.apply # Undo to savepoint +touch app/.undo/id//.apply # Undo single operation +touch app/.undo/to-savepoint/X/.by/user_id/agent-7/.apply # Per-user undo +``` + +The `.apply` trigger works via both `touch` (SETATTR/Chtimes) and `echo` (Write/Close) on NFS and FUSE. + +Pipeline capabilities filter which operations are undone: `.by/user_id/`, `.filter/type/delete/`, `.last/N/`. `.sample/` is rejected on `.apply`. + +#### Execution + +Undo executes in a single PostgreSQL transaction: +1. Query affected files using DISTINCT ON with SkipScan on the `(file_id, log_id ASC)` index +2. DELETE rows created after the target point +3. UPSERT rows from history for edits, renames, and deletes (restores before-state including `parent_id`) +4. INSERT `type='undo'` log entries for each affected file + +Undo is idempotent: undoing to the same savepoint twice produces the same data state (with additional log/history entries). Undo operations are themselves logged, so you can undo an undo. + +#### DDL Limitations + +Undo operates on data (DML) only. If a column was added or removed after the savepoint, the UPSERT may fail. TigerFS detects PostgreSQL error codes 42703 (undefined column) and 42P01 (undefined table) and returns a descriptive error instead of a raw SQL message. + +#### Configuration + +| Setting | Default | Description | +|---------|---------|-------------| +| `undo_list_limit` | 100 | Default listing limit for `.undo/` sub-directories | +| Flag: `--undo-list-limit` | | Override at mount time | +| Env: `TIGERFS_UNDO_LIST_LIMIT` | | Override via environment | + +--- + ## Configuration System ### Configuration Hierarchy (Precedence: Low to High) @@ -1460,7 +1650,8 @@ connection: # Filesystem behavior filesystem: - dir_listing_limit: 10000 # Max rows returned by ls (prevents huge listings) + dir_listing_limit: 1000 # Max rows returned by ls (prevents huge listings) + max_pipeline_depth: 10 # Max chained pipeline ops before capabilities hidden (0=unlimited) trailing_newlines: true # Add \n to column and .count file reads no_filename_extensions: false # Disable type-based extensions (.txt, .json, etc.) attr_timeout: 1 # FUSE attribute cache (seconds, FUSE backend only) @@ -1570,7 +1761,8 @@ tigerfs --foreground --log-level=debug postgres://localhost/mydb /mnt/db **Filesystem Behavior:** ```bash ---max-ls-rows N Large table threshold (default: 10000) +--max-ls-rows N Large table threshold (default: 1000) +--max-pipeline-depth N Max chained pipeline ops in ReadDir (default: 10; 0=unlimited) --unlimited-ls Disable row limit for ls operations --read-only Mount as read-only --allow-other Allow other users to access mount (FUSE) @@ -1584,6 +1776,13 @@ tigerfs --foreground --log-level=debug postgres://localhost/mydb /mnt/db --metadata-refresh SECS Table metadata refresh interval (default: 30) ``` +**Identity & Undo:** +```bash +--user-id ID User identity for undo log entries (also: TIGERFS_USER_ID env) +--auto-savepoint-interval DUR Inactivity gap before auto-savepoint (default: 30m, 0 disables) +--undo-list-limit N Default listing limit for .undo/ sub-directories (default: 100) +``` + **Logging/Debug:** ```bash --log-level LEVEL Log level: debug, info, warn, error (default: info) @@ -1948,6 +2147,30 @@ tigerfs fork /mnt/db --no-mount - Unless `--no-mount`, spawns a background `tigerfs mount` process for the fork - `--json` outputs the fork result (source ID, fork ID, name, connection string, mountpoint) +#### 12. migrate + +**Syntax:** +```bash +tigerfs migrate CONNECTION [--describe] [--dry-run] [--insecure-no-ssl] [--schema SCHEMA] +``` + +**Description:** Detect and run pending database migrations. Migrations are named actions that update database structures for compatibility with newer TigerFS versions. + +**Modes:** +- `--describe`: List pending migrations without generating SQL +- `--dry-run`: Show the SQL that would be executed without running it +- (default): Execute all pending migrations + +**Current migrations (run in order):** + +| Migration | Description | +|-----------|-------------| +| `move-backing-tables` | Move backing tables from user schema to tigerfs schema | +| `relational-directories` | Add parent-pointer directory model (parent_id column) | +| `parent-dir-mtime-trigger` | Add trigger to update parent directory mtime when children change | + +Each migration has a Detect function that checks whether it's needed (queries system catalogs) and a Plan function that generates SQL. Migrations are idempotent -- running `migrate` when nothing is pending is a no-op. + --- ## Connection and Authentication @@ -2306,7 +2529,13 @@ tigerfs mount tiger: /mnt/cloud **mtime (Modification Time):** - Check for `updated_at` or `modified_at` column (common convention) - Use column value if exists -- Fallback to current time if column doesn't exist +- Fallback to `created_at`, then to mount time if neither exists + +**Directory mtime (File-First Workspaces):** +- Directory `modified_at` is updated by a database trigger when children are added, removed, renamed, or moved +- Content-only edits to files do NOT change the parent directory's mtime (POSIX-correct behavior) +- This ensures NFS/FUSE clients detect directory content changes and re-fetch listings +- Cross-mount visibility: another mount sees directory changes within the stat cache TTL (2 seconds) **ctime (Change Time) and atime (Access Time):** - Use same as mtime or current time @@ -2316,6 +2545,7 @@ tigerfs mount tiger: /mnt/cloud - Meaningful timestamps when available - Graceful degradation for tables without timestamp columns - No extra queries just for timestamps +- Directory mtime enables correct NFS/FUSE cache invalidation after undo and cross-mount operations ### Ownership @@ -2992,9 +3222,10 @@ For multi-chunk writes, this produces **O(n²) total DB write volume**: the firs **Mitigations in place:** 1. **wsize=128KB** (macOS mount option): 4x fewer RPCs than macOS default (32KB-64KB). Larger values (e.g., 1MB) can trigger GC deadlocks in test environments where the NFS server runs in the same process as the client. 128KB is a safe balance. -2. **Schema caching**: Schema resolution is cached with `sync.Once`, eliminating ~4-6 DB queries per RPC. -3. **Stat cache**: Dirty cached files return size from memory, avoiding a DB read after each write. -4. **Per-RPC overhead**: Reduced from ~7-9 DB queries to ~1 DB write per WRITE RPC. +2. **noac** (macOS mount option): Disables NFS client attribute caching. Without this, the client caches file attributes (mtime, size) for up to 60 seconds (`acregmax`), causing stale reads after undo operations or cross-mount changes. This does not increase SQL queries -- GETATTR requests are served from TigerFS's in-memory stat cache (2s TTL). The cost is only additional loopback NFS round-trips (~0.1ms each). +3. **Schema caching**: Schema resolution is cached with `sync.Once`, eliminating ~4-6 DB queries per RPC. +4. **Stat cache**: Dirty cached files return size from memory, avoiding a DB read after each write. +5. **Per-RPC overhead**: Reduced from ~7-9 DB queries to ~1 DB write per WRITE RPC. **Behavior:** @@ -3225,6 +3456,48 @@ echo 1 > /mnt/db/.refresh - Manual refresh available for immediate consistency - Most use cases tolerate eventual consistency +### Undo Preview Cache + +Undo preview surfaces (`.undo///.info/summary`, +`.undo///`) are backed by a short-lived +in-process cache (2-second TTL) on top of `QueryUndoAffectedFiles`, +`GetSavepointRow`, `QueryLogEntry`, and history-row reads. The cache +exists because each NFS/FUSE RPC during a single user op (`ls`, +`cat`, `stat`) independently queries the same data, and these reads +are read-only. + +**Apply path is unaffected.** Writing to `.apply` re-queries inside +`ExecuteUndoTransaction` with no cache involvement, so undo +correctness does not depend on cache freshness. + +**Stale-preview window after a concurrent apply.** Under PostgreSQL +READ COMMITTED, a preview read whose `SELECT` started *before* a +concurrent apply transaction COMMITs uses a snapshot taken at +`SELECT`-start. If the apply commits and invalidates the cache while +that read is in flight, the read's result reflects pre-undo state; +the read then re-populates the cache with that stale snapshot. +Subsequent preview lookups can serve stale data until the 2-second +TTL expires. + +**Affected reads:** + +| Cache | Stale data semantically wrong? | +|-------|-------------------------------| +| `affectedFiles` (which files would change for target X) | yes -- the answer is invalidated by the just-applied undo | +| `logEntries`, `savepointRows`, `historyRows` | no -- the underlying tables are append-only or immutable | + +So in practice, only the "files affected by undoing target X" preview +can show a stale list. Listings, log/savepoint/history previews show +data that is technically a snapshot but still semantically correct. + +**Rationale:** the 2-second window is short, the apply path is +unaffected, and eliminating the race would require either holding a +process-wide lock for the duration of every undo transaction (kills +preview concurrency on every other workspace) or threading a +generation counter through every cache read site (significant API +churn). The current behavior is the same eventual-consistency tradeoff +documented above for directory listings. + --- ## Unmounting and Shutdown @@ -3882,9 +4155,9 @@ go install github.com/timescale/tigerfs/cmd/tigerfs@latest ### Phase 2: Feature Documentation (Shipped) -- [docs/markdown-app.md](../docs/markdown-app.md) — Markdown synthesized app guide (creation, usage, column mapping) +- [docs/file-first.md](../docs/file-first.md) — Markdown synthesized app guide (creation, usage, column mapping) - [docs/history.md](../docs/history.md) — Version history feature (browsing, recovery, UUID tracking) -- [docs/native-tables.md](../docs/native-tables.md) — Native table access reference (pipeline queries, DDL) +- [docs/data-first.md](../docs/data-first.md) — Native table access reference (pipeline queries, DDL) - [docs/quickstart.md](../docs/quickstart.md) — Guided scenarios with sample data ### Phase 3: Advanced Documentation diff --git a/docs/tasks-app.md b/docs/tasks-app.md deleted file mode 100644 index bbbd2c2..0000000 --- a/docs/tasks-app.md +++ /dev/null @@ -1,408 +0,0 @@ -# Tasks App - -> **Status: Not Yet Implemented** — This documents the planned Tasks synthesized app (Phase 6.2). The feature is designed but not yet built. - -Manage ordered task lists with status tracking in PostgreSQL. - -## What It Does - -The Tasks App presents database rows as task files with structured filenames. Tasks have hierarchical numbering, status indicators, and automatic reordering. - -**Filename format:** `{number}-{name}-{status}.md` - -``` -/work/ -├── 1-setup-project-x.md # done -├── 1.1-create-repo-x.md # done -├── 1.2-add-readme-~.md # doing (in progress) -├── 2-implement-feature-o.md # todo (open) -└── 2.1-write-tests-o.md # todo -``` - -**Status symbols:** - -| Symbol | Meaning | Stored As | -|--------|---------|-----------| -| `o` | Open/todo | `todo` | -| `~` | In progress | `doing` | -| `x` | Complete | `done` | - -## Why Use It - -- **Hierarchical organization** - Break tasks into subtasks (1.1, 1.2, 1.1.1) -- **Visual status** - See task state at a glance in `ls` output -- **Auto-timestamps** - Track when tasks started/completed -- **Reorderable** - Move tasks by renaming, others shift automatically -- **Agent-friendly** - AI agents manage tasks via simple file operations -- **Persistent** - Tasks survive restarts, stored in PostgreSQL - -## Quick Start - -```bash -# Create a tasks app -echo "tasks" > /mnt/db/.build/work - -# Add tasks -echo "Set up the development environment" > /mnt/db/work/1-setup-o.md -echo "Implement user authentication" > /mnt/db/work/2-auth-o.md -echo "Write API documentation" > /mnt/db/work/3-docs-o.md - -# List tasks -ls /mnt/db/work/ -# 1-setup-o.md 2-auth-o.md 3-docs-o.md -``` - -## Usage - -### Adding Tasks - -```bash -# Simple task -echo "Description here" > /mnt/db/work/1-task-name-o.md - -# Subtask -echo "Subtask description" > /mnt/db/work/1.1-subtask-o.md - -# With full frontmatter -cat > /mnt/db/work/1-setup-o.md << 'EOF' ---- -number: "1" -name: setup -status: todo -assignee: alice ---- - -Set up the development environment: -- Install dependencies -- Configure database -- Run initial migrations -EOF -``` - -### Changing Status - -**Option 1: Rename the file** (quick) - -```bash -# Start working on a task (todo → doing) -mv /mnt/db/work/1-setup-o.md /mnt/db/work/1-setup-~.md - -# Complete a task (doing → done) -mv /mnt/db/work/1-setup-~.md /mnt/db/work/1-setup-x.md -``` - -**Option 2: Edit the frontmatter** (when already editing the file) - -```bash -# Open the file and change status: todo → doing -vim /mnt/db/work/1-setup-o.md -``` - -```yaml ---- -number: "1" -name: setup -status: doing # Changed from "todo" ---- -``` - -The filename updates automatically to match: `1-setup-~.md` - -Status timestamps are set automatically: -- `todo_at` - when status becomes `todo` -- `doing_at` - when status becomes `doing` -- `done_at` - when status becomes `done` - -### Reordering Tasks - -Moving a task to an occupied number shifts others automatically: - -```bash -# Before: 1-setup, 2-auth, 3-docs -# Move docs to position 2 -mv /mnt/db/work/3-docs-o.md /mnt/db/work/2-docs-o.md - -# After: 1-setup, 2-docs, 3-auth (auth shifted to 3) -ls /mnt/db/work/ -``` - -### Closing Gaps - -Gaps are preserved until you explicitly compact: - -```bash -# After deleting task 2, you have: 1, 3, 4 -rm /mnt/db/work/2-some-task-o.md -ls /mnt/db/work/ -# 1-setup-x.md 3-auth-o.md 4-docs-o.md - -# Compact to close gaps -touch /mnt/db/work/.renumber -ls /mnt/db/work/ -# 1-setup-x.md 2-auth-o.md 3-docs-o.md - -# Compact only a subtree -echo "2" > /mnt/db/work/.renumber # Only renumber 2.* -``` - -### Reading Task Details - -```bash -cat /mnt/db/work/1-setup-x.md -``` - -Output: -```markdown ---- -number: "1" -name: setup -status: done -assignee: alice -created_at: 2024-01-15T09:00:00Z -modified_at: 2024-01-15T14:30:00Z -todo_at: 2024-01-15T09:00:00Z -doing_at: 2024-01-15T10:00:00Z -done_at: 2024-01-15T14:30:00Z ---- - -Set up the development environment: -- Install dependencies -- Configure database -- Run initial migrations -``` - -### Filtering Tasks - -Use TigerFS capabilities to filter: - -```bash -# All todo tasks -ls /mnt/db/work/.by/status/todo/ - -# All tasks assigned to alice -ls /mnt/db/work/.by/assignee/alice/ - -# Completed tasks as JSON -cat /mnt/db/work/.by/status/done/.export/json -``` - -## Hierarchical Numbering - -Tasks support unlimited nesting with integers at each level: - -``` -1 # Top-level task -1.1 # First subtask -1.2 # Second subtask -1.2.1 # Sub-subtask -2 # Another top-level -``` - -**Rules:** -- Numbers start at 1 (no zero) -- Integers only (no letters) -- Any depth allowed - -### Creating Subtasks - -```bash -# Create a parent task -echo "Build user authentication" > /mnt/db/work/1-auth-o.md - -# Add subtasks -echo "Design login flow" > /mnt/db/work/1.1-login-design-o.md -echo "Implement login API" > /mnt/db/work/1.2-login-api-o.md -echo "Build login UI" > /mnt/db/work/1.3-login-ui-o.md - -# Add sub-subtasks -echo "Create JWT tokens" > /mnt/db/work/1.2.1-jwt-o.md -echo "Add session storage" > /mnt/db/work/1.2.2-sessions-o.md - -# View the hierarchy -ls /mnt/db/work/ -# 1-auth-o.md -# 1.1-login-design-o.md -# 1.2-login-api-o.md -# 1.2.1-jwt-o.md -# 1.2.2-sessions-o.md -# 1.3-login-ui-o.md -``` - -### Moving Tasks Between Parents - -```bash -# Move task 1.3 to become 2.1 (under a different parent) -mv /mnt/db/work/1.3-login-ui-o.md /mnt/db/work/2.1-login-ui-o.md - -# Promote a subtask to top-level -mv /mnt/db/work/1.2.1-jwt-o.md /mnt/db/work/3-jwt-o.md - -# Demote a task to become a subtask -mv /mnt/db/work/3-jwt-o.md /mnt/db/work/1.2.1-jwt-o.md -``` - -### Shifting is Scoped to Siblings - -When you insert at an occupied number, only siblings at the same level shift: - -```bash -# Current: 1.1, 1.2, 1.3, 2.1, 2.2 -# Insert new task at 1.2 -echo "New task" > /mnt/db/work/1.2-new-task-o.md - -# Result: 1.1, 1.2(new), 1.3(was 1.2), 1.4(was 1.3), 2.1, 2.2 -# Note: 2.x tasks are unaffected -``` - -### Dynamic Padding - -Filenames are zero-padded per-level for correct `ls` sorting: - -```bash -# With 15 subtasks under task 1 -ls /mnt/db/work/ -# 1-auth-o.md -# 1.01-first-o.md -# 1.02-second-o.md -# ... -# 1.15-last-o.md - -# Frontmatter shows unpadded numbers -cat /mnt/db/work/1.01-first-o.md -# number: "1.1" (not "1.01") -``` - -Padding adjusts automatically based on the maximum number at each level. - -## Use Cases - -### Project Management - -```bash -# Create project tasks -echo "tasks" > /mnt/db/.build/project - -# Define milestones and subtasks -echo "Complete MVP" > /mnt/db/project/1-mvp-o.md -echo "User authentication" > /mnt/db/project/1.1-auth-o.md -echo "Login page" > /mnt/db/project/1.1.1-login-o.md -echo "Signup page" > /mnt/db/project/1.1.2-signup-o.md -echo "Dashboard" > /mnt/db/project/1.2-dashboard-o.md - -# Track progress -mv /mnt/db/project/1.1.1-login-o.md /mnt/db/project/1.1.1-login-x.md - -# See what's left -ls /mnt/db/project/.by/status/todo/ -``` - -### Sprint Planning - -```bash -echo "tasks" > /mnt/db/.build/sprint - -# Add sprint items with assignees -cat > /mnt/db/sprint/1-api-endpoints-o.md << 'EOF' ---- -assignee: alice ---- - -Implement REST API endpoints for users resource -EOF - -cat > /mnt/db/sprint/2-frontend-forms-o.md << 'EOF' ---- -assignee: bob ---- - -Build React forms for user registration -EOF - -# Check alice's tasks -ls /mnt/db/sprint/.by/assignee/alice/ -``` - -### Agent Task Management - -AI agents can find and claim the next task: - -```bash -# Find the first todo task -TASK=$(ls /mnt/db/work/*-o.md | head -1 | sed 's/-o.md$//') # "1-research" - -# Claim it -mv "$TASK-o.md" "$TASK-~.md" - -# ... do the work ... - -# Mark complete -mv "$TASK-~.md" "$TASK-x.md" -``` - -**Search by content:** - -```bash -# Find tasks mentioning "database" -grep -l "database" /mnt/db/work/*-o.md -``` - -### Personal Todo List - -```bash -echo "tasks" > /mnt/db/.build/todo - -# Quick task entry -echo "Buy groceries" > /mnt/db/todo/1-groceries-o.md -echo "Call dentist" > /mnt/db/todo/2-dentist-o.md -echo "Review PR #123" > /mnt/db/todo/3-review-o.md - -# Mark done -mv /mnt/db/todo/2-dentist-o.md /mnt/db/todo/2-dentist-x.md - -# Clean up completed -for f in /mnt/db/todo/*-x.md; do rm "$f"; done -touch /mnt/db/todo/.renumber -``` - -### Multi-Agent Coordination - -Multiple agents can coordinate via shared task list: - -```bash -# Create shared work queue -echo "tasks" > /mnt/db/.build/queue - -# Agent A adds work -echo "Process batch 1" > /mnt/db/queue/1-batch1-o.md -echo "Process batch 2" > /mnt/db/queue/2-batch2-o.md - -# Agent B claims a task (atomic via rename) -mv /mnt/db/queue/1-batch1-o.md /mnt/db/queue/1-batch1-~.md - -# Agent B completes -mv /mnt/db/queue/1-batch1-~.md /mnt/db/queue/1-batch1-x.md - -# Agent C claims next available -``` - -## Native Table Access - -Access the underlying table for SQL operations: - -```bash -# Synthesized view (task files) -ls /mnt/db/work/ - -# Native table (row directories) -ls /mnt/db/_work/ -ls /mnt/db/_work/1/status -``` - -## Tips - -1. **Status shortcuts** - Use `o`, `~`, `x` in filenames for quick status changes -2. **Subtasks inherit nothing** - Each task is independent; hierarchy is for organization -3. **Gaps are OK** - Don't feel obligated to renumber; gaps don't affect functionality -4. **Timestamps are automatic** - Just change status; timestamps update automatically -5. **Use assignee** - Great for filtering who should work on what -6. **Combine with grep** - Search task descriptions: `grep -r "urgent" /mnt/db/work/` diff --git a/internal/tigerfs/cmd/config.go b/internal/tigerfs/cmd/config.go index a042bd5..382d82b 100644 --- a/internal/tigerfs/cmd/config.go +++ b/internal/tigerfs/cmd/config.go @@ -98,6 +98,7 @@ func showConfig(w io.Writer, cfg *config.Config) error { Filesystem struct { DirListingLimit int `yaml:"dir_listing_limit"` DirWritingLimit int `yaml:"dir_writing_limit"` + MaxPipelineDepth int `yaml:"max_pipeline_depth"` TrailingNewlines bool `yaml:"trailing_newlines"` NoFilenameExtensions bool `yaml:"no_filename_extensions"` AttrTimeout string `yaml:"attr_timeout"` @@ -155,6 +156,7 @@ func showConfig(w io.Writer, cfg *config.Config) error { // Filesystem display.Filesystem.DirListingLimit = cfg.DirListingLimit display.Filesystem.DirWritingLimit = cfg.DirWritingLimit + display.Filesystem.MaxPipelineDepth = cfg.MaxPipelineDepth display.Filesystem.TrailingNewlines = cfg.TrailingNewlines display.Filesystem.NoFilenameExtensions = cfg.NoFilenameExtensions display.Filesystem.AttrTimeout = cfg.AttrTimeout.String() diff --git a/internal/tigerfs/cmd/config_test.go b/internal/tigerfs/cmd/config_test.go index 58e7436..6e19ea4 100644 --- a/internal/tigerfs/cmd/config_test.go +++ b/internal/tigerfs/cmd/config_test.go @@ -27,7 +27,7 @@ func TestShowConfig(t *testing.T) { TigerCloudProjectID: "proj456", DefaultBackend: "tiger", DefaultMountDir: "/tmp", - DirListingLimit: 10000, + DirListingLimit: 1000, DirWritingLimit: 100000, TrailingNewlines: true, NoFilenameExtensions: false, @@ -80,7 +80,7 @@ func TestShowConfig(t *testing.T) { "host: localhost", "port: 5432", "user: postgres", "database: testdb", "service_id: svc123", "project_id: proj456", "default_backend: tiger", "default_mount_dir: /tmp", - "dir_listing_limit: 10000", "dir_writing_limit: 100000", + "dir_listing_limit: 1000", "dir_writing_limit: 100000", "trailing_newlines: true", "no_filename_extensions: false", "query_timeout: 30s", "dir_filter_limit: 100000", "structural_metadata_refresh_interval: 5m0s", @@ -103,7 +103,7 @@ func TestValidateConfigValues(t *testing.T) { Port: 5432, PoolSize: 10, PoolMaxIdle: 5, - DirListingLimit: 10000, + DirListingLimit: 1000, DefaultFormat: "tsv", BinaryEncoding: "raw", LogLevel: "info", diff --git a/internal/tigerfs/cmd/migrate.go b/internal/tigerfs/cmd/migrate.go index 52279c4..0b7a6c4 100644 --- a/internal/tigerfs/cmd/migrate.go +++ b/internal/tigerfs/cmd/migrate.go @@ -34,8 +34,11 @@ type migration struct { } // migrations is the ordered list of all registered migrations. +// Order matters: earlier migrations run first. var migrations = []migration{ moveBackingTablesMigration(), + addParentPointerMigration(), + addParentDirMtimeTriggerMigration(), } // moveBackingTablesMigration returns the migration that moves synth backing tables @@ -44,7 +47,7 @@ var migrations = []migration{ func moveBackingTablesMigration() migration { return migration{ Name: "move-backing-tables", - Summary: "Move synth backing tables from _name in user schema to name in tigerfs schema", + Summary: "Move backing tables from user schema to tigerfs schema", Detect: func(ctx context.Context, pool *pgxpool.Pool, schema string) ([]string, error) { // Get all tables in user schema rows, err := pool.Query(ctx, @@ -194,6 +197,329 @@ func moveBackingTablesMigration() migration { } } +// addParentPointerMigration returns the migration that converts synth apps from +// path-encoded filenames (ADR-011) to the parent-pointer directory model (ADR-017). +// For each app, it adds parent_id, populates it from existing path hierarchy, +// strips filenames to leaf names, updates constraints/indexes, migrates history +// and log table column names, and recreates the BEFORE trigger. +func addParentPointerMigration() migration { + return migration{ + Name: "relational-directories", + Summary: "Upgrade directory structure for improved performance and undo support", + Detect: func(ctx context.Context, pool *pgxpool.Pool, schema string) ([]string, error) { + // Find synth apps in tigerfs schema that DON'T have parent_id yet + rows, err := pool.Query(ctx, + `SELECT c.relname, d.description + FROM pg_class c + JOIN pg_namespace n ON n.oid = c.relnamespace + LEFT JOIN pg_description d ON d.objoid = c.oid AND d.objsubid = 0 + WHERE n.nspname = $1 AND c.relkind = 'v' + AND d.description LIKE 'tigerfs:%'`, schema) + if err != nil { + return nil, fmt.Errorf("failed to list synth views: %w", err) + } + defer rows.Close() + + var items []string + for rows.Next() { + var viewName string + var comment *string + if err := rows.Scan(&viewName, &comment); err != nil { + return nil, fmt.Errorf("failed to scan view: %w", err) + } + + // Check if backing table in tigerfs schema has parent_id + var hasParentID bool + err := pool.QueryRow(ctx, + `SELECT EXISTS( + SELECT 1 FROM information_schema.columns + WHERE table_schema = 'tigerfs' AND table_name = $1 + AND column_name = 'parent_id' + )`, viewName).Scan(&hasParentID) + if err != nil { + return nil, fmt.Errorf("failed to check parent_id for %s: %w", viewName, err) + } + + // Also check the table actually exists in tigerfs schema + var tableExists bool + err = pool.QueryRow(ctx, + `SELECT EXISTS( + SELECT 1 FROM pg_tables + WHERE schemaname = 'tigerfs' AND tablename = $1 + )`, viewName).Scan(&tableExists) + if err != nil { + return nil, fmt.Errorf("failed to check table existence for %s: %w", viewName, err) + } + + if tableExists && !hasParentID { + items = append(items, viewName) + } + } + return items, nil + }, + Plan: func(ctx context.Context, pool *pgxpool.Pool, schema string, items []string) ([]string, error) { + var stmts []string + + // Create resolve_path function (idempotent) + stmts = append(stmts, synth.GenerateResolvePathSQL()) + + for _, appName := range items { + qt := fmt.Sprintf("%s.%s", db.QuoteIdent(synth.TigerFSSchema), db.QuoteIdent(appName)) + + // --- Source table --- + + // Switch id column from UUIDv4 to UUIDv7 (time-ordered) + stmts = append(stmts, fmt.Sprintf( + `ALTER TABLE %s ALTER COLUMN id SET DEFAULT uuidv7()`, qt)) + + // Add parent_id column + stmts = append(stmts, fmt.Sprintf( + `ALTER TABLE %s ADD COLUMN parent_id UUID`, qt)) + + // Populate parent_id from path hierarchy (PL/pgSQL DO block). + // Processes rows shallowest first; looks up parent by old full-path filename. + stmts = append(stmts, fmt.Sprintf(`DO $migrate$ +DECLARE + r RECORD; + parts TEXT[]; + parent_path TEXT; + found_parent_id UUID; +BEGIN + FOR r IN SELECT id, filename FROM %s + WHERE filename LIKE '%%/%%' + ORDER BY length(filename) - length(replace(filename, '/', '')) + LOOP + parts := string_to_array(r.filename, '/'); + parent_path := array_to_string(parts[1:array_length(parts,1)-1], '/'); + IF parent_path = '' OR parent_path IS NULL THEN + SELECT id INTO found_parent_id FROM %s + WHERE filename = parts[1] AND filetype = 'directory' LIMIT 1; + ELSE + SELECT id INTO found_parent_id FROM %s + WHERE filename = parent_path AND filetype = 'directory' LIMIT 1; + END IF; + UPDATE %s SET parent_id = found_parent_id WHERE id = r.id; + END LOOP; +END $migrate$`, qt, qt, qt, qt)) + + // Strip filenames to leaf names + stmts = append(stmts, fmt.Sprintf( + `UPDATE %s SET filename = split_part(filename, '/', array_length(string_to_array(filename, '/'), 1)) WHERE filename LIKE '%%/%%'`, qt)) + + // Add FK constraint + stmts = append(stmts, fmt.Sprintf( + `ALTER TABLE %s ADD CONSTRAINT %s FOREIGN KEY (parent_id) REFERENCES %s(id) DEFERRABLE INITIALLY IMMEDIATE`, + qt, db.QuoteIdent("fk_"+appName+"_parent"), qt)) + + // Find and drop old UNIQUE constraint, add new one + var oldConstraint *string + _ = pool.QueryRow(ctx, + `SELECT conname FROM pg_constraint + WHERE conrelid = $1::regclass AND contype = 'u' + AND array_length(conkey, 1) = 2 LIMIT 1`, + fmt.Sprintf("tigerfs.%s", appName)).Scan(&oldConstraint) + if oldConstraint != nil { + stmts = append(stmts, fmt.Sprintf( + `ALTER TABLE %s DROP CONSTRAINT %s`, qt, db.QuoteIdent(*oldConstraint))) + } + stmts = append(stmts, fmt.Sprintf( + `ALTER TABLE %s ADD CONSTRAINT %s UNIQUE NULLS NOT DISTINCT (parent_id, filename, filetype) DEFERRABLE INITIALLY IMMEDIATE`, + qt, db.QuoteIdent("uq_"+appName+"_parent_filename"))) + + // Parent index + stmts = append(stmts, fmt.Sprintf( + `CREATE INDEX IF NOT EXISTS %s ON %s (parent_id, filename)`, + db.QuoteIdent("idx_"+appName+"_parent"), qt)) + + // Recreate view to pick up the new parent_id column. + // PostgreSQL views with SELECT * snapshot columns at creation time; + // ALTER TABLE ADD COLUMN does NOT update existing views. + stmts = append(stmts, fmt.Sprintf( + `DROP VIEW IF EXISTS %s`, db.QuoteTable(schema, appName))) + stmts = append(stmts, synth.GenerateViewSQL(schema, appName, synth.TigerFSSchema, appName)) + // Preserve the view comment + var viewComment string + _ = pool.QueryRow(ctx, + `SELECT obj_description(c.oid, 'pg_class') + FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname = $1 AND c.relname = $2`, schema, appName).Scan(&viewComment) + if viewComment != "" { + stmts = append(stmts, fmt.Sprintf( + `COMMENT ON VIEW %s IS '%s'`, + db.QuoteTable(schema, appName), viewComment)) + } + + // --- History table (if exists) --- + var hasHistory bool + _ = pool.QueryRow(ctx, + `SELECT EXISTS(SELECT 1 FROM pg_tables WHERE schemaname = 'tigerfs' AND tablename = $1)`, + appName+"_history").Scan(&hasHistory) + + if hasHistory { + htQt := fmt.Sprintf("%s.%s", db.QuoteIdent(synth.TigerFSSchema), db.QuoteIdent(appName+"_history")) + + stmts = append(stmts, fmt.Sprintf(`ALTER TABLE %s ADD COLUMN IF NOT EXISTS parent_id UUID`, htQt)) + + // Rename columns (check existence first in Plan since we can query) + var hasOldID, hasOldHistID, hasOldOp bool + _ = pool.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM information_schema.columns WHERE table_schema='tigerfs' AND table_name=$1 AND column_name='id')`, appName+"_history").Scan(&hasOldID) + _ = pool.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM information_schema.columns WHERE table_schema='tigerfs' AND table_name=$1 AND column_name='_history_id')`, appName+"_history").Scan(&hasOldHistID) + _ = pool.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM information_schema.columns WHERE table_schema='tigerfs' AND table_name=$1 AND column_name='_operation')`, appName+"_history").Scan(&hasOldOp) + + if hasOldID { + stmts = append(stmts, fmt.Sprintf(`ALTER TABLE %s RENAME COLUMN id TO file_id`, htQt)) + } + if hasOldHistID { + stmts = append(stmts, fmt.Sprintf(`ALTER TABLE %s RENAME COLUMN _history_id TO version_id`, htQt)) + } + if hasOldOp { + stmts = append(stmts, fmt.Sprintf(`ALTER TABLE %s RENAME COLUMN _operation TO operation`, htQt)) + } + + // Populate parent_id from source table + stmts = append(stmts, fmt.Sprintf( + `UPDATE %s h SET parent_id = (SELECT parent_id FROM %s s WHERE s.id = h.file_id)`, + htQt, qt)) + + // Strip history filenames to leaf names + stmts = append(stmts, fmt.Sprintf( + `UPDATE %s SET filename = split_part(filename, '/', array_length(string_to_array(filename, '/'), 1)) WHERE filename LIKE '%%/%%'`, htQt)) + + // Migrate operation values: UPDATE -> edit, DELETE -> delete + stmts = append(stmts, fmt.Sprintf( + `UPDATE %s SET operation = CASE operation WHEN 'UPDATE' THEN 'edit' WHEN 'DELETE' THEN 'delete' ELSE operation END WHERE operation IN ('UPDATE', 'DELETE')`, htQt)) + + // Recreate trigger with new column names and operation logic + stmts = append(stmts, fmt.Sprintf(`DROP TRIGGER IF EXISTS %s ON %s`, + db.QuoteIdent("trg_"+appName+"_history_archive"), qt)) + stmts = append(stmts, fmt.Sprintf(`DROP FUNCTION IF EXISTS %s.%s()`, + db.QuoteIdent(synth.TigerFSSchema), db.QuoteIdent("archive_"+appName+"_history"))) + + // Determine format from view comment + var comment string + _ = pool.QueryRow(ctx, + `SELECT obj_description(c.oid, 'pg_class') + FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname = $1 AND c.relname = $2`, schema, appName).Scan(&comment) + + features := synth.DetectFeaturesFromComment(comment) + historyStmts := synth.GenerateHistorySQL(schema, appName, features.Format) + // Only take the trigger function and trigger (indices 3 and 4) + if len(historyStmts) >= 5 { + stmts = append(stmts, historyStmts[3]) // archive function + stmts = append(stmts, historyStmts[4]) // trigger + } + } + + // --- Log table (if exists) --- + var hasLog bool + _ = pool.QueryRow(ctx, + `SELECT EXISTS(SELECT 1 FROM pg_tables WHERE schemaname = 'tigerfs' AND tablename = $1)`, + appName+"_log").Scan(&hasLog) + + if hasLog { + logQt := fmt.Sprintf("%s.%s", db.QuoteIdent(synth.TigerFSSchema), db.QuoteIdent(appName+"_log")) + + var hasOldHistoryID bool + _ = pool.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM information_schema.columns WHERE table_schema='tigerfs' AND table_name=$1 AND column_name='history_id')`, appName+"_log").Scan(&hasOldHistoryID) + if hasOldHistoryID { + stmts = append(stmts, fmt.Sprintf(`ALTER TABLE %s RENAME COLUMN history_id TO version_id`, logQt)) + } + + // Drop old CHECK, rename values, THEN add new CHECK (order matters: + // can't add new constraint while old values still exist) + stmts = append(stmts, fmt.Sprintf( + `ALTER TABLE %s DROP CONSTRAINT IF EXISTS %s`, + logQt, db.QuoteIdent(appName+"_log_type_check"))) + stmts = append(stmts, fmt.Sprintf( + `UPDATE %s SET type = CASE type WHEN 'insert' THEN 'create' WHEN 'update' THEN 'edit' ELSE type END`, + logQt)) + stmts = append(stmts, fmt.Sprintf( + `ALTER TABLE %s ADD CONSTRAINT %s CHECK (type IN ('create', 'edit', 'rename', 'delete', 'undo'))`, + logQt, db.QuoteIdent(appName+"_log_type_check"))) + } + } + return stmts, nil + }, + } +} + +// addParentDirMtimeTriggerMigration returns the migration that adds a trigger to +// update the parent directory's modified_at when children are added, removed, or +// moved. This gives directories POSIX-correct mtime semantics and ensures the NFS +// client re-fetches directory listings after changes. +func addParentDirMtimeTriggerMigration() migration { + return migration{ + Name: "parent-dir-mtime-trigger", + Summary: "Add trigger to update parent directory mtime when children change", + Detect: func(ctx context.Context, pool *pgxpool.Pool, schema string) ([]string, error) { + // Find synth apps with parent_id (parent-pointer model) but missing + // the parent mtime trigger. + rows, err := pool.Query(ctx, + `SELECT c.relname + FROM pg_class c + JOIN pg_namespace n ON n.oid = c.relnamespace + LEFT JOIN pg_description d ON d.objoid = c.oid AND d.objsubid = 0 + WHERE n.nspname = $1 AND c.relkind = 'v' + AND d.description LIKE 'tigerfs:%'`, schema) + if err != nil { + return nil, fmt.Errorf("failed to list synth views: %w", err) + } + defer rows.Close() + + var items []string + for rows.Next() { + var viewName string + if err := rows.Scan(&viewName); err != nil { + return nil, fmt.Errorf("failed to scan view: %w", err) + } + + // Check prerequisites: backing table exists in tigerfs schema with parent_id + var hasParentID bool + err := pool.QueryRow(ctx, + `SELECT EXISTS( + SELECT 1 FROM information_schema.columns + WHERE table_schema = 'tigerfs' AND table_name = $1 + AND column_name = 'parent_id' + )`, viewName).Scan(&hasParentID) + if err != nil { + return nil, fmt.Errorf("failed to check parent_id for %s: %w", viewName, err) + } + if !hasParentID { + continue + } + + // Check if trigger already exists + triggerName := "trg_" + viewName + "_parent_mtime" + var hasTrigger bool + err = pool.QueryRow(ctx, + `SELECT EXISTS( + SELECT 1 FROM pg_trigger t + JOIN pg_class c ON c.oid = t.tgrelid + JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE t.tgname = $1 AND c.relname = $2 AND n.nspname = 'tigerfs' + )`, triggerName, viewName).Scan(&hasTrigger) + if err != nil { + return nil, fmt.Errorf("failed to check trigger for %s: %w", viewName, err) + } + + if !hasTrigger { + items = append(items, viewName) + } + } + return items, nil + }, + Plan: func(ctx context.Context, pool *pgxpool.Pool, schema string, items []string) ([]string, error) { + var stmts []string + for _, appName := range items { + triggerStmts := synth.GenerateParentDirMtimeTriggerSQL(schema, appName) + stmts = append(stmts, triggerStmts...) + } + return stmts, nil + }, + } +} + // BuildMigrateCmd creates the migrate command. Exported for integration testing. // // The migrate command detects and runs pending database migrations. It supports @@ -321,7 +647,7 @@ Examples: if err := tx.Commit(ctx); err != nil { return fmt.Errorf("migration %s: failed to commit: %w", m.Name, err) } - fmt.Fprintf(w, " Migrated %d items\n", len(items)) + fmt.Fprintf(w, " Migrated %d views\n", len(items)) } if !anyPending { diff --git a/internal/tigerfs/cmd/mount.go b/internal/tigerfs/cmd/mount.go index ee9ef90..d6207ef 100644 --- a/internal/tigerfs/cmd/mount.go +++ b/internal/tigerfs/cmd/mount.go @@ -43,8 +43,12 @@ func buildMountCmd(ctx context.Context) *cobra.Command { var noFilenameExtensions bool var queryTimeout time.Duration var dirFilterLimit int + var maxPipelineDepth int var legacyFuse bool var insecureNoSSL bool + var userID string + var autoSavepointInterval time.Duration + var undoListLimit int cmd := &cobra.Command{ Use: "mount [CONNECTION] [MOUNTPOINT]", @@ -152,6 +156,9 @@ Examples: if dirFilterLimit > 0 { cfg.DirFilterLimit = dirFilterLimit } + if cmd.Flags().Changed("max-pipeline-depth") { + cfg.MaxPipelineDepth = maxPipelineDepth + } if legacyFuse { cfg.LegacyFuse = true } @@ -159,6 +166,26 @@ Examples: cfg.InsecureNoSSL = true } + // User identity: flag > env > empty (anonymous) + if userID != "" { + cfg.UserID = userID + } else if envID := os.Getenv("TIGERFS_USER_ID"); envID != "" { + cfg.UserID = envID + } + + // Auto-savepoint interval: flag overrides config + if autoSavepointInterval != 0 { + cfg.AutoSavepointInterval = autoSavepointInterval + } + if cfg.AutoSavepointInterval < 0 { + return fmt.Errorf("auto-savepoint-interval must be >= 0") + } + + // Undo list limit: flag overrides config + if undoListLimit > 0 { + cfg.UndoListLimit = undoListLimit + } + // Mount the filesystem using platform-specific method // (NFS on macOS, FUSE on Linux). fs, err := mountFilesystem(ctx, cfg, connStr, absMountpoint) @@ -213,8 +240,12 @@ Examples: cmd.Flags().BoolVar(&noFilenameExtensions, "no-filename-extensions", false, "disable automatic file extensions based on column type") cmd.Flags().DurationVar(&queryTimeout, "query-timeout", 0, "global query timeout (e.g., 30s, 1m); 0 uses config default") cmd.Flags().IntVar(&dirFilterLimit, "dir-filter-limit", 0, "row count threshold for .filter/ value listing; 0 uses config default") + cmd.Flags().IntVar(&maxPipelineDepth, "max-pipeline-depth", 0, "max chained pipeline ops before capabilities are hidden; 0 uses config default") cmd.Flags().BoolVar(&legacyFuse, "legacy-fuse", false, "use legacy FUSE node tree (Linux only)") cmd.Flags().BoolVar(&insecureNoSSL, "insecure-no-ssl", false, "allow non-TLS connections to remote databases (insecure)") + cmd.Flags().StringVar(&userID, "user-id", "", "user identity for undo log entries (also: TIGERFS_USER_ID env)") + cmd.Flags().DurationVar(&autoSavepointInterval, "auto-savepoint-interval", 0, "inactivity gap before auto-savepoint (e.g., 30m); 0 uses config default") + cmd.Flags().IntVar(&undoListLimit, "undo-list-limit", 0, "default listing limit for .undo/ sub-directories; 0 uses config default (100)") return cmd } diff --git a/internal/tigerfs/config/config.go b/internal/tigerfs/config/config.go index 7d934e6..37f5b1d 100644 --- a/internal/tigerfs/config/config.go +++ b/internal/tigerfs/config/config.go @@ -41,8 +41,9 @@ type Config struct { EntryTimeout time.Duration `mapstructure:"entry_timeout"` // Query Safety - QueryTimeout time.Duration `mapstructure:"query_timeout"` // Global statement timeout for all queries (default: 30s) - DirFilterLimit int `mapstructure:"dir_filter_limit"` // Row count threshold for .filter/ value listing (default: 100000) + QueryTimeout time.Duration `mapstructure:"query_timeout"` // Global statement timeout for all queries (default: 30s) + DirFilterLimit int `mapstructure:"dir_filter_limit"` // Row count threshold for .filter/ value listing (default: 100000) + MaxPipelineDepth int `mapstructure:"max_pipeline_depth"` // Max chained pipeline ops before capabilities are hidden (default: 10; 0=unlimited) // Metadata MetadataRefreshInterval time.Duration `mapstructure:"metadata_refresh_interval"` // Catalog TTL: schemas/tables/views (default: 10s) @@ -67,6 +68,15 @@ type Config struct { DefaultFormat string `mapstructure:"default_format"` BinaryEncoding string `mapstructure:"binary_encoding"` + // Identity + UserID string `mapstructure:"user_id"` // Mount-level user identity for log entries (--user-id or TIGERFS_USER_ID) + + // Auto-savepoints (ADR-016 Section 4.4) + AutoSavepointInterval time.Duration `mapstructure:"auto_savepoint_interval"` // Inactivity gap before auto-savepoint (default: 30m, 0 disables) + + // Undo (ADR-016 Section 4.3) + UndoListLimit int `mapstructure:"undo_list_limit"` // Default listing limit for .undo/ sub-directories (default: 100) + // FUSE backend selection (Linux only) LegacyFuse bool `mapstructure:"legacy_fuse"` // Use legacy specialized FUSE nodes instead of shared Operations @@ -80,7 +90,7 @@ func Init() error { viper.SetDefault("default_schema", "") // Empty = inherit from PostgreSQL's current_schema() viper.SetDefault("pool_size", 10) viper.SetDefault("pool_max_idle", 5) - viper.SetDefault("dir_listing_limit", 10000) + viper.SetDefault("dir_listing_limit", 1000) viper.SetDefault("dir_writing_limit", 100000) viper.SetDefault("trailing_newlines", true) viper.SetDefault("no_filename_extensions", false) @@ -88,6 +98,7 @@ func Init() error { viper.SetDefault("entry_timeout", 1*time.Second) viper.SetDefault("query_timeout", 30*time.Second) viper.SetDefault("dir_filter_limit", 100000) + viper.SetDefault("max_pipeline_depth", 10) viper.SetDefault("metadata_refresh_interval", 10*time.Second) viper.SetDefault("structural_metadata_refresh_interval", 5*time.Minute) viper.SetDefault("nfs_streaming_threshold", int64(10*1024*1024)) @@ -103,6 +114,8 @@ func Init() error { viper.SetDefault("legacy_fuse", false) viper.SetDefault("default_backend", "") viper.SetDefault("default_mount_dir", "/tmp") + viper.SetDefault("auto_savepoint_interval", 30*time.Minute) + viper.SetDefault("undo_list_limit", 100) viper.SetDefault("config_dir", GetDefaultConfigDir()) // Setup config file diff --git a/internal/tigerfs/config/config_test.go b/internal/tigerfs/config/config_test.go index d974146..99165d5 100644 --- a/internal/tigerfs/config/config_test.go +++ b/internal/tigerfs/config/config_test.go @@ -107,7 +107,7 @@ func TestInit_SetsDefaults(t *testing.T) { {"default_schema", ""}, // Empty = inherit from PostgreSQL's current_schema() {"pool_size", 10}, {"pool_max_idle", 5}, - {"dir_listing_limit", 10000}, + {"dir_listing_limit", 1000}, {"attr_timeout", 1 * time.Second}, {"entry_timeout", 1 * time.Second}, {"query_timeout", 30 * time.Second}, @@ -177,8 +177,8 @@ func TestLoad_UnmarshalConfig(t *testing.T) { if cfg.PoolMaxIdle != 5 { t.Errorf("Expected PoolMaxIdle=5, got %d", cfg.PoolMaxIdle) } - if cfg.DirListingLimit != 10000 { - t.Errorf("Expected DirListingLimit=10000, got %d", cfg.DirListingLimit) + if cfg.DirListingLimit != 1000 { + t.Errorf("Expected DirListingLimit=1000, got %d", cfg.DirListingLimit) } if cfg.AttrTimeout != 1*time.Second { t.Errorf("Expected AttrTimeout=1s, got %v", cfg.AttrTimeout) @@ -542,8 +542,8 @@ func TestConfig_FilesystemFields(t *testing.T) { t.Fatalf("Load() failed: %v", err) } - if cfg.DirListingLimit != 10000 { - t.Errorf("Expected DirListingLimit=10000, got %d", cfg.DirListingLimit) + if cfg.DirListingLimit != 1000 { + t.Errorf("Expected DirListingLimit=1000, got %d", cfg.DirListingLimit) } if cfg.AttrTimeout != 1*time.Second { t.Errorf("Expected AttrTimeout=1s, got %v", cfg.AttrTimeout) diff --git a/internal/tigerfs/db/interfaces.go b/internal/tigerfs/db/interfaces.go index a11e325..134f728 100644 --- a/internal/tigerfs/db/interfaces.go +++ b/internal/tigerfs/db/interfaces.go @@ -95,6 +95,10 @@ type RowWriter interface { // DeleteRow deletes a row by primary key. DeleteRow(ctx context.Context, schema, table string, pk *PKMatch) error + + // DeleteAndUpdate atomically deletes one row and updates another in a single + // transaction. Used for POSIX rename-as-replace (mv over existing file). + DeleteAndUpdate(ctx context.Context, schema, table string, deletePK *PKMatch, updatePK *PKMatch, updateCols []string, updateVals []interface{}) error } // IndexReader provides index metadata and lookup operations. @@ -272,6 +276,10 @@ type HistoryReader interface { // QueryHistoryDistinctFilenames returns distinct filenames from the history table. QueryHistoryDistinctFilenames(ctx context.Context, schema, historyTable string, limit int) ([]string, error) + // QueryHistoryDistinctFilenamesByParent returns distinct filenames from the history table + // filtered by parent_id. Empty parentID means root level. Used by ADR-017. + QueryHistoryDistinctFilenamesByParent(ctx context.Context, schema, historyTable, parentID string, limit int) ([]string, error) + // QueryHistoryDistinctIDs returns distinct row UUIDs from the history table. QueryHistoryDistinctIDs(ctx context.Context, schema, historyTable string, limit int) ([]string, error) @@ -283,13 +291,114 @@ type HistoryReader interface { // Used by synth views with filetype column for directory rename, child checks, etc. type HierarchyWriter interface { // RenameByPrefix atomically renames all rows where column starts with oldPrefix. + // Deprecated: Will be removed when all tables use parent-pointer model (ADR-017). RenameByPrefix(ctx context.Context, schema, table, column, oldPrefix, newPrefix string) (int64, error) // HasChildrenWithPrefix checks if any rows have column values starting with prefix + "/". + // Deprecated: Will be removed when all tables use parent-pointer model (ADR-017). HasChildrenWithPrefix(ctx context.Context, schema, table, column, prefix string) (bool, error) - // InsertIfNotExists inserts a row only if no conflict (ON CONFLICT DO NOTHING). + // InsertIfNotExists inserts a row only if no conflict. + // Uses plain INSERT with unique-violation error handling (no ON CONFLICT) + // to support deferrable constraints required by undo transactions (ADR-017). InsertIfNotExists(ctx context.Context, schema, table string, columns []string, values []interface{}) error + + // GetRowByParentAndName returns a single row matching parent_id + filename. + // Empty parentID means root level (WHERE parent_id IS NULL). + // Combines path resolution with row fetch for ReadFile (ADR-017). + GetRowByParentAndName(ctx context.Context, schema, table, parentID, filename string) ([]string, []interface{}, error) + + // GetRowsByParent returns all rows with the given parent_id, up to limit. + // Empty parentID means root level (WHERE parent_id IS NULL). + // Used by the parent-pointer directory model (ADR-017) for ReadDir. + GetRowsByParent(ctx context.Context, schema, table, parentID string, limit int) ([]string, [][]interface{}, error) +} + +// PathResolver provides path resolution for the parent-pointer directory model (ADR-017). +type PathResolver interface { + // ResolvePath resolves a sequence of path segments to row IDs by calling + // the tigerfs.resolve_path PL/pgSQL function. startParentID is empty for root. + // Returns one PathSegment per resolved segment; fewer than len(segments) + // means a segment didn't resolve (path doesn't exist). + ResolvePath(ctx context.Context, schema, table, startParentID string, segments []string) ([]PathSegment, error) +} + +// LogWriter provides operations for the undo operation log. +// Used by synth write operations to record changes for undo/recovery. +type LogWriter interface { + // InsertLogEntry inserts an operation log entry into the log hypertable. + // Parameters: + // - logTable: the log table name (e.g., "notes_log") + // - userID: the user/agent identity (may be empty for anonymous) + // - opType: operation type ("create", "edit", "rename", "delete", "undo") + // - fileID: stable UUID of the affected row + // - filename: filename at time of operation (denormalized full path for log display) + // - versionID: UUID of the history entry (before-state), empty for creates + // - description: optional human-readable note + InsertLogEntry(ctx context.Context, schema, logTable, userID, opType, fileID, filename, versionID, description string) error + + // QueryNextLogEntry finds the next log entry for a file after a given log_id. + // Returns (version_id, filename) of the next entry, or empty strings if none. + QueryNextLogEntry(ctx context.Context, schema, logTable, fileID, afterLogID string) (string, string, error) + + // QueryFileExists checks if a row with the given id exists in the source table. + QueryFileExists(ctx context.Context, schema, table, fileID string) (bool, error) + + // QueryLatestVersionID returns the most recent version_id for a given + // file_id from the history table. Used to capture the before-state pointer + // after an UPDATE/DELETE fires the BEFORE trigger. + QueryLatestVersionID(ctx context.Context, schema, historyTable, fileID string) (string, error) + + // QueryUndoAffectedFiles returns the first log entry per file after a target point. + // Uses DISTINCT ON with SkipScan on the (file_id, log_id ASC) index. + // The version_id on each entry is the before-state at the target point. + // Optional userID filter limits to operations by a specific user. + QueryUndoAffectedFiles(ctx context.Context, schema, logTable, afterID, userID string, filters []UndoFilter) ([]UndoAffectedFile, error) + + // QueryLogEntry fetches a single log entry by log_id. + QueryLogEntry(ctx context.Context, schema, logTable, logID string) (*UndoAffectedFile, error) + + // ExecuteUndoTransaction executes a batch of undo operations in a single transaction. + // Deletes rows that were created after the target, upserts rows from history + // for edits/renames/deletes, and inserts undo log entries -- all atomically. + ExecuteUndoTransaction(ctx context.Context, params *UndoTransactionParams) error +} + +// UndoAffectedFile represents a file affected by operations after the undo target. +type UndoAffectedFile struct { + FileID string // stable UUID of the affected row + Type string // first operation type after target (create, edit, rename, delete, undo) + VersionID string // version_id of the before-state (empty for creates) + Filename string // filename at time of operation + LogID string // log_id of the first operation (UUIDv7, encodes timestamp) + UserID string // user who performed the operation +} + +// UndoFilter narrows the scope of an undo operation. +type UndoFilter struct { + Column string // e.g., "user_id", "type", "filename" + Value string +} + +// UndoTransactionParams holds all parameters for an atomic undo transaction. +type UndoTransactionParams struct { + Schema string + SourceTable string // e.g., "blog" (the synth app source table) + LogTable string // e.g., "blog_log" + HistoryTable string // e.g., "blog_history" + Description string // description for undo log entries + + // Files to DELETE (created after target point) + DeleteFileIDs []string + DeleteFilenames []string // corresponding filenames for log entries (parallel array) + + // Files to UPSERT from history (edited/renamed/deleted after target) + RestoreVersionIDs []string // version_ids to fetch from history and restore + RestoreFileIDs []string // corresponding file_ids (parallel array) + RestoreFilenames []string // corresponding filenames for log entries + + // Identity for undo log entries + UserID string } // DBClient is the composite interface combining all database capabilities. @@ -309,6 +418,8 @@ type DBClient interface { PipelineReader HierarchyWriter HistoryReader + PathResolver + LogWriter } // Compile-time verification that *Client implements DBClient diff --git a/internal/tigerfs/db/mocks.go b/internal/tigerfs/db/mocks.go index 30a25ab..c8677b9 100644 --- a/internal/tigerfs/db/mocks.go +++ b/internal/tigerfs/db/mocks.go @@ -2,6 +2,7 @@ package db import ( "context" + "fmt" ) // MockDDLExecutor is a mock implementation of DDLExecutor for testing. @@ -170,6 +171,7 @@ type MockRowWriter struct { UpdateColumnFunc func(ctx context.Context, schema, table string, pk *PKMatch, columnName, newValue string) error UpdateColumnCASFunc func(ctx context.Context, schema, table string, pk *PKMatch, setColumn, newValue, whereColumn, whereValue string) error DeleteRowFunc func(ctx context.Context, schema, table string, pk *PKMatch) error + DeleteAndUpdateFunc func(ctx context.Context, schema, table string, deletePK *PKMatch, updatePK *PKMatch, updateCols []string, updateVals []interface{}) error } var _ RowWriter = (*MockRowWriter)(nil) @@ -209,6 +211,13 @@ func (m *MockRowWriter) DeleteRow(ctx context.Context, schema, table string, pk return nil } +func (m *MockRowWriter) DeleteAndUpdate(ctx context.Context, schema, table string, deletePK *PKMatch, updatePK *PKMatch, updateCols []string, updateVals []interface{}) error { + if m.DeleteAndUpdateFunc != nil { + return m.DeleteAndUpdateFunc(ctx, schema, table, deletePK, updatePK, updateCols, updateVals) + } + return nil +} + // MockIndexReader is a mock implementation of IndexReader for testing. type MockIndexReader struct { GetIndexesFunc func(ctx context.Context, schema, table string) ([]Index, error) @@ -578,6 +587,8 @@ type MockHierarchyWriter struct { RenameByPrefixFn func(ctx context.Context, schema, table, column, oldPrefix, newPrefix string) (int64, error) HasChildrenWithPrefixFn func(ctx context.Context, schema, table, column, prefix string) (bool, error) InsertIfNotExistsFn func(ctx context.Context, schema, table string, columns []string, values []interface{}) error + GetRowByParentAndNameFn func(ctx context.Context, schema, table, parentID, filename string) ([]string, []interface{}, error) + GetRowsByParentFn func(ctx context.Context, schema, table, parentID string, limit int) ([]string, [][]interface{}, error) } func (m *MockHierarchyWriter) RenameByPrefix(ctx context.Context, schema, table, column, oldPrefix, newPrefix string) (int64, error) { @@ -601,15 +612,30 @@ func (m *MockHierarchyWriter) InsertIfNotExists(ctx context.Context, schema, tab return nil } +func (m *MockHierarchyWriter) GetRowByParentAndName(ctx context.Context, schema, table, parentID, filename string) ([]string, []interface{}, error) { + if m.GetRowByParentAndNameFn != nil { + return m.GetRowByParentAndNameFn(ctx, schema, table, parentID, filename) + } + return nil, nil, nil +} + +func (m *MockHierarchyWriter) GetRowsByParent(ctx context.Context, schema, table, parentID string, limit int) ([]string, [][]interface{}, error) { + if m.GetRowsByParentFn != nil { + return m.GetRowsByParentFn(ctx, schema, table, parentID, limit) + } + return nil, nil, nil +} + // MockHistoryReader is a mock for HistoryReader. type MockHistoryReader struct { - HasExtensionFn func(ctx context.Context, extName string) (bool, error) - TableExistsFn func(ctx context.Context, schema, table string) (bool, error) - QueryHistoryByFilenameFn func(ctx context.Context, schema, historyTable, filename string, limit int) ([]string, [][]interface{}, error) - QueryHistoryByIDFn func(ctx context.Context, schema, historyTable, rowID string, limit int) ([]string, [][]interface{}, error) - QueryHistoryDistinctFilenamesFn func(ctx context.Context, schema, historyTable string, limit int) ([]string, error) - QueryHistoryDistinctIDsFn func(ctx context.Context, schema, historyTable string, limit int) ([]string, error) - QueryHistoryVersionByTimeFn func(ctx context.Context, schema, historyTable, filterColumn, filterValue string, targetTime interface{}, limit int) ([]string, [][]interface{}, error) + HasExtensionFn func(ctx context.Context, extName string) (bool, error) + TableExistsFn func(ctx context.Context, schema, table string) (bool, error) + QueryHistoryByFilenameFn func(ctx context.Context, schema, historyTable, filename string, limit int) ([]string, [][]interface{}, error) + QueryHistoryByIDFn func(ctx context.Context, schema, historyTable, rowID string, limit int) ([]string, [][]interface{}, error) + QueryHistoryDistinctFilenamesFn func(ctx context.Context, schema, historyTable string, limit int) ([]string, error) + QueryHistoryDistinctFilenamesByParentFn func(ctx context.Context, schema, historyTable, parentID string, limit int) ([]string, error) + QueryHistoryDistinctIDsFn func(ctx context.Context, schema, historyTable string, limit int) ([]string, error) + QueryHistoryVersionByTimeFn func(ctx context.Context, schema, historyTable, filterColumn, filterValue string, targetTime interface{}, limit int) ([]string, [][]interface{}, error) } func (m *MockHistoryReader) HasExtension(ctx context.Context, extName string) (bool, error) { @@ -647,6 +673,13 @@ func (m *MockHistoryReader) QueryHistoryDistinctFilenames(ctx context.Context, s return nil, nil } +func (m *MockHistoryReader) QueryHistoryDistinctFilenamesByParent(ctx context.Context, schema, historyTable, parentID string, limit int) ([]string, error) { + if m.QueryHistoryDistinctFilenamesByParentFn != nil { + return m.QueryHistoryDistinctFilenamesByParentFn(ctx, schema, historyTable, parentID, limit) + } + return nil, nil +} + func (m *MockHistoryReader) QueryHistoryDistinctIDs(ctx context.Context, schema, historyTable string, limit int) ([]string, error) { if m.QueryHistoryDistinctIDsFn != nil { return m.QueryHistoryDistinctIDsFn(ctx, schema, historyTable, limit) @@ -661,6 +694,18 @@ func (m *MockHistoryReader) QueryHistoryVersionByTime(ctx context.Context, schem return nil, nil, nil } +// MockPathResolver implements PathResolver for testing. +type MockPathResolver struct { + ResolvePathFn func(ctx context.Context, schema, table, startParentID string, segments []string) ([]PathSegment, error) +} + +func (m *MockPathResolver) ResolvePath(ctx context.Context, schema, table, startParentID string, segments []string) ([]PathSegment, error) { + if m.ResolvePathFn != nil { + return m.ResolvePathFn(ctx, schema, table, startParentID, segments) + } + return nil, nil +} + type MockDBClient struct { *MockDDLExecutor *MockSchemaReader @@ -675,6 +720,8 @@ type MockDBClient struct { *MockPipelineReader *MockHierarchyWriter *MockHistoryReader + *MockPathResolver + *MockLogWriter } var _ DBClient = (*MockDBClient)(nil) @@ -695,5 +742,55 @@ func NewMockDBClient() *MockDBClient { MockPipelineReader: &MockPipelineReader{}, MockHierarchyWriter: &MockHierarchyWriter{}, MockHistoryReader: &MockHistoryReader{}, + MockPathResolver: &MockPathResolver{}, + MockLogWriter: &MockLogWriter{}, } } + +// MockLogWriter implements LogWriter for testing. +type MockLogWriter struct { + LogEntries []MockLogEntry // Recorded log entries for verification + VersionIDs map[string]string // fileID -> latest versionID +} + +// MockLogEntry records a single log entry for test verification. +type MockLogEntry struct { + Schema, LogTable, UserID, OpType, FileID, Filename, VersionID, Description string +} + +func (m *MockLogWriter) InsertLogEntry(ctx context.Context, schema, logTable, userID, opType, fileID, filename, versionID, description string) error { + m.LogEntries = append(m.LogEntries, MockLogEntry{ + Schema: schema, LogTable: logTable, UserID: userID, OpType: opType, + FileID: fileID, Filename: filename, VersionID: versionID, Description: description, + }) + return nil +} + +func (m *MockLogWriter) QueryNextLogEntry(ctx context.Context, schema, logTable, fileID, afterLogID string) (string, string, error) { + return "", "", nil +} + +func (m *MockLogWriter) QueryFileExists(ctx context.Context, schema, table, fileID string) (bool, error) { + return false, nil +} + +func (m *MockLogWriter) QueryLatestVersionID(ctx context.Context, schema, historyTable, fileID string) (string, error) { + if m.VersionIDs != nil { + if id, ok := m.VersionIDs[fileID]; ok { + return id, nil + } + } + return "", fmt.Errorf("no history entry for %s", fileID) +} + +func (m *MockLogWriter) QueryUndoAffectedFiles(ctx context.Context, schema, logTable, afterID, userID string, filters []UndoFilter) ([]UndoAffectedFile, error) { + return nil, nil +} + +func (m *MockLogWriter) QueryLogEntry(ctx context.Context, schema, logTable, logID string) (*UndoAffectedFile, error) { + return nil, fmt.Errorf("log entry not found: %s", logID) +} + +func (m *MockLogWriter) ExecuteUndoTransaction(ctx context.Context, params *UndoTransactionParams) error { + return nil +} diff --git a/internal/tigerfs/db/pk_match.go b/internal/tigerfs/db/pk_match.go index 8718441..5b52928 100644 --- a/internal/tigerfs/db/pk_match.go +++ b/internal/tigerfs/db/pk_match.go @@ -3,6 +3,8 @@ package db import ( "fmt" "strings" + + "github.com/timescale/tigerfs/internal/tigerfs/format" ) // PKMatch holds the column names and values needed to identify a specific row @@ -226,7 +228,18 @@ func (pk *PrimaryKey) Decode(dirname string) (*PKMatch, error) { // pkDecodeSingle decodes a single-column PK value, only unescaping // filesystem-unsafe characters (slash and null byte). +// +// If the value is a UUIDv7 display name (timestamp+base36 format), it is +// converted back to the standard hex UUID format for database queries. func pkDecodeSingle(encoded string) string { + // Check for UUIDv7 display name format and convert to hex UUID + if format.IsDisplayName(encoded) { + id, err := format.DisplayNameToUUIDv7(encoded) + if err == nil { + return id.String() + } + } + if !strings.Contains(encoded, "%") { return encoded } diff --git a/internal/tigerfs/db/pk_match_test.go b/internal/tigerfs/db/pk_match_test.go index 633fb3a..9598afa 100644 --- a/internal/tigerfs/db/pk_match_test.go +++ b/internal/tigerfs/db/pk_match_test.go @@ -2,6 +2,9 @@ package db import ( "testing" + + "github.com/google/uuid" + "github.com/timescale/tigerfs/internal/tigerfs/format" ) func TestSinglePKMatch(t *testing.T) { @@ -414,3 +417,147 @@ func TestPrimaryKey_RoundTrip_Composite(t *testing.T) { } } } + +func TestPKDecode_UUIDv7DisplayName(t *testing.T) { + // Generate a UUIDv7, convert to display name, then decode via pkDecodeSingle. + // The result should be the standard hex UUID string (for DB queries). + v7, err := uuid.NewV7() + if err != nil { + t.Fatal(err) + } + displayName := format.UUIDv7ToDisplayName(v7) + + pk := &PrimaryKey{Columns: []string{"id"}} + match, err := pk.Decode(displayName) + if err != nil { + t.Fatalf("Decode(%q) error: %v", displayName, err) + } + + expectedHex := v7.String() + if match.Values[0] != expectedHex { + t.Errorf("Decode(%q) = %q, want hex UUID %q", displayName, match.Values[0], expectedHex) + } +} + +func TestPKDecode_HexUUID_Unchanged(t *testing.T) { + // A standard hex UUID (v4 or v7) should pass through Decode unchanged. + hexUUID := "019590a0-1234-7fff-8000-a1b2c3d4e5f6" + pk := &PrimaryKey{Columns: []string{"id"}} + match, err := pk.Decode(hexUUID) + if err != nil { + t.Fatalf("Decode(%q) error: %v", hexUUID, err) + } + if match.Values[0] != hexUUID { + t.Errorf("Decode(%q) = %q, want unchanged", hexUUID, match.Values[0]) + } +} + +func TestPKDecode_NonUUID_Unchanged(t *testing.T) { + // A regular string PK value (not a UUID) should pass through unchanged. + pk := &PrimaryKey{Columns: []string{"name"}} + match, err := pk.Decode("hello-world") + if err != nil { + t.Fatalf("Decode(%q) error: %v", "hello-world", err) + } + if match.Values[0] != "hello-world" { + t.Errorf("Decode(%q) = %q, want unchanged", "hello-world", match.Values[0]) + } +} + +func TestPKEncodeDecode_UUIDv7_RoundTrip(t *testing.T) { + // Full round-trip: UUIDv7 bytes -> scanAndEncodePK (display name) -> + // Decode (back to hex UUID). The hex UUID should match the original. + v7, err := uuid.NewV7() + if err != nil { + t.Fatal(err) + } + + // Simulate what scanAndEncodePK does for a UUIDv7 + bytes := [16]byte(v7) + if !format.IsUUIDv7(bytes) { + t.Fatal("generated UUID is not v7") + } + displayName := format.UUIDv7ToDisplayName(bytes) + + // Decode back (simulating path resolution) + pk := &PrimaryKey{Columns: []string{"id"}} + match, err := pk.Decode(displayName) + if err != nil { + t.Fatalf("Decode(%q) error: %v", displayName, err) + } + + expectedHex := v7.String() + if match.Values[0] != expectedHex { + t.Errorf("round-trip: display=%q decoded=%q want=%q", displayName, match.Values[0], expectedHex) + } +} + +func TestPKEncodeDecode_UUIDv4_RoundTrip(t *testing.T) { + // UUIDv4 should NOT get display name encoding -- it stays as hex + // through the entire encode/decode cycle. + v4 := uuid.New() + hexStr := v4.String() + + // pkDecodeSingle should return hex unchanged (IsDisplayName returns false) + pk := &PrimaryKey{Columns: []string{"id"}} + match, err := pk.Decode(hexStr) + if err != nil { + t.Fatalf("Decode(%q) error: %v", hexStr, err) + } + if match.Values[0] != hexStr { + t.Errorf("v4 UUID decode: got %q, want %q (unchanged)", match.Values[0], hexStr) + } +} + +func TestScanAndEncodePK_UUIDv7(t *testing.T) { + // scanAndEncodePK should produce display name format for UUIDv7 + v7, err := uuid.NewV7() + if err != nil { + t.Fatal(err) + } + values := []interface{}{[16]byte(v7)} + encoded, err := scanAndEncodePK(values, []string{"id"}) + if err != nil { + t.Fatalf("scanAndEncodePK error: %v", err) + } + if !format.IsDisplayName(encoded) { + t.Errorf("scanAndEncodePK for UUIDv7 produced non-display-name: %q", encoded) + } + + // Verify round-trip back to the same UUID + recovered, err := format.DisplayNameToUUIDv7(encoded) + if err != nil { + t.Fatalf("DisplayNameToUUIDv7(%q) error: %v", encoded, err) + } + if v7 != recovered { + t.Errorf("round-trip mismatch: orig=%s encoded=%q recovered=%s", v7, encoded, recovered) + } +} + +func TestScanAndEncodePK_UUIDv4(t *testing.T) { + // scanAndEncodePK should produce standard hex for UUIDv4 + v4 := uuid.New() + values := []interface{}{[16]byte(v4)} + encoded, err := scanAndEncodePK(values, []string{"id"}) + if err != nil { + t.Fatalf("scanAndEncodePK error: %v", err) + } + if format.IsDisplayName(encoded) { + t.Errorf("scanAndEncodePK for UUIDv4 produced display name: %q (should be hex)", encoded) + } + if encoded != v4.String() { + t.Errorf("scanAndEncodePK for UUIDv4: got %q, want %q", encoded, v4.String()) + } +} + +func TestScanAndEncodePK_NonUUID(t *testing.T) { + // Non-UUID PK values should pass through unchanged + values := []interface{}{"hello"} + encoded, err := scanAndEncodePK(values, []string{"name"}) + if err != nil { + t.Fatalf("scanAndEncodePK error: %v", err) + } + if encoded != "hello" { + t.Errorf("scanAndEncodePK for string: got %q, want %q", encoded, "hello") + } +} diff --git a/internal/tigerfs/db/query.go b/internal/tigerfs/db/query.go index 6da2f54..b73d4bf 100644 --- a/internal/tigerfs/db/query.go +++ b/internal/tigerfs/db/query.go @@ -2,15 +2,32 @@ package db import ( "context" + "errors" "fmt" "strings" + "github.com/jackc/pgx/v5/pgconn" "github.com/jackc/pgx/v5/pgxpool" "github.com/timescale/tigerfs/internal/tigerfs/format" "github.com/timescale/tigerfs/internal/tigerfs/logging" "go.uber.org/zap" ) +// isUniqueViolation returns true if the error is a PostgreSQL unique violation (SQLSTATE 23505). +func isUniqueViolation(err error) bool { + var pgErr *pgconn.PgError + return errors.As(err, &pgErr) && pgErr.Code == "23505" +} + +// asPgError extracts a pgconn.PgError from an error chain, or returns nil. +func asPgError(err error) *pgconn.PgError { + var pgErr *pgconn.PgError + if errors.As(err, &pgErr) { + return pgErr + } + return nil +} + // Row represents a database row with column names and values type Row struct { Columns []string @@ -39,9 +56,16 @@ func pkOrderByList(pkColumns []string, direction string) string { // scanAndEncodePK scans multiple PK column values from a row and encodes them // into a single string using the PrimaryKey's encoding rules. +// +// For single-column UUIDv7 primary keys, this produces a human-readable +// timestamp+base36 display name instead of the standard hex UUID format. +// Non-v7 UUIDs continue to display as hex. See ADR-016 Section 11. func scanAndEncodePK(values []interface{}, pkColumns []string) (string, error) { if len(pkColumns) == 1 { - // Single-column: convert directly + // Single-column: check for UUIDv7 before generic conversion + if bytes, ok := values[0].([16]byte); ok && format.IsUUIDv7(bytes) { + return format.UUIDv7ToDisplayName(bytes), nil + } return format.ConvertValueToText(values[0]) } // Multi-column: convert each value, then encode @@ -446,6 +470,47 @@ func (c *Client) DeleteRow(ctx context.Context, schema, table string, pk *PKMatc return DeleteRow(ctx, c.pool, schema, table, pk) } +// DeleteAndUpdate atomically deletes one row and updates another in a single +// PostgreSQL transaction. Used for POSIX rename-as-replace semantics where +// the target file must be removed before the source file can be renamed to it. +// Both BEFORE triggers (history capture) fire within the transaction. +func (c *Client) DeleteAndUpdate(ctx context.Context, schema, table string, + deletePK *PKMatch, updatePK *PKMatch, updateCols []string, updateVals []interface{}) error { + + if c.pool == nil { + return fmt.Errorf("database connection not initialized") + } + + tx, err := c.pool.Begin(ctx) + if err != nil { + return fmt.Errorf("failed to begin transaction: %w", err) + } + defer tx.Rollback(ctx) + + // Step 1: Delete the target row + deleteSQL := fmt.Sprintf(`DELETE FROM %s WHERE %s`, qt(schema, table), deletePK.WhereClause(1)) + _, err = tx.Exec(ctx, deleteSQL, deletePK.WhereArgs()...) + if err != nil { + return fmt.Errorf("failed to delete target row: %w", err) + } + + // Step 2: Update the source row (rename) + setClauses := make([]string, len(updateCols)) + for i, col := range updateCols { + setClauses[i] = fmt.Sprintf(`%s = $%d`, qi(col), i+1) + } + whereStart := len(updateVals) + 1 + updateSQL := fmt.Sprintf(`UPDATE %s SET %s WHERE %s`, + qt(schema, table), strings.Join(setClauses, ", "), updatePK.WhereClause(whereStart)) + allValues := append(updateVals, updatePK.WhereArgs()...) + _, err = tx.Exec(ctx, updateSQL, allValues...) + if err != nil { + return fmt.Errorf("failed to update source row: %w", err) + } + + return tx.Commit(ctx) +} + // GetFirstNRows returns the first N primary key values ordered by PK ascending. // Returns encoded PK strings (comma-delimited for composite PKs). func GetFirstNRows(ctx context.Context, pool *pgxpool.Pool, schema, table string, pkColumns []string, limit int) ([]string, error) { @@ -722,19 +787,442 @@ func (c *Client) InsertIfNotExists(ctx context.Context, schema, table string, co placeholders[i] = fmt.Sprintf("$%d", i+1) } + // Plain INSERT without ON CONFLICT -- deferrable unique constraints (required + // by ADR-017 for undo transactions) don't support ON CONFLICT as arbiter. + // Instead, catch the unique violation error (23505) and treat it as a no-op. query := fmt.Sprintf( - `INSERT INTO %s (%s) VALUES (%s) ON CONFLICT DO NOTHING`, + `INSERT INTO %s (%s) VALUES (%s)`, qt(schema, table), strings.Join(quotedCols, ", "), strings.Join(placeholders, ", "), ) _, err := c.pool.Exec(ctx, query, values...) if err != nil { + if isUniqueViolation(err) { + return nil // Row already exists, no-op + } return fmt.Errorf("failed to insert if not exists: %w", err) } return nil } +// GetRowsByParent returns rows with a specific parent_id, up to limit. +// Empty parentID means root level (WHERE parent_id IS NULL). +// Used by the parent-pointer directory model (ADR-017) for ReadDir. +func (c *Client) GetRowsByParent(ctx context.Context, schema, table, parentID string, limit int) ([]string, [][]interface{}, error) { + if c.pool == nil { + return nil, nil, fmt.Errorf("database connection not initialized") + } + + var query string + var args []interface{} + if parentID == "" { + query = fmt.Sprintf( + `SELECT * FROM %s WHERE "parent_id" IS NULL LIMIT $1`, + qt(schema, table), + ) + args = []interface{}{limit} + } else { + query = fmt.Sprintf( + `SELECT * FROM %s WHERE "parent_id" = $1 LIMIT $2`, + qt(schema, table), + ) + args = []interface{}{parentID, limit} + } + + return c.queryRows(ctx, query, args...) +} + +// GetRowByParentAndName returns a single row matching parent_id + filename. +// Empty parentID means root level (WHERE parent_id IS NULL). +// Returns (columns, row, error); row is nil if not found. +// Used by ReadFile to combine path resolution with row fetch in one query (ADR-017). +func (c *Client) GetRowByParentAndName(ctx context.Context, schema, table, parentID, filename string) ([]string, []interface{}, error) { + if c.pool == nil { + return nil, nil, fmt.Errorf("database connection not initialized") + } + + var query string + var args []interface{} + if parentID == "" { + query = fmt.Sprintf( + `SELECT * FROM %s WHERE "parent_id" IS NULL AND "filename" = $1 LIMIT 1`, + qt(schema, table), + ) + args = []interface{}{filename} + } else { + query = fmt.Sprintf( + `SELECT * FROM %s WHERE "parent_id" = $1 AND "filename" = $2 LIMIT 1`, + qt(schema, table), + ) + args = []interface{}{parentID, filename} + } + + columns, rows, err := c.queryRows(ctx, query, args...) + if err != nil { + return nil, nil, err + } + if len(rows) == 0 { + return columns, nil, nil + } + return columns, rows[0], nil +} + +// QueryNextLogEntry finds the next log entry for a file after a given log_id. +// Returns the version_id and filename of the next entry, or empty strings if none. +// Used by diff symlink resolution to determine the "after" state. +func (c *Client) QueryNextLogEntry(ctx context.Context, schema, logTable, fileID, afterLogID string) (string, string, error) { + if c.pool == nil { + return "", "", fmt.Errorf("database connection not initialized") + } + + query := fmt.Sprintf( + `SELECT COALESCE("version_id"::text, ''), "filename" FROM %s WHERE "file_id" = $1 AND "log_id" > $2 ORDER BY "log_id" ASC LIMIT 1`, + qt(schema, logTable), + ) + + var versionID, filename string + err := c.pool.QueryRow(ctx, query, fileID, afterLogID).Scan(&versionID, &filename) + if err != nil { + if err.Error() == "no rows in result set" { + return "", "", nil // No next entry + } + return "", "", fmt.Errorf("failed to query next log entry: %w", err) + } + return versionID, filename, nil +} + +// QueryFileExists checks if a row with the given id exists in the source table. +// Used by diff symlink resolution to determine if a file still exists. +func (c *Client) QueryFileExists(ctx context.Context, schema, table, fileID string) (bool, error) { + if c.pool == nil { + return false, fmt.Errorf("database connection not initialized") + } + + query := fmt.Sprintf( + `SELECT EXISTS(SELECT 1 FROM %s WHERE "id" = $1)`, + qt(schema, table), + ) + + var exists bool + err := c.pool.QueryRow(ctx, query, fileID).Scan(&exists) + if err != nil { + return false, fmt.Errorf("failed to check file existence: %w", err) + } + return exists, nil +} + +// QueryUndoAffectedFiles returns the first log entry per file after a target point. +// Uses DISTINCT ON to find one entry per file_id, ordered by log_id ASC (oldest first). +// TimescaleDB's SkipScan optimizes this on the (file_id, log_id ASC) index. +func (c *Client) QueryUndoAffectedFiles(ctx context.Context, schema, logTable, afterID, userID string, filters []UndoFilter) ([]UndoAffectedFile, error) { + if c.pool == nil { + return nil, fmt.Errorf("database connection not initialized") + } + + // Build WHERE clause + conditions := []string{fmt.Sprintf("%s > $1", qi("log_id"))} + args := []interface{}{afterID} + argIdx := 2 + + if userID != "" { + conditions = append(conditions, fmt.Sprintf("%s = $%d", qi("user_id"), argIdx)) + args = append(args, userID) + argIdx++ + } + + for _, f := range filters { + conditions = append(conditions, fmt.Sprintf("%s = $%d", qi(f.Column), argIdx)) + args = append(args, f.Value) + argIdx++ + } + + where := strings.Join(conditions, " AND ") + + query := fmt.Sprintf( + `SELECT DISTINCT ON (%s) %s, %s, %s, %s, %s, %s FROM %s WHERE %s ORDER BY %s, %s ASC`, + qi("file_id"), + qi("file_id"), qi("type"), qi("version_id"), qi("filename"), qi("log_id"), qi("user_id"), + qt(schema, logTable), + where, + qi("file_id"), qi("log_id"), + ) + + rows, err := c.pool.Query(ctx, query, args...) + if err != nil { + return nil, fmt.Errorf("failed to query undo affected files: %w", err) + } + defer rows.Close() + + var result []UndoAffectedFile + for rows.Next() { + var f UndoAffectedFile + var versionID *string + var userID *string + if err := rows.Scan(&f.FileID, &f.Type, &versionID, &f.Filename, &f.LogID, &userID); err != nil { + return nil, fmt.Errorf("failed to scan undo affected file: %w", err) + } + if versionID != nil { + f.VersionID = *versionID + } + if userID != nil { + f.UserID = *userID + } + result = append(result, f) + } + return result, nil +} + +// QueryLogEntry fetches a single log entry by log_id. +func (c *Client) QueryLogEntry(ctx context.Context, schema, logTable, logID string) (*UndoAffectedFile, error) { + if c.pool == nil { + return nil, fmt.Errorf("database connection not initialized") + } + + query := fmt.Sprintf( + `SELECT %s, %s, %s, %s, %s, %s FROM %s WHERE %s = $1`, + qi("file_id"), qi("type"), qi("version_id"), qi("filename"), qi("log_id"), qi("user_id"), + qt(schema, logTable), + qi("log_id"), + ) + + var f UndoAffectedFile + var versionID, userID *string + err := c.pool.QueryRow(ctx, query, logID).Scan(&f.FileID, &f.Type, &versionID, &f.Filename, &f.LogID, &userID) + if err != nil { + return nil, fmt.Errorf("log entry not found: %s", logID) + } + if versionID != nil { + f.VersionID = *versionID + } + if userID != nil { + f.UserID = *userID + } + return &f, nil +} + +// ExecuteUndoTransaction executes a batch of undo operations atomically. +// Within a single transaction: deletes created rows, upserts from history, +// and inserts undo log entries. BEFORE triggers fire on each restore, +// creating history entries for the undo itself. +func (c *Client) ExecuteUndoTransaction(ctx context.Context, params *UndoTransactionParams) error { + if c.pool == nil { + return fmt.Errorf("database connection not initialized") + } + + tx, err := c.pool.Begin(ctx) + if err != nil { + return fmt.Errorf("failed to begin undo transaction: %w", err) + } + defer tx.Rollback(ctx) + + // Defer FK and UNIQUE checks to COMMIT so we can DELETE/INSERT/UPSERT + // rows in any order within the undo. This is the deferral path documented + // in ADR-017: parent_id_fkey and the (parent_id, filename, filetype) + // uniqueness constraint are DEFERRABLE INITIALLY IMMEDIATE specifically + // to support this. Without deferral, restoring a child file before its + // parent directory row fails with a parent_id_fkey violation + // (e.g., undoing a delete_dir that contained logged children). + if _, err := tx.Exec(ctx, `SET CONSTRAINTS ALL DEFERRED`); err != nil { + return fmt.Errorf("failed to defer constraints in undo transaction: %w", err) + } + + sourceTable := qt(params.Schema, params.SourceTable) + historyTable := qt(params.Schema, params.HistoryTable) + logTable := qt(params.Schema, params.LogTable) + + // Step 1: DELETE rows that were created after the target point. + for _, fileID := range params.DeleteFileIDs { + _, err := tx.Exec(ctx, + fmt.Sprintf(`DELETE FROM %s WHERE %s = $1`, sourceTable, qi("id")), + fileID, + ) + if err != nil { + return fmt.Errorf("failed to delete created row %s: %w", fileID, err) + } + } + + // Step 2: UPSERT rows from history for edits/renames/deletes. + // Fetch each history row by version_id and restore it. + for i, versionID := range params.RestoreVersionIDs { + fileID := params.RestoreFileIDs[i] + + // Fetch history row -- get all columns dynamically + historyRow, err := tx.Query(ctx, + fmt.Sprintf(`SELECT * FROM %s WHERE %s = $1`, historyTable, qi("version_id")), + versionID, + ) + if err != nil { + return fmt.Errorf("failed to fetch history for version %s: %w", versionID, err) + } + + if !historyRow.Next() { + historyRow.Close() + return fmt.Errorf("history entry not found for version_id %s", versionID) + } + + // Get column descriptions from the result set + fieldDescs := historyRow.FieldDescriptions() + values, err := historyRow.Values() + historyRow.Close() + if err != nil { + return fmt.Errorf("failed to read history row for version %s: %w", versionID, err) + } + + // Build column/value maps, mapping history columns to source columns. + // History has: version_id, file_id, operation, + all source columns. + // Source has: id, + all other columns. file_id in history = id in source. + var sourceCols []string + var sourceVals []interface{} + for j, fd := range fieldDescs { + colName := string(fd.Name) + // Skip history-only columns + if colName == "version_id" || colName == "operation" { + continue + } + // Map file_id (history) → id (source) + if colName == "file_id" { + colName = "id" + } + sourceCols = append(sourceCols, colName) + sourceVals = append(sourceVals, values[j]) + } + + // Build UPSERT: INSERT ... ON CONFLICT (id) DO UPDATE SET ... + placeholders := make([]string, len(sourceCols)) + quotedCols := make([]string, len(sourceCols)) + updateSet := make([]string, 0, len(sourceCols)) + for j, col := range sourceCols { + placeholders[j] = fmt.Sprintf("$%d", j+1) + quotedCols[j] = qi(col) + if col != "id" { // Don't update the PK + updateSet = append(updateSet, fmt.Sprintf("%s = EXCLUDED.%s", qi(col), qi(col))) + } + } + + upsertSQL := fmt.Sprintf( + `INSERT INTO %s (%s) VALUES (%s) ON CONFLICT (%s) DO UPDATE SET %s`, + sourceTable, + strings.Join(quotedCols, ", "), + strings.Join(placeholders, ", "), + qi("id"), + strings.Join(updateSet, ", "), + ) + + _, err = tx.Exec(ctx, upsertSQL, sourceVals...) + if err != nil { + // Detect unique constraint violation: happens when undoing a delete + // after a rename-as-replace -- the renamed file now occupies the filename. + if isUniqueViolation(err) { + return fmt.Errorf("cannot restore file %s: filename already occupied "+ + "(possibly by a rename-as-replace; undo the rename first)", fileID) + } + // Detect DDL schema mismatch: column removed/renamed after the savepoint. + // PostgreSQL returns SQLSTATE 42703 (undefined_column) or 42P01 (undefined_table). + if pgErr := asPgError(err); pgErr != nil && (pgErr.Code == "42703" || pgErr.Code == "42P01") { + return fmt.Errorf("cannot undo: table schema has changed since the target point "+ + "(a column or table may have been added or removed). "+ + "Original error: %w", err) + } + return fmt.Errorf("failed to restore file %s from history: %w", fileID, err) + } + } + + // Step 3: Insert undo log entries for each affected file. + // DELETE targets get a log entry too (we're undoing the creation). + for i, fileID := range params.DeleteFileIDs { + filename := "" + if i < len(params.DeleteFilenames) { + filename = params.DeleteFilenames[i] + } + _, err := tx.Exec(ctx, + fmt.Sprintf( + `INSERT INTO %s (%s, %s, %s, %s, %s, %s) VALUES (uuidv7(), $1, 'undo', $2, $3, $4)`, + logTable, qi("log_id"), qi("file_id"), qi("type"), qi("user_id"), qi("filename"), qi("description"), + ), + fileID, params.UserID, filename, params.Description, + ) + if err != nil { + return fmt.Errorf("failed to insert undo log entry for delete: %w", err) + } + } + for i, fileID := range params.RestoreFileIDs { + // Capture version_id of the state just before our restore (the BEFORE trigger has fired) + var latestVersionID *string + err := tx.QueryRow(ctx, + fmt.Sprintf( + `SELECT %s FROM %s WHERE %s = $1 ORDER BY %s DESC LIMIT 1`, + qi("version_id"), qt(params.Schema, params.HistoryTable), + qi("file_id"), qi("version_id"), + ), + fileID, + ).Scan(&latestVersionID) + if err != nil { + latestVersionID = nil + } + + versionIDVal := "" + if latestVersionID != nil { + versionIDVal = *latestVersionID + } + + _, err = tx.Exec(ctx, + fmt.Sprintf( + `INSERT INTO %s (%s, %s, %s, %s, %s, %s, %s) VALUES (uuidv7(), $1, 'undo', $2, $3, $4, $5)`, + logTable, + qi("log_id"), qi("file_id"), qi("type"), qi("user_id"), + qi("filename"), qi("version_id"), qi("description"), + ), + fileID, params.UserID, params.RestoreFilenames[i], versionIDVal, params.Description, + ) + if err != nil { + return fmt.Errorf("failed to insert undo log entry for restore: %w", err) + } + } + + // Step 4: Bump modified_at on restored rows and their parent dirs. + // + // The bump_parent_mtime AFTER trigger normally keeps directory mtimes + // fresh on child changes, but it can miss bumps during undo: + // * Restored rows inserted via UPSERT's INSERT branch carry their + // historical modified_at (pre-delete value), not now(). + // * With deferred FK ordering, a child can be inserted before its + // parent dir row exists. The trigger's UPDATE on the (missing) + // parent matches 0 rows, and the parent restored later doesn't + // re-trigger the bump on its already-inserted children. + // + // Without an updated mtime, NFS clients with `noac` won't invalidate + // their readdir cache and will serve ghost entries from before the + // undo (file appears in `ls`, but stat/open returns ENOENT). + // + // We force the bump explicitly: each restored row's own mtime, plus + // the mtime of any parent dir that contains a restored row. + if len(params.RestoreFileIDs) > 0 { + _, err := tx.Exec(ctx, + fmt.Sprintf( + `UPDATE %s SET %s = now() WHERE %s = ANY($1::uuid[]) + OR %s IN ( + SELECT DISTINCT %s FROM %s + WHERE %s = ANY($1::uuid[]) AND %s IS NOT NULL + )`, + sourceTable, qi("modified_at"), qi("id"), + qi("id"), + qi("parent_id"), sourceTable, + qi("id"), qi("parent_id"), + ), + params.RestoreFileIDs, + ) + if err != nil { + return fmt.Errorf("failed to bump modified_at after undo: %w", err) + } + } + + if err := tx.Commit(ctx); err != nil { + return fmt.Errorf("failed to commit undo transaction: %w", err) + } + return nil +} + // HasExtension checks if a PostgreSQL extension is installed in the database. func (c *Client) HasExtension(ctx context.Context, extName string) (bool, error) { var exists bool @@ -765,20 +1253,20 @@ func (c *Client) TableExists(ctx context.Context, schema, table string) (bool, e } // QueryHistoryByFilename queries the history table for versions of a file by filename. -// Returns columns and rows ordered by _history_id DESC (most recent first). +// Returns columns and rows ordered by version_id DESC (most recent first). func (c *Client) QueryHistoryByFilename(ctx context.Context, schema, historyTable, filename string, limit int) ([]string, [][]interface{}, error) { query := fmt.Sprintf( - `SELECT * FROM %s WHERE "filename" = $1 ORDER BY "_history_id" DESC LIMIT %d`, + `SELECT * FROM %s WHERE "filename" = $1 ORDER BY "version_id" DESC LIMIT %d`, qt(schema, historyTable), limit, ) return c.queryRows(ctx, query, filename) } -// QueryHistoryByID queries the history table for versions of a row by its UUID. -// Returns columns and rows ordered by _history_id DESC (most recent first). +// QueryHistoryByID queries the history table for versions of a row by its file_id UUID. +// Returns columns and rows ordered by version_id DESC (most recent first). func (c *Client) QueryHistoryByID(ctx context.Context, schema, historyTable, rowID string, limit int) ([]string, [][]interface{}, error) { query := fmt.Sprintf( - `SELECT * FROM %s WHERE "id" = $1 ORDER BY "_history_id" DESC LIMIT %d`, + `SELECT * FROM %s WHERE "file_id" = $1 ORDER BY "version_id" DESC LIMIT %d`, qt(schema, historyTable), limit, ) return c.queryRows(ctx, query, rowID) @@ -807,10 +1295,46 @@ func (c *Client) QueryHistoryDistinctFilenames(ctx context.Context, schema, hist return filenames, nil } -// QueryHistoryDistinctIDs returns distinct row UUIDs from the history table. +// QueryHistoryDistinctFilenamesByParent returns distinct filenames from the history table +// filtered by parent_id. Empty parentID means root level (WHERE parent_id IS NULL). +// Used by the parent-pointer model (ADR-017) for per-directory .history/ listing. +func (c *Client) QueryHistoryDistinctFilenamesByParent(ctx context.Context, schema, historyTable, parentID string, limit int) ([]string, error) { + var query string + var args []interface{} + if parentID == "" { + query = fmt.Sprintf( + `SELECT DISTINCT "filename" FROM %s WHERE "parent_id" IS NULL ORDER BY "filename" LIMIT %d`, + qt(schema, historyTable), limit, + ) + } else { + query = fmt.Sprintf( + `SELECT DISTINCT "filename" FROM %s WHERE "parent_id" = $1 ORDER BY "filename" LIMIT %d`, + qt(schema, historyTable), limit, + ) + args = []interface{}{parentID} + } + + rows, err := c.pool.Query(ctx, query, args...) + if err != nil { + return nil, fmt.Errorf("failed to query distinct filenames by parent: %w", err) + } + defer rows.Close() + + var filenames []string + for rows.Next() { + var fn string + if err := rows.Scan(&fn); err != nil { + return nil, fmt.Errorf("failed to scan filename: %w", err) + } + filenames = append(filenames, fn) + } + return filenames, nil +} + +// QueryHistoryDistinctIDs returns distinct row UUIDs (file_id) from the history table. func (c *Client) QueryHistoryDistinctIDs(ctx context.Context, schema, historyTable string, limit int) ([]string, error) { query := fmt.Sprintf( - `SELECT DISTINCT "id"::text FROM %s ORDER BY "id" LIMIT %d`, + `SELECT DISTINCT "file_id"::text FROM %s ORDER BY "file_id" LIMIT %d`, qt(schema, historyTable), limit, ) rows, err := c.pool.Query(ctx, query) @@ -832,15 +1356,108 @@ func (c *Client) QueryHistoryDistinctIDs(ctx context.Context, schema, historyTab // QueryHistoryVersionByTime finds a history row matching a version ID timestamp. // Version IDs have second precision; UUIDv7 has millisecond precision. This queries -// with a 1-second window around the target timestamp plus filename or id filter. +// with a 1-second window around the target timestamp plus filename or file_id filter. func (c *Client) QueryHistoryVersionByTime(ctx context.Context, schema, historyTable, filterColumn, filterValue string, targetTime interface{}, limit int) ([]string, [][]interface{}, error) { query := fmt.Sprintf( - `SELECT * FROM %s WHERE %s = $1 ORDER BY "_history_id" DESC LIMIT %d`, + `SELECT * FROM %s WHERE %s = $1 ORDER BY "version_id" DESC LIMIT %d`, qt(schema, historyTable), qi(filterColumn), limit, ) return c.queryRows(ctx, query, filterValue) } +// InsertLogEntry inserts an operation log entry into the log hypertable. +func (c *Client) InsertLogEntry(ctx context.Context, schema, logTable, userID, opType, fileID, filename, versionID, description string) error { + query := fmt.Sprintf( + `INSERT INTO %s (user_id, type, file_id, filename, version_id, description) VALUES ($1, $2, $3, $4, $5, $6)`, + qt(schema, logTable), + ) + + // Convert empty strings to nil for nullable columns + var userIDVal, versionIDVal, descVal interface{} + if userID != "" { + userIDVal = userID + } + if versionID != "" { + versionIDVal = versionID + } + if description != "" { + descVal = description + } + + _, err := c.pool.Exec(ctx, query, userIDVal, opType, fileID, filename, versionIDVal, descVal) + if err != nil { + return fmt.Errorf("failed to insert log entry: %w", err) + } + return nil +} + +// QueryLatestVersionID returns the most recent version_id for a given +// file_id from the history table. Returns empty string if no history entry found. +func (c *Client) QueryLatestVersionID(ctx context.Context, schema, historyTable, fileID string) (string, error) { + query := fmt.Sprintf( + `SELECT "version_id" FROM %s WHERE "file_id" = $1 ORDER BY "version_id" DESC LIMIT 1`, + qt(schema, historyTable), + ) + var versionID string + err := c.pool.QueryRow(ctx, query, fileID).Scan(&versionID) + if err != nil { + return "", fmt.Errorf("failed to query latest version ID: %w", err) + } + return versionID, nil +} + +// PathSegment represents one resolved segment from the resolve_path function. +type PathSegment struct { + Depth int // 1-based depth in the resolved path + ID string // UUID of the resolved row + Name string // filename segment that was resolved +} + +// ResolvePath calls the tigerfs.resolve_path PL/pgSQL function to resolve +// a sequence of path segments to row IDs using the parent-pointer model. +// startParentID is the UUID of the starting parent (empty string for root/NULL). +// Returns one PathSegment per resolved segment. If a segment doesn't resolve, +// fewer segments are returned than requested (no error). +func (c *Client) ResolvePath(ctx context.Context, schema, table, startParentID string, segments []string) ([]PathSegment, error) { + if len(segments) == 0 { + return nil, nil + } + + qualifiedTable := fmt.Sprintf("%s.%s", qi(schema), qi(table)) + + // Convert empty startParentID to SQL NULL + var parentArg interface{} + if startParentID != "" { + parentArg = startParentID + } + + query := fmt.Sprintf( + `SELECT depth, resolved_id, resolved_name FROM %s.resolve_path($1::regclass, $2::uuid, $3::text[])`, + qi("tigerfs"), + ) + + rows, err := c.pool.Query(ctx, query, qualifiedTable, parentArg, segments) + if err != nil { + return nil, fmt.Errorf("resolve_path failed: %w", err) + } + defer rows.Close() + + var result []PathSegment + for rows.Next() { + var seg PathSegment + var id [16]byte + if err := rows.Scan(&seg.Depth, &id, &seg.Name); err != nil { + return nil, fmt.Errorf("failed to scan resolve_path result: %w", err) + } + // Convert UUID bytes to hex string + s, _ := format.ConvertValueToText(id) + seg.ID = s + result = append(result, seg) + } + + return result, nil +} + // queryRows executes a query and returns columns and row data. func (c *Client) queryRows(ctx context.Context, query string, args ...interface{}) ([]string, [][]interface{}, error) { rows, err := c.pool.Query(ctx, query, args...) diff --git a/internal/tigerfs/db/query_test.go b/internal/tigerfs/db/query_test.go index 8891acf..702c686 100644 --- a/internal/tigerfs/db/query_test.go +++ b/internal/tigerfs/db/query_test.go @@ -2,13 +2,58 @@ package db import ( "context" + "errors" "fmt" "testing" "time" + "github.com/jackc/pgx/v5/pgconn" "github.com/timescale/tigerfs/internal/tigerfs/config" ) +func TestIsUniqueViolation(t *testing.T) { + tests := []struct { + name string + err error + want bool + }{ + { + name: "unique violation", + err: &pgconn.PgError{Code: "23505"}, + want: true, + }, + { + name: "wrapped unique violation", + err: fmt.Errorf("insert failed: %w", &pgconn.PgError{Code: "23505"}), + want: true, + }, + { + name: "foreign key violation", + err: &pgconn.PgError{Code: "23503"}, + want: false, + }, + { + name: "generic error", + err: errors.New("connection refused"), + want: false, + }, + { + name: "nil error", + err: nil, + want: false, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := isUniqueViolation(tt.err) + if got != tt.want { + t.Errorf("isUniqueViolation(%v) = %v, want %v", tt.err, got, tt.want) + } + }) + } +} + func TestGetRow(t *testing.T) { connStr := getTestConnectionString(t) if connStr == "" { @@ -1838,3 +1883,622 @@ func TestClient_DeleteRow_NilPool(t *testing.T) { t.Errorf("Expected 'database connection not initialized', got: %v", err) } } + +// TestExecuteUndoTransaction_DefersFKConstraint verifies the deferral path +// documented in ADR-017: ExecuteUndoTransaction must call SET CONSTRAINTS ALL +// DEFERRED so rows can be UPSERTed in any order within the transaction. Without +// this, restoring a child row before its parent dir row fires parent_id_fkey +// (SQLSTATE 23503) immediately and aborts the undo. +// +// The test deliberately passes RestoreFileIDs in CHILD-FIRST order so the FK +// would violate without deferral. With the fix in place, the FK is checked at +// COMMIT and both rows survive. +func TestExecuteUndoTransaction_DefersFKConstraint(t *testing.T) { + connStr := getTestConnectionString(t) + if connStr == "" { + t.Skip("No PostgreSQL connection available (set PGHOST or skip)") + } + + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + + cfg := &config.Config{PoolSize: 5, PoolMaxIdle: 2} + client, err := NewClient(ctx, cfg, connStr) + if err != nil { + t.Fatalf("NewClient() failed: %v", err) + } + defer func() { _ = client.Close() }() + + // Use unique table names so parallel runs / leftover state don't collide. + suffix := fmt.Sprintf("%d", time.Now().UnixNano()) + sourceTable := "tundo_src_" + suffix + historyTable := "tundo_hist_" + suffix + logTable := "tundo_log_" + suffix + schema := "public" + + srcQT := QuoteTable(schema, sourceTable) + histQT := QuoteTable(schema, historyTable) + logQT := QuoteTable(schema, logTable) + + // Schema mirrors the relevant pieces of the synth-app shape from ADR-017: + // parent_id FK and (parent_id, filename, filetype) UNIQUE both + // DEFERRABLE INITIALLY IMMEDIATE. No archive trigger -- we'll seed history + // directly so the test stays focused on ExecuteUndoTransaction's behavior. + setupSQL := []string{ + fmt.Sprintf(`CREATE TABLE %s ( + id UUID PRIMARY KEY, + parent_id UUID REFERENCES %s(id) DEFERRABLE INITIALLY IMMEDIATE, + filename TEXT NOT NULL, + filetype TEXT NOT NULL DEFAULT 'file', + body TEXT, + modified_at TIMESTAMPTZ NOT NULL DEFAULT now(), + UNIQUE NULLS NOT DISTINCT (parent_id, filename, filetype) DEFERRABLE INITIALLY IMMEDIATE + )`, srcQT, srcQT), + fmt.Sprintf(`CREATE TABLE %s ( + file_id UUID, + parent_id UUID, + filename TEXT NOT NULL, + filetype TEXT, + body TEXT, + modified_at TIMESTAMPTZ, + version_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY, + operation TEXT NOT NULL + )`, histQT), + fmt.Sprintf(`CREATE TABLE %s ( + log_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY, + file_id UUID NOT NULL, + type TEXT NOT NULL, + user_id TEXT, + filename TEXT NOT NULL, + version_id UUID, + description TEXT + )`, logQT), + } + for _, sql := range setupSQL { + if _, err := client.pool.Exec(ctx, sql); err != nil { + t.Fatalf("setup failed (%s): %v", sql, err) + } + } + defer func() { + // Drop in reverse-dependency order; FK is on the source table itself + // (self-reference), so a single DROP suffices, but we list all three. + _, _ = client.pool.Exec(context.Background(), + fmt.Sprintf(`DROP TABLE IF EXISTS %s, %s, %s`, srcQT, histQT, logQT)) + }() + + // Pre-generate UUIDs so we can refer to them by name and assert. + var parentID, childID, parentVersion, childVersion string + err = client.pool.QueryRow(ctx, + `SELECT uuidv7()::text, uuidv7()::text, uuidv7()::text, uuidv7()::text`, + ).Scan(&parentID, &childID, &parentVersion, &childVersion) + if err != nil { + t.Fatalf("uuid generation failed: %v", err) + } + + // Seed history with delete entries for both rows. The undo will UPSERT + // from these history rows back into the source table. + insertHistory := func(versionID, fileID, parentVal, filename, filetype, body string) { + t.Helper() + var parentArg interface{} + if parentVal == "" { + parentArg = nil + } else { + parentArg = parentVal + } + _, err := client.pool.Exec(ctx, fmt.Sprintf( + `INSERT INTO %s (version_id, file_id, parent_id, filename, filetype, body, modified_at, operation) + VALUES ($1, $2, $3, $4, $5, $6, now() - interval '1 hour', 'delete')`, histQT), + versionID, fileID, parentArg, filename, filetype, body) + if err != nil { + t.Fatalf("seed history (%s): %v", filename, err) + } + } + insertHistory(parentVersion, parentID, "", "parent", "directory", "") + insertHistory(childVersion, childID, parentID, "child.md", "file", "child body") + + // Source table is empty -- both rows were "deleted" (we just simulated it + // by populating history). + + // Call ExecuteUndoTransaction with CHILD-FIRST restore order. Without + // SET CONSTRAINTS ALL DEFERRED this aborts with parent_id_fkey on the + // first UPSERT. + err = client.ExecuteUndoTransaction(ctx, &UndoTransactionParams{ + Schema: schema, + SourceTable: sourceTable, + HistoryTable: historyTable, + LogTable: logTable, + Description: "test undo deferred FK", + RestoreVersionIDs: []string{childVersion, parentVersion}, + RestoreFileIDs: []string{childID, parentID}, + RestoreFilenames: []string{"child.md", "parent"}, + UserID: "test-user", + }) + if err != nil { + t.Fatalf("ExecuteUndoTransaction must succeed with deferred constraints (child-first restore): %v", err) + } + + // Verify both rows are restored and the FK relationship is intact. + var parentExists bool + err = client.pool.QueryRow(ctx, + fmt.Sprintf(`SELECT EXISTS(SELECT 1 FROM %s WHERE id = $1 AND filetype = 'directory')`, srcQT), + parentID).Scan(&parentExists) + if err != nil { + t.Fatalf("query parent: %v", err) + } + if !parentExists { + t.Fatal("parent row should be restored") + } + + var childParent string + err = client.pool.QueryRow(ctx, + fmt.Sprintf(`SELECT parent_id::text FROM %s WHERE id = $1`, srcQT), + childID).Scan(&childParent) + if err != nil { + t.Fatalf("query child: %v", err) + } + if childParent != parentID { + t.Errorf("child.parent_id = %q, want %q", childParent, parentID) + } + + // Verify undo log entries were written for both restored rows. + var logCount int + err = client.pool.QueryRow(ctx, + fmt.Sprintf(`SELECT COUNT(*) FROM %s WHERE type = 'undo' AND file_id IN ($1, $2)`, logQT), + parentID, childID).Scan(&logCount) + if err != nil { + t.Fatalf("query log: %v", err) + } + if logCount != 2 { + t.Errorf("expected 2 undo log entries, got %d", logCount) + } + + // Verify modified_at was bumped to roughly now() on both restored rows. + // History rows have modified_at = now() - 1h; the bump-after-undo step + // should override that. Without the bump, NFS clients with `noac` keep + // serving cached readdir entries from before the undo (the file appears + // in `ls`, but stat/open returns ENOENT). + var ageSec float64 + err = client.pool.QueryRow(ctx, + fmt.Sprintf(`SELECT EXTRACT(EPOCH FROM (now() - MIN(%s))) FROM %s WHERE %s IN ($1, $2)`, + qi("modified_at"), srcQT, qi("id")), + parentID, childID).Scan(&ageSec) + if err != nil { + t.Fatalf("query mtime: %v", err) + } + if ageSec > 60 { + t.Errorf("modified_at not bumped: oldest restored row is %.0fs old (expected <60s)", ageSec) + } +} + +// undoTestEnv holds the per-test PostgreSQL fixture (client, schema, table +// names) used by the ExecuteUndoTransaction unit tests below. +type undoTestEnv struct { + client *Client + ctx context.Context + schema string + sourceTable, historyTable, logTable string + srcQT, histQT, logQT string +} + +// setupUndoTestEnv creates the parent_id-FK, UNIQUE, and history/log tables +// used by the ExecuteUndoTransaction tests, mirroring the synth-app shape from +// ADR-017. Returns the env and a cleanup func that drops the tables and +// closes the client. Skips the test if no test DB is available. +func setupUndoTestEnv(t *testing.T) (*undoTestEnv, func()) { + t.Helper() + connStr := getTestConnectionString(t) + if connStr == "" { + t.Skip("No PostgreSQL connection available (set PGHOST or skip)") + } + + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + + cfg := &config.Config{PoolSize: 5, PoolMaxIdle: 2} + client, err := NewClient(ctx, cfg, connStr) + if err != nil { + cancel() + t.Fatalf("NewClient() failed: %v", err) + } + + suffix := fmt.Sprintf("%d", time.Now().UnixNano()) + env := &undoTestEnv{ + client: client, + ctx: ctx, + schema: "public", + sourceTable: "tundo_src_" + suffix, + historyTable: "tundo_hist_" + suffix, + logTable: "tundo_log_" + suffix, + } + env.srcQT = QuoteTable(env.schema, env.sourceTable) + env.histQT = QuoteTable(env.schema, env.historyTable) + env.logQT = QuoteTable(env.schema, env.logTable) + + setupSQL := []string{ + fmt.Sprintf(`CREATE TABLE %s ( + id UUID PRIMARY KEY, + parent_id UUID REFERENCES %s(id) DEFERRABLE INITIALLY IMMEDIATE, + filename TEXT NOT NULL, + filetype TEXT NOT NULL DEFAULT 'file', + body TEXT, + modified_at TIMESTAMPTZ NOT NULL DEFAULT now(), + UNIQUE NULLS NOT DISTINCT (parent_id, filename, filetype) DEFERRABLE INITIALLY IMMEDIATE + )`, env.srcQT, env.srcQT), + fmt.Sprintf(`CREATE TABLE %s ( + file_id UUID, + parent_id UUID, + filename TEXT NOT NULL, + filetype TEXT, + body TEXT, + modified_at TIMESTAMPTZ, + version_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY, + operation TEXT NOT NULL + )`, env.histQT), + fmt.Sprintf(`CREATE TABLE %s ( + log_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY, + file_id UUID NOT NULL, + type TEXT NOT NULL, + user_id TEXT, + filename TEXT NOT NULL, + version_id UUID, + description TEXT + )`, env.logQT), + } + for _, sql := range setupSQL { + if _, err := client.pool.Exec(ctx, sql); err != nil { + cancel() + _ = client.Close() + t.Fatalf("setup failed (%s): %v", sql, err) + } + } + + cleanup := func() { + _, _ = client.pool.Exec(context.Background(), + fmt.Sprintf(`DROP TABLE IF EXISTS %s, %s, %s`, env.srcQT, env.histQT, env.logQT)) + _ = client.Close() + cancel() + } + return env, cleanup +} + +// seedHistory inserts a history row with operation='delete' (the typical +// archive that an undo would restore from) and modified_at = now() - 1h, so +// the restored row's modified_at is observably old without the bump. +func (env *undoTestEnv) seedHistory(t *testing.T, versionID, fileID, parent, filename, filetype, body string) { + t.Helper() + var parentArg interface{} + if parent != "" { + parentArg = parent + } + _, err := env.client.pool.Exec(env.ctx, fmt.Sprintf( + `INSERT INTO %s (version_id, file_id, parent_id, filename, filetype, body, modified_at, operation) + VALUES ($1, $2, $3, $4, $5, $6, now() - interval '1 hour', 'delete')`, env.histQT), + versionID, fileID, parentArg, filename, filetype, body) + if err != nil { + t.Fatalf("seed history (%s): %v", filename, err) + } +} + +// genUUIDs returns n new UUIDv7 strings. +func (env *undoTestEnv) genUUIDs(t *testing.T, n int) []string { + t.Helper() + out := make([]string, n) + for i := 0; i < n; i++ { + var u string + if err := env.client.pool.QueryRow(env.ctx, `SELECT uuidv7()::text`).Scan(&u); err != nil { + t.Fatalf("uuidv7() failed: %v", err) + } + out[i] = u + } + return out +} + +// TestExecuteUndoTransaction_DefersUniqueConstraint verifies the second half +// of the SET CONSTRAINTS ALL DEFERRED contract: the +// (parent_id, filename, filetype) UNIQUE constraint must also defer to COMMIT. +// +// Scenario: rename-as-replace. A and B both exist at root. The user does +// `mv A B` -- B's row is deleted, A's filename is changed to 'B'. Undoing +// this restoration must produce both rows with their original filenames. +// +// We force RestoreFileIDs in B-first order so B's INSERT runs while A still +// has filename='B' in the source table -- this is a transient UNIQUE +// violation that only resolves when A's UPSERT then reverts filename='A'. +// Without deferral, B's INSERT fails immediately. The pre-existing +// "filename already occupied (rename-as-replace)" error path at query.go is +// bypassed by the deferral path -- this test pins that down. +func TestExecuteUndoTransaction_DefersUniqueConstraint(t *testing.T) { + env, cleanup := setupUndoTestEnv(t) + defer cleanup() + + ids := env.genUUIDs(t, 4) + aID, bID := ids[0], ids[1] + aVer, bVer := ids[2], ids[3] + + // History: A's pre-rename state (filename='A'), B's pre-delete state + // (filename='B'). Both at root (parent=NULL). + env.seedHistory(t, aVer, aID, "", "A", "file", "body A") + env.seedHistory(t, bVer, bID, "", "B", "file", "body B") + + // Source post-rename-as-replace: A row exists with filename='B', B row gone. + _, err := env.client.pool.Exec(env.ctx, + fmt.Sprintf(`INSERT INTO %s (id, parent_id, filename, filetype, body) VALUES ($1, NULL, 'B', 'file', 'body A')`, env.srcQT), + aID) + if err != nil { + t.Fatalf("seed source: %v", err) + } + + // B-first restore order forces the transient UNIQUE violation. + err = env.client.ExecuteUndoTransaction(env.ctx, &UndoTransactionParams{ + Schema: env.schema, + SourceTable: env.sourceTable, + HistoryTable: env.historyTable, + LogTable: env.logTable, + Description: "test undo deferred UNIQUE", + RestoreVersionIDs: []string{bVer, aVer}, + RestoreFileIDs: []string{bID, aID}, + RestoreFilenames: []string{"B", "A"}, + UserID: "test-user", + }) + if err != nil { + t.Fatalf("ExecuteUndoTransaction must succeed with deferred UNIQUE: %v", err) + } + + // Verify both rows have their restored filenames. + rows, err := env.client.pool.Query(env.ctx, + fmt.Sprintf(`SELECT id::text, filename FROM %s WHERE id IN ($1, $2) ORDER BY filename`, env.srcQT), + aID, bID) + if err != nil { + t.Fatalf("query source: %v", err) + } + defer rows.Close() + got := map[string]string{} + for rows.Next() { + var id, filename string + if err := rows.Scan(&id, &filename); err != nil { + t.Fatalf("scan: %v", err) + } + got[id] = filename + } + if got[aID] != "A" { + t.Errorf("A.filename = %q, want %q", got[aID], "A") + } + if got[bID] != "B" { + t.Errorf("B.filename = %q, want %q", got[bID], "B") + } +} + +// TestExecuteUndoTransaction_BumpsParentMtimeWhenOnlyChildRestored isolates +// the SELECT-DISTINCT-parent_id branch of the post-undo modified_at bump. +// +// The parent dir exists throughout (never in the restore set); only a +// child file is restored. The bump must still touch the parent so an NFS +// client with `noac` re-reads its directory listing. +func TestExecuteUndoTransaction_BumpsParentMtimeWhenOnlyChildRestored(t *testing.T) { + env, cleanup := setupUndoTestEnv(t) + defer cleanup() + + ids := env.genUUIDs(t, 3) + parentID, childID, childVer := ids[0], ids[1], ids[2] + + // Parent dir created an hour ago (set explicitly so we can detect + // whether the bump touched it). + _, err := env.client.pool.Exec(env.ctx, fmt.Sprintf( + `INSERT INTO %s (id, parent_id, filename, filetype, modified_at) + VALUES ($1, NULL, 'parent', 'directory', now() - interval '1 hour')`, env.srcQT), + parentID) + if err != nil { + t.Fatalf("seed parent: %v", err) + } + + // Child existed under parent, then was deleted -- only history retains it. + env.seedHistory(t, childVer, childID, parentID, "child.md", "file", "child body") + + // Restore the child only. + err = env.client.ExecuteUndoTransaction(env.ctx, &UndoTransactionParams{ + Schema: env.schema, + SourceTable: env.sourceTable, + HistoryTable: env.historyTable, + LogTable: env.logTable, + Description: "test undo bumps parent mtime", + RestoreVersionIDs: []string{childVer}, + RestoreFileIDs: []string{childID}, + RestoreFilenames: []string{"child.md"}, + UserID: "test-user", + }) + if err != nil { + t.Fatalf("ExecuteUndoTransaction: %v", err) + } + + // Parent's modified_at should be fresh (within 60s of now). + var ageSec float64 + err = env.client.pool.QueryRow(env.ctx, + fmt.Sprintf(`SELECT EXTRACT(EPOCH FROM (now() - modified_at)) FROM %s WHERE id = $1`, env.srcQT), + parentID).Scan(&ageSec) + if err != nil { + t.Fatalf("query parent mtime: %v", err) + } + if ageSec > 60 { + t.Errorf("parent.modified_at not bumped: %.0fs old (expected <60s)", ageSec) + } +} + +// TestExecuteUndoTransaction_DefersFKMultiLevel extends the FK-deferral +// coverage to a 3-level hierarchy (grandparent -> parent -> child). The +// restore order is reversed (child, parent, grandparent), so each row's +// parent_id references a row that hasn't been UPSERTed yet. The whole +// chain must resolve at COMMIT. +// +// Catches a regression if someone narrows the deferral scope so that only +// the immediate FK is deferred; here the FK chain spans two hops. +func TestExecuteUndoTransaction_DefersFKMultiLevel(t *testing.T) { + env, cleanup := setupUndoTestEnv(t) + defer cleanup() + + ids := env.genUUIDs(t, 6) + gID, pID, cID := ids[0], ids[1], ids[2] + gVer, pVer, cVer := ids[3], ids[4], ids[5] + + // History: all three exist as deletes (root grandparent, nested parent + // under it, leaf child under parent). + env.seedHistory(t, gVer, gID, "", "g", "directory", "") + env.seedHistory(t, pVer, pID, gID, "p", "directory", "") + env.seedHistory(t, cVer, cID, pID, "c.md", "file", "child") + + // Source is empty; everything restores via UPSERT-INSERT. + // Reverse order forces the FK chain to be resolved at COMMIT only. + err := env.client.ExecuteUndoTransaction(env.ctx, &UndoTransactionParams{ + Schema: env.schema, + SourceTable: env.sourceTable, + HistoryTable: env.historyTable, + LogTable: env.logTable, + Description: "test undo 3-level FK", + RestoreVersionIDs: []string{cVer, pVer, gVer}, + RestoreFileIDs: []string{cID, pID, gID}, + RestoreFilenames: []string{"c.md", "p", "g"}, + UserID: "test-user", + }) + if err != nil { + t.Fatalf("ExecuteUndoTransaction must succeed with deferred FK chain: %v", err) + } + + // Verify the full chain: g exists at root, p->g, c->p. + var gParent, pParent, cParent *string + row := env.client.pool.QueryRow(env.ctx, + fmt.Sprintf(`SELECT (SELECT parent_id::text FROM %s WHERE id = $1), + (SELECT parent_id::text FROM %s WHERE id = $2), + (SELECT parent_id::text FROM %s WHERE id = $3)`, + env.srcQT, env.srcQT, env.srcQT), + gID, pID, cID) + if err := row.Scan(&gParent, &pParent, &cParent); err != nil { + t.Fatalf("query chain: %v", err) + } + if gParent != nil { + t.Errorf("g.parent_id = %v, want NULL", *gParent) + } + if pParent == nil || *pParent != gID { + t.Errorf("p.parent_id = %v, want %s", pParent, gID) + } + if cParent == nil || *cParent != pID { + t.Errorf("c.parent_id = %v, want %s", cParent, pID) + } +} + +// TestExecuteUndoTransaction_DeleteOnlyUndo guards the empty-RestoreFileIDs +// branch of the post-undo modified_at bump. When a user undoes a single +// 'create' op, the only work is to DELETE the created row from source -- no +// rows are restored. The mtime-bump SQL uses ANY($1::uuid[]) and would +// error on an empty array if the branch wasn't gated. +func TestExecuteUndoTransaction_DeleteOnlyUndo(t *testing.T) { + env, cleanup := setupUndoTestEnv(t) + defer cleanup() + + ids := env.genUUIDs(t, 1) + rID := ids[0] + + // Source has one row; the undo will delete it. + _, err := env.client.pool.Exec(env.ctx, + fmt.Sprintf(`INSERT INTO %s (id, parent_id, filename, filetype, body) VALUES ($1, NULL, 'r.md', 'file', 'data')`, env.srcQT), + rID) + if err != nil { + t.Fatalf("seed source: %v", err) + } + + err = env.client.ExecuteUndoTransaction(env.ctx, &UndoTransactionParams{ + Schema: env.schema, + SourceTable: env.sourceTable, + HistoryTable: env.historyTable, + LogTable: env.logTable, + Description: "test undo delete-only", + DeleteFileIDs: []string{rID}, + DeleteFilenames: []string{"r.md"}, + UserID: "test-user", + }) + if err != nil { + t.Fatalf("ExecuteUndoTransaction (delete-only) must succeed: %v", err) + } + + // Row gone from source. + var exists bool + err = env.client.pool.QueryRow(env.ctx, + fmt.Sprintf(`SELECT EXISTS(SELECT 1 FROM %s WHERE id = $1)`, env.srcQT), + rID).Scan(&exists) + if err != nil { + t.Fatalf("query source: %v", err) + } + if exists { + t.Errorf("row should be deleted by undo") + } + + // Undo log entry was written. + var logCount int + err = env.client.pool.QueryRow(env.ctx, + fmt.Sprintf(`SELECT COUNT(*) FROM %s WHERE file_id = $1 AND type = 'undo'`, env.logQT), + rID).Scan(&logCount) + if err != nil { + t.Fatalf("query log: %v", err) + } + if logCount != 1 { + t.Errorf("expected 1 undo log entry, got %d", logCount) + } +} + +// TestExecuteUndoTransaction_FailsAtCommitOnUnrestorableFK verifies that +// the deferred FK doesn't silently allow orphan rows. If a restore +// targets a child whose parent_id references a row that's neither in +// the source table nor in the restore set, the deferred FK *must* fire +// at COMMIT and the whole undo transaction must roll back. +// +// We never expect this state from normal user operations -- the undo +// classifier always pulls in the parent's row when its children are +// affected -- but the test pins down what happens if a future refactor +// or log inconsistency produces an unrestorable child: a clean FK error +// at COMMIT, no orphan, no silent corruption. The whole point of using +// DEFERRABLE INITIALLY IMMEDIATE (vs just removing the FK) is that +// genuinely-broken state still surfaces; we exercise that here. +func TestExecuteUndoTransaction_FailsAtCommitOnUnrestorableFK(t *testing.T) { + env, cleanup := setupUndoTestEnv(t) + defer cleanup() + + ids := env.genUUIDs(t, 3) + childID, missingParentID, childVer := ids[0], ids[1], ids[2] + + // Seed history with a child whose parent_id points at a UUID that has + // no history row and no source row -- the unrestorable case. + env.seedHistory(t, childVer, childID, missingParentID, "orphan.md", "file", "body") + + err := env.client.ExecuteUndoTransaction(env.ctx, &UndoTransactionParams{ + Schema: env.schema, + SourceTable: env.sourceTable, + HistoryTable: env.historyTable, + LogTable: env.logTable, + Description: "test undo unrestorable FK", + RestoreVersionIDs: []string{childVer}, + RestoreFileIDs: []string{childID}, + RestoreFilenames: []string{"orphan.md"}, + UserID: "test-user", + }) + if err == nil { + t.Fatal("ExecuteUndoTransaction must fail when restoring a child whose parent doesn't exist anywhere") + } + + pgErr := asPgError(err) + if pgErr == nil { + t.Fatalf("expected pg error, got: %v", err) + } + if pgErr.Code != "23503" { + t.Errorf("expected SQLSTATE 23503 (foreign_key_violation), got %q: %v", pgErr.Code, pgErr.Message) + } + + // Source must be empty -- the UPSERT happened during the TX but the + // COMMIT rolled back when the deferred FK check failed. + var count int + err = env.client.pool.QueryRow(env.ctx, + fmt.Sprintf(`SELECT COUNT(*) FROM %s WHERE id = $1`, env.srcQT), + childID).Scan(&count) + if err != nil { + t.Fatalf("query source: %v", err) + } + if count != 0 { + t.Errorf("expected source to be empty after rolled-back undo, got %d row(s)", count) + } +} diff --git a/internal/tigerfs/format/uuidv7.go b/internal/tigerfs/format/uuidv7.go new file mode 100644 index 0000000..7ad4080 --- /dev/null +++ b/internal/tigerfs/format/uuidv7.go @@ -0,0 +1,178 @@ +package format + +import ( + "encoding/binary" + "fmt" + "math/big" + "time" + + "github.com/google/uuid" +) + +// DisplayNameLayout is the time format for the timestamp portion of UUIDv7 +// display names. Uses millisecond precision, UTC, filesystem-safe (no colons). +const DisplayNameLayout = "2006-01-02T150405.000Z" + +// displayNameTSLen is the length of the timestamp portion in a display name. +var displayNameTSLen = len(DisplayNameLayout) // 22 + +// UUIDv7ToDisplayName converts a UUIDv7 to a human-readable display name: +// "-", e.g. "2026-04-07T143000.123Z-zzz0063hd8e5r42". +// +// The format is fully reversible via DisplayNameToUUIDv7. It encodes all 122 +// meaningful bits of the UUID (48 timestamp + 12 rand_a + 62 rand_b). The 4 +// version bits and 2 variant bits are fixed constants reconstructed on decode. +// +// Base36 (0-9a-z) is used for the entropy portion to be case-insensitive safe +// on macOS APFS. +func UUIDv7ToDisplayName(id [16]byte) string { + ts := ExtractUUIDv7Time(id) + entropy := packEntropy(id) + var n big.Int + n.SetBytes(entropy) + return fmt.Sprintf("%s-%s", ts.UTC().Format(DisplayNameLayout), n.Text(36)) +} + +// DisplayNameToUUIDv7 parses a display name back to a UUIDv7. This is the +// inverse of UUIDv7ToDisplayName -- fully reversible with no lookup needed. +func DisplayNameToUUIDv7(name string) (uuid.UUID, error) { + var id uuid.UUID + + // Find the separator between timestamp and entropy. + // The timestamp is fixed-length (22 chars: "2006-01-02T150405.000Z"). + // The entropy follows after a "-". + if len(name) < displayNameTSLen+2 { // timestamp + dash + at least 1 entropy char + return id, fmt.Errorf("invalid UUIDv7 display name %q: too short", name) + } + tsStr := name[:displayNameTSLen] + if name[displayNameTSLen] != '-' { + return id, fmt.Errorf("invalid UUIDv7 display name %q: expected '-' at position %d", name, displayNameTSLen) + } + entropyStr := name[displayNameTSLen+1:] + + // Parse timestamp -> reconstruct bits 0-47 + ts, err := time.Parse(DisplayNameLayout, tsStr) + if err != nil { + return id, fmt.Errorf("invalid UUIDv7 display name %q: %w", name, err) + } + msec := ts.UnixMilli() + + // Reconstruct first 6 bytes from millisecond timestamp + binary.BigEndian.PutUint16(id[0:2], uint16(msec>>32)) + binary.BigEndian.PutUint32(id[2:6], uint32(msec)) + + // Parse base36 entropy -> reconstruct bits 52-63 and 66-127 + var n big.Int + _, ok := n.SetString(entropyStr, 36) + if !ok { + return id, fmt.Errorf("invalid UUIDv7 display name %q: invalid base36 entropy", name) + } + entropy := n.Bytes() + unpackEntropy(id[:], entropy) + + // Set version bits (48-51 = 0111) and variant bits (64-65 = 10) + id[6] = (id[6] & 0x0F) | 0x70 // version 7 + id[8] = (id[8] & 0x3F) | 0x80 // variant 10 + + return id, nil +} + +// IsUUIDv7 checks whether a UUID is version 7 by inspecting the version bits +// at positions 48-51 (the high nibble of byte 6). +func IsUUIDv7(id [16]byte) bool { + return (id[6] >> 4) == 7 +} + +// IsDisplayName checks whether a string looks like a UUIDv7 display name +// (timestamp-base36 format). Used for path resolution to distinguish display +// names from hex UUIDs. +func IsDisplayName(s string) bool { + if len(s) < displayNameTSLen+2 { + return false + } + if s[displayNameTSLen] != '-' { + return false + } + _, err := time.Parse(DisplayNameLayout, s[:displayNameTSLen]) + return err == nil +} + +// ExtractUUIDv7Time extracts the millisecond timestamp from a UUIDv7. +// UUIDv7 stores a Unix timestamp in milliseconds in the first 48 bits. +func ExtractUUIDv7Time(id [16]byte) time.Time { + b := id[:] + msec := int64(binary.BigEndian.Uint16(b[0:2]))<<32 | + int64(binary.BigEndian.Uint32(b[2:6])) + return time.UnixMilli(msec) +} + +// packEntropy extracts the 74 entropy bits (rand_a + rand_b) from a UUIDv7 +// and packs them into a byte slice suitable for big.Int encoding. +func packEntropy(id [16]byte) []byte { + // Extract rand_a (12 bits): low nibble of byte 6 + all of byte 7 + randA := uint16(id[6]&0x0F)<<8 | uint16(id[7]) + + // Extract rand_b (62 bits): low 6 bits of byte 8 + bytes 9-15 + var randB uint64 + randB = uint64(id[8]&0x3F) << 56 + randB |= uint64(id[9]) << 48 + randB |= uint64(id[10]) << 40 + randB |= uint64(id[11]) << 32 + randB |= uint64(id[12]) << 24 + randB |= uint64(id[13]) << 16 + randB |= uint64(id[14]) << 8 + randB |= uint64(id[15]) + + // Pack 74 bits into 10 bytes (80 bits, 6 leading zero bits) + buf := make([]byte, 10) + buf[0] = byte(randA >> 4) + buf[1] = byte(randA<<4) | byte(randB>>58) + buf[2] = byte(randB >> 50) + buf[3] = byte(randB >> 42) + buf[4] = byte(randB >> 34) + buf[5] = byte(randB >> 26) + buf[6] = byte(randB >> 18) + buf[7] = byte(randB >> 10) + buf[8] = byte(randB >> 2) + buf[9] = byte(randB << 6) + + return buf +} + +// unpackEntropy restores the 74 entropy bits from a big.Int byte encoding +// back into the UUID byte array (bytes 6-15), preserving version and variant +// bits which are set by the caller. +func unpackEntropy(id []byte, entropy []byte) { + // Pad entropy to 10 bytes (big.Int.Bytes() strips leading zeros) + padded := make([]byte, 10) + copy(padded[10-len(entropy):], entropy) + + // Extract rand_a (12 bits) from bits 6-17 of the 80-bit field + randA := uint16(padded[0])<<4 | uint16(padded[1]>>4) + + // Extract rand_b (62 bits) from bits 18-79 + var randB uint64 + randB = uint64(padded[1]&0x0F) << 58 + randB |= uint64(padded[2]) << 50 + randB |= uint64(padded[3]) << 42 + randB |= uint64(padded[4]) << 34 + randB |= uint64(padded[5]) << 26 + randB |= uint64(padded[6]) << 18 + randB |= uint64(padded[7]) << 10 + randB |= uint64(padded[8]) << 2 + randB |= uint64(padded[9]) >> 6 + + // Write rand_a into bytes 6-7 (preserving version bits in high nibble of byte 6) + id[6] = (id[6] & 0xF0) | byte(randA>>8) + id[7] = byte(randA) + + // Write rand_b into bytes 8-15 (preserving variant bits in high 2 bits of byte 8) + id[8] = (id[8] & 0xC0) | byte(randB>>56) + id[9] = byte(randB >> 48) + id[10] = byte(randB >> 40) + id[11] = byte(randB >> 32) + id[12] = byte(randB >> 24) + id[13] = byte(randB >> 16) + id[14] = byte(randB >> 8) + id[15] = byte(randB) +} diff --git a/internal/tigerfs/format/uuidv7_test.go b/internal/tigerfs/format/uuidv7_test.go new file mode 100644 index 0000000..65e9e1f --- /dev/null +++ b/internal/tigerfs/format/uuidv7_test.go @@ -0,0 +1,357 @@ +package format + +import ( + "testing" + "time" + + "github.com/google/uuid" +) + +func TestUUIDv7ToDisplayName_RoundTrip(t *testing.T) { + // Generate 100 UUIDv7s and verify lossless round-trip + for i := 0; i < 100; i++ { + id, err := uuid.NewV7() + if err != nil { + t.Fatal(err) + } + name := UUIDv7ToDisplayName(id) + recovered, err := DisplayNameToUUIDv7(name) + if err != nil { + t.Fatalf("DisplayNameToUUIDv7(%q) error: %v", name, err) + } + if id != recovered { + t.Fatalf("round-trip mismatch (iteration %d):\n orig: %s\n display: %s\n recovered: %s", i, id, name, recovered) + } + } +} + +func TestUUIDv7ToDisplayName_Format(t *testing.T) { + id, _ := uuid.NewV7() + name := UUIDv7ToDisplayName(id) + + // Timestamp portion is 22 chars, separator at position 22 + tsLen := len(DisplayNameLayout) + if len(name) < tsLen+2 { + t.Fatalf("display name too short: %q (len=%d)", name, len(name)) + } + if name[tsLen] != '-' { + t.Fatalf("expected '-' at position %d in %q", tsLen, name) + } + + // Entropy portion should be base36 (lowercase alphanumeric only) + entropy := name[tsLen+1:] + for _, c := range entropy { + if !((c >= '0' && c <= '9') || (c >= 'a' && c <= 'z')) { + t.Fatalf("non-base36 character %q in entropy portion of %q", string(c), name) + } + } + + // Timestamp should parse correctly and be close to now + ts := ExtractUUIDv7Time(id) + diff := time.Since(ts) + if diff < 0 || diff > 2*time.Second { + t.Errorf("timestamp %v off from now by %v", ts, diff) + } +} + +func TestUUIDv7ToDisplayName_TimestampMillisecondPrecision(t *testing.T) { + // Create two UUIDv7s with slightly different timestamps and verify + // the millisecond portion differs + id1, _ := uuid.NewV7() + time.Sleep(2 * time.Millisecond) + id2, _ := uuid.NewV7() + + name1 := UUIDv7ToDisplayName(id1) + name2 := UUIDv7ToDisplayName(id2) + + if name1 == name2 { + t.Errorf("two UUIDv7s generated 2ms apart produced identical display names: %q", name1) + } + + // Both should sort chronologically (string comparison) + if name1 > name2 { + t.Errorf("display names don't sort chronologically:\n first: %s\n second: %s", name1, name2) + } +} + +func TestDisplayNameToUUIDv7_Invalid(t *testing.T) { + tests := []struct { + name string + input string + }{ + {"empty", ""}, + {"too short", "abc"}, + {"no separator", "2026-04-07T143000.123Z"}, + {"bad timestamp", "not-a-timestamp.000Z-abc"}, + {"bad entropy", "2026-04-07T143000.123Z-!!!invalid!!!"}, + {"wrong separator position", "2026-04-07T143000.123Zabc"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + _, err := DisplayNameToUUIDv7(tt.input) + if err == nil { + t.Errorf("DisplayNameToUUIDv7(%q) expected error, got nil", tt.input) + } + }) + } +} + +func TestIsUUIDv7(t *testing.T) { + v7, err := uuid.NewV7() + if err != nil { + t.Fatal(err) + } + if !IsUUIDv7(v7) { + t.Errorf("IsUUIDv7(%s) = false for v7, want true", v7) + } + + v4 := uuid.New() // v4 + if IsUUIDv7(v4) { + t.Errorf("IsUUIDv7(%s) = true for v4, want false", v4) + } + + var zero [16]byte + if IsUUIDv7(zero) { + t.Error("IsUUIDv7(zero) = true, want false") + } +} + +func TestIsDisplayName(t *testing.T) { + // Valid display name + id, _ := uuid.NewV7() + name := UUIDv7ToDisplayName(id) + if !IsDisplayName(name) { + t.Errorf("IsDisplayName(%q) = false, want true", name) + } + + // Not display names + tests := []string{ + "", + "abc", + "019590a0-1234-7fff-8000-a1b2c3d4e5f6", // hex UUID + "2026-04-07T143000Z", // old version ID format + "not-a-display-name", + } + for _, s := range tests { + if IsDisplayName(s) { + t.Errorf("IsDisplayName(%q) = true, want false", s) + } + } +} + +func TestUUIDv7ToDisplayName_ZeroEntropy(t *testing.T) { + // UUIDv7 with rand_a=0, rand_b=0. big.Int.Bytes() returns empty for + // zero, so this tests that unpackEntropy handles the padding correctly. + var id [16]byte + // Set a valid timestamp + id[0], id[1], id[2], id[3], id[4], id[5] = 0x01, 0x9D, 0xA4, 0xC1, 0xB8, 0x3B + id[6] = 0x70 // version=7, rand_a high=0 + id[7] = 0x00 // rand_a low=0 + id[8] = 0x80 // variant=10, rand_b high=0 + // bytes 9-15 all zero (rand_b=0) + + name := UUIDv7ToDisplayName(id) + recovered, err := DisplayNameToUUIDv7(name) + if err != nil { + t.Fatalf("round-trip failed for zero entropy: %v", err) + } + if uuid.UUID(id) != recovered { + t.Fatalf("round-trip mismatch for zero entropy:\n orig: %x\n display: %s\n recovered: %x", id, name, recovered) + } +} + +func TestUUIDv7ToDisplayName_MaxEntropy(t *testing.T) { + // UUIDv7 with max rand_a (0xFFF) and max rand_b (62 bits all set) + var id [16]byte + id[0], id[1], id[2], id[3], id[4], id[5] = 0x01, 0x9D, 0xA4, 0xC1, 0xB8, 0x3B + id[6] = 0x7F // version=7, rand_a high=F + id[7] = 0xFF // rand_a low=FF (rand_a = 0xFFF) + id[8] = 0xBF // variant=10, rand_b high 6 bits=111111 + id[9] = 0xFF + id[10] = 0xFF + id[11] = 0xFF + id[12] = 0xFF + id[13] = 0xFF + id[14] = 0xFF + id[15] = 0xFF + + name := UUIDv7ToDisplayName(id) + recovered, err := DisplayNameToUUIDv7(name) + if err != nil { + t.Fatalf("round-trip failed for max entropy: %v", err) + } + if uuid.UUID(id) != recovered { + t.Fatalf("round-trip mismatch for max entropy:\n orig: %x\n display: %s\n recovered: %x", id, name, recovered) + } +} + +func TestUUIDv7ToDisplayName_SmallEntropy(t *testing.T) { + // UUIDv7 with rand_a=0, rand_b=1. The base36 output will be very + // short ("1"), testing that the decoder handles short entropy strings. + var id [16]byte + id[0], id[1], id[2], id[3], id[4], id[5] = 0x01, 0x9D, 0xA4, 0xC1, 0xB8, 0x3B + id[6] = 0x70 // version=7, rand_a=0 + id[7] = 0x00 + id[8] = 0x80 // variant=10, rand_b high=0 + // rand_b = 1 (lowest bit of byte 15) + id[15] = 0x01 + + name := UUIDv7ToDisplayName(id) + recovered, err := DisplayNameToUUIDv7(name) + if err != nil { + t.Fatalf("round-trip failed for small entropy: %v", err) + } + if uuid.UUID(id) != recovered { + t.Fatalf("round-trip mismatch for small entropy:\n orig: %x\n display: %s\n recovered: %x", id, name, recovered) + } + + // The entropy portion should be very short (base36 of a small number) + tsLen := len(DisplayNameLayout) + entropy := name[tsLen+1:] + if len(entropy) > 5 { + t.Errorf("expected short entropy for small value, got %q (len=%d)", entropy, len(entropy)) + } +} + +func TestDisplayNameToUUIDv7_ShortEntropy(t *testing.T) { + // A display name with a single-character entropy "0" should decode correctly + // (represents zero entropy bits) + var id [16]byte + id[0], id[1], id[2], id[3], id[4], id[5] = 0x01, 0x9D, 0xA4, 0xC1, 0xB8, 0x3B + id[6] = 0x70 + id[7] = 0x00 + id[8] = 0x80 + + name := UUIDv7ToDisplayName(id) + recovered, err := DisplayNameToUUIDv7(name) + if err != nil { + t.Fatalf("failed to decode display name with minimal entropy: %v", err) + } + if uuid.UUID(id) != recovered { + t.Fatalf("mismatch:\n orig: %x\n display: %s\n recovered: %x", id, name, recovered) + } +} + +func TestDisplayNameToUUIDv7_HexUUIDIsRejected(t *testing.T) { + // A standard hex UUID string should NOT parse as a display name + hexUUID := "019590a0-1234-7fff-8000-a1b2c3d4e5f6" + _, err := DisplayNameToUUIDv7(hexUUID) + if err == nil { + t.Errorf("DisplayNameToUUIDv7(%q) should have failed for hex UUID input", hexUUID) + } +} + +func TestIsUUIDv7_EdgeCases(t *testing.T) { + tests := []struct { + name string + id [16]byte + expected bool + }{ + { + name: "version 0 (nil-like)", + id: [16]byte{0, 0, 0, 0, 0, 0, 0x00, 0, 0, 0, 0, 0, 0, 0, 0, 0}, + expected: false, + }, + { + name: "version 4", + id: [16]byte{0, 0, 0, 0, 0, 0, 0x40, 0, 0, 0, 0, 0, 0, 0, 0, 0}, + expected: false, + }, + { + name: "version 7 with invalid variant (00)", + id: [16]byte{0, 0, 0, 0, 0, 0, 0x70, 0, 0x00, 0, 0, 0, 0, 0, 0, 0}, + expected: true, // IsUUIDv7 only checks version, not variant + }, + { + name: "version 7 with variant 10 (standard)", + id: [16]byte{0, 0, 0, 0, 0, 0, 0x70, 0, 0x80, 0, 0, 0, 0, 0, 0, 0}, + expected: true, + }, + { + name: "version 1", + id: [16]byte{0, 0, 0, 0, 0, 0, 0x10, 0, 0, 0, 0, 0, 0, 0, 0, 0}, + expected: false, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := IsUUIDv7(tt.id) + if got != tt.expected { + t.Errorf("IsUUIDv7(%x) = %v, want %v", tt.id, got, tt.expected) + } + }) + } +} + +func TestIsDisplayName_EdgeCases(t *testing.T) { + tests := []struct { + name string + input string + expected bool + }{ + {"valid display name", "2026-04-07T143000.123Z-abc", true}, + {"hex UUID", "019590a0-1234-7fff-8000-a1b2c3d4e5f6", false}, + {"old version ID format", "2026-04-07T143000Z", false}, + {"timestamp without entropy", "2026-04-07T143000.123Z", false}, + {"timestamp with wrong separator", "2026-04-07T143000.123Zabc", false}, + {"just long enough (1 char entropy)", "2026-04-07T143000.123Z-0", true}, + {"empty entropy after dash", "2026-04-07T143000.123Z-", false}, // len < tsLen+2 + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := IsDisplayName(tt.input) + if got != tt.expected { + t.Errorf("IsDisplayName(%q) = %v, want %v", tt.input, got, tt.expected) + } + }) + } +} + +func TestUUIDv7ToDisplayName_KnownValue(t *testing.T) { + // Test with a specific known UUID to verify deterministic output. + // Build a UUIDv7 from known bytes. + var id [16]byte + // Timestamp: 2026-04-07 14:30:00.123 UTC = 1775405400123 ms + // 1775405400123 = 0x19D_A4C1_B83B + id[0] = 0x01 + id[1] = 0x9D + id[2] = 0xA4 + id[3] = 0xC1 + id[4] = 0xB8 + id[5] = 0x3B + // Version 7 + id[6] = 0x7F // version=7, rand_a high nibble=F + id[7] = 0xFF // rand_a low byte=FF (rand_a = 0xFFF) + // Variant 10 + id[8] = 0x80 // variant=10, rand_b high 6 bits=000000 + id[9] = 0x00 + id[10] = 0x00 + id[11] = 0x00 + id[12] = 0x00 + id[13] = 0x00 + id[14] = 0x00 + id[15] = 0x01 // rand_b = 1 + + name := UUIDv7ToDisplayName(id) + + // Verify round-trip + recovered, err := DisplayNameToUUIDv7(name) + if err != nil { + t.Fatalf("round-trip failed: %v", err) + } + if uuid.UUID(id) != recovered { + t.Fatalf("round-trip mismatch:\n orig: %x\n display: %s\n recovered: %x", id, name, recovered) + } + + // Verify timestamp portion starts with expected date prefix + ts := ExtractUUIDv7Time(id) + if ts.Year() < 2025 || ts.Year() > 2030 { + t.Errorf("extracted time %v has unexpected year", ts) + } + + // Verify display name starts with the timestamp + expectedPrefix := ts.UTC().Format(DisplayNameLayout) + if name[:len(DisplayNameLayout)] != expectedPrefix { + t.Errorf("display name %q does not start with expected timestamp %q", name, expectedPrefix) + } +} diff --git a/internal/tigerfs/fs/auto_savepoint_test.go b/internal/tigerfs/fs/auto_savepoint_test.go new file mode 100644 index 0000000..f9f9edc --- /dev/null +++ b/internal/tigerfs/fs/auto_savepoint_test.go @@ -0,0 +1,152 @@ +package fs + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/timescale/tigerfs/internal/tigerfs/config" +) + +// --- autoSavepointName tests --- + +func TestSynth_AutoSavepoint_NameWithUserID(t *testing.T) { + ops := &Operations{userID: "agent-7"} + ts := time.Date(2026, 4, 8, 14, 30, 0, 0, time.UTC) + assert.Equal(t, "auto-agent-7-20260408T143000Z", ops.autoSavepointName(ts)) +} + +func TestSynth_AutoSavepoint_NameAnonymous(t *testing.T) { + ops := &Operations{} + ts := time.Date(2026, 4, 8, 14, 30, 0, 0, time.UTC) + assert.Equal(t, "auto-20260408T143000Z", ops.autoSavepointName(ts)) +} + +// --- maybeCreateAutoSavepoint tests --- + +func TestSynth_AutoSavepoint_CreatedOnGap(t *testing.T) { + mockDB := &mockDBClient{ + primaryKeys: map[string]*mockPK{ + "tigerfs.notes_savepoint": {column: "name"}, + }, + lastInsertReturnPK: "auto-test", + } + cfg := &config.Config{ + DirListingLimit: 1000, + AutoSavepointInterval: 5 * time.Minute, + } + ops := NewOperations(cfg, mockDB) + ops.SetUserID("agent-7") + + baseTime := time.Date(2026, 4, 8, 14, 0, 0, 0, time.UTC) + currentTime := baseTime + ops.nowFunc = func() time.Time { return currentTime } + + // Seed the last write time (simulate a previous write) + ops.lastWriteTime = map[string]time.Time{ + "public.notes": baseTime, + } + + // Advance time beyond the interval + currentTime = baseTime.Add(10 * time.Minute) + + ops.maybeCreateAutoSavepoint(context.Background(), "public", "notes") + + // Should have inserted an auto-savepoint + require.True(t, mockDB.insertCalled, "should create auto-savepoint") + require.Len(t, mockDB.insertedRows, 1) + + row := mockDB.insertedRows[0] + colMap := make(map[string]interface{}) + for i, col := range row.columns { + colMap[col] = row.values[i] + } + assert.Equal(t, "auto-agent-7-20260408T141000Z", colMap["name"]) + assert.Equal(t, "agent-7", colMap["user_id"]) + assert.Contains(t, colMap["description"], "inactivity") +} + +func TestSynth_AutoSavepoint_SkippedWithinInterval(t *testing.T) { + mockDB := &mockDBClient{} + cfg := &config.Config{ + DirListingLimit: 1000, + AutoSavepointInterval: 30 * time.Minute, + } + ops := NewOperations(cfg, mockDB) + + baseTime := time.Date(2026, 4, 8, 14, 0, 0, 0, time.UTC) + currentTime := baseTime.Add(5 * time.Minute) // Only 5 min gap + ops.nowFunc = func() time.Time { return currentTime } + + ops.lastWriteTime = map[string]time.Time{ + "public.notes": baseTime, + } + + ops.maybeCreateAutoSavepoint(context.Background(), "public", "notes") + + assert.False(t, mockDB.insertCalled, "should NOT create auto-savepoint within interval") +} + +func TestSynth_AutoSavepoint_DisabledWhenIntervalZero(t *testing.T) { + mockDB := &mockDBClient{} + cfg := &config.Config{ + DirListingLimit: 1000, + AutoSavepointInterval: 0, // disabled + } + ops := NewOperations(cfg, mockDB) + + ops.lastWriteTime = map[string]time.Time{ + "public.notes": time.Now().Add(-1 * time.Hour), + } + + ops.maybeCreateAutoSavepoint(context.Background(), "public", "notes") + + assert.False(t, mockDB.insertCalled, "should NOT create auto-savepoint when interval=0") +} + +func TestSynth_AutoSavepoint_SkippedOnFirstWrite(t *testing.T) { + mockDB := &mockDBClient{} + cfg := &config.Config{ + DirListingLimit: 1000, + AutoSavepointInterval: 5 * time.Minute, + } + ops := NewOperations(cfg, mockDB) + + // No lastWriteTime entry -- first write after mount + ops.maybeCreateAutoSavepoint(context.Background(), "public", "notes") + + assert.False(t, mockDB.insertCalled, "should NOT create auto-savepoint on first write") +} + +func TestSynth_AutoSavepoint_PerTableTracking(t *testing.T) { + mockDB := &mockDBClient{ + primaryKeys: map[string]*mockPK{ + "tigerfs.notes_savepoint": {column: "name"}, + "tigerfs.docs_savepoint": {column: "name"}, + }, + lastInsertReturnPK: "auto-test", + } + cfg := &config.Config{ + DirListingLimit: 1000, + AutoSavepointInterval: 5 * time.Minute, + } + ops := NewOperations(cfg, mockDB) + + baseTime := time.Date(2026, 4, 8, 14, 0, 0, 0, time.UTC) + currentTime := baseTime.Add(10 * time.Minute) + ops.nowFunc = func() time.Time { return currentTime } + + // notes had a recent write, docs had an old write + ops.lastWriteTime = map[string]time.Time{ + "public.notes": baseTime.Add(8 * time.Minute), // 2 min ago -- within interval + "public.docs": baseTime, // 10 min ago -- exceeds interval + } + + ops.maybeCreateAutoSavepoint(context.Background(), "public", "notes") + assert.False(t, mockDB.insertCalled, "notes should NOT trigger (within interval)") + + ops.maybeCreateAutoSavepoint(context.Background(), "public", "docs") + assert.True(t, mockDB.insertCalled, "docs should trigger (exceeds interval)") +} diff --git a/internal/tigerfs/fs/constants.go b/internal/tigerfs/fs/constants.go index 3c13670..095e2c4 100644 --- a/internal/tigerfs/fs/constants.go +++ b/internal/tigerfs/fs/constants.go @@ -7,7 +7,7 @@ package fs // Metadata directory const DirInfo = ".info" -// Metadata files (under .info/ directory) +// Metadata files (under table-level .info/ directory) // Note: These files do NOT have dot prefixes, matching FUSE behavior. const ( FileCount = "count" // Row count @@ -17,6 +17,11 @@ const ( FileIndexes = "indexes" // Index listing ) +// Root-level .info/ files (mount metadata) +const ( + FileUser = "user" // Mount-level user identity for undo log entries +) + // Navigation capabilities // These directories provide different ways to access rows. const ( @@ -67,6 +72,20 @@ const ( DirTables = ".tables" // Backing tables in tigerfs schema ) +// Undo and recovery capabilities (ADR-016) +// Directories for operation log, savepoints, and undo operations. +const ( + DirLog = ".log" // Operation log (data-first on _log table) + DirSavepoint = ".savepoint" // Named savepoints (data-first on _savepoint table) + DirUndo = ".undo" // Undo operations (preview-then-apply) +) + +// Undo control files +const ( + FileApply = ".apply" // Trigger file to execute an undo operation + FileSummary = "summary" // TSV listing of affected files with actions +) + // Control files (DDL staging) // Content files are visible (no dot prefix), trigger files are hidden (dot prefix). const ( @@ -100,17 +119,20 @@ const ( // capabilityDirectories lists all pipeline capability directory names. // Used to prevent these names from being interpreted as column values. var capabilityDirectories = map[string]bool{ - DirBy: true, - DirColumns: true, - DirFilter: true, - DirFirst: true, - DirLast: true, - DirSample: true, - DirAll: true, - DirOrder: true, - DirExport: true, - DirImport: true, - DirInfo: true, + DirBy: true, + DirColumns: true, + DirFilter: true, + DirFirst: true, + DirLast: true, + DirSample: true, + DirAll: true, + DirOrder: true, + DirExport: true, + DirImport: true, + DirInfo: true, + DirLog: true, + DirSavepoint: true, + DirUndo: true, } // IsCapabilityDirectory returns true if name is a reserved capability directory. diff --git a/internal/tigerfs/fs/context.go b/internal/tigerfs/fs/context.go index 87136c9..7fbc148 100644 --- a/internal/tigerfs/fs/context.go +++ b/internal/tigerfs/fs/context.go @@ -87,6 +87,17 @@ type FSContext struct { // IsTerminal indicates this context has reached .export/ and no more // capabilities can be added. IsTerminal bool + + // PipelineDepth tracks the number of pipeline operations applied. + // Used to enforce MaxPipelineDepth and prevent infinite recursion + // from recursive scanners (rm -rf, find, agents). + PipelineDepth int + + // HideCapabilities suppresses pipeline capability directories in ReadDir. + // Used for .log and .savepoint virtual directories inside file-first + // workspaces, where capabilities are accessible by explicit path but + // hidden from listing to prevent recursive scanner blowup. + HideCapabilities bool } // NewFSContext creates a new filesystem context for a table. @@ -163,6 +174,7 @@ func (ctx *FSContext) WithFilter(col, val string, indexed bool) *FSContext { Value: val, Indexed: indexed, }) + clone.PipelineDepth++ return clone } @@ -179,6 +191,7 @@ func (ctx *FSContext) WithOrder(col string, desc bool) *FSContext { clone.OrderBy = col clone.OrderDesc = desc clone.HasOrdered = true + clone.PipelineDepth++ return clone } @@ -202,6 +215,7 @@ func (ctx *FSContext) WithLimit(limit int, limitType LimitType) *FSContext { clone.Limit = limit clone.LimitType = limitType + clone.PipelineDepth++ return clone } @@ -225,6 +239,7 @@ func (ctx *FSContext) WithColumns(columns []string) *FSContext { clone.Columns = make([]string, len(columns)) copy(clone.Columns, columns) clone.HasColumns = true + clone.PipelineDepth++ return clone } diff --git a/internal/tigerfs/fs/history.go b/internal/tigerfs/fs/history.go index b7ef8ce..05fa871 100644 --- a/internal/tigerfs/fs/history.go +++ b/internal/tigerfs/fs/history.go @@ -58,6 +58,13 @@ func (o *Operations) readHistoryFileDispatch(ctx context.Context, parsed *Parsed func (o *Operations) readDirHistory(ctx context.Context, parsed *ParsedPath, info *synth.ViewInfo) ([]Entry, *FSError) { fsCtx := parsed.Context schema := synth.TigerFSSchema + // cacheSchema is the user-facing schema where the live view lives. + // resolveSynthPath uses (cacheSchema, table) as the pathCache key, and + // every other code path keys cache writes/lookups under the user's + // schema, so the history reads must too -- otherwise history's cache + // entries land in a disjoint (tigerfs, table) namespace and get neither + // shared with normal reads nor cleared by normal invalidate calls. + cacheSchema := fsCtx.Schema historyTable := fsCtx.TableName + "_history" now := time.Now() @@ -69,21 +76,56 @@ func (o *Operations) readDirHistory(ctx context.Context, parsed *ParsedPath, inf if parsed.HistoryByID { return o.readDirHistoryByID(ctx, schema, historyTable, parsed, info, now, limit) } - return o.readDirHistoryByFilename(ctx, schema, historyTable, parsed, info, now, limit) + return o.readDirHistoryByFilename(ctx, schema, cacheSchema, historyTable, parsed, info, now, limit) } // readDirHistoryByFilename lists history entries organized by filename. // Uses parsed.PrimaryKey as a directory prefix to show only files at the current level. -func (o *Operations) readDirHistoryByFilename(ctx context.Context, schema, historyTable string, parsed *ParsedPath, info *synth.ViewInfo, now time.Time, limit int) ([]Entry, *FSError) { +// +// schema is the synth schema (tigerfs) where the history table lives; +// cacheSchema is the user's schema, used for pathCache lookups so cache +// keys match live-table reads. +func (o *Operations) readDirHistoryByFilename(ctx context.Context, schema, cacheSchema, historyTable string, parsed *ParsedPath, info *synth.ViewInfo, now time.Time, limit int) ([]Entry, *FSError) { if parsed.HistoryFile == "" { // /{table}/.history/ or /{table}/subdir/.history/ — list filenames at this level - filenames, err := o.db.QueryHistoryDistinctFilenames(ctx, schema, historyTable, limit) + dirPrefix := parsed.PrimaryKey + + var filenames []string + var err error + + // Parent-pointer model (ADR-017): query history by parent_id + if info.Roles.ParentID != "" { + var parentID string + if dirPrefix != "" { + // Resolve directory path in the LIVE table to get its UUID + segments := strings.Split(dirPrefix, "/") + var ok bool + parentID, ok = o.resolveSynthPath(ctx, cacheSchema, parsed.Context.TableName, segments) + if !ok { + return nil, &FSError{Code: ErrNotExist, Message: fmt.Sprintf("directory not found: %s", dirPrefix)} + } + } + filenames, err = o.db.QueryHistoryDistinctFilenamesByParent(ctx, schema, historyTable, parentID, limit) + } else { + // Old model: get all filenames and filter by prefix + filenames, err = o.db.QueryHistoryDistinctFilenames(ctx, schema, historyTable, limit) + } if err != nil { return nil, &FSError{Code: ErrIO, Message: "failed to list history filenames", Cause: err} } - dirPrefix := parsed.PrimaryKey - filtered := filterHistoryAtLevel(filenames, dirPrefix) + var filtered []Entry + if info.Roles.ParentID != "" { + // Parent-pointer: filenames are already scoped to the directory + for _, fn := range filenames { + filtered = append(filtered, Entry{ + Name: fn, IsDir: true, + Mode: os.ModeDir | 0555, ModTime: now, + }) + } + } else { + filtered = filterHistoryAtLevel(filenames, dirPrefix) + } entries := make([]Entry, 0, len(filtered)+1) // Add .by/ entry only at root level (dirPrefix == "") @@ -95,8 +137,7 @@ func (o *Operations) readDirHistoryByFilename(ctx context.Context, schema, histo } // /{table}/.history/foo.md/ — list versions for filename + .id file - // Build full DB filename from directory prefix + local filename - rawFilename := buildHistoryFilename(parsed.PrimaryKey, parsed.HistoryFile) + rawFilename := historyDBFilename(info, parsed.PrimaryKey, parsed.HistoryFile) columns, rows, err := o.db.QueryHistoryByFilename(ctx, schema, historyTable, rawFilename, limit) if err != nil { @@ -107,7 +148,7 @@ func (o *Operations) readDirHistoryByFilename(ctx context.Context, schema, histo // Add .id virtual file entries = append(entries, Entry{Name: ".id", IsDir: false, Mode: 0444, Size: 36, ModTime: now}) - historyIDIdx := columnIndex(columns, "_history_id") + historyIDIdx := columnIndex(columns, "version_id") for _, row := range rows { if historyIDIdx < 0 { continue @@ -155,7 +196,7 @@ func (o *Operations) readDirHistoryByID(ctx context.Context, schema, historyTabl } entries := make([]Entry, 0, len(rows)) - historyIDIdx := columnIndex(columns, "_history_id") + historyIDIdx := columnIndex(columns, "version_id") for _, row := range rows { if historyIDIdx < 0 { continue @@ -205,12 +246,12 @@ func (o *Operations) statHistory(ctx context.Context, parsed *ParsedPath, info * // .history/.by// — version file by UUID if parsed.HistoryByID && parsed.HistoryVersionID != "" { - return o.statHistoryVersion(ctx, schema, historyTable, "id", parsed.HistoryRowID, parsed.HistoryVersionID, info, now) + return o.statHistoryVersion(ctx, schema, historyTable, "file_id", parsed.HistoryRowID, parsed.HistoryVersionID, info, now) } // .history/foo.md/ — filename directory if parsed.HistoryFile != "" && parsed.HistoryVersionID == "" { - rawFilename := buildHistoryFilename(parsed.PrimaryKey, parsed.HistoryFile) + rawFilename := historyDBFilename(info, parsed.PrimaryKey, parsed.HistoryFile) _, rows, err := o.db.QueryHistoryByFilename(ctx, schema, historyTable, rawFilename, 1) if err != nil { return nil, &FSError{Code: ErrIO, Message: "failed to check history by filename", Cause: err} @@ -228,7 +269,7 @@ func (o *Operations) statHistory(ctx context.Context, parsed *ParsedPath, info * // .history/foo.md/ — version file by filename if parsed.HistoryFile != "" && parsed.HistoryVersionID != "" { - rawFilename := buildHistoryFilename(parsed.PrimaryKey, parsed.HistoryFile) + rawFilename := historyDBFilename(info, parsed.PrimaryKey, parsed.HistoryFile) return o.statHistoryVersion(ctx, schema, historyTable, "filename", rawFilename, parsed.HistoryVersionID, info, now) } @@ -264,7 +305,7 @@ func (o *Operations) readHistoryFile(ctx context.Context, parsed *ParsedPath, in // .id file: return the row UUID for this filename if parsed.HistoryFile != "" && parsed.HistoryVersionID == ".id" { - rawFilename := buildHistoryFilename(parsed.PrimaryKey, parsed.HistoryFile) + rawFilename := historyDBFilename(info, parsed.PrimaryKey, parsed.HistoryFile) columns, rows, err := o.db.QueryHistoryByFilename(ctx, schema, historyTable, rawFilename, 1) if err != nil { return nil, &FSError{Code: ErrIO, Message: "failed to query history for .id", Cause: err} @@ -272,9 +313,9 @@ func (o *Operations) readHistoryFile(ctx context.Context, parsed *ParsedPath, in if len(rows) == 0 { return nil, &FSError{Code: ErrNotExist, Message: fmt.Sprintf("no history for %s", parsed.HistoryFile)} } - idIdx := columnIndex(columns, "id") + idIdx := columnIndex(columns, "file_id") if idIdx < 0 { - return nil, &FSError{Code: ErrIO, Message: "id column not found in history table"} + return nil, &FSError{Code: ErrIO, Message: "file_id column not found in history table"} } idStr := synth.ValueToString(rows[0][idIdx]) return []byte(idStr + "\n"), nil @@ -283,11 +324,11 @@ func (o *Operations) readHistoryFile(ctx context.Context, parsed *ParsedPath, in // Version file: read and synthesize content var filterColumn, filterValue string if parsed.HistoryByID { - filterColumn = "id" + filterColumn = "file_id" filterValue = parsed.HistoryRowID } else { filterColumn = "filename" - filterValue = buildHistoryFilename(parsed.PrimaryKey, parsed.HistoryFile) + filterValue = historyDBFilename(info, parsed.PrimaryKey, parsed.HistoryFile) } columns, rows, err := o.db.QueryHistoryVersionByTime(ctx, schema, historyTable, filterColumn, filterValue, parsed.HistoryVersionID, 100) @@ -308,9 +349,9 @@ func (o *Operations) readHistoryFile(ctx context.Context, parsed *ParsedPath, in return content, nil } -// findVersionRow scans rows for one whose _history_id matches the given versionID. +// findVersionRow scans rows for one whose version_id matches the given versionID. func findVersionRow(columns []string, rows [][]interface{}, versionID string) []interface{} { - historyIDIdx := columnIndex(columns, "_history_id") + historyIDIdx := columnIndex(columns, "version_id") if historyIDIdx < 0 { return nil } @@ -333,7 +374,7 @@ func columnIndex(columns []string, name string) int { return -1 } -// historyIDToVersionID converts a _history_id value (UUIDv7) to a version ID string. +// historyIDToVersionID converts a version_id value (UUIDv7) to a display version ID string. func historyIDToVersionID(val interface{}) string { // The value may be a [16]byte, uuid.UUID, or string switch v := val.(type) { @@ -385,6 +426,21 @@ func parseUUID(s string) ([16]byte, error) { // buildHistoryFilename constructs the full DB filename from a directory prefix and local name. // At root level (dirPrefix=""), returns just the localName. // In subdirectories, returns "dirPrefix/localName". +// historyDBFilename returns the filename to use for history table queries. +// For parent-pointer model: extracts the leaf name (history stores leaf names). +// The greedy history path parser may produce multi-segment localName like +// "inbox/task.md" -- only the leaf "task.md" is in the history table. +// For old model: builds full path from directory prefix + local filename. +func historyDBFilename(info *synth.ViewInfo, dirPrefix, localName string) string { + if info.Roles.ParentID != "" { + if idx := strings.LastIndex(localName, "/"); idx >= 0 { + return localName[idx+1:] + } + return localName + } + return buildHistoryFilename(dirPrefix, localName) +} + func buildHistoryFilename(dirPrefix, localName string) string { if dirPrefix == "" { return localName diff --git a/internal/tigerfs/fs/log_interface_test.go b/internal/tigerfs/fs/log_interface_test.go new file mode 100644 index 0000000..015ecd1 --- /dev/null +++ b/internal/tigerfs/fs/log_interface_test.go @@ -0,0 +1,257 @@ +package fs + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/timescale/tigerfs/internal/tigerfs/config" + "github.com/timescale/tigerfs/internal/tigerfs/fs/synth" +) + +// --- synthUndoDirs tests --- + +func TestSynthUndoDirs_WithHistory(t *testing.T) { + info := &synth.ViewInfo{HasHistory: true, CachedMountTime: time.Now()} + dirs := synthUndoDirs(info) + require.Len(t, dirs, 4) + assert.Equal(t, ".history", dirs[0].Name) + assert.Equal(t, ".log", dirs[1].Name) + assert.Equal(t, ".savepoint", dirs[2].Name) + assert.Equal(t, ".undo", dirs[3].Name) + for _, d := range dirs { + assert.True(t, d.IsDir) + } +} + +func TestSynthUndoDirs_WithoutHistory(t *testing.T) { + info := &synth.ViewInfo{HasHistory: false} + dirs := synthUndoDirs(info) + assert.Nil(t, dirs) +} + +// --- uuidToDisplayName tests --- + +func TestUuidToDisplayName_ValidV7Hex(t *testing.T) { + cfg := &config.Config{} + ops := NewOperations(cfg, &mockDBClient{}) + // Use a known UUIDv7 hex -- we just need to verify it converts + // The format package handles the actual conversion + result := ops.uuidToDisplayName("019d7db2-237b-77ab-949a-afe464991e0e") + assert.Contains(t, result, "2026-", "should convert to display name with timestamp") + assert.Contains(t, result, "Z-", "should have Z- separator between timestamp and entropy") +} + +func TestUuidToDisplayName_AlreadyDisplayName(t *testing.T) { + cfg := &config.Config{} + ops := NewOperations(cfg, &mockDBClient{}) + result := ops.uuidToDisplayName("2026-04-07T143000.123Z-zzz0063hd8e5r42") + assert.Equal(t, "2026-04-07T143000.123Z-zzz0063hd8e5r42", result, "should pass through display names") +} + +func TestUuidToDisplayName_NonUUID(t *testing.T) { + cfg := &config.Config{} + ops := NewOperations(cfg, &mockDBClient{}) + result := ops.uuidToDisplayName("not-a-uuid") + assert.Equal(t, "not-a-uuid", result, "should pass through non-UUIDs") +} + +// --- parseUUIDBytes tests --- + +func TestParseUUIDBytes_Valid(t *testing.T) { + b, err := parseUUIDBytes("019d7db2-237b-77ab-949a-afe464991e0e") + require.NoError(t, err) + assert.Len(t, b, 16) +} + +func TestParseUUIDBytes_InvalidLength(t *testing.T) { + _, err := parseUUIDBytes("too-short") + assert.Error(t, err) +} + +func TestParseUUIDBytes_NoDashes(t *testing.T) { + b, err := parseUUIDBytes("019d7db2237b77ab949aafe464991e0e") + require.NoError(t, err) + assert.Len(t, b, 16) +} + +// --- resolveLogDiffSymlink full state matrix (mock-based) --- + +// Helper to create a mock with a log entry row and configurable next/exists results. +func setupLogDiffMock(versionID, fileID, filename, logID string) *mockDBClient { + mock := &mockDBClient{ + primaryKeys: map[string]*mockPK{ + "tigerfs.test_log": {column: "log_id"}, + }, + rowData: map[string]*mockRow{ + "tigerfs.test_log." + logID: { + columns: []string{"log_id", "file_id", "type", "user_id", "filename", "version_id", "description"}, + values: []interface{}{logID, fileID, "edit", nil, filename, nilIfEmpty(versionID), nil}, + }, + }, + } + return mock +} + +func nilIfEmpty(s string) interface{} { + if s == "" { + return nil + } + return s +} + +func TestResolveLogDiffSymlink_Before_NullVersionID(t *testing.T) { + mock := setupLogDiffMock("", "file-1", "hello.md", "log-1") + cfg := &config.Config{} + ops := NewOperations(cfg, mock) + + parsed := &ParsedPath{ + Type: PathColumn, + Context: &FSContext{Schema: "tigerfs", TableName: "test_log"}, + PrimaryKey: "log-1", + Column: "before", + OrigTableName: "test", + } + + target, fsErr := ops.resolveLogDiffSymlink(context.Background(), parsed) + require.Nil(t, fsErr) + assert.Equal(t, "/dev/null", target) +} + +func TestResolveLogDiffSymlink_Before_WithVersionID(t *testing.T) { + mock := setupLogDiffMock("019d7db2-237b-77ab-949a-afe464991e0e", "file-1", "hello.md", "log-1") + cfg := &config.Config{} + ops := NewOperations(cfg, mock) + + parsed := &ParsedPath{ + Type: PathColumn, + Context: &FSContext{Schema: "tigerfs", TableName: "test_log"}, + PrimaryKey: "log-1", + Column: "before", + OrigTableName: "test", + } + + target, fsErr := ops.resolveLogDiffSymlink(context.Background(), parsed) + require.Nil(t, fsErr) + assert.Contains(t, target, "../../.history/hello.md/") + assert.NotEqual(t, "/dev/null", target) +} + +func TestResolveLogDiffSymlink_After_NextEntryWithVersionID(t *testing.T) { + mock := setupLogDiffMock("v-1", "file-1", "hello.md", "log-1") + mock.nextLogVersionID = "019d7db2-337b-77ab-949a-afe464991e0e" + mock.nextLogFilename = "hello.md" + cfg := &config.Config{} + ops := NewOperations(cfg, mock) + + parsed := &ParsedPath{ + Type: PathColumn, + Context: &FSContext{Schema: "tigerfs", TableName: "test_log"}, + PrimaryKey: "log-1", + Column: "after", + OrigTableName: "test", + } + + target, fsErr := ops.resolveLogDiffSymlink(context.Background(), parsed) + require.Nil(t, fsErr) + assert.Contains(t, target, "../../.history/hello.md/") +} + +func TestResolveLogDiffSymlink_After_NextEntryNullVersionID(t *testing.T) { + mock := setupLogDiffMock("v-1", "file-1", "hello.md", "log-1") + mock.nextLogVersionID = "" // NULL -- next op was a create + mock.nextLogFilename = "hello.md" + cfg := &config.Config{} + ops := NewOperations(cfg, mock) + + parsed := &ParsedPath{ + Type: PathColumn, + Context: &FSContext{Schema: "tigerfs", TableName: "test_log"}, + PrimaryKey: "log-1", + Column: "after", + OrigTableName: "test", + } + + target, fsErr := ops.resolveLogDiffSymlink(context.Background(), parsed) + require.Nil(t, fsErr) + assert.Equal(t, "../../hello.md", target) +} + +func TestResolveLogDiffSymlink_After_NoNextEntry_FileExists(t *testing.T) { + mock := setupLogDiffMock("v-1", "file-1", "hello.md", "log-1") + // No next entry (defaults empty) + mock.fileExistsResult = true + cfg := &config.Config{} + ops := NewOperations(cfg, mock) + + parsed := &ParsedPath{ + Type: PathColumn, + Context: &FSContext{Schema: "tigerfs", TableName: "test_log"}, + PrimaryKey: "log-1", + Column: "after", + OrigTableName: "test", + } + + target, fsErr := ops.resolveLogDiffSymlink(context.Background(), parsed) + require.Nil(t, fsErr) + assert.Equal(t, "../../hello.md", target) +} + +func TestResolveLogDiffSymlink_After_NoNextEntry_FileDeleted(t *testing.T) { + mock := setupLogDiffMock("v-1", "file-1", "hello.md", "log-1") + mock.fileExistsResult = false + cfg := &config.Config{} + ops := NewOperations(cfg, mock) + + parsed := &ParsedPath{ + Type: PathColumn, + Context: &FSContext{Schema: "tigerfs", TableName: "test_log"}, + PrimaryKey: "log-1", + Column: "after", + OrigTableName: "test", + } + + target, fsErr := ops.resolveLogDiffSymlink(context.Background(), parsed) + require.Nil(t, fsErr) + assert.Equal(t, "/dev/null", target) +} + +func TestResolveLogDiffSymlink_Current_FileExists(t *testing.T) { + mock := setupLogDiffMock("v-1", "file-1", "hello.md", "log-1") + mock.fileExistsResult = true + cfg := &config.Config{} + ops := NewOperations(cfg, mock) + + parsed := &ParsedPath{ + Type: PathColumn, + Context: &FSContext{Schema: "tigerfs", TableName: "test_log"}, + PrimaryKey: "log-1", + Column: "current", + OrigTableName: "test", + } + + target, fsErr := ops.resolveLogDiffSymlink(context.Background(), parsed) + require.Nil(t, fsErr) + assert.Equal(t, "../../hello.md", target) +} + +func TestResolveLogDiffSymlink_Current_FileDeleted(t *testing.T) { + mock := setupLogDiffMock("v-1", "file-1", "hello.md", "log-1") + mock.fileExistsResult = false + cfg := &config.Config{} + ops := NewOperations(cfg, mock) + + parsed := &ParsedPath{ + Type: PathColumn, + Context: &FSContext{Schema: "tigerfs", TableName: "test_log"}, + PrimaryKey: "log-1", + Column: "current", + OrigTableName: "test", + } + + target, fsErr := ops.resolveLogDiffSymlink(context.Background(), parsed) + require.Nil(t, fsErr) + assert.Equal(t, "/dev/null", target) +} diff --git a/internal/tigerfs/fs/operations.go b/internal/tigerfs/fs/operations.go index c51d43e..a03d6c7 100644 --- a/internal/tigerfs/fs/operations.go +++ b/internal/tigerfs/fs/operations.go @@ -43,8 +43,37 @@ type Operations struct { // Invalidated by write operations; 2-second TTL as safety net. statCache statCache + // pathCache maps (parentID, filename) -> row ID for the parent-pointer + // directory model (ADR-017). Used to avoid calling resolve_path for + // repeated access to the same directory subtree. 2-second TTL. + pathCache pathCache + + // mountPoint is the filesystem mount path (e.g., "/mnt/tigerfs"). + // Used for user-facing log messages. Empty if not set. + mountPoint string + + // userID is the current mount-level identity for undo log entries. + // Set from --user-id flag or TIGERFS_USER_ID env at mount time. + // Can be changed at runtime via writing to .info/user. + // Empty means anonymous (user_id = NULL in log entries). + userID string + userIDModTime time.Time // stable mtime for .info/user Stat responses + // legacyWarnOnce ensures the legacy backing table warning is logged only once. legacyWarnOnce sync.Once + + // Auto-savepoints: track last write time per app to detect inactivity gaps. + // Key is "schema.tableName" (the source table, not _log or _savepoint). + lastWriteTime map[string]time.Time + lastWriteMu sync.Mutex + + // nowFunc returns the current time. Overridable in tests for deterministic timing. + nowFunc func() time.Time + + // undoCache provides short-lived caching for undo preview queries. + // Reduces redundant DB queries when multiple NFS/FUSE RPCs access + // the same undo target within a 2-second window. + undoCache undoCache } // NewOperations creates a new Operations instance. @@ -57,6 +86,27 @@ func NewOperations(cfg *config.Config, dbClient db.DBClient) *Operations { } } +// SetMountPoint records the filesystem mount path for user-facing log messages. +func (o *Operations) SetMountPoint(path string) { + o.mountPoint = path +} + +// SetNowFunc overrides the clock for testing. Pass nil to restore real time. +func (o *Operations) SetNowFunc(fn func() time.Time) { + o.nowFunc = fn +} + +// SetUserID sets the mount-level user identity for undo log entries. +func (o *Operations) SetUserID(id string) { + o.userID = id + o.userIDModTime = time.Now() +} + +// GetUserID returns the current mount-level user identity. +func (o *Operations) GetUserID() string { + return o.userID +} + // statCache caches Entry metadata from ReadDir results. // Stat checks this before querying the DB. ReadFile bypasses it (always fresh). // Invalidated by write operations; 2-second TTL as safety net. @@ -297,6 +347,16 @@ func (o *Operations) readDirWithParsed(ctx context.Context, parsed *ParsedPath) return o.readDirViews(ctx) case PathTable: return o.readDirTable(ctx, parsed) + case PathLog: + // Data-first pipeline on log table + return o.readDirTable(ctx, parsed) + case PathSavepoint: + // With name as PK, readDirTable naturally lists by human-readable name. + return o.readDirTable(ctx, parsed) + case PathUndo: + return o.readDirUndo(ctx, parsed) + case PathRootInfo: + return o.readDirRootInfo(ctx, parsed) case PathInfo: return o.readDirInfo(ctx, parsed) case PathRow: @@ -372,6 +432,7 @@ func (o *Operations) readDirRoot(ctx context.Context) ([]Entry, *FSError) { entries = append(entries, Entry{Name: ".build", IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, Entry{Name: ".create", IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, + Entry{Name: DirInfo, IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, Entry{Name: ".schemas", IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, Entry{Name: ".tables", IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, Entry{Name: ".views", IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, @@ -554,7 +615,7 @@ func (o *Operations) readDirTable(ctx context.Context, parsed *ParsedPath) ([]En // Default limit from config limit := o.config.DirListingLimit if limit <= 0 { - limit = 10000 + limit = 1000 } var rows []string @@ -595,8 +656,17 @@ func (o *Operations) readDirTable(ctx context.Context, parsed *ParsedPath) ([]En // Build entries: capability directories first, then rows entries := make([]Entry, 0, len(rows)+15) + // Check if pipeline depth exceeds the configured maximum. + // When exceeded, hide capability directories to prevent infinite recursion + // from recursive scanners (rm -rf, find, agents). Rows are still listed. + maxDepth := o.config.MaxPipelineDepth + depthExceeded := maxDepth > 0 && fsCtx.PipelineDepth >= maxDepth + // Add capability directories based on what's available in current context - if fsCtx.HasPipelineOperations() { + if fsCtx.HideCapabilities || depthExceeded { + // Depth exceeded: only show .info for metadata, no further pipeline capabilities + entries = append(entries, Entry{Name: DirInfo, IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}) + } else if fsCtx.HasPipelineOperations() { // Use available capabilities based on pipeline state for _, cap := range fsCtx.AvailableCapabilities() { entries = append(entries, Entry{Name: cap, IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}) @@ -604,9 +674,11 @@ func (o *Operations) readDirTable(ctx context.Context, parsed *ParsedPath) ([]En // Always include .info for metadata access entries = append(entries, Entry{Name: DirInfo, IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}) } else { - // Show all capabilities for raw table access + // Show all capabilities for raw table access. + // Note: DirAll is omitted because it's a no-op (same as the table listing) + // and creates infinite recursion for recursive scanners (rm -rf, find, agents). capabilities := []string{ - DirAll, DirBy, DirColumns, DirDelete, DirExport, DirFilter, DirFirst, + DirBy, DirColumns, DirDelete, DirExport, DirFilter, DirFirst, DirFormat, DirImport, DirIndexes, DirInfo, DirLast, DirModify, DirOrder, DirSample, } for _, cap := range capabilities { @@ -718,16 +790,259 @@ func (o *Operations) readDirRow(ctx context.Context, parsed *ParsedPath) ([]Entr cacheEntries[filename] = entry } + // Include format entries in cache so Stat works for .json, .tsv, .csv, .yaml + for _, e := range entries { + if strings.HasPrefix(e.Name, ".") { + cacheEntries[e.Name] = e + } + } + // Prime row-level stat cache so statColumn can use it if len(cacheEntries) > 0 { o.statCache.prime(fsCtx.Schema, fsCtx.TableName+"/"+parsed.PrimaryKey, cacheEntries) } + // Add diff symlinks for log entry rows (before/after/current) + if parsed.OrigTableName != "" && strings.HasSuffix(fsCtx.TableName, "_log") { + entries = append(entries, + Entry{Name: "before", IsDir: false, Mode: os.ModeSymlink | 0777, ModTime: now}, + Entry{Name: "after", IsDir: false, Mode: os.ModeSymlink | 0777, ModTime: now}, + Entry{Name: "current", IsDir: false, Mode: os.ModeSymlink | 0777, ModTime: now}, + ) + } + return entries, nil } +// resolveLogDiffSymlink resolves a before/after/current symlink on a log entry. +// The symlink targets are relative paths from .log// to the app root (../../). +func (o *Operations) resolveLogDiffSymlink(ctx context.Context, parsed *ParsedPath) (string, *FSError) { + fsCtx := parsed.Context + logTable := fsCtx.TableName + appName := parsed.OrigTableName + + // Fetch the log entry to get version_id, file_id, filename + pk, pkErr := o.metaCache.GetPrimaryKey(ctx, fsCtx.Schema, logTable) + if pkErr != nil { + return "", &FSError{Code: ErrIO, Message: "failed to get log table PK", Cause: pkErr} + } + match, decodeErr := pk.Decode(parsed.PrimaryKey) + if decodeErr != nil { + return "", &FSError{Code: ErrInvalidArgument, Message: fmt.Sprintf("invalid log_id: %v", decodeErr)} + } + row, rowErr := o.db.GetRow(ctx, synth.TigerFSSchema, logTable, match) + if rowErr != nil { + return "", &FSError{Code: ErrIO, Message: "failed to fetch log entry", Cause: rowErr} + } + if row == nil { + return "", &FSError{Code: ErrNotExist, Message: "log entry not found"} + } + + // Extract fields from the row + rowMap := make(map[string]interface{}, len(row.Columns)) + for i, col := range row.Columns { + rowMap[col] = row.Values[i] + } + + versionID, _ := format.ConvertValueToText(rowMap["version_id"]) + fileID, _ := format.ConvertValueToText(rowMap["file_id"]) + filename, _ := format.ConvertValueToText(rowMap["filename"]) + logID, _ := format.ConvertValueToText(rowMap["log_id"]) + + switch parsed.Column { + case "before": + if versionID == "" { + return "/dev/null", nil + } + // Convert version_id UUID to display name for .history/ path. + // .history/ is per-directory: tutorials/foo.md → ../../tutorials/.history/foo.md/ + displayName := o.uuidToDisplayName(versionID) + return "../../" + historySymlinkPath(filename, displayName), nil + + case "after": + // Find next log entry for this file + nextVersionID, nextFilename, err := o.db.QueryNextLogEntry(ctx, synth.TigerFSSchema, logTable, fileID, logID) + if err != nil { + return "", &FSError{Code: ErrIO, Message: "failed to query next log entry", Cause: err} + } + if nextVersionID != "" { + // Next entry has a version_id → point to that history version + displayName := o.uuidToDisplayName(nextVersionID) + fn := nextFilename + if fn == "" { + fn = filename + } + return "../../" + historySymlinkPath(fn, displayName), nil + } + if nextFilename != "" { + // Next entry exists but version_id is NULL (was a create) → current file + return "../../" + nextFilename, nil + } + // No next entry → file is either current or deleted + exists, _ := o.db.QueryFileExists(ctx, synth.TigerFSSchema, appName, fileID) + if exists { + return "../../" + filename, nil + } + return "/dev/null", nil + + case "current": + exists, _ := o.db.QueryFileExists(ctx, synth.TigerFSSchema, appName, fileID) + if exists { + return "../../" + filename, nil + } + return "/dev/null", nil + } + + return "", &FSError{Code: ErrInvalidPath, Message: fmt.Sprintf("unknown symlink: %s", parsed.Column)} +} + +// uuidToDisplayName converts a hex UUID string to a UUIDv7 display name. +// Returns the original string if it's not a valid UUIDv7. +func (o *Operations) uuidToDisplayName(hexUUID string) string { + _, err := format.DisplayNameToUUIDv7(hexUUID) + if err == nil { + // It's already a display name + return hexUUID + } + // Try to parse as hex UUID + if len(hexUUID) == 36 { + var id [16]byte + b, parseErr := parseUUIDBytes(hexUUID) + if parseErr == nil { + copy(id[:], b) + if format.IsUUIDv7(id) { + return format.UUIDv7ToDisplayName(id) + } + } + } + return hexUUID +} + +// parseUUIDBytes parses a hex UUID string to bytes. +func parseUUIDBytes(s string) ([]byte, error) { + s = strings.ReplaceAll(s, "-", "") + if len(s) != 32 { + return nil, fmt.Errorf("invalid UUID length: %d", len(s)) + } + b := make([]byte, 16) + for i := 0; i < 16; i++ { + _, err := fmt.Sscanf(s[i*2:i*2+2], "%02x", &b[i]) + if err != nil { + return nil, err + } + } + return b, nil +} + +// historySymlinkPath constructs the correct .history/ path for a file. +// .history/ is per-directory, so "tutorials/foo.md" becomes "tutorials/.history/foo.md/" +// and root-level "hello.md" becomes ".history/hello.md/". +func historySymlinkPath(filename, version string) string { + if idx := strings.LastIndex(filename, "/"); idx >= 0 { + dir := filename[:idx] + leaf := filename[idx+1:] + return dir + "/.history/" + leaf + "/" + version + } + return ".history/" + filename + "/" + version +} + +// writeSavepoint injects user_id from mount identity then delegates to writeRowFile. +// The user can override user_id by including it in the TSV/JSON/CSV body. +func (o *Operations) writeSavepoint(ctx context.Context, parsed *ParsedPath, data []byte) *FSError { + if o.userID != "" && (parsed.Format == "tsv" || parsed.Format == "csv") { + // Inject user_id into TSV/CSV data if not already present. + // Format: first line is headers, second line is values. + sep := "\t" + if parsed.Format == "csv" { + sep = "," + } + trimmed := strings.TrimRight(string(data), "\n") + lines := strings.SplitN(trimmed, "\n", 2) + if len(lines) == 2 && !strings.Contains(lines[0], "user_id") { + lines[0] += sep + "user_id" + lines[1] += sep + o.userID + data = []byte(lines[0] + "\n" + lines[1] + "\n") + } else if trimmed == "" || len(lines) < 2 { + // Empty body -- create minimal TSV with just user_id + data = []byte("user_id\n" + o.userID + "\n") + } + } else if o.userID != "" && parsed.Format == "json" { + // Inject user_id into JSON data if not already present. + s := strings.TrimSpace(string(data)) + if s == "" { + data = []byte(`{"user_id":"` + o.userID + `"}`) + } else if strings.HasPrefix(s, "{") && !strings.Contains(s, "\"user_id\"") { + if s == "{}" { + data = []byte(`{"user_id":"` + o.userID + `"}`) + } else { + s = s[:len(s)-1] + ",\"user_id\":\"" + o.userID + "\"}" + data = []byte(s) + } + } + } else if o.userID != "" && parsed.Format == "yaml" { + // Inject user_id into YAML data if not already present. + s := strings.TrimSpace(string(data)) + if s == "" { + data = []byte("user_id: " + o.userID + "\n") + } else if !strings.Contains(s, "user_id:") { + data = []byte(s + "\nuser_id: " + o.userID + "\n") + } + } + // Delegate to standard row write (handles INSERT/UPDATE, PK merge, format check) + return o.writeRowFile(ctx, parsed, data) +} + // readDirInfo lists the .info metadata directory. // Files match FUSE behavior: count, ddl, schema, columns, indexes (no dot prefix). +// readDirRootInfo lists the root-level .info/ directory (mount metadata). +func (o *Operations) readDirRootInfo(ctx context.Context, parsed *ParsedPath) ([]Entry, *FSError) { + modTime := o.userIDModTime + if modTime.IsZero() { + modTime = time.Now() + } + entries := []Entry{ + {Name: FileUser, IsDir: false, Mode: 0644, Size: int64(len(o.userID) + 1), ModTime: modTime}, + } + return entries, nil +} + +// statRootInfo returns metadata for root-level .info/ paths. +func (o *Operations) statRootInfo(parsed *ParsedPath, now time.Time) (*Entry, *FSError) { + if parsed.InfoFile == "" { + return &Entry{Name: DirInfo, IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, nil + } + if parsed.InfoFile == FileUser { + modTime := o.userIDModTime + if modTime.IsZero() { + modTime = now + } + content := o.userID + "\n" + return &Entry{Name: FileUser, IsDir: false, Mode: 0644, Size: int64(len(content)), ModTime: modTime}, nil + } + return nil, &FSError{Code: ErrNotExist, Message: fmt.Sprintf("unknown info file: %s", parsed.InfoFile)} +} + +// readRootInfoFile reads a root-level .info/ file. +func (o *Operations) readRootInfoFile(parsed *ParsedPath) (*FileContent, *FSError) { + if parsed.InfoFile == FileUser { + return &FileContent{Data: []byte(o.userID + "\n")}, nil + } + return nil, &FSError{Code: ErrNotExist, Message: fmt.Sprintf("unknown info file: %s", parsed.InfoFile)} +} + +// writeRootInfoFile writes a root-level .info/ file. +// Writes to unknown files (e.g., editor temp files like #user#) are silently +// ignored -- they're ephemeral artifacts that don't persist in a virtual filesystem. +func (o *Operations) writeRootInfoFile(parsed *ParsedPath, data []byte) *FSError { + if parsed.InfoFile == FileUser { + o.userID = strings.TrimSpace(string(data)) + o.userIDModTime = time.Now() + return nil + } + return nil // ignore unknown files (editor temp files, etc.) +} + +// readDirInfo lists the table-level .info/ metadata directory. func (o *Operations) readDirInfo(ctx context.Context, parsed *ParsedPath) ([]Entry, *FSError) { now := time.Now() entries := []Entry{ @@ -797,7 +1112,7 @@ func (o *Operations) readDirByCapability(ctx context.Context, parsed *ParsedPath // List distinct values for the column (use DirListingLimit for .by/) limit := o.config.DirListingLimit if limit <= 0 { - limit = 10000 + limit = 1000 } return o.readDistinctColumnValues(ctx, fsCtx, parsed.CapabilityArg, limit) } @@ -862,7 +1177,7 @@ func (o *Operations) readDirFilterCapability(ctx context.Context, parsed *Parsed // List distinct values for the column (use DirFilterLimit for .filter/) limit := o.config.DirFilterLimit if limit <= 0 { - limit = 100000 + limit = 10000 } return o.readDistinctColumnValues(ctx, fsCtx, parsed.CapabilityArg, limit) } @@ -935,15 +1250,11 @@ func (o *Operations) readDirOrderCapability(ctx context.Context, parsed *ParsedP } // readDirPaginationCapability lists options for .first/, .last/, .sample/. +// Returns an empty listing to prevent recursive scanners (rm -rf, find, agents) +// from descending into multiple limit values and triggering parallel queries. +// Users navigate directly via .first/50, .last/100, etc. func (o *Operations) readDirPaginationCapability(ctx context.Context, parsed *ParsedPath) ([]Entry, *FSError) { - now := time.Now() - // Show common limit values - limits := []string{"10", "25", "50", "100", "500", "1000"} - entries := make([]Entry, len(limits)) - for i, l := range limits { - entries[i] = Entry{Name: l, IsDir: true, Mode: os.ModeDir | 0755, ModTime: now} - } - return entries, nil + return []Entry{}, nil } // readDirIndexesCapability lists indexes for /{table}/.indexes/. @@ -1194,6 +1505,24 @@ func (o *Operations) Stat(ctx context.Context, path string) (*Entry, *FSError) { return o.statWithParsed(ctx, parsed, path) } +// Readlink returns the target path of a symlink. Returns ErrInvalidOperation +// if the path is not a symlink. This is a stub that will be wired to specific +// path handlers (e.g., .log/ diff symlinks, .undo/ preview symlinks) in later tasks. +func (o *Operations) Readlink(ctx context.Context, path string) (string, *FSError) { + // First, stat the path to check if it's a symlink + entry, err := o.Stat(ctx, path) + if err != nil { + return "", err + } + if !entry.IsSymlink() { + return "", &FSError{ + Code: ErrInvalidOperation, + Message: "not a symlink", + } + } + return entry.Target, nil +} + // StatWithContext returns metadata using a pre-parsed context. func (o *Operations) StatWithContext(ctx context.Context, fsCtx *FSContext) (*Entry, *FSError) { parsed := &ParsedPath{ @@ -1214,6 +1543,18 @@ func (o *Operations) statWithParsed(ctx context.Context, parsed *ParsedPath, ori case PathRoot: return &Entry{Name: "", IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, nil + case PathRootInfo: + return o.statRootInfo(parsed, now) + + case PathLog: + return &Entry{Name: DirLog, IsDir: true, Mode: os.ModeDir | 0555, ModTime: now}, nil + + case PathSavepoint: + return &Entry{Name: DirSavepoint, IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, nil + + case PathUndo: + return o.statUndo(ctx, parsed) + case PathSchemaList: return &Entry{Name: ".schemas", IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, nil @@ -1463,6 +1804,24 @@ func (o *Operations) statColumn(ctx context.Context, parsed *ParsedPath) (*Entry } } + // Diff symlinks on log entry rows (before/after/current) + if parsed.OrigTableName != "" && strings.HasSuffix(fsCtx.TableName, "_log") { + switch parsed.Column { + case "before", "after", "current": + target, fsErr := o.resolveLogDiffSymlink(ctx, parsed) + if fsErr != nil { + return nil, fsErr + } + return &Entry{ + Name: parsed.Column, + IsDir: false, + Mode: os.ModeSymlink | 0777, + Target: target, + ModTime: time.Now(), + }, nil + } + } + // Check readDirRow-primed cache for this column file. // This avoids per-column DB queries during ls -l in row directories. if entry, ok := o.statCache.lookup(fsCtx.Schema, fsCtx.TableName+"/"+parsed.PrimaryKey, parsed.Column); ok { @@ -1888,6 +2247,8 @@ func (o *Operations) readFileWithParsed(ctx context.Context, parsed *ParsedPath) return o.readRowFile(ctx, parsed) case PathColumn: return o.readColumnFile(ctx, parsed) + case PathRootInfo: + return o.readRootInfoFile(parsed) case PathInfo: return o.readInfoFile(ctx, parsed) case PathExport: @@ -1904,6 +2265,8 @@ func (o *Operations) readFileWithParsed(ctx context.Context, parsed *ParsedPath) return &FileContent{Data: []byte{}}, nil case PathHistory: return o.readHistoryFileDispatch(ctx, parsed) + case PathUndo: + return o.readFileUndo(ctx, parsed) default: return nil, &FSError{ Code: ErrInvalidPath, @@ -2173,7 +2536,7 @@ func (o *Operations) readExportFile(ctx context.Context, parsed *ParsedPath) (*F // Default limit from config limit := o.config.DirListingLimit if limit <= 0 { - limit = 10000 + limit = 1000 } // Validate column names if column projection is specified diff --git a/internal/tigerfs/fs/operations_test.go b/internal/tigerfs/fs/operations_test.go index 0e85250..e55c723 100644 --- a/internal/tigerfs/fs/operations_test.go +++ b/internal/tigerfs/fs/operations_test.go @@ -564,7 +564,9 @@ func TestReadDir_OrderCapability(t *testing.T) { assert.Contains(t, names, "name.desc") } -// TestReadDir_PaginationCapability tests listing options in .first/. +// TestReadDir_PaginationCapability tests that .first/ returns empty listing. +// Pagination directories don't list numeric subdirs to prevent recursive scanners +// (rm -rf, find, agents) from descending into multiple limit values. func TestReadDir_PaginationCapability(t *testing.T) { cfg := &config.Config{} mockDB := &mockDBClient{ @@ -580,14 +582,7 @@ func TestReadDir_PaginationCapability(t *testing.T) { entries, err := ops.ReadDir(context.Background(), "/users/.first") require.Nil(t, err) - require.NotNil(t, entries) - - names := make([]string, len(entries)) - for i, e := range entries { - names[i] = e.Name - } - assert.Contains(t, names, "10") - assert.Contains(t, names, "100") + assert.Empty(t, entries, ".first/ should return empty listing") } // TestReadDir_ExportDirectory tests listing the .export directory. @@ -1086,6 +1081,45 @@ type mockDBClient struct { execInTxSuccess bool execInTxError error + // Log entry tracking + logEntries []mockLogEntry + + // Insert return value (PK of inserted row) + lastInsertReturnPK string + + // Tracks all inserts for verification + insertedRows []mockInsertedRow + + // Version ID return value for QueryLatestVersionID + latestVersionIDs map[string]string // fileID -> versionID + + // Row-by-columns lookup data (for savepoint name-based tests) + rowByColumnsData map[string]*mockRowByColumns + + // NextLogEntry return values (for diff symlink tests) + nextLogVersionID string + nextLogFilename string + fileExistsResult bool + + // Undo test tracking + undoAffectedFiles []db.UndoAffectedFile + undoLogEntry *db.UndoAffectedFile + undoTransactionCalls int + lastUndoParams *db.UndoTransactionParams + fileExistsMap map[string]bool // fileID -> exists (overrides fileExistsResult) + + // ResolvePath tracking + resolvePathResults []db.PathSegment + resolvePathErr error + resolvePathCalls int + resolvePathCallCount int // separate counter for test-specific resolve functions + resolvePathFn func(ctx context.Context, schema, table string, segments []string) ([]db.PathSegment, error) + lastResolveStartParent string + lastResolveSegments []string + + // Rename-as-replace tracking + deleteAndUpdateFunc func(ctx context.Context, schema, table string, deletePK *db.PKMatch, updatePK *db.PKMatch, updateCols []string, updateVals []interface{}) error + // PK match tracking for composite PK tests lastPKMatch *db.PKMatch @@ -1097,6 +1131,26 @@ type mockDBClient struct { getFullDDLCalls int } +type mockLogEntry struct { + userID string + opType string + fileID string + filename string + versionID string +} + +type mockRowByColumns struct { + columns []string + values []interface{} +} + +type mockInsertedRow struct { + schema string + table string + columns []string + values []interface{} +} + type mockPK struct { column string columns []string @@ -1460,6 +1514,14 @@ func (m *mockDBClient) RowExistsByColumns(ctx context.Context, schema, table str } func (m *mockDBClient) GetRowByColumns(ctx context.Context, schema, table string, columns []string, values []interface{}) ([]string, []interface{}, error) { + // Check override map first (for savepoint name-based lookups) + if m.rowByColumnsData != nil && len(columns) > 0 { + overrideKey := fmt.Sprintf("%s.%s.%s=%v", schema, table, columns[0], values[0]) + if data, ok := m.rowByColumnsData[overrideKey]; ok { + return data.columns, data.values, nil + } + } + key := schema + "." + table data, ok := m.allRowsData[key] if !ok { @@ -1515,7 +1577,19 @@ func (m *mockDBClient) InsertRow(ctx context.Context, schema, table string, colu m.insertCalled = true m.lastInsertColumns = columns m.lastInsertValues = values - return "1", nil + // Copy slices to avoid aliasing with caller + colsCopy := make([]string, len(columns)) + copy(colsCopy, columns) + valsCopy := make([]interface{}, len(values)) + copy(valsCopy, values) + m.insertedRows = append(m.insertedRows, mockInsertedRow{ + schema: schema, table: table, columns: colsCopy, values: valsCopy, + }) + pk := "1" + if m.lastInsertReturnPK != "" { + pk = m.lastInsertReturnPK + } + return pk, nil } func (m *mockDBClient) UpdateRow(ctx context.Context, schema, table string, pk *db.PKMatch, columns []string, values []interface{}) error { @@ -1554,6 +1628,16 @@ func (m *mockDBClient) DeleteRow(ctx context.Context, schema, table string, pk * return nil } +func (m *mockDBClient) DeleteAndUpdate(ctx context.Context, schema, table string, deletePK *db.PKMatch, updatePK *db.PKMatch, updateCols []string, updateVals []interface{}) error { + if m.deleteAndUpdateFunc != nil { + return m.deleteAndUpdateFunc(ctx, schema, table, deletePK, updatePK, updateCols, updateVals) + } + m.deleteCalled = true + m.lastDeletePK = deletePK.Values[0] + m.updateCalled = true + return nil +} + // Implement db.DDLExecutor func (m *mockDBClient) Exec(ctx context.Context, sql string, args ...interface{}) error { @@ -1637,6 +1721,10 @@ func (m *mockDBClient) QueryHistoryDistinctFilenames(ctx context.Context, schema return nil, nil } +func (m *mockDBClient) QueryHistoryDistinctFilenamesByParent(ctx context.Context, schema, historyTable, parentID string, limit int) ([]string, error) { + return nil, nil +} + func (m *mockDBClient) QueryHistoryDistinctIDs(ctx context.Context, schema, historyTable string, limit int) ([]string, error) { return nil, nil } @@ -1645,6 +1733,79 @@ func (m *mockDBClient) QueryHistoryVersionByTime(ctx context.Context, schema, hi return nil, nil, nil } +func (m *mockDBClient) InsertLogEntry(ctx context.Context, schema, logTable, userID, opType, fileID, filename, versionID, description string) error { + m.logEntries = append(m.logEntries, mockLogEntry{ + userID: userID, opType: opType, fileID: fileID, filename: filename, versionID: versionID, + }) + return nil +} + +func (m *mockDBClient) QueryNextLogEntry(ctx context.Context, schema, logTable, fileID, afterLogID string) (string, string, error) { + return m.nextLogVersionID, m.nextLogFilename, nil +} + +func (m *mockDBClient) QueryFileExists(ctx context.Context, schema, table, fileID string) (bool, error) { + if m.fileExistsMap != nil { + if v, ok := m.fileExistsMap[fileID]; ok { + return v, nil + } + } + return m.fileExistsResult, nil +} + +func (m *mockDBClient) QueryLatestVersionID(ctx context.Context, schema, historyTable, fileID string) (string, error) { + if m.latestVersionIDs != nil { + if vid, ok := m.latestVersionIDs[fileID]; ok { + return vid, nil + } + } + return "", fmt.Errorf("no history entry for %s", fileID) +} + +func (m *mockDBClient) QueryUndoAffectedFiles(ctx context.Context, schema, logTable, afterID, userID string, filters []db.UndoFilter) ([]db.UndoAffectedFile, error) { + return m.undoAffectedFiles, nil +} + +func (m *mockDBClient) QueryLogEntry(ctx context.Context, schema, logTable, logID string) (*db.UndoAffectedFile, error) { + if m.undoLogEntry != nil { + return m.undoLogEntry, nil + } + return nil, fmt.Errorf("log entry not found: %s", logID) +} + +func (m *mockDBClient) ExecuteUndoTransaction(ctx context.Context, params *db.UndoTransactionParams) error { + m.undoTransactionCalls++ + m.lastUndoParams = params + return nil +} + +func (m *mockDBClient) QueryFileExistsForUndo(fileID string) bool { + if m.fileExistsMap != nil { + if v, ok := m.fileExistsMap[fileID]; ok { + return v + } + } + return m.fileExistsResult +} + +func (m *mockDBClient) GetRowByParentAndName(ctx context.Context, schema, table, parentID, filename string) ([]string, []interface{}, error) { + return nil, nil, nil +} + +func (m *mockDBClient) GetRowsByParent(ctx context.Context, schema, table, parentID string, limit int) ([]string, [][]interface{}, error) { + return nil, nil, nil +} + +func (m *mockDBClient) ResolvePath(ctx context.Context, schema, table, startParentID string, segments []string) ([]db.PathSegment, error) { + m.resolvePathCalls++ + m.lastResolveStartParent = startParentID + m.lastResolveSegments = segments + if m.resolvePathFn != nil { + return m.resolvePathFn(ctx, schema, table, segments) + } + return m.resolvePathResults, m.resolvePathErr +} + // ============================================================================ // Tests for bug fixes - prevent regressions // ============================================================================ @@ -3543,3 +3704,43 @@ func TestCompositePK_TimestampPK(t *testing.T) { assert.Equal(t, "30\n", string(content.Data)) }) } + +// TestReadlink_NonSymlink verifies that Readlink on a regular file or directory +// returns ErrInvalidOperation. +func TestReadlink_NonSymlink(t *testing.T) { + cfg := &config.Config{DirListingLimit: 1000} + mockDB := &mockDBClient{ + tables: map[string][]string{"public": {"users"}}, + columns: map[string][]mockColumn{ + "users": {{name: "id", dataType: "integer"}}, + }, + primaryKeys: map[string]*mockPK{ + "users": {columns: []string{"id"}}, + }, + } + ops := NewOperations(cfg, mockDB) + + // Root directory is not a symlink + _, err := ops.Readlink(context.Background(), "/") + require.NotNil(t, err) + assert.Equal(t, ErrInvalidOperation, err.Code) + + // Table directory is not a symlink + _, err = ops.Readlink(context.Background(), "/users") + require.NotNil(t, err) + assert.Equal(t, ErrInvalidOperation, err.Code) +} + +// TestReadlink_NonexistentPath verifies that Readlink on a path that doesn't +// exist returns ErrNotExist. +func TestReadlink_NonexistentPath(t *testing.T) { + cfg := &config.Config{DirListingLimit: 1000} + mockDB := &mockDBClient{ + tables: map[string][]string{"public": {"users"}}, + } + ops := NewOperations(cfg, mockDB) + + _, err := ops.Readlink(context.Background(), "/nonexistent_table") + require.NotNil(t, err) + assert.Equal(t, ErrNotExist, err.Code) +} diff --git a/internal/tigerfs/fs/path.go b/internal/tigerfs/fs/path.go index cb12cf5..d53f9da 100644 --- a/internal/tigerfs/fs/path.go +++ b/internal/tigerfs/fs/path.go @@ -4,6 +4,8 @@ import ( "fmt" "strconv" "strings" + + "github.com/timescale/tigerfs/internal/tigerfs/format" ) // PathType indicates what kind of filesystem path was parsed. @@ -62,6 +64,23 @@ const ( // PathTablesList is the /.tables/ directory listing backing tables in tigerfs schema. PathTablesList + + // PathRootInfo is the root-level /.info/ directory for mount metadata (user identity). + // Distinct from PathInfo which is table-level /{table}/.info/. + PathRootInfo + + // PathLog is the /{table}/.log/ directory exposing the operation log table. + // Delegates to data-first pipeline on the tigerfs._log table. + PathLog + + // PathSavepoint is the /{table}/.savepoint/ directory exposing savepoints. + // Delegates to data-first pipeline on the tigerfs.
_savepoint table. + PathSavepoint + + // PathUndo is the /{table}/.undo/ directory for preview-then-apply undo. + // Contains routing (id/, to-id/, to-savepoint/), target selection, optional + // pipeline filters, and .apply/.info/summary leaves. + PathUndo ) // ParsedPath holds the result of parsing a filesystem path. @@ -139,6 +158,25 @@ type ParsedPath struct { // processing. Used by synth hierarchy to reconstruct multi-segment filenames. // For example, /memory/projects/web/todo.md → ["projects", "web", "todo.md"]. RawSubPath []string + + // UndoMode is the undo routing mode: "id", "to-id", or "to-savepoint". + // Set when Type is PathUndo. + UndoMode string + + // UndoTarget is the target identifier for undo: a log_id display name + // (for id/ and to-id/) or a savepoint name (for to-savepoint/). + UndoTarget string + + // UndoApply is true when the path ends with .apply (trigger to execute undo). + UndoApply bool + + // UndoFile is a file path within the undo preview directory. + // E.g., "docs/hello.md" in .undo/to-savepoint/before-refactor/docs/hello.md + UndoFile string + + // OrigTableName preserves the original synth app table name when the context + // is redirected to a different table (e.g., .log/ redirects to _log table). + OrigTableName string } // knownFormats maps format extensions to format names. @@ -149,6 +187,17 @@ var knownFormats = map[string]string{ ".yaml": "yaml", } +// IsRowFormatFile returns true if the filename has a known row format extension +// (.json, .csv, .tsv, .yaml). Used by adapters to detect row-level writes. +func IsRowFormatFile(filename string) bool { + for ext := range knownFormats { + if strings.HasSuffix(filename, ext) { + return true + } + } + return false +} + // ParsePath converts a filesystem path to a ParsedPath. // // Examples: @@ -207,6 +256,9 @@ func ParsePath(path string) (*ParsedPath, *FSError) { return parseBuildPath(segments) case ".tables": return parseTablesPath(segments) + case DirInfo: + // Root-level .info/ (mount metadata: user identity, etc.) + return parseRootInfoPath(segments) } // Otherwise, it's a table path (possibly with schema prefix) @@ -225,6 +277,18 @@ func splitPath(path string) []string { return segments } +// parseRootInfoPath handles root-level /.info/ paths (mount metadata). +// Supports: +// - /.info/ → list metadata files (user) +// - /.info/user → read/write mount-level user identity +func parseRootInfoPath(segments []string) (*ParsedPath, *FSError) { + result := &ParsedPath{Type: PathRootInfo} + if len(segments) >= 2 { + result.InfoFile = segments[1] + } + return result, nil +} + // parseSchemaPath handles /.schemas/ paths. // Supports: // - /.schemas/ → list schemas (plus .create) @@ -440,8 +504,10 @@ func processSegments(result *ParsedPath, segments []string) (*ParsedPath, *FSErr for i < len(segments) { seg := segments[i] - // Check for capability directories - if strings.HasPrefix(seg, ".") { + // Check for known capability directories (e.g., .info, .by, .filter, .history). + // Unknown dot-prefixed segments (e.g., .git, .gitignore, .env) fall through + // to the scan-ahead logic and are treated as regular filenames/directories. + if strings.HasPrefix(seg, ".") && isKnownCapability(seg) { consumed, err := processCapability(result, seg, segments[i:]) if err != nil { return nil, err @@ -485,7 +551,7 @@ func isKnownCapability(seg string) bool { switch seg { case DirInfo, DirBy, DirColumns, DirFilter, DirOrder, DirFirst, DirLast, DirSample, DirExport, DirImport, DirAll, DirModify, DirDelete, DirIndexes, - DirFormat, DirHistory: + DirFormat, DirHistory, DirLog, DirSavepoint, DirUndo: return true default: return false @@ -529,10 +595,17 @@ func processCapability(result *ParsedPath, cap string, remaining []string) (int, return processFormat(result, remaining) case DirHistory: return processHistory(result, remaining) + case DirLog: + return processLog(result, remaining) + case DirSavepoint: + return processSavepoint(result, remaining) + case DirUndo: + return processUndo(result, remaining) default: + // Should not reach here -- callers check isKnownCapability() first. return 0, &FSError{ Code: ErrInvalidPath, - Message: fmt.Sprintf("unknown capability: %s", cap), + Message: fmt.Sprintf("internal error: processCapability called with unknown capability: %s", cap), } } } @@ -577,7 +650,9 @@ func processBy(result *ParsedPath, remaining []string) (int, *FSError) { column := remaining[1] value := remaining[2] result.Context = result.Context.WithFilter(column, value, true) - result.Type = PathTable + if result.Type == PathTable || result.Type == PathCapability { + result.Type = PathTable + } return 3, nil } @@ -617,7 +692,9 @@ func processColumns(result *ParsedPath, remaining []string) (int, *FSError) { } result.Context = result.Context.WithColumns(columns) - result.Type = PathTable + if result.Type == PathTable || result.Type == PathCapability { + result.Type = PathTable + } return 2, nil } @@ -649,7 +726,9 @@ func processFilter(result *ParsedPath, remaining []string) (int, *FSError) { column := remaining[1] value := remaining[2] result.Context = result.Context.WithFilter(column, value, false) - result.Type = PathTable + if result.Type == PathTable || result.Type == PathCapability { + result.Type = PathTable + } return 3, nil } @@ -686,7 +765,9 @@ func processOrder(result *ParsedPath, remaining []string) (int, *FSError) { desc = true } result.Context = result.Context.WithOrder(col, desc) - result.Type = PathTable + if result.Type == PathTable || result.Type == PathCapability { + result.Type = PathTable + } return 2, nil } @@ -735,7 +816,11 @@ func processLimit(result *ParsedPath, remaining []string, limitType LimitType) ( } result.Context = result.Context.WithLimit(n, limitType) - result.Type = PathTable + // Preserve the original type for .log/.savepoint/.undo redirected tables. + // Only reset to PathTable if already a table path. + if result.Type == PathTable || result.Type == PathCapability { + result.Type = PathTable + } return 2, nil } @@ -992,10 +1077,284 @@ func processHistory(result *ParsedPath, remaining []string) (int, *FSError) { return consumed, nil } -// isVersionID returns true if a segment looks like a history version ID timestamp. -// Version IDs have the format "2006-01-02T150405Z" (e.g., "2026-02-12T013000Z"). +// processLog handles .log/ paths. Redirects the FSContext to the _log table +// in the tigerfs schema and delegates remaining segments to pipeline parsing. +// The path type becomes PathLog for the listing, or the standard PathRow/PathColumn +// types for accessing specific log entries via pipeline. +func processLog(result *ParsedPath, remaining []string) (int, *FSError) { + if result.Context == nil { + return 0, &FSError{Code: ErrInvalidPath, Message: ".log/ requires a table context"} + } + + // Preserve original table name and redirect to log table + result.OrigTableName = result.Context.TableName + result.Context.TableName = result.Context.TableName + "_log" + result.Context.Schema = "tigerfs" + result.Context.HideCapabilities = true // suppress pipeline dirs in listing + result.Type = PathLog + + if len(remaining) == 1 { + // Just .log/ -- listing + return 1, nil + } + + // Delegate remaining segments to standard pipeline/row parsing. + // This handles .log/.last/10, .log/.by/user_id/agent-7, .log//type, etc. + // Use processSegmentsFrom which returns consumed count. + pConsumed, err := processSegmentsFrom(result, remaining[1:]) + return 1 + pConsumed, err +} + +// processSavepoint handles .savepoint/ paths. Redirects the FSContext to the +// _savepoint table in the tigerfs schema and delegates to pipeline parsing. +func processSavepoint(result *ParsedPath, remaining []string) (int, *FSError) { + if result.Context == nil { + return 0, &FSError{Code: ErrInvalidPath, Message: ".savepoint/ requires a table context"} + } + + result.OrigTableName = result.Context.TableName + result.Context.TableName = result.Context.TableName + "_savepoint" + result.Context.Schema = "tigerfs" + result.Context.HideCapabilities = true // suppress pipeline dirs in listing + result.Type = PathSavepoint + + if len(remaining) == 1 { + return 1, nil + } + + pConsumed, err := processSegmentsFrom(result, remaining[1:]) + return 1 + pConsumed, err +} + +// processSegmentsFrom is like processSegments but returns the number of segments +// consumed instead of a new ParsedPath. Used by .log/ and .savepoint/ which need +// to redirect the context and then delegate remaining parsing. +func processSegmentsFrom(result *ParsedPath, segments []string) (int, *FSError) { + if len(segments) == 0 { + return 0, nil + } + + consumed := 0 + for consumed < len(segments) { + seg := segments[consumed] + + // Capability directory? + if strings.HasPrefix(seg, ".") && isKnownCapability(seg) { + n, err := processCapability(result, seg, segments[consumed:]) + if err != nil { + return 0, err + } + consumed += n + continue + } + + // Not a capability -- treat as row PK (data-first table access) + result.Type = PathRow + pk := seg + for ext, fmt := range knownFormats { + if strings.HasSuffix(pk, ext) { + pk = strings.TrimSuffix(pk, ext) + result.Format = fmt + break + } + } + result.PrimaryKey = pk + consumed++ + + // If there are more segments, check if it's a format file or column name + if consumed < len(segments) { + nextSeg := segments[consumed] + if strings.HasPrefix(nextSeg, ".") && isKnownCapability(nextSeg) { + continue // pipeline after row + } + // Check if it's a row format file (.json, .csv, .tsv, .yaml) + if fmt, ok := knownFormats[nextSeg]; ok { + result.Format = fmt + } else { + result.Type = PathColumn + result.Column = nextSeg + } + consumed++ + } + break + } + + return consumed, nil +} + +// processUndo handles .undo/ paths with routing structure: +// +// .undo/ → listing of undo modes +// .undo/id/ → list recent log entries +// .undo/id// → preview single undo +// .undo/id//.apply → execute single undo +// .undo/id//.info/summary → preview summary +// .undo/id//docs/hello.md → preview file content +// .undo/to-id// → preview undo-to-point +// .undo/to-savepoint// → preview undo-to-savepoint +// .undo/to-savepoint//.by/user_id/agent-7/.apply → filtered undo +// +// Pipeline segments (.by/, .filter/, .last/, etc.) after the target narrow +// which operations are included in the undo scope. +func processUndo(result *ParsedPath, remaining []string) (int, *FSError) { + if result.Context == nil { + return 0, &FSError{Code: ErrInvalidPath, Message: ".undo/ requires a table context"} + } + + result.OrigTableName = result.Context.TableName + result.Type = PathUndo + + if len(remaining) == 1 { + // Just .undo/ -- list modes (id/, to-id/, to-savepoint/) + return 1, nil + } + + // Parse mode + mode := remaining[1] + switch mode { + case "id", "to-id", "to-savepoint": + result.UndoMode = mode + default: + return 0, &FSError{Code: ErrInvalidPath, Message: fmt.Sprintf("unknown undo mode: %s (expected id, to-id, or to-savepoint)", mode)} + } + + if len(remaining) == 2 { + // .undo// -- list targets (log entries or savepoints) + return 2, nil + } + + // Parse target (log_id display name or savepoint name) + result.UndoTarget = remaining[2] + + if len(remaining) == 3 { + // .undo/// -- preview directory + return 3, nil + } + + // Parse remaining segments: pipeline capabilities, .apply, .info/summary, or file paths + consumed := 3 + for i := 3; i < len(remaining); i++ { + seg := remaining[i] + + // Check for .apply trigger + if seg == FileApply { + result.UndoApply = true + consumed = i + 1 + return consumed, nil + } + + // Check for .info/summary + if seg == DirInfo { + if i+1 < len(remaining) { + result.InfoFile = remaining[i+1] + consumed = i + 2 + } else { + result.InfoFile = "." // sentinel: inside .info/ directory + consumed = i + 1 + } + return consumed, nil + } + + // Check for pipeline capability + if isKnownCapability(seg) { + // Process pipeline segments starting at this position. + // Build a sub-remaining slice from this point. + pipelineRemaining := remaining[i:] + pipelineConsumed, err := processCapabilityChain(result, pipelineRemaining) + if err != nil { + return 0, err + } + consumed = i + pipelineConsumed + return consumed, nil + } + + // Not a capability, not .apply, not .info -- it's a file path in the preview + var fileParts []string + for j := i; j < len(remaining); j++ { + seg := remaining[j] + if seg == FileApply { + result.UndoApply = true + consumed = j + 1 + result.UndoFile = strings.Join(fileParts, "/") + return consumed, nil + } + if seg == DirInfo || isKnownCapability(seg) { + break + } + fileParts = append(fileParts, seg) + consumed = j + 1 + } + result.UndoFile = strings.Join(fileParts, "/") + return consumed, nil + } + + return consumed, nil +} + +// processCapabilityChain processes a chain of pipeline capabilities starting +// from the current position. Used by .undo/ to handle pipeline segments after +// the target. Returns the number of segments consumed. +func processCapabilityChain(result *ParsedPath, remaining []string) (int, *FSError) { + consumed := 0 + for consumed < len(remaining) { + seg := remaining[consumed] + + // Check for .apply at any point in the chain + if seg == FileApply { + result.UndoApply = true + consumed++ + return consumed, nil + } + + // Check for .info/summary + if seg == DirInfo { + consumed++ + if consumed < len(remaining) { + result.InfoFile = remaining[consumed] + consumed++ + } + return consumed, nil + } + + if !isKnownCapability(seg) { + break + } + + n, err := processCapability(result, seg, remaining[consumed:]) + if err != nil { + return 0, err + } + consumed += n + } + + // Check if remaining segments after pipeline are .apply or .info + if consumed < len(remaining) { + seg := remaining[consumed] + if seg == FileApply { + result.UndoApply = true + consumed++ + } else if seg == DirInfo { + consumed++ + if consumed < len(remaining) { + result.InfoFile = remaining[consumed] + consumed++ + } + } + } + + return consumed, nil +} + +// isVersionID returns true if a segment looks like a history version ID. +// Recognizes two formats: +// - Legacy: "2006-01-02T150405Z" (18 chars, second-precision timestamp) +// - Display name: "2006-01-02T150405.000Z-" (UUIDv7 display format, ADR-016) func isVersionID(seg string) bool { - // Quick length check: "2006-01-02T150405Z" = 18 chars + // Check for UUIDv7 display name format (e.g., "2026-04-10T173000.123Z-abc123") + if format.IsDisplayName(seg) { + return true + } + // Legacy: "2006-01-02T150405Z" = 18 chars if len(seg) != 18 { return false } diff --git a/internal/tigerfs/fs/path_cache.go b/internal/tigerfs/fs/path_cache.go new file mode 100644 index 0000000..99f9e02 --- /dev/null +++ b/internal/tigerfs/fs/path_cache.go @@ -0,0 +1,97 @@ +package fs + +import ( + "sync" + "time" +) + +// pathCacheTTL is the maximum age of cached path entries before they expire. +// Matches the stat cache TTL -- directory structure changes are visible +// to other mounts within this window. +const pathCacheTTL = 2 * time.Second + +// pathCache maps (parentID, filename) -> row ID for the parent-pointer directory +// model (ADR-017). Used to avoid calling resolve_path for repeated access to the +// same directory subtree. Each table has its own cache namespace. +type pathCache struct { + mu sync.RWMutex + tables map[string]*pathTableCache // key: "schema\x00table" +} + +// pathTableCache holds cached path entries for a single table. +type pathTableCache struct { + entries map[pathCacheKey]pathCacheEntry + created time.Time +} + +// pathCacheKey identifies a single path segment within a parent directory. +// Empty ParentID represents root level (NULL parent_id in the database). +type pathCacheKey struct { + ParentID string // UUID of parent directory, empty for root + Filename string // leaf filename +} + +// pathCacheEntry holds a cached row ID with its expiration. +type pathCacheEntry struct { + ID string // UUID of the resolved row +} + +// lookup returns the cached row ID for a (parentID, filename) pair. +// Returns empty string and false if not found or expired. +func (c *pathCache) lookup(schema, table, parentID, filename string) (string, bool) { + c.mu.RLock() + defer c.mu.RUnlock() + + if c.tables == nil { + return "", false + } + + tc := c.tables[schema+"\x00"+table] + if tc == nil { + return "", false + } + + if time.Since(tc.created) > pathCacheTTL { + return "", false + } + + entry, ok := tc.entries[pathCacheKey{ParentID: parentID, Filename: filename}] + if !ok { + return "", false + } + + return entry.ID, true +} + +// put stores a (parentID, filename) -> id mapping in the cache. +func (c *pathCache) put(schema, table, parentID, filename, id string) { + c.mu.Lock() + defer c.mu.Unlock() + + if c.tables == nil { + c.tables = make(map[string]*pathTableCache) + } + + key := schema + "\x00" + table + tc := c.tables[key] + if tc == nil || time.Since(tc.created) > pathCacheTTL { + tc = &pathTableCache{ + entries: make(map[pathCacheKey]pathCacheEntry), + created: time.Now(), + } + c.tables[key] = tc + } + + tc.entries[pathCacheKey{ParentID: parentID, Filename: filename}] = pathCacheEntry{ID: id} +} + +// invalidate removes all cached entries for a table. Called on directory +// renames/moves/deletes that may change the path structure. +func (c *pathCache) invalidate(schema, table string) { + c.mu.Lock() + defer c.mu.Unlock() + + if c.tables != nil { + delete(c.tables, schema+"\x00"+table) + } +} diff --git a/internal/tigerfs/fs/path_cache_test.go b/internal/tigerfs/fs/path_cache_test.go new file mode 100644 index 0000000..637ed88 --- /dev/null +++ b/internal/tigerfs/fs/path_cache_test.go @@ -0,0 +1,158 @@ +package fs + +import ( + "testing" + "time" + + "github.com/stretchr/testify/assert" +) + +func TestPathCache_LookupMiss(t *testing.T) { + var c pathCache + id, ok := c.lookup("s", "t", "", "foo") + assert.False(t, ok) + assert.Empty(t, id) +} + +func TestPathCache_PutAndLookup(t *testing.T) { + var c pathCache + c.put("s", "t", "", "projects", "uuid-1") + c.put("s", "t", "uuid-1", "web", "uuid-2") + c.put("s", "t", "uuid-2", "todo.md", "uuid-3") + + // All three levels should be cached + id, ok := c.lookup("s", "t", "", "projects") + assert.True(t, ok) + assert.Equal(t, "uuid-1", id) + + id, ok = c.lookup("s", "t", "uuid-1", "web") + assert.True(t, ok) + assert.Equal(t, "uuid-2", id) + + id, ok = c.lookup("s", "t", "uuid-2", "todo.md") + assert.True(t, ok) + assert.Equal(t, "uuid-3", id) +} + +func TestPathCache_SeparateTableNamespaces(t *testing.T) { + var c pathCache + c.put("s", "notes", "", "hello", "uuid-a") + c.put("s", "blog", "", "hello", "uuid-b") + + id, ok := c.lookup("s", "notes", "", "hello") + assert.True(t, ok) + assert.Equal(t, "uuid-a", id) + + id, ok = c.lookup("s", "blog", "", "hello") + assert.True(t, ok) + assert.Equal(t, "uuid-b", id) +} + +func TestPathCache_LookupMissWrongFilename(t *testing.T) { + var c pathCache + c.put("s", "t", "", "foo", "uuid-1") + + _, ok := c.lookup("s", "t", "", "bar") + assert.False(t, ok) +} + +func TestPathCache_LookupMissWrongParent(t *testing.T) { + var c pathCache + c.put("s", "t", "uuid-1", "foo", "uuid-2") + + _, ok := c.lookup("s", "t", "uuid-99", "foo") + assert.False(t, ok) +} + +func TestPathCache_Invalidate(t *testing.T) { + var c pathCache + c.put("s", "t", "", "foo", "uuid-1") + + // Confirm cached + _, ok := c.lookup("s", "t", "", "foo") + assert.True(t, ok) + + // Invalidate + c.invalidate("s", "t") + + _, ok = c.lookup("s", "t", "", "foo") + assert.False(t, ok) +} + +func TestPathCache_InvalidateOnlyAffectsTargetTable(t *testing.T) { + var c pathCache + c.put("s", "notes", "", "foo", "uuid-1") + c.put("s", "blog", "", "foo", "uuid-2") + + c.invalidate("s", "notes") + + _, ok := c.lookup("s", "notes", "", "foo") + assert.False(t, ok) + + id, ok := c.lookup("s", "blog", "", "foo") + assert.True(t, ok) + assert.Equal(t, "uuid-2", id) +} + +func TestPathCache_TTLExpiry(t *testing.T) { + // Temporarily reduce TTL for this test by manipulating the cache internals. + // We test the TTL check by creating a table cache with an old timestamp. + var c pathCache + c.mu.Lock() + c.tables = map[string]*pathTableCache{ + "s\x00t": { + entries: map[pathCacheKey]pathCacheEntry{ + {ParentID: "", Filename: "foo"}: {ID: "uuid-1"}, + }, + created: time.Now().Add(-3 * time.Second), // older than 2s TTL + }, + } + c.mu.Unlock() + + _, ok := c.lookup("s", "t", "", "foo") + assert.False(t, ok, "entries older than TTL should expire") +} + +func TestPathCache_PutResetsExpiredTable(t *testing.T) { + var c pathCache + + // Create an expired table cache + c.mu.Lock() + c.tables = map[string]*pathTableCache{ + "s\x00t": { + entries: map[pathCacheKey]pathCacheEntry{ + {ParentID: "", Filename: "old"}: {ID: "uuid-old"}, + }, + created: time.Now().Add(-3 * time.Second), + }, + } + c.mu.Unlock() + + // Put into expired table should reset it + c.put("s", "t", "", "new", "uuid-new") + + // Old entry should be gone (table was reset) + _, ok := c.lookup("s", "t", "", "old") + assert.False(t, ok) + + // New entry should exist + id, ok := c.lookup("s", "t", "", "new") + assert.True(t, ok) + assert.Equal(t, "uuid-new", id) +} + +func TestPathCache_RootParentID(t *testing.T) { + var c pathCache + + // Root-level entries use empty string for parent_id + c.put("s", "t", "", "root-file.md", "uuid-root") + id, ok := c.lookup("s", "t", "", "root-file.md") + assert.True(t, ok) + assert.Equal(t, "uuid-root", id) + + // Nested entries use parent UUID + c.put("s", "t", "uuid-root", "child.md", "uuid-child") + id, ok = c.lookup("s", "t", "uuid-root", "child.md") + assert.True(t, ok) + assert.Equal(t, "uuid-child", id) +} diff --git a/internal/tigerfs/fs/path_test.go b/internal/tigerfs/fs/path_test.go index d5787a2..f6578a1 100644 --- a/internal/tigerfs/fs/path_test.go +++ b/internal/tigerfs/fs/path_test.go @@ -741,7 +741,6 @@ func TestParsePathInvalid(t *testing.T) { {"users", "no leading slash"}, {"/users/.first/abc", "non-numeric limit"}, {"/users/.first/-1", "negative limit"}, - {"/users/.unknown/foo", "unknown capability"}, } for _, tt := range tests { @@ -754,6 +753,57 @@ func TestParsePathInvalid(t *testing.T) { } } +// TestParsePathDotfiles verifies that unknown dot-prefixed names are treated as regular +// filenames/directories, not rejected as unknown capabilities. +func TestParsePathDotfiles(t *testing.T) { + tests := []struct { + path string + desc string + pathType PathType + pk string + column string + }{ + {"/docs/.gitignore", "dotfile as row", PathRow, ".gitignore", ""}, + {"/docs/.env", "env dotfile", PathRow, ".env", ""}, + {"/docs/.git/config", "dotfile dir with column", PathColumn, ".git", "config"}, + {"/docs/.vscode/settings.json", "dotfile dir with file", PathColumn, ".vscode", "settings.json"}, + {"/users/.unknown/foo", "unknown dot as row+column", PathColumn, ".unknown", "foo"}, + } + + for _, tt := range tests { + t.Run(tt.desc, func(t *testing.T) { + result, err := ParsePath(tt.path) + if err != nil { + t.Fatalf("ParsePath(%q) returned error: %v", tt.path, err) + } + if result.Type != tt.pathType { + t.Errorf("Type = %v, want %v", result.Type, tt.pathType) + } + if result.PrimaryKey != tt.pk { + t.Errorf("PrimaryKey = %q, want %q", result.PrimaryKey, tt.pk) + } + if tt.column != "" && result.Column != tt.column { + t.Errorf("Column = %q, want %q", result.Column, tt.column) + } + }) + } +} + +// TestParsePathDotfileThenCapability verifies that a dotfile followed by a known +// capability directory is correctly parsed via scan-ahead. +func TestParsePathDotfileThenCapability(t *testing.T) { + result, err := ParsePath("/docs/.git/.history") + if err != nil { + t.Fatalf("ParsePath returned error: %v", err) + } + if result.PrimaryKey != ".git" { + t.Errorf("PrimaryKey = %q, want %q", result.PrimaryKey, ".git") + } + if result.Type != PathHistory { + t.Errorf("Type = %v, want PathHistory", result.Type) + } +} + // TestParsePathDisallowedCombinations verifies that invalid capability combinations are rejected. func TestParsePathDisallowedCombinations(t *testing.T) { tests := []struct { @@ -1360,19 +1410,29 @@ func TestSynth_ParsePathCapabilityAfterDirSegments(t *testing.T) { } // TestSynth_IsVersionID verifies version ID format detection. +// Recognizes both legacy 18-char timestamps and UUIDv7 display names (ADR-016). func TestSynth_IsVersionID(t *testing.T) { tests := []struct { seg string want bool }{ + // Legacy format: "2006-01-02T150405Z" (18 chars) {"2026-02-12T013000Z", true}, {"2025-01-01T000000Z", true}, + {"xxxx-xx-xxTxxxxxxZ", true}, // structural match (length + separators) + + // UUIDv7 display name format: "2006-01-02T150405.000Z-" (ADR-016) + {"2026-04-10T173000.123Z-zzz0063hd8e5r42", true}, + {"2025-01-01T000000.000Z-0", true}, + {"2026-12-31T235959.999Z-abc", true}, + + // Invalid {"foo.md", false}, {".id", false}, {"", false}, {"2026-02-12", false}, // too short - {"2026-02-12T013000Zx", false}, // too long - {"xxxx-xx-xxTxxxxxxZ", true}, // structural match (length + separators) + {"2026-02-12T013000Zx", false}, // 19 chars, not legacy, not display name + {"not-a-timestamp.000Z-abc", false}, } for _, tt := range tests { diff --git a/internal/tigerfs/fs/path_undo_test.go b/internal/tigerfs/fs/path_undo_test.go new file mode 100644 index 0000000..9c641df --- /dev/null +++ b/internal/tigerfs/fs/path_undo_test.go @@ -0,0 +1,288 @@ +package fs + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// --- .log/ path parsing --- + +func TestParsePath_Log_Listing(t *testing.T) { + result, err := ParsePath("/notes/.log") + require.Nil(t, err) + assert.Equal(t, PathLog, result.Type) + assert.Equal(t, "tigerfs", result.Context.Schema) + assert.Equal(t, "notes_log", result.Context.TableName) + assert.Equal(t, "notes", result.OrigTableName) +} + +func TestParsePath_Log_Pipeline(t *testing.T) { + result, err := ParsePath("/notes/.log/.last/10") + require.Nil(t, err) + assert.Equal(t, "notes_log", result.Context.TableName) + assert.Equal(t, LimitLast, result.Context.LimitType) + assert.Equal(t, 10, result.Context.Limit) +} + +func TestParsePath_Log_ByUserID(t *testing.T) { + result, err := ParsePath("/notes/.log/.by/user_id/agent-7") + require.Nil(t, err) + assert.Equal(t, "notes_log", result.Context.TableName) + require.Len(t, result.Context.Filters, 1) + assert.Equal(t, "user_id", result.Context.Filters[0].Column) + assert.Equal(t, "agent-7", result.Context.Filters[0].Value) +} + +func TestParsePath_Log_Entry(t *testing.T) { + result, err := ParsePath("/notes/.log/2026-04-08T143012.001Z-g7h8i9j0k1l2a") + require.Nil(t, err) + assert.Equal(t, PathRow, result.Type) + assert.Equal(t, "notes_log", result.Context.TableName) + assert.Equal(t, "2026-04-08T143012.001Z-g7h8i9j0k1l2a", result.PrimaryKey) +} + +func TestParsePath_Log_EntryColumn(t *testing.T) { + result, err := ParsePath("/notes/.log/2026-04-08T143012.001Z-g7h8i9j0k1l2a/type") + require.Nil(t, err) + assert.Equal(t, PathColumn, result.Type) + assert.Equal(t, "notes_log", result.Context.TableName) + assert.Equal(t, "type", result.Column) +} + +func TestParsePath_Log_Export(t *testing.T) { + result, err := ParsePath("/notes/.log/.export/json") + require.Nil(t, err) + assert.Equal(t, "notes_log", result.Context.TableName) + assert.Equal(t, PathExport, result.Type) +} + +func TestParsePath_Log_ByUserExport(t *testing.T) { + result, err := ParsePath("/notes/.log/.by/user_id/agent-7/.export/json") + require.Nil(t, err) + assert.Equal(t, "notes_log", result.Context.TableName) + require.Len(t, result.Context.Filters, 1) + assert.Equal(t, "agent-7", result.Context.Filters[0].Value) + assert.Equal(t, PathExport, result.Type) +} + +// --- .savepoint/ path parsing --- + +func TestParsePath_Savepoint_Listing(t *testing.T) { + result, err := ParsePath("/notes/.savepoint") + require.Nil(t, err) + assert.Equal(t, PathSavepoint, result.Type) + assert.Equal(t, "tigerfs", result.Context.Schema) + assert.Equal(t, "notes_savepoint", result.Context.TableName) + assert.Equal(t, "notes", result.OrigTableName) +} + +func TestParsePath_Savepoint_Entry(t *testing.T) { + result, err := ParsePath("/notes/.savepoint/before-exploration") + require.Nil(t, err) + assert.Equal(t, PathRow, result.Type) + assert.Equal(t, "notes_savepoint", result.Context.TableName) + assert.Equal(t, "before-exploration", result.PrimaryKey) +} + +func TestParsePath_Savepoint_EntryColumn(t *testing.T) { + result, err := ParsePath("/notes/.savepoint/before-exploration/description") + require.Nil(t, err) + assert.Equal(t, PathColumn, result.Type) + assert.Equal(t, "notes_savepoint", result.Context.TableName) + assert.Equal(t, "description", result.Column) +} + +func TestParsePath_Savepoint_Pipeline(t *testing.T) { + result, err := ParsePath("/notes/.savepoint/.by/user_id/agent-7/.last/5") + require.Nil(t, err) + assert.Equal(t, "notes_savepoint", result.Context.TableName) + require.Len(t, result.Context.Filters, 1) + assert.Equal(t, "agent-7", result.Context.Filters[0].Value) + assert.Equal(t, LimitLast, result.Context.LimitType) + assert.Equal(t, 5, result.Context.Limit) +} + +// --- .undo/ path parsing --- + +func TestParsePath_Undo_Root(t *testing.T) { + result, err := ParsePath("/notes/.undo") + require.Nil(t, err) + assert.Equal(t, PathUndo, result.Type) + assert.Empty(t, result.UndoMode) + assert.Equal(t, "notes", result.OrigTableName) +} + +func TestParsePath_Undo_ModeID(t *testing.T) { + result, err := ParsePath("/notes/.undo/id") + require.Nil(t, err) + assert.Equal(t, PathUndo, result.Type) + assert.Equal(t, "id", result.UndoMode) + assert.Empty(t, result.UndoTarget) +} + +func TestParsePath_Undo_ModeToID(t *testing.T) { + result, err := ParsePath("/notes/.undo/to-id") + require.Nil(t, err) + assert.Equal(t, "to-id", result.UndoMode) +} + +func TestParsePath_Undo_ModeToSavepoint(t *testing.T) { + result, err := ParsePath("/notes/.undo/to-savepoint") + require.Nil(t, err) + assert.Equal(t, "to-savepoint", result.UndoMode) +} + +func TestParsePath_Undo_InvalidMode(t *testing.T) { + _, err := ParsePath("/notes/.undo/badmode") + require.NotNil(t, err) + assert.Equal(t, ErrInvalidPath, err.Code) +} + +func TestParsePath_Undo_IDTarget(t *testing.T) { + result, err := ParsePath("/notes/.undo/id/2026-04-08T143015.234Z-i9j0k1l2m3n4b") + require.Nil(t, err) + assert.Equal(t, "id", result.UndoMode) + assert.Equal(t, "2026-04-08T143015.234Z-i9j0k1l2m3n4b", result.UndoTarget) + assert.False(t, result.UndoApply) +} + +func TestParsePath_Undo_IDApply(t *testing.T) { + result, err := ParsePath("/notes/.undo/id/2026-04-08T143015.234Z-i9j0k1l2m3n4b/.apply") + require.Nil(t, err) + assert.Equal(t, "id", result.UndoMode) + assert.Equal(t, "2026-04-08T143015.234Z-i9j0k1l2m3n4b", result.UndoTarget) + assert.True(t, result.UndoApply) +} + +func TestParsePath_Undo_IDInfoSummary(t *testing.T) { + result, err := ParsePath("/notes/.undo/id/2026-04-08T143015.234Z-i9j0k1l2m3n4b/.info/summary") + require.Nil(t, err) + assert.Equal(t, "id", result.UndoMode) + assert.Equal(t, "2026-04-08T143015.234Z-i9j0k1l2m3n4b", result.UndoTarget) + assert.Equal(t, "summary", result.InfoFile) + assert.False(t, result.UndoApply) +} + +func TestParsePath_Undo_ToSavepointTarget(t *testing.T) { + result, err := ParsePath("/notes/.undo/to-savepoint/before-exploration") + require.Nil(t, err) + assert.Equal(t, "to-savepoint", result.UndoMode) + assert.Equal(t, "before-exploration", result.UndoTarget) +} + +func TestParsePath_Undo_ToSavepointApply(t *testing.T) { + result, err := ParsePath("/notes/.undo/to-savepoint/before-exploration/.apply") + require.Nil(t, err) + assert.Equal(t, "to-savepoint", result.UndoMode) + assert.Equal(t, "before-exploration", result.UndoTarget) + assert.True(t, result.UndoApply) +} + +func TestParsePath_Undo_ToSavepointByUserApply(t *testing.T) { + result, err := ParsePath("/notes/.undo/to-savepoint/before-exploration/.by/user_id/agent-7/.apply") + require.Nil(t, err) + assert.Equal(t, "to-savepoint", result.UndoMode) + assert.Equal(t, "before-exploration", result.UndoTarget) + require.Len(t, result.Context.Filters, 1) + assert.Equal(t, "user_id", result.Context.Filters[0].Column) + assert.Equal(t, "agent-7", result.Context.Filters[0].Value) + assert.True(t, result.UndoApply) +} + +func TestParsePath_Undo_ToSavepointByUserInfoSummary(t *testing.T) { + result, err := ParsePath("/notes/.undo/to-savepoint/before-exploration/.by/user_id/agent-7/.info/summary") + require.Nil(t, err) + assert.Equal(t, "to-savepoint", result.UndoMode) + assert.Equal(t, "before-exploration", result.UndoTarget) + require.Len(t, result.Context.Filters, 1) + assert.Equal(t, "agent-7", result.Context.Filters[0].Value) + assert.Equal(t, "summary", result.InfoFile) + assert.False(t, result.UndoApply) +} + +func TestParsePath_Undo_FilterTypeDelete(t *testing.T) { + result, err := ParsePath("/notes/.undo/to-savepoint/before-refactor/.filter/type/delete/.apply") + require.Nil(t, err) + assert.Equal(t, "to-savepoint", result.UndoMode) + assert.Equal(t, "before-refactor", result.UndoTarget) + assert.True(t, result.UndoApply) +} + +func TestParsePath_Undo_Last5InfoSummary(t *testing.T) { + result, err := ParsePath("/notes/.undo/to-savepoint/before-refactor/.last/5/.info/summary") + require.Nil(t, err) + assert.Equal(t, "to-savepoint", result.UndoMode) + assert.Equal(t, "before-refactor", result.UndoTarget) + assert.Equal(t, "summary", result.InfoFile) +} + +func TestParsePath_Undo_PreviewFile(t *testing.T) { + result, err := ParsePath("/notes/.undo/to-savepoint/before-exploration/docs/hello.md") + require.Nil(t, err) + assert.Equal(t, "to-savepoint", result.UndoMode) + assert.Equal(t, "before-exploration", result.UndoTarget) + assert.Equal(t, "docs/hello.md", result.UndoFile) +} + +func TestParsePath_Undo_ToIDTarget(t *testing.T) { + result, err := ParsePath("/notes/.undo/to-id/2026-04-08T143015.234Z-abc123") + require.Nil(t, err) + assert.Equal(t, "to-id", result.UndoMode) + assert.Equal(t, "2026-04-08T143015.234Z-abc123", result.UndoTarget) +} + +func TestParsePath_Undo_ToIDApply(t *testing.T) { + result, err := ParsePath("/notes/.undo/to-id/2026-04-08T143015.234Z-abc123/.apply") + require.Nil(t, err) + assert.Equal(t, "to-id", result.UndoMode) + assert.True(t, result.UndoApply) +} + +func TestParsePath_Undo_ToIDByUserApply(t *testing.T) { + result, err := ParsePath("/notes/.undo/to-id/2026-04-08T143015.234Z-abc123/.by/user_id/agent-7/.apply") + require.Nil(t, err) + assert.Equal(t, "to-id", result.UndoMode) + assert.Equal(t, "2026-04-08T143015.234Z-abc123", result.UndoTarget) + require.Len(t, result.Context.Filters, 1) + assert.Equal(t, "agent-7", result.Context.Filters[0].Value) + assert.True(t, result.UndoApply) +} + +func TestParsePath_Undo_SampleParses(t *testing.T) { + // .sample/ should parse correctly (pipeline works) but .apply should NOT + // be available under .sample/ -- that's a handler concern, not parsing. + result, err := ParsePath("/notes/.undo/to-savepoint/before-refactor/.sample/5") + require.Nil(t, err) + assert.Equal(t, "to-savepoint", result.UndoMode) + assert.Equal(t, "before-refactor", result.UndoTarget) + assert.Equal(t, LimitSample, result.Context.LimitType) + assert.Equal(t, 5, result.Context.Limit) + assert.False(t, result.UndoApply) +} + +// --- .log/ diff symlink column parsing --- + +func TestParsePath_Log_EntryBefore(t *testing.T) { + result, err := ParsePath("/notes/.log/2026-04-08T143012.001Z-g7h8i9j0k1l2a/before") + require.Nil(t, err) + assert.Equal(t, PathColumn, result.Type) + assert.Equal(t, "notes_log", result.Context.TableName) + assert.Equal(t, "2026-04-08T143012.001Z-g7h8i9j0k1l2a", result.PrimaryKey) + assert.Equal(t, "before", result.Column) +} + +func TestParsePath_Log_EntryAfter(t *testing.T) { + result, err := ParsePath("/notes/.log/2026-04-08T143012.001Z-g7h8i9j0k1l2a/after") + require.Nil(t, err) + assert.Equal(t, PathColumn, result.Type) + assert.Equal(t, "after", result.Column) +} + +func TestParsePath_Log_EntryCurrent(t *testing.T) { + result, err := ParsePath("/notes/.log/2026-04-08T143012.001Z-g7h8i9j0k1l2a/current") + require.Nil(t, err) + assert.Equal(t, PathColumn, result.Type) + assert.Equal(t, "current", result.Column) +} diff --git a/internal/tigerfs/fs/savepoint_test.go b/internal/tigerfs/fs/savepoint_test.go new file mode 100644 index 0000000..9b5ac49 --- /dev/null +++ b/internal/tigerfs/fs/savepoint_test.go @@ -0,0 +1,370 @@ +package fs + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/timescale/tigerfs/internal/tigerfs/config" +) + +// --- readDirSavepoint unit tests --- + +func TestReadDirSavepoint_PassesAscendingForFirst(t *testing.T) { + parsed, err := ParsePath("/notes/.savepoint/.first/3") + require.Nil(t, err) + assert.Equal(t, PathSavepoint, parsed.Type) + assert.Equal(t, LimitFirst, parsed.Context.LimitType) + assert.Equal(t, 3, parsed.Context.Limit) +} + +func TestReadDirSavepoint_PassesDescendingForLast(t *testing.T) { + parsed, err := ParsePath("/notes/.savepoint/.last/5") + require.Nil(t, err) + assert.Equal(t, PathSavepoint, parsed.Type) + assert.Equal(t, LimitLast, parsed.Context.LimitType) + assert.Equal(t, 5, parsed.Context.Limit) +} + +func TestReadDirSavepoint_PreservesTypeWithFilter(t *testing.T) { + parsed, err := ParsePath("/notes/.savepoint/.by/user_id/agent-7") + require.Nil(t, err) + assert.Equal(t, PathSavepoint, parsed.Type) + require.Len(t, parsed.Context.Filters, 1) + assert.Equal(t, "user_id", parsed.Context.Filters[0].Column) + assert.Equal(t, "agent-7", parsed.Context.Filters[0].Value) +} + +func TestReadDirSavepoint_PreservesTypeWithFilterAndLimit(t *testing.T) { + parsed, err := ParsePath("/notes/.savepoint/.by/user_id/agent-7/.last/3") + require.Nil(t, err) + assert.Equal(t, PathSavepoint, parsed.Type) + assert.Equal(t, LimitLast, parsed.Context.LimitType) + assert.Equal(t, 3, parsed.Context.Limit) +} + +// --- writeSavepoint unit tests (name as PK, format suffix required) --- + +// Helper to extract column map from inserted row +func insertedColMap(t *testing.T, mockDB *mockDBClient) map[string]interface{} { + t.Helper() + require.Len(t, mockDB.insertedRows, 1) + row := mockDB.insertedRows[0] + m := make(map[string]interface{}) + for i, col := range row.columns { + m[col] = row.values[i] + } + return m +} + +func newSavepointMock() *mockDBClient { + return &mockDBClient{ + primaryKeys: map[string]*mockPK{ + "tigerfs.notes_savepoint": {column: "name"}, + }, + lastInsertReturnPK: "my-checkpoint", + } +} + +func newSavepointMockWithColumns() *mockDBClient { + m := newSavepointMock() + m.columns = map[string][]mockColumn{ + "tigerfs.notes_savepoint": { + {name: "name", dataType: "text"}, + {name: "savepoint_id", dataType: "uuid"}, + {name: "user_id", dataType: "text"}, + {name: "description", dataType: "text"}, + }, + } + return m +} + +// -- Empty body tests (echo "" > name.tsv) -- + +func TestSynth_Savepoint_TSV_EmptyBody(t *testing.T) { + mockDB := newSavepointMock() + ops := NewOperations(&config.Config{DirListingLimit: 1000}, mockDB) + ops.SetUserID("agent-7") + + // echo "" > .savepoint/name.tsv -- empty body, should create with just PK + user_id + data := []byte("\n") + fsErr := ops.WriteFile(context.Background(), "/notes/.savepoint/quick-mark.tsv", data) + require.Nil(t, fsErr) + + m := insertedColMap(t, mockDB) + assert.Equal(t, "quick-mark", m["name"]) + assert.Equal(t, "agent-7", m["user_id"]) +} + +func TestSynth_Savepoint_JSON_EmptyBody(t *testing.T) { + mockDB := newSavepointMock() + ops := NewOperations(&config.Config{DirListingLimit: 1000}, mockDB) + ops.SetUserID("agent-7") + + // echo "" > .savepoint/name.json -- empty body + data := []byte("\n") + fsErr := ops.WriteFile(context.Background(), "/notes/.savepoint/quick-mark.json", data) + require.Nil(t, fsErr) + + m := insertedColMap(t, mockDB) + assert.Equal(t, "quick-mark", m["name"]) + assert.Equal(t, "agent-7", m["user_id"]) +} + +func TestSynth_Savepoint_JSON_EmptyObject(t *testing.T) { + mockDB := newSavepointMock() + ops := NewOperations(&config.Config{DirListingLimit: 1000}, mockDB) + + // echo '{}' > .savepoint/name.json -- empty JSON object, name from PK + data := []byte(`{}`) + fsErr := ops.WriteFile(context.Background(), "/notes/.savepoint/quick-mark.json", data) + require.Nil(t, fsErr) + + m := insertedColMap(t, mockDB) + assert.Equal(t, "quick-mark", m["name"]) +} + +// -- TSV format tests -- + +func TestSynth_Savepoint_TSV_NameAndDescription(t *testing.T) { + mockDB := newSavepointMock() + ops := NewOperations(&config.Config{DirListingLimit: 1000}, mockDB) + + data := []byte("description\nBefore refactoring\n") + fsErr := ops.WriteFile(context.Background(), "/notes/.savepoint/my-checkpoint.tsv", data) + require.Nil(t, fsErr) + + m := insertedColMap(t, mockDB) + assert.Equal(t, "my-checkpoint", m["name"]) + assert.Equal(t, "Before refactoring", m["description"]) +} + +func TestSynth_Savepoint_TSV_NameDescriptionUserID(t *testing.T) { + mockDB := newSavepointMock() + ops := NewOperations(&config.Config{DirListingLimit: 1000}, mockDB) + + data := []byte("description\tuser_id\nBefore refactoring\tagent-9\n") + fsErr := ops.WriteFile(context.Background(), "/notes/.savepoint/my-checkpoint.tsv", data) + require.Nil(t, fsErr) + + m := insertedColMap(t, mockDB) + assert.Equal(t, "my-checkpoint", m["name"]) + assert.Equal(t, "Before refactoring", m["description"]) + assert.Equal(t, "agent-9", m["user_id"]) +} + +func TestSynth_Savepoint_TSV_InjectsUserID(t *testing.T) { + mockDB := newSavepointMock() + ops := NewOperations(&config.Config{DirListingLimit: 1000}, mockDB) + ops.SetUserID("agent-7") + + // user_id NOT in body -- should be auto-injected from mount identity + data := []byte("description\nBefore refactoring\n") + fsErr := ops.WriteFile(context.Background(), "/notes/.savepoint/my-checkpoint.tsv", data) + require.Nil(t, fsErr) + + m := insertedColMap(t, mockDB) + assert.Equal(t, "agent-7", m["user_id"]) + assert.Equal(t, "Before refactoring", m["description"]) + assert.Equal(t, "my-checkpoint", m["name"]) +} + +func TestSynth_Savepoint_TSV_ExplicitUserIDNotOverridden(t *testing.T) { + mockDB := newSavepointMock() + ops := NewOperations(&config.Config{DirListingLimit: 1000}, mockDB) + ops.SetUserID("agent-7") + + // user_id in body should NOT be overridden by mount identity + data := []byte("description\tuser_id\nBefore refactoring\tagent-9\n") + fsErr := ops.WriteFile(context.Background(), "/notes/.savepoint/my-checkpoint.tsv", data) + require.Nil(t, fsErr) + + m := insertedColMap(t, mockDB) + assert.Equal(t, "agent-9", m["user_id"], "explicit user_id should not be overridden") +} + +// -- JSON format tests -- + +func TestSynth_Savepoint_JSON_NameOnly(t *testing.T) { + mockDB := newSavepointMock() + ops := NewOperations(&config.Config{DirListingLimit: 1000}, mockDB) + + data := []byte(`{}`) + fsErr := ops.WriteFile(context.Background(), "/notes/.savepoint/quick-mark.json", data) + require.Nil(t, fsErr) + + m := insertedColMap(t, mockDB) + assert.Equal(t, "quick-mark", m["name"]) +} + +func TestSynth_Savepoint_JSON_NameAndDescription(t *testing.T) { + mockDB := newSavepointMock() + ops := NewOperations(&config.Config{DirListingLimit: 1000}, mockDB) + + data := []byte(`{"description":"Before refactoring"}`) + fsErr := ops.WriteFile(context.Background(), "/notes/.savepoint/my-checkpoint.json", data) + require.Nil(t, fsErr) + + m := insertedColMap(t, mockDB) + assert.Equal(t, "my-checkpoint", m["name"]) + assert.Equal(t, "Before refactoring", m["description"]) +} + +func TestSynth_Savepoint_JSON_NameDescriptionUserID(t *testing.T) { + mockDB := newSavepointMock() + ops := NewOperations(&config.Config{DirListingLimit: 1000}, mockDB) + + data := []byte(`{"description":"Before refactoring","user_id":"agent-9"}`) + fsErr := ops.WriteFile(context.Background(), "/notes/.savepoint/my-checkpoint.json", data) + require.Nil(t, fsErr) + + m := insertedColMap(t, mockDB) + assert.Equal(t, "my-checkpoint", m["name"]) + assert.Equal(t, "Before refactoring", m["description"]) + assert.Equal(t, "agent-9", m["user_id"]) +} + +func TestSynth_Savepoint_JSON_InjectsUserID(t *testing.T) { + mockDB := newSavepointMock() + ops := NewOperations(&config.Config{DirListingLimit: 1000}, mockDB) + ops.SetUserID("agent-7") + + data := []byte(`{"description":"Before refactoring"}`) + fsErr := ops.WriteFile(context.Background(), "/notes/.savepoint/my-checkpoint.json", data) + require.Nil(t, fsErr) + + m := insertedColMap(t, mockDB) + assert.Equal(t, "agent-7", m["user_id"]) +} + +// -- CSV format tests -- + +func TestSynth_Savepoint_CSV_NameAndDescription(t *testing.T) { + mockDB := newSavepointMock() + ops := NewOperations(&config.Config{DirListingLimit: 1000}, mockDB) + + data := []byte("description\nBefore refactoring\n") + fsErr := ops.WriteFile(context.Background(), "/notes/.savepoint/my-checkpoint.csv", data) + require.Nil(t, fsErr) + + m := insertedColMap(t, mockDB) + assert.Equal(t, "my-checkpoint", m["name"]) + assert.Equal(t, "Before refactoring", m["description"]) +} + +func TestSynth_Savepoint_CSV_InjectsUserID(t *testing.T) { + mockDB := newSavepointMock() + ops := NewOperations(&config.Config{DirListingLimit: 1000}, mockDB) + ops.SetUserID("agent-7") + + data := []byte("description\nBefore refactoring\n") + fsErr := ops.WriteFile(context.Background(), "/notes/.savepoint/my-checkpoint.csv", data) + require.Nil(t, fsErr) + + m := insertedColMap(t, mockDB) + assert.Equal(t, "agent-7", m["user_id"]) + assert.Equal(t, "my-checkpoint", m["name"]) +} + +// -- YAML format tests -- + +func TestSynth_Savepoint_YAML_NameAndDescription(t *testing.T) { + mockDB := newSavepointMock() + ops := NewOperations(&config.Config{DirListingLimit: 1000}, mockDB) + + data := []byte("description: Before refactoring\n") + fsErr := ops.WriteFile(context.Background(), "/notes/.savepoint/my-checkpoint.yaml", data) + require.Nil(t, fsErr) + + m := insertedColMap(t, mockDB) + assert.Equal(t, "my-checkpoint", m["name"]) + assert.Equal(t, "Before refactoring", m["description"]) +} + +func TestSynth_Savepoint_YAML_InjectsUserID(t *testing.T) { + mockDB := newSavepointMock() + ops := NewOperations(&config.Config{DirListingLimit: 1000}, mockDB) + ops.SetUserID("agent-7") + + data := []byte("description: Before refactoring\n") + fsErr := ops.WriteFile(context.Background(), "/notes/.savepoint/my-checkpoint.yaml", data) + require.Nil(t, fsErr) + + m := insertedColMap(t, mockDB) + assert.Equal(t, "agent-7", m["user_id"]) + assert.Equal(t, "my-checkpoint", m["name"]) +} + +func TestSynth_Savepoint_YAML_EmptyBody(t *testing.T) { + mockDB := newSavepointMock() + ops := NewOperations(&config.Config{DirListingLimit: 1000}, mockDB) + ops.SetUserID("agent-7") + + data := []byte("\n") + fsErr := ops.WriteFile(context.Background(), "/notes/.savepoint/quick-mark.yaml", data) + require.Nil(t, fsErr) + + m := insertedColMap(t, mockDB) + assert.Equal(t, "quick-mark", m["name"]) + assert.Equal(t, "agent-7", m["user_id"]) +} + +// -- Format file path parsing -- + +func TestSynth_Savepoint_ParsePath_FormatFile(t *testing.T) { + // cat .savepoint/name/.json should parse as PathRow with Format="json" + for _, tc := range []struct { + path, format string + }{ + {"/notes/.savepoint/before-exploration/.json", "json"}, + {"/notes/.savepoint/before-exploration/.tsv", "tsv"}, + {"/notes/.savepoint/before-exploration/.csv", "csv"}, + {"/notes/.savepoint/before-exploration/.yaml", "yaml"}, + } { + parsed, err := ParsePath(tc.path) + require.Nil(t, err, "ParsePath(%s)", tc.path) + assert.Equal(t, PathRow, parsed.Type, "should be PathRow for %s", tc.path) + assert.Equal(t, "before-exploration", parsed.PrimaryKey, "PK for %s", tc.path) + assert.Equal(t, tc.format, parsed.Format, "Format for %s", tc.path) + } +} + +// -- Duplicate name -- + +func TestSynth_Savepoint_DuplicateName(t *testing.T) { + mockDB := newSavepointMock() + // First insert succeeds + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + + data := []byte(`{"description":"first"}`) + fsErr := ops.WriteFile(context.Background(), "/notes/.savepoint/my-save.json", data) + require.Nil(t, fsErr) + + // Second insert with same name -- row exists, so writeRowFile does UPDATE + mockDB.rowData = map[string]*mockRow{ + "tigerfs.notes_savepoint.my-save": { + columns: []string{"name", "savepoint_id", "user_id", "description"}, + values: []interface{}{"my-save", "sp-uuid-1", nil, "first"}, + }, + } + data = []byte(`{"description":"updated"}`) + fsErr = ops.WriteFile(context.Background(), "/notes/.savepoint/my-save.json", data) + require.Nil(t, fsErr, "writing to existing savepoint should update, not fail") + assert.True(t, mockDB.updateCalled, "should have called UpdateRow") +} + +// -- Bare path rejection -- + +func TestSynth_Savepoint_BarePathRejected(t *testing.T) { + mockDB := newSavepointMockWithColumns() + ops := NewOperations(&config.Config{DirListingLimit: 1000}, mockDB) + + data := []byte("Before refactoring\n") + fsErr := ops.WriteFile(context.Background(), "/notes/.savepoint/my-checkpoint", data) + require.NotNil(t, fsErr) + assert.Equal(t, ErrInvalidArgument, fsErr.Code) + assert.Contains(t, fsErr.Message, "without a format suffix") + assert.Contains(t, fsErr.Hint, "my-checkpoint.json", "hint should suggest suffixed alternatives") +} diff --git a/internal/tigerfs/fs/synth/build.go b/internal/tigerfs/fs/synth/build.go index 6a14289..9cec22d 100644 --- a/internal/tigerfs/fs/synth/build.go +++ b/internal/tigerfs/fs/synth/build.go @@ -24,8 +24,10 @@ const TigerFSSchema = "tigerfs" // - created_at: timestamptz with auto-default // - modified_at: timestamptz with auto-default func GenerateMarkdownTableSQL(schema, name string) string { - return fmt.Sprintf(`CREATE TABLE %s.%s ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + qualifiedTable := fmt.Sprintf("%s.%s", db.QuoteIdent(TigerFSSchema), db.QuoteIdent(name)) + return fmt.Sprintf(`CREATE TABLE %s ( + id UUID PRIMARY KEY DEFAULT uuidv7(), + parent_id UUID REFERENCES %s(id) DEFERRABLE INITIALLY IMMEDIATE, filename TEXT NOT NULL, filetype TEXT NOT NULL DEFAULT 'file' CHECK (filetype IN ('file', 'directory')), title TEXT, @@ -35,8 +37,8 @@ func GenerateMarkdownTableSQL(schema, name string) string { encoding TEXT NOT NULL DEFAULT 'utf8' CHECK (encoding IN ('utf8', 'base64')), created_at TIMESTAMPTZ NOT NULL DEFAULT now(), modified_at TIMESTAMPTZ NOT NULL DEFAULT now(), - UNIQUE(filename, filetype) -)`, db.QuoteIdent(TigerFSSchema), db.QuoteIdent(name)) + UNIQUE NULLS NOT DISTINCT (parent_id, filename, filetype) DEFERRABLE INITIALLY IMMEDIATE +)`, qualifiedTable, qualifiedTable) } // GeneratePlainTextTableSQL returns the CREATE TABLE statement for a plain text app. @@ -51,16 +53,18 @@ func GenerateMarkdownTableSQL(schema, name string) string { // - created_at: timestamptz with auto-default // - modified_at: timestamptz with auto-default func GeneratePlainTextTableSQL(schema, name string) string { - return fmt.Sprintf(`CREATE TABLE %s.%s ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + qualifiedTable := fmt.Sprintf("%s.%s", db.QuoteIdent(TigerFSSchema), db.QuoteIdent(name)) + return fmt.Sprintf(`CREATE TABLE %s ( + id UUID PRIMARY KEY DEFAULT uuidv7(), + parent_id UUID REFERENCES %s(id) DEFERRABLE INITIALLY IMMEDIATE, filename TEXT NOT NULL, filetype TEXT NOT NULL DEFAULT 'file' CHECK (filetype IN ('file', 'directory')), body TEXT, encoding TEXT NOT NULL DEFAULT 'utf8' CHECK (encoding IN ('utf8', 'base64')), created_at TIMESTAMPTZ NOT NULL DEFAULT now(), modified_at TIMESTAMPTZ NOT NULL DEFAULT now(), - UNIQUE(filename, filetype) -)`, db.QuoteIdent(TigerFSSchema), db.QuoteIdent(name)) + UNIQUE NULLS NOT DISTINCT (parent_id, filename, filetype) DEFERRABLE INITIALLY IMMEDIATE +)`, qualifiedTable, qualifiedTable) } // GenerateViewSQL returns a CREATE VIEW statement that selects all columns @@ -121,6 +125,59 @@ $$ LANGUAGE plpgsql`, funcName) return []string{createFunc, createTrigger} } +// GenerateParentDirMtimeTriggerSQL returns two SQL statements to create a trigger +// function and trigger that updates the parent directory's modified_at when +// children are added, removed, or moved. This gives directories POSIX-correct +// mtime semantics: mtime changes on child create/delete/rename, not on content edits. +// +// The trigger is AFTER (side-effect on a different row) and uses column-level +// filtering (UPDATE OF parent_id, filename) so content-only edits never fire it. +// No recursion risk: the UPDATE to the parent only changes modified_at, which is +// not in the column filter list. +func GenerateParentDirMtimeTriggerSQL(schema, tableName string) []string { + funcName := fmt.Sprintf("%s.%s", db.QuoteIdent(TigerFSSchema), db.QuoteIdent("bump_"+tableName+"_parent_mtime")) + triggerName := db.QuoteIdent("trg_" + tableName + "_parent_mtime") + qualifiedTable := fmt.Sprintf("%s.%s", db.QuoteIdent(TigerFSSchema), db.QuoteIdent(tableName)) + + createFunc := fmt.Sprintf(`CREATE OR REPLACE FUNCTION %s() +RETURNS TRIGGER AS $$ +BEGIN + IF TG_OP = 'INSERT' THEN + IF NEW.parent_id IS NOT NULL THEN + UPDATE %s SET modified_at = now() + WHERE id = NEW.parent_id AND filetype = 'directory'; + END IF; + ELSIF TG_OP = 'DELETE' THEN + IF OLD.parent_id IS NOT NULL THEN + UPDATE %s SET modified_at = now() + WHERE id = OLD.parent_id AND filetype = 'directory'; + END IF; + ELSIF TG_OP = 'UPDATE' THEN + IF OLD.parent_id IS DISTINCT FROM NEW.parent_id + OR OLD.filename IS DISTINCT FROM NEW.filename THEN + IF OLD.parent_id IS NOT NULL + AND OLD.parent_id IS DISTINCT FROM NEW.parent_id THEN + UPDATE %s SET modified_at = now() + WHERE id = OLD.parent_id AND filetype = 'directory'; + END IF; + IF NEW.parent_id IS NOT NULL THEN + UPDATE %s SET modified_at = now() + WHERE id = NEW.parent_id AND filetype = 'directory'; + END IF; + END IF; + END IF; + RETURN NULL; +END; +$$ LANGUAGE plpgsql`, funcName, qualifiedTable, qualifiedTable, qualifiedTable, qualifiedTable) + + createTrigger := fmt.Sprintf(`CREATE TRIGGER %s + AFTER INSERT OR DELETE OR UPDATE OF parent_id, filename ON %s + FOR EACH ROW EXECUTE FUNCTION %s()`, + triggerName, qualifiedTable, funcName) + + return []string{createFunc, createTrigger} +} + // GenerateBuildSQL returns the complete SQL statements to create a synthesized app. // This includes the backing table, view, view comment, and modified_at trigger. // Returns a slice of individual statements (not delimited) because the trigger @@ -129,11 +186,60 @@ func GenerateBuildSQL(schema, appName string, format SynthFormat) ([]string, err return GenerateBuildSQLWithFeatures(schema, appName, FeatureSet{Format: format}) } +// GenerateResolvePathSQL returns a CREATE OR REPLACE FUNCTION statement for the +// tigerfs.resolve_path PL/pgSQL function. This function resolves a sequence of +// path segments to row IDs using the parent-pointer model (ADR-017). +// +// Parameters: +// - tbl: REGCLASS reference to the source table +// - start_parent: UUID of the starting parent (NULL for root) +// - segments: TEXT[] array of path segments to resolve +// +// Returns a table of (depth, resolved_id, resolved_name) rows, one per resolved +// segment. The Go layer populates its path cache from these results. If any segment +// doesn't resolve, the function returns fewer rows than segments. +func GenerateResolvePathSQL() string { + return fmt.Sprintf(`CREATE OR REPLACE FUNCTION %s.resolve_path( + tbl REGCLASS, + start_parent UUID, + segments TEXT[] +) +RETURNS TABLE(depth INT, resolved_id UUID, resolved_name TEXT) AS $$ +DECLARE + current_id UUID := start_parent; + i INT := 0; + seg TEXT; +BEGIN + FOREACH seg IN ARRAY segments LOOP + i := i + 1; + EXECUTE format('SELECT id FROM %%s WHERE filename = $1 AND parent_id IS NOT DISTINCT FROM $2', tbl) + INTO current_id + USING seg, current_id; + IF current_id IS NULL THEN RETURN; END IF; + depth := i; + resolved_id := current_id; + resolved_name := seg; + RETURN NEXT; + END LOOP; +END; +$$ LANGUAGE plpgsql`, db.QuoteIdent(TigerFSSchema)) +} + +// GenerateParentIndexSQL returns a CREATE INDEX statement for the parent_id + filename +// index. This index supports ReadDir (WHERE parent_id = X) and path resolution lookups. +func GenerateParentIndexSQL(appName string) string { + qualifiedTable := fmt.Sprintf("%s.%s", db.QuoteIdent(TigerFSSchema), db.QuoteIdent(appName)) + return fmt.Sprintf(`CREATE INDEX %s ON %s (parent_id, filename)`, + db.QuoteIdent("idx_"+appName+"_parent"), + qualifiedTable, + ) +} + // GenerateBuildSQLWithFeatures returns SQL statements to create a synthesized app -// with optional features like versioned history. The first statement creates the -// tigerfs schema. The backing table, triggers, and functions are in the tigerfs -// schema; the view is in the user's schema. When features.History is true, -// appends history table, trigger, hypertable, and compression statements. +// with optional features like versioned history. The first statements create the +// tigerfs schema and the resolve_path function. The backing table, triggers, and +// functions are in the tigerfs schema; the view is in the user's schema. When +// features.History is true, appends history hypertable, trigger, log, and savepoint. func GenerateBuildSQLWithFeatures(schema, appName string, features FeatureSet) ([]string, error) { var tableSQL string switch features.Format { @@ -146,12 +252,17 @@ func GenerateBuildSQLWithFeatures(schema, appName string, features FeatureSet) ( } createSchema := fmt.Sprintf(`CREATE SCHEMA IF NOT EXISTS %s`, db.QuoteIdent(TigerFSSchema)) + resolvePathFunc := GenerateResolvePathSQL() + parentIndex := GenerateParentIndexSQL(appName) viewSQL := GenerateViewSQL(schema, appName, TigerFSSchema, appName) commentSQL := GenerateViewCommentSQLWithFeatures(schema, appName, features) triggerStmts := GenerateModifiedAtTriggerSQL(schema, appName) - stmts := []string{createSchema, tableSQL, viewSQL, commentSQL} + parentMtimeStmts := GenerateParentDirMtimeTriggerSQL(schema, appName) + + stmts := []string{createSchema, resolvePathFunc, tableSQL, parentIndex, viewSQL, commentSQL} stmts = append(stmts, triggerStmts...) + stmts = append(stmts, parentMtimeStmts...) if features.History { historyStmts := GenerateHistorySQL(schema, appName, features.Format) @@ -165,15 +276,15 @@ func GenerateBuildSQLWithFeatures(schema, appName string, features FeatureSet) ( // infrastructure for an existing synth app. All history infrastructure // (table, indexes, functions, triggers) lives in the tigerfs schema. // This includes: -// - History table mirroring the source table columns plus metadata -// - Index on (filename, _history_id DESC) for by-filename queries -// - Index on (id, _history_id DESC) for by-UUID queries +// - History hypertable with columnstore (file_id, parent_id, version_id, operation) +// - Index on (filename, version_id DESC) for by-filename queries +// - Index on (file_id, version_id DESC) for by-UUID queries // - Archive trigger function and BEFORE UPDATE OR DELETE trigger -// - TimescaleDB hypertable conversion -// - Compression policy +// - Log hypertable for undo operations (create/edit/rename/delete/undo) +// - Savepoint table for named bookmarks // // The column list varies by format: markdown includes title, author, headers; -// plain text only has the base columns (id, filename, filetype, body, etc.). +// plain text only has the base columns (file_id, filename, filetype, body, etc.). // The archive trigger must match the source table's actual columns. func GenerateHistorySQL(schema, appName string, format SynthFormat) []string { tableName := appName @@ -191,51 +302,71 @@ func GenerateHistorySQL(schema, appName string, format SynthFormat) []string { formatOldValues = "OLD.title, OLD.author, OLD.headers, " } + // History table uses modern CREATE TABLE WITH syntax for hypertable + columnstore. + // This replaces the old create_hypertable() + ALTER TABLE SET + add_compression_policy() calls. + // Column renames from ADR-017: id->file_id, _history_id->version_id, _operation->operation. + // Added: parent_id, CHECK constraints on filetype/encoding/operation. createTable := fmt.Sprintf(`CREATE TABLE %s ( - id UUID, + file_id UUID, + parent_id UUID, filename TEXT NOT NULL, - filetype TEXT,%s + filetype TEXT CHECK (filetype IN ('file', 'directory')),%s body TEXT, - encoding TEXT, + encoding TEXT CHECK (encoding IN ('utf8', 'base64')), created_at TIMESTAMPTZ, modified_at TIMESTAMPTZ, - _history_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY, - _operation TEXT NOT NULL + version_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY, + operation TEXT NOT NULL CHECK (operation IN ('edit', 'rename', 'delete')) +) WITH ( + tsdb.hypertable, + tsdb.partition_column = 'version_id', + tsdb.chunk_interval = '7 days', + tsdb.segmentby = 'file_id', + tsdb.orderby = 'version_id DESC' )`, qualifiedHistory, formatColumns) createIndexFilename := fmt.Sprintf( - `CREATE INDEX %s ON %s (filename, _history_id DESC)`, + `CREATE INDEX %s ON %s (filename, version_id DESC)`, db.QuoteIdent("idx_"+historyTable+"_by_filename"), qualifiedHistory, ) createIndexID := fmt.Sprintf( - `CREATE INDEX %s ON %s (id, _history_id DESC)`, + `CREATE INDEX %s ON %s (file_id, version_id DESC)`, db.QuoteIdent("idx_"+historyTable+"_by_id"), qualifiedHistory, ) - // Archive trigger function — copies OLD row to history table on UPDATE or DELETE. - // Column list must match the source table's actual columns. + // Archive trigger function -- copies OLD row (including parent_id) to history + // table on UPDATE or DELETE. Column list must match the source table's columns. funcName := fmt.Sprintf("%s.%s", db.QuoteIdent(TigerFSSchema), db.QuoteIdent("archive_"+historyTable)) var insertColumns, insertValues string if format == FormatMarkdown { - insertColumns = "id, filename, filetype, title, author, headers, body, encoding, created_at, modified_at" - insertValues = fmt.Sprintf("OLD.id, OLD.filename, OLD.filetype, %sOLD.body,\n OLD.encoding, OLD.created_at, OLD.modified_at", formatOldValues) + insertColumns = "file_id, parent_id, filename, filetype, title, author, headers, body, encoding, created_at, modified_at" + insertValues = fmt.Sprintf("OLD.id, OLD.parent_id, OLD.filename, OLD.filetype, %sOLD.body,\n OLD.encoding, OLD.created_at, OLD.modified_at", formatOldValues) } else { - insertColumns = "id, filename, filetype, body, encoding, created_at, modified_at" - insertValues = "OLD.id, OLD.filename, OLD.filetype, OLD.body,\n OLD.encoding, OLD.created_at, OLD.modified_at" + insertColumns = "file_id, parent_id, filename, filetype, body, encoding, created_at, modified_at" + insertValues = "OLD.id, OLD.parent_id, OLD.filename, OLD.filetype, OLD.body,\n OLD.encoding, OLD.created_at, OLD.modified_at" } createFunc := fmt.Sprintf(`CREATE OR REPLACE FUNCTION %s() RETURNS TRIGGER AS $$ BEGIN INSERT INTO %s (%s, - _history_id, _operation) + version_id, operation) VALUES (%s, - uuidv7(), TG_OP::text); + uuidv7(), + CASE TG_OP + WHEN 'DELETE' THEN 'delete' + WHEN 'UPDATE' THEN + CASE WHEN OLD.filename != NEW.filename + OR OLD.parent_id IS DISTINCT FROM NEW.parent_id + THEN 'rename' + ELSE 'edit' + END + END); IF TG_OP = 'DELETE' THEN RETURN OLD; END IF; @@ -249,32 +380,65 @@ $$ LANGUAGE plpgsql`, funcName, qualifiedHistory, insertColumns, insertValues) FOR EACH ROW EXECUTE FUNCTION %s()`, triggerName, qualifiedTable, funcName) - // TimescaleDB hypertable conversion — UUIDv7 as partition column - createHypertable := fmt.Sprintf( - `SELECT create_hypertable('%s.%s', '_history_id', chunk_time_interval => INTERVAL '1 month')`, - TigerFSSchema, historyTable, + // --- Operation log table (ADR-016 Section 1, ADR-017 updates) --- + // Records every data change for undo operations. Uses UUIDv7 PKs for + // time-ordered entries and SkipScan-optimized queries. + // Column renames: history_id->version_id. Type values are filesystem-centric: + // create/edit/rename/delete/undo (replaces insert/update/delete/undo). + logTable := appName + "_log" + qualifiedLog := fmt.Sprintf("%s.%s", db.QuoteIdent(TigerFSSchema), db.QuoteIdent(logTable)) + + // Log table with hypertable + columnstore settings inline via CREATE TABLE WITH. + // TimescaleDB automatically creates a columnstore policy using the chunk interval + // as the compression interval (no explicit compress_after needed). + createLogTable := fmt.Sprintf(`CREATE TABLE %s ( + log_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY, + file_id UUID NOT NULL, + type TEXT NOT NULL CHECK (type IN ('create', 'edit', 'rename', 'delete', 'undo')), + user_id TEXT, + filename TEXT NOT NULL, + version_id UUID, + description TEXT +) WITH ( + tsdb.hypertable, + tsdb.partition_column = 'log_id', + tsdb.chunk_interval = '7 days', + tsdb.segmentby = 'file_id', + tsdb.orderby = 'log_id ASC' +)`, qualifiedLog) + + // Composite index for SkipScan on the undo DISTINCT ON query + createLogIndex := fmt.Sprintf( + `CREATE INDEX %s ON %s (file_id, log_id ASC)`, + db.QuoteIdent("idx_"+logTable+"_by_file"), + qualifiedLog, ) - // Compression policy — segment by filename, order by _history_id DESC - setCompression := fmt.Sprintf( - `ALTER TABLE %s SET (timescaledb.compress, timescaledb.compress_segmentby = 'filename', timescaledb.compress_orderby = '_history_id DESC')`, - qualifiedHistory, - ) + // --- Savepoint table (ADR-016 Section 2) --- + // Named bookmarks for undo-to-savepoint operations. Regular table (not a + // hypertable) -- savepoints are small and don't need time-series features. + savepointTable := appName + "_savepoint" + qualifiedSavepoint := fmt.Sprintf("%s.%s", db.QuoteIdent(TigerFSSchema), db.QuoteIdent(savepointTable)) - addCompressionPolicy := fmt.Sprintf( - `SELECT add_compression_policy('%s.%s', compress_after => INTERVAL '1 day')`, - TigerFSSchema, historyTable, - ) + createSavepointTable := fmt.Sprintf(`CREATE TABLE %s ( + name TEXT NOT NULL PRIMARY KEY, + savepoint_id UUID NOT NULL DEFAULT uuidv7() UNIQUE, + user_id TEXT, + description TEXT +)`, qualifiedSavepoint) return []string{ + // History infrastructure (table is hypertable via WITH clause) createTable, createIndexFilename, createIndexID, createFunc, createTrigger, - createHypertable, - setCompression, - addCompressionPolicy, + // Undo log infrastructure + createLogTable, + createLogIndex, + // Savepoint infrastructure + createSavepointTable, } } diff --git a/internal/tigerfs/fs/synth/build_test.go b/internal/tigerfs/fs/synth/build_test.go index f343153..fb26203 100644 --- a/internal/tigerfs/fs/synth/build_test.go +++ b/internal/tigerfs/fs/synth/build_test.go @@ -16,14 +16,17 @@ func TestGenerateMarkdownTableSQL(t *testing.T) { if !strings.Contains(sql, "id UUID PRIMARY KEY") { t.Errorf("should have UUID primary key, got:\n%s", sql) } + if !strings.Contains(sql, `parent_id UUID REFERENCES "tigerfs"."posts"(id) DEFERRABLE INITIALLY IMMEDIATE`) { + t.Errorf("should have parent_id FK with DEFERRABLE, got:\n%s", sql) + } if !strings.Contains(sql, "filename TEXT NOT NULL") { t.Errorf("should have filename column, got:\n%s", sql) } if !strings.Contains(sql, "filetype TEXT NOT NULL DEFAULT 'file'") { t.Errorf("should have filetype column, got:\n%s", sql) } - if !strings.Contains(sql, "UNIQUE(filename, filetype)") { - t.Errorf("should have compound UNIQUE constraint, got:\n%s", sql) + if !strings.Contains(sql, "UNIQUE NULLS NOT DISTINCT (parent_id, filename, filetype) DEFERRABLE INITIALLY IMMEDIATE") { + t.Errorf("should have UNIQUE NULLS NOT DISTINCT constraint with DEFERRABLE, got:\n%s", sql) } if !strings.Contains(sql, "title TEXT") { t.Errorf("should have title column, got:\n%s", sql) @@ -54,14 +57,17 @@ func TestGeneratePlainTextTableSQL(t *testing.T) { if !strings.Contains(sql, `"tigerfs"."snippets"`) { t.Errorf("should reference tigerfs.snippets table, got:\n%s", sql) } + if !strings.Contains(sql, `parent_id UUID REFERENCES "tigerfs"."snippets"(id) DEFERRABLE INITIALLY IMMEDIATE`) { + t.Errorf("should have parent_id FK with DEFERRABLE, got:\n%s", sql) + } if !strings.Contains(sql, "filename TEXT NOT NULL") { t.Errorf("should have filename column, got:\n%s", sql) } if !strings.Contains(sql, "filetype TEXT NOT NULL DEFAULT 'file'") { t.Errorf("should have filetype column, got:\n%s", sql) } - if !strings.Contains(sql, "UNIQUE(filename, filetype)") { - t.Errorf("should have compound UNIQUE constraint, got:\n%s", sql) + if !strings.Contains(sql, "UNIQUE NULLS NOT DISTINCT (parent_id, filename, filetype) DEFERRABLE INITIALLY IMMEDIATE") { + t.Errorf("should have UNIQUE NULLS NOT DISTINCT constraint with DEFERRABLE, got:\n%s", sql) } if !strings.Contains(sql, "body TEXT") { t.Errorf("should have body column, got:\n%s", sql) @@ -141,6 +147,71 @@ func TestGenerateModifiedAtTriggerSQL(t *testing.T) { } } +func TestGenerateParentDirMtimeTriggerSQL(t *testing.T) { + stmts := GenerateParentDirMtimeTriggerSQL("public", "posts") + if len(stmts) != 2 { + t.Fatalf("expected 2 statements, got %d", len(stmts)) + } + + funcSQL := stmts[0] + triggerSQL := stmts[1] + + // Function + if !strings.Contains(funcSQL, "CREATE OR REPLACE FUNCTION") { + t.Errorf("should create function, got:\n%s", funcSQL) + } + if !strings.Contains(funcSQL, `"tigerfs"."bump_posts_parent_mtime"`) { + t.Errorf("function should be in tigerfs schema with correct name, got:\n%s", funcSQL) + } + if !strings.Contains(funcSQL, "RETURNS TRIGGER") { + t.Errorf("should return trigger, got:\n%s", funcSQL) + } + if !strings.Contains(funcSQL, "RETURN NULL") { + t.Errorf("AFTER trigger must return NULL, got:\n%s", funcSQL) + } + // Should handle all three operations + if !strings.Contains(funcSQL, "TG_OP = 'INSERT'") { + t.Errorf("should handle INSERT, got:\n%s", funcSQL) + } + if !strings.Contains(funcSQL, "TG_OP = 'DELETE'") { + t.Errorf("should handle DELETE, got:\n%s", funcSQL) + } + if !strings.Contains(funcSQL, "TG_OP = 'UPDATE'") { + t.Errorf("should handle UPDATE, got:\n%s", funcSQL) + } + // Should reference parent_id and filetype + if !strings.Contains(funcSQL, "NEW.parent_id") { + t.Errorf("should reference NEW.parent_id, got:\n%s", funcSQL) + } + if !strings.Contains(funcSQL, "OLD.parent_id") { + t.Errorf("should reference OLD.parent_id, got:\n%s", funcSQL) + } + if !strings.Contains(funcSQL, "filetype = 'directory'") { + t.Errorf("should guard against non-directory parents, got:\n%s", funcSQL) + } + // Should use IS DISTINCT FROM for NULL-safe comparison + if !strings.Contains(funcSQL, "IS DISTINCT FROM") { + t.Errorf("should use IS DISTINCT FROM for NULL-safe comparison, got:\n%s", funcSQL) + } + + // Trigger + if !strings.Contains(triggerSQL, "CREATE TRIGGER") { + t.Errorf("should create trigger, got:\n%s", triggerSQL) + } + if !strings.Contains(triggerSQL, `"trg_posts_parent_mtime"`) { + t.Errorf("trigger name should use correct pattern, got:\n%s", triggerSQL) + } + if !strings.Contains(triggerSQL, "AFTER INSERT OR DELETE OR UPDATE OF parent_id, filename") { + t.Errorf("should be AFTER trigger with column filter, got:\n%s", triggerSQL) + } + if !strings.Contains(triggerSQL, "FOR EACH ROW") { + t.Errorf("should be row-level trigger, got:\n%s", triggerSQL) + } + if !strings.Contains(triggerSQL, `"tigerfs"."posts"`) { + t.Errorf("trigger should reference tigerfs.posts, got:\n%s", triggerSQL) + } +} + func TestGenerateBuildSQL_Markdown(t *testing.T) { stmts, err := GenerateBuildSQL("public", "posts", FormatMarkdown) if err != nil { @@ -153,6 +224,10 @@ func TestGenerateBuildSQL_Markdown(t *testing.T) { if !strings.Contains(stmts[0], `CREATE SCHEMA IF NOT EXISTS "tigerfs"`) { t.Errorf("first statement should create tigerfs schema, got: %s", stmts[0]) } + // Second statement should create resolve_path function + if !strings.Contains(stmts[1], "resolve_path") { + t.Errorf("second statement should create resolve_path function, got: %s", stmts[1]) + } // Should contain all parts if !strings.Contains(allSQL, "CREATE TABLE") { t.Errorf("should contain CREATE TABLE, got:\n%s", allSQL) @@ -176,9 +251,26 @@ func TestGenerateBuildSQL_Markdown(t *testing.T) { if !strings.Contains(allSQL, `"public"."posts" AS SELECT * FROM "tigerfs"."posts"`) { t.Errorf("view should be in public schema referencing tigerfs, got:\n%s", allSQL) } - // Should have 6 statements: schema, table, view, comment, function, trigger - if len(stmts) != 6 { - t.Errorf("expected 6 statements, got %d", len(stmts)) + // Should have 10 statements: schema, resolve_path, table, parent_index, view, comment, + // modified_at function + trigger, parent_mtime function + trigger + if len(stmts) != 10 { + t.Errorf("expected 10 statements, got %d", len(stmts)) + } + + // Parent index for ReadDir and path resolution + if !strings.Contains(allSQL, `"idx_posts_parent"`) { + t.Errorf("should create parent index, got:\n%s", allSQL) + } + if !strings.Contains(allSQL, "(parent_id, filename)") { + t.Errorf("parent index should be on (parent_id, filename), got:\n%s", allSQL) + } + + // Non-history apps should NOT have log or savepoint tables + if strings.Contains(allSQL, "_log") { + t.Errorf("non-history app should not have log table, got:\n%s", allSQL) + } + if strings.Contains(allSQL, "_savepoint") { + t.Errorf("non-history app should not have savepoint table, got:\n%s", allSQL) } } @@ -214,24 +306,60 @@ func TestSynth_GenerateHistorySQL_Markdown(t *testing.T) { if !strings.Contains(allSQL, `"tigerfs"."memory_history"`) { t.Errorf("should reference tigerfs.memory_history table, got:\n%s", allSQL) } - if !strings.Contains(allSQL, "_history_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY") { - t.Errorf("should have _history_id column, got:\n%s", allSQL) + + // ADR-017 column renames: id->file_id, _history_id->version_id, _operation->operation + if !strings.Contains(allSQL, "file_id UUID,") { + t.Errorf("history table should have file_id column (renamed from id), got:\n%s", allSQL) + } + if !strings.Contains(allSQL, "parent_id UUID,") { + t.Errorf("history table should have parent_id column, got:\n%s", allSQL) + } + if !strings.Contains(allSQL, "version_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY") { + t.Errorf("should have version_id column (renamed from _history_id), got:\n%s", allSQL) + } + if !strings.Contains(allSQL, "operation TEXT NOT NULL") { + t.Errorf("should have operation column (renamed from _operation), got:\n%s", allSQL) + } + + // CHECK constraints on filetype, encoding, operation + if !strings.Contains(allSQL, "filetype TEXT CHECK (filetype IN ('file', 'directory'))") { + t.Errorf("history should have filetype CHECK constraint, got:\n%s", allSQL) + } + if !strings.Contains(allSQL, "encoding TEXT CHECK (encoding IN ('utf8', 'base64'))") { + t.Errorf("history should have encoding CHECK constraint, got:\n%s", allSQL) } - if !strings.Contains(allSQL, "_operation TEXT NOT NULL") { - t.Errorf("should have _operation column, got:\n%s", allSQL) + if !strings.Contains(allSQL, "CHECK (operation IN ('edit', 'rename', 'delete'))") { + t.Errorf("history should have operation CHECK constraint, got:\n%s", allSQL) } - // Indexes with clean names + // History table uses modern CREATE TABLE WITH syntax for hypertable + columnstore + if !strings.Contains(allSQL, "tsdb.partition_column = 'version_id'") { + t.Errorf("history hypertable should partition by version_id, got:\n%s", allSQL) + } + if !strings.Contains(allSQL, "tsdb.segmentby = 'file_id'") { + t.Errorf("history columnstore should segment by file_id, got:\n%s", allSQL) + } + if !strings.Contains(allSQL, "tsdb.orderby = 'version_id DESC'") { + t.Errorf("history columnstore should order by version_id DESC, got:\n%s", allSQL) + } + + // Indexes with clean names (using new column names) if !strings.Contains(allSQL, `"idx_memory_history_by_filename"`) { t.Errorf("should create filename index with clean name, got:\n%s", allSQL) } + if !strings.Contains(allSQL, "(filename, version_id DESC)") { + t.Errorf("filename index should use version_id, got:\n%s", allSQL) + } if !strings.Contains(allSQL, `"idx_memory_history_by_id"`) { t.Errorf("should create id index with clean name, got:\n%s", allSQL) } + if !strings.Contains(allSQL, "(file_id, version_id DESC)") { + t.Errorf("id index should use file_id and version_id, got:\n%s", allSQL) + } // Encoding column in history table - if !strings.Contains(allSQL, "encoding TEXT,") { - t.Errorf("history table should contain encoding column, got:\n%s", allSQL) + if !strings.Contains(allSQL, "encoding TEXT CHECK") { + t.Errorf("history table should contain encoding column with CHECK, got:\n%s", allSQL) } // Markdown-specific columns in history table and trigger @@ -248,15 +376,24 @@ func TestSynth_GenerateHistorySQL_Markdown(t *testing.T) { t.Errorf("markdown trigger should copy title, got:\n%s", allSQL) } - // Trigger on tigerfs schema table + // Trigger copies parent_id and uses new column names if !strings.Contains(allSQL, "BEFORE UPDATE OR DELETE") { t.Errorf("should create BEFORE UPDATE OR DELETE trigger, got:\n%s", allSQL) } if !strings.Contains(allSQL, `ON "tigerfs"."memory"`) { t.Errorf("trigger should be on tigerfs.memory, got:\n%s", allSQL) } - if !strings.Contains(allSQL, "TG_OP::text") { - t.Errorf("should record TG_OP as _operation, got:\n%s", allSQL) + if !strings.Contains(allSQL, "OLD.parent_id") { + t.Errorf("trigger should copy OLD.parent_id, got:\n%s", allSQL) + } + if !strings.Contains(allSQL, "version_id, operation") { + t.Errorf("trigger INSERT should use version_id and operation columns, got:\n%s", allSQL) + } + if !strings.Contains(allSQL, "WHEN 'DELETE' THEN 'delete'") { + t.Errorf("trigger should map TG_OP to filesystem-centric types, got:\n%s", allSQL) + } + if !strings.Contains(allSQL, "THEN 'rename'") { + t.Errorf("trigger should detect rename from filename/parent_id changes, got:\n%s", allSQL) } if !strings.Contains(allSQL, "OLD.encoding") { t.Errorf("history trigger should copy encoding column, got:\n%s", allSQL) @@ -267,22 +404,50 @@ func TestSynth_GenerateHistorySQL_Markdown(t *testing.T) { t.Errorf("archive function should be in tigerfs schema, got:\n%s", allSQL) } - // Hypertable uses tigerfs schema - if !strings.Contains(allSQL, "create_hypertable('tigerfs.memory_history'") { - t.Errorf("hypertable should use tigerfs schema, got:\n%s", allSQL) + // Should be 8 statements: history (table with WITH, 2 indexes, func, trigger) + // + log (table, index) + savepoint (table) + if len(stmts) != 8 { + t.Errorf("expected 8 statements, got %d", len(stmts)) } - // Compression - if !strings.Contains(allSQL, "timescaledb.compress") { - t.Errorf("should enable compression, got:\n%s", allSQL) + // --- Log table --- + if !strings.Contains(allSQL, `"tigerfs"."memory_log"`) { + t.Errorf("should create log table in tigerfs schema, got:\n%s", allSQL) + } + if !strings.Contains(allSQL, "log_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY") { + t.Errorf("log table should have log_id UUIDv7 PK, got:\n%s", allSQL) + } + // Log table should have version_id (renamed from history_id) + if !strings.Contains(allSQL, "version_id UUID") { + t.Errorf("log table should have version_id column, got:\n%s", allSQL) + } + // Filesystem-centric type names (ADR-017) + if !strings.Contains(allSQL, "CHECK (type IN ('create', 'edit', 'rename', 'delete', 'undo'))") { + t.Errorf("log table should have filesystem-centric type CHECK constraint, got:\n%s", allSQL) + } + // Log table uses modern CREATE TABLE WITH syntax for hypertable + columnstore + if !strings.Contains(allSQL, "tsdb.hypertable") { + t.Errorf("log table should be a hypertable via CREATE TABLE WITH, got:\n%s", allSQL) + } + if !strings.Contains(allSQL, "tsdb.chunk_interval = '7 days'") { + t.Errorf("log hypertable should have 7-day chunk interval, got:\n%s", allSQL) + } + if !strings.Contains(allSQL, `"idx_memory_log_by_file"`) { + t.Errorf("log table should have (file_id, log_id) index, got:\n%s", allSQL) } - if !strings.Contains(allSQL, "add_compression_policy('tigerfs.memory_history'") { - t.Errorf("compression policy should use tigerfs schema, got:\n%s", allSQL) + if !strings.Contains(allSQL, "tsdb.orderby = 'log_id ASC'") { + t.Errorf("log columnstore should order by log_id ASC, got:\n%s", allSQL) } - // Should be 8 statements: table, 2 indexes, func, trigger, hypertable, compression, policy - if len(stmts) != 8 { - t.Errorf("expected 8 statements, got %d", len(stmts)) + // --- Savepoint table --- + if !strings.Contains(allSQL, `"tigerfs"."memory_savepoint"`) { + t.Errorf("should create savepoint table in tigerfs schema, got:\n%s", allSQL) + } + if !strings.Contains(allSQL, "name TEXT NOT NULL PRIMARY KEY") { + t.Errorf("savepoint table should have name as PK, got:\n%s", allSQL) + } + if !strings.Contains(allSQL, "savepoint_id UUID NOT NULL DEFAULT uuidv7() UNIQUE") { + t.Errorf("savepoint table should have savepoint_id UUIDv7 UNIQUE, got:\n%s", allSQL) } } @@ -317,10 +482,13 @@ func TestSynth_GenerateHistorySQL_PlainText(t *testing.T) { t.Errorf("plain text trigger should not reference OLD.headers, got:\n%s", allSQL) } - // Should still have base columns + // Should still have base columns (OLD.id maps to file_id in history) if !strings.Contains(allSQL, "OLD.id") { t.Errorf("trigger should reference OLD.id, got:\n%s", allSQL) } + if !strings.Contains(allSQL, "OLD.parent_id") { + t.Errorf("trigger should reference OLD.parent_id, got:\n%s", allSQL) + } if !strings.Contains(allSQL, "OLD.filename") { t.Errorf("trigger should reference OLD.filename, got:\n%s", allSQL) } @@ -331,10 +499,18 @@ func TestSynth_GenerateHistorySQL_PlainText(t *testing.T) { t.Errorf("trigger should reference OLD.encoding, got:\n%s", allSQL) } - // Should be 8 statements + // Should be 8 statements (same as markdown -- log/savepoint are format-independent) if len(stmts) != 8 { t.Errorf("expected 8 statements, got %d", len(stmts)) } + + // Log and savepoint tables should exist for plain text too + if !strings.Contains(allSQL, `"tigerfs"."snippets_log"`) { + t.Errorf("should create log table for plain text app, got:\n%s", allSQL) + } + if !strings.Contains(allSQL, `"tigerfs"."snippets_savepoint"`) { + t.Errorf("should create savepoint table for plain text app, got:\n%s", allSQL) + } } func TestSynth_GenerateBuildSQLWithFeatures_History(t *testing.T) { @@ -346,9 +522,10 @@ func TestSynth_GenerateBuildSQLWithFeatures_History(t *testing.T) { allSQL := strings.Join(stmts, "\n") - // Should have base (6: schema+table+view+comment+func+trigger) + history (8) = 14 statements - if len(stmts) != 14 { - t.Errorf("expected 14 statements, got %d", len(stmts)) + // Should have base (10: schema+resolve_path+table+parent_idx+view+comment+func+trigger+parent_mtime_func+parent_mtime_trigger) + // + history (8: table+2idx+func+trigger+log+logidx+savepoint) = 18 statements + if len(stmts) != 18 { + t.Errorf("expected 18 statements, got %d", len(stmts)) } // First statement should create tigerfs schema @@ -366,13 +543,16 @@ func TestSynth_GenerateBuildSQLWithFeatures_History(t *testing.T) { t.Errorf("should create history table in tigerfs schema, got:\n%s", allSQL) } - // History trigger should copy encoding column + // History trigger should copy encoding and parent_id columns if !strings.Contains(allSQL, "OLD.encoding") { t.Errorf("history trigger should copy encoding column, got:\n%s", allSQL) } - // History table should have encoding column - if !strings.Contains(allSQL, "encoding TEXT,") { - t.Errorf("history table should contain encoding column, got:\n%s", allSQL) + if !strings.Contains(allSQL, "OLD.parent_id") { + t.Errorf("history trigger should copy parent_id column, got:\n%s", allSQL) + } + // History table should have encoding column with CHECK + if !strings.Contains(allSQL, "encoding TEXT CHECK") { + t.Errorf("history table should contain encoding column with CHECK, got:\n%s", allSQL) } } @@ -389,7 +569,7 @@ func TestSynth_GenerateHistoryOnlySQL(t *testing.T) { t.Errorf("comment should include history, got: %s", stmts[0]) } - // Should have 1 (comment) + 8 (history) = 9 statements + // Should have 1 (comment) + 8 (history + log + savepoint) = 9 statements if len(stmts) != 9 { t.Errorf("expected 9 statements, got %d", len(stmts)) } @@ -400,6 +580,74 @@ func TestSynth_GenerateHistoryOnlySQL(t *testing.T) { } } +func TestSynth_GenerateResolvePathSQL(t *testing.T) { + sql := GenerateResolvePathSQL() + + // Function should be in tigerfs schema + if !strings.Contains(sql, `"tigerfs".resolve_path`) { + t.Errorf("should create function in tigerfs schema, got:\n%s", sql) + } + // Should accept REGCLASS, UUID, TEXT[] parameters + if !strings.Contains(sql, "tbl REGCLASS") { + t.Errorf("should accept REGCLASS parameter, got:\n%s", sql) + } + if !strings.Contains(sql, "start_parent UUID") { + t.Errorf("should accept start_parent UUID parameter, got:\n%s", sql) + } + if !strings.Contains(sql, "segments TEXT[]") { + t.Errorf("should accept segments TEXT[] parameter, got:\n%s", sql) + } + // Should return table with expected columns + if !strings.Contains(sql, "RETURNS TABLE(depth INT, resolved_id UUID, resolved_name TEXT)") { + t.Errorf("should return (depth, resolved_id, resolved_name), got:\n%s", sql) + } + // Should use EXECUTE with format() for dynamic table (REGCLASS) + if !strings.Contains(sql, "EXECUTE format(") { + t.Errorf("should use EXECUTE format() for dynamic table, got:\n%s", sql) + } + // Should use IS NOT DISTINCT FROM for NULL-safe parent_id comparison + if !strings.Contains(sql, "IS NOT DISTINCT FROM") { + t.Errorf("should use IS NOT DISTINCT FROM for NULL-safe comparison, got:\n%s", sql) + } + // Should be PL/pgSQL + if !strings.Contains(sql, "LANGUAGE plpgsql") { + t.Errorf("should be PL/pgSQL function, got:\n%s", sql) + } + // Should be CREATE OR REPLACE (idempotent) + if !strings.Contains(sql, "CREATE OR REPLACE FUNCTION") { + t.Errorf("should use CREATE OR REPLACE, got:\n%s", sql) + } +} + +func TestSynth_GenerateParentIndexSQL(t *testing.T) { + sql := GenerateParentIndexSQL("memory") + + if !strings.Contains(sql, `"idx_memory_parent"`) { + t.Errorf("index name should be idx_memory_parent, got:\n%s", sql) + } + if !strings.Contains(sql, `"tigerfs"."memory"`) { + t.Errorf("should reference tigerfs.memory table, got:\n%s", sql) + } + if !strings.Contains(sql, "(parent_id, filename)") { + t.Errorf("should index (parent_id, filename), got:\n%s", sql) + } + if !strings.Contains(sql, "CREATE INDEX") { + t.Errorf("should be CREATE INDEX, got:\n%s", sql) + } +} + +func TestSynth_GenerateParentIndexSQL_SpecialName(t *testing.T) { + sql := GenerateParentIndexSQL("my-app") + + // Should properly quote identifiers with special characters + if !strings.Contains(sql, `"idx_my-app_parent"`) { + t.Errorf("should quote index name with special chars, got:\n%s", sql) + } + if !strings.Contains(sql, `"tigerfs"."my-app"`) { + t.Errorf("should quote table name with special chars, got:\n%s", sql) + } +} + func TestSynth_QuoteIdent(t *testing.T) { tests := []struct { input string diff --git a/internal/tigerfs/fs/synth/columns.go b/internal/tigerfs/fs/synth/columns.go index e4531b2..c158eb8 100644 --- a/internal/tigerfs/fs/synth/columns.go +++ b/internal/tigerfs/fs/synth/columns.go @@ -64,6 +64,11 @@ type ColumnRoles struct { // Expected values: "file", "directory". Empty if no matching column found. Filetype string + // ParentID is the column referencing the parent directory row (self-referencing FK). + // NULL for root-level entries. When present with Filetype, enables the relational + // parent-pointer directory model (ADR-017). Empty if no matching column found. + ParentID string + // Encoding is the column indicating how body content is encoded. // Values: "utf8" (plain text, default), "base64" (binary data encoded as base64). // Empty if no matching column found (treated as utf8). @@ -114,6 +119,9 @@ func DetectColumnRoles(columnNames []string, format SynthFormat, pkColumn string // Detect optional filetype column (for hierarchical directory support) roles.Filetype = findMatchingColumn(columnNames, []string{"filetype"}) + // Detect optional parent_id column (for relational directory model, ADR-017) + roles.ParentID = findMatchingColumn(columnNames, []string{"parent_id"}) + // Detect optional encoding column roles.Encoding = findMatchingColumn(columnNames, encodingConventions) @@ -137,6 +145,9 @@ func DetectColumnRoles(columnNames []string, format SynthFormat, pkColumn string if roles.Filetype != "" { excluded[strings.ToLower(roles.Filetype)] = true } + if roles.ParentID != "" { + excluded[strings.ToLower(roles.ParentID)] = true + } if roles.Encoding != "" { excluded[strings.ToLower(roles.Encoding)] = true } diff --git a/internal/tigerfs/fs/synth/columns_test.go b/internal/tigerfs/fs/synth/columns_test.go index 01f35df..759a845 100644 --- a/internal/tigerfs/fs/synth/columns_test.go +++ b/internal/tigerfs/fs/synth/columns_test.go @@ -6,17 +6,18 @@ import ( func TestDetectColumnRoles_Markdown(t *testing.T) { tests := []struct { - name string - columns []string - pk string - wantFile string - wantBody string - wantFront []string - wantModAt string - wantCreAt string - wantExtra string - wantEnc string - wantErr bool + name string + columns []string + pk string + wantFile string + wantBody string + wantFront []string + wantModAt string + wantCreAt string + wantExtra string + wantEnc string + wantParent string + wantErr bool }{ { name: "standard columns", @@ -163,6 +164,18 @@ func TestDetectColumnRoles_Markdown(t *testing.T) { wantCreAt: "created_at", wantEnc: "encoding", }, + { + name: "parent_id column detected and excluded from frontmatter", + columns: []string{"id", "parent_id", "filename", "filetype", "title", "author", "body", "encoding", "created_at", "modified_at"}, + pk: "id", + wantFile: "filename", + wantBody: "body", + wantFront: []string{"title", "author"}, + wantModAt: "modified_at", + wantCreAt: "created_at", + wantEnc: "encoding", + wantParent: "parent_id", + }, } for _, tt := range tests { @@ -198,6 +211,9 @@ func TestDetectColumnRoles_Markdown(t *testing.T) { if roles.Encoding != tt.wantEnc { t.Errorf("Encoding = %q, want %q", roles.Encoding, tt.wantEnc) } + if roles.ParentID != tt.wantParent { + t.Errorf("ParentID = %q, want %q", roles.ParentID, tt.wantParent) + } if len(roles.Frontmatter) != len(tt.wantFront) { t.Fatalf("Frontmatter = %v, want %v", roles.Frontmatter, tt.wantFront) } diff --git a/internal/tigerfs/fs/synth/format.go b/internal/tigerfs/fs/synth/format.go index 17635f6..9039164 100644 --- a/internal/tigerfs/fs/synth/format.go +++ b/internal/tigerfs/fs/synth/format.go @@ -8,12 +8,12 @@ package synth import ( - "encoding/binary" "fmt" "strings" "time" "github.com/google/uuid" + "github.com/timescale/tigerfs/internal/tigerfs/format" ) // SynthFormat represents a synthesized file format. @@ -213,36 +213,47 @@ func detectFormatFromSuffix(viewName string) SynthFormat { return FormatNative } -// VersionIDLayout is the time format for history version IDs. -// Uses a filesystem-safe format without colons: "2006-01-02T150405Z" -const VersionIDLayout = "2006-01-02T150405Z" +// UUIDv7ToDisplayName converts a UUIDv7 to a human-readable display name. +// Delegates to format.UUIDv7ToDisplayName. See ADR-016 Section 11. +func UUIDv7ToDisplayName(id uuid.UUID) string { + return format.UUIDv7ToDisplayName(id) +} + +// DisplayNameToUUIDv7 parses a display name back to a UUIDv7. +// Delegates to format.DisplayNameToUUIDv7. +func DisplayNameToUUIDv7(name string) (uuid.UUID, error) { + return format.DisplayNameToUUIDv7(name) +} + +// IsUUIDv7 checks whether a UUID is version 7. +// Delegates to format.IsUUIDv7. +func IsUUIDv7(id [16]byte) bool { + return format.IsUUIDv7(id) +} -// UUIDv7ToVersionID extracts the embedded timestamp from a UUIDv7 and -// formats it as a filesystem-safe version ID string. +// UUIDv7ToVersionID is the old version ID function name. +// Deprecated: Use UUIDv7ToDisplayName instead. func UUIDv7ToVersionID(id uuid.UUID) string { - ts := extractUUIDv7Time(id) - return ts.UTC().Format(VersionIDLayout) + return format.UUIDv7ToDisplayName(id) } -// VersionIDToTimestamp parses a version ID string back to a time.Time. +// VersionIDToTimestamp parses a version ID (old or new format) back to a time.Time. func VersionIDToTimestamp(versionID string) (time.Time, error) { - t, err := time.Parse(VersionIDLayout, versionID) + // Try new display name format first + if format.IsDisplayName(versionID) { + id, err := format.DisplayNameToUUIDv7(versionID) + if err == nil { + return format.ExtractUUIDv7Time(id), nil + } + } + // Fall back to old format (second precision) + t, err := time.Parse("2006-01-02T150405Z", versionID) if err != nil { return time.Time{}, fmt.Errorf("invalid version ID %q: %w", versionID, err) } return t, nil } -// extractUUIDv7Time extracts the millisecond timestamp from a UUIDv7. -// UUIDv7 stores a Unix timestamp in milliseconds in the first 48 bits. -func extractUUIDv7Time(id uuid.UUID) time.Time { - // First 6 bytes contain the 48-bit Unix timestamp in milliseconds - b := id[:] - msec := int64(binary.BigEndian.Uint16(b[0:2]))<<32 | - int64(binary.BigEndian.Uint32(b[2:6])) - return time.UnixMilli(msec) -} - // detectFormatFromColumns infers the format from column name patterns. func detectFormatFromColumns(columnNames []string) SynthFormat { nameSet := make(map[string]bool, len(columnNames)) diff --git a/internal/tigerfs/fs/synth_ops.go b/internal/tigerfs/fs/synth_ops.go index 68ac977..804d800 100644 --- a/internal/tigerfs/fs/synth_ops.go +++ b/internal/tigerfs/fs/synth_ops.go @@ -138,13 +138,34 @@ func (o *Operations) loadSynthCache(ctx context.Context, schema string) (map[str } } - cache[viewName] = &synth.ViewInfo{ + info := &synth.ViewInfo{ Format: format, Roles: roles, CachedMountTime: mountTime, SupportsHierarchy: roles.Filetype != "", HasHistory: hasHistory, } + cache[viewName] = info + + // Warn if this view uses the legacy directory model (has filetype but no parent_id). + // The relational-directories migration adds parent_id for improved performance. + if info.SupportsHierarchy && roles.ParentID == "" { + var dirPath string + if o.mountPoint != "" { + if schema == o.cachedSchema { + // Default schema: tables at mount root + dirPath = o.mountPoint + "/" + viewName + } else { + // Non-default schema: under .schemas/ + dirPath = o.mountPoint + "/" + DirSchemas + "/" + schema + "/" + viewName + } + } else { + dirPath = viewName + } + logging.Warn("legacy directory format detected", + zap.String("directory", dirPath), + zap.String("hint", "run 'tigerfs migrate' to upgrade")) + } } return cache, nil @@ -190,10 +211,63 @@ func extractModTime(columns []string, values []interface{}, info *synth.ViewInfo return info.CachedMountTime } -// primeSynthStatCache builds and stores a Stat cache from raw query results. -// Called by ReadDir after GetAllRows. Subsequent Stat calls use the cached entries -// instead of re-querying the database, eliminating ~36 of 37 queries for `ls -l`. -func (o *Operations) primeSynthStatCache(schema, table string, columns []string, rows [][]interface{}, info *synth.ViewInfo) { +// synthUndoDirs returns the undo-related directory entries for a history-enabled +// synth app at the workspace ROOT level. Returns nil if history is not enabled. +// Includes .history/, .log/, .savepoint/, .undo/. +func synthUndoDirs(info *synth.ViewInfo) []Entry { + if !info.HasHistory { + return nil + } + t := info.CachedMountTime + return []Entry{ + {Name: DirHistory, IsDir: true, Mode: os.ModeDir | 0555, ModTime: t}, + {Name: DirLog, IsDir: true, Mode: os.ModeDir | 0555, ModTime: t}, + {Name: DirSavepoint, IsDir: true, Mode: os.ModeDir | 0755, ModTime: t}, + {Name: DirUndo, IsDir: true, Mode: os.ModeDir | 0755, ModTime: t}, + } +} + +// synthSubdirDirs returns virtual directory entries for subdirectories. +// Subdirectories do NOT get .log, .savepoint, .undo (those are workspace-level). +// Returns nil -- subdirectories only contain real files and directories. +func synthSubdirDirs(_ *synth.ViewInfo) []Entry { + return nil +} + +// filterReservedNames removes entries whose names conflict with TigerFS virtual +// control directories (e.g., .history, .log, .savepoint, .undo). Virtual directories +// always take precedence over user-created files with the same name. +func filterReservedNames(entries []Entry) []Entry { + filtered := make([]Entry, 0, len(entries)) + for _, e := range entries { + if !isKnownCapability(e.Name) { + filtered = append(filtered, e) + } + } + return filtered +} + +// checkReservedFilename returns an error if the leaf filename matches a TigerFS +// reserved name (virtual control directory). Used by write, mkdir, and rename. +func checkReservedFilename(filename string) *FSError { + leaf := filename + if i := strings.LastIndex(filename, "/"); i >= 0 { + leaf = filename[i+1:] + } + if isKnownCapability(leaf) { + return &FSError{ + Code: ErrPermission, + Message: fmt.Sprintf("filename %q is reserved for TigerFS virtual directory", leaf), + Hint: "choose a different filename to avoid collision with TigerFS control directories", + } + } + return nil +} + +// primeSynthStatCache populates the stat cache from row data. +// pathPrefix is prepended to leaf filenames for the cache key (e.g., "projects/web" +// for subdirectory entries). Empty for root-level entries. +func (o *Operations) primeSynthStatCache(schema, table, pathPrefix string, columns []string, rows [][]interface{}, info *synth.ViewInfo) { entries := make(map[string]Entry, len(rows)) for _, row := range rows { @@ -207,11 +281,12 @@ func (o *Operations) primeSynthStatCache(schema, table string, columns []string, } } if filetypeIdx >= 0 && synth.ValueToString(row[filetypeIdx]) == "directory" { - // Cache directory entry keyed by its filename path - fn := synth.ValueToString(row[findColIdx(columns, info.Roles.Filename)]) + // Cache directory entry keyed by its full path + leafName := synth.ValueToString(row[findColIdx(columns, info.Roles.Filename)]) + fullPath := joinPathPrefix(pathPrefix, leafName) modTime := extractModTime(columns, row, info) - entries[fn] = Entry{ - Name: fn, + entries[fullPath] = Entry{ + Name: leafName, IsDir: true, Mode: 0755, ModTime: modTime, @@ -231,13 +306,14 @@ func (o *Operations) primeSynthStatCache(schema, table string, columns []string, continue } + fullPath := joinPathPrefix(pathPrefix, filename) modTime := extractModTime(columns, row, info) var size int64 if content, err := o.synthesizeContent(columns, row, info); err == nil { size = int64(len(content)) } - entries[filename] = Entry{ + entries[fullPath] = Entry{ Name: filename, IsDir: false, Mode: 0644, @@ -249,6 +325,15 @@ func (o *Operations) primeSynthStatCache(schema, table string, columns []string, o.statCache.prime(schema, table, entries) } +// joinPathPrefix joins a directory prefix with a leaf name. +// Returns leaf unchanged when prefix is empty (root level). +func joinPathPrefix(prefix, leaf string) string { + if prefix == "" { + return leaf + } + return prefix + "/" + leaf +} + // findColIdx returns the index of a column name, or -1 if not found. func findColIdx(columns []string, name string) int { for i, col := range columns { @@ -270,6 +355,18 @@ func (o *Operations) readDirSynthView(ctx context.Context, parsed *ParsedPath, i limit = 10000 } + // Parent-pointer model (ADR-017): query only root-level entries + if info.SupportsHierarchy && info.Roles.ParentID != "" { + columns, rows, err := o.db.GetRowsByParent(ctx, fsCtx.Schema, fsCtx.TableName, "", limit) + if err != nil { + return nil, &FSError{Code: ErrIO, Message: "failed to list root entries", Cause: err} + } + o.primeSynthStatCache(fsCtx.Schema, fsCtx.TableName, "", columns, rows, info) + children := filterReservedNames(o.buildEntriesFromRows(columns, rows, info)) + children = append(synthUndoDirs(info), children...) + return children, nil + } + columns, rows, err := o.db.GetAllRows(ctx, fsCtx.Schema, fsCtx.TableName, limit) if err != nil { return nil, &FSError{ @@ -280,23 +377,19 @@ func (o *Operations) readDirSynthView(ctx context.Context, parsed *ParsedPath, i } // Prime Stat cache so subsequent Stat calls avoid DB queries - o.primeSynthStatCache(fsCtx.Schema, fsCtx.TableName, columns, rows, info) + o.primeSynthStatCache(fsCtx.Schema, fsCtx.TableName, "", columns, rows, info) - // For hierarchical views, filter to root-level entries only + // Old hierarchy model (path-encoded filenames, pre-ADR-017): filter to root-level if info.SupportsHierarchy { - children := o.filterHierarchicalChildren(columns, rows, "", info) - if info.HasHistory { - children = append([]Entry{{Name: DirHistory, IsDir: true, Mode: os.ModeDir | 0555, ModTime: info.CachedMountTime}}, children...) - } + children := filterReservedNames(o.filterHierarchicalChildren(columns, rows, "", info)) + children = append(synthUndoDirs(info), children...) return children, nil } entries := make([]Entry, 0, len(rows)+1) - // Add .history/ if versioned history is enabled - if info.HasHistory { - entries = append(entries, Entry{Name: DirHistory, IsDir: true, Mode: os.ModeDir | 0555, ModTime: info.CachedMountTime}) - } + // Add undo-related directories if versioned history is enabled + entries = append(entries, synthUndoDirs(info)...) for _, row := range rows { var filename string @@ -309,6 +402,11 @@ func (o *Operations) readDirSynthView(ctx context.Context, parsed *ParsedPath, i continue } + // Skip entries whose names collide with virtual control directories + if isKnownCapability(filename) { + continue + } + modTime := extractModTime(columns, row, info) // Synthesize content to get accurate size (CPU-only, no DB query) var size int64 @@ -327,12 +425,61 @@ func (o *Operations) readDirSynthView(ctx context.Context, parsed *ParsedPath, i return entries, nil } +// fetchSynthRowByPath resolves a parent path via cache, then fetches the leaf row +// with a combined parent_id + filename query. This is one round-trip for the leaf +// (vs resolve_path + GetRow = two round-trips). Used by ReadFile where the leaf +// must always be fetched fresh from DB (consistency model: "ReadFile must always hit the DB"). +func (o *Operations) fetchSynthRowByPath(ctx context.Context, schema, table string, info *synth.ViewInfo, fullPath string) ([]string, []interface{}, *FSError) { + parts := strings.Split(fullPath, "/") + leafName := parts[len(parts)-1] + parentSegments := parts[:len(parts)-1] + + // Resolve parent path via cache (0 DB calls if fully cached) + var parentID string + if len(parentSegments) > 0 { + var ok bool + parentID, ok = o.resolveSynthPath(ctx, schema, table, parentSegments) + if !ok { + return nil, nil, &FSError{Code: ErrNotExist, Message: fmt.Sprintf("directory not found: %s", strings.Join(parentSegments, "/"))} + } + } + + // Single query: SELECT * WHERE parent_id = X AND filename = leaf + columns, row, err := o.db.GetRowByParentAndName(ctx, schema, table, parentID, leafName) + if err != nil { + return nil, nil, &FSError{Code: ErrIO, Message: "failed to fetch row", Cause: err} + } + if row == nil { + return nil, nil, &FSError{Code: ErrNotExist, Message: fmt.Sprintf("file not found: %s", fullPath)} + } + + return columns, row, nil +} + +// resolveSynthRow resolves a full path to its row data using the path cache and DB. +// For parent-pointer model only. Returns (columns, row, pkValue, error). +func (o *Operations) resolveSynthRow(ctx context.Context, schema, table string, info *synth.ViewInfo, fullPath string) ([]string, []interface{}, string, *FSError) { + parts := strings.Split(fullPath, "/") + fileID, ok := o.resolveSynthPath(ctx, schema, table, parts) + if !ok { + return nil, nil, "", &FSError{Code: ErrNotExist, Message: fmt.Sprintf("file not found: %s", fullPath)} + } + + row, err := o.db.GetRow(ctx, schema, table, db.SinglePKMatch(info.Roles.PrimaryKey, fileID)) + if err != nil { + return nil, nil, "", &FSError{Code: ErrIO, Message: "failed to fetch row by ID", Cause: err} + } + if row == nil { + return nil, nil, "", &FSError{Code: ErrNotExist, Message: fmt.Sprintf("row not found: %s", fileID)} + } + + return row.Columns, row.Values, fileID, nil +} + // statSynthFile returns metadata for a synthesized file. // For views with hierarchy, also handles directory stat (filetype='directory'). func (o *Operations) statSynthFile(ctx context.Context, parsed *ParsedPath, info *synth.ViewInfo) (*Entry, *FSError) { - // The PrimaryKey field contains the filename (e.g., "hello-world.md") filename := parsed.PrimaryKey - schema := parsed.Context.Schema table := parsed.Context.TableName @@ -343,18 +490,19 @@ func (o *Operations) statSynthFile(ctx context.Context, parsed *ParsedPath, info // Check negative cache (file known to not exist) if o.statCache.isNegative(schema, table, filename) { - return nil, &FSError{ - Code: ErrNotExist, - Message: fmt.Sprintf("file not found: %s (cached)", filename), - } + return nil, &FSError{Code: ErrNotExist, Message: fmt.Sprintf("file not found: %s (cached)", filename)} } - // Single query: look up by filename. For hierarchical views, this may return - // a directory row — we check filetype in the result rather than doing a - // separate existence query. - columns, row, fsErr := o.statSynthRowByFilename(ctx, schema, table, info, filename) + // Parent-pointer model: resolve path to UUID, then fetch row + var columns []string + var row []interface{} + var fsErr *FSError + if info.Roles.ParentID != "" { + columns, row, _, fsErr = o.resolveSynthRow(ctx, schema, table, info, filename) + } else { + columns, row, fsErr = o.statSynthRowByFilename(ctx, schema, table, info, filename) + } if fsErr != nil { - // Cache the negative result so subsequent stats don't re-query if fsErr.Code == ErrNotExist { o.statCache.setNegative(schema, table, filename) } @@ -369,7 +517,7 @@ func (o *Operations) statSynthFile(ctx context.Context, parsed *ParsedPath, info Name: filename, IsDir: true, Mode: 0755, - ModTime: info.CachedMountTime, + ModTime: extractModTime(columns, row, info), } o.statCache.set(schema, table, filename, entry) return &entry, nil @@ -400,10 +548,20 @@ func (o *Operations) statSynthFile(ctx context.Context, parsed *ParsedPath, info } // readFileSynthView reads synthesized file content. +// For parent-pointer model: resolves parent path via cache, then fetches the +// leaf row with a combined parent_id + filename query (one round-trip for the +// leaf instead of resolve_path + GetRow). ADR-017 Section "ReadFile / Stat". func (o *Operations) readFileSynthView(ctx context.Context, parsed *ParsedPath, info *synth.ViewInfo) ([]byte, *FSError) { filename := parsed.PrimaryKey - columns, row, fsErr := o.getSynthRow(ctx, parsed.Context.Schema, parsed.Context.TableName, info, filename) + var columns []string + var row []interface{} + var fsErr *FSError + if info.Roles.ParentID != "" { + columns, row, fsErr = o.fetchSynthRowByPath(ctx, parsed.Context.Schema, parsed.Context.TableName, info, filename) + } else { + columns, row, fsErr = o.getSynthRow(ctx, parsed.Context.Schema, parsed.Context.TableName, info, filename) + } if fsErr != nil { return nil, fsErr } @@ -420,6 +578,194 @@ func (o *Operations) readFileSynthView(ctx context.Context, parsed *ParsedPath, return content, nil } +// logSynthOp records an operation in the undo log for history-enabled synth apps. +// This is a best-effort operation -- log failures are logged but don't fail the +// original write. The userID parameter is empty until user identity is wired (Task 12.5). +// +// Operation types are filesystem-centric (ADR-017): +// - "create": new file/directory created +// - "edit": file content modified +// - "rename": file/directory renamed or moved (parent_id or filename change) +// - "delete": file/directory deleted +// - "undo": undo operation restoring previous state +func (o *Operations) logSynthOp(ctx context.Context, schema, tableName string, info *synth.ViewInfo, opType, fileID, filename string) { + if !info.HasHistory { + return + } + + // Check for auto-savepoint before logging the operation. + o.maybeCreateAutoSavepoint(ctx, schema, tableName) + + logTable := tableName + "_log" + historyTable := tableName + "_history" + + // For edit/rename/delete/undo, capture the version_id of the before-state. + // The BEFORE trigger has already fired, so the most recent history entry + // for this file_id is the one we want. + var versionID string + if opType != "create" && fileID != "" { + vid, err := o.db.QueryLatestVersionID(ctx, synth.TigerFSSchema, historyTable, fileID) + if err != nil { + logging.Debug("failed to capture version_id for log entry", + zap.String("table", tableName), + zap.String("file_id", fileID), + zap.Error(err)) + } else { + versionID = vid + } + } + + userID := o.userID + + err := o.db.InsertLogEntry(ctx, synth.TigerFSSchema, logTable, userID, opType, fileID, filename, versionID, "") + if err != nil { + logging.Warn("failed to insert undo log entry", + zap.String("table", tableName), + zap.String("type", opType), + zap.String("filename", filename), + zap.Error(err)) + } + + // Update last write time after successful log entry. + now := o.now() + o.lastWriteMu.Lock() + if o.lastWriteTime == nil { + o.lastWriteTime = make(map[string]time.Time) + } + o.lastWriteTime[schema+"."+tableName] = now + o.lastWriteMu.Unlock() +} + +// maybeCreateAutoSavepoint checks if the gap since the last write exceeds +// the configured threshold and creates an auto-savepoint if so. +func (o *Operations) maybeCreateAutoSavepoint(ctx context.Context, schema, tableName string) { + interval := o.config.AutoSavepointInterval + if interval <= 0 { + return + } + + key := schema + "." + tableName + now := o.now() + + o.lastWriteMu.Lock() + if o.lastWriteTime == nil { + o.lastWriteTime = make(map[string]time.Time) + } + lastTime, exists := o.lastWriteTime[key] + o.lastWriteMu.Unlock() + + if !exists { + // First write after mount -- nothing to bookmark. + return + } + + if now.Sub(lastTime) < interval { + return + } + + // Gap exceeds threshold -- create auto-savepoint. + name := o.autoSavepointName(now) + savepointTable := tableName + "_savepoint" + + columns := []string{"name"} + values := []interface{}{name} + if o.userID != "" { + columns = append(columns, "user_id") + values = append(values, o.userID) + } + columns = append(columns, "description") + values = append(values, fmt.Sprintf("Auto-savepoint after %s inactivity", now.Sub(lastTime).Truncate(time.Second))) + + _, err := o.db.InsertRow(ctx, synth.TigerFSSchema, savepointTable, columns, values) + if err != nil { + logging.Warn("failed to create auto-savepoint", + zap.String("table", tableName), + zap.String("name", name), + zap.Error(err)) + return + } + + logging.Info("auto-savepoint created", + zap.String("table", tableName), + zap.String("name", name), + zap.Duration("gap", now.Sub(lastTime))) +} + +// autoSavepointName generates a name like "auto-agent-7-20260408T143000Z" +// or "auto-20260408T143000Z" for anonymous users. +func (o *Operations) autoSavepointName(t time.Time) string { + ts := t.UTC().Format("20060102T150405Z") + if o.userID != "" { + return fmt.Sprintf("auto-%s-%s", o.userID, ts) + } + return fmt.Sprintf("auto-%s", ts) +} + +// now returns the current time, using the injectable nowFunc if set. +func (o *Operations) now() time.Time { + if o.nowFunc != nil { + return o.nowFunc() + } + return time.Now() +} + +// resolveSynthPath resolves a sequence of path segments to a row ID using +// the path cache and the resolve_path PL/pgSQL function (ADR-017). +// +// Walks segments from left to right, checking the cache at each level. +// On the first cache miss, calls resolve_path with the remaining segments +// starting from the last cached parent. Populates the cache from the results. +// +// Returns the final row ID and true if the full path resolved, or empty string +// and false if any segment didn't resolve (path doesn't exist). +func (o *Operations) resolveSynthPath(ctx context.Context, schema, table string, segments []string) (string, bool) { + if len(segments) == 0 { + return "", true // root level — no ID, but "exists" + } + + // Walk segments, checking cache at each level + parentID := "" // empty = root (NULL parent_id) + cacheHits := 0 + for _, seg := range segments { + if id, ok := o.pathCache.lookup(schema, table, parentID, seg); ok { + parentID = id + cacheHits++ + } else { + break + } + } + + // All segments resolved from cache + if cacheHits == len(segments) { + return parentID, true + } + + // Call DB for remaining segments + remaining := segments[cacheHits:] + results, err := o.db.ResolvePath(ctx, synth.TigerFSSchema, table, parentID, remaining) + if err != nil { + logging.Debug("resolve_path failed", + zap.String("table", table), + zap.Strings("segments", remaining), + zap.Error(err)) + return "", false + } + + // Populate cache from results + curParent := parentID + for _, seg := range results { + o.pathCache.put(schema, table, curParent, seg.Name, seg.ID) + curParent = seg.ID + } + + // Check if all remaining segments resolved + if len(results) < len(remaining) { + return "", false + } + + return results[len(results)-1].ID, true +} + // writeSynthFile handles writes to synthesized view files (create or update). // For views with hierarchy, auto-creates parent directory rows on insert. // Binary data (null bytes or invalid UTF-8) is base64-encoded for TEXT column storage. @@ -427,6 +773,11 @@ func (o *Operations) writeSynthFile(ctx context.Context, parsed *ParsedPath, inf fsCtx := parsed.Context filename := parsed.PrimaryKey + // Reject filenames that collide with TigerFS virtual directories + if fsErr := checkReservedFilename(filename); fsErr != nil { + return fsErr + } + var colValues map[string]interface{} // Check if data is binary (null bytes or invalid UTF-8) @@ -455,14 +806,67 @@ func (o *Operations) writeSynthFile(ctx context.Context, parsed *ParsedPath, inf } } - // Set the filename column from the path (FS name == DB name) - colValues[info.Roles.Filename] = filename - // For hierarchical views, set filetype='file' explicitly if info.SupportsHierarchy { colValues[info.Roles.Filetype] = "file" } + // Parent-pointer model (ADR-017): use leaf filename + parent_id + if info.Roles.ParentID != "" { + parts := strings.Split(filename, "/") + leafName := parts[len(parts)-1] + colValues[info.Roles.Filename] = leafName + + // Check if file exists by resolving full path + fileID, fileExists := o.resolveSynthPath(ctx, fsCtx.Schema, fsCtx.TableName, parts) + + // Build columns/values slices + columns := make([]string, 0, len(colValues)) + values := make([]interface{}, 0, len(colValues)) + for col, val := range colValues { + columns = append(columns, col) + values = append(values, val) + } + + if fileExists { + // UPDATE by UUID — parent_id and filename don't change on content edit + dbErr := o.db.UpdateRow(ctx, fsCtx.Schema, fsCtx.TableName, db.SinglePKMatch(info.Roles.PrimaryKey, fileID), columns, values) + if dbErr != nil { + return &FSError{Code: ErrIO, Message: "failed to update synth file", Cause: dbErr} + } + o.logSynthOp(ctx, fsCtx.Schema, fsCtx.TableName, info, "edit", fileID, filename) + } else { + // Ensure parent directories exist, get parent UUID + parentID, fsErr := o.resolveSynthParentID(ctx, fsCtx.Schema, fsCtx.TableName, info, filename) + if fsErr != nil { + return fsErr + } + if parentID != "" { + colValues[info.Roles.ParentID] = parentID + // Rebuild columns/values with parent_id + columns = columns[:0] + values = values[:0] + for col, val := range colValues { + columns = append(columns, col) + values = append(values, val) + } + } + + insertedPK, dbErr := o.db.InsertRow(ctx, fsCtx.Schema, fsCtx.TableName, columns, values) + if dbErr != nil { + return &FSError{Code: ErrIO, Message: "failed to create synth file", Cause: dbErr} + } + o.logSynthOp(ctx, fsCtx.Schema, fsCtx.TableName, info, "create", insertedPK, filename) + } + + o.statCache.invalidate(fsCtx.Schema, fsCtx.TableName) + o.pathCache.invalidate(fsCtx.Schema, fsCtx.TableName) + return nil + } + + // Old model (pre-ADR-017): full-path filenames + colValues[info.Roles.Filename] = filename + // Convert map to columns/values slices columns := make([]string, 0, len(colValues)) values := make([]interface{}, 0, len(colValues)) @@ -476,7 +880,6 @@ func (o *Operations) writeSynthFile(ctx context.Context, parsed *ParsedPath, inf rowExists := lookupErr == nil if rowExists { - // UPDATE existing row — need to find the PK value pkValue, fsErr := o.getSynthRowPK(ctx, fsCtx.Schema, fsCtx.TableName, info, filename) if fsErr != nil { return fsErr @@ -484,29 +887,21 @@ func (o *Operations) writeSynthFile(ctx context.Context, parsed *ParsedPath, inf dbErr := o.db.UpdateRow(ctx, fsCtx.Schema, fsCtx.TableName, db.SinglePKMatch(info.Roles.PrimaryKey, pkValue), columns, values) if dbErr != nil { - return &FSError{ - Code: ErrIO, - Message: "failed to update synth file", - Cause: dbErr, - } + return &FSError{Code: ErrIO, Message: "failed to update synth file", Cause: dbErr} } + o.logSynthOp(ctx, fsCtx.Schema, fsCtx.TableName, info, "edit", pkValue, filename) } else { - // For hierarchical views, auto-create parent directories before inserting if info.SupportsHierarchy { - if fsErr := o.ensureSynthParentDirs(ctx, fsCtx.Schema, fsCtx.TableName, info, filename); fsErr != nil { + if _, fsErr := o.resolveSynthParentID(ctx, fsCtx.Schema, fsCtx.TableName, info, filename); fsErr != nil { return fsErr } } - // INSERT new row - _, dbErr := o.db.InsertRow(ctx, fsCtx.Schema, fsCtx.TableName, columns, values) + insertedPK, dbErr := o.db.InsertRow(ctx, fsCtx.Schema, fsCtx.TableName, columns, values) if dbErr != nil { - return &FSError{ - Code: ErrIO, - Message: "failed to create synth file", - Cause: dbErr, - } + return &FSError{Code: ErrIO, Message: "failed to create synth file", Cause: dbErr} } + o.logSynthOp(ctx, fsCtx.Schema, fsCtx.TableName, info, "create", insertedPK, filename) } o.statCache.invalidate(fsCtx.Schema, fsCtx.TableName) @@ -519,50 +914,66 @@ func (o *Operations) deleteSynthFile(ctx context.Context, parsed *ParsedPath, in fsCtx := parsed.Context filename := parsed.PrimaryKey - // For hierarchical views, check if this is a directory + // Parent-pointer model (ADR-017): resolve path then check/delete by UUID + if info.Roles.ParentID != "" { + columns, row, fileID, fsErr := o.resolveSynthRow(ctx, fsCtx.Schema, fsCtx.TableName, info, filename) + if fsErr != nil { + return fsErr + } + + // Check if it's a directory + filetypeIdx := findColIdx(columns, info.Roles.Filetype) + if filetypeIdx >= 0 && synth.ValueToString(row[filetypeIdx]) == "directory" { + // Check for children by parent_id (ADR-017: WHERE parent_id = dir_id) + _, childRows, err := o.db.GetRowsByParent(ctx, fsCtx.Schema, fsCtx.TableName, fileID, 1) + if err != nil { + return &FSError{Code: ErrIO, Message: "failed to check directory children", Cause: err} + } + if len(childRows) > 0 { + return &FSError{Code: ErrNotEmpty, Message: "directory not empty"} + } + } + + err := o.db.DeleteRow(ctx, fsCtx.Schema, fsCtx.TableName, db.SinglePKMatch(info.Roles.PrimaryKey, fileID)) + if err != nil { + return &FSError{Code: ErrIO, Message: "failed to delete synth file", Cause: err} + } + o.logSynthOp(ctx, fsCtx.Schema, fsCtx.TableName, info, "delete", fileID, filename) + o.statCache.invalidate(fsCtx.Schema, fsCtx.TableName) + o.pathCache.invalidate(fsCtx.Schema, fsCtx.TableName) + return nil + } + + // Old model: path-encoded filenames if info.SupportsHierarchy { dirPath := filename - exists, fsErr := o.synthRowExists(ctx, fsCtx.Schema, fsCtx.TableName, info, dirPath, "directory") if fsErr != nil { return fsErr } if exists { - // It's a directory — check for children hasChildren, err := o.db.HasChildrenWithPrefix(ctx, fsCtx.Schema, fsCtx.TableName, info.Roles.Filename, dirPath) if err != nil { - return &FSError{ - Code: ErrIO, - Message: "failed to check directory children", - Cause: err, - } + return &FSError{Code: ErrIO, Message: "failed to check directory children", Cause: err} } if hasChildren { - return &FSError{ - Code: ErrNotEmpty, - Message: "directory not empty", - } + return &FSError{Code: ErrNotEmpty, Message: "directory not empty"} } - // Delete the directory row by looking up its PK pkValue, lookupErr := o.getSynthRowPKByFiletype(ctx, fsCtx.Schema, fsCtx.TableName, info, dirPath, "directory") if lookupErr != nil { return lookupErr } err = o.db.DeleteRow(ctx, fsCtx.Schema, fsCtx.TableName, db.SinglePKMatch(info.Roles.PrimaryKey, pkValue)) if err != nil { - return &FSError{ - Code: ErrIO, - Message: "failed to delete directory", - Cause: err, - } + return &FSError{Code: ErrIO, Message: "failed to delete directory", Cause: err} } + o.logSynthOp(ctx, fsCtx.Schema, fsCtx.TableName, info, "delete", pkValue, dirPath) o.statCache.invalidate(fsCtx.Schema, fsCtx.TableName) return nil } } - // Regular file delete pkValue, fsErr := o.getSynthRowPK(ctx, fsCtx.Schema, fsCtx.TableName, info, filename) if fsErr != nil { return fsErr @@ -570,13 +981,9 @@ func (o *Operations) deleteSynthFile(ctx context.Context, parsed *ParsedPath, in err := o.db.DeleteRow(ctx, fsCtx.Schema, fsCtx.TableName, db.SinglePKMatch(info.Roles.PrimaryKey, pkValue)) if err != nil { - return &FSError{ - Code: ErrIO, - Message: "failed to delete synth file", - Cause: err, - } + return &FSError{Code: ErrIO, Message: "failed to delete synth file", Cause: err} } - + o.logSynthOp(ctx, fsCtx.Schema, fsCtx.TableName, info, "delete", pkValue, filename) o.statCache.invalidate(fsCtx.Schema, fsCtx.TableName) return nil } @@ -804,7 +1211,79 @@ func (o *Operations) synthesizeContent(columns []string, row []interface{}, info // For directories in hierarchical views, performs an atomic prefix rename // that updates the directory row and all its descendants. func (o *Operations) renameSynthFile(ctx context.Context, schema, table string, info *synth.ViewInfo, oldFilename, newFilename string) *FSError { - // For hierarchical views, check if old path is a directory + // Reject renames to reserved TigerFS virtual directory names + if fsErr := checkReservedFilename(newFilename); fsErr != nil { + return fsErr + } + + // Parent-pointer model (ADR-017): rename is a single-row UPDATE + if info.Roles.ParentID != "" { + _, _, fileID, fsErr := o.resolveSynthRow(ctx, schema, table, info, oldFilename) + if fsErr != nil { + return fsErr + } + + // Extract new leaf name (same directory → just change filename) + newParts := strings.Split(newFilename, "/") + newLeaf := newParts[len(newParts)-1] + oldParts := strings.Split(oldFilename, "/") + + // UPDATE filename (and parent_id if moving to different directory) + updateCols := []string{info.Roles.Filename} + updateVals := []interface{}{newLeaf} + + // Check if parent directory changed (move vs rename) + oldParentPath := strings.Join(oldParts[:len(oldParts)-1], "/") + newParentPath := strings.Join(newParts[:len(newParts)-1], "/") + if oldParentPath != newParentPath { + // Move to different directory — resolve new parent + var newParentID string + if newParentPath != "" { + newParentSegs := strings.Split(newParentPath, "/") + var ok bool + newParentID, ok = o.resolveSynthPath(ctx, schema, table, newParentSegs) + if !ok { + return &FSError{Code: ErrNotExist, Message: fmt.Sprintf("target directory not found: %s", newParentPath)} + } + } + updateCols = append(updateCols, info.Roles.ParentID) + if newParentID != "" { + updateVals = append(updateVals, newParentID) + } else { + updateVals = append(updateVals, nil) // root level + } + } + + // Check if target file already exists (POSIX rename-as-replace). + // If it does, atomically delete the target and rename the source. + targetFileID, targetExists := o.resolveSynthPath(ctx, schema, table, newParts) + if targetExists && targetFileID != fileID { + err := o.db.DeleteAndUpdate(ctx, schema, table, + db.SinglePKMatch(info.Roles.PrimaryKey, targetFileID), + db.SinglePKMatch(info.Roles.PrimaryKey, fileID), + updateCols, updateVals) + if err != nil { + return &FSError{Code: ErrIO, Message: "failed to rename synth file", Cause: err} + } + + // Log both operations: delete of target, then rename of source + o.logSynthOp(ctx, schema, table, info, "delete", targetFileID, newFilename) + o.logSynthOp(ctx, schema, table, info, "rename", fileID, newFilename) + } else { + err := o.db.UpdateRow(ctx, schema, table, db.SinglePKMatch(info.Roles.PrimaryKey, fileID), updateCols, updateVals) + if err != nil { + return &FSError{Code: ErrIO, Message: "failed to rename synth file", Cause: err} + } + + o.logSynthOp(ctx, schema, table, info, "rename", fileID, newFilename) + } + + o.statCache.invalidate(schema, table) + o.pathCache.invalidate(schema, table) + return nil + } + + // Old model: path-encoded filenames if info.SupportsHierarchy { oldDirPath := oldFilename newDirPath := newFilename @@ -814,35 +1293,23 @@ func (o *Operations) renameSynthFile(ctx context.Context, schema, table string, return fsErr } if exists { - // Directory rename — atomic prefix swap. - // RenameByPrefix WHERE matches old value, so concurrent renames - // are safe: the loser gets rowsAffected=0. rowsAffected, err := o.db.RenameByPrefix(ctx, schema, table, info.Roles.Filename, oldDirPath, newDirPath) if err != nil { - return &FSError{ - Code: ErrIO, - Message: "failed to rename directory", - Cause: err, - } + return &FSError{Code: ErrIO, Message: "failed to rename directory", Cause: err} } if rowsAffected == 0 { - return &FSError{ - Code: ErrNotExist, - Message: "directory already moved by another process", - } + return &FSError{Code: ErrNotExist, Message: "directory already moved by another process"} } o.statCache.invalidate(schema, table) return nil } } - // Regular file rename — FS name == DB name, no extension normalization needed. columns, row, fsErr := o.getSynthRow(ctx, schema, table, info, oldFilename) if fsErr != nil { return fsErr } - // Extract PK and raw filename from the actual DB row var pkValue, rawOldFilename string for i, col := range columns { switch col { @@ -853,31 +1320,18 @@ func (o *Operations) renameSynthFile(ctx context.Context, schema, table string, } } if pkValue == "" { - return &FSError{ - Code: ErrIO, - Message: fmt.Sprintf("primary key column %q not found in view", info.Roles.PrimaryKey), - } + return &FSError{Code: ErrIO, Message: fmt.Sprintf("primary key column %q not found in view", info.Roles.PrimaryKey)} } - // Atomic rename: UPDATE SET filename = new WHERE pk = X AND filename = old. - // If another process already renamed this file, the WHERE won't match - // and we get "row not found" — exactly one concurrent rename wins. err := o.db.UpdateColumnCAS(ctx, schema, table, db.SinglePKMatch(info.Roles.PrimaryKey, pkValue), info.Roles.Filename, newFilename, info.Roles.Filename, rawOldFilename) if err != nil { if strings.Contains(err.Error(), "not found") { - return &FSError{ - Code: ErrNotExist, - Message: "file already moved by another process", - Cause: err, - } - } - return &FSError{ - Code: ErrIO, - Message: "failed to rename synth file", - Cause: err, + return &FSError{Code: ErrNotExist, Message: "file already moved by another process", Cause: err} } + return &FSError{Code: ErrIO, Message: "failed to rename synth file", Cause: err} } + o.logSynthOp(ctx, schema, table, info, "rename", pkValue, newFilename) o.statCache.invalidate(schema, table) return nil } @@ -889,12 +1343,32 @@ func (o *Operations) readDirSynthHierarchical(ctx context.Context, parsed *Parse fsCtx := parsed.Context prefix := parsed.PrimaryKey - // Get all rows from the view limit := o.config.DirListingLimit if limit <= 0 { limit = 10000 } + // Parent-pointer model (ADR-017): resolve directory path, then query children + if info.Roles.ParentID != "" { + segments := strings.Split(prefix, "/") + parentID, ok := o.resolveSynthPath(ctx, fsCtx.Schema, fsCtx.TableName, segments) + if !ok { + return nil, &FSError{Code: ErrNotExist, Message: fmt.Sprintf("directory not found: %s", prefix)} + } + + columns, rows, err := o.db.GetRowsByParent(ctx, fsCtx.Schema, fsCtx.TableName, parentID, limit) + if err != nil { + return nil, &FSError{Code: ErrIO, Message: "failed to list directory entries", Cause: err} + } + + o.primeSynthStatCache(fsCtx.Schema, fsCtx.TableName, prefix, columns, rows, info) + children := filterReservedNames(o.buildEntriesFromRows(columns, rows, info)) + // Subdirectories: no virtual dirs (.log, .savepoint, .undo are workspace-level only) + children = append(synthSubdirDirs(info), children...) + return children, nil + } + + // Old hierarchy model (path-encoded filenames, pre-ADR-017) columns, rows, err := o.db.GetAllRows(ctx, fsCtx.Schema, fsCtx.TableName, limit) if err != nil { return nil, &FSError{ @@ -904,14 +1378,58 @@ func (o *Operations) readDirSynthHierarchical(ctx context.Context, parsed *Parse } } - // Prime Stat cache so subsequent Stat calls avoid DB queries - o.primeSynthStatCache(fsCtx.Schema, fsCtx.TableName, columns, rows, info) + o.primeSynthStatCache(fsCtx.Schema, fsCtx.TableName, prefix, columns, rows, info) + children := filterReservedNames(o.filterHierarchicalChildren(columns, rows, prefix, info)) + // Old model subdirectories: no virtual dirs + return children, nil +} + +// buildEntriesFromRows converts query result rows into Entry slices for ReadDir. +// Used by the parent-pointer model where GetRowsByParent already filtered to the +// correct directory -- no in-memory filtering needed. Each row's filename column +// contains the leaf name. +func (o *Operations) buildEntriesFromRows(columns []string, rows [][]interface{}, info *synth.ViewInfo) []Entry { + filetypeIdx := findColIdx(columns, info.Roles.Filetype) + entries := make([]Entry, 0, len(rows)) + + for _, row := range rows { + isDir := filetypeIdx >= 0 && synth.ValueToString(row[filetypeIdx]) == "directory" + modTime := extractModTime(columns, row, info) - children := o.filterHierarchicalChildren(columns, rows, prefix, info) - if info.HasHistory { - children = append([]Entry{{Name: DirHistory, IsDir: true, Mode: os.ModeDir | 0555, ModTime: info.CachedMountTime}}, children...) + if isDir { + leafName := synth.ValueToString(row[findColIdx(columns, info.Roles.Filename)]) + entries = append(entries, Entry{ + Name: leafName, + IsDir: true, + Mode: 0755, + ModTime: modTime, + }) + } else { + var filename string + switch info.Format { + case synth.FormatMarkdown: + filename = synth.GetMarkdownFilename(columns, row, info.Roles) + case synth.FormatPlainText: + filename = synth.GetPlainTextFilename(columns, row, info.Roles) + default: + continue + } + + var size int64 + if content, err := o.synthesizeContent(columns, row, info); err == nil { + size = int64(len(content)) + } + entries = append(entries, Entry{ + Name: filename, + IsDir: false, + Mode: 0644, + Size: size, + ModTime: modTime, + }) + } } - return children, nil + + return entries } // filterHierarchicalChildren filters rows to immediate children of a prefix. @@ -1004,63 +1522,126 @@ func (o *Operations) mkdirSynth(ctx context.Context, parsed *ParsedPath, info *s fsCtx := parsed.Context dirPath := parsed.PrimaryKey - // Check if directory already exists + // Reject directory names that collide with TigerFS virtual directories + if fsErr := checkReservedFilename(dirPath); fsErr != nil { + return fsErr + } + + // Parent-pointer model (ADR-017): use leaf name + parent_id + if info.Roles.ParentID != "" { + parts := strings.Split(dirPath, "/") + leafName := parts[len(parts)-1] + + // Check if directory already exists by resolving full path + if _, ok := o.resolveSynthPath(ctx, fsCtx.Schema, fsCtx.TableName, parts); ok { + return &FSError{Code: ErrExists, Message: "directory already exists"} + } + + // Ensure parent directories exist, get parent UUID + parentID, fsErr := o.resolveSynthParentID(ctx, fsCtx.Schema, fsCtx.TableName, info, dirPath) + if fsErr != nil { + return fsErr + } + + columns := []string{info.Roles.Filename, info.Roles.Filetype} + values := []interface{}{leafName, "directory"} + if parentID != "" { + columns = append(columns, info.Roles.ParentID) + values = append(values, parentID) + } + + insertedPK, dbErr := o.db.InsertRow(ctx, fsCtx.Schema, fsCtx.TableName, columns, values) + if dbErr != nil { + return &FSError{Code: ErrIO, Message: "failed to create directory", Cause: dbErr} + } + + o.logSynthOp(ctx, fsCtx.Schema, fsCtx.TableName, info, "create", insertedPK, dirPath) + + o.statCache.invalidate(fsCtx.Schema, fsCtx.TableName) + o.pathCache.invalidate(fsCtx.Schema, fsCtx.TableName) + return nil + } + + // Old model (pre-ADR-017): full-path filenames exists, err := o.synthRowExists(ctx, fsCtx.Schema, fsCtx.TableName, info, dirPath, "directory") if err != nil { return err } if exists { - return &FSError{ - Code: ErrExists, - Message: "directory already exists", - } + return &FSError{Code: ErrExists, Message: "directory already exists"} } - // Auto-create parent directories - if fsErr := o.ensureSynthParentDirs(ctx, fsCtx.Schema, fsCtx.TableName, info, dirPath); fsErr != nil { + if _, fsErr := o.resolveSynthParentID(ctx, fsCtx.Schema, fsCtx.TableName, info, dirPath); fsErr != nil { return fsErr } - // Insert the directory row columns := []string{info.Roles.Filename, info.Roles.Filetype} values := []interface{}{dirPath, "directory"} - _, dbErr := o.db.InsertRow(ctx, fsCtx.Schema, fsCtx.TableName, columns, values) + insertedPK, dbErr := o.db.InsertRow(ctx, fsCtx.Schema, fsCtx.TableName, columns, values) if dbErr != nil { - return &FSError{ - Code: ErrIO, - Message: "failed to create directory", - Cause: dbErr, - } + return &FSError{Code: ErrIO, Message: "failed to create directory", Cause: dbErr} } + o.logSynthOp(ctx, fsCtx.Schema, fsCtx.TableName, info, "create", insertedPK, dirPath) + o.statCache.invalidate(fsCtx.Schema, fsCtx.TableName) return nil } -// ensureSynthParentDirs auto-creates parent directory rows for a given path. -// For "projects/web/todo", creates "projects" and "projects/web" directory rows. -func (o *Operations) ensureSynthParentDirs(ctx context.Context, schema, table string, info *synth.ViewInfo, path string) *FSError { +// resolveSynthParentID returns the UUID of the immediate parent of path, +// asserting that every ancestor segment already exists in the source +// table. It does NOT auto-create missing dirs. +// +// For root-level paths (no slashes), returns ("", nil) -- meaning "the +// row will be at root, parent_id = NULL". +// +// For nested paths, walks ancestors via the path cache and the +// resolve_path PL/pgSQL function. If any segment doesn't resolve, returns +// ErrNotExist; callers should either issue Mkdir per segment first +// (matching POSIX mkdir(2) semantics that the kernel enforces for +// FUSE/NFS clients) or use WriteFileEnsureDirs for mkdir-p behavior. +// +// For the legacy path-encoded model (no parent_id column, pre-ADR-017), +// "ancestor" means a directory row whose filename is the parent path +// prefix; we verify each prefix has a matching row and surface +// ErrNotExist otherwise. +func (o *Operations) resolveSynthParentID(ctx context.Context, schema, table string, info *synth.ViewInfo, path string) (string, *FSError) { parts := strings.Split(path, "/") if len(parts) <= 1 { - return nil // No parents to create + return "", nil // Root level, no parent + } + + // Parent-pointer model (ADR-017): walk segments, return last UUID. + if info.Roles.ParentID != "" { + parentSegments := parts[:len(parts)-1] + parentID, ok := o.resolveSynthPath(ctx, schema, table, parentSegments) + if !ok { + return "", &FSError{ + Code: ErrNotExist, + Message: fmt.Sprintf("parent directory does not exist: %s", strings.Join(parentSegments, "/")), + Hint: "use Mkdir per segment, or WriteFileEnsureDirs to materialize the path", + } + } + return parentID, nil } - // Create each ancestor directory + // Old path-encoded model: each ancestor is a directory row whose + // filename is the prefix path. for i := 1; i < len(parts); i++ { parentPath := strings.Join(parts[:i], "/") - columns := []string{info.Roles.Filename, info.Roles.Filetype} - values := []interface{}{parentPath, "directory"} - err := o.db.InsertIfNotExists(ctx, schema, table, columns, values) - if err != nil { - return &FSError{ - Code: ErrIO, - Message: "failed to create parent directory", - Cause: err, + exists, fsErr := o.synthRowExists(ctx, schema, table, info, parentPath, "directory") + if fsErr != nil { + return "", fsErr + } + if !exists { + return "", &FSError{ + Code: ErrNotExist, + Message: fmt.Sprintf("parent directory does not exist: %s", parentPath), + Hint: "use Mkdir per segment, or WriteFileEnsureDirs to materialize the path", } } } - - return nil + return "", nil } // synthRowExists checks if a row exists in a synth view with the given filename and filetype. diff --git a/internal/tigerfs/fs/synth_ops_test.go b/internal/tigerfs/fs/synth_ops_test.go index d04cb5c..1e4b797 100644 --- a/internal/tigerfs/fs/synth_ops_test.go +++ b/internal/tigerfs/fs/synth_ops_test.go @@ -2,12 +2,14 @@ package fs import ( "context" + "fmt" "testing" "time" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "github.com/timescale/tigerfs/internal/tigerfs/config" + "github.com/timescale/tigerfs/internal/tigerfs/db" "github.com/timescale/tigerfs/internal/tigerfs/fs/synth" ) @@ -108,6 +110,59 @@ func TestSynth_ExtractModTime(t *testing.T) { } } +func TestSynth_StatDirectory_UsesModifiedAt(t *testing.T) { + dirModifiedAt := time.Date(2025, 3, 15, 10, 30, 0, 0, time.UTC) + + mockDB := &mockDBClient{ + tables: map[string][]string{"public": {"_notes"}}, + views: map[string][]string{"public": {"notes"}}, + viewComments: map[string]map[string]string{ + "public": {"notes": "tigerfs:md"}, + }, + columns: map[string][]mockColumn{ + "public.notes": { + {name: "id", dataType: "uuid"}, + {name: "parent_id", dataType: "uuid"}, + {name: "filename", dataType: "text"}, + {name: "filetype", dataType: "text"}, + {name: "title", dataType: "text"}, + {name: "body", dataType: "text"}, + {name: "encoding", dataType: "text"}, + {name: "created_at", dataType: "timestamptz"}, + {name: "modified_at", dataType: "timestamptz"}, + }, + }, + primaryKeys: map[string]*mockPK{ + "public._notes": {column: "id"}, + "public.notes": {column: "id"}, + }, + // resolveSynthPath returns the directory UUID + resolvePathResults: []db.PathSegment{ + {Depth: 1, ID: "uuid-dir-1", Name: "docs"}, + }, + // GetRow returns the directory row with a known modified_at + rowData: map[string]*mockRow{ + "public.notes.uuid-dir-1": { + columns: []string{"id", "parent_id", "filename", "filetype", "title", "body", "encoding", "created_at", "modified_at"}, + values: []interface{}{"uuid-dir-1", nil, "docs", "directory", nil, nil, "utf8", dirModifiedAt, dirModifiedAt}, + }, + }, + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + entry, fsErr := ops.Stat(ctx, "/notes/docs") + require.Nil(t, fsErr, "Stat should succeed for directory") + require.NotNil(t, entry) + + assert.True(t, entry.IsDir, "should be a directory") + assert.Equal(t, "docs", entry.Name) + assert.True(t, entry.ModTime.Equal(dirModifiedAt), + "directory ModTime should come from database modified_at, got %v, want %v", entry.ModTime, dirModifiedAt) +} + // newSynthHierarchicalMockDB creates a mock DB configured with a hierarchical synth markdown view. // The "memory" view has columns [id, filename, filetype, title, author, body] with: // - "projects" (directory) @@ -343,3 +398,425 @@ func TestSynth_NonHierarchicalViewUnchanged(t *testing.T) { assert.Contains(t, names, "second-post.md") assert.Len(t, entries, 2, "flat view should have exactly 2 entries") } + +// TestSynth_LogEntries_WriteSynthFile verifies that write operations on +// history-enabled synth apps create log entries. +func TestSynth_LogEntries_WriteSynthFile(t *testing.T) { + mockDB := &mockDBClient{ + tables: map[string][]string{"public": {"_notes"}}, + views: map[string][]string{"public": {"notes"}}, + viewComments: map[string]map[string]string{ + "public": {"notes": "tigerfs:md,history"}, + }, + columns: map[string][]mockColumn{ + "public.notes": { + {name: "id", dataType: "uuid"}, + {name: "filename", dataType: "text"}, + {name: "title", dataType: "text"}, + {name: "body", dataType: "text"}, + {name: "encoding", dataType: "text"}, + {name: "created_at", dataType: "timestamptz"}, + {name: "modified_at", dataType: "timestamptz"}, + }, + }, + primaryKeys: map[string]*mockPK{ + "public._notes": {column: "id"}, + "public.notes": {column: "id"}, + }, + allRowsData: map[string]*mockAllRows{ + "public.notes": { + columns: []string{"id", "filename", "title", "body", "encoding", "created_at", "modified_at"}, + rows: [][]interface{}{}, + }, + }, + // InsertRow returns a fake PK + lastInsertReturnPK: "uuid-new-1", + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + // INSERT: write a new file + content := "---\ntitle: Hello\n---\n# Hello\n" + writeErr := ops.WriteFile(ctx, "/notes/hello.md", []byte(content)) + require.Nil(t, writeErr, "WriteFile should succeed for insert") + + // Verify a log entry was created for the insert + require.Len(t, mockDB.logEntries, 1, "should have 1 log entry after insert") + assert.Equal(t, "create", mockDB.logEntries[0].opType) + assert.Equal(t, "uuid-new-1", mockDB.logEntries[0].fileID) + assert.Equal(t, "hello.md", mockDB.logEntries[0].filename) + assert.Empty(t, mockDB.logEntries[0].versionID, "create should have no version_id") +} + +// TestSynth_LogEntries_NoHistory verifies that write operations on synth apps +// WITHOUT history do NOT create log entries. +func TestSynth_LogEntries_NoHistory(t *testing.T) { + mockDB := &mockDBClient{ + tables: map[string][]string{"public": {"_notes"}}, + views: map[string][]string{"public": {"notes"}}, + viewComments: map[string]map[string]string{ + "public": {"notes": "tigerfs:md"}, // no ",history" + }, + columns: map[string][]mockColumn{ + "public.notes": { + {name: "id", dataType: "uuid"}, + {name: "filename", dataType: "text"}, + {name: "title", dataType: "text"}, + {name: "body", dataType: "text"}, + {name: "encoding", dataType: "text"}, + {name: "created_at", dataType: "timestamptz"}, + {name: "modified_at", dataType: "timestamptz"}, + }, + }, + primaryKeys: map[string]*mockPK{ + "public._notes": {column: "id"}, + "public.notes": {column: "id"}, + }, + allRowsData: map[string]*mockAllRows{ + "public.notes": { + columns: []string{"id", "filename", "title", "body", "encoding", "created_at", "modified_at"}, + rows: [][]interface{}{}, + }, + }, + lastInsertReturnPK: "uuid-new-1", + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + content := "---\ntitle: Hello\n---\n# Hello\n" + writeErr := ops.WriteFile(ctx, "/notes/hello.md", []byte(content)) + require.Nil(t, writeErr) + + // No history → no log entries + assert.Empty(t, mockDB.logEntries, "non-history app should not create log entries") +} + +// newHistoryMockDB creates a mock DB with a history-enabled synth app that has +// one existing row (hello.md with PK uuid-1). Used for UPDATE/DELETE/RENAME tests. +func newHistoryMockDB() *mockDBClient { + return &mockDBClient{ + tables: map[string][]string{"public": {"_notes"}}, + views: map[string][]string{"public": {"notes"}}, + viewComments: map[string]map[string]string{ + "public": {"notes": "tigerfs:md,history"}, + }, + columns: map[string][]mockColumn{ + "public.notes": { + {name: "id", dataType: "uuid"}, + {name: "filename", dataType: "text"}, + {name: "title", dataType: "text"}, + {name: "body", dataType: "text"}, + {name: "encoding", dataType: "text"}, + {name: "created_at", dataType: "timestamptz"}, + {name: "modified_at", dataType: "timestamptz"}, + }, + }, + primaryKeys: map[string]*mockPK{ + "public._notes": {column: "id"}, + "public.notes": {column: "id"}, + }, + allRowsData: map[string]*mockAllRows{ + "public.notes": { + columns: []string{"id", "filename", "title", "body", "encoding", "created_at", "modified_at"}, + rows: [][]interface{}{ + {"uuid-1", "hello.md", "Hello", "# Hello\n", "utf8", nil, nil}, + }, + }, + }, + latestVersionIDs: map[string]string{ + "uuid-1": "history-uuid-abc", + }, + // rowData is checked by DeleteRow mock + rowData: map[string]*mockRow{ + "public.notes.uuid-1": { + columns: []string{"id", "filename", "title", "body"}, + values: []interface{}{"uuid-1", "hello.md", "Hello", "# Hello\n"}, + }, + }, + } +} + +// TestSynth_LogEntries_EditSynthFile verifies that updating an existing file +// creates a log entry with type=edit and captures the version_id. +func TestSynth_LogEntries_EditSynthFile(t *testing.T) { + mockDB := newHistoryMockDB() + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + + // EDIT: write to an existing file + content := "---\ntitle: Hello Updated\n---\n# Hello Updated\n" + writeErr := ops.WriteFile(context.Background(), "/notes/hello.md", []byte(content)) + require.Nil(t, writeErr, "WriteFile should succeed for edit") + + require.Len(t, mockDB.logEntries, 1, "should have 1 log entry after edit") + assert.Equal(t, "edit", mockDB.logEntries[0].opType) + assert.Equal(t, "uuid-1", mockDB.logEntries[0].fileID) + assert.Equal(t, "hello.md", mockDB.logEntries[0].filename) + assert.Equal(t, "history-uuid-abc", mockDB.logEntries[0].versionID, "edit should capture version_id") +} + +// TestSynth_LogEntries_DeleteSynthFile verifies that deleting a file creates +// a log entry with type=delete and captures the version_id. +func TestSynth_LogEntries_DeleteSynthFile(t *testing.T) { + mockDB := newHistoryMockDB() + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + + deleteErr := ops.Delete(context.Background(), "/notes/hello.md") + require.Nil(t, deleteErr, "Delete should succeed") + + require.Len(t, mockDB.logEntries, 1, "should have 1 log entry after delete") + assert.Equal(t, "delete", mockDB.logEntries[0].opType) + assert.Equal(t, "uuid-1", mockDB.logEntries[0].fileID) + assert.Equal(t, "hello.md", mockDB.logEntries[0].filename) + assert.Equal(t, "history-uuid-abc", mockDB.logEntries[0].versionID, "delete should capture version_id") +} + +// TestSynth_LogEntries_RenameSynthFile verifies that renaming a file creates +// a log entry with type=rename, the new filename, and captures the version_id. +func TestSynth_LogEntries_RenameSynthFile(t *testing.T) { + mockDB := newHistoryMockDB() + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + + renameErr := ops.Rename(context.Background(), "/notes/hello.md", "/notes/goodbye.md") + require.Nil(t, renameErr, "Rename should succeed") + + require.Len(t, mockDB.logEntries, 1, "should have 1 log entry after rename") + assert.Equal(t, "rename", mockDB.logEntries[0].opType) + assert.Equal(t, "uuid-1", mockDB.logEntries[0].fileID) + assert.Equal(t, "goodbye.md", mockDB.logEntries[0].filename, "rename should log the NEW filename") + assert.Equal(t, "history-uuid-abc", mockDB.logEntries[0].versionID, "rename should capture version_id") +} + +// --- resolveSynthPath tests --- + +// TestSynth_ResolvePath_FullCacheMiss verifies that resolveSynthPath calls +// the DB resolve_path when the cache is empty and populates the cache. +func TestSynth_ResolvePath_FullCacheMiss(t *testing.T) { + mockDB := &mockDBClient{ + resolvePathResults: []db.PathSegment{ + {Depth: 1, ID: "uuid-proj", Name: "projects"}, + {Depth: 2, ID: "uuid-web", Name: "web"}, + {Depth: 3, ID: "uuid-todo", Name: "todo.md"}, + }, + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + id, ok := ops.resolveSynthPath(ctx, "public", "notes", []string{"projects", "web", "todo.md"}) + assert.True(t, ok) + assert.Equal(t, "uuid-todo", id) + assert.Equal(t, 1, mockDB.resolvePathCalls, "should call DB once") + + // Cache should now be populated -- second call should not hit DB + id, ok = ops.resolveSynthPath(ctx, "public", "notes", []string{"projects", "web", "todo.md"}) + assert.True(t, ok) + assert.Equal(t, "uuid-todo", id) + assert.Equal(t, 1, mockDB.resolvePathCalls, "should still be 1 DB call (cache hit)") +} + +// TestSynth_ResolvePath_PartialCacheHit verifies that resolveSynthPath +// only queries the DB for unresolved segments. +func TestSynth_ResolvePath_PartialCacheHit(t *testing.T) { + mockDB := &mockDBClient{ + resolvePathResults: []db.PathSegment{ + {Depth: 1, ID: "uuid-notes", Name: "notes.md"}, + }, + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + // Pre-populate cache for "projects" and "web" + ops.pathCache.put("public", "notes", "", "projects", "uuid-proj") + ops.pathCache.put("public", "notes", "uuid-proj", "web", "uuid-web") + + id, ok := ops.resolveSynthPath(ctx, "public", "notes", []string{"projects", "web", "notes.md"}) + assert.True(t, ok) + assert.Equal(t, "uuid-notes", id) + assert.Equal(t, 1, mockDB.resolvePathCalls) + + // Verify DB was called with only the remaining segment and correct startParentID + assert.Equal(t, "uuid-web", mockDB.lastResolveStartParent) + assert.Equal(t, []string{"notes.md"}, mockDB.lastResolveSegments) +} + +// TestSynth_ResolvePath_NonexistentPath verifies that resolveSynthPath +// returns false when a segment doesn't resolve. +func TestSynth_ResolvePath_NonexistentPath(t *testing.T) { + mockDB := &mockDBClient{ + resolvePathResults: []db.PathSegment{ + // Only first segment resolves, "nonexistent" doesn't + {Depth: 1, ID: "uuid-proj", Name: "projects"}, + }, + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + id, ok := ops.resolveSynthPath(ctx, "public", "notes", []string{"projects", "nonexistent", "file.md"}) + assert.False(t, ok) + assert.Empty(t, id) +} + +// TestSynth_ResolvePath_EmptySegments verifies root-level resolution. +func TestSynth_ResolvePath_EmptySegments(t *testing.T) { + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, &mockDBClient{}) + ctx := context.Background() + + id, ok := ops.resolveSynthPath(ctx, "public", "notes", []string{}) + assert.True(t, ok) + assert.Empty(t, id) +} + +// TestSynth_ResolvePath_SingleSegment verifies single-level resolution. +func TestSynth_ResolvePath_SingleSegment(t *testing.T) { + mockDB := &mockDBClient{ + resolvePathResults: []db.PathSegment{ + {Depth: 1, ID: "uuid-file", Name: "hello.md"}, + }, + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + id, ok := ops.resolveSynthPath(ctx, "public", "notes", []string{"hello.md"}) + assert.True(t, ok) + assert.Equal(t, "uuid-file", id) + assert.Equal(t, "", mockDB.lastResolveStartParent, "root-level should pass empty start parent") +} + +// TestSynth_ResolvePath_CacheInvalidation verifies that invalidation +// forces the next resolve to hit the DB. +func TestSynth_ResolvePath_CacheInvalidation(t *testing.T) { + mockDB := &mockDBClient{ + resolvePathResults: []db.PathSegment{ + {Depth: 1, ID: "uuid-1", Name: "file.md"}, + }, + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + // First call populates cache + ops.resolveSynthPath(ctx, "public", "notes", []string{"file.md"}) + assert.Equal(t, 1, mockDB.resolvePathCalls) + + // Invalidate cache + ops.pathCache.invalidate("public", "notes") + + // Second call should hit DB again + ops.resolveSynthPath(ctx, "public", "notes", []string{"file.md"}) + assert.Equal(t, 2, mockDB.resolvePathCalls) +} + +// TestSynth_ResolvePath_DeeplyNested verifies resolution of a 5-level deep path +// (ADR-017 verification scenario #13). +func TestSynth_ResolvePath_DeeplyNested(t *testing.T) { + mockDB := &mockDBClient{ + resolvePathResults: []db.PathSegment{ + {Depth: 1, ID: "uuid-a", Name: "a"}, + {Depth: 2, ID: "uuid-b", Name: "b"}, + {Depth: 3, ID: "uuid-c", Name: "c"}, + {Depth: 4, ID: "uuid-d", Name: "d"}, + {Depth: 5, ID: "uuid-file", Name: "file.md"}, + }, + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + id, ok := ops.resolveSynthPath(ctx, "public", "app", []string{"a", "b", "c", "d", "file.md"}) + assert.True(t, ok) + assert.Equal(t, "uuid-file", id) + + // Verify all 5 levels are cached + cached, ok := ops.pathCache.lookup("public", "app", "", "a") + assert.True(t, ok) + assert.Equal(t, "uuid-a", cached) + + cached, ok = ops.pathCache.lookup("public", "app", "uuid-a", "b") + assert.True(t, ok) + assert.Equal(t, "uuid-b", cached) + + cached, ok = ops.pathCache.lookup("public", "app", "uuid-b", "c") + assert.True(t, ok) + assert.Equal(t, "uuid-c", cached) + + cached, ok = ops.pathCache.lookup("public", "app", "uuid-c", "d") + assert.True(t, ok) + assert.Equal(t, "uuid-d", cached) + + cached, ok = ops.pathCache.lookup("public", "app", "uuid-d", "file.md") + assert.True(t, ok) + assert.Equal(t, "uuid-file", cached) +} + +// TestSynth_ResolvePath_SiblingAccess verifies the sibling resolution pattern +// described in ADR-017: after resolving projects/web/todo.md, resolving +// projects/web/notes.md should only query the DB for the last segment. +func TestSynth_ResolvePath_SiblingAccess(t *testing.T) { + mockDB := &mockDBClient{ + resolvePathResults: []db.PathSegment{ + {Depth: 1, ID: "uuid-proj", Name: "projects"}, + {Depth: 2, ID: "uuid-web", Name: "web"}, + {Depth: 3, ID: "uuid-todo", Name: "todo.md"}, + }, + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + // First access: resolve full path (cold cache) + id, ok := ops.resolveSynthPath(ctx, "public", "notes", []string{"projects", "web", "todo.md"}) + assert.True(t, ok) + assert.Equal(t, "uuid-todo", id) + assert.Equal(t, 1, mockDB.resolvePathCalls) + + // Set up mock for sibling -- only returns the leaf + mockDB.resolvePathResults = []db.PathSegment{ + {Depth: 1, ID: "uuid-notes", Name: "notes.md"}, + } + + // Second access: sibling file in same directory + id, ok = ops.resolveSynthPath(ctx, "public", "notes", []string{"projects", "web", "notes.md"}) + assert.True(t, ok) + assert.Equal(t, "uuid-notes", id) + assert.Equal(t, 2, mockDB.resolvePathCalls) + + // DB should have been called with only the leaf segment, starting from uuid-web + assert.Equal(t, "uuid-web", mockDB.lastResolveStartParent, + "should start from cached parent (web directory)") + assert.Equal(t, []string{"notes.md"}, mockDB.lastResolveSegments, + "should only query the unresolved leaf segment") +} + +// TestSynth_ResolvePath_DBError verifies that a DB error returns false +// without panicking. +func TestSynth_ResolvePath_DBError(t *testing.T) { + mockDB := &mockDBClient{ + resolvePathErr: fmt.Errorf("connection refused"), + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + id, ok := ops.resolveSynthPath(ctx, "public", "notes", []string{"projects", "file.md"}) + assert.False(t, ok) + assert.Empty(t, id) +} diff --git a/internal/tigerfs/fs/synth_parent_test.go b/internal/tigerfs/fs/synth_parent_test.go new file mode 100644 index 0000000..721c808 --- /dev/null +++ b/internal/tigerfs/fs/synth_parent_test.go @@ -0,0 +1,713 @@ +package fs + +import ( + "context" + "fmt" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/timescale/tigerfs/internal/tigerfs/config" + "github.com/timescale/tigerfs/internal/tigerfs/db" + "github.com/timescale/tigerfs/internal/tigerfs/fs/synth" +) + +// --- Helper function tests --- + +func TestJoinPathPrefix(t *testing.T) { + tests := []struct { + prefix string + leaf string + want string + }{ + {"", "file.md", "file.md"}, + {"projects", "file.md", "projects/file.md"}, + {"projects/web", "todo.md", "projects/web/todo.md"}, + {"a/b/c/d", "file.txt", "a/b/c/d/file.txt"}, + } + for _, tt := range tests { + t.Run(tt.want, func(t *testing.T) { + got := joinPathPrefix(tt.prefix, tt.leaf) + assert.Equal(t, tt.want, got) + }) + } +} + +func TestHistoryDBFilename(t *testing.T) { + withParent := &synth.ViewInfo{ + Format: synth.FormatMarkdown, + Roles: &synth.ColumnRoles{ParentID: "parent_id"}, + } + withoutParent := &synth.ViewInfo{ + Format: synth.FormatMarkdown, + Roles: &synth.ColumnRoles{}, + } + + // Parent-pointer model: always returns localName (leaf), ignores prefix + assert.Equal(t, "hello.md", historyDBFilename(withParent, "docs", "hello.md")) + assert.Equal(t, "hello.md", historyDBFilename(withParent, "", "hello.md")) + assert.Equal(t, "hello.md", historyDBFilename(withParent, "a/b/c", "hello.md")) + + // Old model: builds full path + assert.Equal(t, "docs/hello.md", historyDBFilename(withoutParent, "docs", "hello.md")) + assert.Equal(t, "hello.md", historyDBFilename(withoutParent, "", "hello.md")) + assert.Equal(t, "a/b/c/hello.md", historyDBFilename(withoutParent, "a/b/c", "hello.md")) +} + +// --- buildEntriesFromRows tests --- + +func TestSynth_BuildEntriesFromRows_MixedFileAndDir(t *testing.T) { + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, &mockDBClient{}) + + info := &synth.ViewInfo{ + Format: synth.FormatMarkdown, + Roles: &synth.ColumnRoles{ + Filename: "filename", + Filetype: "filetype", + Body: "body", + }, + CachedMountTime: time.Now(), + } + + columns := []string{"id", "parent_id", "filename", "filetype", "body"} + rows := [][]interface{}{ + {"uuid-1", nil, "docs", "directory", nil}, + {"uuid-2", nil, "readme", "file", "# Hello\n"}, + {"uuid-3", nil, "notes", "directory", nil}, + } + + entries := ops.buildEntriesFromRows(columns, rows, info) + require.Len(t, entries, 3) + + // Directory entries + assert.Equal(t, "docs", entries[0].Name) + assert.True(t, entries[0].IsDir) + assert.Equal(t, "notes", entries[2].Name) + assert.True(t, entries[2].IsDir) + + // File entry (markdown adds .md extension via GetMarkdownFilename) + assert.Equal(t, "readme", entries[1].Name) + assert.False(t, entries[1].IsDir) +} + +func TestSynth_BuildEntriesFromRows_Empty(t *testing.T) { + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, &mockDBClient{}) + + info := &synth.ViewInfo{ + Format: synth.FormatMarkdown, + Roles: &synth.ColumnRoles{ + Filename: "filename", + Filetype: "filetype", + Body: "body", + }, + } + + entries := ops.buildEntriesFromRows([]string{"filename", "filetype", "body"}, nil, info) + assert.Empty(t, entries) +} + +func TestSynth_BuildEntriesFromRows_PlainText(t *testing.T) { + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, &mockDBClient{}) + + info := &synth.ViewInfo{ + Format: synth.FormatPlainText, + Roles: &synth.ColumnRoles{ + Filename: "filename", + Filetype: "filetype", + Body: "body", + }, + } + + columns := []string{"filename", "filetype", "body"} + rows := [][]interface{}{ + {"notes", "file", "some content"}, + } + + entries := ops.buildEntriesFromRows(columns, rows, info) + require.Len(t, entries, 1) + assert.Equal(t, "notes", entries[0].Name) + assert.False(t, entries[0].IsDir) +} + +// --- resolveSynthRow tests --- + +func TestSynth_ResolveSynthRow_Found(t *testing.T) { + mockDB := &mockDBClient{ + resolvePathResults: []db.PathSegment{ + {Depth: 1, ID: "uuid-proj", Name: "projects"}, + {Depth: 2, ID: "uuid-file", Name: "todo.md"}, + }, + rowData: map[string]*mockRow{ + "public.notes.uuid-file": { + columns: []string{"id", "filename", "body"}, + values: []interface{}{"uuid-file", "todo.md", "content"}, + }, + }, + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + info := &synth.ViewInfo{ + Roles: &synth.ColumnRoles{PrimaryKey: "id", ParentID: "parent_id"}, + } + + columns, row, pkValue, fsErr := ops.resolveSynthRow(ctx, "public", "notes", info, "projects/todo.md") + require.Nil(t, fsErr) + assert.Equal(t, "uuid-file", pkValue) + assert.Equal(t, []string{"id", "filename", "body"}, columns) + assert.Equal(t, "todo.md", row[1]) +} + +func TestSynth_ResolveSynthRow_NotFound(t *testing.T) { + mockDB := &mockDBClient{ + resolvePathResults: []db.PathSegment{}, // path doesn't resolve + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + info := &synth.ViewInfo{ + Roles: &synth.ColumnRoles{PrimaryKey: "id", ParentID: "parent_id"}, + } + + _, _, _, fsErr := ops.resolveSynthRow(ctx, "public", "notes", info, "nonexistent/file.md") + require.NotNil(t, fsErr) + assert.Equal(t, ErrNotExist, fsErr.Code) +} + +func TestSynth_ResolveSynthRow_DBError(t *testing.T) { + mockDB := &mockDBClient{ + resolvePathErr: fmt.Errorf("connection refused"), + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + info := &synth.ViewInfo{ + Roles: &synth.ColumnRoles{PrimaryKey: "id", ParentID: "parent_id"}, + } + + _, _, _, fsErr := ops.resolveSynthRow(ctx, "public", "notes", info, "file.md") + require.NotNil(t, fsErr) + assert.Equal(t, ErrNotExist, fsErr.Code) +} + +// --- Parent-pointer write operation tests --- + +func newParentPointerMockDB() *mockDBClient { + return &mockDBClient{ + tables: map[string][]string{"public": {"_notes"}}, + views: map[string][]string{"public": {"notes"}}, + viewComments: map[string]map[string]string{ + "public": {"notes": "tigerfs:md,history"}, + }, + columns: map[string][]mockColumn{ + "public.notes": { + {name: "id", dataType: "uuid"}, + {name: "parent_id", dataType: "uuid"}, + {name: "filename", dataType: "text"}, + {name: "filetype", dataType: "text"}, + {name: "title", dataType: "text"}, + {name: "body", dataType: "text"}, + {name: "encoding", dataType: "text"}, + {name: "created_at", dataType: "timestamptz"}, + {name: "modified_at", dataType: "timestamptz"}, + }, + }, + primaryKeys: map[string]*mockPK{ + "public._notes": {column: "id"}, + "public.notes": {column: "id"}, + }, + allRowsData: map[string]*mockAllRows{ + "public.notes": { + columns: []string{"id", "parent_id", "filename", "filetype", "title", "body", "encoding", "created_at", "modified_at"}, + rows: [][]interface{}{}, + }, + }, + latestVersionIDs: map[string]string{}, + lastInsertReturnPK: "uuid-new-1", + } +} + +// TestSynth_ParentPointer_CreateRootFile verifies creating a file at root level +// sets parent_id to nil (root) and uses the leaf filename. +func TestSynth_ParentPointer_CreateRootFile(t *testing.T) { + mockDB := newParentPointerMockDB() + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + content := "---\ntitle: Hello\n---\n# Hello\n" + fsErr := ops.WriteFile(ctx, "/notes/hello.md", []byte(content)) + require.Nil(t, fsErr, "WriteFile at root should succeed") + + // Verify the insert used leaf filename (not full path since it's root) + require.Len(t, mockDB.insertedRows, 1) + insertCols := mockDB.insertedRows[0].columns + insertVals := mockDB.insertedRows[0].values + + // Find filename column value + for i, col := range insertCols { + if col == "filename" { + assert.Equal(t, "hello.md", insertVals[i], "should store leaf filename") + } + // parent_id should NOT be in the insert for root level + assert.NotEqual(t, "parent_id", col, "root-level file should not have parent_id in INSERT") + } + + // Log entry should use "create" type + require.Len(t, mockDB.logEntries, 1) + assert.Equal(t, "create", mockDB.logEntries[0].opType) +} + +// TestSynth_ParentPointer_EditExistingFile verifies that editing an existing file +// uses resolveSynthPath to find the file, then updates by UUID. +func TestSynth_ParentPointer_EditExistingFile(t *testing.T) { + mockDB := newParentPointerMockDB() + // File already exists: resolveSynthPath will find it + mockDB.resolvePathResults = []db.PathSegment{ + {Depth: 1, ID: "uuid-existing", Name: "hello.md"}, + } + mockDB.latestVersionIDs = map[string]string{ + "uuid-existing": "version-abc", + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + content := "---\ntitle: Updated\n---\n# Updated\n" + fsErr := ops.WriteFile(ctx, "/notes/hello.md", []byte(content)) + require.Nil(t, fsErr, "WriteFile edit should succeed") + + // Should have updated by PK, not inserted + assert.Empty(t, mockDB.insertedRows, "edit should UPDATE, not INSERT") + assert.Len(t, mockDB.logEntries, 1) + assert.Equal(t, "edit", mockDB.logEntries[0].opType) + assert.Equal(t, "uuid-existing", mockDB.logEntries[0].fileID) +} + +// TestSynth_ParentPointer_MkdirAtRoot verifies mkdir creates a directory at root level. +func TestSynth_ParentPointer_MkdirAtRoot(t *testing.T) { + mockDB := newParentPointerMockDB() + // resolveSynthPath returns false (dir doesn't exist yet) + mockDB.resolvePathResults = nil + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + fsErr := ops.Mkdir(ctx, "/notes/projects") + require.Nil(t, fsErr, "Mkdir at root should succeed") + + // Should insert with leaf name and filetype=directory + require.Len(t, mockDB.insertedRows, 1) + insertCols := mockDB.insertedRows[0].columns + insertVals := mockDB.insertedRows[0].values + + filenameIdx := -1 + filetypeIdx := -1 + for i, col := range insertCols { + if col == "filename" { + filenameIdx = i + } + if col == "filetype" { + filetypeIdx = i + } + } + require.GreaterOrEqual(t, filenameIdx, 0) + require.GreaterOrEqual(t, filetypeIdx, 0) + assert.Equal(t, "projects", insertVals[filenameIdx]) + assert.Equal(t, "directory", insertVals[filetypeIdx]) +} + +// TestSynth_ParentPointer_DeleteByUUID verifies delete resolves path and deletes by PK. +func TestSynth_ParentPointer_DeleteByUUID(t *testing.T) { + mockDB := newParentPointerMockDB() + // File exists + mockDB.resolvePathResults = []db.PathSegment{ + {Depth: 1, ID: "uuid-file", Name: "hello.md"}, + } + mockDB.rowData = map[string]*mockRow{ + "public.notes.uuid-file": { + columns: []string{"id", "parent_id", "filename", "filetype", "body"}, + values: []interface{}{"uuid-file", nil, "hello.md", "file", "content"}, + }, + } + mockDB.latestVersionIDs = map[string]string{ + "uuid-file": "version-xyz", + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + fsErr := ops.Delete(ctx, "/notes/hello.md") + require.Nil(t, fsErr, "Delete should succeed") + + assert.Equal(t, "delete", mockDB.logEntries[0].opType) + assert.Equal(t, "uuid-file", mockDB.logEntries[0].fileID) +} + +// TestSynth_ParentPointer_RenameSameDir verifies rename within same directory +// only changes the filename column. +func TestSynth_ParentPointer_RenameSameDir(t *testing.T) { + mockDB := newParentPointerMockDB() + // Old file exists + mockDB.resolvePathResults = []db.PathSegment{ + {Depth: 1, ID: "uuid-file", Name: "old.md"}, + } + mockDB.rowData = map[string]*mockRow{ + "public.notes.uuid-file": { + columns: []string{"id", "parent_id", "filename", "filetype", "body"}, + values: []interface{}{"uuid-file", nil, "old.md", "file", "content"}, + }, + } + mockDB.latestVersionIDs = map[string]string{ + "uuid-file": "version-abc", + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + fsErr := ops.Rename(ctx, "/notes/old.md", "/notes/new.md") + require.Nil(t, fsErr, "Rename same dir should succeed") + + assert.Equal(t, "rename", mockDB.logEntries[0].opType) + assert.Equal(t, "uuid-file", mockDB.logEntries[0].fileID) +} + +// --- Rename-as-replace tests --- + +// TestSynth_ParentPointer_RenameReplace verifies that renaming to an existing +// file atomically deletes the target and renames the source (POSIX semantics). +func TestSynth_ParentPointer_RenameReplace(t *testing.T) { + mockDB := newParentPointerMockDB() + // Source file exists + mockDB.resolvePathResults = []db.PathSegment{ + {Depth: 1, ID: "uuid-lock", Name: "config.lock"}, + } + mockDB.rowData = map[string]*mockRow{ + "public.notes.uuid-lock": { + columns: []string{"id", "parent_id", "filename", "filetype", "body"}, + values: []interface{}{"uuid-lock", nil, "config.lock", "file", "new content"}, + }, + } + mockDB.latestVersionIDs = map[string]string{ + "uuid-lock": "version-lock", + "uuid-config": "version-config", + } + + // Target "config" already exists -- resolveSynthPath for ["config"] returns uuid-config + mockDB.resolvePathCallCount = 0 + originalResolveFn := mockDB.resolvePathFn + mockDB.resolvePathFn = func(ctx context.Context, schema, table string, segments []string) ([]db.PathSegment, error) { + // First call: resolve source ("config.lock") + if mockDB.resolvePathCallCount == 0 { + mockDB.resolvePathCallCount++ + return []db.PathSegment{{Depth: 1, ID: "uuid-lock", Name: "config.lock"}}, nil + } + // Second call: resolve target ("config") -- exists + if mockDB.resolvePathCallCount == 1 { + mockDB.resolvePathCallCount++ + return []db.PathSegment{{Depth: 1, ID: "uuid-config", Name: "config"}}, nil + } + if originalResolveFn != nil { + return originalResolveFn(ctx, schema, table, segments) + } + return nil, nil + } + + // Track DeleteAndUpdate calls + deleteAndUpdateCalled := false + mockDB.deleteAndUpdateFunc = func(ctx context.Context, schema, table string, deletePK *db.PKMatch, updatePK *db.PKMatch, updateCols []string, updateVals []interface{}) error { + deleteAndUpdateCalled = true + assert.Equal(t, "uuid-config", deletePK.Values[0], "should delete the target file") + assert.Equal(t, "uuid-lock", updatePK.Values[0], "should update the source file") + return nil + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + fsErr := ops.Rename(ctx, "/notes/config.lock", "/notes/config") + require.Nil(t, fsErr, "Rename-as-replace should succeed") + + assert.True(t, deleteAndUpdateCalled, "should call DeleteAndUpdate for replace") + + // Should have two log entries: delete + rename + require.Len(t, mockDB.logEntries, 2) + assert.Equal(t, "delete", mockDB.logEntries[0].opType) + assert.Equal(t, "uuid-config", mockDB.logEntries[0].fileID) + assert.Equal(t, "rename", mockDB.logEntries[1].opType) + assert.Equal(t, "uuid-lock", mockDB.logEntries[1].fileID) +} + +// TestSynth_ParentPointer_RenameNoReplace verifies that simple rename (no target) +// uses the existing UpdateRow path, not DeleteAndUpdate. +func TestSynth_ParentPointer_RenameNoReplace(t *testing.T) { + mockDB := newParentPointerMockDB() + mockDB.resolvePathResults = []db.PathSegment{ + {Depth: 1, ID: "uuid-file", Name: "old.md"}, + } + mockDB.rowData = map[string]*mockRow{ + "public.notes.uuid-file": { + columns: []string{"id", "parent_id", "filename", "filetype", "body"}, + values: []interface{}{"uuid-file", nil, "old.md", "file", "content"}, + }, + } + mockDB.latestVersionIDs = map[string]string{ + "uuid-file": "version-abc", + } + + // Target doesn't exist -- resolveSynthPath for ["new.md"] returns empty + mockDB.resolvePathCallCount = 0 + mockDB.resolvePathFn = func(ctx context.Context, schema, table string, segments []string) ([]db.PathSegment, error) { + if mockDB.resolvePathCallCount == 0 { + mockDB.resolvePathCallCount++ + return []db.PathSegment{{Depth: 1, ID: "uuid-file", Name: "old.md"}}, nil + } + // Second call: target doesn't exist + mockDB.resolvePathCallCount++ + return nil, nil + } + + deleteAndUpdateCalled := false + mockDB.deleteAndUpdateFunc = func(ctx context.Context, schema, table string, deletePK *db.PKMatch, updatePK *db.PKMatch, updateCols []string, updateVals []interface{}) error { + deleteAndUpdateCalled = true + return nil + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + fsErr := ops.Rename(ctx, "/notes/old.md", "/notes/new.md") + require.Nil(t, fsErr, "Simple rename should succeed") + + assert.False(t, deleteAndUpdateCalled, "should NOT call DeleteAndUpdate for simple rename") + require.Len(t, mockDB.logEntries, 1) + assert.Equal(t, "rename", mockDB.logEntries[0].opType) +} + +// --- Reserved dotfile name tests --- + +// TestSynth_WriteReservedName verifies that creating a file with a reserved name fails. +// Known capabilities are intercepted by the path parser (routed to their handlers, not synth). +// The checkReservedFilename helper provides defense-in-depth for direct API calls. +func TestSynth_WriteReservedName(t *testing.T) { + reserved := []string{".history", ".log", ".savepoint", ".undo", ".info", ".by", ".filter", ".export"} + for _, name := range reserved { + t.Run(name, func(t *testing.T) { + mockDB := newParentPointerMockDB() + mockDB.resolvePathResults = nil + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, fmt.Sprintf("/notes/%s", name), []byte("test")) + require.NotNil(t, fsErr, "WriteFile(%q) should fail for reserved name", name) + }) + } + + // Non-reserved dotfiles should not fail with ErrPermission + allowed := []string{".gitignore", ".env", ".vscode", ".git"} + for _, name := range allowed { + t.Run(name, func(t *testing.T) { + mockDB := newParentPointerMockDB() + mockDB.resolvePathResults = nil + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, fmt.Sprintf("/notes/%s", name), []byte("test")) + if fsErr != nil { + assert.NotEqual(t, ErrPermission, fsErr.Code, + "WriteFile(%q) should not fail with ErrPermission", name) + } + }) + } +} + +// TestSynth_MkdirReservedName verifies that mkdir with a reserved name fails. +func TestSynth_MkdirReservedName(t *testing.T) { + reserved := []string{".history", ".log", ".undo"} + for _, name := range reserved { + t.Run(name, func(t *testing.T) { + mockDB := newParentPointerMockDB() + mockDB.resolvePathResults = nil + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + fsErr := ops.Mkdir(ctx, fmt.Sprintf("/notes/%s", name)) + require.NotNil(t, fsErr, "Mkdir(%q) should fail for reserved name", name) + }) + } + + allowed := []string{".vscode", ".git"} + for _, name := range allowed { + t.Run(name, func(t *testing.T) { + mockDB := newParentPointerMockDB() + mockDB.resolvePathResults = nil + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + fsErr := ops.Mkdir(ctx, fmt.Sprintf("/notes/%s", name)) + if fsErr != nil { + assert.NotEqual(t, ErrPermission, fsErr.Code, + "Mkdir(%q) should not fail with ErrPermission", name) + } + }) + } +} + +// TestSynth_RenameToReservedName verifies that renaming to a reserved name fails. +// Rename targets go through the path parser, so known capabilities are intercepted. +func TestSynth_RenameToReservedName(t *testing.T) { + reserved := []string{".history", ".log", ".undo"} + for _, name := range reserved { + t.Run(name, func(t *testing.T) { + mockDB := newParentPointerMockDB() + mockDB.resolvePathResults = []db.PathSegment{ + {Depth: 1, ID: "uuid-file", Name: "old.md"}, + } + mockDB.rowData = map[string]*mockRow{ + "public.notes.uuid-file": { + columns: []string{"id", "parent_id", "filename", "filetype", "body"}, + values: []interface{}{"uuid-file", nil, "old.md", "file", "content"}, + }, + } + mockDB.latestVersionIDs = map[string]string{ + "uuid-file": "version-xyz", + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + fsErr := ops.Rename(ctx, "/notes/old.md", fmt.Sprintf("/notes/%s", name)) + require.NotNil(t, fsErr, "Rename to %q should fail for reserved name", name) + }) + } + + // Rename to non-reserved dotfile should not fail with ErrPermission + t.Run(".gitignore", func(t *testing.T) { + mockDB := newParentPointerMockDB() + mockDB.resolvePathResults = []db.PathSegment{ + {Depth: 1, ID: "uuid-file", Name: "old.md"}, + } + mockDB.rowData = map[string]*mockRow{ + "public.notes.uuid-file": { + columns: []string{"id", "parent_id", "filename", "filetype", "body"}, + values: []interface{}{"uuid-file", nil, "old.md", "file", "content"}, + }, + } + mockDB.latestVersionIDs = map[string]string{ + "uuid-file": "version-xyz", + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + fsErr := ops.Rename(ctx, "/notes/old.md", "/notes/.gitignore") + if fsErr != nil { + assert.NotEqual(t, ErrPermission, fsErr.Code, + "Rename to .gitignore should not fail with ErrPermission") + } + }) +} + +// TestSynth_RenameFromDotfile verifies that renaming FROM a dotfile works fine. +func TestSynth_RenameFromDotfile(t *testing.T) { + mockDB := newParentPointerMockDB() + mockDB.resolvePathResults = []db.PathSegment{ + {Depth: 1, ID: "uuid-file", Name: ".gitignore"}, + } + mockDB.rowData = map[string]*mockRow{ + "public.notes.uuid-file": { + columns: []string{"id", "parent_id", "filename", "filetype", "body"}, + values: []interface{}{"uuid-file", nil, ".gitignore", "file", "content"}, + }, + } + mockDB.latestVersionIDs = map[string]string{ + "uuid-file": "version-xyz", + } + + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ctx := context.Background() + + fsErr := ops.Rename(ctx, "/notes/.gitignore", "/notes/ignored.txt") + require.Nil(t, fsErr, "Rename from dotfile should succeed") +} + +// TestSynth_FilterReservedNames verifies that reserved names are filtered from ReadDir. +func TestSynth_FilterReservedNames(t *testing.T) { + entries := []Entry{ + {Name: "hello.md"}, + {Name: ".gitignore"}, + {Name: ".history"}, // reserved -- should be filtered + {Name: ".log"}, // reserved -- should be filtered + {Name: ".env"}, + {Name: ".undo"}, // reserved -- should be filtered + } + + filtered := filterReservedNames(entries) + names := make([]string, len(filtered)) + for i, e := range filtered { + names[i] = e.Name + } + + assert.Contains(t, names, "hello.md") + assert.Contains(t, names, ".gitignore") + assert.Contains(t, names, ".env") + assert.NotContains(t, names, ".history") + assert.NotContains(t, names, ".log") + assert.NotContains(t, names, ".undo") +} + +// TestSynth_CheckReservedFilename verifies the reserved filename check helper. +func TestSynth_CheckReservedFilename(t *testing.T) { + tests := []struct { + filename string + wantErr bool + }{ + {".history", true}, + {".log", true}, + {".info", true}, + {".gitignore", false}, + {".env", false}, + {"subdir/.history", true}, // leaf is reserved + {"subdir/.gitignore", false}, // leaf is not reserved + {"hello.md", false}, + } + + for _, tt := range tests { + t.Run(tt.filename, func(t *testing.T) { + err := checkReservedFilename(tt.filename) + if tt.wantErr { + require.NotNil(t, err) + assert.Equal(t, ErrPermission, err.Code) + } else { + assert.Nil(t, err) + } + }) + } +} diff --git a/internal/tigerfs/fs/types.go b/internal/tigerfs/fs/types.go index 5f70549..d7a91a9 100644 --- a/internal/tigerfs/fs/types.go +++ b/internal/tigerfs/fs/types.go @@ -16,26 +16,36 @@ import ( "time" ) -// Entry represents a filesystem entry (file or directory). +// Entry represents a filesystem entry (file, directory, or symlink). // Used by ReadDir and Stat operations to describe filesystem structure. type Entry struct { // Name is the entry name (filename or directory name, not full path). Name string - // IsDir is true for directories, false for files. + // IsDir is true for directories, false for files and symlinks. IsDir bool // Size is the content size in bytes. For directories, this is typically - // a nominal value (e.g., 4096). + // a nominal value (e.g., 4096). For symlinks, this is the length of + // the target path. Size int64 // Mode contains the permission bits and file type. - // Use os.ModeDir for directories. + // Use os.ModeDir for directories, os.ModeSymlink for symlinks. Mode os.FileMode // ModTime is the modification time. For database rows, this may be // derived from a timestamp column or default to mount time. ModTime time.Time + + // Target is the symlink target path. Only set when Mode includes + // os.ModeSymlink. Empty for regular files and directories. + Target string +} + +// IsSymlink returns true if the entry represents a symlink. +func (e *Entry) IsSymlink() bool { + return e.Mode&os.ModeSymlink != 0 } // FileContent holds the content of a file read operation. diff --git a/internal/tigerfs/fs/types_test.go b/internal/tigerfs/fs/types_test.go index 8976f63..1ab8f0d 100644 --- a/internal/tigerfs/fs/types_test.go +++ b/internal/tigerfs/fs/types_test.go @@ -55,6 +55,121 @@ func TestFileContent(t *testing.T) { } } +// TestEntry_IsSymlink verifies symlink detection via Mode bits. +func TestEntry_IsSymlink(t *testing.T) { + tests := []struct { + name string + entry Entry + expected bool + }{ + { + name: "regular file", + entry: Entry{Name: "file.txt", Mode: 0644}, + expected: false, + }, + { + name: "directory", + entry: Entry{Name: "dir", IsDir: true, Mode: os.ModeDir | 0755}, + expected: false, + }, + { + name: "symlink", + entry: Entry{Name: "link", Mode: os.ModeSymlink | 0777, Target: "/dev/null"}, + expected: true, + }, + { + name: "symlink with zero perms", + entry: Entry{Name: "link", Mode: os.ModeSymlink}, + expected: true, + }, + { + name: "zero mode (regular file)", + entry: Entry{Name: "file"}, + expected: false, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := tt.entry.IsSymlink() + if got != tt.expected { + t.Errorf("IsSymlink() = %v, want %v (Mode=%v)", got, tt.expected, tt.entry.Mode) + } + }) + } +} + +// TestEntry_SymlinkFields verifies Target field on symlink entries. +func TestEntry_SymlinkFields(t *testing.T) { + entry := Entry{ + Name: "before", + Mode: os.ModeSymlink | 0777, + Target: "../../.history/docs/hello.md/2026-04-07T143000.123Z-abc", + Size: 53, + } + if !entry.IsSymlink() { + t.Fatal("entry should be a symlink") + } + if entry.Target == "" { + t.Fatal("symlink target should not be empty") + } + if entry.Size != 53 { + t.Errorf("Size = %d, want %d (length of target path)", entry.Size, 53) + } +} + +// TestEntry_IsSymlink_IsDirTakesPrecedence verifies that IsDir=true takes +// precedence in EntryToAttr even if ModeSymlink is also set (invalid state). +func TestEntry_IsSymlink_IsDirTakesPrecedence(t *testing.T) { + // This is an invalid combination -- IsDir and ModeSymlink should not both be set. + // IsSymlink() still returns true (it only checks Mode bits), but code that + // checks IsDir first (like EntryToAttr) will treat it as a directory. + entry := Entry{ + Name: "weird", + IsDir: true, + Mode: os.ModeDir | os.ModeSymlink | 0755, + } + // IsSymlink checks Mode only -- it doesn't know about IsDir + if !entry.IsSymlink() { + t.Error("IsSymlink() should be true when ModeSymlink is set, regardless of IsDir") + } + // But IsDir is also true -- callers that check IsDir first will treat as directory + if !entry.IsDir { + t.Error("IsDir should be true") + } +} + +// TestEntry_Symlink_EmptyTarget verifies behavior when ModeSymlink is set but +// Target is empty. +func TestEntry_Symlink_EmptyTarget(t *testing.T) { + entry := Entry{ + Name: "dangling", + Mode: os.ModeSymlink | 0777, + Target: "", + } + if !entry.IsSymlink() { + t.Error("should be a symlink even with empty target") + } + if entry.Target != "" { + t.Error("target should be empty") + } +} + +// TestEntry_Symlink_SizeMatchesTarget verifies the convention that symlink +// Size should equal len(Target). +func TestEntry_Symlink_SizeMatchesTarget(t *testing.T) { + target := "../../.history/docs/hello.md/2026-04-07T143000.123Z-abc" + entry := Entry{ + Name: "before", + Mode: os.ModeSymlink | 0777, + Target: target, + Size: int64(len(target)), + } + if entry.Size != int64(len(entry.Target)) { + t.Errorf("Size=%d != len(Target)=%d; NFS clients may use Size for buffer allocation", + entry.Size, len(entry.Target)) + } +} + // TestWriteHandle verifies WriteHandle struct and OnClose callback. func TestWriteHandle(t *testing.T) { var closedWith []byte diff --git a/internal/tigerfs/fs/undo.go b/internal/tigerfs/fs/undo.go new file mode 100644 index 0000000..c539fb5 --- /dev/null +++ b/internal/tigerfs/fs/undo.go @@ -0,0 +1,1113 @@ +package fs + +import ( + "context" + "encoding/json" + "fmt" + "os" + "strings" + "time" + + "github.com/google/uuid" + "github.com/timescale/tigerfs/internal/tigerfs/db" + "github.com/timescale/tigerfs/internal/tigerfs/format" + "github.com/timescale/tigerfs/internal/tigerfs/fs/synth" + "github.com/timescale/tigerfs/internal/tigerfs/logging" + "go.uber.org/zap" +) + +// UndoResult holds the outcome of an undo operation. +type UndoResult struct { + FilesDeleted int // rows created after target that were removed + FilesRestored int // rows restored from history (edit/rename/delete) + FilesSkipped int // no-op (e.g., created then already deleted) +} + +// ExecuteUndoToSavepoint undoes all operations after a named savepoint. +// Looks up savepoint_id by name, then delegates to ExecuteUndo. +func (o *Operations) ExecuteUndoToSavepoint(ctx context.Context, schema, tableName, savepointName string, filters []db.UndoFilter) (*UndoResult, error) { + savepointTable := tableName + "_savepoint" + + // Look up savepoint_id by name (name is PK) + row, err := o.db.GetRow(ctx, synth.TigerFSSchema, savepointTable, db.SinglePKMatch("name", savepointName)) + if err != nil { + return nil, fmt.Errorf("savepoint not found: %s", savepointName) + } + + // Extract savepoint_id from the row (pgx returns UUIDs as [16]byte) + var savepointID string + for i, col := range row.Columns { + if col == "savepoint_id" { + savepointID, _ = format.ConvertValueToText(row.Values[i]) + break + } + } + if savepointID == "" { + return nil, fmt.Errorf("savepoint %s has no savepoint_id", savepointName) + } + + desc := fmt.Sprintf("Undo to savepoint %s", savepointName) + return o.ExecuteUndo(ctx, schema, tableName, savepointID, desc, filters) +} + +// ExecuteUndoToLogID undoes all operations after a specific log entry. +// Accepts either a raw UUIDv7 or the display-name form +// ("2026-04-08T143015.234Z-...") -- the latter is what users see in +// .log/.by/... listings. ExecuteUndoSingle has the same resolution; this +// keeps the two log-id entry points symmetric and makes +// `.undo/to-id//.apply` work end-to-end. +func (o *Operations) ExecuteUndoToLogID(ctx context.Context, schema, tableName, logID string, filters []db.UndoFilter) (*UndoResult, error) { + logID = resolveLogID(logID) + desc := fmt.Sprintf("Undo to log entry %s", logID) + return o.ExecuteUndo(ctx, schema, tableName, logID, desc, filters) +} + +// ExecuteUndo undoes all operations after a target point (savepoint_id or log_id). +// Executes all changes in a single PostgreSQL transaction. +// The schema parameter identifies the user-facing schema for cache +// invalidation (where the view lives). The synth backing tables themselves +// always live in synth.TigerFSSchema. +func (o *Operations) ExecuteUndo(ctx context.Context, schema, tableName, afterID, description string, filters []db.UndoFilter) (*UndoResult, error) { + logTable := tableName + "_log" + historyTable := tableName + "_history" + + // Extract optional user_id filter for the query + var userID string + var queryFilters []db.UndoFilter + for _, f := range filters { + if f.Column == "user_id" { + userID = f.Value + } else { + queryFilters = append(queryFilters, f) + } + } + + // Find all affected files + affected, err := o.db.QueryUndoAffectedFiles(ctx, synth.TigerFSSchema, logTable, afterID, userID, queryFilters) + if err != nil { + return nil, fmt.Errorf("failed to query affected files: %w", err) + } + + if len(affected) == 0 { + return &UndoResult{}, nil + } + + // Classify affected files by action + var deleteFileIDs, deleteFilenames []string + var restoreVersionIDs, restoreFileIDs, restoreFilenames []string + skipped := 0 + + for _, f := range affected { + switch f.Type { + case "create": + // Row was created after target -- check if it still exists + exists, err := o.db.QueryFileExists(ctx, synth.TigerFSSchema, tableName, f.FileID) + if err != nil { + logging.Warn("failed to check file existence during undo", + zap.String("file_id", f.FileID), zap.Error(err)) + skipped++ + continue + } + if exists { + deleteFileIDs = append(deleteFileIDs, f.FileID) + deleteFilenames = append(deleteFilenames, f.Filename) + } else { + skipped++ // Created then already deleted -- no-op + } + + case "edit", "rename", "delete", "undo": + if f.VersionID == "" { + logging.Warn("undo: missing version_id for non-create operation", + zap.String("file_id", f.FileID), zap.String("type", f.Type)) + skipped++ + continue + } + restoreVersionIDs = append(restoreVersionIDs, f.VersionID) + restoreFileIDs = append(restoreFileIDs, f.FileID) + restoreFilenames = append(restoreFilenames, f.Filename) + + default: + logging.Warn("undo: unknown operation type", + zap.String("type", f.Type), zap.String("file_id", f.FileID)) + skipped++ + } + } + + if len(deleteFileIDs) == 0 && len(restoreVersionIDs) == 0 { + return &UndoResult{FilesSkipped: skipped}, nil + } + + // Execute all changes atomically + err = o.db.ExecuteUndoTransaction(ctx, &db.UndoTransactionParams{ + Schema: synth.TigerFSSchema, + SourceTable: tableName, + LogTable: logTable, + HistoryTable: historyTable, + Description: description, + DeleteFileIDs: deleteFileIDs, + DeleteFilenames: deleteFilenames, + RestoreVersionIDs: restoreVersionIDs, + RestoreFileIDs: restoreFileIDs, + RestoreFilenames: restoreFilenames, + UserID: o.userID, + }) + if err != nil { + return nil, fmt.Errorf("undo transaction failed: %w", err) + } + + // Invalidate caches under the user's schema (where the view lives and + // where statSynthFile populates the cache). Using synth.TigerFSSchema + // here would leave cached entries -- including negative entries -- in + // place after undo. + o.statCache.invalidate(schema, tableName) + o.undoCache.invalidate() + + result := &UndoResult{ + FilesDeleted: len(deleteFileIDs), + FilesRestored: len(restoreVersionIDs), + FilesSkipped: skipped, + } + + logging.Info("undo completed", + zap.String("table", tableName), + zap.Int("deleted", result.FilesDeleted), + zap.Int("restored", result.FilesRestored), + zap.Int("skipped", result.FilesSkipped)) + + return result, nil +} + +// resolveLogID converts a display name (e.g., "2026-04-08T143015.234Z-i9j0k1l2m3n4b") +// to a raw UUID string. If already a UUID, returns as-is. +func resolveLogID(id string) string { + if format.IsDisplayName(id) { + uuid, err := format.DisplayNameToUUIDv7(id) + if err == nil { + return uuid.String() + } + } + return id +} + +// ExecuteUndoSingle undoes a single log entry. +func (o *Operations) ExecuteUndoSingle(ctx context.Context, schema, tableName, logID string) (*UndoResult, error) { + logTable := tableName + "_log" + historyTable := tableName + "_history" + + logID = resolveLogID(logID) + + // Fetch the log entry + entry, err := o.db.QueryLogEntry(ctx, synth.TigerFSSchema, logTable, logID) + if err != nil { + return nil, fmt.Errorf("log entry not found: %s", logID) + } + + desc := fmt.Sprintf("Undo single operation %s", logID) + + var deleteFileIDs, deleteFilenames []string + var restoreVersionIDs, restoreFileIDs, restoreFilenames []string + skipped := 0 + + switch entry.Type { + case "create": + exists, err := o.db.QueryFileExists(ctx, synth.TigerFSSchema, tableName, entry.FileID) + if err != nil || !exists { + skipped = 1 + } else { + deleteFileIDs = append(deleteFileIDs, entry.FileID) + deleteFilenames = append(deleteFilenames, entry.Filename) + } + + case "edit", "rename", "delete", "undo": + if entry.VersionID == "" { + return nil, fmt.Errorf("cannot undo %s operation: no version_id (before-state not captured)", entry.Type) + } + restoreVersionIDs = append(restoreVersionIDs, entry.VersionID) + restoreFileIDs = append(restoreFileIDs, entry.FileID) + restoreFilenames = append(restoreFilenames, entry.Filename) + + default: + return nil, fmt.Errorf("unknown operation type: %s", entry.Type) + } + + if len(deleteFileIDs) == 0 && len(restoreVersionIDs) == 0 { + return &UndoResult{FilesSkipped: skipped}, nil + } + + err = o.db.ExecuteUndoTransaction(ctx, &db.UndoTransactionParams{ + Schema: synth.TigerFSSchema, + SourceTable: tableName, + LogTable: logTable, + HistoryTable: historyTable, + Description: desc, + DeleteFileIDs: deleteFileIDs, + DeleteFilenames: deleteFilenames, + RestoreVersionIDs: restoreVersionIDs, + RestoreFileIDs: restoreFileIDs, + RestoreFilenames: restoreFilenames, + UserID: o.userID, + }) + if err != nil { + return nil, fmt.Errorf("undo transaction failed: %w", err) + } + + o.statCache.invalidate(schema, tableName) + + return &UndoResult{ + FilesDeleted: len(deleteFileIDs), + FilesRestored: len(restoreVersionIDs), + FilesSkipped: skipped, + }, nil +} + +// --- Filesystem interface (.undo/ navigation, preview, apply) --- + +// readDirUndo handles ReadDir for .undo/ paths. +func (o *Operations) readDirUndo(ctx context.Context, parsed *ParsedPath) ([]Entry, *FSError) { + now := time.Now() + + // Level 0: .undo/ -- list modes + if parsed.UndoMode == "" { + return []Entry{ + {Name: "id", IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, + {Name: "to-id", IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, + {Name: "to-savepoint", IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, + }, nil + } + + // Level 1: .undo// -- list targets (log entries or savepoints) + if parsed.UndoTarget == "" { + return o.readDirUndoTargets(ctx, parsed) + } + + // Level 2: .undo/// -- preview directory + if parsed.UndoFile == "" && parsed.InfoFile == "" && !parsed.UndoApply { + // id/ mode: single operation, just show .info/ and .apply (no preview tree). + // Use .log//before and .log//after for diffing. + if parsed.UndoMode == "id" { + return []Entry{ + {Name: DirInfo, IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, + {Name: FileApply, IsDir: false, Mode: 0644, ModTime: now}, + }, nil + } + return o.readDirUndoPreview(ctx, parsed) + } + + // Level 3: .undo///.info/ -- info directory + if parsed.InfoFile == "." { + return []Entry{ + {Name: FileSummary, IsDir: false, Mode: 0444, ModTime: now}, + }, nil + } + + // Level 3: subdirectory within the preview tree (e.g., tutorials/) + if parsed.UndoFile != "" { + return o.readDirUndoSubdir(ctx, parsed) + } + + return nil, &FSError{Code: ErrInvalidPath, Message: "invalid .undo/ path for ReadDir"} +} + +// readDirUndoTargets lists log entries or savepoints for a given undo mode. +func (o *Operations) readDirUndoTargets(ctx context.Context, parsed *ParsedPath) ([]Entry, *FSError) { + // Redirect context to the appropriate table + switch parsed.UndoMode { + case "id", "to-id": + parsed.Context.TableName = parsed.OrigTableName + "_log" + parsed.Context.Schema = synth.TigerFSSchema + case "to-savepoint": + parsed.Context.TableName = parsed.OrigTableName + "_savepoint" + parsed.Context.Schema = synth.TigerFSSchema + } + + // Apply default limit if none set by pipeline + if parsed.Context.Limit == 0 { + limit := o.config.UndoListLimit + if limit <= 0 { + limit = 100 + } + parsed.Context.Limit = limit + // Default to most recent (descending) + if parsed.Context.LimitType == 0 { + parsed.Context.LimitType = LimitLast + } + } + + return o.readDirTable(ctx, parsed) +} + +// readDirUndoPreview builds a virtual directory listing for the undo preview. +func (o *Operations) readDirUndoPreview(ctx context.Context, parsed *ParsedPath) ([]Entry, *FSError) { + now := time.Now() + + affected, err := o.queryUndoAffected(ctx, parsed) + if err != nil { + return nil, &FSError{Code: ErrIO, Message: "failed to query undo preview", Cause: err} + } + + // Start with .info/ and .apply + entries := []Entry{ + {Name: DirInfo, IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, + {Name: FileApply, IsDir: false, Mode: 0644, ModTime: now}, + } + + // Build top-level entries only. Nested files appear inside their directories. + seenDirs := make(map[string]bool) + + for _, f := range affected { + parts := strings.Split(f.Filename, "/") + + if len(parts) > 1 { + // Nested file -- add top-level directory if not seen + topDir := parts[0] + if !seenDirs[topDir] { + seenDirs[topDir] = true + entries = append(entries, Entry{ + Name: topDir, IsDir: true, Mode: os.ModeDir | 0755, ModTime: now, + }) + } + } else { + // Root-level file + if f.Type == "create" { + entries = append(entries, Entry{ + Name: f.Filename, IsDir: false, Mode: os.ModeSymlink | 0777, Target: "/dev/null", ModTime: now, + }) + } else { + entries = append(entries, Entry{ + Name: f.Filename, IsDir: false, Mode: 0444, ModTime: now, + }) + } + } + } + + return entries, nil +} + +// readDirUndoSubdir lists files within a subdirectory of the undo preview tree. +func (o *Operations) readDirUndoSubdir(ctx context.Context, parsed *ParsedPath) ([]Entry, *FSError) { + now := time.Now() + prefix := parsed.UndoFile + "/" + + affected, err := o.queryUndoAffected(ctx, parsed) + if err != nil { + return nil, &FSError{Code: ErrIO, Message: "failed to query undo preview", Cause: err} + } + + // Collect entries that are direct children of this directory prefix + seenDirs := make(map[string]bool) + var entries []Entry + + for _, f := range affected { + if !strings.HasPrefix(f.Filename, prefix) { + continue + } + // Get the relative path after the prefix + rel := f.Filename[len(prefix):] + // If rel contains a slash, the immediate child is a directory + if idx := strings.Index(rel, "/"); idx >= 0 { + dirName := rel[:idx] + if !seenDirs[dirName] { + seenDirs[dirName] = true + entries = append(entries, Entry{ + Name: dirName, IsDir: true, Mode: os.ModeDir | 0755, ModTime: now, + }) + } + } else { + // Direct child file + if f.Type == "create" { + entries = append(entries, Entry{ + Name: rel, IsDir: false, Mode: os.ModeSymlink | 0777, Target: "/dev/null", ModTime: now, + }) + } else { + entries = append(entries, Entry{ + Name: rel, IsDir: false, Mode: 0444, ModTime: now, + }) + } + } + } + + if len(entries) == 0 { + return nil, &FSError{Code: ErrNotExist, Message: fmt.Sprintf("directory not found in undo preview: %s", parsed.UndoFile)} + } + + return entries, nil +} + +// queryUndoAffected returns the affected files for the current undo path. +func (o *Operations) queryUndoAffected(ctx context.Context, parsed *ParsedPath) ([]db.UndoAffectedFile, error) { + tableName := parsed.OrigTableName + logTable := tableName + "_log" + + // Build filters from pipeline context + var filters []db.UndoFilter + var userID string + var filterKey string + if parsed.Context != nil { + for _, f := range parsed.Context.Filters { + if f.Column == "user_id" { + userID = f.Value + } else { + filters = append(filters, db.UndoFilter{Column: f.Column, Value: f.Value}) + } + filterKey += f.Column + "=" + f.Value + "," + } + } + + // Check affected files cache + if cached, ok := o.undoCache.lookupAffected(parsed.UndoMode, parsed.UndoTarget, filterKey); ok { + return cached, nil + } + + var result []db.UndoAffectedFile + var err error + + switch parsed.UndoMode { + case "id": + // Single operation: return one entry (uses log entry cache) + logID := resolveLogID(parsed.UndoTarget) + entry, qErr := o.cachedQueryLogEntry(ctx, synth.TigerFSSchema, logTable, logID) + if qErr != nil { + return nil, qErr + } + result = []db.UndoAffectedFile{*entry} + + case "to-id": + afterID := resolveLogID(parsed.UndoTarget) + result, err = o.db.QueryUndoAffectedFiles(ctx, synth.TigerFSSchema, logTable, afterID, userID, filters) + + case "to-savepoint": + // Look up savepoint_id by name (uses savepoint cache) + savepointID, spErr := o.cachedSavepointID(ctx, tableName, parsed.UndoTarget) + if spErr != nil { + return nil, spErr + } + result, err = o.db.QueryUndoAffectedFiles(ctx, synth.TigerFSSchema, logTable, savepointID, userID, filters) + + default: + return nil, fmt.Errorf("unknown undo mode: %s", parsed.UndoMode) + } + + if err != nil { + return nil, err + } + + // Cache the result + o.undoCache.storeAffected(parsed.UndoMode, parsed.UndoTarget, filterKey, result) + return result, nil +} + +// statUndo handles Stat for .undo/ paths beyond the basic directory entry. +func (o *Operations) statUndo(ctx context.Context, parsed *ParsedPath) (*Entry, *FSError) { + now := time.Now() + + // .undo/ root or .undo// + if parsed.UndoTarget == "" { + name := DirUndo + if parsed.UndoMode != "" { + name = parsed.UndoMode + } + return &Entry{Name: name, IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, nil + } + + // .undo/// -- validate target exists + if parsed.UndoFile == "" && parsed.InfoFile == "" && !parsed.UndoApply { + if err := o.validateUndoTarget(ctx, parsed); err != nil { + return nil, err + } + return &Entry{Name: parsed.UndoTarget, IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, nil + } + + // For all sub-paths under a target, validate the target exists first + if err := o.validateUndoTarget(ctx, parsed); err != nil { + return nil, err + } + + // .apply + if parsed.UndoApply { + return &Entry{Name: FileApply, IsDir: false, Mode: 0644, ModTime: now}, nil + } + + // .info directory + if parsed.InfoFile == "." { + return &Entry{Name: DirInfo, IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, nil + } + + // .info/summary + if parsed.InfoFile != "" && parsed.InfoFile != "." { + name := parsed.InfoFile + if name == FileSummary || strings.HasPrefix(name, FileSummary+".") { + // Compute actual size by generating the summary content + fc, summaryErr := o.readUndoSummary(ctx, parsed) + size := int64(0) + if summaryErr == nil { + size = int64(len(fc.Data)) + } + return &Entry{Name: name, IsDir: false, Size: size, Mode: 0444, ModTime: now}, nil + } + return nil, &FSError{Code: ErrNotExist, Message: fmt.Sprintf("unknown info file: %s", name)} + } + + // Preview file (not available in id/ mode -- use .log//before instead) + if parsed.UndoFile != "" && parsed.UndoMode == "id" { + return nil, &FSError{Code: ErrNotExist, Message: "preview files not available in .undo/id/ mode; use .log//before for diffs"} + } + if parsed.UndoFile != "" { + // Check if it's an intermediate directory in the preview tree + affected, qErr := o.queryUndoAffected(ctx, parsed) + if qErr != nil { + return nil, &FSError{Code: ErrIO, Message: "failed to query undo preview", Cause: qErr} + } + for _, f := range affected { + if f.Filename == parsed.UndoFile { + if f.Type == "create" { + return &Entry{Name: parsed.UndoFile, IsDir: false, Mode: os.ModeSymlink | 0777, Target: "/dev/null", ModTime: now}, nil + } + // Compute actual size by rendering the preview content + fc, previewErr := o.readUndoPreviewFile(ctx, parsed) + size := int64(0) + if previewErr == nil { + size = int64(len(fc.Data)) + } + return &Entry{Name: parsed.UndoFile, IsDir: false, Size: size, Mode: 0444, ModTime: now}, nil + } + // Check if UndoFile is a directory prefix + if strings.HasPrefix(f.Filename, parsed.UndoFile+"/") { + return &Entry{Name: parsed.UndoFile, IsDir: true, Mode: os.ModeDir | 0755, ModTime: now}, nil + } + } + return nil, &FSError{Code: ErrNotExist, Message: fmt.Sprintf("file not found in undo preview: %s", parsed.UndoFile)} + } + + return nil, &FSError{Code: ErrNotExist, Message: "invalid .undo/ path"} +} + +// readFileUndo handles ReadFile for .undo/ paths. +func (o *Operations) readFileUndo(ctx context.Context, parsed *ParsedPath) (*FileContent, *FSError) { + // .info/summary (or summary.json, summary.csv, etc.) + if parsed.InfoFile != "" { + return o.readUndoSummary(ctx, parsed) + } + + // .apply -- write-only, return empty for NFS SETATTR compatibility + if parsed.UndoApply { + return &FileContent{Data: []byte{}}, nil + } + + // Preview file content (not available in id/ mode) + if parsed.UndoFile != "" { + if parsed.UndoMode == "id" { + return nil, &FSError{Code: ErrNotExist, Message: "preview files not available in .undo/id/ mode; use .log//before for diffs"} + } + return o.readUndoPreviewFile(ctx, parsed) + } + + return nil, &FSError{Code: ErrInvalidPath, Message: "cannot read .undo/ directory as file"} +} + +// readUndoSummary returns the .info/summary file for an undo preview. +func (o *Operations) readUndoSummary(ctx context.Context, parsed *ParsedPath) (*FileContent, *FSError) { + affected, err := o.queryUndoAffected(ctx, parsed) + if err != nil { + return nil, &FSError{Code: ErrIO, Message: "failed to query undo summary", Cause: err} + } + + // Determine output format from InfoFile name (summary.json, summary.csv, etc.) + outputFormat := "tsv" + infoFile := parsed.InfoFile + if idx := strings.LastIndex(infoFile, "."); idx > 0 { + ext := infoFile[idx:] + switch ext { + case ".json": + outputFormat = "json" + case ".csv": + outputFormat = "csv" + case ".yaml": + outputFormat = "yaml" + } + } + + // Gather metadata about the target + meta := o.undoSummaryMetadata(ctx, parsed) + + // Build file rows with enriched columns + var rows []summaryFileRow + for _, f := range affected { + ts := uuidTimestamp(f.LogID) + rows = append(rows, summaryFileRow{ + Type: f.Type, + Filename: f.Filename, + User: f.UserID, + Timestamp: ts, + }) + } + + var data []byte + switch outputFormat { + case "json": + data, err = o.formatSummaryJSON(meta, rows) + case "csv": + data = o.formatSummaryCSV(rows) + case "yaml": + data = o.formatSummaryYAML(meta, rows) + default: // tsv + data = o.formatSummaryTSV(meta, rows) + } + if err != nil { + return nil, &FSError{Code: ErrIO, Message: "failed to format undo summary", Cause: err} + } + + return &FileContent{Data: data}, nil +} + +// undoSummaryMeta holds metadata about the undo target for summary headers. +type undoSummaryMeta struct { + Mode string // "id", "to-id", "to-savepoint" + Target string // savepoint name or log_id display name + TargetTime string // timestamp of the target (from UUIDv7) + User string // user who created the savepoint (empty for log targets) + Description string // savepoint description (empty for log targets) + Affected int // number of affected files +} + +// undoSummaryMetadata gathers metadata about the undo target. +func (o *Operations) undoSummaryMetadata(ctx context.Context, parsed *ParsedPath) undoSummaryMeta { + meta := undoSummaryMeta{ + Mode: parsed.UndoMode, + Target: parsed.UndoTarget, + } + + if parsed.UndoMode == "to-savepoint" { + savepointTable := parsed.OrigTableName + "_savepoint" + row, ok := o.undoCache.lookupSavepoint(synth.TigerFSSchema, savepointTable, parsed.UndoTarget) + if !ok { + var err error + row, err = o.db.GetRow(ctx, synth.TigerFSSchema, savepointTable, db.SinglePKMatch("name", parsed.UndoTarget)) + if err == nil { + o.undoCache.storeSavepoint(synth.TigerFSSchema, savepointTable, parsed.UndoTarget, row) + } + } + if row != nil { + for i, col := range row.Columns { + switch col { + case "savepoint_id": + id, _ := format.ConvertValueToText(row.Values[i]) + meta.TargetTime = uuidTimestamp(id) + case "user_id": + if row.Values[i] != nil { + meta.User, _ = format.ConvertValueToText(row.Values[i]) + } + case "description": + if row.Values[i] != nil { + meta.Description, _ = format.ConvertValueToText(row.Values[i]) + } + } + } + } + } else { + // For id/ and to-id/, extract timestamp from the target log_id + meta.TargetTime = uuidTimestamp(resolveLogID(parsed.UndoTarget)) + } + + return meta +} + +// cachedSavepointID looks up a savepoint's savepoint_id by name, using the cache. +func (o *Operations) cachedSavepointID(ctx context.Context, tableName, savepointName string) (string, error) { + savepointTable := tableName + "_savepoint" + + // Check cache + if row, ok := o.undoCache.lookupSavepoint(synth.TigerFSSchema, savepointTable, savepointName); ok { + return extractSavepointID(row) + } + + // Query DB + row, err := o.db.GetRow(ctx, synth.TigerFSSchema, savepointTable, db.SinglePKMatch("name", savepointName)) + if err != nil { + return "", fmt.Errorf("savepoint not found: %s", savepointName) + } + + // Cache the row + o.undoCache.storeSavepoint(synth.TigerFSSchema, savepointTable, savepointName, row) + + return extractSavepointID(row) +} + +// extractSavepointID extracts the savepoint_id string from a savepoint row. +func extractSavepointID(row *db.Row) (string, error) { + for i, col := range row.Columns { + if col == "savepoint_id" { + id, _ := format.ConvertValueToText(row.Values[i]) + if id != "" { + return id, nil + } + } + } + return "", fmt.Errorf("savepoint row has no savepoint_id") +} + +// cachedQueryLogEntry fetches a log entry by ID, using the cache. +func (o *Operations) cachedQueryLogEntry(ctx context.Context, schema, logTable, logID string) (*db.UndoAffectedFile, error) { + // Check cache + if entry, ok := o.undoCache.lookupLogEntry(schema, logTable, logID); ok { + return entry, nil + } + + // Query DB + entry, err := o.db.QueryLogEntry(ctx, schema, logTable, logID) + if err != nil { + return nil, err + } + + // Cache + o.undoCache.storeLogEntry(schema, logTable, logID, entry) + return entry, nil +} + +// uuidTimestamp extracts a UTC timestamp string from a UUID string. +func uuidTimestamp(uuidStr string) string { + parsed, err := uuid.Parse(uuidStr) + if err != nil { + return "" + } + return format.ExtractUUIDv7Time(parsed).UTC().Format(time.RFC3339) +} + +type summaryFileRow struct { + Type string + Filename string + User string + Timestamp string +} + +func (o *Operations) formatSummaryTSV(meta undoSummaryMeta, rows []summaryFileRow) []byte { + var lines []string + + // Metadata headers as # comments + switch meta.Mode { + case "to-savepoint": + lines = append(lines, "# savepoint: "+meta.Target) + if meta.TargetTime != "" { + lines = append(lines, "# created: "+meta.TargetTime) + } + if meta.User != "" { + lines = append(lines, "# user: "+meta.User) + } + if meta.Description != "" { + lines = append(lines, "# description: "+meta.Description) + } + default: // id, to-id + if meta.TargetTime != "" { + lines = append(lines, "# target: "+meta.TargetTime) + } + } + + fileWord := "files" + if len(rows) == 1 { + fileWord = "file" + } + lines = append(lines, fmt.Sprintf("# affected: %d %s", len(rows), fileWord)) + lines = append(lines, "# type\tfilename\tuser\ttimestamp") + + for _, r := range rows { + lines = append(lines, r.Type+"\t"+r.Filename+"\t"+r.User+"\t"+r.Timestamp) + } + return []byte(strings.Join(lines, "\n") + "\n") +} + +func (o *Operations) formatSummaryCSV(rows []summaryFileRow) []byte { + var lines []string + lines = append(lines, "type,filename,user,timestamp") + for _, r := range rows { + lines = append(lines, r.Type+","+r.Filename+","+r.User+","+r.Timestamp) + } + return []byte(strings.Join(lines, "\n") + "\n") +} + +func (o *Operations) formatSummaryJSON(meta undoSummaryMeta, rows []summaryFileRow) ([]byte, error) { + // Build a structured JSON object + obj := map[string]interface{}{} + switch meta.Mode { + case "to-savepoint": + obj["savepoint"] = meta.Target + if meta.TargetTime != "" { + obj["created"] = meta.TargetTime + } + if meta.User != "" { + obj["user"] = meta.User + } + if meta.Description != "" { + obj["description"] = meta.Description + } + default: + if meta.TargetTime != "" { + obj["target"] = meta.TargetTime + } + } + obj["affected"] = len(rows) + + var files []map[string]string + for _, r := range rows { + files = append(files, map[string]string{ + "type": r.Type, "filename": r.Filename, "user": r.User, "timestamp": r.Timestamp, + }) + } + obj["files"] = files + + data, err := json.MarshalIndent(obj, "", " ") + if err != nil { + return nil, err + } + return append(data, '\n'), nil +} + +func (o *Operations) formatSummaryYAML(meta undoSummaryMeta, rows []summaryFileRow) []byte { + var lines []string + + switch meta.Mode { + case "to-savepoint": + lines = append(lines, "savepoint: "+meta.Target) + if meta.TargetTime != "" { + lines = append(lines, "created: "+meta.TargetTime) + } + if meta.User != "" { + lines = append(lines, "user: "+meta.User) + } + if meta.Description != "" { + lines = append(lines, "description: "+meta.Description) + } + default: + if meta.TargetTime != "" { + lines = append(lines, "target: "+meta.TargetTime) + } + } + + lines = append(lines, fmt.Sprintf("affected: %d", len(rows))) + lines = append(lines, "files:") + for _, r := range rows { + lines = append(lines, fmt.Sprintf(" - type: %s\n filename: %s\n user: %s\n timestamp: %s", + r.Type, r.Filename, r.User, r.Timestamp)) + } + return []byte(strings.Join(lines, "\n") + "\n") +} + +// readUndoPreviewFile returns the content of a file in the undo preview. +// For restore actions: returns the before-state from history. +// For delete actions: returns current file content (the file that will be deleted). +func (o *Operations) readUndoPreviewFile(ctx context.Context, parsed *ParsedPath) (*FileContent, *FSError) { + affected, err := o.queryUndoAffected(ctx, parsed) + if err != nil { + return nil, &FSError{Code: ErrIO, Message: "failed to query undo preview", Cause: err} + } + + // Find the matching file + for _, f := range affected { + if f.Filename != parsed.UndoFile { + continue + } + + if f.Type == "create" { + // File will be deleted on apply -- return current content + tableName := parsed.OrigTableName + return o.readSynthFileByID(ctx, synth.TigerFSSchema, tableName, f.FileID) + } + + // Restore: return before-state from history + if f.VersionID == "" { + return nil, &FSError{Code: ErrIO, Message: "no history version for undo preview"} + } + tableName := parsed.OrigTableName + historyTable := tableName + "_history" + return o.readHistoryByVersionID(ctx, synth.TigerFSSchema, historyTable, tableName, f.VersionID) + } + + return nil, &FSError{Code: ErrNotExist, Message: fmt.Sprintf("file not found in undo preview: %s", parsed.UndoFile)} +} + +// readSynthFileByID reads a synth file by its UUID (for preview of files to be deleted). +// The backing table is in the tigerfs schema, but the synth view info is registered +// under the public schema (where the view lives). +func (o *Operations) readSynthFileByID(ctx context.Context, schema, tableName, fileID string) (*FileContent, *FSError) { + row, err := o.db.GetRow(ctx, schema, tableName, db.SinglePKMatch("id", fileID)) + if err != nil { + return nil, &FSError{Code: ErrNotExist, Message: "file not found"} + } + + // Synth view is registered under public schema, not tigerfs schema. + viewSchema := o.cachedSchema + info := o.getSynthViewInfo(ctx, viewSchema, tableName) + if info == nil { + // Fallback: return raw TSV + data, fmtErr := format.RowToTSV(row.Columns, interfaceSlice(row.Values)) + if fmtErr != nil { + return nil, &FSError{Code: ErrIO, Message: "failed to format row", Cause: fmtErr} + } + return &FileContent{Data: data}, nil + } + + // Render through synth format layer + return o.renderSynthContent(row, info) +} + +// readHistoryByVersionID reads a history entry and renders it as synth content. +func (o *Operations) readHistoryByVersionID(ctx context.Context, schema, historyTable, sourceTable, versionID string) (*FileContent, *FSError) { + // Check history row cache (immutable data, safe to cache) + row, ok := o.undoCache.lookupHistoryRow(schema, historyTable, versionID) + if !ok { + var err error + row, err = o.db.GetRow(ctx, schema, historyTable, db.SinglePKMatch("version_id", versionID)) + if err != nil { + return nil, &FSError{Code: ErrNotExist, Message: fmt.Sprintf("history entry not found: %s", versionID)} + } + o.undoCache.storeHistoryRow(schema, historyTable, versionID, row) + } + + // Map history columns to source columns (file_id -> id, skip version_id/operation) + var columns []string + var values []interface{} + for i, col := range row.Columns { + if col == "version_id" || col == "operation" { + continue + } + if col == "file_id" { + col = "id" + } + columns = append(columns, col) + values = append(values, row.Values[i]) + } + + // Synth view is registered under public schema (where the view lives), + // not the tigerfs schema (where the backing table lives). + viewSchema := o.cachedSchema + info := o.getSynthViewInfo(ctx, viewSchema, sourceTable) + if info == nil { + data, fmtErr := format.RowToTSV(columns, values) + if fmtErr != nil { + return nil, &FSError{Code: ErrIO, Message: "failed to format history row", Cause: fmtErr} + } + return &FileContent{Data: data}, nil + } + + mappedRow := &db.Row{Columns: columns, Values: values} + return o.renderSynthContent(mappedRow, info) +} + +// renderSynthContent renders a row through the synth format layer. +func (o *Operations) renderSynthContent(row *db.Row, info *synth.ViewInfo) (*FileContent, *FSError) { + var data []byte + var err error + switch info.Format { + case synth.FormatMarkdown: + data, err = synth.SynthesizeMarkdown(row.Columns, interfaceSlice(row.Values), info.Roles) + case synth.FormatPlainText: + data, err = synth.SynthesizePlainText(row.Columns, interfaceSlice(row.Values), info.Roles) + default: + data, err = format.RowToTSV(row.Columns, interfaceSlice(row.Values)) + } + if err != nil { + return nil, &FSError{Code: ErrIO, Message: "failed to synthesize content", Cause: err} + } + return &FileContent{Data: data}, nil +} + +// writeUndoApply triggers an undo operation when .apply is written. +func (o *Operations) writeUndoApply(ctx context.Context, parsed *ParsedPath, data []byte) *FSError { + if !parsed.UndoApply { + return &FSError{ + Code: ErrPermission, + Message: ".undo/ is read-only except for .apply", + } + } + + // Reject .sample/ + .apply + if parsed.Context != nil && parsed.Context.LimitType == LimitSample { + return &FSError{ + Code: ErrInvalidArgument, + Message: ".sample/ cannot be combined with .apply (random undo is not supported)", + } + } + + tableName := parsed.OrigTableName + if tableName == "" { + return &FSError{Code: ErrInvalidPath, Message: ".undo/ requires a table context"} + } + + // Cache keys are (user_schema, view_name); the synth backing tables live + // in synth.TigerFSSchema but stat/path lookups happen against the user's + // view. Use the user's schema for invalidation so cached entries + // (including negatives) actually clear after undo. + // + // parsed.Context and parsed.Context.Schema are invariants here: + // processUndo (path.go) errors out on nil Context, and resolveSchema + // (operations.go) errors out when current_schema() can't be resolved. + // Either failure aborts WriteFile before this function is called. + cacheSchema := parsed.Context.Schema + + // Build filters from pipeline context + var filters []db.UndoFilter + if parsed.Context != nil { + for _, f := range parsed.Context.Filters { + filters = append(filters, db.UndoFilter{Column: f.Column, Value: f.Value}) + } + } + + var result *UndoResult + var err error + + switch parsed.UndoMode { + case "id": + result, err = o.ExecuteUndoSingle(ctx, cacheSchema, tableName, parsed.UndoTarget) + case "to-id": + result, err = o.ExecuteUndoToLogID(ctx, cacheSchema, tableName, parsed.UndoTarget, filters) + case "to-savepoint": + result, err = o.ExecuteUndoToSavepoint(ctx, cacheSchema, tableName, parsed.UndoTarget, filters) + default: + return &FSError{Code: ErrInvalidPath, Message: fmt.Sprintf("unknown undo mode: %s", parsed.UndoMode)} + } + + if err != nil { + return &FSError{Code: ErrIO, Message: "undo failed", Cause: err} + } + + logging.Info("undo applied via .apply", + zap.String("table", tableName), + zap.String("mode", parsed.UndoMode), + zap.String("target", parsed.UndoTarget), + zap.Int("files_restored", result.FilesRestored), + zap.Int("files_deleted", result.FilesDeleted), + zap.Int("files_skipped", result.FilesSkipped)) + + // Invalidate caches after undo + o.statCache.invalidate(cacheSchema, tableName) + o.pathCache.invalidate(cacheSchema, tableName) + o.undoCache.invalidate() + + return nil +} + +// validateUndoTarget checks that the undo target (savepoint name or log_id) exists. +func (o *Operations) validateUndoTarget(ctx context.Context, parsed *ParsedPath) *FSError { + tableName := parsed.OrigTableName + + switch parsed.UndoMode { + case "to-savepoint": + // Uses savepoint cache -- populates cache for subsequent queryUndoAffected calls + _, err := o.cachedSavepointID(ctx, tableName, parsed.UndoTarget) + if err != nil { + return &FSError{Code: ErrNotExist, Message: fmt.Sprintf("savepoint not found: %s", parsed.UndoTarget)} + } + case "id", "to-id": + // Uses log entry cache + logTable := tableName + "_log" + logID := resolveLogID(parsed.UndoTarget) + _, err := o.cachedQueryLogEntry(ctx, synth.TigerFSSchema, logTable, logID) + if err != nil { + return &FSError{Code: ErrNotExist, Message: fmt.Sprintf("log entry not found: %s", parsed.UndoTarget)} + } + } + return nil +} + +// interfaceSlice converts []interface{} (already the right type) for format functions. +func interfaceSlice(vals []interface{}) []interface{} { + return vals +} diff --git a/internal/tigerfs/fs/undo_cache.go b/internal/tigerfs/fs/undo_cache.go new file mode 100644 index 0000000..1e39b98 --- /dev/null +++ b/internal/tigerfs/fs/undo_cache.go @@ -0,0 +1,186 @@ +package fs + +import ( + "sync" + "time" + + "github.com/timescale/tigerfs/internal/tigerfs/db" +) + +// undoCacheTTL matches the stat cache TTL (2 seconds). +const undoCacheTTL = 2 * time.Second + +// undoCache provides short-lived caching for undo preview queries. +// Each NFS/FUSE RPC independently queries the same data; caching eliminates +// redundant DB queries within a single user operation (ls, cat, touch). +type undoCache struct { + mu sync.RWMutex + + // affectedFiles caches queryUndoAffected results. + // Key: "mode\x00target\x00filters" (e.g., "to-savepoint\x00before-edits\x00") + affectedFiles map[string]*affectedCacheEntry + + // savepointRows caches savepoint GetRow results (name → full row). + // Key: "schema\x00table\x00name" + savepointRows map[string]*savepointCacheEntry + + // logEntries caches QueryLogEntry results. + // Key: "schema\x00table\x00logID" + logEntries map[string]*logEntryCacheEntry +} + +type affectedCacheEntry struct { + files []db.UndoAffectedFile + created time.Time +} + +type savepointCacheEntry struct { + row *db.Row + created time.Time +} + +type logEntryCacheEntry struct { + entry *db.UndoAffectedFile + created time.Time +} + +// lookupAffected returns cached affected files if present and not expired. +func (c *undoCache) lookupAffected(mode, target, filterKey string) ([]db.UndoAffectedFile, bool) { + c.mu.RLock() + defer c.mu.RUnlock() + + if c.affectedFiles == nil { + return nil, false + } + key := mode + "\x00" + target + "\x00" + filterKey + entry := c.affectedFiles[key] + if entry == nil || time.Since(entry.created) > undoCacheTTL { + return nil, false + } + return entry.files, true +} + +// storeAffected caches affected files for a given undo target. +func (c *undoCache) storeAffected(mode, target, filterKey string, files []db.UndoAffectedFile) { + c.mu.Lock() + defer c.mu.Unlock() + + if c.affectedFiles == nil { + c.affectedFiles = make(map[string]*affectedCacheEntry) + } + key := mode + "\x00" + target + "\x00" + filterKey + c.affectedFiles[key] = &affectedCacheEntry{files: files, created: time.Now()} +} + +// lookupSavepoint returns a cached savepoint row if present and not expired. +func (c *undoCache) lookupSavepoint(schema, table, name string) (*db.Row, bool) { + c.mu.RLock() + defer c.mu.RUnlock() + + if c.savepointRows == nil { + return nil, false + } + key := schema + "\x00" + table + "\x00" + name + entry := c.savepointRows[key] + if entry == nil || time.Since(entry.created) > undoCacheTTL { + return nil, false + } + return entry.row, true +} + +// storeSavepoint caches a savepoint row. +func (c *undoCache) storeSavepoint(schema, table, name string, row *db.Row) { + c.mu.Lock() + defer c.mu.Unlock() + + if c.savepointRows == nil { + c.savepointRows = make(map[string]*savepointCacheEntry) + } + key := schema + "\x00" + table + "\x00" + name + c.savepointRows[key] = &savepointCacheEntry{row: row, created: time.Now()} +} + +// lookupLogEntry returns a cached log entry if present and not expired. +func (c *undoCache) lookupLogEntry(schema, table, logID string) (*db.UndoAffectedFile, bool) { + c.mu.RLock() + defer c.mu.RUnlock() + + if c.logEntries == nil { + return nil, false + } + key := schema + "\x00" + table + "\x00" + logID + entry := c.logEntries[key] + if entry == nil || time.Since(entry.created) > undoCacheTTL { + return nil, false + } + return entry.entry, true +} + +// storeLogEntry caches a log entry. +func (c *undoCache) storeLogEntry(schema, table, logID string, entry *db.UndoAffectedFile) { + c.mu.Lock() + defer c.mu.Unlock() + + if c.logEntries == nil { + c.logEntries = make(map[string]*logEntryCacheEntry) + } + key := schema + "\x00" + table + "\x00" + logID + c.logEntries[key] = &logEntryCacheEntry{entry: entry, created: time.Now()} +} + +// lookupHistoryRow returns a cached history row if present and not expired. +func (c *undoCache) lookupHistoryRow(schema, table, versionID string) (*db.Row, bool) { + c.mu.RLock() + defer c.mu.RUnlock() + + if c.savepointRows == nil { + return nil, false + } + // Reuse savepointRows map for history rows (same structure: key→*db.Row) + key := "history\x00" + schema + "\x00" + table + "\x00" + versionID + entry := c.savepointRows[key] + if entry == nil || time.Since(entry.created) > undoCacheTTL { + return nil, false + } + return entry.row, true +} + +// storeHistoryRow caches a history row. +func (c *undoCache) storeHistoryRow(schema, table, versionID string, row *db.Row) { + c.mu.Lock() + defer c.mu.Unlock() + + if c.savepointRows == nil { + c.savepointRows = make(map[string]*savepointCacheEntry) + } + key := "history\x00" + schema + "\x00" + table + "\x00" + versionID + c.savepointRows[key] = &savepointCacheEntry{row: row, created: time.Now()} +} + +// invalidate clears all undo caches. Called after undo execution. +// +// Concurrency note: there is a narrow window where a concurrent reader +// can re-populate a cache entry with pre-undo data. If a reader's +// QueryUndoAffectedFiles starts before the undo transaction COMMITs (so +// PG's READ COMMITTED snapshot reflects pre-undo state), then the writer +// commits and runs invalidate(), then the reader's query finally returns, +// the reader calls storeAffected() with stale rows. Subsequent lookups +// return that stale snapshot until the 2-second TTL expires. +// +// This is acceptable: the staleness only affects undo *preview* surfaces +// (.info/summary, .undo//), not the apply path itself, +// which re-queries fresh inside ExecuteUndoTransaction. The underlying +// log/savepoint/history rows are append-only or immutable, so stale +// log/savepoint/history cache entries reflect data that's still valid; +// only affectedFiles can become semantically stale. +// +// See docs/spec.md "Undo Preview Cache" for user-facing documentation +// of the same tradeoff. +func (c *undoCache) invalidate() { + c.mu.Lock() + defer c.mu.Unlock() + + c.affectedFiles = nil + c.savepointRows = nil + c.logEntries = nil +} diff --git a/internal/tigerfs/fs/undo_test.go b/internal/tigerfs/fs/undo_test.go new file mode 100644 index 0000000..4752ca7 --- /dev/null +++ b/internal/tigerfs/fs/undo_test.go @@ -0,0 +1,330 @@ +package fs + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/timescale/tigerfs/internal/tigerfs/config" + "github.com/timescale/tigerfs/internal/tigerfs/db" +) + +func newUndoMock() *mockDBClient { + return &mockDBClient{ + primaryKeys: map[string]*mockPK{ + "tigerfs.notes_savepoint": {column: "name"}, + }, + } +} + +func newUndoOps(mockDB *mockDBClient) *Operations { + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ops.SetUserID("agent-7") + return ops +} + +// --- ExecuteUndoSingle: decision matrix --- + +func TestUndo_Single_Create_RowExists(t *testing.T) { + mockDB := newUndoMock() + mockDB.undoLogEntry = &db.UndoAffectedFile{ + FileID: "file-1", Type: "create", VersionID: "", Filename: "hello.md", + } + mockDB.fileExistsMap = map[string]bool{"file-1": true} + ops := newUndoOps(mockDB) + + result, err := ops.ExecuteUndoSingle(context.Background(), "public", "notes", "log-1") + require.NoError(t, err) + assert.Equal(t, 1, result.FilesDeleted) + assert.Equal(t, 0, result.FilesRestored) + + require.NotNil(t, mockDB.lastUndoParams) + assert.Equal(t, []string{"file-1"}, mockDB.lastUndoParams.DeleteFileIDs) + assert.Empty(t, mockDB.lastUndoParams.RestoreVersionIDs) +} + +func TestUndo_Single_Create_RowAlreadyDeleted(t *testing.T) { + mockDB := newUndoMock() + mockDB.undoLogEntry = &db.UndoAffectedFile{ + FileID: "file-1", Type: "create", VersionID: "", Filename: "hello.md", + } + mockDB.fileExistsMap = map[string]bool{"file-1": false} + ops := newUndoOps(mockDB) + + result, err := ops.ExecuteUndoSingle(context.Background(), "public", "notes", "log-1") + require.NoError(t, err) + assert.Equal(t, 0, result.FilesDeleted) + assert.Equal(t, 1, result.FilesSkipped) + assert.Equal(t, 0, mockDB.undoTransactionCalls, "no transaction needed for no-op") +} + +func TestUndo_Single_Edit_RowExists(t *testing.T) { + mockDB := newUndoMock() + mockDB.undoLogEntry = &db.UndoAffectedFile{ + FileID: "file-1", Type: "edit", VersionID: "v-before", Filename: "hello.md", + } + ops := newUndoOps(mockDB) + + result, err := ops.ExecuteUndoSingle(context.Background(), "public", "notes", "log-1") + require.NoError(t, err) + assert.Equal(t, 1, result.FilesRestored) + + require.NotNil(t, mockDB.lastUndoParams) + assert.Equal(t, []string{"v-before"}, mockDB.lastUndoParams.RestoreVersionIDs) + assert.Equal(t, []string{"file-1"}, mockDB.lastUndoParams.RestoreFileIDs) +} + +func TestUndo_Single_Rename_RestoresParentAndFilename(t *testing.T) { + mockDB := newUndoMock() + mockDB.undoLogEntry = &db.UndoAffectedFile{ + FileID: "file-1", Type: "rename", VersionID: "v-before-rename", Filename: "old-name.md", + } + ops := newUndoOps(mockDB) + + result, err := ops.ExecuteUndoSingle(context.Background(), "public", "notes", "log-1") + require.NoError(t, err) + assert.Equal(t, 1, result.FilesRestored) + + require.NotNil(t, mockDB.lastUndoParams) + assert.Equal(t, []string{"v-before-rename"}, mockDB.lastUndoParams.RestoreVersionIDs) + assert.Equal(t, []string{"old-name.md"}, mockDB.lastUndoParams.RestoreFilenames) +} + +func TestUndo_Single_Delete_RestoresFromHistory(t *testing.T) { + mockDB := newUndoMock() + mockDB.undoLogEntry = &db.UndoAffectedFile{ + FileID: "file-1", Type: "delete", VersionID: "v-before-delete", Filename: "removed.md", + } + ops := newUndoOps(mockDB) + + result, err := ops.ExecuteUndoSingle(context.Background(), "public", "notes", "log-1") + require.NoError(t, err) + assert.Equal(t, 1, result.FilesRestored) + + require.NotNil(t, mockDB.lastUndoParams) + assert.Equal(t, []string{"v-before-delete"}, mockDB.lastUndoParams.RestoreVersionIDs) +} + +func TestUndo_Single_UndoType_SameAsEdit(t *testing.T) { + mockDB := newUndoMock() + mockDB.undoLogEntry = &db.UndoAffectedFile{ + FileID: "file-1", Type: "undo", VersionID: "v-before-undo", Filename: "hello.md", + } + ops := newUndoOps(mockDB) + + result, err := ops.ExecuteUndoSingle(context.Background(), "public", "notes", "log-1") + require.NoError(t, err) + assert.Equal(t, 1, result.FilesRestored) + assert.Equal(t, []string{"v-before-undo"}, mockDB.lastUndoParams.RestoreVersionIDs) +} + +func TestUndo_Single_Edit_MissingVersionID(t *testing.T) { + mockDB := newUndoMock() + mockDB.undoLogEntry = &db.UndoAffectedFile{ + FileID: "file-1", Type: "edit", VersionID: "", Filename: "hello.md", + } + ops := newUndoOps(mockDB) + + _, err := ops.ExecuteUndoSingle(context.Background(), "public", "notes", "log-1") + require.Error(t, err) + assert.Contains(t, err.Error(), "no version_id") +} + +func TestUndo_Single_NotFound(t *testing.T) { + mockDB := newUndoMock() + // undoLogEntry is nil -- QueryLogEntry returns error + ops := newUndoOps(mockDB) + + _, err := ops.ExecuteUndoSingle(context.Background(), "public", "notes", "nonexistent") + require.Error(t, err) + assert.Contains(t, err.Error(), "not found") +} + +// --- ExecuteUndo: multi-file --- + +func TestUndo_Multi_MixedOperations(t *testing.T) { + mockDB := newUndoMock() + mockDB.undoAffectedFiles = []db.UndoAffectedFile{ + {FileID: "file-a", Type: "create", VersionID: "", Filename: "new-file.md"}, + {FileID: "file-b", Type: "edit", VersionID: "v-b-before", Filename: "edited.md"}, + {FileID: "file-c", Type: "delete", VersionID: "v-c-before", Filename: "deleted.md"}, + } + mockDB.fileExistsMap = map[string]bool{ + "file-a": true, // created and still exists → DELETE + "file-b": true, // edited + "file-c": false, // deleted + } + ops := newUndoOps(mockDB) + + result, err := ops.ExecuteUndo(context.Background(), "public", "notes", "sp-id-1", "test undo", nil) + require.NoError(t, err) + assert.Equal(t, 1, result.FilesDeleted) // file-a + assert.Equal(t, 2, result.FilesRestored) // file-b, file-c + assert.Equal(t, 0, result.FilesSkipped) + + p := mockDB.lastUndoParams + require.NotNil(t, p) + assert.Equal(t, []string{"file-a"}, p.DeleteFileIDs) + assert.Equal(t, []string{"v-b-before", "v-c-before"}, p.RestoreVersionIDs) + assert.Equal(t, "agent-7", p.UserID) + assert.Equal(t, "test undo", p.Description) +} + +func TestUndo_Multi_CreateThenDeleted_NoOp(t *testing.T) { + mockDB := newUndoMock() + mockDB.undoAffectedFiles = []db.UndoAffectedFile{ + {FileID: "file-a", Type: "create", VersionID: "", Filename: "ephemeral.md"}, + } + mockDB.fileExistsMap = map[string]bool{"file-a": false} // created then deleted + ops := newUndoOps(mockDB) + + result, err := ops.ExecuteUndo(context.Background(), "public", "notes", "sp-id-1", "test", nil) + require.NoError(t, err) + assert.Equal(t, 0, result.FilesDeleted) + assert.Equal(t, 1, result.FilesSkipped) + assert.Equal(t, 0, mockDB.undoTransactionCalls, "no transaction for all-skip") +} + +func TestUndo_Multi_NoAffectedFiles(t *testing.T) { + mockDB := newUndoMock() + mockDB.undoAffectedFiles = nil // no operations after target + ops := newUndoOps(mockDB) + + result, err := ops.ExecuteUndo(context.Background(), "public", "notes", "sp-id-1", "test", nil) + require.NoError(t, err) + assert.Equal(t, 0, result.FilesDeleted) + assert.Equal(t, 0, result.FilesRestored) + assert.Equal(t, 0, result.FilesSkipped) + assert.Equal(t, 0, mockDB.undoTransactionCalls) +} + +// --- ExecuteUndoToSavepoint --- + +func TestUndo_ToSavepoint_LooksUpSavepointID(t *testing.T) { + mockDB := newUndoMock() + mockDB.rowData = map[string]*mockRow{ + "tigerfs.notes_savepoint.before-refactor": { + columns: []string{"name", "savepoint_id", "user_id", "description"}, + values: []interface{}{"before-refactor", "sp-uuid-123", "agent-7", "test"}, + }, + } + mockDB.undoAffectedFiles = []db.UndoAffectedFile{ + {FileID: "file-1", Type: "edit", VersionID: "v-1", Filename: "hello.md"}, + } + ops := newUndoOps(mockDB) + + result, err := ops.ExecuteUndoToSavepoint(context.Background(), "public", "notes", "before-refactor", nil) + require.NoError(t, err) + assert.Equal(t, 1, result.FilesRestored) + assert.Contains(t, mockDB.lastUndoParams.Description, "before-refactor") +} + +func TestUndo_ToSavepoint_NotFound(t *testing.T) { + mockDB := newUndoMock() + // No savepoint row data + ops := newUndoOps(mockDB) + + _, err := ops.ExecuteUndoToSavepoint(context.Background(), "public", "notes", "nonexistent", nil) + require.Error(t, err) + assert.Contains(t, err.Error(), "savepoint not found") +} + +// --- Undo log entry metadata --- + +func TestUndo_LogEntries_HaveUserIDAndDescription(t *testing.T) { + mockDB := newUndoMock() + mockDB.undoLogEntry = &db.UndoAffectedFile{ + FileID: "file-1", Type: "edit", VersionID: "v-1", Filename: "hello.md", + } + ops := newUndoOps(mockDB) + ops.SetUserID("demo-user") + + _, err := ops.ExecuteUndoSingle(context.Background(), "public", "notes", "log-1") + require.NoError(t, err) + + p := mockDB.lastUndoParams + assert.Equal(t, "demo-user", p.UserID) + assert.Contains(t, p.Description, "Undo single operation") +} + +// --- Filter handling --- + +func TestUndo_Multi_UserIDFilter(t *testing.T) { + // The user_id filter is passed through to QueryUndoAffectedFiles. + // We verify it reaches the params correctly. + mockDB := newUndoMock() + mockDB.undoAffectedFiles = nil // empty is fine, we just test the call path + ops := newUndoOps(mockDB) + + filters := []db.UndoFilter{{Column: "user_id", Value: "agent-9"}} + _, err := ops.ExecuteUndo(context.Background(), "public", "notes", "sp-1", "test", filters) + require.NoError(t, err) + // The mock doesn't verify the userID parameter directly, but the code path is exercised. +} + +func TestUndo_Multi_TypeFilter(t *testing.T) { + mockDB := newUndoMock() + mockDB.undoAffectedFiles = nil + ops := newUndoOps(mockDB) + + filters := []db.UndoFilter{{Column: "type", Value: "delete"}} + _, err := ops.ExecuteUndo(context.Background(), "public", "notes", "sp-1", "test", filters) + require.NoError(t, err) +} + +func TestUndo_Multi_CombinedFilters(t *testing.T) { + mockDB := newUndoMock() + mockDB.undoAffectedFiles = nil + ops := newUndoOps(mockDB) + + filters := []db.UndoFilter{ + {Column: "user_id", Value: "agent-7"}, + {Column: "type", Value: "delete"}, + } + _, err := ops.ExecuteUndo(context.Background(), "public", "notes", "sp-1", "test", filters) + require.NoError(t, err) +} + +// --- Edge cases --- + +func TestUndo_Single_UnknownType(t *testing.T) { + mockDB := newUndoMock() + mockDB.undoLogEntry = &db.UndoAffectedFile{ + FileID: "file-1", Type: "unknown", VersionID: "", Filename: "hello.md", + } + ops := newUndoOps(mockDB) + + _, err := ops.ExecuteUndoSingle(context.Background(), "public", "notes", "log-1") + require.Error(t, err) + assert.Contains(t, err.Error(), "unknown operation type") +} + +func TestUndo_Multi_SkipsUnknownType(t *testing.T) { + mockDB := newUndoMock() + mockDB.undoAffectedFiles = []db.UndoAffectedFile{ + {FileID: "file-1", Type: "unknown", VersionID: "", Filename: "weird.md"}, + {FileID: "file-2", Type: "edit", VersionID: "v-2", Filename: "normal.md"}, + } + ops := newUndoOps(mockDB) + + result, err := ops.ExecuteUndo(context.Background(), "public", "notes", "sp-1", "test", nil) + require.NoError(t, err) + assert.Equal(t, 1, result.FilesRestored) // file-2 + assert.Equal(t, 1, result.FilesSkipped) // file-1 (unknown type) +} + +func TestUndo_Multi_SkipsMissingVersionID(t *testing.T) { + mockDB := newUndoMock() + mockDB.undoAffectedFiles = []db.UndoAffectedFile{ + {FileID: "file-1", Type: "edit", VersionID: "", Filename: "no-version.md"}, + {FileID: "file-2", Type: "delete", VersionID: "v-2", Filename: "good.md"}, + } + ops := newUndoOps(mockDB) + + result, err := ops.ExecuteUndo(context.Background(), "public", "notes", "sp-1", "test", nil) + require.NoError(t, err) + assert.Equal(t, 1, result.FilesRestored) // file-2 + assert.Equal(t, 1, result.FilesSkipped) // file-1 (missing version_id) +} diff --git a/internal/tigerfs/fs/user_identity_test.go b/internal/tigerfs/fs/user_identity_test.go new file mode 100644 index 0000000..9a16c42 --- /dev/null +++ b/internal/tigerfs/fs/user_identity_test.go @@ -0,0 +1,209 @@ +package fs + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/timescale/tigerfs/internal/tigerfs/config" + "github.com/timescale/tigerfs/internal/tigerfs/db" +) + +// --- Path parsing tests --- + +func TestParsePath_RootInfo(t *testing.T) { + result, err := ParsePath("/.info") + require.Nil(t, err) + assert.Equal(t, PathRootInfo, result.Type) + assert.Empty(t, result.InfoFile) +} + +func TestParsePath_RootInfoUser(t *testing.T) { + result, err := ParsePath("/.info/user") + require.Nil(t, err) + assert.Equal(t, PathRootInfo, result.Type) + assert.Equal(t, "user", result.InfoFile) +} + +func TestParsePath_RootInfoUnknown(t *testing.T) { + result, err := ParsePath("/.info/nonexistent") + require.Nil(t, err) + assert.Equal(t, PathRootInfo, result.Type) + assert.Equal(t, "nonexistent", result.InfoFile) +} + +// --- Operations user identity tests --- + +func TestOperations_UserID_DefaultEmpty(t *testing.T) { + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, &mockDBClient{}) + assert.Empty(t, ops.GetUserID()) +} + +func TestOperations_UserID_SetGet(t *testing.T) { + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, &mockDBClient{}) + ops.SetUserID("agent-7") + assert.Equal(t, "agent-7", ops.GetUserID()) +} + +func TestOperations_UserID_FromConfig(t *testing.T) { + cfg := &config.Config{DirListingLimit: 1000, UserID: "config-user"} + ops := NewOperations(cfg, &mockDBClient{}) + ops.SetUserID(cfg.UserID) + assert.Equal(t, "config-user", ops.GetUserID()) +} + +// --- .info/user ReadDir/Stat/ReadFile/WriteFile tests --- + +func TestRootInfo_ReadDir(t *testing.T) { + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, &mockDBClient{}) + ops.SetUserID("test-user") + + entries, fsErr := ops.ReadDir(context.Background(), "/.info") + require.Nil(t, fsErr) + require.Len(t, entries, 1) + assert.Equal(t, "user", entries[0].Name) + assert.False(t, entries[0].IsDir) +} + +func TestRootInfo_StatDir(t *testing.T) { + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, &mockDBClient{}) + + entry, fsErr := ops.Stat(context.Background(), "/.info") + require.Nil(t, fsErr) + assert.Equal(t, ".info", entry.Name) + assert.True(t, entry.IsDir) +} + +func TestRootInfo_StatUser(t *testing.T) { + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, &mockDBClient{}) + ops.SetUserID("agent-7") + + entry, fsErr := ops.Stat(context.Background(), "/.info/user") + require.Nil(t, fsErr) + assert.Equal(t, "user", entry.Name) + assert.False(t, entry.IsDir) + assert.Equal(t, int64(len("agent-7\n")), entry.Size) +} + +func TestRootInfo_StatUnknown(t *testing.T) { + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, &mockDBClient{}) + + _, fsErr := ops.Stat(context.Background(), "/.info/nonexistent") + require.NotNil(t, fsErr) + assert.Equal(t, ErrNotExist, fsErr.Code) +} + +func TestRootInfo_ReadUser(t *testing.T) { + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, &mockDBClient{}) + ops.SetUserID("agent-7") + + content, fsErr := ops.ReadFile(context.Background(), "/.info/user") + require.Nil(t, fsErr) + assert.Equal(t, "agent-7\n", string(content.Data)) +} + +func TestRootInfo_ReadUserEmpty(t *testing.T) { + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, &mockDBClient{}) + + content, fsErr := ops.ReadFile(context.Background(), "/.info/user") + require.Nil(t, fsErr) + assert.Equal(t, "\n", string(content.Data)) +} + +func TestRootInfo_WriteUser(t *testing.T) { + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, &mockDBClient{}) + + fsErr := ops.WriteFile(context.Background(), "/.info/user", []byte("agent-9\n")) + require.Nil(t, fsErr) + assert.Equal(t, "agent-9", ops.GetUserID()) +} + +func TestRootInfo_WriteUserTrimsWhitespace(t *testing.T) { + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, &mockDBClient{}) + + fsErr := ops.WriteFile(context.Background(), "/.info/user", []byte(" agent-9 \n")) + require.Nil(t, fsErr) + assert.Equal(t, "agent-9", ops.GetUserID()) +} + +func TestRootInfo_WriteUserClear(t *testing.T) { + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, &mockDBClient{}) + ops.SetUserID("agent-7") + + fsErr := ops.WriteFile(context.Background(), "/.info/user", []byte("\n")) + require.Nil(t, fsErr) + assert.Empty(t, ops.GetUserID(), "writing empty/newline should clear user ID") +} + +// --- Log entry user_id wiring tests --- + +func TestSynth_LogEntry_IncludesUserID(t *testing.T) { + mockDB := newParentPointerMockDB() + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + ops.SetUserID("agent-7") + + content := "---\ntitle: Hello\n---\n# Hello\n" + fsErr := ops.WriteFile(context.Background(), "/notes/hello.md", []byte(content)) + require.Nil(t, fsErr) + + require.Len(t, mockDB.logEntries, 1) + assert.Equal(t, "create", mockDB.logEntries[0].opType) + assert.Equal(t, "agent-7", mockDB.logEntries[0].userID, + "log entry should include the mount-level user ID") +} + +func TestSynth_LogEntry_AnonymousWhenNoUserID(t *testing.T) { + mockDB := newParentPointerMockDB() + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + // No SetUserID -- anonymous + + content := "---\ntitle: Hello\n---\n# Hello\n" + fsErr := ops.WriteFile(context.Background(), "/notes/hello.md", []byte(content)) + require.Nil(t, fsErr) + + require.Len(t, mockDB.logEntries, 1) + assert.Empty(t, mockDB.logEntries[0].userID, + "log entry should have empty user ID when anonymous") +} + +func TestSynth_LogEntry_UserIDChangesAfterWrite(t *testing.T) { + mockDB := newParentPointerMockDB() + mockDB.lastInsertReturnPK = "uuid-new-1" + cfg := &config.Config{DirListingLimit: 1000} + ops := NewOperations(cfg, mockDB) + + // First write as agent-7 + ops.SetUserID("agent-7") + fsErr := ops.WriteFile(context.Background(), "/notes/hello.md", []byte("---\ntitle: Hello\n---\n# Hello\n")) + require.Nil(t, fsErr) + + // Change identity via .info/user + fsErr = ops.WriteFile(context.Background(), "/.info/user", []byte("agent-9\n")) + require.Nil(t, fsErr) + + // Second write -- simulate edit by making resolve find the file + mockDB.resolvePathResults = []db.PathSegment{ + {Depth: 1, ID: "uuid-new-1", Name: "hello.md"}, + } + mockDB.latestVersionIDs = map[string]string{"uuid-new-1": "v-1"} + fsErr = ops.WriteFile(context.Background(), "/notes/hello.md", []byte("---\ntitle: Updated\n---\n# Updated\n")) + require.Nil(t, fsErr) + + require.Len(t, mockDB.logEntries, 2) + assert.Equal(t, "agent-7", mockDB.logEntries[0].userID, "first write should use agent-7") + assert.Equal(t, "agent-9", mockDB.logEntries[1].userID, "second write should use agent-9") +} diff --git a/internal/tigerfs/fs/validate_create_test.go b/internal/tigerfs/fs/validate_create_test.go new file mode 100644 index 0000000..abf9bea --- /dev/null +++ b/internal/tigerfs/fs/validate_create_test.go @@ -0,0 +1,136 @@ +package fs + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/timescale/tigerfs/internal/tigerfs/config" +) + +// newValidateCreateMockDB builds a mock with: +// - "posts" (public schema): synth markdown view (tigerfs:md), backing table "_posts". +// Columns match newSynthMockDB so role detection succeeds and the view registers +// in the synth cache. +// - "products" (public schema): regular data table, PK "id", one existing row id="existing". +// - "products_savepoint" (tigerfs schema): regular table, PK "name" (no rows). +// Exists so /products/.savepoint/xxx can resolve a PK and reach the reject branch. +// +// This lets ValidateCreate exercise: synth-view allow, non-synth new-row reject, +// existing-row allow, and the savepoint-style rejection the original bug reported. +func newValidateCreateMockDB() *mockDBClient { + return &mockDBClient{ + tables: map[string][]string{ + "public": {"_posts", "products"}, + "tigerfs": {"products_savepoint"}, + }, + views: map[string][]string{ + "public": {"posts"}, + }, + viewComments: map[string]map[string]string{ + "public": {"posts": "tigerfs:md"}, + }, + primaryKeys: map[string]*mockPK{ + "public._posts": {column: "id"}, + "public.posts": {column: "id"}, + "public.products": {column: "id"}, + "tigerfs.products_savepoint": {column: "name"}, + }, + columns: map[string][]mockColumn{ + "public.posts": { + {name: "id", dataType: "integer"}, + {name: "filename", dataType: "text"}, + {name: "title", dataType: "text"}, + {name: "author", dataType: "text"}, + {name: "body", dataType: "text"}, + }, + "public.products": { + {name: "id", dataType: "text"}, + {name: "name", dataType: "text"}, + }, + "tigerfs.products_savepoint": { + {name: "name", dataType: "text"}, + }, + }, + rowData: map[string]*mockRow{ + // "/products/existing" should be treated as an existing row. + "public.products.existing": { + columns: []string{"id", "name"}, + values: []interface{}{"existing", "already-here"}, + }, + }, + } +} + +func TestValidateCreate(t *testing.T) { + cases := []struct { + name string + path string + wantReject bool + wantPK string // expected PK in rejection message (used only if wantReject) + }{ + // Synth view — bare filename is user content, not a PK. + {name: "synth view bare path", path: "/posts/new-entry", wantReject: false}, + {name: "synth view md suffix", path: "/posts/new-entry.md", wantReject: false}, + + // Non-synth data table — bare new row collides with future row-directory inode. + {name: "data table bare new row", path: "/products/newrow", wantReject: true, wantPK: "newrow"}, + {name: "data table suffixed new row", path: "/products/newrow.tsv", wantReject: false}, + {name: "data table suffixed json", path: "/products/newrow.json", wantReject: false}, + + // Existing row — writeRowFile will UPDATE; no inode conflict. + {name: "data table bare existing row", path: "/products/existing", wantReject: false}, + + // Savepoint (the original bug). + {name: "savepoint bare path", path: "/products/.savepoint/xxx", wantReject: true, wantPK: "xxx"}, + {name: "savepoint json suffix", path: "/products/.savepoint/xxx.json", wantReject: false}, + + // Non-PathRow types — ValidateCreate must not fire. + {name: "column path", path: "/products/existing/name", wantReject: false}, + + // Unparseable / unknown paths — fail open. + {name: "nonexistent table", path: "/nonexistent/foo", wantReject: false}, + {name: "root", path: "/", wantReject: false}, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + ops := NewOperations(&config.Config{DirListingLimit: 1000}, newValidateCreateMockDB()) + fsErr := ops.ValidateCreate(context.Background(), tc.path) + if tc.wantReject { + require.NotNil(t, fsErr, "expected rejection for %s", tc.path) + assert.Equal(t, ErrInvalidArgument, fsErr.Code) + assert.Contains(t, fsErr.Message, "without a format suffix") + assert.Contains(t, fsErr.Message, tc.wantPK) + assert.Contains(t, fsErr.Hint, tc.wantPK+".json") + assert.Contains(t, fsErr.Hint, tc.wantPK+".tsv") + assert.Contains(t, fsErr.Hint, tc.wantPK+".csv") + require.NotNil(t, fsErr.Cause, "Cause must be set to avoid %%!w() in NFS log") + assert.Equal(t, "format suffix required for new rows", fsErr.Cause.Error()) + } else { + assert.Nil(t, fsErr, "expected allow for %s, got: %+v", tc.path, fsErr) + } + }) + } +} + +// TestValidateCreate_SharesErrorWithWriteRowFile verifies the Create-time +// rejection and the writeRowFile defense-in-depth rejection return the same +// FSError shape -- they must go through newBarePathRejection in lockstep. +func TestValidateCreate_SharesErrorWithWriteRowFile(t *testing.T) { + ops := NewOperations(&config.Config{DirListingLimit: 1000}, newValidateCreateMockDB()) + + fromValidate := ops.ValidateCreate(context.Background(), "/products/newrow") + require.NotNil(t, fromValidate) + + fromWrite := ops.WriteFile(context.Background(), "/products/newrow", []byte("ignored")) + require.NotNil(t, fromWrite) + + assert.Equal(t, fromValidate.Code, fromWrite.Code) + assert.Equal(t, fromValidate.Message, fromWrite.Message) + assert.Equal(t, fromValidate.Hint, fromWrite.Hint) + require.NotNil(t, fromValidate.Cause) + require.NotNil(t, fromWrite.Cause) + assert.Equal(t, fromValidate.Cause.Error(), fromWrite.Cause.Error()) +} diff --git a/internal/tigerfs/fs/write.go b/internal/tigerfs/fs/write.go index df15fe7..92fb289 100644 --- a/internal/tigerfs/fs/write.go +++ b/internal/tigerfs/fs/write.go @@ -2,6 +2,7 @@ package fs import ( "context" + "errors" "fmt" "strings" @@ -24,9 +25,25 @@ import ( // - path: filesystem path to write to // - data: file content (format determined by path extension) // +// Preconditions: +// - For writes inside a synth view (i.e., ///.../), the +// immediate parent directory of path must already exist as a row in +// the source table. WriteFile does NOT auto-create ancestor dirs; +// callers should issue an explicit Mkdir per missing segment first +// (this matches POSIX open(O_CREAT) semantics, which is what the +// kernel enforces for FUSE/NFS clients before TigerFS sees a write). +// Use WriteFileEnsureDirs if you want mkdir-p behavior in one call. +// +// Postconditions on success: +// - The row at path exists with the given data. +// - If history is enabled, exactly one log entry is written: "create" +// when the file was new, "edit" when the file already existed. +// - Stat/path caches for the table are invalidated. +// // Returns nil on success, or an FSError describing the failure. // Common errors: // - ErrInvalidPath: unsupported path type for writes +// - ErrNotExist: an ancestor directory does not exist // - ErrPermission: view is not updatable // - ErrIO: database operation failed func (o *Operations) WriteFile(ctx context.Context, path string, data []byte) *FSError { @@ -39,10 +56,143 @@ func (o *Operations) WriteFile(ctx context.Context, path string, data []byte) *F return o.writeFileWithParsed(ctx, parsed, data) } +// WriteFileEnsureDirs is the mkdir-p variant of WriteFile: any missing +// ancestor directories of path are created via Mkdir before the file is +// written. Each new ancestor produces its own "create" log entry -- +// matching what the kernel would generate from per-segment mkdir(2) +// syscalls in production (the kernel rejects open(O_CREAT) on a path +// whose parent dir is missing, so a deep WriteFile never reaches TigerFS +// from FUSE/NFS without all ancestors already in place). +// +// Use this when a caller -- typically a test or batch-import code that +// drives Operations directly -- wants to materialize a deep path in a +// single call. Prefer WriteFile + explicit Mkdir(s) when you need +// fine-grained control over ordering or error handling, or when the +// in-between dirs already exist and there's nothing to ensure. +// +// Preconditions: +// - The path's first segment must be an existing app/view (Mkdir +// against a non-existent app fails just as it would normally). +// +// Postconditions on success: +// - All ancestors of path exist as directory rows. +// - One "create" log entry is written for each ancestor that did NOT +// already exist before this call. +// - WriteFile's own postconditions (file row + create/edit log entry). +// +// Returns the same errors as Mkdir or WriteFile. ErrAlreadyExists from +// any intermediate Mkdir is treated as "already there, fine" and not +// propagated. +func (o *Operations) WriteFileEnsureDirs(ctx context.Context, path string, data []byte) *FSError { + // Walk segments shallowest-to-deepest, calling Mkdir on each + // ancestor of the file. Index 0 is the app/view name, so we begin + // the loop at 2 (the first dir under the app). The final segment is + // the filename and goes through WriteFile, not Mkdir. + parts := strings.Split(strings.TrimPrefix(path, "/"), "/") + for i := 2; i < len(parts); i++ { + ancestor := "/" + strings.Join(parts[:i], "/") + if fsErr := o.Mkdir(ctx, ancestor); fsErr != nil && fsErr.Code != ErrAlreadyExists { + return fsErr + } + } + return o.WriteFile(ctx, path, data) +} + +// MkdirAll is the mkdir-p variant of Mkdir: every segment of path that +// doesn't already exist is created via Mkdir, in order. The final +// segment is also created as a directory (unlike WriteFileEnsureDirs, +// which expects the final segment to be a file). +// +// Each new segment produces one "create" log entry -- matching what the +// kernel would generate from per-segment mkdir(2) syscalls in +// production. Existing segments are skipped (Mkdir's ErrAlreadyExists +// is treated as success and not propagated). +// +// Use this when a caller wants to materialize a deep directory chain +// in one call. Production code reaching TigerFS through FUSE/NFS rarely +// needs MkdirAll because the kernel always issues per-segment mkdir(2) +// for `mkdir -p` invocations -- but direct-API callers (typically tests +// or batch import code) may want it. +func (o *Operations) MkdirAll(ctx context.Context, path string) *FSError { + parts := strings.Split(strings.TrimPrefix(path, "/"), "/") + for i := 2; i <= len(parts); i++ { + segment := "/" + strings.Join(parts[:i], "/") + if fsErr := o.Mkdir(ctx, segment); fsErr != nil && fsErr.Code != ErrAlreadyExists { + return fsErr + } + } + return nil +} + +// ValidateCreate reports an FSError if creating a regular file at this path +// would conflict with a future directory inode on NFS. +// +// The only case we reject: a bare-path write (no format suffix) that would +// create a new row in a non-synth-view table. The server later exposes that +// row as a directory in READDIRPLUS, but if the kernel already cached a +// file-typed positive dentry from Create, the entry gets suppressed from +// `ls` because the inode types disagree. Failing at Create time prevents +// the file-typed dentry from ever forming. +// +// Returns nil for every other path: synth views (bare filename is user +// content), format-suffix writes (distinct file inode from the row +// directory), non-row paths (columns, DDL, undo, etc.), existing rows, and +// any path the parser cannot fully resolve. Adapters should call this +// before issuing a file handle. +func (o *Operations) ValidateCreate(ctx context.Context, filePath string) *FSError { + parsed, err := o.parsePath(ctx, filePath) + if err != nil { + return nil + } + o.resolveSynthHierarchy(ctx, parsed) + if parsed.Type != PathRow || parsed.Format != "" { + return nil + } + fsCtx := parsed.Context + if fsCtx == nil { + return nil + } + // Synth views route bare-path writes through writeSynthFile; the + // filename is user content, not a PK. Mirrors writeRowFile's dispatch. + if info := o.getSynthViewInfo(ctx, fsCtx.Schema, fsCtx.TableName); info != nil { + return nil + } + pk, mErr := o.metaCache.GetPrimaryKey(ctx, fsCtx.Schema, fsCtx.TableName) + if mErr != nil { + return nil + } + match, dErr := pk.Decode(parsed.PrimaryKey) + if dErr != nil { + return nil + } + if _, gErr := o.db.GetRow(ctx, fsCtx.Schema, fsCtx.TableName, match); gErr == nil { + return nil // row exists; writeRowFile will UPDATE + } + return newBarePathRejection(parsed.PrimaryKey) +} + +// newBarePathRejection builds the FSError returned for bare-path new-row +// writes. Shared by ValidateCreate (adapter-level pre-check) and writeRowFile +// (defense-in-depth guard) so the user-visible message, hint, and cause +// stay in lockstep. +func newBarePathRejection(pk string) *FSError { + return &FSError{ + Code: ErrInvalidArgument, + Message: fmt.Sprintf("cannot create %q without a format suffix", pk), + Hint: fmt.Sprintf("retry as %q, %q, or %q (bare-path writes conflict with the row directory inode on NFS)", pk+".json", pk+".tsv", pk+".csv"), + Cause: errors.New("format suffix required for new rows"), + } +} + // writeFileWithParsed implements write logic for a parsed path. func (o *Operations) writeFileWithParsed(ctx context.Context, parsed *ParsedPath, data []byte) *FSError { switch parsed.Type { case PathRow: + // Savepoint create: echo -e "description\nStarting" > .savepoint/name.tsv + // Thin wrapper injects user_id then delegates to writeRowFile. + if parsed.OrigTableName != "" && strings.HasSuffix(parsed.Context.TableName, "_savepoint") { + return o.writeSavepoint(ctx, parsed, data) + } return o.writeRowFile(ctx, parsed, data) case PathColumn: return o.writeColumnFile(ctx, parsed, data) @@ -50,10 +200,14 @@ func (o *Operations) writeFileWithParsed(ctx context.Context, parsed *ParsedPath return o.writeImportFile(ctx, parsed, data) case PathDDL: return o.writeDDLFile(ctx, parsed, data) + case PathRootInfo: + return o.writeRootInfoFile(parsed, data) case PathBuild: return o.writeBuildFile(ctx, parsed, data) case PathFormat: return o.writeFormatFile(ctx, parsed, data) + case PathUndo: + return o.writeUndoApply(ctx, parsed, data) case PathHistory: return &FSError{ Code: ErrPermission, @@ -120,10 +274,17 @@ func (o *Operations) writeRowFile(ctx context.Context, parsed *ParsedPath, data } if parseErr != nil { - return &FSError{ - Code: ErrInvalidPath, - Message: "failed to parse write data", - Cause: parseErr, + // Empty body with a format suffix is valid -- creates a row with just the PK + // and DEFAULTs for all other columns. e.g., echo "" > .savepoint/name.tsv + if parsed.Format != "" && len(strings.TrimSpace(string(data))) == 0 { + columns = nil + values = nil + } else { + return &FSError{ + Code: ErrInvalidPath, + Message: "failed to parse write data", + Cause: parseErr, + } } } @@ -142,7 +303,37 @@ func (o *Operations) writeRowFile(ctx context.Context, parsed *ParsedPath, data } } } else { - // INSERT new row + // INSERT new row -- require format suffix (.tsv, .json, .csv) for new rows. + // On macOS NFS, bare-path writes (no extension) create a FILE inode that conflicts + // with the DIRECTORY inode returned by READDIRPLUS, causing the entry to silently + // disappear from ls. Format suffixes avoid this because the write path (e.g., + // "test-cat.tsv") differs from the listing name ("test-cat"). + // + // Adapters call ValidateCreate before issuing a file handle; this guard + // stays as defense-in-depth for callers that bypass the adapter (tests, + // direct WriteFile). Both paths share newBarePathRejection so the + // user-visible error stays identical. + if parsed.Format == "" { + return newBarePathRejection(parsed.PrimaryKey) + } + + // Merge PK columns from the path if not already in the data. + // e.g., writing to /categories/test-cat.tsv with body "name\tvalue" should + // include slug=test-cat in the INSERT even though it's not in the TSV body. + for i, pkCol := range match.Columns { + found := false + for _, col := range columns { + if col == pkCol { + found = true + break + } + } + if !found { + columns = append(columns, pkCol) + values = append(values, match.Values[i]) + } + } + _, err = o.db.InsertRow(ctx, fsCtx.Schema, fsCtx.TableName, columns, values) if err != nil { return &FSError{ @@ -635,6 +826,7 @@ func (o *Operations) Delete(ctx context.Context, path string) *FSError { func (o *Operations) deleteWithParsed(ctx context.Context, parsed *ParsedPath) *FSError { switch parsed.Type { case PathRow: + // With name as PK, standard deleteRow works for savepoints too. return o.deleteRow(ctx, parsed) case PathColumn: return o.deleteColumn(ctx, parsed) @@ -848,21 +1040,37 @@ func (o *Operations) Create(ctx context.Context, path string) (*WriteHandle, *FS } } -// Mkdir creates a directory for incremental row creation. +// Mkdir creates a single directory at path. // -// Creating a directory with a primary key value starts the incremental -// row creation workflow: +// For synth views, this inserts a row with filetype='directory' and +// (when history is enabled) writes one "create" log entry. The dir's +// modified_at is set to now() and the parent dir's modified_at is +// bumped via the bump_parent_mtime trigger. +// +// For non-synth tables, mkdir starts the incremental row creation +// workflow: // 1. mkdir /table/pk → creates staging entry // 2. echo "value" > /table/pk/column → sets column value // 3. When all NOT NULL columns are provided, row is auto-committed // +// Preconditions: +// - Path's immediate parent must already exist as a row in the +// source table. Mkdir does NOT auto-create ancestors; if you have +// a deep path to materialize, issue one Mkdir per missing segment +// (matching POSIX mkdir(2) semantics) or use WriteFileEnsureDirs. +// +// Postconditions on success: +// - For synth views: a directory row exists at path; one "create" +// log entry is written if history is enabled. +// // Parameters: // - ctx: context for database operations and cancellation // - path: filesystem path for the new directory // // Returns nil on success, or an FSError describing the failure. // Common errors: -// - ErrExists: row already exists in the database +// - ErrAlreadyExists: a row already exists at path +// - ErrNotExist: an ancestor of path does not exist // - ErrPermission: view is not updatable // - ErrIO: database operation failed func (o *Operations) Mkdir(ctx context.Context, path string) *FSError { diff --git a/internal/tigerfs/fs/write_test.go b/internal/tigerfs/fs/write_test.go index 142e96f..875c0df 100644 --- a/internal/tigerfs/fs/write_test.go +++ b/internal/tigerfs/fs/write_test.go @@ -63,6 +63,147 @@ func TestWriteFile_InsertRow(t *testing.T) { assert.True(t, mockDB.insertCalled) } +// TestWriteFile_InsertRow_TSV_MergesPK tests that TSV insert merges the PK from the filename. +// e.g., echo -e "name\tdescription\nTest\tA test" > categories/test-cat.tsv +// should INSERT with slug='test-cat' even though slug isn't in the TSV headers. +func TestWriteFile_InsertRow_TSV_MergesPK(t *testing.T) { + cfg := &config.Config{} + mockDB := &mockDBClient{ + tables: map[string][]string{ + "public": {"categories"}, + }, + primaryKeys: map[string]*mockPK{ + "public.categories": {column: "slug"}, + }, + lastInsertReturnPK: "test-cat", + } + + ops := NewOperations(cfg, mockDB) + + // TSV with headers: PK (slug) is NOT in the headers + data := []byte("name\tdescription\nTest Category\tA test\n") + err := ops.WriteFile(context.Background(), "/categories/test-cat.tsv", data) + + require.Nil(t, err) + require.True(t, mockDB.insertCalled) + require.Len(t, mockDB.insertedRows, 1) + + row := mockDB.insertedRows[0] + // Should have 3 columns: name, description (from TSV) + slug (merged from filename) + assert.Contains(t, row.columns, "slug", "PK column should be merged from filename") + assert.Contains(t, row.columns, "name") + assert.Contains(t, row.columns, "description") + + // Find slug value + for i, col := range row.columns { + if col == "slug" { + assert.Equal(t, "test-cat", row.values[i], "slug should be the filename PK value") + } + } +} + +// TestWriteFile_InsertRow_TSV_PKInBody tests that PK already in TSV headers is not duplicated. +func TestWriteFile_InsertRow_TSV_PKInBody(t *testing.T) { + cfg := &config.Config{} + mockDB := &mockDBClient{ + tables: map[string][]string{ + "public": {"categories"}, + }, + primaryKeys: map[string]*mockPK{ + "public.categories": {column: "slug"}, + }, + lastInsertReturnPK: "test-cat", + } + + ops := NewOperations(cfg, mockDB) + + // TSV with PK in headers + data := []byte("slug\tname\ntest-cat\tTest Category\n") + err := ops.WriteFile(context.Background(), "/categories/test-cat.tsv", data) + + require.Nil(t, err) + require.Len(t, mockDB.insertedRows, 1) + + row := mockDB.insertedRows[0] + // Count slug occurrences -- should be exactly 1 (not duplicated) + slugCount := 0 + for _, col := range row.columns { + if col == "slug" { + slugCount++ + } + } + assert.Equal(t, 1, slugCount, "PK should not be duplicated when already in TSV body") +} + +// TestWriteFile_InsertRow_BareFormat_Rejected tests that bare path (no extension) +// inserts are rejected to avoid NFS inode cache conflicts. +func TestWriteFile_InsertRow_BareFormat_Rejected(t *testing.T) { + cfg := &config.Config{} + mockDB := &mockDBClient{ + tables: map[string][]string{ + "public": {"categories"}, + }, + columns: map[string][]mockColumn{ + "public.categories": { + {name: "slug", dataType: "text"}, + {name: "name", dataType: "text"}, + {name: "description", dataType: "text"}, + }, + }, + primaryKeys: map[string]*mockPK{ + "public.categories": {column: "slug"}, + }, + } + + ops := NewOperations(cfg, mockDB) + + // Bare path insert should be rejected (no format suffix) + data := []byte("test-cat\tTest Category\tA test\n") + err := ops.WriteFile(context.Background(), "/categories/test-cat", data) + + require.NotNil(t, err, "bare-path INSERT should be rejected") + assert.Equal(t, ErrInvalidArgument, err.Code) + assert.Contains(t, err.Message, "without a format suffix") + assert.Contains(t, err.Hint, "test-cat.json", "hint should suggest suffixed alternatives") + assert.False(t, mockDB.insertCalled, "should not have attempted INSERT") +} + +// TestWriteFile_UpdateRow_BareFormat_Allowed tests that bare path (no extension) +// updates to existing rows still work. +func TestWriteFile_UpdateRow_BareFormat_Allowed(t *testing.T) { + cfg := &config.Config{} + mockDB := &mockDBClient{ + tables: map[string][]string{ + "public": {"categories"}, + }, + columns: map[string][]mockColumn{ + "public.categories": { + {name: "slug", dataType: "text"}, + {name: "name", dataType: "text"}, + {name: "description", dataType: "text"}, + }, + }, + primaryKeys: map[string]*mockPK{ + "public.categories": {column: "slug"}, + }, + rowData: map[string]*mockRow{ + "public.categories.test-cat": { + columns: []string{"slug", "name", "description"}, + values: []interface{}{"test-cat", "Old Name", "Old Desc"}, + }, + }, + } + + ops := NewOperations(cfg, mockDB) + + // Bare path update should work (row already exists, no NFS conflict) + data := []byte("test-cat\tNew Name\tNew Desc\n") + err := ops.WriteFile(context.Background(), "/categories/test-cat", data) + + require.Nil(t, err, "bare-path UPDATE should be allowed") + assert.True(t, mockDB.updateCalled, "should have called UpdateRow") +} + // TestWriteFile_UpdateColumn tests updating a single column. func TestWriteFile_UpdateColumn(t *testing.T) { cfg := &config.Config{} diff --git a/internal/tigerfs/fuse/adapter.go b/internal/tigerfs/fuse/adapter.go index 769ba84..7e1e2cf 100644 --- a/internal/tigerfs/fuse/adapter.go +++ b/internal/tigerfs/fuse/adapter.go @@ -103,6 +103,10 @@ func (a *FSAdapter) EntryToAttr(entry *tigerfs.Entry, out *gofuse.Attr) { // Directory mode: include S_IFDIR flag out.Mode = syscall.S_IFDIR | uint32(entry.Mode&0777) out.Nlink = 2 // directories have . and .. + } else if entry.IsSymlink() { + // Symlink mode: include S_IFLNK flag + out.Mode = syscall.S_IFLNK | 0777 // symlinks are always 0777 + out.Nlink = 1 } else { // File mode: include S_IFREG flag out.Mode = syscall.S_IFREG | uint32(entry.Mode&0777) @@ -186,6 +190,11 @@ func (a *FSAdapter) EntriesToDirEntries(entries []tigerfs.Entry) []gofuse.DirEnt Name: entry.Name, Mode: syscall.S_IFDIR, } + } else if entry.IsSymlink() { + result[i] = gofuse.DirEntry{ + Name: entry.Name, + Mode: syscall.S_IFLNK, + } } else { result[i] = gofuse.DirEntry{ Name: entry.Name, diff --git a/internal/tigerfs/fuse/adapter_test.go b/internal/tigerfs/fuse/adapter_test.go index 26396c6..a4be56f 100644 --- a/internal/tigerfs/fuse/adapter_test.go +++ b/internal/tigerfs/fuse/adapter_test.go @@ -67,6 +67,54 @@ func TestFSAdapter_EntryToAttr_File(t *testing.T) { assert.Equal(t, uint32(1), out.Nlink) // regular files have nlink=1 } +// TestFSAdapter_EntryToAttr_Symlink tests converting a symlink entry. +func TestFSAdapter_EntryToAttr_Symlink(t *testing.T) { + cfg := &config.Config{} + ops := tigerfs.NewOperations(cfg, nil) + adapter := NewFSAdapter(ops) + + entry := &tigerfs.Entry{ + Name: "before", + IsDir: false, + Size: 42, + Mode: os.ModeSymlink | 0777, + Target: "../../.history/docs/hello.md/2026-04-07T143000.123Z-abc", + ModTime: time.Date(2024, 1, 15, 10, 30, 0, 0, time.UTC), + } + + var out gofuse.Attr + adapter.EntryToAttr(entry, &out) + + assert.Equal(t, uint32(syscall.S_IFLNK|0777), out.Mode) + assert.Equal(t, uint64(42), out.Size) + assert.Equal(t, uint32(1), out.Nlink) // symlinks have nlink=1 +} + +// TestFSAdapter_EntriesToDirEntries_Symlink tests directory listing with symlinks. +func TestFSAdapter_EntriesToDirEntries_Symlink(t *testing.T) { + cfg := &config.Config{} + ops := tigerfs.NewOperations(cfg, nil) + adapter := NewFSAdapter(ops) + + entries := []tigerfs.Entry{ + {Name: "docs", IsDir: true, Mode: os.ModeDir | 0755}, + {Name: "hello.md", Mode: 0644}, + {Name: "before", Mode: os.ModeSymlink | 0777, Target: "/dev/null"}, + } + + dirEntries := adapter.EntriesToDirEntries(entries) + require.Len(t, dirEntries, 3) + + assert.Equal(t, "docs", dirEntries[0].Name) + assert.Equal(t, uint32(syscall.S_IFDIR), dirEntries[0].Mode) + + assert.Equal(t, "hello.md", dirEntries[1].Name) + assert.Equal(t, uint32(syscall.S_IFREG), dirEntries[1].Mode) + + assert.Equal(t, "before", dirEntries[2].Name) + assert.Equal(t, uint32(syscall.S_IFLNK), dirEntries[2].Mode) +} + // TestFSAdapter_ErrorToErrno tests error code conversion. func TestFSAdapter_ErrorToErrno(t *testing.T) { cfg := &config.Config{} diff --git a/internal/tigerfs/fuse/all_test.go b/internal/tigerfs/fuse/all_test.go index a5b2c3a..d758ab2 100644 --- a/internal/tigerfs/fuse/all_test.go +++ b/internal/tigerfs/fuse/all_test.go @@ -13,7 +13,7 @@ import ( func TestNewAllRowsNode(t *testing.T) { cfg := &config.Config{ DefaultSchema: "public", - DirListingLimit: 10000, + DirListingLimit: 1000, } partialRows := NewPartialRowTracker(nil) @@ -47,7 +47,7 @@ func TestNewAllRowsNode(t *testing.T) { func TestAllRowsNode_Getattr(t *testing.T) { cfg := &config.Config{ DefaultSchema: "public", - DirListingLimit: 10000, + DirListingLimit: 1000, } allNode := NewAllRowsNode(cfg, nil, nil, "public", "users", nil) @@ -92,7 +92,7 @@ func TestAllRowsNode_Interfaces(t *testing.T) { // TestAllRowsNode_Readdir_WithMock tests Readdir listing all rows func TestAllRowsNode_Readdir_WithMock(t *testing.T) { cfg := &config.Config{ - DirListingLimit: 10000, + DirListingLimit: 1000, } mock := db.NewMockDBClient() @@ -140,7 +140,7 @@ func TestAllRowsNode_Readdir_WithMock(t *testing.T) { // TestAllRowsNode_Readdir_WithMock_Empty tests Readdir with no rows func TestAllRowsNode_Readdir_WithMock_Empty(t *testing.T) { cfg := &config.Config{ - DirListingLimit: 10000, + DirListingLimit: 1000, } mock := db.NewMockDBClient() @@ -176,7 +176,7 @@ func TestAllRowsNode_Readdir_WithMock_Empty(t *testing.T) { // TestAllRowsNode_Readdir_WithMock_PKError tests Readdir when GetPrimaryKey fails func TestAllRowsNode_Readdir_WithMock_PKError(t *testing.T) { cfg := &config.Config{ - DirListingLimit: 10000, + DirListingLimit: 1000, } mock := db.NewMockDBClient() @@ -197,7 +197,7 @@ func TestAllRowsNode_Readdir_WithMock_PKError(t *testing.T) { // TestAllRowsNode_Readdir_WithMock_ListError tests Readdir when ListAllRows fails func TestAllRowsNode_Readdir_WithMock_ListError(t *testing.T) { cfg := &config.Config{ - DirListingLimit: 10000, + DirListingLimit: 1000, } mock := db.NewMockDBClient() diff --git a/internal/tigerfs/fuse/mount_ops.go b/internal/tigerfs/fuse/mount_ops.go index 0618dec..1de2c9b 100644 --- a/internal/tigerfs/fuse/mount_ops.go +++ b/internal/tigerfs/fuse/mount_ops.go @@ -43,6 +43,8 @@ func MountOps(ctx context.Context, cfg *config.Config, connStr, mountpoint strin // 2. Create shared Operations core (same as NFS uses) ops := tigerfs.NewOperations(cfg, dbClient) + ops.SetMountPoint(mountpoint) + ops.SetUserID(cfg.UserID) // 3. Create FSAdapter bridge adapter := NewFSAdapter(ops) diff --git a/internal/tigerfs/fuse/ops_node.go b/internal/tigerfs/fuse/ops_node.go index b9ecfbd..d0cefe9 100644 --- a/internal/tigerfs/fuse/ops_node.go +++ b/internal/tigerfs/fuse/ops_node.go @@ -21,6 +21,7 @@ import ( "github.com/hanwen/go-fuse/v2/fs" "github.com/hanwen/go-fuse/v2/fuse" + fsops "github.com/timescale/tigerfs/internal/tigerfs/fs" "github.com/timescale/tigerfs/internal/tigerfs/logging" "go.uber.org/zap" ) @@ -37,6 +38,7 @@ var _ fs.NodeMkdirer = (*OpsNode)(nil) var _ fs.NodeUnlinker = (*OpsNode)(nil) var _ fs.NodeRmdirer = (*OpsNode)(nil) var _ fs.NodeRenamer = (*OpsNode)(nil) +var _ fs.NodeReadlinker = (*OpsNode)(nil) // OpsNode is a single generic FUSE node type that delegates all operations // to FSAdapter. Every directory and file in the tree is represented by an @@ -104,14 +106,15 @@ func (n *OpsNode) Setattr(ctx context.Context, fh fs.FileHandle, in *fuse.SetAtt } } - // DDL trigger files: fire on touch (mtime update without open file handle). + // Trigger files: fire on touch (mtime update without open file handle). // FUSE `touch` uses utimensat which sends SETATTR with ATIME/MTIME, not OPEN. if fh == nil { if _, ok := in.GetMTime(); ok { baseName := path.Base(n.path) - isTrigger := (baseName == ".test" || baseName == ".commit" || baseName == ".abort") && isDDLOpsPath(n.path) - if isTrigger { - logging.Debug("OpsNode.Setattr: triggering DDL operation via touch", + isTrigger := (baseName == fsops.FileTest || baseName == fsops.FileCommit || baseName == fsops.FileAbort) && isDDLOpsPath(n.path) + isUndoApply := baseName == fsops.FileApply && strings.Contains(n.path, "/"+fsops.DirUndo+"/") + if isTrigger || isUndoApply { + logging.Debug("OpsNode.Setattr: triggering operation via touch", zap.String("path", n.path)) if errno := n.adapter.WriteFile(ctx, n.path, []byte{}); errno != 0 { return errno @@ -140,12 +143,24 @@ func (n *OpsNode) Lookup(ctx context.Context, name string, out *fuse.EntryOut) ( mode := uint32(syscall.S_IFREG) if entry.IsDir { mode = uint32(syscall.S_IFDIR) + } else if entry.IsSymlink() { + mode = uint32(syscall.S_IFLNK) } child := n.NewPersistentInode(ctx, newOpsNode(n.adapter, childPath), fs.StableAttr{Mode: mode}) return child, 0 } +// Readlink returns the target of a symlink. +func (n *OpsNode) Readlink(ctx context.Context) ([]byte, syscall.Errno) { + logging.Debug("OpsNode.Readlink", zap.String("path", n.path)) + target, fsErr := n.adapter.ops.Readlink(ctx, n.path) + if fsErr != nil { + return nil, n.adapter.ErrorToErrno(fsErr) + } + return []byte(target), 0 +} + // Readdir lists directory contents. func (n *OpsNode) Readdir(ctx context.Context) (fs.DirStream, syscall.Errno) { logging.Debug("OpsNode.Readdir", zap.String("path", n.path)) @@ -164,7 +179,7 @@ func (n *OpsNode) Create(ctx context.Context, name string, flags uint32, mode ui child := n.NewPersistentInode(ctx, newOpsNode(n.adapter, childPath), fs.StableAttr{Mode: syscall.S_IFREG}) // Detect trigger and DDL sql files (same logic as Open) - isTrigger := name == ".test" || name == ".commit" || name == ".abort" + isTrigger := name == fsops.FileTest || name == fsops.FileCommit || name == fsops.FileAbort isDDLSQL := name == "sql" && isDDLOpsPath(childPath) handle := &OpsFileHandle{ @@ -234,7 +249,7 @@ func (n *OpsNode) Open(ctx context.Context, flags uint32) (fs.FileHandle, uint32 // DDL trigger files (.test, .commit, .abort) should fire on close baseName := path.Base(n.path) - isTrigger := baseName == ".test" || baseName == ".commit" || baseName == ".abort" + isTrigger := baseName == fsops.FileTest || baseName == fsops.FileCommit || baseName == fsops.FileAbort isDDLSQL := baseName == "sql" && isDDLOpsPath(n.path) if isWrite || isTrigger { @@ -272,7 +287,7 @@ func (n *OpsNode) Open(ctx context.Context, flags uint32) (fs.FileHandle, uint32 // isDDLOpsPath checks if a path is within a DDL staging directory. func isDDLOpsPath(p string) bool { - return strings.Contains(p, "/.create/") || strings.Contains(p, "/.modify/") || strings.Contains(p, "/.delete/") + return strings.Contains(p, "/"+fsops.DirCreate+"/") || strings.Contains(p, "/"+fsops.DirModify+"/") || strings.Contains(p, "/"+fsops.DirDelete+"/") } // OpsFileHandle is a FUSE file handle that buffers writes and commits on Flush. diff --git a/internal/tigerfs/fuse/pagination_test.go b/internal/tigerfs/fuse/pagination_test.go index 9831d48..cd3dbfd 100644 --- a/internal/tigerfs/fuse/pagination_test.go +++ b/internal/tigerfs/fuse/pagination_test.go @@ -13,7 +13,7 @@ import ( func TestNewPaginationNode(t *testing.T) { cfg := &config.Config{ DefaultSchema: "public", - DirListingLimit: 10000, + DirListingLimit: 1000, } partialRows := NewPartialRowTracker(nil) diff --git a/internal/tigerfs/fuse/pipeline_node_test.go b/internal/tigerfs/fuse/pipeline_node_test.go index dbf1c56..305c921 100644 --- a/internal/tigerfs/fuse/pipeline_node_test.go +++ b/internal/tigerfs/fuse/pipeline_node_test.go @@ -12,7 +12,7 @@ import ( // TestPipelineNode_Readdir tests that PipelineNode lists capabilities based on context. func TestPipelineNode_Readdir(t *testing.T) { - cfg := &config.Config{DirListingLimit: 10000} + cfg := &config.Config{DirListingLimit: 1000} mockDB := db.NewMockDBClient() mockDB.MockSchemaReader.GetPrimaryKeyFunc = func(ctx context.Context, schema, table string) (*db.PrimaryKey, error) { @@ -61,7 +61,7 @@ func TestPipelineNode_Readdir(t *testing.T) { // TestPipelineNode_Readdir_AfterOrder tests capabilities after .order/ is applied. func TestPipelineNode_Readdir_AfterOrder(t *testing.T) { - cfg := &config.Config{DirListingLimit: 10000} + cfg := &config.Config{DirListingLimit: 1000} mockDB := db.NewMockDBClient() mockDB.MockSchemaReader.GetPrimaryKeyFunc = func(ctx context.Context, schema, table string) (*db.PrimaryKey, error) { @@ -104,7 +104,7 @@ func TestPipelineNode_Readdir_AfterOrder(t *testing.T) { // TestPipelineNode_Readdir_AfterLimit tests capabilities after .first/N is applied. func TestPipelineNode_Readdir_AfterLimit(t *testing.T) { - cfg := &config.Config{DirListingLimit: 10000} + cfg := &config.Config{DirListingLimit: 1000} mockDB := db.NewMockDBClient() mockDB.MockSchemaReader.GetPrimaryKeyFunc = func(ctx context.Context, schema, table string) (*db.PrimaryKey, error) { @@ -155,7 +155,7 @@ func TestPipelineNode_Readdir_AfterLimit(t *testing.T) { // TestPipelineLimitNode_Lookup tests that limit node parses N correctly. func TestPipelineLimitNode_Lookup(t *testing.T) { - cfg := &config.Config{DirListingLimit: 10000} + cfg := &config.Config{DirListingLimit: 1000} mockDB := db.NewMockDBClient() pipeline := NewPipelineContext("public", "users", "id") @@ -179,7 +179,7 @@ func TestPipelineLimitNode_Lookup(t *testing.T) { // TestPipelineByDirNode_Readdir tests that .by/ shows indexed columns. func TestPipelineByDirNode_Readdir(t *testing.T) { - cfg := &config.Config{DirListingLimit: 10000} + cfg := &config.Config{DirListingLimit: 1000} mockDB := db.NewMockDBClient() mockDB.MockIndexReader.GetSingleColumnIndexesFunc = func(ctx context.Context, schema, table string) ([]db.Index, error) { @@ -224,7 +224,7 @@ func TestPipelineByDirNode_Readdir(t *testing.T) { // TestPipelineOrderDirNode_Readdir tests that .order/ shows all columns. func TestPipelineOrderDirNode_Readdir(t *testing.T) { - cfg := &config.Config{DirListingLimit: 10000} + cfg := &config.Config{DirListingLimit: 1000} mockDB := db.NewMockDBClient() mockDB.MockSchemaReader.GetColumnsFunc = func(ctx context.Context, schema, table string) ([]db.Column, error) { @@ -259,7 +259,7 @@ func TestPipelineOrderDirNode_Readdir(t *testing.T) { // TestPipelineOrderColumnNode_Readdir tests that .order// shows directions. func TestPipelineOrderColumnNode_Readdir(t *testing.T) { - cfg := &config.Config{DirListingLimit: 10000} + cfg := &config.Config{DirListingLimit: 1000} mockDB := db.NewMockDBClient() pipeline := NewPipelineContext("public", "users", "id") @@ -291,7 +291,7 @@ func TestPipelineOrderColumnNode_Readdir(t *testing.T) { // TestPipelineExportDirNode_Readdir tests that .export/ shows formats. func TestPipelineExportDirNode_Readdir(t *testing.T) { - cfg := &config.Config{DirListingLimit: 10000} + cfg := &config.Config{DirListingLimit: 1000} mockDB := db.NewMockDBClient() pipeline := NewPipelineContext("public", "users", "id") @@ -330,7 +330,7 @@ func TestPipelineExportDirNode_Readdir(t *testing.T) { // TestPipelineByColumnNode_Pipeline tests that filter is added with indexed=true. func TestPipelineByColumnNode_Pipeline(t *testing.T) { - cfg := &config.Config{DirListingLimit: 10000} + cfg := &config.Config{DirListingLimit: 1000} mockDB := db.NewMockDBClient() // Create base pipeline diff --git a/internal/tigerfs/fuse/sample_test.go b/internal/tigerfs/fuse/sample_test.go index ec890be..71291c7 100644 --- a/internal/tigerfs/fuse/sample_test.go +++ b/internal/tigerfs/fuse/sample_test.go @@ -14,7 +14,7 @@ import ( func TestNewSampleNode(t *testing.T) { cfg := &config.Config{ DefaultSchema: "public", - DirListingLimit: 10000, + DirListingLimit: 1000, } partialRows := NewPartialRowTracker(nil) diff --git a/internal/tigerfs/fuse/table_test.go b/internal/tigerfs/fuse/table_test.go index 4bf2e85..8388fbb 100644 --- a/internal/tigerfs/fuse/table_test.go +++ b/internal/tigerfs/fuse/table_test.go @@ -13,7 +13,7 @@ import ( func TestNewTableNode(t *testing.T) { cfg := &config.Config{ DefaultSchema: "public", - DirListingLimit: 10000, + DirListingLimit: 1000, } partialRows := NewPartialRowTracker(nil) @@ -53,7 +53,7 @@ func TestTableNode_Interfaces(t *testing.T) { func TestTableNode_Getattr(t *testing.T) { cfg := &config.Config{ DefaultSchema: "public", - DirListingLimit: 10000, + DirListingLimit: 1000, } tableNode := NewTableNode(cfg, nil, nil, "public", "users", nil, nil) diff --git a/internal/tigerfs/fuse/templates.go b/internal/tigerfs/fuse/templates.go index e2d7299..034e7c3 100644 --- a/internal/tigerfs/fuse/templates.go +++ b/internal/tigerfs/fuse/templates.go @@ -36,7 +36,7 @@ CREATE TABLE %s ( -- Primary key (choose one pattern): -- id SERIAL PRIMARY KEY, -- id BIGSERIAL PRIMARY KEY, - -- id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + -- id UUID PRIMARY KEY DEFAULT uuidv7(), -- Add columns here: -- name TEXT NOT NULL, diff --git a/internal/tigerfs/nfs/handler_test.go b/internal/tigerfs/nfs/handler_test.go index 3a1293f..7c21a7f 100644 --- a/internal/tigerfs/nfs/handler_test.go +++ b/internal/tigerfs/nfs/handler_test.go @@ -10,6 +10,7 @@ import ( "github.com/go-git/go-billy/v5/memfs" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" + tigerfs "github.com/timescale/tigerfs/internal/tigerfs/fs" nfs "github.com/willscott/go-nfs" ) @@ -285,3 +286,51 @@ func TestVerifierFor_DifferentContentsProduceDifferentVerifier(t *testing.T) { v2 := h.VerifierFor(path, makeContents("uuid-3", "uuid-4")) assert.NotEqual(t, v1, v2, "different contents should produce different verifiers") } + +// TestOpsFileInfo_Mode_Symlink verifies that opsFileInfo preserves os.ModeSymlink +// in the Mode() return value for symlink entries. +func TestOpsFileInfo_Mode_Symlink(t *testing.T) { + // Symlink entry: Mode must include os.ModeSymlink for go-nfs detection + symlinkEntry := &tigerfs.Entry{ + Name: "before", + Mode: os.ModeSymlink | 0777, + Target: "/dev/null", + } + fi := &opsFileInfo{entry: symlinkEntry, path: "/test/before"} + mode := fi.Mode() + + if mode&os.ModeSymlink == 0 { + t.Errorf("symlink entry Mode() = %v, missing os.ModeSymlink bit", mode) + } + if mode.Perm() != 0777 { + t.Errorf("symlink entry Mode().Perm() = %o, want 0777", mode.Perm()) + } + + // Regular file: Mode must NOT include os.ModeSymlink + fileEntry := &tigerfs.Entry{ + Name: "file.txt", + Mode: 0644, + } + fi2 := &opsFileInfo{entry: fileEntry, path: "/test/file.txt"} + mode2 := fi2.Mode() + + if mode2&os.ModeSymlink != 0 { + t.Errorf("regular file Mode() = %v, should not have os.ModeSymlink", mode2) + } + if mode2.Perm() != 0644 { + t.Errorf("regular file Mode().Perm() = %o, want 0644", mode2.Perm()) + } + + // Directory: Mode must NOT include os.ModeSymlink + dirEntry := &tigerfs.Entry{ + Name: "dir", + IsDir: true, + Mode: os.ModeDir | 0755, + } + fi3 := &opsFileInfo{entry: dirEntry, path: "/test/dir"} + mode3 := fi3.Mode() + + if mode3&os.ModeSymlink != 0 { + t.Errorf("directory Mode() = %v, should not have os.ModeSymlink", mode3) + } +} diff --git a/internal/tigerfs/nfs/mount_darwin.go b/internal/tigerfs/nfs/mount_darwin.go index 5d213d9..767bccb 100644 --- a/internal/tigerfs/nfs/mount_darwin.go +++ b/internal/tigerfs/nfs/mount_darwin.go @@ -51,7 +51,7 @@ func Mount(ctx context.Context, cfg *config.Config, connStr, mountpoint string) logging.Info("Database connection established") // Create NFS server - server, err := NewServer(ctx, cfg, dbClient) + server, err := NewServer(ctx, cfg, dbClient, absMountpoint) if err != nil { dbClient.Close() return nil, fmt.Errorf("failed to create NFS server: %w", err) @@ -88,7 +88,14 @@ func Mount(ctx context.Context, cfg *config.Config, connStr, mountpoint string) // (go-nfs) runs in the same process as the client. 128KB is a safe // balance: 4x fewer RPCs than the default with no GC issues. // - rsize=131072: 128KB read chunks (match wsize for consistency) - mountOpts := fmt.Sprintf("locallocks,vers=3,tcp,port=%d,mountport=%d,soft,timeo=300,retrans=2,noresvport,nolocks,wsize=131072,rsize=131072", port, port) + // - noac: Disable NFS client attribute caching. Without this, the client + // caches file attributes (mtime, size) for up to 60 seconds (acregmax). + // TigerFS content can change server-side at any time (undo operations, + // concurrent mounts), and stale attributes cause the client to serve + // stale file data. This does NOT increase SQL queries -- GETATTR requests + // are served from TigerFS's in-memory stat cache (2s TTL). The cost is + // only additional loopback NFS round-trips (~0.1ms each). + mountOpts := fmt.Sprintf("locallocks,vers=3,tcp,port=%d,mountport=%d,soft,timeo=300,retrans=2,noresvport,nolocks,wsize=131072,rsize=131072,noac", port, port) cmd := exec.CommandContext(ctx, "/sbin/mount_nfs", "-o", mountOpts, diff --git a/internal/tigerfs/nfs/ops_filesystem.go b/internal/tigerfs/nfs/ops_filesystem.go index fb05ae7..894aed7 100644 --- a/internal/tigerfs/nfs/ops_filesystem.go +++ b/internal/tigerfs/nfs/ops_filesystem.go @@ -2,6 +2,7 @@ package nfs import ( "context" + "errors" "fmt" "hash/fnv" "io" @@ -14,7 +15,7 @@ import ( "github.com/go-git/go-billy/v5" "github.com/timescale/tigerfs/internal/tigerfs/config" - "github.com/timescale/tigerfs/internal/tigerfs/fs" + fsops "github.com/timescale/tigerfs/internal/tigerfs/fs" "github.com/timescale/tigerfs/internal/tigerfs/logging" nfsfile "github.com/willscott/go-nfs/file" "go.uber.org/zap" @@ -33,7 +34,7 @@ import ( // See cachedFile for cache entry lifecycle details and docs/spec.md § NFS Write // Limitations for the O(n²) write amplification discussion. type OpsFilesystem struct { - ops *fs.Operations + ops *fsops.Operations cfg *config.Config // cacheMu protects fileCache for concurrent access. @@ -90,7 +91,7 @@ type cachedFile struct { streamed bool // True if file has been partially committed via streaming (append mode) // Database connection for commit operations - ops *fs.Operations + ops *fsops.Operations // File type flags (determine write behavior) isTrigger bool // DDL trigger files (.test, .commit, .abort) - always fire on close @@ -106,7 +107,7 @@ type cachedFile struct { // // The cfg parameter provides NFS cache tuning settings (NFSStreamingThreshold, // NFSMaxRandomWriteSize, NFSCacheReaperInterval, NFSCacheIdleTimeout). -func NewOpsFilesystem(ops *fs.Operations, cfg *config.Config) *OpsFilesystem { +func NewOpsFilesystem(ops *fsops.Operations, cfg *config.Config) *OpsFilesystem { return &OpsFilesystem{ ops: ops, cfg: cfg, @@ -196,7 +197,7 @@ func (f *OpsFilesystem) getOrCreateCachedFile(filePath string, flags int) *cache // Determine file type from path baseName := path.Base(filePath) - isTrigger := baseName == ".test" || baseName == ".commit" || baseName == ".abort" + isTrigger := baseName == fsops.FileTest || baseName == fsops.FileCommit || baseName == fsops.FileAbort isDDLSQL := baseName == "sql" && isDDLPath(filePath) rowFile := isRowFile(baseName) @@ -498,6 +499,25 @@ func (f *OpsFilesystem) Create(filename string) (billy.File, error) { filename = normalizePath(filename) logging.Info("OpsFilesystem.Create", zap.String("filename", filename)) + // Reject bare-path new-row writes here, before we return a file handle. + // If we let Create succeed and fail at Close, the macOS NFS client has + // already cached a positive file-typed dentry for a name the server will + // later expose as a directory, and `ls` suppresses the mismatched entry. + // f.ops is nil in cache-only unit tests; skip validation in that case. + if f.ops != nil { + vctx, vcancel := ctx() + if fsErr := f.ops.ValidateCreate(vctx, filename); fsErr != nil { + vcancel() + logging.Error("OpsFilesystem.Create rejected", + zap.String("filename", filename), + zap.String("message", fsErr.Message), + zap.String("hint", fsErr.Hint), + zap.Error(fsErr.Cause)) + return nil, os.ErrInvalid + } + vcancel() + } + // Get or create cached file entry with O_CREATE|O_TRUNC semantics cached := f.getOrCreateCachedFile(filename, os.O_CREATE|os.O_TRUNC) @@ -549,6 +569,25 @@ func (f *OpsFilesystem) OpenFile(filename string, flag int, perm os.FileMode) (b filename = normalizePath(filename) logging.Info("OpsFilesystem.OpenFile", zap.String("filename", filename), zap.Int("flag", flag)) + // When the client opens with O_CREATE, we may be about to materialize a + // new file handle. Apply the same bare-path rejection as Create so the + // kernel never caches a file-typed dentry for a name the server will + // later expose as a directory. f.ops is nil in cache-only unit tests. + if flag&os.O_CREATE != 0 && f.ops != nil { + vctx, vcancel := ctx() + if fsErr := f.ops.ValidateCreate(vctx, filename); fsErr != nil { + vcancel() + logging.Error("OpsFilesystem.OpenFile rejected", + zap.String("filename", filename), + zap.Int("flag", flag), + zap.String("message", fsErr.Message), + zap.String("hint", fsErr.Hint), + zap.Error(fsErr.Cause)) + return nil, os.ErrInvalid + } + vcancel() + } + isWrite := flag&(os.O_WRONLY|os.O_RDWR|os.O_CREATE|os.O_TRUNC) != 0 // Check if file is already in cache (includes in-flight and truncated files) @@ -636,7 +675,7 @@ func (f *OpsFilesystem) OpenFile(filename string, flag int, perm os.FileMode) (b defer rcancel() content, fsErr := f.ops.ReadFile(rctx, filename) if fsErr != nil { - if fsErr.Code == fs.ErrNotExist { + if fsErr.Code == fsops.ErrNotExist { // If O_CREATE was set, create empty file if flag&os.O_CREATE != 0 { cached = f.getOrCreateCachedFile(filename, flag) @@ -750,11 +789,11 @@ func (f *OpsFilesystem) Stat(filename string) (os.FileInfo, error) { zap.Error(fsErr.Cause)) // Map FSError codes to appropriate OS errors switch fsErr.Code { - case fs.ErrNotExist: + case fsops.ErrNotExist: return nil, os.ErrNotExist - case fs.ErrPermission: + case fsops.ErrPermission: return nil, os.ErrPermission - case fs.ErrInvalidPath, fs.ErrInvalidArgument: + case fsops.ErrInvalidPath, fsops.ErrInvalidArgument: return nil, os.ErrInvalid default: // For other errors, return not exist to avoid false permission denied @@ -786,11 +825,11 @@ func (f *OpsFilesystem) Rename(oldpath, newpath string) error { zap.Int("code", int(fsErr.Code)), zap.String("message", fsErr.Message)) switch fsErr.Code { - case fs.ErrNotExist: + case fsops.ErrNotExist: return os.ErrNotExist - case fs.ErrPermission: + case fsops.ErrPermission: return os.ErrPermission - case fs.ErrInvalidPath, fs.ErrInvalidArgument: + case fsops.ErrInvalidPath, fsops.ErrInvalidArgument: return os.ErrInvalid default: return fmt.Errorf("%s: %w", fsErr.Message, fsErr.Cause) @@ -828,7 +867,7 @@ func (f *OpsFilesystem) Remove(filename string) error { defer dcancel() fsErr := f.ops.Delete(dctx, filename) if fsErr != nil { - if fsErr.Code == fs.ErrNotExist { + if fsErr.Code == fsops.ErrNotExist { return os.ErrNotExist } return fmt.Errorf("%s: %w", fsErr.Message, fsErr.Cause) @@ -863,11 +902,11 @@ func (f *OpsFilesystem) ReadDir(dirname string) ([]os.FileInfo, error) { // Map FSError codes to appropriate OS errors // Use errors that NFS will interpret correctly switch fsErr.Code { - case fs.ErrNotExist: + case fsops.ErrNotExist: return nil, os.ErrNotExist - case fs.ErrPermission: + case fsops.ErrPermission: return nil, os.ErrPermission - case fs.ErrInvalidPath, fs.ErrInvalidArgument: + case fsops.ErrInvalidPath, fsops.ErrInvalidArgument: return nil, os.ErrInvalid default: // For other errors (IO, internal), return a generic error @@ -906,7 +945,7 @@ func (f *OpsFilesystem) MkdirAll(filename string, perm os.FileMode) error { defer mcancel() fsErr := f.ops.Mkdir(mctx, filename) if fsErr != nil { - if fsErr.Code == fs.ErrAlreadyExists { + if fsErr.Code == fsops.ErrAlreadyExists { return nil // MkdirAll doesn't fail if directory exists } return fmt.Errorf("%s: %w", fsErr.Message, fsErr.Cause) @@ -914,19 +953,36 @@ func (f *OpsFilesystem) MkdirAll(filename string, perm os.FileMode) error { return nil } -// Lstat is the same as Stat (no symlinks). +// Lstat returns file info without following symlinks. For symlink entries, +// this returns the symlink's own metadata (with os.ModeSymlink set). +// For non-symlinks, this is identical to Stat. func (f *OpsFilesystem) Lstat(filename string) (os.FileInfo, error) { + // Lstat and Stat currently go through the same ops.Stat path. + // ops.Stat returns symlink entries with os.ModeSymlink set and Target populated. + // The distinction between Lstat and Stat (following vs not following) is handled + // by the kernel/NFS client layer, not by us -- go-nfs calls Lstat for LOOKUP + // and Stat for GETATTR-after-following. return f.Stat(filename) } -// Symlink is not supported. +// Symlink is not supported. TigerFS symlinks are virtual (computed by the fs +// layer for specific paths like .log/ diff links), not user-created. func (f *OpsFilesystem) Symlink(target, link string) error { return fmt.Errorf("symlinks not supported") } -// Readlink is not supported. +// Readlink returns the target path of a symlink. func (f *OpsFilesystem) Readlink(link string) (string, error) { - return "", fmt.Errorf("symlinks not supported") + link = normalizePath(link) + ctx := context.Background() + target, fsErr := f.ops.Readlink(ctx, link) + if fsErr != nil { + if fsErr.Cause != nil { + return "", fmt.Errorf("%s: %w", fsErr.Message, fsErr.Cause) + } + return "", fmt.Errorf("%s", fsErr.Message) + } + return target, nil } // Chroot returns a filesystem rooted at the given path. @@ -968,11 +1024,12 @@ func (f *OpsFilesystem) Chtimes(name string, atime time.Time, mtime time.Time) e name = normalizePath(name) logging.Debug("OpsFilesystem.Chtimes", zap.String("name", name), zap.Time("atime", atime), zap.Time("mtime", mtime)) - // Check if this is a DDL trigger file + // Check if this is a trigger file (DDL or undo .apply) baseName := path.Base(name) - isTrigger := (baseName == ".test" || baseName == ".commit" || baseName == ".abort") && isDDLPath(name) + isTrigger := (baseName == fsops.FileTest || baseName == fsops.FileCommit || baseName == fsops.FileAbort) && isDDLPath(name) + isUndoApply := baseName == fsops.FileApply && strings.Contains(name, "/"+fsops.DirUndo+"/") - if isTrigger { + if isTrigger || isUndoApply { // Check if the trigger was already fired by memFile.Close() (NFS touch // causes both OpenFile+Close and Chtimes, both of which would fire). cached := f.getCachedFile(name) @@ -1006,7 +1063,7 @@ func (f *OpsFilesystem) Chtimes(name string, atime time.Time, mtime time.Time) e // opsFileInfo implements os.FileInfo using fs.Entry. type opsFileInfo struct { - entry *fs.Entry + entry *fsops.Entry path string // Full path for generating unique file IDs } @@ -1018,11 +1075,16 @@ func (fi *opsFileInfo) Size() int64 { return fi.entry.Size } // as the NFS mode field. NFS mode should only contain permission bits (0-07777), // not file type bits. go-nfs correctly uses IsDir() to set the NFS type field, // but the mode field ends up with os.ModeDir (0x80000000) included. -// We return the permission bits only to work around this. -// Note: We still need IsDir() to return true for go-nfs to set the correct NFS type. +// We return the permission bits only for directories and regular files. +// +// EXCEPTION: go-nfs checks Mode() & os.ModeSymlink to detect symlinks +// (in file.go and nfs_onreadlink.go), so we MUST preserve the symlink bit. func (fi *opsFileInfo) Mode() os.FileMode { + if fi.entry.IsSymlink() { + return fi.entry.Mode.Perm() | os.ModeSymlink + } // Return only permission bits (masking off file type) - // go-nfs will still correctly identify this as a directory via IsDir() + // go-nfs will still correctly identify directories via IsDir() return fi.entry.Mode.Perm() } func (fi *opsFileInfo) ModTime() time.Time { return fi.entry.ModTime } @@ -1173,16 +1235,13 @@ func ctx() (context.Context, context.CancelFunc) { // isDDLPath checks if a path is within a DDL staging directory (.create, .modify, .delete). func isDDLPath(p string) bool { - return strings.Contains(p, "/.create/") || strings.Contains(p, "/.modify/") || strings.Contains(p, "/.delete/") + return strings.Contains(p, "/"+fsops.DirCreate+"/") || strings.Contains(p, "/"+fsops.DirModify+"/") || strings.Contains(p, "/"+fsops.DirDelete+"/") } // isRowFile checks if a filename is a row format file (JSON, CSV, TSV, YAML). // Row files represent entire database rows and writes should replace the full content. func isRowFile(filename string) bool { - return strings.HasSuffix(filename, ".json") || - strings.HasSuffix(filename, ".csv") || - strings.HasSuffix(filename, ".tsv") || - strings.HasSuffix(filename, ".yaml") + return fsops.IsRowFormatFile(filename) } // memFile is an in-memory file handle for reading and writing row data. @@ -1518,9 +1577,23 @@ func (f *memFile) Close() error { if fsErr != nil { logging.Error("memFile.Close WriteFile failed", zap.String("path", filePath), + zap.String("message", fsErr.Message), + zap.String("hint", fsErr.Hint), zap.Error(fsErr.Cause)) f.fs.removeFromCache(filePath) - return fmt.Errorf("%s: %w", fsErr.Message, fsErr.Cause) + if fsErr.Cause != nil { + return fmt.Errorf("%s: %w", fsErr.Message, fsErr.Cause) + } + return errors.New(fsErr.Message) + } + + // For virtual directory entries (savepoints, .info/), remove the file + // cache immediately after write. These paths appear as directories in + // ReadDir but the write created a file handle -- keeping the stale file + // cache entry causes go-nfs handle type conflicts (file vs directory). + if strings.Contains(filePath, "/"+fsops.DirSavepoint+"/") || strings.Contains(filePath, "/"+fsops.DirInfo+"/") { + f.fs.removeFromCache(filePath) + return nil } // Mark cache entry clean but keep it alive for Stat to use. diff --git a/internal/tigerfs/nfs/server.go b/internal/tigerfs/nfs/server.go index 8622b3d..db80824 100644 --- a/internal/tigerfs/nfs/server.go +++ b/internal/tigerfs/nfs/server.go @@ -30,11 +30,13 @@ type Server struct { } // NewServer creates a new NFS server. -func NewServer(ctx context.Context, cfg *config.Config, dbClient *db.Client) (*Server, error) { +func NewServer(ctx context.Context, cfg *config.Config, dbClient *db.Client, mountPoint string) (*Server, error) { ctx, cancel := context.WithCancel(ctx) // Create shared fs.Operations ops := fs.NewOperations(cfg, dbClient) + ops.SetMountPoint(mountPoint) + ops.SetUserID(cfg.UserID) // Wrap in billy.Filesystem adapter for go-nfs billyFS := NewOpsFilesystem(ops, cfg) diff --git a/scripts/demo/demo.sh b/scripts/demo/demo.sh index cdffed0..87f3dac 100755 --- a/scripts/demo/demo.sh +++ b/scripts/demo/demo.sh @@ -24,14 +24,16 @@ error() { echo -e "${RED}==>${NC} $1"; } # --------------------------------------------------------------------------- COMMAND="${1:-start}" MODE="" +DEBUG="" for arg in "$@"; do case "$arg" in --docker) MODE="docker" ;; --mac) MODE="mac" ;; + --debug) DEBUG="--log-level debug" ;; start|stop|status|restart|shell) ;; *) - echo "Usage: $0 {start|stop|status|restart|shell} [--docker|--mac]" + echo "Usage: $0 {start|stop|status|restart|shell} [--docker|--mac] [--debug]" echo "" echo "Commands:" echo " start - Start PostgreSQL, mount TigerFS, seed demo apps" @@ -40,9 +42,10 @@ for arg in "$@"; do echo " restart - Stop and start again" echo " shell - Enter the TigerFS environment" echo "" - echo "Modes:" + echo "Options:" echo " --docker Docker containers for both PostgreSQL and TigerFS (default)" echo " --mac PostgreSQL in Docker, TigerFS runs natively on macOS" + echo " --debug Enable debug logging (--log-level debug)" exit 1 ;; esac @@ -134,7 +137,7 @@ docker_start() { info "Mounting TigerFS at $MOUNTPOINT (inside container)..." docker compose -f "$COMPOSE_FILE" exec -T tigerfs \ - tigerfs mount --insecure-no-ssl "$CONN_STR" "$MOUNTPOINT" & + tigerfs mount --insecure-no-ssl $DEBUG "$CONN_STR" "$MOUNTPOINT" & sleep 2 if ! docker_is_mounted; then @@ -242,7 +245,7 @@ mac_start() { mkdir -p "$MOUNTPOINT" info "Mounting TigerFS at $MOUNTPOINT..." - "$REPO_ROOT/bin/tigerfs" mount "$CONN_STR" "$MOUNTPOINT" & + "$REPO_ROOT/bin/tigerfs" mount --insecure-no-ssl $DEBUG "$CONN_STR" "$MOUNTPOINT" & TIGERFS_PID=$! sleep 2 diff --git a/scripts/demo/seed.sh b/scripts/demo/seed.sh index f54abfd..dd73ee1 100755 --- a/scripts/demo/seed.sh +++ b/scripts/demo/seed.sh @@ -34,6 +34,9 @@ info "Creating blog app (markdown, history)..." echo 'markdown,history' > "$MOUNT/.build/blog" sleep 1 # let the build settle +# Create savepoint before any content +echo -e "description\nClean slate before blog content" > "$MOUNT/blog/.savepoint/before-content.tsv" + # Create subdirectories mkdir -p "$MOUNT/blog/tutorials" mkdir -p "$MOUNT/blog/deep-dives" @@ -221,6 +224,9 @@ FROM users; Refresh them with `REFRESH MATERIALIZED VIEW user_stats;` ENDOFFILE +# Create savepoint before edits +echo -e "description\nAll posts created, before any edits" > "$MOUNT/blog/.savepoint/before-edits.tsv" + # Seed blog history by updating posts (captures originals as history entries) info "Seeding blog history..." cat > "$MOUNT/blog/hello-world.md" << 'ENDOFFILE' diff --git a/scripts/test-migration.sh b/scripts/test-migration.sh new file mode 100755 index 0000000..b6d32ac --- /dev/null +++ b/scripts/test-migration.sh @@ -0,0 +1,319 @@ +#!/bin/bash +# test-migration.sh -- Spin up a PostgreSQL container with OLD-format synth data, +# then print the connection string for testing `tigerfs migrate`. +# +# Usage: +# ./scripts/test-migration.sh # start container + load data +# ./scripts/test-migration.sh stop # stop and remove container +# +# After running, test with: +# tigerfs migrate "postgres://demo:demo@localhost:5433/demo" --describe +# tigerfs migrate "postgres://demo:demo@localhost:5433/demo" --dry-run +# tigerfs migrate "postgres://demo:demo@localhost:5433/demo" +# +# Uses port 5433 to avoid conflicting with any existing PostgreSQL on 5432. + +set -e + +CONTAINER_NAME="tigerfs-migration-test" +PORT=5433 +USER=demo +PASS=demo +DB=demo +CONNSTR="postgres://$USER:$PASS@localhost:$PORT/$DB" + +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +RED='\033[0;31m' +NC='\033[0m' + +info() { echo -e "${GREEN}==>${NC} $1"; } +warn() { echo -e "${YELLOW}==>${NC} $1"; } +error() { echo -e "${RED}==>${NC} $1"; } + +# ============================================================================ +# Stop command +# ============================================================================ +if [ "${1:-}" = "stop" ]; then + info "Stopping container $CONTAINER_NAME..." + docker rm -f "$CONTAINER_NAME" 2>/dev/null || true + info "Done." + exit 0 +fi + +# ============================================================================ +# Check if already running +# ============================================================================ +if docker ps -q -f name="$CONTAINER_NAME" | grep -q .; then + warn "Container $CONTAINER_NAME already running on port $PORT" + info "Connection string: $CONNSTR" + info "To stop: $0 stop" + exit 0 +fi + +# Clean up any stopped container with same name +docker rm -f "$CONTAINER_NAME" 2>/dev/null || true + +# ============================================================================ +# Start PostgreSQL container +# ============================================================================ +info "Starting PostgreSQL container on port $PORT..." +docker run -d \ + --name "$CONTAINER_NAME" \ + -p "$PORT:5432" \ + -e POSTGRES_USER="$USER" \ + -e POSTGRES_PASSWORD="$PASS" \ + -e POSTGRES_DB="$DB" \ + timescale/timescaledb-ha:pg18 > /dev/null + +info "Waiting for PostgreSQL to be ready..." +for i in $(seq 1 30); do + if docker exec "$CONTAINER_NAME" pg_isready -U "$USER" -d "$DB" > /dev/null 2>&1; then + break + fi + sleep 1 +done + +if ! docker exec "$CONTAINER_NAME" pg_isready -U "$USER" -d "$DB" > /dev/null 2>&1; then + error "PostgreSQL did not become ready in 30 seconds" + docker logs "$CONTAINER_NAME" + exit 1 +fi + +info "PostgreSQL is ready." + +# ============================================================================ +# Load OLD-format synth data (pre-ADR-017) +# ============================================================================ +info "Loading old-format synth data..." + +docker exec -i "$CONTAINER_NAME" psql -U "$USER" -d "$DB" << 'ENDSQL' + +-- Enable TimescaleDB +CREATE EXTENSION IF NOT EXISTS timescaledb; + +-- Create tigerfs schema +CREATE SCHEMA IF NOT EXISTS tigerfs; + +-- ============================================================================ +-- App 1: docs (markdown with history) -- OLD SCHEMA (no parent_id) +-- ============================================================================ + +-- Source table: old format (UNIQUE on filename+filetype, no parent_id) +CREATE TABLE tigerfs.docs ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + filename TEXT NOT NULL, + filetype TEXT NOT NULL DEFAULT 'file' CHECK (filetype IN ('file', 'directory')), + title TEXT, + author TEXT, + headers JSONB DEFAULT '{}'::jsonb, + body TEXT, + encoding TEXT NOT NULL DEFAULT 'utf8' CHECK (encoding IN ('utf8', 'base64')), + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + modified_at TIMESTAMPTZ NOT NULL DEFAULT now(), + UNIQUE(filename, filetype) +); + +-- View with tigerfs comment +CREATE VIEW public.docs AS SELECT * FROM tigerfs.docs; +COMMENT ON VIEW public.docs IS 'tigerfs:md,history'; + +-- Modified_at trigger +CREATE FUNCTION tigerfs.set_docs_modified_at() RETURNS TRIGGER AS $$ +BEGIN NEW.modified_at = now(); RETURN NEW; END; +$$ LANGUAGE plpgsql; +CREATE TRIGGER trg_docs_modified_at BEFORE UPDATE ON tigerfs.docs + FOR EACH ROW EXECUTE FUNCTION tigerfs.set_docs_modified_at(); + +-- History table: old column names (id, _history_id, _operation) +CREATE TABLE tigerfs.docs_history ( + id UUID, + filename TEXT NOT NULL, + filetype TEXT, + title TEXT, + author TEXT, + headers JSONB, + body TEXT, + encoding TEXT, + created_at TIMESTAMPTZ, + modified_at TIMESTAMPTZ, + _history_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY, + _operation TEXT NOT NULL +); + +-- Old-style archive trigger +CREATE FUNCTION tigerfs.archive_docs_history() RETURNS TRIGGER AS $$ +BEGIN + INSERT INTO tigerfs.docs_history + (id, filename, filetype, title, author, headers, body, encoding, + created_at, modified_at, _history_id, _operation) + VALUES + (OLD.id, OLD.filename, OLD.filetype, OLD.title, OLD.author, OLD.headers, + OLD.body, OLD.encoding, OLD.created_at, OLD.modified_at, + uuidv7(), TG_OP::text); + IF TG_OP = 'DELETE' THEN RETURN OLD; END IF; + RETURN NEW; +END; +$$ LANGUAGE plpgsql; +CREATE TRIGGER trg_docs_history_archive BEFORE UPDATE OR DELETE ON tigerfs.docs + FOR EACH ROW EXECUTE FUNCTION tigerfs.archive_docs_history(); + +-- Old-style log table (history_id, insert/update/delete types) +CREATE TABLE tigerfs.docs_log ( + log_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY, + user_id TEXT, + type TEXT NOT NULL CHECK (type IN ('insert', 'update', 'delete', 'undo')), + file_id UUID NOT NULL, + filename TEXT NOT NULL, + history_id UUID, + description TEXT +) WITH ( + tsdb.hypertable, + tsdb.partition_column = 'log_id', + tsdb.chunk_interval = '7 days', + tsdb.segmentby = 'file_id', + tsdb.orderby = 'log_id ASC' +); +CREATE INDEX idx_docs_log_by_file ON tigerfs.docs_log (file_id, log_id ASC); + +-- Savepoint table +CREATE TABLE tigerfs.docs_savepoint ( + savepoint_id UUID NOT NULL DEFAULT uuidv7() PRIMARY KEY, + user_id TEXT, + name TEXT NOT NULL UNIQUE, + description TEXT +); + +-- ============================================================================ +-- Insert data with OLD path-encoded filenames (slashes in filename column) +-- ============================================================================ + +-- Root-level files +INSERT INTO tigerfs.docs (filename, filetype, title, body) +VALUES ('readme.md', 'file', 'Welcome', '# Welcome to Docs'); + +-- Directories (old style: full path as filename) +INSERT INTO tigerfs.docs (filename, filetype) +VALUES ('getting-started', 'directory'); +INSERT INTO tigerfs.docs (filename, filetype) +VALUES ('reference', 'directory'); + +-- Nested files (old style: full path as filename) +INSERT INTO tigerfs.docs (filename, filetype, title, author, body) +VALUES ('getting-started/installation.md', 'file', 'Installation Guide', 'Alice', + '# Installation\n\nFollow these steps to install.'); +INSERT INTO tigerfs.docs (filename, filetype, title, author, body) +VALUES ('getting-started/quick-start.md', 'file', 'Quick Start', 'Bob', + '# Quick Start\n\nGet up and running in 5 minutes.'); +INSERT INTO tigerfs.docs (filename, filetype, title, author, body) +VALUES ('reference/configuration.md', 'file', 'Configuration', 'Alice', + '# Configuration\n\nAll configuration options explained.'); +INSERT INTO tigerfs.docs (filename, filetype, title, author, body) +VALUES ('reference/api-reference.md', 'file', 'API Reference', 'Charlie', + '# API Reference\n\nComplete API documentation.'); + +-- Simulate some edits to create history entries +UPDATE tigerfs.docs +SET body = '# Installation\n\nUpdated installation instructions for v2.', + title = 'Installation Guide (v2)' +WHERE filename = 'getting-started/installation.md'; + +UPDATE tigerfs.docs +SET body = '# Quick Start\n\nRevised quick start guide.' +WHERE filename = 'getting-started/quick-start.md'; + +-- Insert log entries with OLD type names +INSERT INTO tigerfs.docs_log (file_id, type, filename) +SELECT id, 'insert', filename FROM tigerfs.docs WHERE filetype = 'file'; + +INSERT INTO tigerfs.docs_log (file_id, type, filename, history_id) +SELECT d.id, 'update', d.filename, h._history_id +FROM tigerfs.docs d +JOIN tigerfs.docs_history h ON h.id = d.id +WHERE d.filename = 'getting-started/installation.md' +LIMIT 1; + +INSERT INTO tigerfs.docs_log (file_id, type, filename, history_id) +SELECT d.id, 'update', d.filename, h._history_id +FROM tigerfs.docs d +JOIN tigerfs.docs_history h ON h.id = d.id +WHERE d.filename = 'getting-started/quick-start.md' +LIMIT 1; + +-- Create a savepoint +INSERT INTO tigerfs.docs_savepoint (name, description) +VALUES ('before-v2', 'Savepoint before v2 updates'); + +-- ============================================================================ +-- App 2: snippets (plain text, no history) -- OLD SCHEMA +-- ============================================================================ + +CREATE TABLE tigerfs.snippets ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + filename TEXT NOT NULL, + filetype TEXT NOT NULL DEFAULT 'file' CHECK (filetype IN ('file', 'directory')), + body TEXT, + encoding TEXT NOT NULL DEFAULT 'utf8' CHECK (encoding IN ('utf8', 'base64')), + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + modified_at TIMESTAMPTZ NOT NULL DEFAULT now(), + UNIQUE(filename, filetype) +); + +CREATE VIEW public.snippets AS SELECT * FROM tigerfs.snippets; +COMMENT ON VIEW public.snippets IS 'tigerfs:txt'; + +-- Flat files (no hierarchy) +INSERT INTO tigerfs.snippets (filename, body) VALUES ('todo.txt', 'Buy groceries'); +INSERT INTO tigerfs.snippets (filename, body) VALUES ('notes.txt', 'Meeting at 3pm'); + +-- ============================================================================ +-- Summary +-- ============================================================================ + +DO $$ +DECLARE + doc_count INT; + hist_count INT; + log_count INT; + snippet_count INT; +BEGIN + SELECT count(*) INTO doc_count FROM tigerfs.docs; + SELECT count(*) INTO hist_count FROM tigerfs.docs_history; + SELECT count(*) INTO log_count FROM tigerfs.docs_log; + SELECT count(*) INTO snippet_count FROM tigerfs.snippets; + RAISE NOTICE 'Loaded: % docs rows, % history entries, % log entries, % snippets', + doc_count, hist_count, log_count, snippet_count; +END $$; + +ENDSQL + +echo "" +info "Old-format data loaded successfully." +echo "" +info "Data summary:" +info " docs: 7 rows (2 directories + 5 files), full-path filenames" +info " docs_history: 2 entries (old column names: id, _history_id, _operation)" +info " docs_log: 7 entries (old types: insert, update; old column: history_id)" +info " snippets: 2 rows (flat, no history)" +echo "" +info "To test migration:" +echo "" +echo " # Check what needs migrating:" +echo " tigerfs migrate \"$CONNSTR\" --describe" +echo "" +echo " # Preview the SQL:" +echo " tigerfs migrate \"$CONNSTR\" --dry-run" +echo "" +echo " # Run the migration:" +echo " tigerfs migrate \"$CONNSTR\"" +echo "" +echo " # Verify (should say 'No pending migrations'):" +echo " tigerfs migrate \"$CONNSTR\" --describe" +echo "" +echo " # Mount and verify:" +echo " tigerfs mount \"$CONNSTR\" /tmp/mig-test" +echo " ls /tmp/mig-test/public/docs/" +echo " ls /tmp/mig-test/public/docs/getting-started/" +echo " cat /tmp/mig-test/public/docs/getting-started/installation.md" +echo "" +info "To stop: $0 stop" diff --git a/skills/tigerfs/SKILL.md b/skills/tigerfs/SKILL.md index da44ad6..beeca87 100644 --- a/skills/tigerfs/SKILL.md +++ b/skills/tigerfs/SKILL.md @@ -19,11 +19,14 @@ Each directory in a TigerFS mount is either file-first or data-first: ``` mount/ -├── notes/ # File-first (markdown app) +├── notes/ # File-first (markdown workspace) │ ├── hello.md -│ ├── tutorials/ -│ └── .history/ # Versioned history (if enabled) -├── snippets/ # File-first (plain text app) +│ ├── tutorials/ # Subdirectories: only real files, no virtual dirs +│ ├── .history/ # Versioned history (root level only) +│ ├── .log/ # Operation log (root level only; pipeline hidden from ls) +│ ├── .savepoint/ # Named bookmarks for undo (root level only) +│ └── .undo/ # Preview and apply undo operations (root level only) +├── snippets/ # File-first (plain text workspace) │ └── bash-loop.txt ├── .tables/ # Backing tables in tigerfs schema │ └── notes/ # Data-first access to notes backing table @@ -31,12 +34,13 @@ mount/ │ ├── .info/ │ ├── .by/ │ └── 1/ 2/ 3/ ... -└── .build/ # Create new apps +├── .info/ # Mount-level metadata (user identity) +└── .build/ # Create new workspaces ``` ## File-First -A transactional, shareable filesystem backed by a database. Multiple users and agents can read and write concurrently. Create apps with `.build/`: +A transactional, shareable filesystem backed by a database. Multiple users and agents can read and write concurrently. Create workspaces with `.build/`: ```bash Bash "echo 'markdown' > mount/.build/notes" @@ -51,17 +55,72 @@ See [files.md](files.md) for full details on schemas, column roles, directories, Because the filesystem is transactional and shared, it can implement collaborative workflows: When asked to create a **task list, kanban board, todo, or project tracker**: -> Read [recipes.md](recipes.md) Recipe 1 and follow it exactly. +> Read [recipes.md](recipes.md) Recipe 4 and follow it exactly. Core principle: **directories = states** (`todo/`, `doing/`, `done/`), **`mv` = transitions**. Do NOT use `status` frontmatter fields. When asked to create a **knowledge base, wiki, or documentation store**: -> Read [recipes.md](recipes.md) Recipe 2 and follow it exactly. +> Read [recipes.md](recipes.md) Recipe 5 and follow it exactly. When asked to **save or resume session context**: -> Read [recipes.md](recipes.md) Recipe 3 and follow it exactly. +> Read [recipes.md](recipes.md) Recipe 6 and follow it exactly. When asked to **keep a log of what you do**: -> Read [recipes.md](recipes.md) Recipe 4 and follow it exactly. +> Read [recipes.md](recipes.md) Recipe 7 and follow it exactly. + +When asked to **revert, roll back, or undo changes**: +> Use savepoints and `.undo/`. See the Safe Editing section below and [files.md](files.md) for full details. + +## Safe Editing with Savepoints + +Before making multiple or risky edits, always create a savepoint: + +```bash +Bash "echo '{\"description\":\"Before investigating bug #42\"}' > mount/workspace/.savepoint/before-investigation.json" +``` + +**When to create a savepoint:** investigating a bug, debugging, refactoring, multi-file edits, or any uncertain operation. + +**When to undo:** user asks to revert, agent realizes the approach isn't working, or changes were made to wrong files. + +**Before undoing:** always preview first and get user confirmation. Undo is destructive. + +**Undo is undoable:** undo operations are logged (type='undo'), so you can undo an undo. Create a savepoint before a major undo for extra safety -- if the result isn't what was expected, undo back to that savepoint. + +### Common Workflows + +**"Create a savepoint"** +``` +Bash "echo '{\"description\":\"\"}' > mount/workspace/.savepoint/.json" +``` + +**"What changed since the savepoint?"** +1. Read the summary: `Read "mount/workspace/.undo/to-savepoint//.info/summary"` +2. If <= 5 files affected: for each file, read its before and current state, summarize cumulative changes in English +3. If > 5 files: present the summary table (type, filename, user, timestamp) +4. If user wants raw diffs: `Bash "cd mount/workspace && diff -ru .undo/to-savepoint/ . -x '.*'"` + +**"What changed in this file?"** +1. Find recent edits: `Read "mount/workspace/.log/.by/filename//.last/5/.export/json"` +2. For each edit, read before and after, compare them, summarize in English (e.g., "Added 'Recent Updates' section with 3 new bullet points") +3. Present with log_ids so the user can pick one to undo + +**"Show me the diff"** +- Savepoint: `Bash "cd mount/workspace && diff -ru .undo/to-savepoint/ . -x '.*'"` +- Single file: `Bash "diff -u --color mount/workspace/.log//before mount/workspace/.log//after"` + +**"Undo to the savepoint"** +1. Read the summary: `Read "mount/workspace/.undo/to-savepoint//.info/summary"` +2. If <= 5 files: summarize cumulative changes per file in English +3. Present to user: "This will undo N changes: [summary]. Go ahead?" +4. Only if confirmed: `Bash "touch mount/workspace/.undo/to-savepoint//.apply"` + +**"Undo this one change"** +1. Read the log entry summary: `Read "mount/workspace/.undo/id//.info/summary"` +2. Show the diff: `Bash "diff -u --color mount/workspace/.log//before mount/workspace/.log//after"` +3. Present to user: "This will revert [description]. Go ahead?" +4. Only if confirmed: `Bash "touch mount/workspace/.undo/id//.apply"` + +For advanced cases (filtered undo, per-user undo, pipeline queries), construct paths directly from [files.md](files.md). See [recipes.md](recipes.md) Recipes 1-3 for complete workflow patterns. ## Data-First @@ -80,8 +139,8 @@ Read "mount/users/.by/status/active/.export/json" # Filtered export | Size | Strategy | |------|----------| | **~100 rows or less** | Glob patterns and row-as-directory access are fine | -| **100 - 10,000 rows** | Prefer `.export/` over reading individual rows to avoid 1 query per row. Use `.by/`, `.filter/`, `.first/`, `.last/`, `.sample/` for selective access when possible | -| **10,000+ rows** | Large tables are limited to 10,000 rows by default; use `.all/` if you actually need all rows. Strongly prefer `.export/` over reading individual rows. Use `.by/`, `.filter/`, `.first/`, `.last/`, `.sample/` for selective access whenever possible | +| **100 - 1,000 rows** | Prefer `.export/` over reading individual rows to avoid 1 query per row. Use `.by/`, `.filter/`, `.first/`, `.last/`, `.sample/` for selective access when possible | +| **1,000+ rows** | Large tables are limited to 1,000 rows by default; use `.all/` if you actually need all rows. Strongly prefer `.export/` over reading individual rows. Use `.by/`, `.filter/`, `.first/`, `.last/`, `.sample/` for selective access whenever possible | Always check `.info/count` first to choose the right strategy. @@ -92,13 +151,20 @@ See [data.md](data.md) for the full reference. | Goal | Tool Call | |------|-----------| | **File-First** | | -| List files | `Glob "mount/app/*.md"` or `Glob "mount/app/**/*.md"` (recursive) | -| Read file | `Read "mount/app/file.md"` | -| Write file | `Write "mount/app/file.md"` with content | -| Delete file | `Bash "rm mount/app/file.md"` | -| Search | `Grep pattern="term" path="mount/app/"` | -| History versions | `Glob "mount/app/.history/file.md/*"` | -| Read old version | `Read "mount/app/.history/file.md/"` | +| List files | `Glob "mount/workspace/*.md"` or `Glob "mount/workspace/**/*.md"` (recursive) | +| Read file | `Read "mount/workspace/file.md"` | +| Write file | `Write "mount/workspace/file.md"` with content | +| Delete file | `Bash "rm mount/workspace/file.md"` | +| Search | `Grep pattern="term" path="mount/workspace/"` | +| History versions | `Glob "mount/workspace/.history/file.md/*"` | +| Read old version | `Read "mount/workspace/.history/file.md/"` | +| **Savepoints & Undo** | | +| Create savepoint | `Bash "echo '{\"description\":\"Before refactoring\"}' > mount/workspace/.savepoint/name.json"` | +| Diff all changes since savepoint | `Bash "cd mount/workspace && diff -ru .undo/to-savepoint/name . -x '.*'"` | +| Undo to savepoint | Preview first -- see "Undo to the savepoint" in Common Workflows above | +| Undo one change | Preview first -- see "Undo this one change" in Common Workflows above | +| File drift since a change | `Bash "diff -u --color mount/workspace/.log//before mount/workspace/.log//current"` | +| View recent log | `Read "mount/workspace/.log/.last/10/.export/json"` | | **Data-First** | | | Row count | `Read "mount/t/.info/count"` | | Schema / columns | `Read "mount/t/.info/schema"` or `.info/columns` | @@ -112,12 +178,32 @@ See [data.md](data.md) for the full reference. | Insert row | `Write "mount/t/new.json"` with JSON | | Delete row | `Bash "rm mount/t/pk"` | +## Directory Scanning Safety + +**Virtual directory layout:** `.history/`, `.log/`, `.savepoint/`, `.undo/` only appear at the **workspace root** level, not inside subdirectories. Subdirectories contain only real files and folders. Pipeline capabilities (`.by/`, `.filter/`, `.order/`, `.export/`) inside `.log/` and `.savepoint/` are accessible by explicit path but hidden from `ls` to prevent recursive scanner blowup. + +**Never recursively scan these directories** -- always use targeted access patterns from the Quick Reference above: +- `.log/` -- pipeline capabilities hidden from listing, use `.log/.last/10/.export/json` +- `.savepoint/` -- same as `.log/`, use explicit paths +- `.history/` -- one entry per file version; grows with every edit +- `.undo/` -- preview trees that mirror the affected file hierarchy +- `.by//` -- lists every distinct value; each expands to filtered rows +- `.filter//` -- same as `.by/` with higher limit +- `.export/` -- reading files triggers full table/query dumps +- `.import/` -- write interface; scanning could trigger unintended operations + +**Safe to scan:** +- Regular files and subdirectories in a workspace (e.g., `Glob "mount/workspace/**/*.md"`) +- `.info/` -- small, fixed set of metadata files + ## Anti-Patterns | Don't | Do Instead | |-------|------------| | **File-First** | | | Put `status:` in frontmatter to track state | Use directories as states (`todo/`, `doing/`, `done/`), `mv` to transition | +| Manually reverse edits across files to "undo" | Create savepoints, use `.undo/` to revert atomically | +| Undo without previewing first | Always read `.info/summary` and get user confirmation before applying | | **Data-First** | | | Read individual rows in a loop for large tables | Use `.export/json` or `.export/csv` for bulk access | | Write full row JSON expecting replace semantics | JSON/CSV/TSV writes are **PATCH** -- only specified keys update | @@ -130,7 +216,7 @@ When asked to **create, mount, fork, or manage a database or filesystem**, see [ ## Detailed References -- [files.md](files.md) -- File-first: markdown apps, plain text apps, directories, and history +- [files.md](files.md) -- File-first: workspaces with files and directories, and history - [data.md](data.md) -- Data-first: row-as-file, row-as-directory, metadata, indexes, pipeline queries -- [recipes.md](recipes.md) -- Recipes: kanban boards, knowledge bases, session context, snippets +- [recipes.md](recipes.md) -- Recipes: kanban boards, knowledge bases, session context, snippets, safe exploration, compare approaches, multi-agent undo - [ops.md](ops.md) -- Operations: mount, create, fork, status, unmount diff --git a/skills/tigerfs/data.md b/skills/tigerfs/data.md index cb1989f..076a9aa 100644 --- a/skills/tigerfs/data.md +++ b/skills/tigerfs/data.md @@ -23,7 +23,7 @@ mount/ │ ├── .columns/col1,col2/ # Column projection │ ├── .export/json|csv|tsv # Bulk export │ ├── .import/json|csv|tsv # Bulk import -│ ├── .all/ # Access all rows (bypasses 10,000 row default limit) +│ ├── .all/ # Access all rows (hidden from ls; bypasses 1,000 row limit) │ ├── .indexes/ # Index management (DDL) │ ├── .modify/ # Table modification (DDL) │ ├── .delete/ # Table deletion (DDL) @@ -50,6 +50,21 @@ Read-only files describing the table. **Check `.info/count` first** to choose th | `columns` | Column names, one per line | `id\nname\nemail\nage` | | `indexes` | Index descriptions | `PRIMARY KEY: id\nUNIQUE: email` | +### Mount-Level Metadata (mount/.info/) + +| File | Content | Read/Write | +|------|---------|------------| +| `user` | Current user identity for log entries | Read/Write | + +``` +Read "mount/.info/user" # Current identity +Bash "echo 'agent-7' > mount/.info/user" # Change identity at runtime +``` + +### UUIDv7 Display Format + +Log entries, history versions, and other UUIDs use a human-readable display format: `2026-04-07T143000.123Z-zzz0063hd8e5r42` (UTC timestamp + base36 suffix). These are filesystem-safe, case-insensitive, and sort chronologically. + ## Data Formats All rows can be read and written in four formats by using the corresponding file extension: @@ -74,14 +89,14 @@ Read "mount/users/1/email" # Single column value (raw text) ### Navigating Tables -Directory listings are limited to 10,000 rows by default. Use `.all/` to bypass this limit. +Directory listings are limited to 1,000 rows by default. Use `.all/` to bypass this limit. Note: `.all/` does not appear in `ls` output (to prevent infinite recursion for recursive scanners), but works when accessed directly. Pagination directories (`.first/`, `.last/`, `.sample/`) appear in `ls` but show empty contents -- navigate directly with a number (e.g., `.first/50/`). ``` Glob "mount/users/*.json" # List rows (small tables only) Glob "mount/orders/.first/20/*" # First 20 PKs (ascending) Glob "mount/orders/.last/10/*" # Last 10 PKs (descending, most recent) Glob "mount/orders/.sample/50/*" # 50 random PKs -Glob "mount/big_table/.all/*" # All PKs (bypasses 10,000 row limit) +Glob "mount/big_table/.all/*" # All PKs (bypasses 1,000 row limit) ``` After getting PKs, read individual rows: `Read "mount/orders/1.json"` diff --git a/skills/tigerfs/files.md b/skills/tigerfs/files.md index 8619d3d..0a9d967 100644 --- a/skills/tigerfs/files.md +++ b/skills/tigerfs/files.md @@ -2,18 +2,18 @@ Reference for file-first mode -- reading and writing markdown and plain text files backed by a database. -## Creating Apps +## Creating Workspaces -Apps create a file-first workspace -- a directory of markdown or text files backed by a database table. Create an app when you need a new shared workspace, knowledge base, or document store. +Create a workspace when you need a new shared directory of files backed by a database. ```bash Bash "echo 'markdown' > mount/.build/notes" # Markdown with frontmatter Bash "echo 'markdown,history' > mount/.build/notes" # With versioned history Bash "echo 'plaintext' > mount/.build/snippets" # Plain text, no frontmatter -Bash "echo 'history' > mount/.build/notes" # Add history to existing app +Bash "echo 'history' > mount/.build/notes" # Add history to existing workspace ``` -Each app creates a file-first directory (`mount/notes/`) backed by a table in the `tigerfs` schema. Access the backing table via `mount/.tables/notes/`. To add file-first access to an existing data-first table: `echo 'markdown' > mount/posts/.format/markdown` +Each workspace creates a directory (`mount/notes/`) backed by a table in the `tigerfs` schema. Access the backing table via `mount/.tables/notes/`. To add file-first access to an existing data-first table: `echo 'markdown' > mount/posts/.format/markdown` ## File Structure @@ -87,7 +87,7 @@ Writing `mount/notes/a/b/file.md` auto-creates `a/` and `a/b/`. No need to `mkdi ## Backing Table -Every file-first app has a backing table in the `tigerfs` schema, accessible via the `.tables/` directory. +Every workspace has a backing table in the `tigerfs` schema, accessible via the `.tables/` directory. ``` Read "mount/.tables/notes/.info/schema" # Table schema @@ -100,7 +100,7 @@ See [data.md](data.md) for the full data-first reference. ## Versioned History -Every update and delete is captured as a read-only timestamped snapshot in `.history/`. Requires the `history` feature (see [Creating Apps](#creating-apps)). +Every update and delete is captured as a read-only timestamped snapshot in `.history/`. Requires the `history` feature (see [Creating Workspaces](#creating-workspaces)). Each directory has its own `.history/` scoped to files in that directory: @@ -131,4 +131,102 @@ UUID browsing (`.by/`) is available at the root `.history/` level only, not in s 3. Read the current file: `Read "mount/notes/hello.md"` 4. Compare and report differences. -To restore: read the old version from `.history/`, then Write it back to the current path. +For single-file recovery, find recent edits via `.log/.by/filename/hello.md/.last/5/.export/json`, then undo a specific one with `touch .undo/id//.apply`. For multi-file rollback to a known state, use `touch .undo/to-savepoint//.apply` which handles all affected files atomically. `.history/` is best for reading and comparing old versions; use `.undo/` for restoring. See SKILL.md "Common Workflows" for the full multi-step agent behavior. + +## User Identity + +Each mount has an optional user identity used for log entries, savepoint auto-injection, and per-user undo filtering. + +``` +Read "mount/.info/user" # Read current identity +Bash "echo 'agent-7' > mount/.info/user" # Set identity at runtime +``` + +Set at mount time: `--user-id agent-7` or `TIGERFS_USER_ID=agent-7`. See [ops.md](ops.md). + +## Operation Log + +Every create, edit, rename, and delete on a history-enabled workspace is recorded in `.log/`. Each entry has a stable log_id (UUIDv7), the operation type, affected file, and the user who performed it. + +``` +Glob "mount/notes/.log/.last/10/*" # Recent entries +Read "mount/notes/.log/.last/10/.export/json" # Recent entries as JSON +Glob "mount/notes/.log/.by/user_id/agent-7/.last/5/*" # By user +Glob "mount/notes/.log/.by/type/edit/.last/10/*" # By type +``` + +### Diff Symlinks + +Each log entry directory contains `before`, `after`, and `current` symlinks for diffing: + +```bash +Bash "diff -u --color mount/notes/.log//before mount/notes/.log//after" # What this edit changed +Bash "diff -u --color mount/notes/.log//before mount/notes/.log//current" # Drift since this edit +``` + +History paths are per-directory: `tutorials/.history/getting-started.md/` (not `.history/tutorials/getting-started.md/`). + +Log entry IDs use UUIDv7 display format: `2026-04-07T143000.123Z-zzz0063hd8e5r42` (timestamp + base36 suffix, filesystem-safe). + +## Savepoints + +Named bookmarks for undo-to-savepoint operations. Create one before risky edits. + +```bash +Bash "echo '{\"description\":\"Before investigating bug\"}' > mount/notes/.savepoint/before-investigation.json" +``` + +Savepoint creation requires a format suffix (`.json`, `.tsv`, `.csv`, `.yaml`). JSON is preferred for agents. If `--user-id` is set, `user_id` is auto-injected. + +``` +Glob "mount/notes/.savepoint/*" # List savepoints +Read "mount/notes/.savepoint/before-investigation/description" # Read description +Bash "rm mount/notes/.savepoint/old-savepoint" # Delete savepoint +``` + +### Auto-Savepoints + +TigerFS automatically creates savepoints when it detects an inactivity gap (default 30 minutes). Named `auto--` or `auto-`. Configure via `--auto-savepoint-interval` (set to `0` to disable). + +## Undo + +The `.undo/` directory provides a preview-then-apply interface for reversing operations. + +### Three Modes + +| Mode | Purpose | Listing | +|------|---------|---------| +| `.undo/id//` | Undo a single operation | Summary + apply only (use `.log//before` for diffs) | +| `.undo/to-id//` | Undo all operations after a log entry | Preview tree of affected files | +| `.undo/to-savepoint//` | Undo all operations after a savepoint | Preview tree of affected files | + +### Preview and Apply + +```bash +# What would undo do? +Read "mount/notes/.undo/to-savepoint/before-investigation/.info/summary" + +# Diff all affected files +Bash "cd mount/notes && diff -ru .undo/to-savepoint/before-investigation . -x '.*'" + +# Diff since a specific log entry +Bash "cd mount/notes && diff -ru .undo/to-id/ . -x '.*'" + +# Single-file diff (drift since a specific change) +Bash "diff -u --color mount/notes/.log//before mount/notes/.log//current" + +# Apply undo (destructive -- always preview first) +Bash "touch mount/notes/.undo/to-savepoint/before-investigation/.apply" +``` + +### Per-User Undo + +Only undo a specific user's changes, preserving other users' work: + +```bash +Bash "touch mount/notes/.undo/to-savepoint/before-investigation/.by/user_id/agent-7/.apply" +``` + +### Undo of Undo + +Undo operations are logged. You can undo an undo by targeting its log entry. Create a savepoint before a major undo for extra safety. diff --git a/skills/tigerfs/ops.md b/skills/tigerfs/ops.md index 7e7c9f7..d2ebcf2 100644 --- a/skills/tigerfs/ops.md +++ b/skills/tigerfs/ops.md @@ -21,6 +21,9 @@ Key flags: - `--schema` -- default schema for queries - `--query-timeout` -- global query timeout (e.g., `30s`, `1m`) - `--foreground` -- run in foreground (don't daemonize) +- `--user-id ` -- user identity for undo log entries (also: `TIGERFS_USER_ID` env) +- `--auto-savepoint-interval ` -- inactivity gap before auto-savepoint (default `30m`, `0` disables) +- `--undo-list-limit ` -- default listing limit for `.undo/` sub-directories (default `100`) ## Cloud Backends diff --git a/skills/tigerfs/recipes.md b/skills/tigerfs/recipes.md index 0a7f2b7..dc142c8 100644 --- a/skills/tigerfs/recipes.md +++ b/skills/tigerfs/recipes.md @@ -1,8 +1,115 @@ # Recipes -Practical patterns for file-first workflows. +Practical patterns for file-first workflows. Workflow patterns come first (how to work safely), then application patterns (what to build). -## Recipe 1: Task Board +--- + +# Workflow Patterns + +## Recipe 1: Safe Agent Exploration + +Create a savepoint before investigating, exploring, or making uncertain changes. Auto-savepoints detect session boundaries automatically. + +### Pattern: Savepoint, Explore, Review, Keep or Revert + +```bash +# 1. Create savepoint before starting +Bash "echo '{\"description\":\"Before investigating bug #42\"}' > mount/notes/.savepoint/before-investigation.json" + +# 2. Explore and make changes (edits, creates, deletes) +# ... agent works ... + +# 3. Review what changed +Read "mount/notes/.undo/to-savepoint/before-investigation/.info/summary" +Bash "cd mount/notes && diff -ru .undo/to-savepoint/before-investigation . -x '.*'" + +# 4a. Keep changes (do nothing -- changes are already saved) + +# 4b. Revert all changes (after user confirmation) +Bash "touch mount/notes/.undo/to-savepoint/before-investigation/.apply" +``` + +### Auto-Savepoints + +TigerFS creates savepoints automatically after 30 minutes of inactivity. If an agent starts working after a gap, an auto-savepoint captures the state before the new session. Named `auto--`. + +``` +Glob "mount/notes/.savepoint/auto-*" # List auto-savepoints +``` + +## Recipe 2: Compare Approaches with Savepoints + +Try two implementations and keep the better one. + +```bash +# 1. Savepoint the baseline +Bash "echo '{\"description\":\"Clean baseline\"}' > mount/app/.savepoint/baseline.json" + +# 2. Implement approach A +# ... agent implements DFS approach ... + +# 3. Savepoint after A +Bash "echo '{\"description\":\"DFS implementation complete\"}' > mount/app/.savepoint/after-dfs.json" + +# 4. Undo to baseline (clean slate for approach B) +Bash "touch mount/app/.undo/to-savepoint/baseline/.apply" + +# 5. Implement approach B +# ... agent implements BFS approach ... + +# 6. Compare: B is current, preview A via undo +Read "mount/app/.undo/to-savepoint/after-dfs/.info/summary" + +# 7a. Keep B (already current -- do nothing) + +# 7b. Keep A instead +Bash "touch mount/app/.undo/to-savepoint/after-dfs/.apply" +# This works because undo-to-savepoint restores the state at that savepoint, +# regardless of what happened after (including B's implementation). +``` + +## Recipe 3: Multi-Agent Selective Undo + +Multiple agents work on the same data with separate identities. An orchestrator can selectively undo one agent's changes while preserving another's. + +### Setup: Separate User IDs + +Each agent mounts with its own identity: + +```bash +# Agent 1: research +tigerfs mount --user-id agent-research postgres://... /mnt/research + +# Agent 2: implementation +tigerfs mount --user-id agent-implement postgres://... /mnt/implement +``` + +Both see the same data. Operations are tagged with the agent's user_id in the log. + +### Selective Undo + +```bash +# Create a shared savepoint +Bash "echo '{\"description\":\"Sprint start\"}' > mount/notes/.savepoint/sprint-start.json" + +# Both agents work... +# agent-research explores and edits files +# agent-implement makes changes too + +# View changes by user +Read "mount/notes/.log/.by/user_id/agent-research/.last/10/.export/json" + +# Undo only agent-research's changes (preserves agent-implement's work) +Bash "touch mount/notes/.undo/to-savepoint/sprint-start/.by/user_id/agent-research/.apply" +``` + +**Caveat:** If two agents edit the same file, per-user undo reverts the file to its state before the specified agent's first edit -- which also reverts the other agent's interleaved edits on that same file. + +--- + +# Application Patterns + +## Recipe 4: Task Board Works as: todo list, kanban board, project tracker, shared queue, work coordination. The core pattern: directories = states, files = items, `mv` = transitions, `author` = ownership. @@ -74,7 +181,7 @@ Shows when the task was moved, who edited it, previous content. Use any directory names: `backlog/`, `sprint/`, `review/`, `shipped/`. The directory IS the state. `mv` IS the transition. No status columns needed. -## Recipe 2: Knowledge Base with History +## Recipe 5: Knowledge Base with History ### Setup @@ -132,7 +239,7 @@ Read old version vs current to see what evolved. | `source` | free text | Where you learned this | | `supersedes` | filename | If this replaces an older fact | -## Recipe 3: Session Context (Resuming Work) +## Recipe 6: Session Context (Resuming Work) ### Setup @@ -169,7 +276,7 @@ Read "mount/sessions/2026-02-24-auth-refactor.md" Date + topic: `2026-02-24-auth-refactor.md`. Use `status` frontmatter for filtering. -## Recipe 4: Activity Log +## Recipe 7: Activity Log Append-only log of what agents and users have done. One file per activity, immutable once written. Multiple agents can write simultaneously without conflicts. @@ -201,3 +308,4 @@ Glob "mount/activity/2026-03-21*" # Today's activit Grep pattern="author: agent-a" path="mount/activity/" glob="*.md" # By agent Grep pattern="type: fix" path="mount/activity/" glob="*.md" # By type ``` + diff --git a/test/integration/dir_mtime_test.go b/test/integration/dir_mtime_test.go new file mode 100644 index 0000000..3d39e4b --- /dev/null +++ b/test/integration/dir_mtime_test.go @@ -0,0 +1,506 @@ +package integration + +import ( + "context" + "fmt" + "os" + "path/filepath" + "testing" + "time" + + "github.com/jackc/pgx/v5/pgxpool" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/timescale/tigerfs/internal/tigerfs/config" +) + +// getDirMtime reads a directory row's modified_at directly from PostgreSQL. +func getDirMtime(t *testing.T, pool *pgxpool.Pool, tableName, dirname string) time.Time { + t.Helper() + var mtime time.Time + err := pool.QueryRow(context.Background(), + fmt.Sprintf(`SELECT modified_at FROM tigerfs.%q WHERE filename = $1 AND filetype = 'directory'`, tableName), + dirname, + ).Scan(&mtime) + require.NoError(t, err, "failed to read mtime for dir %s", dirname) + return mtime +} + +// getRootDirMtime reads the root directory's modified_at (parent_id IS NULL). +func getRootDirMtime(t *testing.T, pool *pgxpool.Pool, tableName string) time.Time { + t.Helper() + var mtime time.Time + // Root entries have parent_id IS NULL. There may be multiple (files + dirs at root), + // but for mtime we want a directory that serves as the root container. + // Root-level files have parent_id IS NULL but we want the implicit "root" -- + // use the max modified_at of rows where parent_id IS NULL as a proxy, + // or query a specific root-level directory if one exists. + // For these tests we create explicit subdirectories and query those. + err := pool.QueryRow(context.Background(), + fmt.Sprintf(`SELECT MAX(modified_at) FROM tigerfs.%q WHERE parent_id IS NULL`, tableName), + ).Scan(&mtime) + require.NoError(t, err, "failed to read root mtime") + return mtime +} + +func TestDirMtime_CreateFile(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "dirmtime1") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/dirmtime1", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create a directory + fsErr = ops.Mkdir(ctx, "/dirmtime1/docs") + require.Nil(t, fsErr) + + pool, err := pgxpool.New(ctx, result.ConnStr) + require.NoError(t, err) + defer pool.Close() + + time.Sleep(10 * time.Millisecond) + mtimeBefore := getDirMtime(t, pool, "dirmtime1", "docs") + + // Create a file in the directory + time.Sleep(10 * time.Millisecond) + fsErr = ops.WriteFile(ctx, "/dirmtime1/docs/hello.md", []byte("---\ntitle: Hello\n---\nContent\n")) + require.Nil(t, fsErr) + + mtimeAfter := getDirMtime(t, pool, "dirmtime1", "docs") + assert.True(t, mtimeAfter.After(mtimeBefore), + "parent dir mtime should increase after file creation (before=%v, after=%v)", mtimeBefore, mtimeAfter) +} + +func TestDirMtime_DeleteFile(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "dirmtime2") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/dirmtime2", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + fsErr = ops.Mkdir(ctx, "/dirmtime2/docs") + require.Nil(t, fsErr) + fsErr = ops.WriteFile(ctx, "/dirmtime2/docs/hello.md", []byte("---\ntitle: Hello\n---\nContent\n")) + require.Nil(t, fsErr) + + pool, err := pgxpool.New(ctx, result.ConnStr) + require.NoError(t, err) + defer pool.Close() + + time.Sleep(10 * time.Millisecond) + mtimeBefore := getDirMtime(t, pool, "dirmtime2", "docs") + + // Delete the file + time.Sleep(10 * time.Millisecond) + fsErr = ops.Delete(ctx, "/dirmtime2/docs/hello.md") + require.Nil(t, fsErr) + + mtimeAfter := getDirMtime(t, pool, "dirmtime2", "docs") + assert.True(t, mtimeAfter.After(mtimeBefore), + "parent dir mtime should increase after file deletion (before=%v, after=%v)", mtimeBefore, mtimeAfter) +} + +func TestDirMtime_MoveFile(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "dirmtime3") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/dirmtime3", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + fsErr = ops.Mkdir(ctx, "/dirmtime3/dirA") + require.Nil(t, fsErr) + fsErr = ops.Mkdir(ctx, "/dirmtime3/dirB") + require.Nil(t, fsErr) + fsErr = ops.WriteFile(ctx, "/dirmtime3/dirA/hello.md", []byte("---\ntitle: Hello\n---\nContent\n")) + require.Nil(t, fsErr) + + pool, err := pgxpool.New(ctx, result.ConnStr) + require.NoError(t, err) + defer pool.Close() + + time.Sleep(10 * time.Millisecond) + mtimeA := getDirMtime(t, pool, "dirmtime3", "dirA") + mtimeB := getDirMtime(t, pool, "dirmtime3", "dirB") + + // Move file from dirA to dirB + time.Sleep(10 * time.Millisecond) + fsErr = ops.Rename(ctx, "/dirmtime3/dirA/hello.md", "/dirmtime3/dirB/hello.md") + require.Nil(t, fsErr) + + mtimeAAfter := getDirMtime(t, pool, "dirmtime3", "dirA") + mtimeBAfter := getDirMtime(t, pool, "dirmtime3", "dirB") + assert.True(t, mtimeAAfter.After(mtimeA), + "source dir mtime should increase after move (before=%v, after=%v)", mtimeA, mtimeAAfter) + assert.True(t, mtimeBAfter.After(mtimeB), + "target dir mtime should increase after move (before=%v, after=%v)", mtimeB, mtimeBAfter) +} + +func TestDirMtime_RenameInPlace(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "dirmtime4") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/dirmtime4", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + fsErr = ops.Mkdir(ctx, "/dirmtime4/docs") + require.Nil(t, fsErr) + fsErr = ops.WriteFile(ctx, "/dirmtime4/docs/hello.md", []byte("---\ntitle: Hello\n---\nContent\n")) + require.Nil(t, fsErr) + + pool, err := pgxpool.New(ctx, result.ConnStr) + require.NoError(t, err) + defer pool.Close() + + time.Sleep(10 * time.Millisecond) + mtimeBefore := getDirMtime(t, pool, "dirmtime4", "docs") + + // Rename file within same directory + time.Sleep(10 * time.Millisecond) + fsErr = ops.Rename(ctx, "/dirmtime4/docs/hello.md", "/dirmtime4/docs/world.md") + require.Nil(t, fsErr) + + mtimeAfter := getDirMtime(t, pool, "dirmtime4", "docs") + assert.True(t, mtimeAfter.After(mtimeBefore), + "parent dir mtime should increase after rename (before=%v, after=%v)", mtimeBefore, mtimeAfter) +} + +func TestDirMtime_EditContent(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "dirmtime5") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/dirmtime5", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + fsErr = ops.Mkdir(ctx, "/dirmtime5/docs") + require.Nil(t, fsErr) + fsErr = ops.WriteFile(ctx, "/dirmtime5/docs/hello.md", []byte("---\ntitle: Hello\n---\nOriginal\n")) + require.Nil(t, fsErr) + + pool, err := pgxpool.New(ctx, result.ConnStr) + require.NoError(t, err) + defer pool.Close() + + time.Sleep(10 * time.Millisecond) + mtimeBefore := getDirMtime(t, pool, "dirmtime5", "docs") + + // Edit file content (not filename or parent) + time.Sleep(10 * time.Millisecond) + fsErr = ops.WriteFile(ctx, "/dirmtime5/docs/hello.md", []byte("---\ntitle: Hello\n---\nUpdated content\n")) + require.Nil(t, fsErr) + + mtimeAfter := getDirMtime(t, pool, "dirmtime5", "docs") + assert.Equal(t, mtimeBefore, mtimeAfter, + "parent dir mtime should NOT change on content edit (before=%v, after=%v)", mtimeBefore, mtimeAfter) +} + +func TestDirMtime_UndoCreate(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "dirmtime6") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/dirmtime6", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + fsErr = ops.Mkdir(ctx, "/dirmtime6/docs") + require.Nil(t, fsErr) + fsErr = ops.WriteFile(ctx, "/dirmtime6/docs/hello.md", []byte("---\ntitle: Hello\n---\nContent\n")) + require.Nil(t, fsErr) + + pool, err := pgxpool.New(ctx, result.ConnStr) + require.NoError(t, err) + defer pool.Close() + + time.Sleep(10 * time.Millisecond) + mtimeBefore := getDirMtime(t, pool, "dirmtime6", "docs") + + // Find the most recent create log entry + entries, fsErr := ops.ReadDir(ctx, "/dirmtime6/.log/.by/type/create") + require.Nil(t, fsErr) + var logID string + for _, e := range entries { + if e.Name[0] != '.' { + logID = e.Name + } + } + require.NotEmpty(t, logID, "should find a create log entry") + + // Undo the file creation (DELETE fires trigger) + time.Sleep(10 * time.Millisecond) + _, err = ops.ExecuteUndoSingle(ctx, "public", "dirmtime6", logID) + require.NoError(t, err) + + mtimeAfter := getDirMtime(t, pool, "dirmtime6", "docs") + assert.True(t, mtimeAfter.After(mtimeBefore), + "parent dir mtime should increase after undo of creation (before=%v, after=%v)", mtimeBefore, mtimeAfter) +} + +func TestDirMtime_UndoDelete(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "dirmtime7") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/dirmtime7", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + fsErr = ops.Mkdir(ctx, "/dirmtime7/docs") + require.Nil(t, fsErr) + fsErr = ops.WriteFile(ctx, "/dirmtime7/docs/hello.md", []byte("---\ntitle: Hello\n---\nContent\n")) + require.Nil(t, fsErr) + + // Delete the file + fsErr = ops.Delete(ctx, "/dirmtime7/docs/hello.md") + require.Nil(t, fsErr) + + pool, err := pgxpool.New(ctx, result.ConnStr) + require.NoError(t, err) + defer pool.Close() + + time.Sleep(10 * time.Millisecond) + mtimeBefore := getDirMtime(t, pool, "dirmtime7", "docs") + + // Undo the deletion (INSERT fires trigger -- row is restored) + time.Sleep(10 * time.Millisecond) + entries, fsErr := ops.ReadDir(ctx, "/dirmtime7/.log/.by/type/delete") + require.Nil(t, fsErr) + var logID string + for _, e := range entries { + if e.Name[0] != '.' { + logID = e.Name + } + } + require.NotEmpty(t, logID, "should find a delete log entry") + _, err = ops.ExecuteUndoSingle(ctx, "public", "dirmtime7", logID) + require.NoError(t, err) + + mtimeAfter := getDirMtime(t, pool, "dirmtime7", "docs") + assert.True(t, mtimeAfter.After(mtimeBefore), + "parent dir mtime should increase after undo of deletion (before=%v, after=%v)", mtimeBefore, mtimeAfter) +} + +func TestDirMtime_UndoToSavepoint(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "dirmtime8") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/dirmtime8", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + fsErr = ops.Mkdir(ctx, "/dirmtime8/docs") + require.Nil(t, fsErr) + + // Create savepoint + fsErr = ops.WriteFile(ctx, "/dirmtime8/.savepoint/sp1.json", []byte(`{"description":"before adds"}`)) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create files after savepoint + fsErr = ops.WriteFile(ctx, "/dirmtime8/docs/a.md", []byte("---\ntitle: A\n---\nContent A\n")) + require.Nil(t, fsErr) + fsErr = ops.WriteFile(ctx, "/dirmtime8/docs/b.md", []byte("---\ntitle: B\n---\nContent B\n")) + require.Nil(t, fsErr) + + pool, err := pgxpool.New(ctx, result.ConnStr) + require.NoError(t, err) + defer pool.Close() + + time.Sleep(10 * time.Millisecond) + mtimeBefore := getDirMtime(t, pool, "dirmtime8", "docs") + + // Undo to savepoint (deletes both files) + time.Sleep(10 * time.Millisecond) + fsErr = ops.WriteFile(ctx, "/dirmtime8/.undo/to-savepoint/sp1/.apply", []byte("apply")) + require.Nil(t, fsErr) + + mtimeAfter := getDirMtime(t, pool, "dirmtime8", "docs") + assert.True(t, mtimeAfter.After(mtimeBefore), + "parent dir mtime should increase after undo to savepoint (before=%v, after=%v)", mtimeBefore, mtimeAfter) +} + +// --- Mount-level tests: verify mtime propagates through NFS/FUSE stat --- + +func TestMount_DirMtime_ReflectsCreate(t *testing.T) { + checkFUSEMountCapability(t) + + dbResult := GetTestDBEmpty(t) + if dbResult == nil { + return + } + defer dbResult.Cleanup() + + cfg := &config.Config{ + PoolSize: 5, + PoolMaxIdle: 2, + DefaultSchema: "public", + DirListingLimit: 1000, + AttrTimeout: 1 * time.Second, + EntryTimeout: 1 * time.Second, + MetadataRefreshInterval: 30 * time.Second, + LogLevel: "warn", + TrailingNewlines: true, + } + + mountpoint := t.TempDir() + filesystem := mountWithTimeout(t, cfg, dbResult.ConnStr, mountpoint, 10*time.Second) + if filesystem == nil { + return + } + defer func() { _ = filesystem.Close() }() + time.Sleep(500 * time.Millisecond) + + // Create workspace with history + wsPath := filepath.Join(mountpoint, ".build", "mtimetest") + err := os.WriteFile(wsPath, []byte("markdown,history"), 0644) + require.NoError(t, err) + time.Sleep(200 * time.Millisecond) + + // Create directory + dirPath := filepath.Join(mountpoint, "mtimetest", "docs") + err = os.Mkdir(dirPath, 0755) + require.NoError(t, err) + time.Sleep(200 * time.Millisecond) + + // Record directory mtime before file creation + dirStat, err := os.Stat(dirPath) + require.NoError(t, err) + mtimeBefore := dirStat.ModTime() + + // Create a file in the directory + time.Sleep(100 * time.Millisecond) + filePath := filepath.Join(dirPath, "hello.md") + err = os.WriteFile(filePath, []byte("---\ntitle: Hello\n---\nContent\n"), 0644) + require.NoError(t, err) + time.Sleep(200 * time.Millisecond) + + // Stat directory again -- mtime should have increased + dirStat, err = os.Stat(dirPath) + require.NoError(t, err) + mtimeAfter := dirStat.ModTime() + + assert.True(t, mtimeAfter.After(mtimeBefore), + "dir mtime via os.Stat should increase after file creation (before=%v, after=%v)", mtimeBefore, mtimeAfter) +} + +func TestMount_DirMtime_StableOnEdit(t *testing.T) { + checkFUSEMountCapability(t) + + dbResult := GetTestDBEmpty(t) + if dbResult == nil { + return + } + defer dbResult.Cleanup() + + cfg := &config.Config{ + PoolSize: 5, + PoolMaxIdle: 2, + DefaultSchema: "public", + DirListingLimit: 1000, + AttrTimeout: 1 * time.Second, + EntryTimeout: 1 * time.Second, + MetadataRefreshInterval: 30 * time.Second, + LogLevel: "warn", + TrailingNewlines: true, + } + + mountpoint := t.TempDir() + filesystem := mountWithTimeout(t, cfg, dbResult.ConnStr, mountpoint, 10*time.Second) + if filesystem == nil { + return + } + defer func() { _ = filesystem.Close() }() + time.Sleep(500 * time.Millisecond) + + // Create workspace + wsPath := filepath.Join(mountpoint, ".build", "mtimetest2") + err := os.WriteFile(wsPath, []byte("markdown,history"), 0644) + require.NoError(t, err) + time.Sleep(200 * time.Millisecond) + + // Create directory + file + dirPath := filepath.Join(mountpoint, "mtimetest2", "docs") + err = os.Mkdir(dirPath, 0755) + require.NoError(t, err) + filePath := filepath.Join(dirPath, "hello.md") + err = os.WriteFile(filePath, []byte("---\ntitle: Hello\n---\nOriginal\n"), 0644) + require.NoError(t, err) + time.Sleep(200 * time.Millisecond) + + // Record directory mtime + dirStat, err := os.Stat(dirPath) + require.NoError(t, err) + mtimeBefore := dirStat.ModTime() + + // Edit file content (not filename or parent) + time.Sleep(100 * time.Millisecond) + err = os.WriteFile(filePath, []byte("---\ntitle: Hello\n---\nUpdated content\n"), 0644) + require.NoError(t, err) + time.Sleep(200 * time.Millisecond) + + // Stat directory -- mtime should NOT have changed + dirStat, err = os.Stat(dirPath) + require.NoError(t, err) + mtimeAfter := dirStat.ModTime() + + assert.Equal(t, mtimeBefore, mtimeAfter, + "dir mtime via os.Stat should NOT change on content edit (before=%v, after=%v)", mtimeBefore, mtimeAfter) +} diff --git a/test/integration/history_test.go b/test/integration/history_test.go index c222341..25a74f8 100644 --- a/test/integration/history_test.go +++ b/test/integration/history_test.go @@ -493,13 +493,18 @@ func TestSynth_HistoryPerDirectory(t *testing.T) { []byte("---\ntitle: README\n---\n\nReadme v2.\n")) require.Nil(t, fsErr, "WriteFile readme v2 should succeed") - // 1. Subdirectory listing should include .history/ + // 1. Subdirectory listing should NOT include virtual dirs + // (.log, .savepoint, .undo, .history are workspace-level only) entries, fsErr := ops.ReadDir(ctx, "/hist_pdir/getting-started") require.Nil(t, fsErr, "ReadDir getting-started should succeed") names := fsEntryNames(entries) - assert.Contains(t, names, ".history", "subdirectory should contain .history") + assert.NotContains(t, names, ".history", "subdirectory should NOT contain .history (workspace-level only)") + assert.NotContains(t, names, ".log", "subdirectory should NOT contain .log") + assert.NotContains(t, names, ".savepoint", "subdirectory should NOT contain .savepoint") + assert.NotContains(t, names, ".undo", "subdirectory should NOT contain .undo") - // 2. Subdirectory .history/ should show only local files (not root files) + // 2. Subdirectory .history/ should still be accessible by explicit path + // and show only local files (not root files) entries, fsErr = ops.ReadDir(ctx, "/hist_pdir/getting-started/.history") require.Nil(t, fsErr, "ReadDir getting-started/.history should succeed") names = fsEntryNames(entries) @@ -599,3 +604,74 @@ func TestSynth_HistoryPerDirectoryStatVersion(t *testing.T) { assert.Greater(t, entry.Size, int64(0), "version file should have content") } } + +// TestSynth_HistoryAfterMoveAccessibleByUUID verifies that after moving a file +// between directories, its history versions are accessible via .history/.by// +// (ADR-017 verification scenario #40). +func TestSynth_HistoryAfterMoveAccessibleByUUID(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "hist_move") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/hist_move", []byte("markdown,history\n")) + require.Nil(t, fsErr) + + // Create file in inbox, edit to produce history (mkdir-p inbox/). + v1 := "---\ntitle: Task V1\n---\nVersion 1\n" + fsErr = ops.WriteFileEnsureDirs(ctx, "/hist_move/inbox/task.md", []byte(v1)) + require.Nil(t, fsErr, "create task.md v1") + + time.Sleep(1100 * time.Millisecond) // distinct UUIDv7 + + v2 := "---\ntitle: Task V2\n---\nVersion 2\n" + fsErr = ops.WriteFile(ctx, "/hist_move/inbox/task.md", []byte(v2)) + require.Nil(t, fsErr, "edit task.md v2") + + // Get the file's UUID via .history/.by/ before moving + entries, fsErr := ops.ReadDir(ctx, "/hist_move/.history") + require.Nil(t, fsErr) + names := fsEntryNames(entries) + require.Contains(t, names, ".by", "root .history should have .by") + + // Read .id to get the file UUID + idContent, fsErr := ops.ReadFile(ctx, "/hist_move/.history/inbox/task.md/.id") + require.Nil(t, fsErr, "should read .id file") + fileUUID := strings.TrimSpace(string(idContent.Data)) + require.Len(t, fileUUID, 36, ".id should be a UUID") + + // Move file from inbox to archive + fsErr = ops.Mkdir(ctx, "/hist_move/archive") + require.Nil(t, fsErr) + fsErr = ops.Rename(ctx, "/hist_move/inbox/task.md", "/hist_move/archive/task.md") + require.Nil(t, fsErr, "move task.md to archive") + + // History should still be accessible via .by// + byEntries, fsErr := ops.ReadDir(ctx, "/hist_move/.history/.by") + require.Nil(t, fsErr, "ReadDir .history/.by should succeed") + byNames := fsEntryNames(byEntries) + assert.Contains(t, byNames, fileUUID, ".by/ should list the file's UUID") + + // List versions for this UUID -- should have 2: + // 1. v1→v2 edit (OLD body = "Version 1") + // 2. move from inbox to archive (OLD body = "Version 2", parent_id change) + versionEntries, fsErr := ops.ReadDir(ctx, "/hist_move/.history/.by/"+fileUUID) + require.Nil(t, fsErr, "ReadDir .by// should succeed") + assert.GreaterOrEqual(t, len(versionEntries), 2, + "should have at least 2 versions (edit + move)") + + // Versions are ordered most-recent-first. The OLDEST version (last in list) + // has the v1 content from before the edit. + oldestVersion := versionEntries[len(versionEntries)-1].Name + require.NotEmpty(t, oldestVersion) + + vContent, fsErr := ops.ReadFile(ctx, "/hist_move/.history/.by/"+fileUUID+"/"+oldestVersion) + require.Nil(t, fsErr, "ReadFile oldest version via .by/ should succeed") + assert.Contains(t, string(vContent.Data), "Version 1", + "oldest version via .by// should contain v1 content") +} diff --git a/test/integration/log_test.go b/test/integration/log_test.go new file mode 100644 index 0000000..19913e1 --- /dev/null +++ b/test/integration/log_test.go @@ -0,0 +1,681 @@ +package integration + +import ( + "context" + "fmt" + "strings" + "testing" + "time" + + "github.com/jackc/pgx/v5/pgxpool" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// TestSynth_LogEntries_Integration verifies that write operations on a +// history-enabled synth app create log entries in the _log hypertable. +// This is a full end-to-end test against a real PostgreSQL database. +func TestSynth_LogEntries_Integration(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "logtest") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + // Create a synth app with history + fsErr := ops.WriteFile(ctx, "/.build/logtest", []byte("markdown,history\n")) + require.Nil(t, fsErr, "build logtest app: %v", fsErr) + + // Wait briefly for DDL to complete + time.Sleep(100 * time.Millisecond) + + // INSERT: create a file + content1 := "---\ntitle: First Post\n---\n# First Post\n\nHello world.\n" + fsErr = ops.WriteFile(ctx, "/logtest/first-post.md", []byte(content1)) + require.Nil(t, fsErr, "create first-post.md: %v", fsErr) + + // UPDATE: edit the file + content2 := "---\ntitle: First Post Updated\n---\n# First Post\n\nUpdated content.\n" + fsErr = ops.WriteFile(ctx, "/logtest/first-post.md", []byte(content2)) + require.Nil(t, fsErr, "update first-post.md: %v", fsErr) + + // INSERT: create a second file + content3 := "---\ntitle: Second Post\n---\n# Second Post\n" + fsErr = ops.WriteFile(ctx, "/logtest/second-post.md", []byte(content3)) + require.Nil(t, fsErr, "create second-post.md: %v", fsErr) + + // DELETE: delete the second file + fsErr = ops.Delete(ctx, "/logtest/second-post.md") + require.Nil(t, fsErr, "delete second-post.md: %v", fsErr) + + // RENAME: rename the first file + fsErr = ops.Rename(ctx, "/logtest/first-post.md", "/logtest/hello.md") + require.Nil(t, fsErr, "rename first-post.md to hello.md: %v", fsErr) + + // Query the log table directly to verify entries + pool, err := pgxpool.New(ctx, result.ConnStr) + require.NoError(t, err) + defer pool.Close() + + rows, err := pool.Query(ctx, `SELECT type, filename, + file_id IS NOT NULL AS has_file_id, + version_id IS NOT NULL AS has_version_id + FROM tigerfs.logtest_log ORDER BY log_id ASC`) + require.NoError(t, err) + defer rows.Close() + + type logRow struct { + opType string + filename string + hasFileID bool + hasVersionID bool + } + var entries []logRow + for rows.Next() { + var r logRow + err := rows.Scan(&r.opType, &r.filename, &r.hasFileID, &r.hasVersionID) + require.NoError(t, err) + entries = append(entries, r) + } + + // Should have 5 log entries: create, edit, create, delete, rename + require.Len(t, entries, 5, "expected 5 log entries, got %d: %+v", len(entries), entries) + + // Entry 0: CREATE first-post.md + assert.Equal(t, "create", entries[0].opType) + assert.Equal(t, "first-post.md", entries[0].filename) + assert.True(t, entries[0].hasFileID, "create should have file_id") + assert.False(t, entries[0].hasVersionID, "create should NOT have version_id") + + // Entry 1: EDIT first-post.md + assert.Equal(t, "edit", entries[1].opType) + assert.Equal(t, "first-post.md", entries[1].filename) + assert.True(t, entries[1].hasFileID) + assert.True(t, entries[1].hasVersionID, "edit should have version_id") + + // Entry 2: CREATE second-post.md + assert.Equal(t, "create", entries[2].opType) + assert.Equal(t, "second-post.md", entries[2].filename) + + // Entry 3: DELETE second-post.md + assert.Equal(t, "delete", entries[3].opType) + assert.Equal(t, "second-post.md", entries[3].filename) + assert.True(t, entries[3].hasVersionID, "delete should have version_id") + + // Entry 4: RENAME with NEW filename + assert.Equal(t, "rename", entries[4].opType) + assert.Equal(t, "hello.md", entries[4].filename, "rename should log the new filename") + assert.True(t, entries[4].hasVersionID, "rename should have version_id") + + // Verify the log table is a hypertable (has chunks) + var isHypertable bool + err = pool.QueryRow(ctx, + `SELECT EXISTS(SELECT 1 FROM timescaledb_information.hypertables WHERE hypertable_name = 'logtest_log')`, + ).Scan(&isHypertable) + require.NoError(t, err) + assert.True(t, isHypertable, "logtest_log should be a hypertable") + + // Verify the savepoint table exists + var savepointExists bool + err = pool.QueryRow(ctx, + `SELECT EXISTS(SELECT 1 FROM information_schema.tables WHERE table_schema = 'tigerfs' AND table_name = 'logtest_savepoint')`, + ).Scan(&savepointExists) + require.NoError(t, err) + assert.True(t, savepointExists, "logtest_savepoint table should exist") + + // Verify user_id is NULL (identity not yet wired) + var userIDNull bool + err = pool.QueryRow(ctx, + `SELECT user_id IS NULL FROM tigerfs.logtest_log LIMIT 1`, + ).Scan(&userIDNull) + require.NoError(t, err) + assert.True(t, userIDNull, "user_id should be NULL until identity is wired") + + t.Logf("Log entries verified: %d entries for create/edit/create/delete/rename", len(entries)) + for i, e := range entries { + t.Logf(" [%d] %s %s (file_id=%v, version_id=%v)", i, e.opType, e.filename, e.hasFileID, e.hasVersionID) + } +} + +// TestSynth_LogEntries_NestedFiles verifies that log entries for files in +// subdirectories store the denormalized full path (ADR-017 Section 13.8). +// The log's filename column has different semantics from the source table's +// filename (leaf only) -- the log stores the full path for human-readable display. +func TestSynth_LogEntries_NestedFiles(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "logdir") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/logdir", []byte("markdown,history\n")) + require.Nil(t, fsErr, "build logdir app: %v", fsErr) + time.Sleep(100 * time.Millisecond) + + // MKDIRS: each gets its own 'create' log entry now that mkdirSynth + // logs (mirrors per-segment kernel mkdir(2) calls in production). + require.Nil(t, ops.Mkdir(ctx, "/logdir/projects"), "mkdir projects") + require.Nil(t, ops.Mkdir(ctx, "/logdir/projects/web"), "mkdir projects/web") + + // CREATE: nested file (parents already exist). + content := "---\ntitle: Todo\n---\n# Todo\n" + fsErr = ops.WriteFile(ctx, "/logdir/projects/web/todo.md", []byte(content)) + require.Nil(t, fsErr, "create nested file") + + // EDIT: update the nested file + content2 := "---\ntitle: Todo Updated\n---\n# Todo Updated\n" + fsErr = ops.WriteFile(ctx, "/logdir/projects/web/todo.md", []byte(content2)) + require.Nil(t, fsErr, "edit nested file") + + // RENAME: rename within same directory + fsErr = ops.Rename(ctx, "/logdir/projects/web/todo.md", "/logdir/projects/web/done.md") + require.Nil(t, fsErr, "rename nested file") + + // MOVE: move to different directory + fsErr = ops.Mkdir(ctx, "/logdir/archive") + require.Nil(t, fsErr, "mkdir archive") + fsErr = ops.Rename(ctx, "/logdir/projects/web/done.md", "/logdir/archive/done.md") + require.Nil(t, fsErr, "move file to archive") + + // DELETE: delete the moved file + fsErr = ops.Delete(ctx, "/logdir/archive/done.md") + require.Nil(t, fsErr, "delete file") + + // Query log entries + pool, err := pgxpool.New(ctx, result.ConnStr) + require.NoError(t, err) + defer pool.Close() + + rows, err := pool.Query(ctx, `SELECT type, filename FROM tigerfs.logdir_log ORDER BY log_id ASC`) + require.NoError(t, err) + defer rows.Close() + + type logRow struct { + opType string + filename string + } + var entries []logRow + for rows.Next() { + var r logRow + require.NoError(t, rows.Scan(&r.opType, &r.filename)) + entries = append(entries, r) + } + + // 8 log entries total: 3 mkdirs (projects, projects/web, archive) + // + 5 file ops (create, edit, rename, move-rename, delete). + require.Len(t, entries, 8, "expected 8 log entries, got %d: %+v", len(entries), entries) + + // MKDIR projects (dir, denormalized full path). + assert.Equal(t, "create", entries[0].opType) + assert.Equal(t, "projects", entries[0].filename, "mkdir log full path") + + // MKDIR projects/web. + assert.Equal(t, "create", entries[1].opType) + assert.Equal(t, "projects/web", entries[1].filename, "mkdir log full path") + + // CREATE file: full path of nested file. + assert.Equal(t, "create", entries[2].opType) + assert.Equal(t, "projects/web/todo.md", entries[2].filename, + "create log should store denormalized full path") + + // EDIT: same full path. + assert.Equal(t, "edit", entries[3].opType) + assert.Equal(t, "projects/web/todo.md", entries[3].filename, + "edit log should store denormalized full path") + + // RENAME: new filename in same directory. + assert.Equal(t, "rename", entries[4].opType) + assert.Equal(t, "projects/web/done.md", entries[4].filename, + "rename log should store new full path") + + // MKDIR archive (between rename and move). + assert.Equal(t, "create", entries[5].opType) + assert.Equal(t, "archive", entries[5].filename, "mkdir archive log full path") + + // MOVE: new full path in different directory. + assert.Equal(t, "rename", entries[6].opType) + assert.Equal(t, "archive/done.md", entries[6].filename, + "move log should store new full path in target directory") + + // DELETE: full path at time of deletion. + assert.Equal(t, "delete", entries[7].opType) + assert.Equal(t, "archive/done.md", entries[7].filename, + "delete log should store full path at time of deletion") + + t.Logf("Nested log entries verified:") + for i, e := range entries { + t.Logf(" [%d] %s %s", i, e.opType, e.filename) + } +} + +// TestSynth_LogEntries_DirRenameOneEntry verifies that renaming a directory +// with children produces exactly ONE log entry (ADR-017 verification #44). +// This is the key improvement over the old prefix-based model where directory +// rename was an N-row UPDATE producing N log entries. +func TestSynth_LogEntries_DirRenameOneEntry(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "logdirren") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/logdirren", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create directory with multiple files (mkdir-p the parent on first + // write, then siblings just need WriteFile). + fsErr = ops.WriteFileEnsureDirs(ctx, "/logdirren/mydir/a.md", []byte("---\ntitle: A\n---\nA\n")) + require.Nil(t, fsErr) + fsErr = ops.WriteFile(ctx, "/logdirren/mydir/b.md", []byte("---\ntitle: B\n---\nB\n")) + require.Nil(t, fsErr) + fsErr = ops.WriteFile(ctx, "/logdirren/mydir/c.md", []byte("---\ntitle: C\n---\nC\n")) + require.Nil(t, fsErr) + + // Count log entries before rename + pool, err := pgxpool.New(ctx, result.ConnStr) + require.NoError(t, err) + defer pool.Close() + + var countBefore int + err = pool.QueryRow(ctx, `SELECT count(*) FROM tigerfs.logdirren_log`).Scan(&countBefore) + require.NoError(t, err) + + // Rename the directory (contains 3 files) + fsErr = ops.Rename(ctx, "/logdirren/mydir", "/logdirren/renamed") + require.Nil(t, fsErr, "rename directory should succeed") + + // Count log entries after rename + var countAfter int + err = pool.QueryRow(ctx, `SELECT count(*) FROM tigerfs.logdirren_log`).Scan(&countAfter) + require.NoError(t, err) + + // Should produce exactly ONE new log entry (single-row rename, not N entries) + assert.Equal(t, 1, countAfter-countBefore, + "directory rename should produce exactly 1 log entry (ADR-017: single-row operation)") + + // Verify the entry is a rename with the new directory path + var opType, filename string + err = pool.QueryRow(ctx, + `SELECT type, filename FROM tigerfs.logdirren_log ORDER BY log_id DESC LIMIT 1`, + ).Scan(&opType, &filename) + require.NoError(t, err) + assert.Equal(t, "rename", opType) + assert.Equal(t, "renamed", filename, "should log new directory name") +} + +// TestSynth_LogInterface_ReadDir verifies that .log/ appears in synth app +// ReadDir listings and that the .log/ pipeline works end-to-end. +func TestSynth_LogInterface_ReadDir(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "logui") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + // Create app with history + fsErr := ops.WriteFile(ctx, "/.build/logui", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // .log/ should appear in the app ReadDir + entries, fsErr := ops.ReadDir(ctx, "/logui") + require.Nil(t, fsErr) + names := fsEntryNames(entries) + assert.Contains(t, names, ".log", "synth app should show .log/") + assert.Contains(t, names, ".savepoint", "synth app should show .savepoint/") + assert.Contains(t, names, ".undo", "synth app should show .undo/") + assert.Contains(t, names, ".history", "synth app should show .history/") + + // Create some files to generate log entries + fsErr = ops.WriteFile(ctx, "/logui/hello.md", []byte("---\ntitle: Hello\n---\n# Hello\n")) + require.Nil(t, fsErr) + + time.Sleep(50 * time.Millisecond) + + fsErr = ops.WriteFile(ctx, "/logui/hello.md", []byte("---\ntitle: Updated\n---\n# Updated\n")) + require.Nil(t, fsErr) + + // .log/ listing should have entries + logEntries, fsErr := ops.ReadDir(ctx, "/logui/.log") + require.Nil(t, fsErr, "ReadDir .log/ should succeed") + assert.GreaterOrEqual(t, len(logEntries), 2, "should have at least 2 log entries (create + edit)") + + // Log entries should be directories (rows-as-directories) + for _, e := range logEntries { + assert.True(t, e.IsDir, "log entries should be directories, got: %s", e.Name) + } + + // Find an actual log entry (skip capability directories like .all, .by, etc.) + var entryName string + for _, e := range logEntries { + if !strings.HasPrefix(e.Name, ".") { + entryName = e.Name + break + } + } + require.NotEmpty(t, entryName, "should find a non-capability log entry") + + entryEntries, fsErr := ops.ReadDir(ctx, "/logui/.log/"+entryName) + require.Nil(t, fsErr, "ReadDir .log/ should succeed") + + entryNames := fsEntryNames(entryEntries) + // Should have column files + diff symlinks + assert.Contains(t, entryNames, "before", "log entry should have 'before' symlink") + assert.Contains(t, entryNames, "after", "log entry should have 'after' symlink") + assert.Contains(t, entryNames, "current", "log entry should have 'current' symlink") + + // Read a column value + typeContent, fsErr := ops.ReadFile(ctx, "/logui/.log/"+entryName+"/type") + require.Nil(t, fsErr, "ReadFile .log//type should succeed") + typeStr := strings.TrimSpace(string(typeContent.Data)) + assert.Contains(t, []string{"create", "edit", "rename", "delete", "undo"}, typeStr, + "type column should be a valid operation type") +} + +// TestSynth_LogInterface_DiffSymlinks verifies the before/after/current +// diff symlinks on log entries resolve correctly. +func TestSynth_LogInterface_DiffSymlinks(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "logdiff") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/logdiff", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create a file (log: type=create, version_id=NULL) + fsErr = ops.WriteFile(ctx, "/logdiff/hello.md", []byte("---\ntitle: V1\n---\n# V1\n")) + require.Nil(t, fsErr) + + time.Sleep(1100 * time.Millisecond) // distinct UUIDv7 + + // Edit the file (log: type=edit, version_id=non-NULL) + fsErr = ops.WriteFile(ctx, "/logdiff/hello.md", []byte("---\ntitle: V2\n---\n# V2\n")) + require.Nil(t, fsErr) + + // Get log entries + logEntries, fsErr := ops.ReadDir(ctx, "/logdiff/.log") + require.Nil(t, fsErr) + require.GreaterOrEqual(t, len(logEntries), 2) + + // Find create and edit entries (oldest first since ReadDir returns by PK order) + // Sort to find create vs edit + var createEntry, editEntry string + for _, e := range logEntries { + typeContent, err := ops.ReadFile(ctx, "/logdiff/.log/"+e.Name+"/type") + if err != nil { + continue + } + tp := strings.TrimSpace(string(typeContent.Data)) + switch tp { + case "create": + createEntry = e.Name + case "edit": + editEntry = e.Name + } + } + require.NotEmpty(t, createEntry, "should find a create log entry") + require.NotEmpty(t, editEntry, "should find an edit log entry") + + // Create entry: before should be /dev/null (no before-state) + beforeTarget, fsErr := ops.Readlink(ctx, "/logdiff/.log/"+createEntry+"/before") + require.Nil(t, fsErr, "Readlink before on create should succeed") + assert.Equal(t, "/dev/null", beforeTarget, "create's before should be /dev/null") + + // Edit entry: before should point to .history/ (has version_id) + beforeTarget, fsErr = ops.Readlink(ctx, "/logdiff/.log/"+editEntry+"/before") + require.Nil(t, fsErr, "Readlink before on edit should succeed") + assert.Contains(t, beforeTarget, ".history/", "edit's before should point to .history/") + assert.Contains(t, beforeTarget, "hello.md", "edit's before should reference hello.md") + + // Current on edit: file still exists + currentTarget, fsErr := ops.Readlink(ctx, "/logdiff/.log/"+editEntry+"/current") + require.Nil(t, fsErr, "Readlink current should succeed") + assert.Contains(t, currentTarget, "hello.md", "current should point to live file") + assert.NotEqual(t, "/dev/null", currentTarget, "current should not be /dev/null for existing file") + + // Delete the file to test /dev/null current + fsErr = ops.Delete(ctx, "/logdiff/hello.md") + require.Nil(t, fsErr) + + // Find the delete log entry + logEntries, fsErr = ops.ReadDir(ctx, "/logdiff/.log") + require.Nil(t, fsErr) + var deleteEntry string + for _, e := range logEntries { + typeContent, err := ops.ReadFile(ctx, "/logdiff/.log/"+e.Name+"/type") + if err != nil { + continue + } + if strings.TrimSpace(string(typeContent.Data)) == "delete" { + deleteEntry = e.Name + } + } + require.NotEmpty(t, deleteEntry, "should find a delete log entry") + + // Delete entry: current should be /dev/null (file deleted) + currentTarget, fsErr = ops.Readlink(ctx, "/logdiff/.log/"+deleteEntry+"/current") + require.Nil(t, fsErr, "Readlink current on delete should succeed") + assert.Equal(t, "/dev/null", currentTarget, "current should be /dev/null for deleted file") + + // Delete entry: after should be /dev/null (nothing after delete) + afterTarget, fsErr := ops.Readlink(ctx, "/logdiff/.log/"+deleteEntry+"/after") + require.Nil(t, fsErr, "Readlink after on delete should succeed") + assert.Equal(t, "/dev/null", afterTarget, "delete's after should be /dev/null") +} + +// TestSynth_LogInterface_AfterChain verifies the "after" symlink points to +// .history/ (not current file) when there's a subsequent edit entry. +func TestSynth_LogInterface_AfterChain(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "logchain") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/logchain", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create → Edit1 → Edit2 (three log entries) + fsErr = ops.WriteFile(ctx, "/logchain/hello.md", []byte("---\ntitle: V1\n---\nV1\n")) + require.Nil(t, fsErr) + time.Sleep(1100 * time.Millisecond) + fsErr = ops.WriteFile(ctx, "/logchain/hello.md", []byte("---\ntitle: V2\n---\nV2\n")) + require.Nil(t, fsErr) + time.Sleep(1100 * time.Millisecond) + fsErr = ops.WriteFile(ctx, "/logchain/hello.md", []byte("---\ntitle: V3\n---\nV3\n")) + require.Nil(t, fsErr) + + // Find the first edit entry (middle of the chain) + logEntries, fsErr := ops.ReadDir(ctx, "/logchain/.log") + require.Nil(t, fsErr) + + var editEntries []string + for _, e := range logEntries { + if strings.HasPrefix(e.Name, ".") { + continue + } + typeContent, err := ops.ReadFile(ctx, "/logchain/.log/"+e.Name+"/type") + if err != nil { + continue + } + if strings.TrimSpace(string(typeContent.Data)) == "edit" { + editEntries = append(editEntries, e.Name) + } + } + require.GreaterOrEqual(t, len(editEntries), 2, "should have at least 2 edit entries") + + // First edit's "after" should point to .history/ (because there's a subsequent edit) + afterTarget, fsErr := ops.Readlink(ctx, "/logchain/.log/"+editEntries[0]+"/after") + require.Nil(t, fsErr, "Readlink after on first edit should succeed") + assert.Contains(t, afterTarget, ".history/", "first edit's after should point to .history/ (next edit's before-state)") +} + +// TestSynth_LogInterface_Pipeline verifies that pipeline operations work on .log/. +func TestSynth_LogInterface_Pipeline(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "logpipe") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/logpipe", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create 5 files to generate 5 log entries + for i := 1; i <= 5; i++ { + content := fmt.Sprintf("---\ntitle: File %d\n---\nContent %d\n", i, i) + fsErr = ops.WriteFile(ctx, fmt.Sprintf("/logpipe/file%d.md", i), []byte(content)) + require.Nil(t, fsErr) + } + + // .last/2 should return exactly 2 entries (plus capability dirs) + entries, fsErr := ops.ReadDir(ctx, "/logpipe/.log/.last/2") + require.Nil(t, fsErr, "ReadDir .log/.last/2 should succeed") + + var rowEntries []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + rowEntries = append(rowEntries, e.Name) + } + } + assert.Equal(t, 2, len(rowEntries), ".last/2 should return exactly 2 log entries") +} + +// TestSynth_LogInterface_NestedFilename verifies that log entries for nested files +// store the denormalized full path in the filename column. +func TestSynth_LogInterface_NestedFilename(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "logpath") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/logpath", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create the parent dir then the nested file. Two log entries result + // (one for each Mkdir, one for the file create); we want to assert + // against the FILE create's denormalized filename specifically. + require.Nil(t, ops.Mkdir(ctx, "/logpath/docs"), "mkdir docs") + fsErr = ops.WriteFile(ctx, "/logpath/docs/guide.md", []byte("---\ntitle: Guide\n---\n# Guide\n")) + require.Nil(t, fsErr) + + // Find the most recent log entry (the file create). + logEntries, fsErr := ops.ReadDir(ctx, "/logpath/.log/.last/1") + require.Nil(t, fsErr) + var entryName string + for _, e := range logEntries { + if !strings.HasPrefix(e.Name, ".") { + entryName = e.Name + break + } + } + require.NotEmpty(t, entryName) + + // Read the filename column -- should be full denormalized path. + fnContent, fsErr := ops.ReadFile(ctx, "/logpath/.log/.last/1/"+entryName+"/filename") + require.Nil(t, fsErr, "ReadFile .log/.last/1//filename should succeed") + assert.Equal(t, "docs/guide.md", strings.TrimSpace(string(fnContent.Data)), + "log filename should be denormalized full path") +} + +// TestSynth_LogInterface_UserIdentity verifies that log entries include the user_id +// when set via .info/user. +func TestSynth_LogInterface_UserIdentity(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "loguser") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/loguser", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Set user identity + fsErr = ops.WriteFile(ctx, "/.info/user", []byte("agent-7\n")) + require.Nil(t, fsErr) + + // Create a file + fsErr = ops.WriteFile(ctx, "/loguser/hello.md", []byte("---\ntitle: Hello\n---\n# Hello\n")) + require.Nil(t, fsErr) + + // Find the log entry + logEntries, fsErr := ops.ReadDir(ctx, "/loguser/.log") + require.Nil(t, fsErr) + var entryName string + for _, e := range logEntries { + if !strings.HasPrefix(e.Name, ".") { + entryName = e.Name + break + } + } + require.NotEmpty(t, entryName) + + // Read user_id column + userContent, fsErr := ops.ReadFile(ctx, "/loguser/.log/"+entryName+"/user_id") + require.Nil(t, fsErr, "ReadFile .log//user_id should succeed") + assert.Equal(t, "agent-7", strings.TrimSpace(string(userContent.Data)), + "log entry should include the user identity") +} + +// cleanupTigerFSTablesWithLog extends cleanup to also drop _log and _savepoint tables. +func cleanupLogTables(t *testing.T, connStr string, tableNames ...string) { + t.Helper() + t.Cleanup(func() { + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + pool, err := pgxpool.New(ctx, connStr) + if err != nil { + return + } + defer pool.Close() + for _, name := range tableNames { + pool.Exec(ctx, fmt.Sprintf("DROP TABLE IF EXISTS tigerfs.%s_log CASCADE", name)) + pool.Exec(ctx, fmt.Sprintf("DROP TABLE IF EXISTS tigerfs.%s_savepoint CASCADE", name)) + } + }) +} diff --git a/test/integration/migrate_test.go b/test/integration/migrate_test.go index 56ff752..f99aa28 100644 --- a/test/integration/migrate_test.go +++ b/test/integration/migrate_test.go @@ -303,3 +303,456 @@ func TestSynth_MigrateWithHistory(t *testing.T) { cp.Exec(cleanupCtx, `DROP TABLE IF EXISTS tigerfs."mig_hist_history" CASCADE`) }) } + +// TestSynth_MigrateAddParentPointer tests the relational-directories migration end-to-end: +// 1. Creates old-schema tables (no parent_id) in tigerfs schema with hierarchical data +// 2. Creates old-schema history and log tables with old column names +// 3. Runs migration, verifies detection/dry-run/execution +// 4. Verifies DB state: parent_id chain, leaf filenames, column renames, type renames +// 5. Verifies TigerFS operations work on migrated data (ReadDir, Stat, ReadFile, Write) +// 6. Verifies idempotency +func TestSynth_MigrateAddParentPointer(t *testing.T) { + require.NoError(t, config.Init(), "config.Init should succeed") + + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + + ctx := context.Background() + pool, err := pgxpool.New(ctx, result.ConnStr) + require.NoError(t, err) + defer pool.Close() + + var schema string + err = pool.QueryRow(ctx, "SELECT current_schema()").Scan(&schema) + require.NoError(t, err) + + // Enable TimescaleDB + _, _ = pool.Exec(ctx, "CREATE EXTENSION IF NOT EXISTS timescaledb") + + // --- Setup: Create OLD-schema synth app in tigerfs schema --- + // This simulates a database created before ADR-017 that has already been + // through the move-backing-tables migration (tables in tigerfs schema). + oldSQL := []string{ + `CREATE SCHEMA IF NOT EXISTS tigerfs`, + + // Source table: old schema (no parent_id, UNIQUE(filename, filetype)) + `CREATE TABLE tigerfs."mig_pp" ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + filename TEXT NOT NULL, + filetype TEXT NOT NULL DEFAULT 'file' CHECK (filetype IN ('file', 'directory')), + title TEXT, + author TEXT, + headers JSONB DEFAULT '{}'::jsonb, + body TEXT, + encoding TEXT NOT NULL DEFAULT 'utf8' CHECK (encoding IN ('utf8', 'base64')), + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + modified_at TIMESTAMPTZ NOT NULL DEFAULT now(), + UNIQUE(filename, filetype) + )`, + + // View in user schema + fmt.Sprintf(`CREATE VIEW %q."mig_pp" AS SELECT * FROM tigerfs."mig_pp"`, schema), + fmt.Sprintf(`COMMENT ON VIEW %q."mig_pp" IS 'tigerfs:md,history'`, schema), + + // Insert hierarchical data with old full-path filenames. + // File rows include extension (TigerFS stores "readme.md" not "readme"). + `INSERT INTO tigerfs."mig_pp" (filename, filetype) VALUES ('docs', 'directory')`, + `INSERT INTO tigerfs."mig_pp" (filename, filetype) VALUES ('docs/guides', 'directory')`, + `INSERT INTO tigerfs."mig_pp" (filename, filetype, title, body) VALUES ('readme.md', 'file', 'Root Readme', '# Root')`, + `INSERT INTO tigerfs."mig_pp" (filename, filetype, title, body) VALUES ('docs/install.md', 'file', 'Install Guide', '# Install')`, + `INSERT INTO tigerfs."mig_pp" (filename, filetype, title, body) VALUES ('docs/guides/quickstart.md', 'file', 'Quickstart', '# Quick')`, + + // History table: old column names (id, _history_id, _operation) + `CREATE TABLE tigerfs."mig_pp_history" ( + id UUID, + filename TEXT NOT NULL, + filetype TEXT, + title TEXT, + author TEXT, + headers JSONB, + body TEXT, + encoding TEXT, + created_at TIMESTAMPTZ, + modified_at TIMESTAMPTZ, + _history_id UUID NOT NULL DEFAULT gen_random_uuid() PRIMARY KEY, + _operation TEXT NOT NULL + )`, + + // Insert some history entries (simulating past edits) + `INSERT INTO tigerfs."mig_pp_history" (id, filename, filetype, title, body, _operation) + SELECT id, filename, filetype, 'Old Title', '# Old', 'UPDATE' + FROM tigerfs."mig_pp" WHERE filename = 'readme.md' AND filetype = 'file'`, + `INSERT INTO tigerfs."mig_pp_history" (id, filename, filetype, title, body, _operation) + SELECT id, filename, filetype, 'Old Install', '# Old Install', 'UPDATE' + FROM tigerfs."mig_pp" WHERE filename = 'docs/install.md' AND filetype = 'file'`, + + // Log table: old column names (history_id) and old type values (insert, update) + fmt.Sprintf(`CREATE TABLE tigerfs."mig_pp_log" ( + log_id UUID NOT NULL DEFAULT gen_random_uuid() PRIMARY KEY, + user_id TEXT, + type TEXT NOT NULL CHECK (type IN ('insert', 'update', 'delete', 'undo')), + file_id UUID NOT NULL, + filename TEXT NOT NULL, + history_id UUID, + description TEXT + )`), + + // Insert log entries with old type names + `INSERT INTO tigerfs."mig_pp_log" (file_id, type, filename) + SELECT id, 'insert', filename FROM tigerfs."mig_pp" WHERE filetype = 'file'`, + } + for _, sql := range oldSQL { + _, err := pool.Exec(ctx, sql) + require.NoError(t, err, "setup SQL failed: %s", sql) + } + + // --- Step 1: Detect --- + describeCmd := cmd.BuildMigrateCmd() + var describeBuf bytes.Buffer + describeCmd.SetOut(&describeBuf) + describeCmd.SetErr(&describeBuf) + describeCmd.SetArgs([]string{result.ConnStr, "--describe", "--insecure-no-ssl"}) + err = describeCmd.Execute() + require.NoError(t, err, "migrate --describe should succeed") + assert.Contains(t, describeBuf.String(), "relational-directories") + assert.Contains(t, describeBuf.String(), "mig_pp") + + // --- Step 2: Dry-run --- + dryRunCmd := cmd.BuildMigrateCmd() + var dryRunBuf bytes.Buffer + dryRunCmd.SetOut(&dryRunBuf) + dryRunCmd.SetErr(&dryRunBuf) + dryRunCmd.SetArgs([]string{result.ConnStr, "--dry-run", "--insecure-no-ssl"}) + err = dryRunCmd.Execute() + require.NoError(t, err, "migrate --dry-run should succeed") + dryRunOutput := dryRunBuf.String() + assert.Contains(t, dryRunOutput, "parent_id", "dry-run should mention parent_id") + assert.Contains(t, dryRunOutput, "resolve_path", "dry-run should create resolve_path") + + // --- Step 3: Execute --- + execCmd := cmd.BuildMigrateCmd() + var execBuf bytes.Buffer + execCmd.SetOut(&execBuf) + execCmd.SetErr(&execBuf) + execCmd.SetArgs([]string{result.ConnStr, "--insecure-no-ssl"}) + err = execCmd.Execute() + require.NoError(t, err, "migrate should succeed: %s", execBuf.String()) + + // --- Step 4: Verify source table --- + + // parent_id column should exist + var hasParentID bool + err = pool.QueryRow(ctx, `SELECT EXISTS( + SELECT 1 FROM information_schema.columns + WHERE table_schema='tigerfs' AND table_name='mig_pp' AND column_name='parent_id' + )`).Scan(&hasParentID) + require.NoError(t, err) + assert.True(t, hasParentID, "source table should have parent_id column") + + // Filenames should be leaf names only + var filenames []string + rows, err := pool.Query(ctx, `SELECT filename FROM tigerfs."mig_pp" ORDER BY filename`) + require.NoError(t, err) + for rows.Next() { + var fn string + require.NoError(t, rows.Scan(&fn)) + filenames = append(filenames, fn) + } + rows.Close() + // Should be leaf names: docs, guides, install.md, quickstart.md, readme.md (alphabetical) + assert.Equal(t, []string{"docs", "guides", "install.md", "quickstart.md", "readme.md"}, filenames, + "filenames should be leaf names only (no slashes)") + + // Verify parent_id chain: docs is root, guides is child of docs + var docsID, guidesParentID string + err = pool.QueryRow(ctx, `SELECT id::text FROM tigerfs."mig_pp" WHERE filename='docs' AND filetype='directory'`).Scan(&docsID) + require.NoError(t, err) + err = pool.QueryRow(ctx, `SELECT parent_id::text FROM tigerfs."mig_pp" WHERE filename='guides' AND filetype='directory'`).Scan(&guidesParentID) + require.NoError(t, err) + assert.Equal(t, docsID, guidesParentID, "guides should be child of docs") + + // install.md should be child of docs + var installParentID string + err = pool.QueryRow(ctx, `SELECT parent_id::text FROM tigerfs."mig_pp" WHERE filename='install.md' AND filetype='file'`).Scan(&installParentID) + require.NoError(t, err) + assert.Equal(t, docsID, installParentID, "install.md should be child of docs") + + // readme.md should be at root (parent_id IS NULL) + var readmeParentNull bool + err = pool.QueryRow(ctx, `SELECT parent_id IS NULL FROM tigerfs."mig_pp" WHERE filename='readme.md' AND filetype='file'`).Scan(&readmeParentNull) + require.NoError(t, err) + assert.True(t, readmeParentNull, "readme should have NULL parent_id (root level)") + + // --- Step 5: Verify history table --- + var hasFileID, hasVersionID, hasOperation bool + err = pool.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM information_schema.columns WHERE table_schema='tigerfs' AND table_name='mig_pp_history' AND column_name='file_id')`).Scan(&hasFileID) + require.NoError(t, err) + assert.True(t, hasFileID, "history should have file_id (renamed from id)") + + err = pool.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM information_schema.columns WHERE table_schema='tigerfs' AND table_name='mig_pp_history' AND column_name='version_id')`).Scan(&hasVersionID) + require.NoError(t, err) + assert.True(t, hasVersionID, "history should have version_id (renamed from _history_id)") + + err = pool.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM information_schema.columns WHERE table_schema='tigerfs' AND table_name='mig_pp_history' AND column_name='operation')`).Scan(&hasOperation) + require.NoError(t, err) + assert.True(t, hasOperation, "history should have operation (renamed from _operation)") + + // History filenames should be leaf names + var histFilename string + err = pool.QueryRow(ctx, `SELECT filename FROM tigerfs."mig_pp_history" LIMIT 1`).Scan(&histFilename) + require.NoError(t, err) + assert.False(t, strings.Contains(histFilename, "/"), "history filenames should be leaf names, got: %s", histFilename) + + // --- Step 6: Verify log table --- + var hasLogVersionID bool + err = pool.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM information_schema.columns WHERE table_schema='tigerfs' AND table_name='mig_pp_log' AND column_name='version_id')`).Scan(&hasLogVersionID) + require.NoError(t, err) + assert.True(t, hasLogVersionID, "log should have version_id (renamed from history_id)") + + // Log type values should be updated + var logTypes []string + logRows, err := pool.Query(ctx, `SELECT DISTINCT type FROM tigerfs."mig_pp_log" ORDER BY type`) + require.NoError(t, err) + for logRows.Next() { + var tp string + require.NoError(t, logRows.Scan(&tp)) + logTypes = append(logTypes, tp) + } + logRows.Close() + assert.Equal(t, []string{"create"}, logTypes, "old 'insert' types should be renamed to 'create'") + + // --- Step 7: Verify TigerFS operations work on migrated data --- + ops := setupFSOperations(t, result.ConnStr) + + // ReadDir root should show readme.md and docs/ + entries, fsErr := ops.ReadDir(ctx, "/mig_pp") + require.Nil(t, fsErr, "ReadDir root should succeed") + names := fsEntryNames(entries) + assert.Contains(t, names, "readme.md", "root should show readme.md") + assert.Contains(t, names, "docs", "root should show docs/") + assert.NotContains(t, names, "install.md", "root should NOT show nested files") + + // ReadDir docs/ should show install.md and guides/ + entries, fsErr = ops.ReadDir(ctx, "/mig_pp/docs") + require.Nil(t, fsErr, "ReadDir docs should succeed") + names = fsEntryNames(entries) + assert.Contains(t, names, "install.md") + assert.Contains(t, names, "guides") + + // ReadFile should return content + content, fsErr := ops.ReadFile(ctx, "/mig_pp/docs/guides/quickstart.md") + require.Nil(t, fsErr, "ReadFile nested file should succeed") + assert.Contains(t, string(content.Data), "# Quick") + + // Write a new file in migrated app + newContent := "---\ntitle: New File\n---\n# New\n" + fsErr = ops.WriteFile(ctx, "/mig_pp/docs/new-file.md", []byte(newContent)) + require.Nil(t, fsErr, "WriteFile in migrated app should succeed") + + // Verify new file is accessible + entries, fsErr = ops.ReadDir(ctx, "/mig_pp/docs") + require.Nil(t, fsErr) + assert.Contains(t, fsEntryNames(entries), "new-file.md") + + // --- Step 8: Idempotency --- + idempCmd := cmd.BuildMigrateCmd() + var idempBuf bytes.Buffer + idempCmd.SetOut(&idempBuf) + idempCmd.SetErr(&idempBuf) + idempCmd.SetArgs([]string{result.ConnStr, "--describe", "--insecure-no-ssl"}) + err = idempCmd.Execute() + require.NoError(t, err) + assert.Contains(t, idempBuf.String(), "No pending migrations", "should be idempotent") + + // Cleanup + t.Cleanup(func() { + cleanupCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + cp, err := pgxpool.New(cleanupCtx, result.ConnStr) + if err != nil { + return + } + defer cp.Close() + cp.Exec(cleanupCtx, `DROP TABLE IF EXISTS tigerfs."mig_pp" CASCADE`) + cp.Exec(cleanupCtx, `DROP TABLE IF EXISTS tigerfs."mig_pp_history" CASCADE`) + cp.Exec(cleanupCtx, `DROP TABLE IF EXISTS tigerfs."mig_pp_log" CASCADE`) + cp.Exec(cleanupCtx, `DROP VIEW IF EXISTS "mig_pp" CASCADE`) + }) +} + +// TestSynth_MigrateAddParentDirMtimeTrigger tests migration for workspaces that have +// parent_id but lack the parent directory mtime trigger. +func TestSynth_MigrateAddParentDirMtimeTrigger(t *testing.T) { + require.NoError(t, config.Init(), "config.Init should succeed") + + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + + ctx := context.Background() + + pool, err := pgxpool.New(ctx, result.ConnStr) + require.NoError(t, err) + defer pool.Close() + + var schema string + err = pool.QueryRow(ctx, "SELECT current_schema()").Scan(&schema) + require.NoError(t, err) + + // Setup: Create a workspace with parent_id but WITHOUT the parent mtime trigger. + // This simulates an existing workspace created before this feature. + setupSQL := []string{ + `CREATE SCHEMA IF NOT EXISTS tigerfs`, + `CREATE TABLE tigerfs."mig_mtime" ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + parent_id UUID REFERENCES tigerfs."mig_mtime"(id), + filename TEXT NOT NULL, + filetype TEXT NOT NULL DEFAULT 'file', + title TEXT, + body TEXT, + encoding TEXT NOT NULL DEFAULT 'utf8', + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + modified_at TIMESTAMPTZ NOT NULL DEFAULT now(), + UNIQUE NULLS NOT DISTINCT (parent_id, filename, filetype) + )`, + fmt.Sprintf(`CREATE VIEW %q."mig_mtime" AS SELECT * FROM tigerfs."mig_mtime"`, schema), + fmt.Sprintf(`COMMENT ON VIEW %q."mig_mtime" IS 'tigerfs:md'`, schema), + // Add the modified_at BEFORE trigger (existing workspaces have this) + `CREATE OR REPLACE FUNCTION tigerfs."set_mig_mtime_modified_at"() + RETURNS TRIGGER AS $$ + BEGIN NEW.modified_at = now(); RETURN NEW; END; + $$ LANGUAGE plpgsql`, + `CREATE TRIGGER "trg_mig_mtime_modified_at" + BEFORE UPDATE ON tigerfs."mig_mtime" + FOR EACH ROW EXECUTE FUNCTION tigerfs."set_mig_mtime_modified_at"()`, + } + for _, sql := range setupSQL { + _, err := pool.Exec(ctx, sql) + require.NoError(t, err, "setup SQL failed: %s", sql) + } + + // Step 1: Detect should find this workspace + descCmd := cmd.BuildMigrateCmd() + var descBuf bytes.Buffer + descCmd.SetOut(&descBuf) + descCmd.SetErr(&descBuf) + descCmd.SetArgs([]string{result.ConnStr, "--describe", "--insecure-no-ssl"}) + err = descCmd.Execute() + require.NoError(t, err) + assert.Contains(t, descBuf.String(), "parent-dir-mtime-trigger") + assert.Contains(t, descBuf.String(), "mig_mtime") + + // Step 2: Dry-run should show trigger SQL + dryCmd := cmd.BuildMigrateCmd() + var dryBuf bytes.Buffer + dryCmd.SetOut(&dryBuf) + dryCmd.SetErr(&dryBuf) + dryCmd.SetArgs([]string{result.ConnStr, "--dry-run", "--insecure-no-ssl"}) + err = dryCmd.Execute() + require.NoError(t, err) + assert.Contains(t, dryBuf.String(), "bump_mig_mtime_parent_mtime") + assert.Contains(t, dryBuf.String(), "AFTER INSERT OR DELETE OR UPDATE") + + // Step 3: Execute migration + execCmd := cmd.BuildMigrateCmd() + var execBuf bytes.Buffer + execCmd.SetOut(&execBuf) + execCmd.SetErr(&execBuf) + execCmd.SetArgs([]string{result.ConnStr, "--insecure-no-ssl"}) + err = execCmd.Execute() + require.NoError(t, err) + assert.Contains(t, execBuf.String(), "parent-dir-mtime-trigger") + + // Step 4: Verify trigger exists + var hasTrigger bool + err = pool.QueryRow(ctx, + `SELECT EXISTS( + SELECT 1 FROM pg_trigger t + JOIN pg_class c ON c.oid = t.tgrelid + JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE t.tgname = 'trg_mig_mtime_parent_mtime' + AND c.relname = 'mig_mtime' AND n.nspname = 'tigerfs' + )`).Scan(&hasTrigger) + require.NoError(t, err) + assert.True(t, hasTrigger, "trigger should exist after migration") + + // Step 5: Verify trigger works -- create a dir, then insert a file, check parent mtime + _, err = pool.Exec(ctx, `INSERT INTO tigerfs."mig_mtime" (id, filename, filetype) VALUES (gen_random_uuid(), 'testdir', 'directory')`) + require.NoError(t, err) + + time.Sleep(10 * time.Millisecond) + var dirMtimeBefore time.Time + err = pool.QueryRow(ctx, `SELECT modified_at FROM tigerfs."mig_mtime" WHERE filename = 'testdir' AND filetype = 'directory'`).Scan(&dirMtimeBefore) + require.NoError(t, err) + + var dirID string + err = pool.QueryRow(ctx, `SELECT id FROM tigerfs."mig_mtime" WHERE filename = 'testdir' AND filetype = 'directory'`).Scan(&dirID) + require.NoError(t, err) + + time.Sleep(10 * time.Millisecond) + _, err = pool.Exec(ctx, `INSERT INTO tigerfs."mig_mtime" (id, parent_id, filename, filetype, body) VALUES (gen_random_uuid(), $1, 'test.md', 'file', 'hello')`, dirID) + require.NoError(t, err) + + var dirMtimeAfter time.Time + err = pool.QueryRow(ctx, `SELECT modified_at FROM tigerfs."mig_mtime" WHERE filename = 'testdir' AND filetype = 'directory'`).Scan(&dirMtimeAfter) + require.NoError(t, err) + assert.True(t, dirMtimeAfter.After(dirMtimeBefore), "dir mtime should increase after child insert") + + // Step 6: Idempotency -- second describe should find nothing + idemp := cmd.BuildMigrateCmd() + var idempBuf bytes.Buffer + idemp.SetOut(&idempBuf) + idemp.SetErr(&idempBuf) + idemp.SetArgs([]string{result.ConnStr, "--describe", "--insecure-no-ssl"}) + err = idemp.Execute() + require.NoError(t, err) + assert.Contains(t, idempBuf.String(), "No pending migrations") + + t.Cleanup(func() { + cleanupCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + cp, err := pgxpool.New(cleanupCtx, result.ConnStr) + if err != nil { + return + } + defer cp.Close() + cp.Exec(cleanupCtx, `DROP TABLE IF EXISTS tigerfs."mig_mtime" CASCADE`) + cp.Exec(cleanupCtx, `DROP VIEW IF EXISTS "mig_mtime" CASCADE`) + }) +} + +// TestSynth_MigrateParentDirMtimeTrigger_NotNeeded tests that workspaces +// created with the trigger already present don't show up in migration detect. +func TestSynth_MigrateParentDirMtimeTrigger_NotNeeded(t *testing.T) { + require.NoError(t, config.Init(), "config.Init should succeed") + + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "mig_mtime_new") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + // Create a workspace with the full build path (includes trigger) + fsErr := ops.WriteFile(ctx, "/.build/mig_mtime_new", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Describe should find no pending migrations + descCmd := cmd.BuildMigrateCmd() + var descBuf bytes.Buffer + descCmd.SetOut(&descBuf) + descCmd.SetErr(&descBuf) + descCmd.SetArgs([]string{result.ConnStr, "--describe", "--insecure-no-ssl"}) + err := descCmd.Execute() + require.NoError(t, err) + assert.Contains(t, descBuf.String(), "No pending migrations") +} diff --git a/test/integration/savepoint_test.go b/test/integration/savepoint_test.go new file mode 100644 index 0000000..a000254 --- /dev/null +++ b/test/integration/savepoint_test.go @@ -0,0 +1,610 @@ +package integration + +import ( + "context" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/timescale/tigerfs/internal/tigerfs/config" + "github.com/timescale/tigerfs/internal/tigerfs/db" + "github.com/timescale/tigerfs/internal/tigerfs/fs" +) + +// TestSynth_Savepoint_CRUD verifies create, read, update, and delete of savepoints. +func TestSynth_Savepoint_CRUD(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "sptest") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/sptest", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create savepoint with description (TSV format) + fsErr = ops.WriteFile(ctx, "/sptest/.savepoint/before-exploration.tsv", []byte("description\nStarting exploration\n")) + require.Nil(t, fsErr, "create savepoint with description") + + // Create savepoint via JSON (name only, no description) + fsErr = ops.WriteFile(ctx, "/sptest/.savepoint/quick-mark.json", []byte("{}")) + require.Nil(t, fsErr, "create savepoint via JSON") + + // List savepoints -- with name as PK, entries are human-readable names + entries, fsErr := ops.ReadDir(ctx, "/sptest/.savepoint") + require.Nil(t, fsErr, "ReadDir .savepoint/ should succeed") + var rowEntries []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + rowEntries = append(rowEntries, e.Name) + } + } + assert.GreaterOrEqual(t, len(rowEntries), 2, "should have at least 2 savepoint entries") + assert.Contains(t, rowEntries, "before-exploration", "should list by name") + assert.Contains(t, rowEntries, "quick-mark", "should list by name") + + // Read column: description + desc, fsErr := ops.ReadFile(ctx, "/sptest/.savepoint/before-exploration/description") + require.Nil(t, fsErr, "ReadFile description should succeed") + assert.Equal(t, "Starting exploration", strings.TrimSpace(string(desc.Data))) + + // Read column: savepoint_id (should be a UUID) + spID, fsErr := ops.ReadFile(ctx, "/sptest/.savepoint/before-exploration/savepoint_id") + require.Nil(t, fsErr, "ReadFile savepoint_id should succeed") + assert.GreaterOrEqual(t, len(strings.TrimSpace(string(spID.Data))), 36, "savepoint_id should be a UUID") + + // Update description + fsErr = ops.WriteFile(ctx, "/sptest/.savepoint/before-exploration/description", []byte("Updated description\n")) + require.Nil(t, fsErr, "update description should succeed") + + desc, fsErr = ops.ReadFile(ctx, "/sptest/.savepoint/before-exploration/description") + require.Nil(t, fsErr) + assert.Equal(t, "Updated description", strings.TrimSpace(string(desc.Data))) + + // Delete savepoint + fsErr = ops.Delete(ctx, "/sptest/.savepoint/quick-mark") + require.Nil(t, fsErr, "delete savepoint should succeed") + + // Verify deleted savepoint is gone + _, fsErr = ops.Stat(ctx, "/sptest/.savepoint/quick-mark") + require.NotNil(t, fsErr, "deleted savepoint should not be found") + + // Original savepoint still accessible + desc, fsErr = ops.ReadFile(ctx, "/sptest/.savepoint/before-exploration/description") + require.Nil(t, fsErr, "non-deleted savepoint should still be accessible") +} + +// TestSynth_Savepoint_PipelineLast verifies that .last/N pipeline works on savepoints. +func TestSynth_Savepoint_PipelineLast(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "sporder") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/sporder", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create 3 savepoints + fsErr = ops.WriteFile(ctx, "/sporder/.savepoint/first.json", []byte("{}")) + require.Nil(t, fsErr) + time.Sleep(50 * time.Millisecond) + fsErr = ops.WriteFile(ctx, "/sporder/.savepoint/second.json", []byte("{}")) + require.Nil(t, fsErr) + time.Sleep(50 * time.Millisecond) + fsErr = ops.WriteFile(ctx, "/sporder/.savepoint/third.json", []byte("{}")) + require.Nil(t, fsErr) + + // .last/2 should return exactly 2 data entries (plus capability dirs) + entries, fsErr := ops.ReadDir(ctx, "/sporder/.savepoint/.last/2") + require.Nil(t, fsErr, "ReadDir .last/2 should succeed") + var rowEntries []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + rowEntries = append(rowEntries, e.Name) + } + } + assert.Equal(t, 2, len(rowEntries), ".last/2 should return exactly 2 savepoint entries") +} + +// TestSynth_Savepoint_FilterByUser verifies that .by/user_id/ filters work. +func TestSynth_Savepoint_FilterByUser(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "spuser") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/spuser", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create savepoints as agent-7 + ops.SetUserID("agent-7") + fsErr = ops.WriteFile(ctx, "/spuser/.savepoint/agent7-first.tsv", []byte("description\nFirst by agent 7\n")) + require.Nil(t, fsErr) + time.Sleep(50 * time.Millisecond) + fsErr = ops.WriteFile(ctx, "/spuser/.savepoint/agent7-second.tsv", []byte("description\nSecond by agent 7\n")) + require.Nil(t, fsErr) + + // Create savepoints as agent-9 + ops.SetUserID("agent-9") + fsErr = ops.WriteFile(ctx, "/spuser/.savepoint/agent9-only.tsv", []byte("description\nOnly by agent 9\n")) + require.Nil(t, fsErr) + + // Verify column read works + uid, fsErr := ops.ReadFile(ctx, "/spuser/.savepoint/agent7-first/user_id") + require.Nil(t, fsErr) + assert.Equal(t, "agent-7", strings.TrimSpace(string(uid.Data))) + + uid, fsErr = ops.ReadFile(ctx, "/spuser/.savepoint/agent9-only/user_id") + require.Nil(t, fsErr) + assert.Equal(t, "agent-9", strings.TrimSpace(string(uid.Data))) +} + +// TestSynth_Savepoint_UserIDPopulated verifies that creating a savepoint +// populates user_id from the mount-level identity. +func TestSynth_Savepoint_UserIDPopulated(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "spuid") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/spuid", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + ops.SetUserID("demo-user") + fsErr = ops.WriteFile(ctx, "/spuid/.savepoint/my-save.tsv", []byte("description\ncheckpoint\n")) + require.Nil(t, fsErr) + + uid, fsErr := ops.ReadFile(ctx, "/spuid/.savepoint/my-save/user_id") + require.Nil(t, fsErr) + assert.Equal(t, "demo-user", strings.TrimSpace(string(uid.Data)), + "savepoint should have the mount-level user_id") +} + +// TestSynth_Savepoint_StatNotFound verifies that stat on nonexistent savepoint returns ENOENT. +func TestSynth_Savepoint_StatNotFound(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "spnf") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/spnf", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + _, fsErr = ops.Stat(ctx, "/spnf/.savepoint/nonexistent") + require.NotNil(t, fsErr) + assert.Equal(t, fs.ErrNotExist, fsErr.Code, "should return ErrNotExist for nonexistent savepoint") +} + +// TestSynth_Savepoint_BarePathRejected verifies that bare-path savepoint creation is rejected. +func TestSynth_Savepoint_BarePathRejected(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "spbare") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/spbare", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Bare path (no format suffix) should be rejected + fsErr = ops.WriteFile(ctx, "/spbare/.savepoint/my-save", []byte("checkpoint\n")) + require.NotNil(t, fsErr, "bare-path savepoint creation should be rejected") + assert.Equal(t, fs.ErrInvalidArgument, fsErr.Code) +} + +// TestSynth_Savepoint_MultipleFormats verifies savepoint creation with different formats. +func TestSynth_Savepoint_MultipleFormats(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "spfmt") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/spfmt", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // TSV format + fsErr = ops.WriteFile(ctx, "/spfmt/.savepoint/via-tsv.tsv", []byte("description\nCreated via TSV\n")) + require.Nil(t, fsErr, "TSV savepoint creation should succeed") + + // JSON format + fsErr = ops.WriteFile(ctx, "/spfmt/.savepoint/via-json.json", []byte(`{"description":"Created via JSON"}`)) + require.Nil(t, fsErr, "JSON savepoint creation should succeed") + + // CSV format + fsErr = ops.WriteFile(ctx, "/spfmt/.savepoint/via-csv.csv", []byte("description\nCreated via CSV\n")) + require.Nil(t, fsErr, "CSV savepoint creation should succeed") + + // Verify all three exist and have correct descriptions + for _, tc := range []struct { + name, desc string + }{ + {"via-tsv", "Created via TSV"}, + {"via-json", "Created via JSON"}, + {"via-csv", "Created via CSV"}, + } { + d, fsErr := ops.ReadFile(ctx, "/spfmt/.savepoint/"+tc.name+"/description") + require.Nil(t, fsErr, "ReadFile description for %s should succeed", tc.name) + assert.Equal(t, tc.desc, strings.TrimSpace(string(d.Data)), "description for %s", tc.name) + } +} + +// TestSynth_Savepoint_NameBasedListing verifies that ReadDir returns human-readable +// names (not UUIDs) now that name is the PK. +func TestSynth_Savepoint_NameBasedListing(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "spnames") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/spnames", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + fsErr = ops.WriteFile(ctx, "/spnames/.savepoint/alpha.json", []byte(`{"description":"first"}`)) + require.Nil(t, fsErr) + fsErr = ops.WriteFile(ctx, "/spnames/.savepoint/beta.json", []byte(`{"description":"second"}`)) + require.Nil(t, fsErr) + + entries, fsErr := ops.ReadDir(ctx, "/spnames/.savepoint") + require.Nil(t, fsErr) + + var names []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + names = append(names, e.Name) + } + } + assert.Contains(t, names, "alpha", "should list by human-readable name") + assert.Contains(t, names, "beta", "should list by human-readable name") + // Names should NOT look like UUIDs + for _, n := range names { + assert.Less(t, len(n), 36, "entry %q should be a name, not a UUID", n) + } +} + +// TestSynth_Savepoint_FormatFileRead verifies that cat .savepoint/name/.json +// returns the full row serialized in the requested format. +func TestSynth_Savepoint_FormatFileRead(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "spfmtrd") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/spfmtrd", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + fsErr = ops.WriteFile(ctx, "/spfmtrd/.savepoint/my-save.tsv", []byte("description\nTest checkpoint\n")) + require.Nil(t, fsErr) + + // Read as JSON + content, fsErr := ops.ReadFile(ctx, "/spfmtrd/.savepoint/my-save/.json") + require.Nil(t, fsErr, "ReadFile .json should succeed") + data := string(content.Data) + assert.Contains(t, data, "my-save", "JSON should contain name") + assert.Contains(t, data, "Test checkpoint", "JSON should contain description") + + // Read as TSV + content, fsErr = ops.ReadFile(ctx, "/spfmtrd/.savepoint/my-save/.tsv") + require.Nil(t, fsErr, "ReadFile .tsv should succeed") + assert.Contains(t, string(content.Data), "my-save", "TSV should contain name") +} + +// TestSynth_Savepoint_YAMLFormat verifies savepoint creation with .yaml suffix. +func TestSynth_Savepoint_YAMLFormat(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "spyaml") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/spyaml", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + fsErr = ops.WriteFile(ctx, "/spyaml/.savepoint/yaml-test.yaml", []byte("description: Created via YAML\n")) + require.Nil(t, fsErr, "YAML savepoint creation should succeed") + + desc, fsErr := ops.ReadFile(ctx, "/spyaml/.savepoint/yaml-test/description") + require.Nil(t, fsErr) + assert.Equal(t, "Created via YAML", strings.TrimSpace(string(desc.Data))) +} + +// TestSynth_Savepoint_EmptyBodyCreation verifies that echo "" > .savepoint/name.tsv +// creates a savepoint with just the name PK. +func TestSynth_Savepoint_EmptyBodyCreation(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "spempty") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/spempty", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Empty body -- should create with just name + auto-generated savepoint_id + fsErr = ops.WriteFile(ctx, "/spempty/.savepoint/empty-save.json", []byte("{}")) + require.Nil(t, fsErr, "empty body savepoint creation should succeed") + + // Verify name column + name, fsErr := ops.ReadFile(ctx, "/spempty/.savepoint/empty-save/name") + require.Nil(t, fsErr) + assert.Equal(t, "empty-save", strings.TrimSpace(string(name.Data))) + + // Verify savepoint_id was auto-generated + spID, fsErr := ops.ReadFile(ctx, "/spempty/.savepoint/empty-save/savepoint_id") + require.Nil(t, fsErr) + assert.GreaterOrEqual(t, len(strings.TrimSpace(string(spID.Data))), 36, "should have auto-generated UUID") +} + +// TestSynth_Savepoint_DeleteAndRecreate verifies that a savepoint can be deleted +// and a new one created with the same name. +func TestSynth_Savepoint_DeleteAndRecreate(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "spreuse") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/spreuse", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create + fsErr = ops.WriteFile(ctx, "/spreuse/.savepoint/reusable.tsv", []byte("description\nOriginal\n")) + require.Nil(t, fsErr) + + desc, fsErr := ops.ReadFile(ctx, "/spreuse/.savepoint/reusable/description") + require.Nil(t, fsErr) + assert.Equal(t, "Original", strings.TrimSpace(string(desc.Data))) + + // Delete + fsErr = ops.Delete(ctx, "/spreuse/.savepoint/reusable") + require.Nil(t, fsErr) + + _, fsErr = ops.Stat(ctx, "/spreuse/.savepoint/reusable") + require.NotNil(t, fsErr, "deleted savepoint should not be found") + + // Recreate with same name, different description + fsErr = ops.WriteFile(ctx, "/spreuse/.savepoint/reusable.tsv", []byte("description\nRecreated\n")) + require.Nil(t, fsErr, "recreating deleted savepoint should succeed") + + desc, fsErr = ops.ReadFile(ctx, "/spreuse/.savepoint/reusable/description") + require.Nil(t, fsErr) + assert.Equal(t, "Recreated", strings.TrimSpace(string(desc.Data))) +} + +// TestSynth_Savepoint_Ordering verifies that .last/N returns entries in +// descending PK order (alphabetical by name since name is PK). +func TestSynth_Savepoint_Ordering(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "spord") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/spord", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create savepoints with names that sort alphabetically + fsErr = ops.WriteFile(ctx, "/spord/.savepoint/aaa-first.json", []byte("{}")) + require.Nil(t, fsErr) + fsErr = ops.WriteFile(ctx, "/spord/.savepoint/bbb-second.json", []byte("{}")) + require.Nil(t, fsErr) + fsErr = ops.WriteFile(ctx, "/spord/.savepoint/ccc-third.json", []byte("{}")) + require.Nil(t, fsErr) + + // .last/2 should return the last 2 alphabetically (bbb, ccc) + entries, fsErr := ops.ReadDir(ctx, "/spord/.savepoint/.last/2") + require.Nil(t, fsErr) + var names []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + names = append(names, e.Name) + } + } + assert.Equal(t, 2, len(names), ".last/2 should return 2 entries") + assert.Contains(t, names, "bbb-second") + assert.Contains(t, names, "ccc-third") + assert.NotContains(t, names, "aaa-first", "aaa-first should not be in .last/2") +} + +// TestSynth_AutoSavepoint_CreatedOnGap verifies that an auto-savepoint is created +// when the inactivity gap exceeds the configured threshold. Uses injectable clock +// to avoid real sleeps. +func TestSynth_AutoSavepoint_CreatedOnGap(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "spauto") + + // Create ops with auto-savepoint interval of 5 minutes + cfg := &config.Config{ + DirListingLimit: 10000, + QueryTimeout: 30, + PoolSize: 5, + PoolMaxIdle: 2, + InsecureNoSSL: true, + AutoSavepointInterval: 5 * time.Minute, + } + ctx := context.Background() + dbClient, err := db.NewClient(ctx, cfg, result.ConnStr) + require.NoError(t, err) + t.Cleanup(func() { dbClient.Close() }) + + ops := fs.NewOperations(cfg, dbClient) + + // Injectable clock + baseTime := time.Date(2026, 4, 8, 14, 0, 0, 0, time.UTC) + currentTime := baseTime + ops.SetNowFunc(func() time.Time { return currentTime }) + + // Create the app + fsErr := ops.WriteFile(ctx, "/.build/spauto", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // First write at T+0 -- no auto-savepoint (first write, no previous session) + fsErr = ops.WriteFile(ctx, "/spauto/first.md", []byte("---\ntitle: First\n---\nContent\n")) + require.Nil(t, fsErr) + + // Second write at T+1m -- within interval, no auto-savepoint + currentTime = baseTime.Add(1 * time.Minute) + fsErr = ops.WriteFile(ctx, "/spauto/second.md", []byte("---\ntitle: Second\n---\nContent\n")) + require.Nil(t, fsErr) + + // Verify no auto-savepoints yet + entries, fsErr := ops.ReadDir(ctx, "/spauto/.savepoint") + require.Nil(t, fsErr) + var autoNames []string + for _, e := range entries { + if strings.HasPrefix(e.Name, "auto-") { + autoNames = append(autoNames, e.Name) + } + } + assert.Empty(t, autoNames, "no auto-savepoints should exist yet") + + // Third write at T+10m -- exceeds 5m interval, should trigger auto-savepoint + currentTime = baseTime.Add(10 * time.Minute) + fsErr = ops.WriteFile(ctx, "/spauto/third.md", []byte("---\ntitle: Third\n---\nContent\n")) + require.Nil(t, fsErr) + + // Verify auto-savepoint was created + entries, fsErr = ops.ReadDir(ctx, "/spauto/.savepoint") + require.Nil(t, fsErr) + autoNames = nil + for _, e := range entries { + if strings.HasPrefix(e.Name, "auto-") { + autoNames = append(autoNames, e.Name) + } + } + assert.Equal(t, 1, len(autoNames), "should have exactly 1 auto-savepoint") + if len(autoNames) > 0 { + assert.Contains(t, autoNames[0], "auto-", "name should start with auto-") + // Verify it has a description mentioning inactivity + desc, fsErr := ops.ReadFile(ctx, "/spauto/.savepoint/"+autoNames[0]+"/description") + require.Nil(t, fsErr) + assert.Contains(t, string(desc.Data), "inactivity") + } +} + +// TestSynth_AutoSavepoint_DisabledWhenZero verifies that interval=0 disables auto-savepoints. +func TestSynth_AutoSavepoint_DisabledWhenZero(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "spnone") + + cfg := &config.Config{ + DirListingLimit: 10000, + QueryTimeout: 30, + PoolSize: 5, + PoolMaxIdle: 2, + InsecureNoSSL: true, + AutoSavepointInterval: 0, // disabled + } + ctx := context.Background() + dbClient, err := db.NewClient(ctx, cfg, result.ConnStr) + require.NoError(t, err) + t.Cleanup(func() { dbClient.Close() }) + + ops := fs.NewOperations(cfg, dbClient) + + baseTime := time.Date(2026, 4, 8, 14, 0, 0, 0, time.UTC) + currentTime := baseTime + ops.SetNowFunc(func() time.Time { return currentTime }) + + fsErr := ops.WriteFile(ctx, "/.build/spnone", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // First write + fsErr = ops.WriteFile(ctx, "/spnone/first.md", []byte("---\ntitle: First\n---\nContent\n")) + require.Nil(t, fsErr) + + // Second write with huge gap -- should NOT trigger auto-savepoint + currentTime = baseTime.Add(24 * time.Hour) + fsErr = ops.WriteFile(ctx, "/spnone/second.md", []byte("---\ntitle: Second\n---\nContent\n")) + require.Nil(t, fsErr) + + entries, fsErr := ops.ReadDir(ctx, "/spnone/.savepoint") + require.Nil(t, fsErr) + for _, e := range entries { + assert.False(t, strings.HasPrefix(e.Name, "auto-"), + "no auto-savepoints should exist when interval=0, found: %s", e.Name) + } +} diff --git a/test/integration/setup.go b/test/integration/setup.go index f94e11c..4c3972d 100644 --- a/test/integration/setup.go +++ b/test/integration/setup.go @@ -402,6 +402,8 @@ func cleanupTigerFSTables(t *testing.T, connStr string, tableNames ...string) { for _, name := range tableNames { pool.Exec(ctx, fmt.Sprintf("DROP TABLE IF EXISTS tigerfs.%s CASCADE", name)) pool.Exec(ctx, fmt.Sprintf("DROP TABLE IF EXISTS tigerfs.%s_history CASCADE", name)) + pool.Exec(ctx, fmt.Sprintf("DROP TABLE IF EXISTS tigerfs.%s_log CASCADE", name)) + pool.Exec(ctx, fmt.Sprintf("DROP TABLE IF EXISTS tigerfs.%s_savepoint CASCADE", name)) } }) } diff --git a/test/integration/synthesized_test.go b/test/integration/synthesized_test.go index b16b915..bd5d540 100644 --- a/test/integration/synthesized_test.go +++ b/test/integration/synthesized_test.go @@ -738,9 +738,9 @@ func TestSynth_HierarchicalMkdirNested(t *testing.T) { fsErr := ops.WriteFile(ctx, "/.build/mem_mknest", []byte("markdown\n")) require.Nil(t, fsErr, "build should succeed") - // Create a nested directory — should auto-create parent - fsErr = ops.Mkdir(ctx, "/mem_mknest/projects/web") - require.Nil(t, fsErr, "Mkdir nested should succeed: %v", fsErr) + // Create a nested directory chain via MkdirAll (mkdir-p semantics). + fsErr = ops.MkdirAll(ctx, "/mem_mknest/projects/web") + require.Nil(t, fsErr, "MkdirAll nested should succeed: %v", fsErr) // Verify parent was auto-created entry, fsErr := ops.Stat(ctx, "/mem_mknest/projects") @@ -774,10 +774,11 @@ func TestSynth_HierarchicalWriteFile(t *testing.T) { fsErr := ops.WriteFile(ctx, "/.build/mem_write", []byte("markdown\n")) require.Nil(t, fsErr, "build should succeed") - // Write a file in a subdirectory — should auto-create parent dirs + // Write a file in a subdirectory using WriteFileEnsureDirs (mkdir-p + // the ancestors as part of the same call). content := "---\ntitle: Todo\nauthor: alice\n---\n\n# Todo\n\nFix bugs.\n" - fsErr = ops.WriteFile(ctx, "/mem_write/projects/web/todo.md", []byte(content)) - require.Nil(t, fsErr, "WriteFile nested should succeed: %v", fsErr) + fsErr = ops.WriteFileEnsureDirs(ctx, "/mem_write/projects/web/todo.md", []byte(content)) + require.Nil(t, fsErr, "WriteFileEnsureDirs nested should succeed: %v", fsErr) // Verify parent directories were auto-created entry, fsErr := ops.Stat(ctx, "/mem_write/projects") @@ -811,8 +812,8 @@ func TestSynth_HierarchicalReadDir(t *testing.T) { require.Nil(t, fsErr, "build should succeed") // Create structure: projects/web/todo.md, projects/web/notes.md, readme.md - fsErr = ops.Mkdir(ctx, "/mem_readdir/projects/web") - require.Nil(t, fsErr, "Mkdir should succeed") + fsErr = ops.MkdirAll(ctx, "/mem_readdir/projects/web") + require.Nil(t, fsErr, "MkdirAll should succeed") fsErr = ops.WriteFile(ctx, "/mem_readdir/projects/web/todo.md", []byte("---\ntitle: Todo\n---\n\nContent.\n")) @@ -863,10 +864,10 @@ func TestSynth_HierarchicalStatFile(t *testing.T) { fsErr := ops.WriteFile(ctx, "/.build/mem_stat", []byte("markdown\n")) require.Nil(t, fsErr, "build should succeed") - // Write a nested file + // Write a nested file (mkdir-p the ancestors first via the helper). content := "---\ntitle: Deep File\n---\n\nDeep content.\n" - fsErr = ops.WriteFile(ctx, "/mem_stat/deep/nested/file.md", []byte(content)) - require.Nil(t, fsErr, "WriteFile should succeed") + fsErr = ops.WriteFileEnsureDirs(ctx, "/mem_stat/deep/nested/file.md", []byte(content)) + require.Nil(t, fsErr, "WriteFileEnsureDirs should succeed") // Stat the file entry, fsErr := ops.Stat(ctx, "/mem_stat/deep/nested/file.md") @@ -947,10 +948,10 @@ func TestSynth_HierarchicalDeleteFile(t *testing.T) { fsErr := ops.WriteFile(ctx, "/.build/mem_delf", []byte("markdown\n")) require.Nil(t, fsErr, "build should succeed") - // Create a file in a directory - fsErr = ops.WriteFile(ctx, "/mem_delf/docs/readme.md", + // Create a file in a directory (mkdir-p the parent via the helper). + fsErr = ops.WriteFileEnsureDirs(ctx, "/mem_delf/docs/readme.md", []byte("---\ntitle: Readme\n---\n\nContent.\n")) - require.Nil(t, fsErr, "WriteFile should succeed") + require.Nil(t, fsErr, "WriteFileEnsureDirs should succeed") // Delete the file fsErr = ops.Delete(ctx, "/mem_delf/docs/readme.md") @@ -1011,10 +1012,10 @@ func TestSynth_HierarchicalRmdirNonEmpty(t *testing.T) { fsErr := ops.WriteFile(ctx, "/.build/mem_rmne", []byte("markdown\n")) require.Nil(t, fsErr, "build should succeed") - // Create a directory with a file in it - fsErr = ops.WriteFile(ctx, "/mem_rmne/docs/readme.md", + // Create a directory with a file in it (mkdir-p the parent). + fsErr = ops.WriteFileEnsureDirs(ctx, "/mem_rmne/docs/readme.md", []byte("---\ntitle: Readme\n---\n\nContent.\n")) - require.Nil(t, fsErr, "WriteFile should succeed") + require.Nil(t, fsErr, "WriteFileEnsureDirs should succeed") // Attempt to delete non-empty directory should fail fsErr = ops.Delete(ctx, "/mem_rmne/docs") @@ -1037,10 +1038,10 @@ func TestSynth_HierarchicalRenameFile(t *testing.T) { fsErr := ops.WriteFile(ctx, "/.build/mem_renf", []byte("markdown\n")) require.Nil(t, fsErr, "build should succeed") - // Create a file in a directory - fsErr = ops.WriteFile(ctx, "/mem_renf/docs/old-name.md", + // Create a file in a directory (mkdir-p the parent). + fsErr = ops.WriteFileEnsureDirs(ctx, "/mem_renf/docs/old-name.md", []byte("---\ntitle: My Doc\n---\n\nContent here.\n")) - require.Nil(t, fsErr, "WriteFile should succeed") + require.Nil(t, fsErr, "WriteFileEnsureDirs should succeed") // Rename the file fsErr = ops.Rename(ctx, "/mem_renf/docs/old-name.md", "/mem_renf/docs/new-name.md") @@ -1072,10 +1073,11 @@ func TestSynth_HierarchicalRenameDir(t *testing.T) { fsErr := ops.WriteFile(ctx, "/.build/mem_rend", []byte("markdown\n")) require.Nil(t, fsErr, "build should succeed") - // Create a directory with files - fsErr = ops.WriteFile(ctx, "/mem_rend/old-dir/file1.md", + // Create a directory with files (mkdir-p the parent for the first + // write; the second write only needs the parent that already exists). + fsErr = ops.WriteFileEnsureDirs(ctx, "/mem_rend/old-dir/file1.md", []byte("---\ntitle: File One\n---\n\nContent 1.\n")) - require.Nil(t, fsErr, "WriteFile file1 should succeed") + require.Nil(t, fsErr, "WriteFileEnsureDirs file1 should succeed") fsErr = ops.WriteFile(ctx, "/mem_rend/old-dir/file2.md", []byte("---\ntitle: File Two\n---\n\nContent 2.\n")) @@ -1119,10 +1121,11 @@ func TestSynth_DeeplyNested(t *testing.T) { fsErr := ops.WriteFile(ctx, "/.build/mem_deep", []byte("markdown\n")) require.Nil(t, fsErr, "build should succeed") - // Write a deeply nested file — auto-creates all parent directories + // Write a deeply nested file via WriteFileEnsureDirs (creates all + // parent directories). content := "---\ntitle: Deep File\n---\n\nVery deep content.\n" - fsErr = ops.WriteFile(ctx, "/mem_deep/a/b/c/d/deep.md", []byte(content)) - require.Nil(t, fsErr, "WriteFile deeply nested should succeed: %v", fsErr) + fsErr = ops.WriteFileEnsureDirs(ctx, "/mem_deep/a/b/c/d/deep.md", []byte(content)) + require.Nil(t, fsErr, "WriteFileEnsureDirs deeply nested should succeed: %v", fsErr) // Verify all intermediate directories exist for _, dir := range []string{"/mem_deep/a", "/mem_deep/a/b", "/mem_deep/a/b/c", "/mem_deep/a/b/c/d"} { @@ -1168,10 +1171,10 @@ func TestSynth_MixedFlatAndHierarchical(t *testing.T) { []byte("---\ntitle: Flat\n---\n\nFlat content.\n")) require.Nil(t, fsErr, "WriteFile flat should succeed") - // Create a nested file (creates directory) - fsErr = ops.WriteFile(ctx, "/mem_mixed/subdir/nested.md", + // Create a nested file (mkdir-p the parent via the helper). + fsErr = ops.WriteFileEnsureDirs(ctx, "/mem_mixed/subdir/nested.md", []byte("---\ntitle: Nested\n---\n\nNested content.\n")) - require.Nil(t, fsErr, "WriteFile nested should succeed") + require.Nil(t, fsErr, "WriteFileEnsureDirs nested should succeed") // Root should show both flat file and directory entries, fsErr := ops.ReadDir(ctx, "/mem_mixed") @@ -1207,9 +1210,9 @@ func TestSynth_HierarchicalPlainText(t *testing.T) { fsErr := ops.WriteFile(ctx, "/.build/notes_hier", []byte("txt\n")) require.Nil(t, fsErr, "build should succeed: %v", fsErr) - // Create a nested file - fsErr = ops.WriteFile(ctx, "/notes_hier/work/meeting.txt", []byte("Discuss quarterly goals.\n")) - require.Nil(t, fsErr, "WriteFile nested txt should succeed: %v", fsErr) + // Create a nested file (mkdir-p the parent via the helper). + fsErr = ops.WriteFileEnsureDirs(ctx, "/notes_hier/work/meeting.txt", []byte("Discuss quarterly goals.\n")) + require.Nil(t, fsErr, "WriteFileEnsureDirs nested txt should succeed: %v", fsErr) // Verify parent directory was auto-created entry, fsErr := ops.Stat(ctx, "/notes_hier/work") @@ -1253,6 +1256,496 @@ func TestSynth_MkdirAlreadyExists(t *testing.T) { assert.Equal(t, fs.ErrExists, fsErr.Code, "should be EEXIST error") } +// ============================================================================ +// Parent-Pointer Model Tests (ADR-017) +// ============================================================================ + +// TestSynth_MoveFileBetweenDirs verifies moving a file from one directory +// to another by renaming across directories (ADR-017 verification #5). +func TestSynth_MoveFileBetweenDirs(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "mem_move") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/mem_move", []byte("markdown\n")) + require.Nil(t, fsErr) + + // Create source and target directories + fsErr = ops.Mkdir(ctx, "/mem_move/inbox") + require.Nil(t, fsErr) + fsErr = ops.Mkdir(ctx, "/mem_move/archive") + require.Nil(t, fsErr) + + // Create file in inbox + content := "---\ntitle: Task\n---\n# Task\n" + fsErr = ops.WriteFile(ctx, "/mem_move/inbox/task.md", []byte(content)) + require.Nil(t, fsErr) + + // Verify file is in inbox + entries, fsErr := ops.ReadDir(ctx, "/mem_move/inbox") + require.Nil(t, fsErr) + assert.Contains(t, fsEntryNames(entries), "task.md") + + // Move file from inbox to archive + fsErr = ops.Rename(ctx, "/mem_move/inbox/task.md", "/mem_move/archive/task.md") + require.Nil(t, fsErr, "move file between dirs should succeed") + + // Verify file is no longer in inbox + entries, fsErr = ops.ReadDir(ctx, "/mem_move/inbox") + require.Nil(t, fsErr) + assert.NotContains(t, fsEntryNames(entries), "task.md", "inbox should be empty after move") + + // Verify file is now in archive + entries, fsErr = ops.ReadDir(ctx, "/mem_move/archive") + require.Nil(t, fsErr) + assert.Contains(t, fsEntryNames(entries), "task.md", "archive should contain moved file") + + // Verify file content is preserved + fileContent, fsErr := ops.ReadFile(ctx, "/mem_move/archive/task.md") + require.Nil(t, fsErr) + assert.Contains(t, string(fileContent.Data), "# Task") +} + +// TestSynth_MoveDirBetweenParents verifies moving a directory from one parent +// to another (ADR-017 verification #9). +func TestSynth_MoveDirBetweenParents(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "mem_mvdir") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/mem_mvdir", []byte("markdown\n")) + require.Nil(t, fsErr) + + // Create: parent1/child/file.md, parent2/ + fsErr = ops.Mkdir(ctx, "/mem_mvdir/parent1") + require.Nil(t, fsErr) + fsErr = ops.Mkdir(ctx, "/mem_mvdir/parent1/child") + require.Nil(t, fsErr) + content := "---\ntitle: File\n---\nContent\n" + fsErr = ops.WriteFile(ctx, "/mem_mvdir/parent1/child/file.md", []byte(content)) + require.Nil(t, fsErr) + fsErr = ops.Mkdir(ctx, "/mem_mvdir/parent2") + require.Nil(t, fsErr) + + // Move child directory from parent1 to parent2 + fsErr = ops.Rename(ctx, "/mem_mvdir/parent1/child", "/mem_mvdir/parent2/child") + require.Nil(t, fsErr, "move directory should succeed") + + // parent1 should no longer have child + entries, fsErr := ops.ReadDir(ctx, "/mem_mvdir/parent1") + require.Nil(t, fsErr) + assert.NotContains(t, fsEntryNames(entries), "child") + + // parent2 should now have child + entries, fsErr = ops.ReadDir(ctx, "/mem_mvdir/parent2") + require.Nil(t, fsErr) + assert.Contains(t, fsEntryNames(entries), "child") + + // File inside moved directory should still be accessible + entries, fsErr = ops.ReadDir(ctx, "/mem_mvdir/parent2/child") + require.Nil(t, fsErr) + assert.Contains(t, fsEntryNames(entries), "file.md") + + fileContent, fsErr := ops.ReadFile(ctx, "/mem_mvdir/parent2/child/file.md") + require.Nil(t, fsErr) + assert.Contains(t, string(fileContent.Data), "Content") +} + +// TestSynth_EmptyDirReadDir verifies ReadDir on an empty directory returns +// an empty list (ADR-017 verification #20). +func TestSynth_EmptyDirReadDir(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "mem_empty") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/mem_empty", []byte("markdown\n")) + require.Nil(t, fsErr) + + fsErr = ops.Mkdir(ctx, "/mem_empty/vacant") + require.Nil(t, fsErr) + + entries, fsErr := ops.ReadDir(ctx, "/mem_empty/vacant") + require.Nil(t, fsErr, "ReadDir empty dir should succeed") + assert.Empty(t, entries, "empty directory should have no entries") +} + +// TestSynth_SameLeafNameDifferentDirs verifies that files with the same +// leaf name in different directories coexist independently. This is a +// critical test for the parent-pointer model correctness. +func TestSynth_SameLeafNameDifferentDirs(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "mem_samename") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/mem_samename", []byte("markdown\n")) + require.Nil(t, fsErr) + + // Create two directories + fsErr = ops.Mkdir(ctx, "/mem_samename/docs") + require.Nil(t, fsErr) + fsErr = ops.Mkdir(ctx, "/mem_samename/guides") + require.Nil(t, fsErr) + + // Create readme.md in BOTH directories with DIFFERENT content + docsContent := "---\ntitle: Docs Readme\n---\n# Docs\n" + fsErr = ops.WriteFile(ctx, "/mem_samename/docs/readme.md", []byte(docsContent)) + require.Nil(t, fsErr) + + guidesContent := "---\ntitle: Guides Readme\n---\n# Guides\n" + fsErr = ops.WriteFile(ctx, "/mem_samename/guides/readme.md", []byte(guidesContent)) + require.Nil(t, fsErr) + + // Both directories should list readme.md + entries, fsErr := ops.ReadDir(ctx, "/mem_samename/docs") + require.Nil(t, fsErr) + assert.Contains(t, fsEntryNames(entries), "readme.md") + + entries, fsErr = ops.ReadDir(ctx, "/mem_samename/guides") + require.Nil(t, fsErr) + assert.Contains(t, fsEntryNames(entries), "readme.md") + + // Read each and verify content is distinct + docsFile, fsErr := ops.ReadFile(ctx, "/mem_samename/docs/readme.md") + require.Nil(t, fsErr, "ReadFile docs/readme.md should succeed") + assert.Contains(t, string(docsFile.Data), "# Docs") + assert.NotContains(t, string(docsFile.Data), "# Guides") + + guidesFile, fsErr := ops.ReadFile(ctx, "/mem_samename/guides/readme.md") + require.Nil(t, fsErr, "ReadFile guides/readme.md should succeed") + assert.Contains(t, string(guidesFile.Data), "# Guides") + assert.NotContains(t, string(guidesFile.Data), "# Docs") + + // Edit one, verify the other is unaffected + updatedDocs := "---\ntitle: Docs Updated\n---\n# Docs Updated\n" + fsErr = ops.WriteFile(ctx, "/mem_samename/docs/readme.md", []byte(updatedDocs)) + require.Nil(t, fsErr) + + guidesFile, fsErr = ops.ReadFile(ctx, "/mem_samename/guides/readme.md") + require.Nil(t, fsErr) + assert.Contains(t, string(guidesFile.Data), "# Guides", "editing docs/readme.md should not affect guides/readme.md") +} + +// TestSynth_RenameDirChildrenUnaffected verifies that renaming a directory +// does NOT change child rows -- children are unaffected because only the +// directory row's filename changes (ADR-017 single-row rename). +func TestSynth_RenameDirChildrenUnaffected(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "mem_renchild") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/mem_renchild", []byte("markdown\n")) + require.Nil(t, fsErr) + + // Create directory with files (mkdir-p the parent on first write). + content1 := "---\ntitle: A\n---\nFile A\n" + fsErr = ops.WriteFileEnsureDirs(ctx, "/mem_renchild/mydir/a.md", []byte(content1)) + require.Nil(t, fsErr) + content2 := "---\ntitle: B\n---\nFile B\n" + fsErr = ops.WriteFile(ctx, "/mem_renchild/mydir/b.md", []byte(content2)) + require.Nil(t, fsErr) + + // Rename directory + fsErr = ops.Rename(ctx, "/mem_renchild/mydir", "/mem_renchild/renamed") + require.Nil(t, fsErr, "rename dir should succeed") + + // Children should be accessible under new name + entries, fsErr := ops.ReadDir(ctx, "/mem_renchild/renamed") + require.Nil(t, fsErr, "ReadDir renamed dir should succeed") + names := fsEntryNames(entries) + assert.Contains(t, names, "a.md") + assert.Contains(t, names, "b.md") + + // File content should be preserved + file, fsErr := ops.ReadFile(ctx, "/mem_renchild/renamed/a.md") + require.Nil(t, fsErr) + assert.Contains(t, string(file.Data), "File A") +} + +// TestSynth_NestedReadDirAtEachLevel verifies ReadDir returns correct entries +// at every level of a 3-level hierarchy. +func TestSynth_NestedReadDirAtEachLevel(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "mem_levels") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/mem_levels", []byte("markdown\n")) + require.Nil(t, fsErr) + + // Build: root-file.md, L1/L1-file.md, L1/L2/L2-file.md, L1/L2/L3/L3-file.md. + // Use WriteFileEnsureDirs for the deep paths; each call mkdir-p's any + // missing ancestors before writing the leaf file. + fsErr = ops.WriteFile(ctx, "/mem_levels/root-file.md", []byte("---\ntitle: Root\n---\nRoot\n")) + require.Nil(t, fsErr) + fsErr = ops.WriteFileEnsureDirs(ctx, "/mem_levels/L1/L1-file.md", []byte("---\ntitle: L1\n---\nL1\n")) + require.Nil(t, fsErr) + fsErr = ops.WriteFileEnsureDirs(ctx, "/mem_levels/L1/L2/L2-file.md", []byte("---\ntitle: L2\n---\nL2\n")) + require.Nil(t, fsErr) + fsErr = ops.WriteFileEnsureDirs(ctx, "/mem_levels/L1/L2/L3/L3-file.md", []byte("---\ntitle: L3\n---\nL3\n")) + require.Nil(t, fsErr) + + // Root level: should have root-file.md and L1 directory + entries, fsErr := ops.ReadDir(ctx, "/mem_levels") + require.Nil(t, fsErr) + names := fsEntryNames(entries) + assert.Contains(t, names, "root-file.md") + assert.Contains(t, names, "L1") + assert.NotContains(t, names, "L2", "root should NOT show nested dirs") + assert.NotContains(t, names, "L1-file.md", "root should NOT show nested files") + + // L1 level: should have L1-file.md and L2 directory + entries, fsErr = ops.ReadDir(ctx, "/mem_levels/L1") + require.Nil(t, fsErr) + names = fsEntryNames(entries) + assert.Contains(t, names, "L1-file.md") + assert.Contains(t, names, "L2") + assert.NotContains(t, names, "root-file.md", "L1 should NOT show root files") + assert.NotContains(t, names, "L3", "L1 should NOT show L3") + + // L2 level: should have L2-file.md and L3 directory + entries, fsErr = ops.ReadDir(ctx, "/mem_levels/L1/L2") + require.Nil(t, fsErr) + names = fsEntryNames(entries) + assert.Contains(t, names, "L2-file.md") + assert.Contains(t, names, "L3") + + // L3 level: should have only L3-file.md + entries, fsErr = ops.ReadDir(ctx, "/mem_levels/L1/L2/L3") + require.Nil(t, fsErr) + names = fsEntryNames(entries) + assert.Equal(t, []string{"L3-file.md"}, names, "L3 should have exactly one file") +} + +// TestSynth_DeleteNestedFileParentPersists verifies that deleting a file +// in a subdirectory does not delete the parent directory. +func TestSynth_DeleteNestedFileParentPersists(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "mem_delnest") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/mem_delnest", []byte("markdown\n")) + require.Nil(t, fsErr) + + // Create two files in a directory (mkdir-p the parent on first write). + fsErr = ops.WriteFileEnsureDirs(ctx, "/mem_delnest/mydir/a.md", []byte("---\ntitle: A\n---\nA\n")) + require.Nil(t, fsErr) + fsErr = ops.WriteFile(ctx, "/mem_delnest/mydir/b.md", []byte("---\ntitle: B\n---\nB\n")) + require.Nil(t, fsErr) + + // Delete one file + fsErr = ops.Delete(ctx, "/mem_delnest/mydir/a.md") + require.Nil(t, fsErr) + + // Directory should still exist with the other file + entries, fsErr := ops.ReadDir(ctx, "/mem_delnest/mydir") + require.Nil(t, fsErr, "directory should still exist after deleting a child") + names := fsEntryNames(entries) + assert.Contains(t, names, "b.md") + assert.NotContains(t, names, "a.md") + + // Root should still show the directory + entries, fsErr = ops.ReadDir(ctx, "/mem_delnest") + require.Nil(t, fsErr) + assert.Contains(t, fsEntryNames(entries), "mydir") +} + +// TestSynth_HistoryAfterRename verifies that .history/ shows a file under +// its NEW name after rename, and versions are accessible (ADR-017 #38). +func TestSynth_HistoryAfterRename(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "mem_histren") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/mem_histren", []byte("markdown,history\n")) + require.Nil(t, fsErr) + + // Create and edit a file (generates history) + v1 := "---\ntitle: V1\n---\nVersion 1\n" + fsErr = ops.WriteFile(ctx, "/mem_histren/original.md", []byte(v1)) + require.Nil(t, fsErr) + + time.Sleep(1100 * time.Millisecond) // ensure distinct UUIDv7 + + v2 := "---\ntitle: V2\n---\nVersion 2\n" + fsErr = ops.WriteFile(ctx, "/mem_histren/original.md", []byte(v2)) + require.Nil(t, fsErr) + + // Rename the file + fsErr = ops.Rename(ctx, "/mem_histren/original.md", "/mem_histren/renamed.md") + require.Nil(t, fsErr) + + // .history/ should list the NEW filename (renamed.md) + // History entries are keyed by the DB filename, which is now "renamed.md" + entries, fsErr := ops.ReadDir(ctx, "/mem_histren/.history") + require.Nil(t, fsErr) + names := fsEntryNames(entries) + // The old name "original.md" had history entries -- those entries still have + // filename="original.md" in the history table. After rename, new entries would + // use "renamed.md". Both should appear in the history listing. + assert.Contains(t, names, "original.md", "history should show old filename (from pre-rename versions)") +} + +// TestSynth_HistoryDirCacheUsesUserSchema verifies that readDirHistory's +// pathCache lookups key under the user's schema, matching every other +// pathCache writer. Without that, history-path entries land in a disjoint +// (tigerfs, table) namespace that survives invalidations from the live- +// path code (delete, mkdir, rename), so a deleted-then-recreated dir +// returns the OLD dir's history under the new dir's path until the +// 2-second TTL expires. +// +// Repro: +// 1. mkdir A, write A/x.md, edit A/x.md (history rows reference A.id). +// 2. ReadDir .history/A/ to populate the path cache for "A". +// 3. delete A/x.md, delete A, mkdir A (new dir, new id). All three +// invalidate the pathCache. +// 4. ReadDir .history/A/. Should resolve "A" to the NEW id and return +// no history entries (the new A has no children with history). +// +// Without the fix, step 4 hits a stale (tigerfs, table) cache entry +// pointing at OLD_A_ID and returns x.md's history under the new A's +// path -- visibly wrong to the user. +func TestSynth_HistoryDirCacheUsesUserSchema(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "histcache") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + require.Nil(t, ops.WriteFile(ctx, "/.build/histcache", []byte("markdown,history\n"))) + time.Sleep(100 * time.Millisecond) + + // Build state: A/x.md with at least one history version. + require.Nil(t, ops.Mkdir(ctx, "/histcache/A")) + require.Nil(t, ops.WriteFile(ctx, "/histcache/A/x.md", + []byte("---\ntitle: X V1\n---\nv1\n"))) + time.Sleep(1100 * time.Millisecond) // distinct UUIDv7 + require.Nil(t, ops.WriteFile(ctx, "/histcache/A/x.md", + []byte("---\ntitle: X V2\n---\nv2\n"))) + + // Prime the path cache for the current "A" -> OLD_ID by reading the + // history listing under that path. + primeEntries, fsErr := ops.ReadDir(ctx, "/histcache/A/.history") + require.Nil(t, fsErr) + primeNames := fsEntryNames(primeEntries) + require.Contains(t, primeNames, "x.md", + "sanity: x.md should appear in A's history before recreate") + + // Tear A down completely. + require.Nil(t, ops.Delete(ctx, "/histcache/A/x.md")) + require.Nil(t, ops.Delete(ctx, "/histcache/A")) + + // Recreate A. New row, new UUID. + require.Nil(t, ops.Mkdir(ctx, "/histcache/A")) + + // Read history under the new A. The pathCache must resolve "A" to the + // NEW UUID (because every prior write/delete/mkdir invalidated the + // pathCache under the user's schema). With the bug, the stale entry + // under (tigerfs, table) keys survives and returns OLD_A_ID, so the + // history query for parent_id = OLD_A_ID returns x.md. + postEntries, fsErr := ops.ReadDir(ctx, "/histcache/A/.history") + require.Nil(t, fsErr) + postNames := fsEntryNames(postEntries) + if assert.NotContains(t, postNames, "x.md", + "history under recreated A should not include the old A's children "+ + "(stale pathCache entry under (tigerfs, table) keys)") { + // extra logging if assertion passed + t.Logf("history under recreated A: %v (expected empty)", postNames) + } +} + +// TestSynth_RootFilesAndDirsCoexist verifies that root-level files and +// directories coexist correctly in ReadDir. +func TestSynth_RootFilesAndDirsCoexist(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "mem_rootmix") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/mem_rootmix", []byte("markdown\n")) + require.Nil(t, fsErr) + + // Create root-level file + fsErr = ops.WriteFile(ctx, "/mem_rootmix/readme.md", []byte("---\ntitle: Root\n---\nRoot file\n")) + require.Nil(t, fsErr) + + // Create root-level directory with a file inside (mkdir-p docs/). + fsErr = ops.WriteFileEnsureDirs(ctx, "/mem_rootmix/docs/guide.md", []byte("---\ntitle: Guide\n---\nGuide\n")) + require.Nil(t, fsErr) + + // ReadDir root should show both + entries, fsErr := ops.ReadDir(ctx, "/mem_rootmix") + require.Nil(t, fsErr) + names := fsEntryNames(entries) + assert.Contains(t, names, "readme.md", "root should have file") + assert.Contains(t, names, "docs", "root should have directory") + + // Verify types + for _, e := range entries { + if e.Name == "readme.md" { + assert.False(t, e.IsDir, "readme.md should be a file") + } + if e.Name == "docs" { + assert.True(t, e.IsDir, "docs should be a directory") + } + } +} + // ============================================================================ // Helpers // ============================================================================ diff --git a/test/integration/undo_test.go b/test/integration/undo_test.go new file mode 100644 index 0000000..b560af4 --- /dev/null +++ b/test/integration/undo_test.go @@ -0,0 +1,1748 @@ +package integration + +import ( + "context" + "encoding/json" + "fmt" + "io" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/timescale/tigerfs/internal/tigerfs/config" + "github.com/timescale/tigerfs/internal/tigerfs/db" + "github.com/timescale/tigerfs/internal/tigerfs/fs" + "github.com/timescale/tigerfs/internal/tigerfs/nfs" +) + +// --- Mkdir / WriteFileEnsureDirs logging --- + +// TestMkdir_LogsCreateEntry verifies that mkdirSynth writes a 'create' +// log entry, mirroring what WriteFile does for new files. Without this, +// undo can't roll back a Mkdir-created directory because the dir is +// invisible to QueryUndoAffectedFiles. +func TestMkdir_LogsCreateEntry(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "mkdirlog") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + require.Nil(t, ops.WriteFile(ctx, "/.build/mkdirlog", []byte("markdown,history\n"))) + time.Sleep(100 * time.Millisecond) + + // Mkdir is the only post-build op, so all of its log entries are new. + require.Nil(t, ops.Mkdir(ctx, "/mkdirlog/A")) + + // .log/.by/type/create should now contain exactly one entry whose + // filename is the dir we just made. + entries, fsErr := ops.ReadDir(ctx, "/mkdirlog/.log/.by/type/create") + require.Nil(t, fsErr) + var createIDs []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + createIDs = append(createIDs, e.Name) + } + } + if len(createIDs) != 1 { + t.Fatalf("expected exactly 1 create log entry after Mkdir, got %d (%v)", len(createIDs), createIDs) + } +} + +// TestWriteFileEnsureDirs_LogsEachIntermediateDir verifies that +// WriteFileEnsureDirs goes through Mkdir for every missing ancestor, +// producing one 'create' log entry per dir plus one for the file -- +// the same shape the kernel would generate via per-segment mkdir(2) +// calls in production. +func TestWriteFileEnsureDirs_LogsEachIntermediateDir(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "ensuredirs") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + require.Nil(t, ops.WriteFile(ctx, "/.build/ensuredirs", []byte("markdown,history\n"))) + time.Sleep(100 * time.Millisecond) + + // Single deep write through the helper. Expect 4 create log entries: + // A, A/B, A/B/C, plus the file itself. + require.Nil(t, ops.WriteFileEnsureDirs(ctx, + "/ensuredirs/A/B/C/x.md", + []byte("---\ntitle: X\n---\nbody x\n"))) + + entries, fsErr := ops.ReadDir(ctx, "/ensuredirs/.log/.by/type/create") + require.Nil(t, fsErr) + var createCount int + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + createCount++ + } + } + if createCount != 4 { + t.Errorf("expected 4 create log entries (3 dirs + 1 file), got %d", createCount) + } + + // File and dirs all exist after the call. + for _, p := range []string{"/ensuredirs/A", "/ensuredirs/A/B", "/ensuredirs/A/B/C", "/ensuredirs/A/B/C/x.md"} { + _, fsErr := ops.Stat(ctx, p) + if fsErr != nil { + t.Errorf("%s should exist after WriteFileEnsureDirs: %v", p, fsErr) + } + } +} + +// TestWriteFileEnsureDirs_PreservesExistingDirs verifies that when some +// ancestors already exist, WriteFileEnsureDirs skips them (Mkdir's +// ErrAlreadyExists is treated as "fine") and only logs creates for the +// dirs it actually had to make. +func TestWriteFileEnsureDirs_PreservesExistingDirs(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "ensuredirs2") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + require.Nil(t, ops.WriteFile(ctx, "/.build/ensuredirs2", []byte("markdown,history\n"))) + time.Sleep(100 * time.Millisecond) + + // Pre-create A and A/B. + require.Nil(t, ops.Mkdir(ctx, "/ensuredirs2/A")) + require.Nil(t, ops.Mkdir(ctx, "/ensuredirs2/A/B")) + + // Now call the helper. It should mkdir only A/B/C and write the file. + require.Nil(t, ops.WriteFileEnsureDirs(ctx, + "/ensuredirs2/A/B/C/x.md", + []byte("---\ntitle: X\n---\nbody x\n"))) + + entries, fsErr := ops.ReadDir(ctx, "/ensuredirs2/.log/.by/type/create") + require.Nil(t, fsErr) + var createCount int + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + createCount++ + } + } + // Expected: 2 from the pre-mkdirs + 1 for A/B/C from the helper + // + 1 for the file = 4 total. + if createCount != 4 { + t.Errorf("expected 4 create log entries total (A, A/B pre-existing + A/B/C + file), got %d", createCount) + } +} + +// --- Single operation undo --- + +func TestUndo_SingleCreate(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undo1") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undo1", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create a file + ops.WriteFile(ctx, "/undo1/hello.md", []byte("---\ntitle: Hello\n---\nOriginal\n")) + require.Nil(t, fsErr) + + // Verify it exists + _, fsErr = ops.Stat(ctx, "/undo1/hello.md") + require.Nil(t, fsErr, "file should exist after creation") + + // Find the log entry for the create + entries, fsErr := ops.ReadDir(ctx, "/undo1/.log/.by/type/create") + require.Nil(t, fsErr) + var logIDs []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + logIDs = append(logIDs, e.Name) + } + } + require.GreaterOrEqual(t, len(logIDs), 1, "should have at least one create log entry") + + // Undo the create -- should delete the file + t.Logf("Undoing log entry: %s", logIDs[len(logIDs)-1]) + undoResult, err := ops.ExecuteUndoSingle(ctx, "public", "undo1", logIDs[len(logIDs)-1]) + require.NoError(t, err) + t.Logf("Undo result: deleted=%d, restored=%d, skipped=%d", + undoResult.FilesDeleted, undoResult.FilesRestored, undoResult.FilesSkipped) + assert.Equal(t, 1, undoResult.FilesDeleted) + + // Verify file is gone via ReadDir (more reliable than Stat for synth views) + entries, fsErr = ops.ReadDir(ctx, "/undo1") + require.Nil(t, fsErr) + var fileNames []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") && strings.HasSuffix(e.Name, ".md") { + fileNames = append(fileNames, e.Name) + } + } + assert.Empty(t, fileNames, "file should be gone after undoing create, but found: %v", fileNames) +} + +func TestUndo_SingleEdit(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undo2") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undo2", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create and then edit + ops.WriteFile(ctx, "/undo2/hello.md", []byte("---\ntitle: Hello\n---\nVersion 1\n")) + ops.WriteFile(ctx, "/undo2/hello.md", []byte("---\ntitle: Hello\n---\nVersion 2\n")) + + // Verify current content + fc, fsErr := ops.ReadFile(ctx, "/undo2/hello.md") + require.Nil(t, fsErr) + assert.Contains(t, string(fc.Data), "Version 2") + + // Find the edit log entry + entries, fsErr := ops.ReadDir(ctx, "/undo2/.log/.by/type/edit") + require.Nil(t, fsErr) + var logIDs []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + logIDs = append(logIDs, e.Name) + } + } + require.GreaterOrEqual(t, len(logIDs), 1) + + // Undo the edit -- should restore Version 1 + undoResult, err := ops.ExecuteUndoSingle(ctx, "public", "undo2", logIDs[len(logIDs)-1]) + require.NoError(t, err) + assert.Equal(t, 1, undoResult.FilesRestored) + + fc, fsErr = ops.ReadFile(ctx, "/undo2/hello.md") + require.Nil(t, fsErr) + assert.Contains(t, string(fc.Data), "Version 1", "should be restored to pre-edit state") +} + +func TestUndo_SingleDelete(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undo3") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undo3", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create then delete + ops.WriteFile(ctx, "/undo3/hello.md", []byte("---\ntitle: Hello\n---\nContent\n")) + ops.Delete(ctx, "/undo3/hello.md") + + // Verify deleted + _, fsErr = ops.Stat(ctx, "/undo3/hello.md") + require.NotNil(t, fsErr, "file should be deleted") + + // Find the delete log entry + entries, fsErr := ops.ReadDir(ctx, "/undo3/.log/.by/type/delete") + require.Nil(t, fsErr) + var logIDs []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + logIDs = append(logIDs, e.Name) + } + } + require.GreaterOrEqual(t, len(logIDs), 1) + + // Undo the delete -- should restore the file + undoResult, err := ops.ExecuteUndoSingle(ctx, "public", "undo3", logIDs[len(logIDs)-1]) + require.NoError(t, err) + assert.Equal(t, 1, undoResult.FilesRestored) + + fc, fsErr := ops.ReadFile(ctx, "/undo3/hello.md") + require.Nil(t, fsErr, "file should be restored after undoing delete") + assert.Contains(t, string(fc.Data), "Content") +} + +// TestUndo_Apply_InvalidatesNegativeStatCache pins down the cache-schema +// bug in writeUndoApply / ExecuteUndo. statSynthFile populates statCache +// under the user's schema (parsed.Context.Schema, e.g. "public"); both +// invalidate calls in the undo path used to pass synth.TigerFSSchema +// instead, leaving cached entries (especially negative entries) in place +// after undo. +// +// Repro: +// 1. Create then delete a file. +// 2. Stat the deleted file -- returns ENOENT and *populates* a negative +// stat-cache entry under the user's schema. +// 3. Undo the delete via .apply. +// 4. Stat the file again. Without the fix, step 3's invalidate runs +// under "tigerfs" and doesn't touch the "public" cache, so step 4 +// returns "(cached)" ENOENT. With the fix, the cache is cleared and +// step 4 sees the restored row. +func TestUndo_Apply_InvalidatesNegativeStatCache(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undocache") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undocache", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create then delete -- the delete log entry is what we'll undo. + require.Nil(t, ops.WriteFile(ctx, "/undocache/hello.md", + []byte("---\ntitle: Hello\n---\nContent\n"))) + require.Nil(t, ops.Delete(ctx, "/undocache/hello.md")) + + // Stat the deleted file. The error is expected and required: this is + // what populates the negative stat-cache entry under "public". + _, fsErr = ops.Stat(ctx, "/undocache/hello.md") + require.NotNil(t, fsErr, "stat must miss (file is deleted)") + require.Equal(t, fs.ErrNotExist, fsErr.Code, "expected ENOENT, got %v", fsErr) + + // Apply undo via the production .apply path. We need the delete log id. + entries, fsErr := ops.ReadDir(ctx, "/undocache/.log/.by/type/delete") + require.Nil(t, fsErr) + var deleteIDs []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + deleteIDs = append(deleteIDs, e.Name) + } + } + require.GreaterOrEqual(t, len(deleteIDs), 1) + fsErr = ops.WriteFile(ctx, "/undocache/.undo/id/"+deleteIDs[len(deleteIDs)-1]+"/.apply", []byte("")) + require.Nil(t, fsErr, "undo .apply should succeed") + + // Stat must now succeed -- the file is back, and the post-undo cache + // invalidation must have cleared the negative entry the earlier Stat + // installed. Without the fix, this returns ErrNotExist with message + // "file not found: hello.md (cached)". + entry, fsErr := ops.Stat(ctx, "/undocache/hello.md") + if fsErr != nil { + t.Fatalf("stat after undo should succeed; got code=%v message=%q "+ + "(stale negative-cache entry from before the undo was not invalidated)", + fsErr.Code, fsErr.Message) + } + if entry.Name != "hello.md" { + t.Errorf("entry.Name = %q, want %q", entry.Name, "hello.md") + } +} + +// --- Multi-file undo to savepoint --- + +// TestUndo_ToSavepoint_DefersFKConstraint verifies that the deferred-FK +// path (SET CONSTRAINTS ALL DEFERRED) applies on the to-savepoint route +// just like it does on to-id. Same scenario as +// TestUndo_ToID_RestoresDeletedDirWithFile but driven via .savepoint. +// +// The file is created at root before the dir, then moved in. file_id < +// dir_id, so when undo restores both rows, the file is UPSERTed first +// (file_id ASC). Its parent_id references the not-yet-restored dir; +// without deferral that violates parent_id_fkey at INSERT time. +func TestUndo_ToSavepoint_DefersFKConstraint(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "spfk") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + require.Nil(t, ops.WriteFile(ctx, "/.build/spfk", []byte("markdown,history\n"))) + time.Sleep(100 * time.Millisecond) + + // 1. File at root first (oldest file_id). + require.Nil(t, ops.WriteFile(ctx, "/spfk/x.md", + []byte("---\ntitle: X\n---\nbody x\n"))) + + // 2. Dir A (newer file_id). + require.Nil(t, ops.Mkdir(ctx, "/spfk/A")) + + // 3. Move x.md into A. file_id stays old; parent_id becomes A.id. + require.Nil(t, ops.Rename(ctx, "/spfk/x.md", "/spfk/A/x.md")) + + // 4. Savepoint here -- captures "x.md lives at A/x.md". + require.Nil(t, ops.WriteFile(ctx, "/spfk/.savepoint/before-deletes.json", + []byte(`{"description":"x.md lives at A/x.md"}`))) + + // 5. Delete file then dir. + require.Nil(t, ops.Delete(ctx, "/spfk/A/x.md")) + require.Nil(t, ops.Delete(ctx, "/spfk/A")) + + // 6. Apply undo via the to-savepoint route. Without deferred FK + // this fails when x.md (older file_id) is UPSERTed before A. + require.Nil(t, ops.WriteFile(ctx, "/spfk/.undo/to-savepoint/before-deletes/.apply", + []byte(""))) + + // 7. State at savepoint should be restored: x.md inside A. + fc, fsErr := ops.ReadFile(ctx, "/spfk/A/x.md") + require.Nil(t, fsErr, "x.md should be restored under A") + assert.Contains(t, string(fc.Data), "body x") + + entries, fsErr := ops.ReadDir(ctx, "/spfk/A") + require.Nil(t, fsErr, "A should be readable after undo") + var names []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + names = append(names, e.Name) + } + } + assert.Contains(t, names, "x.md") +} + +func TestUndo_ToSavepoint_MultiFile(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undomulti") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undomulti", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create initial files + ops.WriteFile(ctx, "/undomulti/existing.md", []byte("---\ntitle: Existing\n---\nOriginal content\n")) + + // Create savepoint + ops.WriteFile(ctx, "/undomulti/.savepoint/checkpoint.json", []byte("{}")) + time.Sleep(50 * time.Millisecond) + + // After savepoint: create A, edit existing, delete existing + ops.WriteFile(ctx, "/undomulti/new-file.md", []byte("---\ntitle: New\n---\nNew content\n")) + ops.WriteFile(ctx, "/undomulti/existing.md", []byte("---\ntitle: Existing\n---\nModified content\n")) + + // Verify state before undo + _, fsErr = ops.Stat(ctx, "/undomulti/new-file.md") + require.Nil(t, fsErr, "new-file should exist") + fc, fsErr := ops.ReadFile(ctx, "/undomulti/existing.md") + require.Nil(t, fsErr) + assert.Contains(t, string(fc.Data), "Modified content") + + // Undo to savepoint + undoResult, err := ops.ExecuteUndoToSavepoint(ctx, "public", "undomulti", "checkpoint", nil) + require.NoError(t, err) + t.Logf("Undo result: deleted=%d, restored=%d, skipped=%d", + undoResult.FilesDeleted, undoResult.FilesRestored, undoResult.FilesSkipped) + + // new-file should be gone (created after savepoint) + // Use ReadDir instead of Stat -- synth path cache may be stale + dirEntries, fsErr := ops.ReadDir(ctx, "/undomulti") + require.Nil(t, fsErr) + var mdFiles []string + for _, e := range dirEntries { + if strings.HasSuffix(e.Name, ".md") { + mdFiles = append(mdFiles, e.Name) + } + } + assert.NotContains(t, mdFiles, "new-file.md", "new-file should be deleted after undo") + + // existing should be restored to original content + fc, fsErr = ops.ReadFile(ctx, "/undomulti/existing.md") + require.Nil(t, fsErr) + assert.Contains(t, string(fc.Data), "Original content", "existing should be restored") +} + +// TestUndo_LogReadFreshAfterSavepoint pins down a staleness bug observed +// in stress-test runs: after a heavy undo_to_savepoint, an *immediate* +// read of /.log/.last/N/.export/json sometimes returns a snapshot of the +// log table from before the undo's new log rows were committed. +// Empirically the kernel-level NFS path triggers it in ~3/500 iters +// with recovery taking 100-500ms. +// +// This test bypasses NFS entirely (calls ops.ReadFile directly), so a +// failure here would mean the staleness lives in the TigerFS query/ +// cache path. A passing test means the issue is downstream of ops -- +// likely NFS attribute/data cache or kernel-side file-handle reuse. +// +// Setup is shaped to match the stress-test pattern that triggers it +// most reliably: ~100 mixed ops on multiple files (so the undo has +// real work to do and produces several new log rows), then one +// undo_to_savepoint, then immediately read the log export and assert +// the newest log_id is > all log_ids that existed before the undo. +func TestUndo_LogReadFreshAfterSavepoint(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "logfresh") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + require.Nil(t, ops.WriteFile(ctx, "/.build/logfresh", []byte("markdown,history\n"))) + time.Sleep(100 * time.Millisecond) + + // Savepoint at iter 0 -- the undo will roll back everything below. + require.Nil(t, ops.WriteFile(ctx, "/logfresh/.savepoint/sp.json", []byte("{}"))) + time.Sleep(50 * time.Millisecond) + + // 100 ops mixing creates and edits across several files. Sized to + // produce enough log rows that an undo has substantial work, which + // is what the stress-test data shows as the trigger condition. + for i := 0; i < 50; i++ { + path := fmt.Sprintf("/logfresh/file-%02d.md", i%5) + body := fmt.Sprintf("---\ntitle: F%d\n---\nrev %d\n", i%5, i) + require.Nil(t, ops.WriteFile(ctx, path, []byte(body)), + "write iter %d", i) + } + + // Capture the log_id of the newest pre-undo entry. After the undo, + // the *post-undo* newest log_id (the last undo log row) MUST be > this. + preUndoNewestID := readNewestLogID(t, ctx, ops, "/logfresh") + require.NotEmpty(t, preUndoNewestID) + + // Run the heavy undo. + _, err := ops.ExecuteUndoToSavepoint(ctx, "public", "logfresh", "sp", nil) + require.NoError(t, err) + + // Read the log export immediately, with a tight retry budget so we + // can tell freshness from eventual consistency. Each retry is a + // fresh ops.ReadFile call. + var newestAfterUndo string + for attempt := 1; attempt <= 20; attempt++ { + newestAfterUndo = readNewestLogID(t, ctx, ops, "/logfresh") + if newestAfterUndo > preUndoNewestID { + t.Logf("read fresh on attempt %d (~%dms)", attempt, (attempt-1)*25) + break + } + time.Sleep(25 * time.Millisecond) + } + + require.Greater(t, newestAfterUndo, preUndoNewestID, + "after undo_to_savepoint, /.log/.last/N must reflect at least one new log_id; "+ + "read %q vs pre-undo newest %q (staleness in ops layer would mean TigerFS-side cache, not NFS)", + newestAfterUndo, preUndoNewestID) +} + +// readNewestLogID reads /.log/.last/1/.export/json and returns the +// log_id of the single returned entry (which is the table's newest by +// the LimitLast ORDER BY). Returns "" if the read fails or the JSON +// has no entries. +func readNewestLogID(t *testing.T, ctx context.Context, ops *fs.Operations, wsRoot string) string { + t.Helper() + fc, fsErr := ops.ReadFile(ctx, wsRoot+"/.log/.last/1/.export/json") + if fsErr != nil { + return "" + } + var entries []struct { + LogID string `json:"log_id"` + } + if err := json.Unmarshal(fc.Data, &entries); err != nil || len(entries) == 0 { + return "" + } + return entries[0].LogID +} + +// TestUndo_LogReadFreshAfterSavepoint_ViaNFSAdapter is the second +// boundary test in the iter-107 staleness investigation. It mirrors +// TestUndo_LogReadFreshAfterSavepoint exactly, but wraps the same +// fs.Operations in an in-process *nfs.OpsFilesystem and reads the log +// export through OpenFile + io.ReadAll instead of ops.ReadFile. This +// exercises the NFS adapter path (file-cache lookup, OpenFile-per-read +// pattern, billy.File semantics) without involving go-nfs RPC handling +// or the kernel client. +// +// Combined with the ops-only sibling test, this triangulates where the +// staleness lives: +// +// - ops.ReadFile fresh (already proven by sibling test) +// - OpsFilesystem fresh (this test) --> bug is in go-nfs / kernel +// - OpsFilesystem stale (this test) --> bug is in OpsFilesystem +// +// Either outcome is a 2x reduction of the hypothesis space. +func TestUndo_LogReadFreshAfterSavepoint_ViaNFSAdapter(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "logfresh2") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + // Wrap the same ops in the production NFS adapter. The adapter has + // its own file cache (cachedFile) and OpenFile/Close lifecycle that + // could conceivably hold stale state across opens. + nfsFS := nfs.NewOpsFilesystem(ops, &config.Config{ + DirListingLimit: 10000, + QueryTimeout: 30, + }) + defer nfsFS.Close() + + require.Nil(t, ops.WriteFile(ctx, "/.build/logfresh2", []byte("markdown,history\n"))) + time.Sleep(100 * time.Millisecond) + + require.Nil(t, ops.WriteFile(ctx, "/logfresh2/.savepoint/sp.json", []byte("{}"))) + time.Sleep(50 * time.Millisecond) + + // Same workload shape as the ops-only test: 50 ops across 5 files + // to give the savepoint undo real work. + for i := 0; i < 50; i++ { + path := fmt.Sprintf("/logfresh2/file-%02d.md", i%5) + body := fmt.Sprintf("---\ntitle: F%d\n---\nrev %d\n", i%5, i) + require.Nil(t, ops.WriteFile(ctx, path, []byte(body)), + "write iter %d", i) + } + + preUndoNewestID := readNewestLogIDViaNFS(t, nfsFS, "/logfresh2") + require.NotEmpty(t, preUndoNewestID) + + _, err := ops.ExecuteUndoToSavepoint(ctx, "public", "logfresh2", "sp", nil) + require.NoError(t, err) + + // Same retry budget as the ops sibling. If the NFS adapter + // introduces staleness, we should see attempts > 1 here while the + // ops sibling consistently hits attempt 1. + var newestAfterUndo string + for attempt := 1; attempt <= 20; attempt++ { + newestAfterUndo = readNewestLogIDViaNFS(t, nfsFS, "/logfresh2") + if newestAfterUndo > preUndoNewestID { + t.Logf("read fresh on attempt %d (~%dms)", attempt, (attempt-1)*25) + break + } + time.Sleep(25 * time.Millisecond) + } + + require.Greater(t, newestAfterUndo, preUndoNewestID, + "after undo_to_savepoint, /.log/.last/N read via OpsFilesystem must reflect "+ + "at least one new log_id. read=%q pre-undo-newest=%q. "+ + "If THIS test is the one that flakes (ops sibling stays clean), "+ + "the staleness lives in OpsFilesystem; otherwise it's in go-nfs/kernel.", + newestAfterUndo, preUndoNewestID) +} + +// readNewestLogIDViaNFS reads through the NFS adapter using its +// production OpenFile/Read/Close lifecycle, not ops.ReadFile. +func readNewestLogIDViaNFS(t *testing.T, nfsFS *nfs.OpsFilesystem, wsRoot string) string { + t.Helper() + file, err := nfsFS.OpenFile(wsRoot+"/.log/.last/1/.export/json", 0, 0) + if err != nil { + return "" + } + defer file.Close() + data, err := io.ReadAll(file) + if err != nil { + return "" + } + var entries []struct { + LogID string `json:"log_id"` + } + if err := json.Unmarshal(data, &entries); err != nil || len(entries) == 0 { + return "" + } + return entries[0].LogID +} + +// --- Undo by user --- + +func TestUndo_ByUser(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undouser") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undouser", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create initial files + ops.WriteFile(ctx, "/undouser/shared.md", []byte("---\ntitle: Shared\n---\nOriginal\n")) + + // Create savepoint + ops.WriteFile(ctx, "/undouser/.savepoint/before-edits.json", []byte("{}")) + time.Sleep(50 * time.Millisecond) + + // agent-7 edits shared.md + ops.SetUserID("agent-7") + ops.WriteFile(ctx, "/undouser/shared.md", []byte("---\ntitle: Shared\n---\nEdited by agent-7\n")) + + // agent-9 creates a new file + ops.SetUserID("agent-9") + ops.WriteFile(ctx, "/undouser/agent9-file.md", []byte("---\ntitle: Agent9\n---\nBy agent-9\n")) + + // Undo only agent-7's changes + ops.SetUserID("agent-7") + filters := []db.UndoFilter{{Column: "user_id", Value: "agent-7"}} + undoResult, err := ops.ExecuteUndoToSavepoint(ctx, "public", "undouser", "before-edits", filters) + require.NoError(t, err) + t.Logf("Undo by user result: deleted=%d, restored=%d, skipped=%d", + undoResult.FilesDeleted, undoResult.FilesRestored, undoResult.FilesSkipped) + + // shared.md should be restored to original (agent-7's edit undone) + fc, fsErr := ops.ReadFile(ctx, "/undouser/shared.md") + require.Nil(t, fsErr) + assert.Contains(t, string(fc.Data), "Original", "agent-7's edit should be undone") + + // agent-9's file should still exist (not affected by agent-7 undo) + _, fsErr = ops.Stat(ctx, "/undouser/agent9-file.md") + assert.Nil(t, fsErr, "agent-9's file should be preserved") +} + +// --- Undo of undo (ADR Section 3.4) --- + +func TestUndo_UndoOfUndo(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undoundo") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undoundo", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Step 1: Create with v1 + ops.WriteFile(ctx, "/undoundo/hello.md", []byte("---\ntitle: Hello\n---\nVersion 1\n")) + + // Step 2: Edit to v2 + ops.WriteFile(ctx, "/undoundo/hello.md", []byte("---\ntitle: Hello\n---\nVersion 2\n")) + + // Find the edit log entry (L1) + entries, fsErr := ops.ReadDir(ctx, "/undoundo/.log/.by/type/edit") + require.Nil(t, fsErr) + var editLogIDs []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + editLogIDs = append(editLogIDs, e.Name) + } + } + require.GreaterOrEqual(t, len(editLogIDs), 1) + editLogID := editLogIDs[len(editLogIDs)-1] + + // Step 3: Undo the edit (restores v1) + _, err := ops.ExecuteUndoSingle(ctx, "public", "undoundo", editLogID) + require.NoError(t, err) + + fc, fsErr := ops.ReadFile(ctx, "/undoundo/hello.md") + require.Nil(t, fsErr) + assert.Contains(t, string(fc.Data), "Version 1", "after first undo, should be v1") + + // Find the undo log entry (L2) + entries, fsErr = ops.ReadDir(ctx, "/undoundo/.log/.by/type/undo") + require.Nil(t, fsErr) + var undoLogIDs []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + undoLogIDs = append(undoLogIDs, e.Name) + } + } + require.GreaterOrEqual(t, len(undoLogIDs), 1) + undoLogID := undoLogIDs[len(undoLogIDs)-1] + + // Step 4: Undo the undo (restores v2) + _, err = ops.ExecuteUndoSingle(ctx, "public", "undoundo", undoLogID) + require.NoError(t, err) + + fc, fsErr = ops.ReadFile(ctx, "/undoundo/hello.md") + require.Nil(t, fsErr) + assert.Contains(t, string(fc.Data), "Version 2", "after undo-of-undo, should be v2") +} + +// --- Idempotent undo to savepoint (ADR Section 3.5) --- + +func TestUndo_Idempotent(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undoidem") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undoidem", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create initial state + ops.WriteFile(ctx, "/undoidem/hello.md", []byte("---\ntitle: Hello\n---\nOriginal\n")) + ops.WriteFile(ctx, "/undoidem/.savepoint/checkpoint.json", []byte("{}")) + time.Sleep(50 * time.Millisecond) + + // Modify after savepoint + ops.WriteFile(ctx, "/undoidem/hello.md", []byte("---\ntitle: Hello\n---\nModified\n")) + + // Undo to savepoint -- first time + _, err := ops.ExecuteUndoToSavepoint(ctx, "public", "undoidem", "checkpoint", nil) + require.NoError(t, err) + + fc, fsErr := ops.ReadFile(ctx, "/undoidem/hello.md") + require.Nil(t, fsErr) + assert.Contains(t, string(fc.Data), "Original") + + // Undo to same savepoint again -- should produce same data state + _, err = ops.ExecuteUndoToSavepoint(ctx, "public", "undoidem", "checkpoint", nil) + require.NoError(t, err) + + fc, fsErr = ops.ReadFile(ctx, "/undoidem/hello.md") + require.Nil(t, fsErr) + assert.Contains(t, string(fc.Data), "Original", "idempotent: same result on second undo") +} + +// --- Undo with no operations after target --- + +func TestUndo_NoOpsAfterSavepoint(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undonoop") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undonoop", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + ops.WriteFile(ctx, "/undonoop/hello.md", []byte("---\ntitle: Hello\n---\nContent\n")) + ops.WriteFile(ctx, "/undonoop/.savepoint/latest.json", []byte("{}")) + + // No operations after savepoint -- undo should be a no-op + undoResult, err := ops.ExecuteUndoToSavepoint(ctx, "public", "undonoop", "latest", nil) + require.NoError(t, err) + assert.Equal(t, 0, undoResult.FilesDeleted) + assert.Equal(t, 0, undoResult.FilesRestored) + assert.Equal(t, 0, undoResult.FilesSkipped) + + // File should still be there unchanged + fc, fsErr := ops.ReadFile(ctx, "/undonoop/hello.md") + require.Nil(t, fsErr) + assert.Contains(t, string(fc.Data), "Content") +} + +// --- Undo log entries verification --- + +func TestUndo_CreatesUndoLogEntries(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undolog") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undolog", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + ops.WriteFile(ctx, "/undolog/hello.md", []byte("---\ntitle: Hello\n---\nV1\n")) + ops.WriteFile(ctx, "/undolog/.savepoint/sp1.json", []byte("{}")) + time.Sleep(50 * time.Millisecond) + ops.WriteFile(ctx, "/undolog/hello.md", []byte("---\ntitle: Hello\n---\nV2\n")) + + // Count undo log entries before + entries, fsErr := ops.ReadDir(ctx, "/undolog/.log/.by/type/undo") + undoBefore := 0 + if fsErr == nil { + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + undoBefore++ + } + } + } + + // Undo to savepoint + _, err := ops.ExecuteUndoToSavepoint(ctx, "public", "undolog", "sp1", nil) + require.NoError(t, err) + + // Count undo log entries after + entries, fsErr = ops.ReadDir(ctx, "/undolog/.log/.by/type/undo") + require.Nil(t, fsErr) + undoAfter := 0 + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + undoAfter++ + } + } + assert.Greater(t, undoAfter, undoBefore, "undo should create new log entries with type='undo'") +} + +// --- Filtered undo --- + +func TestUndo_FilterByType(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undofilt") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undofilt", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Create two files, savepoint, then delete one and edit the other + ops.WriteFile(ctx, "/undofilt/keep.md", []byte("---\ntitle: Keep\n---\nKeep this\n")) + ops.WriteFile(ctx, "/undofilt/restore-me.md", []byte("---\ntitle: Restore\n---\nRestore this\n")) + ops.WriteFile(ctx, "/undofilt/.savepoint/sp1.json", []byte("{}")) + time.Sleep(50 * time.Millisecond) + + ops.WriteFile(ctx, "/undofilt/keep.md", []byte("---\ntitle: Keep\n---\nEdited\n")) + ops.Delete(ctx, "/undofilt/restore-me.md") + + // Undo only deletes (not edits) + filters := []db.UndoFilter{{Column: "type", Value: "delete"}} + undoResult, err := ops.ExecuteUndoToSavepoint(ctx, "public", "undofilt", "sp1", filters) + require.NoError(t, err) + t.Logf("Filtered undo: deleted=%d, restored=%d, skipped=%d", + undoResult.FilesDeleted, undoResult.FilesRestored, undoResult.FilesSkipped) + + // restore-me.md should be back + _, fsErr = ops.Stat(ctx, "/undofilt/restore-me.md") + assert.Nil(t, fsErr, "deleted file should be restored") + + // keep.md should still have the edited content (edit was not undone) + fc, fsErr := ops.ReadFile(ctx, "/undofilt/keep.md") + require.Nil(t, fsErr) + assert.Contains(t, string(fc.Data), "Edited", "edit should NOT be undone by type=delete filter") +} + +// Note: Rename undo (parent_id restoration) requires the NFS adapter's Rename method, +// which is not exposed on fs.Operations. Tested via mount-based integration tests. + +// --- .undo/ filesystem interface tests --- + +func TestUndo_Interface_ReadDirRoot(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undoui") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undoui", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // ReadDir .undo/ should list id/, to-id/, to-savepoint/ + entries, fsErr := ops.ReadDir(ctx, "/undoui/.undo") + require.Nil(t, fsErr) + var names []string + for _, e := range entries { + names = append(names, e.Name) + } + assert.Contains(t, names, "id") + assert.Contains(t, names, "to-id") + assert.Contains(t, names, "to-savepoint") +} + +func TestUndo_Interface_ReadDirSavepoints(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undoui2") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undoui2", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + ops.WriteFile(ctx, "/undoui2/.savepoint/sp1.json", []byte("{}")) + ops.WriteFile(ctx, "/undoui2/.savepoint/sp2.json", []byte("{}")) + + // ReadDir .undo/to-savepoint/ should list savepoints + entries, fsErr := ops.ReadDir(ctx, "/undoui2/.undo/to-savepoint") + require.Nil(t, fsErr) + var names []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + names = append(names, e.Name) + } + } + assert.Contains(t, names, "sp1") + assert.Contains(t, names, "sp2") +} + +func TestUndo_Interface_Summary(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undoui3") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undoui3", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + ops.WriteFile(ctx, "/undoui3/existing.md", []byte("---\ntitle: Existing\n---\nOriginal\n")) + ops.WriteFile(ctx, "/undoui3/.savepoint/checkpoint.json", []byte("{}")) + time.Sleep(50 * time.Millisecond) + + ops.WriteFile(ctx, "/undoui3/new-file.md", []byte("---\ntitle: New\n---\nNew content\n")) + ops.WriteFile(ctx, "/undoui3/existing.md", []byte("---\ntitle: Existing\n---\nModified\n")) + + // Read .info/summary (TSV) + fc, fsErr := ops.ReadFile(ctx, "/undoui3/.undo/to-savepoint/checkpoint/.info/summary") + require.Nil(t, fsErr, "ReadFile .info/summary should succeed") + summary := string(fc.Data) + t.Logf("Summary TSV:\n%s", summary) + + // Metadata headers + assert.Contains(t, summary, "# savepoint: checkpoint") + assert.Contains(t, summary, "# affected: 2 files") + + // Column header comment + assert.Contains(t, summary, "# type\tfilename\tuser\ttimestamp") + + // Data rows with operation type (not "restore"/"delete") + assert.Contains(t, summary, "create\tnew-file.md") + assert.Contains(t, summary, "edit\texisting.md") + + // Timestamps should be present (RFC3339 format) + assert.Contains(t, summary, "202") // year prefix in timestamp +} + +func TestUndo_Interface_SummaryJSON(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undojs") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undojs", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + ops.WriteFile(ctx, "/undojs/hello.md", []byte("---\ntitle: Hello\n---\nV1\n")) + ops.WriteFile(ctx, "/undojs/.savepoint/sp1.tsv", []byte("description\nTest savepoint\n")) + time.Sleep(50 * time.Millisecond) + ops.WriteFile(ctx, "/undojs/hello.md", []byte("---\ntitle: Hello\n---\nV2\n")) + + fc, fsErr := ops.ReadFile(ctx, "/undojs/.undo/to-savepoint/sp1/.info/summary.json") + require.Nil(t, fsErr, "ReadFile summary.json should succeed") + summary := string(fc.Data) + t.Logf("Summary JSON:\n%s", summary) + + // Structured fields + assert.Contains(t, summary, `"savepoint"`) + assert.Contains(t, summary, `"sp1"`) + assert.Contains(t, summary, `"description"`) + assert.Contains(t, summary, `"Test savepoint"`) + assert.Contains(t, summary, `"affected"`) + assert.Contains(t, summary, `"files"`) + assert.Contains(t, summary, `"type"`) + assert.Contains(t, summary, `"edit"`) + assert.Contains(t, summary, `"hello.md"`) +} + +func TestUndo_Interface_SummaryCSV(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undocsv") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undocsv", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + ops.WriteFile(ctx, "/undocsv/hello.md", []byte("---\ntitle: Hello\n---\nV1\n")) + ops.WriteFile(ctx, "/undocsv/.savepoint/sp1.json", []byte("{}")) + time.Sleep(50 * time.Millisecond) + ops.WriteFile(ctx, "/undocsv/hello.md", []byte("---\ntitle: Hello\n---\nV2\n")) + + fc, fsErr := ops.ReadFile(ctx, "/undocsv/.undo/to-savepoint/sp1/.info/summary.csv") + require.Nil(t, fsErr, "ReadFile summary.csv should succeed") + summary := string(fc.Data) + t.Logf("Summary CSV:\n%s", summary) + + // Header row (no # comments in CSV) + assert.True(t, strings.HasPrefix(summary, "type,filename,user,timestamp\n"), + "CSV should start with header row") + // No metadata comments + assert.NotContains(t, summary, "#") + // Data row + assert.Contains(t, summary, "edit,hello.md,") +} + +func TestUndo_Interface_SummaryToID(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undotid") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undotid", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + ops.WriteFile(ctx, "/undotid/hello.md", []byte("---\ntitle: Hello\n---\nV1\n")) + ops.WriteFile(ctx, "/undotid/hello.md", []byte("---\ntitle: Hello\n---\nV2\n")) + + // Get a log entry to use as to-id target + entries, fsErr := ops.ReadDir(ctx, "/undotid/.log/.first/1") + require.Nil(t, fsErr) + var logIDs []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + logIDs = append(logIDs, e.Name) + } + } + require.GreaterOrEqual(t, len(logIDs), 1) + + fc, fsErr := ops.ReadFile(ctx, "/undotid/.undo/to-id/"+logIDs[0]+"/.info/summary") + require.Nil(t, fsErr, "ReadFile to-id summary should succeed") + summary := string(fc.Data) + t.Logf("Summary to-id:\n%s", summary) + + // to-id mode should show # target: instead of # savepoint: + assert.Contains(t, summary, "# target:") + assert.NotContains(t, summary, "# savepoint:") + assert.Contains(t, summary, "# affected:") +} + +func TestUndo_Interface_SummaryWithUserID(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undouid") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undouid", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + ops.SetUserID("agent-7") + ops.WriteFile(ctx, "/undouid/hello.md", []byte("---\ntitle: Hello\n---\nV1\n")) + ops.WriteFile(ctx, "/undouid/.savepoint/sp1.json", []byte("{}")) + time.Sleep(50 * time.Millisecond) + ops.WriteFile(ctx, "/undouid/hello.md", []byte("---\ntitle: Hello\n---\nV2\n")) + + fc, fsErr := ops.ReadFile(ctx, "/undouid/.undo/to-savepoint/sp1/.info/summary") + require.Nil(t, fsErr) + summary := string(fc.Data) + t.Logf("Summary with user:\n%s", summary) + + // Per-file user column should show agent-7 + assert.Contains(t, summary, "agent-7") +} + +func TestUndo_Interface_PreviewContent(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undoui4") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undoui4", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + ops.WriteFile(ctx, "/undoui4/hello.md", []byte("---\ntitle: Hello\n---\nVersion 1\n")) + ops.WriteFile(ctx, "/undoui4/.savepoint/checkpoint.json", []byte("{}")) + time.Sleep(50 * time.Millisecond) + ops.WriteFile(ctx, "/undoui4/hello.md", []byte("---\ntitle: Hello\n---\nVersion 2\n")) + + // Preview should show Version 1 (the before-state) + fc, fsErr := ops.ReadFile(ctx, "/undoui4/.undo/to-savepoint/checkpoint/hello.md") + require.Nil(t, fsErr, "ReadFile preview should succeed") + assert.Contains(t, string(fc.Data), "Version 1", "preview should show before-state") + assert.NotContains(t, string(fc.Data), "Version 2") +} + +func TestUndo_Interface_ApplyViaSavepoint(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undoui5") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undoui5", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + ops.WriteFile(ctx, "/undoui5/hello.md", []byte("---\ntitle: Hello\n---\nOriginal\n")) + ops.WriteFile(ctx, "/undoui5/.savepoint/checkpoint.json", []byte("{}")) + time.Sleep(50 * time.Millisecond) + ops.WriteFile(ctx, "/undoui5/hello.md", []byte("---\ntitle: Hello\n---\nModified\n")) + + // Apply undo via .apply + fsErr = ops.WriteFile(ctx, "/undoui5/.undo/to-savepoint/checkpoint/.apply", []byte("")) + require.Nil(t, fsErr, "WriteFile .apply should succeed") + + // Verify content restored + fc, fsErr := ops.ReadFile(ctx, "/undoui5/hello.md") + require.Nil(t, fsErr) + assert.Contains(t, string(fc.Data), "Original", "content should be restored after .apply") +} + +func TestUndo_Interface_ApplyViaID(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undoui6") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undoui6", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + ops.WriteFile(ctx, "/undoui6/hello.md", []byte("---\ntitle: Hello\n---\nV1\n")) + ops.WriteFile(ctx, "/undoui6/hello.md", []byte("---\ntitle: Hello\n---\nV2\n")) + + // Find the edit log entry + entries, fsErr := ops.ReadDir(ctx, "/undoui6/.log/.by/type/edit") + require.Nil(t, fsErr) + var logIDs []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + logIDs = append(logIDs, e.Name) + } + } + require.GreaterOrEqual(t, len(logIDs), 1) + + // Apply single undo via .undo/id//.apply + fsErr = ops.WriteFile(ctx, "/undoui6/.undo/id/"+logIDs[len(logIDs)-1]+"/.apply", []byte("")) + require.Nil(t, fsErr, "WriteFile .apply via id should succeed") + + fc, fsErr := ops.ReadFile(ctx, "/undoui6/hello.md") + require.Nil(t, fsErr) + assert.Contains(t, string(fc.Data), "V1", "should be restored to V1") +} + +// TestUndo_Interface_ApplyViaToID_DisplayName verifies that the production +// .apply path accepts a display-name log_id (the format users see in +// .log/.by/... listings), not just a raw UUIDv7. ExecuteUndoSingle has +// resolved display names since inception; ExecuteUndoToLogID was missing +// the same call, so any user who copied a log id from `ls .log/.by/...` +// into `.undo/to-id//.apply` would silently get FilesRestored=0 +// (pgx encodes the string as text, the implicit cast on log_id::text turns +// the WHERE log_id > $1 comparison lexicographic, and UUID texts beginning +// with '0' always sort below display names beginning with '2'). +func TestUndo_Interface_ApplyViaToID_DisplayName(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undotidd") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undotidd", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + require.Nil(t, ops.WriteFile(ctx, "/undotidd/hello.md", []byte("---\ntitle: Hello\n---\nV1\n"))) + require.Nil(t, ops.WriteFile(ctx, "/undotidd/hello.md", []byte("---\ntitle: Hello\n---\nV2\n"))) + + // Capture the create log entry's display-name id from .log/.by/type/create + // (this is the canonical UI surface; display names always sort by time). + entries, fsErr := ops.ReadDir(ctx, "/undotidd/.log/.by/type/create") + require.Nil(t, fsErr) + var displayNames []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + displayNames = append(displayNames, e.Name) + } + } + require.GreaterOrEqual(t, len(displayNames), 1) + target := displayNames[len(displayNames)-1] + if !strings.HasPrefix(target, "20") { + t.Fatalf("expected display-name format (e.g. 2026-...) but got %q -- "+ + "this test is meaningless if .log/ already returns raw UUIDs", target) + } + + // Apply undo via .undo/to-id//.apply -- the path the + // user would naturally construct from a directory listing. + fsErr = ops.WriteFile(ctx, "/undotidd/.undo/to-id/"+target+"/.apply", []byte("")) + require.Nil(t, fsErr, "WriteFile .apply via to-id (display name) should succeed") + + // V2's edit is the only op after the create. Undoing back to the create + // must roll the edit back, leaving V1 in the file. + fc, fsErr := ops.ReadFile(ctx, "/undotidd/hello.md") + require.Nil(t, fsErr) + assert.Contains(t, string(fc.Data), "V1", + "undo_to_id with display-name target must restore V1; if file is still V2 the path silently no-op'd") +} + +// TestUndo_ToID_LargeEditOnly verifies that undo_to_id targeting a create +// log_id rolls back ONLY the subsequent edit, leaving the file at its +// post-create state. The body is 10 MB so the version-history restoration +// path is exercised at non-trivial size. +// +// At this layer each ops.WriteFile produces one log entry (1 create + 1 +// edit). NFS-driven multi-chunk fan-out -- where a single user-level +// create produces 1 + ceil(size/wsize) log entries -- lives a layer above +// and is exercised by the stress test under test/stress/. The DB-level +// undo path is identical in both cases. +func TestUndo_ToID_LargeEditOnly(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undolarge") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + require.Nil(t, ops.WriteFile(ctx, "/.build/undolarge", []byte("markdown,history\n"))) + time.Sleep(100 * time.Millisecond) + + // 10 MB body filled with a deterministic non-uniform pattern so the + // post-undo readback can't accidentally match via zero-fill. + createBody := make([]byte, 10*1024*1024) + for i := range createBody { + createBody[i] = 'A' + byte(i%26) + } + createContent := append([]byte("---\ntitle: BigCreate\n---\n"), createBody...) + require.Nil(t, ops.WriteFileEnsureDirs(ctx, "/undolarge/dir/big.md", createContent)) + + // Capture the create log_id of the file (last create entry; the dir + // create from WriteFileEnsureDirs is chronologically earlier). + entries, fsErr := ops.ReadDir(ctx, "/undolarge/.log/.by/type/create") + require.Nil(t, fsErr) + var createIDs []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + createIDs = append(createIDs, e.Name) + } + } + require.GreaterOrEqual(t, len(createIDs), 1) + createLogID := createIDs[len(createIDs)-1] + + // Edit with distinguishable, smaller body so length checks can tell + // post-undo (10 MB create) from pre-undo (5 MB edit). + editBody := make([]byte, 5*1024*1024) + for i := range editBody { + editBody[i] = 'a' + byte(i%26) + } + editContent := append([]byte("---\ntitle: BigEdit\n---\n"), editBody...) + require.Nil(t, ops.WriteFile(ctx, "/undolarge/dir/big.md", editContent)) + + // Sanity: the edit took effect before we undo it. + fc, fsErr := ops.ReadFile(ctx, "/undolarge/dir/big.md") + require.Nil(t, fsErr) + require.Contains(t, string(fc.Data), "BigEdit", + "pre-undo body should reflect the edit, not the create") + + // undo_to_id targeting the create's log_id must undo only the edit, + // keeping the file present in its post-create state. + fsErr = ops.WriteFile(ctx, "/undolarge/.undo/to-id/"+createLogID+"/.apply", []byte("")) + require.Nil(t, fsErr, "WriteFile .apply via to-id should succeed") + + fc, fsErr = ops.ReadFile(ctx, "/undolarge/dir/big.md") + require.Nil(t, fsErr, "file must still exist after undoing only the edit") + assert.Contains(t, string(fc.Data), "BigCreate", + "post-undo body must contain the create-time title (edit was undone)") + assert.NotContains(t, string(fc.Data), "BigEdit", + "post-undo body must NOT contain edit content") + // 10 MB create vs 5 MB edit: bracket the size to disambiguate. + assert.Greater(t, len(fc.Data), 9*1024*1024, + "post-undo body should be ~10MB (create), not ~5MB (edit)") +} + +func TestUndo_Interface_ApplyNoOp(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undoui7") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undoui7", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + ops.WriteFile(ctx, "/undoui7/hello.md", []byte("---\ntitle: Hello\n---\nContent\n")) + ops.WriteFile(ctx, "/undoui7/.savepoint/latest.json", []byte("{}")) + + // .apply on empty set should succeed (no-op) + fsErr = ops.WriteFile(ctx, "/undoui7/.undo/to-savepoint/latest/.apply", []byte("")) + require.Nil(t, fsErr, ".apply on empty set should be a no-op") +} + +func TestUndo_Interface_InvalidTarget(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undoui8") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undoui8", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // .apply with invalid savepoint should fail + fsErr = ops.WriteFile(ctx, "/undoui8/.undo/to-savepoint/nonexistent/.apply", []byte("")) + require.NotNil(t, fsErr, ".apply with invalid savepoint should fail") +} + +// --- Target validation tests (cd into nonexistent targets should fail) --- + +func TestUndo_Interface_StatInvalidSavepoint(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undoval1") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undoval1", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Stat on nonexistent savepoint should return ENOENT + _, fsErr = ops.Stat(ctx, "/undoval1/.undo/to-savepoint/nonexistent") + require.NotNil(t, fsErr, "Stat on nonexistent savepoint should fail") + assert.Equal(t, fs.ErrNotExist, fsErr.Code) +} + +func TestUndo_Interface_StatInvalidLogID(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undoval2") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undoval2", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Stat on nonexistent log_id in id/ should return ENOENT + _, fsErr = ops.Stat(ctx, "/undoval2/.undo/id/nonexistent") + require.NotNil(t, fsErr, "Stat on nonexistent log_id should fail") + assert.Equal(t, fs.ErrNotExist, fsErr.Code) +} + +func TestUndo_Interface_StatInvalidToID(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undoval3") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undoval3", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Stat on nonexistent log_id in to-id/ should return ENOENT + _, fsErr = ops.Stat(ctx, "/undoval3/.undo/to-id/nonexistent") + require.NotNil(t, fsErr, "Stat on nonexistent to-id target should fail") + assert.Equal(t, fs.ErrNotExist, fsErr.Code) +} + +func TestUndo_Interface_StatValidSavepoint(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undoval4") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undoval4", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + ops.WriteFile(ctx, "/undoval4/.savepoint/valid-sp.json", []byte("{}")) + + // Stat on existing savepoint should succeed + entry, fsErr := ops.Stat(ctx, "/undoval4/.undo/to-savepoint/valid-sp") + require.Nil(t, fsErr, "Stat on existing savepoint should succeed") + assert.True(t, entry.IsDir) + assert.Equal(t, "valid-sp", entry.Name) +} + +// TestUndo_ToID_RestoresDeletedDirWithFile exercises the deferred-constraint +// path documented in ADR-017. Without SET CONSTRAINTS ALL DEFERRED in +// ExecuteUndoTransaction, restoring a child file row whose parent_id points +// to a directory row that hasn't been UPSERTed yet trips parent_id_fkey +// (SQLSTATE 23503) and aborts the undo with EIO. This was the failure +// reported by the stress test seed 1777065886566418000. +// +// To force the FK-ordering case, we create the file at root *before* the +// directory. Restore order in ExecuteUndoTransaction is by file_id ASC, so +// the older file_id (the file) is upserted first, with parent_id pointing to +// the not-yet-restored dir row. We use the production .apply path so cache +// invalidation matches the FUSE/NFS layer behavior. The undo target is read +// from .log/.last/.../.export/json so it's a raw UUID -- the format +// ExecuteUndo expects when called from writeUndoApply. +func TestUndo_ToID_RestoresDeletedDirWithFile(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undofk") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undofk", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // 1. Create x.md at root -- its file_id (UUIDv7) is the oldest. + fsErr = ops.WriteFile(ctx, "/undofk/x.md", []byte("---\ntitle: X\n---\nbody x\n")) + require.Nil(t, fsErr) + + // 2. Create dir A -- newer file_id. + fsErr = ops.Mkdir(ctx, "/undofk/A") + require.Nil(t, fsErr) + + // 3. Move x.md into A. After this, x.md.parent_id = A.id but its + // file_id stays unchanged (older than A's). When the undo restores + // rows in file_id ASC order, x.md is upserted before A -- forcing + // the FK to violate immediately without deferral. + fsErr = ops.Rename(ctx, "/undofk/x.md", "/undofk/A/x.md") + require.Nil(t, fsErr) + + // Target: the most recent log entry, which is the rename. After undoing + // back to this point, both x.md (under A) and A should be restored. + targetLogID := mostRecentLogIDRaw(t, ops, ctx, "/undofk") + + // 4. Delete file then dir (rmdir requires empty). Sanity-stat the + // deleted dir afterward; this populates the negative stat-cache + // entry, which the undo's invalidation must clear (covered + // independently by TestUndo_Apply_InvalidatesNegativeStatCache). + require.Nil(t, ops.Delete(ctx, "/undofk/A/x.md")) + require.Nil(t, ops.Delete(ctx, "/undofk/A")) + _, fsErr = ops.Stat(ctx, "/undofk/A") + require.NotNil(t, fsErr, "A should be deleted before undo") + + // Pause so the dir's archived (pre-delete) modified_at is measurably + // older than the imminent undo. This makes the post-undo ModTime check + // below robust to wall-clock skew between the PG container and host: + // without the bump, ModTime ~= pre-delete time = sleepFloor; with the + // bump, ModTime is at least sleep duration newer. + const undoBumpSleep = 500 * time.Millisecond + sleepFloor := time.Now() + time.Sleep(undoBumpSleep) + + // 5. Apply undo via the production .apply path. The path segment is + // the raw UUID (matches what writeUndoApply passes to ExecuteUndo). + applyPath := "/undofk/.undo/to-id/" + targetLogID + "/.apply" + fsErr = ops.WriteFile(ctx, applyPath, []byte("")) + require.Nil(t, fsErr, "undo .apply must succeed with deferred constraints") + + // 6. Both the dir and the file inside should be back. + fc, fsErr := ops.ReadFile(ctx, "/undofk/A/x.md") + require.Nil(t, fsErr, "file x.md should be restored under A") + assert.Contains(t, string(fc.Data), "body x") + + entries, fsErr := ops.ReadDir(ctx, "/undofk/A") + require.Nil(t, fsErr, "directory A should be readable after undo") + var names []string + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + names = append(names, e.Name) + } + } + assert.Contains(t, names, "x.md", "A should contain x.md after undo") + + // 7. Directory A's ModTime must reflect the undo. In this scenario the + // only restored child (x.md) has an older file_id than A, so x.md + // is UPSERTed before A and the bump_parent_mtime AFTER trigger + // fires against a not-yet-existing A (0 rows updated). Without + // the explicit post-undo modified_at bump, A's mtime stays at the + // historical (pre-delete) value -- and NFS clients with `noac` + // won't invalidate their readdir cache. + // + // The pre-delete mtime was captured before sleepFloor; the bumped + // mtime should be at least undoBumpSleep newer. Use half the sleep + // as the tolerance to absorb container/host clock skew. + dirEntry, fsErr := ops.Stat(ctx, "/undofk/A") + require.Nil(t, fsErr, "stat A after undo") + tolerance := undoBumpSleep / 2 + if dirEntry.ModTime.Before(sleepFloor.Add(tolerance)) { + t.Errorf("dir A ModTime not bumped by undo: ModTime=%v, expected after sleepFloor+tolerance=%v", + dirEntry.ModTime, sleepFloor.Add(tolerance)) + } +} + +// TestUndo_ToID_RestoresMultipleFilesInDeletedDir verifies multi-file +// restoration end-to-end: delete one of several siblings, undo, and ensure +// the directory listing surfaces every file (with content intact). Catches +// regressions in row restoration or path resolution that wouldn't show up +// in single-file tests. +func TestUndo_ToID_RestoresMultipleFilesInDeletedDir(t *testing.T) { + result := GetTestDBEmpty(t) + if result == nil { + return + } + defer result.Cleanup() + cleanupTigerFSTables(t, result.ConnStr, "undomulti") + + ops := setupFSOperations(t, result.ConnStr) + ctx := context.Background() + + fsErr := ops.WriteFile(ctx, "/.build/undomulti", []byte("markdown,history\n")) + require.Nil(t, fsErr) + time.Sleep(100 * time.Millisecond) + + // Build state: dir A with three sibling files. + require.Nil(t, ops.Mkdir(ctx, "/undomulti/A")) + require.Nil(t, ops.WriteFile(ctx, "/undomulti/A/one.md", + []byte("---\ntitle: One\n---\nbody one\n"))) + require.Nil(t, ops.WriteFile(ctx, "/undomulti/A/two.md", + []byte("---\ntitle: Two\n---\nbody two\n"))) + require.Nil(t, ops.WriteFile(ctx, "/undomulti/A/three.md", + []byte("---\ntitle: Three\n---\nbody three\n"))) + + // Target: the most recent log entry (create of three.md). We undo back + // to this point, so the upcoming delete of two.md gets rolled back. + targetLogID := mostRecentLogIDRaw(t, ops, ctx, "/undomulti") + + // Delete two.md. + require.Nil(t, ops.Delete(ctx, "/undomulti/A/two.md")) + + // Apply undo via the production .apply path. + applyPath := "/undomulti/.undo/to-id/" + targetLogID + "/.apply" + fsErr = ops.WriteFile(ctx, applyPath, []byte("")) + require.Nil(t, fsErr, "undo .apply must succeed") + + // All three siblings should be present. + entries, fsErr := ops.ReadDir(ctx, "/undomulti/A") + require.Nil(t, fsErr, "read A after undo") + names := map[string]bool{} + for _, e := range entries { + if !strings.HasPrefix(e.Name, ".") { + names[e.Name] = true + } + } + for _, want := range []string{"one.md", "two.md", "three.md"} { + if !names[want] { + t.Errorf("missing %s after undo (got %v)", want, names) + } + } + + // two.md's content is restored. + fc, fsErr := ops.ReadFile(ctx, "/undomulti/A/two.md") + require.Nil(t, fsErr, "read restored two.md") + assert.Contains(t, string(fc.Data), "body two") +} + +// mostRecentLogIDRaw returns the raw UUID of the most recent log entry for +// the given app, read via .log/.last/1/.export/json. ReadDir on .log/ returns +// display names (timestamp-encoded), but the undo apply path expects a raw +// UUID, so we go through the JSON export to get the underlying value. +func mostRecentLogIDRaw(t *testing.T, ops *fs.Operations, ctx context.Context, appPath string) string { + t.Helper() + fc, fsErr := ops.ReadFile(ctx, appPath+"/.log/.last/1/.export/json") + require.Nil(t, fsErr, "read latest log JSON") + var entries []struct { + LogID string `json:"log_id"` + } + require.NoError(t, json.Unmarshal(fc.Data, &entries)) + require.NotEmpty(t, entries, "log should have at least one entry") + require.NotEmpty(t, entries[0].LogID, "log_id must not be empty") + return entries[0].LogID +} diff --git a/test/stress/README.md b/test/stress/README.md new file mode 100644 index 0000000..c68de33 --- /dev/null +++ b/test/stress/README.md @@ -0,0 +1,253 @@ +# tigerfs-stress + +Comprehensive stress test for TigerFS file-first workspaces. + +## What It Is + +`tigerfs-stress` is a self-contained, deterministic stress test that exercises TigerFS file-first workspaces through randomized sequences of filesystem operations and undo rollbacks. It verifies correctness after every operation by comparing the actual mounted filesystem against an in-memory expected state model. + +### Why + +Integration tests cover individual operations and specific edge cases, but real-world usage involves long, unpredictable sequences of mixed operations with interleaved undos. This stress test catches ordering bugs, state tracking errors, undo corruption, and data loss that targeted tests miss. + +### Architecture + +The stress tester is a standalone Go binary that operates as a **pure filesystem client** -- all operations use `os.WriteFile`, `os.Rename`, `os.Remove`, etc. against the real NFS mount. It has no tigerfs internal imports. Undo is triggered by writing to `.undo/id//.apply` through the filesystem. + +``` + NFS mount +tigerfs-stress ── os.WriteFile ─────────────> TigerFS ──> PostgreSQL + │ │ + │ in-memory expected state │ actual data + │ (path -> md5 hash) │ (rows in DB) + │ │ + └── ValidateWorkspace() ── os.ReadFile ─────┘ + + md5 compare +``` + +### Infrastructure Lifecycle + +1. **Start**: Build tigerfs binary, spin up Docker PostgreSQL (TimescaleDB), mount TigerFS to a temp directory, create a workspace with versioned history +2. **Test**: Run N iterations of randomized operations with verification +3. **Teardown**: Kill tigerfs, unmount, docker compose down, remove temp dir + +All infrastructure is managed automatically. Teardown happens on normal exit, on Ctrl-C (SIGINT/SIGTERM), or via the `stop` command from another terminal. + +### Operation Loop + +Each iteration: +1. **Select** a random operation from a weighted table (respecting preconditions) +2. **Push** the current expected state onto a stack (for undo rollback) +3. **Execute** the operation on the real mounted filesystem +4. **Update** the in-memory expected state +5. **Validate**: walk the real filesystem, hash every file, compare to expected + +Operations are weighted to produce realistic mixes: + +| Operation | Weight | Description | +|-----------|--------|-------------| +| create_file | 25 | New markdown file with random content | +| edit_file | 25 | Modify existing file body | +| rename_file | 10 | Rename within same directory | +| move_file | 10 | Move to different directory | +| delete_file | 10 | Remove existing file | +| create_dir | 5 | New subdirectory | +| rename_dir | 5 | Rename directory (cascades to contents) | +| create_savepoint | 5 | Named savepoint with snapshot | +| undo_single | 3 | Undo most recent operation | +| undo_to_id | 2 | Undo all operations after a log entry | +| undo_to_savepoint | 2 | Undo all operations after a savepoint | + +### State Tracking + +The expected state is a map of `relative_path -> md5_hash`. Only hashes are stored in memory -- actual file content lives in PostgreSQL via TigerFS. This keeps memory bounded even with `--large-files` mode. + +Before every operation, the current state is pushed onto a stack: +- **undo_single**: pops one entry (reverts to before the last operation) +- **undo_to_id**: restores to the stack entry at that log position +- **undo_to_savepoint**: restores to the state when the savepoint was created + +### Determinism + +All randomness flows through a single `math/rand.Rand` instance initialized with the provided seed. No goroutines, no time-dependent operations in the test loop, no global random. Given the same seed and flags, runs produce the same operation sequence in most cases. + +**Caveat**: determinism is best-effort, not strict. `canExecute(undo_single, ...)` consults `MostRecentLogIsAtomic()` which depends on `LogCount` -- the number of log entries an op produced -- and that count is read back over NFS via `.log/.last/N/.export/json`. NFS-write commit timing (go-nfs fabricates Open/Write/Close per WRITE RPC, and the test process and NFS server share a process) can change whether the read returns N or N+1 entries on identical inputs. When `canExecute`'s verdict flips, the PRNG is consumed at a different rate and the trace diverges. Most runs of the same seed produce identical traces; pathological cases produce different ones. **For non-reproducible failures, rely on the failure dump (below) rather than seed-based replay.** + +### File Size Distribution + +File sizes follow a **log-normal distribution**, which models real-world file size patterns (many small files, few large ones): + +| Mode | Max Size | Typical | Range | +|------|----------|---------|-------| +| Default | 100KB | ~5KB | 64B - 100KB | +| `--large-files` | 10MB | ~22KB | 64B - 10MB | + +### Directory Density + +| Mode | Max Files/Dir | Max Subdirs/Dir | +|------|--------------|-----------------| +| Default | 10 | 3 | +| `--many-files` | 1000 | 20 | + +## Prerequisites + +- Docker (for PostgreSQL with TimescaleDB) +- Go 1.22+ +- macOS or Linux + +## Build + +```bash +go build -o bin/tigerfs-stress ./test/stress +``` + +## Usage + +```bash +# Run with defaults (random seed, 20 iterations) +bin/tigerfs-stress start + +# Reproducible run +bin/tigerfs-stress start --seed 42 --iterations 50 + +# Large-scale stress test +bin/tigerfs-stress start --large-files --many-files --iterations 100 --validate-every 5 + +# Debug mode (verbose tigerfs logging to tigerfs.log) +bin/tigerfs-stress start --debug --iterations 10 + +# Keep infrastructure running after test for manual inspection +bin/tigerfs-stress start --keep --seed 42 + +# Capture diagnostic snapshots at specific iterations (run continues) +bin/tigerfs-stress start --seed 42 --iterations 1000 --dump-at 100,500,778 + +# Kill a running test from another terminal +bin/tigerfs-stress stop +``` + +## Flags + +| Flag | Default | Description | +|------|---------|-------------| +| `--seed N` | random | PRNG seed for reproducibility | +| `--iterations N` | 20 | Number of operation rounds | +| `--debug` | false | Pass `--log-level debug` to tigerfs | +| `--keep` | false | Don't tear down on exit | +| `--workspace NAME` | testws | Workspace name | +| `--validate-every N` | 1 | Validate every N ops (undo always validates) | +| `--large-files` | false | Enable large files up to 10MB (default max: 100KB) | +| `--many-files` | false | Enable dense directories up to 1000 files/dir (default: 10) | +| `--dump-at LIST` | (none) | Comma-separated iteration numbers to write a snapshot dump after (e.g., `100,250,778`) | + +## Output + +- **stdout**: Step-by-step progress (`[STEP 1/20] create_file docs/intro.md (4.8KB)`) +- **stderr**: Errors with seed and the full replay command (all workload-affecting flags included) +- **tigerfs.log**: TigerFS stdout/stderr (written to working directory) +- **Failure dump**: `/tmp/tigerfs-stress-failure---/` (validation failures only; see below) +- **Exit code**: 0 = pass, 1 = verification failure, 2 = infrastructure failure + +## Stopping a Run + +- **Ctrl-C**: Triggers clean teardown (SIGINT trapped) +- **`bin/tigerfs-stress stop`**: Reads `/tmp/tigerfs-stress.info` and tears down infrastructure (PostgreSQL, mount, mountpoint dir, info file). Does **not** delete failure dumps -- those live in a separate sibling directory and persist until you remove them manually. +- **`--keep`**: Skips teardown; infrastructure stays running for manual inspection + +A validation failure also implicitly keeps the infrastructure running so you can inspect live state alongside the dump. Run `bin/tigerfs-stress stop` when you're done. + +## Diagnostic Dumps + +Three ways a diagnostic dump gets written: + +1. **Auto on validation failure** -- the runner writes a `failure` dump (with `failure_kind: "validation"`) when ValidateWorkspace returns mismatches. Per-iteration, post-undo, or final validation all trigger it. +2. **Auto on operation failure** -- the runner writes a `failure` dump (with `failure_kind: "operation"`) when an op (create_file, edit_file, etc.) returns an error such as EIO. The op trace records the failing op with a `[FAILED: ]` marker. +3. **Manual via `--dump-at N[,M,...]`** -- writes a `snapshot` dump after the listed iterations (post-validation, before the next op). The run continues. Useful for forensics on non-reproducible runs. + +In all three cases the infrastructure is left running so you can correlate the dump against the live database and mount. + +Both kinds use the same machinery and produce the same set of files. They differ only in the directory prefix (`failure-` vs `snapshot-`), the `kind` field in `summary.json`, and the `summary.txt` heading. `find /tmp -name 'tigerfs-stress-failure-*'` keeps returning real failures only. + +Dump location: `/tmp/tigerfs-stress----/` + +Files in the dump: + +| File | Contents | +|------|----------| +| `analysis.txt` | Pre-computed cross-references and anomaly findings (workspace status, stack island structure, log_count distribution, op counts, and any flagged regressions). **Open this first.** | +| `summary.txt` | Human-readable overview: seed, iteration, failing op, issue counts, replay command, dump path, mountpoint, postgres URL, grouped issue summary. | +| `summary.json` | Same as `summary.txt`, machine-readable for downstream tooling. | +| `expected_state.json` | The stress tester's `WorkspaceState` at the moment of failure (path -> md5 hash, plus dirs). | +| `actual_state.json` | Live filesystem snapshot (every file md5'd, every dir listed). | +| `diff.txt` | Sorted, grouped issues: missing files, unexpected files, hash mismatches, missing dirs, unexpected dirs. Each group shows the count and lists every entry. | +| `diff.json` | Same diff as structured `[]ValidationIssue` (kind/path/expected_hash/actual_hash). | +| `stack.json` | Every `StackEntry` with `LogID`, `LogCount`, captured `Files`/`Dirs`. `LogCount > 1` indicates an op that fanned out into multiple log entries (large files, NFS multi-chunk writes). | +| `operations.log` | Line-per-step trace of the entire run, mirroring stdout but kept structured (op name, description, log-entry fan-out marker). | +| `operations.json` | Same trace as `[]OpRecord`, for programmatic analysis. | +| `db_state.json` | Snapshot of the four undo-related tables: workspace rows, log, savepoints, last 200 history versions. Captured via a fresh pgx connection (separate from tigerfs's pool). | +| `db_error.txt` | Written instead of `db_state.json` if the DB capture failed (e.g., conn refused). The rest of the dump is still complete. | + +The dump is best-effort: an I/O error on one file doesn't abort the rest. A warning is printed to stderr and the dump continues. + +### Anomaly detection + +`analysis.txt` includes automated checks that catch issues invisible to validation. Heuristics currently encoded: + +| Heuristic | Catches | +|-----------|---------| +| `log_count` exceeds `ceil(write_size / 128KB) + 2` for create/edit | `lastLogID` regression after a heavy undo (the original iter-107 bug pattern) | +| `create_savepoint` with `log_count > 0` | savepoint logging changed (regression in expected behavior) | +| Stack entry's `LogID` < earlier entry's `LogID` (UUIDv7 lexicographic) | Stack bookkeeping crossed iterations in the wrong order | +| `MissingFile` + `UnexpectedFile` with the same content hash | Rename / move applied by TigerFS but not WorkspaceState (or vice versa) | + +Bias is toward false positives -- a noisy flag is cheap to dismiss; a missed regression hides until the next 1000-iter run also passes silently. + +### Inspecting a dump + +```bash +# Read the auto-generated analysis first +cat /tmp/tigerfs-stress-failure---/analysis.txt + +# Then the failure overview +cat /tmp/tigerfs-stress-failure---/summary.txt + +# Look at the structured diff +cat /tmp/tigerfs-stress-failure---/diff.txt + +# Cross-reference an expected vs. actual hash for a specific file +jq '.["path/to/file.md"]' .../expected_state.json +jq '.["path/to/file.md"]' .../actual_state.json + +# Walk the stack to see what state each iteration captured +jq '.entries[] | {iteration, log_id, log_count, file_count: (.files | length)}' .../stack.json + +# See the DB-side view of what TigerFS recorded +jq '.log[] | {log_id, type, filename}' .../db_state.json +``` + +Because the infrastructure is still running, you can also `psql` directly into the test PostgreSQL using the `conn_str` from `summary.json`, or `ls`/`cat` the live mountpoint to compare against the dump's actual_state. + +## Replaying Failures + +The full replay command is printed both at startup (in the failure message) and inside `summary.txt`: + +``` +Replay with: bin/tigerfs-stress start --seed 1713490823456 --iterations 100 --validate-every 1 --large-files +``` + +Replays include every workload-affecting flag (`--seed`, `--iterations`, `--validate-every`, `--large-files`, `--many-files`, `--workspace`) so the run is bit-for-bit comparable to the original. See the determinism caveat above: NFS-timing-sensitive failures may not reproduce on the same seed; in those cases the failure dump is the durable record. + +## Unit Tests + +The stress tester's own logic is unit tested (no Docker or TigerFS required): + +```bash +go test ./test/stress/... +``` + +Tests cover: state deep copy, stack push/pop/savepoint, ValidateWorkspace against temp dirs, content generation determinism, operation selection weights, precondition checks. + +## See Also + +- [ADR-018: Stress Test](../../docs/adr/018-stress-test.md) +- Phase 14 in [Implementation Tasks](../../docs/implementation/implementation-tasks.md) diff --git a/test/stress/diagnostics.go b/test/stress/diagnostics.go new file mode 100644 index 0000000..83bd631 --- /dev/null +++ b/test/stress/diagnostics.go @@ -0,0 +1,669 @@ +package main + +import ( + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "sort" + "strings" + "time" + + "github.com/jackc/pgx/v5" +) + +// OpRecord captures one iteration's operation for the failure dump's +// operations.log -- a structured replay of the run that reproduces what +// `[STEP N/M] ...` printed at runtime, plus log_id metadata that wouldn't +// otherwise survive process exit. +type OpRecord struct { + Iteration int `json:"iteration"` + OpName string `json:"op_name"` + Desc string `json:"desc"` // human-readable, matches stdout + NewLogIDs []string `json:"new_log_ids"` // log_ids produced by this op (>=2 indicates fan-out) + Validated bool `json:"validated"` + UndoTarget string `json:"undo_target,omitempty"` // for undo_* ops: the targeted log_id / savepoint +} + +// stackDump is the JSON shape for stack.json. Mirrors StackEntry/StateStack +// but with exported field names suitable for offline analysis. +type stackDumpEntry struct { + Iteration int `json:"iteration"` + LogID string `json:"log_id,omitempty"` + LogCount int `json:"log_count"` + Files map[string]string `json:"files"` + Dirs map[string]bool `json:"dirs"` +} + +type stackDump struct { + Entries []stackDumpEntry `json:"entries"` + Savepoints map[string]int `json:"savepoints"` +} + +// dbDump is the JSON shape for db_state.json. We snapshot the four tables +// the undo system uses; *_history is capped because it can grow without +// bound and we only need recent versions for diagnosis. +type dbDump struct { + Workspace string `json:"workspace"` + Schema string `json:"schema"` + Rows []map[string]interface{} `json:"rows"` + Log []map[string]interface{} `json:"log"` + Savepoints []map[string]interface{} `json:"savepoints"` + HistoryLastN []map[string]interface{} `json:"history_last_n"` + HistoryLimit int `json:"history_limit"` +} + +// DumpKind tags a dump as either a validation failure (auto-fired by the +// runner) or a manual snapshot (--dump-at). Distinguishing them in the dump +// directory prefix and summary lets `find /tmp -name 'tigerfs-stress-failure-*'` +// keep returning real failures only. +type DumpKind string + +const ( + DumpKindFailure DumpKind = "failure" + DumpKindSnapshot DumpKind = "snapshot" +) + +// dumpSummary is the JSON shape for summary.json -- one-stop overview +// describing the dump for downstream tooling and humans alike. Keep it +// flat: anything that fits here doesn't need to be cross-referenced from +// the other dump files. +// +// FailureKind is "" for snapshots, "validation" when ValidateWorkspace +// returned mismatches, "operation" when an op (create_file, edit_file, +// etc.) returned an error. Provided so downstream tooling can grep +// failures by category without parsing free-form text. +// +// ErrorMessage carries the abort reason -- the validation issue list +// for kind=validation, the underlying op error (e.g. EIO) for +// kind=operation. Empty for snapshots. +type dumpSummary struct { + Kind DumpKind `json:"kind"` + FailureKind string `json:"failure_kind,omitempty"` + Seed int64 `json:"seed"` + Iteration int `json:"iteration"` + TotalIterations int `json:"total_iterations"` + Workspace string `json:"workspace"` + LargeFiles bool `json:"large_files"` + ManyFiles bool `json:"many_files"` + ValidateEvery int `json:"validate_every"` + Op string `json:"op"` + IssueCount int `json:"issue_count"` + ReplayCommand string `json:"replay_command"` + Mountpoint string `json:"mountpoint"` + ConnStr string `json:"conn_str"` + DumpDir string `json:"dump_dir"` + GeneratedAt string `json:"generated_at"` + ErrorMessage string `json:"error_message,omitempty"` +} + +// historyDumpLimit caps how many *_history rows we serialize. Versions can +// pile up in a long run; the most-recent ones are what's diagnostic. +const historyDumpLimit = 200 + +// WriteDump captures everything a future investigator might want about a +// stress-test moment: expected vs actual workspace state, the full state +// stack, the live DB tables, the structured op trace, and a sorted diff +// of any divergence. Used both for auto-fired failure dumps (kind = +// DumpKindFailure, valErr non-nil) and manual --dump-at snapshots (kind = +// DumpKindSnapshot, valErr nil). +// +// Returns the dump directory path and any I/O error encountered along the +// way. Best-effort: we keep going past individual file write errors so +// the caller gets as much data as possible. +// +// The dump dir is +// +// /tmp/tigerfs-stress----/ +// +// to disambiguate concurrent runs and let users keep multiple dumps +// around. Caller is expected to print the returned path. +func WriteDump(kind DumpKind, failureKind string, cfg *Config, infra *Infra, state *WorkspaceState, stack *StateStack, opLog []OpRecord, runErr error, op string, iteration int) (string, error) { + dumpDir := fmt.Sprintf("/tmp/tigerfs-stress-%s-%d-%d-%d", kind, cfg.Seed, iteration, time.Now().Unix()) + if err := os.MkdirAll(dumpDir, 0755); err != nil { + return "", fmt.Errorf("create dump dir: %w", err) + } + + // Snapshot the live filesystem and compute the structured diff. Both + // feed the actual_state.json and diff.txt files. + wsPath := filepath.Join(infra.Mountpoint, cfg.Workspace) + actualFiles, actualDirs, snapErr := snapshotWorkspace(wsPath) + if snapErr != nil { + // Don't bail -- record the error and continue with the data we + // already have. A failed snapshot is a useful symptom, not a + // reason to abort dumping the rest. + writeText(dumpDir, "snapshot_error.txt", snapErr.Error()) + } + issues := diffWorkspace(state, actualFiles, actualDirs) + + // summary.json + summary.txt -- the at-a-glance view + summary := dumpSummary{ + Kind: kind, + FailureKind: failureKind, + Seed: cfg.Seed, + Iteration: iteration, + TotalIterations: cfg.Iterations, + Workspace: cfg.Workspace, + LargeFiles: cfg.LargeFiles, + ManyFiles: cfg.ManyFiles, + ValidateEvery: cfg.ValidateEvery, + Op: op, + IssueCount: len(issues), + ReplayCommand: replayCommand(cfg), + Mountpoint: infra.Mountpoint, + ConnStr: infra.ConnStr, + DumpDir: dumpDir, + GeneratedAt: time.Now().Format(time.RFC3339), + ErrorMessage: shortErrorMessage(runErr), + } + writeJSON(dumpDir, "summary.json", summary) + writeText(dumpDir, "summary.txt", renderSummaryText(summary, issues)) + + // expected_state.json -- WorkspaceState at the moment of failure + writeJSON(dumpDir, "expected_state.json", map[string]interface{}{ + "files": state.Files, + "dirs": state.Dirs, + }) + + // actual_state.json -- live filesystem snapshot + writeJSON(dumpDir, "actual_state.json", map[string]interface{}{ + "files": actualFiles, + "dirs": actualDirs, + }) + + // diff.txt + diff.json -- structured divergence + writeJSON(dumpDir, "diff.json", issues) + writeText(dumpDir, "diff.txt", renderDiffText(issues)) + + // stack.json -- every StackEntry and savepoint (in-package access). + writeJSON(dumpDir, "stack.json", buildStackDump(stack)) + + // operations.log + operations.json -- full op trace + writeJSON(dumpDir, "operations.json", opLog) + writeText(dumpDir, "operations.log", renderOpLogText(opLog)) + + // db_state.json -- live PostgreSQL view of the four undo-related + // tables. Best-effort: a DB error here doesn't fail the whole dump. + if dump, err := captureDBState(cfg, infra); err != nil { + writeText(dumpDir, "db_error.txt", err.Error()) + } else { + writeJSON(dumpDir, "db_state.json", dump) + } + + // analysis.txt -- pre-computed cross-reference and anomaly checks. + // Last so it's based on the same in-memory state captured above. + writeText(dumpDir, "analysis.txt", analyzeDump(state, stack, opLog, issues)) + + return dumpDir, nil +} + +// renderSummaryText produces a human-friendly summary.txt. Designed to be +// the first file someone opens after a dump -- everything else can be +// looked up via the paths and command listed here. +func renderSummaryText(s dumpSummary, issues []ValidationIssue) string { + var b strings.Builder + title := "tigerfs-stress " + string(s.Kind) + " dump" + fmt.Fprintf(&b, "%s\n", title) + fmt.Fprintf(&b, "%s\n\n", strings.Repeat("=", len(title))) + fmt.Fprintf(&b, "Generated: %s\n", s.GeneratedAt) + fmt.Fprintf(&b, "Seed: %d\n", s.Seed) + fmt.Fprintf(&b, "Iteration: %d / %d\n", s.Iteration, s.TotalIterations) + fmt.Fprintf(&b, "Op: %s\n", s.Op) + fmt.Fprintf(&b, "Issues: %d\n", s.IssueCount) + fmt.Fprintf(&b, "Workspace: %s\n", s.Workspace) + fmt.Fprintf(&b, "LargeFiles: %v\n", s.LargeFiles) + fmt.Fprintf(&b, "ManyFiles: %v\n", s.ManyFiles) + fmt.Fprintf(&b, "ValidateEvery: %d\n", s.ValidateEvery) + fmt.Fprintf(&b, "\nOpen analysis.txt first -- it lists anomalies and the run shape at a glance.\n") + fmt.Fprintf(&b, "\nDump directory: %s\n", s.DumpDir) + fmt.Fprintf(&b, "Mountpoint: %s\n", s.Mountpoint) + fmt.Fprintf(&b, "Postgres: %s\n", s.ConnStr) + fmt.Fprintf(&b, "\nReplay: %s\n", s.ReplayCommand) + if len(issues) > 0 { + fmt.Fprintf(&b, "\nIssue summary:\n%s\n", renderDiffText(issues)) + } + if s.ErrorMessage != "" { + // Heading adapts to whether this was a validation diff or an + // op-level error; both flow through the same field. + heading := "Error" + switch s.FailureKind { + case "validation": + heading = "Validation error" + case "operation": + heading = "Operation error" + } + fmt.Fprintf(&b, "\n%s:\n %s\n", heading, s.ErrorMessage) + } + return b.String() +} + +// renderDiffText groups issues by kind for quick scanning. Within each +// group, paths are already sorted by diffWorkspace, so a side-by-side +// rename ("missing X" / "unexpected Y at different path") usually appears +// near each other. +func renderDiffText(issues []ValidationIssue) string { + if len(issues) == 0 { + return "(no issues)" + } + groups := map[ValidationIssueKind][]ValidationIssue{} + for _, iss := range issues { + groups[iss.Kind] = append(groups[iss.Kind], iss) + } + order := []ValidationIssueKind{ + IssueMissingFile, IssueUnexpectedFile, IssueHashMismatch, + IssueMissingDir, IssueUnexpectedDir, + } + var b strings.Builder + for _, k := range order { + group := groups[k] + if len(group) == 0 { + continue + } + fmt.Fprintf(&b, "\n [%s] (%d)\n", k, len(group)) + for _, iss := range group { + fmt.Fprintf(&b, " %s\n", formatIssue(iss)) + } + } + return b.String() +} + +// renderOpLogText produces operations.log -- the line-per-step trace +// that's easy to grep. JSON form (operations.json) is also written for +// programmatic analysis. +func renderOpLogText(opLog []OpRecord) string { + var b strings.Builder + for _, r := range opLog { + marker := "" + if !r.Validated { + marker = " [no-validate]" + } + ids := "" + if len(r.NewLogIDs) > 1 { + ids = fmt.Sprintf(" [%d log entries]", len(r.NewLogIDs)) + } + fmt.Fprintf(&b, "[STEP %d] %s%s%s\n", r.Iteration, r.Desc, marker, ids) + } + return b.String() +} + +// shortErrorMessage extracts just the human summary from a wrapped run +// error -- the goroutine stack inside `%w(...)` is noise here. Used for +// both validation-diff messages and op-level error chains. +func shortErrorMessage(err error) string { + if err == nil { + return "" + } + msg := err.Error() + // Cap at first 4 KB; the structured diff has the full data. + if len(msg) > 4096 { + msg = msg[:4096] + "... (truncated; see diff.txt)" + } + return msg +} + +// buildStackDump copies StateStack internals into the JSON-friendly +// stackDump shape. Same package so we can read unexported fields directly. +func buildStackDump(stack *StateStack) stackDump { + out := stackDump{ + Entries: make([]stackDumpEntry, 0, len(stack.entries)), + Savepoints: map[string]int{}, + } + for _, e := range stack.entries { + out.Entries = append(out.Entries, stackDumpEntry{ + Iteration: e.Iteration, + LogID: e.LogID, + LogCount: e.LogCount, + Files: e.State.Files, + Dirs: e.State.Dirs, + }) + } + for k, v := range stack.savepoints { + out.Savepoints[k] = v + } + return out +} + +// captureDBState opens a fresh pgx connection and snapshots the four +// tables that participate in undo. Lives in its own connection (not the +// pool tigerfs uses) so this is safe even mid-failure. +func captureDBState(cfg *Config, infra *Infra) (*dbDump, error) { + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + + conn, err := pgx.Connect(ctx, infra.ConnStr) + if err != nil { + return nil, fmt.Errorf("connect: %w", err) + } + defer conn.Close(ctx) + + out := &dbDump{ + Workspace: cfg.Workspace, + Schema: "tigerfs", + HistoryLimit: historyDumpLimit, + } + + // Live rows (current workspace contents). Body length only -- full + // bodies can be megabytes per row and the diff doesn't need them. + rows, err := queryAsMaps(ctx, conn, + fmt.Sprintf(`SELECT id::text, parent_id::text AS parent_id, filename, filetype, length(body) AS body_len, modified_at::text AS modified_at FROM tigerfs.%s ORDER BY filename, id`, cfg.Workspace)) + if err != nil { + return out, fmt.Errorf("query rows: %w", err) + } + out.Rows = rows + + logRows, err := queryAsMaps(ctx, conn, + fmt.Sprintf(`SELECT log_id::text, file_id::text, type, filename, version_id::text AS version_id, description FROM tigerfs.%s_log ORDER BY log_id`, cfg.Workspace)) + if err != nil { + return out, fmt.Errorf("query log: %w", err) + } + out.Log = logRows + + spRows, err := queryAsMaps(ctx, conn, + fmt.Sprintf(`SELECT name, log_id::text, created_at::text AS created_at FROM tigerfs.%s_savepoint ORDER BY created_at`, cfg.Workspace)) + if err != nil { + // _savepoint table may not always exist; degrade gracefully. + out.Savepoints = nil + } else { + out.Savepoints = spRows + } + + histRows, err := queryAsMaps(ctx, conn, + fmt.Sprintf(`SELECT version_id::text, file_id::text, filename, length(body) AS body_len, modified_at::text AS modified_at, operation FROM tigerfs.%s_history ORDER BY version_id DESC LIMIT %d`, cfg.Workspace, historyDumpLimit)) + if err != nil { + return out, fmt.Errorf("query history: %w", err) + } + out.HistoryLastN = histRows + + return out, nil +} + +// queryAsMaps executes a query and returns each row as a column->value +// map. Field names come from the result description so the JSON output +// matches the SQL projection exactly. +func queryAsMaps(ctx context.Context, conn *pgx.Conn, query string) ([]map[string]interface{}, error) { + rows, err := conn.Query(ctx, query) + if err != nil { + return nil, err + } + defer rows.Close() + + fields := rows.FieldDescriptions() + cols := make([]string, len(fields)) + for i, f := range fields { + cols[i] = string(f.Name) + } + + var out []map[string]interface{} + for rows.Next() { + vals, err := rows.Values() + if err != nil { + return nil, err + } + m := make(map[string]interface{}, len(cols)) + for i, c := range cols { + m[c] = vals[i] + } + out = append(out, m) + } + if err := rows.Err(); err != nil { + return nil, err + } + // Stable ordering for the cols list -- map ranges are random and a + // stable JSON output makes diffing dumps easier. + sort.Strings(cols) + return out, nil +} + +// writeJSON marshals v indented and writes it to dumpDir/name. Errors +// are logged to stderr but do not propagate -- a single missing file +// shouldn't take the whole dump down. +func writeJSON(dumpDir, name string, v interface{}) { + data, err := json.MarshalIndent(v, "", " ") + if err != nil { + fmt.Fprintf(os.Stderr, "[WARN] dump: marshal %s: %v\n", name, err) + return + } + if err := os.WriteFile(filepath.Join(dumpDir, name), data, 0644); err != nil { + fmt.Fprintf(os.Stderr, "[WARN] dump: write %s: %v\n", name, err) + } +} + +func writeText(dumpDir, name, content string) { + if err := os.WriteFile(filepath.Join(dumpDir, name), []byte(content), 0644); err != nil { + fmt.Fprintf(os.Stderr, "[WARN] dump: write %s: %v\n", name, err) + } +} + +// nfsChunkBytes is the NFS write chunk size (matches mount option +// wsize=131072). A user-level write of N bytes fans out into +// ceil(N/nfsChunkBytes) WRITE+CLOSE RPCs, each producing one log entry, +// so this is the upper bound on a single op's log_count. +const nfsChunkBytes = 128 * 1024 + +// analyzeDump renders analysis.txt -- a pre-computed cross-reference of +// the most useful dump observations and a battery of anomaly checks. +// Called from WriteDump so every dump is self-explanatory without ad-hoc +// queries: at a glance the reader sees workspace status, stack island +// structure, log_count distribution, op counts, and any flagged +// anomalies (false positives tolerated; false negatives are not). +func analyzeDump(state *WorkspaceState, stack *StateStack, opLog []OpRecord, issues []ValidationIssue) string { + var b strings.Builder + + fmt.Fprintf(&b, "=== Workspace ===\n") + if len(issues) == 0 { + fmt.Fprintf(&b, "Validation: PASSED\n") + } else { + fmt.Fprintf(&b, "Validation: FAILED (%d issues; see diff.txt)\n", len(issues)) + } + fmt.Fprintf(&b, "Files: %d Dirs: %d\n\n", len(state.Files), len(state.Dirs)) + + fmt.Fprintf(&b, "=== Stack islands ===\n") + islands := stackIslands(stack) + parts := make([]string, 0, len(islands)) + for _, isl := range islands { + if isl.start == isl.end { + parts = append(parts, fmt.Sprintf("[%d]", isl.start)) + } else { + parts = append(parts, fmt.Sprintf("[%d..%d]", isl.start, isl.end)) + } + } + fmt.Fprintf(&b, "%d entries: %s\n", len(stack.entries), strings.Join(parts, " ")) + if len(stack.savepoints) > 0 { + spNames := make([]string, 0, len(stack.savepoints)) + for n := range stack.savepoints { + spNames = append(spNames, n) + } + sort.Strings(spNames) + fmt.Fprintf(&b, "Savepoints (%d): %s\n", len(spNames), strings.Join(spNames, ", ")) + } + fmt.Fprintln(&b) + + fmt.Fprintf(&b, "=== log_count distribution ===\n") + dist := map[int]int{} + for _, e := range stack.entries { + dist[e.LogCount]++ + } + keys := make([]int, 0, len(dist)) + for k := range dist { + keys = append(keys, k) + } + sort.Ints(keys) + for _, k := range keys { + fmt.Fprintf(&b, " log_count=%-3d %d entries\n", k, dist[k]) + } + fmt.Fprintln(&b) + + fmt.Fprintf(&b, "=== Anomalies ===\n") + anomalies := detectAnomalies(opLog, stack, issues) + if len(anomalies) == 0 { + fmt.Fprintf(&b, "(none detected)\n") + } else { + for _, a := range anomalies { + fmt.Fprintf(&b, "WARNING %s\n", a) + } + } + fmt.Fprintln(&b) + + fmt.Fprintf(&b, "=== Op counts ===\n") + opCounts := map[string]int{} + for _, op := range opLog { + opCounts[op.OpName]++ + } + opNames := make([]string, 0, len(opCounts)) + for n := range opCounts { + opNames = append(opNames, n) + } + sort.Strings(opNames) + for _, n := range opNames { + fmt.Fprintf(&b, " %-20s %d\n", n, opCounts[n]) + } + + return b.String() +} + +// island groups consecutive iterations from the stack. Gaps between +// islands correspond to undo operations that trimmed the stack -- their +// pattern is one of the most useful "what's the run shape" signals. +type island struct{ start, end int } + +func stackIslands(stack *StateStack) []island { + if len(stack.entries) == 0 { + return nil + } + out := []island{{stack.entries[0].Iteration, stack.entries[0].Iteration}} + for i := 1; i < len(stack.entries); i++ { + iter := stack.entries[i].Iteration + if iter == out[len(out)-1].end+1 { + out[len(out)-1].end = iter + } else { + out = append(out, island{iter, iter}) + } + } + return out +} + +// detectAnomalies runs heuristics over the op trace, stack, and issues. +// Each finding produces a short "what looks wrong" string for analysis.txt. +// Bias toward false positives: a noisy flag is a cheap way to surface +// something the reader will quickly dismiss; a missed regression is +// invisible until the next 1000-iter run also passes silently. +func detectAnomalies(opLog []OpRecord, stack *StateStack, issues []ValidationIssue) []string { + var out []string + + // 1. log_count vs op-type expectation. NFS chunks at nfsChunkBytes; + // a small create can produce at most 1 log entry, a large one + // ceil(size/chunk). Anything wildly higher implies lastLogID + // regressed during a prior undo and the next op picked up old + // entries as if they were new. + for _, op := range opLog { + actual := len(op.NewLogIDs) + if actual <= 1 { + continue + } + if op.OpName == "create_savepoint" { + out = append(out, fmt.Sprintf("iter %d: create_savepoint logged %d entries (expected 0)", op.Iteration, actual)) + continue + } + if strings.HasPrefix(op.OpName, "undo_") { + continue // undo bookkeeping is handled separately + } + expected := 1 + switch op.OpName { + case "create_file", "edit_file": + if size, ok := parseSizeBytes(op.Desc); ok { + expected = (size + nfsChunkBytes - 1) / nfsChunkBytes + if expected < 1 { + expected = 1 + } + } + } + // Tolerance of +2 absorbs off-by-one chunk-boundary cases (e.g., + // a 128KB file can land as 1 or 2 chunks depending on NFS client + // framing) without burying real regressions. + if actual > expected+2 { + out = append(out, fmt.Sprintf( + "iter %d: log_count=%d for %q (expected ~%d, off by %d) -- likely lastLogID regression in a prior undo", + op.Iteration, actual, op.Desc, expected, actual-expected)) + } + } + + // 2. UUIDv7 monotonicity in the stack. Each subsequent stack entry + // captures a later state, so its LogID (when set) must be > + // the previous logged entry's LogID. A regression means stack + // bookkeeping crossed an iteration boundary in the wrong order. + var prevID string + var prevIter int + for _, e := range stack.entries { + if e.LogID == "" { + continue + } + if prevID != "" && e.LogID < prevID { + out = append(out, fmt.Sprintf( + "stack iter %d: log_id %s < prior iter %d's %s (UUIDv7 regression)", + e.Iteration, e.LogID, prevIter, prevID)) + } + prevID = e.LogID + prevIter = e.Iteration + } + + // 3. Rename / move artifacts in failure dumps. Pair MissingFile + + // UnexpectedFile by content hash -- a match means the file is + // really still on disk, just at a different path. The likely + // cause is TigerFS and the stress-test diverging on a rename + // or move (one applied it, the other didn't). + if len(issues) > 0 { + unexpectedByHash := map[string]string{} + for _, iss := range issues { + if iss.Kind == IssueUnexpectedFile { + unexpectedByHash[iss.ActualHash] = iss.Path + } + } + for _, iss := range issues { + if iss.Kind != IssueMissingFile { + continue + } + if foundAt, ok := unexpectedByHash[iss.ExpectedHash]; ok { + out = append(out, fmt.Sprintf( + "rename artifact: %q expected, found at %q (hash %s match) -- TigerFS and stress-test diverged on a rename/move", + iss.Path, foundAt, shortHash(iss.ExpectedHash))) + } + } + } + + return out +} + +// parseSizeBytes extracts the trailing "(NNN.NB)" / "(NN.NKB)" / +// "(N.NMB)" / "(N.NGB)" from an op description like +// "create_file foo (134.5KB)". Returns (bytes, true) on success; +// (0, false) when there's no parseable size suffix (e.g., for a +// dir-content summary like "(3 files, 0 subdirs)"). +func parseSizeBytes(desc string) (int, bool) { + open := strings.LastIndex(desc, "(") + closeIdx := strings.LastIndex(desc, ")") + if open == -1 || closeIdx <= open { + return 0, false + } + inner := desc[open+1 : closeIdx] + var num float64 + var unit string + n, err := fmt.Sscanf(inner, "%f%s", &num, &unit) + if err != nil || n != 2 { + return 0, false + } + mult := 0 + switch unit { + case "B": + mult = 1 + case "KB": + mult = 1024 + case "MB": + mult = 1024 * 1024 + case "GB": + mult = 1024 * 1024 * 1024 + default: + return 0, false + } + return int(num * float64(mult)), true +} diff --git a/test/stress/docker-compose.yml b/test/stress/docker-compose.yml new file mode 100644 index 0000000..e153ef1 --- /dev/null +++ b/test/stress/docker-compose.yml @@ -0,0 +1,11 @@ +services: + postgres: + image: timescale/timescaledb-ha:pg18 + environment: + POSTGRES_USER: testundo + POSTGRES_PASSWORD: testundo + POSTGRES_DB: testundo + ports: + - "5433:5432" + tmpfs: + - /var/lib/postgresql/data diff --git a/test/stress/infra.go b/test/stress/infra.go new file mode 100644 index 0000000..940b39f --- /dev/null +++ b/test/stress/infra.go @@ -0,0 +1,375 @@ +package main + +import ( + "encoding/json" + "fmt" + "os" + "os/exec" + "os/signal" + "path/filepath" + "runtime" + "strconv" + "strings" + "syscall" + "time" +) + +const ( + infoFilePath = "/tmp/tigerfs-stress.info" + pgPort = "5433" + pgUser = "testundo" + pgPassword = "testundo" + pgDatabase = "testundo" + pgReadyTimeout = 30 * time.Second + mountWaitTimeout = 15 * time.Second +) + +// Infra holds the state of the running infrastructure. +type Infra struct { + Mountpoint string + TigerFSPid int + ComposePath string + RepoRoot string + ConnStr string // postgres URL used by the mounted tigerfs (also reusable for diagnostics) + tigerfsCmd *exec.Cmd + sigChan chan os.Signal +} + +// RunInfo is serialized to the info file for the stop command. +type RunInfo struct { + TigerFSPid int `json:"tigerfs_pid"` + Mountpoint string `json:"mountpoint"` + ComposePath string `json:"compose_path"` + RepoRoot string `json:"repo_root"` +} + +// SetupInfra builds tigerfs, starts Docker PostgreSQL, mounts the filesystem, +// and creates the workspace. +func SetupInfra(cfg *Config) (*Infra, error) { + repoRoot, err := findRepoRoot() + if err != nil { + return nil, fmt.Errorf("find repo root: %w", err) + } + + composePath := filepath.Join(repoRoot, "test", "stress", "docker-compose.yml") + mountpoint := fmt.Sprintf("/tmp/tigerfs-stress-%d", time.Now().Unix()) + tigerBinary := filepath.Join(repoRoot, "bin", "tigerfs") + + infra := &Infra{ + Mountpoint: mountpoint, + ComposePath: composePath, + RepoRoot: repoRoot, + sigChan: make(chan os.Signal, 1), + } + + // Step 1: Build tigerfs + fmt.Print("Building tigerfs... ") + if err := runCmd(repoRoot, "go", "build", "-o", tigerBinary, "./cmd/tigerfs"); err != nil { + return nil, fmt.Errorf("build tigerfs: %w", err) + } + fmt.Println("done") + + // Step 2: Start Docker PostgreSQL + fmt.Print("Starting PostgreSQL... ") + if err := runCmd(repoRoot, "docker", "compose", "-f", composePath, "up", "-d"); err != nil { + return nil, fmt.Errorf("docker compose up: %w", err) + } + fmt.Println("done") + + // Step 3: Wait for PostgreSQL + fmt.Print("Waiting for PostgreSQL... ") + if err := waitForPostgres(); err != nil { + infra.teardownDocker() + return nil, fmt.Errorf("postgres not ready: %w", err) + } + fmt.Println("ready") + + // Step 4: Create mountpoint + if err := os.MkdirAll(mountpoint, 0755); err != nil { + infra.teardownDocker() + return nil, fmt.Errorf("create mountpoint: %w", err) + } + + // Step 5: Mount TigerFS + fmt.Print("Mounting TigerFS... ") + connStr := fmt.Sprintf("postgres://%s:%s@localhost:%s/%s", pgUser, pgPassword, pgPort, pgDatabase) + infra.ConnStr = connStr + + args := []string{"mount", "--insecure-no-ssl", "--user-id", "stress-test"} + if cfg.Debug { + args = append(args, "--log-level", "debug") + } + args = append(args, connStr, mountpoint) + + logFile, err := os.Create("tigerfs.log") + if err != nil { + infra.teardownDocker() + return nil, fmt.Errorf("create tigerfs.log: %w", err) + } + + infra.tigerfsCmd = exec.Command(tigerBinary, args...) + infra.tigerfsCmd.Stdout = logFile + infra.tigerfsCmd.Stderr = logFile + infra.tigerfsCmd.Dir = repoRoot + + if err := infra.tigerfsCmd.Start(); err != nil { + logFile.Close() + infra.teardownDocker() + return nil, fmt.Errorf("start tigerfs: %w", err) + } + infra.TigerFSPid = infra.tigerfsCmd.Process.Pid + fmt.Printf("pid=%d ", infra.TigerFSPid) + + // Wait for mount to appear + if err := waitForMount(mountpoint); err != nil { + infra.killTigerFS() + infra.teardownDocker() + return nil, fmt.Errorf("mount not ready: %w", err) + } + fmt.Println("mounted") + + // Step 6: Create workspace + fmt.Printf("Creating workspace '%s'... ", cfg.Workspace) + buildPath := filepath.Join(mountpoint, ".build", cfg.Workspace) + if err := os.WriteFile(buildPath, []byte("markdown,history\n"), 0644); err != nil { + infra.killTigerFS() + infra.unmount() + infra.teardownDocker() + return nil, fmt.Errorf("create workspace: %w", err) + } + + // Verify workspace exists + wsPath := filepath.Join(mountpoint, cfg.Workspace) + if _, err := os.Stat(wsPath); err != nil { + infra.killTigerFS() + infra.unmount() + infra.teardownDocker() + return nil, fmt.Errorf("workspace not found after creation: %w", err) + } + fmt.Println("done") + + // Step 7: Write run info for stop command + if err := infra.writeInfo(); err != nil { + fmt.Fprintf(os.Stderr, "[WARN] Failed to write info file: %v\n", err) + } + + // Step 8: Set up signal handling + signal.Notify(infra.sigChan, syscall.SIGINT, syscall.SIGTERM) + + return infra, nil +} + +// Teardown shuts down all infrastructure in order. +func (infra *Infra) Teardown() { + fmt.Println("\nTearing down...") + infra.killTigerFS() + infra.unmount() + infra.removeMountpoint() + infra.teardownDocker() + infra.removeInfo() + fmt.Println("Teardown complete.") +} + +// WaitForSignal blocks until SIGINT or SIGTERM is received. +func (infra *Infra) WaitForSignal() { + sig := <-infra.sigChan + fmt.Printf("\nReceived %s\n", sig) +} + +func (infra *Infra) killTigerFS() { + if infra.tigerfsCmd == nil || infra.tigerfsCmd.Process == nil { + return + } + + pid := infra.tigerfsCmd.Process.Pid + fmt.Printf(" Stopping tigerfs (pid %d)... ", pid) + + // Try graceful shutdown first + infra.tigerfsCmd.Process.Signal(syscall.SIGTERM) + + // Wait up to 5 seconds + done := make(chan error, 1) + go func() { done <- infra.tigerfsCmd.Wait() }() + + select { + case <-done: + fmt.Println("stopped") + case <-time.After(5 * time.Second): + fmt.Print("force killing... ") + infra.tigerfsCmd.Process.Kill() + <-done + fmt.Println("killed") + } +} + +func (infra *Infra) unmount() { + fmt.Printf(" Unmounting %s... ", infra.Mountpoint) + + var err error + if runtime.GOOS == "darwin" { + err = exec.Command("diskutil", "unmount", "force", infra.Mountpoint).Run() + } else { + err = exec.Command("umount", infra.Mountpoint).Run() + } + + if err != nil { + fmt.Printf("warning: %v\n", err) + } else { + fmt.Println("done") + } +} + +func (infra *Infra) removeMountpoint() { + os.RemoveAll(infra.Mountpoint) +} + +func (infra *Infra) teardownDocker() { + fmt.Print(" Stopping Docker... ") + if err := runCmd(infra.RepoRoot, "docker", "compose", "-f", infra.ComposePath, "down", "-v"); err != nil { + fmt.Printf("warning: %v\n", err) + } else { + fmt.Println("done") + } +} + +func (infra *Infra) writeInfo() error { + info := RunInfo{ + TigerFSPid: infra.TigerFSPid, + Mountpoint: infra.Mountpoint, + ComposePath: infra.ComposePath, + RepoRoot: infra.RepoRoot, + } + data, err := json.Marshal(info) + if err != nil { + return err + } + return os.WriteFile(infoFilePath, data, 0644) +} + +func (infra *Infra) removeInfo() { + os.Remove(infoFilePath) +} + +// StopInfra reads the info file and tears down a running stress test. +func StopInfra() error { + data, err := os.ReadFile(infoFilePath) + if err != nil { + return fmt.Errorf("no running test found (cannot read %s): %w", infoFilePath, err) + } + + var info RunInfo + if err := json.Unmarshal(data, &info); err != nil { + return fmt.Errorf("parse info file: %w", err) + } + + fmt.Printf("Found running test: pid=%d mountpoint=%s\n", info.TigerFSPid, info.Mountpoint) + + // Kill tigerfs process + fmt.Printf(" Killing tigerfs (pid %d)... ", info.TigerFSPid) + proc, err := os.FindProcess(info.TigerFSPid) + if err == nil { + proc.Signal(syscall.SIGTERM) + time.Sleep(2 * time.Second) + proc.Kill() + fmt.Println("done") + } else { + fmt.Printf("not found: %v\n", err) + } + + // Unmount + fmt.Printf(" Unmounting %s... ", info.Mountpoint) + if runtime.GOOS == "darwin" { + exec.Command("diskutil", "unmount", "force", info.Mountpoint).Run() + } else { + exec.Command("umount", info.Mountpoint).Run() + } + fmt.Println("done") + + // Remove mountpoint + os.RemoveAll(info.Mountpoint) + + // Docker down + fmt.Print(" Stopping Docker... ") + if err := runCmd(info.RepoRoot, "docker", "compose", "-f", info.ComposePath, "down", "-v"); err != nil { + fmt.Printf("warning: %v\n", err) + } else { + fmt.Println("done") + } + + // Remove info file + os.Remove(infoFilePath) + + return nil +} + +// Helper functions + +func findRepoRoot() (string, error) { + // Walk up from current dir looking for go.mod + dir, err := os.Getwd() + if err != nil { + return "", err + } + for { + if _, err := os.Stat(filepath.Join(dir, "go.mod")); err == nil { + return dir, nil + } + parent := filepath.Dir(dir) + if parent == dir { + return "", fmt.Errorf("could not find go.mod in any parent directory") + } + dir = parent + } +} + +func runCmd(dir string, name string, args ...string) error { + cmd := exec.Command(name, args...) + cmd.Dir = dir + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + return cmd.Run() +} + +func waitForPostgres() error { + deadline := time.Now().Add(pgReadyTimeout) + for time.Now().Before(deadline) { + cmd := exec.Command("pg_isready", "-h", "localhost", "-p", pgPort, "-U", pgUser, "-d", pgDatabase) + if cmd.Run() == nil { + return nil + } + time.Sleep(1 * time.Second) + } + return fmt.Errorf("PostgreSQL not ready after %s", pgReadyTimeout) +} + +func waitForMount(mountpoint string) error { + deadline := time.Now().Add(mountWaitTimeout) + for time.Now().Before(deadline) { + // Check if mount appears in mount table + out, err := exec.Command("mount").Output() + if err == nil && strings.Contains(string(out), mountpoint) { + // Also verify we can stat a path inside the mount + if _, err := os.Stat(filepath.Join(mountpoint, ".info")); err == nil { + return nil + } + } + time.Sleep(500 * time.Millisecond) + } + return fmt.Errorf("mount not ready after %s", mountWaitTimeout) +} + +// processExists checks if a process with the given PID is running. +func processExists(pid int) bool { + proc, err := os.FindProcess(pid) + if err != nil { + return false + } + // On Unix, FindProcess always succeeds. Send signal 0 to check. + err = proc.Signal(syscall.Signal(0)) + return err == nil +} + +// formatPID returns a string representation of a PID for display. +func formatPID(pid int) string { + return strconv.Itoa(pid) +} diff --git a/test/stress/main.go b/test/stress/main.go new file mode 100644 index 0000000..b5bae06 --- /dev/null +++ b/test/stress/main.go @@ -0,0 +1,192 @@ +// tigerfs-stress is a comprehensive stress test for TigerFS file-first workspaces. +// It exercises all filesystem operations (create, edit, rename, move, delete) and +// all undo operations (single, to-id, to-savepoint) with deterministic PRNG-seeded +// randomization and hash-based verification. +// +// Build: go build -o bin/tigerfs-stress ./test/stress +// Usage: bin/tigerfs-stress start [--seed N] [--iterations N] [--debug] +// +// bin/tigerfs-stress stop +package main + +import ( + "flag" + "fmt" + "os" + "strconv" + "strings" + "time" +) + +// Config holds all CLI options for the stress test. +// +// DumpAtSpec is the raw `--dump-at` flag value (kept for the startup +// banner / replay command). DumpAt is the parsed lookup set used during +// the run: a key per iteration where a manual snapshot should fire. +type Config struct { + Seed int64 + Iterations int + Debug bool + Keep bool + Workspace string + ValidateEvery int + LargeFiles bool + ManyFiles bool + DumpAtSpec string + DumpAt map[int]bool +} + +func main() { + if len(os.Args) < 2 { + printUsage() + os.Exit(2) + } + + switch os.Args[1] { + case "start": + cfg := parseStartFlags(os.Args[2:]) + os.Exit(runStart(cfg)) + case "stop": + os.Exit(runStop()) + case "-h", "--help", "help": + printUsage() + os.Exit(0) + default: + fmt.Fprintf(os.Stderr, "Unknown command: %s\n\n", os.Args[1]) + printUsage() + os.Exit(2) + } +} + +func parseStartFlags(args []string) *Config { + cfg := &Config{} + fs := flag.NewFlagSet("start", flag.ExitOnError) + + fs.Int64Var(&cfg.Seed, "seed", 0, "PRNG seed (0 = generate from time)") + fs.IntVar(&cfg.Iterations, "iterations", 20, "number of operation rounds") + fs.BoolVar(&cfg.Debug, "debug", false, "pass --log-level debug to tigerfs") + fs.BoolVar(&cfg.Keep, "keep", false, "don't tear down Docker/mount on exit") + fs.StringVar(&cfg.Workspace, "workspace", "testws", "workspace name") + fs.IntVar(&cfg.ValidateEvery, "validate-every", 1, "validate workspace every N ops (undo always validates)") + fs.BoolVar(&cfg.LargeFiles, "large-files", false, "enable large file generation (up to 10MB)") + fs.BoolVar(&cfg.ManyFiles, "many-files", false, "enable dense directories (up to 1000 files/dir)") + fs.StringVar(&cfg.DumpAtSpec, "dump-at", "", "comma-separated iteration numbers to write a snapshot dump after (e.g., 100,250)") + + fs.Parse(args) + + // Generate seed if not provided + if cfg.Seed == 0 { + cfg.Seed = time.Now().UnixNano() + } + + cfg.DumpAt = parseDumpAtSpec(cfg.DumpAtSpec, cfg.Iterations) + + return cfg +} + +// parseDumpAtSpec splits a comma-separated iteration list into a +// deduplicated lookup set. Empty input returns nil (no snapshots). +// Invalid entries (non-integer, <=0, or > max) are warned to stderr and +// skipped so a typo doesn't silently neutralize the flag. +func parseDumpAtSpec(spec string, maxIter int) map[int]bool { + if spec == "" { + return nil + } + out := map[int]bool{} + for _, part := range strings.Split(spec, ",") { + part = strings.TrimSpace(part) + if part == "" { + continue + } + n, err := strconv.Atoi(part) + if err != nil || n <= 0 { + fmt.Fprintf(os.Stderr, "[WARN] --dump-at: ignoring invalid iteration %q\n", part) + continue + } + if n > maxIter { + fmt.Fprintf(os.Stderr, "[WARN] --dump-at %d: past --iterations %d, will never fire\n", n, maxIter) + continue + } + out[n] = true + } + if len(out) == 0 { + return nil + } + return out +} + +func runStart(cfg *Config) int { + fmt.Printf("=== tigerfs-stress ===\n") + fmt.Printf("Seed: %d\n", cfg.Seed) + fmt.Printf("Iterations: %d\n", cfg.Iterations) + fmt.Printf("Workspace: %s\n", cfg.Workspace) + fmt.Printf("Debug: %v\n", cfg.Debug) + fmt.Printf("LargeFiles: %v\n", cfg.LargeFiles) + fmt.Printf("ManyFiles: %v\n", cfg.ManyFiles) + fmt.Printf("Validate: every %d ops\n", cfg.ValidateEvery) + if len(cfg.DumpAt) > 0 { + fmt.Printf("DumpAt: %s\n", cfg.DumpAtSpec) + } + fmt.Println() + + // Set up infrastructure + infra, err := SetupInfra(cfg) + if err != nil { + fmt.Fprintf(os.Stderr, "[ERROR] Infrastructure setup failed: %v\n", err) + return 2 + } + + fmt.Printf("Infrastructure ready. Mountpoint: %s\n", infra.Mountpoint) + fmt.Printf("Workspace path: %s/%s\n", infra.Mountpoint, cfg.Workspace) + fmt.Println() + + // Run test iterations. RunAndExit returns KeepInfra=true on a + // validation failure (so the user can inspect live state alongside + // the failure dump); we honour both that signal and the explicit + // --keep flag. + res := RunAndExit(cfg, infra) + if cfg.Keep || res.KeepInfra { + fmt.Printf("\nInfrastructure left running at %s (use 'bin/tigerfs-stress stop' to tear down)\n", infra.Mountpoint) + } else { + infra.Teardown() + } + return res.ExitCode +} + +func runStop() int { + err := StopInfra() + if err != nil { + fmt.Fprintf(os.Stderr, "[ERROR] Stop failed: %v\n", err) + return 2 + } + fmt.Println("Stopped.") + return 0 +} + +func printUsage() { + fmt.Fprintf(os.Stderr, `tigerfs-stress - stress test for TigerFS file-first workspaces + +Usage: + tigerfs-stress start [OPTIONS] Start infrastructure, run test, teardown + tigerfs-stress stop Kill running test and teardown infrastructure + +Options: + --seed N PRNG seed for reproducibility (0 = random) + --iterations N Number of operation rounds (default: 20) + --debug Pass --log-level debug to tigerfs + --keep Don't tear down on exit + --workspace NAME Workspace name (default: testws) + --validate-every N Validate every N ops (default: 1; undo always validates) + --large-files Enable large files up to 10MB (default max: 100KB) + --many-files Enable dense directories up to 1000 files/dir (default: 10) + --dump-at LIST Write a snapshot dump after these iterations (e.g., 100,250) + +Examples: + tigerfs-stress start + tigerfs-stress start --seed 42 --iterations 50 + tigerfs-stress start --large-files --many-files --iterations 100 --validate-every 5 + tigerfs-stress start --debug --keep --seed 42 + tigerfs-stress start --seed 42 --iterations 1000 --dump-at 100,500,778 + tigerfs-stress stop +`) +} diff --git a/test/stress/operations.go b/test/stress/operations.go new file mode 100644 index 0000000..a577128 --- /dev/null +++ b/test/stress/operations.go @@ -0,0 +1,883 @@ +package main + +import ( + "fmt" + "math" + "math/rand" + "os" + "path/filepath" + "sort" + "strings" +) + +// SizeConfig controls the file size distribution. +type SizeConfig struct { + MaxBytes int + MeanLog float64 // mu of underlying normal (log-space) + StdDevLog float64 // sigma of underlying normal (log-space) +} + +// DensityConfig controls directory density limits. +type DensityConfig struct { + MaxFilesPerDir int + MaxSubdirsPerDir int +} + +var ( + defaultSizeConfig = SizeConfig{ + MaxBytes: 100 * 1024, // 100KB + MeanLog: 8.5, // ~5KB typical + StdDevLog: 1.5, // spread: 500B - 50KB + } + largeSizeConfig = SizeConfig{ + MaxBytes: 10 * 1024 * 1024, // 10MB + MeanLog: 10.0, // ~22KB typical + StdDevLog: 2.5, // spread: 1KB - 1MB common, occasional 10MB + } + defaultDensityConfig = DensityConfig{ + MaxFilesPerDir: 10, + MaxSubdirsPerDir: 3, + } + manyFilesDensityConfig = DensityConfig{ + MaxFilesPerDir: 1000, + MaxSubdirsPerDir: 20, + } +) + +// OpConfig bundles all configuration for operations. +type OpConfig struct { + Size SizeConfig + Density DensityConfig +} + +// NewOpConfig creates an OpConfig from CLI flags. +func NewOpConfig(largeFiles, manyFiles bool) *OpConfig { + cfg := &OpConfig{ + Size: defaultSizeConfig, + Density: defaultDensityConfig, + } + if largeFiles { + cfg.Size = largeSizeConfig + } + if manyFiles { + cfg.Density = manyFilesDensityConfig + } + return cfg +} + +// Pools tracks available files and directories for operation targeting. +type Pools struct { + Files []string // relative paths of existing files + Dirs []string // relative paths of existing directories (empty string = root) +} + +// NewPools creates pools with just the root directory. +func NewPools() *Pools { + return &Pools{ + Files: nil, + Dirs: []string{""}, // root + } +} + +// AddFile adds a file to the pool. +func (p *Pools) AddFile(relPath string) { + p.Files = append(p.Files, relPath) +} + +// RemoveFile removes a file from the pool. +func (p *Pools) RemoveFile(relPath string) { + for i, f := range p.Files { + if f == relPath { + p.Files = append(p.Files[:i], p.Files[i+1:]...) + return + } + } +} + +// AddDir adds a directory to the pool. +func (p *Pools) AddDir(relPath string) { + p.Dirs = append(p.Dirs, relPath) +} + +// RemoveDir removes a directory and all files/subdirs under it from pools. +func (p *Pools) RemoveDir(relPath string) { + prefix := relPath + "/" + + // Remove the dir itself + for i, d := range p.Dirs { + if d == relPath { + p.Dirs = append(p.Dirs[:i], p.Dirs[i+1:]...) + break + } + } + + // Remove subdirs + filtered := p.Dirs[:0] + for _, d := range p.Dirs { + if !strings.HasPrefix(d, prefix) { + filtered = append(filtered, d) + } + } + p.Dirs = filtered + + // Remove files + filteredFiles := p.Files[:0] + for _, f := range p.Files { + if !strings.HasPrefix(f, prefix) && filepath.Dir(f) != relPath { + filteredFiles = append(filteredFiles, f) + } + } + // Also remove files directly in the dir + var finalFiles []string + for _, f := range filteredFiles { + dir := filepath.Dir(f) + if dir == "." { + dir = "" + } + if dir == relPath { + continue + } + finalFiles = append(finalFiles, f) + } + p.Files = finalFiles +} + +// RenameFile renames a file in the pool. +func (p *Pools) RenameFile(oldPath, newPath string) { + for i, f := range p.Files { + if f == oldPath { + p.Files[i] = newPath + return + } + } +} + +// RenameDir renames a directory and updates all nested paths. +func (p *Pools) RenameDir(oldPath, newPath string) { + oldPrefix := oldPath + "/" + newPrefix := newPath + "/" + + for i, d := range p.Dirs { + if d == oldPath { + p.Dirs[i] = newPath + } else if strings.HasPrefix(d, oldPrefix) { + p.Dirs[i] = newPrefix + strings.TrimPrefix(d, oldPrefix) + } + } + + for i, f := range p.Files { + if strings.HasPrefix(f, oldPrefix) { + p.Files[i] = newPrefix + strings.TrimPrefix(f, oldPrefix) + } + } +} + +// NonRootDirs returns directories excluding the root. +func (p *Pools) NonRootDirs() []string { + var result []string + for _, d := range p.Dirs { + if d != "" { + result = append(result, d) + } + } + return result +} + +// generateFileSize returns a random file size using log-normal distribution. +func generateFileSize(rng *rand.Rand, cfg SizeConfig) int { + logSize := cfg.MeanLog + cfg.StdDevLog*rng.NormFloat64() + size := int(math.Exp(logSize)) + if size < 64 { + size = 64 + } + if size > cfg.MaxBytes { + size = cfg.MaxBytes + } + return size +} + +// generateContent creates deterministic markdown content of approximately targetSize bytes. +// Content always ends at a newline boundary to avoid truncation artifacts (null bytes +// from mid-string slicing that PostgreSQL TEXT columns can't store). +func generateContent(rng *rand.Rand, title string, targetSize int) string { + var buf strings.Builder + buf.Grow(targetSize + 200) + fmt.Fprintf(&buf, "---\ntitle: %s\n---\n\n", title) + lineNum := 0 + for buf.Len() < targetSize { + nWords := 5 + rng.Intn(15) + fmt.Fprintf(&buf, "Line %d: %s\n", lineNum, randomWords(rng, nWords)) + lineNum++ + } + // Don't truncate mid-line -- return at the last complete line. + // Content may slightly exceed targetSize (by up to one line ~200 chars). + return buf.String() +} + +// Word pool for content generation (deterministic, no external dependencies). +var wordPool = []string{ + "the", "quick", "brown", "fox", "jumps", "over", "lazy", "dog", + "alpha", "beta", "gamma", "delta", "epsilon", "zeta", "eta", "theta", + "lorem", "ipsum", "dolor", "sit", "amet", "consectetur", "adipiscing", + "data", "query", "index", "table", "schema", "column", "row", "view", + "create", "update", "delete", "insert", "select", "filter", "order", + "first", "last", "sample", "export", "import", "mount", "build", + "file", "directory", "path", "name", "hash", "content", "version", + "undo", "redo", "save", "restore", "checkpoint", "rollback", "commit", +} + +// randomWords returns n random words from the word pool. +func randomWords(rng *rand.Rand, n int) string { + words := make([]string, n) + for i := range words { + words[i] = wordPool[rng.Intn(len(wordPool))] + } + return strings.Join(words, " ") +} + +// randomName generates a random filename-safe name. +func randomName(rng *rand.Rand) string { + prefixes := []string{ + "doc", "note", "memo", "report", "guide", "spec", + "draft", "plan", "log", "ref", "brief", "summary", + } + prefix := prefixes[rng.Intn(len(prefixes))] + suffix := rng.Intn(10000) + return fmt.Sprintf("%s-%04d", prefix, suffix) +} + +// randomDirName generates a random directory name. +func randomDirName(rng *rand.Rand) string { + names := []string{ + "docs", "notes", "drafts", "specs", "guides", "refs", + "archive", "inbox", "review", "staging", "projects", "topics", + } + name := names[rng.Intn(len(names))] + suffix := rng.Intn(1000) + return fmt.Sprintf("%s-%03d", name, suffix) +} + +// readBackHash reads a file back from TigerFS and returns the hash of +// what TigerFS returns. This is necessary because TigerFS synth views +// re-synthesize content (parse frontmatter into columns, reconstruct +// on read), so the returned content may differ from what was written. +// +// Uses explicit open/close to ensure the NFS client fetches fresh data +// from the server rather than serving from the write cache. +func readBackHash(fullPath string) (string, error) { + data, err := os.ReadFile(fullPath) + if err != nil { + return "", fmt.Errorf("read back %s: %w", fullPath, err) + } + return HashContent(data), nil +} + +// --- Filesystem Operations --- +// Each operation takes the workspace path, rng, pools, state, and config. +// It performs the real filesystem operation and updates the expected state. +// Returns a description string for logging, or an error. + +// OpCreateFile creates a new markdown file in a random directory. +// Returns the description, the number of bytes written, and any error. +func OpCreateFile(wsPath string, rng *rand.Rand, pools *Pools, state *WorkspaceState, cfg *OpConfig) (string, int, error) { + // Pick a directory with capacity + dir := pickDirWithCapacity(rng, pools, state, cfg.Density.MaxFilesPerDir) + name := randomName(rng) + ".md" + relPath := name + if dir != "" { + relPath = dir + "/" + name + } + + size := generateFileSize(rng, cfg.Size) + content := generateContent(rng, name, size) + written := len(content) + + fullPath := filepath.Join(wsPath, relPath) + if err := os.MkdirAll(filepath.Dir(fullPath), 0755); err != nil { + return "", 0, fmt.Errorf("mkdir for %s: %w", relPath, err) + } + if err := os.WriteFile(fullPath, []byte(content), 0644); err != nil { + return "", 0, fmt.Errorf("write %s: %w", relPath, err) + } + + // Read back from TigerFS to get the synthesized content hash + // (TigerFS re-synthesizes markdown from structured columns) + hash, err := readBackHash(fullPath) + if err != nil { + return "", 0, err + } + + state.SetFile(relPath, hash) + pools.AddFile(relPath) + + return fmt.Sprintf("create_file %s (%s)", relPath, formatSize(size)), written, nil +} + +// OpEditFile modifies an existing file. +func OpEditFile(wsPath string, rng *rand.Rand, pools *Pools, state *WorkspaceState, cfg *OpConfig) (string, error) { + if len(pools.Files) == 0 { + return "", fmt.Errorf("no files to edit") + } + + idx := rng.Intn(len(pools.Files)) + relPath := pools.Files[idx] + size := generateFileSize(rng, cfg.Size) + content := generateContent(rng, filepath.Base(relPath), size) + + fullPath := filepath.Join(wsPath, relPath) + if err := os.WriteFile(fullPath, []byte(content), 0644); err != nil { + return "", fmt.Errorf("write %s: %w", relPath, err) + } + + // Read back synthesized content + hash, err := readBackHash(fullPath) + if err != nil { + return "", err + } + + state.SetFile(relPath, hash) + + return fmt.Sprintf("edit_file %s (%s)", relPath, formatSize(size)), nil +} + +// OpRenameFile renames a file within the same directory. +func OpRenameFile(wsPath string, rng *rand.Rand, pools *Pools, state *WorkspaceState, _ *OpConfig) (string, error) { + if len(pools.Files) == 0 { + return "", fmt.Errorf("no files to rename") + } + + idx := rng.Intn(len(pools.Files)) + oldRelPath := pools.Files[idx] + dir := filepath.Dir(oldRelPath) + if dir == "." { + dir = "" + } + + newName := randomName(rng) + ".md" + newRelPath := newName + if dir != "" { + newRelPath = dir + "/" + newName + } + + oldFull := filepath.Join(wsPath, oldRelPath) + newFull := filepath.Join(wsPath, newRelPath) + if err := os.Rename(oldFull, newFull); err != nil { + return "", fmt.Errorf("rename %s -> %s: %w", oldRelPath, newRelPath, err) + } + + state.RenameFile(oldRelPath, newRelPath) + pools.RenameFile(oldRelPath, newRelPath) + + return fmt.Sprintf("rename_file %s -> %s", oldRelPath, newRelPath), nil +} + +// OpMoveFile moves a file to a different directory. +func OpMoveFile(wsPath string, rng *rand.Rand, pools *Pools, state *WorkspaceState, _ *OpConfig) (string, error) { + if len(pools.Files) == 0 || len(pools.Dirs) < 2 { + return "", fmt.Errorf("need files and multiple dirs to move") + } + + fileIdx := rng.Intn(len(pools.Files)) + oldRelPath := pools.Files[fileIdx] + oldDir := filepath.Dir(oldRelPath) + if oldDir == "." { + oldDir = "" + } + + // Enumerate dirs other than the file's current parent and pick one + // uniformly. Random-with-retries can flake here: with only 2 dirs + // (root + one), each attempt has a 50% chance of picking oldDir, so + // 10 attempts fail ~0.1% of the time. canExecute already guarantees + // len(pools.Dirs) >= 2, and oldDir is one of those entries, so the + // candidate slice is always non-empty. + candidates := make([]string, 0, len(pools.Dirs)-1) + for _, d := range pools.Dirs { + if d != oldDir { + candidates = append(candidates, d) + } + } + if len(candidates) == 0 { + return "", fmt.Errorf("no different directory available for move_file %s", oldRelPath) + } + newDir := candidates[rng.Intn(len(candidates))] + + baseName := filepath.Base(oldRelPath) + newRelPath := baseName + if newDir != "" { + newRelPath = newDir + "/" + baseName + } + + oldFull := filepath.Join(wsPath, oldRelPath) + newFull := filepath.Join(wsPath, newRelPath) + if err := os.MkdirAll(filepath.Dir(newFull), 0755); err != nil { + return "", fmt.Errorf("mkdir for move: %w", err) + } + if err := os.Rename(oldFull, newFull); err != nil { + return "", fmt.Errorf("move %s -> %s: %w", oldRelPath, newRelPath, err) + } + + state.RenameFile(oldRelPath, newRelPath) + pools.RenameFile(oldRelPath, newRelPath) + + return fmt.Sprintf("move_file %s -> %s", oldRelPath, newRelPath), nil +} + +// OpDeleteFile deletes an existing file. +func OpDeleteFile(wsPath string, rng *rand.Rand, pools *Pools, state *WorkspaceState, _ *OpConfig) (string, error) { + if len(pools.Files) == 0 { + return "", fmt.Errorf("no files to delete") + } + + idx := rng.Intn(len(pools.Files)) + relPath := pools.Files[idx] + + fullPath := filepath.Join(wsPath, relPath) + if err := os.Remove(fullPath); err != nil { + return "", fmt.Errorf("delete %s: %w", relPath, err) + } + + state.RemoveFile(relPath) + pools.RemoveFile(relPath) + + return fmt.Sprintf("delete_file %s", relPath), nil +} + +// OpCreateDir creates a new subdirectory. +func OpCreateDir(wsPath string, rng *rand.Rand, pools *Pools, state *WorkspaceState, cfg *OpConfig) (string, error) { + // Pick a parent directory with subdir capacity + parent := pickDirWithSubdirCapacity(rng, pools, state, cfg.Density.MaxSubdirsPerDir) + name := randomDirName(rng) + relPath := name + if parent != "" { + relPath = parent + "/" + name + } + + fullPath := filepath.Join(wsPath, relPath) + if err := os.MkdirAll(fullPath, 0755); err != nil { + return "", fmt.Errorf("mkdir %s: %w", relPath, err) + } + + state.AddDir(relPath) + pools.AddDir(relPath) + + return fmt.Sprintf("create_dir %s", relPath), nil +} + +// OpRenameDir renames a non-root directory. +func OpRenameDir(wsPath string, rng *rand.Rand, pools *Pools, state *WorkspaceState, _ *OpConfig) (string, error) { + nonRoot := pools.NonRootDirs() + if len(nonRoot) == 0 { + return "", fmt.Errorf("no non-root dirs to rename") + } + + idx := rng.Intn(len(nonRoot)) + oldRelPath := nonRoot[idx] + parent := filepath.Dir(oldRelPath) + if parent == "." { + parent = "" + } + oldName := filepath.Base(oldRelPath) + + // randomDirName has 12 prefixes * 1000 suffixes = 12k options; collisions + // with the existing name are rare but possible. Re-roll up to a few + // times to avoid `os.Rename(A, A)` which fails with EEXIST. + var newName string + for attempt := 0; attempt < 5; attempt++ { + newName = randomDirName(rng) + if newName != oldName { + break + } + } + if newName == oldName { + return "", fmt.Errorf("rename_dir: could not generate a different name for %s after retries", oldRelPath) + } + + newRelPath := newName + if parent != "" { + newRelPath = parent + "/" + newName + } + + oldFull := filepath.Join(wsPath, oldRelPath) + newFull := filepath.Join(wsPath, newRelPath) + if err := os.Rename(oldFull, newFull); err != nil { + return "", fmt.Errorf("rename dir %s -> %s: %w", oldRelPath, newRelPath, err) + } + + state.RenameDir(oldRelPath, newRelPath) + pools.RenameDir(oldRelPath, newRelPath) + + return fmt.Sprintf("rename_dir %s -> %s", oldRelPath, newRelPath), nil +} + +// OpMoveDir moves a non-root directory (which may contain files/subdirs) +// into a different parent directory. Sources are biased toward dirs with +// contents so the recursive case is regularly exercised. The destination +// must not be the source's current parent, the source itself, or a +// descendant of the source. If the biased source has no valid destination, +// other non-root sources are tried in random order before giving up. +func OpMoveDir(wsPath string, rng *rand.Rand, pools *Pools, state *WorkspaceState, _ *OpConfig) (string, error) { + srcs := orderedMoveDirSources(rng, pools, state) + if len(srcs) == 0 { + return "", fmt.Errorf("no non-root dirs to move") + } + + var src, newRelPath string + for _, candidate := range srcs { + dests := validMoveDirDests(candidate, pools, state) + if len(dests) == 0 { + continue + } + src = candidate + dest := dests[rng.Intn(len(dests))] + baseName := filepath.Base(src) + newRelPath = baseName + if dest != "" { + newRelPath = dest + "/" + baseName + } + break + } + if newRelPath == "" { + return "", fmt.Errorf("no valid (src, dest) pair for move_dir") + } + + fileCount, subdirCount := countDirContents(src, state) + + oldFull := filepath.Join(wsPath, src) + newFull := filepath.Join(wsPath, newRelPath) + if err := os.MkdirAll(filepath.Dir(newFull), 0755); err != nil { + return "", fmt.Errorf("mkdir for move_dir: %w", err) + } + if err := os.Rename(oldFull, newFull); err != nil { + return "", fmt.Errorf("move_dir %s -> %s: %w", src, newRelPath, err) + } + + state.RenameDir(src, newRelPath) + pools.RenameDir(src, newRelPath) + + return fmt.Sprintf("move_dir %s -> %s (%d files, %d subdirs)", src, newRelPath, fileCount, subdirCount), nil +} + +// OpDeleteDir recursively deletes a non-root directory and everything inside. +// +// TigerFS records one log entry per delete (parent_id_fkey forces leaves to +// go first), so a delete_dir produces N entries in total. Each one is an +// independent undo target -- ExecuteUndoSingle restores exactly one row, +// not "the whole delete_dir." If we push a single stack entry covering the +// whole op, undo_single's pop returns "state before delete_dir" while +// TigerFS only restores one row, and validation fails with state mismatches +// (cannot recover, since deferred-FK requires the parent dir to exist by +// COMMIT time -- restoring just a child of a deleted dir would orphan it). +// +// Instead, walk the tree explicitly and push one stack entry per deletion. +// The first deletion uses the entry the runner already pushed before this +// op; each subsequent deletion gets a fresh entry capturing the state right +// before that particular row was removed. undo_single can then target any +// individual deletion safely. +func OpDeleteDir(wsPath string, rng *rand.Rand, pools *Pools, state *WorkspaceState, _ *OpConfig, stack *StateStack, iteration int, stats *Stats) (string, error) { + src := pickNonRootDirBiased(rng, pools, state) + if src == "" { + return "", fmt.Errorf("no non-root dirs to delete") + } + + fileCount, subdirCount := countDirContents(src, state) + deletions, err := collectDeletionOrder(wsPath, src) + if err != nil { + return "", fmt.Errorf("walk %s for deletion: %w", src, err) + } + + // lastSeenLogID enforces monotonicity across the per-row reads below. + // Empirically the same staleness window that affects post-undo reads + // also affects these per-row reads in tight succession (snapshot dump + // at iter 471 caught three reads stale by 1.5s, with each subsequent + // read advancing slightly but still trailing the actual newest). + // On regression the helper returns the prior; we treat that as "do + // not tag this entry" so a stale id never lands on a newer stack + // entry (which would point undo_single at the wrong log row). + var lastSeenLogID string + + for i, item := range deletions { + // First deletion uses the runner-supplied stack entry; subsequent + // deletions get a fresh entry with the in-progress state. + if i > 0 { + stack.Push(state, iteration) + } + + fullPath := filepath.Join(wsPath, item.path) + if err := os.Remove(fullPath); err != nil { + return "", fmt.Errorf("delete %s: %w", item.path, err) + } + if item.isDir { + state.RemoveDir(item.path) + } else { + state.RemoveFile(item.path) + } + + // Pin the freshly-produced log_id on the entry that just + // captured the pre-deletion state, but only if it advances past + // the prior tagged id. If the read regresses (stale snapshot) + // or stays equal (helper kept prior on retry exhaustion), we + // leave this entry unlogged so undo_single will skip it -- it's + // safer to lose targetability for one deletion than to misroute + // undo_single onto a completely unrelated log row. + logID := readLatestLogIDMonotonic(wsPath, lastSeenLogID, iteration, + fmt.Sprintf("delete_dir per-row %s", item.path), stats) + if logID != "" && logID > lastSeenLogID { + stack.SetLastLogID(logID) + lastSeenLogID = logID + } + } + + pools.RemoveDir(src) + + return fmt.Sprintf("delete_dir %s (%d files, %d subdirs)", src, fileCount, subdirCount), nil +} + +// deletionItem is one entry in collectDeletionOrder's output. +type deletionItem struct { + path string + isDir bool +} + +// collectDeletionOrder walks the *actual* filesystem under src and returns +// every entry in post-order (children before their parent), with src itself +// last. Hidden entries (those starting with ".") are skipped -- TigerFS +// virtual paths such as .log/ aren't real children to remove. +// +// We deliberately walk the FS rather than reading WorkspaceState. After the +// mkdirSynth-logging fix, the test's tracked state should stay in sync with +// TigerFS across undos, so a state-based traversal would also work in the +// common case. Walking the FS is kept as defensive coding: any future state +// drift (a new unlogged op, a tracking bug in move_dir/rename_dir, etc.) +// would otherwise surface as cryptic ENOTEMPTY/EIO failures from os.Remove +// hitting unexpected children. The cost is a few NFS-mediated readdirs per +// delete_dir, which is negligible next to the N+1 deletes the op already +// makes. +func collectDeletionOrder(wsPath, src string) ([]deletionItem, error) { + var out []deletionItem + var walk func(rel string) error + walk = func(rel string) error { + entries, err := os.ReadDir(filepath.Join(wsPath, rel)) + if err != nil { + return err + } + // Sort for determinism within a directory. + sort.Slice(entries, func(i, j int) bool { + return entries[i].Name() < entries[j].Name() + }) + for _, e := range entries { + name := e.Name() + if strings.HasPrefix(name, ".") { + continue + } + childRel := filepath.Join(rel, name) + if e.IsDir() { + if err := walk(childRel); err != nil { + return err + } + out = append(out, deletionItem{path: childRel, isDir: true}) + } else { + out = append(out, deletionItem{path: childRel, isDir: false}) + } + } + return nil + } + + if err := walk(src); err != nil { + return nil, err + } + out = append(out, deletionItem{path: src, isDir: true}) + return out, nil +} + +// OpCreateSavepoint creates a named savepoint. +func OpCreateSavepoint(wsPath string, rng *rand.Rand, _ *Pools, _ *WorkspaceState, _ *OpConfig, iteration int, stack *StateStack) (string, error) { + name := fmt.Sprintf("sp-%d-%04d", iteration, rng.Intn(10000)) + spPath := filepath.Join(wsPath, ".savepoint", name+".json") + content := fmt.Sprintf(`{"description":"Savepoint at iteration %d"}`, iteration) + + if err := os.WriteFile(spPath, []byte(content), 0644); err != nil { + return "", fmt.Errorf("create savepoint %s: %w", name, err) + } + + stack.SaveSavepoint(name) + + return fmt.Sprintf("create_savepoint %s", name), nil +} + +// RebuildPools reconstructs pools from the workspace state. +// Used after undo operations to sync pools with the restored state. +func RebuildPools(state *WorkspaceState) *Pools { + pools := &Pools{ + Dirs: []string{""}, // root always exists + } + for relPath := range state.Files { + pools.Files = append(pools.Files, relPath) + } + for relPath := range state.Dirs { + pools.Dirs = append(pools.Dirs, relPath) + } + return pools +} + +// --- Helpers --- + +func pickDirWithCapacity(rng *rand.Rand, pools *Pools, state *WorkspaceState, maxFiles int) string { + // Try random directories, find one with capacity + for attempts := 0; attempts < 20; attempts++ { + dir := pools.Dirs[rng.Intn(len(pools.Dirs))] + if state.FileCount(dir) < maxFiles { + return dir + } + } + // Fallback: root always works (or first dir with space) + return pools.Dirs[0] +} + +func pickDirWithSubdirCapacity(rng *rand.Rand, pools *Pools, state *WorkspaceState, maxSubdirs int) string { + for attempts := 0; attempts < 20; attempts++ { + dir := pools.Dirs[rng.Intn(len(pools.Dirs))] + if state.SubdirCount(dir) < maxSubdirs { + return dir + } + } + return pools.Dirs[0] +} + +// orderedMoveDirSources returns non-root dirs in the order the picker should +// try them for move_dir: the biased preferred candidate first (prefers dirs +// with nested contents), then the remaining non-root dirs in random order. +// This lets OpMoveDir keep the non-empty bias while still finding a valid +// (src, dest) pair whenever one exists. +func orderedMoveDirSources(rng *rand.Rand, pools *Pools, state *WorkspaceState) []string { + first := pickNonRootDirBiased(rng, pools, state) + if first == "" { + return nil + } + ordered := []string{first} + rest := make([]string, 0, len(pools.NonRootDirs())-1) + for _, d := range pools.NonRootDirs() { + if d != first { + rest = append(rest, d) + } + } + rng.Shuffle(len(rest), func(i, j int) { rest[i], rest[j] = rest[j], rest[i] }) + return append(ordered, rest...) +} + +// validMoveDirDests returns all pool directories that are legal destinations +// for moving src: not src's current parent, not src itself, not a descendant +// of src, and not already occupied by a dir of the same basename. +func validMoveDirDests(src string, pools *Pools, state *WorkspaceState) []string { + oldParent := filepath.Dir(src) + if oldParent == "." { + oldParent = "" + } + baseName := filepath.Base(src) + descendantPrefix := src + "/" + + var out []string + for _, dest := range pools.Dirs { + if dest == oldParent || dest == src { + continue + } + if strings.HasPrefix(dest, descendantPrefix) { + continue + } + proposed := baseName + if dest != "" { + proposed = dest + "/" + baseName + } + if _, exists := state.Dirs[proposed]; exists { + continue + } + out = append(out, dest) + } + return out +} + +// canMoveDir reports whether at least one valid (src, dest) pair exists. +// Used by canExecute so opMoveDir is only selected when feasible. +func canMoveDir(pools *Pools, state *WorkspaceState) bool { + for _, src := range pools.NonRootDirs() { + if len(validMoveDirDests(src, pools, state)) > 0 { + return true + } + } + return false +} + +// pickNonRootDirBiased chooses a non-root directory, preferring ones that +// contain files or subdirectories so move_dir / delete_dir exercise recursive +// behavior. Empty dirs are still reachable as a secondary bucket. Returns "" +// if no non-root dirs exist. +func pickNonRootDirBiased(rng *rand.Rand, pools *Pools, state *WorkspaceState) string { + nonRoot := pools.NonRootDirs() + if len(nonRoot) == 0 { + return "" + } + + var withContent, empty []string + for _, d := range nonRoot { + if dirHasContents(d, state) { + withContent = append(withContent, d) + } else { + empty = append(empty, d) + } + } + + // When both groups exist, prefer non-empty dirs 70% of the time. + if len(withContent) > 0 && len(empty) > 0 { + if rng.Intn(10) < 7 { + return withContent[rng.Intn(len(withContent))] + } + return empty[rng.Intn(len(empty))] + } + if len(withContent) > 0 { + return withContent[rng.Intn(len(withContent))] + } + return empty[rng.Intn(len(empty))] +} + +// dirHasContents reports whether the given directory has any nested files +// or subdirectories in the expected state. +func dirHasContents(dir string, state *WorkspaceState) bool { + prefix := dir + "/" + for f := range state.Files { + if strings.HasPrefix(f, prefix) { + return true + } + } + for d := range state.Dirs { + if strings.HasPrefix(d, prefix) { + return true + } + } + return false +} + +// countDirContents returns the number of files and subdirectories nested +// anywhere beneath dir (recursive). +func countDirContents(dir string, state *WorkspaceState) (files, subdirs int) { + prefix := dir + "/" + for f := range state.Files { + if strings.HasPrefix(f, prefix) { + files++ + } + } + for d := range state.Dirs { + if strings.HasPrefix(d, prefix) { + subdirs++ + } + } + return +} + +func formatSize(bytes int) string { + if bytes < 1024 { + return fmt.Sprintf("%dB", bytes) + } + if bytes < 1024*1024 { + return fmt.Sprintf("%.1fKB", float64(bytes)/1024) + } + return fmt.Sprintf("%.1fMB", float64(bytes)/(1024*1024)) +} diff --git a/test/stress/operations_test.go b/test/stress/operations_test.go new file mode 100644 index 0000000..f9dade6 --- /dev/null +++ b/test/stress/operations_test.go @@ -0,0 +1,493 @@ +package main + +import ( + "math/rand" + "os" + "path/filepath" + "strings" + "testing" +) + +func TestGenerateFileSize_DefaultBounds(t *testing.T) { + rng := rand.New(rand.NewSource(42)) + cfg := defaultSizeConfig + + for i := 0; i < 1000; i++ { + size := generateFileSize(rng, cfg) + if size < 64 { + t.Errorf("iteration %d: size %d below minimum 64", i, size) + } + if size > cfg.MaxBytes { + t.Errorf("iteration %d: size %d above max %d", i, size, cfg.MaxBytes) + } + } +} + +func TestGenerateFileSize_LargeBounds(t *testing.T) { + rng := rand.New(rand.NewSource(42)) + cfg := largeSizeConfig + + for i := 0; i < 1000; i++ { + size := generateFileSize(rng, cfg) + if size < 64 { + t.Errorf("iteration %d: size %d below minimum 64", i, size) + } + if size > cfg.MaxBytes { + t.Errorf("iteration %d: size %d above max %d", i, size, cfg.MaxBytes) + } + } +} + +func TestGenerateFileSize_Deterministic(t *testing.T) { + sizes1 := make([]int, 20) + sizes2 := make([]int, 20) + + rng1 := rand.New(rand.NewSource(123)) + rng2 := rand.New(rand.NewSource(123)) + + for i := range sizes1 { + sizes1[i] = generateFileSize(rng1, defaultSizeConfig) + sizes2[i] = generateFileSize(rng2, defaultSizeConfig) + } + + for i := range sizes1 { + if sizes1[i] != sizes2[i] { + t.Errorf("iteration %d: sizes differ (%d vs %d) with same seed", i, sizes1[i], sizes2[i]) + } + } +} + +func TestGenerateContent_Deterministic(t *testing.T) { + rng1 := rand.New(rand.NewSource(42)) + rng2 := rand.New(rand.NewSource(42)) + + c1 := generateContent(rng1, "test", 500) + c2 := generateContent(rng2, "test", 500) + + if c1 != c2 { + t.Error("same seed should produce identical content") + } +} + +func TestGenerateContent_ApproximateSize(t *testing.T) { + rng := rand.New(rand.NewSource(42)) + + for _, targetSize := range []int{64, 500, 5000, 50000} { + content := generateContent(rng, "test", targetSize) + // Content ends at a natural line boundary, so it will be at least targetSize + // but may overshoot by up to one line (~200 bytes max). + if len(content) < targetSize { + t.Errorf("content length %d shorter than target %d", len(content), targetSize) + } + maxOvershoot := 300 + if len(content) > targetSize+maxOvershoot { + t.Errorf("content length %d overshoots target %d by more than %d", len(content), targetSize, maxOvershoot) + } + } +} + +func TestRandomName_Valid(t *testing.T) { + rng := rand.New(rand.NewSource(42)) + + for i := 0; i < 100; i++ { + name := randomName(rng) + if name == "" { + t.Errorf("iteration %d: empty name", i) + } + if len(name) > 20 { + t.Errorf("iteration %d: name too long: %s", i, name) + } + } +} + +func TestRandomName_Deterministic(t *testing.T) { + rng1 := rand.New(rand.NewSource(42)) + rng2 := rand.New(rand.NewSource(42)) + + for i := 0; i < 20; i++ { + n1 := randomName(rng1) + n2 := randomName(rng2) + if n1 != n2 { + t.Errorf("iteration %d: names differ with same seed: %s vs %s", i, n1, n2) + } + } +} + +func TestRandomWords(t *testing.T) { + rng := rand.New(rand.NewSource(42)) + words := randomWords(rng, 5) + if words == "" { + t.Error("randomWords returned empty string") + } + // Should have spaces + parts := len(words) - len(strings.ReplaceAll(words, " ", "")) + if parts != 4 { // 5 words = 4 spaces + t.Errorf("expected 4 spaces in 5 words, got %d", parts) + } +} + +func TestPools_Basic(t *testing.T) { + pools := NewPools() + + if len(pools.Dirs) != 1 || pools.Dirs[0] != "" { + t.Error("new pools should have root dir") + } + + pools.AddFile("a.md") + pools.AddFile("b.md") + pools.AddDir("docs") + + if len(pools.Files) != 2 { + t.Errorf("expected 2 files, got %d", len(pools.Files)) + } + + pools.RemoveFile("a.md") + if len(pools.Files) != 1 || pools.Files[0] != "b.md" { + t.Error("RemoveFile didn't work correctly") + } + + nonRoot := pools.NonRootDirs() + if len(nonRoot) != 1 || nonRoot[0] != "docs" { + t.Error("NonRootDirs should return docs") + } +} + +func TestPools_RenameDir(t *testing.T) { + pools := NewPools() + pools.AddDir("docs") + pools.AddDir("docs/sub") + pools.AddFile("docs/a.md") + pools.AddFile("docs/sub/b.md") + pools.AddFile("root.md") + + pools.RenameDir("docs", "notes") + + found := false + for _, d := range pools.Dirs { + if d == "docs" { + t.Error("old dir still in pool") + } + if d == "notes" { + found = true + } + } + if !found { + t.Error("new dir not in pool") + } + + for _, f := range pools.Files { + if strings.HasPrefix(f, "docs/") { + t.Errorf("file %s still has old prefix", f) + } + } +} + +func TestDirHasContents(t *testing.T) { + state := NewWorkspaceState() + state.AddDir("docs") + state.AddDir("empty") + state.AddDir("docs/sub") + state.SetFile("docs/sub/a.md", "h1") + + if !dirHasContents("docs", state) { + t.Error("docs should have contents (subdir + file)") + } + if !dirHasContents("docs/sub", state) { + t.Error("docs/sub should have contents (a.md)") + } + if dirHasContents("empty", state) { + t.Error("empty should have no contents") + } +} + +func TestCountDirContents(t *testing.T) { + state := NewWorkspaceState() + state.AddDir("docs") + state.AddDir("docs/sub") + state.AddDir("docs/sub/deep") + state.SetFile("docs/a.md", "h1") + state.SetFile("docs/sub/b.md", "h2") + state.SetFile("docs/sub/deep/c.md", "h3") + state.SetFile("other.md", "h4") // outside docs + + files, subdirs := countDirContents("docs", state) + if files != 3 { + t.Errorf("docs files: got %d, want 3", files) + } + if subdirs != 2 { + t.Errorf("docs subdirs: got %d, want 2", subdirs) + } + + files, subdirs = countDirContents("docs/sub", state) + if files != 2 { + t.Errorf("docs/sub files: got %d, want 2", files) + } + if subdirs != 1 { + t.Errorf("docs/sub subdirs: got %d, want 1", subdirs) + } +} + +func TestPickNonRootDirBiased_Empty(t *testing.T) { + pools := NewPools() + state := NewWorkspaceState() + rng := rand.New(rand.NewSource(1)) + if got := pickNonRootDirBiased(rng, pools, state); got != "" { + t.Errorf("no non-root dirs: got %q, want empty", got) + } +} + +func TestPickNonRootDirBiased_OnlyEmpty(t *testing.T) { + pools := NewPools() + pools.AddDir("a") + pools.AddDir("b") + state := NewWorkspaceState() + state.AddDir("a") + state.AddDir("b") + rng := rand.New(rand.NewSource(1)) + + for i := 0; i < 10; i++ { + got := pickNonRootDirBiased(rng, pools, state) + if got != "a" && got != "b" { + t.Errorf("iter %d: got %q, want a or b", i, got) + } + } +} + +func TestPickNonRootDirBiased_PrefersNonEmpty(t *testing.T) { + // Set up: one dir with contents, three empty dirs. + // Expect the non-empty dir to be chosen ~70% of the time. + pools := NewPools() + pools.AddDir("full") + pools.AddDir("empty1") + pools.AddDir("empty2") + pools.AddDir("empty3") + pools.AddFile("full/a.md") + + state := NewWorkspaceState() + state.AddDir("full") + state.AddDir("empty1") + state.AddDir("empty2") + state.AddDir("empty3") + state.SetFile("full/a.md", "h") + + rng := rand.New(rand.NewSource(42)) + fullCount := 0 + const iters = 1000 + for i := 0; i < iters; i++ { + if pickNonRootDirBiased(rng, pools, state) == "full" { + fullCount++ + } + } + // Target 70% +/- 5% (bias is 70/30 between groups, but within the empty + // group each dir gets 10% so "full" alone vs each "empty_i" is 70 vs 10). + if fullCount < 650 || fullCount > 750 { + t.Errorf("expected ~700/1000 for non-empty dir, got %d", fullCount) + } +} + +func TestCanMoveDir_SingleRootChild(t *testing.T) { + // Regression: a single non-root dir whose parent is root has no valid + // destination -- root is its current parent, and the dir itself is + // excluded. canMoveDir must return false in this case. + pools := NewPools() + pools.AddDir("docs") + state := NewWorkspaceState() + state.AddDir("docs") + if canMoveDir(pools, state) { + t.Error("canMoveDir should return false when only source is a root-child with no sibling") + } +} + +func TestCanMoveDir_TwoRootChildren(t *testing.T) { + pools := NewPools() + pools.AddDir("a") + pools.AddDir("b") + state := NewWorkspaceState() + state.AddDir("a") + state.AddDir("b") + if !canMoveDir(pools, state) { + t.Error("canMoveDir should return true with two sibling root-child dirs") + } +} + +func TestCanMoveDir_NestedHasRootDest(t *testing.T) { + // Nested dir can always move to root. + pools := NewPools() + pools.AddDir("a") + pools.AddDir("a/b") + state := NewWorkspaceState() + state.AddDir("a") + state.AddDir("a/b") + if !canMoveDir(pools, state) { + t.Error("canMoveDir should return true when a nested dir can move to root") + } +} + +func TestValidMoveDirDests_ExcludesSelfParentAndDescendants(t *testing.T) { + pools := NewPools() + pools.AddDir("a") + pools.AddDir("a/b") + pools.AddDir("a/b/c") + pools.AddDir("other") + state := NewWorkspaceState() + state.AddDir("a") + state.AddDir("a/b") + state.AddDir("a/b/c") + state.AddDir("other") + + // Moving "a/b": current parent is "a", self is "a/b", descendants are "a/b/c". + // Valid dests: "" (root), "other". + dests := validMoveDirDests("a/b", pools, state) + got := map[string]bool{} + for _, d := range dests { + got[d] = true + } + if !got[""] || !got["other"] || len(got) != 2 { + t.Errorf("validMoveDirDests(a/b): got %v, want [\"\" other]", dests) + } +} + +func TestCollectDeletionOrder(t *testing.T) { + // Real filesystem layout: + // /A/ + // leaf.md + // sub/ + // deep.md + // /other.md (must NOT appear in order; outside src) + // /A/.hidden (must NOT appear; hidden entries are skipped) + tmp := t.TempDir() + mustMkdir := func(p string) { + if err := os.MkdirAll(filepath.Join(tmp, p), 0o755); err != nil { + t.Fatal(err) + } + } + mustWrite := func(p string) { + if err := os.WriteFile(filepath.Join(tmp, p), []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + } + mustMkdir("A/sub") + mustWrite("A/leaf.md") + mustWrite("A/sub/deep.md") + mustWrite("A/.hidden") + mustWrite("other.md") + + order, err := collectDeletionOrder(tmp, "A") + if err != nil { + t.Fatalf("collectDeletionOrder: %v", err) + } + + // src must always be last. + if order[len(order)-1].path != "A" || !order[len(order)-1].isDir { + t.Fatalf("last item must be src dir 'A', got %+v", order[len(order)-1]) + } + + // Nothing outside A's subtree, and no hidden entries. + for _, it := range order { + if it.path != "A" && !strings.HasPrefix(it.path, "A/") { + t.Errorf("unexpected path outside subtree: %s", it.path) + } + if strings.Contains(it.path, "/.") || strings.HasPrefix(filepath.Base(it.path), ".") { + t.Errorf("hidden entry should be skipped: %s", it.path) + } + } + + // Children-before-parent invariant: every directory entry must appear + // after every entry whose path begins with that dir + "/". + indexOf := map[string]int{} + for i, it := range order { + indexOf[it.path] = i + } + for _, it := range order { + if !it.isDir { + continue + } + dirIdx := indexOf[it.path] + prefix := it.path + "/" + for _, child := range order { + if strings.HasPrefix(child.path, prefix) && indexOf[child.path] > dirIdx { + t.Errorf("child %s (idx %d) appears AFTER its parent %s (idx %d)", + child.path, indexOf[child.path], it.path, dirIdx) + } + } + } + + // Spot-check exact contents. + want := map[string]bool{ + "A/sub/deep.md": false, + "A/sub": true, + "A/leaf.md": false, + "A": true, + } + if len(order) != len(want) { + t.Errorf("len(order) = %d, want %d (got %v)", len(order), len(want), order) + } + for _, it := range order { + expectedDir, ok := want[it.path] + if !ok { + t.Errorf("unexpected entry %s", it.path) + continue + } + if it.isDir != expectedDir { + t.Errorf("entry %s isDir=%v, want %v", it.path, it.isDir, expectedDir) + } + } +} + +func TestCollectDeletionOrder_EmptyDir(t *testing.T) { + tmp := t.TempDir() + if err := os.MkdirAll(filepath.Join(tmp, "empty"), 0o755); err != nil { + t.Fatal(err) + } + order, err := collectDeletionOrder(tmp, "empty") + if err != nil { + t.Fatal(err) + } + if len(order) != 1 || order[0].path != "empty" || !order[0].isDir { + t.Errorf("empty dir should yield only [{empty,true}], got %v", order) + } +} + +// TestCollectDeletionOrder_PhantomDir simulates the case where TigerFS has +// a directory the stress test's WorkspaceState doesn't track (mkdirSynth +// doesn't log, so undo doesn't roll Mkdir-created dirs back). The walker +// must find such phantom dirs via os.ReadDir so they get deleted along +// with the rest, otherwise os.Remove on the parent fails with ENOTEMPTY +// (surfaced as EIO through the NFS adapter). +func TestCollectDeletionOrder_PhantomDir(t *testing.T) { + tmp := t.TempDir() + if err := os.MkdirAll(filepath.Join(tmp, "A/phantom"), 0o755); err != nil { + t.Fatal(err) + } + order, err := collectDeletionOrder(tmp, "A") + if err != nil { + t.Fatal(err) + } + var sawPhantom bool + for _, it := range order { + if it.path == "A/phantom" && it.isDir { + sawPhantom = true + } + } + if !sawPhantom { + t.Errorf("phantom dir A/phantom not in deletion order: %v", order) + } +} + +func TestFormatSize(t *testing.T) { + tests := []struct { + bytes int + want string + }{ + {100, "100B"}, + {1024, "1.0KB"}, + {5120, "5.0KB"}, + {1048576, "1.0MB"}, + } + for _, tt := range tests { + got := formatSize(tt.bytes) + if got != tt.want { + t.Errorf("formatSize(%d) = %s, want %s", tt.bytes, got, tt.want) + } + } +} diff --git a/test/stress/runner.go b/test/stress/runner.go new file mode 100644 index 0000000..c67d059 --- /dev/null +++ b/test/stress/runner.go @@ -0,0 +1,470 @@ +package main + +import ( + "errors" + "fmt" + "math/rand" + "os" + "path/filepath" + "strings" +) + +// Operation types with weights for random selection. +type opType int + +const ( + opCreateFile opType = iota + opEditFile + opRenameFile + opMoveFile + opDeleteFile + opCreateDir + opRenameDir + opMoveDir + opDeleteDir + opCreateSavepoint + opUndoSingle + opUndoToID + opUndoToSavepoint +) + +type weightedOp struct { + op opType + weight int + name string +} + +var operationTable = []weightedOp{ + {opCreateFile, 25, "create_file"}, + {opEditFile, 25, "edit_file"}, + {opRenameFile, 10, "rename_file"}, + {opMoveFile, 10, "move_file"}, + {opDeleteFile, 10, "delete_file"}, + {opCreateDir, 5, "create_dir"}, + {opRenameDir, 5, "rename_dir"}, + {opMoveDir, 5, "move_dir"}, + {opDeleteDir, 3, "delete_dir"}, + {opCreateSavepoint, 5, "create_savepoint"}, + {opUndoSingle, 3, "undo_single"}, + {opUndoToID, 2, "undo_to_id"}, + {opUndoToSavepoint, 2, "undo_to_savepoint"}, +} + +// totalWeight is the sum of all operation weights. +var totalWeight int + +func init() { + for _, wo := range operationTable { + totalWeight += wo.weight + } +} + +// selectOperation picks a random operation based on weights. +func selectOperation(rng *rand.Rand) opType { + r := rng.Intn(totalWeight) + cumulative := 0 + for _, wo := range operationTable { + cumulative += wo.weight + if r < cumulative { + return wo.op + } + } + return opCreateFile // fallback +} + +// opName returns the human-readable name of an operation. +func opName(op opType) string { + for _, wo := range operationTable { + if wo.op == op { + return wo.name + } + } + return "unknown" +} + +// canExecute checks if an operation's preconditions are met. +func canExecute(op opType, pools *Pools, state *WorkspaceState, stack *StateStack) bool { + switch op { + case opCreateFile: + return len(pools.Dirs) > 0 + case opEditFile, opRenameFile, opDeleteFile: + return len(pools.Files) > 0 + case opMoveFile: + return len(pools.Files) > 0 && len(pools.Dirs) >= 2 + case opCreateDir: + return len(pools.Dirs) > 0 + case opRenameDir: + return len(pools.NonRootDirs()) > 0 + case opMoveDir: + // Need at least one feasible (src, dest) pair. Checking pool counts + // alone is insufficient -- e.g., a single root-child dir has no valid + // destination because root is its current parent. + return canMoveDir(pools, state) + case opDeleteDir: + return len(pools.NonRootDirs()) > 0 + case opCreateSavepoint: + return true + case opUndoSingle: + // undo_single only undoes the most recent log entry. If the most + // recent op produced multiple log entries (e.g., a multi-chunk NFS + // write of a large file), undoing just one leaves the file in a + // partial-content state that WorkspaceState (md5-keyed) can't track. + // Require atomic (1-log-entry) ops only. + return stack.MostRecentLogIsAtomic() + case opUndoToID: + return stack.LoggedCount() >= 2 + case opUndoToSavepoint: + return stack.HasSavepoints() + } + return false +} + +// Run-failure kinds. Kept as small string constants so they're stable in +// summary.json (downstream tooling can switch on them) and self-explanatory +// when read back from a dump. +const ( + runFailureValidation = "validation" // ValidateWorkspace returned mismatches + runFailureOperation = "operation" // executeOperation returned an error (EIO, etc.) +) + +// RunFailure is returned from RunIterations when the run cannot continue. +// Carries the dump directory path so RunAndExit can surface it in the +// failure message and main.go can skip teardown to leave infrastructure +// live for inspection. +// +// Kind distinguishes the cause: +// - "validation": the live filesystem diverged from the expected state +// - "operation": an op (create_file, edit_file, etc.) returned an error +// +// Both cases produce the same dump format -- the differences live in the +// summary.txt heading and the dump's `failure_kind` field. +type RunFailure struct { + Kind string + DumpDir string + Iteration int + Seed int64 + Desc string + Err error +} + +func (v *RunFailure) Error() string { + return fmt.Sprintf("[STEP %d] Seed=%d %s failure after %s:\n%v", + v.Iteration, v.Seed, v.Kind, v.Desc, v.Err) +} + +func (v *RunFailure) Unwrap() error { return v.Err } + +// RunIterations executes the main test loop. +func RunIterations(cfg *Config, infra *Infra) error { + rng := rand.New(rand.NewSource(cfg.Seed)) + opCfg := NewOpConfig(cfg.LargeFiles, cfg.ManyFiles) + wsPath := filepath.Join(infra.Mountpoint, cfg.Workspace) + + state := NewWorkspaceState() + stack := NewStateStack() + pools := NewPools() + stats := NewStats() + opLog := make([]OpRecord, 0, cfg.Iterations) + + var lastLogID string + + for i := 1; i <= cfg.Iterations; i++ { + // Select operation (with re-roll for unmet preconditions) + op := selectValidOperation(rng, pools, state, stack) + + // Determine if this is an undo operation + isUndo := op == opUndoSingle || op == opUndoToID || op == opUndoToSavepoint + + // Push state before non-undo operations + preStackLen := stack.Len() + if !isUndo { + stack.Push(state, i) + } + + // Execute operation + desc, restoredState, err := executeOperation(op, wsPath, rng, pools, state, opCfg, stack, i, stats) + if err != nil { + // Operational failure (EIO, precondition mismatch, etc.). + // Append a marker OpRecord so the dump's op trace shows + // what failed and why; otherwise the trace would just end + // abruptly at iter N-1 with no indication of what went + // wrong at iter N. + failingDesc := fmt.Sprintf("%s [FAILED: %v]", opName(op), err) + opLog = append(opLog, OpRecord{ + Iteration: i, + OpName: opName(op), + Desc: failingDesc, + Validated: false, + }) + dumpDir, dumpErr := WriteDump(DumpKindFailure, runFailureOperation, cfg, infra, state, stack, opLog, err, failingDesc, i) + if dumpErr != nil { + fmt.Fprintf(os.Stderr, "[WARN] failed to write diagnostics dump: %v\n", dumpErr) + } + return &RunFailure{ + Kind: runFailureOperation, + DumpDir: dumpDir, + Iteration: i, + Seed: cfg.Seed, + Desc: failingDesc, + Err: fmt.Errorf("%s failed: %w", opName(op), err), + } + } + stats.RecordOp(opName(op)) + + fmt.Printf("[STEP %d/%d] %s\n", i, cfg.Iterations, desc) + + // Record log_ids for non-undo operations. A single op may produce + // multiple log entries (e.g., a multi-chunk NFS write of a large + // file fans out into 1 create + N edits). Capture all of them so + // undo_single can be gated to atomic (1-log-entry) ops. + // + // Skip when the op grew the stack itself (OpDeleteDir pushes one + // entry per deletion via SetLastLogID, each tagged LogCount=1). + // Overwriting the final entry with total-deletions LogCount would + // incorrectly mark it non-atomic. + var newIDs []string + if !isUndo && stack.Len() == preStackLen+1 { + newIDs = readLogIDsSince(wsPath, lastLogID) + if len(newIDs) > 0 { + stack.SetLogIDsForLastEntry(newIDs) + lastLogID = newIDs[len(newIDs)-1] + } + // If newIDs is empty, this op didn't log (e.g., create_savepoint). + // LogID stays empty on the stack entry. + } else if !isUndo { + // Op managed its own stack growth (OpDeleteDir). Refresh + // lastLogID to the latest log entry so subsequent ops see + // all of the op's log entries as "old". + // + // Same staleness window as the post-undo read -- the dump + // at iter 472 (log_count=65) was caused by *this* call + // returning a stale value, cascading into the next op's + // readLogIDsSince. Wrap with the monotonic helper so a + // regressed read keeps the prior known-good lastLogID + // instead of cascading old entries forward. + lastLogID = readLatestLogIDMonotonic(wsPath, lastLogID, i, desc, stats) + } + + // For undo operations, restore state and rebuild pools. + // + // Defensive: readLatestLogID over NFS has been observed to + // return stale results after a heavy undo_to_savepoint -- the + // TigerFS-side `.log/.last/N/.export/json` virtual file + // occasionally yields a snapshot from before the undo's log + // rows were visible, despite noac mounts and unique paths. + // When the returned id is *older* than what we already saw, + // retry briefly. If it never recovers, keep the old lastLogID + // (we know it's at least correct as a lower bound) and warn. + // Without this guard, a regressed lastLogID makes the next + // non-undo op's readLogIDsSince attribute every previously + // observed-but-newer log entry to that op (the iter-107 + // log_count=61 anomaly was a 60-entry regression of this kind). + if isUndo && restoredState != nil { + state = restoredState + pools = RebuildPools(state) + lastLogID = readLatestLogIDMonotonic(wsPath, lastLogID, i, desc, stats) + } + + // Validate + shouldValidate := isUndo || (cfg.ValidateEvery > 0 && i%cfg.ValidateEvery == 0) + opLog = append(opLog, OpRecord{ + Iteration: i, + OpName: opName(op), + Desc: desc, + NewLogIDs: newIDs, + Validated: shouldValidate, + }) + if shouldValidate { + if err := ValidateWorkspace(wsPath, state); err != nil { + dumpDir, dumpErr := WriteDump(DumpKindFailure, runFailureValidation, cfg, infra, state, stack, opLog, err, desc, i) + if dumpErr != nil { + fmt.Fprintf(os.Stderr, "[WARN] failed to write diagnostics dump: %v\n", dumpErr) + } + return &RunFailure{ + Kind: runFailureValidation, + DumpDir: dumpDir, + Iteration: i, + Seed: cfg.Seed, + Desc: desc, + Err: err, + } + } + } + + // --dump-at: capture a manual snapshot at this iteration. + // Fires after validation so the dump reflects post-op state. + // Doesn't stop the run; multiple --dump-at iterations produce + // multiple snapshots in one run. + if cfg.DumpAt[i] { + snapDir, snapErr := WriteDump(DumpKindSnapshot, "", cfg, infra, state, stack, opLog, nil, desc, i) + if snapErr != nil { + fmt.Fprintf(os.Stderr, "[WARN] --dump-at %d: snapshot dump failed: %v\n", i, snapErr) + } else { + fmt.Fprintf(os.Stderr, " [dump-at %d] snapshot written to %s\n", i, snapDir) + } + } + } + + // Final validation + fmt.Println() + fmt.Print("Final validation... ") + if err := ValidateWorkspace(wsPath, state); err != nil { + dumpDir, dumpErr := WriteDump(DumpKindFailure, runFailureValidation, cfg, infra, state, stack, opLog, err, "final-validation", cfg.Iterations) + if dumpErr != nil { + fmt.Fprintf(os.Stderr, "[WARN] failed to write diagnostics dump: %v\n", dumpErr) + } + return &RunFailure{ + Kind: runFailureValidation, + DumpDir: dumpDir, + Iteration: cfg.Iterations, + Seed: cfg.Seed, + Desc: "final-validation", + Err: err, + } + } + fmt.Println("PASSED") + + fmt.Printf("\nCompleted %d iterations with seed %d. All validations passed.\n", cfg.Iterations, cfg.Seed) + + stats.Print() + return nil +} + +// selectValidOperation picks an operation that can be executed. +// Falls back to create_file if nothing else works. +func selectValidOperation(rng *rand.Rand, pools *Pools, state *WorkspaceState, stack *StateStack) opType { + // Try up to 50 times to find a valid operation + for attempt := 0; attempt < 50; attempt++ { + op := selectOperation(rng) + if canExecute(op, pools, state, stack) { + return op + } + } + // Fallback: create_file is almost always valid (dirs always exist) + if canExecute(opCreateFile, pools, state, stack) { + return opCreateFile + } + // Ultimate fallback: create_dir + return opCreateDir +} + +// executeOperation dispatches to the appropriate operation function. +// Returns (description, restoredState, error). restoredState is non-nil only +// for undo operations -- the caller should use it to replace the current state. +// stats is updated with per-op metadata (currently just created-file sizes). +func executeOperation(op opType, wsPath string, rng *rand.Rand, pools *Pools, + state *WorkspaceState, cfg *OpConfig, stack *StateStack, iteration int, stats *Stats) (string, *WorkspaceState, error) { + + switch op { + case opCreateFile: + desc, size, err := OpCreateFile(wsPath, rng, pools, state, cfg) + if err == nil { + stats.RecordCreatedFileSize(size) + } + return desc, nil, err + case opEditFile: + desc, err := OpEditFile(wsPath, rng, pools, state, cfg) + return desc, nil, err + case opRenameFile: + desc, err := OpRenameFile(wsPath, rng, pools, state, cfg) + return desc, nil, err + case opMoveFile: + desc, err := OpMoveFile(wsPath, rng, pools, state, cfg) + return desc, nil, err + case opDeleteFile: + desc, err := OpDeleteFile(wsPath, rng, pools, state, cfg) + return desc, nil, err + case opCreateDir: + desc, err := OpCreateDir(wsPath, rng, pools, state, cfg) + return desc, nil, err + case opRenameDir: + desc, err := OpRenameDir(wsPath, rng, pools, state, cfg) + return desc, nil, err + case opMoveDir: + desc, err := OpMoveDir(wsPath, rng, pools, state, cfg) + return desc, nil, err + case opDeleteDir: + desc, err := OpDeleteDir(wsPath, rng, pools, state, cfg, stack, iteration, stats) + return desc, nil, err + case opCreateSavepoint: + desc, err := OpCreateSavepoint(wsPath, rng, pools, state, cfg, iteration, stack) + return desc, nil, err + case opUndoSingle: + return OpUndoSingle(wsPath, stack) + case opUndoToID: + return OpUndoToID(wsPath, rng, stack) + case opUndoToSavepoint: + return OpUndoToSavepoint(wsPath, stack) + default: + return "", nil, fmt.Errorf("unknown operation: %d", op) + } +} + +// RunResult is the outcome of a stress run. KeepInfra is true when the +// caller should skip teardown so the user can inspect the live state +// (DB rows, mounted workspace, FS-level traces) -- currently set on +// validation failure where the dump directory references the live infra. +type RunResult struct { + ExitCode int + KeepInfra bool +} + +// RunAndExit runs the test iterations and returns the result. The caller +// (main) decides whether to skip teardown based on KeepInfra. +func RunAndExit(cfg *Config, infra *Infra) RunResult { + err := RunIterations(cfg, infra) + if err == nil { + return RunResult{ExitCode: 0} + } + + fmt.Fprintf(os.Stderr, "\n[ERROR] %v\n", err) + + var rf *RunFailure + if errors.As(err, &rf) && rf.DumpDir != "" { + // Highlight the dump dir prominently -- it's the first place the + // user should look. Surrounding boxes draw the eye in dense + // terminal output where the trace can be 1000+ lines. + fmt.Fprintln(os.Stderr) + fmt.Fprintln(os.Stderr, strings.Repeat("=", 70)) + fmt.Fprintf(os.Stderr, " Failure dump directory: %s\n", rf.DumpDir) + fmt.Fprintf(os.Stderr, " Failure kind: %s\n", rf.Kind) + fmt.Fprintf(os.Stderr, " Open %s/summary.txt first.\n", rf.DumpDir) + fmt.Fprintln(os.Stderr, strings.Repeat("=", 70)) + fmt.Fprintln(os.Stderr) + fmt.Fprintf(os.Stderr, "Infrastructure left running for inspection.\n") + fmt.Fprintf(os.Stderr, "Mountpoint: %s\n", infra.Mountpoint) + fmt.Fprintf(os.Stderr, "Postgres: %s\n", infra.ConnStr) + fmt.Fprintf(os.Stderr, "Tear down with: bin/tigerfs-stress stop\n") + fmt.Fprintf(os.Stderr, "\nReplay with: %s\n", replayCommand(cfg)) + return RunResult{ExitCode: 1, KeepInfra: true} + } + + fmt.Fprintf(os.Stderr, "Replay with: %s\n", replayCommand(cfg)) + return RunResult{ExitCode: 1} +} + +// replayCommand reconstructs the full CLI invocation needed to re-run a +// failing seed. Must include every flag that affects the workload (seed, +// iterations, validate-every, large-files, many-files, workspace) so the +// run is bit-for-bit reproducible. +func replayCommand(cfg *Config) string { + parts := []string{ + "bin/tigerfs-stress start", + fmt.Sprintf("--seed %d", cfg.Seed), + fmt.Sprintf("--iterations %d", cfg.Iterations), + fmt.Sprintf("--validate-every %d", cfg.ValidateEvery), + } + if cfg.LargeFiles { + parts = append(parts, "--large-files") + } + if cfg.ManyFiles { + parts = append(parts, "--many-files") + } + if cfg.Workspace != "testws" { + parts = append(parts, fmt.Sprintf("--workspace %s", cfg.Workspace)) + } + if cfg.DumpAtSpec != "" { + parts = append(parts, fmt.Sprintf("--dump-at %s", cfg.DumpAtSpec)) + } + return strings.Join(parts, " ") +} diff --git a/test/stress/runner_test.go b/test/stress/runner_test.go new file mode 100644 index 0000000..f31a7c3 --- /dev/null +++ b/test/stress/runner_test.go @@ -0,0 +1,141 @@ +package main + +import ( + "math/rand" + "testing" +) + +func TestSelectOperation_Deterministic(t *testing.T) { + rng1 := rand.New(rand.NewSource(42)) + rng2 := rand.New(rand.NewSource(42)) + + ops1 := make([]opType, 100) + ops2 := make([]opType, 100) + + for i := range ops1 { + ops1[i] = selectOperation(rng1) + ops2[i] = selectOperation(rng2) + } + + for i := range ops1 { + if ops1[i] != ops2[i] { + t.Errorf("iteration %d: ops differ with same seed (%s vs %s)", i, opName(ops1[i]), opName(ops2[i])) + } + } +} + +func TestSelectOperation_AllTypesAppear(t *testing.T) { + rng := rand.New(rand.NewSource(42)) + seen := make(map[opType]bool) + + // With enough iterations, all types should appear + for i := 0; i < 10000; i++ { + seen[selectOperation(rng)] = true + } + + for _, wo := range operationTable { + if !seen[wo.op] { + t.Errorf("operation %s never selected in 10000 iterations", wo.name) + } + } +} + +func TestCanExecute_EmptyState(t *testing.T) { + pools := NewPools() // only root dir + state := NewWorkspaceState() + stack := NewStateStack() + + // Should be executable with empty state + if !canExecute(opCreateFile, pools, state, stack) { + t.Error("create_file should be valid with root dir") + } + if !canExecute(opCreateDir, pools, state, stack) { + t.Error("create_dir should be valid with root dir") + } + if !canExecute(opCreateSavepoint, pools, state, stack) { + t.Error("create_savepoint should always be valid") + } + + // Should NOT be executable + if canExecute(opEditFile, pools, state, stack) { + t.Error("edit_file should be invalid with no files") + } + if canExecute(opDeleteFile, pools, state, stack) { + t.Error("delete_file should be invalid with no files") + } + if canExecute(opRenameFile, pools, state, stack) { + t.Error("rename_file should be invalid with no files") + } + if canExecute(opMoveFile, pools, state, stack) { + t.Error("move_file should be invalid with no files") + } + if canExecute(opRenameDir, pools, state, stack) { + t.Error("rename_dir should be invalid with no non-root dirs") + } + if canExecute(opUndoSingle, pools, state, stack) { + t.Error("undo_single should be invalid with empty stack") + } + if canExecute(opUndoToID, pools, state, stack) { + t.Error("undo_to_id should be invalid with empty stack") + } + if canExecute(opUndoToSavepoint, pools, state, stack) { + t.Error("undo_to_savepoint should be invalid with no savepoints") + } +} + +func TestCanExecute_WithFiles(t *testing.T) { + pools := NewPools() + pools.AddFile("test.md") + pools.AddDir("docs") + state := NewWorkspaceState() + state.AddDir("docs") + stack := NewStateStack() + stack.Push(NewWorkspaceState(), 0) + stack.SetLastLogID("log-0") + stack.Push(NewWorkspaceState(), 1) + stack.SetLastLogID("log-1") + stack.SaveSavepoint("sp1") + + if !canExecute(opEditFile, pools, state, stack) { + t.Error("edit_file should be valid with files") + } + if !canExecute(opMoveFile, pools, state, stack) { + t.Error("move_file should be valid with files + 2 dirs") + } + if !canExecute(opRenameDir, pools, state, stack) { + t.Error("rename_dir should be valid with non-root dirs") + } + if !canExecute(opUndoSingle, pools, state, stack) { + t.Error("undo_single should be valid with stack entries") + } + if !canExecute(opUndoToID, pools, state, stack) { + t.Error("undo_to_id should be valid with 2+ stack entries") + } + if !canExecute(opUndoToSavepoint, pools, state, stack) { + t.Error("undo_to_savepoint should be valid with savepoints") + } +} + +func TestSelectValidOperation_FallsBack(t *testing.T) { + rng := rand.New(rand.NewSource(42)) + pools := NewPools() // only root dir, no files + state := NewWorkspaceState() + stack := NewStateStack() + + // Should always return something valid (create_file or create_dir) + for i := 0; i < 100; i++ { + op := selectValidOperation(rng, pools, state, stack) + if !canExecute(op, pools, state, stack) { + t.Errorf("iteration %d: selectValidOperation returned %s which can't execute", i, opName(op)) + } + } +} + +func TestOpName(t *testing.T) { + if opName(opCreateFile) != "create_file" { + t.Error("wrong name for opCreateFile") + } + if opName(opUndoToSavepoint) != "undo_to_savepoint" { + t.Error("wrong name for opUndoToSavepoint") + } +} diff --git a/test/stress/state.go b/test/stress/state.go new file mode 100644 index 0000000..35c184a --- /dev/null +++ b/test/stress/state.go @@ -0,0 +1,654 @@ +package main + +import ( + "crypto/md5" + "fmt" + "io/fs" + "math/rand" + "os" + "path/filepath" + "sort" + "strings" +) + +// WorkspaceState tracks the expected state of a workspace as a map of +// relative file paths to their md5 content hashes, plus a set of directories. +type WorkspaceState struct { + Files map[string]string // relative path -> md5 hex hash + Dirs map[string]bool // relative path -> exists +} + +// NewWorkspaceState creates an empty workspace state. +func NewWorkspaceState() *WorkspaceState { + return &WorkspaceState{ + Files: make(map[string]string), + Dirs: make(map[string]bool), + } +} + +// DeepCopy returns an independent copy of the workspace state. +func (ws *WorkspaceState) DeepCopy() *WorkspaceState { + clone := &WorkspaceState{ + Files: make(map[string]string, len(ws.Files)), + Dirs: make(map[string]bool, len(ws.Dirs)), + } + for k, v := range ws.Files { + clone.Files[k] = v + } + for k, v := range ws.Dirs { + clone.Dirs[k] = v + } + return clone +} + +// SetFile records a file with the given content hash. +func (ws *WorkspaceState) SetFile(relPath, hash string) { + ws.Files[relPath] = hash +} + +// RemoveFile removes a file from the expected state. +func (ws *WorkspaceState) RemoveFile(relPath string) { + delete(ws.Files, relPath) +} + +// AddDir records a directory. +func (ws *WorkspaceState) AddDir(relPath string) { + ws.Dirs[relPath] = true +} + +// RemoveDir removes a directory and all files/subdirs under it. +func (ws *WorkspaceState) RemoveDir(relPath string) { + delete(ws.Dirs, relPath) + prefix := relPath + "/" + for k := range ws.Files { + if strings.HasPrefix(k, prefix) { + delete(ws.Files, k) + } + } + for k := range ws.Dirs { + if strings.HasPrefix(k, prefix) { + delete(ws.Dirs, k) + } + } +} + +// RenameFile moves a file from oldPath to newPath in the expected state. +func (ws *WorkspaceState) RenameFile(oldPath, newPath string) { + if hash, ok := ws.Files[oldPath]; ok { + delete(ws.Files, oldPath) + ws.Files[newPath] = hash + } +} + +// RenameDir moves a directory and all its contents from oldPath to newPath. +// +// Mutations are collected before the loops finish: adding keys to a map +// while ranging over it is undefined per the Go spec, and freshly added +// keys may or may not be visited. +func (ws *WorkspaceState) RenameDir(oldPath, newPath string) { + delete(ws.Dirs, oldPath) + ws.Dirs[newPath] = true + + oldPrefix := oldPath + "/" + newPrefix := newPath + "/" + + // Collect file moves first, then apply. + type fileMove struct{ oldKey, newKey, hash string } + var fileMoves []fileMove + for k, v := range ws.Files { + if strings.HasPrefix(k, oldPrefix) { + fileMoves = append(fileMoves, fileMove{ + oldKey: k, + newKey: newPrefix + strings.TrimPrefix(k, oldPrefix), + hash: v, + }) + } + } + for _, m := range fileMoves { + delete(ws.Files, m.oldKey) + ws.Files[m.newKey] = m.hash + } + + // Collect subdirectory moves first, then apply. + type dirMove struct{ oldKey, newKey string } + var dirMoves []dirMove + for k := range ws.Dirs { + if strings.HasPrefix(k, oldPrefix) { + dirMoves = append(dirMoves, dirMove{ + oldKey: k, + newKey: newPrefix + strings.TrimPrefix(k, oldPrefix), + }) + } + } + for _, m := range dirMoves { + delete(ws.Dirs, m.oldKey) + ws.Dirs[m.newKey] = true + } +} + +// FileCount returns the number of files in the given directory (non-recursive). +func (ws *WorkspaceState) FileCount(dirPath string) int { + count := 0 + prefix := dirPath + "/" + if dirPath == "" { + prefix = "" + } + for k := range ws.Files { + rel := k + if prefix != "" { + if !strings.HasPrefix(k, prefix) { + continue + } + rel = strings.TrimPrefix(k, prefix) + } + // Only count direct children (no further slashes) + if !strings.Contains(rel, "/") { + count++ + } + } + return count +} + +// SubdirCount returns the number of direct subdirectories in the given directory. +func (ws *WorkspaceState) SubdirCount(dirPath string) int { + count := 0 + prefix := dirPath + "/" + if dirPath == "" { + prefix = "" + } + for k := range ws.Dirs { + rel := k + if prefix != "" { + if !strings.HasPrefix(k, prefix) { + continue + } + rel = strings.TrimPrefix(k, prefix) + } else { + rel = k + } + if rel == "" { + continue + } + // Only count direct children + if !strings.Contains(rel, "/") { + count++ + } + } + return count +} + +// StackEntry is a snapshot of the workspace state at a point in time. +// +// LogID holds the log_id of the operation that ran AFTER this state was +// captured (empty for non-logged ops like create_savepoint). LogCount holds +// how many log entries that operation produced -- usually 1, but a large +// create_file or edit_file fans out into multiple log entries because go-nfs +// fabricates Open/Write/Close per WRITE RPC, so a multi-chunk write commits +// once per chunk. undo_single only undoes the LAST of those entries; the +// intermediate states (file with N chunks of content) are not represented in +// WorkspaceState (which tracks md5 hashes), so undo_single is unsafe for +// multi-log entries -- use undo_to_id or undo_to_savepoint instead. +type StackEntry struct { + State *WorkspaceState + Iteration int + LogID string // log_id of the most recent log entry produced by this op + LogCount int // number of log entries produced by this op (0 if non-logged) +} + +// StateStack tracks workspace state history for undo operations. +type StateStack struct { + entries []StackEntry + savepoints map[string]int // savepoint name -> stack index +} + +// NewStateStack creates an empty state stack. +func NewStateStack() *StateStack { + return &StateStack{ + entries: nil, + savepoints: make(map[string]int), + } +} + +// Push saves the current state before an operation. +func (s *StateStack) Push(state *WorkspaceState, iteration int) { + s.entries = append(s.entries, StackEntry{ + State: state.DeepCopy(), + Iteration: iteration, + }) +} + +// Pop removes and returns the most recent state (for undo_single). +// Returns nil if the stack is empty. +func (s *StateStack) Pop() *WorkspaceState { + if len(s.entries) == 0 { + return nil + } + entry := s.entries[len(s.entries)-1] + s.entries = s.entries[:len(s.entries)-1] + return entry.State +} + +// Len returns the number of entries on the stack. +func (s *StateStack) Len() int { + return len(s.entries) +} + +// SetLastLogID sets the LogID on the most recent stack entry and marks it +// as a single-log-entry operation. Used by ops that push one stack entry per +// log entry (OpDeleteDir walks the deletion tree and pushes per row). +func (s *StateStack) SetLastLogID(logID string) { + if len(s.entries) > 0 { + s.entries[len(s.entries)-1].LogID = logID + s.entries[len(s.entries)-1].LogCount = 1 + } +} + +// SetLogIDsForLastEntry records the log_ids produced by the most recent op. +// LogID is set to the newest id (the one undo_single would target). LogCount +// captures the total -- if > 1, the most recent op fanned out into multiple +// log entries (typically multi-chunk NFS writes on large files), and +// undo_single cannot reach a workspace-trackable state by undoing just one +// of them. +func (s *StateStack) SetLogIDsForLastEntry(ids []string) { + if len(s.entries) == 0 || len(ids) == 0 { + return + } + s.entries[len(s.entries)-1].LogID = ids[len(ids)-1] + s.entries[len(s.entries)-1].LogCount = len(ids) +} + +// MostRecentLogIsAtomic returns true if the most recent logged stack entry +// produced exactly one log entry. Returns false if no logged entries exist +// or if the most recent logged op fanned out into multiple log entries. +// Gates undo_single, which can only safely undo single-log-entry operations +// (intermediate states from chunked writes aren't representable in +// WorkspaceState). +func (s *StateStack) MostRecentLogIsAtomic() bool { + for i := len(s.entries) - 1; i >= 0; i-- { + if s.entries[i].LogID != "" { + return s.entries[i].LogCount == 1 + } + } + return false +} + +// LoggedCount returns the number of entries with non-empty LogIDs. +func (s *StateStack) LoggedCount() int { + count := 0 + for _, e := range s.entries { + if e.LogID != "" { + count++ + } + } + return count +} + +// PopToLogID finds the stack entry matching the given LogID (searching from top), +// returns its State (the state before the operation was applied), and trims the +// stack to exclude that entry and everything above it. +func (s *StateStack) PopToLogID(logID string) *WorkspaceState { + for i := len(s.entries) - 1; i >= 0; i-- { + if s.entries[i].LogID == logID { + state := s.entries[i].State + s.entries = s.entries[:i] + for name, spIdx := range s.savepoints { + if spIdx > i { + delete(s.savepoints, name) + } + } + return state.DeepCopy() + } + } + return nil +} + +// RestoreAfterLogID finds the entry with the given LogID, then returns the state +// AFTER that operation (entries[idx+1].State) and trims the stack to [0..idx]. +// This is used for undo_to_id where TigerFS keeps the target operation. +func (s *StateStack) RestoreAfterLogID(logID string) *WorkspaceState { + targetIdx := -1 + for i := len(s.entries) - 1; i >= 0; i-- { + if s.entries[i].LogID == logID { + targetIdx = i + break + } + } + if targetIdx < 0 || targetIdx+1 >= len(s.entries) { + return nil + } + + afterState := s.entries[targetIdx+1].State.DeepCopy() + s.entries = s.entries[:targetIdx+1] + for name, spIdx := range s.savepoints { + if spIdx > targetIdx+1 { + delete(s.savepoints, name) + } + } + return afterState +} + +// RandomLoggedTarget picks a random logged stack entry that isn't the most recent +// logged entry (so there's at least one logged operation to undo after it). +// Returns the LogID and true, or empty string and false if not enough logged entries. +func (s *StateStack) RandomLoggedTarget(rng *rand.Rand) (string, bool) { + var logged []int + for i, e := range s.entries { + if e.LogID != "" { + logged = append(logged, i) + } + } + if len(logged) < 2 { + return "", false + } + // Pick any logged entry except the last one + idx := rng.Intn(len(logged) - 1) + return s.entries[logged[idx]].LogID, true +} + +// MostRecentLogID returns the LogID of the most recent logged stack entry, +// or empty string if none have LogIDs. +func (s *StateStack) MostRecentLogID() string { + for i := len(s.entries) - 1; i >= 0; i-- { + if s.entries[i].LogID != "" { + return s.entries[i].LogID + } + } + return "" +} + +// SaveSavepoint records a savepoint at the current stack position. +func (s *StateStack) SaveSavepoint(name string) { + s.savepoints[name] = len(s.entries) +} + +// RestoreToSavepoint returns the state at the given savepoint and trims +// the stack back to that point. Returns nil if savepoint not found. +func (s *StateStack) RestoreToSavepoint(name string) *WorkspaceState { + idx, ok := s.savepoints[name] + if !ok { + return nil + } + if idx == 0 { + // Savepoint was at the very beginning + s.entries = s.entries[:0] + return NewWorkspaceState() + } + state := s.entries[idx-1].State + s.entries = s.entries[:idx] + + // Remove savepoints that are after this point + for spName, spIdx := range s.savepoints { + if spIdx > idx { + delete(s.savepoints, spName) + } + } + + return state.DeepCopy() +} + +// RestoreToIndex returns the state at the given stack index and trims +// the stack back to that point. Returns nil if index is out of bounds. +func (s *StateStack) RestoreToIndex(idx int) *WorkspaceState { + if idx < 0 || idx >= len(s.entries) { + return nil + } + state := s.entries[idx].State + s.entries = s.entries[:idx+1] + + // Remove savepoints that are after this point + for name, spIdx := range s.savepoints { + if spIdx > idx+1 { + delete(s.savepoints, name) + } + } + + return state.DeepCopy() +} + +// MostRecentSavepoint returns the name of the most recently created savepoint, +// or empty string if none exist. +func (s *StateStack) MostRecentSavepoint() string { + best := "" + bestIdx := -1 + for name, idx := range s.savepoints { + if idx > bestIdx { + best = name + bestIdx = idx + } + } + return best +} + +// HasSavepoints returns true if any savepoints exist. +func (s *StateStack) HasSavepoints() bool { + return len(s.savepoints) > 0 +} + +// HashContent returns the md5 hex hash of the given content. +func HashContent(content []byte) string { + h := md5.Sum(content) + return fmt.Sprintf("%x", h) +} + +// snapshotWorkspace walks the live filesystem at wsPath and returns +// (files: relpath -> md5 hash, dirs: relpath -> true). Dotfiles and +// dot-prefixed directories (.log, .savepoint, .undo, .history) are TigerFS +// virtuals, not real children of the workspace, and are skipped. +func snapshotWorkspace(wsPath string) (map[string]string, map[string]bool, error) { + files := make(map[string]string) + dirs := make(map[string]bool) + + err := filepath.WalkDir(wsPath, func(path string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + + relPath, err := filepath.Rel(wsPath, path) + if err != nil { + return err + } + + if relPath == "." { + return nil + } + + name := d.Name() + if strings.HasPrefix(name, ".") { + if d.IsDir() { + return filepath.SkipDir + } + return nil + } + + if d.IsDir() { + dirs[relPath] = true + return nil + } + + content, err := os.ReadFile(path) + if err != nil { + return fmt.Errorf("read %s: %w", relPath, err) + } + files[relPath] = HashContent(content) + return nil + }) + if err != nil { + return nil, nil, fmt.Errorf("walk workspace: %w", err) + } + return files, dirs, nil +} + +// ValidationIssueKind tags one of four divergence types between the +// stress-test expectation and the live workspace. Used both for the +// human-readable summary and for the failure-dump diff. +type ValidationIssueKind string + +const ( + IssueMissingFile ValidationIssueKind = "missing_file" + IssueUnexpectedFile ValidationIssueKind = "unexpected_file" + IssueHashMismatch ValidationIssueKind = "hash_mismatch" + IssueMissingDir ValidationIssueKind = "missing_dir" + IssueUnexpectedDir ValidationIssueKind = "unexpected_dir" +) + +// ValidationIssue describes a single state divergence. ExpectedHash and +// ActualHash are populated only for file issues (empty for dir issues). +type ValidationIssue struct { + Kind ValidationIssueKind `json:"kind"` + Path string `json:"path"` + ExpectedHash string `json:"expected_hash,omitempty"` + ActualHash string `json:"actual_hash,omitempty"` +} + +// diffWorkspace compares expected vs snapshot maps and returns a sorted +// list of issues. Sort order: kind first (so all "missing" group together), +// then path within each kind. Stable across runs for diffability. +func diffWorkspace(expected *WorkspaceState, actualFiles map[string]string, actualDirs map[string]bool) []ValidationIssue { + var issues []ValidationIssue + + for relPath, expectedHash := range expected.Files { + actualHash, ok := actualFiles[relPath] + if !ok { + issues = append(issues, ValidationIssue{ + Kind: IssueMissingFile, + Path: relPath, + ExpectedHash: expectedHash, + }) + continue + } + if actualHash != expectedHash { + issues = append(issues, ValidationIssue{ + Kind: IssueHashMismatch, + Path: relPath, + ExpectedHash: expectedHash, + ActualHash: actualHash, + }) + } + } + for relPath, actualHash := range actualFiles { + if _, ok := expected.Files[relPath]; !ok { + issues = append(issues, ValidationIssue{ + Kind: IssueUnexpectedFile, + Path: relPath, + ActualHash: actualHash, + }) + } + } + for relPath := range expected.Dirs { + if !actualDirs[relPath] { + issues = append(issues, ValidationIssue{Kind: IssueMissingDir, Path: relPath}) + } + } + for relPath := range actualDirs { + if _, ok := expected.Dirs[relPath]; !ok { + issues = append(issues, ValidationIssue{Kind: IssueUnexpectedDir, Path: relPath}) + } + } + + sort.Slice(issues, func(i, j int) bool { + if issues[i].Kind != issues[j].Kind { + return issues[i].Kind < issues[j].Kind + } + return issues[i].Path < issues[j].Path + }) + return issues +} + +// ValidateWorkspace compares the actual filesystem state at wsPath against +// the expected state. Returns nil if they match, or an error whose message +// lists all mismatches. +// +// Validates four invariants: +// - every file in expected.Files exists on disk and hashes correctly, +// - no extra files are on disk, +// - every dir in expected.Dirs exists on disk, +// - no extra dirs are on disk. +func ValidateWorkspace(wsPath string, expected *WorkspaceState) error { + actualFiles, actualDirs, err := snapshotWorkspace(wsPath) + if err != nil { + return err + } + issues := diffWorkspace(expected, actualFiles, actualDirs) + if len(issues) == 0 { + return nil + } + lines := make([]string, len(issues)) + for i, iss := range issues { + lines[i] = formatIssue(iss) + } + return fmt.Errorf("validation failed (%d issues):\n %s", len(issues), strings.Join(lines, "\n ")) +} + +// formatIssue produces the one-line human-readable summary of an issue, +// matching the historical text format so existing log greps still work. +func formatIssue(iss ValidationIssue) string { + switch iss.Kind { + case IssueMissingFile: + return fmt.Sprintf("missing file: %s (expected hash %s)", iss.Path, shortHash(iss.ExpectedHash)) + case IssueUnexpectedFile: + return fmt.Sprintf("unexpected file: %s", iss.Path) + case IssueHashMismatch: + return fmt.Sprintf("hash mismatch: %s (expected %s, got %s)", iss.Path, shortHash(iss.ExpectedHash), shortHash(iss.ActualHash)) + case IssueMissingDir: + return fmt.Sprintf("missing dir: %s", iss.Path) + case IssueUnexpectedDir: + return fmt.Sprintf("unexpected dir: %s", iss.Path) + } + return fmt.Sprintf("unknown issue: %+v", iss) +} + +func shortHash(h string) string { + if len(h) >= 8 { + return h[:8] + } + return h +} + +// SnapshotHash computes a deterministic hash of the entire workspace. +// Files are sorted by path; each contributes "relpath:md5hash\n". +// The concatenation is then md5-hashed. +func SnapshotHash(wsPath string) (string, error) { + var entries []string + + err := filepath.WalkDir(wsPath, func(path string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + + relPath, err := filepath.Rel(wsPath, path) + if err != nil { + return err + } + + if relPath == "." { + return nil + } + + name := d.Name() + if strings.HasPrefix(name, ".") { + if d.IsDir() { + return filepath.SkipDir + } + return nil + } + + if d.IsDir() { + return nil + } + + content, err := os.ReadFile(path) + if err != nil { + return fmt.Errorf("read %s: %w", relPath, err) + } + entries = append(entries, fmt.Sprintf("%s:%s", relPath, HashContent(content))) + return nil + }) + if err != nil { + return "", err + } + + sort.Strings(entries) + combined := strings.Join(entries, "\n") + "\n" + h := md5.Sum([]byte(combined)) + return fmt.Sprintf("%x", h), nil +} diff --git a/test/stress/state_test.go b/test/stress/state_test.go new file mode 100644 index 0000000..e4d8944 --- /dev/null +++ b/test/stress/state_test.go @@ -0,0 +1,451 @@ +package main + +import ( + "testing" +) + +func TestDeepCopy_Independent(t *testing.T) { + orig := NewWorkspaceState() + orig.SetFile("a.md", "hash1") + orig.AddDir("docs") + + clone := orig.DeepCopy() + + // Modify clone + clone.SetFile("b.md", "hash2") + clone.AddDir("notes") + clone.RemoveFile("a.md") + + // Original should be unchanged + if _, ok := orig.Files["a.md"]; !ok { + t.Error("original lost a.md after modifying clone") + } + if _, ok := orig.Files["b.md"]; ok { + t.Error("original gained b.md from clone modification") + } + if _, ok := orig.Dirs["notes"]; ok { + t.Error("original gained notes/ from clone modification") + } +} + +func TestDeepCopy_Empty(t *testing.T) { + orig := NewWorkspaceState() + clone := orig.DeepCopy() + + clone.SetFile("x.md", "hash") + if len(orig.Files) != 0 { + t.Error("empty original affected by clone") + } +} + +func TestSetFile_RemoveFile(t *testing.T) { + ws := NewWorkspaceState() + ws.SetFile("docs/hello.md", "abc123") + + if ws.Files["docs/hello.md"] != "abc123" { + t.Error("SetFile didn't store hash") + } + + ws.RemoveFile("docs/hello.md") + if _, ok := ws.Files["docs/hello.md"]; ok { + t.Error("RemoveFile didn't remove file") + } +} + +func TestRenameFile(t *testing.T) { + ws := NewWorkspaceState() + ws.SetFile("old.md", "hash1") + + ws.RenameFile("old.md", "new.md") + + if _, ok := ws.Files["old.md"]; ok { + t.Error("old path still exists after rename") + } + if ws.Files["new.md"] != "hash1" { + t.Error("new path has wrong hash") + } +} + +func TestRenameDir(t *testing.T) { + ws := NewWorkspaceState() + ws.AddDir("docs") + ws.AddDir("docs/sub") + ws.SetFile("docs/a.md", "h1") + ws.SetFile("docs/sub/b.md", "h2") + ws.SetFile("other.md", "h3") + + ws.RenameDir("docs", "notes") + + if _, ok := ws.Dirs["docs"]; ok { + t.Error("old dir still exists") + } + if !ws.Dirs["notes"] { + t.Error("new dir not created") + } + if !ws.Dirs["notes/sub"] { + t.Error("subdirectory not moved") + } + if ws.Files["notes/a.md"] != "h1" { + t.Error("file not moved to new dir") + } + if ws.Files["notes/sub/b.md"] != "h2" { + t.Error("nested file not moved") + } + if ws.Files["other.md"] != "h3" { + t.Error("unrelated file affected") + } +} + +func TestRemoveDir_CascadesContents(t *testing.T) { + ws := NewWorkspaceState() + ws.AddDir("docs") + ws.AddDir("docs/sub") + ws.SetFile("docs/a.md", "h1") + ws.SetFile("docs/sub/b.md", "h2") + ws.SetFile("root.md", "h3") + + ws.RemoveDir("docs") + + if _, ok := ws.Dirs["docs"]; ok { + t.Error("dir not removed") + } + if _, ok := ws.Dirs["docs/sub"]; ok { + t.Error("subdir not removed") + } + if _, ok := ws.Files["docs/a.md"]; ok { + t.Error("file in dir not removed") + } + if _, ok := ws.Files["docs/sub/b.md"]; ok { + t.Error("file in subdir not removed") + } + if ws.Files["root.md"] != "h3" { + t.Error("unrelated file affected") + } +} + +func TestFileCount(t *testing.T) { + ws := NewWorkspaceState() + ws.SetFile("a.md", "h1") + ws.SetFile("b.md", "h2") + ws.SetFile("docs/c.md", "h3") + ws.AddDir("docs") + + if got := ws.FileCount(""); got != 2 { + t.Errorf("root FileCount = %d, want 2", got) + } + if got := ws.FileCount("docs"); got != 1 { + t.Errorf("docs FileCount = %d, want 1", got) + } +} + +func TestSubdirCount(t *testing.T) { + ws := NewWorkspaceState() + ws.AddDir("docs") + ws.AddDir("notes") + ws.AddDir("docs/sub") + + if got := ws.SubdirCount(""); got != 2 { + t.Errorf("root SubdirCount = %d, want 2", got) + } + if got := ws.SubdirCount("docs"); got != 1 { + t.Errorf("docs SubdirCount = %d, want 1", got) + } +} + +func TestStackPushPop(t *testing.T) { + stack := NewStateStack() + + ws1 := NewWorkspaceState() + ws1.SetFile("a.md", "h1") + stack.Push(ws1, 0) + + ws2 := NewWorkspaceState() + ws2.SetFile("a.md", "h1") + ws2.SetFile("b.md", "h2") + stack.Push(ws2, 1) + + if stack.Len() != 2 { + t.Errorf("stack len = %d, want 2", stack.Len()) + } + + popped := stack.Pop() + if popped == nil { + t.Fatal("Pop returned nil") + } + if len(popped.Files) != 2 { + t.Errorf("popped state has %d files, want 2", len(popped.Files)) + } + + if stack.Len() != 1 { + t.Errorf("stack len after pop = %d, want 1", stack.Len()) + } + + popped2 := stack.Pop() + if len(popped2.Files) != 1 { + t.Errorf("second pop has %d files, want 1", len(popped2.Files)) + } + + if stack.Pop() != nil { + t.Error("Pop on empty stack should return nil") + } +} + +func TestStackSavepoint(t *testing.T) { + stack := NewStateStack() + + // Push 3 states, savepoint after state 1 + ws0 := NewWorkspaceState() + stack.Push(ws0, 0) + + ws1 := NewWorkspaceState() + ws1.SetFile("a.md", "h1") + stack.Push(ws1, 1) + stack.SaveSavepoint("sp1") + + ws2 := NewWorkspaceState() + ws2.SetFile("a.md", "h1") + ws2.SetFile("b.md", "h2") + stack.Push(ws2, 2) + + ws3 := NewWorkspaceState() + ws3.SetFile("a.md", "h1") + ws3.SetFile("b.md", "h2") + ws3.SetFile("c.md", "h3") + stack.Push(ws3, 3) + + if stack.Len() != 4 { + t.Fatalf("stack len = %d, want 4", stack.Len()) + } + + // Restore to savepoint (should go back to state after push index 1) + restored := stack.RestoreToSavepoint("sp1") + if restored == nil { + t.Fatal("RestoreToSavepoint returned nil") + } + if len(restored.Files) != 1 { + t.Errorf("restored state has %d files, want 1", len(restored.Files)) + } + if restored.Files["a.md"] != "h1" { + t.Error("restored state has wrong content") + } + if stack.Len() != 2 { + t.Errorf("stack len after restore = %d, want 2", stack.Len()) + } +} + +func TestStackSavepoint_NotFound(t *testing.T) { + stack := NewStateStack() + if stack.RestoreToSavepoint("nonexistent") != nil { + t.Error("RestoreToSavepoint should return nil for unknown savepoint") + } +} + +func TestStackMostRecentSavepoint(t *testing.T) { + stack := NewStateStack() + + if stack.MostRecentSavepoint() != "" { + t.Error("MostRecentSavepoint should be empty for new stack") + } + + ws := NewWorkspaceState() + stack.Push(ws, 0) + stack.SaveSavepoint("early") + + stack.Push(ws, 1) + stack.Push(ws, 2) + stack.SaveSavepoint("late") + + if got := stack.MostRecentSavepoint(); got != "late" { + t.Errorf("MostRecentSavepoint = %q, want %q", got, "late") + } +} + +func TestStackRestoreToIndex(t *testing.T) { + stack := NewStateStack() + + for i := 0; i < 5; i++ { + ws := NewWorkspaceState() + ws.SetFile("file.md", HashContent([]byte(string(rune('a'+i))))) + stack.Push(ws, i) + } + + restored := stack.RestoreToIndex(2) + if restored == nil { + t.Fatal("RestoreToIndex returned nil") + } + if stack.Len() != 3 { + t.Errorf("stack len after RestoreToIndex(2) = %d, want 3", stack.Len()) + } +} + +func TestSetLastLogID(t *testing.T) { + stack := NewStateStack() + ws := NewWorkspaceState() + stack.Push(ws, 1) + stack.SetLastLogID("abc") + + if stack.entries[0].LogID != "abc" { + t.Errorf("LogID = %q, want %q", stack.entries[0].LogID, "abc") + } + + // SetLastLogID on empty stack should not panic + empty := NewStateStack() + empty.SetLastLogID("xyz") +} + +func TestLoggedCount(t *testing.T) { + stack := NewStateStack() + ws := NewWorkspaceState() + + stack.Push(ws, 1) + stack.SetLastLogID("a") + stack.Push(ws, 2) + // entry 2 has no LogID (non-logged operation) + stack.Push(ws, 3) + stack.SetLastLogID("b") + + if got := stack.LoggedCount(); got != 2 { + t.Errorf("LoggedCount = %d, want 2", got) + } +} + +func TestPopToLogID(t *testing.T) { + stack := NewStateStack() + + ws1 := NewWorkspaceState() + ws1.SetFile("a.md", "h1") + stack.Push(ws1, 1) + stack.SetLastLogID("log-1") + + ws2 := NewWorkspaceState() + ws2.SetFile("a.md", "h1") + ws2.SetFile("b.md", "h2") + stack.Push(ws2, 2) + // No LogID (create_dir) + + ws3 := NewWorkspaceState() + ws3.SetFile("a.md", "h1") + ws3.SetFile("b.md", "h2") + ws3.SetFile("c.md", "h3") + stack.Push(ws3, 3) + stack.SetLastLogID("log-3") + + // Pop to log-3: should return state before op 3 (ws3) and trim to 2 entries + restored := stack.PopToLogID("log-3") + if restored == nil { + t.Fatal("PopToLogID returned nil") + } + if len(restored.Files) != 3 { + t.Errorf("restored has %d files, want 3", len(restored.Files)) + } + if stack.Len() != 2 { + t.Errorf("stack len = %d, want 2", stack.Len()) + } +} + +func TestPopToLogID_SkipsNonLogged(t *testing.T) { + stack := NewStateStack() + ws := NewWorkspaceState() + + stack.Push(ws, 1) + stack.SetLastLogID("log-1") + + ws2 := NewWorkspaceState() + ws2.SetFile("a.md", "h1") + stack.Push(ws2, 2) + stack.SetLastLogID("log-2") + + ws3 := NewWorkspaceState() + ws3.SetFile("a.md", "h1") + ws3.AddDir("docs") + stack.Push(ws3, 3) + // No LogID (create_dir at step 3) + + // Pop to log-2: should skip the non-logged entry at top, find log-2 + restored := stack.PopToLogID("log-2") + if restored == nil { + t.Fatal("PopToLogID returned nil") + } + if len(restored.Files) != 1 { + t.Errorf("restored has %d files, want 1", len(restored.Files)) + } + if stack.Len() != 1 { + t.Errorf("stack len = %d, want 1", stack.Len()) + } +} + +func TestRestoreAfterLogID(t *testing.T) { + stack := NewStateStack() + + ws0 := NewWorkspaceState() + stack.Push(ws0, 1) + stack.SetLastLogID("log-1") + + ws1 := NewWorkspaceState() + ws1.SetFile("a.md", "h1") + stack.Push(ws1, 2) + stack.SetLastLogID("log-2") + + ws2 := NewWorkspaceState() + ws2.SetFile("a.md", "h1") + ws2.SetFile("b.md", "h2") + stack.Push(ws2, 3) + stack.SetLastLogID("log-3") + + // Restore after log-1: should return ws1 (state after op 1 = state before op 2) + restored := stack.RestoreAfterLogID("log-1") + if restored == nil { + t.Fatal("RestoreAfterLogID returned nil") + } + if len(restored.Files) != 1 || restored.Files["a.md"] != "h1" { + t.Errorf("wrong restored state: %v", restored.Files) + } + if stack.Len() != 1 { + t.Errorf("stack len = %d, want 1", stack.Len()) + } +} + +func TestRebuildPools(t *testing.T) { + state := NewWorkspaceState() + state.SetFile("a.md", "h1") + state.SetFile("docs/b.md", "h2") + state.AddDir("docs") + + pools := RebuildPools(state) + + if len(pools.Files) != 2 { + t.Errorf("pools has %d files, want 2", len(pools.Files)) + } + if len(pools.Dirs) != 2 { // root + docs + t.Errorf("pools has %d dirs, want 2", len(pools.Dirs)) + } + hasRoot := false + hasDocs := false + for _, d := range pools.Dirs { + if d == "" { + hasRoot = true + } + if d == "docs" { + hasDocs = true + } + } + if !hasRoot || !hasDocs { + t.Error("pools missing root or docs dir") + } +} + +func TestHashContent(t *testing.T) { + h1 := HashContent([]byte("hello")) + h2 := HashContent([]byte("hello")) + h3 := HashContent([]byte("world")) + + if h1 != h2 { + t.Error("same content should produce same hash") + } + if h1 == h3 { + t.Error("different content should produce different hash") + } + if len(h1) != 32 { + t.Errorf("hash length = %d, want 32 (hex md5)", len(h1)) + } +} diff --git a/test/stress/stats.go b/test/stress/stats.go new file mode 100644 index 0000000..21e8217 --- /dev/null +++ b/test/stress/stats.go @@ -0,0 +1,277 @@ +package main + +import ( + "fmt" + "math" + "sort" + "strings" + "time" +) + +// MonotonicWarning records a single readLatestLogIDMonotonic regression +// event so the runner can emit a per-run summary at the end. The +// warnings stream individually to stderr as they happen; the summary is +// the at-a-glance view of "is the NFS-layer staleness behavior stable?" +type MonotonicWarning struct { + Iteration int + OpDesc string // op that just ran when the regression was detected + Retries int // number of retries until recovery (or hit the cap) + Recovered bool // false = retry budget exhausted, prior was kept + Elapsed time.Duration // wall-clock time spent in retries (Retries * monotonicRetryDelay) +} + +// Stats tracks operation counts, created-file sizes, and monotonicity +// warnings over a stress-test run. Printed after final validation to +// give a summary of what the run exercised. +type Stats struct { + ops map[string]int + createSizes []int + warnings []MonotonicWarning +} + +// NewStats returns an empty Stats. +func NewStats() *Stats { + return &Stats{ops: make(map[string]int)} +} + +// RecordOp increments the counter for the given operation name (see opName). +func (s *Stats) RecordOp(name string) { + s.ops[name]++ +} + +// RecordCreatedFileSize records the size in bytes of a newly created file. +// Only called for create_file; edits and other writes are not tracked. +func (s *Stats) RecordCreatedFileSize(bytes int) { + s.createSizes = append(s.createSizes, bytes) +} + +// RecordMonotonicWarning appends a regression event for end-of-run +// summary. Called from readLatestLogIDMonotonic each time a regressed +// read is observed (whether or not it later recovers). +func (s *Stats) RecordMonotonicWarning(w MonotonicWarning) { + s.warnings = append(s.warnings, w) +} + +// Print writes operation counts/percentages and a file-size histogram to stdout. +func (s *Stats) Print() { + total := 0 + for _, n := range s.ops { + total += n + } + + fmt.Println() + fmt.Println("=== Operation Statistics ===") + fmt.Println() + + s.printGroup("File Operations:", []string{ + "create_file", "edit_file", "rename_file", "move_file", "delete_file", + }, nil, total) + fmt.Println() + s.printGroup("Directory Operations:", []string{ + "create_dir", "rename_dir", "move_dir", "delete_dir", + }, nil, total) + fmt.Println() + s.printGroup("Other Operations:", []string{ + "create_savepoint", "undo_single", "undo_to_id", "undo_to_savepoint", + }, nil, total) + fmt.Println() + fmt.Printf(" %-20s %5d %5.1f%%\n", "Total:", total, 100.0) + + s.printHistogram() + s.printMonotonicWarnings(total) +} + +// printMonotonicWarnings renders the per-run monotonicity-warning +// summary -- count, recovery rate, retry distribution, and which op +// categories triggered the regressions. Surfaces whether NFS-layer +// staleness is stable across runs without requiring the reader to +// scroll through the trace looking for `[warn iter ...]` lines. +// +// The "total" param is the total op count from Print(), used to +// compute the regression rate as a percentage of all ops. +func (s *Stats) printMonotonicWarnings(total int) { + fmt.Println() + fmt.Println("=== Monotonicity Warnings ===") + if len(s.warnings) == 0 { + fmt.Println("(none -- no readLatestLogID regressions observed)") + return + } + + recovered := 0 + for _, w := range s.warnings { + if w.Recovered { + recovered++ + } + } + rate := 0.0 + if total > 0 { + rate = 100.0 * float64(len(s.warnings)) / float64(total) + } + fmt.Printf("Total: %d regressions across %d ops (%.2f%%)\n", + len(s.warnings), total, rate) + fmt.Printf("Recovered: %d / %d (%d kept prior lastLogID after retry exhaustion)\n", + recovered, len(s.warnings), len(s.warnings)-recovered) + + // Recovery-time distribution: bucket by retry count so the reader + // can spot whether most recoveries are quick (1-3 retries) or + // approaching the cap (a sign the budget needs to grow). + buckets := map[string]int{} + for _, w := range s.warnings { + switch { + case !w.Recovered: + buckets["stuck (>cap)"]++ + case w.Retries <= 3: + buckets["fast (≤3 retries, ~300ms)"]++ + case w.Retries <= 10: + buckets["medium (4-10 retries, ~1s)"]++ + default: + buckets["slow (>10 retries, >1s)"]++ + } + } + for _, label := range []string{ + "fast (≤3 retries, ~300ms)", + "medium (4-10 retries, ~1s)", + "slow (>10 retries, >1s)", + "stuck (>cap)", + } { + if c := buckets[label]; c > 0 { + fmt.Printf(" %-30s %d\n", label, c) + } + } + + // Which op kind preceded each regression -- helps spot whether the + // staleness only follows particular op types (e.g., undo_to_savepoint + // was the original suspect; deletion-loop reads also contribute). + opKindCounts := map[string]int{} + for _, w := range s.warnings { + // w.OpDesc is the full description like "undo_to_savepoint sp-..." + // or "create_file foo.md (1KB)"; first token is the op kind. + kind := w.OpDesc + if idx := strings.IndexByte(kind, ' '); idx > 0 { + kind = kind[:idx] + } + opKindCounts[kind]++ + } + if len(opKindCounts) > 0 { + fmt.Println("Op kinds preceding regressions:") + var kinds []string + for k := range opKindCounts { + kinds = append(kinds, k) + } + sort.Strings(kinds) + for _, k := range kinds { + fmt.Printf(" %-22s %d\n", k, opKindCounts[k]) + } + } +} + +func (s *Stats) printGroup(title string, names []string, notes map[string]string, total int) { + fmt.Println(title) + for _, name := range names { + count := s.ops[name] + pct := 0.0 + if total > 0 { + pct = 100.0 * float64(count) / float64(total) + } + note := "" + if n, ok := notes[name]; ok { + note = " " + n + } + fmt.Printf(" %-20s %5d %5.1f%%%s\n", name, count, pct, note) + } +} + +func (s *Stats) printHistogram() { + n := len(s.createSizes) + fmt.Println() + fmt.Println("=== File Size Histogram ===") + fmt.Printf("Files created: %d\n\n", n) + if n == 0 { + return + } + + type bucket struct { + lo, hi int64 + label string + count int + } + buckets := []bucket{ + {0, 1024, " 64 B - 1 KB", 0}, + {1024, 4 * 1024, " 1 KB - 4 KB", 0}, + {4 * 1024, 16 * 1024, " 4 KB - 16 KB", 0}, + {16 * 1024, 64 * 1024, " 16 KB - 64 KB", 0}, + {64 * 1024, 256 * 1024, " 64 KB - 256 KB", 0}, + {256 * 1024, 1024 * 1024, " 256 KB - 1 MB", 0}, + {1024 * 1024, 10 * 1024 * 1024, " 1 MB - 10 MB", 0}, + {10 * 1024 * 1024, 100 * 1024 * 1024, " 10 MB - 100 MB", 0}, + } + + for _, sz := range s.createSizes { + v := int64(sz) + for i := range buckets { + if v >= buckets[i].lo && v < buckets[i].hi { + buckets[i].count++ + break + } + } + } + + last := -1 + maxCount := 0 + for i, b := range buckets { + if b.count > 0 { + last = i + if b.count > maxCount { + maxCount = b.count + } + } + } + if last < 0 { + return + } + + const maxBar = 30 + fmt.Println("Range Count Distribution") + for i := 0; i <= last; i++ { + barLen := 0 + if maxCount > 0 { + barLen = (buckets[i].count * maxBar) / maxCount + } + bar := strings.Repeat("█", barLen) + fmt.Printf(" %-17s %5d %s\n", buckets[i].label, buckets[i].count, bar) + } + + sorted := make([]int, n) + copy(sorted, s.createSizes) + sort.Ints(sorted) + var sum int64 + for _, v := range sorted { + sum += int64(v) + } + mean := int(float64(sum) / float64(n)) + + fmt.Println() + fmt.Println("Size Statistics:") + fmt.Printf(" Count: %d\n", n) + fmt.Printf(" Min: %s\n", formatSize(sorted[0])) + fmt.Printf(" Max: %s\n", formatSize(sorted[n-1])) + fmt.Printf(" Mean: %s\n", formatSize(mean)) + fmt.Printf(" P50: %s\n", formatSize(percentile(sorted, 50))) + fmt.Printf(" P90: %s\n", formatSize(percentile(sorted, 90))) + fmt.Printf(" P99: %s\n", formatSize(percentile(sorted, 99))) +} + +// percentile returns the p-th percentile (nearest-rank) of a sorted slice. +func percentile(sorted []int, p int) int { + if len(sorted) == 0 { + return 0 + } + idx := int(math.Ceil(float64(p)/100.0*float64(len(sorted)))) - 1 + if idx < 0 { + idx = 0 + } + if idx >= len(sorted) { + idx = len(sorted) - 1 + } + return sorted[idx] +} diff --git a/test/stress/stats_test.go b/test/stress/stats_test.go new file mode 100644 index 0000000..cae2e07 --- /dev/null +++ b/test/stress/stats_test.go @@ -0,0 +1,65 @@ +package main + +import "testing" + +func TestStats_RecordOp(t *testing.T) { + s := NewStats() + s.RecordOp("create_file") + s.RecordOp("create_file") + s.RecordOp("edit_file") + if s.ops["create_file"] != 2 { + t.Errorf("create_file count: got %d, want 2", s.ops["create_file"]) + } + if s.ops["edit_file"] != 1 { + t.Errorf("edit_file count: got %d, want 1", s.ops["edit_file"]) + } + if s.ops["rename_file"] != 0 { + t.Errorf("rename_file count: got %d, want 0", s.ops["rename_file"]) + } +} + +func TestStats_RecordCreatedFileSize(t *testing.T) { + s := NewStats() + s.RecordCreatedFileSize(100) + s.RecordCreatedFileSize(200) + s.RecordCreatedFileSize(300) + if len(s.createSizes) != 3 { + t.Fatalf("createSizes length: got %d, want 3", len(s.createSizes)) + } + if s.createSizes[0] != 100 || s.createSizes[1] != 200 || s.createSizes[2] != 300 { + t.Errorf("createSizes: got %v, want [100 200 300]", s.createSizes) + } +} + +func TestPercentile(t *testing.T) { + sorted := []int{10, 20, 30, 40, 50, 60, 70, 80, 90, 100} + cases := []struct { + p, want int + }{ + {10, 10}, + {50, 50}, + {90, 90}, + {99, 100}, + {100, 100}, + } + for _, c := range cases { + if got := percentile(sorted, c.p); got != c.want { + t.Errorf("p%d: got %d, want %d", c.p, got, c.want) + } + } +} + +func TestPercentile_Empty(t *testing.T) { + if got := percentile(nil, 50); got != 0 { + t.Errorf("empty p50: got %d, want 0", got) + } +} + +func TestPercentile_Single(t *testing.T) { + sorted := []int{42} + for _, p := range []int{1, 50, 99, 100} { + if got := percentile(sorted, p); got != 42 { + t.Errorf("single p%d: got %d, want 42", p, got) + } + } +} diff --git a/test/stress/undo.go b/test/stress/undo.go new file mode 100644 index 0000000..a46944c --- /dev/null +++ b/test/stress/undo.go @@ -0,0 +1,241 @@ +package main + +import ( + "encoding/json" + "fmt" + "math/rand" + "os" + "path/filepath" + "time" +) + +// LogEntry represents a single entry from .log/.last/N/.export/json. +type LogEntry struct { + LogID string `json:"log_id"` + FileID string `json:"file_id"` + Type string `json:"type"` + Filename string `json:"filename"` + UserID string `json:"user_id"` + VersionID string `json:"version_id"` +} + +// logReadSeq is an incrementing counter used to bust NFS attribute caching. +// Each call to readLogEntries uses a unique .last/N path, preventing the +// macOS NFS client from serving stale cached data for virtual log files. +var logReadSeq int + +// readLogEntries reads the N most recent log entries from the workspace. +func readLogEntries(wsPath string, n int) ([]LogEntry, error) { + logPath := filepath.Join(wsPath, ".log", fmt.Sprintf(".last/%d/.export/json", n)) + data, err := os.ReadFile(logPath) + if err != nil { + return nil, fmt.Errorf("read log: %w", err) + } + + var entries []LogEntry + if err := json.Unmarshal(data, &entries); err != nil { + return nil, fmt.Errorf("parse log JSON: %w", err) + } + return entries, nil +} + +// readLatestLogEntry reads the most recent log entry, using an incrementing +// path counter to bypass NFS attribute caching. +func readLatestLogEntry(wsPath string) (*LogEntry, error) { + logReadSeq++ + entries, err := readLogEntries(wsPath, logReadSeq) + if err != nil { + return nil, err + } + if len(entries) == 0 { + return nil, fmt.Errorf("log is empty") + } + return &entries[0], nil +} + +// readLatestLogID reads the most recent log entry's ID (cache-busting). +func readLatestLogID(wsPath string) string { + entry, err := readLatestLogEntry(wsPath) + if err != nil { + return "" + } + return entry.LogID +} + +// monotonicRetryCount caps how many times readLatestLogIDMonotonic will +// re-poll before giving up. Each retry waits monotonicRetryDelay; the +// total wait budget is roughly count*delay. Sized for the empirically +// observed lag distribution: regressions recover anywhere from 150ms +// to ~1.55s, with the runtime distribution biased toward the short end. +// 2.5s leaves ~60% headroom over the observed max while staying under +// any reasonable per-iteration budget. +const ( + monotonicRetryCount = 25 + monotonicRetryDelay = 100 * time.Millisecond +) + +// readLatestLogIDMonotonic wraps readLatestLogID with a safety net for +// the empirically observed staleness path: +// +// After a heavy undo_to_savepoint, an immediate read of +// `.log/.last/N/.export/json` over NFS sometimes returns a snapshot +// from before the undo's log rows were visible -- so the "newest" +// log_id reported is older than ones we've already seen. The post-undo +// log_id can only ever be greater than every prior log_id (UUIDv7 is +// time-ordered), so a smaller value is provably stale. +// +// The fix: re-poll briefly. If recovery happens within the retry +// budget, return the larger value. If the read keeps regressing past +// the budget, log a warning and return the prior known good (we don't +// regress the runner's lastLogID; downstream ops would over-attribute +// previously-seen log entries to themselves). +// +// Stats is optional (may be nil for call sites that don't have one +// available, like test fixtures). When non-nil, every observed +// regression is recorded for the end-of-run summary. +func readLatestLogIDMonotonic(wsPath, priorLastLogID string, iter int, desc string, stats *Stats) string { + got := readLatestLogID(wsPath) + if got == "" { + // Read failed entirely; keep prior. (Also empty/never-set.) + return priorLastLogID + } + if priorLastLogID == "" || got >= priorLastLogID { + return got + } + // Regression. Retry with brief sleeps. + for attempt := 1; attempt <= monotonicRetryCount; attempt++ { + time.Sleep(monotonicRetryDelay) + got = readLatestLogID(wsPath) + if got >= priorLastLogID { + elapsed := time.Duration(attempt) * monotonicRetryDelay + fmt.Fprintf(os.Stderr, + " [warn iter %d] readLatestLogID regressed after %q; recovered after %d retries (~%v)\n", + iter, desc, attempt, elapsed) + if stats != nil { + stats.RecordMonotonicWarning(MonotonicWarning{ + Iteration: iter, OpDesc: desc, Retries: attempt, + Recovered: true, Elapsed: elapsed, + }) + } + return got + } + } + elapsed := time.Duration(monotonicRetryCount) * monotonicRetryDelay + fmt.Fprintf(os.Stderr, + " [warn iter %d] readLatestLogID regressed after %q and did NOT recover within %d retries; keeping prior %s (got stuck at %s)\n", + iter, desc, monotonicRetryCount, priorLastLogID, got) + if stats != nil { + stats.RecordMonotonicWarning(MonotonicWarning{ + Iteration: iter, OpDesc: desc, Retries: monotonicRetryCount, + Recovered: false, Elapsed: elapsed, + }) + } + return priorLastLogID +} + +// logScanDepth is the minimum number of log entries to scan when looking for +// new entries since the last known log_id. Sized to comfortably cover any +// single user-level op -- the largest fan-out we see is ~10 log entries from +// a 1MB create_file (one entry per 128KB NFS chunk). +const logScanDepth = 50 + +// readLogIDsSince returns log_ids of every entry strictly newer than +// sinceLogID, in chronological (oldest-first) order. Used after non-undo ops +// to discover whether a single user-level op produced multiple log entries +// (NFS multi-chunk writes fan out into N entries), so undo_single can be +// gated when it can't reach a workspace-trackable state. +// +// The N passed to readLogEntries doubles as a cache-buster: the path +// `.last/N/.export/json` must be unique on every call or the macOS NFS +// client serves stale attribute/data cache. Use logReadSeq + logScanDepth +// so N is both unique (logReadSeq increments per call) and large enough to +// cover any single op's fan-out. +func readLogIDsSince(wsPath, sinceLogID string) []string { + logReadSeq++ + entries, err := readLogEntries(wsPath, logReadSeq+logScanDepth) + if err != nil { + return nil + } + // readLogEntries returns newest-first; iterate reverse for oldest-first. + var ids []string + for i := len(entries) - 1; i >= 0; i-- { + if entries[i].LogID > sinceLogID { + ids = append(ids, entries[i].LogID) + } + } + return ids +} + +// OpUndoSingle undoes the most recent logged operation. +// Uses the stack (not the TigerFS log) to identify the target, avoiding issues +// with NFS caching and undo log entries that TigerFS adds after undo operations. +// Returns the expected state after undo (state before the undone operation). +func OpUndoSingle(wsPath string, stack *StateStack) (string, *WorkspaceState, error) { + logID := stack.MostRecentLogID() + if logID == "" { + return "", nil, fmt.Errorf("no logged operations to undo") + } + + // Apply undo + applyPath := filepath.Join(wsPath, ".undo", "id", logID, ".apply") + if err := os.WriteFile(applyPath, []byte("apply"), 0644); err != nil { + return "", nil, fmt.Errorf("apply undo id/%s: %w", logID, err) + } + + // Pop the stack entry and any non-logged entries above it. + // The returned state is the state before the undone operation. + restored := stack.PopToLogID(logID) + if restored == nil { + return "", nil, fmt.Errorf("log_id %s not found in stack", logID) + } + + return fmt.Sprintf("undo_single %s", logID), restored, nil +} + +// OpUndoToID undoes all logged operations after a random logged stack entry. +// Returns the expected state after undo (state after the target operation). +func OpUndoToID(wsPath string, rng *rand.Rand, stack *StateStack) (string, *WorkspaceState, error) { + // Pick a random logged entry (not the most recent logged one) + logID, ok := stack.RandomLoggedTarget(rng) + if !ok { + return "", nil, fmt.Errorf("need at least 2 logged operations for undo_to_id") + } + + // Apply undo -- TigerFS undoes everything AFTER this log_id + applyPath := filepath.Join(wsPath, ".undo", "to-id", logID, ".apply") + if err := os.WriteFile(applyPath, []byte("apply"), 0644); err != nil { + return "", nil, fmt.Errorf("apply undo to-id/%s: %w", logID, err) + } + + // Restore state: the entry AFTER the target has the state we want + // (state after the target operation = state before the next operation). + restored := stack.RestoreAfterLogID(logID) + if restored == nil { + return "", nil, fmt.Errorf("could not restore state for undo to-id/%s", logID) + } + + return fmt.Sprintf("undo_to_id %s", logID), restored, nil +} + +// OpUndoToSavepoint undoes all operations after the most recent savepoint. +// Returns the expected state after undo (state at the savepoint). +func OpUndoToSavepoint(wsPath string, stack *StateStack) (string, *WorkspaceState, error) { + name := stack.MostRecentSavepoint() + if name == "" { + return "", nil, fmt.Errorf("no savepoints to undo to") + } + + // Apply undo + applyPath := filepath.Join(wsPath, ".undo", "to-savepoint", name, ".apply") + if err := os.WriteFile(applyPath, []byte("apply"), 0644); err != nil { + return "", nil, fmt.Errorf("apply undo to-savepoint/%s: %w", name, err) + } + + // Restore state from savepoint + restored := stack.RestoreToSavepoint(name) + if restored == nil { + return "", nil, fmt.Errorf("savepoint %q not found in stack", name) + } + + return fmt.Sprintf("undo_to_savepoint %s", name), restored, nil +} diff --git a/test/stress/validate_test.go b/test/stress/validate_test.go new file mode 100644 index 0000000..69c62f4 --- /dev/null +++ b/test/stress/validate_test.go @@ -0,0 +1,667 @@ +package main + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "testing" +) + +func setupTestDir(t *testing.T, files map[string]string) string { + t.Helper() + dir := t.TempDir() + for relPath, content := range files { + fullPath := filepath.Join(dir, relPath) + os.MkdirAll(filepath.Dir(fullPath), 0755) + if err := os.WriteFile(fullPath, []byte(content), 0644); err != nil { + t.Fatalf("setup file %s: %v", relPath, err) + } + } + return dir +} + +func TestValidateWorkspace_Passing(t *testing.T) { + files := map[string]string{ + "hello.md": "# Hello\n", + "docs/intro.md": "# Intro\n", + } + dir := setupTestDir(t, files) + + expected := NewWorkspaceState() + for relPath, content := range files { + expected.SetFile(relPath, HashContent([]byte(content))) + } + // docs/ exists on disk because setupTestDir creates parents implicitly; + // expected state must declare it (validation now checks dirs too). + expected.AddDir("docs") + + err := ValidateWorkspace(dir, expected) + if err != nil { + t.Errorf("ValidateWorkspace should pass: %v", err) + } +} + +func TestValidateWorkspace_MissingFile(t *testing.T) { + files := map[string]string{ + "hello.md": "# Hello\n", + } + dir := setupTestDir(t, files) + + expected := NewWorkspaceState() + expected.SetFile("hello.md", HashContent([]byte("# Hello\n"))) + expected.SetFile("missing.md", HashContent([]byte("gone"))) + + err := ValidateWorkspace(dir, expected) + if err == nil { + t.Error("ValidateWorkspace should fail for missing file") + } +} + +func TestValidateWorkspace_UnexpectedFile(t *testing.T) { + files := map[string]string{ + "hello.md": "# Hello\n", + "extra.md": "# Extra\n", + } + dir := setupTestDir(t, files) + + expected := NewWorkspaceState() + expected.SetFile("hello.md", HashContent([]byte("# Hello\n"))) + // extra.md not in expected + + err := ValidateWorkspace(dir, expected) + if err == nil { + t.Error("ValidateWorkspace should fail for unexpected file") + } +} + +func TestValidateWorkspace_HashMismatch(t *testing.T) { + files := map[string]string{ + "hello.md": "# Hello v2\n", + } + dir := setupTestDir(t, files) + + expected := NewWorkspaceState() + expected.SetFile("hello.md", HashContent([]byte("# Hello v1\n"))) // wrong hash + + err := ValidateWorkspace(dir, expected) + if err == nil { + t.Error("ValidateWorkspace should fail for hash mismatch") + } +} + +func TestValidateWorkspace_SkipsDotfiles(t *testing.T) { + dir := t.TempDir() + + // Create a regular file + os.WriteFile(filepath.Join(dir, "hello.md"), []byte("# Hello\n"), 0644) + + // Create dotfile and dot-directory (should be skipped) + os.WriteFile(filepath.Join(dir, ".hidden"), []byte("secret"), 0644) + os.MkdirAll(filepath.Join(dir, ".git", "objects"), 0755) + os.WriteFile(filepath.Join(dir, ".git", "config"), []byte("[core]"), 0644) + + expected := NewWorkspaceState() + expected.SetFile("hello.md", HashContent([]byte("# Hello\n"))) + + err := ValidateWorkspace(dir, expected) + if err != nil { + t.Errorf("ValidateWorkspace should skip dotfiles: %v", err) + } +} + +func TestValidateWorkspace_EmptyWorkspace(t *testing.T) { + dir := t.TempDir() + expected := NewWorkspaceState() + + err := ValidateWorkspace(dir, expected) + if err != nil { + t.Errorf("empty workspace should validate: %v", err) + } +} + +func TestValidateWorkspace_NestedDirs(t *testing.T) { + files := map[string]string{ + "a/b/c/deep.md": "deep content", + "a/top.md": "top content", + } + dir := setupTestDir(t, files) + + expected := NewWorkspaceState() + for relPath, content := range files { + expected.SetFile(relPath, HashContent([]byte(content))) + } + // Declare every dir setupTestDir's MkdirAll created. + for _, d := range []string{"a", "a/b", "a/b/c"} { + expected.AddDir(d) + } + + err := ValidateWorkspace(dir, expected) + if err != nil { + t.Errorf("nested dirs should validate: %v", err) + } +} + +func TestValidateWorkspace_MissingDir(t *testing.T) { + dir := t.TempDir() // empty workspace + + expected := NewWorkspaceState() + expected.AddDir("expected-dir") + + err := ValidateWorkspace(dir, expected) + if err == nil { + t.Error("ValidateWorkspace should fail when an expected dir is missing") + } +} + +func TestValidateWorkspace_UnexpectedDir(t *testing.T) { + dir := t.TempDir() + if err := os.MkdirAll(filepath.Join(dir, "rogue-dir"), 0o755); err != nil { + t.Fatal(err) + } + + expected := NewWorkspaceState() // no dirs declared + + err := ValidateWorkspace(dir, expected) + if err == nil { + t.Error("ValidateWorkspace should fail when an unexpected dir is on disk") + } +} + +func TestValidateWorkspace_EmptyExpectedDir(t *testing.T) { + // An empty dir on disk that's also declared in expected.Dirs is fine. + dir := t.TempDir() + if err := os.MkdirAll(filepath.Join(dir, "empty"), 0o755); err != nil { + t.Fatal(err) + } + + expected := NewWorkspaceState() + expected.AddDir("empty") + + if err := ValidateWorkspace(dir, expected); err != nil { + t.Errorf("declared empty dir should validate: %v", err) + } +} + +func TestSnapshotHash_Deterministic(t *testing.T) { + files := map[string]string{ + "a.md": "content a", + "b.md": "content b", + "dir/c.md": "content c", + } + dir := setupTestDir(t, files) + + h1, err := SnapshotHash(dir) + if err != nil { + t.Fatalf("SnapshotHash: %v", err) + } + + h2, err := SnapshotHash(dir) + if err != nil { + t.Fatalf("SnapshotHash: %v", err) + } + + if h1 != h2 { + t.Error("same workspace should produce same snapshot hash") + } +} + +func TestSnapshotHash_DifferentContent(t *testing.T) { + dir1 := setupTestDir(t, map[string]string{"a.md": "v1"}) + dir2 := setupTestDir(t, map[string]string{"a.md": "v2"}) + + h1, _ := SnapshotHash(dir1) + h2, _ := SnapshotHash(dir2) + + if h1 == h2 { + t.Error("different content should produce different snapshot hash") + } +} + +func TestSnapshotHash_SkipsDotfiles(t *testing.T) { + dir1 := setupTestDir(t, map[string]string{"a.md": "content"}) + + // dir2 has same file plus a dotfile + dir2 := setupTestDir(t, map[string]string{"a.md": "content"}) + os.WriteFile(filepath.Join(dir2, ".hidden"), []byte("secret"), 0644) + + h1, _ := SnapshotHash(dir1) + h2, _ := SnapshotHash(dir2) + + if h1 != h2 { + t.Error("dotfiles should be excluded from snapshot hash") + } +} + +// TestDiffWorkspace_AllIssueKinds verifies that diffWorkspace surfaces +// every kind of divergence the dump consumes, with stable sort order. +func TestDiffWorkspace_AllIssueKinds(t *testing.T) { + expected := NewWorkspaceState() + expected.SetFile("keep.md", "h1") + expected.SetFile("missing.md", "h2") + expected.SetFile("changed.md", "h3-expected") + expected.AddDir("keepdir") + expected.AddDir("missingdir") + + actualFiles := map[string]string{ + "keep.md": "h1", + "changed.md": "h3-actual", + "unexpected.md": "h4", + } + actualDirs := map[string]bool{ + "keepdir": true, + "unexpectedir": true, + } + + issues := diffWorkspace(expected, actualFiles, actualDirs) + if len(issues) != 5 { + t.Fatalf("want 5 issues, got %d: %+v", len(issues), issues) + } + + // Sorted by kind alphabetically. hash_mismatch < missing_dir < + // missing_file < unexpected_dir < unexpected_file. + wantKinds := []ValidationIssueKind{ + IssueHashMismatch, IssueMissingDir, IssueMissingFile, + IssueUnexpectedDir, IssueUnexpectedFile, + } + for i, k := range wantKinds { + if issues[i].Kind != k { + t.Errorf("issue %d: kind=%s, want %s", i, issues[i].Kind, k) + } + } +} + +// TestWriteDump_FailureKind exercises the dump writer end-to-end against +// a real local filesystem (no DB, no infra) for the failure case. DB +// capture failure is expected and surfaces as db_error.txt; everything +// else must materialize. +func TestWriteDump_FailureKind(t *testing.T) { + mountDir, cfg, infra, state, stack, opLog := setupDumpScenario(t) + + dumpDir, err := WriteDump(DumpKindFailure, "validation", cfg, infra, state, stack, opLog, + strErr("validation failed (1 issues): missing file: expected-but-missing.md"), + "create_file foo", 1) + if err != nil { + t.Fatalf("WriteDump: %v", err) + } + if dumpDir == "" { + t.Fatal("dumpDir empty") + } + if !strings.Contains(dumpDir, "tigerfs-stress-failure-") { + t.Errorf("failure dump should use 'failure' prefix in dir name, got %s", dumpDir) + } + + // Every dump file (except the optional db_state.json which is + // expected to fail in this test) must be present and non-empty. + for _, name := range []string{"summary.txt", "summary.json", "expected_state.json", "actual_state.json", "diff.txt", "diff.json", "stack.json", "operations.log", "operations.json", "db_error.txt"} { + info, err := os.Stat(filepath.Join(dumpDir, name)) + if err != nil { + t.Errorf("%s missing: %v", name, err) + continue + } + if info.Size() == 0 { + t.Errorf("%s empty", name) + } + } + + // summary.txt must contain the dump dir, replay command, the issue + // summary, and the kind heading -- those are the user-facing + // signposts. + body, err := os.ReadFile(filepath.Join(dumpDir, "summary.txt")) + if err != nil { + t.Fatal(err) + } + for _, want := range []string{"--seed 999", "--large-files", "--validate-every 1", "missing_file", dumpDir, "failure dump"} { + if !strings.Contains(string(body), want) { + t.Errorf("summary.txt missing %q", want) + } + } + + // summary.json should round-trip cleanly so downstream tools can + // read it without ad-hoc parsing. + jb, _ := os.ReadFile(filepath.Join(dumpDir, "summary.json")) + var s dumpSummary + if err := json.Unmarshal(jb, &s); err != nil { + t.Fatalf("summary.json: %v", err) + } + if s.Kind != DumpKindFailure || s.FailureKind != "validation" || s.Seed != 999 || s.IssueCount == 0 || s.DumpDir != dumpDir { + t.Errorf("summary.json fields: %+v", s) + } + if s.ErrorMessage == "" { + t.Error("failure summary should populate ErrorMessage") + } + + keepOrCleanup(t, dumpDir) + _ = mountDir +} + +// TestWriteDump_SnapshotKind exercises the same machinery via the manual +// --dump-at path: no validation error, "snapshot" prefix in the dir name, +// empty ValidationMessage. +func TestWriteDump_SnapshotKind(t *testing.T) { + mountDir, cfg, infra, state, stack, opLog := setupDumpScenario(t) + + dumpDir, err := WriteDump(DumpKindSnapshot, "", cfg, infra, state, stack, opLog, + nil, "create_file foo", 1) + if err != nil { + t.Fatalf("WriteDump: %v", err) + } + if !strings.Contains(dumpDir, "tigerfs-stress-snapshot-") { + t.Errorf("snapshot dump should use 'snapshot' prefix in dir name, got %s", dumpDir) + } + + body, err := os.ReadFile(filepath.Join(dumpDir, "summary.txt")) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(body), "snapshot dump") { + t.Errorf("snapshot summary.txt should say 'snapshot dump', got:\n%s", body) + } + if strings.Contains(string(body), "Validation error:") { + t.Errorf("snapshot summary.txt should not include 'Validation error:' (no error to report)") + } + + jb, _ := os.ReadFile(filepath.Join(dumpDir, "summary.json")) + var s dumpSummary + if err := json.Unmarshal(jb, &s); err != nil { + t.Fatalf("summary.json: %v", err) + } + if s.Kind != DumpKindSnapshot || s.FailureKind != "" { + t.Errorf("summary.json kind = %q failure_kind = %q, want snapshot/empty", s.Kind, s.FailureKind) + } + if s.ErrorMessage != "" { + t.Errorf("snapshot summary.json should have empty ErrorMessage, got %q", s.ErrorMessage) + } + + keepOrCleanup(t, dumpDir) + _ = mountDir +} + +// TestWriteDump_OperationFailureKind verifies the third trigger path: +// an op-level error (e.g. EIO) should also produce a dump with +// failure_kind="operation". The dump format and machinery are identical +// to validation failures; only the FailureKind tag and the summary.txt +// heading differ. Without this path, op failures tear down infra +// without leaving diagnostic data -- the gap that motivated this test. +func TestWriteDump_OperationFailureKind(t *testing.T) { + mountDir, cfg, infra, state, stack, opLog := setupDumpScenario(t) + + dumpDir, err := WriteDump(DumpKindFailure, "operation", cfg, infra, state, stack, opLog, + strErr("write doc-1681.md: open /tmp/.../doc-1681.md: input/output error"), + "create_file foo [FAILED: input/output error]", 1) + if err != nil { + t.Fatalf("WriteDump: %v", err) + } + if !strings.Contains(dumpDir, "tigerfs-stress-failure-") { + t.Errorf("op-failure dump should still use 'failure' prefix, got %s", dumpDir) + } + + body, err := os.ReadFile(filepath.Join(dumpDir, "summary.txt")) + if err != nil { + t.Fatal(err) + } + // summary.txt should label this as an operation error, not a + // validation error -- they're different categories of failure. + if !strings.Contains(string(body), "Operation error:") { + t.Errorf("op-failure summary.txt should say 'Operation error:', got:\n%s", body) + } + if strings.Contains(string(body), "Validation error:") { + t.Errorf("op-failure summary.txt must not say 'Validation error:'") + } + if !strings.Contains(string(body), "input/output error") { + t.Errorf("op-failure summary.txt should include the underlying error message, got:\n%s", body) + } + + jb, _ := os.ReadFile(filepath.Join(dumpDir, "summary.json")) + var s dumpSummary + if err := json.Unmarshal(jb, &s); err != nil { + t.Fatalf("summary.json: %v", err) + } + if s.Kind != DumpKindFailure || s.FailureKind != "operation" { + t.Errorf("summary.json kind=%q failure_kind=%q, want failure/operation", s.Kind, s.FailureKind) + } + if s.ErrorMessage == "" || !strings.Contains(s.ErrorMessage, "input/output error") { + t.Errorf("summary.json error_message must carry the op error, got %q", s.ErrorMessage) + } + + keepOrCleanup(t, dumpDir) + _ = mountDir +} + +// setupDumpScenario builds the common test fixture for dump tests: a +// staged workspace, a Config, an Infra (with intentionally invalid DB +// conn so the DB capture fails cleanly), an expected state that diverges +// from the workspace, a stack with one entry, and an op log. +func setupDumpScenario(t *testing.T) (string, *Config, *Infra, *WorkspaceState, *StateStack, []OpRecord) { + t.Helper() + mountDir := t.TempDir() + wsDir := filepath.Join(mountDir, "testws") + if err := os.MkdirAll(wsDir, 0755); err != nil { + t.Fatal(err) + } + os.WriteFile(filepath.Join(wsDir, "actual.md"), []byte("on-disk"), 0644) + + cfg := &Config{ + Seed: 999, Iterations: 10, ValidateEvery: 1, + LargeFiles: true, ManyFiles: false, Workspace: "testws", + } + infra := &Infra{ + Mountpoint: mountDir, + ConnStr: "postgres://invalid:invalid@127.0.0.1:1/none", + } + state := NewWorkspaceState() + state.SetFile("expected-but-missing.md", HashContent([]byte("v"))) + stack := NewStateStack() + stack.Push(NewWorkspaceState(), 1) + stack.SetLastLogID("019dcca8-0000-7000-8000-000000000001") + opLog := []OpRecord{ + {Iteration: 1, OpName: "create_file", Desc: "create_file foo (1KB)", NewLogIDs: []string{"019dcca8-0000-7000-8000-000000000001"}, Validated: true}, + } + return mountDir, cfg, infra, state, stack, opLog +} + +func keepOrCleanup(t *testing.T, dumpDir string) { + t.Helper() + if os.Getenv("STRESS_KEEP_DUMP") == "" { + os.RemoveAll(dumpDir) + } else { + t.Logf("dump kept at %s (STRESS_KEEP_DUMP set)", dumpDir) + } +} + +// TestParseDumpAtSpec covers the small parser used by --dump-at: empty +// input, comma+space mix, dedup, and rejection of garbage / OOB values. +func TestParseDumpAtSpec(t *testing.T) { + cases := []struct { + name string + spec string + max int + want map[int]bool + }{ + {"empty", "", 100, nil}, + {"single", "5", 100, map[int]bool{5: true}}, + {"multiple", "10,20,30", 100, map[int]bool{10: true, 20: true, 30: true}}, + {"with spaces", " 10 , 20 ", 100, map[int]bool{10: true, 20: true}}, + {"dedup", "5,5,5", 100, map[int]bool{5: true}}, + {"reject zero/neg", "0,-3,5", 100, map[int]bool{5: true}}, + {"reject non-int", "abc,5", 100, map[int]bool{5: true}}, + {"reject past max", "5,500", 100, map[int]bool{5: true}}, + {"all rejected -> nil", "abc,0,-1", 100, nil}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := parseDumpAtSpec(tc.spec, tc.max) + if len(got) != len(tc.want) { + t.Fatalf("parseDumpAtSpec(%q, %d) = %v, want %v", tc.spec, tc.max, got, tc.want) + } + for k := range tc.want { + if !got[k] { + t.Errorf("missing key %d", k) + } + } + }) + } +} + +// strErr is a tiny helper for tests that need a non-nil error with a +// specific message. +func strErr(msg string) error { return &simpleErr{msg} } + +type simpleErr struct{ msg string } + +func (e *simpleErr) Error() string { return e.msg } + +// TestParseSizeBytes covers the trailing "(N.NUNIT)" parser. +func TestParseSizeBytes(t *testing.T) { + cases := []struct { + desc string + wantBytes int + wantOk bool + }{ + {"create_file foo (1.1KB)", 1126, true}, // 1.1*1024 + {"create_file foo (56.2KB)", 57548, true}, + {"edit_file bar (10.0MB)", 10 * 1024 * 1024, true}, + {"create_file baz (134.5KB)", 137728, true}, + {"create_file q (134B)", 134, true}, + {"delete_dir foo (3 files, 1 subdirs)", 0, false}, + {"create_savepoint sp-1", 0, false}, + {"undo_to_savepoint sp-1", 0, false}, + {"unbalanced parens (oops", 0, false}, + {"empty parens ()", 0, false}, + } + for _, tc := range cases { + t.Run(tc.desc, func(t *testing.T) { + b, ok := parseSizeBytes(tc.desc) + if ok != tc.wantOk { + t.Errorf("ok = %v, want %v (bytes=%d)", ok, tc.wantOk, b) + } + if ok && b != tc.wantBytes { + t.Errorf("bytes = %d, want %d", b, tc.wantBytes) + } + }) + } +} + +// TestStackIslands covers the iteration-clustering helper used to render +// "stack: [1] [107..131] [220..227]" in analysis.txt. +func TestStackIslands(t *testing.T) { + mk := func(iters ...int) *StateStack { + s := NewStateStack() + for _, it := range iters { + s.entries = append(s.entries, StackEntry{Iteration: it, State: NewWorkspaceState()}) + } + return s + } + cases := []struct { + name string + stack *StateStack + wantStr string + }{ + {"empty", mk(), ""}, + {"single", mk(5), "[5,5]"}, + {"contiguous", mk(1, 2, 3, 4), "[1,4]"}, + {"two islands", mk(1, 2, 5, 6, 7), "[1,2] [5,7]"}, + {"three islands", mk(1, 107, 108, 220, 221, 222), "[1,1] [107,108] [220,222]"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := stackIslands(tc.stack) + parts := make([]string, len(got)) + for i, isl := range got { + parts[i] = fmt.Sprintf("[%d,%d]", isl.start, isl.end) + } + if joined := strings.Join(parts, " "); joined != tc.wantStr { + t.Errorf("got %q, want %q", joined, tc.wantStr) + } + }) + } +} + +// TestDetectAnomalies_Iter107Style covers the headline case the analyzer +// is built for: a tiny create with log_count blown up by a prior undo's +// lastLogID regression. +func TestDetectAnomalies_Iter107Style(t *testing.T) { + opLog := []OpRecord{ + {Iteration: 107, OpName: "create_file", Desc: "create_file log-5022.md (1.1KB)", + NewLogIDs: makeFakeLogIDs(61)}, // pathological + } + got := detectAnomalies(opLog, NewStateStack(), nil) + if len(got) != 1 { + t.Fatalf("want 1 anomaly, got %d: %v", len(got), got) + } + if !strings.Contains(got[0], "iter 107") || !strings.Contains(got[0], "log_count=61") { + t.Errorf("anomaly text doesn't mention iter and count: %q", got[0]) + } +} + +// TestDetectAnomalies_LargeWriteIsNotAnomaly verifies a legitimate +// multi-chunk write is NOT flagged. 10MB / 128KB = 80 entries is the +// expected fan-out, not a regression. +func TestDetectAnomalies_LargeWriteIsNotAnomaly(t *testing.T) { + opLog := []OpRecord{ + {Iteration: 471, OpName: "edit_file", Desc: "edit_file note-9832.md (10.0MB)", + NewLogIDs: makeFakeLogIDs(80)}, + {Iteration: 432, OpName: "create_file", Desc: "create_file memo-0047.md (307.8KB)", + NewLogIDs: makeFakeLogIDs(3)}, + } + got := detectAnomalies(opLog, NewStateStack(), nil) + if len(got) != 0 { + t.Errorf("expected no anomalies for legitimate fan-out, got: %v", got) + } +} + +// TestDetectAnomalies_SavepointWithLogEntries verifies that +// create_savepoint logging anything is flagged (savepoints are recorded +// in a separate table and must not appear in testws_log). +func TestDetectAnomalies_SavepointWithLogEntries(t *testing.T) { + opLog := []OpRecord{ + {Iteration: 1, OpName: "create_savepoint", Desc: "create_savepoint sp-1", + NewLogIDs: makeFakeLogIDs(2)}, + } + got := detectAnomalies(opLog, NewStateStack(), nil) + if len(got) != 1 || !strings.Contains(got[0], "create_savepoint") { + t.Errorf("expected savepoint anomaly, got: %v", got) + } +} + +// TestDetectAnomalies_UUIDv7Regression catches a stack entry whose +// log_id is older than its predecessor -- a sign of bookkeeping +// crossing iteration boundaries in the wrong order. +func TestDetectAnomalies_UUIDv7Regression(t *testing.T) { + stack := NewStateStack() + stack.entries = []StackEntry{ + {Iteration: 5, State: NewWorkspaceState(), LogID: "019dccdd-d800-...", LogCount: 1}, + {Iteration: 6, State: NewWorkspaceState(), LogID: "019dccdd-d700-...", LogCount: 1}, // regression + } + got := detectAnomalies(nil, stack, nil) + if len(got) != 1 || !strings.Contains(got[0], "UUIDv7 regression") { + t.Errorf("expected UUIDv7 regression anomaly, got: %v", got) + } +} + +// TestDetectAnomalies_RenameArtifact validates the failure-dump +// heuristic that pairs MissingFile + UnexpectedFile by content hash to +// identify TigerFS/stress-test rename divergences. +func TestDetectAnomalies_RenameArtifact(t *testing.T) { + issues := []ValidationIssue{ + {Kind: IssueMissingFile, Path: "old/file.md", ExpectedHash: "deadbeef00000000"}, + {Kind: IssueUnexpectedFile, Path: "new/file.md", ActualHash: "deadbeef00000000"}, + {Kind: IssueMissingFile, Path: "unrelated.md", ExpectedHash: "abc123"}, + } + got := detectAnomalies(nil, NewStateStack(), issues) + if len(got) != 1 || !strings.Contains(got[0], "rename artifact") { + t.Errorf("expected single rename artifact, got: %v", got) + } + if !strings.Contains(got[0], "old/file.md") || !strings.Contains(got[0], "new/file.md") { + t.Errorf("anomaly text missing path pair: %q", got[0]) + } +} + +func makeFakeLogIDs(n int) []string { + out := make([]string, n) + for i := 0; i < n; i++ { + out[i] = fmt.Sprintf("019dccdd-fake-%04d", i) + } + return out +}