main #1
Open
main #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
robelest
commented
Nov 3, 2025
robelest
commented
- feat: Implement semantic-release with main → prod workflow
- fix: Resolve biome linter issues
- Remove entire onDelete handler (~80 lines of complex logic) - Remove deletedKeys tracking and cleanup code - Change handleDelete to use collection.update() instead of collection.delete() - Treat 'deleted' as a boolean field, just like 'isCompleted' - Simplify subscription handler to simple update/insert (no special delete logic) - Add UI filter to hide soft-deleted items - Sync SSR initialData to Yjs for consistent CRDT state - Sync all subscription updates to Yjs before TanStack DB Result: 89 lines removed, much simpler architecture that works just like checkbox toggle
- Update README.md and packages/replicate/README.md with soft delete pattern - Add comprehensive 'Delete Pattern: Soft Delete' section to both READMEs - Update CLAUDE.md with soft delete best practices and architecture - Update CHANGELOG.md with unreleased changes documenting the refactoring - Update SvelteKit example app to use soft delete (collection.update with deleted: true) - Update SvelteKit convex tasks.ts to remove deleteDocument, add stream endpoint - Update SvelteKit page components to filter deleted items in UI - Update SvelteKit SSR loader to filter deleted items - Remove all references to deleteDocument, deleteDocumentHelper, and onDelete handler - Document that deleted field works exactly like isCompleted (just a boolean) - Update API reference sections to reflect soft delete approach - Add examples showing handleDelete using collection.update instead of collection.delete
- Changed from tracking IDs (Set) to full items (Map) in subscription handler
- TanStack DB write() requires { type: 'delete', value: T } with full item
- Previous code only had IDs, causing write() to fail with sync error
- Added onDelete handler to capture Yjs deletion deltas
- Updated component to append-only event log with operationType field
- Changed server helper to perform hard deletes from main table
- Enhanced error logging with full stack traces and context
- Updated example app to use collection.delete() for hard deletes
- Removed deleted/deletedAt fields from schema (hard delete pattern)
This fixes the issue where deleted items would reappear on Client A
and cause Client B disconnections due to subscription sync failures.
- Removed all 109 emojis from documentation and code comments - Updated CLAUDE.md delete section for hard delete implementation - Updated README.md to reflect hard delete pattern (v0.3.0+) - Removed outdated IMPLEMENTATION-PLAN-v0.3.0.md (849 lines) - Removed outdated MIGRATION-SINGLE-PACKAGE.md (184 lines) - Deleted duplicate packages/replicate/README.md (870 lines) - Created ROADMAP.md for future features: - v0.4.0: Document recovery and audit trail features - v0.5.0: Advanced sync features (partial sync, delta compression) - v1.0.0: Production hardening (encryption, migrations, time-travel) - Updated code comments in collection.ts to remove emojis - Updated examples/sveltekit/IMPLEMENTATION_SUMMARY.md - Updated RELEASING.md to remove emojis Total: Removed 2003 lines of redundant/outdated documentation
BREAKING CHANGE: Monorepo structure flattened - Moved packages/replicate/src → src/ (root level) - Moved packages/replicate/rslib.config.ts → rslib.config.ts (root) - Merged packages/replicate/package.json into root package.json - Deleted packages/ directory entirely - Updated examples to use 'link:../..' protocol instead of 'workspace:*' Benefits: - Simpler structure (no confusing nested package directory) - Root IS the package (matches actual project structure) - Single package.json at root - Cleaner imports and references - No workspace protocol needed (examples use direct links) Structure before: packages/replicate/src/ → The actual package source package.json (root) → Just workspace wrapper Structure after: src/ → The actual package source (at root) package.json (root) → The actual package config examples/ → Examples link directly to root This is the final structure - root is both the package and the project.
robelest
force-pushed
the
main
branch
3 times, most recently
from
a3f0613 to
c9fce53
Compare
…hitecture misrepresentations - Fix package name from non-existent split packages to single @trestleinc/replicate - Correct all import paths: /client, /server, /ssr, /convex.config - Document event-sourced component architecture (append-only event log) - Prominently feature replicatedTable helper that auto-injects version/timestamp - Fix API reference to match actual exports (remove non-existent helpers) - Remove outdated documentation files (CHANGELOG.md, ROADMAP.md, example docs) - Update all code examples with correct import paths and actual working code
Analyzed and documented solutions for Jamie Turner's critical questions: - Long Histories: State vector-based sync prevents data loss - Schema Migrations: Two-phase approach (server + client) - Protocol Evolution: Local storage migration on package updates - Reset Handling: Synthesize CRDT deltas for manual edits - Authorization: OUT OF SCOPE - developer responsibility Status: 2/7 implemented, 4/7 fully planned, 1/7 out of scope 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Implements protocol versioning system to handle breaking changes to ConvexReplicate API: - Automatic lazy initialization when first collection is created - Protocol version checking via FunctionReference pattern (consistent with other APIs) - IndexedDB storage for local version metadata - Migration system for protocol upgrades - Comprehensive test suite (17 tests, all passing) Key features: - Users only need to add `getProtocolVersion` to API object - No manual initialization required (happens automatically) - Follows existing Convex API patterns (convexClient.query + FunctionReference) - Added wrapper queries in both examples (tanstack-start, sveltekit) Also includes: - Global rename: collectionName → collection for consistency - Updated all documentation (CLAUDE.md, Production.md, README.md) - Added vitest config and test infrastructure 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
…age wrapper Fixes critical architecture bug where stream query was querying main table instead of component for CRDT bytes. This broke CRDT conflict resolution, which requires binary CRDT deltas (not materialized documents). **Breaking Change:** - Removed helper functions (insertDocumentHelper, updateDocumentHelper, etc.) - Replaced with ReplicateStorage class using factory methods - Updated all examples to use new pattern **Server-Side Changes:** - Add ReplicateStorage<T> class for type-safe component operations - Implement factory methods: createStreamQuery, createSSRQuery, createInsertMutation, createUpdateMutation, createDeleteMutation - Stream query now calls component.public.stream for CRDT bytes - Removed src/server/replication.ts (old helper pattern) - Removed src/client/storage.ts (confused r2 pattern duplicate) **Client-Side Changes:** - Remove SSR Yjs initialization to prevent "Unexpected content type" error - SSR data now only populates TanStack DB (not Yjs) - Add Yjs observer to sync CRDT deltas to TanStack DB - Apply CRDT updates with 'subscription' origin to prevent double-sync - Add checkpoint persistence in localStorage for bandwidth optimization **Architecture Flow:** 1. SSR data → TanStack DB (instant render) 2. CRDT stream → Yjs (conflict resolution via Y.applyUpdate) 3. Yjs observer → TanStack DB (overwrites SSR with resolved state) **Documentation:** - Update CLAUDE.md with ReplicateStorage pattern and usage examples - Update Production.md to reflect Protocol Evolution as fully implemented - Add comprehensive comments in example app This implementation correctly separates concerns: - SSR provides fast initial render via materialized documents - CRDT stream provides conflict-free sync via Yjs binary deltas - Yjs observer bridges CRDT resolution to TanStack DB 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Add snapshot-based compaction system with gap detection and eliminate metadata table overhead. Key changes: - Add compaction.ts with compact() and cleanupSnapshots() mutations - Implement gap detection in stream query with state vector support - Eliminate metadata table - compute storage size from snapshot + recent deltas - Rename maxCrdtStorageBytes to replicationLimit (default: 10MB) - Add checkCompactionThreshold() for conditional 80% threshold triggers - Update ReplicateStorage to pass replicationLimit through all mutations Performance improvements: - Zero write amplification (removed metadata updates per mutation) - Exact calculations vs approximate counters - Bounded query cost (~100-1000 rows) only when approaching threshold - Self-correcting architecture with no synchronization bugs 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- Remove threshold monitoring (calculateStorageSize, checkCompactionThreshold) - Remove replicationLimit parameter from mutations - Add compactAllCollections mutation for batch processing - Rename ReplicateStorage option: replicationLimit → compactionCutoffDays - Add cron schedule example (daily compaction, weekly snapshot cleanup) - Add internal mutation wrappers for cron jobs (required pattern) - Extract compactCollection helper for code reuse - Update CLAUDE.md with cron setup instructions Benefits: - Zero per-mutation overhead (no size checks) - Predictable timing and costs - Simpler architecture 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- Fix ArgumentValidationError by adding vector param to stream wrapper - Rename stateVector → vector throughout codebase for brevity - Change documentId from v.union(v.string(), v.null()) to v.optional(v.string()) The wrapper in src/server/storage.ts was missing the vector parameter that clients send for gap detection, causing validation errors. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
…tion Replace manual cron files with constructor-based schedule configuration using @convex-dev/crons component. Users now configure compaction intervals and retention policies directly in the Replicate constructor, then call a one-time init function to register schedules dynamically. Key changes: - Add createScheduleInit() factory method for runtime schedule registration - Rename ReplicateStorage → Replicate for consistency - Rename parameters to use consistent minute-based units (compactRetention, pruneRetention) - Remove need for manual convex/crons.ts and convex/replicate.ts wrapper files - Enable per-collection customization of both schedule intervals and retention policies - Install @convex-dev/crons as dependency for dynamic scheduling This eliminates boilerplate and enables flexible per-collection control over compaction behavior. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Remove @convex-dev/crons dependency and dynamic registration complexity. Users now write standard convex/crons.ts files using Convex's built-in cronJobs() system. Breaking changes: - Remove compactInterval, compactRetention, pruneInterval, pruneRetention from Replicate constructor - Remove createScheduleInit() factory method - Remove registerSchedule and unregisterSchedule component mutations - Rename cutoffDays → retentionDays for consistency New pattern: - Export compact and prune functions from collection files - Schedule them in convex/crons.ts using native Convex cron syntax - Pure server-side, no client interaction required Benefits: - Removes external dependency (@convex-dev/crons) - Uses standard Convex pattern everyone already knows - Full control over schedules using Convex's cron syntax - Significantly less code to maintain 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
…ction Consolidate production readiness analysis into main README: - Add comprehensive "Jamie's Seven Questions" deep-dive section - Update Reset Handling status from "FULLY PLANNED" to "FULLY IMPLEMENTED" - Document actual gap detection implementation (lines 123-258 in public.ts) - Update overall status from 5/7 to 6/7 fully implemented (86%) - Remove outdated Production.md file (content now in README) Key insights documented: - Gap detection IS reset handling (no complex client code needed) - State vectors preserve offline changes during reset - Transparent snapshot serving when deltas unavailable - Much simpler than originally planned implementation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Authorization is fully implemented via standard Convex patterns: - Optional hooks (checkRead, checkWrite, checkDelete) for permission checks - Full access to ctx.auth in mutations/queries - Works with any Convex auth provider (Clerk, Auth0, etc.) Jamie's question was "how to handle auth in local-first?" - we answered it: use Convex's built-in ctx.auth system with optional permission hooks. Simplified the documentation to be honest about what we provide: - Not a complex auth framework (don't need one!) - Just optional callbacks + standard Convex patterns - Clean, simple, flexible 7/7 (100%) complete 🎉 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- Add OperationType enum (delta/diff/snapshot) for type-safe operation handling - Add YjsOrigin enum for tracking mutation sources (local vs remote) - Fix observer to always use 'update' operation (prevents insert/update mismatch when initial data only in TanStack DB) - Remove echo detection system to allow proper CRDT conflict resolution - Move shared types to component/shared.ts following r2 pattern - Fix linting warnings (replace non-null assertions with optional chaining) - Add comprehensive debug logging for troubleshooting sync issues This improves client-to-client sync reliability. Client B now properly receives and displays changes from Client A. Echo detection was removed as it prevented clients from receiving authoritative CRDT-merged state. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- Remove observer dependency for Yjs→TanStack DB sync - Implement sequential initialization: IndexedDB → SSR merge → Subscription - Change SSR from replace to CRDT merge semantics - Sync ALL Yjs documents after subscription deltas (handles no-op merges) - Add explicit snapshot restore sync to TanStack DB - Rename createConvexCollection to handleReconnect for clarity This fixes multi-client sync issues where subscription deltas created documents in Yjs but didn't propagate to TanStack DB's reactive layer. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
…oops Two critical bugs were preventing Client A from receiving Client B's updates: 1. **ArrayBuffer Byte Offset Bug** - Using `pendingUpdate.buffer` sent entire underlying ArrayBuffer - Uint8Array might reference bytes 10-150, but .buffer sent all 1024 bytes - Changed to `pendingUpdate.slice().buffer` for clean copy - Fixed in onInsert, onUpdate, and onDelete handlers 2. **Remote Update Echo Loop** - updateV2 listener captured ALL events including remote subscriptions - Remote updates were being re-sent to Convex on next mutation - Added origin filter to only capture local mutations - Ignores YjsOrigin.Subscription, YjsOrigin.Snapshot, YjsOrigin.SSRInit Result: Multi-client sync now works correctly. Client A successfully applies Client B's deltas without no-op CRDT merges. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- Add y-indexeddb dependency for Yjs IndexedDB persistence - Update example app to use handleReconnect (renamed from createConvexCollection) - Improve example app layer documentation (3-layer architecture) - Add custom LogTape formatter for better debug output in example - Update lockfiles 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Replace Effect.ts with plain JavaScript implementations: - Rewrite error classes without Data.TaggedError (maintain _tag discriminator) - Simplify SeqService from 78 to 29 lines using plain async functions - Remove Effect imports from collection.ts - Delete unused effect.ts test helper Remove dependencies: - effect: 3.19.13 - @effect/platform: 0.94.0 - @effect/vitest: latest Update documentation to reflect simplified sync architecture using setTimeout debouncing instead of Effect actor model. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Update Convex dependency from 1.31.2 to 1.31.3 across the monorepo. Also change root package.json resolutions/overrides from exact version 1.31.2 to range ^1.31.2 to allow patch updates. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Replace ESLint with Oxlint for faster, simpler linting. This change removes ESLint configuration and dependencies in favor of Oxlint with Oxfmt for formatting, reducing tooling complexity while maintaining code quality standards. Changes: - Remove eslint.config.js and all ESLint-related dependencies - Add oxlint and oxfmt with configuration files (.oxfmtrc.json, oxlint.json) - Update build script to run oxfmt and oxlint before vite build - Reformat all source files with tab-based indentation - Update Convex generated files to match new formatting - Clean up package.json (reorder dependencies, remove lint scripts) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Restructure the project from a single-package architecture to a monorepo using bun workspaces. This enables better separation of concerns and independent versioning of client, server, and shared packages. Key changes: - Move all source code from src/ to packages/replicate/src/ - Configure bun workspaces in root package.json - Remove old build tooling (tsdown, vitest configs) - Migrate to Oxlint for all code formatting - Update all examples to reference monorepo structure - Reformat all files with new Oxlint configuration - Update documentation to reflect new architecture This change prepares the codebase for v2 release with improved modularity and developer experience. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Co-Authored-By: Claude Opus 4.5 <[email protected]>
Changes: - Add pushLocal parameter to recover() - only push on reconnection, not init - Add retry logic with exponential backoff (3 retries, 1s/2s/3s delays) - Add online/offline event listeners for reconnection detection - Update tests to account for retry delays This fixes the severe performance regression where online sync was taking 10-20 seconds. The recovery function was pushing local state for ALL documents on every page load, flooding the Convex mutation queue and causing all subsequent user mutations to queue behind. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Changes: - Add deltaCounts table to cache delta count per document - Replace full collection scan with cached count lookup - Increment/decrement count during mutations for O(1) threshold checks - Remove O(n) scan from stream() subscription query This removes a significant performance bottleneck where every subscription update triggered a full collection scan to check compaction eligibility. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Changes: - Remove await on collectionRef.update() for fire-and-forget sync - Make awareness throttleMs configurable (default 50ms) - Set IntervalEditor to 300ms debounce/throttle for accurate cursor tracking - Update default debounce from 200ms to 50ms Fire-and-forget removes network round-trip blocking from the sync path. Yjs CRDTs ensure eventual consistency, so we don't need to await. The 300ms setting balances responsiveness with cursor position accuracy. Co-Authored-By: Claude Opus 4.5 <[email protected]>
…e pattern Unifies the API surface by enforcing consistent patterns across all configurations: - Getter functions for all dynamic/reactive values (user identity) - Explicit .create() factories for all persistence methods - Consolidated presence logic into single file Breaking changes: - User identity now requires getter pattern everywhere: user: () => UserIdentity - Persistence factories require explicit .create(): persistence.web.sqlite.create() - Removed legacy 3-argument collection.create() overload - Removed TKey generic (getKey always returns string) - Deleted awareness.ts and document.ts (merged into services/presence.ts) Benefits: - Reduces codebase by ~1200 lines (90% duplicate code eliminated) - Single opinionated pattern for each concern - Improved type inference and IDE autocomplete - Clearer API surface with explicit factory methods All tests pass (53/53), build clean with 0 warnings. Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Merge shared/{types,validators}.ts into shared/index.ts
- Merge server/replicate.ts into server/index.ts
- Fix conflicting SessionInfo export in client/collection.ts
- Fix generic type arguments in ConvexCollectionConfig
- Fix Vite type definitions in test.setup.ts
Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add Project Structure section showing consolidated index.ts layout. Document getter/factory/namespace pattern used across the API. Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Move logger from client/ and component/ to shared/logger.ts - Configure LogTape with ANSI colored console output - Update all imports to use $/shared/logger - Update documentation with new logger location Co-Authored-By: Claude Opus 4.5 <[email protected]>
Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Replace pglite persistence with web.sqlite throughout examples - Rewrite sync protocol section for v2 architecture - Document debounce-based sync (50ms internal, 200ms prose) - Explain state-vector-aware compaction - Add local-first startup flow - Add experimental features section - Encrypted storage with WebAuthn PRF and passphrase support - Schema migrations with versioned schemas - Update API reference - Add identity utilities (color.generate, name.anonymous) - Correct persistence provider exports - Add schema.define() for migrations Co-Authored-By: Claude Opus 4.5 <[email protected]>
Document SSR pagination hydration with PaginatedMaterial/PaginatedPage types, pagination state properties (status, canLoadMore, count), and subscribe pattern. Note that client-side load() is not yet implemented. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Client-side:
- Add throttleMs to prose options (cursor update throttling)
- Add user and anonymousPresence to collection config
- Document AnonymousPresenceConfig interface
Server-side:
- Add view function for auth-based query filtering
- Add compaction.retain for snapshot history
- Add evalSession hook for presence authorization
- Add onDelta hook for delta query logging
- Update compaction options (threshold as number, not Size)
Encryption:
- Add mode option ("local" vs "e2e")
- Add configuration options table
- Clarify lock.idle is in minutes
Co-Authored-By: Claude Opus 4.5 <[email protected]>
Major v2 release includes: - New persistence API (web.sqlite, native.sqlite) - Debounce-based sync architecture (no Effect.ts) - State-vector-aware compaction - Experimental encryption (WebAuthn PRF) - Experimental schema migrations - View function for auth-based filtering - Identity utilities for presence Co-Authored-By: Claude Opus 4.5 <[email protected]>
robelest
force-pushed
the
main
branch
2 times, most recently
from
d7245a5 to
37f3610
Compare
Fixes CVE-2025-59288 by removing the vulnerable dependency entirely. These were added for planned browser tests that were never implemented. Co-Authored-By: Claude Opus 4.5 <[email protected]>
…atibility Using `latest` tag instead of workspace reference allows examples to be deployed independently without requiring the full monorepo structure. Co-Authored-By: Claude Opus 4.5 <[email protected]>
npm is stricter about override conflicts than bun. The overrides were only needed for monorepo deduplication, not standalone deployment. Co-Authored-By: Claude Opus 4.5 <[email protected]>
npm has strict peer dependency resolution that conflicts with the project's dependencies. Using bun provider matches the project's intended package manager. Co-Authored-By: Claude Opus 4.5 <[email protected]>
robelest
force-pushed
the
main
branch
3 times, most recently
from
cb70bee to
af713e8
Compare
Railpack doesn't have a bun provider, but we can use node provider and override the install command to use bun. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.