🔄 Replicator Plugin

Real-time change data capture (CDC) to PostgreSQL, BigQuery, SQS, webhooks, and more.

Navigation: Getting Started → | Configuration → | Usage Patterns → | Best Practices →

⚡ TLDR

Real-time CDC (Change Data Capture) replication to multiple destinations - each operation replicated individually in near real-time.

import { Database } from 's3db.js';
import { ReplicatorPlugin } from 's3db.js';

const db = new Database({ connectionString: 's3://...' });
await db.connect();

// One line to replicate everything!
await db.usePlugin(new ReplicatorPlugin({
  replicators: [{
    driver: 'postgresql',           // Or: bigquery, sqs, webhook, etc.
    resources: ['users', 'orders'],
    config: {
      connectionString: process.env.DATABASE_URL,
      schemaSync: { enabled: true }  // Auto-create tables
    }
  }]
}));

// All operations automatically replicated!
const users = await db.resources.users;
await users.insert({ name: 'Alice', email: 'alice@example.com' });
// ✅ Replicated to PostgreSQL in ~2 seconds

Key features:

✅ Real-Time CDC: Each insert/update/delete replicated individually (<10ms latency)
✅ Multi-target: S3DB, BigQuery, PostgreSQL, MySQL, DynamoDB, MongoDB, SQS, Webhooks
✅ Data transformation with custom functions
✅ Automatic retry with exponential backoff
✅ Schema sync - Auto-create and update database tables
✅ Selective replication - Replicate only what's needed
✅ Event monitoring - Track all operations

📦 Dependencies

Required:

pnpm install s3db.js

Optional Drivers (install what you need):

# PostgreSQL
pnpm install pg

# MySQL / MariaDB / PlanetScale
pnpm install mysql2

# Google BigQuery
pnpm install @google-cloud/bigquery

# AWS SQS
pnpm install @aws-sdk/client-sqs

# MongoDB
pnpm install mongodb

# AWS DynamoDB
pnpm install @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb

# Turso (SQLite Edge)
pnpm install @libsql/client

# Webhooks & S3DB: No installation needed!

🚀 Quick Start (3 minutes)

1. Install Dependencies

pnpm install s3db.js pg

2. Setup Replicator

import { Database } from 's3db.js';
import { ReplicatorPlugin } from 's3db.js';

const db = new Database({
  connectionString: 's3://key:secret@my-bucket'
});
await db.connect();

await db.usePlugin(new ReplicatorPlugin({
  replicators: [{
    driver: 'postgresql',
    resources: ['users', 'orders'],
    config: {
      connectionString: 'postgresql://user:pass@localhost/analytics',
      schemaSync: { enabled: true }  // Auto-create tables
    }
  }]
}));

console.log('✅ Replication running');

3. Use Normally (Replication is Automatic!)

const users = await db.resources.users;

// Insert - automatically replicated
await users.insert({ name: 'Bob', email: 'bob@example.com' });
// ✅ Appeared in PostgreSQL in ~2s

// Update - automatically replicated
await users.update('user-id', { name: 'Bob Updated' });
// ✅ Updated in PostgreSQL in ~2s

📚 Documentation Guides

All documentation is organized into focused guides:

🎯 For First-Time Users

Getting Started (10 min) - Installation, quick start, common targets
- What is real-time CDC
- Installation & dependencies
- 5-minute quick start
- Your first replication setup
- Common replication targets
- Error handling basics

⚙️ Configuration & Setup

Configuration Guide (15 min) - All configuration options & drivers
- Default configuration object
- Plugin-level options
- 3 configuration patterns (Development, PostgreSQL, Multi-destination)
- Schema sync setup
- Complete driver reference (S3DB, PostgreSQL, BigQuery, SQS, Webhook, etc.)
- Resource mapping options
- Performance tuning

💡 Real-World Scenarios

Usage Patterns (25 min) - 6 progressive patterns with complete code
- Pattern 1: Simple Backup (S3DB → S3DB)
- Pattern 2: Data Transformation
- Pattern 3: Multi-Destination Replication
- Pattern 4: Error Handling & Monitoring
- Pattern 5: Selective Replication with Filters
- Pattern 6: Production Multi-Region Sync
- Copy-paste recipes

✅ Best Practices & Troubleshooting

Best Practices & FAQ (25 min) - Production deployment
- 6 essential best practices with code examples
- Error handling strategies
- Common issues & solutions
- 20+ FAQ entries across 5 categories
- Production deployment checklist

🎯 Key Features

Real-Time CDC

Every database operation replicated immediately:

// All these are replicated in near real-time
await users.insert(newUser);      // ~2 seconds to PostgreSQL
await users.update(id, changes);  // ~2 seconds to PostgreSQL
await users.delete(id);           // ~2 seconds to PostgreSQL

Multi-Destination

Replicate to 4+ targets with different transformations:

new ReplicatorPlugin({
  replicators: [
    // Backup to S3DB
    { driver: 's3db', resources: ['users'], config: { ... } },
    // Analytics to PostgreSQL
    { driver: 'postgresql', resources: ['users'], config: { ... } },
    // Dashboards to BigQuery
    { driver: 'bigquery', resources: ['users'], config: { ... } },
    // Events to SQS
    { driver: 'sqs', resources: ['users'], config: { ... } }
  ]
})

Schema Sync

Automatically create and update database tables:

{
  driver: 'postgresql',
  config: {
    connectionString: '...',
    schemaSync: {
      enabled: true,           // Auto-create tables
      strategy: 'alter',       // Add missing columns
      onMismatch: 'warn'       // Warn on schema mismatch
    }
  }
}

Data Transformation

Transform data before replication:

{
  resources: {
    users: {
      resource: 'user_profiles',  // Different table name
      transform: (data) => ({
        user_id: data.id,
        email: data.email,
        // Omit: password, apiKey, sensitive fields
        created_date: new Date(data.createdAt).toISOString().split('T')[0]
      })
    }
  }
}

Selective Actions & Filtering

Replicate only what you need:

{
  resources: {
    users: {
      actions: ['inserted'],    // Only new users
      shouldReplicate: (data) => data.active === true  // Only active
    },
    logs: {
      actions: []               // Never replicate logs
    }
  }
}

🔄 Typical Workflows

1. Analytics Pipeline

Replicate operational data to PostgreSQL for analytics:

{
  driver: 'postgresql',
  resources: {
    orders: { resource: 'analytics_orders' },
    users: { resource: 'analytics_users' }
  },
  config: { connectionString: process.env.ANALYTICS_DB }
}

2. Event Streaming

Stream events to SQS for microservices:

{
  driver: 'sqs',
  resources: ['orders', 'payments'],
  config: {
    queueName: process.env.SQS_QUEUE_NAME,
    region: 'us-east-1'
  }
}

When the SQS replicator boots, it resolves the queue and creates it automatically if it does not exist yet.

3. Real-Time Backup

Backup to another S3 bucket:

{
  driver: 's3db',
  resources: ['users', 'orders'],
  config: { connectionString: 's3://backup-bucket' }
}

4. Multi-Region HA

Backup to multiple regions:

[
  { driver: 's3db', resources: [...], config: { connectionString: 's3://us-east-1-backup' } },
  { driver: 's3db', resources: [...], config: { connectionString: 's3://eu-west-1-backup' } }
]

❓ Quick FAQ

Q: What's the difference between Replicator and Backup plugins?

Replicator: Real-time per-operation sync (fast, multiple destinations) Backup: Periodic snapshots (slow, disaster recovery)

See detailed comparison for more.

Q: How fast is replication?

Near real-time: ~2-5 seconds from insert to destination. Minimal latency (<10ms) to start replication process.

For analytics that's real-time enough!

Q: Can I replicate to multiple PostgreSQL databases?

Yes! Just configure multiple replicators:

new ReplicatorPlugin({
  replicators: [
    { driver: 'postgresql', resources: [...], config: { connectionString: 'db1' } },
    { driver: 'postgresql', resources: [...], config: { connectionString: 'db2' } }
  ]
})

Q: What if destination goes down?

Automatic retry with exponential backoff (up to 3 times by default). Failed operations stored in log resource.

→ See recovery strategies

Q: Can I skip certain records?

Yes, use shouldReplicate filter:

{
  resources: {
    orders: {
      shouldReplicate: (data) => data.total > 100  // Only large orders
    }
  }
}

🗂️ Common Use Cases

Use Case	Driver	Guide
Dev/Testing	S3DB	Getting Started
Analytics	PostgreSQL	Usage Patterns
Dashboards	BigQuery	Usage Patterns
Event Stream	SQS	Usage Patterns
Backup	S3DB	Getting Started
Production	Multi-target	Best Practices

🚀 Next Steps

New to replication? → Getting Started
Want to configure? → Configuration Guide
Need code examples? → Usage Patterns
Going to production? → Best Practices
Troubleshooting? → Best Practices FAQ

📖 Full Documentation Index

Topic	Guide	Time
Setup	Getting Started	10 min
Configuration	Configuration Guide	15 min
Examples	Usage Patterns	25 min
Production	Best Practices	25 min

Total Reading Time: ~75 minutes for complete understanding

🔗 Related Plugins

Backup Plugin - Periodic snapshots for disaster recovery
TTL Plugin - Auto-expire old data
Audit Plugin - Track all changes
Cache Plugin - Speed up queries

💬 Need Help?

📖 Check the FAQ - Most questions answered
🔍 Read the guide index - Find what you need
🎯 Try usage patterns - Copy-paste solutions
🐛 Found a bug? Open an issue on GitHub
💡 Have a question? Check detailed guides or ask the community

Ready to replicate? Start with Getting Started →

Navigation Menu

Uh oh!

🔄 Replicator Plugin

⚡ TLDR

📦 Dependencies

🚀 Quick Start (3 minutes)

1. Install Dependencies

2. Setup Replicator

3. Use Normally (Replication is Automatic!)

📚 Documentation Guides

🎯 For First-Time Users

⚙️ Configuration & Setup

💡 Real-World Scenarios

✅ Best Practices & Troubleshooting

🎯 Key Features

Real-Time CDC

Multi-Destination

Schema Sync

Data Transformation

Selective Actions & Filtering

🔄 Typical Workflows

1. Analytics Pipeline

2. Event Streaming

3. Real-Time Backup

4. Multi-Region HA

❓ Quick FAQ

🗂️ Common Use Cases

🚀 Next Steps

📖 Full Documentation Index

🔗 Related Plugins

💬 Need Help?

Uh oh!

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

🔄 Replicator Plugin

⚡ TLDR

📦 Dependencies

🚀 Quick Start (3 minutes)

1. Install Dependencies

2. Setup Replicator

3. Use Normally (Replication is Automatic!)

📚 Documentation Guides

🎯 For First-Time Users

⚙️ Configuration & Setup

💡 Real-World Scenarios

✅ Best Practices & Troubleshooting

🎯 Key Features

Real-Time CDC

Multi-Destination

Schema Sync

Data Transformation

Selective Actions & Filtering

🔄 Typical Workflows

1. Analytics Pipeline

2. Event Streaming

3. Real-Time Backup

4. Multi-Region HA

❓ Quick FAQ

🗂️ Common Use Cases

🚀 Next Steps

📖 Full Documentation Index

🔗 Related Plugins

💬 Need Help?