Skip to content

feat: replay_replication() has no incremental checkpoint or durable outcomes #7

Description

@prrao87

This isn't a bug; it's more of a design limitation, so worth filing as a feature request.

Summary

ladybug.replay_replication() reads the full replication log every time it
runs. It does not remember which entries a previous call applied.

This works when materializing an empty graph from the full log. It does not work
well for keeping an existing graph up to date. A second call tries every old
INSERT again. Since INSERTs use CREATE, they fail on duplicate primary keys.

The log also has no status or error columns. After a replay, callers cannot tell
which entries were applied, which failed, or which still need to run without
reading the session's NOTICE messages.

Could replay_replication() support an incremental mode that starts after the
last processed log entry? If the function is intended only for rebuilding an
empty graph, documenting that restriction would also help.

Environment

  • PostgreSQL 18.4 (Homebrew), macOS arm64
  • pg_ladybug main @ e5182153f1d4b588e7fd758d69c1406a6649a79f
  • liblbug 0.19.1

Reproduction

CREATE DATABASE bugrepro;
\c bugrepro

CREATE EXTENSION pg_ladybug;

CREATE TABLE node_city (
    id   bigint PRIMARY KEY,
    name text NOT NULL
);

SELECT ladybug.register_node('City', 'node_city', 'id', NULL, 'demo');
SELECT ladybug.enable_replication('demo', 'INSERT,UPDATE,DELETE');

INSERT INTO node_city VALUES (1, 'Toronto'), (2, 'Vancouver');

Run both replays in the same session:

SET ladybug.storage_path = '/tmp/bugrepro.lbdb';
SET ladybug.pg_connstr   = 'host=/path/to/socket port=5432 dbname=bugrepro user=me';

SELECT * FROM ladybug.cypher(
  'CREATE NODE TABLE City(id INT64, name STRING, PRIMARY KEY(id))'
) AS t(ok text);

SELECT ladybug.replay_replication('demo');   -- first run
SELECT ladybug.replay_replication('demo');   -- second run

Result

The first replay returns 2. The second replay returns 0 after attempting
both INSERTs again:

NOTICE:  ladybug: replay skipped: CREATE (n:City {id: 1, name: 'Toronto'})
DETAIL:  ladybug: query failed: Runtime exception: Found duplicated primary key value 1, ...
NOTICE:  ladybug: replay skipped: CREATE (n:City {id: 2, name: 'Vancouver'})
DETAIL:  ladybug: query failed: Runtime exception: Found duplicated primary key value 2, ...
NOTICE:  ladybug: replayed 0 change(s), skipped 2 (ensure native ladybug node/rel tables exist)
 replay_replication
--------------------
                  0

The log is unchanged:

SELECT count(*) FROM ladybug.replication_log('demo');   -- still 2

A second replay from a new PostgreSQL connection also triggers the backend
crash described in the separate SIGABRT report. Keeping both calls in one
session isolates the incremental replay behavior without that crash.

Why this matters

There is no reliable way to apply only new changes. Duplicate entries and
genuine failures both appear as skipped statements, and the result exists only
in NOTICE output.

In a larger test, the log contained 996 entries. Replay reported 160 applied
and 836 skipped, but it left no durable record of which entries failed.

Possible approach

  • Store a per-graph checkpoint, or accept a starting log ID and return the last
    processed ID.
  • Record the status and error for each entry.
  • Allow failed entries to be retried without replaying successful entries.
  • Document full-log replay as a rebuild operation if that remains its intended
    purpose.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions