Skip to content

Delete-echo repair tool can hang holding the database lock during read-back #925

Open

Description

@kriszyp

The offline delete-echo repair tool's read-back step runs a child Node process with spawnSync and no timeout, while the tool holds the database's writable RocksDB instance (and so its LOCK). If the child stalls, spawnSync blocks the tool indefinitely, withDatabaseHeld never reaches its finally, and the lock stays held until an operator kills the process. Harper cannot start on that database in the meantime.

Where — replication/repairDeleteEchoRuns.ts:1004 at 6f3ee94349 (origin/main), called from applyDatabase() inside withDatabaseHeld(databasePath, …). The integration test's own CLI helper (integrationTests/cluster/deleteEchoRun.test.mjs, runRepairTool) already passes timeout: 120000.

Trigger — a stalled read-back child, for example an iterator that stops making progress on unusual data. The tool is typically run during incident recovery, which is when a stuck lock hurts most.

Versions — harper-pro#895 (merged 2026-09-25); v5.3.0-beta.3.

Fix — pass a timeout (and killSignal: 'SIGKILL') to the read-back spawnSync, and report a timed-out read-back as a failed verification rather than a crash.

Found by the post-push review on harper-pro#895 (unaddressed thread) and verified during the 2026-09-26 DLC post-mortem. It is also one of the shapes the new hang-and-leak pre-push self-check (skills-internal#282) flags.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    P2

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions