Skip to content

bug(install): PM2 registration is not persisted for resurrect #144

Description

@raphaelmarra

Summary

autopg install can report a successful PM2 installation while leaving
autopg-server absent from ~/.pm2/dump.pm2. After the PM2 systemd service
restarts and runs pm2 resurrect, Autopg is silently not restored, while its
database consumers may still be restored and enter crash loops.

This caused an approximately 30-hour production outage in an Omni/RAVI
WhatsApp deployment. No data was lost, but the channel remained offline until
the existing Autopg data directory was re-registered manually.

Environment

  • Autopg 3.0.7, current stable release
  • PM2 6.0.14
  • Ubuntu 24.04, Linux 6.8 x86_64
  • Autopg supervised by PM2 and PM2 supervised by systemd
  • Persistent data directory and non-default port 8432

What happened

  1. The active PM2 dump was created on 2026-07-28.
  2. autopg install registered autopg-server on 2026-08-14, and
    autopg status reported it online.
  3. The PM2 systemd service restarted on 2026-08-19.
  4. pm2 resurrect restored omni-nats, omni-api, ravi, and
    ravi-channels, but not autopg-server, because the dump had never been
    updated.
  5. Port 8432 remained closed. omni-api failed after 30 database attempts per
    cycle and accumulated more than 2,100 restarts.

Sanitized pre-recovery state:

autopg.installed=true
autopg.status=stopped
autopg.pid=null
listener.8432=absent
listener.8882=absent
dump.pm2=omni-nats,omni-api,ravi,ravi-channels
omni-api.restarts=2115

Re-registering Autopg against the unchanged data directory immediately made
the database ready, restored the existing WhatsApp credentials, and brought
the channel back without a new QR code. Running pm2 save afterward produced
a dump containing all five required processes.

Reproduction

On a host with an existing PM2 dump:

pm2 save
autopg install --port 8432 --data ~/.autopg/data --no-ui
autopg status --json
jq -r '.[].name' ~/.pm2/dump.pm2

The live process is online, but the pre-existing dump does not necessarily
contain autopg-server. A later PM2 daemon or systemd restart can therefore
drop Autopg from the restored process set.

Expected behavior

A successful autopg install should not leave operators with a live-only PM2
registration that will disappear at the next resurrect.

Because pm2 save persists the entire PM2 inventory and may have broader side
effects, any of these designs would close the safety gap:

  • persist the process automatically when safe;
  • provide an explicit, documented persistence flag and fail or warn loudly
    when the live process is missing from the dump;
  • make autopg doctor compare the live PM2 entry with dump.pm2 and return a
    failing, actionable check.

Acceptance criteria

  • A successful install cannot silently leave autopg-server absent from the
    process set used by the next pm2 resurrect.
  • The operator receives a machine-readable failure or warning when live PM2
    state and dump.pm2 differ.
  • The check covers name, executable, port, socket, data directory, and
    supervision limits without touching PostgreSQL data.
  • Re-running install with the same settings remains idempotent.
  • An integration test starts from a pre-existing dump, installs Autopg, and
    verifies the persisted/read-back state.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions