postgres-mutex

A distributed mutex for Postgres. One table, heartbeat-based liveness, automatic recovery from crashed lock holders. No ZooKeeper, no Redis, no Consul, no etcd.

If you already have Postgres, you shouldn't need any of those just to run a job on exactly one instance at a time.

Quickstart

pip install postgres-mutex

from postgres_mutex import Mutex

mutex = Mutex(dsn="postgres://user:pass@host/db", lock_name="nightly-report-job")
mutex.create_schema()  # creates the mutex_lock table if it doesn't exist; idempotent

with mutex.acquire(blocking=False) as acquired:
    if acquired:
        run_the_job()
    else:
        print("another instance is running the job")

Async works the same way:

async with mutex.acquire_async(blocking=True, timeout=30):
    await run_the_job_async()

Or skip the if acquired branch entirely with the decorator:

@mutex.singleton("nightly-report-job")
def run_the_job(): ...

Supplying connection info

Two ways to give Mutex a way to reach Postgres — pick one:

dsn (the common case) — any libpq connection string, passed straight to psycopg untouched. Mutex opens and owns one dedicated connection for it.

Mutex(dsn="postgres://user:pass@host:5432/dbname?sslmode=require", lock_name="job")

Because it's handed to psycopg unmodified, all the usual libpq conventions work: keyword form ("host=localhost dbname=mydb user=me password=secret"), partial DSNs filled in from PGHOST / PGPORT / PGUSER / PGPASSWORD / PGDATABASE / PGSSLMODE, and .pgpass for passwords you don't want in the DSN at all. Where that string comes from — an env var, a secrets manager, a config file — is up to you; postgres-mutex just needs a valid one.

pool / async_pool — if your app already manages a psycopg_pool ConnectionPool / AsyncConnectionPool (e.g. sitting in front of pgbouncer), hand it to Mutex instead of a dsn so it doesn't open an extra always-on connection of its own. Mutex borrows a connection for the duration of each operation (acquire, heartbeat, release) and gives it back — it never owns, opens, or closes the pool.

from psycopg_pool import ConnectionPool

pool = ConnectionPool("postgres://...")  # owned and closed by your app
mutex = Mutex(lock_name="nightly-report-job", pool=pool)

Pass async_pool instead (or as well, if the same lock is driven from both sync and async code) for acquire_async / heartbeat_async / release_async. Requires the psycopg-pool package (pip install postgres-mutex[pool]).

How it works

One table:

CREATE TABLE mutex_lock (
  lock_name      VARCHAR(64) NOT NULL,
  locked         INT NOT NULL DEFAULT 1,
  instance_id    VARCHAR(64) NOT NULL,
  last_heartbeat TIMESTAMPTZ NOT NULL,
  acquired_at    TIMESTAMPTZ NOT NULL,
  CONSTRAINT uq_mutex_lock UNIQUE (locked, lock_name),
  CONSTRAINT chk_locked CHECK (locked = 1)
);

UNIQUE(locked, lock_name) combined with CHECK (locked = 1) means only one row can ever exist per lock_name. Acquiring is just "try to insert a row" — whoever's INSERT lands first wins, enforced by the database itself. No CAS loop, no advisory locks, no race conditions to reason about.

The lock holder heartbeats every 10 seconds (UPDATE ... SET last_heartbeat = now()). Every field used for staleness detection — now(), the interval comparisons — is computed inside a single Postgres statement, so results depend only on the Postgres server's clock, never on any client's wall clock. Skewed or jumping client clocks can't corrupt lock state.

Dual-threshold stale handling

Most distributed-lock tutorials use one timeout. That's the wrong call:

• 30 seconds without a heartbeat → an alert fires (via your metrics hook / on_alert callback) so on-call can look — is the holder just slow, or actually dead?
• 10 minutes without a heartbeat → the lock auto-releases; safe to assume the holder crashed.

If you release at 30 seconds, a holder that's merely slow (GC pause, a long query, a noisy neighbor) loses the lock to another instance that's likely to hit the exact same slowness — you can end up in a churn loop where nobody makes progress. The alert window buys a human time to intervene before the system self-heals on its own.

Clean shutdown

On graceful shutdown, release() deletes the row outright, so the next instance can acquire immediately instead of waiting out the stale threshold.

API

Mutex(
    dsn: str | None = None,               # or pass pool / async_pool instead
    lock_name: str,
    *,
    pool: psycopg_pool.ConnectionPool | None = None,
    async_pool: psycopg_pool.AsyncConnectionPool | None = None,
    instance_id: str | None = None,       # default: "<hostname>-<random>"
    table: str = "mutex_lock",
    heartbeat_interval: float = 10.0,
    alert_threshold: float = 30.0,
    stale_threshold: float = 600.0,
    poll_interval: float = 0.5,           # blocking-acquire retry interval
    metrics: MetricsHook | None = None,
)

• mutex.acquire(blocking=False, timeout=None) — sync context manager, yields bool
• mutex.acquire_async(blocking=True, timeout=None) — async context manager
• mutex.release() / mutex.release_async()
• mutex.heartbeat() / mutex.heartbeat_async() — manual heartbeat (usually automatic)
• mutex.singleton(lock_name=None, *, blocking=False, timeout=None) — decorator
• mutex.singleton_async(...) — async decorator
• mutex.create_schema() / mutex.create_schema_async() — idempotent CREATE TABLE IF NOT EXISTS

Metrics

Implement whichever subset of MetricsHook you care about (Prometheus, OpenTelemetry, statsd, or your own):

class PrometheusMetrics:
    def acquired(self, lock_name, *, instance_id): ...
    def released(self, lock_name, *, instance_id): ...
    def contended(self, lock_name, *, instance_id): ...
    def stale_reclaimed(self, lock_name, *, instance_id, previous_holder): ...
    def alert(self, lock_name, *, holder, seconds_since_heartbeat): ...
    def heartbeat_latency(self, lock_name, *, instance_id, seconds): ...


mutex = Mutex(dsn, "nightly-report-job", metrics=PrometheusMetrics())

A broken metrics implementation can never break the lock — every call is wrapped in a best-effort try/except.

Why not X?

	Extra infra?	Survives a crash cleanly?	Blocks other queries?	Notes
postgres-mutex	No — uses Postgres you already have	Yes — heartbeat + dual-threshold auto-release	No	Single table, ~one round trip per acquire
Redis Redlock	Yes — Redis (ideally 5 independent nodes)	Depends on TTL tuning	No	Correctness has been actively debated for multi-node deployments
ShedLock	No extra infra, but adds a dependency + its own table conventions	Yes, similar TTL model	No	Good option if you're already on it; postgres-mutex is a smaller, dependency-light alternative
SELECT ... FOR UPDATE	No	No — a crashed holder's transaction rollback releases it, but a hung connection holds the lock indefinitely and blocks readers	Yes — blocks queries waiting on the row/table	Simplest option for short critical sections inside one transaction; not a good fit for "hold across a long job"
ZooKeeper / Consul / etcd	Yes — a whole coordination service to run and operate	Yes — ephemeral nodes / sessions	No	Correct and battle-tested, but a lot of infrastructure for "run this job on one instance"

Failure modes, explicitly

• Holder crashes: heartbeat stops. Alert fires at alert_threshold (default 30s). Lock is reclaimable by any other instance at stale_threshold (default 600s).
• Holder is just slow (GC pause, long query): same alert fires at 30s — that's the point, so a human can tell "slow" from "dead" before the system takes action.
• Network partition (holder alive, can't reach Postgres): heartbeats fail silently and retry; from every other instance's point of view this is indistinguishable from a crash, so the same dual-threshold logic applies. If the partition heals before stale_threshold, the original holder keeps the lock. If not, another instance reclaims it, and the original holder's next heartbeat raises LockNotHeldError — write your job logic to check for that on long-running work if split-brain during the alert window is unacceptable for your use case.
• Clock skew between instances: irrelevant. Every staleness check runs now() - last_heartbeat inside one Postgres statement — only the Postgres server's clock is ever consulted.
• instance_id reused across two live processes: don't do this — release/heartbeat scoping is by (lock_name, instance_id), so two processes sharing an instance_id can each mutate the other's lock state. The default (hostname-<random>) avoids this; set your own only if it's genuinely unique per process.

What's not in v1

• No fencing tokens
• No read/write locks — single mutex only
• No lock hierarchies
• No cross-database coordination
• Postgres only (MySQL/SQLite not planned for v1)

Development

pip install -e ".[dev]"
pytest              # spins up real Postgres via testcontainers — needs Docker
ruff check .
mypy

License

MIT