feat: Implement distributed Redis lock before provisioning mqinfra

[alexluong]

When a node starts up, it needs to ensure the required infrastructure (queues, topics, etc.) exists. Since we might have multiple nodes starting simultaneously, we need to coordinate who creates the infrastructure.

Here's how it works:

The retry loop: Each node tries up to 5 times to handle the infrastructure setup, with a 5-second wait between attempts. This gives nodes time to coordinate without hammering Redis.

The flow for each attempt:

1. First, we check if the infrastructure already exists - if it does, we're done! No lock needed.
2. If infrastructure doesn't exist, we try to grab a distributed lock in Redis (using SET NX with a 10-second expiry)
3. If we get the lock, great! We provision the infrastructure and release the lock
4. If someone else has the lock, we wait 5 seconds and try the whole thing again

Why this works well:

• Most of the time (after initial setup), infrastructure already exists, so nodes skip everything and start immediately
• During initial cluster deployment, one node will win the lock race and create the infrastructure while others wait
• The 10-second lock expiry protects against crashed nodes holding locks forever
• If something goes catastrophically wrong and a node can't start after 5 attempts, it fails health checks and gets recycled

This flow is really only relevant during the very first deployment. After that, the infrastructure exists and every node just breezes through the "does it exist?" check.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
outpost-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 25, 2025 2:31am
outpost-website	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 25, 2025 2:31am