AWS Region Failover

Region Failover

When to failover

Primary region (us-east-2) is having significant issues, which are preventing Core from handling requets.

Failover caveats

• S3 uploads do not instantly replicate between regions, so there may be some data inconsistency when failing over for ~15 minutes.
• If the primary Redis instance is also down, we will lose in-progress locks, so if there is still intermittent traffic/active locks, you should hold off on failover until critical locks are released in Redis (can check in Upstash).

How to failover

• Go to Cloudfront and change the origins to point to the us-west-2 versions.
• Ideally force restart the us-east-2 and us-west-2 lambdas if possible by setting a random env variable, which will cause it to re-check Redis.

This page is a stub: you can help by bothering @devksingh4 to actually finish writing it.