Skip to content

deploy checklist v2

Jump to bottom
compwron edited this page Nov 1, 2026 · 1 revision

Proposed 2026-11-01 as a replacement for deploy-checklist, which is unchanged. Discuss in Slack before switching over.

Whoever starts a deploy owns it until production is verified, or until it's rolled back. Pair if you can. Plan for about an hour, most of it waiting.

Production output rule: anything you paste into Slack, issues or PRs is counts and statuses only. No emails, names, IPs or record contents.

1. Before you deploy

  • Main is green. The latest commit on main has passing Actions runs: rspec, ruby_lint, erb_lint, npm_lint_and_test, factory_bot_lint. Don't deploy a red main.
  • Know what's shipping. Compare the live release with main: 
    heroku releases -a casa-production -n 5
    The top Deploy <sha> row is what's live. Check the SHA here, not /health.json: its latest_deploy_time also changes on config-var releases. Then list the changes, replacing <live-sha>: https://github.com/rubyforgood/casa/compare/<live-sha>...main
  • Review migrations. Look for anything new under db/migrate/. The release phase runs db:migrate automatically, then after_party:run (see Procfile). Check for:
    • table rewrites or long locks on big tables (case_contacts, login_activities), even though Strong Migrations catches most of these
    • anything irreversible (dropped columns or tables). Rolling back the code does not roll back the schema.
  • Review after_party tasks. Look for anything new under lib/tasks/deployment/. These run once, against all orgs. Does it delete or update data? Has someone counted how many rows it touches in production?
  • Config. Does any change need a new env var or Scheduler entry? Set env vars before deploying.
  • Post in Slack: "Deploying <live-sha> → <new-sha>, N PRs, migrations: yes/no."

2. Backup

  • Take a manual Postgres backup from the Durability tab, or run: 
    heroku pg:backups:capture -a casa-production
  • Note the backup ID (e.g. b123). You'll need it if a migration damages data.

3. Verify on QA

  • Confirm casa-qa is running the same SHA you're about to ship. Check with heroku releases -a casa-qa -n 3.
  • Exercise each user-facing change from the compare list, as the right role.
  • Check Bugsnag and #bot-bugsnag for new QA errors.

4. Deploy

  • Click Promote to production in the pipeline. This ships the exact slug you tested on QA.
    • If you're not using the pipeline: run git push heroku main. This builds a new slug, so QA didn't test that exact build.
  • Watch the release phase finish: 
    heroku releases -a casa-production -n 3
    The new version should show as succeeded, not failed. If the release phase fails (a migration or after_party error), Heroku keeps the old code running. Stop, read heroku releases:output -a casa-production, and fix forward.

5. Smoke test production (about 5 minutes)

  • https://casavolunteertracking.org/health returns OK.
  • Sign in as a casa admin and load:
    • the dashboard
    • /supervisors
    • /volunteers
    • one case page
    • the case contacts list
  • If you have a test volunteer login, open "New case contact" and check that the form loads. Don't submit it in a real org.
  • Sign in at /all_casa_admins/sign_in and load metrics.
  • Sign out.

6. Watch for 30 minutes

  • Bugsnag / #bot-bugsnag: no new error groups. Bugsnag reports roughly every 30 minutes.
  • Scout: no endpoint suddenly slower, and no allocation spikes.
  • Memory: no new R14 (memory quota exceeded) errors in Papertrail.
  • Background jobs are draining. The queue should be about 0, because Scheduler runs jobs:workoff every 10 minutes: 
    heroku run -a casa-production -- bin/rails runner 'puts "jobs=#{Delayed::Job.count} failed=#{Delayed::Job.where.not(last_error: nil).count} oldest_run_at=#{Delayed::Job.minimum(:run_at).inspect}"'
  • Scheduler: every entry is present and shows a "last run" in the past day.
  • Brevo transactional stats: sends are going out, with no jump in blocked or bounced mail.

Something wrong? Go to Rollback.

7. Announce (only after step 6 is clean)

  • Create a GitHub release:
    • tag: today's date
    • target: the deployed SHA
    • generate release notes, then sort them into categories
  • Update the in-app patch notes. Log into prod as an all casa admin, then sign out. Check them as a casa admin in the inbox.
  • Email stakeholders from casa@rubyforgood with what changed, plus screenshots.
  • Reply to the Slack deploy thread: "Done, vNNN, <sha>, smoke test and 30-minute watch clean."

Rollback

Code only (no migration in this deploy, or the migration was additive):

heroku rollback -a casa-production

This goes back to the previous release. It does not undo migrations or after_party tasks. Then re-check step 5.

A migration or after_party task broke or deleted data:

  1. Put the app in maintenance mode: heroku maintenance:on -a casa-production. Tell Slack.
  2. Get a second maintainer on the call. Restoring overwrites everything written since the backup.
  3. Restore the backup from step 2 by following Heroku's guide.
  4. Roll back the code with heroku rollback, run heroku maintenance:off, then smoke test again.

After any rollback:

  • Open an issue: what broke, which release, and how it was found (counts only, no user data).
  • Revert the commit or fix forward on main before the next deploy.

Clone this wiki locally