feat(ci): automated rollback on deployment failure with health-check gating

[Pranav-chaudhari-2006]

Summary

Failed deployments can leave the service in a broken state with no automatic recovery. This PR introduces a multi-layer automated rollback system that triggers rollbacks based on health checks and deployment outcomes — covering GitHub Actions CI/CD, Kubernetes, and Docker Compose environments.

---

Changes Made

🆕 New Files

.github/workflows/deploy.yml

A full production deployment pipeline that runs automatically after the CI workflow succeeds on main.

Pipeline flow:

Build & tag Docker image (sha-<short> + latest)
  └─► Push to GHCR
        └─► kubectl set image (rolling update)
              └─► kubectl rollout status --timeout=3m
                    ├─ FAILURE → kubectl rollout undo ──► Step Summary ──► ❌ workflow fails
                    └─ SUCCESS → Poll /healthz/ready for 60s
                          ├─ TIMEOUT → kubectl rollout undo ──► Step Summary ──► ❌ workflow fails
                          └─ HTTP 200 → ✅ Deployment summary written

• Images are tagged with both latest and a pinned sha-<commit> tag, so the previous image is always reachable for rollback
• A GitHub Step Summary is written on rollback, showing the failed image, the reverted image, and next-step instructions

---

deploy/scripts/rollback.sh

A standalone Bash script for manual or emergency rollbacks from any CI system or terminal.

# Roll back to the previous revision
bash deploy/scripts/rollback.sh

# Roll back to a specific revision
bash deploy/scripts/rollback.sh --revision 5 --namespace production

# Override the health endpoint
bash deploy/scripts/rollback.sh --health-url https://api.yourdomain.com --timeout 120

What it does:

1. Captures current image + revision before undoing
2. Calls kubectl rollout undo
3. Waits for the undo rollout to stabilise
4. Polls /healthz/ready for up to 90 s to confirm the rollback itself is healthy
5. Exits non-zero if the rollback image is also unhealthy (prevents silent failures)
6. Annotates the deployment with the rollback reason for audit history

---

.github/workflows/staging-smoke.yml

A lightweight smoke-test workflow that runs on every push to develop and blocks production promotion if any check fails.

Checks performed:

Check	Endpoint	Pass condition
Liveness	GET /healthz/live	HTTP 200
Readiness	GET /healthz/ready	HTTP 200
API smoke	POST /api/chat	HTTP 200, 401, or 422

---

✏️ Modified Files

deploy/k8s/deployment.example.yaml

Added fields that arm Kubernetes's own rollout controller as a native auto-rollback line of defence:

Field	Value	Effect
strategy.type	RollingUpdate	Explicitly documented
maxUnavailable	0	Never drop below full replica count during rollout
maxSurge	1	Bring up 1 extra pod before killing an old one
minReadySeconds	15	Pod must stay healthy 15 s before it counts as ready
progressDeadlineSeconds	180	K8s marks rollout Failed → auto-triggers rollout undo after 3 min

progressDeadlineSeconds is the native Kubernetes rollback trigger — it fires independently of the GitHub Actions workflow, providing a second automatic rollback layer.

---

docker-compose.yml

Health-check and restart improvements for local and staging Docker Compose environments:

# backend
healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8000/healthz/ready"]
  interval: 10s
  timeout: 5s
  retries: 5
  start_period: 15s
restart: on-failure:3   # stops crash-restart loops after 3 failures

# frontend
depends_on:
  backend:
    condition: service_healthy   # won't start until backend is confirmed healthy
restart: on-failure:3

---

Setup & Requirements

GitHub Secrets & Variables

Name	Type	Value
KUBECONFIG_B64	Secret	base64 -w0 ~/.kube/config
SERVICE_HEALTH_URL	Variable	e.g. https://api.yourdomain.com
STAGING_URL	Variable	e.g. https://staging.yourdomain.com

Go to Settings → Secrets and variables → Actions to add these.

Permissions

The deploy.yml workflow uses GITHUB_TOKEN for GHCR pushes — no additional PAT needed. The production environment should be configured in Settings → Environments with any required approval rules.

---

How to Run / Test

Trigger the deploy pipeline

Push to main after CI passes — deploy.yml runs automatically.
Or trigger it manually: Actions → Deploy with Automated Rollback → Run workflow.

Trigger the staging smoke tests

Push to develop — staging-smoke.yml runs automatically.
Or trigger manually via Actions → Staging Smoke Tests → Run workflow (you can override the staging URL in the inputs).

Run the rollback script manually

# Make executable (Linux/macOS)
chmod +x deploy/scripts/rollback.sh

# Run with defaults
bash deploy/scripts/rollback.sh

# Full options
bash deploy/scripts/rollback.sh \
  --namespace production \
  --deployment qyverixai \
  --health-url https://api.yourdomain.com \
  --timeout 120 \
  --revision 5

Test the Docker Compose health checks locally

docker compose up --build
# Backend must pass /healthz/ready before frontend container starts
docker compose ps   # check STATUS column — should show "(healthy)"

---

Staging Rollback Test Checklist

Before enabling in production, validate these scenarios in staging:

• Push a non-existent image tag → confirm kubectl rollout status fails and rollback fires
• Push an image whose /healthz/ready always returns 503 → confirm health-gate catches it
• Run staging-smoke.yml manually → confirm all three checks pass
• Run rollback.sh --revision <n> → confirm it rolls to the correct revision
• Confirm rollback annotations appear in kubectl rollout history

---

References

• 📄 deploy.yml — Production deploy workflow with auto-rollback
• 📄 staging-smoke.yml — Staging smoke-test gate
• 📄 rollback.sh — Manual rollback script
• 📄 deployment.example.yaml — Kubernetes manifest with rollout strategy
• 📄 docker-compose.yml — Local/staging health-check config
• 📖 Kubernetes Deployment Rolling Update docs
• 📖 GitHub Actions workflow_run trigger

---

Rollback Decision Tree

Deploy triggered on main
│
├─[K8s] progressDeadlineSeconds=180 fires (pods not ready in 3 min)
│         └─► K8s controller auto-issues rollout undo
│
├─[CI] kubectl rollout status times out
│         └─► deploy.yml → kubectl rollout undo + GitHub Step Summary
│
├─[CI] /healthz/ready doesn't return 200 within 60s
│         └─► deploy.yml → kubectl rollout undo + GitHub Step Summary
│
└─[Manual] On-call engineer
              └─► bash deploy/scripts/rollback.sh

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR improves deployment reliability by adding health-gated startup in Docker Compose, introducing Kubernetes rollout safety knobs, and adding GitHub Actions workflows/scripts to support automated and manual rollback flows.

Changes:

• Add Docker Compose healthchecks and gate frontend startup on backend readiness.
• Add a manual Kubernetes rollback script plus example deployment rollout strategy settings.
• Add GitHub Actions workflows for staging smoke tests and production deploy with auto-rollback.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
docker-compose.yml	Adds backend healthcheck + restart policy and gates frontend on backend health.
deploy/scripts/rollback.sh	Introduces a manual rollback helper that undoes a deployment and polls readiness.
deploy/k8s/deployment.example.yaml	Documents and configures safer rolling update parameters for Deployments.
.github/workflows/staging-smoke.yml	Adds a staging smoke workflow to probe health endpoints and a basic API call.
.github/workflows/deploy.yml	Adds build/push + deploy workflow with an explicit health gate and auto-rollback.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.