Infra disk exhaustion incident (cloud + CI previews)

[ClaireGz]

What happened (chronologically)

1. The disk filled up on our infra — both the cloud app volume (/app/projects) and the CI PR-preview host ran out of space ("no space left on device").
2. On the cloud, importing a GitHub project started failing with ENOSPC: no space left on device when creating the import temp dir.
3. Still on the cloud, "Pull latest" also failed — with no space, git couldn't even write its .git/config.lock file, surfacing a confusing git error to the user (see Cloud git sync breaks permanently on stale .git/config.lock #1174).
4. On CI, the PR preview host was out of disk, so Docker couldn't pull/extract the PR image → the preview container was never created → nginx had nothing to route to → 502 (with a stale "Not Secure" fallback). The deploy script swallowed the Docker error and reported "✅ success", so CI stayed green and nobody was alerted.
5. Disk was freed by manually running the Nightly cleanup workflow (.github/workflows/remove-old-artifacts.yml). Cloud and CI are working again.

What was impacted

• Cloud: GitHub project import and "Pull latest" both broken for affected projects.
• CI: PR preview deployments returned 502 / broken cert, while CI falsely reported success.

Status

Resolved for now (disk manually cleaned). Kept open because the recovery was manual and the fallback needs to be improved so this doesn't recur silently.

[ClaireGz]

Update: unblocked

Disk was freed by manually running the Nightly cleanup workflow (.github/workflows/remove-old-artifacts.yml) — cloud is working again. Keeping this open for the fallback, since today it only recovered because someone noticed and ran the job by hand.

Gaps in the current fallback

The nightly cleanup has real holes:

1. Only runs once a day (1am), reactively to nothing. Disk can hit 100% at any point between runs and stay broken until 1am or a manual trigger. There's no on-demand / self-healing path.
2. docker rmi -f $(docker images -aq) doesn't reclaim everything. It removes images but not stopped containers, build cache, or unused volumes. On a box that builds an image per PR, build cache is usually the biggest hog — so the box can still fill even with the nightly running.
3. It only cleans the CI preview VM. It does not touch the cloud app volume (/app/projects) — the leaked .github-import-* / .project-import-backup-* temp dirs have no scheduled cleanup at all.
4. Failures are silent. The deploy script swallows the Docker ENOSPC and reports "✅ success," so nothing signals that disk is the problem.

Better fallback (proposed)

• Preflight + reactive prune in the deploy job: before each PR preview build/deploy, check df; if low, run docker system prune -af --volumes and continue. Self-heals on demand instead of waiting for 1am.
• Use a full prune in the nightly: replace docker rmi -f $(docker images -aq) with docker system prune -af --volumes (and/or docker builder prune -af) to reclaim build cache + volumes.
• Fail loud on ENOSPC: stop the deploy script masking Docker errors as success — on "no space", prune-and-retry once, then fail the job visibly if still stuck.
• Disk alerting at ~80% on both the preview VM and the cloud app volume, so we get warning before 100%.
• Cloud app volume cleanup: code-level guaranteed removal of .github-import-* / .project-import-backup-* temp dirs (nightly doesn't cover this host) — see Cloud git sync breaks permanently on stale .git/config.lock #1174.