flake: [Infrastructure] Terraform provider install 503 - provisioner/terraform TestProvision/ai-task-app-id

[flake-investigator]

CI Run: https://github.com/coder/coder/actions/runs/20134964926
Commit: 8ead6f795d182b747309740b300ddd567f31a225 (author: Danielle Maywood)

Failing job: test-go-race-pg (Linux)
Failure time: 2025-12-11T13:50:34Z (same window as Slack alert)

Test failure:

• Package: provisioner/terraform
• Test: TestProvision/ai-task-app-id
• Step: terraform init during provider install

Log evidence:
...snip...
log: [INFO] Installing provider version: coder/coder v2.13.1...
log: [ERROR] Error: Failed to install provider
log: [ERROR] Error while installing coder/coder v2.13.1: github.com: bad response code: 503
initialize terraform: exit status 1
...snip...

Root cause classification: Infrastructure (external provider registry returned 503). Not a product/test logic failure; transient network/service outage during provider install.

Duplicate search summary:

• repo:coder/internal "Failed to install provider" terraform — no results
• repo:coder/internal "bad response code: 503" — no results
• repo:coder/internal TestProvision/ai-task-app-id — no results

Assignment analysis:

• Component area: provisioner/terraform tests and provider installation caching/infra
• Recent ownership context in provisioner/terraform: multiple infra-related changes by Steven Masley (Emyrk) and others (provider caching, test infra). Assigning to Steven for initial triage; reassign as needed.

Action requested:

• Validate provider cache/proxy behavior covered this path; consider retry logic around provider install to mitigate transient 5xx
• If already addressed, ensure this test path uses the shared caching helpers so we avoid external fetches during CI.

[flake-investigator]

Reoccurrence detected. CI run: https://github.com/coder/coder/actions/runs/20138644218

Failing job: gen
Evidence excerpt:

Error: Failed to install provider
Error while installing kreuzwerker/docker v2.25.0: github.com: bad response
code: 503

Classification: Infrastructure (external provider registry/GitHub 503 during provider install).

[flake-investigator]

Recurrence on main CI within minutes of this comment.

CI Run: https://github.com/coder/coder/actions/runs/20138644218
Commit: 6f86f67754e53b9cc4d8d55d3b255369b09a9007 (main)

Failing job(s):

• test-go-pg (ubuntu-latest)
• gen

Key log evidence (test-go-pg):

log: [INFO] Installing provider version: coder/coder v2.13.1...
log: [ERROR] Error: Failed to install provider
log: [ERROR] Error while installing coder/coder v2.13.1: github.com: bad response code: 503
initialize terraform: exit status 1

Key log evidence (gen - terraform testdata generation):

Error: Failed to install provider
Error while installing kreuzwerker/docker v2.25.0: github.com: bad response
code: 503

Additional context: the same workflow’s build job failed while downloading rcodesign from GitHub with:

ERROR 503: Service Unavailable. (wget from github.com)

Classification: Infrastructure outage at GitHub/provider registry causing transient 5xx during provider/binary fetch. Not a product or test logic regression.

Action hint: consider short retry around provider install steps or greater cache reliance to mitigate transient upstream 5xx.

[flake-investigator]

New occurrence from main.

• CI run: https://github.com/coder/coder/actions/runs/20143099625
• Commit: 2e4aa729bee1b6aacecf5b38be3b5ccffcc8eb6c (author: Mathias Fredriksson)
• Failing jobs: gen, test-go-race-pg
• Failure time window: 2025-12-11T18:20–18:26Z (same window as Slack alert)

Key log evidence (provider install via terraform init):

• gen:
  Error while installing coder/coder v2.13.1: github.com: bad response code: 504
• test-go-race-pg:
  provisioner/terraform TestProvision/ai-task-app-id
  log: [ERROR] Error: Failed to install provider
  log: [ERROR] Error while installing coder/coder v2.13.1: github.com: bad response code: 504
  initialize terraform: exit status 1

Classification: Infrastructure (external provider registry 504). No data race, no product test assertion involved. Same signature as this issue.

Action hint: consider retry/backoff around provider install or ensure test path consistently uses cached provider to avoid external fetch in CI.

[flake-investigator]

Recurrence observed in main.

• CI Run: https://github.com/coder/coder/actions/runs/20148032849
• Commit: a87a44412fe208c77476bcbc503c75b6ed6f5341 (author: Andrew Aquino)
• Job: test-go-race-pg (Linux)
• Failure time: 2025-12-11T21:37:47Z (same window as Slack alert)

Failing tests and evidence:

• Package: provisioner/terraform
• Tests:
  • TestProvision/ai-task-app-id
  • TestProvision/external-agent
  • Parent: TestProvision (aggregate)

Key log lines:

log: [INFO] Installing provider version: coder/coder v2.13.1...
log: [ERROR] Error: Failed to install provider
log: [ERROR] Error while installing coder/coder v2.13.1: github.com: bad response code: 504
initialize terraform: exit status 1

Classification: Infrastructure (external provider registry/network 5xx during terraform init). No panic/OOM or race detector warnings present in the logs.

Notes:

• Matches the same failure mode as this issue (prior 503). Suggest adding retry/backoff around provider install in test harness and/or ensuring provider cache/proxy prevents external fetch in CI path.
• Assignment unchanged (infra/provisioner test infra ownership).