Skip to content

ESCALATION_POLICY.md

Latest commit

 

History

History
35 lines (26 loc) · 1.59 KB

ESCALATION_POLICY.md

File metadata and controls

35 lines (26 loc) · 1.59 KB
Raw

Escalation Policy

This document outlines the standard escalation path for all critical production alerts generated by the teachLink_backend alerting service. When an alert triggers (e.g. PAYMENT_FAILURE_RATE_CRITICAL), it is automatically sent to Slack and PagerDuty.

1. Primary On-Call (L1 Support)

  • Time to Acknowledge (TTA): 15 minutes
  • Responsibilities:
    • Acknowledge the alert in PagerDuty immediately.
    • Review the corresponding runbook in docs/RUNBOOKS.md.
    • Perform initial triage and attempt mitigation (e.g. failing over a database connection, restarting a pod, toggling a feature flag).
  • If unacknowledged after 15 minutes, the alert automatically escalates to L2.

2. Secondary On-Call (L2 Support)

  • Time to Acknowledge (TTA): 15 minutes
  • Responsibilities:
    • Intervene if L1 is unavailable or unable to resolve the issue.
    • In-depth investigation into backend logs and infrastructure metrics.
    • Coordination with platform and infrastructure teams if necessary.
  • If unacknowledged after 15 minutes, the alert automatically escalates to L3.

3. Engineering Manager / Director (L3 Support)

  • Time to Acknowledge (TTA): ASAP
  • Responsibilities:
    • Take over incident command.
    • Send out internal/external status communications (Statuspage).
    • Mobilize the necessary engineering task force to resolve the outage.

Slack Communication Protocol

  • All active incidents should be discussed in the #incidents-active Slack channel.
  • Once the incident is resolved, a post-mortem document should be drafted and reviewed within 48 hours.