Skip to content

[Maintenance][Alerts] Post-window verification: alert (and ticket) when a device that was online before a maintenance window is still offline after it #5236

Description

@ToddHebebrand

Description

Split from #4981 (mega-issue), and the maintenance-window half of #4330. After a scheduled reboot or maintenance window, nothing in Breeze checks that the device came back:

Alert if a server that was previously online is offline at the end of that maintenance window.

When the job runs it creates a ticket to say the server is rebooting, then monitors and waits for the RMM to show it online again. If online it closes the ticket; if offline for more than 20 min it sends an urgent alert via Teams.

Today maintenanceRebootWorker.ts only issues reboots to online devices with a pending-reboot flag during an active window; there is no window-close hook, no "was online before / still offline after" comparison, and no post-maintenance alert or ticket. The generic offline alert (alertConditions/handlers/offline.ts) is typically suppressed during the window, so a server that never returns is silent until someone notices.

Proposed Fix

  1. Snapshot at window open: record which targeted devices were online when the window (or reboot batch) started.
  2. Verify at window close (+ grace): for each device that was online at open and has not heartbeated by windowEnd + graceMinutes (default 20), raise a maintenance_reboot_failed alert at high/critical severity through the normal alert pipeline (so Teams/email/webhook routing and escalation apply, and automations can trigger on it).
  3. Optional ticket lifecycle: when a ticket target is configured, open a "Scheduled reboot in progress" ticket at batch start and auto-close it when the device is back; leave it open and escalate if verification fails. (Depends on [Alerts][Ticketing] Ticket as an alert target: 'ticket' notification channel type and 'create_ticket' automation action #5238, ticket as an alert/automation target.)

Affected Files

  • apps/api/src/jobs/maintenanceRebootWorker.ts
  • apps/api/src/services/alertConditions/ (new handler) or a dedicated post-maintenance job
  • apps/api/src/db/schema/configurationPolicies.ts (grace + verification flags on configPolicyMaintenanceSettings)
  • apps/web/src/components/configurationPolicies/featureTabs/MaintenanceTab.tsx

Reported By

stressedout9064 on Discord (#feature-suggestions, 2026-09-04 and 2026-08-30) — split from #4981; overlaps #4330 (reboot-and-monitor script). Related: #5235 (window ordering/dependency).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions