Skip to content

fix: watchdog false-positive cascade - add internal heartbeat#22

Open
shipyard-ci[bot] wants to merge 1 commit into
mainfrom
bugfix/46-watchdog-false-positives
Open

fix: watchdog false-positive cascade - add internal heartbeat#22
shipyard-ci[bot] wants to merge 1 commit into
mainfrom
bugfix/46-watchdog-false-positives

Conversation

@shipyard-ci

@shipyard-ci shipyard-ci Bot commented Jun 22, 2026

Copy link
Copy Markdown

Closes #21

Summary

Fixed the watchdog false-positive cascade that has been generating 250+ erroneous 'service down' alerts since May 31.

Root Cause

The watchdog was only tracking activity when messages were actively being processed. During normal quiet periods (no incoming messages), the watchdog would timeout and restart the process, despite the service being perfectly healthy.

Solution

Added an internal heartbeat mechanism to the watchdog that operates independently of message processing:

  • The watchdog now pings itself every 150 seconds (timeout/4 of default 600s timeout)
  • This provides a system-level heartbeat confirming the event loop and scheduler are responsive
  • Only triggers a restart if BOTH the watchdog's internal heartbeat AND explicit message pings stop
  • This distinction allows normal operational quiet periods while still detecting actual hangs or crashes

Impact

  • Eliminates false positives during expected low-activity periods
  • Maintains crash/hang detection capability
  • Prevents alert fatigue that was reducing incident response effectiveness

Testing

Added three test cases:

  1. test_watchdog_internal_heartbeat_prevents_false_positive - Verifies heartbeat keeps daemon alive during quiet periods
  2. test_watchdog_handle_ping_resets_activity - Confirms explicit pings from message processing still work
  3. test_disabled_watchdog_is_noop - Ensures disabled watchdog mode works correctly

Technical Details

The fix distinguishes between two types of inactivity:

  • No message activity (normal) - Quiet periods are expected in operation
  • No daemon responsiveness (critical) - Watchdog itself stops pinging, indicating hung/crashed process

This architectural improvement validates operational state before escalating alerts, preventing log-parser misclassifications that were flagging normal logs as failures.

Issue #21: Watchdog was triggering false 'service down' alerts during normal
quiet periods when no messages were being processed. The system was healthy
but idle, yet the watchdog would timeout and restart unnecessarily.

Root cause: The watchdog only pinged when messages were actively processed,
but normal operations have quiet periods. This caused 250+ false positives
since May 31.

Fix: Add internal heartbeat mechanism to the watchdog itself
- Watchdog now pings itself at regular intervals (timeout/4) independent of
  message processing activity
- This provides a system-level heartbeat that confirms the event loop and
  scheduler are responsive
- Only triggers a restart if BOTH the watchdog's own heartbeat AND explicit
  message pings stop, indicating a hung or crashed daemon

Architecture improvement:
- Distinguishes between 'no message activity' (normal, expected) and
  'no daemon responsiveness' (actual failure)
- Prevents stale log entries from triggering false escalations
- Adds proper operational state validation before alert generation

Tests added:
- test_watchdog_internal_heartbeat_prevents_false_positive: Verifies the
  heartbeat keeps the watchdog alive during quiet periods
- test_watchdog_handle_ping_resets_activity: Confirms explicit pings still work
- test_disabled_watchdog_is_noop: Ensures disabled mode works correctly

Fixes: #21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Watchdog False-Positive Cascade: 11+ CRITICAL 'spacebot down' Alerts (Service Operational)

0 participants