fix: watchdog false-positive cascade - add internal heartbeat#22
Open
shipyard-ci[bot] wants to merge 1 commit into
Open
fix: watchdog false-positive cascade - add internal heartbeat#22shipyard-ci[bot] wants to merge 1 commit into
shipyard-ci[bot] wants to merge 1 commit into
Conversation
Issue #21: Watchdog was triggering false 'service down' alerts during normal quiet periods when no messages were being processed. The system was healthy but idle, yet the watchdog would timeout and restart unnecessarily. Root cause: The watchdog only pinged when messages were actively processed, but normal operations have quiet periods. This caused 250+ false positives since May 31. Fix: Add internal heartbeat mechanism to the watchdog itself - Watchdog now pings itself at regular intervals (timeout/4) independent of message processing activity - This provides a system-level heartbeat that confirms the event loop and scheduler are responsive - Only triggers a restart if BOTH the watchdog's own heartbeat AND explicit message pings stop, indicating a hung or crashed daemon Architecture improvement: - Distinguishes between 'no message activity' (normal, expected) and 'no daemon responsiveness' (actual failure) - Prevents stale log entries from triggering false escalations - Adds proper operational state validation before alert generation Tests added: - test_watchdog_internal_heartbeat_prevents_false_positive: Verifies the heartbeat keeps the watchdog alive during quiet periods - test_watchdog_handle_ping_resets_activity: Confirms explicit pings still work - test_disabled_watchdog_is_noop: Ensures disabled mode works correctly Fixes: #21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #21
Summary
Fixed the watchdog false-positive cascade that has been generating 250+ erroneous 'service down' alerts since May 31.
Root Cause
The watchdog was only tracking activity when messages were actively being processed. During normal quiet periods (no incoming messages), the watchdog would timeout and restart the process, despite the service being perfectly healthy.
Solution
Added an internal heartbeat mechanism to the watchdog that operates independently of message processing:
Impact
Testing
Added three test cases:
Technical Details
The fix distinguishes between two types of inactivity:
This architectural improvement validates operational state before escalating alerts, preventing log-parser misclassifications that were flagging normal logs as failures.