Consecutive-failure tracking is backed by a Redis set per destination, holding the IDs of failed attempts; the consecutive-failure count is just the size of that set. Each failure adds its attempt ID to the set and resets a 24h TTL on the whole key (Redis TTLs are per-key, not per-member — the clock covers the entire set). Because the TTL resets from the last failure, the streak only survives if adjacent failures stay within 24h of each other.
When a gap exceeds 24h, the set expires and the count silently restarts:
| time |
event |
TTL set to expire at |
key alive when next failure lands? |
set after |
count |
| t=0h |
fail |
t=24h |
— (new) |
{a1} |
1 |
| t=25h |
fail |
t=49h |
no — expired at t=24h |
{a2} |
1 |
The first failure's set was dropped before the second arrived, so the second failure starts over at count 1 instead of 2.
This is a blind spot for very low-traffic destinations (e.g. one failing event every day or two): adjacent failures land more than 24h apart, so the count never accumulates enough to cross the alert thresholds or auto-disable, despite the destination being effectively 100% failing. Exhausted-retries alerts still fire on their own path, so it's not fully silent.
Opening this to confirm the 24h sliding window is intended behavior and document it (so it's not later mistaken for a bug), and to decide whether the low-traffic case warrants any change.
Consecutive-failure tracking is backed by a Redis set per destination, holding the IDs of failed attempts; the consecutive-failure count is just the size of that set. Each failure adds its attempt ID to the set and resets a 24h TTL on the whole key (Redis TTLs are per-key, not per-member — the clock covers the entire set). Because the TTL resets from the last failure, the streak only survives if adjacent failures stay within 24h of each other.
When a gap exceeds 24h, the set expires and the count silently restarts:
{a1}{a2}The first failure's set was dropped before the second arrived, so the second failure starts over at count 1 instead of 2.
This is a blind spot for very low-traffic destinations (e.g. one failing event every day or two): adjacent failures land more than 24h apart, so the count never accumulates enough to cross the alert thresholds or auto-disable, despite the destination being effectively 100% failing. Exhausted-retries alerts still fire on their own path, so it's not fully silent.
Opening this to confirm the 24h sliding window is intended behavior and document it (so it's not later mistaken for a bug), and to decide whether the low-traffic case warrants any change.