More control over service degradations

[heipei]

We receive frequent alerts due to alleged service degradation which can't be reproduced on our end. Our servers run in Germany (Hetzner, Falkenstein), and we perform checks using Frankfurt and other European regions. Sometimes we do receive degradation alerts from Ireland/Milan since the HTTP request took more than 3 seconds. These are always one-time incidents, the next check works fine, and we don't see slow response times in our access logs and almost never receive degradation alerts from Frankfurt.

It would be great if there could be additional thresholds for degradation alerts, e.g. only alert after three successive degradations were observed, or only if the degradation is observed from all regions simultaneously.

For now we're limiting the checks to only Frankfurt and Paris and enable double checks in order to hopefully reduce the amount of incorrect alerts we receive. Please let me know if you need more information.

[tnolet]

@heipei we are looking at (way) more flexible alerting the coming quarters. I added this ticket to the overarching one. https://github.com/orgs/checkly/projects/4/views/4?pane=issue&itemId=21238722