Runners monitor and alert

[shashidhar087]

Has anybody thought about runner monitoring and sending an alert when number of idle runners scale down below set threshold value ? We have implemented the monitoring and alert feature internally in our org and the feature is working fine since last few weeks. We use runner pool feature and use some of the pool parameters to monitor the runners.

I am happy to raise a PR and we can discuss if anything need to be improved so that everybody can use the feature.

Just to give high level picture, lambda function is monitoring number of idle runners per environment, per org and puts custom metric value to cloudwatch which is integrated to cloudwatch alarm when the metric count goes less than set threshold value (25% of pool count). SNS topic subscription to email or any endpoint to receieve notifications. We are using OpsGenie as our alert tool.

[shashidhar087]

Has anybody thought about runner monitoring and sending an alert when number of idle runners scale down below set threshold value ? We have implemented the monitoring and alert feature internally in our org and the feature is working fine since last few weeks. We use runner pool feature and use some of the pool parameters to monitor the runners.

I am happy to raise a PR and we can discuss if anything need to be improved so that everybody can use the feature.

Just to give high level picture, lambda function is monitoring number of idle runners per environment, per org and puts custom metric value to cloudwatch which is integrated to cloudwatch alarm when the metric count goes less than set threshold value (25% of pool count). SNS topic subscription to email or any endpoint to receieve notifications. We are using OpsGenie as our alert tool.

@npalm @ScottGuymer

[ScottGuymer]

We are very interested in monitoring solutions around this module and is something we have yet to put a lot of real thought into.

Very curious how this is working for you and what other things you intend to monitor.

[shashidhar087]

I shall open a PR in the next few days on what we have done till now on the monitoring.

[shashidhar087]

https://github.com/philips-labs/terraform-aws-github-runner/pull/1972