Investigate quicker ways to deploy monitoring scripts

[FedericoCeratto]

Setting up daemons that write to node_exporter is time consuming

We can test netdata plugins after #268

[SuperQ]

Why not make the daemons that expose Prometheus metrics directly, rather than write out to the node_exporter?

[FedericoCeratto]

@SuperQ it would make the daemons significantly more complex, having to run a dedicated thread each to expose the metrics, and more burdensome to deploy, having to assign a unique port to each, allow incoming traffic and configure Prometheus to scrape each of them.
It's difficult to justify when the code doing the real monitoring work is 1 line of bash. (e.g. finding and counting files, running one SQL query, ...)

[SuperQ]

The kinds of monitoring tasks you're talking about are already implemented by good quality code. For example, there are several SQL query exporters for exposing SQL query data.

Add monitoring endpoints to existing daemons is not "significantly more complex". These techniques are well documented and common.

[hellais]

Each service is now exposing metrics directly and we are working toward discontinuing netdata in favour of this. I am closing this issue in favour of the several monitoring related ones we have elsewhere:

• Improve monitoring of ooniservices devops#157
• Setup monitoring of ECS nodes devops#104
• Clean up current monitoring and alerting systems #882