Setup hot-standby replica for clickhouse cluster #131
Description
Currently our clickhouse cluster has 3 nodes, 2 data nodes and 3 zookeeper nodes.
Originally we were planning to add a third data node (data2.htz-fsn.prod.ooni.nu), however it's not possible to do that with the current disk setup on the host, since the disks are unevenly sized and a raid5 configuration would lead to too much wasted disk space. We could go for raid0, but that seems a bit risky to do on nvme drives.
In light of this, we should for the moment keep the cluster to be 2 nodes (data3 being the cold replica since it's HDD backed) and discontinue data2 to keep costs down.
For the time being we keep data2.htz-fsn.prod.ooni.nu a reserved hostname label for when we decide to finalize setting it up.
We probably want to do that once we have finished the backend-fsn and backend-hel migration so we can cut down the cost of our dedicated hosts.