Possibility to add load balancer for API

[Cr4mble]

Hi, thanks for the great project!
I'm currently updating from v1.1.5 to v.2.0.8.
As of v2.x.x a load balancer is not created anymore in front of the three master nodes in HA mode.

We have two k3s clusters on Hetzner and on one of these there is an ArgoCD that should connect to the second cluster.
If i do a argocd cluster add to add the second cluster to ArgoCD i can select the cluster from my local kubeconfig.
As of the new version there are the three master pods and not the load balancer. If i now add one of these nodes ArgoCD will always try to connect to the IP address of this node.
If this master node goes down ArgoCD is not able to connect to the cluster anymore and the HA is gone from this perspective.

Is there a possibility to add a flag that i can decide that i want an HA mode with and without a load balancer in front?

Thanks, Cheers!

[sven-borkert]

I just wanted to ask the same. The documentation mentions an API loadbalancer, but none gets created. Am I supposed to manually create it, or what's the idea here?

[vitobotta]

I understand your use case. It's a valid point and I will see if I can reintroduce the load balancer as an optional setting.

There are reasons why 2.x doesn't create a load balancer:

• hetzner-k3s allows you to restrict access to the Kubernetes API to specific networks, for better security. The problem is that when using a load balancer in front of the masters, this doesn't work because load balancers are not supported by Hetzner firewalls yet, so that means that security was weakened with HA clusters before. By replacing the load balancer with a multi context kubeconfig, instead, you are connecting to a master directly, so the firewall rules apply nicely.
• many people asked to remove the requirement for the load balancer to save some bucks. This is a secondary reason but anyway it was requested by many.

I can make enabling the load balancer again optional for use cases like yours, perhaps with a warning that firewall rules for the API won't work when connecting to it via load balancer.

[vitobotta]

I just wanted to ask the same. The documentation mentions an API loadbalancer, but none gets created. Am I supposed to manually create it, or what's the idea here?

The load balancer should no longer be mentioned in the docs but only in the release notes for the reasons I mentioned in the other comment. Where do you see the docs mentioning that a load balancer is still created?

[Cr4mble]

@vitobotta from a security perspective it's absolutely not right to not restrict traffic to the control planes which is exposed via LoadBalancer, i totally agree on that. Would be nice if Hetzner would implement this on the LoadBalancer level cause it's a common use case to restrict it.

In our case we have a LoadBalancer in front of the three control planes and we restrict the traffic to certain IPs. If i do not have the whitelisted IP i get a timeout. So this is working perfectly. I think the LoadBalancer of Hetzner is forwarding our IP to the control plane and then it's getting blocked on the control plane level cause of the firewall.

So my theory is:
Traffic to LoadBalancer with not whitelisted IP -> forwarding to control planes as there is no firewall rule
Traffic is getting to the Control Plane -> it's getting blocked here cause of the firewall attached to it

[vitobotta]

That's interesting. Did you enable proxy protocol on the load balancer? Otherwise I don't understand how the firewall would see the actual IP of the client instead of that of the load balancer itself. Also, normally proxy protocol support must be also enabled on the upstream. Can you clarify a bit your configuration/setup?

[sven-borkert]

The load balancer should no longer be mentioned in the docs but only in the release notes for the reasons I mentioned in the other comment. Where do you see the docs mentioning that a load balancer is still created?

I think I just read this comment in "Creating_a_cluster.md":

"this will firewall port 6443 on the nodes; it will NOT firewall the API load balancer"

I don't need an API loadbalancer to be created, I was just wondering if I missed something. Never mind. Thank you for this great tool, it's very helpful for me. 👍

[vitobotta]

The load balancer should no longer be mentioned in the docs but only in the release notes for the reasons I mentioned in the other comment. Where do you see the docs mentioning that a load balancer is still created?

I think I just read this comment in "Creating_a_cluster.md":

"this will firewall port 6443 on the nodes; it will NOT firewall the API load balancer"

I don't need an API loadbalancer to be created, I was just wondering if I missed something. Never mind. Thank you for this great tool, it's very helpful for me. 👍

Thanks, I amended that section of the document. I guess we can close this then. But do you mind clarifying if you have enabled the proxy protocol in the load balancer? I could add some info to the docs for people who like you prefer a load balancer as optional thing.

[Cr4mble]

That's interesting. Did you enable proxy protocol on the load balancer? Otherwise I don't understand how the firewall would see the actual IP of the client instead of that of the load balancer itself. Also, normally proxy protocol support must be also enabled on the upstream. Can you clarify a bit your configuration/setup?

My bad, my theory is gone. We had a test cluster where i added some IP addresses for whitelisting. This test cluster had only one control plane and no load balancer. There the connection was not possible cause of the firewall on node layer. We have another more realistic cluster with three control planes and a load balancer but there i can connect from every IP.
I tested this on the wrong cluster.
Sorry for the false alert.
Would be nice of Hetzner if they implement this firewall on load balancer layer.

Do you think that you can activate the load balancer feature an version 2.x.x for people who are aware of this risk but need this for ArgoCD HA for example?

Cheers

[vitobotta]

Yeah we can keep this issue open as a reminder. I will see if I can reimplement native support in the tool but I can't promise when since I have a long list of things for the next release already.