LoadBalancer doesn't add any droplets (crashes on a droplet not used in the cluster)

[Richard87]

Hi!

I have just created a new cluster with kubeadm, and added do-ccm. When creating a loadbalancer it failed, it got stuck while adding droplets to the loadbalancer:

kubectl -n kube-system logs -f digitalocean-cloud-controller-manager-56bd986844-jn286
W0708 16:30:39.020273       1 client_config.go:533] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W0708 16:30:39.048026       1 controllermanager.go:105] detected a cluster without a ClusterID.  A ClusterID will be required in the future.  Please tag your cluster to avoid any future issues
W0708 16:30:39.051714       1 authentication.go:55] Authentication is disabled
I0708 16:30:39.051824       1 insecure_serving.go:44] Serving insecurely on [::]:10253
I0708 16:30:39.054121       1 node_controller.go:86] Sending events to api server.
I0708 16:30:39.055122       1 pvlcontroller.go:107] Starting PersistentVolumeLabelController
I0708 16:30:39.055139       1 controller_utils.go:1019] Waiting for caches to sync for persistent volume label controller
I0708 16:30:39.055951       1 controllermanager.go:258] Will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true.
I0708 16:30:39.056171       1 service_controller.go:183] Starting service controller
I0708 16:30:39.056192       1 controller_utils.go:1019] Waiting for caches to sync for service controller
I0708 16:30:39.155327       1 controller_utils.go:1026] Caches are synced for persistent volume label controller
I0708 16:30:39.156331       1 controller_utils.go:1026] Caches are synced for service controller
I0708 16:30:39.156534       1 service_controller.go:636] Detected change in list of current cluster nodes. New node set: map[worker1:{} worker2:{} worker3:{}]
I0708 16:30:39.163151       1 service_controller.go:644] Successfully updated 1 out of 1 load balancers to direct traffic to the updated set of nodes
I0708 16:52:36.106744       1 event.go:218] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"worker2", UID:"efd803d9-82bf-11e8-ac1a-e21a8d63bb7d", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'Deleting Node worker2 because it's not present according to cloud provider' Node worker2 event: DeletingNode
I0708 16:53:46.963070       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"nginx", UID:"d456d2d9-82ce-11e8-ac1a-e21a8d63bb7d", APIVersion:"v1", ResourceVersion:"14539", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
I0708 16:53:46.963132       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"nginx", UID:"d456d2d9-82ce-11e8-ac1a-e21a8d63bb7d", APIVersion:"v1", ResourceVersion:"14539", FieldPath:""}): type: 'Normal' reason: 'Type' NodePort -> LoadBalancer
E0708 16:53:47.768379       1 loadbalancers.go:269] error getting node addresses for host.example.com: could not get private ip: <nil>
E0708 16:53:47.768421       1 loadbalancers.go:269] error getting node addresses for host.example.com: could not get private ip: <nil>
I0708 16:53:59.166272       1 service_controller.go:636] Detected change in list of current cluster nodes. New node set: map[worker1:{} worker3:{}]
E0708 16:53:59.377640       1 loadbalancers.go:269] error getting node addresses for host.example.com: could not get private ip: <nil>
E0708 16:53:59.377865       1 loadbalancers.go:269] error getting node addresses forhost.example.com: could not get private ip: <nil>
I0708 16:54:53.863448       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"nginx", UID:"d456d2d9-82ce-11e8-ac1a-e21a8d63bb7d", APIVersion:"v1", ResourceVersion:"14539", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer
E0708 16:54:56.402307       1 service_controller.go:660] External error while updating load balancer: PUT https://api.digitalocean.com/v2/load_balancers/93b9c153-a39f-415c-8bd4-763ffac1e6e5: 422 (request "c66e6aab-3784-4edd-91ad-cbfc57705882") Load Balancer can't be updated while it processes previous actions.
I0708 16:54:56.402554       1 service_controller.go:644] Successfully updated 2 out of 3 load balancers to direct traffic to the updated set of nodes
I0708 16:54:56.402706       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"nginx", UID:"d456d2d9-82ce-11e8-ac1a-e21a8d63bb7d", APIVersion:"v1", ResourceVersion:"14539", FieldPath:""}): type: 'Warning' reason: 'LoadBalancerUpdateFailed' Error updating load balancer with new hosts map[worker1:{} worker3:{}]: PUT https://api.digitalocean.com/v2/load_balancers/93b9c153-a39f-415c-8bd4-763ffac1e6e5: 422 (request "c66e6aab-3784-4edd-91ad-cbfc57705882") Load Balancer can't be updated while it processes previous actions
E0708 16:56:37.022723       1 loadbalancers.go:269] error getting node addresses for host.example.com: could not get private ip: <nil>
E0708 16:56:37.022771       1 loadbalancers.go:269] error getting node addresses for host.example.com: could not get private ip: <nil>
I0708 16:56:38.566833       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"nginx", UID:"d456d2d9-82ce-11e8-ac1a-e21a8d63bb7d", APIVersion:"v1", ResourceVersion:"14539", FieldPath:""}): type: 'Normal' reason: 'UpdatedLoadBalancer' Updated load balancer with new hosts

host.example.com is a different droplet on the same account (there are a totalt of 5 droplets, 1 master, 3 workers and host.example.com)

Also, After running kubeadm join and started kubelet, I modified the kubelet config and restarted the service, I don't know if this is okay/enough to make do-ccm work properly?

[Richard87]

I added the droplets manually (int the web-interface on digitalocean), and it worked niceley, but today I noticed all the droplets had been removed from the loadbalancer...

[andrewsykim]

Hi, thanks for reporting this issue.

error getting node addresses for host.example.com: could not get private ip:

this is an error mentioned in a previous issue #85, there's a fairly simple fix for this which I can get around to later this week. However, it shouldn't be affectign your loadbalancer in any way (just a bad log message)

External error while updating load balancer: PUT https://api.digitalocean.com/v2/load_balancers/93b9c153-a39f-415c-8bd4-763ffac1e6e5: 422 (request "c66e6aab-3784-4edd-91ad-cbfc57705882") Load Balancer can't be updated while it processes previous actions.

This is the first time I'm seeing this error, will dig into it a bit more and get back to you.

[andrewsykim]

Do you have health checks enabled on your loadbalancer? Do you know if those health checks are all passing?

[Richard87]

Hi!

I didn't touch the loadbalancer at all, but it seems do-ccm didn't manage the droplets in my cluster, at the time they wasen't started with --cloud-provider external, se my other issue for more issues there ;)

So do-ccm didn't add any droplets att all to the loadbalancer, but it seems the rest of the config of the loadbalancer was correct (port/protocol). I assume if I figgure out the problem with cloud-provider it would fix this issue...

[andrewsykim]

Thanks for the details, I'll dig into this a bit more and get back to you.

[andrewsykim]

@Richard87 tried to dig into the issue a bit but it seems like you deleted your LoadBalancer, would it be possible for you to spin up a new one so we can debug it?

[Richard87]

I'm not running do-ccm now because of the other issue #111 , so I'm not sure if I can?

Do you want me to just start do-ccm while all the nodes are configured normally ( eg. without --cloud-provider external )?

[Richard87]

Hi!

That worked nicely :)

But, it would be nice if the loud-balancer would get a human redable name... (maybe from the loadbalance service+hash, then store digitaloceans loadbalancer id as an annotation?

[andrewsykim]

I agree, but unfortunately loadbalancer names are generated from upstream Kubernetes code (it's the UUID of the Service resource), see #102 for more details.