How does Cluster Collector work?

[perezjasonr]

Sorry but I couldnt find it in the docs.

When I do a bootstrap a k8s cluster, then run an in-cluster kubent pod using local kubectl

I get:

8:00PM INF Retrieved 115 resources from collector name=Cluster
8:00PM INF Retrieved 0 resources from collector name="Helm v2"
8:00PM WRN failed to discover supported resources for : groupVersion shouldn't be empty
8:00PM INF Retrieved 716 resources from collector name="Helm v3"
8:00PM INF Loaded ruleset name=custom.rego.tmpl
8:00PM INF Loaded ruleset name=deprecated-1-16.rego
8:00PM INF Loaded ruleset name=deprecated-1-22.rego
8:00PM INF Loaded ruleset name=deprecated-1-25.rego
8:00PM INF Loaded ruleset name=deprecated-1-26.rego
8:00PM INF Loaded ruleset name=deprecated-future.rego
...
...
deprecated output

the exact same cluster, but done in cicd (gitlab), where the gitlab runner has kubectl, and kubeconfig in the runner then does the exact same pod, I get:

8:45PM INF Retrieved 0 resources from collector name=Cluster
8:45PM INF Retrieved 0 resources from collector name="Helm v2"
8:45PM WRN failed to discover supported resources for : groupVersion shouldn't be empty
8:45PM INF Retrieved 759 resources from collector name="Helm v3"
8:45PM INF Loaded ruleset name=custom.rego.tmpl
8:45PM INF Loaded ruleset name=deprecated-1-16.rego
8:45PM INF Loaded ruleset name=deprecated-1-22.rego
8:45PM INF Loaded ruleset name=deprecated-1-25.rego
8:45PM INF Loaded ruleset name=deprecated-1-26.rego
8:45PM INF Loaded ruleset name=deprecated-future.rego

the cluster is in the same cloud platform, bootstrapped with the same method (its AKS).

I'm trying to figure out why its 0 from collector Cluster in the gitlab runner. The only difference in my mind is where kubectl calls are coming from. One is from a gitlab runner, one is just from my local workstation. but the gitlab runner basically is handed the same stuff, its essentially just a mini setup (kubectl and kubeconf) put in a runner... but then runs the same commands meaning, it creates the clusterrole, binding, and sa, runs the kubent pod. I would think from the perspective of in-cluster kubent job, everything would be the same.

[perezjasonr]

Also what is covered in the "cluster" collector? I couldn't find an explanation for this. Is this referring to kube components and maybe some other things?

[dark0dave]

Cluster collector calls the cluster to get k8s resources via the rest api. As to why its not working from the gitlab runner I am not sure. Its likely attempting to call the master control plane, and it might not have access.

[perezjasonr]

Thank you for the response, Is it different in scope from the helm releases (helm collector)? Or is there overlap?

[dark0dave]

Yes its different. Helm deploys to a different location, which is the reason both of those collectors are separate.

[perezjasonr]

Do you happen to know what kinds of things are covered in the cluster collector? just trying to get an idea of what I'm not getting if I accept not being able to reach the master via rest api

cni? kube components? network policies? just guessing

[dark0dave]

anything not deployed by helm, basically

[dark0dave]

I'd look here: https://github.com/doitintl/kube-no-trouble/blob/master/pkg/collector/cluster.go#L72

[dark0dave]

those are the resources which kubent looks for by default, we also support additional kinds, but you will need to provide a list via the -a command kubent to find those.

[dark0dave]

I find it unlikely that this is a kubent problem, so might close this one off if thats ok

[perezjasonr]

Ok thank you for the clarification. ALso thats a very helpful link
so idea is things like manually done deployments/pods, any non helm object it seems. But since a helm release might have a deployment for example or a statefulset there is a little overlap right? or for example CRDs are often in helm charts and thats looking at CRDs.

edit: actually pods arent there?

[perezjasonr]

Well the goal was to figure how what exactly cluster collector is doing (not necessarily saying its a kubent issue per se), to better troubleshoot this, if your certain it works by hitting master node rest api then yes it can be closed. Wondering if i can change to debug or something to see it complain about reaching something.

[dark0dave]

We merge all the collected data together, so it doesn't really matter if there is overlap. We don't scan for pods as pods are a super low level resource, in general its best practice not to create them manually.

[perezjasonr]

Ok it makes sense. I'm gonna try to run it with --log-level debug at least once before closing to see if we can confirm complaints to master node rest, at least in my mind that resolves the ticket idea of "how does it work (and why isn't it working in my scenario)"

[perezjasonr]

After running it with debug i found out the issue was rbac and it couldn't get the cluster resources due to their apiGroups.

[stepanstipl]

Glad you were able to figure this out @perezjasonr 👍, I have created an issue (#426) to consider logging this at a warning level - I think it might be more user-friendly and make possible permission issue immediately obvious.