Hey 👋 CNI question

[rjshrjndrn]

Does KubeSolo support cilium? I really would like to run without iptables, and a little network visibility when needed will be nice.

[stevensbkang]

Hello @rjshrjndrn, since kubesolo doesn't rely on any Kubernetes specific CNIs, cilium should work.

Just gave it a try and it seems fine:

k get nodes
NAME                                          STATUS   ROLES    AGE     VERSION
qa-brm-5d89aefb0ef9-01.portainercloud.local   Ready    <none>   5m32s   v1.32.5-k3s1

cilium install --version 1.17.4
ℹ️  Using Cilium version 1.17.4
🔮 Auto-detected cluster name: kubesolo
🔮 Auto-detected kube-proxy has not been installed
ℹ️  Cilium will fully replace all functionalities of kube-proxy

cilium status --wait
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium                   Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet              cilium-envoy             Desired: 1, Ready: 1/1, Available: 1/1
Deployment             cilium-operator          Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium                   Running: 1
                       cilium-envoy             Running: 1
                       cilium-operator          Running: 1
                       clustermesh-apiserver    
                       hubble-relay             
Cluster Pods:          0/2 managed by Cilium
Helm chart version:    1.17.4
Image versions         cilium             quay.io/cilium/cilium:v1.17.4@sha256:24a73fe795351cf3279ac8e84918633000b52a9654ff73a6b0d7223bcff4a67a: 1
                       cilium-envoy       quay.io/cilium/cilium-envoy:v1.32.6-1746661844-0f602c28cb2aa57b29078195049fb257d5b5246c@sha256:a04218c6879007d60d96339a441c448565b6f86650358652da27582e0efbf182: 1
                       cilium-operator    quay.io/cilium/operator-generic:v1.17.4@sha256:a3906412f477b09904f46aac1bed28eb522bef7899ed7dd81c15f78b7aa1b9b5: 1

k get pods -A
NAMESPACE            NAME                                      READY   STATUS    RESTARTS   AGE
kube-system          cilium-envoy-zmfkc                        1/1     Running   0          5m25s
kube-system          cilium-gw99x                              1/1     Running   0          5m25s
kube-system          cilium-operator-57b894b47d-mwnjz          1/1     Running   0          5m24s
kube-system          coredns-846c8566cb-7xxdz                  1/1     Running   0          10m
local-path-storage   local-path-provisioner-657d8d4c7f-hz25m   1/1     Running   0          10m

[rjshrjndrn]

Hmm.. Are you sure the traffic s flowing through cilium?
Cluster Pods: 0/2 managed by Cilium

[stevensbkang]

Nope, as it is!

BTW, just looking at Cilium logs again, it seems to replace kube-proxy:

Cilium will fully replace all functionalities of kube-proxy

You may want to pass --set kubeProxyReplacement=false during the cilium installation since kube-proxy cannot be disabled when you start kubesolo today.

Please share how the testing goes, very keen to see the results 😄

[stevensbkang]

This is what I see now:

cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium                   Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet              cilium-envoy             Desired: 1, Ready: 1/1, Available: 1/1
Deployment             cilium-operator          Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium                   Running: 1
                       cilium-envoy             Running: 1
                       cilium-operator          Running: 1
                       clustermesh-apiserver    
                       hubble-relay             
Cluster Pods:          3/5 managed by Cilium
Helm chart version:    1.17.4
Image versions         cilium             quay.io/cilium/cilium:v1.17.4@sha256:24a73fe795351cf3279ac8e84918633000b52a9654ff73a6b0d7223bcff4a67a: 1
                       cilium-envoy       quay.io/cilium/cilium-envoy:v1.32.6-1746661844-0f602c28cb2aa57b29078195049fb257d5b5246c@sha256:a04218c6879007d60d96339a441c448565b6f86650358652da27582e0efbf182: 1
                       cilium-operator    quay.io/cilium/operator-generic:v1.17.4@sha256:a3906412f477b09904f46aac1bed28eb522bef7899ed7dd81c15f78b7aa1b9b5: 1

[rjshrjndrn]

Awesome. I'll give it a go. BTW, any specific reason kube-proxy shouldn't be replaced?

[stevensbkang]

By design. We built kubesolo for IoT/IIoT devices where a Kubernetes CNI such as Cilium or Flannel would be an overkill