Calico CNI. There is always few control plane nodes can not access the node ip/cluster ip. NodePort could all have access #9353

RogerTaylor130 · 2024-10-17T03:34:30Z

RogerTaylor130
Oct 17, 2024

Cluster Env:

3 master nodes, and one worker node. Stacked Etcd.

kubeadm init --pod-network-cidr 192.168.0.0/16 --control-plane-endpoint nlb-iiucdxea6hym5n9fax.cn-hongkong.nlb.aliyuncs.com --upload-certs

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.2/manifests/tigera-operator.yaml

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.2/manifests/custom-resources.yaml

[root@master02 ~]#  k get no -owide
NAME       STATUS   ROLES           AGE     VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                              KERNEL-VERSION             CONTAINER-RUNTIME
master01   Ready    control-plane   6m38s   v1.31.1   172.22.22.47   <none>        Alibaba Cloud Linux 3.2104 U10 (OpenAnolis Edition)   5.10.134-17.2.al8.x86_64   containerd://1.7.22
master02   Ready    control-plane   5m53s   v1.31.1   172.29.84.5    <none>        Alibaba Cloud Linux 3.2104 U10 (OpenAnolis Edition)   5.10.134-17.2.al8.x86_64   containerd://1.7.22
master03   Ready    control-plane   5m45s   v1.31.1   172.22.22.48   <none>        Alibaba Cloud Linux 3.2104 U10 (OpenAnolis Edition)   5.10.134-17.2.al8.x86_64   containerd://1.7.22
worker01   Ready    <none>          4m25s   v1.31.1   172.22.22.46   <none>        Alibaba Cloud Linux 3.2104 U10 (OpenAnolis Edition)   5.10.134-17.2.al8.x86_64   containerd://1.7.22

[root@master01 ~]# k run test --image nginx
pod/test created
[root@master01 ~]# k get po -owide
NAME   READY   STATUS    RESTARTS   AGE   IP            NODE       NOMINATED NODE   READINESS GATES
test   1/1     Running   0          10s   192.168.5.4   worker01   <none>           <none>


[root@master02 ~]# k get po -A -owide
NAMESPACE          NAME                                       READY   STATUS    RESTARTS   AGE     IP               NODE       NOMINATED NODE   READINESS GATES
calico-apiserver   calico-apiserver-6b8c7bb6c5-fw8gl          1/1     Running   0          3m16s   192.168.241.66   master01   <none>           <none>
calico-apiserver   calico-apiserver-6b8c7bb6c5-tqnvx          1/1     Running   0          3m16s   192.168.5.2      worker01   <none>           <none>
calico-system      calico-kube-controllers-649bf9bff6-w2g2w   1/1     Running   0          4m21s   192.168.235.2    master03   <none>           <none>
calico-system      calico-node-lf2vx                          1/1     Running   0          4m21s   172.22.22.46     worker01   <none>           <none>
calico-system      calico-node-m9g4x                          1/1     Running   0          4m21s   172.22.22.47     master01   <none>           <none>
calico-system      calico-node-mwftr                          1/1     Running   0          4m21s   172.22.22.48     master03   <none>           <none>
calico-system      calico-node-t98v8                          1/1     Running   0          4m21s   172.29.84.5      master02   <none>           <none>
calico-system      calico-typha-5d49cd87b-9wv5l               1/1     Running   0          4m21s   172.22.22.46     worker01   <none>           <none>
calico-system      calico-typha-5d49cd87b-b499m               1/1     Running   0          4m12s   172.22.22.47     master01   <none>           <none>
calico-system      csi-node-driver-gqjxf                      2/2     Running   0          4m21s   192.168.235.1    master03   <none>           <none>
calico-system      csi-node-driver-h58vc                      2/2     Running   0          4m21s   192.168.5.1      worker01   <none>           <none>
calico-system      csi-node-driver-lsms6                      2/2     Running   0          4m21s   192.168.241.65   master01   <none>           <none>
calico-system      csi-node-driver-szdhl                      2/2     Running   0          4m21s   192.168.59.193   master02   <none>           <none>
default            test                                       1/1     Running   0          86s     192.168.5.4      worker01   <none>           <none>
kube-system        coredns-7c65d6cfc9-9pkpf                   1/1     Running   0          6m51s   192.168.235.4    master03   <none>           <none>
kube-system        coredns-7c65d6cfc9-x5lp6                   1/1     Running   0          6m51s   192.168.235.3    master03   <none>           <none>
kube-system        etcd-master01                              1/1     Running   0          6m51s   172.22.22.47     master01   <none>           <none>
kube-system        etcd-master02                              1/1     Running   0          6m6s    172.29.84.5      master02   <none>           <none>
kube-system        etcd-master03                              1/1     Running   0          5m58s   172.22.22.48     master03   <none>           <none>
kube-system        kube-apiserver-master01                    1/1     Running   0          6m51s   172.22.22.47     master01   <none>           <none>
kube-system        kube-apiserver-master02                    1/1     Running   0          6m6s    172.29.84.5      master02   <none>           <none>
kube-system        kube-apiserver-master03                    1/1     Running   0          5m58s   172.22.22.48     master03   <none>           <none>
kube-system        kube-controller-manager-master01           1/1     Running   0          6m51s   172.22.22.47     master01   <none>           <none>
kube-system        kube-controller-manager-master02           1/1     Running   0          6m6s    172.29.84.5      master02   <none>           <none>
kube-system        kube-controller-manager-master03           1/1     Running   0          5m57s   172.22.22.48     master03   <none>           <none>
kube-system        kube-proxy-26v9w                           1/1     Running   0          4m40s   172.22.22.46     worker01   <none>           <none>
kube-system        kube-proxy-7gwb5                           1/1     Running   0          6m8s    172.29.84.5      master02   <none>           <none>
kube-system        kube-proxy-fxt5s                           1/1     Running   0          6m51s   172.22.22.47     master01   <none>           <none>
kube-system        kube-proxy-glqdb                           1/1     Running   0          6m      172.22.22.48     master03   <none>           <none>
kube-system        kube-scheduler-master01                    1/1     Running   0          6m51s   172.22.22.47     master01   <none>           <none>
kube-system        kube-scheduler-master02                    1/1     Running   0          6m6s    172.29.84.5      master02   <none>           <none>
kube-system        kube-scheduler-master03                    1/1     Running   0          5m50s   172.22.22.48     master03   <none>           <none>
tigera-operator    tigera-operator-89c775547-wzhtr            1/1     Running   0          4m30s   172.22.22.46     worker01   <none>           <none>

Network Access:
Control planes 2 can access the service via node IP & cluster IP & NodePort
Control plane 1 and 3 can not access the service via node IP & cluster IP & NodePort

[root@master01 ~]# k get po -owide
NAME   READY   STATUS    RESTARTS   AGE     IP            NODE       NOMINATED NODE   READINESS GATES
test   1/1     Running   0          3m17s   192.168.5.4   worker01   <none>           <none>


[root@master03 ~]# curl 192.168.5.4
curl: (7) Failed to connect to 192.168.5.4 port 80: Connection timed out
[root@master03 ~]# curl 192.168.5.4
curl: (7) Failed to connect to 192.168.5.4 port 80: Connection timed out

[root@master01 ~]# curl 192.168.5.4
curl: (7) Failed to connect to 192.168.5.4 port 80: Connection timed out
[root@master01 ~]# curl 192.168.5.4
curl: (7) Failed to connect to 192.168.5.4 port 80: Connection timed out

[root@master02 ~]# curl 192.168.5.4
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Pods On different Nodes:

[root@master02 ~]#  k get po -A -owide | grep master01
calico-apiserver   calico-apiserver-6b8c7bb6c5-fw8gl          1/1     Running   0          6m4s    192.168.241.66   master01   <none>           <none>
calico-system      calico-node-m9g4x                          1/1     Running   0          7m9s    172.22.22.47     master01   <none>           <none>
calico-system      calico-typha-5d49cd87b-b499m               1/1     Running   0          7m      172.22.22.47     master01   <none>           <none>
calico-system      csi-node-driver-lsms6                      2/2     Running   0          7m9s    192.168.241.65   master01   <none>           <none>
kube-system        etcd-master01                              1/1     Running   0          9m39s   172.22.22.47     master01   <none>           <none>
kube-system        kube-apiserver-master01                    1/1     Running   0          9m39s   172.22.22.47     master01   <none>           <none>
kube-system        kube-controller-manager-master01           1/1     Running   0          9m39s   172.22.22.47     master01   <none>           <none>
kube-system        kube-proxy-fxt5s                           1/1     Running   0          9m39s   172.22.22.47     master01   <none>           <none>
kube-system        kube-scheduler-master01                    1/1     Running   0          9m39s   172.22.22.47     master01   <none>           <none>



[root@master02 ~]# k get po -A -owide | grep master02
calico-system      calico-node-t98v8                          1/1     Running   0          7m22s   172.29.84.5      master02   <none>           <none>
calico-system      csi-node-driver-szdhl                      2/2     Running   0          7m22s   192.168.59.193   master02   <none>           <none>
kube-system        etcd-master02                              1/1     Running   0          9m7s    172.29.84.5      master02   <none>           <none>
kube-system        kube-apiserver-master02                    1/1     Running   0          9m7s    172.29.84.5      master02   <none>           <none>
kube-system        kube-controller-manager-master02           1/1     Running   0          9m7s    172.29.84.5      master02   <none>           <none>
kube-system        kube-proxy-7gwb5                           1/1     Running   0          9m9s    172.29.84.5      master02   <none>           <none>
kube-system        kube-scheduler-master02                    1/1     Running   0          9m7s    172.29.84.5      master02   <none>           <none>



[root@master02 ~]# k get po -A -owide | grep master03
calico-system      calico-kube-controllers-649bf9bff6-w2g2w   1/1     Running   0          7m36s   192.168.235.2    master03   <none>           <none>
calico-system      calico-node-mwftr                          1/1     Running   0          7m36s   172.22.22.48     master03   <none>           <none>
calico-system      csi-node-driver-gqjxf                      2/2     Running   0          7m36s   192.168.235.1    master03   <none>           <none>
kube-system        coredns-7c65d6cfc9-9pkpf                   1/1     Running   0          10m     192.168.235.4    master03   <none>           <none>
kube-system        coredns-7c65d6cfc9-x5lp6                   1/1     Running   0          10m     192.168.235.3    master03   <none>           <none>
kube-system        etcd-master03                              1/1     Running   0          9m13s   172.22.22.48     master03   <none>           <none>
kube-system        kube-apiserver-master03                    1/1     Running   0          9m13s   172.22.22.48     master03   <none>           <none>
kube-system        kube-controller-manager-master03           1/1     Running   0          9m12s   172.22.22.48     master03   <none>           <none>
kube-system        kube-proxy-glqdb                           1/1     Running   0          9m15s   172.22.22.48     master03   <none>           <none>
kube-system        kube-scheduler-master03                    1/1     Running   0          9m5s    172.22.22.48     master03   <none>           <none>


[root@master02 ~]# k get po -A -owide | grep worker01
calico-apiserver   calico-apiserver-6b8c7bb6c5-tqnvx          1/1     Running   0          6m47s   192.168.5.2      worker01   <none>           <none>
calico-system      calico-node-lf2vx                          1/1     Running   0          7m52s   172.22.22.46     worker01   <none>           <none>
calico-system      calico-typha-5d49cd87b-9wv5l               1/1     Running   0          7m52s   172.22.22.46     worker01   <none>           <none>
calico-system      csi-node-driver-h58vc                      2/2     Running   0          7m52s   192.168.5.1      worker01   <none>           <none>
default            test                                       1/1     Running   0          4m57s   192.168.5.4      worker01   <none>           <none>
kube-system        kube-proxy-26v9w                           1/1     Running   0          8m11s   172.22.22.46     worker01   <none>           <none>
tigera-operator    tigera-operator-89c775547-wzhtr            1/1     Running   0          8m1s    172.22.22.46     worker01   <none>           <none>

Answered by frozenprocess

Oct 17, 2024

Sounds like an encapsulation problem to me, try:

kubectl replace -f -<<EOF
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    ipPools:
    - name: default-ipv4-ippool
      blockSize: 26
      cidr: 192.168.0.0/16
      encapsulation: VXLAN
      natOutgoing: Enabled
      node selector: all()
EOF

To verify the change you can use kubectl get ippool -o yaml
vxlan should be set to "Always"

View full answer

RogerTaylor130 · 2024-10-17T03:39:29Z

RogerTaylor130
Oct 17, 2024
Author

IP Addr:

[root@master01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:16:3e:04:0d:9c brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname ens5
    inet 172.22.22.47/20 brd 172.22.31.255 scope global dynamic noprefixroute eth0
       valid_lft 315358536sec preferred_lft 315358536sec
    inet6 fe80::216:3eff:fe04:d9c/64 scope link
       valid_lft forever preferred_lft forever
3: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 66:58:bc:42:2d:24 brd ff:ff:ff:ff:ff:ff
    inet 192.168.241.64/32 scope global vxlan.calico
       valid_lft forever preferred_lft forever
    inet6 fe80::6458:bcff:fe42:2d24/64 scope link
       valid_lft forever preferred_lft forever
6: calibfa7be85083@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-eabad0bb-bed8-a4ba-9391-bb1d6a57bf0a
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
7: cali28600765078@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-3a7c6593-856a-206c-d4bc-273740fda615
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever


[root@master02 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:16:3e:08:cc:9a brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname ens5
    inet 172.29.84.5/20 brd 172.29.95.255 scope global dynamic noprefixroute eth0
       valid_lft 315358703sec preferred_lft 315358703sec
    inet6 fe80::216:3eff:fe08:cc9a/64 scope link
       valid_lft forever preferred_lft forever
3: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 66:9a:be:a5:63:87 brd ff:ff:ff:ff:ff:ff
    inet 192.168.59.192/32 scope global vxlan.calico
       valid_lft forever preferred_lft forever
    inet6 fe80::649a:beff:fea5:6387/64 scope link
       valid_lft forever preferred_lft forever
6: cali5cb13fcc9f4@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-a8b9af05-a0e9-97bf-c31e-d386850b07b7
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever



[root@master03 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:16:3e:00:19:7a brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname ens5
    inet 172.22.22.48/20 brd 172.22.31.255 scope global dynamic noprefixroute eth0
       valid_lft 315358576sec preferred_lft 315358576sec
    inet6 fe80::216:3eff:fe00:197a/64 scope link
       valid_lft forever preferred_lft forever
3: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 66:a0:18:04:a5:5a brd ff:ff:ff:ff:ff:ff
    inet 192.168.235.0/32 scope global vxlan.calico
       valid_lft forever preferred_lft forever
    inet6 fe80::64a0:18ff:fe04:a55a/64 scope link
       valid_lft forever preferred_lft forever
6: calic9dc5275731@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-6d1f157a-123d-d686-10ee-54e18faf6e07
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
7: cali32b70a31f69@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-57724e56-3621-7d1e-ef0a-549864014df6
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
8: calid8f2310fc04@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-50c8fabd-dd60-834c-6abf-e5d75c7d2400
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
9: cali5e899cf9300@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-1a5490ee-15c0-4d91-8f16-8f5d2a3a30c5
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever


[root@worker01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:16:3e:06:c0:d1 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname ens5
    inet 172.22.22.46/20 brd 172.22.31.255 scope global dynamic noprefixroute eth0
       valid_lft 315358403sec preferred_lft 315358403sec
    inet6 fe80::216:3eff:fe06:c0d1/64 scope link
       valid_lft forever preferred_lft forever
3: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 66:8c:33:86:44:ce brd ff:ff:ff:ff:ff:ff
    inet 192.168.5.0/32 scope global vxlan.calico
       valid_lft forever preferred_lft forever
    inet6 fe80::648c:33ff:fe86:44ce/64 scope link
       valid_lft forever preferred_lft forever
6: cali3357095d954@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-605dfa14-f90d-2f77-39b3-49695c252795
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
7: cali8665e9a6c8c@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-bcee0e2e-d54b-4eb7-f95c-21a1471eba4f
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
9: cali1037a54e65e@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-75bd25c3-054a-2168-aaee-baeddc69c215
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever

0 replies

RogerTaylor130 · 2024-10-17T03:43:40Z

RogerTaylor130
Oct 17, 2024
Author

Please let me know the information you want to help debug!

0 replies

RogerTaylor130 · 2024-10-17T03:45:52Z

RogerTaylor130
Oct 17, 2024
Author

Logs:

Two calico ApiSercer

[root@master01 ~]# k logs -f -n calico-apiserver calico-apiserver-6b8c7bb6c5-tqnvx
Version:      v3.28.2
Build date:   2024-09-18T17:49:30+0000
Git tag ref:  v3.28.2
Git commit:   9a96ee39f
I1017 03:27:37.557532       1 plugins.go:158] Loaded 2 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,MutatingAdmissionWebhook.
I1017 03:27:37.558153       1 plugins.go:161] Loaded 2 validating admission controller(s) successfully in the following order: ValidatingAdmissionPolicy,ValidatingAdmissionWebhook.
I1017 03:27:37.645623       1 handler.go:275] Adding GroupVersion projectcalico.org v3 to ResourceManager
I1017 03:27:37.645801       1 run_server.go:66] Running the API server
I1017 03:27:37.646650       1 run_server.go:55] Starting watch extension
W1017 03:27:37.737846       1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected.
W1017 03:27:37.737899       1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected.
I1017 03:27:37.744543       1 secure_serving.go:213] Serving securely on [::]:5443
I1017 03:27:37.744784       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I1017 03:27:37.744805       1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
I1017 03:27:37.744932       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/calico-apiserver-certs/tls.crt::/calico-apiserver-certs/tls.key"
I1017 03:27:37.745197       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I1017 03:27:37.746333       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I1017 03:27:37.746540       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1017 03:27:37.746687       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I1017 03:27:37.746698       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1017 03:27:37.845075       1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController
I1017 03:27:37.847455       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1017 03:27:37.847455       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file


[root@master02 ~]# k logs -f -n calico-apiserver calico-apiserver-6b8c7bb6c5-fw8gl
Version:      v3.28.2
Build date:   2024-09-18T17:49:30+0000
Git tag ref:  v3.28.2
Git commit:   9a96ee39f
I1017 03:27:35.853829       1 plugins.go:158] Loaded 2 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,MutatingAdmissionWebhook.
I1017 03:27:35.853896       1 plugins.go:161] Loaded 2 validating admission controller(s) successfully in the following order: ValidatingAdmissionPolicy,ValidatingAdmissionWebhook.
I1017 03:27:35.981861       1 handler.go:275] Adding GroupVersion projectcalico.org v3 to ResourceManager
I1017 03:27:35.981953       1 run_server.go:66] Running the API server
I1017 03:27:35.981964       1 run_server.go:55] Starting watch extension
W1017 03:27:36.052738       1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected.
W1017 03:27:36.052753       1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected.
I1017 03:27:36.058940       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I1017 03:27:36.058973       1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
I1017 03:27:36.059013       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I1017 03:27:36.059041       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1017 03:27:36.059063       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I1017 03:27:36.059070       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1017 03:27:36.059406       1 secure_serving.go:213] Serving securely on [::]:5443
I1017 03:27:36.059446       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/calico-apiserver-certs/tls.crt::/calico-apiserver-certs/tls.key"
I1017 03:27:36.059558       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I1017 03:27:36.159492       1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController
I1017 03:27:36.159516       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1017 03:27:36.159492       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file

controller

[root@master03 ~]# k logs -f -n calico-system calico-kube-controllers-649bf9bff6-w2g2w
2024-10-17 03:27:18.494 [INFO][1] main.go 98: Loaded configuration from environment config=&config.Config{LogLevel:"info", WorkloadEndpointWorkers:1, ProfileWorkers:1, PolicyWorkers:1, NodeWorkers:1, Kubeconfig:"", DatastoreType:"kubernetes"}
2024-10-17 03:27:18.496 [WARNING][1] winutils.go 150: Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2024-10-17 03:27:18.496 [INFO][1] main.go 122: Ensuring Calico datastore is initialized
2024-10-17 03:27:18.510 [INFO][1] main.go 147: Calico datastore is initialized
2024-10-17 03:27:18.510 [INFO][1] main.go 184: Getting initial config snapshot from datastore
2024-10-17 03:27:18.609 [INFO][1] main.go 187: Got initial config snapshot
2024-10-17 03:27:18.609 [INFO][1] watchersyncer.go 89: Start called
2024-10-17 03:27:18.609 [INFO][1] main.go 201: Starting status report routine
2024-10-17 03:27:18.609 [INFO][1] main.go 210: Starting Prometheus metrics server on port 9094
2024-10-17 03:27:18.609 [INFO][1] main.go 486: Starting informer informer=&cache.sharedIndexInformer{indexer:(*cache.cache)(0xc0002fd6e0), controller:cache.Controller(nil), processor:(*cache.sharedProcessor)(0xc0002feeb0), cacheMutationDetector:cache.dummyMutationDetector{}, listerWatcher:(*cache.ListWatch)(0xc0002fd6c8), objectType:(*v1.Pod)(0xc000436488), objectDescription:"", resyncCheckPeriod:0, defaultEventHandlerResyncPeriod:0, clock:(*clock.RealClock)(0x33e3260), started:false, stopped:false, startedLock:sync.Mutex{state:0, sema:0x0}, blockDeltas:sync.Mutex{state:0, sema:0x0}, watchErrorHandler:(cache.WatchErrorHandler)(nil), transform:(cache.TransformFunc)(nil)}
2024-10-17 03:27:18.609 [INFO][1] main.go 486: Starting informer informer=&cache.sharedIndexInformer{indexer:(*cache.cache)(0xc0002fd758), controller:cache.Controller(nil), processor:(*cache.sharedProcessor)(0xc0002fef00), cacheMutationDetector:cache.dummyMutationDetector{}, listerWatcher:(*cache.ListWatch)(0xc0002fd740), objectType:(*v1.Node)(0xc0002fa608), objectDescription:"", resyncCheckPeriod:0, defaultEventHandlerResyncPeriod:0, clock:(*clock.RealClock)(0x33e3260), started:false, stopped:false, startedLock:sync.Mutex{state:0, sema:0x0}, blockDeltas:sync.Mutex{state:0, sema:0x0}, watchErrorHandler:(cache.WatchErrorHandler)(nil), transform:(cache.TransformFunc)(nil)}
2024-10-17 03:27:18.609 [INFO][1] main.go 492: Starting controller ControllerType="Node"
2024-10-17 03:27:18.609 [INFO][1] controller.go 158: Starting Node controller
I1017 03:27:18.609904       1 shared_informer.go:311] Waiting for caches to sync for nodes
2024-10-17 03:27:18.609 [INFO][1] watchersyncer.go 130: Sending status update Status=wait-for-ready
2024-10-17 03:27:18.609 [INFO][1] syncer.go 86: Node controller syncer status updated: wait-for-ready
2024-10-17 03:27:18.609 [INFO][1] watchersyncer.go 149: Starting main event processing loop
2024-10-17 03:27:18.609 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/ippools"
2024-10-17 03:27:18.610 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/nodes"
2024-10-17 03:27:18.610 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/clusterinformations"
2024-10-17 03:27:18.611 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/ipam/v2/assignment/"
2024-10-17 03:27:18.611 [INFO][1] resources.go 350: Main client watcher loop
2024-10-17 03:27:18.621 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/ippools"
2024-10-17 03:27:18.621 [INFO][1] watchersyncer.go 130: Sending status update Status=resync
2024-10-17 03:27:18.621 [INFO][1] syncer.go 86: Node controller syncer status updated: resync
2024-10-17 03:27:18.621 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches
2024-10-17 03:27:18.622 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/clusterinformations"
2024-10-17 03:27:18.622 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches
2024-10-17 03:27:18.625 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/ipam/v2/assignment/"
2024-10-17 03:27:18.626 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches
2024-10-17 03:27:18.626 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/nodes"
2024-10-17 03:27:18.626 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches
2024-10-17 03:27:18.626 [INFO][1] watchersyncer.go 221: All watchers have sync'd data - sending data and final sync
2024-10-17 03:27:18.627 [INFO][1] watchersyncer.go 130: Sending status update Status=in-sync
2024-10-17 03:27:18.627 [INFO][1] syncer.go 86: Node controller syncer status updated: in-sync
I1017 03:27:18.710333       1 shared_informer.go:318] Caches are synced for nodes
I1017 03:27:18.710377       1 shared_informer.go:311] Waiting for caches to sync for pods
I1017 03:27:18.710383       1 shared_informer.go:318] Caches are synced for pods
2024-10-17 03:27:18.710 [INFO][1] ipam.go 251: Will run periodic IPAM sync every 7m30s
2024-10-17 03:27:18.710 [INFO][1] ipam.go 315: Syncer is InSync, kicking sync channel status=in-sync
2024-10-17 03:27:18.765 [INFO][1] hostendpoints.go 173: successfully synced all hostendpoints

0 replies

RogerTaylor130 · 2024-10-17T03:55:46Z

RogerTaylor130
Oct 17, 2024
Author

typha
calico-typha-5d49cd87b-9wv5l.log
calico-typha-5d49cd87b-b499m.log

0 replies

RogerTaylor130 · 2024-10-17T04:31:21Z

RogerTaylor130
Oct 17, 2024
Author

drivers

[root@master02 ~]# k logs -f -n calico-system csi-node-driver-lsms6
Defaulted container "calico-csi" out of: calico-csi, csi-node-driver-registrar
[root@master02 ~]# k logs -f -n calico-system csi-node-driver-gqjxf
Defaulted container "calico-csi" out of: calico-csi, csi-node-driver-registrar

[root@master02 ~]# k logs -f -n calico-system csi-node-driver-h58vc
Defaulted container "calico-csi" out of: calico-csi, csi-node-driver-registrar

[root@master02 ~]# k logs -f -n calico-system csi-node-driver-szdhl
Defaulted container "calico-csi" out of: calico-csi, csi-node-driver-registrar

And calico-node
calico-node-mwftr.log
calico-node-t98v8.log
calico-node-lf2vx.log
calico-node-m9g4x.log

0 replies

frozenprocess · 2024-10-17T20:59:22Z

frozenprocess
Oct 17, 2024
Maintainer

Sounds like an encapsulation problem to me, try:

kubectl replace -f -<<EOF
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    ipPools:
    - name: default-ipv4-ippool
      blockSize: 26
      cidr: 192.168.0.0/16
      encapsulation: VXLAN
      natOutgoing: Enabled
      node selector: all()
EOF

To verify the change you can use kubectl get ippool -o yaml
vxlan should be set to "Always"

2 replies

RogerTaylor130 Oct 28, 2024
Author

Sorry for the late reply... Changed from VXLANCrossSubnet to VXLAN resolved the problem perfectly. Thank you so much!

RogerTaylor130 Oct 31, 2024
Author

Hi, @frozenprocess. After reviewing the official documentation, I have a few questions regarding VXLANCrossSubnet and VXLAN. I hope you could offer some advice! 🌞

Here’s what I understand so far:

VXLAN and IPIP are both overlay network solutions.
VXLANCrossSubnet is a cross-subnet overlay network solution.

Whether it’s VXLAN, IPIP, or VXLANCrossSubnet, they are all types of overlay network solutions, but they differ in implementation and function details. For instance, the main difference between VXLANCrossSubnet and VXLAN is that the former does not encapsulate traffic within the same subnet, while the latter does.

Sounds like an encapsulation issue to me; try: encapsulation: VXLAN.

In my case, the master nodes also got an IP address in the range of CIDR definition. Why can’t I access the pod IP from all master nodes, given that VXLANCrossSubnet will encapsulate traffic if it’s in different subnets? Is it because the encapsulation only happens between pod and pod, not node and pod?

BR

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Project Calico

Calico CNI. There is always few control plane nodes can not access the node ip/cluster ip. NodePort could all have access #9353

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments 2 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Project Calico

Calico CNI. There is always few control plane nodes can not access the node ip/cluster ip. NodePort could all have access #9353

RogerTaylor130 Oct 17, 2024

Replies: 6 comments · 2 replies

RogerTaylor130 Oct 17, 2024 Author

RogerTaylor130 Oct 17, 2024 Author

RogerTaylor130 Oct 17, 2024 Author

RogerTaylor130 Oct 17, 2024 Author

RogerTaylor130 Oct 17, 2024 Author

frozenprocess Oct 17, 2024 Maintainer

RogerTaylor130 Oct 28, 2024 Author

RogerTaylor130 Oct 31, 2024 Author

RogerTaylor130
Oct 17, 2024

Replies: 6 comments 2 replies

RogerTaylor130
Oct 17, 2024
Author

RogerTaylor130
Oct 17, 2024
Author

RogerTaylor130
Oct 17, 2024
Author

RogerTaylor130
Oct 17, 2024
Author

RogerTaylor130
Oct 17, 2024
Author

frozenprocess
Oct 17, 2024
Maintainer

RogerTaylor130 Oct 28, 2024
Author

RogerTaylor130 Oct 31, 2024
Author