Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't deploy calico-3.29.1 from China #9585

Open
huangzeqi opened this issue Dec 11, 2024 · 13 comments
Open

Can't deploy calico-3.29.1 from China #9585

huangzeqi opened this issue Dec 11, 2024 · 13 comments

Comments

@huangzeqi
Copy link

huangzeqi commented Dec 11, 2024

Here I download calico-3.29.1 to my first k8s master node. Use below command to setup first control plane node. And then use "kubectl create -f tigera-operator.yaml" and "kubectl create -f custom-resources.yaml"[cidr: 192.168.0.0/16 is adjusted to match k8s --pod-network-cid] to deploy calico. But it reports related pods are in "ContainerCreating" status but it keeps always. I think it's caused by "docker.io" is not accessible from China. So which files should I replace with the standby registry download site, such as from "quay.io". Actually I try pull related version images to local, but looks calico doesn't read it from local. So is there an official way to realize? I tried calico 3.28 and 3.29.1, both version failed in my side.

[root@k8sma manifests]# docker images | egrep -i "quay.io|^calico" | sort -rn
quay.io/tigera/operator                                           v1.34.0    01249e32d0f6   7 months ago    73.6MB
quay.io/calico/typha                                              v3.29.1    4cb3738506f5   2 weeks ago     72.2MB
quay.io/calico/typha                                              latest     364b090cfcbf   3 years ago     128MB
quay.io/calico/pod2daemon-flexvol                                 v3.29.1    2b7452b763ec   2 weeks ago     13.9MB
quay.io/calico/pod2daemon-flexvol                                 latest     0d3b19c2d4d5   3 years ago     21.3MB
quay.io/calico/node                                               v3.29.1    feb26d4585d6   2 weeks ago     397MB
quay.io/calico/node                                               latest     9b7965ed4504   3 years ago     214MB
quay.io/calico/kube-controllers                                   v3.29.1    6331715a2ae9   2 weeks ago     80.7MB
quay.io/calico/kube-controllers                                   latest     5235846386af   3 years ago     132MB
quay.io/calico/cni                                                v3.29.1    7dd6ea186aba   2 weeks ago     215MB
quay.io/calico/cni                                                latest     2c8aa43a8d6d   3 years ago     239MB
calico/typha                                                      v3.29.1    4cb3738506f5   2 weeks ago     72.2MB
calico/typha                                                      v3.28.0    a9372c0f51b5   7 months ago    71.1MB
calico/pod2daemon-flexvol                                         v3.29.1    2b7452b763ec   2 weeks ago     13.9MB
calico/pod2daemon-flexvol                                         v3.28.0    587b28ecfc62   7 months ago    13.4MB
calico/node                                                       v3.29.1    feb26d4585d6   2 weeks ago     397MB
calico/node                                                       v3.28.0    4e42b6f329bc   7 months ago    353MB
calico/kube-controllers                                           v3.29.1    6331715a2ae9   2 weeks ago     80.7MB
calico/kube-controllers                                           v3.28.0    428d92b02253   7 months ago    79.1MB
calico/flannel-migration-controller                               v3.28.0    c027e5a36c9a   7 months ago    128MB
calico/dikastes                                                   v3.28.0    7431b5d64b5e   7 months ago    41.9MB
calico/cni                                                        v3.29.1    7dd6ea186aba   2 weeks ago     215MB
calico/cni                                                        v3.28.0    107014d9f4c8   7 months ago    209MB
[root@k8sma manifests]# kubectl get pods --all-namespaces 
NAMESPACE          NAME                                READY   STATUS              RESTARTS   AGE
calico-apiserver   calico-apiserver-58b5784d5f-pt5qj   0/1     ContainerCreating   0          115s
calico-apiserver   calico-apiserver-58b5784d5f-slb5d   0/1     ContainerCreating   0          115s
kube-system        coredns-cb4864fb5-rmgcb             0/1     ContainerCreating   0          5m15s
kube-system        coredns-cb4864fb5-vdb94             0/1     ContainerCreating   0          5m15s
kube-system        etcd-k8sma                          1/1     Running             20         5m23s
kube-system        kube-apiserver-k8sma                1/1     Running             1          5m23s
kube-system        kube-controller-manager-k8sma       1/1     Running             1          5m23s
kube-system        kube-proxy-vrfrh                    1/1     Running             0          5m15s
kube-system        kube-scheduler-k8sma                1/1     Running             1          5m23s
tigera-operator    tigera-operator-7bc55997bb-pwzs2    1/1     Running             0          2m9s
[root@k8sma manifests]# kubectl describe pod calico-apiserver-58b5784d5f-pt5qj -n calico-apiserver 
Name:             calico-apiserver-58b5784d5f-pt5qj
Namespace:        calico-apiserver
Priority:         0
Service Account:  calico-apiserver
Node:             k8sma/192.168.31.111
Start Time:       Wed, 11 Dec 2024 12:54:32 +0800
Labels:           apiserver=true
                  app.kubernetes.io/name=calico-apiserver
                  k8s-app=calico-apiserver
                  pod-template-hash=58b5784d5f
Annotations:      tigera-operator.hash.operator.tigera.io/calico-apiserver-certs: d6f930fa1298a0636bb27ff0a68386460a41751f
Status:           Pending
IP:               
IPs:              <none>
Controlled By:    ReplicaSet/calico-apiserver-58b5784d5f
Containers:
  calico-apiserver:
    Container ID:    
    Image:           docker.io/calico/apiserver:v3.29.1
    Image ID:        
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Args:
      --secure-port=5443
      --tls-private-key-file=/calico-apiserver-certs/tls.key
      --tls-cert-file=/calico-apiserver-certs/tls.crt
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Readiness:      http-get https://:5443/readyz delay=0s timeout=5s period=60s #success=1 #failure=3
    Environment:
      DATASTORE_TYPE:           kubernetes
      KUBERNETES_SERVICE_HOST:  192.0.0.1
      KUBERNETES_SERVICE_PORT:  443
      MULTI_INTERFACE_MODE:     none
    Mounts:
      /calico-apiserver-certs from calico-apiserver-certs (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xgsxt (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   False 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  calico-apiserver-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  calico-apiserver-certs
    Optional:    false
  kube-api-access-xgsxt:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node-role.kubernetes.io/control-plane:NoSchedule
                             node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Normal   Scheduled               2m42s                default-scheduler  Successfully assigned calico-apiserver/calico-apiserver-58b5784d5f-pt5qj to k8sma
  Warning  FailedMount             2m42s                kubelet            MountVolume.SetUp failed for volume "calico-apiserver-certs" : secret "calico-apiserver-certs" not found
  Warning  FailedCreatePodSandBox  2m41s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a809f7394b9563b41f0367573e77ce4ccc4e12714909128999d8b2685d42d85f": plugin type="calico" failed (add): error getting ClusterInformation: Get "https://192.0.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
  Normal   SandboxChanged          4s (x13 over 2m40s)  kubelet            Pod sandbox changed, it will be killed and re-created.
kubeadm init --apiserver-advertise-address=192.168.31.111 --control-plane-endpoint "192.168.31.250:6553" --image-repository registry.aliyuncs.com/google_containers --upload-certs --service-cidr=192.1.0.0/12 --pod-network-cidr=192.2.0.0/16 --cri-socket=unix:///var/run/containerd/containerd.sock
[root@k8sma manifests]# kubectl create -f ./tigera-operator.yaml 
namespace/tigera-operator created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpfilters.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/tiers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/adminnetworkpolicies.policy.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created
serviceaccount/tigera-operator created
clusterrole.rbac.authorization.k8s.io/tigera-operator created
clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created
deployment.apps/tigera-operator created
[root@k8sma manifests]# kubectl create -f ./custom-resources.yaml 
installation.operator.tigera.io/default created
apiserver.operator.tigera.io/default created

@caseydavenport
Copy link
Member

Warning FailedCreatePodSandBox 2m41s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a809f7394b9563b41f0367573e77ce4ccc4e12714909128999d8b2685d42d85f": plugin type="calico" failed (add): error getting ClusterInformation: Get "https://192.0.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

This doesn't look like an image pull issue to me. I think there might be something wrong with the certificates you've used when creating your cluster?

@huangzeqi
Copy link
Author

My planned control plane cluster is based on a two master nodes k8s with keep alive and haproxy. I below is my first step to deploy the first control plane node. Then deploy calico with tigera-operator.yaml and custom-resources.yaml.

I confirm that from China we can't access docker.io related resources. So my idea is to replace docker.io with quay.io in all calico related yaml configration files. I download calico-3.29.1 tar ball and deploy based on its manifest folder. So is there a clear instruction that which patterns should be replaced, if I want to change image pull address?

I use "kubeadm init --apiserver-advertise-address=192.168.31.111 --control-plane-endpoint "192.168.31.250:6553" --image-repository registry.aliyuncs.com/google_containers --upload-certs --service-cidr=192.1.0.0/12 --pod-network-cidr=192.2.0.0/16 --cri-socket=unix:///var/run/containerd/containerd.sock" to create a the cluster on the first node.

@huangzeqi
Copy link
Author

截屏2024-12-12 09 00 36

@huangzeqi
Copy link
Author

From above screen copy, you can say I even pull v3.29.1 related images to local and make tag to link as docker.io. But in my local calico related pods deployment, it just can't hook with these images. That's why I'm considering to replace "docker.io" with "quay.io" directly in calico-3.29.1/manifests for the related yaml files. Question is which files should be replace and which patters should?

@caseydavenport
Copy link
Member

Does this doc on using a custom registry help at all? https://docs.tigera.io/calico/latest/operations/image-options/alternate-registry#concepts

You will need to tell the Calico operator to use quay.io/ images instead of docker.io.

You should just be able to replace all instances of docker.io with quay.io instead.

@huangzeqi
Copy link
Author

Thanks for the document, I refer to your shared KB and I prepared all resources in local registry docker hub. And I can deploy tigera-operator.yaml and custom-resources.yaml. From output it looks all pods are running. But when I check "calico-node-xxxx" pod events and it gives warning: "Warning Unhealthy 62s (x3 over 80s) kubelet Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused". When I use "calicoctl node status" and it shows "No IPv4 peers found". And actually I met similar issue when I tried a previous calico version based on release-v3.28.0. So somewhere in my local environment should be configured to fix the issue?

[root@k8sma manifests]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
calico-apiserver calico-apiserver-6b76fc7fc4-svpxf 1/1 Running 0 105s
calico-apiserver calico-apiserver-6b76fc7fc4-w2bdl 1/1 Running 0 105s
calico-system calico-kube-controllers-6bd9b6fb88-5p5rb 1/1 Running 0 84s
calico-system calico-node-249wg 1/1 Running 0 85s
calico-system calico-typha-76766d7fc4-zkg7z 1/1 Running 0 89s
calico-system csi-node-driver-7zrc6 2/2 Running 0 2m7s
kube-system coredns-cb4864fb5-48d5d 1/1 Running 0 10m
kube-system coredns-cb4864fb5-hwfvv 1/1 Running 0 10m
kube-system etcd-k8sma 1/1 Running 25 10m
kube-system kube-apiserver-k8sma 1/1 Running 6 10m
kube-system kube-controller-manager-k8sma 1/1 Running 6 10m
kube-system kube-proxy-wlx45 1/1 Running 0 10m
kube-system kube-scheduler-k8sma 1/1 Running 6 10m
tigera-operator tigera-operator-7f57f79bb6-llvbt 1/1 Running 0 2m16s

kubectl describe pod calico-node-249wg -n calico-system

......
......
Events:
Type Reason Age From Message


Normal Scheduled 93s default-scheduler Successfully assigned calico-system/calico-node-249wg to k8sma
Normal Pulled 92s kubelet Container image "docker.io/calico/pod2daemon-flexvol:v3.29.1" already present on machine
Normal Created 91s kubelet Created container flexvol-driver
Normal Started 91s kubelet Started container flexvol-driver
Normal Pulled 89s kubelet Container image "docker.io/calico/cni:v3.29.1" already present on machine
Normal Created 89s kubelet Created container install-cni
Normal Started 88s kubelet Started container install-cni
Normal Pulled 82s kubelet Container image "docker.io/calico/node:v3.29.1" already present on machine
Normal Created 81s kubelet Created container calico-node
Normal Started 81s kubelet Started container calico-node
Warning Unhealthy 62s (x3 over 80s) kubelet Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused

[root@k8sma manifests]# calicoctl node status
Calico process is running.

IPv4 BGP status
No IPv4 peers found.

IPv6 BGP status
No IPv6 peers found.

@caseydavenport
Copy link
Member

Warning Unhealthy 62s (x3 over 80s) kubelet Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused

This tends to suggest a problem with calico/node startup that is preventing BIRD from launching.

What does kubectl logs on that calico/node pod say?

@huangzeqi
Copy link
Author

huangzeqi commented Dec 14, 2024

Hello, the error likes below. And I tried to adjust calico plugin auto-detection with my ens.* network adapters, so I added below configurations in custom-resources.yaml. Then re-deploy calico plugin. But looks same error.

apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  # Configures Calico networking.
  calicoNetwork:
    ipPools:
    - name: default-ipv4-ippool
      blockSize: 26
      cidr: 192.2.0.0/16
      encapsulation: VXLANCrossSubnet
      natOutgoing: Enabled
      nodeSelector: all()
    nodeAddressAutodetectionV4:
      interface: ens.*



[root@k8sma ~]# kubectl get pods --all-namespaces 
NAMESPACE          NAME                                      READY   STATUS    RESTARTS      AGE
calico-apiserver   calico-apiserver-6c689fd8b8-965g4         1/1     Running   0             21m
calico-apiserver   calico-apiserver-6c689fd8b8-qb485         1/1     Running   1 (20m ago)   21m
calico-system      calico-kube-controllers-66ff99977-x59ng   1/1     Running   0             21m
calico-system      calico-node-frdd6                         1/1     Running   0             21m
calico-system      calico-typha-5fdcd5cd78-6djp4             1/1     Running   0             21m
calico-system      csi-node-driver-lh75w                     2/2     Running   0             21m
kube-system        coredns-cb4864fb5-nq9bf                   1/1     Running   0             21m
kube-system        coredns-cb4864fb5-x7l59                   1/1     Running   0             21m
kube-system        etcd-k8sma                                1/1     Running   26            21m
kube-system        kube-apiserver-k8sma                      1/1     Running   7             21m
kube-system        kube-controller-manager-k8sma             1/1     Running   7             21m
kube-system        kube-proxy-wkfgt                          1/1     Running   0             21m
kube-system        kube-scheduler-k8sma                      1/1     Running   7             21m
tigera-operator    tigera-operator-7f57f79bb6-w942x          1/1     Running   0             21m

[root@k8sma manifests]# kubectl logs -p calico-node-frdd6 -n calico-system --previous
Defaulted container "calico-node" out of: calico-node, flexvol-driver (init), install-cni (init)
Error from server (BadRequest): previous terminated container "calico-node" in pod "calico-node-frdd6" not found

@huangzeqi
Copy link
Author

huangzeqi commented Dec 14, 2024

Something more:

[root@k8sma ~]# kubectl get events --sort-by='.metadata.creationTimestamp' -n calico-system
LAST SEEN   TYPE      REASON                   OBJECT                                         MESSAGE
17m         Normal    NoPods                   poddisruptionbudget/calico-typha               No matching pods found
17m         Normal    ScalingReplicaSet        deployment/calico-typha                        Scaled up replica set calico-typha-5fdcd5cd78 to 1
17m         Normal    SuccessfulCreate         replicaset/calico-typha-5fdcd5cd78             Created pod: calico-typha-5fdcd5cd78-6djp4
17m         Normal    Scheduled                pod/calico-typha-5fdcd5cd78-6djp4              Successfully assigned calico-system/calico-typha-5fdcd5cd78-6djp4 to k8sma
17m         Normal    Scheduled                pod/calico-node-frdd6                          Successfully assigned calico-system/calico-node-frdd6 to k8sma
17m         Normal    Created                  pod/calico-typha-5fdcd5cd78-6djp4              Created container calico-typha
17m         Normal    Pulled                   pod/calico-typha-5fdcd5cd78-6djp4              Container image "docker.io/calico/typha:v3.29.1" already present on machine
17m         Normal    SuccessfulCreate         daemonset/calico-node                          Created pod: calico-node-frdd6
17m         Normal    SuccessfulCreate         daemonset/csi-node-driver                      Created pod: csi-node-driver-lh75w
17m         Normal    Scheduled                pod/csi-node-driver-lh75w                      Successfully assigned calico-system/csi-node-driver-lh75w to k8sma
17m         Normal    ScalingReplicaSet        deployment/calico-kube-controllers             Scaled up replica set calico-kube-controllers-66ff99977 to 1
17m         Normal    Pulled                   pod/calico-node-frdd6                          Container image "docker.io/calico/pod2daemon-flexvol:v3.29.1" already present on machine
17m         Normal    SuccessfulCreate         replicaset/calico-kube-controllers-66ff99977   Created pod: calico-kube-controllers-66ff99977-x59ng
17m         Normal    Scheduled                pod/calico-kube-controllers-66ff99977-x59ng    Successfully assigned calico-system/calico-kube-controllers-66ff99977-x59ng to k8sma
17m         Normal    Created                  pod/calico-node-frdd6                          Created container flexvol-driver
16m         Normal    SandboxChanged           pod/csi-node-driver-lh75w                      Pod sandbox changed, it will be killed and re-created.
16m         Normal    Created                  pod/calico-node-frdd6                          Created container install-cni
16m         Normal    Pulled                   pod/calico-node-frdd6                          Container image "docker.io/calico/cni:v3.29.1" already present on machine
16m         Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-66ff99977-x59ng    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b34090cc772ba8c28426ad6041125afb07e8bc100c06a57bb00cefd8371524e6": plugin type="calico" failed (add): error getting ClusterInformation: Get "https://192.0.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
17m         Normal    Started                  pod/calico-typha-5fdcd5cd78-6djp4              Started container calico-typha
17m         Warning   FailedCreatePodSandBox   pod/csi-node-driver-lh75w                      Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "47927514de99b8ca29933dd523c4439543609dc98788632209f44079f5b91b38": plugin type="calico" failed (add): error getting ClusterInformation: Get "https://192.0.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
17m         Normal    Started                  pod/calico-node-frdd6                          Started container flexvol-driver
16m         Normal    SandboxChanged           pod/calico-kube-controllers-66ff99977-x59ng    Pod sandbox changed, it will be killed and re-created.
16m         Normal    Started                  pod/calico-node-frdd6                          Started container install-cni
16m         Normal    Pulled                   pod/calico-node-frdd6                          Container image "docker.io/calico/node:v3.29.1" already present on machine
16m         Normal    Created                  pod/calico-node-frdd6                          Created container calico-node
16m         Normal    Started                  pod/calico-node-frdd6                          Started container calico-node
16m         Normal    Created                  pod/csi-node-driver-lh75w                      Created container calico-csi
16m         Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-66ff99977-x59ng    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6a4b77e5c545f8dd00026ffbb326240176ef00a50e11a470c2d2ac5bfd58bce7": plugin type="calico" failed (add): error adding host side routes for interface: cali20fa88199e2, error: route (Ifindex: 37, Dst: 192.2.33.68/32, Scope: link) already exists for an interface other than 'cali20fa88199e2': route (Ifindex: 22, Dst: 192.2.33.68/32, Scope: link, Iface: cali523cda9dfbf)
16m         Normal    Pulled                   pod/csi-node-driver-lh75w                      Container image "docker.io/calico/csi:v3.29.1" already present on machine
16m         Normal    Created                  pod/csi-node-driver-lh75w                      Created container csi-node-driver-registrar
16m         Normal    Pulled                   pod/csi-node-driver-lh75w                      Container image "docker.io/calico/node-driver-registrar:v3.29.1" already present on machine
16m         Normal    Started                  pod/csi-node-driver-lh75w                      Started container calico-csi
16m         Normal    Started                  pod/csi-node-driver-lh75w                      Started container csi-node-driver-registrar
16m         Warning   FailedCreatePodSandBox   pod/calico-kube-controllers-66ff99977-x59ng    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6b9934a055b8b22f5411758f8704bf0d70410a796e85ce1309affa89a8838832": plugin type="calico" failed (add): error adding host side routes for interface: cali20fa88199e2, error: route (Ifindex: 41, Dst: 192.2.33.72/32, Scope: link) already exists for an interface other than 'cali20fa88199e2': route (Ifindex: 28, Dst: 192.2.33.72/32, Scope: link, Iface: cali6a10f958c7b)
16m         Warning   Unhealthy                pod/calico-node-frdd6                          Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused
16m         Normal    Created                  pod/calico-kube-controllers-66ff99977-x59ng    Created container calico-kube-controllers
16m         Normal    Pulled                   pod/calico-kube-controllers-66ff99977-x59ng    Container image "docker.io/calico/kube-controllers:v3.29.1" already present on machine
16m         Normal    Started                  pod/calico-kube-controllers-66ff99977-x59ng    Started container calico-kube-controllers
16m         Warning   Unhealthy                pod/calico-kube-controllers-66ff99977-x59ng    Readiness probe failed: Failed to read status file /status/status.json: unexpected end of JSON input

@huangzeqi
Copy link
Author

[root@k8sma ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:18:37:71 brd ff:ff:ff:ff:ff:ff
    altname enp3s0
    inet 192.168.31.111/24 brd 192.168.31.255 scope global noprefixroute ens160
       valid_lft forever preferred_lft forever
    inet 192.168.31.250/32 scope global ens160
       valid_lft forever preferred_lft forever
3: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:18:37:7b brd ff:ff:ff:ff:ff:ff
    altname enp11s0
    inet 192.168.8.111/24 brd 192.168.8.255 scope global noprefixroute ens192
       valid_lft forever preferred_lft forever
4: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:18:37:85 brd ff:ff:ff:ff:ff:ff
    altname enp19s0
    inet 192.168.137.111/24 brd 192.168.137.255 scope global noprefixroute ens224
       valid_lft forever preferred_lft forever
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:2a:0f:60:05 brd ff:ff:ff:ff:ff:ff
7: veth7b5d3de@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 56:bf:9b:da:31:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
14: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 66:ae:15:71:28:d6 brd ff:ff:ff:ff:ff:ff
    inet 192.2.33.67/32 scope global vxlan.calico
       valid_lft forever preferred_lft forever
34: calia6a51746110@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-776f615b-9e93-8d16-0baa-05219779b5a4
35: calia12229c4c57@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-13199a3a-c00c-d8de-b239-9495a3f7d57e
38: cali6831d25b850@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-cbdb0fee-ff4d-6c03-789a-221f61f9f613
39: calie1d3997d3c4@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-6b11e0d9-3a68-28d8-8cdd-8a6fbd8f0313
40: cali6bbd494d2fd@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-bb0e7f36-43da-50aa-956c-dbacc076c480
44: cali20fa88199e2@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-4814f063-f65e-94ec-3f49-aa83cfb93edc

@huangzeqi
Copy link
Author

@caseydavenport
Copy link
Member

[root@k8sma manifests]# kubectl logs -p calico-node-frdd6 -n calico-system --previous
Defaulted container "calico-node" out of: calico-node, flexvol-driver (init), install-cni (init)
Error from server (BadRequest): previous terminated container "calico-node" in pod "calico-node-frdd6" not found

Could you run this without the --previous option so that we can see the current logs?

@BurlyLuo
Copy link

BurlyLuo commented Jan 4, 2025

Already test at china network environment. worked as expected

[root@rowan> calico-tmp]# kk exec -it ds/calico-node -- calico-node -v 
Defaulted container "calico-node" out of: calico-node, upgrade-ipam (init), install-cni (init), mount-bpffs (init)
v3.29.1
[root@rowan> calico-tmp]# all -o wide 
NAMESPACE            NAME                                                READY   STATUS    RESTARTS   AGE     IP              NODE                        NOMINATED NODE   READINESS GATES
kube-system          calico-kube-controllers-5b49947fc-6nw9h             1/1     Running   0          4m38s   10.244.51.196   calico-ipip-control-plane   <none>           <none>
kube-system          calico-node-gvg87                                   1/1     Running   0          4m38s   172.18.0.2      calico-ipip-control-plane   <none>           <none>
kube-system          calico-node-k6n6q                                   1/1     Running   0          4m38s   172.18.0.3      calico-ipip-worker          <none>           <none>
kube-system          coredns-5d78c9869d-9zvc6                            1/1     Running   0          5m49s   10.244.51.194   calico-ipip-control-plane   <none>           <none>
kube-system          coredns-5d78c9869d-l4b8q                            1/1     Running   0          5m49s   10.244.51.193   calico-ipip-control-plane   <none>           <none>
kube-system          etcd-calico-ipip-control-plane                      1/1     Running   0          6m3s    172.18.0.2      calico-ipip-control-plane   <none>           <none>
kube-system          kube-apiserver-calico-ipip-control-plane            1/1     Running   0          6m3s    172.18.0.2      calico-ipip-control-plane   <none>           <none>
kube-system          kube-controller-manager-calico-ipip-control-plane   1/1     Running   0          6m3s    172.18.0.2      calico-ipip-control-plane   <none>           <none>
kube-system          kube-proxy-ksjbb                                    1/1     Running   0          5m44s   172.18.0.3      calico-ipip-worker          <none>           <none>
kube-system          kube-proxy-xhg74                                    1/1     Running   0          5m49s   172.18.0.2      calico-ipip-control-plane   <none>           <none>
kube-system          kube-scheduler-calico-ipip-control-plane            1/1     Running   0          6m3s    172.18.0.2      calico-ipip-control-plane   <none>           <none>
local-path-storage   local-path-provisioner-6bc4bddd6b-62xz7             1/1     Running   0          5m49s   10.244.51.195   calico-ipip-control-plane   <none>           <none>
[root@rowan> calico-tmp]# 

BurlyLuo added a commit to BurlyLuo/wcni-kind that referenced this issue Jan 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants