Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

can't populate calico related pods in the second node in a high-availability k8s control-plane nodes #9646

Open
huanghaiqing1 opened this issue Dec 26, 2024 · 2 comments
Open

can't populate calico related pods in the second node in a high-availability k8s control-plane nodes #9646

huanghaiqing1 opened this issue Dec 26, 2024 · 2 comments
Labels
kind/support

Comments

@huanghaiqing1
Copy link

huanghaiqing1 commented Dec 26, 2024 •
edited by caseydavenport
Loading

Here I setup a high-availability k8s environment for POC. k8sma and k8smb are both master nodes. k8sma also plays a role as insecure-registries hub in local. k8s HA-clutster is based on keepalived and haproxy. I unzip tar-ball of calico-3.29.1 from gitlab and replace all images location with my local insecure-registries hub's address for yaml files under: /calico-3.29.1/manifests. And deploy tigera-operator.yaml and custom-resources.yaml on the first node with no problems[all are in running status]. When deploy k8s second master node, calico pods such as calico-node-w4tfl and csi-node-driver-2xmqm are in pending status. Related events show they are still trying to pull image "docker.io/calico/pod2daemon-flexvol:v3.29.1". But these calico pods are created automatically by k8s and populate from first master node, how to adjust their image path settings from k8s? Both master nodes' /etc/docker/daemon.json have registered insecure-registries: k8sma:5000[my local docker hub is baesd on it. It's the first k8s master node]. No firewalld acl set within all k8s nodes. Attached zip file is /calico-3.29.1/manifests, which is adjusted for my POC environment including configure calico related images download from local.

Warning Failed 33m kubelet Failed to pull image "docker.io/calico/pod2daemon-flexvol:v3.29.1": rpc error: code = DeadlineExceeded desc = failed to pull and unpack image "docker.io/calico/pod2daemon-flexvol:v3.29.1": failed to resolve reference "docker.io/calico/pod2daemon-flexvol:v3.29.1": failed to do request: Head "https://registry-1.docker.io/v2/calico/pod2daemon-flexvol/manifests/v3.29.1": dial tcp 69.63.186.31:443: i/o timeout

[root@k8sma ~]# docker inspect k8sma:5000/calico/pod2daemon-flexvol:v3.29.1
[
    {
        "Id": "sha256:2b7452b763ec8833ca0386ada5fd066e552a9b3b02b8538a5e34cc3d6d3840a6",
        "RepoTags": [
            "calico/pod2daemon-flexvol:v3.29.1",
            "k8sma:5000/calico/pod2daemon-flexvol:v3.29.1",
            "localhost:5000/calico/pod2daemon-flexvol:v3.29.1",
            "quay.io/calico/pod2daemon-flexvol:v3.29.1"
        ],
        "RepoDigests": [
            "calico/pod2daemon-flexvol@sha256:b39582ca0014605e8887a70e2f2fb103c8149d28d1aa6482fb5589e3c7c06197",
            "localhost:5000/calico/pod2daemon-flexvol@sha256:12dc82d6954594600519bc68031b009a56f971f11bc75e53c71fbb5071620e29",
            "quay.io/calico/pod2daemon-flexvol@sha256:b39582ca0014605e8887a70e2f2fb103c8149d28d1aa6482fb5589e3c7c06197"
        ],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2024-11-20T21:53:00.953460504Z",
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": null,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": [
                "/usr/local/bin/flexvol.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "description": "Calico FlexVolume driver installer to setup secure connections from Kubernetes pods to local daemons",
                "maintainer": "[email protected]",
                "name": "Calico FlexVolume driver installer",
                "release": "1",
                "summary": "Calico FlexVolume driver installer to setup secure connections from Kubernetes pods to local daemons",
                "vendor": "Project Calico",
                "version": "v3.29.1"
            }
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 13897586,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/805d09b4eac1310f3f8c943fd91da0be26c88ade70df33949da3fc7fa9d64db0/diff",
                "MergedDir": "/var/lib/docker/overlay2/6df1c55802e46b3cb5bf3f3a74c619c3c30d680c09061dc69b8e12ff9a9349ca/merged",
                "UpperDir": "/var/lib/docker/overlay2/6df1c55802e46b3cb5bf3f3a74c619c3c30d680c09061dc69b8e12ff9a9349ca/diff",
                "WorkDir": "/var/lib/docker/overlay2/6df1c55802e46b3cb5bf3f3a74c619c3c30d680c09061dc69b8e12ff9a9349ca/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:da4037918328aa59c3b346fb1ebedc50f04e7200ef4e164da39942fe02326441",
                "sha256:c7f5b2a14d583fdd3056d497ad65725cdb5f1b598be9ba5aa6f3ff1d1f65f136"
            ]
        },
        "Metadata": {
            "LastTagTime": "2024-12-24T13:52:20.284142669+08:00"
        }
    }
]

[root@k8sma ~]# kubectl get pods --all-namespaces
NAMESPACE          NAME                                       READY   STATUS              RESTARTS       AGE
calico-apiserver   calico-apiserver-75dcb6c777-6zdrl          1/1     Running             5 (63m ago)    42h
calico-apiserver   calico-apiserver-75dcb6c777-df6w5          1/1     Running             4 (63m ago)    42h
calico-system      calico-kube-controllers-66c6bf9fb8-fjwb8   1/1     Running             2 (63m ago)    40h
calico-system      calico-node-6r7cm                          1/1     Running             3 (63m ago)    42h
calico-system      calico-node-w4tfl                          0/1     Init:ErrImagePull   0              40h
calico-system      calico-typha-6d4dd75bc9-b69ln              1/1     Running             2 (63m ago)    40h
calico-system      csi-node-driver-2xmqm                      0/2     ContainerCreating   0              42h
calico-system      csi-node-driver-wvz52                      2/2     Running             6 (63m ago)    42h
kube-system        coredns-cb4864fb5-j6ksd                    1/1     Running             3 (63m ago)    43h
kube-system        coredns-cb4864fb5-whrkp                    1/1     Running             3 (63m ago)    43h
kube-system        etcd-k8sma                                 1/1     Running             63 (63m ago)   43h
kube-system        etcd-k8smb                                 1/1     Running             5 (63m ago)    42h
kube-system        kube-apiserver-k8sma                       1/1     Running             23 (63m ago)   43h
kube-system        kube-apiserver-k8smb                       1/1     Running             7 (63m ago)    42h
kube-system        kube-controller-manager-k8sma              1/1     Running             24 (63m ago)   43h
kube-system        kube-controller-manager-k8smb              1/1     Running             6 (63m ago)    42h
kube-system        kube-proxy-5nkch                           1/1     Running             3 (63m ago)    42h
kube-system        kube-proxy-xqf2v                           1/1     Running             5 (63m ago)    43h
kube-system        kube-scheduler-k8sma                       1/1     Running             24 (63m ago)   43h
kube-system        kube-scheduler-k8smb                       1/1     Running             6 (63m ago)    42h
tigera-operator    tigera-operator-5cbff6c885-5q8xh           1/1     Running             3 (63m ago)    42h

[root@k8sma ~]# calicoctl node status
Calico process is running.

IPv4 BGP status
No IPv4 peers found.

IPv6 BGP status
No IPv6 peers found.

[root@k8sma ~]# kubectl get nodes -o wide
NAME    STATUS     ROLES           AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE          KERNEL-VERSION              CONTAINER-RUNTIME
k8sma   Ready      control-plane   43h   v1.30.6   192.168.31.111   <none>        CentOS Stream 8   4.18.0-553.6.1.el8.x86_64   containerd://1.6.32
k8smb   NotReady   control-plane   42h   v1.30.6   192.168.31.112   <none>        CentOS Stream 8   4.18.0-553.6.1.el8.x86_64   containerd://1.6.32

[root@k8sma ~]# docker images | grep -i calico | grep -i "v3.29.1" | sort
calico/apiserver                                                  v3.29.1    421726ace5ed   5 weeks ago     101MB
calico/cni                                                        v3.29.1    7dd6ea186aba   5 weeks ago     215MB
calico/csi                                                        v3.29.1    bda8c42e0475   5 weeks ago     18.9MB
calico/ctl                                                        v3.29.1    93e62d075104   5 weeks ago     72.3MB
calico/dikastes                                                   v3.29.1    0027f49eb041   5 weeks ago     53.1MB
calico/kube-controllers                                           v3.29.1    6331715a2ae9   5 weeks ago     80.7MB
calico/node-driver-registrar                                      v3.29.1    8b7d18f262d5   5 weeks ago     24.3MB
calico/node                                                       v3.29.1    feb26d4585d6   5 weeks ago     397MB
calico/pod2daemon-flexvol                                         v3.29.1    2b7452b763ec   5 weeks ago     13.9MB
calico/typha                                                      v3.29.1    4cb3738506f5   5 weeks ago     72.2MB
k8sma:5000/calico/apiserver                                       v3.29.1    421726ace5ed   5 weeks ago     101MB
k8sma:5000/calico/cni                                             v3.29.1    7dd6ea186aba   5 weeks ago     215MB
k8sma:5000/calico/csi                                             v3.29.1    bda8c42e0475   5 weeks ago     18.9MB
k8sma:5000/calico/ctl                                             v3.29.1    93e62d075104   5 weeks ago     72.3MB
k8sma:5000/calico/dikastes                                        v3.29.1    0027f49eb041   5 weeks ago     53.1MB
k8sma:5000/calico/kube-controllers                                v3.29.1    6331715a2ae9   5 weeks ago     80.7MB
k8sma:5000/calico/node-driver-registrar                           v3.29.1    8b7d18f262d5   5 weeks ago     24.3MB
k8sma:5000/calico/node                                            v3.29.1    feb26d4585d6   5 weeks ago     397MB
k8sma:5000/calico/pod2daemon-flexvol                              v3.29.1    2b7452b763ec   5 weeks ago     13.9MB
k8sma:5000/calico/typha                                           v3.29.1    0027f49eb041   5 weeks ago     53.1MB
localhost:5000/calico/apiserver                                   v3.29.1    421726ace5ed   5 weeks ago     101MB
localhost:5000/calico/cni                                         v3.29.1    7dd6ea186aba   5 weeks ago     215MB
localhost:5000/calico/csi                                         v3.29.1    bda8c42e0475   5 weeks ago     18.9MB
localhost:5000/calico/ctl                                         v3.29.1    93e62d075104   5 weeks ago     72.3MB
localhost:5000/calico/dikastes                                    v3.29.1    0027f49eb041   5 weeks ago     53.1MB
localhost:5000/calico/kube-controllers                            v3.29.1    6331715a2ae9   5 weeks ago     80.7MB
localhost:5000/calico/node-driver-registrar                       v3.29.1    8b7d18f262d5   5 weeks ago     24.3MB
localhost:5000/calico/node                                        v3.29.1    feb26d4585d6   5 weeks ago     397MB
localhost:5000/calico/pod2daemon-flexvol                          v3.29.1    2b7452b763ec   5 weeks ago     13.9MB
localhost:5000/calico/typha                                       v3.29.1    4cb3738506f5   5 weeks ago     72.2MB
quay.io/calico/cni                                                v3.29.1    7dd6ea186aba   5 weeks ago     215MB
quay.io/calico/kube-controllers                                   v3.29.1    6331715a2ae9   5 weeks ago     80.7MB
quay.io/calico/node                                               v3.29.1    feb26d4585d6   5 weeks ago     397MB
quay.io/calico/pod2daemon-flexvol                                 v3.29.1    2b7452b763ec   5 weeks ago     13.9MB
quay.io/calico/typha                                              v3.29.1    4cb3738506f5   5 weeks ago     72.2MB
@huanghaiqing1
Copy link
Author

huanghaiqing1 commented Dec 26, 2024 •
edited
Loading

manifests.zip

@caseydavenport
Copy link
Member

Warning Failed 33m kubelet Failed to pull image "docker.io/calico/pod2daemon-flexvol:v3.29.1": rpc error: code = DeadlineExceeded desc = failed to pull and unpack image "docker.io/calico/pod2daemon-flexvol:v3.29.1": failed to resolve reference "docker.io/calico/pod2daemon-flexvol:v3.29.1": failed to do request: Head "https://registry-1.docker.io/v2/calico/pod2daemon-flexvol/manifests/v3.29.1": dial tcp 69.63.186.31:443: i/o timeout

It sounds like your container runtime is attempting to pull the images from DockerHub, and isn't able to. Shouldn't it be trying to pull from your local registry?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/support
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@caseydavenport @huanghaiqing1