VXLAN not working when tunnel address is borrowed

[JornShen]

we meet a problems which calico node can not reach other node's pod. And we use calico vxlan mode. And the problem node's vxlan device ip is borrowed from other node's ipamblock.

we ping other node's pods in this node, the packcet dont reply back to this problem node, because reply dst suit the route that isn't not its.

like that:
the problems node is :

root@nc81iogersfelltbdupg0:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:48:51:53 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.56/24 brd 192.168.0.255 scope global eth0
       valid_lft forever preferred_lft forever
6: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether 66:74:9e:0b:bd:11 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.75/32 scope global vxlan.calico
       valid_lft forever preferred_lft forever
7: cali8b0a20152a9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-98e7be4e-9420-983c-3afe-b54e50750d51
8: cali0145d32f180@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-1f0beb63-2e60-8275-2242-286b6d76ea54

vxlan device ip is 172.18.0.75

it route is:

root@nc81iogersfelltbdupg0:~# ip r
default via 192.168.0.1 dev eth0
172.18.0.0/26 via 172.18.0.0 dev vxlan.calico onlink
172.18.0.64/26 via 172.18.0.64 dev vxlan.calico onlink
172.18.0.76 dev cali8b0a20152a9 scope link
172.18.0.77 dev cali0145d32f180 scope link
172.18.0.192/26 via 172.18.0.192 dev vxlan.calico onlink
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.56

and the other node is

root@wttest-1-master-2:~# ip r
default via 192.168.0.1 dev eth0
blackhole 172.18.0.0/26 proto 80
172.18.0.1 dev cali865d3e7c719 scope link
172.18.0.2 dev cali67b4aa67d0e scope link
172.18.0.64/26 via 172.18.0.64 dev vxlan.calico onlink
172.18.0.76 via 172.18.0.75 dev vxlan.calico onlink
172.18.0.77 via 172.18.0.75 dev vxlan.calico onlink
172.18.0.192/26 via 172.18.0.192 dev vxlan.calico onlink
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.198

we can see in other node dont have route like 172.18.0.75 via 172.18.0.75 dev vxlan.calico onlink, and packet back error

I use calico latest verison, And it perform the same.

And I see the code in calico dont deal with the scene (that vxlan ip is borrow from others).
if the route is IPv4VXLANTunnelAddr, it dont add any route and all pass it.

the traffice is node -> pod(other node), and node's vxlan ip is borrowed from other node ipamblock.

[JornShen]

@caseydavenport

[JornShen]

@fasaxc

[JornShen]

Is it a bug for this ? @caseydavenport

[caseydavenport]

It does sound like a bug, although I'm not sure the root cause off the top of my head. I would have thought we'd be resilient to VXLAN tunnel addresses being borrowed, but perhaps not.

As a workaround, you might want to turn off IP borrowing by enabling IPAM strict affinity, or look into why IP addresses are being borrowed in the first place (it usually indicates that your IP pool configuration isn't ideal and you're running out of blocks, or that you have leaked some IP addresses)

[JornShen]

@caseydavenport This problems appear when my k8s nodes have use up all the ipamblock(all the ipamblock has been attached by existing node), Now I want to extend a node for k8s. And the new node doesn't have ipamblock and have to borrow other ipamblock（because IPAM strict affinity is close by default).

It is easy to reproduce this problems. you can make a try.

[caseydavenport]

Yep, that's the scenario in which borrowing occurs. My point was that we should be resilient to borrowing when it happens.

Do you know why you're using all of the IPAM blocks? Do you have an exceptionally large cluster or an exceptionally small IP pool?

[JornShen]

yeah, that would occurs. You mean when IPAM blocks is not enough, the cluster should be forbided to extend the node? @caseydavenport

[caseydavenport]

You mean when IPAM blocks is not enough, the cluster should be forbided to extend the node?

By default no, but there is an option to tell Calico to disable IP borrowing when it runs out of blocks.

[JornShen]

So which config you recommend @caseydavenport , shall we use IPAM strict affinity by default ? But if we use it, fixed ip feature maybe be affected..

[caseydavenport]

But if we use it, fixed ip feature maybe be affected..

Correct, if you're using features that require borrowing then strict affinity isn't an option.

And I see the code in calico dont deal with the scene (that vxlan ip is borrow from others). if the route is IPv4VXLANTunnelAddr, it dont add any route and all pass it.

Can you point me to the code snippet you were looking at?

[JornShen]

But if we use it, fixed ip feature maybe be affected..

Correct, if you're using features that require borrowing then strict affinity isn't an option.

And I see the code in calico dont deal with the scene (that vxlan ip is borrow from others). if the route is IPv4VXLANTunnelAddr, it dont add any route and all pass it.

Can you point me to the code snippet you were looking at?

maybe in vxlan_mgr.go, function OnUpdate.

[Davidrjx]

similar problem, vxlan.calico interface has UNKNOWN state and icmp can not reach it from another node

[fasaxc]

Looks like more discussion has happened on the duplicate bug #6160 so let's close this in favour of that one.