Slow iperf3 over Calico VXLAN network

[vmax]

Expected Behavior

Speed test results roughly match between Calico and direct connection

Current Behavior

Speed test over iperf3 is either abysmally slow (in Kbit/s range) or connectivity is non-existent

Possible Solution

To debug, I've tried establishing a direct VXLAN tunnel between two nodes with a different VNI and IP range and the speed matched that of the direct connection:

node1:

ip link add vxlan42 type vxlan id 42 dev eth0 dstport 5000
ip addr add 192.168.100.1/24 dev vxlan42
bridge fdb add 00:00:00:00:00:00 dst NODE2PUBLICIP dev vxlan42
ip link set vxlan42 up

node2:

ip link add vxlan42 type vxlan id 42 dev eno1 dstport 5000
ip addr add 192.168.100.2/24 dev vxlan42
bridge fdb add 00:00:00:00:00:00 dst NODE1PUBLICIP dev vxlan42
ip link set vxlan42 up

test results:

[  5] local 192.168.100.1 port 5201 connected to 192.168.100.2 port 60808
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  79.1 MBytes   663 Mbits/sec                  
[  5]   1.00-2.00   sec  90.6 MBytes   760 Mbits/sec                  
[  5]   2.00-3.00   sec  78.8 MBytes   661 Mbits/sec                  
[  5]   3.00-4.00   sec  71.4 MBytes   599 Mbits/sec                  
[  5]   4.00-5.00   sec  74.8 MBytes   627 Mbits/sec                  
[  5]   5.00-6.00   sec  77.1 MBytes   647 Mbits/sec                  
[  5]   6.00-7.00   sec  58.6 MBytes   492 Mbits/sec                  
[  5]   7.00-8.00   sec  61.2 MBytes   514 Mbits/sec                  
[  5]   8.00-9.00   sec  63.4 MBytes   532 Mbits/sec                  
[  5]   9.00-10.00  sec  51.1 MBytes   429 Mbits/sec                  
[  5]  10.00-10.01  sec   384 KBytes   258 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec   706 MBytes   592 Mbits/sec                  receiver

Steps to Reproduce (for bugs)

0. Two nodes, one have vxlan.calico address 10.233.107.0/32; other one has 10.233.123.0/32; they both are also accessible over public Internet
1. iperf3 -s on first node
2. iperf3 -c 10.233.107.0 on the second node:

Connecting to host 10.233.107.0, port 5201
[  5] local 10.233.123.0 port 45674 connected to 10.233.107.0 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   105 KBytes   860 Kbits/sec    2   13.7 KBytes       
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    0   13.7 KBytes       
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   13.7 KBytes       
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    0   13.7 KBytes       
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   13.7 KBytes       
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    6   4.10 KBytes       
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    0   4.10 KBytes       
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   4.10 KBytes       
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   4.10 KBytes       
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   4.10 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   105 KBytes  86.1 Kbits/sec    8             sender
[  5]   0.00-10.01  sec  4.10 KBytes  3.35 Kbits/sec                  receiver

3. iperf3 -c NODE1PUBLICIP from the second node:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  80.2 MBytes   672 Mbits/sec  6987    452 KBytes       
[  5]   1.00-2.00   sec  33.8 MBytes   283 Mbits/sec    0    508 KBytes       
[  5]   2.00-3.00   sec  38.8 MBytes   325 Mbits/sec    0    563 KBytes       
[  5]   3.00-4.00   sec  41.2 MBytes   346 Mbits/sec    0    617 KBytes       
[  5]   4.00-5.00   sec  46.2 MBytes   388 Mbits/sec    0    673 KBytes       
[  5]   5.00-6.00   sec  48.8 MBytes   409 Mbits/sec    0    725 KBytes       
[  5]   6.00-7.00   sec  52.5 MBytes   440 Mbits/sec    0    779 KBytes       
[  5]   7.00-8.00   sec  56.2 MBytes   472 Mbits/sec    0    834 KBytes       
[  5]   8.00-9.00   sec  61.2 MBytes   514 Mbits/sec    0    888 KBytes       
[  5]   9.00-10.00  sec  65.0 MBytes   545 Mbits/sec    0    945 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   524 MBytes   439 Mbits/sec  6987             sender
[  5]   0.00-10.01  sec   522 MBytes   437 Mbits/sec                  receiver

Context

Network connectivity between pods on some nodes in our cluster is disrupted. We couldn't quite isolate which nodes are affected; one hypothesis is that they are on Hetzner and have ARM architecture. Attached is the pcap of the failing test (a lot of TCP retransmissions there).
calico-issue.pcap.gz

Would be happy with any debugging tips!

Your Environment

• Calico version: v3.27.3
• Calico dataplane (iptables, windows etc.): iptables, VXLAN overlay
• Orchestrator version (e.g. kubernetes, mesos, rkt): k8s
• Operating System and version: Ubuntu 22.04.3 LTS
• Link to your project (optional):

[caseydavenport]

Some things to look into:

• Is fragmentation occurring? Might be an MTU issue - https://docs.tigera.io/calico/latest/networking/configuring/mtu
• Could be VXLAN checksum offload, which has caused problems in certain kernel versions. Disabling checksum offload on vxlan.calico could help?
• Similarly, there was an interaction with kube-proxy for some connections that was fixed in v3.29.1: Execute MASQ rules after kube-proxy to avoid conflict - reenable vxlan csum offload #9091