[BPF] failed to dump the conntrack table

[ioworker0]

version: v3.27.3
dataplane: eBPF

I modified the bpfMapSizeConntrack parameter in the global Felix configuration and then deleted a calico-node. After that, the following errors occurred:

...
2024-12-30 13:48:43.975 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:0, PolicyIdx:25, Iface:"cali3d68ded7758", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:27, Type:"workload", ToOrFrom:"from", HookLayout:hook.Layout{0:96, 2:97, 3:98, 4:99, 6:100, 7:101, 8:90, 10:91, 11:92, 12:93, 14:94, 15:95}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
2024-12-30 13:48:43.975 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:0, PolicyIdx:17, Iface:"cali681df24beb0", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:19, Type:"workload", ToOrFrom:"from", HookLayout:hook.Layout{0:96, 2:97, 3:98, 4:99, 6:100, 7:101, 8:90, 10:91, 11:92, 12:93, 14:94, 15:95}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
2024-12-30 13:48:43.975 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:0, PolicyIdx:29, Iface:"calicf78b17719a", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:31, Type:"workload", ToOrFrom:"from", HookLayout:hook.Layout{0:96, 2:97, 3:98, 4:99, 6:100, 7:101, 8:90, 10:91, 11:92, 12:93, 14:94, 15:95}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
2024-12-30 13:48:43.975 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:1, PolicyIdx:22, Iface:"cali3148282adac", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:21, Type:"workload", ToOrFrom:"to", HookLayout:hook.Layout{0:84, 2:85, 3:86, 4:87, 6:88, 7:89, 8:78, 10:79, 11:80, 12:81, 14:82, 15:83}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
2024-12-30 13:48:43.975 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:1, PolicyIdx:24, Iface:"cali3d68ded7758", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:26, Type:"workload", ToOrFrom:"to", HookLayout:hook.Layout{0:84, 2:85, 3:86, 4:87, 6:88, 7:89, 8:78, 10:79, 11:80, 12:81, 14:82, 15:83}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
2024-12-30 13:48:43.976 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:0, PolicyIdx:39, Iface:"cali1f715c523aa", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:36, Type:"workload", ToOrFrom:"from", HookLayout:hook.Layout{0:96, 2:97, 3:98, 4:99, 6:100, 7:101, 8:90, 10:91, 11:92, 12:93, 14:94, 15:95}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
2024-12-30 13:48:43.976 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:0, PolicyIdx:32, Iface:"cali52c6ce56514", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:34, Type:"workload", ToOrFrom:"from", HookLayout:hook.Layout{0:96, 2:97, 3:98, 4:99, 6:100, 7:101, 8:90, 10:91, 11:92, 12:93, 14:94, 15:95}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
2024-12-30 13:48:43.976 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:0, PolicyIdx:23, Iface:"cali3148282adac", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:20, Type:"workload", ToOrFrom:"from", HookLayout:hook.Layout{0:96, 2:97, 3:98, 4:99, 6:100, 7:101, 8:90, 10:91, 11:92, 12:93, 14:94, 15:95}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
2024-12-30 13:48:43.977 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:1, PolicyIdx:12, Iface:"cali9135c56a46c", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:14, Type:"workload", ToOrFrom:"to", HookLayout:hook.Layout{0:84, 2:85, 3:86, 4:87, 6:88, 7:89, 8:78, 10:79, 11:80, 12:81, 14:82, 15:83}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
2024-12-30 13:48:43.977 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:1, PolicyIdx:33, Iface:"cali52c6ce56514", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:35, Type:"workload", ToOrFrom:"to", HookLayout:hook.Layout{0:84, 2:85, 3:86, 4:87, 6:88, 7:89, 8:78, 10:79, 11:80, 12:81, 14:82, 15:83}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
2024-12-30 13:48:43.977 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:1, PolicyIdx:38, Iface:"cali1f715c523aa", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:37, Type:"workload", ToOrFrom:"to", HookLayout:hook.Layout{0:84, 2:85, 3:86, 4:87, 6:88, 7:89, 8:78, 10:79, 11:80, 12:81, 14:82, 15:83}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
2024-12-30 13:48:43.977 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:1, PolicyIdx:28, Iface:"calicf78b17719a", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:30, Type:"workload", ToOrFrom:"to", HookLayout:hook.Layout{0:84, 2:85, 3:86, 4:87, 6:88, 7:89, 8:78, 10:79, 11:80, 12:81, 14:82, 15:83}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
2024-12-30 13:48:43.979 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:0, PolicyIdx:13, Iface:"cali9135c56a46c", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:15, Type:"workload", ToOrFrom:"from", HookLayout:hook.Layout{0:96, 2:97, 3:98, 4:99, 6:100, 7:101, 8:90, 10:91, 11:92, 12:93, 14:94, 15:95}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc/globals/cali_v4_ct3': parameter mismatch
libbpf: map 'cali_v4_ct3': error reusing pinned map
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: map 'cali_v4_ct3': failed to create: Invalid argument(-22)
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
libbpf: failed to load object '/usr/lib/calico/bpf/tc_preamble.o'
2024-12-30 13:48:43.979 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:1, PolicyIdx:40, Iface:"cali0ecd3ac8233", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:42, Type:"workload", ToOrFrom:"to", HookLayout:hook.Layout{0:84, 2:85, 3:86, 4:87, 6:88, 7:89, 8:78, 10:79, 11:80, 12:81, 14:82, 15:83}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
2024-12-30 13:48:43.979 [WARNING][103] felix/attach.go 164: Failed to load program attachPoint=&tc.AttachPoint{AttachPoint:bpf.AttachPoint{Hook:0, PolicyIdx:41, Iface:"cali0ecd3ac8233", LogLevel:"debug"}, LogFilter:"", LogFilterIdx:43, Type:"workload", ToOrFrom:"from", HookLayout:hook.Layout{0:96, 2:97, 3:98, 4:99, 6:100, 7:101, 8:90, 10:91, 11:92, 12:93, 14:94, 15:95}, HostIP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0xa0, 0x5a, 0x61}, HostTunnelIP:net.IP(nil), IntfIP:net.IP{0xa9, 0xfe, 0x1, 0x1}, FIB:true, ToHostDrop:false, DSR:true, DSROptoutCIDRs:false, TunnelMTU:0x5dc, VXLANPort:0x12b5, WgPort:0x0, ExtToServiceConnmark:0x0, PSNATStart:0x4e20, PSNATEnd:0x752f, IPv6Enabled:false, RPFEnforceOption:0x2, NATin:0x0, NATout:0x0, UDPOnly:false}
...

These errors can be resolved by deleting the file /sys/fs/bpf/tc/globals/cali_v4_ct3. But, when I tried to dump the conntrack table, another error occurred: ErrIterationFinished during the first iteration.

// PinnedMap &{MapParameters:{PinDir: Type:hash KeySize:16 ValueSize:88 MaxEntries:100000 Name:cali_v4_ct Flags:1 Version:3 UpdatedByBPF:true} fdLoaded:false fd:0 oldfd:0 perCPU:false oldSize:0 UpgradeFn:0x1a07d20 GetMapParams:0x2aa3460 
func (b *PinnedMap) Iter(f IterCallback) error {
...
    // valueSize: 88 KeySize: 16 MaxEntries: 100000 MapFD: 8
    it, err := NewIterator(b.MapFD(), b.KeySize, valueSize, b.MaxEntries)
....
	var action IteratorAction
	for {
		k, v, err := it.Next()
...
		if err != nil {
			if err == ErrIterationFinished {
				return nil
			}
			return errors.Errorf("iterating the map failed: %s", err)
		}
...
}

[ioworker0]

cc @tomastigera