Live capture on libvirt p2p links

[ssasso]

Live packet capture, i.e. with tcpdump or wireshark, is essentials for troubleshooting stuff, but also for better learn all the different protocols.

When using libvirt p2p links, the packets are "raw encapsulated" inside UDP.

It's annoying to force wireshark to decapsulate "udp port X" as ethernet on every new capture, so I created a script to pass the right info to the wireshark command line (and to perform remote sniffing - I am using netlab remotely, but of course this can be run directly with a local wireshark instance) (-d "udp.port==10002,eth"):

#!/bin/bash

NETLAB_HOST=172.23.128.5
NODE_ID=$1
IF_INDEX=$2

HOST="127.1.$NODE_ID.$IF_INDEX"
PORT=$((10000 + $IF_INDEX))

echo "HOST IS $HOST"
echo "PORT IS $PORT"

ssh root@$NETLAB_HOST "tcpdump -l -U -i lo -w - 'host $HOST and udp port $PORT'" | ./wshark3.sh -d "udp.port==$PORT,eth" -k -i -                   

I think we can
1 - document this trick on the documentation
2 - in the future create a command wrapper through "netlab" command tool.

[ipspace]

Yeah, we could have "netlab capture" that would run tcpdump. Can tcpdump do decapsulation on the fly or is there something else we could add there to do the decapsulation? It would be nice for netlab capture to produce a clean stream of packets

[ssasso]

Unfortunately it's not a real decapsulation, but a "parse this udp as eth" - and tcpdump does not support it.

On a CLI the same can be shown with tshark, i.e. tshark -i lo -f "host 127.1.12.2 and udp port 10002" -d "udp.port==10002,eth" -

however, if you save the pcap file, it will have the additional "UDP" headers:

(and, if you open it later, you need to apply the "parsing filter" again).

OTOH there could be a workaround, implementing two "methods":

1. live capture will use tshark (or remote wireshark)
2. async captures, to simply save pcap files, can still use tshark, and then edit the pcap file with editcap to remove the initial IP+UDP headers.

I already tried to use a pipeline tcpdump | editcap and obtain a live capture without the udp part, but it's not working due to bufferings of editcap itself.

[ssasso]

(and if you are asking "how can they do live sniffing on GNS3"... they implemented an user space "bridge" that they always put in the middle in every link, with the capability to sniff in real time ;) https://github.com/GNS3/ubridge )

[ssasso]

Example with "offline" edit with editcap:

Before:

After:

editcap -D0 -C42 -L 01-before.pcap 02-after.pcap

captures.zip

[jbemmel]

We may be able to use eBPF filters to selectively monitor traffic and strip UDP headers if needed.
See https://blogs.oracle.com/linux/post/bpf-using-bpf-to-do-packet-transformation for some context

I have played a bit with eBPF here: https://github.com/jbemmel/srl-evpn-proxy/blob/main/src/static-vxlan-agent/filter-vxlan-arp.c
where I used it to capture ARP packets sent via VXLAN

For example, here https://android.googlesource.com/kernel/common/+/f88eb7c0d002/samples/bpf/xdp_tx_iptunnel_kern.c is an XDP example program that adds an IP header and tunnels packets. Removing an UDP header should be similar. CloudFlare has created xdpcap (https://github.com/cloudflare/xdpcap) which might help to capture the output of such an XDP program

[ipspace]

I would prefer to have a ready-made tool. It's already a nightmare supporting the existing bag of tools (and upstreams changing CLI parameters etc don't help) and users who follow the parts of the instructions they like while skipping other bits. Adding homegrown tools playing with eBPF is asking for trouble.

Obviously, if someone wants to create such a tool and package (and support) it separately from netlab, then I have absolutely no problems pointing users to it and wishing them good luck ;)