Support capturing via fentry / fexit trace points #51

arthurfabre · 2021-04-06T09:08:39Z

Linux now supports attaching BPF programs just before (fentry) and after (fexit) an XDP program.
This allows packets to be captured before and after an XDP program runs, from unmodified XDP programs (no "hook" map).

xdpdump already supports this, but without filtering: https://github.com/xdp-project/xdp-tools/tree/master/xdp-dump/.

The text was updated successfully, but these errors were encountered:

wenlxie · 2023-09-19T14:18:19Z

I did a try to do implementation for this.
Here are the steps:

Get the prog info from id by NewProgramFromID()
The id is the xdp program's prog id
p, err := NewProgramFromID(id)
Then I set the program spec (which used to load the program)to be:

 	progSpec.Name = "fentry"
	progSpec.AttachTarget =  xdp program get by NewProgramFromID(id)
	progSpec.AttachTo = xdp program's name
	progSpec.AttachType = ebpf.AttachTraceFEntry

The key part is for how to generate the instructions for the progSpec.Instructions
The previous asm code are generated for xdp programs with xdp_md as args.
But for fentry to hook xdp programs, the input arg had been changed to xdp_buff, which means the asm code need to be changed.
At least that this need to be changed:

xdpcap/cmd/xdpcap/program.go

Lines 96 to 99 in 417804f

    
           asm.LoadMem(asm.R0, asm.R6, 0, asm.Word), 
        
           // Packet end 
        
           asm.LoadMem(asm.R1, asm.R6, 4, asm.Word),

I tried to change the asm code to make it works for xdp_buff, but got stucked.

I changed to asm code for the data and data_end to be :

		// Packet start
		asm.LoadMem(asm.R9, asm.R6, 0, asm.DWord),
		asm.LoadMem(asm.R0, asm.R9, 0, asm.DWord),

		// Packet end
		asm.LoadMem(asm.R1, asm.R9, 8, asm.DWord),

But I got error

permission denied: 21: (69) r2 = *(u16 *)(r0 +12): R0 invalid mem access 'inv' (26 line(s) omitted)

@arthurfabre Any thoughts for this issue? Could you help for how to make these asm codes works for xdp_buff? Thanks
Looks like fentry has more strict verify than xdp

Here are the asm instructions:

 0: MovReg dst: r6 src: r1
	  1: LoadMapPtr dst: r1 fd: 23
	  3: MovReg dst: r2 src: rfp
	  4: AddImm dst: r2 imm: -4
	  5: StMemW dst: r2 src: r0 off: 0 imm: 0
	  6: Call FnMapLookupElem
	  7: JEqImm dst: r0 off: -1 imm: 0 <exit>
	  8: MovReg dst: r7 src: r0
	  9: LdXMemDW dst: r9 src: r6 off: 0 imm: 0
	 10: LdXMemDW dst: r0 src: r9 off: 0 imm: 0
	 11: LdXMemDW dst: r1 src: r9 off: 8 imm: 0
	 12: MovReg dst: r8 src: r1
	 13: SubReg dst: r8 src: r0
	 14: JLEImm dst: r8 off: -1 imm: 12 <exit>
	 15: LdXMemDW dst: r2 src: r7 off: 0 imm: 0
	 16: AddImm dst: r2 imm: 1
	 17: StXMemDW dst: r7 src: r2 off: 0 imm: 0
filter_block_0:
	 18: MovReg dst: r4 src: r0
	 19: AddImm dst: r4 imm: 21
	 20: JGTReg dst: r4 off: -1 src: r1 <filter_nomatch>
	 21: LdXMemH dst: r2 src: r0 off: 12 imm: 0
	 22: SwapBE dst: r2 imm: 16
	 23: JNEImm dst: r2 off: -1 imm: 34525 <filter_block_7>
filter_block_2:
	 24: LdXMemB dst: r2 src: r0 off: 20 imm: 0
	 25: JEqImm dst: r2 off: -1 imm: 6 <filter_block_10>
filter_block_4:
	 26: JNEImm dst: r2 off: -1 imm: 44 <filter_block_11>
filter_block_5:
	 27: MovReg dst: r4 src: r0
	 28: AddImm dst: r4 imm: 55
	 29: JGTReg dst: r4 off: -1 src: r1 <filter_nomatch>
	 30: LdXMemB dst: r2 src: r0 off: 54 imm: 0
	 31: JEqImm dst: r2 off: -1 imm: 6 <filter_block_10>
	 32: JaImm dst: r0 off: -1 imm: 0 <filter_block_11>
filter_block_7:
	 33: JNEImm dst: r2 off: -1 imm: 2048 <filter_block_11>
filter_block_8:
	 34: MovReg dst: r4 src: r0
	 35: AddImm dst: r4 imm: 24
	 36: JGTReg dst: r4 off: -1 src: r1 <filter_nomatch>
	 37: LdXMemB dst: r2 src: r0 off: 23 imm: 0
	 38: JNEImm dst: r2 off: -1 imm: 6 <filter_block_11>
filter_block_10:
	 39: Mov32Imm dst: r2 imm: 1
	 40: JaImm dst: r0 off: -1 imm: 0 <result>
filter_block_11:
	 41: Mov32Imm dst: r2 imm: 0
	 42: JaImm dst: r0 off: -1 imm: 0 <result>
filter_nomatch:
	 43: MovImm dst: r2 imm: 0
	 44: JaImm dst: r0 off: -1 imm: 0 <result>
result:
	 45: JEqImm dst: r2 off: -1 imm: 0 <exit>
	 46: LdXMemDW dst: r0 src: r7 off: 8 imm: 0
	 47: AddImm dst: r0 imm: 1
	 48: StXMemDW dst: r7 src: r0 off: 8 imm: 0
	 49: MovReg dst: r1 src: r6
	 50: LoadMapPtr dst: r2 fd: 3
	 52: MovReg dst: r3 src: r8
	 53: LShImm dst: r3 imm: 32
	 54: LdImmDW dst: r0 imm: 4294967295
	 56: OrReg dst: r3 src: r0
	 57: MovReg dst: r4 src: rfp
	 58: AddImm dst: r4 imm: -8
	 59: StXMemDW dst: r4 src: r8 off: 0 imm: 0
	 60: AddImm dst: r4 imm: -8
	 61: StMemDW dst: r4 src: r0 off: 0 imm: 1
	 62: MovImm dst: r5 imm: 16
	 63: Call FnPerfEventOutput
	 64: JEqImm dst: r0 off: -1 imm: 0 <exit>
	 65: LdXMemDW dst: r0 src: r7 off: 16 imm: 0
	 66: AddImm dst: r0 imm: 1
	 67: StXMemDW dst: r7 src: r0 off: 16 imm: 0
exit:
	 68: MovImm dst: r0 imm: 1
	 69: Exit

wenlxie · 2023-09-20T04:09:44Z

@ptzianos Could you also help to take a look. Thanks

arthurfabre · 2023-09-21T14:53:40Z

Thanks for giving it a go! Looking at the definition of xdp_buff (https://elixir.bootlin.com/linux/latest/source/include/net/xdp.h#L81), data is a normal pointer to the packet, you don't need to immediately dereference it (it happens later on every time we access the packet).

I think the only difference between xdp_buff and xdp_md is that xdp_buff uses 64 bit pointers on a 64 bit system but xdp_md' always uses 32 bits. Maybe something like this works on a 64 bit system:

 // Packet start
 asm.LoadMem(asm.R0, asm.R6, 0, asm.DWord), 
  
 // Packet end 
 asm.LoadMem(asm.R1, asm.R6, 8, asm.DWord),

wenlxie · 2023-09-21T15:06:42Z

Thanks for giving it a go! Looking at the definition of xdp_buff (https://elixir.bootlin.com/linux/latest/source/include/net/xdp.h#L81), data is a normal pointer to the packet, you don't need to immediately dereference it (it happens later on every time we access the packet).

I think the only difference between xdp_buff and xdp_md is that xdp_buff uses 64 bit pointers on a 64 bit system but xdp_md' always uses 32 bits. Maybe something like this works on a 64 bit system:
 // Packet start
 asm.LoadMem(asm.R0, asm.R6, 0, asm.DWord), 
  
 // Packet end 
 asm.LoadMem(asm.R1, asm.R6, 8, asm.DWord), 

@arthurfabre
Actually I tried this, then I got the error: func 'xxxxxxx' doesn't have 2-th argument

Trace type prog should have the r1 for the address to arg arrays, which is different with xdp prog.

wenlxie · 2023-09-25T04:46:26Z

@arthurfabre

Add more details for the issue I met now:

My tcp filter expr is just "tcp"
In the genenrated cbpf to ebpf codes, it failed to pass the verifier in kernel, the error is: permission denied: 21: (69) r2 = *(u16 *)(r0 +12): R0 invalid mem access 'inv' (26 line(s) omitted)
What I changed is to change the asm code to read the xdp_buff->data and xdp_buff->data_end

		// Packet start
		asm.LoadMem(asm.R9, asm.R6, 0, asm.DWord),
		asm.LoadMem(asm.R0, asm.R9, 0, asm.DWord),

		// Packet end
		asm.LoadMem(asm.R1, asm.R9, 8, asm.DWord),

The error code part is it is try to check for the type field in the mac header.

	 19: AddImm dst: r4 imm: 21
	 20: JGTReg dst: r4 off: -1 src: r1 <filter_nomatch>
	 21: LdXMemH dst: r2 src: r0 off: 12 imm: 0

Since there are already check code ( data + 21 > data_end) before load the data in data+ 12 , so I'd think the generated asm code is fine

So I still think this may related with the code that I changed to get the xdp_buff->data and xdp_buff->data_end
But not sure for how to change for this part.
I checked the xdp-dump tools,
I tried xdp-dump and xlated the trace program that used to hook for xdp prog

When fetch for data and data_end, there is always a convert to long operation. Then I guess maybe this is the issue why I got the error.

I tried to check for the xlated code the trace prograom, I got following codes:

nt trace_on_entry(unsigned long long * ctx):
; int BPF_PROG(trace_on_entry, struct xdp_buff *xdp)
   0: (79) r1 = *(u64 *)(r1 +0)
; void *data = (void *)(long)xdp->data;
   1: (79) r3 = *(u64 *)(r1 +0)
; void *data_end = (void *)(long)xdp->data_end;
   2: (79) r2 = *(u64 *)(r1 +8)
; if (data >= data_end ||
   3: (3d) if r3 >= r2 goto pc+38
; trace_cfg.capture_if_ifindex != xdp->rxq->dev->ifindex)
   4: (79) r4 = *(u64 *)(r1 +32)
; trace_cfg.capture_if_ifindex != xdp->rxq->dev->ifindex)
   5: (79) r4 = *(u64 *)(r4 +0)
; trace_cfg.capture_if_ifindex != xdp->rxq->dev->ifindex)
   6: (61) r5 = *(u32 *)(r4 +208)
; trace_cfg.capture_if_ifindex != xdp->rxq->dev->ifindex)
   7: (18) r4 = map[id:2635][0]+0
   9: (61) r0 = *(u32 *)(r4 +0)
; if (data >= data_end ||
  10: (5d) if r0 != r5 goto pc+31
; metadata.prog_index = trace_cfg.capture_prog_index;
  11: (61) r5 = *(u32 *)(r4 +8)
; metadata.prog_index = trace_cfg.capture_prog_index;
  12: (6b) *(u16 *)(r10 -10) = r5
; metadata.ifindex = xdp->rxq->dev->ifindex;
  13: (79) r5 = *(u64 *)(r1 +32)
; metadata.pkt_len = (__u16)(data_end - data);
  14: (1f) r2 -= r3
; metadata.ifindex = xdp->rxq->dev->ifindex;
  15: (79) r3 = *(u64 *)(r5 +0)
; metadata.ifindex = xdp->rxq->dev->ifindex;
  16: (61) r3 = *(u32 *)(r3 +208)
; metadata.ifindex = xdp->rxq->dev->ifindex;
  17: (63) *(u32 *)(r10 -24) = r3
; metadata.cap_len = min(metadata.pkt_len, trace_cfg.capture_snaplen);
  18: (61) r3 = *(u32 *)(r4 +4)
  19: (bf) r4 = r2
  20: (57) r4 &= 65535
  21: (2d) if r3 > r4 goto pc+1
  22: (bf) r4 = r3
; metadata.rx_queue = xdp->rxq->queue_index;
  23: (79) r5 = *(u64 *)(r1 +32)
; ((__u64) metadata.cap_len << 32) |
  24: (bf) r3 = r4
  25: (67) r3 <<= 32
  26: (18) r0 = 0xffffffff
; ((__u64) metadata.cap_len << 32) |
  28: (4f) r3 |= r0
; metadata.rx_queue = xdp->rxq->queue_index;
  29: (61) r5 = *(u32 *)(r5 +8)
; metadata.pkt_len = (__u16)(data_end - data);
  30: (6b) *(u16 *)(r10 -16) = r2
  31: (b7) r2 = 0
;
  32: (6b) *(u16 *)(r10 -12) = r2
; metadata.action = action;
  33: (63) *(u32 *)(r10 -8) = r2
; metadata.cap_len = min(metadata.pkt_len, trace_cfg.capture_snaplen);
  34: (6b) *(u16 *)(r10 -14) = r4
; metadata.rx_queue = xdp->rxq->queue_index;
  35: (63) *(u32 *)(r10 -20) = r5
  36: (bf) r4 = r10
; metadata.rx_queue = xdp->rxq->queue_index;
  37: (07) r4 += -24
; bpf_xdp_output(xdp, &xdpdump_perf_map,
  38: (18) r2 = map[id:2634]
  40: (b7) r5 = 20
  41: (85) call bpf_xdp_event_output#8687200
; int BPF_PROG(trace_on_entry, struct xdp_buff *xdp)
  42: (b7) r0 = 0
  43: (95) exit

Look like the long convert is actually no needed.

So got a bit confused here.

@arthurfabre Any insights for this? Appreciate that if you can some time and have a check.

wenlxie · 2023-09-25T04:51:52Z

I also genearted the cbpf to c code

// True if packet matches, false otherwise
__attribute__((__always_inline__)) static inline
uint32_t test(const uint8_t *const data, const uint8_t *const data_end) {
	__attribute__((unused))
	uint32_t a, x, m[16];
	__attribute__((unused))
	const uint8_t *indirect;


block_0:
__attribute__((unused));
	if (data + 21 > data_end) return 0;
	a = ntohs(*((uint16_t *) (data + 12)));
	if (a != 34525) goto block_7;

block_2:
__attribute__((unused));
	if (data + 21 > data_end) return 0;
	a = *(data + 20);
	if (a == 6) goto block_10;

block_4:
__attribute__((unused));
	if (a != 44) goto block_11;

block_5:
__attribute__((unused));
	if (data + 55 > data_end) return 0;
	a = *(data + 54);
	if (a == 6) goto block_10; else goto block_11;

block_7:
__attribute__((unused));
	if (a != 2048) goto block_11;

block_8:
__attribute__((unused));
	if (data + 24 > data_end) return 0;
	a = *(data + 23);
	if (a != 6) goto block_11;

block_10:
__attribute__((unused));
	return 1;

block_11:
__attribute__((unused));
	return 0;

}

The block0 can't pass the ebpf verify

block_0:
__attribute__((unused));
	if (data + 21 > data_end) return 0;
	a = ntohs(*((uint16_t *) (data + 12)));
	if (a != 34525) goto block_7;

arthurfabre mentioned this issue Apr 6, 2021

Does xdpcap work similarly to xdpdump, where it can capture packets before and after xdp program execution? #50

Closed

mertyildiran mentioned this issue Jul 31, 2022

Add AF_PACKET support kubeshark/kubeshark#1052

Merged

wenlxie mentioned this issue Sep 21, 2023

eBPf Code generated by this tool can't pass the verifier cloudflare/cbpfc#32

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support capturing via fentry / fexit trace points #51

Support capturing via fentry / fexit trace points #51

arthurfabre commented Apr 6, 2021

wenlxie commented Sep 19, 2023 •

edited

Loading

wenlxie commented Sep 20, 2023 •

edited

Loading

arthurfabre commented Sep 21, 2023 •

edited

Loading

wenlxie commented Sep 21, 2023 •

edited

Loading

wenlxie commented Sep 25, 2023 •

edited

Loading

wenlxie commented Sep 25, 2023

Support capturing via fentry / fexit trace points #51

Support capturing via fentry / fexit trace points #51

Comments

arthurfabre commented Apr 6, 2021

wenlxie commented Sep 19, 2023 • edited Loading

wenlxie commented Sep 20, 2023 • edited Loading

arthurfabre commented Sep 21, 2023 • edited Loading

wenlxie commented Sep 21, 2023 • edited Loading

wenlxie commented Sep 25, 2023 • edited Loading

wenlxie commented Sep 25, 2023

wenlxie commented Sep 19, 2023 •

edited

Loading

wenlxie commented Sep 20, 2023 •

edited

Loading

arthurfabre commented Sep 21, 2023 •

edited

Loading

wenlxie commented Sep 21, 2023 •

edited

Loading

wenlxie commented Sep 25, 2023 •

edited

Loading