Interactive debugging with the target process in libafl_qemu

[noobone123]

Dear Developers, hello! I have recently been using the usermode of libafl_qemu to fuzz a VPN program, and the emulation and hooking features have been incredibly helpful in this process. Thank you for your work.
However, I currently have a need to perform interactive debugging with the target process within libafl_qemu. Specifically, I would like to remotely connect to the target process using gdb-multiarch, similar to how it can be done with regular qemu-user or the qiling-framework. I am wondering if this is something that can be achieved within libafl_qemu. Thank you~

[domenukk]

Yes all options from regular qemu can be used, such as

./libafl_qemu_launcher -i ./in <other params> -- -g 1234 ./target

and then inside qemu, do

target remote :1234

[noobone123]

Thank you for your response @domenukk . I successfully executed the aforementioned commands, and I can now use GDB to attach to a process. However, I still have some questions and would like to seek your further assistance.
Here’s how I executed the commands and proceeded with the debugging:

1. Taking the qemu_launcher in the fuzzer directory as an example, I first executed the following command in a terminal: ./target/output --log ./target/output/log.txt --cores 0-7 --asan-cores 0-3 --cmplog-cores 2-5 --iterations 10000 --tui -- -g 1234 ./target/x86_64/libpng-harness-development, Unlike in the Makefile.toml, I added -g 1234 to expose the debugging port. Now, the qemu_launcher process indeed appears to be hanging.
   
2. I started a new terminal and successfully connected to a process using the target remote :1234 command in GDB (I apologize that I’m not actually sure which process I connected to, as qemu_launcher seems to launch a broker process and multiple clients). The connection is shown below:
   
3. As mentioned above, I’m unsure which process I connected to. Then I tried to add breakpoints in GDB using b <func_name> (e.g., b LLVMFuzzerTestOneInput), GDB informed me that No symbol table is loaded. Use the "file" command.. Furthermore, as shown in the figure above, the program is about to step into a function at address 0x555555558240. Based on this address and my analysis of ./target/x86_64/libpng-harness-development, I believe it should be the _start function of libpng-harness-development (as shown below). By manually calculating the pie_base and adding breakpoints, I successfully entered the main function and the LLVMFuzzerTestOneInput function. However, I don’t quite understand why the symbol table in the program wasn’t loaded, and some commands in pwndbg also failed to work properly (e.g., piebase).
   Another interesting observation is that the GDB command ni often causes the debugged fuzzer to resume execution directly (similar to the effect of the continue command), whereas si doesn’t seem to exhibit this behavior.

.text:0000000000002240 public _start
.text:0000000000002240 _start proc near
.text:0000000000002240 ; __unwind {
.text:0000000000002240 endbr64
.text:0000000000002244 xor     ebp, ebp
.text:0000000000002246 mov     r9, rdx         ; rtld_fini
.text:0000000000002249 pop     rsi             ; argc
.text:000000000000224A mov     rdx, rsp        ; ubp_av
.text:000000000000224D and     rsp, 0FFFFFFFFFFFFFFF0h
.text:0000000000002251 push    rax
.text:0000000000002252 push    rsp             ; stack_end
.text:0000000000002253 xor     r8d, r8d        ; fini
.text:0000000000002256 xor     ecx, ecx        ; init
.text:0000000000002258 lea     rdi, main       ; main
.text:000000000000225F call    cs:__libc_start_main_ptr
.text:0000000000002265 hlt
.text:0000000000002265 ; } // starts at 2240

Finally, Let me summarize my questions:

1. Which process am I connected to after executing target remote :1234 in GDB? I’m unsure whether I’m connected to the broker process, one of the client processes, or the target program itself (libpng-harness-development).
2. Why are the program’s symbols not loaded properly during debugging, and why do the aforementioned issues occur in GDB?I’m confused as to why GDB reports No symbol table is loaded and why commands like piebase in pwndbg fail to work. Additionally, I don’t understand why the ni command often causes the fuzzer to resume execution, while si does not.
3. What is the relationship between the debugged process and qemu_launcher? This is my most pressing question. I don’t fully understand how the debugged process interacts with qemu_launcher. Specifically:
   • The harness in qemu_launcher sets breakpoints in the target program, modifies the arguments of LLVMFuzzerTestOneInput, and injects new inputs before each execution. In the future, there may also be hooks added by the harness.
   • When I debug the target program with GDB, am I debugging a single execution within the harness? (Which means the current input during debugging is influenced by the harness’s code). If not, what is the relationship between debugged process and the harness? Will the hooks I wrote in Rust affect my debugged process?

I apologize for asking so many unprofessional questions. I tried to read the source code of libafl_qemu, but due to my limited knowledge, I still haven’t fully understood the aforementioned points. I greatly look forward to and appreciate your answers!

[domenukk]

I would advice you to not connect to an instance that does any fuzzing.
Instead, call the qemu harness directly with a given input. Or at least use a SimpleEventManager instead of a launcher to avoid any threading headaches.

See this example:

LibAFL/fuzzers/binary_only/qemu_launcher/src/instance.rs

Line 216 in deb7655

if let Some(rerun_input) = &self.options.rerun_input {

[noobone123]

Thank you @domenukk for your help. After setting the fuzzer to simplemgr and specifying a single core, the aforementioned issue was indeed resolved. Now, GDB can debug the target program normally, and the symbols are loaded correctly.

During the debugging process, I also encountered some strange issues, but I think I’ve mostly figured them out now. Therefore, I’d like to summarize them here for future users who, like me, might have no prior experience.
The strange issues I encountered can be attributed to a conflict between the control of the target program by libafl_qemu (the fuzzer) and the control by GDB. I believe it’s important to clarify the concept of control here. During debugging, both the fuzzer and GDB may control the state of the target program. Therefore, when their controls conflict, we might observe strange phenomena in GDB (e.g., sudden changes in register or memory values).

However, the transfer of control follows certain patterns. From my observations, when the fuzzer calls unsafe { qemu.run() ... }, the target program executes, and when we are debugging, control is handed over to GDB. In some special cases, control may return to the fuzzer. For example, if we set a breakpoint in the fuzzer, when the program under GDB’s control hits that breakpoint, the program will be controlled by the fuzzer. After that, a series of operations performed by the fuzzer may alter the state of the target program until the next call to unsafe { qemu.run() ... }. According to the comment in the qemu.run() function: This function will run the emulator until the next breakpoint / sync exit, or until finish., a sync exit will also trigger a transfer of control (I haven’t delved deeper into libafl_qemu yet, so I’m not sure if the hook mechanism is implemented through sync exits).

In summary, when debugging the target program, we need to consider the transfer of control to avoid being caught off guard by strange phenomena.

Finally, I observed another interesting phenomenon. When a breakpoint set in the fuzzer (e.g., 0xdddd) is triggered in GDB, the control is not transferred to the fuzzer when pc == 0xdddd as I initially expected. Instead, control is handed over to the fuzzer only after the instruction at 0xdddd has been executed (in GDB). However, this doesn’t seem to affect the fuzzer’s logic, as the PC value read later in the fuzzer is indeed 0xdddd.

[noobone123]

Today, I delved deeper into the Emulator and Executor code in libafl_qemu and have some new insights to share.

The fuzzer ultimately calls the run_target function in QemuExecutor for each input test, and run_target invokes the user-defined harness closure function. If we implement a custom Module in EmulatorModules (previously known as QemuHelpers and hooks), each iteration of the fuzzing process roughly follows this sequence:

first_exec()
pre_exec()
harness()
post_exec()

I also noticed that the Emulator implements a run function, which transfers control to QEMU and calls pre_qemu_exec and post_qemu_exec whenever control is transferred. However, this function doesn’t seem to be called automatically. Instead, we can call it within the harness. In this case, each iteration of the fuzzing process would look something like this:

first_exec()
pre_exec()
=== harness run ===
Emulator.run() : pre_qemu_exec() -> qemu.run() -> post_qemu_exec()
... (Emulator.run() can be called multiple times as needed)
... (other code in the harness)
=== harness end ===
post_exec()

The above content isn’t actually complicated, but as a beginner, I found it difficult to infer the behavior of each component just from their names, and I often got confused. I hope these simple summaries can help future beginners, as libafl is truly a fantastic project.

[domenukk]

PRs that add documentation are always welcome :P