QEMU 1 : QEMU Emulation of Raspberry Pi (Sort Of) and Attaching Debugger

I had raspberry pi 2b model lying in my desk. But there are two roadblocks, one the architecture is 32 bit. I should not call it roadblock, but IMO it’s better to dive in 64bit system as the future is 64bit systems.
Second, after spending days, I figured out that there is something wrong with the UART port on this device. Luckily my colleagues are amazing, they suggested to not spend money in buying a new board, but use QEMU emulation for raspberry pi initially until I figure my way around. Use GDB debugger to debug the bootup process.

So I turned to QEMU for emulation of ARM64 device emulation.
QEMU supports different types of boards. For example raspberry pi boards are mentioned as raspi4b. For the simulation purpose, I am using generic virt machine instead of raspi4b. But I am using latest 64bit Raspbian OS image. It can be said it not a full blown virtualization of raspberry pi, but serves environment for diving into internals of ARM64 bit architecture.

‘virt‘ is a generic board in QEMU and is not really a hardware. A generates a generic device tree blob (.dtb) is also supplied to this generic platform.  This dtb provides information about the addresses, interrupt lines and other configuration of the various devices in the system.
It is to be noted, for this board, emulated hardware flash memory starts from 0x00000000 and RAM starts from 0x40000000. Usually the device tree blob (dtb) is loaded at the start of RAM address. See details here.

For the steps of emulating raspberry Pi, I followed exact steps mention in cGandom/RaspberryPi4-qemu.md Github page. This is a wonderful step by step guide of setting up QEMU emulator for Raspberry PI.
But Things to note here is-
1. You are allowed to change user id and password as per your wish. I used default username “pi” and “raspberry” as password.
2. Also note that CONFIG_GDB_SCRIPTS has to be enabled for debugging purposes. New config enablement can be done from kernel “menuconfig“; just after you have executed make defconfig step in above mentioned page.
3. You can pick latest Raspbian OS image.

Below is the command formed to launch QEMU emulator with debugging mode enabled-

qemu-system-aarch64 -S -s -machine virt -cpu cortex-a72 -smp 1 -m 2G \
        -kernel Image -append root=/dev/vda2 rootfstype=ext4 rw panic=0 \
         console=ttyAMA0 nographic nokaslr maxcpus=1 \
        -drive format=raw,file=2024-11-19-raspios-bookworm-arm64.img,if=none,id=hd0,cache=writeback \
        -device virtio-blk,drive=hd0,bootindex=0 \
        -netdev user,id=mynet,hostfwd=tcp::2222-:22 \
        -device virtio-net-pci,netdev=mynet \
        -monitor telnet:127.0.0.1:5555,server,nowait

Breakdown of above command
1. I have selected machine as virt. The default virt machine is cortex-a15, I have switched it to cortex-a78 to match raspberry pi 4b (sort of).
2. After building kernel, save the vmlinux to preferably in home directory. Try in other directory, but some weird error may pop up, which I don’t want to go into. Vmlinux image is found in same place where your compiled image is present: arch/arm64/boot/vmlinux
3. I have added “nokaslr” to disable kernel space randomization. Otherwise symbols addresses in vmlinux image and actual address may not match due to randomized base address.
4. The -s option enables debugging mode while launching QEMU.
5. The -S option is added to pause execution at startup until continue (“c”) command is issued from debugger.
6. I have set maxcpus=1. This option limits number of cores in the system to 1. This way debugging becomes way easier than debugging multicore system.

To enable debugging

1. Launch GDB from second terminal. Use vmlinux image saved as mentioned above-

gdb multiarch ~/vmlinux

2. Inside gdb session, connect with QEMU at default address localhost:1234

(gdb) target remote :1234

3. Once connected, with gdb set desired breakpoints using command b <function name>.
Example I have placed a breakpoint at rest_init function-

b rest_init

4. Press c to for QEMU to resume execution. It will continue to boot until your breakpoint is hit.

Everything was fine until I placed a breakpoint at first address of kernel start. In my experiment I placed breakpoint at the first function “primary_entry“. This is one of the first function when kernel boots. This function is present in head.S file arch/arm64/kernel/head.S. To my surprise, I did not hit my breakpoint.
So after spending another day, I found out that someone was dealing with our exact problem. According to this post, early assembly code is not indexed. Also it has something to do with MMU not being enabled so we have to deal with physical addresses.

Anyways, we can add symbols to our debugger by running below command-

(gdb) add-symbol-file vmlinux -s .head.text 0x40200000 -s .text 0x40210000
add symbol table from file "vmlinux" at
.head.text_addr = 0x40200000
.text_addr = 0x40210000
(y or n) y
Reading symbols from vmlinux...

This command is telling gdb to load the symbols for the .head.text section at the address 0x40200000 and the .text section at the address 0x40210000. GDB will load symbols from vmlinux and give addresses for .head.text as 0x40200000 and .text as 0x40210000
Note: The .head.text section is the section in assembly code used for setting up first bootup code. Once the early setup is done, initialization will be transferred to kernel code at .text section.

Caution:
Without going in details, where this info in the above post is derived from I am making following assumptions.
Supplied dtb to “virt” board is assumed to be of max size 2MB. Hence the global .head.text section starts after 2MB mark at 0x40200000, and .text be of max 64KB, hence .text section starts at 0x40210000.

What are these sections, we will be talking about these sections in next post, when we are creating our bare metal kernel. But for starters, it has to do with linker scripts. These sections manage memory of our programs.

Now we can add break points to first text section. We can now place breakpoint at first address of execution as “b _text”. We can also place breakpoint at primary_entry as well.

(gdb) b _text
Breakpoint 3 at 0x40200000: _text.
(gdb) b primary_entry
Breakpoint 4 at 0xffff800081ab30a0
(gdb) c
Breakpoint 3.1, _text () at arch/arm64/kernel/head.S:60
60 efi_signature_nop // special NOP to identity as PE/COFF executable

Additional Notes

1. Next instruction being executed can be stepped using nexti (or ni) command. The way it is different from step command (or s command) is, during function call, step jumps to first line of function and stops.
2. At any time arm registers can be viewed using i r commands. This is helpful in viewing register contents change on instruction execution.

References

1. https://stackoverflow.com/questions/79307197/how-to-set-a-breakpoint-at-arm64-kernel-startup-entry-point-using-qemu-and-gdb
2. https://gist.github.com/cGandom/23764ad5517c8ec1d7cd904b923ad863
3. Emulating a Raspberry Pi in QEMU | Interrupt
4. https://elixir.bootlin.com/linux/v6.6.74/source/arch/arm64/kernel/head.S
5. https://www.raspberrypi.com/documentation/computers/linux_kernel.html
6. https://www.kernel.org/doc/html/v4.15/dev-tools/gdb-kernel-debugging.html
7. https://www.qemu.org/docs/master/system/arm/virt.html