# GDB step debugging

To GDB step debug the Linux kernel, first run:

    ./run -d

If you want to break immediately at a symbol, e.g. `start_kernel` of the boot sequence, run on another shell:

    ./rungdb start_kernel

Now QEMU will stop there, and you can use the normal GDB commands:

    l
    n
    c

To skip the boot, run just:

    ./rungdb

and when you want to break, do `Ctrl + C` from GDB.

To have some fun, you can first run inside QEMU:

    /count.sh

which counts to infinity to stdout, and then in GDB:

    Ctrl + C
    break sys_write
    continue
    continue
    continue

And you now control the counting from GDB.

See also:

- <http://stackoverflow.com/questions/11408041/how-to-debug-the-linux-kernel-with-gdb-and-qemu/33203642#33203642>
- <http://stackoverflow.com/questions/4943857/linux-kernel-live-debugging-how-its-done-and-what-tools-are-used/42316607#42316607>

`O=0` is an impossible dream, `O=2` being the default: <https://stackoverflow.com/questions/29151235/how-to-de-optimize-the-linux-kernel-to-and-compile-it-with-o0> So get ready for some weird jumps, and `<value optimized out>` fun. Why, Linux, why.

## Kernel module debugging

Loadable kernel modules are a bit trickier since the kernel can place them at different memory locations depending on load order.

So we cannot set the breakpoints before `insmod`.

However, the Linux kernel GDB scripts offer the `lx-symbols` command, which takes care of that beautifully for us:

    ./run -d
    ./rungdb

In QEMU:

    insmod /fops.ko

In GDB, hit `Ctrl + C`, and note how it says:

    scanning for modules in ../kernel_module-1.0/
    loading @0xffffffffa0000000: ../kernel_module-1.0//fops.ko

That's `lx-symbols` working! Now simply:

    b fop_write
    c

In QEMU:

    printf a >/sys/kernel/debug/lkmc_fops/f

and GDB now breaks at our `fop_write` function!

Just don't forget to remove your breakpoints after `rmmod`, or they will point to stale memory locations.

TODO: why does `break work_func` for `insmod kthread.ko` not break the first time I `insmod`, but breaks the second time?

See also: <http://stackoverflow.com/questions/28607538/how-to-debug-linux-kernel-modules-with-qemu/44095831#44095831>

### Bypassing lx-symbols

Useless, but a good way to show how hardcore you are. From inside QEMU:

    insmod /fops.ko
    cat /proc/modules

This will give a line of form:

    fops 2327 0 - Live 0xfffffffa00000000

And then tell GDB where the module was loaded with:

    Ctrl + C
    add-symbol-file ../kernel_module-1.0/fops.ko 0xfffffffa00000000

## Debug kernel early boot

TODO: why can't we break at early startup stuff such as:

    ./rungdb extract_kernel
    ./rungdb main

See also: <https://stackoverflow.com/questions/2589845/what-are-the-first-operations-that-the-linux-kernel-executes-on-boot>

## call

GDB can call functions as explained at: <https://stackoverflow.com/questions/1354731/how-to-evaluate-functions-in-gdb>

However this is failing for us:

- some symbols are not visible to `call` even though `b` sees them
- for those that are, `call` fails with an E14 error

E.g.: if we break on `sys_write` on `/count.sh`:

    >>> call printk(0, "asdf")
    Could not fetch register "orig_rax"; remote failure reply 'E14'
    >>> b printk
    Breakpoint 2 at 0xffffffff81091bca: file kernel/printk/printk.c, line 1824.
    >>> call fdget_pos(fd)
    No symbol "fdget_pos" in current context.
    >>> b fdget_pos
    Breakpoint 3 at 0xffffffff811615e3: fdget_pos. (9 locations)
    >>>

even though `fdget_pos` is the first thing `sys_write` does:

    581 SYSCALL_DEFINE3(write, unsigned int, fd, const char __user *, buf,
    582         size_t, count)
    583 {
    584     struct fd f = fdget_pos(fd);

See also: <https://github.com/cirosantilli/linux-kernel-module-cheat/issues/19>