From 321761f177143d1c928c83fdf5e47bc6ba8dad40 Mon Sep 17 00:00:00 2001 From: Ciro Santilli Date: Wed, 28 Feb 2018 02:48:57 +0000 Subject: [PATCH] Get rid of action to make things simpler --- README.adoc | 128 ++++++++++++++++++++++++++-------------------------- 1 file changed, 63 insertions(+), 65 deletions(-) diff --git a/README.adoc b/README.adoc index 305bbba..afac398 100644 --- a/README.adoc +++ b/README.adoc @@ -13,9 +13,7 @@ Run one command, get a QEMU Buildroot BusyBox virtual machine built from source toc::[] -== Action - -=== Getting started +== Getting started Reserve 12Gb of disk and run: @@ -57,7 +55,7 @@ image:screenshot.png[image] All available modules can be found in the link:kernel_module/[`kernel_module` directory]. -==== Module documentation +=== Module documentation .... head kernel_module/modulename.c @@ -76,7 +74,7 @@ The sources of those tests will further clarify what the corresponding kernel mo git ls-files | grep modulename .... -==== Rebuild +=== Rebuild If you make changes to the kernel modules or most configurations tracked on this repository, you can just use again: @@ -101,7 +99,7 @@ Linux and QEMU rebuilds are so common that we have dedicated shortcut flags for ./build -l -q .... -==== Clean the build +=== Clean the build You did something crazy, and nothing seems to work anymore? @@ -130,7 +128,7 @@ rm -rf buildroot/output.x86_64~/build/host-qemu-custom This is sometimes necessary when changing the version of the submodules, and then builds fail. We should try to understand why and report bugs. -==== Filesystem persistency +=== Filesystem persistency The root filesystem is persistent across: @@ -160,7 +158,7 @@ the disk image gets overwritten by a fresh filesystem and you lose all changes. Remember that if you forcibly turn QEMU off without `sync` or `poweroff` from inside the VM, e.g. by closing the QEMU window, disk changes may not be saved. -==== Message control +=== Message control We use `printk` a lot, and it shows on the QEMU terminal by default. If that annoys you (e.g. you want to see stdout separately), do: @@ -183,7 +181,7 @@ but I never managed to increase that buffer: The superior alternative is to use text mode or a telnet connection. -==== Text mode +=== Text mode Show serial console directly on the current terminal, without opening a QEMU window: @@ -245,7 +243,7 @@ Limitations: This is however fortunate when running QEMU with GDB, as the Ctrl + C reaches GDB and breaks. * Very early kernel messages such as `early console in extract_kernel` only show on the GUI, since at such early stages, not even the serial has been setup. -==== Automatic startup commands +=== Automatic startup commands When debugging a module, it becomes tedious to wait for build and re-type: @@ -283,7 +281,7 @@ and they will be run automatically before the login prompt. Scripts under `/etc/init.d` are run by `/etc/init.d/rcS`, which gets called by the line `::sysinit:/etc/init.d/rcS` in `/etc/inittab`. -==== Kernel version +=== Kernel version We try to use the latest possible kernel major release version. @@ -312,7 +310,7 @@ or on host: cat buildroot/output.*~/build/linux-custom/.config .... -==== Kernel command line parameters +=== Kernel command line parameters Bootloaders can pass a string as input to the Linux kernel when it is booting to control its behaviour, much like the `execve` system call does to userland processes. @@ -348,7 +346,7 @@ When dealing with real boards, extra command line options are provided on some m * GRUB configuration files: https://askubuntu.com/questions/19486/how-do-i-add-a-kernel-boot-parameter * Raspberry pi `/boot/cmdline.txt` on a magic partition: https://raspberrypi.stackexchange.com/questions/14839/how-to-change-the-kernel-commandline-for-archlinuxarm-on-raspberry-pi-effectly -==== QEMU GUI is unresponsive +=== QEMU GUI is unresponsive Sometimes in Ubuntu 14.04, after the QEMU SDL GUI starts, it does not get updated after keyboard strokes, and there are artifacts like disappearing text. @@ -364,7 +362,7 @@ This started happening when we switched to building QEMU through Buildroot, and Using text mode is another workaround if you don't need GUI features. -==== Debug QEMU +=== Debug QEMU When you start interacting with QEMU hardware, it is useful to see what is going on inside of QEMU itself. @@ -392,7 +390,7 @@ Just make sure that you never click inside the QEMU window when doing that, othe You can still send key presses to QEMU however even without the mouse capture, just either click on the title bar, or alt tab to give it focus. [[gdb]] -=== GDB step debugging +== GDB step debugging To GDB step debug the Linux kernel, first run: @@ -447,7 +445,7 @@ See also: `O=0` is an impossible dream, `O=2` being the default: https://stackoverflow.com/questions/29151235/how-to-de-optimize-the-linux-kernel-to-and-compile-it-with-o0 So get ready for some weird jumps, and `` fun. Why, Linux, why. -==== Kernel module debugging +=== Kernel module debugging Loadable kernel modules are a bit trickier since the kernel can place them at different memory locations depending on load order. @@ -490,7 +488,7 @@ TODO: why does `break work_func` for `insmod kthread.ko` not break the first tim See also: http://stackoverflow.com/questions/28607538/how-to-debug-linux-kernel-modules-with-qemu/44095831#44095831 -===== Bypassing lx-symbols +==== Bypassing lx-symbols Useless, but a good way to show how hardcore you are. From inside QEMU: @@ -512,7 +510,7 @@ Ctrl + C add-symbol-file ../kernel_module-1.0/fops.ko 0xfffffffa00000000 .... -==== Debug kernel early boot +=== Debug kernel early boot TODO: why can't we break at early startup stuff such as: @@ -523,7 +521,7 @@ TODO: why can't we break at early startup stuff such as: See also: https://stackoverflow.com/questions/2589845/what-are-the-first-operations-that-the-linux-kernel-executes-on-boot -==== GDB call +=== GDB call GDB can call functions as explained at: https://stackoverflow.com/questions/1354731/how-to-evaluate-functions-in-gdb @@ -557,7 +555,7 @@ even though `fdget_pos` is the first thing `sys_write` does: See also: https://github.com/cirosantilli/linux-kernel-module-cheat/issues/19 -=== KGDB +== KGDB KGDB is kernel dark magic that allows you to GDB the kernel on real hardware without any extra hardware support. @@ -604,7 +602,7 @@ See also: * https://github.com/torvalds/linux/blob/v4.9/Documentation/DocBook/kgdb.tmpl * https://stackoverflow.com/questions/22004616/qemu-kernel-debugging-with-kgdb/44197715#44197715 -==== KGDB kernel modules +=== KGDB kernel modules In QEMU: @@ -626,7 +624,7 @@ and you now control the count. TODO: if I `-ex lx-symbols` to the `gdb` command, just like done for QEMU `-gdb`, the kernel oops. How to automate this step? -==== KDB +=== KDB If you modify `runqemu` to use: @@ -662,7 +660,7 @@ The other KDB commands allow you to instruction steps, view memory, registers an But TODO I don't think you can see where you are in the kernel source code and line step as from GDB, since the kernel source is not available on guest (ah, if only debugging information supported full source). -=== gdbserver +== gdbserver Step debug userland processes to understand how they are talking to the kernel. @@ -705,7 +703,7 @@ Feature request: https://bugs.busybox.net/show_bug.cgi?id=10386 An implementation overview can be found at: https://reverseengineering.stackexchange.com/questions/8829/cross-debugging-for-mips-elf-with-qemu-toolchain/16214#16214 -==== gdbserver different archs +=== gdbserver different archs As usual, different archs work with: @@ -713,7 +711,7 @@ As usual, different archs work with: ./rungdbserver -a arm kernel_module-1.0/user/myinsmod.out .... -==== gdbserver BusyBox +=== gdbserver BusyBox BusyBox executables are all symlinks, so if you do on guest: @@ -727,7 +725,7 @@ on host you need: ./rungdbserver busybox-1.26.2/busybox .... -==== gdbserver shared libraries +=== gdbserver shared libraries Our setup gives you the rare opportunity to step debug libc and other system libraries e.g. with: @@ -752,7 +750,7 @@ which automatically finds unstripped shared libraries on the host for us. See also: https://stackoverflow.com/questions/8611194/debugging-shared-libraries-with-gdbserver/45252113#45252113 -==== Debug userland process without gdbserver +=== Debug userland process without gdbserver QEMU `-gdb` GDB breakpoints are set on virtual addresses, so you can in theory debug userland processes as well. @@ -841,11 +839,11 @@ continue This is of least reliable setup as there might be other processes that use the given virtual address. -=== Other architectures +== Other architectures The portability of the kernel and toolchains is amazing: change an option and most things magically work on completely different hardware. -==== arm +=== arm First build: @@ -899,7 +897,7 @@ Error occurred in Python command: 'utf8' codec can't decode byte 0x9f in positio + and then after inserting the module, symbols are not found, presumably because `lx-symbols` never ran. -==== aarch64 +=== aarch64 .... ./build -a aarch64 @@ -932,7 +930,7 @@ TODOs: no module object found for '' .... -==== mips64 +=== mips64 .... ./build -a mips64 @@ -948,7 +946,7 @@ TODOs: ** https://unix.stackexchange.com/questions/354127/qemu-mips-and-debian * <> does not work properly, does not find `start_kernel` -=== init +== init When the Linux kernel finishes booting, it runs an executable as the first and only userland process. @@ -962,7 +960,7 @@ systemd is a "popular" `/init` implementation for desktop distros as of 2017. BusyBox provides its own minimalistic init implementation which Buildroot uses by default. -==== Custom init +=== Custom init Is the default BusyBox `/init` too bloated for you, minimalism freak? @@ -983,7 +981,7 @@ Also remember that if your init returns, the kernel will panic, there are just t * run forever in a loop or long sleep * `poweroff` the machine -==== Disable networking +=== Disable networking The default BusyBox init scripts enable networking, and there is a 15 second timeout in case your network is down or if your kernel / emulator setup does not support it. @@ -999,7 +997,7 @@ To restore it, run: ./build -t initscripts-reconfigure .... -==== The init environment +=== The init environment The docs make it clear https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html @@ -1013,7 +1011,7 @@ And you can try it out with: ./run -e 'init=/init_env_poweroff.sh - asdf=qwer zxcv' -n .... -=== modprobe +== modprobe If you are feeling fancy, you can also insert modules with: @@ -1059,7 +1057,7 @@ Kernel modules built from the Linux mainline tree with `CONFIG_SOME_MOD=m`, are modprobe dummy-irq .... -=== X11 +== X11 Only tested successfully in `x86_64`. @@ -1084,7 +1082,7 @@ More details: https://unix.stackexchange.com/questions/70931/how-to-install-x11- Not sure how well that graphics stack represents real systems, but if it does it would be a good way to understand how it works. -==== X11 ARM +=== X11 ARM On ARM, `startx` hangs at a message: @@ -1109,7 +1107,7 @@ A friend told me this but I haven't tried it yet: * `xf86-video-modesetting` is likely the missing ingredient, but it does not seem possible to activate it from Buildroot currently without patching things. * `xf86-video-fbdev` should work as well, but we need to make sure fbdev is enabled, and maybe add some line to the `Xorg.conf` -=== Count boot instructions +== Count boot instructions * https://www.quora.com/How-many-instructions-does-a-typical-Linux-kernel-boot-take * https://github.com/cirosantilli/chat/issues/31 @@ -1205,7 +1203,7 @@ Call Trace: in which the boot appears to hang for a considerable time. * Confirm that the kernel enters at `0x1000000`, or where it enters. Once we have this, we can exclude what comes before in the BIOS. -=== initrd +== initrd The kernel can boot from an CPIO file, which is a directory serialization format much like tar: https://superuser.com/questions/343915/tar-vs-cpio-what-is-the-difference @@ -1244,7 +1242,7 @@ Buildroot forces that option when `BR2_TARGET_ROOTFS_CPIO=y` is given https://unix.stackexchange.com/questions/89923/how-does-linux-load-the-initrd-image asks how the mechanism works in more detail. -==== initrd in desktop distros +=== initrd in desktop distros Most modern desktop distributions have an initrd in their root disk to do early setup. @@ -1258,7 +1256,7 @@ I think GRUB then knows read common disk formats, and then loads that initrd to Related: https://stackoverflow.com/questions/6405083/initrd-and-booting-the-linux-kernel -==== initramfs +=== initramfs initramfs is just like <>, but you also glue the image directly to the kernel image itself. @@ -1276,7 +1274,7 @@ In the background, it uses `BR2_TARGET_ROOTFS_INITRAMFS`, and this makes the ker http://nairobi-embedded.org/initramfs_tutorial.html shows a full manual setup. -=== ftrace +== ftrace Trace a single function: @@ -1371,7 +1369,7 @@ TODO: what do `+` and `!` mean? Each `enable` under the `events/` tree enables a certain set of functions, the higher the `enable` more functions are enabled. -=== QEMU user mode +== QEMU user mode This has nothing to do with the Linux kernel, but it is cool: @@ -1412,7 +1410,7 @@ qemu-arm -g 1234 -L . bin/ls TODO: find source. Lazy now. -=== Snapshot +== Snapshot https://stackoverflow.com/questions/40227651/does-qemu-emulator-have-checkpoint-function/48724371#48724371 @@ -1478,11 +1476,11 @@ And the output is `0`. Our setup does not allow for snapshotting while using <>. -=== GEM5 +== GEM5 GEM5 is a system simulator, much like QEMU: http://gem5.org/ -==== GEM5 vs QEMU +=== GEM5 vs QEMU * advantages of GEM5: ** simulates a generic more realistic pipelined and optionally out of order CPU cycle by cycle, including a realistic DRAM memory access model with latencies, caches and page table manipulations. This allows us to: @@ -1515,7 +1513,7 @@ This suits chip makers that want to distribute forks with secret IP to their cus + On the other hand, the chip makers tend to upstream less, and the project becomes more crappy in average :-) -===== GEM5 vs QEMU performance +==== GEM5 vs QEMU performance We have benchmarked a Linux kernel boot at commit da79d6c6cde0fbe5473ce868c9be4771160a003b with the commands: @@ -1549,7 +1547,7 @@ on a Lenovo P51 laptop with: * 512GB SSD PCIe TLC OPAL2 * Ubuntu 17.10 -==== GEM5 ARM +=== GEM5 ARM .... ./configure && ./build -a arm -g @@ -1562,7 +1560,7 @@ On another shell: ./gem5-shell .... -===== GEM5 run benchmark +==== GEM5 run benchmark OK, this is why we used GEM5 in the first place, performance measurements! @@ -1616,7 +1614,7 @@ Whenever we run `m5 dumpstats` or `m5 exit`, a section with the following format ---------- End Simulation Statistics ---------- .... -====== Enable compiler optimizations +===== Enable compiler optimizations If you are benchmarking compiled programs instead of hand written assembly, remember that we configure Buildroot to disable optimizations by default with: @@ -1636,7 +1634,7 @@ and do a full rebuild. TODO is it possible to compile a single package with optimizations enabled? In any case, this wouldn't be very representative, since calls to an unoptimized libc will also have an impact on performance. Kernel-wise it should be fine though, since the kernel requires `O=2`. -====== Interesting benchmarks +===== Interesting benchmarks Buildroot built-in libraries, mostly under Libraries > Other: @@ -1661,7 +1659,7 @@ External open source benchmarks. We will try to create Buildroot packages for th * http://parsec.cs.princeton.edu/ Mentioned on docs: http://gem5.org/PARSEC_benchmarks * http://www.m5sim.org/Splash_benchmarks -====== GEM5 change system parameters +===== GEM5 change system parameters Besides optimizing a program for a given CPU setup, chip developers can also do the inverse, and optimize the chip for a given benchmark! @@ -1776,7 +1774,7 @@ TODO: why doesn't this exist: ls /sys/devices/system/cpu/cpu0/cpufreq .... -===== GEM5 kernel command line parameters +==== GEM5 kernel command line parameters Analogous <>: @@ -1801,7 +1799,7 @@ Kernel command line: .... [[gem5-gdb]] -===== GEM5 GDB step debugging +==== GEM5 GDB step debugging Analogous <>, on the first shell: @@ -1831,7 +1829,7 @@ And we now see the boot messages, and then get a shell. Now try the `/continue.s TODO: how to stop at `start_kernel`? GEM5 listens for GDB by default, and therefore does not wait for a GDB connection to start like QEMU does. So when GDB connects we might have already passed `start_kernel`. Maybe `--debug-break=0` can be used? -===== GEM5 checkpoint +==== GEM5 checkpoint Analogous to QEMU's <>, but better since it can be started from inside the guest, so we can easily checkpoint after a specific guest event, e.g. just before `init` is done. @@ -1893,7 +1891,7 @@ Then there is no need to pass the kernel command line again to GEM5 for replay: since boot has already happened, and the parameters are already in the RAM of the snapshot. -====== GEM5 restore checkpoint with a different CPU +===== GEM5 restore checkpoint with a different CPU GEM5 can switch to a different CPU model when restoring a checkpoint. @@ -1917,7 +1915,7 @@ And then restore the checkpoint with a different CPU: ./run -a arm -g -- --caches -r 1 --restore-with-cpu=HPI .... -===== Pass extra options to GEM5 +==== Pass extra options to GEM5 Pass options to the `fs.py` script: @@ -1940,7 +1938,7 @@ Pass options to the `gem5` executable itself: ./run -G '-h' -g .... -===== Run multiple GEM5 instances at once +==== Run multiple GEM5 instances at once GEM5 just assigns new ports if some ports are occupied, so we can do: @@ -1959,7 +1957,7 @@ And a second instance: TODO Now we just need to network them up to have some more fun! -===== QEMU and GEM5 with the same kernel configuration +==== QEMU and GEM5 with the same kernel configuration We would like to be able to run both GEM5 and QEMU with the same kernel build to avoid duplication, but TODO we haven't been able to get that working yet. @@ -1967,7 +1965,7 @@ This documents our failed attempts so far. As a result, we currently have to create two full `buildroot/output*` directories, which means two full GCC builds. -====== QEMU with GEM5 kernel configuration +===== QEMU with GEM5 kernel configuration To test this, hack up `run` to use the `buildroot/output.arm-gem5~` directory, and then run: @@ -1987,7 +1985,7 @@ and the display shows: Guest has not initialized the display (yet). .... -====== GEM5 with QEMU kernel configuration +===== GEM5 with QEMU kernel configuration Test it out with: @@ -2016,19 +2014,19 @@ Escape character is '^]'. I have also tried to copy the exact same kernel command line options used by QEMU, but nothing changed. -===== GEM5 limitations +==== GEM5 limitations * networking not working * `gdbserver`: https://stackoverflow.com/questions/48941494/how-to-do-port-forwarding-from-guest-to-host-in-gem5 -==== GEM5 aarch64 +=== GEM5 aarch64 .... ./configure && ./build -a aarch64 -g ./run -a aarch64 -g .... -==== GEM5 x86 +=== GEM5 x86 .... ./configure && ./build -a x86_64 -g