diff --git a/.gitignore b/.gitignore index b79e7b3..0d977b3 100644 --- a/.gitignore +++ b/.gitignore @@ -7,5 +7,6 @@ gitignore* # Specific files. /data +/images-*.zip /out /out.* diff --git a/README.adoc b/README.adoc index f94b499..6915523 100644 --- a/README.adoc +++ b/README.adoc @@ -15,9 +15,11 @@ toc::[] == Getting started -=== Getting started Ubuntu +=== Getting started natively -This is the most native setup, and therefore the best one if you are on one of the supported Ubuntu: 16.04 or 18.04. +This is the best one if you are on one of the supported Ubuntu: 16.04 or 18.04. + +It will likely work on other Linux distros if you install the required packages, but this is not currently well tested, but patches are welcome. You can also try <> if you are on other distros. Reserve 12Gb of disk and run: @@ -27,9 +29,16 @@ cd linux-kernel-module-cheat ./configure && ./build && ./run .... +It is also trivial to build for different supported <>. + The first configure will take a while (30 minutes to 2 hours) to clone and build, see <> for more details. -If you don't want to wait, you could also try to compile the examples and run them on your host computer as explained on at <>, but as explained on that section, that is dangerous, limited, and will likely not work. +If you don't want to wait, you could also try the following faster but much more limited methods: + +* <> +* <> + +but you will soon find that they are simply not enough if you anywhere near serious about systems programming. After QEMU opens up, you can start playing with the kernel modules: @@ -108,9 +117,9 @@ We will try to support the following Ubuntu versions at least: * otherwise, support both latest LTS and the latest non-LTS [[docker]] -=== Getting started Docker +=== Getting started with Docker -This is a good option if you are on a Linux host, but the <> failed due to your weird host distribution. +This is a good option if you are on a Linux host, but the <> failed due to your weird host distribution. Note however that most things in this repository are highly Linux-portable, should just work once you have found the corresponding `apt-get` package manager commands in the link:configure[] for your distro. In theory :-) @@ -214,6 +223,114 @@ After this, to start using Docker again will you need another: ./rundocker setup .... +[[prebuilt]] +=== Getting started with prebuilts + +We don't currently provide a full prebuilt because it would be too big to host freely. + +However, if you just want to quickly try out running our kernel modules, try this: + +. Download QEMU and this repo: ++ +.... +sudo apt-get install qemu-system-x86 +git clone https://github.com/cirosantilli/linux-kernel-module-cheat +cd linux-kernel-module-cheat +.... +. go to the latest release link:https://github.com/cirosantilli/linux-kernel-module-cheat/releases[], download the `images-*.zip` file and extract it: ++ +.... +unzip images-*.zip +.... ++ +It is link:https://stackoverflow.com/questions/24987542/is-there-a-link-to-github-for-downloading-a-file-in-the-latest-release-of-a-repo/50540591#50540591[not possible to automate this step without the API], and I'm not venturing there this time +. run with the `-P` option: ++ +.... +./run -P +.... + +Limitations of this method: + +* can't GDB step debug the kernel, since the source and cross toolchain with GDB are not available. Buildroot cannot easily use a host toolchain: <>. ++ +Maybe we could work around this by just downloading the kernel source somehow, and using a host prebuilt GDB, but we felt that it would be too messy and unreliable. +* can't create new modules or modify the existing ones, since no cross toolchain +* can't use things that rely on our QEMU fork, e.g. in-fork <> or <> +* you won't get the latest version of this repository. Our <> attempt to automate builds failed, and storing a release for every commit would likely make GitHub mad at us. +* <> is not currently supported, although it should not be too hard to do. One annoyance is that there is no Debian package for it, so you have to compile your own, so you might as well just build the image itself. + +[[host]] +=== Getting started on host + +This method runs the kernel modules directly on your host computer without a VM, and saves you the compilation time and disk usage of the virtual machine method. + +It has however severe limitations, and you will soon see that the compilation time and disk usage are well worth it: + +* can't control which kernel version and build options to use. So some of the modules will likely not compile because of kernel API changes, since https://stackoverflow.com/questions/37098482/how-to-build-a-linux-kernel-module-so-that-it-is-compatible-with-all-kernel-rele/45429681#45429681[the Linux kernel does not have a stable kernel module API]. +* bugs can easily break you system. E.g.: +** segfaults can trivially lead to a kernel crash, and require a reboot +** your disk could get erased. Yes, this can also happen with `sudo` from userland. But you should not use `sudo` when developing newbie programs. And for the kernel you don't have the choice not to use `sudo`. +** even more subtle system corruption such as https://unix.stackexchange.com/questions/78858/cannot-remove-or-reinsert-kernel-module-after-error-while-inserting-it-without-r[not being able to rmmod] +* can't control which hardware is used, notably the CPU architecture +* can't step debug it with <> easily. The alternatives are JTAG or <>, but those are less reliable, and JTAG requires extra hardware. + +Still interested? + +.... +cd kernel_module +./make-host.sh +.... + +If the compilation of any of the C files fails because of kernel or toolchain differences that we don't control on the host, just rename it to remove the `.c` extension and try again: + +.... +mv broken.c broken.c~ +./build_host +.... + +Once you manage to compile, and have come to terms with the fact that this may blow up your host, try it out with: + +.... +sudo insmod hello.ko + +# Our module is there. +sudo lsmod | grep hello + +# Last message should be: hello init +dmest -T + +sudo rmmod hello + +# Last message should be: hello exit +dmesg -T + +# Not present anymore +sudo lsmod | grep hello +.... + +Once you are done with this method, you must clean up the in-tree build objects before you decide to do the right thing and move on to the superior `./build` Buildroot method: + +.... +cd "kernel_module" +./make-host.sh clean +.... + +otherwise they will cause problems. + +==== Hello host + +Minimal host build system example: + +.... +cd hello_host +make +insmod hello.ko +dmesg +rmmod hello.ko +dmesg +.... + === Text mode By default, we show the serial console directly on the current terminal, without opening a QEMU window. @@ -6294,9 +6411,7 @@ Finally, do a clone of the relevant repository out of tree and reproduce the bug In this section document how benchmark builds and runs of this repo, and how to investigate what the bottleneck is. -Ideally, we should setup an automated build server that benchmarks those things continuously for us. - -We tried to automate it on Travis with link:.travis.yml[] but it hits the current 50 minute job timeout: https://travis-ci.org/cirosantilli/linux-kernel-module-cheat/builds/296454523 And I bet it would likely hit a disk maxout either way if it went on. +Ideally, we should setup an automated build server that benchmarks those things continuously for us, but our <> attempt failed. So currently, we are running benchmarks manually when it seems reasonable and uploading them to: https://github.com/cirosantilli/linux-kernel-module-cheat-regression @@ -6308,6 +6423,10 @@ Run all benchmarks and upload the results: ./bench-all -A .... +=== Travis + +We tried to automate it on Travis with link:.travis.yml[] but it hits the current 50 minute job timeout: https://travis-ci.org/cirosantilli/linux-kernel-module-cheat/builds/296454523 And I bet it would likely hit a disk maxout either way if it went on. + === Benchmark this repo benchmarks ==== Benchmark Linux kernel boot @@ -6427,8 +6546,6 @@ Our philosophy is: * try to keep the toolchain (GCC, Binutils) unchanged, otherwise a full rebuild is required. + So we generally just enable all toolchain options by default, even though this adds a bit of time to the build. -+ -The biggest build time hog is always GCC, and it does not look like we can use a precompiled one: https://stackoverflow.com/questions/10833672/buildroot-environment-with-host-toolchain * if something is very valuable, we just add it by default even if it increases the Build time, notably GDB and QEMU * runtime is sacred. + @@ -6441,6 +6558,11 @@ We do our best to reduce the instruction and feature count to the bare minimum n + One possibility we could play with is to build loadable modules instead of built-in modules to reduce runtime, but make it easier to get started with the modules. +[[prebuilt-toolchain]] +====== Buildroot use prebuilt host toolchain + +The biggest build time hog is always GCC, and it does not look like we can use a precompiled one: https://stackoverflow.com/questions/10833672/buildroot-environment-with-host-toolchain + ===== Benchmark Buildroot build baseline This is the minimal build we could expect to get away with. @@ -6525,76 +6647,6 @@ gem5: ** https://stackoverflow.com/questions/47997565/gem5-system-requirements-for-decent-performance/48941793#48941793 ** https://github.com/gem5/gem5/issues/25 -== Run kernel modules on host - -This method runs the kernel modules directly on your host computer without a VM, and saves you the compilation time and disk usage of the virtual machine method. - -It has however severe limitations, and you will soon see that the compilation time and disk usage are well worth it: - -* can't control which kernel version and build options to use. So some of the modules will likely not compile because of kernel API changes, since https://stackoverflow.com/questions/37098482/how-to-build-a-linux-kernel-module-so-that-it-is-compatible-with-all-kernel-rele/45429681#45429681[the Linux kernel does not have a stable kernel module API]. -* bugs can easily break you system. E.g.: -** segfaults can trivially lead to a kernel crash, and require a reboot -** your disk could get erased. Yes, this can also happen with `sudo` from userland. But you should not use `sudo` when developing newbie programs. And for the kernel you don't have the choice not to use `sudo`. -** even more subtle system corruption such as https://unix.stackexchange.com/questions/78858/cannot-remove-or-reinsert-kernel-module-after-error-while-inserting-it-without-r[not being able to rmmod] -* can't control which hardware is used, notably the CPU architecture -* can't step debug it with <> easily. The alternatives are JTAG or <>, but those are less reliable, and JTAG requires extra hardware. - -Still interested? - -.... -cd kernel_module -./make-host.sh -.... - -If the compilation of any of the C files fails because of kernel or toolchain differences that we don't control on the host, just rename it to remove the `.c` extension and try again: - -.... -mv broken.c broken.c~ -./build_host -.... - -Once you manage to compile, and have come to terms with the fact that this may blow up your host, try it out with: - -.... -sudo insmod hello.ko - -# Our module is there. -sudo lsmod | grep hello - -# Last message should be: hello init -dmest -T - -sudo rmmod hello - -# Last message should be: hello exit -dmesg -T - -# Not present anymore -sudo lsmod | grep hello -.... - -Once you are done with this method, you must clean up the in-tree build objects before you decide to do the right thing and move on to the superior `./build` Buildroot method: - -.... -cd "kernel_module" -./make-host.sh clean -.... - -otherwise they will cause problems. - -=== Hello host - -Minimal host build system sanity check example. - -.... -cd hello_host -make -insmod hello.ko -dmesg -rmmod hello.ko -dmesg -.... - == Conversation === kmod diff --git a/build b/build index 4474698..1d4950b 100755 --- a/build +++ b/build @@ -171,11 +171,7 @@ BR2_ROOTFS_POST_SCRIPT_ARGS=\"${post_script_args}\" make O="$buildroot_out_dir" olddefconfig fi -mkdir -p \ - "$gem5_out_dir" \ - "$qemu_out_dir" \ - "$p9_dir" \ -; +common_mkdir cd "$buildroot_dir" # HOST_QEMU_OPTS is a hack that happens to work because the QEMU package luckly uses += at all times. # It shouldn't be necessary in the first place: https://bugs.busybox.net/show_bug.cgi?id=9936 diff --git a/common b/common index 086d63f..6e054a5 100644 --- a/common +++ b/common @@ -62,6 +62,14 @@ set_common_vars() { common_trace_txt_file="${common_out_run_dir}/trace.txt" fi } +common_mkdir() ( + mkdir -p \ + "$build_dir" \ + "$gem5_out_dir" \ + "$qemu_out_dir" \ + "$p9_dir" \ + ; +) root_dir="$(pwd)" out_dir="${root_dir}/out" common_bench_boot="${out_dir}/bench-boot.txt" diff --git a/run b/run index 0ddc28b..8f3a6ca 100755 --- a/run +++ b/run @@ -25,6 +25,7 @@ initrd=false initramfs=false memory=256M nographic=true +prebuilt=false root= tmux=false tmux_args= @@ -33,7 +34,7 @@ trace_enabled=false # just to prevent QEMU from emitting a warning that '' is not valid. trace_type=pr_manager_run vnc= -while getopts a:c:DdE:e:F:f:G:ghIiKkm:T:U:uVx OPT; do +while getopts a:c:DdE:e:F:f:G:ghIiKkm:PT:U:uVx OPT; do case "$OPT" in a) arch="$OPTARG" @@ -93,6 +94,9 @@ while getopts a:c:DdE:e:F:f:G:ghIiKkm:T:U:uVx OPT; do m) memory="$OPTARG" ;; + P) + prebuilt=true + ;; T) trace_enabled=true trace_type="$OPTARG" @@ -205,10 +209,16 @@ else if "$kvm"; then extra_flags="${extra_flags} -enable-kvm" fi + if "$prebuilt"; then + common_mkdir + qemu_executable="qemu-system-${arch}" + else + qemu_executable="${buildroot_out_dir}/host/usr/bin/qemu-system-${arch}" + fi extra_flags="${extra_flags_qemu} ${extra_flags}" qemu_common="\ ${debug_vm} \ -'${buildroot_out_dir}/host/usr/bin/qemu-system-${arch}' \\ +${qemu_executable} \\ -device rtl8139,netdev=net0 \\ -gdb tcp::1234 \\ -m '${memory}' \\ @@ -234,18 +244,23 @@ ${vnc}" root='root=/dev/vda' fi fi - # The base QEMU commands are found under board/qemu/*/readme.tx case "$arch" in x86_64) if "$kgdb"; then extra_append="${extra_append} kgdboc=ttyS0,115200" fi + if "$prebuilt"; then + custom_devices= + else + custom_devices="-device lkmc_pci_min \\ +" + fi cmd="\ ${qemu_common} \ -M pc \\ -append '${root} nopat ${extra_append}' \\ -device edu \\ --device lkmc_pci_min \\ +${custom_devices} \\ -kernel '${images_dir}/bzImage' \\ ${extra_flags} \ " diff --git a/run-usage.adoc b/run-usage.adoc index 8b4ceb9..8171a2a 100644 --- a/run-usage.adoc +++ b/run-usage.adoc @@ -40,6 +40,7 @@ The default is the minimum amount that boots all archs without extra options added. Anything lower will lead some arch to fail to boot. Any +|`-P` | |Run the downloaded prebuilt images. |`-T` |`TRACE_TYPES` |Set trace events to be enabled. If not given, gem5 tracing is completely disabled, while QEMU tracing is enabled but uses default traces that are very rare and don't affect diff --git a/zip-img b/zip-img index 30f8eb7..a4bb013 100755 --- a/zip-img +++ b/zip-img @@ -1,10 +1,6 @@ #!/usr/bin/env bash set -eu -# Ideally should be obtained from common. -# But the paths there are absolute, and get recorded by in zip. -# and generating relative paths seems hard: -# https://stackoverflow.com/questions/2564634/convert-absolute-path-into-relative-path-given-a-current-directory-using-bash -outfile="out/images-$(git log -1 --format="%H").zip" +outfile="out/out.zip" rm -f "$outfile" for arch in x86_64 arm aarch64; do img_dir="out/${arch}/buildroot/images"