mirror of
https://github.com/cirosantilli/linux-kernel-module-cheat.git
synced 2026-01-23 02:05:57 +01:00
prebuilt: allow running prebuilt images
This commit is contained in:
Ciro Santilli
1
.gitignore
vendored
1
.gitignore
vendored
@@ -7,5 +7,6 @@ gitignore*
|
||||
|
||||
# Specific files.
|
||||
/data
|
||||
/images-*.zip
|
||||
/out
|
||||
/out.*
|
||||
|
||||
212
README.adoc
212
README.adoc
@@ -15,9 +15,11 @@ toc::[]
|
||||
|
||||
== Getting started
|
||||
|
||||
=== Getting started Ubuntu
|
||||
=== Getting started natively
|
||||
|
||||
This is the most native setup, and therefore the best one if you are on one of the supported Ubuntu: 16.04 or 18.04.
|
||||
This is the best one if you are on one of the supported Ubuntu: 16.04 or 18.04.
|
||||
|
||||
It will likely work on other Linux distros if you install the required packages, but this is not currently well tested, but patches are welcome. You can also try <<docker>> if you are on other distros.
|
||||
|
||||
Reserve 12Gb of disk and run:
|
||||
|
||||
@@ -27,9 +29,16 @@ cd linux-kernel-module-cheat
|
||||
./configure && ./build && ./run
|
||||
....
|
||||
|
||||
It is also trivial to build for different supported <<cpu-architecture,CPU architectures>>.
|
||||
|
||||
The first configure will take a while (30 minutes to 2 hours) to clone and build, see <<benchmark-builds>> for more details.
|
||||
|
||||
If you don't want to wait, you could also try to compile the examples and run them on your host computer as explained on at <<run-kernel-modules-on-host>>, but as explained on that section, that is dangerous, limited, and will likely not work.
|
||||
If you don't want to wait, you could also try the following faster but much more limited methods:
|
||||
|
||||
* <<prebuilt>>
|
||||
* <<host>>
|
||||
|
||||
but you will soon find that they are simply not enough if you anywhere near serious about systems programming.
|
||||
|
||||
After QEMU opens up, you can start playing with the kernel modules:
|
||||
|
||||
@@ -108,9 +117,9 @@ We will try to support the following Ubuntu versions at least:
|
||||
* otherwise, support both latest LTS and the latest non-LTS
|
||||
|
||||
[[docker]]
|
||||
=== Getting started Docker
|
||||
=== Getting started with Docker
|
||||
|
||||
This is a good option if you are on a Linux host, but the <<getting-started-ubuntu,native build>> failed due to your weird host distribution.
|
||||
This is a good option if you are on a Linux host, but the <<getting-started-natively,native build>> failed due to your weird host distribution.
|
||||
|
||||
Note however that most things in this repository are highly Linux-portable, should just work once you have found the corresponding `apt-get` package manager commands in the link:configure[] for your distro. In theory :-)
|
||||
|
||||
@@ -214,6 +223,114 @@ After this, to start using Docker again will you need another:
|
||||
./rundocker setup
|
||||
....
|
||||
|
||||
[[prebuilt]]
|
||||
=== Getting started with prebuilts
|
||||
|
||||
We don't currently provide a full prebuilt because it would be too big to host freely.
|
||||
|
||||
However, if you just want to quickly try out running our kernel modules, try this:
|
||||
|
||||
. Download QEMU and this repo:
|
||||
+
|
||||
....
|
||||
sudo apt-get install qemu-system-x86
|
||||
git clone https://github.com/cirosantilli/linux-kernel-module-cheat
|
||||
cd linux-kernel-module-cheat
|
||||
....
|
||||
. go to the latest release link:https://github.com/cirosantilli/linux-kernel-module-cheat/releases[], download the `images-*.zip` file and extract it:
|
||||
+
|
||||
....
|
||||
unzip images-*.zip
|
||||
....
|
||||
+
|
||||
It is link:https://stackoverflow.com/questions/24987542/is-there-a-link-to-github-for-downloading-a-file-in-the-latest-release-of-a-repo/50540591#50540591[not possible to automate this step without the API], and I'm not venturing there this time
|
||||
. run with the `-P` option:
|
||||
+
|
||||
....
|
||||
./run -P
|
||||
....
|
||||
|
||||
Limitations of this method:
|
||||
|
||||
* can't GDB step debug the kernel, since the source and cross toolchain with GDB are not available. Buildroot cannot easily use a host toolchain: <<prebuilt-toolchain>>.
|
||||
+
|
||||
Maybe we could work around this by just downloading the kernel source somehow, and using a host prebuilt GDB, but we felt that it would be too messy and unreliable.
|
||||
* can't create new modules or modify the existing ones, since no cross toolchain
|
||||
* can't use things that rely on our QEMU fork, e.g. in-fork <<hardware-models>> or <<tracing>>
|
||||
* you won't get the latest version of this repository. Our <<travis>> attempt to automate builds failed, and storing a release for every commit would likely make GitHub mad at us.
|
||||
* <<gem5>> is not currently supported, although it should not be too hard to do. One annoyance is that there is no Debian package for it, so you have to compile your own, so you might as well just build the image itself.
|
||||
|
||||
[[host]]
|
||||
=== Getting started on host
|
||||
|
||||
This method runs the kernel modules directly on your host computer without a VM, and saves you the compilation time and disk usage of the virtual machine method.
|
||||
|
||||
It has however severe limitations, and you will soon see that the compilation time and disk usage are well worth it:
|
||||
|
||||
* can't control which kernel version and build options to use. So some of the modules will likely not compile because of kernel API changes, since https://stackoverflow.com/questions/37098482/how-to-build-a-linux-kernel-module-so-that-it-is-compatible-with-all-kernel-rele/45429681#45429681[the Linux kernel does not have a stable kernel module API].
|
||||
* bugs can easily break you system. E.g.:
|
||||
** segfaults can trivially lead to a kernel crash, and require a reboot
|
||||
** your disk could get erased. Yes, this can also happen with `sudo` from userland. But you should not use `sudo` when developing newbie programs. And for the kernel you don't have the choice not to use `sudo`.
|
||||
** even more subtle system corruption such as https://unix.stackexchange.com/questions/78858/cannot-remove-or-reinsert-kernel-module-after-error-while-inserting-it-without-r[not being able to rmmod]
|
||||
* can't control which hardware is used, notably the CPU architecture
|
||||
* can't step debug it with <<gdb,GDB>> easily. The alternatives are JTAG or <<kgdb>>, but those are less reliable, and JTAG requires extra hardware.
|
||||
|
||||
Still interested?
|
||||
|
||||
....
|
||||
cd kernel_module
|
||||
./make-host.sh
|
||||
....
|
||||
|
||||
If the compilation of any of the C files fails because of kernel or toolchain differences that we don't control on the host, just rename it to remove the `.c` extension and try again:
|
||||
|
||||
....
|
||||
mv broken.c broken.c~
|
||||
./build_host
|
||||
....
|
||||
|
||||
Once you manage to compile, and have come to terms with the fact that this may blow up your host, try it out with:
|
||||
|
||||
....
|
||||
sudo insmod hello.ko
|
||||
|
||||
# Our module is there.
|
||||
sudo lsmod | grep hello
|
||||
|
||||
# Last message should be: hello init
|
||||
dmest -T
|
||||
|
||||
sudo rmmod hello
|
||||
|
||||
# Last message should be: hello exit
|
||||
dmesg -T
|
||||
|
||||
# Not present anymore
|
||||
sudo lsmod | grep hello
|
||||
....
|
||||
|
||||
Once you are done with this method, you must clean up the in-tree build objects before you decide to do the right thing and move on to the superior `./build` Buildroot method:
|
||||
|
||||
....
|
||||
cd "kernel_module"
|
||||
./make-host.sh clean
|
||||
....
|
||||
|
||||
otherwise they will cause problems.
|
||||
|
||||
==== Hello host
|
||||
|
||||
Minimal host build system example:
|
||||
|
||||
....
|
||||
cd hello_host
|
||||
make
|
||||
insmod hello.ko
|
||||
dmesg
|
||||
rmmod hello.ko
|
||||
dmesg
|
||||
....
|
||||
|
||||
=== Text mode
|
||||
|
||||
By default, we show the serial console directly on the current terminal, without opening a QEMU window.
|
||||
@@ -6294,9 +6411,7 @@ Finally, do a clone of the relevant repository out of tree and reproduce the bug
|
||||
|
||||
In this section document how benchmark builds and runs of this repo, and how to investigate what the bottleneck is.
|
||||
|
||||
Ideally, we should setup an automated build server that benchmarks those things continuously for us.
|
||||
|
||||
We tried to automate it on Travis with link:.travis.yml[] but it hits the current 50 minute job timeout: https://travis-ci.org/cirosantilli/linux-kernel-module-cheat/builds/296454523 And I bet it would likely hit a disk maxout either way if it went on.
|
||||
Ideally, we should setup an automated build server that benchmarks those things continuously for us, but our <<travis>> attempt failed.
|
||||
|
||||
So currently, we are running benchmarks manually when it seems reasonable and uploading them to: https://github.com/cirosantilli/linux-kernel-module-cheat-regression
|
||||
|
||||
@@ -6308,6 +6423,10 @@ Run all benchmarks and upload the results:
|
||||
./bench-all -A
|
||||
....
|
||||
|
||||
=== Travis
|
||||
|
||||
We tried to automate it on Travis with link:.travis.yml[] but it hits the current 50 minute job timeout: https://travis-ci.org/cirosantilli/linux-kernel-module-cheat/builds/296454523 And I bet it would likely hit a disk maxout either way if it went on.
|
||||
|
||||
=== Benchmark this repo benchmarks
|
||||
|
||||
==== Benchmark Linux kernel boot
|
||||
@@ -6427,8 +6546,6 @@ Our philosophy is:
|
||||
* try to keep the toolchain (GCC, Binutils) unchanged, otherwise a full rebuild is required.
|
||||
+
|
||||
So we generally just enable all toolchain options by default, even though this adds a bit of time to the build.
|
||||
+
|
||||
The biggest build time hog is always GCC, and it does not look like we can use a precompiled one: https://stackoverflow.com/questions/10833672/buildroot-environment-with-host-toolchain
|
||||
* if something is very valuable, we just add it by default even if it increases the Build time, notably GDB and QEMU
|
||||
* runtime is sacred.
|
||||
+
|
||||
@@ -6441,6 +6558,11 @@ We do our best to reduce the instruction and feature count to the bare minimum n
|
||||
+
|
||||
One possibility we could play with is to build loadable modules instead of built-in modules to reduce runtime, but make it easier to get started with the modules.
|
||||
|
||||
[[prebuilt-toolchain]]
|
||||
====== Buildroot use prebuilt host toolchain
|
||||
|
||||
The biggest build time hog is always GCC, and it does not look like we can use a precompiled one: https://stackoverflow.com/questions/10833672/buildroot-environment-with-host-toolchain
|
||||
|
||||
===== Benchmark Buildroot build baseline
|
||||
|
||||
This is the minimal build we could expect to get away with.
|
||||
@@ -6525,76 +6647,6 @@ gem5:
|
||||
** https://stackoverflow.com/questions/47997565/gem5-system-requirements-for-decent-performance/48941793#48941793
|
||||
** https://github.com/gem5/gem5/issues/25
|
||||
|
||||
== Run kernel modules on host
|
||||
|
||||
This method runs the kernel modules directly on your host computer without a VM, and saves you the compilation time and disk usage of the virtual machine method.
|
||||
|
||||
It has however severe limitations, and you will soon see that the compilation time and disk usage are well worth it:
|
||||
|
||||
* can't control which kernel version and build options to use. So some of the modules will likely not compile because of kernel API changes, since https://stackoverflow.com/questions/37098482/how-to-build-a-linux-kernel-module-so-that-it-is-compatible-with-all-kernel-rele/45429681#45429681[the Linux kernel does not have a stable kernel module API].
|
||||
* bugs can easily break you system. E.g.:
|
||||
** segfaults can trivially lead to a kernel crash, and require a reboot
|
||||
** your disk could get erased. Yes, this can also happen with `sudo` from userland. But you should not use `sudo` when developing newbie programs. And for the kernel you don't have the choice not to use `sudo`.
|
||||
** even more subtle system corruption such as https://unix.stackexchange.com/questions/78858/cannot-remove-or-reinsert-kernel-module-after-error-while-inserting-it-without-r[not being able to rmmod]
|
||||
* can't control which hardware is used, notably the CPU architecture
|
||||
* can't step debug it with <<gdb,GDB>> easily. The alternatives are JTAG or <<kgdb>>, but those are less reliable, and JTAG requires extra hardware.
|
||||
|
||||
Still interested?
|
||||
|
||||
....
|
||||
cd kernel_module
|
||||
./make-host.sh
|
||||
....
|
||||
|
||||
If the compilation of any of the C files fails because of kernel or toolchain differences that we don't control on the host, just rename it to remove the `.c` extension and try again:
|
||||
|
||||
....
|
||||
mv broken.c broken.c~
|
||||
./build_host
|
||||
....
|
||||
|
||||
Once you manage to compile, and have come to terms with the fact that this may blow up your host, try it out with:
|
||||
|
||||
....
|
||||
sudo insmod hello.ko
|
||||
|
||||
# Our module is there.
|
||||
sudo lsmod | grep hello
|
||||
|
||||
# Last message should be: hello init
|
||||
dmest -T
|
||||
|
||||
sudo rmmod hello
|
||||
|
||||
# Last message should be: hello exit
|
||||
dmesg -T
|
||||
|
||||
# Not present anymore
|
||||
sudo lsmod | grep hello
|
||||
....
|
||||
|
||||
Once you are done with this method, you must clean up the in-tree build objects before you decide to do the right thing and move on to the superior `./build` Buildroot method:
|
||||
|
||||
....
|
||||
cd "kernel_module"
|
||||
./make-host.sh clean
|
||||
....
|
||||
|
||||
otherwise they will cause problems.
|
||||
|
||||
=== Hello host
|
||||
|
||||
Minimal host build system sanity check example.
|
||||
|
||||
....
|
||||
cd hello_host
|
||||
make
|
||||
insmod hello.ko
|
||||
dmesg
|
||||
rmmod hello.ko
|
||||
dmesg
|
||||
....
|
||||
|
||||
== Conversation
|
||||
|
||||
=== kmod
|
||||
|
||||
6
build
6
build
@@ -171,11 +171,7 @@ BR2_ROOTFS_POST_SCRIPT_ARGS=\"${post_script_args}\"
|
||||
make O="$buildroot_out_dir" olddefconfig
|
||||
fi
|
||||
|
||||
mkdir -p \
|
||||
"$gem5_out_dir" \
|
||||
"$qemu_out_dir" \
|
||||
"$p9_dir" \
|
||||
;
|
||||
common_mkdir
|
||||
cd "$buildroot_dir"
|
||||
# HOST_QEMU_OPTS is a hack that happens to work because the QEMU package luckly uses += at all times.
|
||||
# It shouldn't be necessary in the first place: https://bugs.busybox.net/show_bug.cgi?id=9936
|
||||
|
||||
8
common
8
common
@@ -62,6 +62,14 @@ set_common_vars() {
|
||||
common_trace_txt_file="${common_out_run_dir}/trace.txt"
|
||||
fi
|
||||
}
|
||||
common_mkdir() (
|
||||
mkdir -p \
|
||||
"$build_dir" \
|
||||
"$gem5_out_dir" \
|
||||
"$qemu_out_dir" \
|
||||
"$p9_dir" \
|
||||
;
|
||||
)
|
||||
root_dir="$(pwd)"
|
||||
out_dir="${root_dir}/out"
|
||||
common_bench_boot="${out_dir}/bench-boot.txt"
|
||||
|
||||
23
run
23
run
@@ -25,6 +25,7 @@ initrd=false
|
||||
initramfs=false
|
||||
memory=256M
|
||||
nographic=true
|
||||
prebuilt=false
|
||||
root=
|
||||
tmux=false
|
||||
tmux_args=
|
||||
@@ -33,7 +34,7 @@ trace_enabled=false
|
||||
# just to prevent QEMU from emitting a warning that '' is not valid.
|
||||
trace_type=pr_manager_run
|
||||
vnc=
|
||||
while getopts a:c:DdE:e:F:f:G:ghIiKkm:T:U:uVx OPT; do
|
||||
while getopts a:c:DdE:e:F:f:G:ghIiKkm:PT:U:uVx OPT; do
|
||||
case "$OPT" in
|
||||
a)
|
||||
arch="$OPTARG"
|
||||
@@ -93,6 +94,9 @@ while getopts a:c:DdE:e:F:f:G:ghIiKkm:T:U:uVx OPT; do
|
||||
m)
|
||||
memory="$OPTARG"
|
||||
;;
|
||||
P)
|
||||
prebuilt=true
|
||||
;;
|
||||
T)
|
||||
trace_enabled=true
|
||||
trace_type="$OPTARG"
|
||||
@@ -205,10 +209,16 @@ else
|
||||
if "$kvm"; then
|
||||
extra_flags="${extra_flags} -enable-kvm"
|
||||
fi
|
||||
if "$prebuilt"; then
|
||||
common_mkdir
|
||||
qemu_executable="qemu-system-${arch}"
|
||||
else
|
||||
qemu_executable="${buildroot_out_dir}/host/usr/bin/qemu-system-${arch}"
|
||||
fi
|
||||
extra_flags="${extra_flags_qemu} ${extra_flags}"
|
||||
qemu_common="\
|
||||
${debug_vm} \
|
||||
'${buildroot_out_dir}/host/usr/bin/qemu-system-${arch}' \\
|
||||
${qemu_executable} \\
|
||||
-device rtl8139,netdev=net0 \\
|
||||
-gdb tcp::1234 \\
|
||||
-m '${memory}' \\
|
||||
@@ -234,18 +244,23 @@ ${vnc}"
|
||||
root='root=/dev/vda'
|
||||
fi
|
||||
fi
|
||||
# The base QEMU commands are found under board/qemu/*/readme.tx
|
||||
case "$arch" in
|
||||
x86_64)
|
||||
if "$kgdb"; then
|
||||
extra_append="${extra_append} kgdboc=ttyS0,115200"
|
||||
fi
|
||||
if "$prebuilt"; then
|
||||
custom_devices=
|
||||
else
|
||||
custom_devices="-device lkmc_pci_min \\
|
||||
"
|
||||
fi
|
||||
cmd="\
|
||||
${qemu_common} \
|
||||
-M pc \\
|
||||
-append '${root} nopat ${extra_append}' \\
|
||||
-device edu \\
|
||||
-device lkmc_pci_min \\
|
||||
${custom_devices} \\
|
||||
-kernel '${images_dir}/bzImage' \\
|
||||
${extra_flags} \
|
||||
"
|
||||
|
||||
@@ -40,6 +40,7 @@
|
||||
The default is the minimum amount that boots all archs without extra
|
||||
options added. Anything lower will lead some arch to fail to boot.
|
||||
Any
|
||||
|`-P` | |Run the downloaded prebuilt images.
|
||||
|`-T` |`TRACE_TYPES` |Set trace events to be enabled.
|
||||
If not given, gem5 tracing is completely disabled, while QEMU tracing
|
||||
is enabled but uses default traces that are very rare and don't affect
|
||||
|
||||
6
zip-img
6
zip-img
@@ -1,10 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
set -eu
|
||||
# Ideally should be obtained from common.
|
||||
# But the paths there are absolute, and get recorded by in zip.
|
||||
# and generating relative paths seems hard:
|
||||
# https://stackoverflow.com/questions/2564634/convert-absolute-path-into-relative-path-given-a-current-directory-using-bash
|
||||
outfile="out/images-$(git log -1 --format="%H").zip"
|
||||
outfile="out/out.zip"
|
||||
rm -f "$outfile"
|
||||
for arch in x86_64 arm aarch64; do
|
||||
img_dir="out/${arch}/buildroot/images"
|
||||
|
||||
Reference in New Issue
Block a user