The gem5 tests require building statically with build id static, see also: Section 10.7, “gem5 syscall emulation mode”. TODO automate this better.
gem5 run benchmark: how to run a benchmark in gem5 full system, including how to boot Linux, checkpoint and restore to skip the boot on a fast CPU
m5out directory: understand the output files that gem5 produces, which contain information about your run
m5ops: magic guest instructions used to control gem5
+Add new files to the Buildroot image: how to add your own files to the image if you have a benchmark that we don’t already support out of the box (also send a pull request!)
The examples that work include most C examples that don’t rely on complicated syscalls such as threads, and almost all the Userland assembly examples.
The exact list of userland programs that work in baremetal is specified in path_properties with the baremetal property, but you can also easily find it out with a baremetal test dry run:
The exact list of userland programs that work in baremetal is specified in path_properties.py with the baremetal property, but you can also easily find it out with a baremetal test dry run:
The gem5 tests require building statically with build id static, see also: Section 10.7, “gem5 syscall emulation mode”. TODO automate this better.
See: Section 33.14, “Test this repo” for more useful testing tips.
+See: Section 33.15, “Test this repo” for more useful testing tips.
modules built with Buildroot, see: Section 33.13.2.1, “kernel_modules buildroot package”
+modules built with Buildroot, see: Section 33.14.2.1, “kernel_modules buildroot package”
modules built from the kernel tree itself, see: Section 15.12.2, “dummy-irq”
@@ -9785,7 +9797,7 @@ git log | grep -E ' Linux [0-9]+\.' | headThis also makes this repo the perfect setup to develop the Linux kernel.
In case something breaks while updating the Linux kernel, you can try to bisect it to understand the root cause, see: Section 33.15, “Bisection”.
+In case something breaks while updating the Linux kernel, you can try to bisect it to understand the root cause, see: Section 33.16, “Bisection”.
First, use use the branching procedure described at: Section 33.17, “Update a forked submodule”
Because the kernel is so central to this repository, almost all tests must be re-run, so basically just follow the full testing procedure described at: Section 33.14, “Test this repo”. The only tests that can be skipped are essentially the Baremetal tests.
+Because the kernel is so central to this repository, almost all tests must be re-run, so basically just follow the full testing procedure described at: Section 33.15, “Test this repo”. The only tests that can be skipped are essentially the Baremetal tests.
Before comitting, don’t forget to update:
@@ -21999,7 +22011,60 @@ make menuconfigThere are basically two choices:
+create a Buildroot package: Add new Buildroot packages
+drop your files directly in rootfs_overlay and follow instructions from that section
+If you need to cross compile input files such as C for the guest, then Buildroot packages are definitely the cleaner option as they make cross compilation easy.
+However, for a quick initial prototype, it should be fine to just manually compile your files and drop them in rootfs_overlay.
+Ideally, you should still use the Buildroot cross compiler for this which ensures compatibility.
+The best way to do that is to use either run-toolchain or getvar.
+In case you can’t for some reason, e.g. if you need to use your own custom toolchain, you should:
+make sure that you have built your toolchain to match the our kernel version. It often just works even if they are not perfectly matched however, partly because the Linux kernel is highly backwards compatible
+build statically with -static to avoid binary compatibility issues with our own glibc
Related threads:
+First, see if you can’t get away without actually adding a new package, for example:
If none of those methods are flexible enough for you, you can just fork or hack up buildroot_packages/sample_package the sample package to do what you want.
For how to use that package, see: Section 33.13.2, “buildroot_packages directory”.
+For how to use that package, see: Section 33.14.2, “buildroot_packages directory”.
Then iterate trying to do what you want and reading the manual until it works: https://buildroot.org/downloads/manual/manual.html
We already even had a C SVE test in-tree, but it was disabled because the old toolchain does not support it.
So once the new GCC 8 toolchain was built, we can first enable that test by editing the path_properties file to not skip C SVE tests anymore:
+So once the new GCC 8 toolchain was built, we can first enable that test by editing the path_properties.py file to not skip C SVE tests anymore:
In baremetal, we detect if tests failed by parsing logs for the Magic failure string.
See: Section 33.14, “Test this repo” for more useful testing tips.
+See: Section 33.15, “Test this repo” for more useful testing tips.
The getvar helper script can print the values of internal LKMC variables.
+Within our Python scripts such as common.py, those variable are visible as self.env[<var>].
For example, to find the Buildroot output directory for an aarch64 build, you could use:
./getvar --arch aarch64 buildroot_build_dir+
which as of LKMC b15a0e455d691afa49f3b813ad9b09394dfb02b7 outputs:
+/path/to/linux-kernel-module-cheat/out/buildroot/build/default/aarch64+
You can also list all available variables in one go with just:
+./getvar+
Using getvar makes it possible to make Bash scripts more portable if for example directory structure changes across LKMC versions.
+For this reason, we use it in particular often in this README to reduce the need for refactoring.
+While you could just manually find/learn the path to toolchain tools, e.g. in LKMC b15a0e455d691afa49f3b813ad9b09394dfb02b7 they are
+./out/buildroot/build/default/aarch64/host/bin/aarch64-buildroot-linux-gnu-gcc userland/c/hello.c +./out/buildroot/build/default/aarch64/host/bin/aarch64-buildroot-linux-gnu-objdump -D a.out+
you can save some typing and get portability across directory structure changes with our run-toolchain helper:
+./run-toolchain --arch aarch64 gcc -- userland/c/hello.c +./run-toolchain --arch aarch64 objdump -- -D a.out+
Alternatively, if you just need a variable to feed into your own Build system, you can also use getvar:
+./getvar --arch aarch64 toolchain_prefix+
which outputs as of LKMC b15a0e455d691afa49f3b813ad9b09394dfb02b7:
+/path/to/linux-kernel-module-cheat/out/buildroot/build/default/aarch64/host/usr/bin/aarch64-buildroot-linux-gnu+
It is not possible to rebuild the root filesystem while running QEMU because QEMU holds the file qcow2 file:
When doing long simulations sweeping across multiple system parameters, it becomes fundamental to do multiple simulations in parallel.
To run multiple gem5 checkouts, see: Section 33.12.3.1, “gem5 worktree”.
+To run multiple gem5 checkouts, see: Section 33.13.3.1, “gem5 worktree”.
Implementation note: we create multiple namespaces for two things:
@@ -34088,7 +34230,7 @@ less "$(./getvar --arch aarch64 --emulator gem5 --run-id 1 termout_file)"It often happens that you are comparing two versions of the build, a good and a bad one, and trying to figure out why the bad one is bad.
Our build variants system allows you to keep multiple built versions of all major components, so that you can easily switching between running one or the other.
If you want to keep two builds around, one for the latest Linux version, and the other for Linux v4.16:
To run both kernels simultaneously, one on each QEMU instance, see: Section 33.11, “Simultaneous runs”.
+To run both kernels simultaneously, one on each QEMU instance, see: Section 33.12, “Simultaneous runs”.
Analogous to the Linux kernel build variants but with the --qemu-build-id option instead:
Analogous to the Linux kernel build variants but with the --gem5-build-id option instead:
Therefore, you can’t forget to checkout to the sources to that of the corresponding build before running, unless you explicitly tell gem5 to use a non-default source tree with gem5 worktree. This becomes inevitable when you want to launch multiple simultaneous runs at different checkouts.
--gem5-build-id goes a long way, but if you want to seamlessly switch between two gem5 tress without checking out multiple times, then --gem5-worktree is for you.
Suppose that you are working on a private fork of gem5, but you want to use this repository to develop it as well.
Allows you to have multiple versions of the GCC toolchain or root filesystem.
lkmc/ contains sources and headers that are shared across kernel modules, userland and baremetal examples.
Another option would have been to name it as includes/lkmc, but that would make paths longer, and we might want to store source code in that directory as well in the future.
When factoring out functionality across userland examples, there are two main options:
Source: buildroot_packages/.
A custom build script can give you more flexibility: e.g. the package can be made work with other root filesystems more easily, have better 9P support, and rebuild faster as it evades some Buildroot boilerplate.
Has the following structure:
Patches in this directory are never applied automatically: it is up to users to manually apply them before usage following the instructions in this documentation.
Source: rootfs_overlay.
This way you can just hack away the scripts and try them out immediately without any further operations.
This path can be found with:
LKMC first collects all the files that it will dump into the guest there, and then in the very last step dumps everything into the final image.
In Buildroot, this is done by pointing BR2_ROOTFS_OVERLAY to that directory.
In Buildroot, this is done by pointing BR2_ROOTFS_OVERLAY to that directory, which is documented at: https://buildroot.org/downloads/manual/manual.html#rootfs-custom
This does not include native image modification mechanisms such as Buildroot packages, which we let Buildroot itself manage.
@@ -34578,7 +34720,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -The files:
Print out several parameters that normally change randomly from boot to boot:
-./run --eval-after './linux/rand_check.out;./linux/poweroff.out'-
Source: userland/linux/rand_check.c
-This can be used to check the determinism of:
-lkmc_home refers to the target base directory in which we put all our custom built stuff, such as userland executables and kernel modules.
Whenever a relative path is used inside a guest sample command, e.g. insmod hello.ko or ./hello.out, it means that the path lives in lkmc_home unless stated otherwise.
In order to build and run each userland and baremetal example properly, we need per-file metadata such as compiler flags and required number of cores.
+This data is stored is stored in path_properties.py at path_properties_tuples.
Maybe we should embed it magically into source files directories to make it easier to see? But one big Python dict was easier to implement so we started like this. And it allows factoring chunks out easily.
+The format is as follows:
+'path_component': (
+ {'property': value},
+ {
+ 'child_path_component':
+ {
+ {'child_property': },
+ {}
+ }
+ }
+)
+and as a shortcut, paths that don’t have any children can be written directly as:
+'path_component': {'property': value}
+Properties of parent directories apply to all children.
+Lists coming from parent directories are extended instead of overwritten by children, this is especially useful for C compiler flags.
+To quickly determine which properties a path has, you can use getprops, e.g.:
+./getprops userland/c/hello.c+
which outputs values such as:
+allowed_archs=None +allowed_emulators=None +arm_aarch32=False +arm_sve=False +baremetal=True+
Print out several parameters that normally change randomly from boot to boot:
+./run --eval-after './linux/rand_check.out;./linux/poweroff.out'+
Source: userland/linux/rand_check.c
+This can be used to check the determinism of:
+Run almost all tests:
test does not all possible tests, because there are too many possible variations and that would take forever. The rationale is the same as for ./build all and is explained in ./build --help.
You can select multiple archs and emulators of interest, as for an other command, with:
By default, continue running even after the first failure happens, and they show a summary at the end.
TODO: we really need a mechanism to automatically generate the test list automatically e.g. based on path_properties, currently there are many tests missing, and we have to add everything manually which is very annoying.
+TODO: we really need a mechanism to automatically generate the test list automatically e.g. based on path_properties.py, currently there are many tests missing, and we have to add everything manually which is very annoying.
We could just generate it on the fly on the host, and forward it to guest through CLI arguments.
@@ -34803,7 +35008,7 @@ echo $?We have some pexpect automated tests for GDB for both userland and baremetal programs!
We do not know of any way to set the emulator exit status in QEMU arm full system.
For the Linux kernel, do the following manual tests for now.
You should also test that the Internet works:
build-userland and test-executables have a wide variety of target selection modes, and it was hard to keep them all working without some tests:
When updating the Linux kernel, QEMU and gem5, things sometimes break.
In order to build and run each userland and baremetal example properly, we need per-file metadata such as compiler flags and required number of cores.
-This data is stored is stored in path_properties.py at path_properties_tuples.
Maybe we should embed it magically into source files directories to make it easier to see? But one big Python dict was easier to implement so we started like this. And it allows factoring chunks out easily.
-The format is as follows:
-'path_component': (
- {'property': value},
- {
- 'child_path_component':
- {
- {'child_property': },
- {}
- }
- }
-)
-and as a shortcut, paths that don’t have any children can be written directly as:
-'path_component': {'property': value}
-Properties of parent directories apply to all children.
-Lists coming from parent directories are extended instead of overwritten by children, this is especially useful for C compiler flags.
-To quickly determine which properties a path has, you can use getprops, e.g.:
-./getprops userland/c/hello.c-
which outputs values such as:
-allowed_archs=None -allowed_emulators=None -arm_aarch32=False -arm_sve=False -baremetal=True-
This is a template update procedure for submodules for which we have some patches on on top of mainline.