Linux Kernel Module Cheat

This setup has been mostly tested on Ubuntu. For other host operating systems see: Section 29.1, “Supported hosts”. For greater stability, consider using the latest release instead of master: https://github.com/cirosantilli/linux-kernel-module-cheat/releases

Reserve 12Gb of disk and run:

You don’t need to clone recursively even though we have .git submodules: download-dependencies fetches just the submodules that you need for this build to save time.

If something goes wrong, see: Section 29.2, “Common build issues” and use our issue tracker: https://github.com/cirosantilli/linux-kernel-module-cheat/issues

The initial build will take a while (30 minutes to 2 hours) to clone and build, see Benchmark builds for more details.

If you don’t want to wait, you could also try the following faster but much more limited methods:

but you will soon find that they are simply not enough if you anywhere near serious about systems programming.

After ./run, QEMU opens up leaving you in the /lkmc/ directory, and you can start playing with the kernel modules inside the simulated system:

This should print to the screen:

which are printk messages from init and cleanup methods of those modules.

Sources:

Quit QEMU with:

See also: Section 13.1.1, “Quit QEMU from text mode”.

All available modules can be found in the kernel_modules directory.

It is super easy to build for different CPU architectures, just use the --arch option:

To avoid typing --arch aarch64 many times, you can set the default arch as explained at: Section 29.4, “Default command line arguments”

I now urge you to read the following sections which contain widely applicable information:

Once you use GDB step debug and tmux, your terminal will look a bit like this:

Besides a seamless initial build, this project also aims to make it effortless to modify and rebuild several major components of the system, to serve as an awesome development setup.

This is our reference setup, and the best supported one, use it unless you have good reason not to.

It was historically the first one we did, and all sections have been tested with this setup unless explicitly noted.

Read the following sections for further introductory material:

This setup is like the QEMU Buildroot setup, but it uses gem5 instead of QEMU as a system simulator.

QEMU tries to run as fast as possible and give correct results at the end, but it does not tell us how many CPU cycles it takes to do something, just the number of instructions it ran. This kind of simulation is known as functional simulation.

The number of instructions executed is a very poor estimator of performance because in modern computers, a lot of time is spent waiting for memory requests rather than the instructions themselves.

gem5 on the other hand, can simulate the system in more detail than QEMU, including:

and can therefore be used to estimate system performance, see: Section 18.2, “gem5 run benchmark” for an example.

The downside of gem5 much slower than QEMU because of the greater simulation detail.

See gem5 vs QEMU for a more thorough comparison.

For the most part, if you just add the --emulator gem5 option or *-gem5 suffix to all commands and everything should magically work.

If you haven’t built Buildroot yet for QEMU Buildroot setup, you can build from the beginning with:

If you have already built previously, don’t be afraid: gem5 and QEMU use almost the same root filesystem and kernel, so ./build will be fast.

Remember that the gem5 boot is considerably slower than QEMU since the simulation is more detailed.

To get a terminal, either open a new shell and run:

You can quit the shell without killing gem5 by typing tilde followed by a period:

If you are inside tmux, which I highly recommend, you can both run gem5 stdout and open the guest terminal on a split window with:

See also: Section 2.3.1, “tmux gem5”.

At the end of boot, it might not be very clear that you have the shell since some printk messages may appear in front of the prompt like this:

but if you look closely, the PS1 prompt marker # is there already, just hit enter and a clear prompt line will appear.

If you forgot to open the shell and gem5 exit, you can inspect the terminal output post-mortem at:

More gem5 information is present at: Section 18, “gem5”

Good next steps are:

This setup uses prebuilt binaries that we upload to GitHub from time to time.

We don’t currently provide a full prebuilt because it would be too big to host freely, notably because of the cross toolchain.

Our prebuilts currently include:

For more details, see our our release procedure.

Advantage of this setup: saves time and disk space on the initial install, which is expensive in largely due to building the toolchain.

The limitations are severe however:

This setup might be good enough for those developing simulators, as that requires less image modification. But once again, if you are serious about this, why not just let your computer build the full featured setup while you take a coffee or a nap? :-)

Checkout to the latest tag and use the Ubuntu packaged QEMU to boot Linux:

You have to checkout to the latest tag to ensure that the scripts match the release format: https://stackoverflow.com/questions/1404796/how-to-get-the-latest-tag-name-in-current-branch-in-git

This is known not to work for aarch64 on an Ubuntu 16.04 host with QEMU 2.5.0, presumably because QEMU is too old, the terminal does not show any output. I haven’t investigated why.

Or to run a baremetal example instead:

Be saner and use our custom built QEMU instead:

This also allows you to modify QEMU if you’re into that sort of thing.

To build the kernel modules as in Your first kernel module hack do:

TODO: for now the only way to test those modules out without building Buildroot is with 9p, since we currently rely on Buildroot to manipulate the root filesystem.

Command explanation:

To modify the Linux kernel, build and use it as usual:

Minimal host build system example:

In order to test the kernel and emulators, userland content in the form of executables and scripts is of course required, and we store it mostly under:

When we started this repository, it only contained content that interacted very closely with the kernel, or that had required performance analysis.

However, we soon started to notice that this had an increasing overlap with other userland test repositories: we were duplicating build and test infrastructure and even some examples.

Therefore, we decided to consolidate other userland tutorials that we had scattered around into this repository.

Notable userland content included / moving into this repository includes:

There are several ways to run our Userland content, notably:

This setup does not use the Linux kernel nor Buildroot at all: it just runs your very own minimal OS.

x86_64 is not currently supported, only arm and aarch64: I had made some x86 bare metal examples at: https://github.com/cirosantilli/x86-bare-metal-examples but I’m lazy to port them here now. Pull requests are welcome.

The main reason this setup is included in this project, despite the word "Linux" being on the project name, is that a lot of the emulator boilerplate can be reused for both use cases.

This setup allows you to make a tiny OS and that runs just a few instructions, use it to fully control the CPU to better understand the simulators for example, or develop your own OS if you are into that.

You can also use C and a subset of the C standard library because we enable Newlib by default. See also: https://electronics.stackexchange.com/questions/223929/c-standard-libraries-on-bare-metal/400077#400077

Our C bare-metal compiler is built with crosstool-NG. If you have already built Buildroot previously, you will end up with two GCCs installed. Unfortunately I don’t see a solution for this, since we need separate toolchains for Newlib on baremetal and glibc on Linux: https://stackoverflow.com/questions/38956680/difference-between-arm-none-eabi-and-arm-linux-gnueabi/38989869#38989869

Every .c file inside baremetal/ and .S file inside baremetal/arch/<arch>/ generates a separate baremetal image.

For example, to run baremetal/arch/aarch64/dump_regs.c in QEMU do:

And the terminal prints the values of certain system registers. This example prints registers that are only accessible from EL1 or higher, and thus could not be run in userland.

In addition to the examples under baremetal/, several of the userland examples can also be run in baremetal! This is largely due to the awesomeness of Newlib.

The examples that work include most C examples that don’t rely on complicated syscalls such as threads, and almost all the Userland assembly examples.

The exact list of userland programs that work in baremetal is specified in path_properties with the baremetal property, but you can also easily find it out with a baremetal test dry run:

For example, we can run the C hello world userland/c/hello.c simply as:

and that outputs to the serial port the string:

which QEMU shows on the host terminal.

To modify a baremetal program, simply edit the file, e.g.

and rebuild:

./build qemu-baremetal that we run previously is only needed for the initial build. That script calls build-baremetal for us, in addition to building prerequisites such as QEMU and crosstool-NG.

./build-baremetal uses crosstool-NG, and so it must be preceded by build-crosstool-ng, which ./build qemu-baremetal also calls.

Now let’s run userland/arch/aarch64/add.S:

This time, the terminal does not print anything, which indicates success: if you look into the source, you will see that we just have an assertion there.

You can see a sample assertion fail in userland/c/assert_fail.c:

and the terminal contains:

and the exit status of our script is 1:

You can run all the baremetal examples in one go and check that all assertions passed with:

To use gem5 instead of QEMU do:

and then as usual open a shell with:

Or as usual, tmux users can do both in one go with:

TODO: the carriage returns are a bit different than in QEMU, see: Section 26.4, “gem5 baremetal carriage return”.

Note that ./build-baremetal requires the --emulator gem5 option, and generates separate executable images for both, as can be seen from:

This is unlike the Linux kernel that has a single image for both QEMU and gem5:

The reason for that is that on baremetal we don’t parse the device tress from memory like the Linux kernel does, which tells the kernel for example the UART address, and many other system parameters.

gem5 also supports the RealViewPBX machine, which represents an older hardware compared to the default VExpress_GEM5_V1:

This generates yet new separate images with new magic constants:

But just stick to newer and better VExpress_GEM5_V1 unless you have a good reason to use RealViewPBX.

When doing baremetal programming, it is likely that you will want to learn userland assembly first, see: Section 21, “Userland assembly”.

For more information on baremetal, see the section: Section 26, “Baremetal”.

The following subjects are particularly important:

Just don’t forget to pass --arch to ./run-gdb, e.g.:

and:

https://stackoverflow.com/questions/29151235/how-to-de-optimize-the-linux-kernel-to-and-compile-it-with-o0

O=0 is an impossible dream, O=2 being the default.

So get ready for some weird jumps, and <value optimized out> fun. Why, Linux, why.

If you are using gem5 instead of QEMU, --tmux has a different effect by default: it opens the gem5 terminal instead of the debugger:

To open a new pane with GDB instead of the terminal, use:

which is equivalent to:

--tmux-program implies --tmux, so we can just write:

If you also want to see both GDB and the terminal with gem5, then you will need to open a separate shell manually as usual with ./gem5-shell.

From inside tmux, you can create new terminals on a new window with Ctrl-B C split a pane yet again vertically with Ctrl-B % or horizontally with Ctrl-B ".

TODO on arm 51e31cdc2933a774c2a0dc62664ad8acec1d2dbe it does not always work, and lx-symbols fails with the message:

Can’t reproduce on x86_64 and aarch64 are fine.

It is kind of random: if you just insmod manually and then immediately ./run-gdb --arch arm, then it usually works.

But this fails most of the time: shell 1:

shell 2:

then hit Ctrl-C on shell 2, and voila.

Then:

says that the load address is:

so it is close to the failing 0xbf0000cc.

readelf:

does not give any interesting hits at cc, no symbol was placed that far.

TODO find a more convenient method. We have working methods, but they are not ideal.

This is not very easy, since by the time the module finishes loading, and lx-symbols can work properly, module_init has already finished running!

Possibly asked at:

Useless, but a good way to show how hardcore you are. Disable lx-symbols with:

From inside guest:

as mentioned at:

This will give a line of form:

And then tell GDB where the module was loaded with:

Alternatively, if the module panics before you can read /proc/modules, there is a pr_debug which shows the load address:

And then search for a line of type:

Tested on 4f4749148273c282e80b58c59db1b47049e190bf + 1.

One possibility is to run:

and then find the second address (the first one does not work, already too late maybe):

and break there:

but TODO: it does not show the source assembly under arch/arm: https://stackoverflow.com/questions/11423784/qemu-arm-linux-kernel-boot-debug-no-source-code

I also tried to hack run-gdb with:

and no I do have the symbols from arch/arm/boot/compressed/vmlinux', but the breaks still don’t work.

This is the userland debug setup most likely to work, since at init time there is only one userland executable running.

For executables from the userland/ directory such as userland/posix/count.c:

Then, as soon as boot ends, we are left inside a debug session that looks just like what gdbserver would produce.

BusyBox custom init process:

This follows BusyBox' convention of calling the main for each executable as <exec>_main since the busybox executable has many "mains".

BusyBox default init process:

init cannot be debugged with gdbserver without modifying the source, or else /sbin/init exits early with:

Non-init process:

This is the least reliable setup as there might be other processes that use the given virtual address.

List all processes:

Sample output:

The second and third fields are obviously PID and process name.

The first one is more interesting, and contains the address of the task_struct in memory.

This can be confirmed with:

which contains the correct PID for all threads I’ve tried:

TODO get the PC of the kthreads: https://stackoverflow.com/questions/26030910/find-program-counter-of-process-in-kernel Then we would be able to see where the threads are stopped in the code!

On ARM, I tried:

but task_pt_regs is a #define and GDB cannot see defines without -ggdb3: https://stackoverflow.com/questions/2934006/how-do-i-print-a-defined-constant-in-gdb which are apparently not set?

Bibliography:

This error means that the GDB server, e.g. in QEMU, sent more registers than the GDB client expected.

This can happen for the following reasons:

The XML target description format is described a bit further at: https://stackoverflow.com/questions/46415059/how-to-observe-aarch64-system-registers-in-qemu/53043044#53043044

You can also use KDB directly from the graphic window with:

This setup could be used to debug the kernel on machines without serial, such as modern desktops.

This works because --graphics adds kbd (which stands for KeyBoarD!) to kgdboc.

TODO neither arm and aarch64 are working as of 1cd1e58b023791606498ca509256cc48e95e4f5b + 1.

arm seems to place and hit the breakpoint correctly, but no matter how many go commands I do, the count.sh stdout simply does not show.

aarch64 seems to place the breakpoint correctly, but after the first go the kernel oopses with warning:

and stack trace:

My theory is that every serious ARM developer has JTAG, and no one ever tests this, and the kernel code is just broken.

This example illustrates how reading from the x86 control registers with mov crX, rax can only be done from kernel land on ring0.

From kernel land:

works and output the registers, for example:

However if we try to do it from userland:

stdout gives:

and dmesg outputs:

Sources:

In both cases, we attempt to run the exact same code which is shared on the ring0.h header file.

Bibliography:

TODO Can you run arm executables in the aarch64 guest? https://stackoverflow.com/questions/22460589/armv8-running-legacy-32-bit-applications-on-64-bit-os/51466709#51466709

I’ve tried:

but it fails with:

Just using BusyBox' poweroff at the end of the init does not work and the kernel panics:

because BusyBox' poweroff tries to do some fancy stuff like killing init, likely to allow userland to shutdown nicely.

But this fails when we are init itself!

BusyBox' poweroff works more brutally and effectively if you add -f:

but why not just use our minimal ./linux/poweroff.out and be done with it?

Source: userland/linux/poweroff.c

This also illustrates how to shutdown the computer from C: https://stackoverflow.com/questions/28812514/how-to-shutdown-linux-using-c-or-qt-without-call-to-system

I dare you to guess what this does:

Source: userland/posix/sleep_forever.c

This executable is a convenient simple init that does not panic and sleeps instead.

Get a reasonable answer to "how long does boot take in guest time?":

Source: userland/linux/time_boot.c

That executable writes to dmesg directly through /dev/kmsg a message of type:

which tells us that boot took 2.188242 seconds based on the dmesg timestamp.

Bibliography: https://stackoverflow.com/questions/12683169/measure-time-taken-for-linux-kernel-from-bootup-to-userpace/46517014#46517014

The annoying dash - gets passed as a parameter to init, which makes it impossible to use this method for most non custom executables.

Arguments with dots that come after - are still treated specially (of the form subsystem.somevalue) and disappear, from args, e.g.:

outputs:

so see how a.b is gone.

The simple workaround is to just create a shell script that does it, e.g. as we’ve done at: rootfs_overlay/lkmc/gem5_exit.sh.

Wait, where do HOME and TERM come from? (greps the kernel). Ah, OK, the kernel sets those by default: https://github.com/torvalds/linux/blob/94710cac0ef4ee177a63b5227664b38c95bbf703/init/main.c#L173

On top of the Linux kernel, the BusyBox /bin/sh shell will also define other variables.

We can explore the shenanigans that the shell adds on top of the Linux kernel with:

From there we observe that:

gives:

therefore adding SHLVL and PWD to the default kernel exported variables.

Furthermore, to increase confusion, if you list all non-exported shell variables https://askubuntu.com/questions/275965/how-to-list-all-variables-names-and-their-current-values with:

then it shows more variables, notably:

See: Section 7.3, “rootfs”

It’s nice when the obvious just works, right?

and on another shell:

Or alternatively, if you are using tmux, do everything in one go with:

To stop at the very first instruction of a freestanding program, just use --no-continue. A good example of this is shown at: Section 21.5.1, “Freestanding programs”.

glibc has a check for kernel version, likely obtained from the uname syscall, and if the kernel is not new enough, it quits.

Both gem5 and QEMU however allow setting the reported uname version from the command line, which we do to always match our toolchain.

QEMU by default copies the host uname value, but we always override it in our scripts.

Determining the right number to use for the kernel version is of course highly non-trivial and would require an extensive userland test suite, which most emulator don’t have.

Source: userland/posix/uname.c.

The QEMU source that does this is at: https://github.com/qemu/qemu/blob/v3.1.0/linux-user/syscall.c#L8931

Bibliography:

The ID is just hardcoded on the source:

For some reason QEMU / glibc x86_64 picks up the host libc, which breaks things.

Other archs work as they different host libc is skipped. User mode static executables also work.

We have worked around this with with https://bugs.launchpad.net/qemu/+bug/1701798/comments/12 from the thread: https://bugs.launchpad.net/qemu/+bug/1701798 by creating the file: rootfs_overlay/etc/ld.so.cache which is a symlink to a file that cannot exist: /dev/null/nonexistent.

Reproduction:

Outcome:

To get things working again, restore ld.so.cache with:

I’ve also tested on an Ubuntu 16.04 guest and the failure is different one:

A non-QEMU-specific example of stack smashing is shown at: https://stackoverflow.com/questions/1345670/stack-smashing-detected/51897264#51897264

Tested at: 2e32389ebf1bedd89c682aa7b8fe42c3c0cf96e5 + 1.

One limitation of static executables is that Buildroot mostly only builds dynamic versions of libraries (the libc is an exception).

So programs that rely on those libraries might not compile as GCC can’t find the .a version of the library.

For example, if we try to build BLAS statically:

it fails with:

g++ and pthreads also causes issues: https://stackoverflow.com/questions/35116327/when-g-static-link-pthread-cause-segmentation-fault-why

As a consequence, the following fails:

with error:

and if we manually build and run natively on host it segfaults.

If we hack the compilation command to do instead:

then it works. We should automate that at some point.

As of gem5 7fa4c946386e7207ad5859e8ade0bbfc14000d91, the crappy se.py script does not forward the exit status of syscall emulation mode, you can test it with:

Source: userland/c/false.c.

Then manually run the generated gem5 CLI, and do:

and the output is always 0.

Instead, it just outputs a message to stdout just like for m5 fail:

which we parse in run and then exit with the correct result ourselves…​

Related thread: https://stackoverflow.com/questions/56032347/is-there-a-way-to-identify-if-gem5-run-got-over-successfully

gem5 shows its own stdout to terminal, and does not allow you to type stdin to programs.

Instead, you must pass stdin non-interactively with the through a file with the --se.py --input option, e.g.:

leads to gem5 output:

Source: userland/c/getchar.c

Let’s see if user mode runs considerably faster than full system or not.

First we build Dhrystone manually statically since dynamic linking is broken in gem5 as explained at: Section 10.6, “gem5 syscall emulation mode”.

gem5 user mode:

gem5 full system:

QEMU user mode:

QEMU full system:

Result on P51 at bad30f513c46c1b0995d3a10c0d9bc2a33dc4fa0:

At 8d8307ac0710164701f6e14c99a69ee172ccbb70 + 1, I noticed that if you run userland/posix/count.c:

it first waits for 3 seconds, then the program exits, and then it dumps all the stdout at once, instead of counting once every second as expected.

The same can be reproduced by copying the raw QEMU command and piping it through tee, so I don’t think it is a bug in our setup:

TODO: investigate further and then possibly post on QEMU mailing list.

Name of a predecessor set of tools.

kmod’s modprobe can also load modules under different names to avoid conflicts, e.g.:

https://superuser.com/questions/1087859/how-to-quit-the-qemu-monitor-when-not-using-a-gui

However, our QEMU setup captures Ctrl + C and other common signals and sends them to the guest, which makes it hard to quit QEMU for the first time since there is no GUI either.

The simplest way to quit QEMU, is to do:

Alternative methods include:

Scroll up in QEMU graphic mode:

but I never managed to increase that buffer:

The superior alternative is to use text mode and GNU screen or tmux.

For aarch64 we also need to configure the kernel with linux_config/display:

This is because the gem5 aarch64 defconfig does not enable HDLCD like the 32 bit one arm one for some reason.

TODO get working. There is an unmerged patchset at: https://gem5-review.googlesource.com/c/public/gem5/+/11036/1

The DP650 is a newer display hardware than HDLCD. TODO is its interface publicly documented anywhere? Since it has a gem5 model and in-tree Linux kernel support, that information cannot be secret?

The key option to enable support in Linux is DRM_MALI_DISPLAY=y which we enable at linux_config/display.

Build the kernel exactly as for Graphic mode gem5 aarch64 and then run with:

We cannot use mainline Linux because the gem5 arm Linux kernel patches are required at least to provide the CONFIG_DRM_VIRT_ENCODER option.

gem5 emulates the HDLCD ARM Holdings hardware for arm and aarch64.

The kernel uses HDLCD to implement the DRM interface, the required kernel config options are present at: linux_config/display.

TODO: minimize out the --custom-config-file. If we just remove it on arm: it does not work with a failing dmesg:

So what other options are missing from gem5_defconfig? It would be cool to minimize it out to better understand the options.

TODO 9076c1d9bcc13b6efdb8ef502274f846d8d4e6a1 I’m 100% sure that it was working before, but I didn’t run it forever, and it stopped working at some point. Needs bisection, on whatever commit last touched x11 stuff.

-show-cursor did not help, I just get to see the host cursor, but the guest cursor still does not move.

Doing:

shows that interrupts do happen when mouse and keyboard presses are done, so I expect that it is some wrong either with:

/var/log/Xorg.0.log contains the following interesting lines:

The file /dev/inputs/mice does not exist.

Note that our current link:kernel_confi_fragment sets:

for gem5, so you might want to remove those lines to debug this.

On ARM, startx hangs at a message:

and nothing shows on the screen, and:

says:

A friend told me this but I haven’t tried it yet:

First Enable networking.

Then in the host, start a server:

And then in the guest, find the IP we need to hit with:

which gives:

so we use in the guest:

Bibliography: https://serverfault.com/questions/769874/how-to-forward-a-port-from-guest-to-host-in-qemu-kvm/951835#951835

All of 9P and NFS (and sshfs) allow sharing directories between guest and host.

Advantages of 9P

Advantages of NFS:

As usual, we have already set everything up for you. On host:

Guest:

Host:

The main ingredients for this are:

Bibliography:

TODO seems possible! Lets do it:

TODO: get working.

9P is better with emulation, but let’s just get this working for fun.

First make sure that this works: Section 14.3.2, “Guest to host networking”.

Then, build the kernel with NFS support:

Now on host:

Now edit /etc/exports to contain:

and restart the server:

Now on guest:

TODO: failing with:

And now the /tmp directory from host is not mounted on guest!

If you don’t want to start the NFS server after the next boot automatically so save resources, do:

To modify a single option on top of our default kernel configs, do:

Kernel modules depend on certain kernel configs, and therefore in general you might have to clean and rebuild the kernel modules after changing the kernel config:

and then proceed as in Your first kernel module hack.

You might often get way without rebuilding the kernel modules however.

To use an extra kernel config fragment file on top of our defaults, do:

To use just your own exact .config instead of our defaults ones, use:

There is also a shortcut --custom-config-file to use the gem5 arm Linux kernel patches.

The following options can all be used together, sorted by decreasing config setting power precedence:

To do a clean menu config yourself and use that for the build, do:

But remember that every new build re-configures the kernel by default, so to keep your configs you will need to use on further builds:

So what you likely want to do instead is to save that as a new defconfig and use it later as:

You can also use other config generating targets such as defconfig with the same method as shown at: Section 15.1.3.1.1, “Linux kernel defconfig”.

Get the build config in guest:

or with our shortcut:

or to conveniently grep for a specific option case insensitively:

Source: rootfs_overlay/lkmc/conf.sh.

This is enabled by:

From host:

Just for fun https://stackoverflow.com/questions/14958192/how-to-get-the-config-from-a-linux-kernel-image/14958263#14958263:

although this can be useful when someone gives you a random image.

By default, build-linux generates a .config that is a mixture of:

To find out which kernel configs are being used exactly, simply run:

and look for the merge_config.sh call. This script from the Linux kernel tree, as the name suggests, merges multiple configuration files into one as explained at: https://unix.stackexchange.com/questions/224887/how-to-script-make-menuconfig-to-automate-linux-kernel-build-configuration/450407#450407

For each arch, the base of our configs are named as:

e.g.: linux_config/buildroot-x86_64.

These configs are extracted directly from a Buildroot build with update-buildroot-kernel-configs.

Note that Buildroot can sed override some of the configurations, e.g. it forces CONFIG_BLK_DEV_INITRD=y when BR2_TARGET_ROOTFS_CPIO is on. For this reason, those configs are not simply copy pasted from Buildroot files, but rather from a Buildroot kernel build, and then minimized with make savedefconfig: https://stackoverflow.com/questions/27899104/how-to-create-a-defconfig-file-from-a-config

On top of those, we add the following by default:

We try to use the latest possible kernel major release version.

In QEMU:

or in the source:

During update all you kernel modules may break since the kernel API is not stable.

They are usually trivial breaks of things moving around headers or to sub-structs.

The userland, however, should simply not break, as Linus enforces strict backwards compatibility of userland interfaces.

This backwards compatibility is just awesome, it makes getting and running the latest master painless.

This also makes this repo the perfect setup to develop the Linux kernel.

In case something breaks while updating the Linux kernel, you can try to bisect it to understand the root cause, see: Section 29.14, “Bisection”.

The kernel is not forward compatible, however, so downgrading the Linux kernel requires downgrading the userland too to the latest Buildroot branch that supports it.

The default Linux kernel version is bumped in Buildroot with commit messages of type:

So you can try:

Those commits change BR2_LINUX_KERNEL_LATEST_VERSION in /linux/Config.in.

You should then look up if there is a branch that supports that kernel. Staying on branches is a good idea as they will get backports, in particular ones that fix the build as newer host versions come out.

Finally, after downgrading Buildroot, if something does not work, you might also have to make some changes to how this repo uses Buildroot, as the Buildroot configuration options might have changed.

We don’t expect those changes to be very difficult. A good way to approach the task is to:

Then, if you spot an option that is wrong, some grepping in this repo should quickly point you to the code you need to modify.

It also possible that you will need to apply some patches from newer Buildroot versions for it to build, due to incompatibilities with the host Ubuntu packages and that Buildroot version. Just read the error message, and try:

Successful port reports:

Double quotes can be used to escape spaces as in opt="a b", but double quotes themselves cannot be escaped, e.g. opt"a\"b"

This even lead us to use base64 encoding with --eval!

There are two methods:

core_param suggests how they are different:

By default, the Linux kernel mounts the root filesystem as readonly. TODO rationale?

This cannot be observed in the default BusyBox init, because by default our rootfs_overlay/etc/inittab does:

Analogously, Ubuntu 18.04 does in its fstab something like:

which uses default mount rw flags.

We have however removed those setups init setups to keep things more minimal, and replaced them with the rw kernel boot parameter makes the root mounted as writable.

To observe the default readonly behaviour, hack the run script to remove replace init, and then run on a raw shell:

Now try to do:

which fails with:

We can also observe the read-onlyness with:

which contains:

and so it is Read Only as shown by ro.

Disable userland address space randomization. Test it out by running rand_check.out twice:

If we remove it from our run script by hacking it up, the addresses shown by linux/rand_check.out vary across boots.

Equivalent to:

enables all log levels, and is basically the same as:

except that you don’t need to know what is the maximum level.

https://stackoverflow.com/questions/28936199/why-is-pr-debug-of-the-linux-kernel-not-giving-any-output/49835405#49835405

Debug messages are not printable by default without recompiling.

But the awesome CONFIG_DYNAMIC_DEBUG=y option which we enable by default allows us to do:

and we have a shortcut at:

Source: rootfs_overlay/lkmc/pr_debug.sh.

Syntax: https://www.kernel.org/doc/html/v4.11/admin-guide/dynamic-debug-howto.html

Wildcards are also accepted, e.g. enable all messages from all files:

TODO: why is this not working:

Enable messages in specific modules:

Source: kernel_modules/myprintk.c

This outputs the pr_debug message:

but TODO: it also shows debug messages even without enabling them explicitly:

and it shows as enabled:

Enable pr_debug for boot messages as well, before we can reach userland and write to /proc:

Get ready for the noisiest boot ever, I think it overflows the printk buffer and funny things happen.

The Linux kernel allows passing module parameters at insertion time through the init_module and finit_module system calls.

The insmod tool exposes that as:

Parameters are declared in the module as:

Automated test:

Outcome: the test passes:

Sources:

As shown in the example, module parameters can also be read and modified at runtime from sysfs.

We can obtain the help text of the parameters with:

The output contains:

One module can depend on symbols of another module that are exported with EXPORT_SYMBOL:

Outcome: the test passes:

Sources:

The kernel deduces dependencies based on the EXPORT_SYMBOL that each module uses.

Symbols exported by EXPORT_SYMBOL can be seen with:

sample output:

This requires CONFIG_KALLSYMS_ALL=y.

Dependency information is stored by the kernel module build system in the .ko files' MODULE_INFO, e.g.:

contains:

We can double check with:

The output contains:

Module dependencies are also stored at:

Output: