Make this repo good enough to move in cpp-cheat, x86-assembly-cheat and arm-assembly-cheat in

This commit is a large squash, the full development branch is available at: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/asm This notably means a refactor of the userland build and testing, to support: - improved assembly infrastructure unified across arm and x86 - native in-tree build and test helpers - parallel building and testing, which implies thread_pool.py - selection of what to build and test from the CLI - path_properties.py to indicate how to build and run different examples - in full system, move all userland stuff into /lkmc - prefix everything that we defined across files with LKMC - --gdb uber convenient helper - remove import imp which was deprecated Full commit messages from the branch follow: 1: userland: add assembly support Move arm assembly cheat here, and start some work on x86 cheat as well. 2: document userland asm syscall interfaces 3: userland assembly: structure readme 4: x86 fail works 5: asm: more links 6: userland: add ported to all archs 7: move all our stuff into /lkmc in guest Motivation: userland is getting several new subdirectories, it would be too insane to just dump all of that in the guest root filesystem. To alleviate the cd pain, .profile puts user inside /lkmc by default. 8: start the big userland migration 9: migrate all 10: bak 11: build-userland-in-tree is now a Python command ./build calls it, we did this to allow --download-dependencies to work perfectly. 12: rename include to lkmc 13: mtops.h is perfect now 14: userland: make build perfect 15: preparing test_user_mode, need to generalize stuff as usual 16: asm: prefix every linux specific with linux/ 17: userland: maybe it really works 18: userland: fix kernel version to work on older ubuntu Expose --kernel-version to allow customization. Update LTP info. 19: userland: build really truly working now userland test: start work, in a working state, but no features 20: test-user-mode: make perfect like build-userland Multithreading and target selection. 21: userland: get a bit closer to perfection 22: thread_pool: support passing thread IDs Then use that to fix gem5 error log read race. 23: userland: native testing 24: userland: path properties getting nice! 25: userland: move posix/environ from cpp-cheat 26: gem5: --debug-flags without =, looks nicer whenever it can be done 27: run: rename --wait-gdb in --gdb-wait, --gdb prefix might become a thing 28: run: create --tmux-program gdb to open gem5 GDB 29: run: create the uber convenient --gdb option 30: userland: move getchar from cpp-cheat 31: prebuilt: kernel boot aarch64 does not work on Ubuntu 16.04 32: userland: x86_64 linux hello world make PIE 33: userland: try to make userland executable selection saner Only allow existing files to be built, stop extension expansion madness. cli_function: get_cli print booleans properly, was printing without --no- for negations. 34: userland: only link to lkmc.o if needed 35: path_properties: make data very compact with only tuples and dicts Spend 2 hours of my life thinking about low value tree walks ;-) 36: userland: move more userland/arch/ logic into property tree 37: userland: make libs work Working for build, but now test-user-mode-in-tree is not using --in-tree, TODO fix later on. 38: userland: make libs really work 39: userland: document path_properties 40: userland: classify linux 41: waste your life 42: common: fix absolute path runs --gdb: allow running from arbitrary directory 43: baremetal: arm allow using floating point instructions 44: baremetal: stat preparing to make perfect like userland/ 45: run: fix image check logic accounting for userland Was failing if I try to run userland (with abspath) when out/ directory is not present. 46: cli-function: raise if the config file is given and does not exist 47: common: define missing 'ld' variable, this broke m5 build 48: rum: --qemu-which host now works for user mode as well as system Don't fall back on host QEMU automatically, too much insanity. 49: userland: refix silly mistakes 50: userland: use path_properties flags for all builds, including lkmc. and userland/arch/main.c Without this in particular, --gdb fails on assembly because main.c was not being built with -ggdb3. 51: userland: start refactor to show failing values on failure! aarch64 basically done, but missing: - other archs - maybe convert main.c into C++ to use templates? - full review of ASSERT_EQ calling convention issues not seen by tests by chance - documentation 52: readme: releases are more stable... 53: submodules: sort gitmodules 54: test-baremetal: same interface as test-user-mode In particular, runs tests in parallel, and allows selecting given tests 55: baremetal: allow arbitrary exit status with the magic string test-baremetal: fix missing setting x0 return value Examples were just returning on ret without setting x0, which led to failures... those were not noticed because of how broken the testing system was ;-) 56: baremetal: ah, actually nope, it didn't work :-( Workaround for now. Works on asserts, but not on exit 1. Some other day, maybe. https://github.com/cirosantilli/linux-kernel-module-cheat/issues/59 57: panic on panic: improve behaviour description 58: baremetal: get exit status working with on_exit :-) 59: baremetal: implement C assert 60: test-baremetal: remove commented out exit status workaround 61: test-user-mode: handle exit status for signals. Fix #61. 62: aarch64: fix ASSERT_EQ_REG tests on gem5 Was doing an 8-byte aligned store, which gem5 dislikes. But the ARMARM says bad things may happen there, notably a signal: "D1.8.2 SP alignment checking" so gem5 is not really too wrong, QEMU just happens to work by chance. 63: userland assembly: build empty.S and fail.S to toplevel and run fail.S with path_properties exit_status They were just duplicating stuff needlessly while we don't support non-native in-tree builds, which leads to executable conflicts for C file anyways. 64: gem5: use a single build tree for all build types gem5 already has different object names for each build type it seems, so let's just make sure that works and save some disk space. 65: userland x86_64: ASSERT_EQ show actual and expected values 66: assert_fail.c: add to readme index 67: userland x86_64: implement ASSERT_MEMCMP 68: userland x86_64: allow ASSERT_EQ to take just about anything 69: gas data sizes 70: gas_data_sizes.S: make PIE for all ISAs 71: x86: paddq 72: x86 paddq: test entire family 73: Get rid of imp, started giving deprecation warning every time in Python 3.7 in Ubuntu 19.04. Please python stop torturing me with refactors. Make ./run -u blow up if executable not found, otherwise I go crazy. Get ./test-gdb back to life after the ./run relative path refactor, forgot to test this. 74: fix run-toolchain, qemu-monitor, trace-boot, trace2line, bisect-linux-boot-gem5. Fixes part of #63 I'm sad no one reported qemu-monitor break, that one is kind of important. count.out arguments broke it as an init program, since the kernel adds trash parameters to every init. Is anyone using this repo, I wonder? Keep pushing, keep pushing. One day it gets good enough, and the whole world will see. 75: x86 assembly: addpd 76: Fix import_path circular dependency by splitting it out. Use import thread_pool instead from, from is evil. Fix poweroff.out path for ./trace-boot. 77: run: rename cryptic tmu to tmux-split, ./run is good now so I never use it anymore explicitly 78: assembly SIMD add: make uniform for all ISAs, mark as entry point to learning SIMD 79: start moving arm-assembly-cheat readme in here 80: arm assembly: move some more in 81: move more arm in 82: userland: attempt to fix all assembly example links to README 83: assembly: improve organization of simd add 84: ld2 move in 85: Make userland / assembly getting started more uniform / visible Forward --gcc-which to ./run --tmux. Use gdb-multiarch for --gcc-which host. 86: userland: disable PIE explicitly on command line for all executables 87: userland: make userland content a better landing page 88: build: check git version from --version and degrade gracefully 89: build: make --dry-run work again on all 90: import_path: importlib explicit for Ubuntu 16.04 91: make all submodules point to my forks git servers are insane, submodule implementation is crap, what can you do 92: build: log warning on git too old for --update 93: build-linux: do olddefconfig even if no fragments In particular, gem5 kernel 4.15 needs it 94: userland content: improve a bit landing page for cpp-cheat
2026-01-22 17:55:57 +01:00 · 2019-05-16 00:00:00 +00:00
parent 7a5ca339a3
commit d1003f1cb2
388 changed files with 9972 additions and 3432 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -1,9 +1,10 @@
 # Extensions and prefixes.
 *.tmp
 tmp.*
+*.tmp.*
 *~
-?*.gitignore
-gitignore*
+*.gitignore
+gitignore.*

 # Specific files.
 /data
@@ -22,6 +23,9 @@ __pycache__
 *.o
 *.out

+# Data to be plotted output.
+*.dat
+
 # Kernel modules.
 .cache.mk
 *.ko.cmd
--- a/.gitmodules
+++ b/.gitmodules
@@ -3,19 +3,28 @@
 	url = git://git.kernel.org/pub/scm/linux/kernel/git/mark/boot-wrapper-aarch64.git
 [submodule "submodules/binutils-gdb"]
 	path = submodules/binutils-gdb
+    # url = git://sourceware.org/git/binutils-gdb.git
 	url = https://github.com/cirosantilli/binutils-gdb
 [submodule "submodules/buildroot"]
 	path = submodules/buildroot
+    # url = git://git.busybox.net/buildroot
 	url = https://github.com/cirosantilli/buildroot
 	ignore = dirty
 [submodule "submodules/crosstool-ng"]
 	path = submodules/crosstool-ng
-	url = https://github.com/crosstool-ng/crosstool-ng
+    # url = https://github.com/crosstool-ng/crosstool-ng
+	url = https://github.com/cirosantilli/crosstool-ng
+[submodule "submodules/gcc"]
+	path = submodules/gcc
+    # url = git://gcc.gnu.org/git/gcc.git
+	url = https://github.com/cirosantilli/gcc
 [submodule "submodules/gem5"]
 	path = submodules/gem5
-	url = https://gem5.googlesource.com/public/gem5
+    # url = https://gem5.googlesource.com/public/gem5
+	url = https://github.com/cirosantilli/gem5
 [submodule "submodules/glibc"]
 	path = submodules/glibc
+    # url = git://sourceware.org/git/glibc.git
 	url = https://github.com/cirosantilli/glibc
 # The true upstream does not accept git submodule update --init --depth 1
 # git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
@@ -23,16 +32,16 @@
 # https://unix.stackexchange.com/questions/338578/linux-kernel-source-code-size-difference
 [submodule "submodules/linux"]
 	path = submodules/linux
+    # usl = git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
 	url = https://github.com/cirosantilli/linux
 [submodule "submodules/parsec-benchmark"]
 	path = submodules/parsec-benchmark
 	url = https://github.com/cirosantilli/parsec-benchmark
 [submodule "submodules/qemu"]
 	path = submodules/qemu
+    # url = https://github.com/qemu/qemu
 	url = https://github.com/cirosantilli/qemu
 [submodule "submodules/xen"]
 	path = submodules/xen
-	url = git://xenbits.xen.org/xen.git
-[submodule "submodules/gcc"]
-	path = submodules/gcc
-	url = https://github.com/cirosantilli/gcc
+    # url = git://xenbits.xen.org/xen.git
+	url = https://github.com/cirosantilli/xen
--- a/README.adoc
+++ b/README.adoc
--- a/baremetal/add.c
+++ b/baremetal/add.c
@@ -1,13 +0,0 @@
-#include <lkmc.h>
-
-int main(void) {
-    int i, j, k;
-    i = 1;
-    /* test-gdb-op1 */
-    j = 2;
-    /* test-gdb-op2 */
-    k = i + j;
-    /* test-gdb-result */
-    if (k != 3)
-        lkmc_assert_fail();
-}
--- a/baremetal/add.c
+++ b/baremetal/add.c
@@ -0,0 +1 @@
+../lkmc/add.c
--- a/baremetal/add.py
+++ b/baremetal/add.py
@@ -1,9 +0,0 @@
-def test(self):
-    self.sendline('tbreak main')
-    self.sendline('continue')
-    self.continue_to('op1')
-    assert self.get_int('i') == 1
-    self.continue_to('op2')
-    assert self.get_int('j') == 2
-    self.continue_to('result')
-    assert self.get_int('k') == 3
--- a/baremetal/add.py
+++ b/baremetal/add.py
@@ -0,0 +1 @@
+../lkmc/add.py
--- a/baremetal/arch/aarch64/add.S
+++ b/baremetal/arch/aarch64/add.S
@@ -1,12 +1,13 @@
 .global main
 main:
    /* 1 + 2 == 3 */
-    mov x0, #1
+    mov x0, 1
    /* test-gdb-op1 */
-    add x1, x0, #2
+    add x1, x0, 2
    /* test-gdb-result */
-    cmp x1, #3
+    cmp x1, 3
    beq 1f
    bl lkmc_assert_fail
 1:
+    mov x0, 0
    ret
--- a/baremetal/arch/aarch64/c_from_as.S
+++ b/baremetal/arch/aarch64/c_from_as.S
@@ -1,5 +1,5 @@
 /* Call a C function. */
 .global main
 main:
-    mov x0, #0
+    mov x0, 0
    bl exit
--- a/baremetal/arch/aarch64/fadd.S
+++ b/baremetal/arch/aarch64/fadd.S
@@ -3,43 +3,45 @@
 .global main
 main:
    /* 1.5 + 2.5 == 4.0 */
-    fmov d0, #1.5
+    fmov d0, 1.5
    /* test-gdb-d0 */
-    fmov d1, #2.5
+    fmov d1, 2.5
    /* test-gdb-d1 */
    fadd d2, d0, d1
    /* test-gdb-d2 */
-    fmov d3, #4.0
+    fmov d3, 4.0
    fcmp d2, d3
    beq 1f
    bl lkmc_assert_fail
 1:

    /* Now in 32-bit. */
-    fmov s0, #1.5
+    fmov s0, 1.5
    /* test-gdb-s0 */
-    fmov s1, #2.5
+    fmov s1, 2.5
    /* test-gdb-s1 */
    fadd s2, s0, s1
    /* test-gdb-s2 */
    fadd s2, s0, s1
-    fmov s3, #4.0
+    fmov s3, 4.0
    fcmp s2, s3
    beq 1f
    bl lkmc_assert_fail
 1:

    /* Higher registers. */
-    fmov d28, #1.5
+    fmov d28, 1.5
    /* test-gdb-d28 */
-    fmov d29, #2.5
+    fmov d29, 2.5
    /* test-gdb-d29 */
    fadd d30, d28, d29
    /* test-gdb-d30 */
-    fmov d31, #4.0
+    fmov d31, 4.0
    /* test-gdb-d31 */
    fcmp d30, d31
    beq 1f
    bl lkmc_assert_fail
 1:
+
+    mov x0, 0
    ret
--- a/baremetal/arch/aarch64/multicore.S
+++ b/baremetal/arch/aarch64/multicore.S
@@ -3,7 +3,7 @@
 .global main
 main:
    /* Reset spinlock. */
-    mov x0, #0
+    mov x0, 0
    ldr x1, =spinlock
    str x0, [x1]

@@ -66,6 +66,7 @@ spinlock_start:
    wfe
    cbz x0, spinlock_start

+    mov x0, 0
    ret

 spinlock:
--- a/baremetal/arch/aarch64/no_bootloader/gem5_exit.S
+++ b/baremetal/arch/aarch64/no_bootloader/gem5_exit.S
@@ -1,3 +1,5 @@
+#include <lkmc/m5ops.h>
+
 .global mystart
 mystart:
-    mov x0, #0; .inst 0XFF000110 | (0x21 << 16);
+    LKMC_M5OPS_EXIT_ASM
--- a/baremetal/arch/aarch64/no_bootloader/semihost_exit.S
+++ b/baremetal/arch/aarch64/no_bootloader/semihost_exit.S
@@ -6,7 +6,7 @@ mystart:
    movk x1, 2, lsl 16
    ldr x2, =semihost_args
    str x1, [x2, 0]
-    mov x0, #0
+    mov x0, 0
    str x0, [x2, 8]
    mov x1, x2
    mov w0, 0x18
--- a/baremetal/arch/aarch64/regs.S
+++ b/baremetal/arch/aarch64/regs.S
@@ -4,26 +4,26 @@
 */
 .global main
 main:
-    mov x0, #1
+    mov x0, 1
    /* test-gdb-x0 */
-    mov x1, #2
+    mov x1, 2
    /* test-gdb-x1 */

-    mov x29, #1
+    mov x29, 1
    /* test-gdb-x29 */
-    mov x30, #2
+    mov x30, 2
    /* test-gdb-x30 */

-    fmov d0, #1.5
+    fmov d0, 1.5
    /* test-gdb-d0 */
-    fmov d1, #2.5
+    fmov d1, 2.5
    /* test-gdb-d1 */

-    fmov d30, #1.5
+    fmov d30, 1.5
    /* test-gdb-d30 */
-    fmov d31, #2.5
+    fmov d31, 2.5
    /* test-gdb-d31 */

    /* Exit required since we messed up with x30 which is the lr. */
-    mov x0, #0
+    mov x0, 0
    bl exit
--- a/baremetal/arch/aarch64/return.S
+++ b/baremetal/arch/aarch64/return.S
@@ -1,4 +1,5 @@
 /* Return to ensure that the post main works. */
 .global main
 main:
+    mov x0, 0
    ret
--- a/baremetal/arch/aarch64/return1.S
+++ b/baremetal/arch/aarch64/return1.S
@@ -0,0 +1,5 @@
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#magic-failure-string */
+.global main
+main:
+    mov x0, 1
+    ret
--- a/baremetal/arch/aarch64/svc_asm.S
+++ b/baremetal/arch/aarch64/svc_asm.S
@@ -16,6 +16,7 @@ main:
 1:

    /* Go home. */
+    mov x0, 0
    ret

 LKMC_GLOBAL(lkmc_vector_trap_handler)
--- a/baremetal/arch/arm/add.S
+++ b/baremetal/arch/arm/add.S
@@ -9,4 +9,5 @@ main:
    beq 1f
    bl lkmc_assert_fail
 1:
+    mov r0, #0
    bx lr
--- a/baremetal/arch/arm/dump_regs.c
+++ b/baremetal/arch/arm/dump_regs.c
@@ -0,0 +1,15 @@
+/* I want to move el and all other "what's the initial value of such system register" into here. */
+
+#include <stdio.h>
+#include <inttypes.h>
+
+int main(void) {
+    uint32_t cpsr;
+    uint32_t mvfr1;
+    __asm__ ("mrs %0, cpsr" : "=r" (cpsr) : :);
+    /* TODO this is blowing up an exception, how to I read from it? */
+    /*__asm__ ("vmrs %0, mvfr1" : "=r" (mvfr1) : :);*/
+    printf("cpsr  %" PRIx32 "\n", cpsr);
+    /*printf("mvfr1 %" PRIx32 "\n", mvfr1);*/
+    return 0;
+}
--- a/baremetal/arch/arm/gem5_assert.S
+++ b/baremetal/arch/arm/gem5_assert.S
@@ -1,8 +1,10 @@
 /* assert 0x12345678 + 1 == 0x12345679 */
+
+#include <lkmc/m5ops.h>
+
 .global main
 main:
-    movw r0, #:lower16:myvar
-    movt r0, #:upper16:myvar
+    adr r0, myvar
    ldr r1, [r0]
    add r1, r1, #1
    str r1, [r0]
@@ -10,10 +12,8 @@ main:
    movt r2, #0x1234
    cmp r1, r2
    beq ok
-    /* m5 fail 1 */
-    mov r0, #0; mov r1, #0; mov r2, #1; mov r3, #0; .inst 0xEE000110 | (0x22 << 16);
+    LKMC_M5OPS_FAIL_1_ASM
 ok:
-    /* m5 exit */
-    mov r0, #0; mov r1, #0; .inst 0xEE000110 | (0x21 << 16);
+    LKMC_M5OPS_EXIT_ASM
 myvar:
    .word 0x12345678
--- a/baremetal/arch/arm/multicore.S
+++ b/baremetal/arch/arm/multicore.S
@@ -32,6 +32,7 @@ spinlock_start:
    wfe
    cmp r0, #0
    beq spinlock_start
+    mov r0, #0
    bx lr
 spinlock:
    .skip 4
--- a/baremetal/arch/arm/regs.S
+++ b/baremetal/arch/arm/regs.S
@@ -5,4 +5,5 @@ main:
    /* test-gdb-r0 */
    mov r1, #2
    /* test-gdb-r1 */
+    mov r0, #0
    bx lr
--- a/baremetal/arch/arm/return1.S
+++ b/baremetal/arch/arm/return1.S
@@ -0,0 +1,5 @@
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#magic-failure-string */
+.global main
+main:
+    mov r0, #1
+    bx lr
--- a/baremetal/assert_fail.c
+++ b/baremetal/assert_fail.c
@@ -0,0 +1 @@
+../lkmc/assert_fail.c
--- a/baremetal/exit.c
+++ b/baremetal/exit.c
@@ -1,6 +0,0 @@
-#include <stdio.h>
-#include <stdlib.h>
-
-int main(void) {
-    exit(0);
-}
--- a/baremetal/exit0.c
+++ b/baremetal/exit0.c
@@ -0,0 +1,7 @@
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#magic-failure-string */
+
+#include <stdlib.h>
+
+int main(void) {
+    exit(0);
+}
--- a/baremetal/exit1.c
+++ b/baremetal/exit1.c
@@ -0,0 +1,7 @@
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#magic-failure-string */
+
+#include <stdlib.h>
+
+int main(void) {
+    exit(1);
+}
--- a/baremetal/interactive/prompt.c
+++ b/baremetal/interactive/prompt.c
@@ -1,3 +1,5 @@
+/* Test that input request through serial also works. */
+
 #include <stdio.h>
 #include <stdlib.h>

--- a/baremetal/hello.c
+++ b/baremetal/hello.c
@@ -1,6 +0,0 @@
-#include <stdio.h>
-
-int main(void) {
-    puts("hello");
-    return 0;
-}
--- a/baremetal/hello.c
+++ b/baremetal/hello.c
@@ -0,0 +1 @@
+../lkmc/hello.c
--- a/baremetal/interactive/infinite_loop.c
+++ b/baremetal/interactive/infinite_loop.c
--- a/baremetal/interactive/README.adoc
+++ b/baremetal/interactive/README.adoc
@@ -1 +0,0 @@
-This folder contains examples that are not very testable: either are supposed to return 0, or are interactive, etc.
--- a/baremetal/interactive/exit1.c
+++ b/baremetal/interactive/exit1.c
@@ -1,6 +0,0 @@
-#include <stdio.h>
-#include <stdlib.h>
-
-int main(void) {
-    exit(1);
-}
--- a/baremetal/interactive/return1.c
+++ b/baremetal/interactive/return1.c
@@ -1 +0,0 @@
-int main(void) { return 1; }
--- a/baremetal/lib/aarch64.S
+++ b/baremetal/lib/aarch64.S
@@ -14,6 +14,12 @@ mystart:
    /* Prepare the stack for main, mandatory for C code. */
    ldr x0, =stack_top
    mov sp, x0
+
+    /* https://github.com/cirosantilli/linux-kernel-module-cheat#magic-failure-string */
+    adr x0, lkmc_baremetal_on_exit_callback
+    bl on_exit
+
+    /* Run main. */
    bl main

    /* If main returns, exit. */
--- a/baremetal/lib/arm.S
+++ b/baremetal/lib/arm.S
@@ -1,5 +1,16 @@
+#include <lkmc.h>
+
 .global mystart
 mystart:
+    /* Prepare the stack for main, mandatory for C code. */
    ldr sp, =stack_top
+
+    /* https://github.com/cirosantilli/linux-kernel-module-cheat#magic-failure-string */
+    ldr r0, =lkmc_baremetal_on_exit_callback
+    bl on_exit
+
+    /* Run main. */
    bl main
+
+    /* If main returns, exit. */
    bl exit
--- a/baremetal/lib/syscalls.c
+++ b/baremetal/lib/syscalls.c
@@ -2,6 +2,8 @@
 #include <stdlib.h>
 #include <sys/stat.h>

+#include <lkmc/m5ops.h>
+
 enum {
    UART_FR_RXFE = 0x10,
 };
@@ -11,13 +13,55 @@ enum {

 int _close(int file) { return -1; }

+void _exit(int status) {
+#if defined(GEM5)
+    LKMC_M5OPS_EXIT;
+#else
+#if defined(__arm__)
+    __asm__ __volatile__ (
+        "mov r0, #0x18\n"
+        "ldr r1, =#0x20026\n"
+        "svc 0x00123456\n"
+        :
+        :
+        : "r0", "r1"
+    );
+#elif defined(__aarch64__)
+    /* TODO actually use the exit value here, just for fun. */
+    __asm__ __volatile__ (
+        "mov x1, #0x26\n" \
+        "movk x1, #2, lsl #16\n" \
+        "str x1, [sp,#0]\n" \
+        "mov x0, #0\n" \
+        "str x0, [sp,#8]\n" \
+        "mov x1, sp\n" \
+        "mov w0, #0x18\n" \
+        "hlt 0xf000\n"
+        :
+        :
+        : "x0", "x1"
+    );
+#endif
+#endif
+}
+
 int _fstat(int file, struct stat *st) {
    st->st_mode = S_IFCHR;
    return 0;
 }

+/* Required by assert. */
+int _getpid(void) { return 0; }
+
+/* Required by assert. */
+int _kill(pid_t pid, int sig) {
+    exit(128 + sig);
+}
+
 int _isatty(int file) { return 1; }
+
 int _lseek(int file, int ptr, int dir) { return 0; }
+
 int _open(const char *name, int flags, int mode) { return -1; }

 int _read(int file, char *ptr, int len) {
@@ -59,33 +103,3 @@ int _write(int file, char *ptr, int len) {
    }
    return len;
 }
-
-/* Only 0 is supported for now, arm semihosting cannot handle other values. */
-void _exit(int status) {
-#if defined(GEM5)
-#if defined(__arm__)
-    __asm__ __volatile__ ("mov r0, #0; mov r1, #0; .inst 0xEE000110 | (0x21 << 16);" : : : "r0", "r1");
-#elif defined(__aarch64__)
-    __asm__ __volatile__ ("mov x0, #0; .inst 0XFF000110 | (0x21 << 16);" : : : "x0");
-#endif
-#else
-#if defined(__arm__)
-    __asm__ __volatile__ ("mov r0, #0x18; ldr r1, =#0x20026; svc 0x00123456" : : : "r0", "r1");
-#elif defined(__aarch64__)
-    /* TODO actually use the exit value here, just for fun. */
-    __asm__ __volatile__ (
-        "mov x1, #0x26\n" \
-        "movk x1, #2, lsl #16\n" \
-        "str x1, [sp,#0]\n" \
-        "mov x0, #0\n" \
-        "str x0, [sp,#8]\n" \
-        "mov x1, sp\n" \
-        "mov w0, #0x18\n" \
-        "hlt 0xf000\n"
-        :
-        :
-        : "x0", "x1"
-    );
-#endif
-#endif
-}
--- a/baremetal/lkmc_assert_fail.c
+++ b/baremetal/lkmc_assert_fail.c
@@ -0,0 +1 @@
+../lkmc/lkmc_assert_fail.c
--- a/baremetal/return1.c
+++ b/baremetal/return1.c
@@ -0,0 +1,2 @@
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#magic-failure-string */
+int main(void) { return 1; }
--- a/baremetal/return2.c
+++ b/baremetal/return2.c
@@ -0,0 +1,2 @@
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#magic-failure-string */
+int main(void) { return 2; }
--- a/43
+++ b/43
@@ -1,34 +1,11 @@
-#!/usr/bin/env python3
-
-import imp
-import os
-import shutil
-import sys
-
-import common
-build_linux = imp.load_source('build-linux', os.path.join(kwargs['root_dir'], 'build_linux'))
-run = imp.load_source('run', os.path.join(kwargs['root_dir'], 'run'))
-
-parser = self.get_argparse(
-    argparse_args={
-        'description': '''Bisect the Linux kernel on gem5 boots.
-
-More information at: https://github.com/cirosantilli/linux-kernel-module-cheat#bisection
-'''},
-    default_args={
-        'emulators': ['gem5'],
-        'linux_build_id': 'bisect',
-    },
-)
-args = self.setup(parser)
-# We need a clean rebuild because rebuilds at different revisions:
-# - may fail
-# - may not actually rebuild all files, e.g. on header changes
-self.rmrf(kwargs['linux_build_dir'])
-build_linux.LinuxComponent().do_build(args)
-status = run.main(args, {
-    'eval': 'm5 exit',
-})
-if status == 125 or status == 127:
+#!/usr/bin/env bash
+set -eu
+./build-linux --clean "$@"
+./build-linux "$@"
+set +e
+./run --eval 'm5 exit' "$@" || status=$?
+# https://stackoverflow.com/questions/4713088/how-to-use-git-bisect/22592593#22592593
+if [ "$status" -eq 125 ] || [ "$status" -gt 127 ]; then
  status=1
-sys.exit(status)
+fi
+exit "$status"
--- a/2
+++ b/2
@@ -3,4 +3,4 @@ set -eu
 git submodule update --recursive
 cd ../..
 ./build-qemu --arch aarch64 --qemu-build-id bisect
-./run --arch aarch64 --kernel-cli 'init=/poweroff.out' --qemu-build-id bisect
+./run --arch aarch64 --kernel-cli 'init=/lkmc/linux/poweroff.out' --qemu-build-id bisect
--- a/20
+++ b/20
@@ -1,9 +1,20 @@
 #!/usr/bin/env python3
+
 import common
-parser = self.get_argparse(
-    argparse_args={'description':'Convert a BST vs heap stat file into a gnuplot input'}
+
+class Main(common.LkmcCliFunction):
+    def __init__(self):
+        super().__init__(
+            defaults={
+                'emulator': 'gem5',
+                'show_time': False,
+            },
+            description='''\
+Convert a BST vs heap stat file into a gnuplot input
+''',
        )
-args = self.setup(parser)
+
+    def timed_main(self):
        stats = self.get_stats()
        it = iter(stats)
        i = 1
@@ -15,3 +26,6 @@ for stat in it:
                break
            print('{} {} {}'.format(i, stat, next_stat))
            i += 1
+
+if __name__ == '__main__':
+    Main().cli()
--- a/bst-vs-heap.gnuplot
+++ b/bst-vs-heap.gnuplot
@@ -0,0 +1,25 @@
+#!/usr/bin/env gnuplot
+set terminal png size 1024, 2048
+set output "bst-vs-heap.tmp.png"
+set multiplot layout 5,1 title "Heap vs BST vs Hash map insert time"
+set xlabel "size"
+set ylabel "nanoseconds"
+
+set title "Heap"
+plot "bst_vs_heap.dat" using 1:2 notitle
+
+set title "Heap (zoom)"
+set yrange [0:25]
+plot "bst_vs_heap.dat" using 1:2 notitle
+
+set title "BST"
+set yrange [*:*]
+plot "bst_vs_heap.dat" using 1:3 notitle
+
+#set title "Hash map"
+#set yrange [*:*]
+#plot "bst_vs_heap.dat" using 1:4 notitle
+#
+#set title "Hash map zoom"
+#set yrange [0:350]
+#plot "bst_vs_heap.dat" using 1:4 notitle
--- a/85
+++ b/85
@@ -7,9 +7,12 @@ import cli_function
 import collections
 import common
 import copy
+import subprocess
 import shell_helpers
 from shell_helpers import LF

+import lkmc
+
 class _Component:
    '''
    Yes, we are re-inventing a crappy dependency resolution system,
@@ -107,10 +110,8 @@ so looping over all of them would waste time.
        )
        buildroot_component = _Component(
            self._build_file('build-buildroot'),
-            submodules = {
-                'buildroot',
-            },
            submodules_shallow = {
+                'buildroot',
                'binutils-gdb',
                'gcc',
                'glibc',
@@ -164,7 +165,7 @@ so looping over all of them would waste time.
                # Generate graphs of config.ini under m5out.
                'pydot',
            },
-            'submodules': {'gem5'},
+            'submodules_shallow': {'gem5'},
        }

        self.name_to_component_map = {
@@ -227,7 +228,7 @@ so looping over all of them would waste time.
                    'python-dev',
                    'texinfo',
                },
-                submodules={'crosstool-ng'},
+                submodules_shallow={'crosstool-ng'},
            ),
            'doc': _Component(
                self._build_file('build-doc'),
@@ -272,7 +273,7 @@ so looping over all of them would waste time.
            'm5': _Component(
                self._build_file('build-m5'),
                dependencies=['buildroot'],
-                submodules={'gem5'},
+                submodules_shallow={'gem5'},
            ),
            'overlay': _Component(dependencies=[
                'copy-overlay',
@@ -284,7 +285,7 @@ so looping over all of them would waste time.
                'overlay',
            ]),
            'parsec-benchmark': _Component(
-                submodules={'parsec-benchmark'},
+                submodules_shallow={'parsec-benchmark'},
                dependencies=['buildroot'],
            ),
            'qemu': _Component(
@@ -339,6 +340,14 @@ so looping over all of them would waste time.
                self._build_file('build-userland'),
                dependencies=['buildroot'],
            ),
+            'userland-host': _Component(
+                self._build_file('build-userland-in-tree'),
+                apt_get_pkgs={
+                    'libdrm-dev',
+                    'libeigen3-dev',
+                    'libopenblas-dev',
+                },
+            ),
            'userland-gem5': _Component(
                self._build_file('build-userland', static=True, userland_build_id='static'),
                dependencies=['buildroot'],
@@ -394,8 +403,8 @@ Which components to build. Default: qemu-buildroot
        def f():
            args = self.get_common_args()
            args.update(extra_args)
-            args['print_time'] = False
-            self.import_path_main(component_file)(**args)
+            args['show_time'] = False
+            lkmc.import_path.import_path_main(component_file)(**args)
        return f

    def timed_main(self):
@@ -504,46 +513,54 @@ Which components to build. Default: qemu-buildroot
                    ['python3', '-m', 'pip', 'install', '--user', LF] +
                    self.sh.add_newlines(sorted(python3_pkgs))
                )
-            git_cmd_common = ['git', 'submodule', 'update', '--init', '--recursive']
-            if submodules:
-                # == Other nice git options for when distros move to newer Git
-                #
-                # Currently not on Ubuntu 16.04:
-                #
-                # `--progress`: added on Git 2.10:
-                #
+            git_version_tuple = tuple(int(x) for x in subprocess.check_output(['git', '--version']).decode().split(' ')[-1].split('.'))
+            git_cmd_common = [
+                'git', LF,
+                'submodule', LF,
+                'update', LF,
+                '--init', LF,
+                '--recursive', LF,
+            ]
+            if git_version_tuple >= (2, 9, 0):
+                # https://stackoverflow.com/questions/26957237/how-to-make-git-clone-faster-with-multiple-threads/52327638#52327638
+                git_cmd_common.extend(['--jobs', str(len(os.sched_getaffinity(0))), LF])
+            if git_version_tuple >= (2, 10, 0):
                # * https://stackoverflow.com/questions/32944468/how-to-show-progress-for-submodule-fetching
                # * https://stackoverflow.com/questions/4640020/progress-indicator-for-git-clone
-                #
-                # `--jobs"`: https://stackoverflow.com/questions/26957237/how-to-make-git-clone-faster-with-multiple-threads/52327638#52327638
-                self.sh.run_cmd(
-                    git_cmd_common + ['--', LF] +
-                    self.sh.add_newlines([os.path.join(common.consts['submodules_dir'], x) for x in sorted(submodules)])
-                )
+                git_cmd_common.extend(['--progress', LF])
+            def submodule_ids_to_cmd(submodules):
+                return self.sh.add_newlines([os.path.join(common.consts['submodules_dir'], x) for x in sorted(submodules)])
+            if submodules:
+                self.sh.run_cmd(git_cmd_common + ['--', LF] + submodule_ids_to_cmd(submodules))
            if submodules_shallow:
-                # == Shallow cloning.
-                #
                # TODO Ideally we should shallow clone --depth 1 all of them.
                #
                # However, most git servers out there are crap or craply configured
                # and don't allow shallow cloning except for branches.
                #
-                # So for now, let's shallow clone only the Linux kernel, which has by far
-                # the largest .git repo history, and full clone the others.
+                # So for now I'm keeping all mirrors in my GitHub.
+                # and always have a lkmc-* branch pointint to it.
                #
-                # Then we will maintain a GitHub Linux kernel mirror / fork that always has a
-                # lkmc branch, and point to it, so that it will always succeed.
+                # However, QEMU has a bunch of submodules itself, and I'm not in the mood
+                # to mirror all of them...
                #
                # See also:
                #
                # * https://stackoverflow.com/questions/3489173/how-to-clone-git-repository-with-specific-revision-changeset
                # * https://stackoverflow.com/questions/2144406/git-shallow-submodules/47374702#47374702
                # * https://unix.stackexchange.com/questions/338578/why-is-the-git-clone-of-the-linux-kernel-source-code-much-larger-than-the-extrac
-                #
-                self.sh.run_cmd(
-                    git_cmd_common + ['--depth', '1', '--', LF] +
-                    self.sh.add_newlines([os.path.join(common.consts['submodules_dir'], x) for x in sorted(submodules_shallow)])
-                )
+                cmd = git_cmd_common.copy()
+                if git_version_tuple > (2, 7, 4):
+                    # Then there is a bug in Ubuntu 16.04 git 2.7.4 where --depth 1 fails...
+                    # OMG git submodules implementation sucks:
+                    # * https://stackoverflow.com/questions/2155887/git-submodule-head-reference-is-not-a-tree-error/25875273#25875273
+                    # * https://github.com/boostorg/boost/issues/245
+                    cmd.extend(['--depth', '1', LF])
+                else:
+                    self.log_warn('your git is too old for git submodule update --depth 1')
+                    self.log_warn('update to git 2.17 or newer and you will save clone time')
+                    self.log_warn('see: https://github.com/cirosantilli/linux-kernel-module-cheat/issues/44')
+                self.sh.run_cmd(cmd + ['--', LF] + submodule_ids_to_cmd(submodules_shallow))

        # Do the build.
        for component in selected_components:
--- a/72
+++ b/72
@@ -15,6 +15,9 @@ Build the baremetal examples with crosstool-NG.
 ''',
            supported_archs=common.consts['crosstool_ng_supported_archs']
        )
+        self._add_argument('--ccflags')
+        self._add_argument('--force-rebuild')
+        self._add_argument('--optimization-level')

    def build(self):
        build_dir = self.get_build_dir()
@@ -39,11 +42,25 @@ Build the baremetal examples with crosstool-NG.
        cflags = [
            '-I', self.env['baremetal_source_lib_dir'], LF,
            '-I', self.env['root_dir'], LF,
-            '-O0', LF,
+            '-O{}'.format(self.env['optimization_level']), LF,
            '-ggdb3', LF,
            '-mcpu={}'.format(self.env['mcpu']), LF,
            '-nostartfiles', LF,
        ]
+        if self.env['arch'] == 'arm':
+            cflags.extend([
+                '-mhard-float', LF,
+                # This uses the soft float ABI for calling functions from objets in Newlib which
+                # our crosstool-NG config compiles with soft floats, while emiting hard float
+                # from C and allowing us to use it from assembly, e.g. for the VMRS instruction:
+                # which would otherwise fail "with selected processor does not support XXX in ARM mode"
+                # Bibliography:
+                # - https://stackoverflow.com/questions/9753749/arm-compilation-error-vfp-registered-used-by-executable-not-object-file
+                # - https://stackoverflow.com/questions/41131432/cross-compiling-error-selected-processor-does-not-support-fmrx-r3-fpexc-in/41131782#41131782
+                # - https://embeddedartistry.com/blog/2017/10/9/r1q7pksku2q3gww9rpqef0dnskphtc
+                '-mfloat-abi=softfp', LF,
+                '-mfpu=crypto-neon-fp-armv8', LF,
+            ])
        cflags_after = ['-lm']
        if self.env['emulator'] == 'gem5':
            if self.env['machine'] == 'VExpress_GEM5_V1':
@@ -54,7 +71,10 @@ Build the baremetal examples with crosstool-NG.
                uart_address = 0x10009000
            else:
                raise Exception('unknown machine: ' + self.env['machine'])
-            cflags.extend(['-D', 'GEM5'.format(uart_address), LF])
+            cflags.extend([
+                '-D', 'GEM5'.format(uart_address), LF,
+                '-DLKMC_M5OPS_ENABLE=1', LF,
+            ])
        else:
            entry_address = 0x40000000
            uart_address = 0x09000000
@@ -67,9 +87,10 @@ Build the baremetal examples with crosstool-NG.
                self.env['asm_ext']
            )
        )
+        cflags.extend(self.sh.shlex_split(self.env['ccflags']))
        if self.need_rebuild([src], bootloader_obj):
            self.sh.run_cmd(
-                [self.env['gcc'],  LF] +
+                [self.env['gcc_path'],  LF] +
                cflags +
                [
                    '-c', LF,
@@ -84,11 +105,11 @@ Build the baremetal examples with crosstool-NG.
        ]:
            if self.need_rebuild([src, self.env['common_h']], obj):
                self.sh.run_cmd(
-                    [self.env['gcc'],  LF] +
+                    [self.env['gcc_path'],  LF] +
                    cflags +
                    [
-                        '-c', LF,
                        '-D', 'UART0_ADDR={:#x}'.format(uart_address), LF,
+                        '-c', LF,
                        '-o', obj, LF,
                        src, LF,
                    ] +
@@ -112,43 +133,34 @@ Build the baremetal examples with crosstool-NG.
                    in_name, in_ext = os.path.splitext(in_basename)
                    if  (
                        os.path.isfile(in_path) and
-                        in_ext in (self.env['c_ext'], self.env['asm_ext'])
+                        in_ext in self.env['build_in_exts']
                    ):
-                        main_obj = os.path.join(
-                            out_dir,
-                            '{}{}'.format(
-                                in_name,
-                                self.env['obj_ext']
-                            )
-                        )
-                        src = os.path.join(self.env['baremetal_source_dir'], in_path)
-                        if self.need_rebuild([src, self.env['common_h']], main_obj):
-                            self.sh.run_cmd(
-                                [self.env['gcc'],  LF] +
-                                cflags +
-                                [
-                                    '-c', LF,
-                                    '-o', main_obj, LF,
-                                    src, LF,
-                                ] +
-                                cflags_after
-                            )
-                        objs = common_objs_bootloader + [main_obj]
                        out = os.path.join(out_dir, in_name + self.env['baremetal_build_ext'])
-                        if self.need_rebuild(objs + [self.env['baremetal_link_script']], out):
+                        src = os.path.join(self.env['baremetal_source_dir'], in_path)
+                        if self.need_rebuild(
+                            common_objs_bootloader +
+                            [
+                                src,
+                                self.env['baremetal_link_script'],
+                                self.env['common_h']
+                            ],
+                            out
+                        ):
                            self.sh.run_cmd(
-                                [self.env['gcc'],  LF] +
+                                [self.env['gcc_path'],  LF] +
                                cflags +
                                [
                                    '-Wl,--section-start=.text={:#x}'.format(entry_address), LF,
                                    '-o', out, LF,
                                    '-T', self.env['baremetal_link_script'], LF,
                                ] +
-                                self.sh.add_newlines(objs) +
+                                [
+                                    src, LF,
+                                ] +
+                                self.sh.add_newlines(common_objs_bootloader) +
                                cflags_after
                            )

-
    def get_build_dir(self):
        return self.env['baremetal_build_dir']

--- a/5
+++ b/5
@@ -70,6 +70,7 @@ Extra arguments to be passed to the Buildroot make,
 usually extra Buildroot targets.
 '''
        )
+        self._add_argument('--force-rebuild')

    def build(self):
        build_dir = self.get_build_dir()
@@ -147,6 +148,8 @@ usually extra Buildroot targets.
            cwd=self.env['buildroot_source_dir'],
        )
        self.make_build_dirs()
+        if self.env['force_rebuild']:
+            extra_make_args.extend(['-B', LF])
        if not self.env['no_all']:
            extra_make_args.extend(['all', LF])
        self.sh.run_cmd(
@@ -158,7 +161,7 @@ usually extra Buildroot targets.
            ] +
            extra_make_args
            ,
-            out_file=os.path.join(self.env['buildroot_build_dir'], 'lkmc.log'),
+            out_file=os.path.join(self.env['buildroot_build_dir'], self.env['repo_short_id'] + '.log'),
            delete_env=['LD_LIBRARY_PATH'],
            cwd=self.env['buildroot_source_dir'],
        )
--- a/2
+++ b/2
@@ -61,7 +61,7 @@ Build crosstool-NG with Newlib for bare metal compilation
                'build', LF,
                'CT_JOBS={}'.format(str(self.env['nproc'])), LF,
            ],
-            out_file=os.path.join(build_dir, 'lkmc.log'),
+            out_file=os.path.join(build_dir, self.env['repo_short_id'] + '.log'),
            delete_env=['LD_LIBRARY_PATH'],
            extra_paths=[self.env['ccache_dir']],
        )
--- a/3
+++ b/3
@@ -9,7 +9,7 @@ class Main(common.LkmcCliFunction):
    def __init__(self):
        super().__init__(
            defaults = {
-                'print_time': False,
+                'show_time': False,
            },
            description='''\
 https://github.com/cirosantilli/linux-kernel-module-cheat#build-the-documentation
@@ -27,6 +27,7 @@ https://github.com/cirosantilli/linux-kernel-module-cheat#build-the-documentatio
        )
        error_re = re.compile('^asciidoctor: WARNING: ')
        exit_status = 0
+        if not self.env['dry_run']:
            with open(self.env['build_doc_log']) as f:
                for line in f:
                    if error_re.search(line):
--- a/1
+++ b/1
@@ -7,7 +7,6 @@ import tarfile
 import common
 from shell_helpers import LF

-
 class DockerComponent(self.Component):
    def get_argparse_args(self):
        return {
--- a/1
+++ b/1
@@ -64,6 +64,7 @@ https://github.com/cirosantilli/linux-kernel-module-cheat-regression#gem5-unit-t
                    self.env['gem5_source_dir'], LF,
                ])
            else:
+                if not self.env['dry_run']:
                    raise Exception('gem5 submodule not checked out')
        if self.env['verbose']:
            verbose = ['--verbose', LF]
--- a/16
+++ b/16
@@ -81,21 +81,19 @@ Run `make modules_install` after `make`.
            metavar='extra-make-args',
            nargs='*'
        )
+        self._add_argument('--force-rebuild')

    def build(self):
        build_dir = self.get_build_dir()
        os.makedirs(build_dir, exist_ok=True)
-        tool = 'gcc'
-        gcc = self.get_toolchain_tool(tool)
-        prefix = gcc[:-len(tool)]
        common_args = {
            'cwd': self.env['linux_source_dir'],
        }
        ccache = shutil.which('ccache')
        if ccache is not None:
-            cc = '{} {}'.format(ccache, gcc)
+            cc = '{} {}'.format(ccache, self.env['gcc_path'])
        else:
-            cc = gcc
+            cc = self.env['gcc_path']
        if self.env['verbose']:
            verbose = ['V=1']
        else:
@@ -104,10 +102,12 @@ Run `make modules_install` after `make`.
            'make', LF,
            '-j', str(self.env['nproc']), LF,
            'ARCH={}'.format(self.env['linux_arch']), LF,
-            'CROSS_COMPILE={}'.format(prefix), LF,
+            'CROSS_COMPILE={}-'.format(self.env['toolchain_prefix']), LF,
            'CC={}'.format(cc), LF,
            'O={}'.format(build_dir), LF,
        ] + verbose
+        if self.env['force_rebuild']:
+            common_make_args.extend(['-B', LF])
        if self.env['configure']:
            if self.env['custom_config_target']:
                base_config_given = True
@@ -199,7 +199,7 @@ Run `make modules_install` after `make`.
                extra_env={
                    'KBUILD_BUILD_VERSION': '1',
                    'KBUILD_BUILD_TIMESTAMP': 'Thu Jan  1 00:00:00 UTC 1970',
-                    'KBUILD_BUILD_USER': 'lkmc',
+                    'KBUILD_BUILD_USER': self.env['repo_short_id'],
                    'KBUILD_BUILD_HOST': common.git_sha(self.env['linux_source_dir']),
                },
                **common_args
@@ -209,7 +209,7 @@ Run `make modules_install` after `make`.
                    (
                        common_make_args +
                        [
-                            'INSTALL_MOD_PATH={}'.format(self.env['out_rootfs_overlay_dir']), LF,
+                            'INSTALL_MOD_PATH={}'.format(self.env['out_rootfs_overlay_lkmc_dir']), LF,
                            'modules_install', LF,
                        ]
                    ),
--- a/4
+++ b/4
@@ -16,8 +16,8 @@ class Main(common.BuildCliFunction):
            'make', LF,
            '-j', str(self.env['nproc']), LF,
            '-f', 'Makefile.{}'.format(arch), LF,
-            'CC={}'.format(self.env['gcc']), LF,
-            'LD={}'.format(self.env['ld']), LF,
+            'CC={}'.format(self.env['gcc_path']), LF,
+            'LD={}'.format(self.env['ld_path']), LF,
            'PWD={}'.format(self.env['gem5_m5_source_dir']), LF,
        ]

--- a/33
+++ b/33
@@ -3,6 +3,7 @@
 import distutils.dir_util
 import os
 import platform
+import shlex
 import shutil

 import common
@@ -19,6 +20,9 @@ See also: https://github.com/cirosantilli/linux-kernel-module-cheat#host
        self.add_argument(
            '--make-args',
            default='',
+            help='''
+Pass custom options to make.
+''',
        )
        self.add_argument(
            '--host',
@@ -34,6 +38,7 @@ Place the modules on a separate magic directory from non --host builds.
            help='Which kernel modules to build. Default: build all',
            nargs='*',
        )
+        self._add_argument('--force-rebuild')

    def build(self):
        build_dir = self.get_build_dir()
@@ -72,22 +77,23 @@ Place the modules on a separate magic directory from non --host builds.
            build_subdir = self.env['kernel_modules_build_host_subdir']
        else:
            build_subdir = self.env['kernel_modules_build_subdir']
-        tool = 'gcc'
-        gcc = self.get_toolchain_tool(tool)
-        prefix = gcc[:-len(tool)]
        ccache = shutil.which('ccache')
        if ccache is not None:
-            cc = '{} {}'.format(ccache, gcc)
+            cc = '{} {}'.format(ccache, self.env['gcc_path'])
        else:
-            cc = gcc
-        if self.env['verbose']:
-            verbose = ['V=1']
-        else:
-            verbose = []
+            cc = self.env['gcc_path']
        if self.env['host']:
            linux_dir = os.path.join('/lib', 'modules', platform.uname().release, 'build')
        else:
            linux_dir = self.env['linux_build_dir']
+        ccflags = [
+            '-I', self.env['root_dir'], LF,
+        ]
+        make_args_extra = []
+        if self.env['verbose']:
+            make_args_extra.extend(['V=1', LF])
+        if self.env['force_rebuild']:
+            make_args_extra.extend(['-B', LF])
        self.sh.run_cmd(
            (
                [
@@ -95,20 +101,21 @@ Place the modules on a separate magic directory from non --host builds.
                    '-j', str(self.env['nproc']), LF,
                    'ARCH={}'.format(self.env['linux_arch']), LF,
                    'CC={}'.format(cc), LF,
-                    'CROSS_COMPILE={}'.format(prefix), LF,
+                    'CCFLAGS={}'.format(self.sh.cmd_to_string(ccflags)), LF,
+                    'CROSS_COMPILE={}-'.format(self.env['toolchain_prefix']), LF,
                    'LINUX_DIR={}'.format(linux_dir), LF,
                    'M={}'.format(build_subdir), LF,
                    'OBJECT_FILES={}'.format(' '.join(object_files)), LF,
                ] +
-                self.sh.shlex_split(self.env['make_args']) +
-                verbose
+                make_args_extra +
+                self.sh.shlex_split(self.env['make_args'])
            ),
            cwd=os.path.join(self.env['kernel_modules_build_subdir']),
        )
        if not self.env['host']:
            self.sh.copy_dir_if_update_non_recursive(
                srcdir=self.env['kernel_modules_build_subdir'],
-                destdir=self.env['out_rootfs_overlay_dir'],
+                destdir=self.env['out_rootfs_overlay_lkmc_dir'],
                filter_ext=self.env['kernel_module_ext'],
            )

--- a/326
+++ b/326
@@ -2,295 +2,141 @@

 import os
 import shlex
-
-import common
-import threading
 import subprocess
-from shell_helpers import LF
+import threading

-error = False
+from shell_helpers import LF
+import common
+import thread_pool

 class Main(common.BuildCliFunction):
-    def __init__(self):
-        super().__init__(
-            description='''\
+    def __init__(self, *args, **kwargs):
+        if not 'description' in kwargs:
+            kwargs['description'] = '''\
 Build our compiled userland examples.
 '''
-        )
-        self.default_cstd = 'c11'
-        self.default_cxxstd = 'c++17'
-        self.add_argument(
-            '--has-package',
-            action='append',
-            default=[],
-            help='''\
-Indicate that a given package is present in the root filesystem, which
-allows us to build examples that rely on it.
-''',
-        )
-        self.add_argument(
-            '--in-tree',
-            default=False,
-            help='''\
-Magic build mode tailored to build from within the source tree:
-
-* place build output inside soure tree to conveniently run it
-* if not targets are given, build use the current working directory
-''',
-        )
+        super().__init__(*args, **kwargs)
        self.add_argument(
            'targets',
            default=[],
            help='''\
-Build only the given userland programs or all programs in the given directories.
+Select to build only the given userland programs, or all programs under
+the given directories.
+
+Default: build all.
+
+Must point to either sources or directories under userland/, or to LKMC
+toplevel which is a synonym for userland/.

 Default: build all examples that have their package dependencies met, e.g.:
+
 -   userland/arch/ programs only build if the target arch matches
 -   an OpenBLAS example can only be built if the target root filesystem
-    has the OpenBLAS libraries and headers installed, which you must inform with --has-package
+    has the OpenBLAS libraries and headers installed, which you must inform
+    with --package
 ''',
            nargs='*',
        )
-
-    def _build_one(
-        self,
-        in_path,
-        out_path,
-        ccflags,
-        ccflags_after=None,
-        cstd=None,
-        cxxstd=None,
-        extra_deps=None,
-        extra_objs=None,
-        link=True,
-        raise_on_failure=True,
-        thread_limiter=None,
-    ):
-        try:
-            if extra_deps is None:
-                extra_deps = []
-            if extra_objs is None:
-                extra_objs = []
-            if ccflags_after is None:
-                ccflags_after = []
-            ret = 0
-            if self.need_rebuild([in_path] + extra_objs + extra_deps, out_path):
-                ccflags = ccflags.copy()
-                if not link:
-                    ccflags.extend(['-c', LF])
-                in_ext = os.path.splitext(in_path)[1]
-                do_compile = True
-                if in_ext == self.env['c_ext']:
-                    cc = self.env['gcc']
-                    if cstd is None:
-                        std = self.default_cstd
-                    else:
-                        std = cstd
-                    ccflags.extend([
-                        '-fopenmp', LF,
-                    ])
-                elif in_ext == self.env['cxx_ext']:
-                    cc = self.env['gxx']
-                    if cxxstd is None:
-                        std = self.default_cxxstd
-                    else:
-                        std = cxxstd
-                else:
-                    do_compile = False
-                if do_compile:
-                    ret = self.sh.run_cmd(
-                        (
-                            [
-                                cc, LF,
-                            ] +
-                            ccflags +
-                            [
-                                '-std={}'.format(std), LF,
-                                '-o', out_path, LF,
-                                in_path, LF,
-                            ] +
-                            self.sh.add_newlines(extra_objs) +
-                            [
-                                '-lm', LF,
-                                '-pthread', LF,
-                            ] +
-                            ccflags_after
-                        ),
-                        extra_paths=[self.env['ccache_dir']],
-                        raise_on_failure=raise_on_failure,
-                    )
-        finally:
-            if thread_limiter is not None:
-                thread_limiter.release()
-        if ret != 0:
-            self.error = True
-        return ret
-
-    def _get_targets(self):
-        if self.env['_args_given']['targets']:
-            targets = self.env['targets']
-            if self.env['in_tree']:
-                cwd = os.getcwd()
-                targets = [os.path.join(cwd, target) for target in targets]
-            return targets
-        else:
-            if self.env['in_tree']:
-                return [os.getcwd()]
-            else:
-                return [self.env['userland_source_dir']]
+        self._add_argument('--ccflags')
+        self._add_argument('--force-rebuild')
+        self._add_argument('--optimization-level')

    def build(self):
        build_dir = self.get_build_dir()
-        os.makedirs(build_dir, exist_ok=True)
-        has_packages = set(self.env['has_package'])
-        ccflags = [
+        cc_flags = [
            '-I', self.env['root_dir'], LF,
-            '-I', self.env['userland_source_dir'], LF,
-            '-O0', LF,
-            '-Wall', LF,
-            '-Werror', LF,
-            '-Wextra', LF,
-            '-Wno-unused-function', LF,
-            '-ggdb3', LF,
-        ]
+            '-O{}'.format(self.env['optimization_level']), LF,
+        ] + self.sh.shlex_split(self.env['ccflags'])
        if self.env['static']:
-            ccflags.extend(['-static', LF])
-        common_obj = os.path.join(
+            cc_flags.extend(['-static', LF])
+        extra_obj_lkmc_common = os.path.join(
            build_dir,
            self.env['common_basename_noext'] + self.env['obj_ext']
        )
        self._build_one(
            in_path=self.env['common_c'],
-            out_path=common_obj,
-            ccflags=ccflags,
+            out_path=extra_obj_lkmc_common,
+            cc_flags=cc_flags,
            extra_deps=[self.env['common_h']],
            link=False,
        )
-        pkgs = {
-            'eigen': {
-                # TODO: was failing with:
-                # fatal error: Eigen/Dense: No such file or directory as of
-                # 975ce0723ee3fa1fea1766e6683e2f3acb8558d6
-                # http://lists.busybox.net/pipermail/buildroot/2018-June/222914.html
-                'ccflags': [
-                    '-I',
-                    os.path.join(
-                        self.env['buildroot_staging_dir'],
-                        'usr',
-                        'include',
-                        'eigen3'
-                    ),
-                    LF
-                ],
-                # Header only.
-                'ccflags_after': [],
-            },
-            'libdrm': {},
-            'openblas': {},
-        }
-        rootdir_abs_len = len(self.env['userland_source_dir'])
-        thread_limiter = threading.BoundedSemaphore(self.env['nproc'])
-        self.error = False
-        for target in self._get_targets():
-            target = self.resolve_userland_source(target)
-            for path, in_dirnames, in_filenames in self.sh.walk(target):
-                in_dirnames.sort()
-                path_abs = os.path.abspath(path)
-                dirpath_relative_root = path_abs[rootdir_abs_len + 1:]
-                dirpath_relative_root_components = dirpath_relative_root.split(os.sep)
-                if (
-                    len(dirpath_relative_root_components) < 2 or
-                    dirpath_relative_root_components[0] != 'arch' or
-                    dirpath_relative_root_components[1] == self.env['arch']
-                ):
-                    out_dir = os.path.join(
+        extra_obj_userland_asm = os.path.join(
            build_dir,
-                        dirpath_relative_root
+            'arch',
+            'main' + self.env['obj_ext']
        )
-                    os.makedirs(out_dir, exist_ok=True)
-                    ccflags_dir = ccflags.copy()
-                    if dirpath_relative_root_components == ['gcc']:
-                        cstd = 'gnu11'
-                        cxxstd = 'gnu++17'
-                    else:
-                        cstd = self.default_cstd
-                        cxxstd = self.default_cxxstd
-                        # -pedantic complains even if we use -std=gnu11.
-                        ccflags_dir.extend(['-pedantic', LF])
+        extra_obj_userland_asm_relpath = os.path.join(
+            'arch',
+            'main' + self.env['c_ext']
+        )
+        self._build_one(
+            in_path=os.path.join(
+                self.env['userland_source_dir'],
+                extra_obj_userland_asm_relpath
+            ),
+            out_path=extra_obj_userland_asm,
+            cc_flags=cc_flags,
+            extra_deps=[self.env['common_h']],
+            link=False,
+        )
+        with thread_pool.ThreadPool(
+            self._build_one,
+            nthreads=self.env['nproc'],
+        ) as my_thread_pool:
+            try:
+                for target in self.env['targets']:
+                    for path, in_dirnames, in_filenames in self.sh.walk(target):
                        for in_filename in in_filenames:
-                        in_path = os.path.join(path, in_filename)
-                        in_name, in_ext = os.path.splitext(in_filename)
-                        out_path = os.path.join(
-                            out_dir,
-                            in_name + self.env['userland_build_ext']
-                        )
-                        pkg_key = in_name.split('_')[0]
-                        ccflags_file = ccflags_dir.copy()
-                        ccflags_after = []
-                        if pkg_key in pkgs:
-                            if pkg_key not in has_packages:
+                            in_ext = os.path.splitext(in_filename)[1]
+                            if not in_ext in self.env['build_in_exts']:
                                continue
-                            pkg = pkgs[pkg_key]
-                            if 'ccflags' in pkg:
-                                ccflags_file.extend(pkg['ccflags'])
-                            else:
-                                pkg_config_output = subprocess.check_output([
-                                    self.env['buildroot_pkg_config'],
-                                    '--cflags',
-                                    pkg_key
-                                ]).decode()
-                                ccflags_file.extend(self.sh.shlex_split(pkg_config_output))
-                            if 'ccflags_after' in pkg:
-                                ccflags_file.extend(pkg['ccflags_after'])
-                            else:
-                                pkg_config_output = subprocess.check_output([
-                                    self.env['buildroot_pkg_config'],
-                                    '--libs',
-                                    pkg_key
-                                ]).decode()
-                                ccflags_after.extend(self.sh.shlex_split(pkg_config_output))
-                        thread_limiter.acquire()
-                        if self.error:
-                            return 1
-                        thread = threading.Thread(
-                            target=self._build_one,
-                            kwargs={
+                            in_path = os.path.join(path, in_filename)
+                            error = my_thread_pool.submit({
                                'in_path': in_path,
-                                'out_path': out_path,
-                                'ccflags': ccflags_file,
-                                'cstd': cstd,
-                                'cxxstd': cxxstd,
-                                'extra_objs': [common_obj],
-                                'ccflags_after': ccflags_after,
-                                'raise_on_failure': False,
-                                'thread_limiter': thread_limiter,
-                            }
-                        )
-                        thread.start()
+                                'out_path': self.resolve_userland_executable(in_path),
+                                'cc_flags': cc_flags,
+                                'extra_objs_lkmc_common': [extra_obj_lkmc_common],
+                                'extra_objs_userland_asm': [extra_obj_userland_asm],
+                            })
+                            if error is not None:
+                                raise common.ExitLoop()
+            except common.ExitLoop:
+                pass
+        error = my_thread_pool.get_error()
+        if error is not None:
+            print(error)
+            return 1
+        if not self.env['in_tree']:
            self.sh.copy_dir_if_update(
                srcdir=build_dir,
-            destdir=self.env['out_rootfs_overlay_dir'],
-            filter_ext=self.env['userland_build_ext'],
+                destdir=self.env['out_rootfs_overlay_lkmc_dir'],
+                filter_ext=self.env['userland_executable_ext'],
            )
        return 0

    def clean(self):
        if self.env['in_tree']:
-            for target in self._get_targets():
-                for path, dirnames, filenames in os.walk(target):
-                    filenames.sort()
-                    dirnames.sort()
+            for target in self.env['targets']:
+                if os.path.exists(target):
+                    if os.path.isfile(target):
+                        self.sh.rmrf(self.resolve_userland_executable(target))
+                    else:
+                        for path, dirnames, filenames in self.sh.walk(target):
                            for filename in filenames:
                                if os.path.splitext(filename)[1] in self.env['userland_out_exts']:
                                    self.sh.rmrf(os.path.join(path, filename))
        else:
-            self.sh.rmrf(self.get_build_dir())
+            for target in self.env['targets']:
+                self.sh.rmrf(self.resolve_userland_executable(target))

    def get_build_dir(self):
-        if self.env['in_tree']:
-            return self.env['userland_source_dir']
-        else:
        return self.env['userland_build_dir']

+    def setup_one(self):
+        self.env['targets'] = self.resolve_targets(
+            self.env['userland_source_dir'],
+            self.env['targets']
+        )
+
 if __name__ == '__main__':
    Main().cli()
--- a/26
+++ b/26
@@ -1,2 +1,24 @@
-#!/usr/bin/env bash
-"$(git rev-parse --show-toplevel)/build-userland" --gcc-which host --in-tree "$@"
+#!/usr/bin/env python3
+
+import os
+import subprocess
+
+import lkmc.import_path
+
+build_userland = lkmc.import_path.import_path_relative_root('build-userland')
+
+class Main(build_userland.Main):
+    def __init__(self):
+        super().__init__(
+            description='''\
+https://github.com/cirosantilli/linux-kernel-module-cheat#userland-setup-getting-started-natively
+''',
+            defaults={
+                'gcc_which': 'host',
+                'in_tree': True,
+                'targets': ['.'],
+            }
+        )
+
+if __name__ == '__main__':
+    Main().cli()
--- a/cli_function.py
+++ b/cli_function.py
@@ -11,10 +11,11 @@ made to this file.
 import argparse
 import bisect
 import collections
-import imp
 import os
 import sys

+import lkmc.import_path
+
 class _Argument:
    def __init__(
            self,
@@ -113,8 +114,7 @@ class _Argument:

 class CliFunction:
    '''
-    Represent a function that can be called either from Python code, or
-    from the command line.
+    A function that can be called either from Python code, or from the command line.

    Features:

@@ -134,6 +134,10 @@ class CliFunction:

    * that decorator API is insane
    * CLI + Python for single functions was wontfixed: https://github.com/pallets/click/issues/40
+    +
+    Oh, and I commented on that issue pointing to this alternative and they deleted my comment:
+    https://github.com/pallets/click/issues/40#event-2088718624 Lol. It could have been useful
+    for other Googlers and as an implementation reference.
    '''
    def __call__(self, **kwargs):
        '''
@@ -147,15 +151,15 @@ class CliFunction:
    def _do_main(self, kwargs):
        return self.main(**self._get_args(kwargs))

-    def __init__(self, config_file=None, description=None, extra_config_params=None):
+    def __init__(self, default_config_file=None, description=None, extra_config_params=None):
        self._arguments = collections.OrderedDict()
-        self._config_file = config_file
+        self._default_config_file = default_config_file
        self._description = description
        self.extra_config_params = extra_config_params
-        if self._config_file is not None:
+        if self._default_config_file is not None:
            self.add_argument(
                '--config-file',
-                default=self._config_file,
+                default=self._default_config_file,
                help='Path to the configuration file to use'
            )

@@ -172,20 +176,23 @@ class CliFunction:
        args_with_defaults = kwargs.copy()
        # Add missing args from config file.
        config_file = None
+        args_given = {}
        if 'config_file' in args_with_defaults and args_with_defaults['config_file'] is not None:
            config_file = args_with_defaults['config_file']
+            args_given['config_file'] = True
        else:
-            config_file = self._config_file
-        args_given = {}
+            config_file = self._default_config_file
+            args_given['config_file'] = False
        for key in self._arguments:
            args_given[key] = not (
                not key in args_with_defaults or
                args_with_defaults[key] is None or
                self._arguments[key].nargs == '*' and args_with_defaults[key] == []
            )
-        if config_file is not None and os.path.exists(config_file):
+        if config_file is not None:
+            if os.path.exists(config_file):
                config_configs = {}
-            config = imp.load_source('config', config_file)
+                config = lkmc.import_path.import_path(config_file)
                if self.extra_config_params is None:
                    config.set_args(config_configs)
                else:
@@ -196,6 +203,8 @@ class CliFunction:
                    if not args_given[key]:
                        args_with_defaults[key] = config_configs[key]
                        args_given[key] = True
+            elif args_given['config_file']:
+                raise Exception('Config file does not exist: ' + config_file)
        # Add missing args from hard-coded defaults.
        for key in self._arguments:
            argument = self._arguments[key]
@@ -290,7 +299,10 @@ class CliFunction:
                if value != default:
                    if argument.is_option:
                        if argument.is_bool:
+                            if value:
                                vals = [(argument.longname,)]
+                            else:
+                                vals = [('--no-' + argument.longname[2:],)]
                        elif 'action' in argument.kwargs and argument.kwargs['action'] == 'append':
                            vals = [(argument.longname, str(val)) for val in value]
                        else:
@@ -326,7 +338,7 @@ if __name__ == '__main__':
    class OneCliFunction(CliFunction):
        def __init__(self):
            super().__init__(
-                config_file='cli_function_test_config.py',
+                default_config_file='cli_function_test_config.py',
                description = '''\
 Description of this
 amazing function!
@@ -454,7 +466,8 @@ amazing function!
    # get_cli
    assert one_cli_function.get_cli(pos_mandatory=1, asdf='B') == [('--asdf', 'B'), ('--bool-cli',), ('1',)]
    assert one_cli_function.get_cli(pos_mandatory=1, asdf='B', qwer='R') == [('--asdf', 'B'), ('--bool-cli',), ('--qwer', 'R'), ('1',)]
-    assert one_cli_function.get_cli(pos_mandatory=1, bool_true=False) == [('--bool-cli',), ('--bool-true',), ('1',)]
+    assert one_cli_function.get_cli(pos_mandatory=1, bool_true=False) == [('--bool-cli',), ('--no-bool-true',), ('1',)]
+    assert one_cli_function.get_cli(pos_mandatory=1, bool_false=True) == [('--bool-cli',), ('--bool-false',), ('1',)]
    assert one_cli_function.get_cli(pos_mandatory=1, pos_optional=2, args_star=['asdf', 'qwer']) == [('--bool-cli',), ('1',), ('2',), ('asdf',), ('qwer',)]
    assert one_cli_function.get_cli(pos_mandatory=1, append=['2', '3']) == [('--append', '2'), ('--append', '3',), ('--bool-cli',), ('1',)]

--- a/common.py
+++ b/common.py
--- a/9
+++ b/9
@@ -15,12 +15,9 @@ https://github.com/cirosantilli/linux-kernel-module-cheat#rootfs_overlay
 ''')

    def build(self):
-        # TODO: print rsync equivalent, move into shell_helpers.
-        distutils.dir_util.copy_tree(
-            self.env['rootfs_overlay_dir'],
-            self.env['out_rootfs_overlay_dir'],
-            preserve_symlinks=True,
-            update=1,
+        self.sh.copy_dir_if_update(
+            srcdir=self.env['rootfs_overlay_dir'],
+            destdir=self.env['out_rootfs_overlay_dir'],
        )

 if __name__ == '__main__':
--- a/2
+++ b/2
@@ -6,7 +6,7 @@ class Main(common.LkmcCliFunction):
    def __init__(self):
        super().__init__(
            defaults={
-                'print_time': False,
+                'show_time': False,
            },
            description='''\
 Get the value of a gem5 stat from the stats.txt file.
--- a/2
+++ b/2
@@ -6,7 +6,7 @@ class Main(common.LkmcCliFunction):
    def __init__(self):
        super().__init__(
            defaults = {
-                'print_time': False,
+                'show_time': False,
            },
            description='''\
 Print the value of a self.env['py'] variable.
--- a/include/README.adoc
+++ b/include/README.adoc
@@ -1 +0,0 @@
-https://github.com/cirosantilli/linux-kernel-module-cheat#include-directory
--- a/kernel_modules/Makefile
+++ b/kernel_modules/Makefile
@@ -1,15 +1,17 @@
-ifeq ($(OBJECT_FILES),)
-# Hardcoding  LKMC_MODULE_SUBDIRS here because is not defined.
-obj-m += $(addsuffix .o, $(notdir $(basename $(filter-out %.mod.c, $(wildcard $(BR2_EXTERNAL_LKMC_PATH)/kernel_modules/*.c)))))
-else
 # Trying to do:
 # $(MAKE) -C '$(LINUX_DIR)' M='$(M)' hello.ko hello2.ko
 # to restrict which modules are built leads to failures
 # when doing parallel builds. The only solution I could find
 # was to let the host select obj-m itself.
 obj-m += $(OBJECT_FILES)
-endif
-ccflags-y := -DDEBUG -g -std=gnu99 -Werror -Wno-declaration-after-statement -Wframe-larger-than=1000000000
+ccflags-y := \
+  -DDEBUG \
+  -ggdb3 \
+  -std=gnu99 \
+  -Werror \
+  -Wframe-larger-than=1000000000 \
+  -Wno-declaration-after-statement \
+  $(CCFLAGS)

 .PHONY: all

--- a/kernel_modules/anonymous_inode.c
+++ b/kernel_modules/anonymous_inode.c
@@ -9,7 +9,7 @@
 #include <linux/printk.h> /* printk */
 #include <linux/uaccess.h> /* copy_from_user */

-#include "../include/anonymous_inode.h"
+#include <lkmc/anonymous_inode.h>

 static struct dentry *debugfs_file;
 static u32 myval = 1;
--- a/kernel_modules/ioctl.c
+++ b/kernel_modules/ioctl.c
@@ -5,7 +5,7 @@
 #include <linux/printk.h> /* printk */
 #include <linux/uaccess.h> /* copy_from_user, copy_to_user */

-#include "../include/ioctl.h"
+#include <lkmc/ioctl.h>

 static struct dentry *debugfs_file;

--- a/kernel_modules/netlink.c
+++ b/kernel_modules/netlink.c
@@ -6,7 +6,7 @@
 #include <linux/skbuff.h>
 #include <net/sock.h>

-#include "../include/netlink.h"
+#include <lkmc/netlink.h>

 struct sock *nl_sk = NULL;

--- a/kernel_modules/pmccntr.c
+++ b/kernel_modules/pmccntr.c
@@ -1,4 +1,4 @@
-/* https://github.com/cirosantilli/linux-kernel-module-cheat#pmccntr */
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#arm-pmccntr */

 #include <linux/debugfs.h>
 #include <linux/errno.h> /* EFAULT */
--- a/kernel_modules/ring0.c
+++ b/kernel_modules/ring0.c
@@ -3,13 +3,13 @@
 #include <linux/module.h>
 #include <linux/kernel.h>

-#include "../include/ring0.h"
+#include <lkmc/ring0.h>

 static int myinit(void)
 {
 #if defined(__x86_64__) || defined(__i386__)
-	Ring0Regs ring0_regs;
-	ring0_get_control_regs(&ring0_regs);
+	LkmcRing0Regs ring0_regs;
+	lkmc_ring0_get_control_regs(&ring0_regs);
 	pr_info("cr0 = 0x%8.8llX\n", (unsigned long long)ring0_regs.cr0);
 	pr_info("cr2 = 0x%8.8llX\n", (unsigned long long)ring0_regs.cr2);
 	pr_info("cr3 = 0x%8.8llX\n", (unsigned long long)ring0_regs.cr3);
--- a/lkmc.c
+++ b/lkmc.c
@@ -11,22 +11,15 @@ void lkmc_assert(bool condition) {
        lkmc_assert_fail();
 }

-void lkmc_assert_fail() {
-    puts("lkmc_test_fail");
+void lkmc_assert_fail(void) {
    exit(1);
 }

-bool lkmc_vector_equal(size_t n, double *v1, double *v2, double max_err) {
-    double sum = 0.0;
-    double diff;
-    size_t i;
-    for (i = 0; i < n; ++i) {
-        diff = v1[i] - v2[i];
-        sum += diff * diff;
+void lkmc_baremetal_on_exit_callback(int status, void *arg) {
+    (void)arg;
+    if (status != 0) {
+        printf("lkmc_exit_status_%d\n", status);
    }
-    if (sqrt(sum)/n > max_err)
-        return false;
-    return true;
 }

 #if defined(__aarch64__)
--- a/lkmc.h
+++ b/lkmc.h
@@ -11,7 +11,7 @@

 void lkmc_assert(bool);
 void lkmc_assert_fail();
-bool lkmc_vector_equal(size_t n, double *v1, double *v2, double max_err);
+void lkmc_baremetal_on_exit_callback(int status, void *arg);
 #endif

 /* https://stackoverflow.com/questions/1489932/how-to-concatenate-twice-with-the-c-preprocessor-and-expand-a-macro-as-in-arg */
--- a/lkmc/README.adoc
+++ b/lkmc/README.adoc
@@ -0,0 +1 @@
+https://github.com/cirosantilli/linux-kernel-module-cheat#lkmc-directory
--- a/lkmc/init.py
+++ b/lkmc/init.py
--- a/lkmc/add.c
+++ b/lkmc/add.c
@@ -0,0 +1,13 @@
+#include <lkmc.h>
+
+int main(void) {
+    int i, j, k;
+    i = 1;
+    /* test-gdb-op1 */
+    j = 2;
+    /* test-gdb-op2 */
+    k = i + j;
+    /* test-gdb-result */
+    if (k != 3)
+        lkmc_assert_fail();
+}
--- a/lkmc/add.py
+++ b/lkmc/add.py
@@ -0,0 +1,9 @@
+def test(self):
+    self.sendline('tbreak main')
+    self.sendline('continue')
+    self.continue_to('op1')
+    assert self.get_int('i') == 1
+    self.continue_to('op2')
+    assert self.get_int('j') == 2
+    self.continue_to('result')
+    assert self.get_int('k') == 3
--- a/include/anonymous_inode.h
+++ b/include/anonymous_inode.h
@@ -1,5 +1,7 @@
-#ifndef IOCTL_H
-#define IOCTL_H
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#anonymous-inode */
+
+#ifndef LKMC_ANONYMOUS_INODE_H
+#define LKMC_ANONYMOUS_INODE_H

 #include <linux/ioctl.h>

--- a/lkmc/assert_fail.c
+++ b/lkmc/assert_fail.c
@@ -0,0 +1,18 @@
+/* Let's see what happens when an assert fails.
+ *
+ * Outcome on Ubuntu 19.04 shows the failure line:
+ *
+ *     assert_fail.out: /path/to/linux-kernel-module-cheat/userland/c/assert_fail.c:15: main: Assertion `0' failed.
+ *
+ * and exit status 134 == 128 + 6, which corresponds to SIGABORT (6).
+ */
+
+#include <assert.h>
+#include <stdlib.h>
+#include <stdio.h>
+
+int main(void) {
+    assert(0);
+    puts("here");
+    return EXIT_SUCCESS;
+}
--- a/lkmc/hello.c
+++ b/lkmc/hello.c
@@ -0,0 +1,9 @@
+/* Print hello to stdout ;-) */
+
+#include <stdio.h>
+#include <stdlib.h>
+
+int main(void) {
+    puts("hello");
+    return EXIT_SUCCESS;
+}
--- a/lkmc/import_path.py
+++ b/lkmc/import_path.py
@@ -0,0 +1,34 @@
+#!/usr/bin/env python3
+
+import importlib.machinery
+import importlib.util
+import os
+import sys
+
+root_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+def import_path(path):
+    '''
+    https://stackoverflow.com/questions/2601047/import-a-python-module-without-the-py-extension
+    https://stackoverflow.com/questions/31773310/what-does-the-first-argument-of-the-imp-load-source-method-do
+    '''
+    module_name = os.path.basename(path).replace('-', '_')
+    spec = importlib.util.spec_from_loader(
+        module_name,
+        importlib.machinery.SourceFileLoader(module_name, path)
+    )
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    sys.modules[module_name] = module
+    return module
+
+def import_path_relative_root(basename):
+    return import_path(os.path.join(root_dir, basename))
+
+def import_path_main(basename):
+    '''
+    Import an object of the Main class of a given file.
+
+    By convention, we call the main object of all our CLI scripts as Main.
+    '''
+    return import_path_relative_root(basename).Main()
--- a/include/ioctl.h
+++ b/include/ioctl.h
@@ -1,5 +1,7 @@
-#ifndef IOCTL_H
-#define IOCTL_H
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#ioctl */
+
+#ifndef LKMC_IOCTL_H
+#define LKMC_IOCTL_H

 #include <linux/ioctl.h>

--- a/baremetal/interactive/assert_fail.c
+++ b/baremetal/interactive/assert_fail.c
@@ -3,4 +3,3 @@
 int main(void) {
    lkmc_assert_fail();
 }
-
--- a/lkmc/m5ops.h
+++ b/lkmc/m5ops.h
@@ -0,0 +1,56 @@
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#m5ops-instructions */
+
+#ifndef LKMC_M5OPS_H
+#define LKMC_M5OPS_H
+
+#if LKMC_M5OPS_ENABLE == 1
+
+#if defined(__arm__)
+
+#define LKMC_M5OPS_CHECKPOINT_ASM mov r0, #0; mov r1, #0; mov r2, #0; mov r3, #0; .inst 0xEE000110 | (0x43 << 16)
+#define LKMC_M5OPS_DUMPSTATS_ASM  mov r0, #0; mov r1, #0; mov r2, #0; mov r3, #0; .inst 0xEE000110 | (0x41 << 16)
+#define LKMC_M5OPS_EXIT_ASM       mov r0, #0; mov r1, #0;                         .inst 0xEE000110 | (0x21 << 16)
+#define LKMC_M5OPS_FAIL_1_ASM     mov r0, #0; mov r1, #0; mov r2, #1; mov r3, #0; .inst 0xEE000110 | (0x22 << 16)
+#define LKMC_M5OPS_RESETSTATS_ASM mov r0, #0; mov r1, #0; mov r2, #0; mov r3, #0; .inst 0xEE000110 | (0x40 << 16)
+
+#define LKMC_M5OPS_CHECKPOINT __asm__ __volatile__ ("mov r0, #0; mov r1, #0; mov r2, #0; mov r3, #0; .inst 0xEE000110 | (0x43 << 16);" : : : "r0", "r1", "r2", "r3")
+#define LKMC_M5OPS_DUMPSTATS  __asm__ __volatile__ ("mov r0, #0; mov r1, #0; mov r2, #0; mov r3, #0; .inst 0xEE000110 | (0x41 << 16);" : : : "r0", "r1", "r2", "r3")
+#define LKMC_M5OPS_EXIT       __asm__ __volatile__ ("mov r0, #0; mov r1, #0;                         .inst 0xEE000110 | (0x21 << 16);" : : : "r0", "r1"            )
+#define LKMC_M5OPS_FAIL_1     __asm__ __volatile__ ("mov r0, #0; mov r1, #0; mov r2, #1; mov r3, #0; .inst 0xEE000110 | (0x22 << 16);" : : : "r0", "r1", "r2", "r3")
+#define LKMC_M5OPS_RESETSTATS __asm__ __volatile__ ("mov r0, #0; mov r1, #0; mov r2, #0; mov r3, #0; .inst 0xEE000110 | (0x40 << 16);" : : : "r0", "r1", "r2", "r3")
+
+#elif defined(__aarch64__)
+
+#define LKMC_M5OPS_CHECKPOINT_ASM mov x0, #0; mov x1, #0; .inst 0xFF000110 | (0x43 << 16);
+#define LKMC_M5OPS_DUMPSTATS_ASM  mov x0, #0; mov x1, #0; .inst 0xFF000110 | (0x41 << 16);
+#define LKMC_M5OPS_EXIT_ASM       mov x0, #0;             .inst 0XFF000110 | (0x21 << 16);
+#define LKMC_M5OPS_FAIL_1_ASM     mov x0, #0; mov x1, #1; .inst 0xFF000110 | (0x22 << 16);
+#define LKMC_M5OPS_RESETSTATS_ASM mov x0, #0; mov x1, #0; .inst 0XFF000110 | (0x40 << 16);
+
+#define LKMC_M5OPS_CHECKPOINT __asm__ __volatile__ ("mov x0, #0; mov x1, #0; .inst 0xFF000110 | (0x43 << 16);" : : : "x0", "x1")
+#define LKMC_M5OPS_DUMPSTATS  __asm__ __volatile__ ("mov x0, #0; mov x1, #0; .inst 0xFF000110 | (0x41 << 16);" : : : "x0", "x1")
+#define LKMC_M5OPS_EXIT       __asm__ __volatile__ ("mov x0, #0;             .inst 0XFF000110 | (0x21 << 16);" : : : "x0"      )
+#define LKMC_M5OPS_FAIL_1     __asm__ __volatile__ ("mov x0, #0; mov x1, #1; .inst 0xFF000110 | (0x22 << 16);" : : : "x0", "x1")
+#define LKMC_M5OPS_RESETSTATS __asm__ __volatile__ ("mov x0, #0; mov x1, #0; .inst 0XFF000110 | (0x40 << 16);" : : : "x0", "x1")
+
+#else
+#error m5ops not implemented for the current arch
+#endif
+
+#else
+
+#define LKMC_M5OPS_CHECKPOINT_ASM
+#define LKMC_M5OPS_DUMPSTATS_ASM
+#define LKMC_M5OPS_EXIT_ASM
+#define LKMC_M5OPS_FAIL_1_ASM
+#define LKMC_M5OPS_RESETSTATS_ASM
+
+#define LKMC_M5OPS_CHECKPOINT
+#define LKMC_M5OPS_DUMPSTATS
+#define LKMC_M5OPS_EXIT
+#define LKMC_M5OPS_FAIL_1
+#define LKMC_M5OPS_RESETSTATS
+
+#endif
+
+#endif
--- a/lkmc/math.h
+++ b/lkmc/math.h
@@ -0,0 +1,20 @@
+#ifndef LKMC_MATH_H
+#define LKMC_MATH_H
+
+#include <math.h>
+#include <stdbool.h>
+
+bool lkmc_vector_equal(size_t n, double *v1, double *v2, double max_err) {
+    double sum = 0.0;
+    double diff;
+    size_t i;
+    for (i = 0; i < n; ++i) {
+        diff = v1[i] - v2[i];
+        sum += diff * diff;
+    }
+    if (sqrt(sum)/n > max_err)
+        return false;
+    return true;
+}
+
+#endif
--- a/include/netlink.h
+++ b/include/netlink.h
@@ -1,5 +1,7 @@
-#ifndef NETLINK_H
-#define NETLINK_H
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#netlink-sockets */
+
+#ifndef LKMC_NETLINK_H
+#define LKMC_NETLINK_H

 /* Socket identifier, matches userland. TODO can be anything?
 * Is there a more scalable way to do it? E.g. ioctl device,
--- a/userland/common_userland.h
+++ b/userland/common_userland.h
@@ -1,5 +1,5 @@
-#ifndef COMMON_H
-#define COMMON_H
+#ifndef LKMC_PAGEMAP_H
+#define LKMC_PAGEMAP_H

 #define _XOPEN_SOURCE 700
 #include <fcntl.h> /* open */
@@ -17,7 +17,7 @@ typedef struct {
    unsigned int file_page : 1;
    unsigned int swapped : 1;
    unsigned int present : 1;
-} PagemapEntry;
+} LkmcPagemapEntry;

 /* Parse the pagemap entry for the given virtual address.
 *
@@ -26,8 +26,7 @@ typedef struct {
 * @param[in]  vaddr      virtual address to get entry for
 * @return                0 for success, 1 for failure
 */
-int pagemap_get_entry(PagemapEntry *entry, int pagemap_fd, uintptr_t vaddr)
-{
+int lkmc_pagemap_get_entry(LkmcPagemapEntry *entry, int pagemap_fd, uintptr_t vaddr) {
    size_t nread;
    ssize_t ret;
    uint64_t data;
@@ -62,8 +61,7 @@ int pagemap_get_entry(PagemapEntry *entry, int pagemap_fd, uintptr_t vaddr)
 * @param[in]  vaddr virtual address to get entry for
 * @return           0 for success, 1 for failure
 */
-int virt_to_phys_user(uintptr_t *paddr, pid_t pid, uintptr_t vaddr)
-{
+int lkmc_pagemap_virt_to_phys_user(uintptr_t *paddr, pid_t pid, uintptr_t vaddr) {
    char pagemap_file[BUFSIZ];
    int pagemap_fd;

@@ -72,8 +70,8 @@ int virt_to_phys_user(uintptr_t *paddr, pid_t pid, uintptr_t vaddr)
    if (pagemap_fd < 0) {
        return 1;
    }
-    PagemapEntry entry;
-    if (pagemap_get_entry(&entry, pagemap_fd, vaddr)) {
+    LkmcPagemapEntry entry;
+    if (lkmc_pagemap_get_entry(&entry, pagemap_fd, vaddr)) {
        return 1;
    }
    close(pagemap_fd);
--- a/include/ring0.h
+++ b/include/ring0.h
@@ -1,29 +1,31 @@
-#if defined(__x86_64__) || defined(__i386__)
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#ring0 */

+#ifndef LKMC_RING0_H
+#define LKMC_RING0_H
+#if defined(__x86_64__) || defined(__i386__)
 #ifdef THIS_MODULE
 #include <linux/kernel.h>
 #if defined(__x86_64__)
-typedef u64 T;
+typedef u64 LkmcRing0RegsType;
 #elif defined(__i386__)
-typedef u32 T;
+typedef u32 LkmcRing0RegsType;
 #endif
 #else
 #include <stdint.h>
 #if defined(__x86_64__)
-typedef uint64_t T;
+typedef uint64_t LkmcRing0RegsType;
 #elif defined(__i386__)
-typedef uint32_t T;
+typedef uint32_t LkmcRing0RegsType;
 #endif
 #endif

 typedef struct {
-    T cr0;
-    T cr2;
-    T cr3;
-} Ring0Regs;
+    LkmcRing0RegsType cr0;
+    LkmcRing0RegsType cr2;
+    LkmcRing0RegsType cr3;
+} LkmcRing0Regs;

-void ring0_get_control_regs(Ring0Regs *ring0_regs)
-{
+void lkmc_ring0_get_control_regs(LkmcRing0Regs *ring0_regs) {
 #if defined(__x86_64__)
    __asm__ __volatile__ (
        "mov %%cr0, %%rax;"
@@ -70,5 +72,5 @@ void ring0_get_control_regs(Ring0Regs *ring0_regs)
    );
 #endif
 }
-
+#endif
 #endif
--- a/path_properties.py
+++ b/path_properties.py
@@ -0,0 +1,413 @@
+#!/usr/bin/env python3
+
+import os
+
+from shell_helpers import LF
+
+class PathProperties:
+    default_c_std = 'c11'
+    default_cxx_std = 'c++17'
+    default_properties = {
+        'allowed_archs': None,
+        'c_std': default_c_std,
+        'cc_flags': [
+            '-Wall', LF,
+            '-Werror', LF,
+            '-Wextra', LF,
+            '-Wno-unused-function', LF,
+            '-fopenmp', LF,
+            '-ggdb3', LF,
+            # PIE causes the following problems:
+            # * QEMU GDB step debug does not find breakpoints:
+            #   https://stackoverflow.com/questions/51310756/how-to-gdb-step-debug-a-dynamically-linked-executable-in-qemu-user-mode/51343326#51343326
+            # * when writing assembly code, we have to constantly think about it:
+            #   https://stackoverflow.com/questions/2463150/what-is-the-fpie-option-for-position-independent-executables-in-gcc-and-ld/51308031#51308031
+            # As of 91986fb2955f96e06d1c5ffcc5536ba9f0af1fd9, our Buildroot toolchain
+            # does not have it enabled by default, but the Ubuntu 18.04 host toolchain does.
+            '-fno-pie', LF,
+            '-no-pie', LF,
+        ],
+        'cc_flags_after': [],
+        'cc_pedantic': True,
+        'cxx_std': default_cxx_std,
+        # Expected program exit status. When signals are raised, this refers
+        # to the native exit status. as reported by Bash #?.
+        'exit_status': 0,
+        'extra_objs_baremetal_bootloader': False,
+        # We should get rid of this if we ever properly implement dependency graphs.
+        'extra_objs_lkmc_common': False,
+        'extra_objs_userland_asm': False,
+        'interactive': False,
+        # The script takes a perceptible amount of time to run. Possibly an infinite loop.
+        'more_than_1s': False,
+        # The path should not be built. E.g., it is symlinked into multiple archs.
+        'no_build': False,
+        # The path does not generate an executable in itself, e.g.
+        # it only generates intermediate object files. Therefore it
+        # should not be run while testing.
+        'no_executable': False,
+        # the test receives a signal. We skip those tests for now,
+        # on userland because we are lazy to figure out the exact semantics
+        # of how Python + QEMU + gem5 determine the exit status of signals.
+        'receives_signal': False,
+        # The script requires a non-trivial argument to be passed to run properly.
+        'requires_argument': False,
+        'requires_dynamic_library': False,
+        'requires_m5ops': False,
+        # gem5 fatal: syscall getcpu (#168) unimplemented.
+        'requires_syscall_getcpu': False,
+        'requires_semihosting': False,
+        # Requires certain of our custom kernel modules to be inserted to run.
+        'requires_kernel_modules': False,
+        # The example requires sudo, which usually implies that it can do something
+        # deeply to the system it runs on, which would preventing further interactive
+        # or test usage of the system, for example poweroff or messing up the GUI.
+        'requires_sudo': False,
+        # We were lazy to properly classify why we are skipping these tests.
+        # TODO get it done.
+        'skip_run_unclassified': False,
+        # Aruments added automatically to run when running tests,
+        # but not on manual running.
+        'test_run_args': {
+            'ctrl_c_host': True,
+            'show_stdout': False,
+            'show_time': False,
+            'background': True,
+        },
+    }
+
+    '''
+    Encodes properties of userland and baremetal paths.
+    For directories, it applies to all files under the directory.
+    Used to determine how to build and test the examples.
+    '''
+    def __init__(
+        self,
+        properties
+    ):
+        for key in properties:
+            if not key in self.default_properties:
+                raise ValueError('Unknown key: {}'.format(key))
+        self.properties = properties.copy()
+
+    def __getitem__(self, key):
+        return self.properties[key]
+
+    def __repr__(self):
+        return str(self.properties)
+
+    def set_path_components(self, path_components):
+        self.path_components = path_components
+
+    def should_be_built(self, env, link=False):
+        if len(self.path_components) > 1 and \
+                self.path_components[1] == 'libs' and \
+                not env['package_all'] and \
+                not self.path_components[2] in env['package']:
+            return False
+        return \
+            not self['no_build'] and \
+            (
+                self['allowed_archs'] is None or
+                env['arch'] in self['allowed_archs']
+            ) and \
+            not (
+                link and
+                self['no_executable']
+            )
+
+    def should_be_tested(self, env):
+        return (
+            self.should_be_built(env) and
+            not self['interactive'] and
+            not self['more_than_1s'] and
+            not self['no_executable'] and
+            not self['requires_argument'] and
+            not self['requires_kernel_modules'] and
+            not self['requires_sudo'] and
+            not self['skip_run_unclassified'] and
+            not (
+                env['emulator'] == 'gem5' and
+                (
+                    self['requires_dynamic_library'] or
+                    self['requires_semihosting'] or
+                    self['requires_syscall_getcpu']
+                )
+            ) and
+            not (
+                env['emulator'] == 'qemu' and
+                (
+                    self['requires_m5ops']
+                )
+            )
+        )
+
+    def update(self, other):
+        other_tmp_properties = other.properties.copy()
+        if 'cc_flags' in self.properties and 'cc_flags' in other_tmp_properties:
+            other_tmp_properties['cc_flags'] = \
+                self.properties['cc_flags'] + \
+                other_tmp_properties['cc_flags']
+        if 'test_run_args' in self.properties and 'test_run_args' in other_tmp_properties:
+            other_tmp_properties['test_run_args'] = {
+                **self.properties['test_run_args'],
+                **other_tmp_properties['test_run_args']
+            }
+        return self.properties.update(other_tmp_properties)
+
+class PrefixTree:
+    def __init__(self, path_properties_dict=None, children=None):
+        if path_properties_dict is None:
+            path_properties_dict = {}
+        if children is None:
+            children = {}
+        self.children = children
+        self.path_properties = PathProperties(path_properties_dict)
+
+    @staticmethod
+    def make_from_tuples(tuples):
+        def tree_from_tuples(tuple_):
+            if not type(tuple_) is tuple:
+                tuple_ = (tuple_, {})
+            cur_properties, cur_children = tuple_
+            return PrefixTree(cur_properties, cur_children)
+        top_tree = tree_from_tuples(tuples)
+        todo_trees = [top_tree]
+        while todo_trees:
+            cur_tree = todo_trees.pop()
+            cur_children = cur_tree.children
+            for child_key in cur_children:
+                new_tree = tree_from_tuples(cur_children[child_key])
+                cur_children[child_key] = new_tree
+                todo_trees.append(new_tree)
+        return top_tree
+
+def get(path):
+    cur_node = path_properties_tree
+    path_components = path.split(os.sep)
+    path_properties = PathProperties(cur_node.path_properties.properties.copy())
+    for path_component in path_components:
+        if path_component in cur_node.children:
+            cur_node = cur_node.children[path_component]
+            path_properties.update(cur_node.path_properties)
+        else:
+            break
+    path_properties.set_path_components(path_components)
+    return path_properties
+
+gnu_extension_properties = {
+    'c_std': 'gnu11',
+    'cc_pedantic': False,
+    'cxx_std': 'gnu++17'
+}
+freestanding_properties = {
+    'cc_flags': [
+        '-ffreestanding', LF,
+        '-nostdlib', LF,
+        '-static', LF,
+    ],
+    'extra_objs_userland_asm': False,
+}
+# See: https://github.com/cirosantilli/linux-kernel-module-cheat#path-properties
+path_properties_tuples = (
+    PathProperties.default_properties,
+    {
+        'baremetal': (
+            {},
+            {
+                'arch': (
+                    {},
+                    {
+                        'arm': (
+                            {'allowed_archs': {'arm'}},
+                            {
+                                'gem5_assert.S': {'requires_m5ops': True},
+                                'multicore.S': {'test_run_args': {'cpus': 2}},
+                                'no_bootloader': (
+                                    {'extra_objs_baremetal_bootloader': False},
+                                    {
+                                        'gem5_exit.S': {'requires_m5ops': True},
+                                        'semihost_exit.S': {'requires_semihosting': True},
+                                    }
+                                ),
+                                'return1.S': {'exit_status': 1},
+                                'semihost_exit.S': {'requires_semihosting': True},
+                            },
+
+                        ),
+                        'aarch64': (
+                            {'allowed_archs': {'aarch64'}},
+                            {
+                                'multicore.S': {'test_run_args': {'cpus': 2}},
+                                'no_bootloader': (
+                                    {'extra_objs_baremetal_bootloader': False},
+                                    {
+                                        'gem5_exit.S': {'requires_m5ops': True},
+                                        'semihost_exit.S': {'requires_semihosting': True},
+                                    }
+                                ),
+                                'return1.S': {'exit_status': 1},
+                                'semihost_exit.S': {'requires_semihosting': True},
+                            },
+                        )
+                    }
+                ),
+                'assert_fail.c': {'exit_status': 134},
+                'lkmc_assert_fail.c': {'exit_status': 1},
+                'exit1.c': {'exit_status': 1},
+                'infinite_loop.c': {'more_than_1s': True},
+                'lib': (
+                    {'no_executable': True},
+                    {}
+                ),
+                'getchar.c': {'interactive': True},
+                'return1.c': {'exit_status': 1},
+                'return2.c': {'exit_status': 2},
+            }
+        ),
+        'userland': (
+            {
+                'cc_flags_after': [
+                    '-lm', LF,
+                    '-pthread', LF,
+                ],
+            },
+            {
+                'arch': (
+                    {
+                        'extra_objs_userland_asm': True,
+                    },
+                    {
+                        'arm': (
+                            {
+                                'allowed_archs': {'arm'},
+                                'cc_flags': [
+                                    '-Xassembler', '-mcpu=cortex-a72', LF,
+                                    # To prevent:
+                                    # > vfp.S: Error: selected processor does not support <FPU instruction> in ARM mode
+                                    # https://stackoverflow.com/questions/41131432/cross-compiling-error-selected-processor-does-not-support-fmrx-r3-fpexc-in/52875732#52875732
+                                    # We aim to take the most extended mode currently available that works on QEMU.
+                                    '-Xassembler', '-mfpu=crypto-neon-fp-armv8.1', LF,
+                                    '-Xassembler', '-meabi=5', LF,
+                                    # Treat inline assembly as arm instead of thumb
+                                    # The opposite of -mthumb.
+                                    '-marm', LF,
+                                    # Make gcc generate .syntax unified for inline assembly.
+                                    # However, it gets ignored if -marm is given, which a GCC bug that was recently fixed:
+                                    # https://stackoverflow.com/questions/54078112/how-to-write-syntax-unified-ual-armv7-inline-assembly-in-gcc/54132097#54132097
+                                    # So we just write divided inline assembly for now.
+                                    '-masm-syntax-unified', LF,
+                                ]
+                            },
+                            {
+                                'c': (
+                                    {
+                                        'extra_objs_userland_asm': False,
+                                    },
+                                    {
+                                        'freestanding': freestanding_properties,
+                                    },
+                                ),
+                                'freestanding': freestanding_properties,
+                            }
+                        ),
+                        'aarch64': (
+                            {'allowed_archs': {'aarch64'}},
+                            {
+                                'c': (
+                                    {
+                                        'extra_objs_userland_asm': False,
+                                    },
+                                    {
+                                        'freestanding': freestanding_properties,
+                                    },
+                                ),
+                                'freestanding': freestanding_properties,
+                            }
+                        ),
+                        'fail.S': {'exit_status': 1},
+                        'main.c': {
+                            'extra_objs_userland_asm': False,
+                            'no_executable': True
+                        },
+                        'x86_64': (
+                            {'allowed_archs': {'x86_64'}},
+                            {
+                                'c': (
+                                    {
+                                        'extra_objs_userland_asm': False,
+                                    },
+                                    {
+                                        'freestanding': freestanding_properties,
+                                        'ring0.c': {
+                                            'exit_status': 139,
+                                            'receives_signal': True
+                                        }
+                                    }
+                                ),
+                                'freestanding': freestanding_properties,
+                                'lkmc_assert_eq_fail.S': {'exit_status': 1},
+                                'lkmc_assert_memcmp_fail.S': {'exit_status': 1},
+                            }
+                        ),
+                    }
+                ),
+                'c': (
+                    {},
+                    {
+                        'assert_fail.c': {
+                            'exit_status': 134,
+                            'receives_signal': True,
+                        },
+                        'false.c': {'exit_status': 1},
+                        'getchar.c': {'interactive': True},
+                        'infinite_loop.c': {'more_than_1s': True},
+                    }
+                ),
+                'gcc': gnu_extension_properties,
+                'kernel_modules': {**gnu_extension_properties, **{'requires_kernel_modules': True}},
+                'lkmc': (
+                    {'extra_objs_lkmc_common': True},
+                    {
+                        'assert_fail.c': {'exit_status': 1},
+                    }
+                ),
+                'libs': (
+                    {'requires_dynamic_library': True},
+                    {
+                        'libdrm': {'requires_sudo': True},
+                    }
+                ),
+                'linux': (
+                    gnu_extension_properties,
+                    {
+                        'ctrl_alt_del.c': {'requires_sudo': True},
+                        'init_env_poweroff.c': {'requires_sudo': True},
+                        'myinsmod.c': {'requires_sudo': True},
+                        'myrmmod.c': {'requires_sudo': True},
+                        'pagemap_dump.c': {'requires_argument': True},
+                        'poweroff.c': {'requires_sudo': True},
+                        'proc_events.c': {'requires_sudo': True},
+                        'proc_events.c': {'requires_sudo': True},
+                        'sched_getaffinity.c': {'requires_syscall_getcpu': True},
+                        'sched_getaffinity_threads.c': {
+                            'requires_syscall_getcpu': True,
+                            'more_than_1s': True,
+                        },
+                        'time_boot.c': {'requires_sudo': True},
+                        'virt_to_phys_user.c': {'requires_argument': True},
+                    }
+                ),
+                'posix': (
+                    {},
+                    {
+                        'count.c': {'more_than_1s': True},
+                        'sleep_forever.c': {'more_than_1s': True},
+                        'virt_to_phys_test.c': {'more_than_1s': True},
+                    }
+                ),
+            }
+        ),
+    }
+)
+path_properties_tree = PrefixTree.make_from_tuples(path_properties_tuples)
--- a/27
+++ b/27
@@ -5,41 +5,44 @@ import sys
 import telnetlib

 import common
-from shell_helpers import LF

-prompt = b'\n(qemu) '
-
-parser = self.get_argparse({
-    'description': '''\
+class Main(common.LkmcCliFunction):
+    def __init__(self):
+        super().__init__(
+            description='''\
 Run a command on the QEMU monitor of a running QEMU instance

 If the stdin is a terminal, open an interact shell. Otherwise,
 run commands from stdin and quit.
-'''
-})
-parser.add_argument(
+''',
+        )
+        self.add_argument(
            'command',
            help='If given, run this command and quit',
            nargs='*',
        )
-args = self.setup(parser)

+    def timed_main(self):
        def write_and_read(tn, cmd, prompt):
            tn.write(cmd.encode('utf-8'))
            return '\n'.join(tn.read_until(prompt).decode('utf-8').splitlines()[1:])[:-len(prompt)]

-with telnetlib.Telnet('localhost', kwargs['qemu_monitor_port']) as tn:
+        with telnetlib.Telnet('localhost', self.env['qemu_monitor_port']) as tn:
+            prompt = b'\n(qemu) '
            # Couldn't disable server echo, so just removing the write for now.
            # https://stackoverflow.com/questions/12421799/how-to-disable-telnet-echo-in-python-telnetlib
            # sock = tn.get_socket()
            # sock.send(telnetlib.IAC + telnetlib.WILL + telnetlib.ECHO)
            if os.isatty(sys.stdin.fileno()):
-        if kwargs['command'] == []:
+                if self.env['command'] == []:
                    print(tn.read_until(prompt).decode('utf-8'), end='')
                    tn.interact()
                else:
                    tn.read_until(prompt)
-            print(write_and_read(tn, ' '.join(kwargs['command']) + '\n', prompt))
+                    print(write_and_read(tn, ' '.join(self.env['command']) + '\n', prompt))
            else:
                tn.read_until(prompt)
                print(write_and_read(tn, sys.stdin.read() + '\n', prompt))
+
+if __name__ == '__main__':
+    Main().cli()
--- a/2
+++ b/2
@@ -12,7 +12,7 @@ class Main(common.LkmcCliFunction):
 https://github.com/cirosantilli/linux-kernel-module-cheat#release-zip
 ''',
            defaults = {
-                'print_time': False,
+                'show_time': False,
            }
        )
        self.zip_files = []
--- a/rootfs_overlay/.profile
+++ b/rootfs_overlay/.profile
@@ -0,0 +1,4 @@
+# https://github.com/cirosantilli/linux-kernel-module-cheat#busybox-shell-initrc-files
+echo "hello .profile"
+export PS1='\w\n\u@\h# '
+cd /lkmc
--- a/rootfs_overlay/anonymous_inode.sh
+++ b/rootfs_overlay/anonymous_inode.sh
@@ -1,5 +0,0 @@
-#!/bin/sh
-set -e
-insmod anonymous_inode.ko
-[ "$(/anonymous_inode.out /sys/kernel/debug/lkmc_anonymous_inode 3)" = "$(printf '1\n10\n100')" ]
-rmmod anonymous_inode
--- a/rootfs_overlay/conf.sh
+++ b/rootfs_overlay/conf.sh
@@ -1,2 +0,0 @@
-#!/bin/sh
-zcat /proc/config.gz | grep -Ei "${1:-}"
--- a/rootfs_overlay/etc/init.d/S98
+++ b/rootfs_overlay/etc/init.d/S98
@@ -1,5 +1,7 @@
 #!/bin/sh
+# https://github.com/cirosantilli/linux-kernel-module-cheat#init-busybox
 echo "hello S98"
+cd "$lkmc_home"
 if [ -n "$lkmc_eval" ]; then
  eval "$lkmc_eval"
 elif [ -n "$lkmc_eval_base64" ]; then
--- a/rootfs_overlay/eval.sh
+++ b/rootfs_overlay/eval.sh
@@ -1,18 +0,0 @@
-#!/bin/sh
-echo "$lkmc_eval"
-eval "$lkmc_eval"
-
-# Ideally, this script would do just:
-#
-## Get rid of the '-'.
-#shift
-#echo "$@"
-#
-# However, the kernel CLI parsing is crap, and the 4.14 docs lie.
-#
-# In particular, not all that is passed after "-" goes to an argument to init,
-# e.g. stuff with dots like "- ./poweroff.out" still gets treated specially and
-# does not go to init.
-#
-# This also likely means that the above solution is also unreliable in some cases,
-# and that in the end you just have to add a script to the root filesystem.
--- a/rootfs_overlay/eval_base64.sh
+++ b/rootfs_overlay/eval_base64.sh
@@ -1,2 +0,0 @@
-#!/bin/sh
-eval "$(printf "$lkmc_eval" | base64 -d)"
--- a/rootfs_overlay/ioctl.sh
+++ b/rootfs_overlay/ioctl.sh
@@ -1,7 +0,0 @@
-#!/bin/sh
-set -e
-f=/sys/kernel/debug/lkmc_ioctl
-insmod ioctl.ko
-[ "$(/ioctl.out "$f" 0 1)" = 2 ]
-[ "$(/ioctl.out "$f" 1 1 1)" = '2 0' ]
-rmmod ioctl
--- a/rootfs_overlay/lkmc/anonymous_inode.sh
+++ b/rootfs_overlay/lkmc/anonymous_inode.sh
@@ -0,0 +1,6 @@
+#!/bin/sh
+# https://github.com/cirosantilli/linux-kernel-module-cheat#anonymous-inode
+set -e
+insmod anonymous_inode.ko
+[ "$(./kernel_modules/anonymous_inode.out /sys/kernel/debug/lkmc_anonymous_inode 3)" = "$(printf '1\n10\n100')" ]
+rmmod anonymous_inode
--- a/rootfs_overlay/lkmc/character_device.sh
+++ b/rootfs_overlay/lkmc/character_device.sh
--- a/rootfs_overlay/lkmc/character_device_create.sh
+++ b/rootfs_overlay/lkmc/character_device_create.sh
--- a/rootfs_overlay/lkmc/conf.sh
+++ b/rootfs_overlay/lkmc/conf.sh
@@ -0,0 +1,3 @@
+#!/bin/sh
+# https://github.com/cirosantilli/linux-kernel-module-cheat#find-the-kernel-config
+zcat /proc/config.gz | grep -Ei "${1:-}"
--- a/rootfs_overlay/lkmc/count.sh
+++ b/rootfs_overlay/lkmc/count.sh
@@ -1,4 +1,5 @@
 #!/bin/sh
+# Count to infinity with 1 second sleep between each increment.
 # Generate infinitely many system calls :-)
 i=0
 while true; do
--- a/Show More
+++ b/Show More
				`@@ -1 +0,0 @@`
				`This folder contains examples that are not very testable: either are supposed to return 0, or are interactive, etc.`
				`@@ -1 +0,0 @@`
				`https://github.com/cirosantilli/linux-kernel-module-cheat#include-directory`
				`@@ -0,0 +1 @@`
				`https://github.com/cirosantilli/linux-kernel-module-cheat#lkmc-directory`