Make this repo good enough to move in cpp-cheat, x86-assembly-cheat and arm-assembly-cheat in

This commit is a large squash, the full development branch is available at:
https://github.com/cirosantilli/linux-kernel-module-cheat/tree/asm

This notably means a refactor of the userland build and testing, to support:

- improved assembly infrastructure unified across arm and x86
- native in-tree build and test helpers
- parallel building and testing, which implies thread_pool.py
- selection of what to build and test from the CLI
- path_properties.py to indicate how to build and run different examples
- in full system, move all userland stuff into /lkmc
- prefix everything that we defined across files with LKMC
- --gdb uber convenient helper
- remove import imp which was deprecated

Full commit messages from the branch follow:

1:

userland: add assembly support

Move arm assembly cheat here, and start some work on x86 cheat as well.

2:

document userland asm syscall interfaces

3:

userland assembly: structure readme

4:

x86 fail works

5:

asm: more links

6:

userland: add ported to all archs

7:

move all our stuff into /lkmc in guest

Motivation: userland is getting several new subdirectories, it would be
too insane to just dump all of that in the guest root filesystem.

To alleviate the cd pain, .profile puts user inside /lkmc by default.

8:

start the big userland migration

9:

migrate all

10:

bak

11:

build-userland-in-tree is now a Python command

./build calls it, we did this to allow --download-dependencies to work
perfectly.

12:

rename include to lkmc

13:

mtops.h is perfect now

14:

userland: make build perfect

15:

preparing test_user_mode, need to generalize stuff as usual

16:

asm: prefix every linux specific with linux/

17:

userland: maybe it really works

18:

userland: fix kernel version to work on older ubuntu

Expose --kernel-version to allow customization.

Update LTP info.

19:

userland: build really truly working now

userland test: start work, in a working state, but no features

20:

test-user-mode: make perfect like build-userland

Multithreading and target selection.

21:

userland: get a bit closer to perfection

22:

thread_pool: support passing thread IDs

Then use that to fix gem5 error log read race.

23:

userland: native testing

24:

userland: path properties getting nice!

25:

userland: move posix/environ from cpp-cheat

26:

gem5: --debug-flags without =, looks nicer whenever it can be done

27:

run: rename --wait-gdb in --gdb-wait, --gdb prefix might become a thing

28:

run: create --tmux-program gdb to open gem5 GDB

29:

run: create the uber convenient --gdb option

30:

userland: move getchar from cpp-cheat

31:

prebuilt: kernel boot aarch64 does not work on Ubuntu 16.04

32:

userland: x86_64 linux hello world make PIE

33:

userland: try to make userland executable selection saner

Only allow existing files to be built, stop extension expansion madness.

cli_function: get_cli print booleans properly, was printing without --no-
for negations.

34:

userland: only link to lkmc.o if needed

35:

path_properties: make data very compact with only tuples and dicts

Spend 2 hours of my life thinking about low value tree walks ;-)

36:

userland: move more userland/arch/ logic into property tree

37:

userland: make libs work

Working for build, but now test-user-mode-in-tree is not using --in-tree,
TODO fix later on.

38:

userland: make libs really work

39:

userland: document path_properties

40:

userland: classify linux

41:

waste your life

42:

common: fix absolute path runs

--gdb: allow running from arbitrary directory

43:

baremetal: arm allow using floating point instructions

44:

baremetal: stat preparing to make perfect like userland/

45:

run: fix image check logic accounting for userland

Was failing if I try to run userland (with abspath) when out/
directory is not present.

46:

cli-function: raise if the config file is given and does not exist

47:

common: define missing 'ld' variable, this broke m5 build

48:

rum: --qemu-which host now works for user mode as well as system

Don't fall back on host QEMU automatically, too much insanity.

49:

userland: refix silly mistakes

50:

userland: use path_properties flags for all builds, including lkmc. and userland/arch/main.c

Without this in particular, --gdb fails on assembly because main.c
was not being built with -ggdb3.

51:

userland: start refactor to show failing values on failure!

aarch64 basically done, but missing:

- other archs
- maybe convert main.c into C++ to use templates?
- full review of ASSERT_EQ calling convention issues not seen by tests
  by chance
- documentation

52:

readme: releases are more stable...

53:

submodules: sort gitmodules

54:

test-baremetal: same interface as test-user-mode

In particular, runs tests in parallel, and allows selecting given tests

55:

baremetal: allow arbitrary exit status with the magic string

test-baremetal: fix missing setting x0 return value

Examples were just returning on ret without setting x0, which led to
failures... those were not noticed because of how broken the testing system
was ;-)

56:

baremetal: ah, actually nope, it didn't work :-(

Workaround for now. Works on asserts, but not on exit 1.

Some other day, maybe.

https://github.com/cirosantilli/linux-kernel-module-cheat/issues/59

57:

panic on panic: improve behaviour description

58:

baremetal: get exit status working with on_exit :-)

59:

baremetal: implement C assert

60:

test-baremetal: remove commented out exit status workaround

61:

test-user-mode: handle exit status for signals. Fix #61.

62:

aarch64: fix ASSERT_EQ_REG tests on gem5

Was doing an 8-byte aligned store, which gem5 dislikes.

But the ARMARM says bad things may happen there, notably a signal:
"D1.8.2 SP alignment checking" so gem5 is not really too wrong,
QEMU just happens to work by chance.

63:

userland assembly: build empty.S and fail.S to toplevel and run fail.S with path_properties exit_status

They were just duplicating stuff needlessly while we don't support non-native in-tree builds,
which leads to executable conflicts for C file anyways.

64:

gem5: use a single build tree for all build types

gem5 already has different object names for each build type it seems, so
let's just make sure that works and save some disk space.

65:

userland x86_64: ASSERT_EQ show actual and expected values

66:

assert_fail.c: add to readme index

67:

userland x86_64: implement ASSERT_MEMCMP

68:

userland x86_64: allow ASSERT_EQ to take just about anything

69:

gas data sizes

70:

gas_data_sizes.S: make PIE for all ISAs

71:

x86: paddq

72:

x86 paddq: test entire family

73:

Get rid of imp, started giving deprecation warning every time in Python 3.7 in Ubuntu 19.04.

Please python stop torturing me with refactors.

Make ./run -u blow up if executable not found, otherwise I go crazy.

Get ./test-gdb back to life after the ./run relative path refactor, forgot to test this.

74:

fix run-toolchain, qemu-monitor, trace-boot, trace2line, bisect-linux-boot-gem5. Fixes part of #63

I'm sad no one reported qemu-monitor break, that one is kind of important.

count.out arguments broke it as an init program, since the kernel adds trash
parameters to every init.

Is anyone using this repo, I wonder? Keep pushing, keep pushing.
One day it gets good enough, and the whole world will see.

75:

x86 assembly: addpd

76:

Fix import_path circular dependency by splitting it out.

Use import thread_pool instead from, from is evil.

Fix poweroff.out path for ./trace-boot.

77:

run: rename cryptic tmu to tmux-split, ./run is good now so I never use it anymore explicitly

78:

assembly SIMD add: make uniform for all ISAs, mark as entry point to learning SIMD

79:

start moving arm-assembly-cheat readme in here

80:

arm assembly: move some more in

81:

move more arm in

82:

userland: attempt to fix all assembly example links to README

83:

assembly: improve organization of simd add

84:

ld2 move in

85:

Make userland / assembly getting started more uniform / visible

Forward --gcc-which to ./run --tmux.

Use gdb-multiarch for --gcc-which host.

86:

userland: disable PIE explicitly on command line for all executables

87:

userland: make userland content a better landing page

88:

build: check git version from --version and degrade gracefully

89:

build: make --dry-run work again on all

90:

import_path: importlib explicit for Ubuntu 16.04

91:

make all submodules point to my forks

git servers are insane, submodule implementation is crap, what can you do

92:

build: log warning on git too old for --update

93:

build-linux: do olddefconfig even if no fragments

In particular, gem5 kernel 4.15 needs it

94:

userland content: improve a bit landing page for cpp-cheat

.gitignore

@@ -1,9 +1,10 @@
 # Extensions and prefixes.
 *.tmp
 tmp.*
+*.tmp.*
 *~
-?*.gitignore
+*.gitignore
-gitignore*
+gitignore.*
 
 # Specific files.
 /data
@@ -22,6 +23,9 @@ __pycache__
 *.o
 *.out
 
+# Data to be plotted output.
+*.dat
+
 # Kernel modules.
 .cache.mk
 *.ko.cmd

.gitmodules

@@ -3,19 +3,28 @@
 	url = git://git.kernel.org/pub/scm/linux/kernel/git/mark/boot-wrapper-aarch64.git
 [submodule "submodules/binutils-gdb"]
 	path = submodules/binutils-gdb
+    # url = git://sourceware.org/git/binutils-gdb.git
 	url = https://github.com/cirosantilli/binutils-gdb
 [submodule "submodules/buildroot"]
 	path = submodules/buildroot
+    # url = git://git.busybox.net/buildroot
 	url = https://github.com/cirosantilli/buildroot
 	ignore = dirty
 [submodule "submodules/crosstool-ng"]
 	path = submodules/crosstool-ng
-	url = https://github.com/crosstool-ng/crosstool-ng
+    # url = https://github.com/crosstool-ng/crosstool-ng
+	url = https://github.com/cirosantilli/crosstool-ng
+[submodule "submodules/gcc"]
+	path = submodules/gcc
+    # url = git://gcc.gnu.org/git/gcc.git
+	url = https://github.com/cirosantilli/gcc
 [submodule "submodules/gem5"]
 	path = submodules/gem5
-	url = https://gem5.googlesource.com/public/gem5
+    # url = https://gem5.googlesource.com/public/gem5
+	url = https://github.com/cirosantilli/gem5
 [submodule "submodules/glibc"]
 	path = submodules/glibc
+    # url = git://sourceware.org/git/glibc.git
 	url = https://github.com/cirosantilli/glibc
 # The true upstream does not accept git submodule update --init --depth 1
 # git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
@@ -23,16 +32,16 @@
 # https://unix.stackexchange.com/questions/338578/linux-kernel-source-code-size-difference
 [submodule "submodules/linux"]
 	path = submodules/linux
+    # usl = git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
 	url = https://github.com/cirosantilli/linux
 [submodule "submodules/parsec-benchmark"]
 	path = submodules/parsec-benchmark
 	url = https://github.com/cirosantilli/parsec-benchmark
 [submodule "submodules/qemu"]
 	path = submodules/qemu
+    # url = https://github.com/qemu/qemu
 	url = https://github.com/cirosantilli/qemu
 [submodule "submodules/xen"]
 	path = submodules/xen
-	url = git://xenbits.xen.org/xen.git
+    # url = git://xenbits.xen.org/xen.git
-[submodule "submodules/gcc"]
+	url = https://github.com/cirosantilli/xen
-	path = submodules/gcc
-	url = https://github.com/cirosantilli/gcc

baremetal/add.c

@@ -1,13 +0,0 @@
-#include <lkmc.h>
-
-int main(void) {
-    int i, j, k;
-    i = 1;
-    /* test-gdb-op1 */
-    j = 2;
-    /* test-gdb-op2 */
-    k = i + j;
-    /* test-gdb-result */
-    if (k != 3)
-        lkmc_assert_fail();
-}

baremetal/add.c

@@ -0,0 +1 @@
+../lkmc/add.c

baremetal/add.py

@@ -1,9 +0,0 @@
-def test(self):
-    self.sendline('tbreak main')
-    self.sendline('continue')
-    self.continue_to('op1')
-    assert self.get_int('i') == 1
-    self.continue_to('op2')
-    assert self.get_int('j') == 2
-    self.continue_to('result')
-    assert self.get_int('k') == 3

baremetal/add.py

@@ -0,0 +1 @@
+../lkmc/add.py

baremetal/arch/aarch64/add.S

@@ -1,12 +1,13 @@
 .global main
 main:
     /* 1 + 2 == 3 */
-    mov x0, #1
+    mov x0, 1
     /* test-gdb-op1 */
-    add x1, x0, #2
+    add x1, x0, 2
     /* test-gdb-result */
-    cmp x1, #3
+    cmp x1, 3
     beq 1f
     bl lkmc_assert_fail
 1:
+    mov x0, 0
     ret

baremetal/arch/aarch64/c_from_as.S

@@ -1,5 +1,5 @@
 /* Call a C function. */
 .global main
 main:
-    mov x0, #0
+    mov x0, 0
     bl exit

baremetal/arch/aarch64/fadd.S

@@ -3,43 +3,45 @@
 .global main
 main:
     /* 1.5 + 2.5 == 4.0 */
-    fmov d0, #1.5
+    fmov d0, 1.5
     /* test-gdb-d0 */
-    fmov d1, #2.5
+    fmov d1, 2.5
     /* test-gdb-d1 */
     fadd d2, d0, d1
     /* test-gdb-d2 */
-    fmov d3, #4.0
+    fmov d3, 4.0
     fcmp d2, d3
     beq 1f
     bl lkmc_assert_fail
 1:
 
     /* Now in 32-bit. */
-    fmov s0, #1.5
+    fmov s0, 1.5
     /* test-gdb-s0 */
-    fmov s1, #2.5
+    fmov s1, 2.5
     /* test-gdb-s1 */
     fadd s2, s0, s1
     /* test-gdb-s2 */
     fadd s2, s0, s1
-    fmov s3, #4.0
+    fmov s3, 4.0
     fcmp s2, s3
     beq 1f
     bl lkmc_assert_fail
 1:
 
     /* Higher registers. */
-    fmov d28, #1.5
+    fmov d28, 1.5
     /* test-gdb-d28 */
-    fmov d29, #2.5
+    fmov d29, 2.5
     /* test-gdb-d29 */
     fadd d30, d28, d29
     /* test-gdb-d30 */
-    fmov d31, #4.0
+    fmov d31, 4.0
     /* test-gdb-d31 */
     fcmp d30, d31
     beq 1f
     bl lkmc_assert_fail
 1:
+
+    mov x0, 0
     ret

baremetal/arch/aarch64/multicore.S

@@ -3,7 +3,7 @@
 .global main
 main:
     /* Reset spinlock. */
-    mov x0, #0
+    mov x0, 0
     ldr x1, =spinlock
     str x0, [x1]
 
@@ -66,6 +66,7 @@ spinlock_start:
     wfe
     cbz x0, spinlock_start
 
+    mov x0, 0
     ret
 
 spinlock:

baremetal/arch/aarch64/no_bootloader/gem5_exit.S

@@ -1,3 +1,5 @@
+#include <lkmc/m5ops.h>
+
 .global mystart
 mystart:
-    mov x0, #0; .inst 0XFF000110 | (0x21 << 16);
+    LKMC_M5OPS_EXIT_ASM

baremetal/arch/aarch64/no_bootloader/semihost_exit.S

@@ -6,7 +6,7 @@ mystart:
     movk x1, 2, lsl 16
     ldr x2, =semihost_args
     str x1, [x2, 0]
-    mov x0, #0
+    mov x0, 0
     str x0, [x2, 8]
     mov x1, x2
     mov w0, 0x18

baremetal/arch/aarch64/regs.S

@@ -4,26 +4,26 @@
  */
 .global main
 main:
-    mov x0, #1
+    mov x0, 1
     /* test-gdb-x0 */
-    mov x1, #2
+    mov x1, 2
     /* test-gdb-x1 */
 
-    mov x29, #1
+    mov x29, 1
     /* test-gdb-x29 */
-    mov x30, #2
+    mov x30, 2
     /* test-gdb-x30 */
 
-    fmov d0, #1.5
+    fmov d0, 1.5
     /* test-gdb-d0 */
-    fmov d1, #2.5
+    fmov d1, 2.5
     /* test-gdb-d1 */
 
-    fmov d30, #1.5
+    fmov d30, 1.5
     /* test-gdb-d30 */
-    fmov d31, #2.5
+    fmov d31, 2.5
     /* test-gdb-d31 */
 
     /* Exit required since we messed up with x30 which is the lr. */
-    mov x0, #0
+    mov x0, 0
     bl exit

baremetal/arch/aarch64/return.S

@@ -1,4 +1,5 @@
 /* Return to ensure that the post main works. */
 .global main
 main:
+    mov x0, 0
     ret

baremetal/arch/aarch64/return1.S

@@ -0,0 +1,5 @@
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#magic-failure-string */
+.global main
+main:
+    mov x0, 1
+    ret

baremetal/arch/aarch64/svc_asm.S

@@ -16,6 +16,7 @@ main:
 1:
 
     /* Go home. */
+    mov x0, 0
     ret
 
 LKMC_GLOBAL(lkmc_vector_trap_handler)

baremetal/arch/arm/add.S

@@ -9,4 +9,5 @@ main:
     beq 1f
     bl lkmc_assert_fail
 1:
+    mov r0, #0
     bx lr

baremetal/arch/arm/dump_regs.c

@@ -0,0 +1,15 @@
+/* I want to move el and all other "what's the initial value of such system register" into here. */
+
+#include <stdio.h>
+#include <inttypes.h>
+
+int main(void) {
+    uint32_t cpsr;
+    uint32_t mvfr1;
+    __asm__ ("mrs %0, cpsr" : "=r" (cpsr) : :);
+    /* TODO this is blowing up an exception, how to I read from it? */
+    /*__asm__ ("vmrs %0, mvfr1" : "=r" (mvfr1) : :);*/
+    printf("cpsr  %" PRIx32 "\n", cpsr);
+    /*printf("mvfr1 %" PRIx32 "\n", mvfr1);*/
+    return 0;
+}

baremetal/arch/arm/gem5_assert.S

@@ -1,8 +1,10 @@
 /* assert 0x12345678 + 1 == 0x12345679 */
+
+#include <lkmc/m5ops.h>
+
 .global main
 main:
-    movw r0, #:lower16:myvar
+    adr r0, myvar
-    movt r0, #:upper16:myvar
     ldr r1, [r0]
     add r1, r1, #1
     str r1, [r0]
@@ -10,10 +12,8 @@ main:
     movt r2, #0x1234
     cmp r1, r2
     beq ok
-    /* m5 fail 1 */
+    LKMC_M5OPS_FAIL_1_ASM
-    mov r0, #0; mov r1, #0; mov r2, #1; mov r3, #0; .inst 0xEE000110 | (0x22 << 16);
 ok:
-    /* m5 exit */
+    LKMC_M5OPS_EXIT_ASM
-    mov r0, #0; mov r1, #0; .inst 0xEE000110 | (0x21 << 16);
 myvar:
     .word 0x12345678

baremetal/arch/arm/multicore.S

@@ -32,6 +32,7 @@ spinlock_start:
     wfe
     cmp r0, #0
     beq spinlock_start
+    mov r0, #0
     bx lr
 spinlock:
     .skip 4

baremetal/arch/arm/regs.S

@@ -5,4 +5,5 @@ main:
     /* test-gdb-r0 */
     mov r1, #2
     /* test-gdb-r1 */
+    mov r0, #0
     bx lr

baremetal/arch/arm/return1.S

@@ -0,0 +1,5 @@
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#magic-failure-string */
+.global main
+main:
+    mov r0, #1
+    bx lr

baremetal/assert_fail.c

@@ -0,0 +1 @@
+../lkmc/assert_fail.c

baremetal/exit.c

@@ -1,6 +0,0 @@
-#include <stdio.h>
-#include <stdlib.h>
-
-int main(void) {
-    exit(0);
-}

baremetal/exit0.c

@@ -0,0 +1,7 @@
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#magic-failure-string */
+
+#include <stdlib.h>
+
+int main(void) {
+    exit(0);
+}

baremetal/exit1.c

@@ -0,0 +1,7 @@
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#magic-failure-string */
+
+#include <stdlib.h>
+
+int main(void) {
+    exit(1);
+}

baremetal/getchar.c

@@ -1,3 +1,5 @@
+/* Test that input request through serial also works. */
+
 #include <stdio.h>
 #include <stdlib.h>
 

baremetal/hello.c

@@ -1,6 +0,0 @@
-#include <stdio.h>
-
-int main(void) {
-    puts("hello");
-    return 0;
-}

baremetal/hello.c

@@ -0,0 +1 @@
+../lkmc/hello.c

baremetal/interactive/README.adoc

@@ -1 +0,0 @@
-This folder contains examples that are not very testable: either are supposed to return 0, or are interactive, etc.

baremetal/interactive/exit1.c

@@ -1,6 +0,0 @@
-#include <stdio.h>
-#include <stdlib.h>
-
-int main(void) {
-    exit(1);
-}

baremetal/interactive/return1.c

@@ -1 +0,0 @@
-int main(void) { return 1; }

baremetal/lib/aarch64.S

@@ -14,6 +14,12 @@ mystart:
     /* Prepare the stack for main, mandatory for C code. */
     ldr x0, =stack_top
     mov sp, x0
+
+    /* https://github.com/cirosantilli/linux-kernel-module-cheat#magic-failure-string */
+    adr x0, lkmc_baremetal_on_exit_callback
+    bl on_exit
+
+    /* Run main. */
     bl main
 
     /* If main returns, exit. */

baremetal/lib/arm.S

@@ -1,5 +1,16 @@
+#include <lkmc.h>
+
 .global mystart
 mystart:
+    /* Prepare the stack for main, mandatory for C code. */
     ldr sp, =stack_top
+
+    /* https://github.com/cirosantilli/linux-kernel-module-cheat#magic-failure-string */
+    ldr r0, =lkmc_baremetal_on_exit_callback
+    bl on_exit
+
+    /* Run main. */
     bl main
+
+    /* If main returns, exit. */
     bl exit

baremetal/lib/syscalls.c

@@ -2,6 +2,8 @@
 #include <stdlib.h>
 #include <sys/stat.h>
 
+#include <lkmc/m5ops.h>
+
 enum {
     UART_FR_RXFE = 0x10,
 };
@@ -11,13 +13,55 @@ enum {
 
 int _close(int file) { return -1; }
 
+void _exit(int status) {
+#if defined(GEM5)
+    LKMC_M5OPS_EXIT;
+#else
+#if defined(__arm__)
+    __asm__ __volatile__ (
+        "mov r0, #0x18\n"
+        "ldr r1, =#0x20026\n"
+        "svc 0x00123456\n"
+        :
+        :
+        : "r0", "r1"
+    );
+#elif defined(__aarch64__)
+    /* TODO actually use the exit value here, just for fun. */
+    __asm__ __volatile__ (
+        "mov x1, #0x26\n" \
+        "movk x1, #2, lsl #16\n" \
+        "str x1, [sp,#0]\n" \
+        "mov x0, #0\n" \
+        "str x0, [sp,#8]\n" \
+        "mov x1, sp\n" \
+        "mov w0, #0x18\n" \
+        "hlt 0xf000\n"
+        :
+        :
+        : "x0", "x1"
+    );
+#endif
+#endif
+}
+
 int _fstat(int file, struct stat *st) {
     st->st_mode = S_IFCHR;
     return 0;
 }
 
+/* Required by assert. */
+int _getpid(void) { return 0; }
+
+/* Required by assert. */
+int _kill(pid_t pid, int sig) {
+    exit(128 + sig);
+}
+
 int _isatty(int file) { return 1; }
+
 int _lseek(int file, int ptr, int dir) { return 0; }
+
 int _open(const char *name, int flags, int mode) { return -1; }
 
 int _read(int file, char *ptr, int len) {
@@ -59,33 +103,3 @@ int _write(int file, char *ptr, int len) {
     }
     return len;
 }
-
-/* Only 0 is supported for now, arm semihosting cannot handle other values. */
-void _exit(int status) {
-#if defined(GEM5)
-#if defined(__arm__)
-    __asm__ __volatile__ ("mov r0, #0; mov r1, #0; .inst 0xEE000110 | (0x21 << 16);" : : : "r0", "r1");
-#elif defined(__aarch64__)
-    __asm__ __volatile__ ("mov x0, #0; .inst 0XFF000110 | (0x21 << 16);" : : : "x0");
-#endif
-#else
-#if defined(__arm__)
-    __asm__ __volatile__ ("mov r0, #0x18; ldr r1, =#0x20026; svc 0x00123456" : : : "r0", "r1");
-#elif defined(__aarch64__)
-    /* TODO actually use the exit value here, just for fun. */
-    __asm__ __volatile__ (
-        "mov x1, #0x26\n" \
-        "movk x1, #2, lsl #16\n" \
-        "str x1, [sp,#0]\n" \
-        "mov x0, #0\n" \
-        "str x0, [sp,#8]\n" \
-        "mov x1, sp\n" \
-        "mov w0, #0x18\n" \
-        "hlt 0xf000\n"
-        :
-        :
-        : "x0", "x1"
-    );
-#endif
-#endif
-}

baremetal/lkmc_assert_fail.c

@@ -0,0 +1 @@
+../lkmc/lkmc_assert_fail.c

baremetal/return1.c

@@ -0,0 +1,2 @@
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#magic-failure-string */
+int main(void) { return 1; }

baremetal/return2.c

@@ -0,0 +1,2 @@
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#magic-failure-string */
+int main(void) { return 2; }

bisect-linux-boot-gem5

@@ -1,34 +1,11 @@
-#!/usr/bin/env python3
+#!/usr/bin/env bash
-
+set -eu
-import imp
+./build-linux --clean "$@"
-import os
+./build-linux "$@"
-import shutil
+set +e
-import sys
+./run --eval 'm5 exit' "$@" || status=$?
-
+# https://stackoverflow.com/questions/4713088/how-to-use-git-bisect/22592593#22592593
-import common
+if [ "$status" -eq 125 ] || [ "$status" -gt 127 ]; then
-build_linux = imp.load_source('build-linux', os.path.join(kwargs['root_dir'], 'build_linux'))
+  status=1
-run = imp.load_source('run', os.path.join(kwargs['root_dir'], 'run'))
+fi
-
+exit "$status"
-parser = self.get_argparse(
-    argparse_args={
-        'description': '''Bisect the Linux kernel on gem5 boots.
-
-More information at: https://github.com/cirosantilli/linux-kernel-module-cheat#bisection
-'''},
-    default_args={
-        'emulators': ['gem5'],
-        'linux_build_id': 'bisect',
-    },
-)
-args = self.setup(parser)
-# We need a clean rebuild because rebuilds at different revisions:
-# - may fail
-# - may not actually rebuild all files, e.g. on header changes
-self.rmrf(kwargs['linux_build_dir'])
-build_linux.LinuxComponent().do_build(args)
-status = run.main(args, {
-    'eval': 'm5 exit',
-})
-if status == 125 or status == 127:
-    status = 1
-sys.exit(status)

bisect-qemu-linux-boot

@@ -3,4 +3,4 @@ set -eu
 git submodule update --recursive
 cd ../..
 ./build-qemu --arch aarch64 --qemu-build-id bisect
-./run --arch aarch64 --kernel-cli 'init=/poweroff.out' --qemu-build-id bisect
+./run --arch aarch64 --kernel-cli 'init=/lkmc/linux/poweroff.out' --qemu-build-id bisect

bst-vs-heap

@@ -1,17 +1,31 @@
 #!/usr/bin/env python3
+
 import common
-parser = self.get_argparse(
+
-    argparse_args={'description':'Convert a BST vs heap stat file into a gnuplot input'}
+class Main(common.LkmcCliFunction):
-)
+    def __init__(self):
-args = self.setup(parser)
+        super().__init__(
-stats = self.get_stats()
+            defaults={
-it = iter(stats)
+                'emulator': 'gem5',
-i = 1
+                'show_time': False,
-for stat in it:
+            },
-    try:
+            description='''\
-        next_stat = next(it)
+Convert a BST vs heap stat file into a gnuplot input
-    except StopIteration:
+''',
-        # Automatic dumpstats at end may lead to odd number of stats.
+        )
-        break
+
-    print('{} {} {}'.format(i, stat, next_stat))
+    def timed_main(self):
-    i += 1
+        stats = self.get_stats()
+        it = iter(stats)
+        i = 1
+        for stat in it:
+            try:
+                next_stat = next(it)
+            except StopIteration:
+                # Automatic dumpstats at end may lead to odd number of stats.
+                break
+            print('{} {} {}'.format(i, stat, next_stat))
+            i += 1
+
+if __name__ == '__main__':
+    Main().cli()

bst-vs-heap.gnuplot

@@ -0,0 +1,25 @@
+#!/usr/bin/env gnuplot
+set terminal png size 1024, 2048
+set output "bst-vs-heap.tmp.png"
+set multiplot layout 5,1 title "Heap vs BST vs Hash map insert time"
+set xlabel "size"
+set ylabel "nanoseconds"
+
+set title "Heap"
+plot "bst_vs_heap.dat" using 1:2 notitle
+
+set title "Heap (zoom)"
+set yrange [0:25]
+plot "bst_vs_heap.dat" using 1:2 notitle
+
+set title "BST"
+set yrange [*:*]
+plot "bst_vs_heap.dat" using 1:3 notitle
+
+#set title "Hash map"
+#set yrange [*:*]
+#plot "bst_vs_heap.dat" using 1:4 notitle
+#
+#set title "Hash map zoom"
+#set yrange [0:350]
+#plot "bst_vs_heap.dat" using 1:4 notitle

build

@@ -7,9 +7,12 @@ import cli_function
 import collections
 import common
 import copy
+import subprocess
 import shell_helpers
 from shell_helpers import LF
 
+import lkmc
+
 class _Component:
     '''
     Yes, we are re-inventing a crappy dependency resolution system,
@@ -107,10 +110,8 @@ so looping over all of them would waste time.
         )
         buildroot_component = _Component(
             self._build_file('build-buildroot'),
-            submodules = {
-                'buildroot',
-            },
             submodules_shallow = {
+                'buildroot',
                 'binutils-gdb',
                 'gcc',
                 'glibc',
@@ -164,7 +165,7 @@ so looping over all of them would waste time.
                 # Generate graphs of config.ini under m5out.
                 'pydot',
             },
-            'submodules': {'gem5'},
+            'submodules_shallow': {'gem5'},
         }
 
         self.name_to_component_map = {
@@ -227,7 +228,7 @@ so looping over all of them would waste time.
                     'python-dev',
                     'texinfo',
                 },
-                submodules={'crosstool-ng'},
+                submodules_shallow={'crosstool-ng'},
             ),
             'doc': _Component(
                 self._build_file('build-doc'),
@@ -272,7 +273,7 @@ so looping over all of them would waste time.
             'm5': _Component(
                 self._build_file('build-m5'),
                 dependencies=['buildroot'],
-                submodules={'gem5'},
+                submodules_shallow={'gem5'},
             ),
             'overlay': _Component(dependencies=[
                 'copy-overlay',
@@ -284,7 +285,7 @@ so looping over all of them would waste time.
                 'overlay',
             ]),
             'parsec-benchmark': _Component(
-                submodules={'parsec-benchmark'},
+                submodules_shallow={'parsec-benchmark'},
                 dependencies=['buildroot'],
             ),
             'qemu': _Component(
@@ -339,6 +340,14 @@ so looping over all of them would waste time.
                 self._build_file('build-userland'),
                 dependencies=['buildroot'],
             ),
+            'userland-host': _Component(
+                self._build_file('build-userland-in-tree'),
+                apt_get_pkgs={
+                    'libdrm-dev',
+                    'libeigen3-dev',
+                    'libopenblas-dev',
+                },
+            ),
             'userland-gem5': _Component(
                 self._build_file('build-userland', static=True, userland_build_id='static'),
                 dependencies=['buildroot'],
@@ -394,8 +403,8 @@ Which components to build. Default: qemu-buildroot
         def f():
             args = self.get_common_args()
             args.update(extra_args)
-            args['print_time'] = False
+            args['show_time'] = False
-            self.import_path_main(component_file)(**args)
+            lkmc.import_path.import_path_main(component_file)(**args)
         return f
 
     def timed_main(self):
@@ -504,46 +513,54 @@ Which components to build. Default: qemu-buildroot
                     ['python3', '-m', 'pip', 'install', '--user', LF] +
                     self.sh.add_newlines(sorted(python3_pkgs))
                 )
-            git_cmd_common = ['git', 'submodule', 'update', '--init', '--recursive']
+            git_version_tuple = tuple(int(x) for x in subprocess.check_output(['git', '--version']).decode().split(' ')[-1].split('.'))
-            if submodules:
+            git_cmd_common = [
-                # == Other nice git options for when distros move to newer Git
+                'git', LF,
-                #
+                'submodule', LF,
-                # Currently not on Ubuntu 16.04:
+                'update', LF,
-                #
+                '--init', LF,
-                # `--progress`: added on Git 2.10:
+                '--recursive', LF,
-                #
+            ]
+            if git_version_tuple >= (2, 9, 0):
+                # https://stackoverflow.com/questions/26957237/how-to-make-git-clone-faster-with-multiple-threads/52327638#52327638
+                git_cmd_common.extend(['--jobs', str(len(os.sched_getaffinity(0))), LF])
+            if git_version_tuple >= (2, 10, 0):
                 # * https://stackoverflow.com/questions/32944468/how-to-show-progress-for-submodule-fetching
                 # * https://stackoverflow.com/questions/4640020/progress-indicator-for-git-clone
-                #
+                git_cmd_common.extend(['--progress', LF])
-                # `--jobs"`: https://stackoverflow.com/questions/26957237/how-to-make-git-clone-faster-with-multiple-threads/52327638#52327638
+            def submodule_ids_to_cmd(submodules):
-                self.sh.run_cmd(
+                return self.sh.add_newlines([os.path.join(common.consts['submodules_dir'], x) for x in sorted(submodules)])
-                    git_cmd_common + ['--', LF] +
+            if submodules:
-                    self.sh.add_newlines([os.path.join(common.consts['submodules_dir'], x) for x in sorted(submodules)])
+                self.sh.run_cmd(git_cmd_common + ['--', LF] + submodule_ids_to_cmd(submodules))
-                )
             if submodules_shallow:
-                # == Shallow cloning.
-                #
                 # TODO Ideally we should shallow clone --depth 1 all of them.
                 #
                 # However, most git servers out there are crap or craply configured
                 # and don't allow shallow cloning except for branches.
                 #
-                # So for now, let's shallow clone only the Linux kernel, which has by far
+                # So for now I'm keeping all mirrors in my GitHub.
-                # the largest .git repo history, and full clone the others.
+                # and always have a lkmc-* branch pointint to it.
                 #
-                # Then we will maintain a GitHub Linux kernel mirror / fork that always has a
+                # However, QEMU has a bunch of submodules itself, and I'm not in the mood
-                # lkmc branch, and point to it, so that it will always succeed.
+                # to mirror all of them...
                 #
                 # See also:
                 #
                 # * https://stackoverflow.com/questions/3489173/how-to-clone-git-repository-with-specific-revision-changeset
                 # * https://stackoverflow.com/questions/2144406/git-shallow-submodules/47374702#47374702
                 # * https://unix.stackexchange.com/questions/338578/why-is-the-git-clone-of-the-linux-kernel-source-code-much-larger-than-the-extrac
-                #
+                cmd = git_cmd_common.copy()
-                self.sh.run_cmd(
+                if git_version_tuple > (2, 7, 4):
-                    git_cmd_common + ['--depth', '1', '--', LF] +
+                    # Then there is a bug in Ubuntu 16.04 git 2.7.4 where --depth 1 fails...
-                    self.sh.add_newlines([os.path.join(common.consts['submodules_dir'], x) for x in sorted(submodules_shallow)])
+                    # OMG git submodules implementation sucks:
-                )
+                    # * https://stackoverflow.com/questions/2155887/git-submodule-head-reference-is-not-a-tree-error/25875273#25875273
+                    # * https://github.com/boostorg/boost/issues/245
+                    cmd.extend(['--depth', '1', LF])
+                else:
+                    self.log_warn('your git is too old for git submodule update --depth 1')
+                    self.log_warn('update to git 2.17 or newer and you will save clone time')
+                    self.log_warn('see: https://github.com/cirosantilli/linux-kernel-module-cheat/issues/44')
+                self.sh.run_cmd(cmd + ['--', LF] + submodule_ids_to_cmd(submodules_shallow))
 
         # Do the build.
         for component in selected_components:

build-baremetal

@@ -15,6 +15,9 @@ Build the baremetal examples with crosstool-NG.
 ''',
             supported_archs=common.consts['crosstool_ng_supported_archs']
         )
+        self._add_argument('--ccflags')
+        self._add_argument('--force-rebuild')
+        self._add_argument('--optimization-level')
 
     def build(self):
         build_dir = self.get_build_dir()
@@ -39,11 +42,25 @@ Build the baremetal examples with crosstool-NG.
         cflags = [
             '-I', self.env['baremetal_source_lib_dir'], LF,
             '-I', self.env['root_dir'], LF,
-            '-O0', LF,
+            '-O{}'.format(self.env['optimization_level']), LF,
             '-ggdb3', LF,
             '-mcpu={}'.format(self.env['mcpu']), LF,
             '-nostartfiles', LF,
         ]
+        if self.env['arch'] == 'arm':
+            cflags.extend([
+                '-mhard-float', LF,
+                # This uses the soft float ABI for calling functions from objets in Newlib which
+                # our crosstool-NG config compiles with soft floats, while emiting hard float
+                # from C and allowing us to use it from assembly, e.g. for the VMRS instruction:
+                # which would otherwise fail "with selected processor does not support XXX in ARM mode"
+                # Bibliography:
+                # - https://stackoverflow.com/questions/9753749/arm-compilation-error-vfp-registered-used-by-executable-not-object-file
+                # - https://stackoverflow.com/questions/41131432/cross-compiling-error-selected-processor-does-not-support-fmrx-r3-fpexc-in/41131782#41131782
+                # - https://embeddedartistry.com/blog/2017/10/9/r1q7pksku2q3gww9rpqef0dnskphtc
+                '-mfloat-abi=softfp', LF,
+                '-mfpu=crypto-neon-fp-armv8', LF,
+            ])
         cflags_after = ['-lm']
         if self.env['emulator'] == 'gem5':
             if self.env['machine'] == 'VExpress_GEM5_V1':
@@ -54,7 +71,10 @@ Build the baremetal examples with crosstool-NG.
                 uart_address = 0x10009000
             else:
                 raise Exception('unknown machine: ' + self.env['machine'])
-            cflags.extend(['-D', 'GEM5'.format(uart_address), LF])
+            cflags.extend([
+                '-D', 'GEM5'.format(uart_address), LF,
+                '-DLKMC_M5OPS_ENABLE=1', LF,
+            ])
         else:
             entry_address = 0x40000000
             uart_address = 0x09000000
@@ -67,9 +87,10 @@ Build the baremetal examples with crosstool-NG.
                 self.env['asm_ext']
             )
         )
+        cflags.extend(self.sh.shlex_split(self.env['ccflags']))
         if self.need_rebuild([src], bootloader_obj):
             self.sh.run_cmd(
-                [self.env['gcc'],  LF] +
+                [self.env['gcc_path'],  LF] +
                 cflags +
                 [
                     '-c', LF,
@@ -84,11 +105,11 @@ Build the baremetal examples with crosstool-NG.
         ]:
             if self.need_rebuild([src, self.env['common_h']], obj):
                 self.sh.run_cmd(
-                    [self.env['gcc'],  LF] +
+                    [self.env['gcc_path'],  LF] +
                     cflags +
                     [
-                        '-c', LF,
                         '-D', 'UART0_ADDR={:#x}'.format(uart_address), LF,
+                        '-c', LF,
                         '-o', obj, LF,
                         src, LF,
                     ] +
@@ -112,43 +133,34 @@ Build the baremetal examples with crosstool-NG.
                     in_name, in_ext = os.path.splitext(in_basename)
                     if  (
                         os.path.isfile(in_path) and
-                        in_ext in (self.env['c_ext'], self.env['asm_ext'])
+                        in_ext in self.env['build_in_exts']
                     ):
-                        main_obj = os.path.join(
-                            out_dir,
-                            '{}{}'.format(
-                                in_name,
-                                self.env['obj_ext']
-                            )
-                        )
-                        src = os.path.join(self.env['baremetal_source_dir'], in_path)
-                        if self.need_rebuild([src, self.env['common_h']], main_obj):
-                            self.sh.run_cmd(
-                                [self.env['gcc'],  LF] +
-                                cflags +
-                                [
-                                    '-c', LF,
-                                    '-o', main_obj, LF,
-                                    src, LF,
-                                ] +
-                                cflags_after
-                            )
-                        objs = common_objs_bootloader + [main_obj]
                         out = os.path.join(out_dir, in_name + self.env['baremetal_build_ext'])
-                        if self.need_rebuild(objs + [self.env['baremetal_link_script']], out):
+                        src = os.path.join(self.env['baremetal_source_dir'], in_path)
+                        if self.need_rebuild(
+                            common_objs_bootloader +
+                            [
+                                src,
+                                self.env['baremetal_link_script'],
+                                self.env['common_h']
+                            ],
+                            out
+                        ):
                             self.sh.run_cmd(
-                                [self.env['gcc'],  LF] +
+                                [self.env['gcc_path'],  LF] +
                                 cflags +
                                 [
                                     '-Wl,--section-start=.text={:#x}'.format(entry_address), LF,
                                     '-o', out, LF,
                                     '-T', self.env['baremetal_link_script'], LF,
                                 ] +
-                                self.sh.add_newlines(objs) +
+                                [
+                                    src, LF,
+                                ] +
+                                self.sh.add_newlines(common_objs_bootloader) +
                                 cflags_after
                             )
 
-
     def get_build_dir(self):
         return self.env['baremetal_build_dir']
 

build-buildroot

@@ -70,6 +70,7 @@ Extra arguments to be passed to the Buildroot make,
 usually extra Buildroot targets.
 '''
         )
+        self._add_argument('--force-rebuild')
 
     def build(self):
         build_dir = self.get_build_dir()
@@ -147,6 +148,8 @@ usually extra Buildroot targets.
             cwd=self.env['buildroot_source_dir'],
         )
         self.make_build_dirs()
+        if self.env['force_rebuild']:
+            extra_make_args.extend(['-B', LF])
         if not self.env['no_all']:
             extra_make_args.extend(['all', LF])
         self.sh.run_cmd(
@@ -158,7 +161,7 @@ usually extra Buildroot targets.
             ] +
             extra_make_args
             ,
-            out_file=os.path.join(self.env['buildroot_build_dir'], 'lkmc.log'),
+            out_file=os.path.join(self.env['buildroot_build_dir'], self.env['repo_short_id'] + '.log'),
             delete_env=['LD_LIBRARY_PATH'],
             cwd=self.env['buildroot_source_dir'],
         )

build-crosstool-ng

@@ -61,7 +61,7 @@ Build crosstool-NG with Newlib for bare metal compilation
                 'build', LF,
                 'CT_JOBS={}'.format(str(self.env['nproc'])), LF,
             ],
-            out_file=os.path.join(build_dir, 'lkmc.log'),
+            out_file=os.path.join(build_dir, self.env['repo_short_id'] + '.log'),
             delete_env=['LD_LIBRARY_PATH'],
             extra_paths=[self.env['ccache_dir']],
         )

build-doc

@@ -9,7 +9,7 @@ class Main(common.LkmcCliFunction):
     def __init__(self):
         super().__init__(
             defaults = {
-                'print_time': False,
+                'show_time': False,
             },
             description='''\
 https://github.com/cirosantilli/linux-kernel-module-cheat#build-the-documentation
@@ -27,11 +27,12 @@ https://github.com/cirosantilli/linux-kernel-module-cheat#build-the-documentatio
         )
         error_re = re.compile('^asciidoctor: WARNING: ')
         exit_status = 0
-        with open(self.env['build_doc_log']) as f:
+        if not self.env['dry_run']:
-            for line in f:
+            with open(self.env['build_doc_log']) as f:
-                if error_re.search(line):
+                for line in f:
-                    exit_status = 1
+                    if error_re.search(line):
-                    break
+                        exit_status = 1
+                        break
         return exit_status
 
 if __name__ == '__main__':

build-docker

@@ -7,7 +7,6 @@ import tarfile
 import common
 from shell_helpers import LF
 
-
 class DockerComponent(self.Component):
     def get_argparse_args(self):
         return {

build-gem5

@@ -64,7 +64,8 @@ https://github.com/cirosantilli/linux-kernel-module-cheat-regression#gem5-unit-t
                     self.env['gem5_source_dir'], LF,
                 ])
             else:
-                raise Exception('gem5 submodule not checked out')
+                if not self.env['dry_run']:
+                    raise Exception('gem5 submodule not checked out')
         if self.env['verbose']:
             verbose = ['--verbose', LF]
         else:

build-linux

@@ -81,21 +81,19 @@ Run `make modules_install` after `make`.
             metavar='extra-make-args',
             nargs='*'
         )
+        self._add_argument('--force-rebuild')
 
     def build(self):
         build_dir = self.get_build_dir()
         os.makedirs(build_dir, exist_ok=True)
-        tool = 'gcc'
-        gcc = self.get_toolchain_tool(tool)
-        prefix = gcc[:-len(tool)]
         common_args = {
             'cwd': self.env['linux_source_dir'],
         }
         ccache = shutil.which('ccache')
         if ccache is not None:
-            cc = '{} {}'.format(ccache, gcc)
+            cc = '{} {}'.format(ccache, self.env['gcc_path'])
         else:
-            cc = gcc
+            cc = self.env['gcc_path']
         if self.env['verbose']:
             verbose = ['V=1']
         else:
@@ -104,10 +102,12 @@ Run `make modules_install` after `make`.
             'make', LF,
             '-j', str(self.env['nproc']), LF,
             'ARCH={}'.format(self.env['linux_arch']), LF,
-            'CROSS_COMPILE={}'.format(prefix), LF,
+            'CROSS_COMPILE={}-'.format(self.env['toolchain_prefix']), LF,
             'CC={}'.format(cc), LF,
             'O={}'.format(build_dir), LF,
         ] + verbose
+        if self.env['force_rebuild']:
+            common_make_args.extend(['-B', LF])
         if self.env['configure']:
             if self.env['custom_config_target']:
                 base_config_given = True
@@ -182,13 +182,13 @@ Run `make modules_install` after `make`.
                     ] +
                     self.sh.add_newlines(config_fragments)
                 )
-                self.sh.run_cmd(
+            self.sh.run_cmd(
-                    (
+                (
-                        common_make_args +
+                    common_make_args +
-                        ['olddefconfig', LF]
+                    ['olddefconfig', LF]
-                    ),
+                ),
-                    **common_args
+                **common_args
-                )
+            )
         if self.env['build']:
             self.sh.run_cmd(
                 (
@@ -199,7 +199,7 @@ Run `make modules_install` after `make`.
                 extra_env={
                     'KBUILD_BUILD_VERSION': '1',
                     'KBUILD_BUILD_TIMESTAMP': 'Thu Jan  1 00:00:00 UTC 1970',
-                    'KBUILD_BUILD_USER': 'lkmc',
+                    'KBUILD_BUILD_USER': self.env['repo_short_id'],
                     'KBUILD_BUILD_HOST': common.git_sha(self.env['linux_source_dir']),
                 },
                 **common_args
@@ -209,7 +209,7 @@ Run `make modules_install` after `make`.
                     (
                         common_make_args +
                         [
-                            'INSTALL_MOD_PATH={}'.format(self.env['out_rootfs_overlay_dir']), LF,
+                            'INSTALL_MOD_PATH={}'.format(self.env['out_rootfs_overlay_lkmc_dir']), LF,
                             'modules_install', LF,
                         ]
                     ),

build-m5

@@ -16,8 +16,8 @@ class Main(common.BuildCliFunction):
             'make', LF,
             '-j', str(self.env['nproc']), LF,
             '-f', 'Makefile.{}'.format(arch), LF,
-            'CC={}'.format(self.env['gcc']), LF,
+            'CC={}'.format(self.env['gcc_path']), LF,
-            'LD={}'.format(self.env['ld']), LF,
+            'LD={}'.format(self.env['ld_path']), LF,
             'PWD={}'.format(self.env['gem5_m5_source_dir']), LF,
         ]
 

build-modules

@@ -3,6 +3,7 @@
 import distutils.dir_util
 import os
 import platform
+import shlex
 import shutil
 
 import common
@@ -19,6 +20,9 @@ See also: https://github.com/cirosantilli/linux-kernel-module-cheat#host
         self.add_argument(
             '--make-args',
             default='',
+            help='''
+Pass custom options to make.
+''',
         )
         self.add_argument(
             '--host',
@@ -34,6 +38,7 @@ Place the modules on a separate magic directory from non --host builds.
             help='Which kernel modules to build. Default: build all',
             nargs='*',
         )
+        self._add_argument('--force-rebuild')
 
     def build(self):
         build_dir = self.get_build_dir()
@@ -72,22 +77,23 @@ Place the modules on a separate magic directory from non --host builds.
             build_subdir = self.env['kernel_modules_build_host_subdir']
         else:
             build_subdir = self.env['kernel_modules_build_subdir']
-        tool = 'gcc'
-        gcc = self.get_toolchain_tool(tool)
-        prefix = gcc[:-len(tool)]
         ccache = shutil.which('ccache')
         if ccache is not None:
-            cc = '{} {}'.format(ccache, gcc)
+            cc = '{} {}'.format(ccache, self.env['gcc_path'])
         else:
-            cc = gcc
+            cc = self.env['gcc_path']
-        if self.env['verbose']:
-            verbose = ['V=1']
-        else:
-            verbose = []
         if self.env['host']:
             linux_dir = os.path.join('/lib', 'modules', platform.uname().release, 'build')
         else:
             linux_dir = self.env['linux_build_dir']
+        ccflags = [
+            '-I', self.env['root_dir'], LF,
+        ]
+        make_args_extra = []
+        if self.env['verbose']:
+            make_args_extra.extend(['V=1', LF])
+        if self.env['force_rebuild']:
+            make_args_extra.extend(['-B', LF])
         self.sh.run_cmd(
             (
                 [
@@ -95,20 +101,21 @@ Place the modules on a separate magic directory from non --host builds.
                     '-j', str(self.env['nproc']), LF,
                     'ARCH={}'.format(self.env['linux_arch']), LF,
                     'CC={}'.format(cc), LF,
-                    'CROSS_COMPILE={}'.format(prefix), LF,
+                    'CCFLAGS={}'.format(self.sh.cmd_to_string(ccflags)), LF,
+                    'CROSS_COMPILE={}-'.format(self.env['toolchain_prefix']), LF,
                     'LINUX_DIR={}'.format(linux_dir), LF,
                     'M={}'.format(build_subdir), LF,
                     'OBJECT_FILES={}'.format(' '.join(object_files)), LF,
                 ] +
-                self.sh.shlex_split(self.env['make_args']) +
+                make_args_extra +
-                verbose
+                self.sh.shlex_split(self.env['make_args'])
             ),
             cwd=os.path.join(self.env['kernel_modules_build_subdir']),
         )
         if not self.env['host']:
             self.sh.copy_dir_if_update_non_recursive(
                 srcdir=self.env['kernel_modules_build_subdir'],
-                destdir=self.env['out_rootfs_overlay_dir'],
+                destdir=self.env['out_rootfs_overlay_lkmc_dir'],
                 filter_ext=self.env['kernel_module_ext'],
             )
 

build-userland

@@ -2,295 +2,141 @@
 
 import os
 import shlex
-
-import common
-import threading
 import subprocess
-from shell_helpers import LF
+import threading
 
-error = False
+from shell_helpers import LF
+import common
+import thread_pool
 
 class Main(common.BuildCliFunction):
-    def __init__(self):
+    def __init__(self, *args, **kwargs):
-        super().__init__(
+        if not 'description' in kwargs:
-            description='''\
+            kwargs['description'] = '''\
 Build our compiled userland examples.
 '''
-        )
+        super().__init__(*args, **kwargs)
-        self.default_cstd = 'c11'
-        self.default_cxxstd = 'c++17'
-        self.add_argument(
-            '--has-package',
-            action='append',
-            default=[],
-            help='''\
-Indicate that a given package is present in the root filesystem, which
-allows us to build examples that rely on it.
-''',
-        )
-        self.add_argument(
-            '--in-tree',
-            default=False,
-            help='''\
-Magic build mode tailored to build from within the source tree:
-
-* place build output inside soure tree to conveniently run it
-* if not targets are given, build use the current working directory
-''',
-        )
         self.add_argument(
             'targets',
             default=[],
             help='''\
-Build only the given userland programs or all programs in the given directories.
+Select to build only the given userland programs, or all programs under
+the given directories.
+
+Default: build all.
+
+Must point to either sources or directories under userland/, or to LKMC
+toplevel which is a synonym for userland/.
 
 Default: build all examples that have their package dependencies met, e.g.:
+
 -   userland/arch/ programs only build if the target arch matches
 -   an OpenBLAS example can only be built if the target root filesystem
-    has the OpenBLAS libraries and headers installed, which you must inform with --has-package
+    has the OpenBLAS libraries and headers installed, which you must inform
+    with --package
 ''',
             nargs='*',
         )
-
+        self._add_argument('--ccflags')
-    def _build_one(
+        self._add_argument('--force-rebuild')
-        self,
+        self._add_argument('--optimization-level')
-        in_path,
-        out_path,
-        ccflags,
-        ccflags_after=None,
-        cstd=None,
-        cxxstd=None,
-        extra_deps=None,
-        extra_objs=None,
-        link=True,
-        raise_on_failure=True,
-        thread_limiter=None,
-    ):
-        try:
-            if extra_deps is None:
-                extra_deps = []
-            if extra_objs is None:
-                extra_objs = []
-            if ccflags_after is None:
-                ccflags_after = []
-            ret = 0
-            if self.need_rebuild([in_path] + extra_objs + extra_deps, out_path):
-                ccflags = ccflags.copy()
-                if not link:
-                    ccflags.extend(['-c', LF])
-                in_ext = os.path.splitext(in_path)[1]
-                do_compile = True
-                if in_ext == self.env['c_ext']:
-                    cc = self.env['gcc']
-                    if cstd is None:
-                        std = self.default_cstd
-                    else:
-                        std = cstd
-                    ccflags.extend([
-                        '-fopenmp', LF,
-                    ])
-                elif in_ext == self.env['cxx_ext']:
-                    cc = self.env['gxx']
-                    if cxxstd is None:
-                        std = self.default_cxxstd
-                    else:
-                        std = cxxstd
-                else:
-                    do_compile = False
-                if do_compile:
-                    ret = self.sh.run_cmd(
-                        (
-                            [
-                                cc, LF,
-                            ] +
-                            ccflags +
-                            [
-                                '-std={}'.format(std), LF,
-                                '-o', out_path, LF,
-                                in_path, LF,
-                            ] +
-                            self.sh.add_newlines(extra_objs) +
-                            [
-                                '-lm', LF,
-                                '-pthread', LF,
-                            ] +
-                            ccflags_after
-                        ),
-                        extra_paths=[self.env['ccache_dir']],
-                        raise_on_failure=raise_on_failure,
-                    )
-        finally:
-            if thread_limiter is not None:
-                thread_limiter.release()
-        if ret != 0:
-            self.error = True
-        return ret
-
-    def _get_targets(self):
-        if self.env['_args_given']['targets']:
-            targets = self.env['targets']
-            if self.env['in_tree']:
-                cwd = os.getcwd()
-                targets = [os.path.join(cwd, target) for target in targets]
-            return targets
-        else:
-            if self.env['in_tree']:
-                return [os.getcwd()]
-            else:
-                return [self.env['userland_source_dir']]
 
     def build(self):
         build_dir = self.get_build_dir()
-        os.makedirs(build_dir, exist_ok=True)
+        cc_flags = [
-        has_packages = set(self.env['has_package'])
-        ccflags = [
             '-I', self.env['root_dir'], LF,
-            '-I', self.env['userland_source_dir'], LF,
+            '-O{}'.format(self.env['optimization_level']), LF,
-            '-O0', LF,
+        ] + self.sh.shlex_split(self.env['ccflags'])
-            '-Wall', LF,
-            '-Werror', LF,
-            '-Wextra', LF,
-            '-Wno-unused-function', LF,
-            '-ggdb3', LF,
-        ]
         if self.env['static']:
-            ccflags.extend(['-static', LF])
+            cc_flags.extend(['-static', LF])
-        common_obj = os.path.join(
+        extra_obj_lkmc_common = os.path.join(
             build_dir,
             self.env['common_basename_noext'] + self.env['obj_ext']
         )
         self._build_one(
             in_path=self.env['common_c'],
-            out_path=common_obj,
+            out_path=extra_obj_lkmc_common,
-            ccflags=ccflags,
+            cc_flags=cc_flags,
             extra_deps=[self.env['common_h']],
             link=False,
         )
-        pkgs = {
+        extra_obj_userland_asm = os.path.join(
-            'eigen': {
+            build_dir,
-                # TODO: was failing with:
+            'arch',
-                # fatal error: Eigen/Dense: No such file or directory as of
+            'main' + self.env['obj_ext']
-                # 975ce0723ee3fa1fea1766e6683e2f3acb8558d6
-                # http://lists.busybox.net/pipermail/buildroot/2018-June/222914.html
-                'ccflags': [
-                    '-I',
-                    os.path.join(
-                        self.env['buildroot_staging_dir'],
-                        'usr',
-                        'include',
-                        'eigen3'
-                    ),
-                    LF
-                ],
-                # Header only.
-                'ccflags_after': [],
-            },
-            'libdrm': {},
-            'openblas': {},
-        }
-        rootdir_abs_len = len(self.env['userland_source_dir'])
-        thread_limiter = threading.BoundedSemaphore(self.env['nproc'])
-        self.error = False
-        for target in self._get_targets():
-            target = self.resolve_userland_source(target)
-            for path, in_dirnames, in_filenames in self.sh.walk(target):
-                in_dirnames.sort()
-                path_abs = os.path.abspath(path)
-                dirpath_relative_root = path_abs[rootdir_abs_len + 1:]
-                dirpath_relative_root_components = dirpath_relative_root.split(os.sep)
-                if (
-                    len(dirpath_relative_root_components) < 2 or
-                    dirpath_relative_root_components[0] != 'arch' or
-                    dirpath_relative_root_components[1] == self.env['arch']
-                ):
-                    out_dir = os.path.join(
-                        build_dir,
-                        dirpath_relative_root
-                    )
-                    os.makedirs(out_dir, exist_ok=True)
-                    ccflags_dir = ccflags.copy()
-                    if dirpath_relative_root_components == ['gcc']:
-                        cstd = 'gnu11'
-                        cxxstd = 'gnu++17'
-                    else:
-                        cstd = self.default_cstd
-                        cxxstd = self.default_cxxstd
-                        # -pedantic complains even if we use -std=gnu11.
-                        ccflags_dir.extend(['-pedantic', LF])
-                    for in_filename in in_filenames:
-                        in_path = os.path.join(path, in_filename)
-                        in_name, in_ext = os.path.splitext(in_filename)
-                        out_path = os.path.join(
-                            out_dir,
-                            in_name + self.env['userland_build_ext']
-                        )
-                        pkg_key = in_name.split('_')[0]
-                        ccflags_file = ccflags_dir.copy()
-                        ccflags_after = []
-                        if pkg_key in pkgs:
-                            if pkg_key not in has_packages:
-                                continue
-                            pkg = pkgs[pkg_key]
-                            if 'ccflags' in pkg:
-                                ccflags_file.extend(pkg['ccflags'])
-                            else:
-                                pkg_config_output = subprocess.check_output([
-                                    self.env['buildroot_pkg_config'],
-                                    '--cflags',
-                                    pkg_key
-                                ]).decode()
-                                ccflags_file.extend(self.sh.shlex_split(pkg_config_output))
-                            if 'ccflags_after' in pkg:
-                                ccflags_file.extend(pkg['ccflags_after'])
-                            else:
-                                pkg_config_output = subprocess.check_output([
-                                    self.env['buildroot_pkg_config'],
-                                    '--libs',
-                                    pkg_key
-                                ]).decode()
-                                ccflags_after.extend(self.sh.shlex_split(pkg_config_output))
-                        thread_limiter.acquire()
-                        if self.error:
-                            return 1
-                        thread = threading.Thread(
-                            target=self._build_one,
-                            kwargs={
-                                'in_path': in_path,
-                                'out_path': out_path,
-                                'ccflags': ccflags_file,
-                                'cstd': cstd,
-                                'cxxstd': cxxstd,
-                                'extra_objs': [common_obj],
-                                'ccflags_after': ccflags_after,
-                                'raise_on_failure': False,
-                                'thread_limiter': thread_limiter,
-                            }
-                        )
-                        thread.start()
-        self.sh.copy_dir_if_update(
-            srcdir=build_dir,
-            destdir=self.env['out_rootfs_overlay_dir'],
-            filter_ext=self.env['userland_build_ext'],
         )
+        extra_obj_userland_asm_relpath = os.path.join(
+            'arch',
+            'main' + self.env['c_ext']
+        )
+        self._build_one(
+            in_path=os.path.join(
+                self.env['userland_source_dir'],
+                extra_obj_userland_asm_relpath
+            ),
+            out_path=extra_obj_userland_asm,
+            cc_flags=cc_flags,
+            extra_deps=[self.env['common_h']],
+            link=False,
+        )
+        with thread_pool.ThreadPool(
+            self._build_one,
+            nthreads=self.env['nproc'],
+        ) as my_thread_pool:
+            try:
+                for target in self.env['targets']:
+                    for path, in_dirnames, in_filenames in self.sh.walk(target):
+                        for in_filename in in_filenames:
+                            in_ext = os.path.splitext(in_filename)[1]
+                            if not in_ext in self.env['build_in_exts']:
+                                continue
+                            in_path = os.path.join(path, in_filename)
+                            error = my_thread_pool.submit({
+                                'in_path': in_path,
+                                'out_path': self.resolve_userland_executable(in_path),
+                                'cc_flags': cc_flags,
+                                'extra_objs_lkmc_common': [extra_obj_lkmc_common],
+                                'extra_objs_userland_asm': [extra_obj_userland_asm],
+                            })
+                            if error is not None:
+                                raise common.ExitLoop()
+            except common.ExitLoop:
+                pass
+        error = my_thread_pool.get_error()
+        if error is not None:
+            print(error)
+            return 1
+        if not self.env['in_tree']:
+            self.sh.copy_dir_if_update(
+                srcdir=build_dir,
+                destdir=self.env['out_rootfs_overlay_lkmc_dir'],
+                filter_ext=self.env['userland_executable_ext'],
+            )
         return 0
 
     def clean(self):
         if self.env['in_tree']:
-            for target in self._get_targets():
+            for target in self.env['targets']:
-                for path, dirnames, filenames in os.walk(target):
+                if os.path.exists(target):
-                    filenames.sort()
+                    if os.path.isfile(target):
-                    dirnames.sort()
+                        self.sh.rmrf(self.resolve_userland_executable(target))
-                    for filename in filenames:
+                    else:
-                        if os.path.splitext(filename)[1] in self.env['userland_out_exts']:
+                        for path, dirnames, filenames in self.sh.walk(target):
-                            self.sh.rmrf(os.path.join(path, filename))
+                            for filename in filenames:
+                                if os.path.splitext(filename)[1] in self.env['userland_out_exts']:
+                                    self.sh.rmrf(os.path.join(path, filename))
         else:
-            self.sh.rmrf(self.get_build_dir())
+            for target in self.env['targets']:
+                self.sh.rmrf(self.resolve_userland_executable(target))
 
     def get_build_dir(self):
-        if self.env['in_tree']:
+        return self.env['userland_build_dir']
-            return self.env['userland_source_dir']
+
-        else:
+    def setup_one(self):
-            return self.env['userland_build_dir']
+        self.env['targets'] = self.resolve_targets(
+            self.env['userland_source_dir'],
+            self.env['targets']
+        )
 
 if __name__ == '__main__':
     Main().cli()

build-userland-in-tree

@@ -1,2 +1,24 @@
-#!/usr/bin/env bash
+#!/usr/bin/env python3
-"$(git rev-parse --show-toplevel)/build-userland" --gcc-which host --in-tree "$@"
+
+import os
+import subprocess
+
+import lkmc.import_path
+
+build_userland = lkmc.import_path.import_path_relative_root('build-userland')
+
+class Main(build_userland.Main):
+    def __init__(self):
+        super().__init__(
+            description='''\
+https://github.com/cirosantilli/linux-kernel-module-cheat#userland-setup-getting-started-natively
+''',
+            defaults={
+                'gcc_which': 'host',
+                'in_tree': True,
+                'targets': ['.'],
+            }
+        )
+
+if __name__ == '__main__':
+    Main().cli()

cli_function.py

@@ -11,10 +11,11 @@ made to this file.
 import argparse
 import bisect
 import collections
-import imp
 import os
 import sys
 
+import lkmc.import_path
+
 class _Argument:
     def __init__(
             self,
@@ -113,8 +114,7 @@ class _Argument:
 
 class CliFunction:
     '''
-    Represent a function that can be called either from Python code, or
+    A function that can be called either from Python code, or from the command line.
-    from the command line.
 
     Features:
 
@@ -134,6 +134,10 @@ class CliFunction:
 
     * that decorator API is insane
     * CLI + Python for single functions was wontfixed: https://github.com/pallets/click/issues/40
+    +
+    Oh, and I commented on that issue pointing to this alternative and they deleted my comment:
+    https://github.com/pallets/click/issues/40#event-2088718624 Lol. It could have been useful
+    for other Googlers and as an implementation reference.
     '''
     def __call__(self, **kwargs):
         '''
@@ -147,15 +151,15 @@ class CliFunction:
     def _do_main(self, kwargs):
         return self.main(**self._get_args(kwargs))
 
-    def __init__(self, config_file=None, description=None, extra_config_params=None):
+    def __init__(self, default_config_file=None, description=None, extra_config_params=None):
         self._arguments = collections.OrderedDict()
-        self._config_file = config_file
+        self._default_config_file = default_config_file
         self._description = description
         self.extra_config_params = extra_config_params
-        if self._config_file is not None:
+        if self._default_config_file is not None:
             self.add_argument(
                 '--config-file',
-                default=self._config_file,
+                default=self._default_config_file,
                 help='Path to the configuration file to use'
             )
 
@@ -172,30 +176,35 @@ class CliFunction:
         args_with_defaults = kwargs.copy()
         # Add missing args from config file.
         config_file = None
+        args_given = {}
         if 'config_file' in args_with_defaults and args_with_defaults['config_file'] is not None:
             config_file = args_with_defaults['config_file']
+            args_given['config_file'] = True
         else:
-            config_file = self._config_file
+            config_file = self._default_config_file
-        args_given = {}
+            args_given['config_file'] = False
         for key in self._arguments:
             args_given[key] = not (
                 not key in args_with_defaults or
                 args_with_defaults[key] is None or
                 self._arguments[key].nargs == '*' and args_with_defaults[key] == []
             )
-        if config_file is not None and os.path.exists(config_file):
+        if config_file is not None:
-            config_configs = {}
+            if os.path.exists(config_file):
-            config = imp.load_source('config', config_file)
+                config_configs = {}
-            if self.extra_config_params is None:
+                config = lkmc.import_path.import_path(config_file)
-                config.set_args(config_configs)
+                if self.extra_config_params is None:
-            else:
+                    config.set_args(config_configs)
-                config.set_args(config_configs, self.extra_config_params)
+                else:
-            for key in config_configs:
+                    config.set_args(config_configs, self.extra_config_params)
-                if key not in self._arguments:
+                for key in config_configs:
-                    raise Exception('Unknown key in config file: ' + key)
+                    if key not in self._arguments:
-                if not args_given[key]:
+                        raise Exception('Unknown key in config file: ' + key)
-                    args_with_defaults[key] = config_configs[key]
+                    if not args_given[key]:
-                    args_given[key] = True
+                        args_with_defaults[key] = config_configs[key]
+                        args_given[key] = True
+            elif args_given['config_file']:
+                raise Exception('Config file does not exist: ' + config_file)
         # Add missing args from hard-coded defaults.
         for key in self._arguments:
             argument = self._arguments[key]
@@ -290,7 +299,10 @@ class CliFunction:
                 if value != default:
                     if argument.is_option:
                         if argument.is_bool:
-                            vals = [(argument.longname,)]
+                            if value:
+                                vals = [(argument.longname,)]
+                            else:
+                                vals = [('--no-' + argument.longname[2:],)]
                         elif 'action' in argument.kwargs and argument.kwargs['action'] == 'append':
                             vals = [(argument.longname, str(val)) for val in value]
                         else:
@@ -326,7 +338,7 @@ if __name__ == '__main__':
     class OneCliFunction(CliFunction):
         def __init__(self):
             super().__init__(
-                config_file='cli_function_test_config.py',
+                default_config_file='cli_function_test_config.py',
                 description = '''\
 Description of this
 amazing function!
@@ -454,7 +466,8 @@ amazing function!
     # get_cli
     assert one_cli_function.get_cli(pos_mandatory=1, asdf='B') == [('--asdf', 'B'), ('--bool-cli',), ('1',)]
     assert one_cli_function.get_cli(pos_mandatory=1, asdf='B', qwer='R') == [('--asdf', 'B'), ('--bool-cli',), ('--qwer', 'R'), ('1',)]
-    assert one_cli_function.get_cli(pos_mandatory=1, bool_true=False) == [('--bool-cli',), ('--bool-true',), ('1',)]
+    assert one_cli_function.get_cli(pos_mandatory=1, bool_true=False) == [('--bool-cli',), ('--no-bool-true',), ('1',)]
+    assert one_cli_function.get_cli(pos_mandatory=1, bool_false=True) == [('--bool-cli',), ('--bool-false',), ('1',)]
     assert one_cli_function.get_cli(pos_mandatory=1, pos_optional=2, args_star=['asdf', 'qwer']) == [('--bool-cli',), ('1',), ('2',), ('asdf',), ('qwer',)]
     assert one_cli_function.get_cli(pos_mandatory=1, append=['2', '3']) == [('--append', '2'), ('--append', '3',), ('--bool-cli',), ('1',)]
 

copy-overlay

@@ -15,12 +15,9 @@ https://github.com/cirosantilli/linux-kernel-module-cheat#rootfs_overlay
 ''')
 
     def build(self):
-        # TODO: print rsync equivalent, move into shell_helpers.
+        self.sh.copy_dir_if_update(
-        distutils.dir_util.copy_tree(
+            srcdir=self.env['rootfs_overlay_dir'],
-            self.env['rootfs_overlay_dir'],
+            destdir=self.env['out_rootfs_overlay_dir'],
-            self.env['out_rootfs_overlay_dir'],
-            preserve_symlinks=True,
-            update=1,
         )
 
 if __name__ == '__main__':

gem5-stat

@@ -6,7 +6,7 @@ class Main(common.LkmcCliFunction):
     def __init__(self):
         super().__init__(
             defaults={
-                'print_time': False,
+                'show_time': False,
             },
             description='''\
 Get the value of a gem5 stat from the stats.txt file.

getvar

@@ -6,7 +6,7 @@ class Main(common.LkmcCliFunction):
     def __init__(self):
         super().__init__(
             defaults = {
-                'print_time': False,
+                'show_time': False,
             },
             description='''\
 Print the value of a self.env['py'] variable.

include/README.adoc

@@ -1 +0,0 @@
-https://github.com/cirosantilli/linux-kernel-module-cheat#include-directory

kernel_modules/Makefile

@@ -1,15 +1,17 @@
-ifeq ($(OBJECT_FILES),)
-# Hardcoding  LKMC_MODULE_SUBDIRS here because is not defined.
-obj-m += $(addsuffix .o, $(notdir $(basename $(filter-out %.mod.c, $(wildcard $(BR2_EXTERNAL_LKMC_PATH)/kernel_modules/*.c)))))
-else
 # Trying to do:
 # $(MAKE) -C '$(LINUX_DIR)' M='$(M)' hello.ko hello2.ko
 # to restrict which modules are built leads to failures
 # when doing parallel builds. The only solution I could find
 # was to let the host select obj-m itself.
 obj-m += $(OBJECT_FILES)
-endif
+ccflags-y := \
-ccflags-y := -DDEBUG -g -std=gnu99 -Werror -Wno-declaration-after-statement -Wframe-larger-than=1000000000
+  -DDEBUG \
+  -ggdb3 \
+  -std=gnu99 \
+  -Werror \
+  -Wframe-larger-than=1000000000 \
+  -Wno-declaration-after-statement \
+  $(CCFLAGS)
 
 .PHONY: all
 

kernel_modules/anonymous_inode.c

@@ -9,7 +9,7 @@
 #include <linux/printk.h> /* printk */
 #include <linux/uaccess.h> /* copy_from_user */
 
-#include "../include/anonymous_inode.h"
+#include <lkmc/anonymous_inode.h>
 
 static struct dentry *debugfs_file;
 static u32 myval = 1;

kernel_modules/ioctl.c

@@ -5,7 +5,7 @@
 #include <linux/printk.h> /* printk */
 #include <linux/uaccess.h> /* copy_from_user, copy_to_user */
 
-#include "../include/ioctl.h"
+#include <lkmc/ioctl.h>
 
 static struct dentry *debugfs_file;
 

kernel_modules/netlink.c

@@ -6,7 +6,7 @@
 #include <linux/skbuff.h>
 #include <net/sock.h>
 
-#include "../include/netlink.h"
+#include <lkmc/netlink.h>
 
 struct sock *nl_sk = NULL;
 

kernel_modules/pmccntr.c

@@ -1,4 +1,4 @@
-/* https://github.com/cirosantilli/linux-kernel-module-cheat#pmccntr */
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#arm-pmccntr */
 
 #include <linux/debugfs.h>
 #include <linux/errno.h> /* EFAULT */

kernel_modules/ring0.c

@@ -3,13 +3,13 @@
 #include <linux/module.h>
 #include <linux/kernel.h>
 
-#include "../include/ring0.h"
+#include <lkmc/ring0.h>
 
 static int myinit(void)
 {
 #if defined(__x86_64__) || defined(__i386__)
-	Ring0Regs ring0_regs;
+	LkmcRing0Regs ring0_regs;
-	ring0_get_control_regs(&ring0_regs);
+	lkmc_ring0_get_control_regs(&ring0_regs);
 	pr_info("cr0 = 0x%8.8llX\n", (unsigned long long)ring0_regs.cr0);
 	pr_info("cr2 = 0x%8.8llX\n", (unsigned long long)ring0_regs.cr2);
 	pr_info("cr3 = 0x%8.8llX\n", (unsigned long long)ring0_regs.cr3);

lkmc.c

@@ -11,22 +11,15 @@ void lkmc_assert(bool condition) {
         lkmc_assert_fail();
 }
 
-void lkmc_assert_fail() {
+void lkmc_assert_fail(void) {
-    puts("lkmc_test_fail");
     exit(1);
 }
 
-bool lkmc_vector_equal(size_t n, double *v1, double *v2, double max_err) {
+void lkmc_baremetal_on_exit_callback(int status, void *arg) {
-    double sum = 0.0;
+    (void)arg;
-    double diff;
+    if (status != 0) {
-    size_t i;
+        printf("lkmc_exit_status_%d\n", status);
-    for (i = 0; i < n; ++i) {
-        diff = v1[i] - v2[i];
-        sum += diff * diff;
     }
-    if (sqrt(sum)/n > max_err)
-        return false;
-    return true;
 }
 
 #if defined(__aarch64__)

lkmc.h

@@ -11,7 +11,7 @@
 
 void lkmc_assert(bool);
 void lkmc_assert_fail();
-bool lkmc_vector_equal(size_t n, double *v1, double *v2, double max_err);
+void lkmc_baremetal_on_exit_callback(int status, void *arg);
 #endif
 
 /* https://stackoverflow.com/questions/1489932/how-to-concatenate-twice-with-the-c-preprocessor-and-expand-a-macro-as-in-arg */

lkmc/README.adoc

@@ -0,0 +1 @@
+https://github.com/cirosantilli/linux-kernel-module-cheat#lkmc-directory

lkmc/add.c

@@ -0,0 +1,13 @@
+#include <lkmc.h>
+
+int main(void) {
+    int i, j, k;
+    i = 1;
+    /* test-gdb-op1 */
+    j = 2;
+    /* test-gdb-op2 */
+    k = i + j;
+    /* test-gdb-result */
+    if (k != 3)
+        lkmc_assert_fail();
+}

lkmc/add.py

@@ -0,0 +1,9 @@
+def test(self):
+    self.sendline('tbreak main')
+    self.sendline('continue')
+    self.continue_to('op1')
+    assert self.get_int('i') == 1
+    self.continue_to('op2')
+    assert self.get_int('j') == 2
+    self.continue_to('result')
+    assert self.get_int('k') == 3

lkmc/anonymous_inode.h

@@ -1,5 +1,7 @@
-#ifndef IOCTL_H
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#anonymous-inode */
-#define IOCTL_H
+
+#ifndef LKMC_ANONYMOUS_INODE_H
+#define LKMC_ANONYMOUS_INODE_H
 
 #include <linux/ioctl.h>
 

lkmc/assert_fail.c

@@ -0,0 +1,18 @@
+/* Let's see what happens when an assert fails.
+ *
+ * Outcome on Ubuntu 19.04 shows the failure line:
+ *
+ *     assert_fail.out: /path/to/linux-kernel-module-cheat/userland/c/assert_fail.c:15: main: Assertion `0' failed.
+ *
+ * and exit status 134 == 128 + 6, which corresponds to SIGABORT (6).
+ */
+
+#include <assert.h>
+#include <stdlib.h>
+#include <stdio.h>
+
+int main(void) {
+    assert(0);
+    puts("here");
+    return EXIT_SUCCESS;
+}

lkmc/hello.c

@@ -0,0 +1,9 @@
+/* Print hello to stdout ;-) */
+
+#include <stdio.h>
+#include <stdlib.h>
+
+int main(void) {
+    puts("hello");
+    return EXIT_SUCCESS;
+}

lkmc/import_path.py

@@ -0,0 +1,34 @@
+#!/usr/bin/env python3
+
+import importlib.machinery
+import importlib.util
+import os
+import sys
+
+root_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+def import_path(path):
+    '''
+    https://stackoverflow.com/questions/2601047/import-a-python-module-without-the-py-extension
+    https://stackoverflow.com/questions/31773310/what-does-the-first-argument-of-the-imp-load-source-method-do
+    '''
+    module_name = os.path.basename(path).replace('-', '_')
+    spec = importlib.util.spec_from_loader(
+        module_name,
+        importlib.machinery.SourceFileLoader(module_name, path)
+    )
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    sys.modules[module_name] = module
+    return module
+
+def import_path_relative_root(basename):
+    return import_path(os.path.join(root_dir, basename))
+
+def import_path_main(basename):
+    '''
+    Import an object of the Main class of a given file.
+
+    By convention, we call the main object of all our CLI scripts as Main.
+    '''
+    return import_path_relative_root(basename).Main()

lkmc/ioctl.h

@@ -1,5 +1,7 @@
-#ifndef IOCTL_H
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#ioctl */
-#define IOCTL_H
+
+#ifndef LKMC_IOCTL_H
+#define LKMC_IOCTL_H
 
 #include <linux/ioctl.h>
 

lkmc/lkmc_assert_fail.c

@@ -3,4 +3,3 @@
 int main(void) {
     lkmc_assert_fail();
 }
-

lkmc/m5ops.h

@@ -0,0 +1,56 @@
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#m5ops-instructions */
+
+#ifndef LKMC_M5OPS_H
+#define LKMC_M5OPS_H
+
+#if LKMC_M5OPS_ENABLE == 1
+
+#if defined(__arm__)
+
+#define LKMC_M5OPS_CHECKPOINT_ASM mov r0, #0; mov r1, #0; mov r2, #0; mov r3, #0; .inst 0xEE000110 | (0x43 << 16)
+#define LKMC_M5OPS_DUMPSTATS_ASM  mov r0, #0; mov r1, #0; mov r2, #0; mov r3, #0; .inst 0xEE000110 | (0x41 << 16)
+#define LKMC_M5OPS_EXIT_ASM       mov r0, #0; mov r1, #0;                         .inst 0xEE000110 | (0x21 << 16)
+#define LKMC_M5OPS_FAIL_1_ASM     mov r0, #0; mov r1, #0; mov r2, #1; mov r3, #0; .inst 0xEE000110 | (0x22 << 16)
+#define LKMC_M5OPS_RESETSTATS_ASM mov r0, #0; mov r1, #0; mov r2, #0; mov r3, #0; .inst 0xEE000110 | (0x40 << 16)
+
+#define LKMC_M5OPS_CHECKPOINT __asm__ __volatile__ ("mov r0, #0; mov r1, #0; mov r2, #0; mov r3, #0; .inst 0xEE000110 | (0x43 << 16);" : : : "r0", "r1", "r2", "r3")
+#define LKMC_M5OPS_DUMPSTATS  __asm__ __volatile__ ("mov r0, #0; mov r1, #0; mov r2, #0; mov r3, #0; .inst 0xEE000110 | (0x41 << 16);" : : : "r0", "r1", "r2", "r3")
+#define LKMC_M5OPS_EXIT       __asm__ __volatile__ ("mov r0, #0; mov r1, #0;                         .inst 0xEE000110 | (0x21 << 16);" : : : "r0", "r1"            )
+#define LKMC_M5OPS_FAIL_1     __asm__ __volatile__ ("mov r0, #0; mov r1, #0; mov r2, #1; mov r3, #0; .inst 0xEE000110 | (0x22 << 16);" : : : "r0", "r1", "r2", "r3")
+#define LKMC_M5OPS_RESETSTATS __asm__ __volatile__ ("mov r0, #0; mov r1, #0; mov r2, #0; mov r3, #0; .inst 0xEE000110 | (0x40 << 16);" : : : "r0", "r1", "r2", "r3")
+
+#elif defined(__aarch64__)
+
+#define LKMC_M5OPS_CHECKPOINT_ASM mov x0, #0; mov x1, #0; .inst 0xFF000110 | (0x43 << 16);
+#define LKMC_M5OPS_DUMPSTATS_ASM  mov x0, #0; mov x1, #0; .inst 0xFF000110 | (0x41 << 16);
+#define LKMC_M5OPS_EXIT_ASM       mov x0, #0;             .inst 0XFF000110 | (0x21 << 16);
+#define LKMC_M5OPS_FAIL_1_ASM     mov x0, #0; mov x1, #1; .inst 0xFF000110 | (0x22 << 16);
+#define LKMC_M5OPS_RESETSTATS_ASM mov x0, #0; mov x1, #0; .inst 0XFF000110 | (0x40 << 16);
+
+#define LKMC_M5OPS_CHECKPOINT __asm__ __volatile__ ("mov x0, #0; mov x1, #0; .inst 0xFF000110 | (0x43 << 16);" : : : "x0", "x1")
+#define LKMC_M5OPS_DUMPSTATS  __asm__ __volatile__ ("mov x0, #0; mov x1, #0; .inst 0xFF000110 | (0x41 << 16);" : : : "x0", "x1")
+#define LKMC_M5OPS_EXIT       __asm__ __volatile__ ("mov x0, #0;             .inst 0XFF000110 | (0x21 << 16);" : : : "x0"      )
+#define LKMC_M5OPS_FAIL_1     __asm__ __volatile__ ("mov x0, #0; mov x1, #1; .inst 0xFF000110 | (0x22 << 16);" : : : "x0", "x1")
+#define LKMC_M5OPS_RESETSTATS __asm__ __volatile__ ("mov x0, #0; mov x1, #0; .inst 0XFF000110 | (0x40 << 16);" : : : "x0", "x1")
+
+#else
+#error m5ops not implemented for the current arch
+#endif
+
+#else
+
+#define LKMC_M5OPS_CHECKPOINT_ASM
+#define LKMC_M5OPS_DUMPSTATS_ASM
+#define LKMC_M5OPS_EXIT_ASM
+#define LKMC_M5OPS_FAIL_1_ASM
+#define LKMC_M5OPS_RESETSTATS_ASM
+
+#define LKMC_M5OPS_CHECKPOINT
+#define LKMC_M5OPS_DUMPSTATS
+#define LKMC_M5OPS_EXIT
+#define LKMC_M5OPS_FAIL_1
+#define LKMC_M5OPS_RESETSTATS
+
+#endif
+
+#endif

lkmc/math.h

@@ -0,0 +1,20 @@
+#ifndef LKMC_MATH_H
+#define LKMC_MATH_H
+
+#include <math.h>
+#include <stdbool.h>
+
+bool lkmc_vector_equal(size_t n, double *v1, double *v2, double max_err) {
+    double sum = 0.0;
+    double diff;
+    size_t i;
+    for (i = 0; i < n; ++i) {
+        diff = v1[i] - v2[i];
+        sum += diff * diff;
+    }
+    if (sqrt(sum)/n > max_err)
+        return false;
+    return true;
+}
+
+#endif

lkmc/netlink.h

@@ -1,5 +1,7 @@
-#ifndef NETLINK_H
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#netlink-sockets */
-#define NETLINK_H
+
+#ifndef LKMC_NETLINK_H
+#define LKMC_NETLINK_H
 
 /* Socket identifier, matches userland. TODO can be anything?
  * Is there a more scalable way to do it? E.g. ioctl device,

lkmc/pagemap.h

@@ -1,5 +1,5 @@
-#ifndef COMMON_H
+#ifndef LKMC_PAGEMAP_H
-#define COMMON_H
+#define LKMC_PAGEMAP_H
 
 #define _XOPEN_SOURCE 700
 #include <fcntl.h> /* open */
@@ -17,7 +17,7 @@ typedef struct {
     unsigned int file_page : 1;
     unsigned int swapped : 1;
     unsigned int present : 1;
-} PagemapEntry;
+} LkmcPagemapEntry;
 
 /* Parse the pagemap entry for the given virtual address.
  *
@@ -26,8 +26,7 @@ typedef struct {
  * @param[in]  vaddr      virtual address to get entry for
  * @return                0 for success, 1 for failure
  */
-int pagemap_get_entry(PagemapEntry *entry, int pagemap_fd, uintptr_t vaddr)
+int lkmc_pagemap_get_entry(LkmcPagemapEntry *entry, int pagemap_fd, uintptr_t vaddr) {
-{
     size_t nread;
     ssize_t ret;
     uint64_t data;
@@ -62,8 +61,7 @@ int pagemap_get_entry(PagemapEntry *entry, int pagemap_fd, uintptr_t vaddr)
  * @param[in]  vaddr virtual address to get entry for
  * @return           0 for success, 1 for failure
  */
-int virt_to_phys_user(uintptr_t *paddr, pid_t pid, uintptr_t vaddr)
+int lkmc_pagemap_virt_to_phys_user(uintptr_t *paddr, pid_t pid, uintptr_t vaddr) {
-{
     char pagemap_file[BUFSIZ];
     int pagemap_fd;
 
@@ -72,8 +70,8 @@ int virt_to_phys_user(uintptr_t *paddr, pid_t pid, uintptr_t vaddr)
     if (pagemap_fd < 0) {
         return 1;
     }
-    PagemapEntry entry;
+    LkmcPagemapEntry entry;
-    if (pagemap_get_entry(&entry, pagemap_fd, vaddr)) {
+    if (lkmc_pagemap_get_entry(&entry, pagemap_fd, vaddr)) {
         return 1;
     }
     close(pagemap_fd);

lkmc/ring0.h

@@ -1,29 +1,31 @@
-#if defined(__x86_64__) || defined(__i386__)
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#ring0 */
 
+#ifndef LKMC_RING0_H
+#define LKMC_RING0_H
+#if defined(__x86_64__) || defined(__i386__)
 #ifdef THIS_MODULE
 #include <linux/kernel.h>
 #if defined(__x86_64__)
-typedef u64 T;
+typedef u64 LkmcRing0RegsType;
 #elif defined(__i386__)
-typedef u32 T;
+typedef u32 LkmcRing0RegsType;
 #endif
 #else
 #include <stdint.h>
 #if defined(__x86_64__)
-typedef uint64_t T;
+typedef uint64_t LkmcRing0RegsType;
 #elif defined(__i386__)
-typedef uint32_t T;
+typedef uint32_t LkmcRing0RegsType;
 #endif
 #endif
 
 typedef struct {
-    T cr0;
+    LkmcRing0RegsType cr0;
-    T cr2;
+    LkmcRing0RegsType cr2;
-    T cr3;
+    LkmcRing0RegsType cr3;
-} Ring0Regs;
+} LkmcRing0Regs;
 
-void ring0_get_control_regs(Ring0Regs *ring0_regs)
+void lkmc_ring0_get_control_regs(LkmcRing0Regs *ring0_regs) {
-{
 #if defined(__x86_64__)
     __asm__ __volatile__ (
         "mov %%cr0, %%rax;"
@@ -70,5 +72,5 @@ void ring0_get_control_regs(Ring0Regs *ring0_regs)
     );
 #endif
 }
-
+#endif
 #endif

path_properties.py

@@ -0,0 +1,413 @@
+#!/usr/bin/env python3
+
+import os
+
+from shell_helpers import LF
+
+class PathProperties:
+    default_c_std = 'c11'
+    default_cxx_std = 'c++17'
+    default_properties = {
+        'allowed_archs': None,
+        'c_std': default_c_std,
+        'cc_flags': [
+            '-Wall', LF,
+            '-Werror', LF,
+            '-Wextra', LF,
+            '-Wno-unused-function', LF,
+            '-fopenmp', LF,
+            '-ggdb3', LF,
+            # PIE causes the following problems:
+            # * QEMU GDB step debug does not find breakpoints:
+            #   https://stackoverflow.com/questions/51310756/how-to-gdb-step-debug-a-dynamically-linked-executable-in-qemu-user-mode/51343326#51343326
+            # * when writing assembly code, we have to constantly think about it:
+            #   https://stackoverflow.com/questions/2463150/what-is-the-fpie-option-for-position-independent-executables-in-gcc-and-ld/51308031#51308031
+            # As of 91986fb2955f96e06d1c5ffcc5536ba9f0af1fd9, our Buildroot toolchain
+            # does not have it enabled by default, but the Ubuntu 18.04 host toolchain does.
+            '-fno-pie', LF,
+            '-no-pie', LF,
+        ],
+        'cc_flags_after': [],
+        'cc_pedantic': True,
+        'cxx_std': default_cxx_std,
+        # Expected program exit status. When signals are raised, this refers
+        # to the native exit status. as reported by Bash #?.
+        'exit_status': 0,
+        'extra_objs_baremetal_bootloader': False,
+        # We should get rid of this if we ever properly implement dependency graphs.
+        'extra_objs_lkmc_common': False,
+        'extra_objs_userland_asm': False,
+        'interactive': False,
+        # The script takes a perceptible amount of time to run. Possibly an infinite loop.
+        'more_than_1s': False,
+        # The path should not be built. E.g., it is symlinked into multiple archs.
+        'no_build': False,
+        # The path does not generate an executable in itself, e.g.
+        # it only generates intermediate object files. Therefore it
+        # should not be run while testing.
+        'no_executable': False,
+        # the test receives a signal. We skip those tests for now,
+        # on userland because we are lazy to figure out the exact semantics
+        # of how Python + QEMU + gem5 determine the exit status of signals.
+        'receives_signal': False,
+        # The script requires a non-trivial argument to be passed to run properly.
+        'requires_argument': False,
+        'requires_dynamic_library': False,
+        'requires_m5ops': False,
+        # gem5 fatal: syscall getcpu (#168) unimplemented.
+        'requires_syscall_getcpu': False,
+        'requires_semihosting': False,
+        # Requires certain of our custom kernel modules to be inserted to run.
+        'requires_kernel_modules': False,
+        # The example requires sudo, which usually implies that it can do something
+        # deeply to the system it runs on, which would preventing further interactive
+        # or test usage of the system, for example poweroff or messing up the GUI.
+        'requires_sudo': False,
+        # We were lazy to properly classify why we are skipping these tests.
+        # TODO get it done.
+        'skip_run_unclassified': False,
+        # Aruments added automatically to run when running tests,
+        # but not on manual running.
+        'test_run_args': {
+            'ctrl_c_host': True,
+            'show_stdout': False,
+            'show_time': False,
+            'background': True,
+        },
+    }
+
+    '''
+    Encodes properties of userland and baremetal paths.
+    For directories, it applies to all files under the directory.
+    Used to determine how to build and test the examples.
+    '''
+    def __init__(
+        self,
+        properties
+    ):
+        for key in properties:
+            if not key in self.default_properties:
+                raise ValueError('Unknown key: {}'.format(key))
+        self.properties = properties.copy()
+
+    def __getitem__(self, key):
+        return self.properties[key]
+
+    def __repr__(self):
+        return str(self.properties)
+
+    def set_path_components(self, path_components):
+        self.path_components = path_components
+
+    def should_be_built(self, env, link=False):
+        if len(self.path_components) > 1 and \
+                self.path_components[1] == 'libs' and \
+                not env['package_all'] and \
+                not self.path_components[2] in env['package']:
+            return False
+        return \
+            not self['no_build'] and \
+            (
+                self['allowed_archs'] is None or
+                env['arch'] in self['allowed_archs']
+            ) and \
+            not (
+                link and
+                self['no_executable']
+            )
+
+    def should_be_tested(self, env):
+        return (
+            self.should_be_built(env) and
+            not self['interactive'] and
+            not self['more_than_1s'] and
+            not self['no_executable'] and
+            not self['requires_argument'] and
+            not self['requires_kernel_modules'] and
+            not self['requires_sudo'] and
+            not self['skip_run_unclassified'] and
+            not (
+                env['emulator'] == 'gem5' and
+                (
+                    self['requires_dynamic_library'] or
+                    self['requires_semihosting'] or
+                    self['requires_syscall_getcpu']
+                )
+            ) and
+            not (
+                env['emulator'] == 'qemu' and
+                (
+                    self['requires_m5ops']
+                )
+            )
+        )
+
+    def update(self, other):
+        other_tmp_properties = other.properties.copy()
+        if 'cc_flags' in self.properties and 'cc_flags' in other_tmp_properties:
+            other_tmp_properties['cc_flags'] = \
+                self.properties['cc_flags'] + \
+                other_tmp_properties['cc_flags']
+        if 'test_run_args' in self.properties and 'test_run_args' in other_tmp_properties:
+            other_tmp_properties['test_run_args'] = {
+                **self.properties['test_run_args'],
+                **other_tmp_properties['test_run_args']
+            }
+        return self.properties.update(other_tmp_properties)
+
+class PrefixTree:
+    def __init__(self, path_properties_dict=None, children=None):
+        if path_properties_dict is None:
+            path_properties_dict = {}
+        if children is None:
+            children = {}
+        self.children = children
+        self.path_properties = PathProperties(path_properties_dict)
+
+    @staticmethod
+    def make_from_tuples(tuples):
+        def tree_from_tuples(tuple_):
+            if not type(tuple_) is tuple:
+                tuple_ = (tuple_, {})
+            cur_properties, cur_children = tuple_
+            return PrefixTree(cur_properties, cur_children)
+        top_tree = tree_from_tuples(tuples)
+        todo_trees = [top_tree]
+        while todo_trees:
+            cur_tree = todo_trees.pop()
+            cur_children = cur_tree.children
+            for child_key in cur_children:
+                new_tree = tree_from_tuples(cur_children[child_key])
+                cur_children[child_key] = new_tree
+                todo_trees.append(new_tree)
+        return top_tree
+
+def get(path):
+    cur_node = path_properties_tree
+    path_components = path.split(os.sep)
+    path_properties = PathProperties(cur_node.path_properties.properties.copy())
+    for path_component in path_components:
+        if path_component in cur_node.children:
+            cur_node = cur_node.children[path_component]
+            path_properties.update(cur_node.path_properties)
+        else:
+            break
+    path_properties.set_path_components(path_components)
+    return path_properties
+
+gnu_extension_properties = {
+    'c_std': 'gnu11',
+    'cc_pedantic': False,
+    'cxx_std': 'gnu++17'
+}
+freestanding_properties = {
+    'cc_flags': [
+        '-ffreestanding', LF,
+        '-nostdlib', LF,
+        '-static', LF,
+    ],
+    'extra_objs_userland_asm': False,
+}
+# See: https://github.com/cirosantilli/linux-kernel-module-cheat#path-properties
+path_properties_tuples = (
+    PathProperties.default_properties,
+    {
+        'baremetal': (
+            {},
+            {
+                'arch': (
+                    {},
+                    {
+                        'arm': (
+                            {'allowed_archs': {'arm'}},
+                            {
+                                'gem5_assert.S': {'requires_m5ops': True},
+                                'multicore.S': {'test_run_args': {'cpus': 2}},
+                                'no_bootloader': (
+                                    {'extra_objs_baremetal_bootloader': False},
+                                    {
+                                        'gem5_exit.S': {'requires_m5ops': True},
+                                        'semihost_exit.S': {'requires_semihosting': True},
+                                    }
+                                ),
+                                'return1.S': {'exit_status': 1},
+                                'semihost_exit.S': {'requires_semihosting': True},
+                            },
+
+                        ),
+                        'aarch64': (
+                            {'allowed_archs': {'aarch64'}},
+                            {
+                                'multicore.S': {'test_run_args': {'cpus': 2}},
+                                'no_bootloader': (
+                                    {'extra_objs_baremetal_bootloader': False},
+                                    {
+                                        'gem5_exit.S': {'requires_m5ops': True},
+                                        'semihost_exit.S': {'requires_semihosting': True},
+                                    }
+                                ),
+                                'return1.S': {'exit_status': 1},
+                                'semihost_exit.S': {'requires_semihosting': True},
+                            },
+                        )
+                    }
+                ),
+                'assert_fail.c': {'exit_status': 134},
+                'lkmc_assert_fail.c': {'exit_status': 1},
+                'exit1.c': {'exit_status': 1},
+                'infinite_loop.c': {'more_than_1s': True},
+                'lib': (
+                    {'no_executable': True},
+                    {}
+                ),
+                'getchar.c': {'interactive': True},
+                'return1.c': {'exit_status': 1},
+                'return2.c': {'exit_status': 2},
+            }
+        ),
+        'userland': (
+            {
+                'cc_flags_after': [
+                    '-lm', LF,
+                    '-pthread', LF,
+                ],
+            },
+            {
+                'arch': (
+                    {
+                        'extra_objs_userland_asm': True,
+                    },
+                    {
+                        'arm': (
+                            {
+                                'allowed_archs': {'arm'},
+                                'cc_flags': [
+                                    '-Xassembler', '-mcpu=cortex-a72', LF,
+                                    # To prevent:
+                                    # > vfp.S: Error: selected processor does not support <FPU instruction> in ARM mode
+                                    # https://stackoverflow.com/questions/41131432/cross-compiling-error-selected-processor-does-not-support-fmrx-r3-fpexc-in/52875732#52875732
+                                    # We aim to take the most extended mode currently available that works on QEMU.
+                                    '-Xassembler', '-mfpu=crypto-neon-fp-armv8.1', LF,
+                                    '-Xassembler', '-meabi=5', LF,
+                                    # Treat inline assembly as arm instead of thumb
+                                    # The opposite of -mthumb.
+                                    '-marm', LF,
+                                    # Make gcc generate .syntax unified for inline assembly.
+                                    # However, it gets ignored if -marm is given, which a GCC bug that was recently fixed:
+                                    # https://stackoverflow.com/questions/54078112/how-to-write-syntax-unified-ual-armv7-inline-assembly-in-gcc/54132097#54132097
+                                    # So we just write divided inline assembly for now.
+                                    '-masm-syntax-unified', LF,
+                                ]
+                            },
+                            {
+                                'c': (
+                                    {
+                                        'extra_objs_userland_asm': False,
+                                    },
+                                    {
+                                        'freestanding': freestanding_properties,
+                                    },
+                                ),
+                                'freestanding': freestanding_properties,
+                            }
+                        ),
+                        'aarch64': (
+                            {'allowed_archs': {'aarch64'}},
+                            {
+                                'c': (
+                                    {
+                                        'extra_objs_userland_asm': False,
+                                    },
+                                    {
+                                        'freestanding': freestanding_properties,
+                                    },
+                                ),
+                                'freestanding': freestanding_properties,
+                            }
+                        ),
+                        'fail.S': {'exit_status': 1},
+                        'main.c': {
+                            'extra_objs_userland_asm': False,
+                            'no_executable': True
+                        },
+                        'x86_64': (
+                            {'allowed_archs': {'x86_64'}},
+                            {
+                                'c': (
+                                    {
+                                        'extra_objs_userland_asm': False,
+                                    },
+                                    {
+                                        'freestanding': freestanding_properties,
+                                        'ring0.c': {
+                                            'exit_status': 139,
+                                            'receives_signal': True
+                                        }
+                                    }
+                                ),
+                                'freestanding': freestanding_properties,
+                                'lkmc_assert_eq_fail.S': {'exit_status': 1},
+                                'lkmc_assert_memcmp_fail.S': {'exit_status': 1},
+                            }
+                        ),
+                    }
+                ),
+                'c': (
+                    {},
+                    {
+                        'assert_fail.c': {
+                            'exit_status': 134,
+                            'receives_signal': True,
+                        },
+                        'false.c': {'exit_status': 1},
+                        'getchar.c': {'interactive': True},
+                        'infinite_loop.c': {'more_than_1s': True},
+                    }
+                ),
+                'gcc': gnu_extension_properties,
+                'kernel_modules': {**gnu_extension_properties, **{'requires_kernel_modules': True}},
+                'lkmc': (
+                    {'extra_objs_lkmc_common': True},
+                    {
+                        'assert_fail.c': {'exit_status': 1},
+                    }
+                ),
+                'libs': (
+                    {'requires_dynamic_library': True},
+                    {
+                        'libdrm': {'requires_sudo': True},
+                    }
+                ),
+                'linux': (
+                    gnu_extension_properties,
+                    {
+                        'ctrl_alt_del.c': {'requires_sudo': True},
+                        'init_env_poweroff.c': {'requires_sudo': True},
+                        'myinsmod.c': {'requires_sudo': True},
+                        'myrmmod.c': {'requires_sudo': True},
+                        'pagemap_dump.c': {'requires_argument': True},
+                        'poweroff.c': {'requires_sudo': True},
+                        'proc_events.c': {'requires_sudo': True},
+                        'proc_events.c': {'requires_sudo': True},
+                        'sched_getaffinity.c': {'requires_syscall_getcpu': True},
+                        'sched_getaffinity_threads.c': {
+                            'requires_syscall_getcpu': True,
+                            'more_than_1s': True,
+                        },
+                        'time_boot.c': {'requires_sudo': True},
+                        'virt_to_phys_user.c': {'requires_argument': True},
+                    }
+                ),
+                'posix': (
+                    {},
+                    {
+                        'count.c': {'more_than_1s': True},
+                        'sleep_forever.c': {'more_than_1s': True},
+                        'virt_to_phys_test.c': {'more_than_1s': True},
+                    }
+                ),
+            }
+        ),
+    }
+)
+path_properties_tree = PrefixTree.make_from_tuples(path_properties_tuples)

qemu-monitor

@@ -5,41 +5,44 @@ import sys
 import telnetlib
 
 import common
-from shell_helpers import LF
 
-prompt = b'\n(qemu) '
+class Main(common.LkmcCliFunction):
-
+    def __init__(self):
-parser = self.get_argparse({
+        super().__init__(
-    'description': '''\
+            description='''\
 Run a command on the QEMU monitor of a running QEMU instance
 
 If the stdin is a terminal, open an interact shell. Otherwise,
 run commands from stdin and quit.
-'''
+''',
-})
+        )
-parser.add_argument(
+        self.add_argument(
-    'command',
+            'command',
-    help='If given, run this command and quit',
+            help='If given, run this command and quit',
-    nargs='*',
+            nargs='*',
-)
+        )
-args = self.setup(parser)
 
-def write_and_read(tn, cmd, prompt):
+    def timed_main(self):
-    tn.write(cmd.encode('utf-8'))
+        def write_and_read(tn, cmd, prompt):
-    return '\n'.join(tn.read_until(prompt).decode('utf-8').splitlines()[1:])[:-len(prompt)]
+            tn.write(cmd.encode('utf-8'))
+            return '\n'.join(tn.read_until(prompt).decode('utf-8').splitlines()[1:])[:-len(prompt)]
 
-with telnetlib.Telnet('localhost', kwargs['qemu_monitor_port']) as tn:
+        with telnetlib.Telnet('localhost', self.env['qemu_monitor_port']) as tn:
-    # Couldn't disable server echo, so just removing the write for now.
+            prompt = b'\n(qemu) '
-    # https://stackoverflow.com/questions/12421799/how-to-disable-telnet-echo-in-python-telnetlib
+            # Couldn't disable server echo, so just removing the write for now.
-    # sock = tn.get_socket()
+            # https://stackoverflow.com/questions/12421799/how-to-disable-telnet-echo-in-python-telnetlib
-    # sock.send(telnetlib.IAC + telnetlib.WILL + telnetlib.ECHO)
+            # sock = tn.get_socket()
-    if os.isatty(sys.stdin.fileno()):
+            # sock.send(telnetlib.IAC + telnetlib.WILL + telnetlib.ECHO)
-        if kwargs['command'] == []:
+            if os.isatty(sys.stdin.fileno()):
-            print(tn.read_until(prompt).decode('utf-8'), end='')
+                if self.env['command'] == []:
-            tn.interact()
+                    print(tn.read_until(prompt).decode('utf-8'), end='')
-        else:
+                    tn.interact()
-            tn.read_until(prompt)
+                else:
-            print(write_and_read(tn, ' '.join(kwargs['command']) + '\n', prompt))
+                    tn.read_until(prompt)
-    else:
+                    print(write_and_read(tn, ' '.join(self.env['command']) + '\n', prompt))
-        tn.read_until(prompt)
+            else:
-        print(write_and_read(tn, sys.stdin.read() + '\n', prompt))
+                tn.read_until(prompt)
+                print(write_and_read(tn, sys.stdin.read() + '\n', prompt))
+
+if __name__ == '__main__':
+    Main().cli()

release-zip

@@ -12,7 +12,7 @@ class Main(common.LkmcCliFunction):
 https://github.com/cirosantilli/linux-kernel-module-cheat#release-zip
 ''',
             defaults = {
-                'print_time': False,
+                'show_time': False,
             }
         )
         self.zip_files = []

rootfs_overlay/.profile

@@ -0,0 +1,4 @@
+# https://github.com/cirosantilli/linux-kernel-module-cheat#busybox-shell-initrc-files
+echo "hello .profile"
+export PS1='\w\n\u@\h# '
+cd /lkmc

rootfs_overlay/anonymous_inode.sh

@@ -1,5 +0,0 @@
-#!/bin/sh
-set -e
-insmod anonymous_inode.ko
-[ "$(/anonymous_inode.out /sys/kernel/debug/lkmc_anonymous_inode 3)" = "$(printf '1\n10\n100')" ]
-rmmod anonymous_inode

rootfs_overlay/conf.sh

@@ -1,2 +0,0 @@
-#!/bin/sh
-zcat /proc/config.gz | grep -Ei "${1:-}"

rootfs_overlay/etc/init.d/S98

@@ -1,5 +1,7 @@
 #!/bin/sh
+# https://github.com/cirosantilli/linux-kernel-module-cheat#init-busybox
 echo "hello S98"
+cd "$lkmc_home"
 if [ -n "$lkmc_eval" ]; then
   eval "$lkmc_eval"
 elif [ -n "$lkmc_eval_base64" ]; then

rootfs_overlay/eval.sh

@@ -1,18 +0,0 @@
-#!/bin/sh
-echo "$lkmc_eval"
-eval "$lkmc_eval"
-
-# Ideally, this script would do just:
-#
-## Get rid of the '-'.
-#shift
-#echo "$@"
-#
-# However, the kernel CLI parsing is crap, and the 4.14 docs lie.
-#
-# In particular, not all that is passed after "-" goes to an argument to init,
-# e.g. stuff with dots like "- ./poweroff.out" still gets treated specially and
-# does not go to init.
-#
-# This also likely means that the above solution is also unreliable in some cases,
-# and that in the end you just have to add a script to the root filesystem.

rootfs_overlay/eval_base64.sh

@@ -1,2 +0,0 @@
-#!/bin/sh
-eval "$(printf "$lkmc_eval" | base64 -d)"

rootfs_overlay/ioctl.sh

@@ -1,7 +0,0 @@
-#!/bin/sh
-set -e
-f=/sys/kernel/debug/lkmc_ioctl
-insmod ioctl.ko
-[ "$(/ioctl.out "$f" 0 1)" = 2 ]
-[ "$(/ioctl.out "$f" 1 1 1)" = '2 0' ]
-rmmod ioctl

rootfs_overlay/lkmc/anonymous_inode.sh

@@ -0,0 +1,6 @@
+#!/bin/sh
+# https://github.com/cirosantilli/linux-kernel-module-cheat#anonymous-inode
+set -e
+insmod anonymous_inode.ko
+[ "$(./kernel_modules/anonymous_inode.out /sys/kernel/debug/lkmc_anonymous_inode 3)" = "$(printf '1\n10\n100')" ]
+rmmod anonymous_inode

rootfs_overlay/lkmc/conf.sh

@@ -0,0 +1,3 @@
+#!/bin/sh
+# https://github.com/cirosantilli/linux-kernel-module-cheat#find-the-kernel-config
+zcat /proc/config.gz | grep -Ei "${1:-}"

rootfs_overlay/lkmc/count.sh

@@ -1,4 +1,5 @@
 #!/bin/sh
+# Count to infinity with 1 second sleep between each increment.
 # Generate infinitely many system calls :-)
 i=0
 while true; do