Build the Linux kernel independently from Buildroot

This will allow for other types of root filesystems that don't rely on Buildroot
to be added and used in the future.

Propagate --verbose on all build scripts to see full GCC commands.

build-all: allow for neat subsets

also 9p share rootfs_overlay. TODO document.

README.adoc

@@ -75,6 +75,8 @@ cd linux-kernel-module-cheat
 ./configure --qemu && \
   ./build-qemu &&
   ./build-buildroot && \
+  ./build-linux && \
+  ./build-modules && \
   ./run && \
 :;
 ....
@@ -98,7 +100,7 @@ see this: https://askubuntu.com/questions/496549/error-you-must-put-some-source-
 
 It does not work if you just download the `.zip` from GitHub because we use link:.gitmodules[Git submodules], you must clone this repo. `./configure` then fetches only the required submodules for you.
 
-QEMU opens up and you can start playing with the kernel modules inside the simulated system:
+QEMU opens up and you can start playing with the kernel modules inside the simulated system: TODO fix path to 9p:
 
 ....
 insmod /hello.ko
@@ -210,10 +212,12 @@ pr_info("hello init hacked\n");
 and then rebuild the kernel modules and re-run to see it take effect:
 
 ....
-./build-buildroot --kernel-modules
+./build-modules
 ./run --eval-busybox 'insmod /hello.ko'
 ....
 
+TODO mention 9p
+
 Congratulations, you are now officially a kernel module hacker!
 
 We use `./build-buildroot` because the kernel modules go inside the root filesystem, and it is Buildroot that generates and updates our root filesystem. The kernel modules are link:packages/kernel_modules[inside a Buildroot package].
@@ -245,7 +249,7 @@ pr_info("I'VE HACKED THE LINUX KERNEL!!!");
 Then rebuild the Linux kernel and reboot:
 
 ....
-./build-buildroot && ./run
+./build-linux && ./run
 ....
 
 and, surely enough, your message has appeared at the beginning of the boot.
@@ -321,6 +325,8 @@ All of this makes QEMU the natural choice of default system simulator.
 
 === gem5 Buildroot setup
 
+TODO document kernel modules don't work on 9p
+
 ==== About the gem5 Buildroot setup
 
 This setup is like the <<qemu-buildroot-setup>>, but it uses link:http://gem5.org/[gem5] instead of QEMU as a system simulator.
@@ -535,7 +541,6 @@ The limitations are severe however:
 * can't <<gdb,GDB step debug the kernel>>, since the source and cross toolchain with GDB are not available. Buildroot cannot easily use a host toolchain: <<prebuilt-toolchain>>.
 +
 Maybe we could work around this by just downloading the kernel source somehow, and using a host prebuilt GDB, but we felt that it would be too messy and unreliable.
-* can't create new modules or modify the existing ones, since no cross toolchain
 * you won't get the latest version of this repository. Our <<travis>> attempt to automate builds failed, and storing a release for every commit would likely make GitHub mad at us.
 * <<gem5>> is not currently supported, although it should not be too hard to do. Annoyances:
 +
@@ -583,6 +588,23 @@ then do whatever that checked out README says.
 
 If you are curious to see what the releases contain in detail, have a look at our <<release,release procedure>>.
 
+To build the kernel modules, simply do:
+
+....
+./build-linux -- modules_prepare
+./build-modules
+./run
+....
+
+`modules_prepare` does the minimal build procedure required on the kernel for us to be able to compile the kernel modules, and is way faster than doing a full kernel build. A full kernel build would also work however.
+
+To modify the Linux kernel, build and use it as usual:
+
+....
+./build-linux
+./run
+....
+
 ////
 For gem5, do:
 
@@ -601,11 +623,11 @@ The Linux kernel is required for `extract-vmlinux` to convert the compressed ker
 [[host]]
 === Host kernel module setup
 
-**THIS IS DANGEROUS, YOU HAVE BEEN WARNED**
+**THIS IS DANGEROUS (AND FUN), YOU HAVE BEEN WARNED**
 
 This method runs the kernel modules directly on your host computer without a VM, and saves you the compilation time and disk usage of the virtual machine method.
 
-It has however severe limitations, and you will soon see that the compilation time and disk usage are well worth it:
+It has however severe limitations:
 
 * can't control which kernel version and build options to use. So some of the modules will likely not compile because of kernel API changes, since https://stackoverflow.com/questions/37098482/how-to-build-a-linux-kernel-module-so-that-it-is-compatible-with-all-kernel-rele/45429681#45429681[the Linux kernel does not have a stable kernel module API].
 * bugs can easily break you system. E.g.:
@@ -618,20 +640,34 @@ It has however severe limitations, and you will soon see that the compilation ti
 Still interested?
 
 ....
-cd packages/kernel_modules
+./build-modules --host
-./make-host.sh
 ....
 
-If the compilation of any of the C files fails because of kernel or toolchain differences that we don't control on the host, just rename it to remove the `.c` extension and try again:
+Compilation will likely fail for some modules because of kernel or toolchain differences that we can't control on the host.
+
+The best solution is to compile just your modules with:
 
 ....
+./build-modules --host -- hello hello2
+....
+
+which is equivalent to:
+
+....
+./build-modules --host -- packages/kernel/modules/hello.c packages/kernel/modules/hello2.c
+....
+
+Or just remove the `.c` extension from the failing files and try again:
+
+....
+cd "$(./getvar kernel_modules_src_dir)"
 mv broken.c broken.c~
-./build_host
 ....
 
 Once you manage to compile, and have come to terms with the fact that this may blow up your host, try it out with:
 
 ....
+cd "$(./getvar kernel_modules_build_host_dir)"
 sudo insmod hello.ko
 
 # Our module is there.
@@ -649,15 +685,6 @@ dmesg -T
 sudo lsmod | grep hello
 ....
 
-Once you are done with this method, you must clean up the in-tree build objects before you decide to do the right thing and move on to the superior `./build` Buildroot method:
-
-....
-cd packages/kernel_modules
-./make-host.sh clean
-....
-
-otherwise they will cause problems.
-
 ==== Hello host
 
 Minimal host build system example:
@@ -3362,6 +3389,8 @@ although this can be useful when someone gives you a random image.
 [[kernel-configs-about]]
 ==== About our Linux kernel configs
 
+TODO: explain link:update-buildroot-kernel-config[]
+
 We have managed to come up with minimalistic kernel configs that work for both QEMU and gem5 (oh, the hours of bisection).
 
 Our configs are all based on Buildroot's configs, which were designed for QEMU, and then on top of those we also add:
@@ -7276,7 +7305,7 @@ Bibliography:
 
 ==== 9P gem5
 
-Seems possible! Lets do it:
+TODO seems possible! Lets do it:
 
 * http://gem5.org/wiki/images/b/b8/Summit2017_wa_devlib.pdf
 * http://gem5.org/WA-gem5
@@ -10853,23 +10882,21 @@ We use it for:
 +
 C files for example need compilation, and must go through the regular package system, e.g. through link:packages/kernel_modules/user[].
 
-=== CONTRIBUTING
+=== Test this repo
 
-==== Testing
+This section describes how to run the most complete set of tests possible.
 
-Testing that should be done for every functional patch.
+It takes too much time to be feasible for every patch, but it should be done for every release.
 
-===== Guest testing
+==== Automated tests
-
-Run all tests:
 
 ....
 ./build-all
-./test
+./test --size 3
 echo $?
 ....
 
-Should output 0.
+should output 0.
 
 Sources:
 
@@ -10893,7 +10920,7 @@ Test that the Internet works:
 
 Source: link:rootfs_overlay/test_all.sh[].
 
-===== Host testing
+===== Test GDB
 
 Shell 1:
 
@@ -10914,7 +10941,7 @@ Then proceed to do the following tests:
 * `/count.sh` and `b __x64_sys_write`
 * `insmod /timer.ko` and `b lkmc_timer_callback`
 
-==== Bisection
+=== Bisection
 
 When updating the Linux kernel, QEMU and gem5, things sometimes break.
 
@@ -10953,7 +10980,7 @@ git submodule update
 
 An example of Linux kernel commit bisection on gem5 boots can be found at: link:bisect-linux-boot-gem5[].
 
-==== Update a forked submodule
+=== Update a forked submodule
 
 This is a template update procedure for submodules for which we have some patches on on top of mainline.
 
@@ -10979,7 +11006,7 @@ git rebase --onto "$next_mainline_revision" "$last_mainline_revision"
 git commit -m "linux: update to ${next_mainline_revision}"
 ....
 
-==== Sanity checks
+=== Sanity checks
 
 Basic C and C++ hello worlds:
 
@@ -11000,7 +11027,7 @@ Sources:
 * link:packages/kernel_modules/user/hello.c[]
 * link:packages/kernel_modules/user/hello_cpp.c[]
 
-===== rand_check.out
+==== rand_check.out
 
 Print out several parameters that normally change randomly from boot to boot:
 
@@ -11015,7 +11042,7 @@ This can be used to check the determinism of:
 * <<norandmaps>>
 * <<qemu-record-and-replay>>
 
-==== Release
+=== Release
 
 Create a release:
 
@@ -11042,7 +11069,7 @@ This should in particular enable to easily update <<prebuilt>>.
 
 TODO also run tests and only release if they pass.
 
-===== release-zip
+==== release-zip
 
 Create a zip containing all files required for <<prebuilt>>
 

bench-all

@@ -117,7 +117,7 @@ fi
 if "$bench_linux_boot"; then
   cd "${root_dir}"
   "${root_dir}/build-all"
-  "${root_dir}/bench-boot" -t 3
+  "${root_dir}/bench-boot" --size 3
   cp "$(${root_dir}/getvar bench_boot)" "$new_dir"
 fi
 

bench-boot

@@ -2,21 +2,18 @@
 set -eu
 root_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)"
 test_size=1
-OPTIND=1
+while [ $# -gt 0 ]; do
-while getopts t: OPT; do
+  case "$1" in
-  case "$OPT" in
+    --size)
-    t)
       # 1: a few seconds and important
       # 2: < 5 minutes and important or a few seconds and not too important
       # 3: all
-      test_size="$OPTARG"
+      test_size="$2"
-      ;;
+      shift 2
-    ?)
-      exit 2
       ;;
   esac
 done
-shift "$(($OPTIND - 1))"
+
 if [ $# -gt 1 ]; then
   extra_args=" $*"
 else

bisect-linux-boot-gem5

@@ -24,7 +24,7 @@ args = common.setup(parser)
 # We need a clean rebuild because rebuilds at different revisions:
 # - may fail
 # - may not actually rebuild all files, e.g. on header changes
-common.rmrf(common.linux_variant_dir)
+common.rmrf(common.linux_build_dir)
 assert build.main(args) == 0
 status = run.main(args, {
     'eval': 'm5 exit',

build-all

@@ -1,28 +1,89 @@
 #!/usr/bin/env bash
+
+# Build everything, or a subset of everything.
+#
+# Without any args, build everything for all archs:
+#
+#     ./build-all
+#
+# This will build QEMU, gem5, Buildroot, Linux, etc.
+# for x86_64, arm and aarch64.
+#
+# With --archs, build everything for just the given archs:
+#
+#    ./build-all --archs 'arm aarch64'
+#
+# Other options make this script build only the given projects. E.g., to build
+# just Linux and QEMU for all archs, but not gem5, Buildroot, etc.:
+#
+#    ./build-all --linux --qemu
+
 set -eu
 archs='x86_64 arm aarch64'
-gem5=true
+baremetal=false
-while getopts A:G OPT; do
+buildroot=false
-  case "$OPT" in
+gem5=false
-    A)
+linux=false
-      archs="$OPTARG"
+qemu=false
-      ;;
+while [ $# -gt 0 ]; do
-    G)
+  case "$1" in
-      gem5=false
+    --archs)
+      archs="$2"
+      shift 2
       ;;
+    --baremetal)
+      baremetal=true
+      shift
+    ;;
+    --buildroot)
+      buildroot=true
+      shift
+    ;;
+    --gem5)
+      gem5=true
+      shift
+    ;;
+    --linux)
+      linux=true
+      shift
+    ;;
+    --qemu)
+      qemu=true
+      shift
+    ;;
   esac
 done
-shift "$(($OPTIND - 1))"
+if ! (
+    "$baremetal" || \
+    "$buildroot" || \
+    "$gem5" || \
+    "$linux" || \
+    "$qemu"
+  )
+then
+  baremetal=true
+  buildroot=true
+  gem5=true
+  linux=true
+  qemu=true
+fi
 for arch in $archs; do
-  ./build-qemu --arch "$arch"
+  if "$qemu"; then
-  if "$gem5"; then
+    ./build-qemu --arch "$arch"
-    ./build-gem5 --arch "$arch"
   fi
-  ./build-buildroot --arch "$arch" --gem5 --kernel-modules -l "$@"
+  if "$baremetal" && [ ! "$arch" = x86_64 ]; then
-  if [ ! "$arch" = x86_64 ]; then
     ./build-crosstool-ng --arch "$arch"
     ./build-baremetal --arch "$arch"
     ./build-baremetal --arch "$arch" --gem5
     ./build-baremetal --arch "$arch" --gem5 --machine RealViewPBX
   fi
+  if "$buildroot"; then
+    ./build-buildroot --arch "$arch" --gem5 --kernel-modules "$@"
+  fi
+  if "$gem5"; then
+    ./build-gem5 --arch "$arch"
+  fi
+  if "$linux"; then
+    ./build-linux --arch "$arch"
+  fi
 done

build-baremetal

@@ -76,7 +76,7 @@ def main(args, extra_args=None):
             gcc = 'arm-none-eabi-gcc'
         else:
             os.environ['PATH'] = common.crosstool_ng_bin_dir + os.environ['PATH']
-            gcc = common.get_toolchain_tool('gcc')
+            gcc = common.get_toolchain_tool('gcc', allowed_toolchains=['crosstool-ng'])
         if args.gem5:
             if common.machine == 'VExpress_GEM5_V1':
                 entry_address = 0x80000000

build-buildroot

@@ -13,7 +13,7 @@ import common
 
 defaults = {
     'baseline': False,
-    'buildroot_bare_kernel': False,
+    'build_linux': False,
     'buildroot_config': [],
     'buildroot_config_fragment': [],
     'initramfs': False,
@@ -27,7 +27,6 @@ defaults = {
     'no_all': False,
     'nproc': None,
     'skip_configure': False,
-    'verbose': False,
     'extra_make_args': [],
 }
 
@@ -47,15 +46,25 @@ Takes precedence over any Buildroot config files.
         '-b', '--buildroot-config-fragment', default=defaults['buildroot_config_fragment'], action='append',
         help='''Also use the given Buildroot configuration fragment file.
 Pass multiple times to use multiple fragment files.'''
+    )
+    parser.add_argument(
+        '--build-linux', default=defaults['build_linux'], action='store_true',
+        help='''\
+Enable building the Linux kernel with Buildroot. This is done mostly
+to extract Buildroot's default kernel configurations when updating Buildroot.
+That kernel will not be use by our scripts.
+'''
     )
     parser.add_argument(
         '--baseline', default=defaults['baseline'], action='store_true',
         help='''Do a default-ish Buildroot defconfig build, without any of our extra options.
-Mostly to track how much slower we are than a basic build.'''
+Mostly to track how much slower we are than a basic build.
+'''
     )
     parser.add_argument(
         '-C', '--kernel-config', default=defaults['kernel_config'], action='append',
-        help='''Add a single kernel config configs to the current build.
+        help='''\
+Add a single kernel config configs to the current build.
 Example value: 'CONFIG_FORTIFY_SOURCE=y'.
 Can be used multiple times to add multiple configs.
 Takes precedence over any Buildroot config files.
@@ -63,8 +72,10 @@ Takes precedence over any Buildroot config files.
     )
     parser.add_argument(
         '-c', '--kernel-config-fragment', default=defaults['kernel_config_fragment'], action='append',
-        help='''Also use the given kernel configuration fragment file.
+        help='''\
-Pass multiple times to use multiple fragment files.'''
+Also use the given kernel configuration fragment file.
+Pass multiple times to use multiple fragment files.
+'''
     )
     parser.add_argument(
         '-I', '--initramfs', default=defaults['initramfs'], action='store_true',
@@ -78,41 +89,35 @@ Pass multiple times to use multiple fragment files.'''
     )
     parser.add_argument(
         '-K', '--kernel-custom-config-file', default=defaults['kernel_custom_config_file'],
-        help='''Ignore all default kernel configurations and use this file instead.
+        help='''\
-Still uses options explicitly passed with `-C` and `-c` on top of it.'''
+Ignore all default kernel configurations and use this file instead.
+Still uses options explicitly passed with `-C` and `-c` on top of it.
+'''
     )
     kernel_module_group.add_argument(
         '-k', '--kernel-modules', default=defaults['kernel_modules'], action='store_true',
         help='Reconfigure and rebuild the kernel modules package'
     )
-    parser.add_argument(
-        '-l', '--linux-reconfigure', default=defaults['linux_reconfigure'], action='store_true',
-        help='''Reconfigure and rebuild the Linux kernel.
-Touches kernel configuration files to overcome:
-https://stackoverflow.com/questions/49260466/why-when-i-change-br2-linux-kernel-custom-config-file-and-run-make-linux-reconfi'''
-    )
     parser.add_argument(
         '--no-all', default=defaults['no_all'], action='store_true',
-        help='''Don't build the all target which normally gets build by default.
+        help='''\
-That target builds the root filesystem and all its dependencies.'''
+Don't build the all target which normally gets build by default.
-    )
+That target builds the root filesystem and all its dependencies.
-    kernel_module_group.add_argument(
+'''
-        '--no-kernel-modules', default=defaults['no_kernel_modules'], action='store_true',
-        help="Don't build the kernel modules package"
     )
     parser.add_argument(
         '--skip-configure', default=defaults['skip_configure'], action='store_true',
-        help='''Skip the Buildroot configuration. Saves a few seconds,
+        help='''\
-but requires you to know what you are doing :-)'''
+Skip the Buildroot configuration. Saves a few seconds,
-    )
+but requires you to know what you are doing :-)
-    parser.add_argument(
+'''
-        '-v', '--verbose', default=defaults['verbose'], action='store_true',
-        help='Do a verbose build'
     )
     parser.add_argument(
         'extra_make_args', default=defaults['extra_make_args'], metavar='extra-make-args', nargs='*',
-        help='''Extra arguments to be passed to the Buildroot make,
+        help='''\
-usually extra Buildroot targets.'''
+Extra arguments to be passed to the Buildroot make,
+usually extra Buildroot targets.
+'''
     )
     return parser
 
@@ -125,6 +130,7 @@ def main(args, extra_args=None):
         os.makedirs(common.out_dir, exist_ok=True)
         extra_make_args = args.extra_make_args.copy()
         if args.kernel_modules:
+            assert(args.build_linux)
             extra_make_args.append('kernel_modules-reconfigure')
         if args.linux_reconfigure:
             extra_make_args.append('linux-reconfigure')
@@ -171,6 +177,10 @@ def main(args, extra_args=None):
                 'BR2_DL_DIR="{}"'.format(common.buildroot_download_dir),
             ])
             common.write_configs(common.buildroot_config_file, buildroot_configs)
+            if args.build_linux:
+                buildroot_configs.extend([
+                    '# BR2_LINUX_KERNEL is not set',
+                ])
             if not args.baseline:
                 buildroot_configs.extend([
                     'BR2_GLOBAL_PATCH_DIR="{}"'.format(
@@ -180,13 +190,13 @@ def main(args, extra_args=None):
                     'BR2_PACKAGE_OVERRIDE_FILE="{}"'.format(
                         path_relative_to_buildroot(os.path.join(common.root_dir, 'buildroot_override'))),
                     'BR2_ROOTFS_OVERLAY="{}"'.format(
-                        path_relative_to_buildroot(os.path.join(common.root_dir, 'rootfs_overlay'))),
+                        path_relative_to_buildroot(common.rootfs_overlay_dir)),
                     'BR2_ROOTFS_POST_BUILD_SCRIPT="{}"'.format(
                         path_relative_to_buildroot(os.path.join(common.root_dir, 'rootfs-post-build-script'))),
                     'BR2_ROOTFS_USERS_TABLES="{}"'.format(
                         path_relative_to_buildroot(os.path.join(common.root_dir, 'user_table'))),
                 ])
-                if not args.no_kernel_modules:
+                if args.kernel_modules:
                     buildroot_configs.append('BR2_PACKAGE_KERNEL_MODULES=y')
                 if args.gem5:
                     buildroot_configs.append('BR2_PACKAGE_GEM5=y')
@@ -241,13 +251,12 @@ def main(args, extra_args=None):
                             raise Exception('Kernel config fragment file does not exist: {}'.format(args.kernel_custom_config_file))
                         default_kernel_config_fragments = []
                     else:
-                        kernel_config_fragment_dir = os.path.join(common.root_dir, 'kernel_config')
                         default_kernel_config_fragments = ['min', 'default']
                         if args.linux_reconfigure:
                             # https://stackoverflow.com/questions/49260466/why-when-i-change-br2-linux-kernel-custom-config-file-and-run-make-linux-reconfi
-                            pathlib.Path(os.path.join(kernel_config_fragment_dir, 'min')).touch()
+                            pathlib.Path(os.path.join(common.linux_config_dir, 'min')).touch()
                     for i, default_kernel_config_fragment in enumerate(default_kernel_config_fragments):
-                        default_kernel_config_fragments[i] = os.path.join(kernel_config_fragment_dir, default_kernel_config_fragment)
+                        default_kernel_config_fragments[i] = os.path.join(common.linux_config_dir, default_kernel_config_fragment)
                 kernel_config_fragments.extend(default_kernel_config_fragments)
                 for i, frag in enumerate(kernel_config_fragments):
                     kernel_config_fragments[i] = path_relative_to_buildroot(frag)
@@ -263,17 +272,6 @@ def main(args, extra_args=None):
                     cwd=common.buildroot_src_dir,
                 ) == 0
 
-        # Manage Linux kernel and QEMU variants.
-        def symlink_buildroot_variant(custom_dir, variant_dir):
-            if os.path.islink(custom_dir):
-                os.unlink(custom_dir)
-            elif os.path.isdir(custom_dir):
-                # Migration for existing builds.
-                shutil.move(custom_dir, variant_dir)
-            os.makedirs(variant_dir, exist_ok=True)
-            os.symlink(variant_dir, custom_dir)
-        symlink_buildroot_variant(common.linux_build_dir, common.linux_variant_dir)
-
         # Do the actual build.
         common.mkdir()
         if not args.no_all:

build-gem5

@@ -38,6 +38,10 @@ else:
                 '-b', os.path.join('wt', args.gem5_build_id),
                 common.gem5_src_dir
             ]) == 0
+    if args.verbose:
+        verbose = ['--verbose']
+    else:
+        verbose = []
     if args.arch == 'x86_64':
         dummy_img_path = os.path.join(disks_dir, 'linux-bigswap2.img')
         with open(dummy_img_path, 'wb') as dummy_img_file:
@@ -73,14 +77,16 @@ else:
         assert common.run_cmd(['make', '-C', bootloader64_dir]) == 0
         shutil.copy2(os.path.join(bootloader64_dir, 'boot_emm.arm64'), binaries_dir)
     assert common.run_cmd(
-        [
+        (
-            'scons',
+            [
-            # TODO factor with build.
+                'scons',
-            '-j', str(multiprocessing.cpu_count()),
+                '-j', str(multiprocessing.cpu_count()),
-            '--ignore-style',
+                '--ignore-style',
-            common.gem5_executable
+                common.gem5_executable
-        ] +
+            ] +
-        args.extra_scons_args,
+            verbose +
+            args.extra_scons_args
+        ),
         cwd=common.gem5_src_dir,
         extra_paths=[common.ccache_dir],
     ) == 0

build-linux

@@ -0,0 +1,86 @@
+#!/usr/bin/env python3
+
+import multiprocessing
+import os
+import shutil
+import subprocess
+import time
+
+import common
+
+parser = common.get_argparse()
+common.add_build_arguments(parser)
+parser.add_argument(
+    'extra_make_args',
+    default=[],
+    metavar='extra-make-args',
+    nargs='*'
+)
+args = common.setup(parser)
+if args.clean:
+    common.rmrf(common.linux_build_dir)
+else:
+    start_time = time.time()
+    os.makedirs(common.linux_build_dir, exist_ok=True)
+    shutil.copy2(
+        os.path.join(common.linux_config_dir, 'buildroot-{}'.format(args.arch)),
+        os.path.join(common.linux_build_dir, '.config'),
+    )
+    tool = 'gcc'
+    gcc = common.get_toolchain_tool(tool)
+    prefix = gcc[:-len(tool)]
+    common_args = {
+        'cwd': common.linux_src_dir,
+    }
+    ccache = shutil.which('ccache')
+    if ccache is not None:
+        cc = '{} {}'.format(ccache, gcc)
+    else:
+        cc = gcc
+    common_make_args = [
+        'ARCH={}'.format(common.linux_arch),
+        'CROSS_COMPILE={}'.format(prefix),
+        'CC={}'.format(cc),
+        'O={}'.format(common.linux_build_dir),
+    ]
+    if args.verbose:
+        verbose = ['V=1']
+    else:
+        verbose = []
+    assert common.run_cmd(
+        [
+            os.path.join(common.linux_src_dir, 'scripts', 'kconfig', 'merge_config.sh'),
+            '-m',
+            '-O', common.linux_build_dir,
+            os.path.join(common.linux_build_dir, '.config'),
+            os.path.join(common.linux_config_dir, 'min'),
+            os.path.join(common.linux_config_dir, 'default'),
+        ],
+    ) == 0
+    assert common.run_cmd(
+        (
+            [
+                'make',
+                '-j', str(multiprocessing.cpu_count()),
+            ] +
+            common_make_args +
+            [
+                'olddefconfig',
+            ]
+        ),
+        **common_args,
+    ) == 0
+    assert common.run_cmd(
+        (
+            [
+                'make',
+                '-j', str(multiprocessing.cpu_count()),
+            ] +
+            common_make_args +
+            verbose +
+            args.extra_make_args
+        ),
+        **common_args,
+    ) == 0
+    end_time = time.time()
+    common.print_time(end_time - start_time)

build-modules

@@ -0,0 +1,122 @@
+#!/usr/bin/env python3
+
+import distutils.file_util
+import multiprocessing
+import os
+import platform
+import shutil
+import subprocess
+import time
+
+import common
+
+parser = common.get_argparse()
+common.add_build_arguments(parser)
+parser.add_argument(
+    '--host',
+    action='store_true',
+    default=False,
+    help='Build the modules for the host instead of guest',
+)
+parser.add_argument(
+    'kernel_modules',
+    default=[],
+    help='Which kernel modules to build. Default: build all',
+    metavar='kernel-modules',
+    nargs='*',
+)
+args = common.setup(parser)
+if args.host:
+    build_dir = os.path.join(common.kernel_modules_build_host_dir)
+else:
+    build_dir = os.path.join(common.kernel_modules_build_dir)
+if args.clean:
+    common.rmrf(build_dir)
+else:
+    start_time = time.time()
+    os.makedirs(build_dir, exist_ok=True)
+
+    # Build kernel modules.
+    #
+    # I kid you not, out-of-tree build is not possible, O= does not work as for the kernel build:
+    #
+    # * https://stackoverflow.com/questions/5718899/building-an-out-of-tree-linux-kernel-module-in-a-separate-object-directory
+    # * https://stackoverflow.com/questions/12244979/build-kernel-module-into-a-specific-directory
+    # * https://stackoverflow.com/questions/18386182/out-of-tree-kernel-modules-multiple-module-single-makefile-same-source-file
+    #
+    # This copies only modified files as per:
+    # https://stackoverflow.com/questions/5718899/building-an-out-of-tree-linux-kernel-module-in-a-separate-object-directory
+    all_kernel_modules = []
+    for basename in os.listdir(common.kernel_modules_src_dir):
+        src = os.path.join(common.kernel_modules_src_dir, basename)
+        if os.path.isfile(src):
+            distutils.file_util.copy_file(
+                src,
+                os.path.join(build_dir, basename),
+                update=1,
+            )
+            noext, ext = os.path.splitext(basename)
+            if ext == common.c_ext:
+                all_kernel_modules.append(noext)
+    if args.kernel_modules == []:
+        kernel_modules = all_kernel_modules
+    else:
+        kernel_modules = map(lambda x: os.path.splitext(os.path.split(x)[1])[0], args.kernel_modules)
+    object_files = map(lambda x: x + common.obj_ext, kernel_modules)
+    tool = 'gcc'
+    if args.host:
+        allowed_toolchains = ['host']
+    else:
+        allowed_toolchains = None
+    gcc = common.get_toolchain_tool(tool, allowed_toolchains=allowed_toolchains)
+    prefix = gcc[:-len(tool)]
+    ccache = shutil.which('ccache')
+    if ccache is not None:
+        cc = '{} {}'.format(ccache, gcc)
+    else:
+        cc = gcc
+    if args.verbose:
+        verbose = ['V=1']
+    else:
+        verbose = []
+    if args.host:
+        linux_dir = os.path.join('/lib', 'modules', platform.uname().release, 'build')
+    else:
+        linux_dir = common.linux_build_dir
+    assert common.run_cmd(
+        (
+            [
+                'make',
+                '-j', str(multiprocessing.cpu_count()),
+                'ARCH={}'.format(common.linux_arch),
+                'CC={}'.format(cc),
+                'CROSS_COMPILE={}'.format(prefix),
+                'LINUX_DIR={}'.format(linux_dir),
+                'M={}'.format(build_dir),
+                'OBJECT_FILES={}'.format(' '.join(object_files)),
+            ] +
+            verbose
+        ),
+        cwd=common.kernel_modules_src_dir,
+    ) == 0
+
+    # Build userland tools.
+    if args.host:
+        allowed_toolchains = ['host']
+    else:
+        allowed_toolchains = ['buildroot']
+    cc = common.get_toolchain_tool('gcc', allowed_toolchains=allowed_toolchains)
+    cxx = common.get_toolchain_tool('g++', allowed_toolchains=allowed_toolchains)
+    assert common.run_cmd(
+        [
+            'make',
+            '-j', str(multiprocessing.cpu_count()),
+            'CC={}'.format(cc),
+            'CXX={}'.format(cxx),
+            'OUT_DIR={}'.format(os.path.join(build_dir, 'user')),
+        ],
+        cwd=os.path.join(common.kernel_modules_src_dir, 'user'),
+        extra_paths=[common.ccache_dir],
+    ) == 0
+    end_time = time.time()
+    common.print_time(end_time - start_time)

build-qemu

@@ -21,6 +21,10 @@ if args.clean:
 else:
     start_time = time.time()
     os.makedirs(common.qemu_build_dir, exist_ok=True)
+    if args.verbose:
+        verbose = ['V=1']
+    else:
+        verbose = []
     assert common.run_cmd(
         [
             os.path.join(common.qemu_src_dir, 'configure'),
@@ -35,11 +39,14 @@ else:
         cwd=common.qemu_build_dir
     ) == 0
     assert common.run_cmd(
-        [
+        (
-            'make',
+            [
-            # TODO factor with build.
+                'make',
-            '-j', str(multiprocessing.cpu_count()),
+                '-j', str(multiprocessing.cpu_count()),
-        ],
+
+            ] +
+            verbose
+        ),
         cwd=common.qemu_build_dir,
         extra_paths=[common.ccache_dir],
     ) == 0

common.py

@@ -32,6 +32,8 @@ submodules_dir = os.path.join(root_dir, 'submodules')
 buildroot_src_dir = os.path.join(submodules_dir, 'buildroot')
 crosstool_ng_src_dir = os.path.join(submodules_dir, 'crosstool-ng')
 linux_src_dir = os.path.join(submodules_dir, 'linux')
+linux_config_dir = os.path.join(this.root_dir, 'kernel_config')
+rootfs_overlay_dir = os.path.join(this.root_dir, 'rootfs_overlay')
 extract_vmlinux = os.path.join(linux_src_dir, 'scripts', 'extract-vmlinux')
 qemu_src_dir = os.path.join(submodules_dir, 'qemu')
 parsec_benchmark_src_dir = os.path.join(submodules_dir, 'parsec-benchmark')
@@ -47,6 +49,10 @@ sha = subprocess.check_output(['git', '-C', root_dir, 'log', '-1', '--format=%H'
 release_dir = os.path.join(this.out_dir, 'release')
 release_zip_file = os.path.join(this.release_dir, 'lkmc-{}.zip'.format(this.sha))
 github_repo_id = 'cirosantilli/linux-kernel-module-cheat'
+asm_ext = '.S'
+c_ext = '.c'
+kernel_module_ext = '.ko'
+obj_ext = '.o'
 config_file = os.path.join(data_dir, 'config')
 if os.path.exists(config_file):
     config = imp.load_source('config', config_file)
@@ -101,6 +107,11 @@ def get_argparse(default_args=None, argparse_args=None):
         '--baremetal',
         help='Use Baremetal examples instead of Linux kernel ones'
     )
+    parser.add_argument(
+        '--buildroot-build-id',
+        default=default_build_id,
+        help='Buildroot build ID. Allows you to keep multiple separate gem5 builds. Default: %(default)s'
+    )
     parser.add_argument(
         '--crosstool-ng-build-id', default=default_build_id,
         help='Crosstool-NG build ID. Allows you to keep multiple separate crosstool-NG builds. Default: %(default)s'
@@ -109,6 +120,12 @@ def get_argparse(default_args=None, argparse_args=None):
         '-g', '--gem5', default=False, action='store_true',
         help='Use gem5 instead of QEMU'
     )
+    parser.add_argument(
+        '--gem5-src',
+        help='''\
+Use the given directory as the gem5 source tree. Ignore `--gem5-worktree`.
+'''
+    )
     parser.add_argument(
         '-L', '--linux-build-id', default=default_build_id,
         help='Linux build ID. Allows you to keep multiple separate Linux builds. Default: %(default)s'
@@ -173,15 +190,14 @@ Default: the run ID (-n) if that is an integer, otherwise 0.
         '-Q', '--qemu-build-id', default=default_build_id,
         help='QEMU build ID. Allows you to keep multiple separate QEMU builds. Default: %(default)s'
     )
-    parser.add_argument(
-        '--buildroot-build-id',
-        default=default_build_id,
-        help='Buildroot build ID. Allows you to keep multiple separate gem5 builds. Default: %(default)s'
-    )
     parser.add_argument(
         '-t', '--gem5-build-type', default='opt',
         help='gem5 build type, most often used for "debug" builds. Default: %(default)s'
     )
+    parser.add_argument(
+        '-v', '--verbose', default=False, action='store_true',
+        help='Show full compilation commands when they are not shown by default.'
+    )
     if hasattr(this, 'configs'):
         defaults = this.configs.copy()
     else:
@@ -224,12 +240,39 @@ def get_stats(stat_re=None, stats_file=None):
                     ret.append(cols[1])
     return ret
 
-def get_toolchain_tool(tool):
+def get_toolchain_prefix(tool, allowed_toolchains=None):
-    global this
+    buildroot_full_prefix = os.path.join(this.host_bin_dir, this.buildroot_toolchain_prefix)
-    if this.baremetal is None:
+    buildroot_exists = os.path.exists('{}-{}'.format(buildroot_full_prefix, tool))
-        return glob.glob(os.path.join(this.host_bin_dir, '*-buildroot-*-{}'.format(tool)))[0]
+    crosstool_ng_full_prefix = os.path.join(this.crosstool_ng_bin_dir, this.crosstool_ng_toolchain_prefix)
+    crosstool_ng_exists = os.path.exists('{}-{}'.format(crosstool_ng_full_prefix, tool))
+    host_tool = '{}-{}'.format(this.ubuntu_toolchain_prefix, tool)
+    host_path = shutil.which(host_tool)
+    if host_path is not None:
+        host_exists = True
+        host_full_prefix = host_path[:-(len(tool)+1)]
     else:
-        return os.path.join(this.crosstool_ng_bin_dir, '{}-{}'.format(this.crosstool_ng_prefix, tool))
+        host_exists = False
+        host_full_prefix = None
+    known_toolchains = {
+        'crosstool-ng': (crosstool_ng_exists, crosstool_ng_full_prefix),
+        'buildroot': (buildroot_exists, buildroot_full_prefix),
+        'host': (host_exists, host_full_prefix),
+    }
+    if allowed_toolchains is None:
+        if this.baremetal is None:
+            allowed_toolchains = ['buildroot', 'crosstool-ng', 'host']
+        else:
+            allowed_toolchains = ['crosstool-ng', 'buildroot', 'host']
+    tried = []
+    for toolchain in allowed_toolchains:
+        exists, prefix = known_toolchains[toolchain]
+        tried.append('{}-{}'.format(prefix, tool))
+        if exists:
+            return prefix
+    raise Exception('Tool not found. Tried:\n' + '\n'.join(tried))
+
+def get_toolchain_tool(tool, allowed_toolchains=None):
+    return '{}-{}'.format(this.get_toolchain_prefix(tool, allowed_toolchains), tool)
 
 def github_make_request(
         authenticate=False,
@@ -273,7 +316,7 @@ def mkdir():
     os.makedirs(this.qemu_run_dir, exist_ok=True)
     os.makedirs(this.p9_dir, exist_ok=True)
 
-def print_cmd(cmd, cwd, cmd_file=None, extra_env=None, extra_paths=None):
+def print_cmd(cmd, cwd=None, cmd_file=None, extra_env=None, extra_paths=None):
     '''
     Format a command given as a list of strings so that it can
     be viewed nicely and executed by bash directly and print it to stdout.
@@ -283,7 +326,8 @@ def print_cmd(cmd, cwd, cmd_file=None, extra_env=None, extra_paths=None):
     '''
     newline_separator = ' \\\n'
     out = []
-    out.append('cd {} &&{}'.format(shlex.quote(cwd), newline_separator))
+    if cwd is not None:
+        out.append('cd {} &&{}'.format(shlex.quote(cwd), newline_separator))
     if extra_paths is not None:
         out.append('PATH="{}:${{PATH}}"'.format(':'.join(extra_paths)) + newline_separator)
     for key in extra_env:
@@ -395,7 +439,7 @@ def run_cmd(
     if 'cwd' in kwargs:
         cwd = kwargs['cwd']
     else:
-        cwd = os.getcwd()
+        cwd = None
     env = os.environ.copy()
     env.update(extra_env)
     if extra_paths is not None:
@@ -458,7 +502,9 @@ def setup(parser):
         this.armv = 7
         this.gem5_arch = 'ARM'
         this.mcpu = 'cortex-a15'
-        this.crosstool_ng_prefix = 'arm-unknown-eabi'
+        this.buildroot_toolchain_prefix = 'arm-buildroot-linux-uclibcgnueabihf'
+        this.crosstool_ng_toolchain_prefix = 'arm-unknown-eabi'
+        this.ubuntu_toolchain_prefix = 'arm-linux-gnueabihf'
         if args.gem5:
             if this.machine is None:
                 this.machine = 'VExpress_GEM5_V1'
@@ -469,7 +515,9 @@ def setup(parser):
         this.armv = 8
         this.gem5_arch = 'ARM'
         this.mcpu = 'cortex-a57'
-        this.crosstool_ng_prefix = 'aarch64-unknown-elf'
+        this.buildroot_toolchain_prefix = 'aarch64-buildroot-linux-uclibc'
+        this.crosstool_ng_toolchain_prefix = 'aarch64-unknown-elf'
+        this.ubuntu_toolchain_prefix = 'aarch64-linux-gnu'
         if args.gem5:
             if this.machine is None:
                 this.machine = 'VExpress_GEM5_V1'
@@ -477,8 +525,10 @@ def setup(parser):
             if this.machine is None:
                 this.machine = 'virt'
     elif args.arch == 'x86_64':
-        this.crosstool_ng_prefix = 'TODO'
+        this.crosstool_ng_toolchain_prefix = 'x86_64-unknown-elf'
         this.gem5_arch = 'X86'
+        this.buildroot_toolchain_prefix = 'x86_64-buildroot-linux-uclibc'
+        this.ubuntu_toolchain_prefix = 'x86_64-linux-gnu'
         if args.gem5:
             if this.machine is None:
                 this.machine = 'TODO'
@@ -490,9 +540,6 @@ def setup(parser):
     this.buildroot_download_dir = os.path.join(this.buildroot_out_dir, 'download')
     this.buildroot_config_file = os.path.join(this.buildroot_build_dir, '.config')
     this.build_dir = os.path.join(this.buildroot_build_dir, 'build')
-    this.linux_build_dir = os.path.join(this.build_dir, 'linux-custom')
-    this.linux_variant_dir = '{}.{}'.format(this.linux_build_dir, args.linux_build_id)
-    this.vmlinux = os.path.join(this.linux_variant_dir, "vmlinux")
     this.qemu_build_dir = os.path.join(this.out_dir, 'qemu', args.qemu_build_id)
     this.qemu_executable_basename = 'qemu-system-{}'.format(args.arch)
     this.qemu_executable = os.path.join(this.qemu_build_dir, '{}-softmmu'.format(args.arch), this.qemu_executable_basename)
@@ -560,15 +607,28 @@ def setup(parser):
     this.gem5_se_file = os.path.join(this.gem5_config_dir, 'example', 'se.py')
     this.gem5_fs_file = os.path.join(this.gem5_config_dir, 'example', 'fs.py')
     this.run_cmd_file = os.path.join(this.run_dir, 'run.sh')
-    if args.arch == 'arm':
-        this.linux_image = os.path.join('arch', 'arm', 'boot', 'zImage')
-    elif args.arch == 'aarch64':
-        this.linux_image = os.path.join('arch', 'arm64', 'boot', 'Image')
-    elif args.arch == 'x86_64':
-        this.linux_image = os.path.join('arch', 'x86', 'boot', 'bzImage')
-    this.linux_image = os.path.join(this.linux_variant_dir, linux_image)
 
-    # Ports.
+    # Linux
+    this.linux_buildroot_build_dir = os.path.join(this.build_dir, 'linux-custom')
+    this.linux_build_dir = os.path.join(this.out_dir, 'linux', args.linux_build_id, args.arch)
+    this.vmlinux = os.path.join(this.linux_build_dir, "vmlinux")
+    if args.arch == 'arm':
+        this.linux_arch = 'arm'
+        this.linux_image = os.path.join('arch', this.linux_arch, 'boot', 'zImage')
+    elif args.arch == 'aarch64':
+        this.linux_arch = 'arm64'
+        this.linux_image = os.path.join('arch', this.linux_arch, 'boot', 'Image')
+    elif args.arch == 'x86_64':
+        this.linux_arch = 'x86'
+        this.linux_image = os.path.join('arch', this.linux_arch, 'boot', 'bzImage')
+    this.linux_image = os.path.join(this.linux_build_dir, linux_image)
+
+    # Kernel modules.
+    this.kernel_modules_build_base_dir = os.path.join(this.out_dir, 'kernel_modules')
+    this.kernel_modules_build_dir = os.path.join(this.kernel_modules_build_base_dir, args.arch)
+    this.kernel_modules_build_host_dir = os.path.join(this.kernel_modules_build_base_dir, 'host')
+
+    # Ports
     if args.port_offset is None:
         try:
             args.port_offset = int(args.run_id)
@@ -590,9 +650,6 @@ def setup(parser):
     this.baremetal_lib_basename = 'lib'
     this.baremetal_src_dir = os.path.join(this.root_dir, 'baremetal')
     this.baremetal_src_lib_dir = os.path.join(this.baremetal_src_dir, this.baremetal_lib_basename)
-    this.c_ext = '.c'
-    this.asm_ext = '.S'
-    this.obj_ext = '.o'
     if args.gem5:
         this.simulator_name = 'gem5'
     else:

configure

@@ -82,6 +82,8 @@ build-essential \
 coreutils \
 cpio \
 expect \
+gcc-aarch64-linux-gnu \
+gcc-arm-linux-gnueabi \
 git \
 moreutils \
 rsync \
@@ -93,8 +95,6 @@ wget \
   if "$gem5"; then
     pkgs="${pkgs} \
 ccache \
-gcc-aarch64-linux-gnu \
-gcc-arm-linux-gnueabi \
 libgoogle-perftools-dev \
 protobuf-compiler \
 python-dev \

kernel_config/buildroot-aarch64

@@ -0,0 +1,98 @@
+CONFIG_SYSVIPC=y
+CONFIG_POSIX_MQUEUE=y
+CONFIG_NO_HZ_IDLE=y
+CONFIG_HIGH_RES_TIMERS=y
+CONFIG_TASKSTATS=y
+CONFIG_IKCONFIG=y
+CONFIG_IKCONFIG_PROC=y
+CONFIG_SCHED_AUTOGROUP=y
+CONFIG_BLK_DEV_INITRD=y
+CONFIG_KALLSYMS_ALL=y
+CONFIG_PROFILING=y
+CONFIG_MODULES=y
+CONFIG_MODULE_UNLOAD=y
+CONFIG_MODVERSIONS=y
+CONFIG_MODULE_SRCVERSION_ALL=y
+CONFIG_BLK_DEV_BSGLIB=y
+CONFIG_ARCH_VEXPRESS=y
+CONFIG_PCI=y
+CONFIG_PCI_HOST_GENERIC=y
+CONFIG_TRANSPARENT_HUGEPAGE=y
+CONFIG_BINFMT_MISC=y
+CONFIG_COMPAT=y
+CONFIG_NET=y
+CONFIG_PACKET=y
+CONFIG_PACKET_DIAG=y
+CONFIG_UNIX=y
+CONFIG_NET_KEY=y
+CONFIG_INET=y
+CONFIG_IP_MULTICAST=y
+CONFIG_IP_ADVANCED_ROUTER=y
+CONFIG_BRIDGE=m
+CONFIG_NET_SCHED=y
+CONFIG_VSOCKETS=y
+CONFIG_NET_9P=y
+CONFIG_NET_9P_VIRTIO=y
+CONFIG_NET_9P_DEBUG=y
+CONFIG_DEVTMPFS=y
+CONFIG_DEVTMPFS_MOUNT=y
+CONFIG_CONNECTOR=y
+CONFIG_VIRTIO_BLK=y
+CONFIG_DUMMY_IRQ=m
+CONFIG_BLK_DEV_SD=y
+CONFIG_CHR_DEV_SG=y
+CONFIG_SCSI_CONSTANTS=y
+CONFIG_SCSI_LOGGING=y
+CONFIG_SCSI_SCAN_ASYNC=y
+CONFIG_SCSI_VIRTIO=y
+CONFIG_ATA=y
+CONFIG_ATA_PIIX=y
+CONFIG_NETDEVICES=y
+CONFIG_DUMMY=y
+CONFIG_MACVLAN=y
+CONFIG_VIRTIO_NET=y
+CONFIG_NLMON=y
+CONFIG_8139CP=y
+# CONFIG_INPUT_MOUSE is not set
+CONFIG_VT_HW_CONSOLE_BINDING=y
+CONFIG_SERIAL_AMBA_PL011=y
+CONFIG_SERIAL_AMBA_PL011_CONSOLE=y
+CONFIG_VIRTIO_CONSOLE=y
+CONFIG_HW_RANDOM=y
+CONFIG_HW_RANDOM_VIRTIO=y
+CONFIG_FB=y
+CONFIG_LOGO=y
+CONFIG_RTC_CLASS=y
+CONFIG_UIO=m
+CONFIG_UIO_PDRV_GENIRQ=m
+CONFIG_UIO_DMEM_GENIRQ=m
+CONFIG_UIO_PCI_GENERIC=m
+CONFIG_VIRTIO_PCI=y
+CONFIG_VIRTIO_MMIO=y
+CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
+CONFIG_MAILBOX=y
+CONFIG_PL320_MBOX=y
+CONFIG_EXT4_FS=y
+CONFIG_OVERLAY_FS=y
+CONFIG_TMPFS=y
+CONFIG_TMPFS_POSIX_ACL=y
+CONFIG_CONFIGFS_FS=y
+CONFIG_SQUASHFS=y
+CONFIG_9P_FS=y
+CONFIG_9P_FS_POSIX_ACL=y
+CONFIG_9P_FS_SECURITY=y
+CONFIG_DYNAMIC_DEBUG=y
+CONFIG_DEBUG_INFO=y
+CONFIG_GDB_SCRIPTS=y
+CONFIG_DEBUG_KERNEL=y
+CONFIG_IRQSOFF_TRACER=y
+CONFIG_SCHED_TRACER=y
+CONFIG_HWLAT_TRACER=y
+CONFIG_FTRACE_SYSCALLS=y
+CONFIG_STACK_TRACER=y
+CONFIG_FUNCTION_PROFILER=y
+CONFIG_KGDB=y
+CONFIG_KGDB_TESTS=y
+CONFIG_KGDB_KDB=y
+CONFIG_KDB_KEYBOARD=y
+# CONFIG_STRICT_DEVMEM is not set

kernel_config/buildroot-arm

@@ -0,0 +1,180 @@
+# CONFIG_LOCALVERSION_AUTO is not set
+CONFIG_SYSVIPC=y
+CONFIG_IKCONFIG=y
+CONFIG_IKCONFIG_PROC=y
+CONFIG_LOG_BUF_SHIFT=14
+CONFIG_CGROUPS=y
+CONFIG_CPUSETS=y
+# CONFIG_UTS_NS is not set
+# CONFIG_IPC_NS is not set
+# CONFIG_PID_NS is not set
+# CONFIG_NET_NS is not set
+CONFIG_BLK_DEV_INITRD=y
+CONFIG_KALLSYMS_ALL=y
+CONFIG_PROFILING=y
+CONFIG_OPROFILE=y
+CONFIG_MODULES=y
+CONFIG_MODULE_UNLOAD=y
+CONFIG_MODVERSIONS=y
+CONFIG_MODULE_SRCVERSION_ALL=y
+# CONFIG_LBDAF is not set
+# CONFIG_BLK_DEV_BSG is not set
+# CONFIG_IOSCHED_DEADLINE is not set
+# CONFIG_IOSCHED_CFQ is not set
+CONFIG_ARCH_VEXPRESS=y
+CONFIG_ARCH_VEXPRESS_DCSCB=y
+CONFIG_ARCH_VEXPRESS_TC2_PM=y
+CONFIG_PCI=y
+CONFIG_PCI_HOST_GENERIC=y
+CONFIG_SMP=y
+CONFIG_HAVE_ARM_ARCH_TIMER=y
+CONFIG_MCPM=y
+CONFIG_VMSPLIT_2G=y
+CONFIG_NR_CPUS=8
+CONFIG_ARM_PSCI=y
+CONFIG_CMA=y
+CONFIG_ZBOOT_ROM_TEXT=0x0
+CONFIG_ZBOOT_ROM_BSS=0x0
+CONFIG_CMDLINE="console=ttyAMA0"
+CONFIG_CPU_IDLE=y
+CONFIG_VFP=y
+CONFIG_NEON=y
+# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
+CONFIG_NET=y
+CONFIG_PACKET=y
+CONFIG_UNIX=y
+CONFIG_INET=y
+CONFIG_IP_PNP=y
+CONFIG_IP_PNP_DHCP=y
+CONFIG_IP_PNP_BOOTP=y
+# CONFIG_IPV6 is not set
+# CONFIG_WIRELESS is not set
+CONFIG_NET_9P=y
+CONFIG_NET_9P_VIRTIO=y
+CONFIG_NET_9P_DEBUG=y
+CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
+CONFIG_DEVTMPFS=y
+CONFIG_DEVTMPFS_MOUNT=y
+CONFIG_CONNECTOR=y
+CONFIG_MTD=y
+CONFIG_MTD_CMDLINE_PARTS=y
+CONFIG_MTD_BLOCK=y
+CONFIG_MTD_CFI=y
+CONFIG_MTD_CFI_INTELEXT=y
+CONFIG_MTD_CFI_AMDSTD=y
+CONFIG_MTD_PHYSMAP=y
+CONFIG_MTD_PHYSMAP_OF=y
+CONFIG_MTD_PLATRAM=y
+CONFIG_MTD_UBI=y
+CONFIG_VIRTIO_BLK=y
+CONFIG_DUMMY_IRQ=m
+# CONFIG_SCSI_PROC_FS is not set
+CONFIG_BLK_DEV_SD=y
+CONFIG_SCSI_VIRTIO=y
+CONFIG_ATA=y
+# CONFIG_SATA_PMP is not set
+CONFIG_ATA_PIIX=y
+CONFIG_NETDEVICES=y
+CONFIG_VIRTIO_NET=y
+CONFIG_8139CP=y
+CONFIG_SMC91X=y
+CONFIG_SMSC911X=y
+# CONFIG_WLAN is not set
+CONFIG_INPUT_EVDEV=y
+# CONFIG_INPUT_MOUSE is not set
+# CONFIG_SERIO_SERPORT is not set
+CONFIG_SERIO_AMBAKMI=y
+CONFIG_LEGACY_PTY_COUNT=16
+CONFIG_SERIAL_AMBA_PL011=y
+CONFIG_SERIAL_AMBA_PL011_CONSOLE=y
+CONFIG_VIRTIO_CONSOLE=y
+CONFIG_HW_RANDOM=y
+CONFIG_HW_RANDOM_VIRTIO=y
+CONFIG_I2C=y
+CONFIG_I2C_VERSATILE=y
+CONFIG_SENSORS_VEXPRESS=y
+CONFIG_REGULATOR_VEXPRESS=y
+CONFIG_FB=y
+CONFIG_FB_ARMCLCD=y
+CONFIG_FRAMEBUFFER_CONSOLE=y
+CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y
+CONFIG_LOGO=y
+# CONFIG_LOGO_LINUX_MONO is not set
+# CONFIG_LOGO_LINUX_VGA16 is not set
+CONFIG_SOUND=y
+CONFIG_SND=y
+# CONFIG_SND_DRIVERS is not set
+CONFIG_SND_ARMAACI=y
+CONFIG_HID_DRAGONRISE=y
+CONFIG_HID_GYRATION=y
+CONFIG_HID_TWINHAN=y
+CONFIG_HID_NTRIG=y
+CONFIG_HID_PANTHERLORD=y
+CONFIG_HID_PETALYNX=y
+CONFIG_HID_SAMSUNG=y
+CONFIG_HID_SONY=y
+CONFIG_HID_SUNPLUS=y
+CONFIG_HID_GREENASIA=y
+CONFIG_HID_SMARTJOYPLUS=y
+CONFIG_HID_TOPSEED=y
+CONFIG_HID_THRUSTMASTER=y
+CONFIG_HID_ZEROPLUS=y
+CONFIG_USB=y
+CONFIG_USB_ANNOUNCE_NEW_DEVICES=y
+CONFIG_USB_MON=y
+CONFIG_USB_STORAGE=y
+CONFIG_USB_ISP1760=y
+CONFIG_MMC=y
+CONFIG_MMC_ARMMMCI=y
+CONFIG_NEW_LEDS=y
+CONFIG_LEDS_CLASS=y
+CONFIG_LEDS_GPIO=y
+CONFIG_LEDS_TRIGGERS=y
+CONFIG_LEDS_TRIGGER_HEARTBEAT=y
+CONFIG_LEDS_TRIGGER_CPU=y
+CONFIG_RTC_CLASS=y
+CONFIG_RTC_DRV_PL031=y
+CONFIG_UIO=m
+CONFIG_UIO_PDRV_GENIRQ=m
+CONFIG_UIO_DMEM_GENIRQ=m
+CONFIG_UIO_PCI_GENERIC=m
+CONFIG_VIRTIO_PCI=y
+CONFIG_VIRTIO_BALLOON=y
+CONFIG_VIRTIO_MMIO=y
+CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
+CONFIG_EXT2_FS=y
+CONFIG_EXT3_FS=y
+CONFIG_OVERLAY_FS=y
+CONFIG_VFAT_FS=y
+CONFIG_TMPFS=y
+CONFIG_CONFIGFS_FS=y
+CONFIG_JFFS2_FS=y
+CONFIG_UBIFS_FS=y
+CONFIG_CRAMFS=y
+CONFIG_SQUASHFS=y
+CONFIG_SQUASHFS_LZO=y
+CONFIG_NFS_FS=y
+CONFIG_ROOT_NFS=y
+CONFIG_9P_FS=y
+CONFIG_9P_FS_POSIX_ACL=y
+CONFIG_9P_FS_SECURITY=y
+CONFIG_NLS_CODEPAGE_437=y
+CONFIG_NLS_ISO8859_1=y
+CONFIG_DYNAMIC_DEBUG=y
+CONFIG_DEBUG_INFO=y
+CONFIG_GDB_SCRIPTS=y
+CONFIG_DEBUG_KERNEL=y
+CONFIG_DETECT_HUNG_TASK=y
+# CONFIG_SCHED_DEBUG is not set
+CONFIG_IRQSOFF_TRACER=y
+CONFIG_SCHED_TRACER=y
+CONFIG_HWLAT_TRACER=y
+CONFIG_FTRACE_SYSCALLS=y
+CONFIG_STACK_TRACER=y
+CONFIG_FUNCTION_PROFILER=y
+CONFIG_KGDB=y
+CONFIG_KGDB_TESTS=y
+CONFIG_KGDB_KDB=y
+CONFIG_KDB_KEYBOARD=y
+CONFIG_DEBUG_USER=y
+# CONFIG_CRYPTO_HW is not set

kernel_config/buildroot-x86_64

@@ -0,0 +1,93 @@
+CONFIG_SYSVIPC=y
+CONFIG_IKCONFIG=y
+CONFIG_IKCONFIG_PROC=y
+CONFIG_CGROUPS=y
+CONFIG_BLK_DEV_INITRD=y
+CONFIG_KALLSYMS_ALL=y
+CONFIG_MODULES=y
+CONFIG_MODULE_UNLOAD=y
+CONFIG_MODVERSIONS=y
+CONFIG_MODULE_SRCVERSION_ALL=y
+CONFIG_SMP=y
+CONFIG_HYPERVISOR_GUEST=y
+CONFIG_PARAVIRT=y
+CONFIG_NET=y
+CONFIG_PACKET=y
+CONFIG_UNIX=y
+CONFIG_INET=y
+# CONFIG_WIRELESS is not set
+CONFIG_NET_9P=y
+CONFIG_NET_9P_VIRTIO=y
+CONFIG_NET_9P_DEBUG=y
+CONFIG_DEVTMPFS=y
+CONFIG_DEVTMPFS_MOUNT=y
+CONFIG_CONNECTOR=y
+CONFIG_VIRTIO_BLK=y
+CONFIG_DUMMY_IRQ=m
+CONFIG_BLK_DEV_SD=y
+CONFIG_SCSI_VIRTIO=y
+CONFIG_ATA=y
+CONFIG_ATA_PIIX=y
+CONFIG_NETDEVICES=y
+CONFIG_VIRTIO_NET=y
+CONFIG_NE2K_PCI=y
+CONFIG_8139CP=y
+# CONFIG_WLAN is not set
+CONFIG_INPUT_EVDEV=y
+# CONFIG_INPUT_MOUSE is not set
+CONFIG_SERIAL_8250=y
+CONFIG_SERIAL_8250_CONSOLE=y
+CONFIG_VIRTIO_CONSOLE=y
+CONFIG_HW_RANDOM_VIRTIO=m
+CONFIG_DRM=y
+CONFIG_DRM_QXL=y
+CONFIG_DRM_BOCHS=y
+CONFIG_DRM_VIRTIO_GPU=y
+CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y
+CONFIG_LOGO=y
+CONFIG_SOUND=y
+CONFIG_SND=y
+CONFIG_SND_HDA_INTEL=y
+CONFIG_SND_HDA_GENERIC=y
+CONFIG_USB=y
+CONFIG_USB_XHCI_HCD=y
+CONFIG_USB_EHCI_HCD=y
+CONFIG_USB_UHCI_HCD=y
+CONFIG_USB_STORAGE=y
+CONFIG_UIO=m
+CONFIG_UIO_PDRV_GENIRQ=m
+CONFIG_UIO_DMEM_GENIRQ=m
+CONFIG_UIO_PCI_GENERIC=m
+CONFIG_VIRTIO_PCI=y
+CONFIG_VIRTIO_BALLOON=y
+CONFIG_VIRTIO_INPUT=y
+CONFIG_VIRTIO_MMIO=y
+CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
+CONFIG_EXT4_FS=y
+CONFIG_AUTOFS4_FS=y
+CONFIG_OVERLAY_FS=y
+CONFIG_TMPFS=y
+CONFIG_TMPFS_POSIX_ACL=y
+CONFIG_CONFIGFS_FS=y
+CONFIG_SQUASHFS=y
+CONFIG_9P_FS=y
+CONFIG_9P_FS_POSIX_ACL=y
+CONFIG_9P_FS_SECURITY=y
+CONFIG_DYNAMIC_DEBUG=y
+CONFIG_DEBUG_INFO=y
+CONFIG_GDB_SCRIPTS=y
+CONFIG_DEBUG_KERNEL=y
+CONFIG_IRQSOFF_TRACER=y
+CONFIG_SCHED_TRACER=y
+CONFIG_HWLAT_TRACER=y
+CONFIG_FTRACE_SYSCALLS=y
+CONFIG_STACK_TRACER=y
+CONFIG_FUNCTION_PROFILER=y
+CONFIG_KGDB=y
+CONFIG_KGDB_TESTS=y
+CONFIG_KGDB_LOW_LEVEL_TRAP=y
+CONFIG_KGDB_KDB=y
+CONFIG_KDB_KEYBOARD=y
+# CONFIG_STRICT_DEVMEM is not set
+CONFIG_X86_PTDUMP=y
+CONFIG_UNWINDER_FRAME_POINTER=y

kernel_config/default

@@ -2,7 +2,7 @@
 CONFIG_BLK_DEV_INITRD=y
 CONFIG_DYNAMIC_DEBUG=y
 CONFIG_MODULE_SRCVERSION_ALL=y
-CONFIG_STRICT_DEVMEM=n
+# CONFIG_STRICT_DEVMEM is not set
 
 # Filesystems.
 CONFIG_DEBUG_FS=y
@@ -24,10 +24,10 @@ CONFIG_KGDB_KDB=y
 CONFIG_KGDB_LOW_LEVEL_TRAP=y
 CONFIG_KGDB_SERIAL_CONSOLE=y
 CONFIG_KGDB_TESTS=y
-CONFIG_KGDB_TESTS_ON_BOOT=n
+# CONFIG_KGDB_TESTS_ON_BOOT is not set
 CONFIG_MAGIC_SYSRQ=y
 CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1
-CONFIG_SERIAL_KGDB_NMI=n
+# CONFIG_SERIAL_KGDB_NMI is not set
 
 # Non-static variables show up on /proc/kallsyms
 # https://stackoverflow.com/questions/20196636/does-kallsyms-have-all-the-symbol-of-kernel-functions/44614878#44614878
@@ -89,10 +89,6 @@ CONFIG_LOGO=y
 
 ## Networking
 
-# Will everything blow up?
-# https://superuser.com/questions/684005/how-does-one-permanently-disable-gnu-linux-networking/1255015#1255015
-#CONFIG_NET=n
-
 # If given, we can use QEMU 2.9.0 default x86 networking without any -net or -netdev options,
 # since E1000 is the default networking device as mentioned at:
 # https://en.wikibooks.org/w/index.php?title=QEMU/Networking&oldid=3268753
@@ -121,7 +117,6 @@ CONFIG_UIO_PCI_GENERIC=m
 
 ## ARM
 
-# Like CONFIG_X86_PTDUMP for ARM.
 CONFIG_ARM64_PTDUMP=y
 
 # For record and replay.

packages/kernel_modules/Makefile

@@ -1,10 +1,7 @@
-obj-m += $(addsuffix .o, $(notdir $(basename $(filter-out %.mod.c, $(wildcard $(BR2_EXTERNAL_KERNEL_MODULES_PATH)/*.c)))))
+obj-m += $(OBJECT_FILES)
 ccflags-y := -DDEBUG -g -std=gnu99 -Werror -Wno-declaration-after-statement -Wframe-larger-than=1000000000
 
-.PHONY: all clean
+.PHONY: all
 
 all:
-	$(MAKE) -C '/lib/modules/$(shell uname -r)/build' M='$(PWD)' modules
+	$(MAKE) -C '$(LINUX_DIR)' M='$(M)'
-
-clean:
-	$(MAKE) -C '$(LINUX_DIR)' M='$(PWD)' clean

packages/kernel_modules/make-host.sh

@@ -1,24 +0,0 @@
-#!/usr/bin/env bash
-
-# We can almost do everything from the Makefile itself by using default values for
-#
-#    LINUX_DIR ?= "/lib/modules/$(uname -r)/build"
-#    BR2_EXTERNAL_KERNEL_MODULES_PATH="$(pwd)"
-#
-# The problem with that is that if you define those variables in your environment,
-# the build breaks, so this is more portable.
-#
-# Trying to add `-i` to overcome incompatible modules will fail,
-# because any build failure prevents the generation of all `.mod.c` files.
-
-j="$(($(nproc) - 2))"
-while getopts j: OPT; do
-  case "$OPT" in
-    'j')
-      j="$OPTARG"
-    ;;
-  esac
-done
-shift $(($OPTIND - 1))
-make -j "$j" KERNEL_MODULES_PATH="$(pwd)" LINUX_DIR="/lib/modules/$(uname -r)/build" "$@"
-make -C user/ -j "$j" "$@"

packages/kernel_modules/user/Makefile

@@ -1,12 +1,13 @@
-.PHONY: all clean
+.PHONY: all clean mkdir
 
 CFLAGS_EXTRA = -fopenmp -std=c99
 CXXFLAGS_EXTRA = -std=c++17
 CCFLAGS_EXTRA = -Wall -Werror -Wextra
 IN_EXT_C = .c
 IN_EXT_CXX = .cpp
-LIBS =
+LIBS = -lm
 OUT_EXT = .out
+OUT_DIR = .
 
 OUTS := $(foreach IN_EXT,$(IN_EXT_C) $(IN_EXT_CXX),$(addsuffix $(OUT_EXT), $(basename $(wildcard *$(IN_EXT)))))
 ifeq ($(BR2_PACKAGE_EIGEN),y)
@@ -30,14 +31,18 @@ ifeq ($(BR2_PACKAGE_OPENBLAS),y)
 else
 	OUTS := $(filter-out openblas$(OUT_EXT),$(OUTS))
 endif
+OUTS := $(addprefix $(OUT_DIR)/,$(OUTS))
 
-all: $(OUTS)
+all: mkdir $(OUTS)
 
-%$(OUT_EXT): %$(IN_EXT_C)
+$(OUT_DIR)/%$(OUT_EXT): %$(IN_EXT_C)
 	$(CC) $(CFLAGS) $(CFLAGS_EXTRA) -o '$@' '$<' $(LIBS)
 
-%$(OUT_EXT): %$(IN_EXT_CXX)
+$(OUT_DIR)/%$(OUT_EXT): %$(IN_EXT_CXX)
 	$(CXX) $(CXXFLAGS) $(CXXFLAGS_EXTRA) -o '$@' '$<' $(LIBS)
 
 clean:
 	rm -f *'$(OUT_EXT)'
+
+mkdir:
+	mkdir -p '$(OUT_DIR)'

rootfs-post-build-script

@@ -4,8 +4,9 @@
 # which gets called by the default target.
 target_dir="$1"
 mkdir -p \
-  "${target_dir}/mnt/9p" \
+  "${target_dir}/mnt/9p/data" \
-  "${target_dir}/mnt/out" \
+  "${target_dir}/mnt/9p/out" \
+  "${target_dir}/mnt/9p/rootfs_overlay" \
 ;
 # Maybe there is a cleaner way to get rid of those files,
 # like disabling some Buildroot packages, but no patience.

rootfs_overlay/etc/fstab

@@ -1,11 +1,12 @@
-# <file system>	<mount pt>	<type>	<options>	<dump>	<pass>
+# <file system>		<mount pt>				<type>	<options>						<dump>	<pass>
-/dev/root		/		ext2	rw,noauto	0	1
+/dev/root			/						ext2	rw,noauto						0		1
-proc			/proc		proc	defaults	0	0
+proc				/proc					proc	defaults						0		0
-devpts			/dev/pts	devpts	defaults,gid=5,mode=620	0	0
+devpts				/dev/pts				devpts	defaults,gid=5,mode=620			0		0
-tmpfs			/dev/shm	tmpfs	mode=0777	0	0
+tmpfs				/dev/shm				tmpfs	mode=0777						0		0
-tmpfs			/tmp		tmpfs	mode=1777	0	0
+tmpfs				/tmp					tmpfs	mode=1777						0		0
-tmpfs			/run		tmpfs	mode=0755,nosuid,nodev	0	0
+tmpfs				/run					tmpfs	mode=0755,nosuid,nodev			0		0
-sysfs			/sys		sysfs	defaults	0	0
+sysfs				/sys					sysfs	defaults						0		0
-debugfs			/sys/kernel/debug	debugfs	defaults	0	0
+debugfs				/sys/kernel/debug		debugfs	defaults						0		0
-host_scratch	/mnt/9p				9p		trans=virtio,version=9p2000.L	0	0
+host_data			/mnt/9p/data			9p		trans=virtio,version=9p2000.L	0		0
-host_out		/mnt/out			9p		trans=virtio,version=9p2000.L	0	0
+host_out			/mnt/9p/out				9p		trans=virtio,version=9p2000.L	0		0
+host_rootfs_overlay	/mnt/9p/rootfs_overlay	9p		trans=virtio,version=9p2000.L	0		0

run

@@ -217,8 +217,9 @@ def main(args, extra_args=None):
                 '-no-reboot',
                 '-smp', str(args.cpus),
                 '-trace', 'enable={},file={}'.format(trace_type, common.qemu_trace_file),
-                '-virtfs', 'local,path={},mount_tag=host_scratch,security_model=mapped,id=host_scratch'.format(common.p9_dir),
+                '-virtfs', 'local,path={},mount_tag=host_data,security_model=mapped,id=host_data'.format(common.p9_dir),
-                '-virtfs', 'local,path={},mount_tag=host_out,security_model=mapped,id=host_out'.format(common.build_dir),
+                '-virtfs', 'local,path={},mount_tag=host_out,security_model=mapped,id=host_out'.format(common.out_dir),
+                '-virtfs', 'local,path={},mount_tag=host_rootfs_overlay,security_model=mapped,id=host_rootfs_overlay'.format(common.rootfs_overlay_dir),
             ] +
             serial_monitor +
             vnc

run-gdb

@@ -44,15 +44,17 @@ def main(args, extra_args=None):
         break_at = []
     if args.baremetal is None:
         image = common.vmlinux
+        allowed_toolchains = ['buildroot', 'crosstool-ng', 'host']
     else:
         image = common.image
+        allowed_toolchains = ['crosstool-ng', 'buildroot', 'host']
     cmd = (
-        [common.get_toolchain_tool('gdb')] +
+        [common.get_toolchain_tool('gdb', allowed_toolchains=allowed_toolchains)] +
         before +
         ['-q']
     )
     if args.baremetal is None:
-        cmd.extend(['-ex', 'add-auto-load-safe-path {}'.format(common.linux_variant_dir)])
+        cmd.extend(['-ex', 'add-auto-load-safe-path {}'.format(common.linux_build_dir)])
     if args.sim:
         target = 'sim'
     else:
@@ -85,7 +87,7 @@ def main(args, extra_args=None):
         # which gets put on the kernel build root when python debugging scripts are enabled.
         cmd.extend(['-ex', 'continue'] + lx_symbols)
     cmd.extend(after)
-    return common.run_cmd(cmd, cmd_file=os.path.join(common.run_dir, 'run-gdb.sh'), cwd=common.linux_variant_dir)
+    return common.run_cmd(cmd, cmd_file=os.path.join(common.run_dir, 'run-gdb.sh'), cwd=common.linux_build_dir)
 
 if __name__ == '__main__':
     parser = common.get_argparse(argparse_args={'description': 'Connect with GDB to an emulator to debug Linux itself'})

run-toolchain

@@ -36,10 +36,15 @@ parser.add_argument(
     nargs='*'
 )
 args = common.setup(parser)
+if args.baremetal is None:
+    image = common.vmlinux
+else:
+    image = common.image
+tool= common.get_toolchain_tool(args.tool, allowed_toolchains=allowed_toolchains)
 if args.dry:
-    print(common.get_toolchain_tool(args.tool))
+    print(tool)
 else:
     sys.exit(common.run_cmd(
-        [common.get_toolchain_tool(args.tool)] + args.extra_args,
+        [tool] + args.extra_args,
         cmd_file=os.path.join(common.run_dir, 'run-toolchain.sh'),
     ))

test

@@ -1,4 +1,4 @@
 #!/usr/bin/env bash
 set -eu
-./bench-boot -t "${1:-1}"
+./bench-boot --size "${1:-1}"
 ./test-kernel-modules

update-buildroot-kernel-configs

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+set -eux
+root_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)"
+getvar="${root_dir}/getvar"
+for arch in x86_64 arm aarch64; do
+  linux_arch="$("${getvar}" --arch "$arch" linux_arch)"
+  linux_buildroot_build_dir="$("${getvar}" --arch "$arch" linux_buildroot_build_dir)"
+  linux_config_dir="$("${getvar}" --arch "$arch" linux_config_dir)"
+  "${root_dir}/build-buildroot" --baseline --build-linux --no-all -- linux-configure
+  cd "$linux_build_dir"
+  make ARCH="$linux_arch" savedefconfig
+  cp defconfig "${linux_config_dir}/buildroot-${arch}"
+done