linux: introduce build variants selectable at runtime

README.adoc

@@ -6384,19 +6384,56 @@ If none of those methods are flexible enough for you, create a new package as fo
 +
 if you make any changes to that package after the initial build: <<rebuild>>
 
-=== Build multiple versions of the same package
+=== Build variants
 
-If you are comparing two versions of have a package that takes considerable time to build, one on each branch, you may want to keep two builds around to make things even faster.
+It often happens that you are comparing two versions of the build, a good and a bad one, and trying to figure out why the bad one is bad.
 
-One simple approach is to simply use two checkouts of this repository, or to use the `-s` option:
+This section describes some techniques that can help to reduce the build time and disk usage in those situations.
+
+==== Full builds variants
+
+The most coarse thing you can do is to keep two full checkouts of this repository, possibly with `git subtree`.
+
+This approach has the advantage of being simple and robust, but it wastes a lot of space and time for the full rebuild, since <<ccache>> does not make compilation instantaneous due to configuration file reading.
+
+The next less coarse approach, is to use the `-s` option:
 
 ....
 ./build -s mybranch
 ....
 
-While <<ccache>> does help to speed up the full rebuild, it still does not necessarily make compilation instantaneous due to configuration file reading.
+which generates a full new build under `out/` named for example as `out/x86_64-mybranch`, but at least avoids copying up the source.
 
-One alternative is to do:
+TODO: only `-s` works for `./build`, e.g. if you want to `./run` afterwards you need to manually `mv` build around. This should be easy to patch however.
+
+==== Linux kernel build variants
+
+Since the Linux kernel is so important to us, we have created a convenient dedicated mechanism for it.
+
+For example, if you want to keep two builds around, one for the latest Linux version, and the other for Linux `v4.16`:
+
+....
+./build
+git -C linux checkout v4.16
+./build -L v4.16
+git -C linux checkout -
+./run
+./run -L v4.16
+....
+
+The `-L` option should be passed to all scripts that support it, much like `-a` for the <<cpu-architecture>>, e.g. to step debug:
+
+.....
+./rungdb -L v4.16
+.....
+
+It should also be easy to extend this method to all `-custom` `_OVERRIDE_SRCDIR` packages, which luckily tend to be the ones that we care the most about.
+
+This technique is implemented semi-hackishly by moving symlinks around inside the Buildroot build dir at build time, and selecting the right build directory at runtime.
+
+==== Generic package build variants
+
+This hack-ish technique allows us to rebuild just one package at a time:
 
 ....
 ./build KERNEL_MODULE_VERSION=mybranch
@@ -6408,6 +6445,8 @@ and now you can see that a new version of `kernel_module` was built and put insi
 ls out/x86_64/buildroot/build/kernel_module-mybranch
 ....
 
+Unfortunately we don't have a nice runtime selection with `./run` implemented currently, you have to manually move packages around.
+
 TODO: is there a way to do it nicely for `*_OVERRIDE_SRCDIR` packages from link:buildroot_override[]? I tried:
 
 ....
@@ -6426,7 +6465,7 @@ and theI tried:
 ./build -l LINUX_VERSION=mybranch LINUX_SITE="$(pwd)/linux"
 ....
 
-but it feels hack-ish, and the build was slower than normal, looks like the build was single threaded?
+but it feels hackish, and the build was slower than normal, looks like the build was single threaded?
 
 === BR2_TARGET_ROOTFS_EXT2_SIZE
 

build

@@ -20,7 +20,7 @@ post_script_args=
 qemu_sdl='--enable-sdl --with-sdlabi=2.0'
 suffix=
 v=0
-while getopts 'a:B:b:C:c:fGgj:hIiK:klp:qSs:v' OPT; do
+while getopts 'a:B:b:C:c:fGgj:hIiK:kL:lp:qSs:v' OPT; do
   case "$OPT" in
     a)
       arch="$OPTARG"
@@ -73,6 +73,9 @@ BR2_TARGET_ROOTFS_INITRAMFS=n
       extra_make_args="${extra_make_args} kernel_module-reconfigure \\
 "
       ;;
+    L)
+      common_linux_variant="$OPTARG"
+      ;;
     l)
       linux_reconfigure=true
       extra_make_args="${extra_make_args} linux-reconfigure \\
@@ -101,7 +104,7 @@ BR2_TARGET_ROOTFS_INITRAMFS=n
 done
 shift $(($OPTIND - 1))
 extra_make_args="${extra_make_args} $@"
-set_common_vars "$arch" "$gem5" "$suffix"
+set_common_vars -L "$common_linux_variant" "$arch" "$gem5" "$suffix"
 config_file="${buildroot_out_dir}/.config"
 case "$arch" in
   x86_64)
@@ -167,6 +170,14 @@ BR2_ROOTFS_POST_SCRIPT_ARGS=\"${post_script_args}\"
 fi
 
 common_mkdir
+if [ -h "$common_linux_custom_dir" ]; then
+  rm "$common_linux_custom_dir"
+elif [ -d "$common_linux_custom_dir" ]; then
+  # Migration for existing builds.
+  mv "$common_linux_custom_dir" "$common_linux_variant_dir"
+fi
+mkdir -p "$common_linux_variant_dir"
+ln -s "$common_linux_variant_dir" "$common_linux_custom_dir"
 cd "$buildroot_dir"
 # HOST_QEMU_OPTS is a hack that happens to work because the QEMU package luckly uses += at all times.
 # It shouldn't be necessary in the first place: https://bugs.busybox.net/show_bug.cgi?id=9936

build-usage.adoc

@@ -26,10 +26,13 @@
                              Mnemonic: `fast`.
 |`-g` |                     |Enable gem5 build or force its rebuild.
 |`-h` |                     |Show this help message.
+|`-L` |                     |Linux kernel build variant.
 |`-I` |                     |Enable initramfs for the current build.
 |`-i` |                     |Enable initrd for the current build.
 |`-K` |`KERNEL_CONFIG_FILE` |Use `KERNEL_CONFIG_FILE` as the exact Linux kernel
-                             configuration. Ignore the default kernel config fragments.
+                             configuration. Ignore the default kernel config fragments,
+                             but still add options explicitly passed with `-C` and `-c`.
+                             on top of it.
 |`-p` |                     |Pass extra arguments to the `rootfs_post_build_script`.
 |`-S` |                     |Don't build QEMU with SDL support.
                              Graphics such as X11 won't work, only the terminal.

common

@@ -20,6 +20,17 @@ common_bench_cmd() (
   printf "exit_status $?\n" >> "$results_file"
 )
 set_common_vars() {
+  linux_variant=
+  OPTIND=1
+  while getopts L: OPT; do
+    case "$OPT" in
+      L)
+        linux_variant="$OPTARG"
+        ;;
+    esac
+  done
+  echo $OPTIND
+  shift "$(($OPTIND - 1))"
   arch="$1"
   gem5="${2:-false}"
   case "$arch" in
@@ -61,6 +72,24 @@ set_common_vars() {
     common_out_run_dir="$qemu_out_dir"
     common_trace_txt_file="${common_out_run_dir}/trace.txt"
   fi
+  common_linux_custom_dir="${build_dir}/linux-custom"
+  common_linux_variant_dir="${common_linux_custom_dir}.${linux_variant}"
+  common_vmlinux="${common_linux_variant_dir}/vmlinux"
+  case "$arch" in
+    arm)
+      common_linux_image=arch/arm/boot/zImage
+      ;;
+    aarch64)
+      common_linux_image=arch/arm64/boot/Image
+      ;;
+    mips64)
+      common_linux_image=vmlinux
+      ;;
+    x86_64)
+      common_linux_image=arch/x86/boot/bzImage
+      ;;
+  esac
+  common_linux_image="${common_linux_variant_dir}/${common_linux_image}"
 }
 common_mkdir() (
   mkdir -p \
@@ -70,6 +99,7 @@ common_mkdir() (
     "$p9_dir" \
   ;
 )
+common_linux_variant=default
 root_dir="$(pwd)"
 out_dir="${root_dir}/out"
 common_bench_boot="${out_dir}/bench-boot.txt"

run

@@ -34,7 +34,7 @@ trace_enabled=false
 # just to prevent QEMU from emitting a warning that '' is not valid.
 trace_type=pr_manager_run
 vnc=
-while getopts a:c:DdE:e:F:f:G:ghIiKkm:PT:U:uVx OPT; do
+while getopts a:c:DdE:e:F:f:G:ghIiKkL:m:PT:U:uVx OPT; do
   case "$OPT" in
     a)
       arch="$OPTARG"
@@ -91,6 +91,9 @@ while getopts a:c:DdE:e:F:f:G:ghIiKkm:PT:U:uVx OPT; do
 "
       kgdb=true
       ;;
+    L)
+      common_linux_variant="$OPTARG"
+      ;;
     m)
       memory="$OPTARG"
       ;;
@@ -121,7 +124,7 @@ while getopts a:c:DdE:e:F:f:G:ghIiKkm:PT:U:uVx OPT; do
 done
 shift "$(($OPTIND - 1))"
 extra_flags="$extra_flags $@"
-set_common_vars "$arch" "$gem5"
+set_common_vars -L "$common_linux_variant" "$arch" "$gem5"
 images_dir="${buildroot_out_dir}/images"
 if "$debug" && "$kvm"; then
   echo 'error: -d and -K are incompatible' 1>&2
@@ -175,7 +178,7 @@ ${gem5opts} \
 -d '${m5out_dir}' \\
 '${gem5_src_dir}/configs/example/fs.py' \\
 --disk-image='${images_dir}/rootfs.ext2' \\
---kernel='${buildroot_out_dir}/build/linux-custom/vmlinux' \\
+--kernel='${common_vmlinux}' \\
 --mem-size='${memory}' \\
 --num-cpus='${cpus}' \\
 --script='${readfile_file}' \\
@@ -221,6 +224,7 @@ ${debug_vm} \
 ${qemu_executable} \\
 -device rtl8139,netdev=net0 \\
 -gdb tcp::1234 \\
+-kernel '${common_linux_image}' \\
 -m '${memory}' \\
 -monitor telnet::45454,server,nowait \\
 -netdev user,hostfwd=tcp::45455-:45455,hostfwd=tcp::45456-:22,id=net0 \\
@@ -261,7 +265,6 @@ ${qemu_common} \
 -append '${root} nopat ${extra_append}' \\
 -device edu \\
 ${custom_devices} \\
--kernel '${images_dir}/bzImage' \\
 ${extra_flags} \
 "
     ;;
@@ -275,7 +278,6 @@ ${qemu_common} \
 -append '${root} ${extra_append}' \\
 -cpu cortex-a15 \\
 -device virtio-gpu-pci \\
--kernel '${images_dir}/zImage' \\
 ${extra_flags} \
 "
     ;;
@@ -304,7 +306,7 @@ ${qemu_common} \
 -M malta \\
 -append '${root} ${extra_append}' \\
 -cpu I6400 \\
--kernel '${images_dir}/vmlinux' \\
+-kernel '${common_vmlinux}' \\
 ${extra_flags} \
 "
     ;;
@@ -314,7 +316,7 @@ if "$tmux"; then
   if "$gem5"; then
     eval "./tmu 'sleep 2;./gem5-shell ${tmux_args};'"
   elif  "$debug"; then
-    eval "./tmu ./rungdb -a '${arch}' ${tmux_args}"
+    eval "./tmu ./rungdb -a '${arch} -L ${common_linux_variant}' ${tmux_args}"
   fi
 fi
 "${root_dir}/eeval" "$cmd" "${common_out_run_dir}/run.sh"

run-usage.adoc

@@ -36,6 +36,7 @@
 |`-i` |              |Run with initrd.
 |`-K` |              |Use KVM. Only works if guest arch == host arch.
 |`-k` |              |Enable KGDB.
+|`-L` |              |Linux kernel build variant.
 |`-m` |              |Set the memory size of the guest. E.g.: `-m 512M`. Default: `256M`.
                       The default is the minimum amount that boots all archs without extra
                       options added. Anything lower will lead some arch to fail to boot.

rungdb

@@ -7,7 +7,7 @@ before=
 lx_symbols="-ex 'lx-symbols ../kernel_module-1.0/' \\
 "
 kgdb=false
-while getopts A:a:b:gkL OPT; do
+while getopts A:a:b:gkL:X OPT; do
   case "$OPT" in
     A)
       after="$OPTARG"
@@ -25,6 +25,9 @@ while getopts A:a:b:gkL OPT; do
       kgdb=true
       ;;
     L)
+      common_linux_variant="$OPTARG"
+      ;;
+    X)
       lx_symbols=
       ;;
     ?)
@@ -45,7 +48,7 @@ if "$gem5"; then
 else
   port=1234
 fi
-set_common_vars "$arch" "$gem5"
+set_common_vars -L "$common_linux_variant" "$arch" "$gem5"
 gdb="${host_dir}/usr/bin/${arch}-linux-gdb \\
 ${before}"
 if "$kgdb"; then
@@ -87,5 +90,5 @@ ${brk} \
 ${lx_symbols} \
 "
 fi
-"${root_dir}/eeval" "cd '${build_dir}/linux-custom/' && \\
+"${root_dir}/eeval" "cd '${common_linux_variant_dir}' && \\
 $cmd $after" "${common_out_run_dir}/rungdb.sh"

rungdb-user

@@ -35,11 +35,11 @@ set_common_vars "$arch" "$gem5"
 executable="${build_dir}/${executable_rel}"
 addr="$("${root_dir}/runtc" readelf -h "$executable" | awk '/Entry/{ print $NF }' )"
 ex="-ex \"add-symbol-file $executable $addr\""
-# -L or else lx-symbols throws for arm:
+# -X or else lx-symbols throws for arm:
 # gdb.MemoryError: Cannot access memory at address 0xbf0040cc
 # TODO understand better.
 #
 # Also, lx-symbols overrides the add-symbol-file commands.
-cmd="./rungdb -a '${arch}' -b '${ex}' ${gem5_opt} -L ${brk}"
+cmd="./rungdb -a '${arch}' -b '${ex}' ${gem5_opt} -X ${brk}"
 echo "$cmd"
 eval "$cmd"