gem5: track build and run variants separately with -M and -N

Otherwise, checking out branches is too insane, as it does not
update the worktrees, even though the gem5/gem5 module was updated.

gem5: expose build types, document debug builds.

simultaneous runs: store stdout and stderr on a file to allow running
all from a single terminal on the background cleanly.

README.adoc

@@ -1154,8 +1154,8 @@ The default run id is `0`.
 This method also allows us to keep run outputs in separate directories for later inspection, e.g.:
 
 ....
-./run -aA -g -n 0
+./run -aA -g -n 0 &>/dev/null &
-./run -aA -g -n 1
+./run -aA -g -n 1 &>/dev/null &
 ....
 
 produces two separate `m5out` directories:
@@ -1165,6 +1165,13 @@ less out/aarch64/gem5/default/0/m5out
 less out/aarch64/gem5/default/1/m5out
 ....
 
+and the gem5 host executable stdout and stderr can be found at:
+
+....
+less out/aarch64/gem5/default/0/termout.txt
+less out/aarch64/gem5/default/1/termout.txt
+....
+
 You can also add a prefix to the build ID before a period:
 
 ....
@@ -1194,6 +1201,8 @@ Like <<cpu-architecture>>, you will need to pass the `-n` option to anything tha
 ./rungdb -n 1
 ....
 
+To run multiple gem5 checkouts, see: <<gem5-simultaneous-runs-with-build-variants>>.
+
 Implementation note: we create multiple namespaces for two things:
 
 * run output directory
@@ -5108,7 +5117,7 @@ gem5 full system:
 
 ....
 printf 'm5 exit' > data/readfile
-./run -aa -g -F 'm5 checkpoint;m5 readfile > a.sh;sh a.sh'
+./run -aa -g -F '/gem5.sh'
 printf 'm5 resetstats;dhrystone 100000;m5 exit' > data/readfile
 time ./run -aa -gu -- -r 1
 ....
@@ -5517,7 +5526,7 @@ arch=aarch64
 # Generate a checkpoint after Linux boots.
 # The boot takes a while, be patient young Padawan.
 printf 'm5 exit' > data/readfile
-./run -a "$arch" -g -F '/5 checkpoint;m5 readfile > a.sh;sh a.sh'
+./run -a "$arch" -g -F '/gem5.sh'
 
 # Restore the checkpoint, and run the benchmark with parameter 1.000.
 # We skip the boot completely, saving time!
@@ -6172,7 +6181,7 @@ This integer value is just pure `fs.py` sugar, the backend at `m5.instantiate` j
 
 You want to automate running several tests from a single pristine post-boot state.
 
-The problem is that after the checkpoint, the memory and disk states are fixed, so you can't for example:
+The problem is that boot takes forever, and after the checkpoint, the memory and disk states are fixed, so you can't for example:
 
 * hack up an existing rc script, since the disk is fixed
 * inject new kernel boot command line options, since those have already been put into memory by the bootloader
@@ -6188,10 +6197,12 @@ printf 'echo "second benchmark";m5 exit' > data/readfile
 ./run -a aarch64 -g -- -r 1
 ....
 
-Other possibilities include:
+Since this is such a common setup, we provide helper for it at: link:rootfs_overlay/gem5.sh[rootfs_overlay/gem5.sh].
+
+Other loophole possibilities include:
 
 * <<9p>>
-* create multiple disk images, and mount the benchmark one
+* link:https://stackoverflow.com/questions/50862906/how-to-attach-multiple-disk-images-in-a-simulation-with-gem5-fs-py/51037661#51037661[create multiple disk images], and mount the benchmark from on one of them
 * `expect` as mentioned at: https://stackoverflow.com/questions/7013137/automating-telnet-session-using-bash-scripts
 +
 ....
@@ -6798,28 +6809,69 @@ Analogous to the <<linux-kernel-build-variants>> but with the `-M` option instea
 ....
 ./build -g
 git -C gem5/gem5 checkout some-branch
-./build -M some-branch -g
+./build -g -M some-branch
 git -C gem5/gem5 checkout -
 ./run -g
+git -C gem5/gem5 checkout some-branch
 ./run -M some-branch -g
 ....
 
-Since we control the gem5 build however, unlike Linux which uses Buildroot, we make it awesomer, and use `git worktree` instead of a mere `rsync` for the copy.
+Don't forget however that gem5 has Python scripts in its source code tree, and that those must match the source code of a given build.
 
-It works like this:
+Therefore, you can't forget to checkout to the sources to that of the corresponding build before running, unless you explicitly tell gem5 to use a non-default source tree with `-N`.
 
-* when you don't pass the `-M` option, which is the same as the `-M default` variant, we use the source code from under the `gem5/gem5` submodule for the build
+This becomes inevitable when you want to launch <<gem5-simultaneous-runs-with-build-variants>>.
-* otherwise, if you pass `-M some-branch`, we generate a git worktree checkout under `data/gem5/some-branch`, with branch name `wt/some-branch`.
-+
-The initial revision for that worktree is whatever `gem5/gem5` is currently points to.
-+
-However, if the worktree already exists, we leave it untouched.
-+
-Therefore, you can safely go to that directory and edit the source there without fear that it will get deleted.
-+
-The `wt/` branch name prefix stands for `WorkTree`, and is done to allow us to checkout to a test `some-branch` branch under `gem5/gem5` and still use `-M some-branch`, without aconflict for the worktree branch.
 
-All build outputs end up at: `out/common/gem5/<variant>` regardless.
+===== gem5 simultaneous runs with build variants
+
+In order to checkout multiple gem5 builds and run them simultaneously, you also need to use the `-N`:
+
+....
+./build -g
+git -C gem5/gem5 checkout some-branch
+./build -g -M some-branch -N some-branch
+git -C gem5/gem5 checkout -
+./run -g -n 0 &>/dev/null &
+./run -g -M some-branch -N some-branch -n 1 &>/dev/null &
+....
+
+The `-N <woktree-id>` determines the location of the gem5 tree to be used for both:
+
+* the input C files of the build at build time
+* the Python scripts to be used at runtime
+
+The difference between `-M` and `-N` is that `-M` specifies the gem5 build output directory, while `-N` specifies the source input directory.
+
+When `-N` is not given, source tree under `gem5/gem5` is used.
+
+If `-N <worktree-id>` is given, the directory used is `data/gem5/<worktree-id>`, and:
+
+* if that directory does not exist, create a `git worktree` at a branch `wt/<worktree-id>` on current commit of `gem5/gem5` there.
++
+The `wt/` branch name prefix stands for `WorkTree`, and is done to allow us to checkout to a test `some-branch` branch under `gem5/gem5` and still use `-N some-branch`, without conflict for the worktree branch, which can only be checked out once.
+* otherwise, leave that worktree untouched, without updating it
+
+`-N` is only required if you have multiple gem5 checkouts, e.g. it would not be required for multiple builds of the same tree, e.g. a <<gem5-debug-build>> and a non-debug one.
+
+===== gem5 debug build
+
+Built and run `gem5.debug`, which has optimizations turned off unlike the default `gem5.opt`:
+
+....
+./build -aA -g -M debug -t debug
+./run -aA -g -M debug -t debug
+....
+
+`-M` is optional just to prevent it from overwriting the `opt` build.
+
+A Linux kernel boot was about 14 times slower than opt at 71e927e63bda6507d5a528f22c78d65099bdf36f between the commands:
+
+....
+./run -aA -E 'm5 exit' -g -L v4.16
+./run -aA -E 'm5 exit' -g -M debug -t debug -L v4.16
+....
+
+Therefore the performance different is very big, making debug mode almost unusable.
 
 ==== Generic package build variants
 

build

@@ -21,7 +21,7 @@ post_script_args=
 qemu_sdl='--enable-sdl --with-sdlabi=2.0'
 suffix=
 v=0
-while getopts 'a:B:b:C:c:fGgj:hIiK:kL:lM:p:qSs:v' OPT; do
+while getopts 'a:B:b:C:c:fGgj:hIiK:kL:lM:p:qSst::v' OPT; do
   case "$OPT" in
     a)
       arch="$OPTARG"
@@ -98,6 +98,9 @@ BR2_TARGET_ROOTFS_INITRAMFS=n
     s)
       suffix="$OPTARG"
       ;;
+    t)
+      common_gem5_build_type="$OPTARG"
+      ;;
     v)
       v=1
       ;;
@@ -220,8 +223,9 @@ env \\
 make \\
 O='${buildroot_out_dir}' \\
 HOST_QEMU_OPTS='--enable-debug --enable-trace-backends=simple ${qemu_sdl}' \\
-GEM5_LKMC_SRCDIR="$common_gem5_src_dir" \\
+GEM5_LKMC_GEM5_BUILD_TYPE="$common_gem5_build_type" \\
 GEM5_LKMC_OUTDIR="$common_gem5_out_dir" \\
+GEM5_LKMC_SRCDIR="$common_gem5_src_dir" \\
 V='${v}' \\
 ${extra_make_args} \
 all \\

build-usage.adoc

@@ -26,13 +26,14 @@
                              Mnemonic: `fast`.
 |`-g` |                     |Enable gem5 build or force its rebuild.
 |`-h` |                     |Show this help message.
-|`-L` |                     |Linux kernel build variant.
+|`-L` |`VARIANT`             |Linux kernel build variant.
 |`-I` |                     |Enable initramfs for the current build.
 |`-i` |                     |Enable initrd for the current build.
 |`-K` |`KERNEL_CONFIG_FILE` |Use `KERNEL_CONFIG_FILE` as the exact Linux kernel
                              configuration. Ignore the default kernel config fragments,
                              but still add options explicitly passed with `-C` and `-c`.
                              on top of it.
+|`-M` |`VARIANT`            |gem5 build variant.
 |`-p` |                     |Pass extra arguments to the `rootfs_post_build_script`.
 |`-S` |                     |Don't build QEMU with SDL support.
                              Graphics such as X11 won't work, only the terminal.

common

@@ -69,24 +69,28 @@ set_common_vars() {
   common_images_dir="${buildroot_out_dir}/images"
   host_dir="${buildroot_out_dir}/host"
   common_qemu_run_dir="${out_arch_dir}/qemu/${common_run_id}"
+  common_qemu_termout_file="${common_qemu_run_dir}/termout.txt"
   common_linux_custom_dir="${build_dir}/linux-custom"
   common_linux_variant_dir="${common_linux_custom_dir}.${linux_variant}"
   common_vmlinux="${common_linux_variant_dir}/vmlinux"
-  if [ "$gem5_variant" = default ]; then
+  if [ -n "$common_gem5_worktree" ]; then
-    common_gem5_src_dir="${root_dir}/gem5/gem5"
+    common_gem5_src_dir="${common_gem5_non_default_src_root_dir}/${common_gem5_worktree}"
   else
-    common_gem5_src_dir="${common_gem5_non_default_src_root_dir}/${gem5_variant}"
+    common_gem5_src_dir="${root_dir}/gem5/gem5"
   fi
   common_gem5_out_dir="${common_dir}/gem5/${gem5_variant}"
   common_gem5_m5term="${common_gem5_out_dir}/m5term"
   common_gem5_build_dir="${common_gem5_out_dir}/build"
   common_gem5_system_dir="${common_gem5_out_dir}/system"
   common_gem5_run_dir="${out_arch_dir}/gem5/${gem5_variant}/${common_run_id}"
+  common_gem5_termout_file="${common_gem5_run_dir}/termout.txt"
   common_m5out_dir="${common_gem5_run_dir}/m5out"
   if "$gem5"; then
     common_run_dir="$common_gem5_run_dir"
+    common_termout_file="$common_gem5_termout_file"
   else
     common_run_dir="$common_qemu_run_dir"
+    common_termout_file="$common_qemu_termout_file"
   fi
   common_trace_txt_file="${common_m5out_dir}/trace.txt"
   case "$arch" in
@@ -136,8 +140,10 @@ data_dir="${root_dir}/data"
 p9_dir="${data_dir}/9p"
 readfile_file="${data_dir}/readfile"
 common_dir="${out_dir}/common"
+common_gem5_build_type=opt
 common_gem5_default_src_dir="${root_dir}/gem5/gem5"
 common_gem5_non_default_src_root_dir="${data_dir}/gem5"
+common_gem5_worktree=
 common_gem5_variant=default
 common_run_id=0
 f="${data_dir}/cli"

gem5-bench-cache

@@ -68,7 +68,7 @@ if "$generate_checkpoints"; then
   # Create the checkpoints after the kernel boot.
   rm -rf "${common_m5out_dir}"/cpt.*;
   printf 'm5 exit' > "${readfile_file}"
-  cpt_cmd="-E 'm5 checkpoint;m5 readfile > a.sh;sh a.sh'"
+  cpt_cmd="-E '/gem5.sh'"
   # 1
   ./eeval "$cmd $cpt_cmd"
   # RESTORE_INVESTIGATION

gem5/build

@@ -1,10 +1,11 @@
 #!/usr/bin/env bash
 set -eux
 arch=x86_64
+build_type=opt
 cross_compile=
 j=
 outdir="$(pwd)"
-while getopts a:c:j:o: OPT; do
+while getopts a:c:j:o:t: OPT; do
   case "$OPT" in
     a)
       arch="$OPTARG"
@@ -18,6 +19,12 @@ while getopts a:c:j:o: OPT; do
     o)
       outdir="$OPTARG"
       ;;
+    t)
+      build_type="$OPTARG"
+      ;;
+    ?)
+      exit 2
+      ;;
   esac
 done
 shift "$(($OPTIND - 1))"
@@ -30,7 +37,7 @@ disks_dir="${system_dir}/disks"
 mkdir -p "$binaries_dir" "$disks_dir"
 export PATH="/usr/lib/ccache:${PATH}"
 if [ "$arch" = x86_64 ]; then
-  scons -j "$j" --ignore-style "${outdir}/build/X86/gem5.opt"
+  scons -j "$j" --ignore-style "${outdir}/build/X86/gem5.${build_type}"
   f="${disks_dir}/linux-bigswap2.img"
   dd if=/dev/zero of="$f" bs=1024 count=65536
   mkswap "$f"
@@ -38,7 +45,7 @@ if [ "$arch" = x86_64 ]; then
   # I'm not even joking. No one has ever built x86 gem5 without the magic dist dir present.
   touch "${binaries_dir}/x86_64-vmlinux-2.6.22.9"
 elif [ "$arch" = arm ] || [ "$arch" = aarch64 ]; then
-  scons -j "$j" --ignore-style "${outdir}/build/ARM/gem5.opt"
+  scons -j "$j" --ignore-style "${outdir}/build/ARM/gem5.${build_type}"
   make -C ./system/arm/dt/
   mkdir -p "${system_dir}/arm/dt"
   # || true in case they are the same directory.

gem5/external.mk

@@ -16,7 +16,7 @@ endif
 
 define GEM5_BUILD_CMDS
 	# TODO cannot pass "-c '$(TARGET_CROSS)'" here because the ARM build uses aarch64 for the bootloader...
-	cd '$(GEM5_LKMC_SRCDIR)' && '$(GEM5_SITE)/build' -a '$(ARCH)' -j '$(BR2_JLEVEL)' -o '$(GEM5_LKMC_OUTDIR)'
+	cd '$(GEM5_LKMC_SRCDIR)' && '$(GEM5_SITE)/build' -a '$(ARCH)' -j '$(BR2_JLEVEL)' -o '$(GEM5_LKMC_OUTDIR)' -t '$(GEM5_LKMC_GEM5_BUILD_TYPE)'
 
 	# TODO cannot use TARGET_CONFIGURE_OPTS here because it overrides the CFLAGS on m5,
 	# which have an include. We should patch gem5 to add a += instead of = there.

rootfs_overlay/gem5.sh

@@ -0,0 +1,7 @@
+#!/bin/sh
+m5 checkpoint
+script=/tmp/readfile
+m5 readfile > "$script"
+if [ -s "$script" ]; then
+  sh "$script"
+fi

run

@@ -36,7 +36,7 @@ trace_enabled=false
 # just to prevent QEMU from emitting a warning that '' is not valid.
 trace_type=pr_manager_run
 vnc=
-while getopts a:c:DdE:e:F:f:G:ghIiKkL:M:m:n:PT:U:uVX:x OPT; do
+while getopts a:c:DdE:e:F:f:G:ghIiKkL:M:m:N:n:PT:t:U:uVX:x OPT; do
   case "$OPT" in
     a)
       arch="$OPTARG"
@@ -100,6 +100,9 @@ while getopts a:c:DdE:e:F:f:G:ghIiKkL:M:m:n:PT:U:uVX:x OPT; do
     m)
       memory="$OPTARG"
       ;;
+    N)
+      common_gem5_worktree="$OPTARG"
+      ;;
     n)
       common_run_id="$OPTARG"
       ;;
@@ -110,6 +113,9 @@ while getopts a:c:DdE:e:F:f:G:ghIiKkL:M:m:n:PT:U:uVX:x OPT; do
       trace_enabled=true
       trace_type="$OPTARG"
       ;;
+    t)
+      common_gem5_build_type="$OPTARG"
+      ;;
     U)
       tmux_args="$OPTARG"
       ;;
@@ -188,7 +194,7 @@ if "$gem5"; then
   gem5_common="\
 M5_PATH='${common_gem5_system_dir}' \\
 ${debug_vm} \
-'${common_gem5_build_dir}/${gem5_arch}/gem5.opt' \\
+'${common_gem5_build_dir}/${gem5_arch}/gem5.${common_gem5_build_type}' \\
 --debug-file=trace.txt \\
 ${gem5opts} \
 -d '${common_m5out_dir}' \\
@@ -356,4 +362,7 @@ if "$tmux"; then
     eval "./tmu ./rungdb -a '${arch} -L ${common_linux_variant}' -n ${common_run_id} ${tmux_args}"
   fi
 fi
+cmd="${cmd} \\
+|& tee ${common_termout_file} \
+"
 "${root_dir}/eeval" "$cmd" "${common_run_dir}/run.sh"

run-usage.adoc

@@ -36,11 +36,13 @@
 |`-i`   |              |Run with initrd.
 |`-K`   |              |Use KVM. Only works if guest arch == host arch.
 |`-k`   |              |Enable KGDB.
-|`-L`   |              |Linux kernel build variant.
+|`-L`   |`VARIANT`     |Linux kernel build variant.
+|`-M`   |`VARIANT`     |gem5 build output variant.
 |`-m`   |              |Set the memory size of the guest. E.g.: `-m 512M`. Default: `256M`.
                         The default is the minimum amount that boots all archs without extra
                         options added. Anything lower will lead some arch to fail to boot.
                         Any
+|`-N`   |`VARIANT`     |gem5 source input variant.
 |`-n`   |              |Run ID.
 |`-P`   |              |Run the downloaded prebuilt images.
 |`-T`   |`TRACE_TYPES` |Set trace events to be enabled.