Merge branch 'master' of github.com:cirosantilli/linux-kernel-module-cheat

.gitignore

@@ -8,6 +8,7 @@
 *~
 .tmp_versions
 /rootfs_overlay/etc/init.d/S99
+/rootfs_overlay/ignore.sh
 Module.symvers
 README.html
 modules.order

.gitmodules

@@ -11,3 +11,6 @@
 [submodule "gem5/gem5"]
 	path = gem5/gem5
 	url = https://gem5.googlesource.com/public/gem5
+[submodule "parsec-benchmark/parsec-benchmark"]
+	path = parsec-benchmark/parsec-benchmark
+	url = https://github.com/cirosantilli/parsec-benchmark

README.adoc

@@ -23,10 +23,7 @@ cd linux-kernel-module-cheat
 ./configure && ./build && ./run
 ....
 
-The first build will take a while (https://stackoverflow.com/questions/10833672/buildroot-environment-with-host-toolchain[GCC], Linux kernel), e.g.:
+The first configure will take a while (30 minutes to 2 hours) to clone and build, see <<benchmarking-this-repo>> for more details.
-
-* 2 hours on a mid end 2012 laptop
-* 30 minutes on a high end 2017 desktop
 
 If you don't want to wait, you could also try to compile the examples and run them on your host computer as explained on at <<run-on-host>>, but as explained on that section, that is dangerous, limited, and will likely not work.
 
@@ -76,29 +73,36 @@ git ls-files | grep modulename
 
 === Rebuild
 
-If you make changes to the kernel modules or most configurations tracked on this repository, you can just use again:
+After making changes to a package, you must explicitly tell it to be rebuilt.
+
+For example, you you modify the kernel modules, you must rebuild with:
 
 ....
-./build
+./build -k
-./run
 ....
 
-and the modified files will be rebuilt.
+which is just an alias for:
-
-If you change any package besides `kernel_module`, you must also request those packages to be reconfigured or rebuilt with extra targets, e.g.:
 
 ....
-./build -t linux-reconfigure -t host-qemu-reconfigure
+./build -- kernel_module-reconfigure
 ....
 
-Those aren't turned on by default because they take quite a few seconds.
+where `kernel_module` is the name of out Buildroot package that contains the kernel modules.
 
-Linux and QEMU rebuilds are so common that we have dedicated shortcut flags for them:
+Other important targets are:
+
+....
+./build -- linux-reconfigure host-qemu-reconfigure
+....
+
+which are aliased respectively to:
 
 ....
 ./build -l -q
 ....
 
+We don't rebuild by default because, even with `make` incremental rebuilds, the timestamp check takes a few annoying seconds.
+
 === Clean the build
 
 You did something crazy, and nothing seems to work anymore?
@@ -158,6 +162,8 @@ the disk image gets overwritten by a fresh filesystem and you lose all changes.
 
 Remember that if you forcibly turn QEMU off without `sync` or `poweroff` from inside the VM, e.g. by closing the QEMU window, disk changes may not be saved.
 
+When booting from <<initrd>> however without a disk, persistency is lost.
+
 === Message control
 
 We use `printk` a lot, and it shows on the QEMU terminal by default. If that annoys you (e.g. you want to see stdout separately), do:
@@ -240,13 +246,23 @@ Instead, you can either run them from a minimal init:
 ./run -e 'init=/eval.sh - lkmc_eval="insmod /hello.ko;/poweroff.out"' -n
 ....
 
+or if the script is large, add it to a gitignored file that will go into the guest:
+
+....
+echo '
+insmod /hello.ko
+/poweroff.out
+' > rootfs_overlay/ignore.sh
+./run -e 'init=/ignore.sh' -n
+....
+
 or run them at the end of the BusyBox init, which does things like setting up networking:
 
 ....
 ./run -e '- lkmc_eval="insmod /hello.ko;wget -S google.com;poweroff.out;"'
 ....
 
-or add them to a new `init.d` entry:
+or add them to a new `init.d` entry to run at the end o the BusyBox init:
 
 ....
 cp rootfs_overlay/etc/init.d/S98 rootfs_overlay/etc/init.d/S99
@@ -369,6 +385,32 @@ Just make sure that you never click inside the QEMU window when doing that, othe
 
 You can still send key presses to QEMU however even without the mouse capture, just either click on the title bar, or alt tab to give it focus.
 
+=== What command was actually run?
+
+When asking for help on upstream repositories outside of this repository, you will need to provide the commands that you are running in detail without referencing our scripts.
+
+For example, QEMU developers will only want to see the final QEMU command that you are running.
+
+We make that easy by building commands as strings, and then echoing them before evaling.
+
+So for example when you run:
+
+....
+./run -a arm
+....
+
+Stdout shows a line with the full command of type:
+
+....
+./buildroot/output.arm~/host/usr/bin/qemu-system-arm -m 128M -monitor telnet::45454,server,nowait -netdev user,hostfwd=tcp::45455-:45455,id=net0 -smp 1  -M versatilepb -append 'root=/dev/sda nokaslr norandmaps printk.devkmsg=on printk.time=y' -device rtl8139,netdev=net0 -dtb ./buildroot/output.arm~/images/versatile-pb.dtb -kernel ./buildroot/output.arm~/images/zImage -serial stdio    -drive file='./buildroot/output.arm~/images/rootfs.ext2.qcow2,if=scsi,format=qcow2'
+....
+
+This line is also saved to a file for convenience:
+
+....
+cat ./run.log
+....
+
 [[gdb]]
 == GDB step debugging
 
@@ -819,7 +861,7 @@ continue
 
 This is of least reliable setup as there might be other processes that use the given virtual address.
 
-== Architecture
+== Architectures
 
 The portability of the kernel and toolchains is amazing: change an option and most things magically work on completely different hardware.
 
@@ -968,7 +1010,7 @@ To disable networking, use:
 To restore it, run:
 
 ....
-./build -t initscripts-reconfigure
+./build -- initscripts-reconfigure
 ....
 
 === The init environment
@@ -1031,6 +1073,32 @@ Kernel modules built from the Linux mainline tree with `CONFIG_SOME_MOD=m`, are
 modprobe dummy-irq
 ....
 
+== KVM
+
+You can make QEMU or gem5 <<gem5-vs-qemu-performance,run faster>> by passing enabling KVM with:
+
+....
+./run -K
+....
+
+but it was broken in gem5 with pending patches: https://www.mail-archive.com/gem5-users@gem5.org/msg15046.html
+
+KVM uses the link:https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine[KVM Linux kernel feature] of the host to run most instructions natively.
+
+We don't enable KVM by default because:
+
+* only works if the architecture of the guest equals that of the host.
++
+We have only tested / supported it on x86, but it is rumoured that QEMU and gem5 also have ARM KVM support if you are link:https://www.youtube.com/watch?v=8ItXpmLsINs[running an ARM desktop for some weird reason] :-)
+* limits visibility, since more things are running natively:
+** can't use GDB
+** can't do instruction tracing
+* kernel boots are already fast enough without `-enable-kvm`
+
+The main use case for `-enable-kvm` in this repository is to test if something that takes a long time to run is functionally correct.
+
+For example, when porting a benchmark to Buildroot, you can first use QEMU's KVM to test that benchmarks is producing the correct results, before analysing them more deeply in gem5, which runs much slower.
+
 == X11
 
 Only tested successfully in `x86_64`.
@@ -1038,7 +1106,7 @@ Only tested successfully in `x86_64`.
 Build:
 
 ....
-./build -x
+./build -i buildroot_config_fragment_x11
 ./run
 ....
 
@@ -1248,7 +1316,7 @@ One obvious use case is having an encrypted root filesystem: you keep the initrd
 
 I think GRUB then knows read common disk formats, and then loads that initrd to memory with a `/boot/grub/grub.cfg` directive of type:
 
-    initrd	/initrd.img-4.4.0-108-generic
+    initrd /initrd.img-4.4.0-108-generic
 
 Related: https://stackoverflow.com/questions/6405083/initrd-and-booting-the-linux-kernel
 
@@ -1485,12 +1553,14 @@ To get started, have a look at the "Hardware device drivers" secion under link:k
 
 == gem5
 
+=== gem5 getting started
+
 gem5 is a system simulator, much <<gem5-vs-qemu,like QEMU>>: http://gem5.org/
 
 For the most part, just add the `-g` option to the QEMU commands and everything should magically work:
 
 ....
-./configure && ./build -a arm -g
+./configure -gq && ./build -a arm -g
 ./run -a arm -g
 ....
 
@@ -1555,7 +1625,7 @@ time ./run -a arm -e 'init=/poweroff.out'
 time ./run -a arm -e 'm5 exit' -g
 time ./run -a arm -e 'm5 exit' -g -- --caches --cpu-type=HPI
 time ./run -a x86_64 -e 'init=/poweroff.out'
-time ./run -a x86_64 -e 'init=/poweroff.out' - -enable-kvm
+time ./run -a x86_64 -e 'init=/poweroff.out' -- -enable-kvm
 time ./run -a x86_64 -e 'init=/poweroff.out' -g
 ....
 
@@ -1572,12 +1642,7 @@ and the results were:
 |gem5 X86_64 |5 minutes 30 seconds| 82
 |===
 
-on a Lenovo P51 laptop with:
+tested on the <<p51>>.
-
-* Intel Core i7-7820HQ Processor (8MB Cache, up to 3.90GHz) (4 cores 8 threads)
-* 32GB(16+16) DDR4 2400MHz SODIMM
-* 512GB SSD PCIe TLC OPAL2
-* Ubuntu 17.10
 
 === gem5 run benchmark
 
@@ -1599,6 +1664,8 @@ It works like this:
 * the first commond boots linux with the default simplified `AtomicSimpleCPU`, and generates a <<gem5-checkpoint,checkpoint>> after the kernel boots and before running the benchmark
 * the second command restores the checkpoint with the more detailed `HPI` CPU model, and runs the benchmark. We don't boot with it because that is much slower.
 
+ARM employees have just been modifying benchmarking code with instrumentation directly: https://github.com/arm-university/arm-gem5-rsk/blob/aa3b51b175a0f3b6e75c9c856092ae0c8f2a7cdc/parsec_patches/xcompile-patch.diff#L230
+
 A few imperfections of our benchmarking method are:
 
 * when we do `m5 resetstats` and `m5 exit`, there is some time passed before the `exec` system call returns and the actual benchmark starts and ends
@@ -1819,6 +1886,146 @@ External open source benchmarks. We will try to create Buildroot packages for th
 * http://parsec.cs.princeton.edu/ Mentioned on docs: http://gem5.org/PARSEC_benchmarks
 * http://www.m5sim.org/Splash_benchmarks
 
+===== PARSEC benchmark
+
+We have ported parts of the link:http://parsec.cs.princeton.edu[PARSEC benchmark] for cross compilation at: https://github.com/cirosantilli/parsec-benchmark See the documentation on that repo to find out which benchmarks have been ported.
+
+This repo makes it trivial to get started with it:
+
+....
+configure -gpq && ./build -a arm -g -i buildroot_config_fragment_parsec
+./run -a arm -g
+....
+
+Once inside the guest, we could in theory launch PARSEC exactly as you would launch it on the host:
+
+....
+cd /parsec/
+bash
+. env.sh
+parsecmgmt -a run -p splash2x.fmm -i test
+....
+
+TODO: `splash2x.barnes` segfaults on `arsecmgmt -a run -p splash2x.fmm -i simsmall` inside QEMU. Why? Other benchmarks ran fine.
+
+....
+[PARSEC] [---------- Beginning of output ----------]
+Generating input file input_1...
+Running /parsec/ext/splash2x/apps/barnes/inst/arm-linux.gcc/bin/barnes 1 < input_1:
+reading input file :
+
+Segmentation fault
+....
+
+However, while this is fine inside QEMU, it is not practical in gem5, since the `parsecmgmt` Bash scripts just takes too long to run in that case!
+
+So instead, you must find out the raw executable command, and run it manually yourself.
+
+This command can be found from the `Running` line that `parsecmgmt` outputs when running the programs.
+
+"Luckily", we run the run scripts while creating the image to extract the inputs, so you can just do a find in your shell history to find the run command and find a line of type:
+
+....
+Running /parsec/ext/splash2x/apps/fmm/inst/arm-linux.gcc/bin/fmm 1 < input_1:
+....
+
+which teaches you that you can run `fmm` as:
+
+....
+cd /parsec/ext/splash2x/apps/fmm/run
+../inst/arm-linux.gcc/bin/fmm 1 < input_1
+....
+
+We are also collecting more raw commands for testing at: link:parsec-benchmark/test.sh[]
+
+And so inside of `gem5`, you likely want to do:
+
+....
+cd /parsec/ext/splash2x/apps/fmm/run
+m5 checkpoint
+m5 resetstats && /parsec/ext/splash2x/apps/fmm/inst/arm-linux.gcc/bin/fmm 1 < input_1 && m5 dumpstats
+....
+
+You will always want to `cd` into the `run` directory first, which is where the input is located.
+
+====== PARSEC change the input size
+
+One limitation is that only one input size is available on the guest for a given build.
+
+To change that, edit link:buildroot_config_fragment_parsec[] to contain for example:
+
+....
+BR2_PACKAGE_PARSEC_BENCHMARK_INPUT_SIZE=simsmall
+....
+
+and then rebuild with:
+
+....
+./build -a arm -g -i buildroot_config_fragment_parsec -- parsec-benchmark-reconfigure
+....
+
+This limitation exists because `parsecmgmt` generates the input files just before running via the Bash scripts, but we can't run `parsecmgmt` on gem5 as it is too slow!
+
+One option would be to do that inside the guest with QEMU, but this would required a full rebuild due to <<gem5-and-qemu-with-the-same-kernel-configuration>>.
+
+Also, we can't generate all input sizes at once, because many of them have the same name and would overwrite one another... Parsec clearly needs a redesign for embedded, maybe we will patch it later.
+
+====== PARSEC uninstall
+
+If you want to remove PARSEC later, Buildroot doesn't provide an automated package removal mechanism as documented at: link:https://github.com/buildroot/buildroot/blob/2017.08/docs/manual/rebuilding-packages.txt#L90[], but the following procedure should be satisfactory:
+
+....
+rm -rf \
+  ./buildroot/dl/parsec-* \
+  ./buildroot/output.arm-gem5~/build/parsec-* \
+  ./buildroot/output.arm-gem5~/build/packages-file-list.txt \
+  ./buildroot/output.arm-gem5~/images/rootfs.* \
+  ./buildroot/output.arm-gem5~/target/parsec-* \
+;
+./build -a arm -g
+....
+
+====== PARSEC benchmark hacking
+
+If you end up going inside link:parsec-benchmark/parsec-benchmark[] to hack up the benchmark (you will!), these tips will be helpful.
+
+Buildroot was not designed to deal with large images, and currently cross rebuilds are a bit slow, due to some image generation and validation steps.
+
+A few workarounds are:
+
+* develop in host first as much as you can. Our PARSEC fork supports it.
++
+If you do this, don't forget to do a:
++
+....
+cd parsec-benchmark/parsec-benchmark
+git clean -xdf .
+....
+before going for the cross compile build.
++
+* patch Buildroot to work well, and keep cross compiling all the way. This should be totally viable, and we should do it.
++
+Don't forget to explicitly rebuild PARSEC with:
++
+....
+./build -a arm -g -i buildroot_config_fragment_parsec parsec-benchmark-reconfigure
+....
++
+You may also want to test if your patches are still functionally correct inside of QEMU first, which is a faster emulator.
+* sell your soul, and compile natively inside the guest. We won't do this, not only because it is evil, but also because Buildroot explicitly does not support it: https://buildroot.org/downloads/manual/manual.html#faq-no-compiler-on-target ARM employees have been known to do this: https://github.com/arm-university/arm-gem5-rsk/blob/aa3b51b175a0f3b6e75c9c856092ae0c8f2a7cdc/parsec_patches/qemu-patch.diff
+
+TODO Buildroot is slow because of the `pkg-generic` `GLOBAL_INSTRUMENTATION_HOOKS` sanitation which go over the entire tree doing complex operations... I no like, in particular `check_bin_arch` and `check_host_rpath`.
+
+The pause is followed by:
+
+....
+buildroot/output.arm~/build/parsec-benchmark-custom/.stamp_target_installed
+....
+
+so which shows that the whole delay is inside our install itself.
+
+I put an `echo f` in `check_bin_arch`, and it just loops forever, does not stop on a particular package.
+
 === gem5 kernel command line parameters
 
 Analogous <<kernel-command-line-parameters,to QEMU>>:
@@ -2050,7 +2257,7 @@ info: Entering event queue @ 0.  Starting simulation...
 and the `telnet` at:
 
 ....
-2017-12-28-11-59-51@ciro@ciro-p51$ ./gem5-shell
+$ ./gem5-shell
 Trying 127.0.0.1...
 Connected to localhost.
 Escape character is '^]'.
@@ -2337,7 +2544,13 @@ diff .config.olg .config
 
 Copy and paste the diff additions to `buildroot_config_fragment`.
 
-==== What is making my build so slow?
+==== Benchmarking this repo
+
+In this section document how fast the build and clone are, and how to investigate them.
+
+Send a pull request if you try it out on something significantly different.
+
+===== What is making my build so slow?
 
 ....
 cd buildroot/output.x86_64~
@@ -2350,6 +2563,41 @@ Our philosophy is:
 * if something adds little to the build time, build it in by default
 * otherwise, make it optional
 
+The biggest time hog is always GCC, can we use a precompiled one? https://stackoverflow.com/questions/10833672/buildroot-environment-with-host-toolchain
+
+===== Benchmark machines
+
+The build times are calculated after doing link:https://buildroot.org/downloads/manual/manual.html#_offline_builds[`make source`], which downloads the sources, and basically benchmarks the Internet.
+
+====== P51
+
+Build time at 2c12b21b304178a81c9912817b782ead0286d282: 28 minutes
+
+Lenovo ThinkPad link:https://www3.lenovo.com/gb/en/laptops/thinkpad/p-series/P51/p/22TP2WPWP51[P51 laptop]:
+
+* 2500 USD in 2018 (high end)
+* Intel Core i7-7820HQ Processor (8MB Cache, up to 3.90GHz) (4 cores 8 threads)
+* 32GB(16+16) DDR4 2400MHz SODIMM
+* 512GB SSD PCIe TLC OPAL2
+* Ubuntu 17.10
+
+====== T430
+
+Build time: 2 hours.
+
+TODO specs, SHA.
+
+===== Benchmark internets
+
+====== 38Mbps
+
+2c12b21b304178a81c9912817b782ead0286d282:
+
+* shallow clone of all submodules: 4 minutes.
+* `make source`: 2 minutes
+
+Google M-lab speed test: 36.4Mbps
+
 === About
 
 This project is for people who want to learn and modify low level system components:

build

@@ -1,48 +1,53 @@
 #!/usr/bin/env bash
-set -e
+set -eu
 arch=x86_64
-extra_targets=''
+configure=true
+config_fragments=buildroot_config_fragment
+extra_make_args=''
 gem5=false
 j="$(($(nproc) - 2))"
 post_script_args=''
 qemu_sdl='--enable-sdl --with-sdlabi=2.0'
-x11=false
 v=0
-while getopts 'a:gj:lp:qSt:v' OPT; do
+while getopts 'a:Cgj:i:klp:qS:v' OPT; do
   case "$OPT" in
     a)
       arch="$OPTARG"
     ;;
+    C)
+      configure=false
+    ;;
     g)
       gem5=true
     ;;
     j)
       j="$OPTARG"
     ;;
+    i)
+      config_fragments="$config_fragments $OPTARG"
+    ;;
+    k)
+      extra_make_args="$extra_make_args kernel_module-reconfigure"
+    ;;
     l)
-      extra_targets="$extra_args linux-reconfigure"
+      extra_make_args="$extra_make_args linux-reconfigure"
     ;;
     p)
       post_script_args="$OPTARG"
     ;;
     q)
-      extra_targets="$extra_args host-qemu-reconfigure"
+      extra_make_args="$extra_make_args host-qemu-reconfigure"
     ;;
     S)
       qemu_sdl=''
     ;;
-    t)
-      extra_targets="$extra_args $OPTARG"
-    ;;
-    x)
-      x11=true
-    ;;
     v)
       v=1
     ;;
   esac
 done
 shift $(($OPTIND - 1))
+extra_make_args="$extra_make_args $@"
 case "$arch" in
   x86_64)
     defconfig=qemu_x86_64_defconfig
@@ -62,60 +67,71 @@ if "$gem5"; then
   arch_dir="${arch}-gem5"
   # Networking was not working, so disable it to speed things up.
   post_script_args="$post_script_args -n"
+else
+  config_fragments="$config_fragments buildroot_config_fragment_qemu"
+fi
+root_dir="$(pwd)"
+buildroot_dir="${root_dir}/buildroot"
+out_dir="${buildroot_dir}/output.${arch_dir}~"
+config_file="${out_dir}/.config"
+
+if "$configure"; then
+  cd "${buildroot_dir}"
+  for p in $(find "${root_dir}/buildroot_patches/" -maxdepth 1 -name '*.patch' -print); do
+      patch -N -r - -p 1 <"$p" || :
+  done
+  make O="$out_dir" BR2_EXTERNAL="${root_dir}/kernel_module:${root_dir}/gem5:${root_dir}/parsec-benchmark" "$defconfig"
+  # TODO Can't get rid of these for now.
+  # http://stackoverflow.com/questions/44078245/is-it-possible-to-use-config-fragments-with-buildroots-config
+  for config_fragment in $config_fragments; do
+    cat "../$config_fragment" >> "${config_file}"
+  done
+  printf "
+BR2_JLEVEL=$j
+BR2_ROOTFS_POST_SCRIPT_ARGS=\"$post_script_args\"
+" >> "${config_file}"
+  if "$gem5"; then
+    printf "\
+BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE=\"../kernel_config_${arch_dir}\"
+BR2_PACKAGE_GEM5=y
+" >> "${config_file}"
+  else
+    printf "\
+BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES=\"../kernel_config_fragment\"
+" >> "${config_file}"
+  fi
+  if [ "$arch" = 'mips64' ]; then
+    # Workaround for:
+    # http://lists.busybox.net/pipermail/buildroot/2017-August/201053.html
+    sed -Ei 's/^BR2_PACKAGE_LINUX_TOOLS_GPIO/BR2_PACKAGE_LINUX_TOOLS_GPIO=n/' "${config_file}"
+  fi
+  make O="$out_dir" olddefconfig
 fi
 
-cd kernel_module
+cd "${root_dir}/kernel_module"
 ./make-host.sh -j "$j" clean
-cd ../buildroot
+cd "${buildroot_dir}"
-for p in $(find '../buildroot_patches/' -maxdepth 1 -name '*.patch' -print); do
-    patch -N -r - -p 1 <"$p" || :
-done
-outdir="output.${arch_dir}~"
-make O="$outdir" BR2_EXTERNAL="$(pwd)/../kernel_module:$(pwd)/../gem5" "$defconfig"
-# TODO Can't get rid of this for now.
-# http://stackoverflow.com/questions/44078245/is-it-possible-to-use-config-fragments-with-buildroots-config
-cat ../buildroot_config_fragment >> "${outdir}/.config"
-if "$gem5"; then
-  echo "\
-BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE=\"../kernel_config_${arch_dir}\"
-" >> "${outdir}/.config"
-else
-  echo "\
-BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES=\"../kernel_config_fragment\"
-" >> "${outdir}/.config"
-fi
-if "$x11"; then
-  cat ../buildroot_config_fragment_x11 >> "${outdir}/.config"
-fi
-if ! "$gem5"; then
-  cat ../buildroot_config_fragment_qemu >> "${outdir}/.config"
-fi
-if [ "$arch" = 'mips64' ]; then
-  # Workaround for:
-  # http://lists.busybox.net/pipermail/buildroot/2017-August/201053.html
-  sed -Ei 's/^BR2_PACKAGE_LINUX_TOOLS_GPIO/BR2_PACKAGE_LINUX_TOOLS_GPIO=n/' "${outdir}/.config"
-fi
-make O="$outdir" olddefconfig
 # HOST_QEMU_OPTS is a hack that happens to work because the QEMU package luckly uses += at all times.
 # It shouldn't be necessary in the first place: https://bugs.busybox.net/show_bug.cgi?id=9936
 #
 # Even if were an autotools package, there is no general way currently to pass extra configs to it:
 # https://stackoverflow.com/questions/44341188/how-to-pass-extra-custom-configure-autotools-options-to-a-buildroot-package/44341225#44341225
-time \
+#
+# BR2_ options may be given on the command line here, and they do have direct "define" effects.
+# But this is generally bad, as it skips the Kconfig mechanism, e.g. it does not set defaults properly.
+cmd="time \
   env \
   -u LD_LIBRARY_PATH \
   make \
-  O="$outdir" \
+  O='$out_dir' \
-  BR2_JLEVEL="$j" \
+  HOST_QEMU_OPTS='--enable-debug --extra-cflags=-DDEBUG_PL061=1 --enable-trace-backends=simple $qemu_sdl' \
-  BR2_PACKAGE_GEM5="$("$gem5" && echo y || echo n)" \
+  V='$v' \
-  BR2_ROOTFS_POST_SCRIPT_ARGS="$post_script_args" \
+  $extra_make_args \
-  HOST_QEMU_OPTS="--enable-debug --extra-cflags='-DDEBUG_PL061=1' --enable-trace-backends=simple $qemu_sdl" \
-  V="$v" \
-  kernel_module-rebuild \
-  $extra_targets \
   all \
-;
+"
-cd ..
+echo "$cmd" | tee "${root_dir}/build.log"
+eval "$cmd"
 if "$gem5"; then
+  cd "${root_dir}"
   ./build-gem5 -a "$arch"
 fi

buildroot_config_fragment

@@ -1,11 +1,13 @@
-BR2_ENABLE_LOCALE=y
+# Custom packages
+BR2_PACKAGE_KERNEL_MODULE=y
+
 BR2_GCC_ENABLE_GRAPHITE=y
 BR2_GCC_ENABLE_LTO=y
 BR2_GCC_ENABLE_OPENMP=y
 BR2_GLOBAL_PATCH_DIR="../global_patch_dir"
 BR2_PACKAGE_BUSYBOX_CONFIG_FRAGMENT_FILES="../busybox_config_fragment"
 BR2_PACKAGE_DHRYSTONE=y
-BR2_PACKAGE_KERNEL_MODULE=y
+BR2_PACKAGE_FILE=y
 BR2_PACKAGE_OVERRIDE_FILE="../buildroot_override"
 # For qemu-ga on guest. TODO: do something with it, and document it.
 BR2_PACKAGE_QEMU=y

buildroot_config_fragment_parsec

@@ -0,0 +1,8 @@
+BR2_PACKAGE_PARSEC_BENCHMARK=y
+#BR2_PACKAGE_PARSEC_BENCHMARK_BUILD_LIST="splash2x"
+#BR2_PACKAGE_PARSEC_BENCHMARK_INPUT_SIZE="simsmall"
+# Because PARSEC + its data are huge. TODO: can't we automate calculating the size?
+# Problems will arise if someone tries to use two such benchmarks.
+# Cannot be selected automatically from Kconfig:
+# https://stackoverflow.com/questions/40309054/how-to-select-the-value-of-a-string-option-from-another-option-in-kbuild-kconfig/49096538#49096538
+BR2_TARGET_ROOTFS_EXT2_SIZE="128M"

buildroot_override

@@ -1,3 +1,4 @@
 HOST_QEMU_OVERRIDE_SRCDIR = ../qemu
 LINUX_OVERRIDE_SRCDIR = ../linux
+PARSEC_BENCHMARK_OVERRIDE_SRCDIR = ../parsec-benchmark/parsec-benchmark
 QEMU_OVERRIDE_SRCDIR = ../qemu

configure

@@ -1,9 +1,21 @@
 #!/usr/bin/env bash
 set -e
 interactive_pkgs=libsdl2-dev
+gem5=false
+qemu=true
+submodules='buildroot linux'
 y=''
-while getopts t OPT; do
+while getopts gqpt OPT; do
   case "$OPT" in
+    g)
+      gem5=true
+      ;;
+    q)
+      qemu=false
+      ;;
+    p)
+      submodules="$submodules parsec-benchmark/parsec-benchmark"
+      ;;
     t)
       interactive_pkgs=''
       y='-y'
@@ -12,45 +24,56 @@ while getopts t OPT; do
 done
 shift $(($OPTIND - 1))
 
+## Submodules
+
+if "$qemu"; then
+  submodules="$submodules qemu"
+fi
+if "$gem5"; then
+  submodules="$submodules gem5/gem5"
+fi
 (
   set -e
-  # Shallow clonning saves a considerable ammount of time, specially because of the linux kernel.
+  # Shallow cloning saves a considerable amount of time, specially because of the linux kernel.
-  # However, git submodules are buggy as usual, and this is the best way i've found to get it done:
+  # However, git submodules are buggy as usual, and this is the best way I've found to get it done:
   # https://stackoverflow.com/questions/2144406/git-shallow-submodules/47374702#47374702
   # In particular:
   # - `shallow = true` on the submodule has no effect for the non default educational branches of our submodules
   # - QEMU's submodules point to commits that are neither under branches nor tags, and so `--shallow-submodules` fails
-  git submodule update --depth 1 --jobs 4 --init
+  git submodule update --depth 1 --jobs 4 --init -- $submodules
-  cd qemu
+  if "$qemu"; then
-  git submodule update --init
+    cd qemu
+    git submodule update --init
+  fi
 ) &
-wait $! || git submodule update --init
+wait $! || git submodule update --init -- $submodules
+
+## apt-get
 
 pkgs="\
 automake \
 build-essential \
 coreutils \
 "
-
+if "$gem5"; then
-# GEM5
+  pkgs="$pkgs \
-pkgs="$pkgs \
+  g++-6 \
-g++-6 \
+  gcc-6 \
-gcc-6 \
+  gcc-aarch64-linux-gnu \
-gcc-aarch64-linux-gnu \
+  gcc-arm-linux-gnueabi \
-gcc-arm-linux-gnueabi \
+  libgoogle-perftools-dev \
-libgoogle-perftools-dev \
+  protobuf-compiler \
-protobuf-compiler \
+  "
-"
+fi
-
 command -v apt-get >/dev/null 2>&1 || {
-    cat <<EOF
+  cat <<EOF
 apt-get not found. You're on your own for installing dependencies.
 
 On Ubuntu they are:
 
 $pkgs
 EOF
-    exit 0
+  exit 0
 }
 
 # Without this started failing in kernel 4.15 with:
@@ -67,4 +90,6 @@ sudo apt-get install $y \
   $pkgs \
   $interactive_pkgs \
 ;
-sudo apt-get build-dep $y qemu
+if "$qemu"; then
+  sudo apt-get build-dep $y qemu
+fi

gem5/Config.in

@@ -1,4 +1,7 @@
 config BR2_PACKAGE_GEM5
 	bool "gem5"
 	help
-		GEM5
+	  gem5 system simulator. Only builds the m5 guest instrumentation
+	  tool for now, not the simulator itself.
+
+	  http://gem5.org

parsec

@@ -1,14 +0,0 @@
-#!/usr/bin/env bash
-wget http://parsec.cs.princeton.edu/download/3.0/parsec-3.0.tar.gz
-tar -xvzf parsec-3.0.tar.gz
-cd parsec-3.0
-. env.sh
-parsecmgmt -a build -p streamcluster
-parsecmgmt -a run -p streamcluster
-
-parsecmgmt -a build -p all
-
-wget http://parsec.cs.princeton.edu/download/3.0/parsec-3.0-input-sim.tar.gz
-tar -xzf parsec-3.0-input-sim.tar.gz
-parsecmgmt -a run -p streamcluster -i simsmall
-parsecmgmt -a run -p streamcluster -i simlarge -n 2

parsec-benchmark/Config.in

@@ -0,0 +1,29 @@
+config BR2_PACKAGE_PARSEC_BENCHMARK
+	bool "PARSEC_BENCHMARK"
+# Parsec shell scripts use a hardcoded /bin/bash
+# One option would be to try and use /bin/sh.
+# But symlinking fails because of BusyBox' symlink mechanism.
+# The other option would be to patch Parsec to use /bin/sh and be POSIX compliant.
+# But let's take the path of smallest resistance for now.
+	select BR2_PACKAGE_BUSYBOX_SHOW_OTHERS
+	select BR2_PACKAGE_BASH
+	help
+	  Parsec system benchmark.
+
+	  http://parsec.cs.princeton.edu/
+
+if BR2_PACKAGE_PARSEC_BENCHMARK
+
+config BR2_PACKAGE_PARSEC_BENCHMARK_BUILD_LIST
+	string "build_list"
+	default splash2x
+	help
+	  Space separated list of parsec packages to build.
+
+config BR2_PACKAGE_PARSEC_BENCHMARK_INPUT_SIZE
+	string "input_size"
+	default test
+	help
+	  Which input size to generate on the host for the guest.
+
+endif

parsec-benchmark/external.desc

@@ -0,0 +1 @@
+name: PARSEC_BENCHMARK

parsec-benchmark/external.mk

@@ -0,0 +1,56 @@
+################################################################################
+#
+# PARSEC_BENCHMARK
+#
+################################################################################
+
+PARSEC_BENCHMARK_VERSION = master
+PARSEC_BENCHMARK_SITE = git@github.com:cirosantilli/parsec-benchmark.git
+PARSEC_BENCHMARK_SITE_METHOD = git
+
+define PARSEC_BENCHMARK_BUILD_CMDS
+  # TODO make this nicer, only untar when extract step is done.
+  # EXTRACT_CMDS and EXTRA_DOWNLOADS would be good candidates,
+  # but they don't run with OVERRIDE_SRCDIR.
+  '$(PARSEC_BENCHMARK_PKGDIR)/parsec-benchmark/get-inputs' $(if $(filter $(V),1),-v,) '$(DL_DIR)' '$(@D)/'
+  # We run the benchmarks with the wrong arch here to generate the inputs on the host.
+  # This is because on gem5 this takes too long to do.
+  cd $(@D) && . env.sh && for pkg in $(BR2_PACKAGE_PARSEC_BENCHMARK_BUILD_LIST); do \
+    export HOSTCC='$(HOSTCC)'; \
+    export M4='$(HOST_DIR)/usr/bin/m4'; \
+    export MAKE='$(MAKE)'; \
+    export OSTYPE=linux; \
+    export TARGET_CROSS='$(TARGET_CROSS)'; \
+    export HOSTTYPE='$(BR2_ARCH)'; \
+    parsecmgmt -a build -p $$pkg; \
+    parsecmgmt -a run -p $$pkg -i $(BR2_PACKAGE_PARSEC_BENCHMARK_INPUT_SIZE); \
+  done
+endef
+
+define PARSEC_BENCHMARK_INSTALL_TARGET_CMDS
+  # This is a bit coarse and makes the image larger with useless source code.
+  #
+  # But according to du, the source accounts for only 1/5 of the total size,
+  # so benchmarks dominate, and it doesn't matter much.
+  #
+  # Also it is not so critical for simulators anyways unlike real embedded systems.
+  #
+  # One possibility to make this better may be to install only the 'inst/' and 'input/'
+  # folders for each package + toplevel '/bin/' and '/config/', but of course we won't
+  # know if this works until time consuming testing is done :-)
+  mkdir -p '$(TARGET_DIR)/parsec/'
+
+  $(INSTALL) -D -m 0755 '$(PARSEC_BENCHMARK_PKGDIR)/test.sh' '$(TARGET_DIR)/parsec/'
+
+  rsync -am $(if $(filter $(V),1),-v,) --include '*/' \
+    --include '/bin/***' \
+    --include '/config/***' \
+    --include '/env.sh' \
+    --include 'inst/***' \
+    --include 'run/***' \
+    --exclude '*' '$(@D)/' '$(TARGET_DIR)/parsec/' \
+  ;
+  # rsync finished.
+endef
+
+$(eval $(generic-package))

parsec-benchmark/test.sh

@@ -0,0 +1,26 @@
+#!/bin/sh
+
+# Since Parsec does not stop on errors and we need some raw commands for gem5,
+# let's do some of our own unit tests here. Thses should be run on QEMU
+# to be reasonably fast.
+#
+# Must be run with BR2_PACKAGE_PARSEC_BENCHMARK_INPUT_SIZE=test
+
+set -ex
+
+ncpus=1
+
+cd /parsec/ext/splash2x/apps/fmm/run
+../inst/*/bin/fmm "$ncpus" < input_1
+
+cd /parsec/ext/splash2x/apps/raytrace/run
+../inst/*/bin/raytrace -s -p"$ncpus" -a4 teapot.env
+
+cd /parsec/ext/splash2x/apps/volrend/run
+../inst/*/bin/volrend "$ncpus" head-scaleddown4 4 < input_1
+
+cd /parsec/ext/splash2x/apps/water_nsquared/run
+../inst/*/bin/water_nsquared "$ncpus" < input_1
+
+cd /parsec/ext/splash2x/apps/water_spatial/run
+../inst/*/bin/water_spatial "$ncpus" < input_1

run

@@ -7,6 +7,7 @@ arch=x86_64
 cpus=1
 debug_vm=''
 kgdb=false
+kvm=false
 nographic=false
 # norandmaps: Don't use address space randomization. Equivalent to echo 0 > /proc/sys/kernel/randomize_va_space.
 # printk.time=y: log in format: "[time ] msg" for all printk messages.
@@ -19,7 +20,7 @@ gem5=false
 gem5opts=''
 initrd=false
 root=''
-while getopts a:c:Dde:G:giknt:x OPT; do
+while getopts a:c:Dde:G:giKknt:x OPT; do
   case "$OPT" in
     a)
       arch="$OPTARG"
@@ -36,6 +37,9 @@ while getopts a:c:Dde:G:giknt:x OPT; do
     e)
       extra_append="$extra_append $OPTARG"
       ;;
+    K)
+      kvm=true
+      ;;
     k)
       extra_append="$extra_append kgdbwait"
       # For those who want to try KDB.
@@ -66,6 +70,9 @@ if $gem5; then
   outdir="$(pwd)/buildroot/output.${arch}-gem5~"
   gem5_dir="$(pwd)/gem5/gem5"
   if [ "$arch" = x86_64 ]; then
+    if "$kvm"; then
+      extra_flags="$extra_flags --cpu-type=X86KvmCPU"
+    fi
     cmd="\
 M5_PATH='$(pwd)/gem5/gem5-system' \
 '${gem5_dir}/build/X86/gem5.opt' \
@@ -95,6 +102,9 @@ $extra_flags \
   fi
 else
   buildroot_out_dir="./buildroot/output.${arch}~"
+  if "$kvm"; then
+    extra_flags="$extra_flags -enable-kvm"
+  fi
   extra_flags="$extra_flags_qemu $extra_flags"
   images_dir="$buildroot_out_dir/images"
   qemu_common="\