run: support multiple simultaneous runs and run output directories

2026-01-23 02:05:57 +01:00 · 2018-06-12 10:29:56 +01:00
parent 37f64de29f
commit 28c70a3824
12 changed files with 171 additions and 88 deletions
--- a/README.adoc
+++ b/README.adoc
@@ -1129,6 +1129,74 @@ ____

 `finit` is newer and was added only in v3.8. More rationale: https://lwn.net/Articles/519010/

+=== Simultaneous runs
+
+When doing long simulations, it becomes fundamental to do multiple simulations in parallel.
+
+This is specially true for gem5, which runs much slower than QEMU, and cannot use multiple host cores to speed up the simulation: https://github.com/cirosantilli-work/gem5-issues/issues/15
+
+First shell:
+
+....
+./run
+....
+
+Another shell:
+
+....
+./run -n 1
+....
+
+The default run id is `0`.
+
+This method also allows us to keep run outputs in separate directories for later inspection, e.g.:
+
+....
+./run -aA -g -n 0
+./run -aA -g -n 1
+....
+
+produces two separate `m5out` directories:
+
+....
+less out/aarch64/gem5/default/0/m5out
+less out/aarch64/gem5/default/1/m5out
+....
+
+You can also add a prefix to the build ID before a period:
+
+....
+./run -aA -g -n some-experiment.1
+....
+
+which then uses the output directory:
+
+....
+less out/aarch64/gem5/default/some-experiment.1/m5out
+....
+
+and makes it easier to remember afterwards which directory contains what.
+
+However this still takes up the same ports as:
+
+....
+./run -aA -g -n 1
+....
+
+so you cannot run both at the same time.
+
+Like <<cpu-architecture>>, you will need to pass the `-n` option to anything that needs to know runtime information, e.g. <<gdb>>:
+
+....
+./run -n 1
+./rungdb -n 1
+....
+
+Implementation note: we create multiple namespaces for two things:
+
+* ports: gem5 automatically increments ports until it finds a free one, but QEMU does not, and just crashes
+* run output directory
+
 [[gdb]]
 == GDB step debug

@@ -1600,7 +1668,7 @@ One possibility is to run:
 and then find the second address (the first one does not work, already too late maybe):

 ....
-less ./out/arm/qemu/trace.txt
+less ./out/arm/qemu/0/trace.txt
 ....

 and break there:
@@ -2482,7 +2550,7 @@ To enable initrd instead of the default ext2 disk image, do:
 Notice how it boots fine, even though this leads to not giving QEMU the `-drive` option, as can be verified with:

 ....
-cat ./out/x86_64/qemu/run.sh
+cat ./out/x86_64/qemu/0/run.sh
 ....

 Also as expected, there is no filesystem persistency, since we are doing everything in memory:
@@ -5069,7 +5137,13 @@ The most interesting are events which show instructions that QEMU ran, for which
 You can then inspect the instructions with:

 ....
-less ./out/x86_64/qemu/trace.txt
+less ./out/x86_64/qemu/0/trace.txt
+....
+
+Get the list of available trace events:
+
+....
+./run -T help
 ....

 Enable other specific trace events:
@@ -5077,13 +5151,7 @@ Enable other specific trace events:
 ....
 ./run -T trace1,trace2
 ./qemu-trace2txt -a "$arch"
-less ./out/x86_64/qemu/trace.txt
-....
-
-Get the list of available trace events:
-
-....
-./run -T help
+less ./out/x86_64/qemu/0/trace.txt
 ....

 This functionality relies on the following setup:
@@ -5115,7 +5183,7 @@ We can further use Binutils' `addr2line` to get the line that corresponds to eac

 ....
 ./trace-boot -a x86_64 && ./trace2line -a x86_64
-less ./out/x86_64/qemu/trace-lines.txt
+less ./out/x86_64/qemu/0/trace-lines.txt
 ....

 The format is as follows:
@@ -5180,7 +5248,7 @@ But it also provides a tracing mechanism documented at: link:http://www.gem5.org

 ....
 ./run -a aarch64 -E 'm5 exit' -g -T Exec
-less out/aarch64/gem5/m5out/trace.txt
+less out/aarch64/gem5/default/0/m5out/trace.txt
 ....

 List all available debug flags:
@@ -5405,7 +5473,7 @@ Now you can play a fun little game with your friends:
 To find out why your program is slow, a good first step is to have a look at the statistics for the run:

 ....
-cat out/aarch64/gem5/m5out/stats.txt
+cat out/aarch64/gem5/default/0/m5out/stats.txt
 ....

 Whenever we run `m5 dumpstats` or `m5 exit`, a section with the following format is added to that file:
@@ -5949,6 +6017,8 @@ In the guest, wait for the boot to end and run:
 m5 checkpoint
 ....

+where <<m5>> is a guest utility present inside the gem5 tree which we cross-compiled and installed into the guest.
+
 To restore the checkpoint, kill the VM and run:

 ....
@@ -5990,19 +6060,11 @@ Then there is no need to pass the kernel command line again to gem5 for replay:

 since boot has already happened, and the parameters are already in the RAM of the snapshot.

-Our scripts "namespace" with the checkpoint by architecture with `--checkpoint-dir`, so if you make two checkpoints:
+Checkpoints are stored inside the `m5out` directory at:

-* one in x86
-* the other in arm
-
-Then you would still restore both of them with `-- -r 1`.
-
-This makes it easier to remember which checkpoint is which, especially since there appears to be no runtime way to set the checkpoint names.
-
-Internals:
-
-* the checkpoints are stored under `out/$arch/gem5/m5out/cpt.$todo_whatisthis`
-* <<m5>> is a guest utility present inside the gem5 tree which we cross-compiled and installed into the guest
+....
+out/<arch>/gem5/<gem5-variant>/<run-id>/m5out/cpt.<checkpoint-time>
+....

 [[gem5-restore-new-scrip]]
 ==== gem5 checkpoint restore and run a different script
@@ -6108,25 +6170,6 @@ Cycles instead of instructions:

 Otherwise the simulation runs forever by default.

-=== Run multiple gem5 instances at once
-
-gem5 just assigns new ports if some ports are occupied, so we can do:
-
-....
-./run -g
-# Same as ./gem5-shell 0
-./gem5-shell
-....
-
-And a second instance:
-
-....
-./run -g
-./gem5-shell 1
-....
-
-TODO Now we just need to network them up to have some more fun! See dist-gem5: http://publish.illinois.edu/icsl-pdgem5/
-
 === m5

 `m5` is a guest command line utility that is installed and run on the guest.
@@ -6165,7 +6208,7 @@ m5 writefile myfileguest myfilehost
 Host:

 ....
-cat out/aarch64/gem5/m5out/myfilehost
+cat out/aarch64/gem5/default/0/m5out/myfilehost
 ....

 Does not work for subdirectories, gem5 crashes:
@@ -6653,13 +6696,13 @@ Stdout shows a line with the full command of type:
 and this line is also saved to a file for convenience:

 ....
-cat ./out/arm/qemu/run.sh
+cat ./out/arm/qemu/0/run.sh
 ....

 or for gem5:

 ....
-cat ./out/arm/gem5/run.sh
+cat ./out/arm/gem5/default/0/run.sh
 ....

 Next, you will also want to give the relevant images to save them time. Zip the images with:
@@ -7073,13 +7116,16 @@ The action seems to be happening at: `hw/arm/virt.c`.
 **** `out/<arch>/buildroot/build/linux-custom`: symlink to a variant, custom madness that we do on top of Buildroot: <<linux-kernel-build-variants>>
 **** `out/<arch>/buildroot/build/linux-custom.<variant>`: what `linux-custom` points to
 *** `out/<arch>/qemu`: QEMU runtime outputs
+*** `out/<arch>/qemu/<run-id>/run.sh`: full CLI used to run QEMU. See: <<report-upstream-bugs>>
+*** `out/<arch>/gem5/<gem5-variant>/<run-id>/`: gem5 runtime outputs
+**** `out/<arch>/gem5/<gem5-variant>/<run-id>/m5out`
+**** `out/<arch>/gem5/<gem5-variant>/<run-id>/run.sh`: full CLI used to run gem5. See: <<report-upstream-bugs>>
 ** `out/common`: cross arch outputs, for when we can gain a lot of time and space by sharing things that are common across different archs.
 *** `out/common/dl/`: Buildroot caches downloaded source there due to `BR2_DL_DIR`
 *** `out/common/gem5/`: `arm` and `aarch64` have the same build.
-**** `out/common/gem5/<variant>/`
-***** `out/common/gem5/<variant>/build/`
-***** `out/common/gem5/<variant>/m5out`
-***** `out/common/gem5/<variant>/system`
+**** `out/common/gem5/<gem5-variant>/`: gem5 build output. In common to share the ARM and aarch64 builds.
+***** `out/common/gem5/<gem5-variant>/build/`: main build outputs, including the `gem5.opt` executable and object files
+***** `out/common/gem5/<gem5-variant>/system/`: `M5_PATH` directory, with DTBs and bootloaders

 ==== buildroot_patches

--- a/35
+++ b/35
@@ -23,7 +23,7 @@ set_common_vars() {
  linux_variant=
  gem5_variant=
  OPTIND=1
-  while getopts L:M: OPT; do
+  while getopts L:M:n: OPT; do
    case "$OPT" in
      L)
        linux_variant="$OPTARG"
@@ -31,6 +31,9 @@ set_common_vars() {
      M)
        gem5_variant="$OPTARG"
        ;;
+      n)
+        common_run_id="$OPTARG"
+        ;;
    esac
  done
  shift "$(($OPTIND - 1))"
@@ -65,7 +68,7 @@ set_common_vars() {
  build_dir="${buildroot_out_dir}/build"
  common_images_dir="${buildroot_out_dir}/images"
  host_dir="${buildroot_out_dir}/host"
-  qemu_out_dir="${out_arch_dir}/qemu"
+  common_qemu_run_dir="${out_arch_dir}/qemu/${common_run_id}"
  common_linux_custom_dir="${build_dir}/linux-custom"
  common_linux_variant_dir="${common_linux_custom_dir}.${linux_variant}"
  common_vmlinux="${common_linux_variant_dir}/vmlinux"
@@ -77,14 +80,14 @@ set_common_vars() {
  common_gem5_out_dir="${common_dir}/gem5/${gem5_variant}"
  common_gem5_build_dir="${common_gem5_out_dir}/build"
  common_gem5_system_dir="${common_gem5_out_dir}/system"
-  common_m5out_dir="${common_gem5_out_dir}/m5out"
+  common_gem5_run_dir="${out_arch_dir}/gem5/${gem5_variant}/${common_run_id}"
+  common_m5out_dir="${common_gem5_run_dir}/m5out"
  if "$gem5"; then
-    common_out_run_dir="$common_gem5_out_dir"
-    common_trace_txt_file="${common_m5out_dir}/trace.txt"
+    common_run_dir="$common_gem5_run_dir"
  else
-    common_out_run_dir="$qemu_out_dir"
-    common_trace_txt_file="${common_out_run_dir}/trace.txt"
+    common_run_dir="$common_qemu_run_dir"
  fi
+  common_trace_txt_file="${common_run_dir}/trace.txt"
  case "$arch" in
    arm)
      common_linux_image=arch/arm/boot/zImage
@@ -100,12 +103,27 @@ set_common_vars() {
      ;;
  esac
  common_linux_image="${common_linux_variant_dir}/${common_linux_image}"
+
+  # Ports.
+  common_run_id_number="$(echo "$common_run_id" | cut -d . -f 2)"
+  if "$gem5"; then
+    common_gem5_telnet_port="$((3456 + $common_run_id_number))"
+    common_gdb_port="$((7000 + $common_run_id_number))"
+  else
+    common_qemu_base_port="$((45454 + 10 * $common_run_id_number))"
+    common_qemu_monitor_port="$(($common_qemu_base_port + 0))"
+    common_qemu_hostfwd_generic_port="$(($common_qemu_base_port + 1))"
+    common_qemu_hostfwd_ssh_port="$(($common_qemu_base_port + 2))"
+    common_qemu_gdb_port="$(($common_qemu_base_port + 3))"
+    common_gdb_port="$common_qemu_gdb_port"
+  fi
 }
 common_mkdir() (
  mkdir -p \
    "$build_dir" \
    "$common_gem5_out_dir" \
-    "$qemu_out_dir" \
+    "$common_gem5_run_dir" \
+    "$common_qemu_run_dir" \
    "$p9_dir" \
  ;
 )
@@ -120,6 +138,7 @@ common_dir="${out_dir}/common"
 common_gem5_default_src_dir="${root_dir}/gem5/gem5"
 common_gem5_non_default_src_root_dir="${data_dir}/gem5"
 common_gem5_variant=default
+common_run_id=0
 f="${data_dir}/cli"
 if [ -f "$f" ]; then
  . "$f"
--- a/2
+++ b/2
@@ -1,5 +1,6 @@
 #!/usr/bin/env bash
 # echo and eval
+set -e
 a=
 while getopts a OPT; do
  case "$OPT" in
@@ -15,6 +16,7 @@ done
 shift "$(($OPTIND - 1))"
 cmd="$1"
 outfile="${2:-/dev/null}"
+mkdir -p "$(dirname "$outfile")"
 echo "$cmd" | tee $a "$outfile"
 eval "$cmd"
 exit "$?"
--- a/2
+++ b/2
@@ -19,7 +19,7 @@ set_common_vars "$arch" true
 cmd="./run -a $arch -g"
 cache_small='--caches --l2cache --l1d_size=1024   --l1i_size=1024   --l2_size=1024   --l3_size=1024  '
 cache_large='--caches --l2cache --l1d_size=1024kB --l1i_size=1024kB --l2_size=1024kB --l3_size=1024kB'
-results_file="${common_gem5_out_dir}/bench-cache.txt"
+results_file="${common_gem5_run_dir}/bench-cache.txt"

 bench() (
  common_bench_cmd "$1" "$results_file"
--- a/17
+++ b/17
@@ -1,4 +1,15 @@
 #!/usr/bin/env bash
-offset="${1:-0}"
-base=3456
-telnet localhost $(($base + $offset))
+. common
+while getopts n: OPT; do
+  case "$OPT" in
+    n)
+      common_run_id="$OPTARG"
+      ;;
+    ?)
+      exit 2
+      ;;
+  esac
+done
+shift "$(($OPTIND - 1))"
+set_common_vars -n "$common_run_id" "$arch" true
+telnet localhost "$common_gem5_telnet_port"
--- a/2
+++ b/2
@@ -12,5 +12,5 @@ done
 set_common_vars "$arch" false
 ./qemu/scripts/simpletrace.py \
  "${build_dir}/host-qemu-custom/trace-events-all" \
-  "${qemu_out_dir}/trace.bin" \
+  "${common_qemu_run_dir}/trace.bin" \
 > "${common_trace_txt_file}"
--- a/24
+++ b/24
@@ -34,7 +34,7 @@ trace_enabled=false
 # just to prevent QEMU from emitting a warning that '' is not valid.
 trace_type=pr_manager_run
 vnc=
-while getopts a:c:DdE:e:F:f:G:ghIiKkL:M:m:PT:U:uVx OPT; do
+while getopts a:c:DdE:e:F:f:G:ghIiKkL:M:m:n:PT:U:uVx OPT; do
  case "$OPT" in
    a)
      arch="$OPTARG"
@@ -87,8 +87,6 @@ while getopts a:c:DdE:e:F:f:G:ghIiKkL:M:m:PT:U:uVx OPT; do
      extra_append="$extra_append kgdbwait"
      # For those who want to try KDB.
      #extra_append="$extra_append kgdbwait kgdboc=kbd"
-      extra_flags_qemu="$extra_flags_qemu -serial tcp::1234,server,nowait \\
-"
      kgdb=true
      ;;
    L)
@@ -100,6 +98,9 @@ while getopts a:c:DdE:e:F:f:G:ghIiKkL:M:m:PT:U:uVx OPT; do
    m)
      memory="$OPTARG"
      ;;
+    n)
+      common_run_id="$OPTARG"
+      ;;
    P)
      prebuilt=true
      ;;
@@ -127,7 +128,7 @@ while getopts a:c:DdE:e:F:f:G:ghIiKkL:M:m:PT:U:uVx OPT; do
 done
 shift "$(($OPTIND - 1))"
 extra_flags="$extra_flags $@"
-set_common_vars -L "$common_linux_variant" -M "$common_gem5_variant" "$arch" "$gem5"
+set_common_vars -L "$common_linux_variant" -M "$common_gem5_variant" -n "$common_run_id" "$arch" "$gem5"
 if "$debug" && "$kvm"; then
  echo 'error: -d and -K are incompatible' 1>&2
  exit 1
@@ -202,6 +203,7 @@ ${extra_flags} \
 "
  fi
 else
+  mkdir -p "$common_qemu_run_dir"
  if [ -z "$debug_vm" ]; then
    serial_monitor='-serial mon:stdio \
 '
@@ -211,6 +213,10 @@ else
  if "$kvm"; then
    extra_flags="${extra_flags} -enable-kvm"
  fi
+  if "$kgdb"; then
+    extra_flags_qemu="$extra_flags_qemu -serial 'tcp::${common_gdb_port},server,nowait' \\
+"
+  fi
  if "$prebuilt"; then
    common_mkdir
    qemu_executable="qemu-system-${arch}"
@@ -222,14 +228,14 @@ else
 ${debug_vm} \
 ${qemu_executable} \\
 -device rtl8139,netdev=net0 \\
-gdb tcp::1234 \\
+-gdb 'tcp::${common_gdb_port}' \\
 -kernel '${common_linux_image}' \\
 -m '${memory}' \\
-monitor telnet::45454,server,nowait \\
-netdev user,hostfwd=tcp::45455-:45455,hostfwd=tcp::45456-:22,id=net0 \\
+-monitor 'telnet::${common_qemu_monitor_port},server,nowait' \\
+-netdev 'user,hostfwd=tcp::${common_qemu_hostfwd_generic_port}-:${common_qemu_hostfwd_generic_port},hostfwd=tcp::${common_qemu_hostfwd_ssh_port}-:22,id=net0' \\
 ${serial_monitor} \
 -smp '${cpus}' \\
-trace 'enable=${trace_type},file=${qemu_out_dir}/trace.bin' \\
+-trace 'enable=${trace_type},file=${common_qemu_run_dir}/trace.bin' \\
 -virtfs 'local,path=${p9_dir},mount_tag=host_scratch,security_model=mapped,id=host_scratch' \\
 -virtfs 'local,path=${buildroot_out_dir}/build,mount_tag=host_out,security_model=mapped,id=host_out' \\
 ${vnc}"
@@ -318,4 +324,4 @@ if "$tmux"; then
    eval "./tmu ./rungdb -a '${arch} -L ${common_linux_variant}' ${tmux_args}"
  fi
 fi
-"${root_dir}/eeval" "$cmd" "${common_out_run_dir}/run.sh"
+"${root_dir}/eeval" "$cmd" "${common_gem5_run_dir}/run.sh"
--- a/run-usage.adoc
+++ b/run-usage.adoc
@@ -41,6 +41,7 @@
                      The default is the minimum amount that boots all archs without extra
                      options added. Anything lower will lead some arch to fail to boot.
                      Any
+|`-n` |              |Run ID.
 |`-P` |              |Run the downloaded prebuilt images.
 |`-T` |`TRACE_TYPES` |Set trace events to be enabled.
                      If not given, gem5 tracing is completely disabled, while QEMU tracing
--- a/18
+++ b/18
@@ -7,7 +7,7 @@ before=
 lx_symbols="-ex 'lx-symbols ../kernel_module-1.0/' \\
 "
 kgdb=false
-while getopts A:a:b:gkL:X OPT; do
+while getopts A:a:b:gkL:n:X OPT; do
  case "$OPT" in
    A)
      after="$OPTARG"
@@ -27,6 +27,9 @@ while getopts A:a:b:gkL:X OPT; do
    L)
      common_linux_variant="$OPTARG"
      ;;
+    n)
+      common_run_id="$OPTARG"
+      ;;
    X)
      lx_symbols=
      ;;
@@ -43,12 +46,7 @@ if [ "$#" -gt 0 ]; then
 else
  brk=
 fi
-if "$gem5"; then
-  port=7000
-else
-  port=1234
-fi
-set_common_vars -L "$common_linux_variant" "$arch" "$gem5"
+set_common_vars -L "$common_linux_variant" -n "$common_run_id" "$arch" "$gem5"
 gdb="${host_dir}/usr/bin/${arch}-linux-gdb \\
 ${before}"
 if "$kgdb"; then
@@ -57,7 +55,7 @@ ${gdb} \
 -q \\
 -ex 'add-auto-load-safe-path $(pwd)' \\
 -ex 'file vmlinux' \\
-ex 'target remote localhost:${port}' \\
+-ex 'target remote localhost:${common_gdb_port}' \\
 "
 else
  # ## lx-symbols
@@ -84,11 +82,11 @@ ${gdb} \
 -q \\
 -ex 'add-auto-load-safe-path $(pwd)' \\
 -ex 'file vmlinux' \\
-ex 'target remote localhost:${port}' \\
+-ex 'target remote localhost:${common_gdb_port}' \\
 ${brk} \
 -ex 'continue' \\
 ${lx_symbols} \
 "
 fi
 "${root_dir}/eeval" "cd '${common_linux_variant_dir}' && \\
-$cmd $after" "${common_out_run_dir}/rungdb.sh"
+$cmd $after" "${common_run_dir}/rungdb.sh"
--- a/2
+++ b/2
@@ -18,7 +18,7 @@ set_common_vars "$arch" "$gem5"
 "${host_dir}/usr/bin/${arch}-linux-gdb" \
  -q \
  -ex "set sysroot ${buildroot_out_dir}/staging" \
-  -ex 'target remote localhost:45455' \
+  -ex "target remote localhost:${common_qemu_hostfwd_generic_port}" \
  -ex 'tb main' \
  -ex 'c' \
  "${build_dir}/${executable}" \
--- a/4
+++ b/4
@@ -26,6 +26,6 @@ else
  echo "instructions $(wc -l "${common_trace_txt_file}" | cut -d' ' -f1)"
  entry_addr=$("${root_dir}/runtc" readelf -h "${build_dir}/linux-custom/vmlinux" | grep 'Entry point address' | sed -E 's/.*: *//')
  echo "entry_address ${entry_addr}"
-  sed "/${entry_addr}/q" "${common_trace_txt_file}" >"${qemu_out_dir}/trace-boot.txt"
-  echo "instructions_firmware $(wc -l "${qemu_out_dir}/trace-boot.txt" | cut -d' ' -f1)"
+  sed "/${entry_addr}/q" "${common_trace_txt_file}" >"${common_qemu_run_dir}/trace-boot.txt"
+  echo "instructions_firmware $(wc -l "${common_qemu_run_dir}/trace-boot.txt" | cut -d' ' -f1)"
 fi
--- a/2
+++ b/2
@@ -24,4 +24,4 @@ kernel_dir="${build_dir}/linux-custom"
  xargs "${host_dir}/bin/${arch}-linux-addr2line" -e "${kernel_dir}/vmlinux" -fp | \
  sed -E "s|at ${kernel_dir}/(\./\|)||" | \
  uniq -c \
-> "${common_out_run_dir}/trace-lines.txt"
+> "${common_run_dir}/trace-lines.txt"