diff --git a/README.adoc b/README.adoc index 90ab3b2..078ecf6 100644 --- a/README.adoc +++ b/README.adoc @@ -1129,6 +1129,74 @@ ____ `finit` is newer and was added only in v3.8. More rationale: https://lwn.net/Articles/519010/ +=== Simultaneous runs + +When doing long simulations, it becomes fundamental to do multiple simulations in parallel. + +This is specially true for gem5, which runs much slower than QEMU, and cannot use multiple host cores to speed up the simulation: https://github.com/cirosantilli-work/gem5-issues/issues/15 + +First shell: + +.... +./run +.... + +Another shell: + +.... +./run -n 1 +.... + +The default run id is `0`. + +This method also allows us to keep run outputs in separate directories for later inspection, e.g.: + +.... +./run -aA -g -n 0 +./run -aA -g -n 1 +.... + +produces two separate `m5out` directories: + +.... +less out/aarch64/gem5/default/0/m5out +less out/aarch64/gem5/default/1/m5out +.... + +You can also add a prefix to the build ID before a period: + +.... +./run -aA -g -n some-experiment.1 +.... + +which then uses the output directory: + +.... +less out/aarch64/gem5/default/some-experiment.1/m5out +.... + +and makes it easier to remember afterwards which directory contains what. + +However this still takes up the same ports as: + +.... +./run -aA -g -n 1 +.... + +so you cannot run both at the same time. + +Like <>, you will need to pass the `-n` option to anything that needs to know runtime information, e.g. <>: + +.... +./run -n 1 +./rungdb -n 1 +.... + +Implementation note: we create multiple namespaces for two things: + +* ports: gem5 automatically increments ports until it finds a free one, but QEMU does not, and just crashes +* run output directory + [[gdb]] == GDB step debug @@ -1600,7 +1668,7 @@ One possibility is to run: and then find the second address (the first one does not work, already too late maybe): .... -less ./out/arm/qemu/trace.txt +less ./out/arm/qemu/0/trace.txt .... and break there: @@ -2482,7 +2550,7 @@ To enable initrd instead of the default ext2 disk image, do: Notice how it boots fine, even though this leads to not giving QEMU the `-drive` option, as can be verified with: .... -cat ./out/x86_64/qemu/run.sh +cat ./out/x86_64/qemu/0/run.sh .... Also as expected, there is no filesystem persistency, since we are doing everything in memory: @@ -5069,7 +5137,13 @@ The most interesting are events which show instructions that QEMU ran, for which You can then inspect the instructions with: .... -less ./out/x86_64/qemu/trace.txt +less ./out/x86_64/qemu/0/trace.txt +.... + +Get the list of available trace events: + +.... +./run -T help .... Enable other specific trace events: @@ -5077,13 +5151,7 @@ Enable other specific trace events: .... ./run -T trace1,trace2 ./qemu-trace2txt -a "$arch" -less ./out/x86_64/qemu/trace.txt -.... - -Get the list of available trace events: - -.... -./run -T help +less ./out/x86_64/qemu/0/trace.txt .... This functionality relies on the following setup: @@ -5115,7 +5183,7 @@ We can further use Binutils' `addr2line` to get the line that corresponds to eac .... ./trace-boot -a x86_64 && ./trace2line -a x86_64 -less ./out/x86_64/qemu/trace-lines.txt +less ./out/x86_64/qemu/0/trace-lines.txt .... The format is as follows: @@ -5180,7 +5248,7 @@ But it also provides a tracing mechanism documented at: link:http://www.gem5.org .... ./run -a aarch64 -E 'm5 exit' -g -T Exec -less out/aarch64/gem5/m5out/trace.txt +less out/aarch64/gem5/default/0/m5out/trace.txt .... List all available debug flags: @@ -5405,7 +5473,7 @@ Now you can play a fun little game with your friends: To find out why your program is slow, a good first step is to have a look at the statistics for the run: .... -cat out/aarch64/gem5/m5out/stats.txt +cat out/aarch64/gem5/default/0/m5out/stats.txt .... Whenever we run `m5 dumpstats` or `m5 exit`, a section with the following format is added to that file: @@ -5949,6 +6017,8 @@ In the guest, wait for the boot to end and run: m5 checkpoint .... +where <> is a guest utility present inside the gem5 tree which we cross-compiled and installed into the guest. + To restore the checkpoint, kill the VM and run: .... @@ -5990,19 +6060,11 @@ Then there is no need to pass the kernel command line again to gem5 for replay: since boot has already happened, and the parameters are already in the RAM of the snapshot. -Our scripts "namespace" with the checkpoint by architecture with `--checkpoint-dir`, so if you make two checkpoints: +Checkpoints are stored inside the `m5out` directory at: -* one in x86 -* the other in arm - -Then you would still restore both of them with `-- -r 1`. - -This makes it easier to remember which checkpoint is which, especially since there appears to be no runtime way to set the checkpoint names. - -Internals: - -* the checkpoints are stored under `out/$arch/gem5/m5out/cpt.$todo_whatisthis` -* <> is a guest utility present inside the gem5 tree which we cross-compiled and installed into the guest +.... +out//gem5///m5out/cpt. +.... [[gem5-restore-new-scrip]] ==== gem5 checkpoint restore and run a different script @@ -6108,25 +6170,6 @@ Cycles instead of instructions: Otherwise the simulation runs forever by default. -=== Run multiple gem5 instances at once - -gem5 just assigns new ports if some ports are occupied, so we can do: - -.... -./run -g -# Same as ./gem5-shell 0 -./gem5-shell -.... - -And a second instance: - -.... -./run -g -./gem5-shell 1 -.... - -TODO Now we just need to network them up to have some more fun! See dist-gem5: http://publish.illinois.edu/icsl-pdgem5/ - === m5 `m5` is a guest command line utility that is installed and run on the guest. @@ -6165,7 +6208,7 @@ m5 writefile myfileguest myfilehost Host: .... -cat out/aarch64/gem5/m5out/myfilehost +cat out/aarch64/gem5/default/0/m5out/myfilehost .... Does not work for subdirectories, gem5 crashes: @@ -6545,7 +6588,7 @@ However, if the worktree already exists, we leave it untouched. + Therefore, you can safely go to that directory and edit the source there without fear that it will get deleted. + -The `wt/` branch name prefix stands for `WorkTree`, and is done to allow us to checkout to a test `some-branch` branch under `gem5/gem5` and still use `-M some-branch`, without a conflict for the worktree branch. +The `wt/` branch name prefix stands for `WorkTree`, and is done to allow us to checkout to a test `some-branch` branch under `gem5/gem5` and still use `-M some-branch`, without aconflict for the worktree branch. All build outputs end up at: `out/common/gem5/` regardless. @@ -6653,13 +6696,13 @@ Stdout shows a line with the full command of type: and this line is also saved to a file for convenience: .... -cat ./out/arm/qemu/run.sh +cat ./out/arm/qemu/0/run.sh .... or for gem5: .... -cat ./out/arm/gem5/run.sh +cat ./out/arm/gem5/default/0/run.sh .... Next, you will also want to give the relevant images to save them time. Zip the images with: @@ -7073,13 +7116,16 @@ The action seems to be happening at: `hw/arm/virt.c`. **** `out//buildroot/build/linux-custom`: symlink to a variant, custom madness that we do on top of Buildroot: <> **** `out//buildroot/build/linux-custom.`: what `linux-custom` points to *** `out//qemu`: QEMU runtime outputs +*** `out//qemu//run.sh`: full CLI used to run QEMU. See: <> +*** `out//gem5///`: gem5 runtime outputs +**** `out//gem5///m5out` +**** `out//gem5///run.sh`: full CLI used to run gem5. See: <> ** `out/common`: cross arch outputs, for when we can gain a lot of time and space by sharing things that are common across different archs. *** `out/common/dl/`: Buildroot caches downloaded source there due to `BR2_DL_DIR` *** `out/common/gem5/`: `arm` and `aarch64` have the same build. -**** `out/common/gem5//` -***** `out/common/gem5//build/` -***** `out/common/gem5//m5out` -***** `out/common/gem5//system` +**** `out/common/gem5//`: gem5 build output. In common to share the ARM and aarch64 builds. +***** `out/common/gem5//build/`: main build outputs, including the `gem5.opt` executable and object files +***** `out/common/gem5//system/`: `M5_PATH` directory, with DTBs and bootloaders ==== buildroot_patches diff --git a/common b/common index f6b9b3a..cc6b7a3 100644 --- a/common +++ b/common @@ -23,7 +23,7 @@ set_common_vars() { linux_variant= gem5_variant= OPTIND=1 - while getopts L:M: OPT; do + while getopts L:M:n: OPT; do case "$OPT" in L) linux_variant="$OPTARG" @@ -31,6 +31,9 @@ set_common_vars() { M) gem5_variant="$OPTARG" ;; + n) + common_run_id="$OPTARG" + ;; esac done shift "$(($OPTIND - 1))" @@ -65,7 +68,7 @@ set_common_vars() { build_dir="${buildroot_out_dir}/build" common_images_dir="${buildroot_out_dir}/images" host_dir="${buildroot_out_dir}/host" - qemu_out_dir="${out_arch_dir}/qemu" + common_qemu_run_dir="${out_arch_dir}/qemu/${common_run_id}" common_linux_custom_dir="${build_dir}/linux-custom" common_linux_variant_dir="${common_linux_custom_dir}.${linux_variant}" common_vmlinux="${common_linux_variant_dir}/vmlinux" @@ -77,14 +80,14 @@ set_common_vars() { common_gem5_out_dir="${common_dir}/gem5/${gem5_variant}" common_gem5_build_dir="${common_gem5_out_dir}/build" common_gem5_system_dir="${common_gem5_out_dir}/system" - common_m5out_dir="${common_gem5_out_dir}/m5out" + common_gem5_run_dir="${out_arch_dir}/gem5/${gem5_variant}/${common_run_id}" + common_m5out_dir="${common_gem5_run_dir}/m5out" if "$gem5"; then - common_out_run_dir="$common_gem5_out_dir" - common_trace_txt_file="${common_m5out_dir}/trace.txt" + common_run_dir="$common_gem5_run_dir" else - common_out_run_dir="$qemu_out_dir" - common_trace_txt_file="${common_out_run_dir}/trace.txt" + common_run_dir="$common_qemu_run_dir" fi + common_trace_txt_file="${common_run_dir}/trace.txt" case "$arch" in arm) common_linux_image=arch/arm/boot/zImage @@ -100,12 +103,27 @@ set_common_vars() { ;; esac common_linux_image="${common_linux_variant_dir}/${common_linux_image}" + + # Ports. + common_run_id_number="$(echo "$common_run_id" | cut -d . -f 2)" + if "$gem5"; then + common_gem5_telnet_port="$((3456 + $common_run_id_number))" + common_gdb_port="$((7000 + $common_run_id_number))" + else + common_qemu_base_port="$((45454 + 10 * $common_run_id_number))" + common_qemu_monitor_port="$(($common_qemu_base_port + 0))" + common_qemu_hostfwd_generic_port="$(($common_qemu_base_port + 1))" + common_qemu_hostfwd_ssh_port="$(($common_qemu_base_port + 2))" + common_qemu_gdb_port="$(($common_qemu_base_port + 3))" + common_gdb_port="$common_qemu_gdb_port" + fi } common_mkdir() ( mkdir -p \ "$build_dir" \ "$common_gem5_out_dir" \ - "$qemu_out_dir" \ + "$common_gem5_run_dir" \ + "$common_qemu_run_dir" \ "$p9_dir" \ ; ) @@ -120,6 +138,7 @@ common_dir="${out_dir}/common" common_gem5_default_src_dir="${root_dir}/gem5/gem5" common_gem5_non_default_src_root_dir="${data_dir}/gem5" common_gem5_variant=default +common_run_id=0 f="${data_dir}/cli" if [ -f "$f" ]; then . "$f" diff --git a/eeval b/eeval index 4ab747c..d913261 100755 --- a/eeval +++ b/eeval @@ -1,5 +1,6 @@ #!/usr/bin/env bash # echo and eval +set -e a= while getopts a OPT; do case "$OPT" in @@ -15,6 +16,7 @@ done shift "$(($OPTIND - 1))" cmd="$1" outfile="${2:-/dev/null}" +mkdir -p "$(dirname "$outfile")" echo "$cmd" | tee $a "$outfile" eval "$cmd" exit "$?" diff --git a/gem5-bench-cache b/gem5-bench-cache index 51dcaf7..65cc1a5 100755 --- a/gem5-bench-cache +++ b/gem5-bench-cache @@ -19,7 +19,7 @@ set_common_vars "$arch" true cmd="./run -a $arch -g" cache_small='--caches --l2cache --l1d_size=1024 --l1i_size=1024 --l2_size=1024 --l3_size=1024 ' cache_large='--caches --l2cache --l1d_size=1024kB --l1i_size=1024kB --l2_size=1024kB --l3_size=1024kB' -results_file="${common_gem5_out_dir}/bench-cache.txt" +results_file="${common_gem5_run_dir}/bench-cache.txt" bench() ( common_bench_cmd "$1" "$results_file" diff --git a/gem5-shell b/gem5-shell index 29355b6..dd81350 100755 --- a/gem5-shell +++ b/gem5-shell @@ -1,4 +1,15 @@ #!/usr/bin/env bash -offset="${1:-0}" -base=3456 -telnet localhost $(($base + $offset)) +. common +while getopts n: OPT; do + case "$OPT" in + n) + common_run_id="$OPTARG" + ;; + ?) + exit 2 + ;; + esac +done +shift "$(($OPTIND - 1))" +set_common_vars -n "$common_run_id" "$arch" true +telnet localhost "$common_gem5_telnet_port" diff --git a/qemu-trace2txt b/qemu-trace2txt index 19196f5..b847b39 100755 --- a/qemu-trace2txt +++ b/qemu-trace2txt @@ -12,5 +12,5 @@ done set_common_vars "$arch" false ./qemu/scripts/simpletrace.py \ "${build_dir}/host-qemu-custom/trace-events-all" \ - "${qemu_out_dir}/trace.bin" \ + "${common_qemu_run_dir}/trace.bin" \ > "${common_trace_txt_file}" diff --git a/run b/run index 7a80345..e58d9c3 100755 --- a/run +++ b/run @@ -34,7 +34,7 @@ trace_enabled=false # just to prevent QEMU from emitting a warning that '' is not valid. trace_type=pr_manager_run vnc= -while getopts a:c:DdE:e:F:f:G:ghIiKkL:M:m:PT:U:uVx OPT; do +while getopts a:c:DdE:e:F:f:G:ghIiKkL:M:m:n:PT:U:uVx OPT; do case "$OPT" in a) arch="$OPTARG" @@ -87,8 +87,6 @@ while getopts a:c:DdE:e:F:f:G:ghIiKkL:M:m:PT:U:uVx OPT; do extra_append="$extra_append kgdbwait" # For those who want to try KDB. #extra_append="$extra_append kgdbwait kgdboc=kbd" - extra_flags_qemu="$extra_flags_qemu -serial tcp::1234,server,nowait \\ -" kgdb=true ;; L) @@ -100,6 +98,9 @@ while getopts a:c:DdE:e:F:f:G:ghIiKkL:M:m:PT:U:uVx OPT; do m) memory="$OPTARG" ;; + n) + common_run_id="$OPTARG" + ;; P) prebuilt=true ;; @@ -127,7 +128,7 @@ while getopts a:c:DdE:e:F:f:G:ghIiKkL:M:m:PT:U:uVx OPT; do done shift "$(($OPTIND - 1))" extra_flags="$extra_flags $@" -set_common_vars -L "$common_linux_variant" -M "$common_gem5_variant" "$arch" "$gem5" +set_common_vars -L "$common_linux_variant" -M "$common_gem5_variant" -n "$common_run_id" "$arch" "$gem5" if "$debug" && "$kvm"; then echo 'error: -d and -K are incompatible' 1>&2 exit 1 @@ -202,6 +203,7 @@ ${extra_flags} \ " fi else + mkdir -p "$common_qemu_run_dir" if [ -z "$debug_vm" ]; then serial_monitor='-serial mon:stdio \ ' @@ -211,6 +213,10 @@ else if "$kvm"; then extra_flags="${extra_flags} -enable-kvm" fi + if "$kgdb"; then + extra_flags_qemu="$extra_flags_qemu -serial 'tcp::${common_gdb_port},server,nowait' \\ +" + fi if "$prebuilt"; then common_mkdir qemu_executable="qemu-system-${arch}" @@ -222,14 +228,14 @@ else ${debug_vm} \ ${qemu_executable} \\ -device rtl8139,netdev=net0 \\ --gdb tcp::1234 \\ +-gdb 'tcp::${common_gdb_port}' \\ -kernel '${common_linux_image}' \\ -m '${memory}' \\ --monitor telnet::45454,server,nowait \\ --netdev user,hostfwd=tcp::45455-:45455,hostfwd=tcp::45456-:22,id=net0 \\ +-monitor 'telnet::${common_qemu_monitor_port},server,nowait' \\ +-netdev 'user,hostfwd=tcp::${common_qemu_hostfwd_generic_port}-:${common_qemu_hostfwd_generic_port},hostfwd=tcp::${common_qemu_hostfwd_ssh_port}-:22,id=net0' \\ ${serial_monitor} \ -smp '${cpus}' \\ --trace 'enable=${trace_type},file=${qemu_out_dir}/trace.bin' \\ +-trace 'enable=${trace_type},file=${common_qemu_run_dir}/trace.bin' \\ -virtfs 'local,path=${p9_dir},mount_tag=host_scratch,security_model=mapped,id=host_scratch' \\ -virtfs 'local,path=${buildroot_out_dir}/build,mount_tag=host_out,security_model=mapped,id=host_out' \\ ${vnc}" @@ -318,4 +324,4 @@ if "$tmux"; then eval "./tmu ./rungdb -a '${arch} -L ${common_linux_variant}' ${tmux_args}" fi fi -"${root_dir}/eeval" "$cmd" "${common_out_run_dir}/run.sh" +"${root_dir}/eeval" "$cmd" "${common_gem5_run_dir}/run.sh" diff --git a/run-usage.adoc b/run-usage.adoc index 0a7debd..d39e7fc 100644 --- a/run-usage.adoc +++ b/run-usage.adoc @@ -41,6 +41,7 @@ The default is the minimum amount that boots all archs without extra options added. Anything lower will lead some arch to fail to boot. Any +|`-n` | |Run ID. |`-P` | |Run the downloaded prebuilt images. |`-T` |`TRACE_TYPES` |Set trace events to be enabled. If not given, gem5 tracing is completely disabled, while QEMU tracing diff --git a/rungdb b/rungdb index a1411b0..a2d91da 100755 --- a/rungdb +++ b/rungdb @@ -7,7 +7,7 @@ before= lx_symbols="-ex 'lx-symbols ../kernel_module-1.0/' \\ " kgdb=false -while getopts A:a:b:gkL:X OPT; do +while getopts A:a:b:gkL:n:X OPT; do case "$OPT" in A) after="$OPTARG" @@ -27,6 +27,9 @@ while getopts A:a:b:gkL:X OPT; do L) common_linux_variant="$OPTARG" ;; + n) + common_run_id="$OPTARG" + ;; X) lx_symbols= ;; @@ -43,12 +46,7 @@ if [ "$#" -gt 0 ]; then else brk= fi -if "$gem5"; then - port=7000 -else - port=1234 -fi -set_common_vars -L "$common_linux_variant" "$arch" "$gem5" +set_common_vars -L "$common_linux_variant" -n "$common_run_id" "$arch" "$gem5" gdb="${host_dir}/usr/bin/${arch}-linux-gdb \\ ${before}" if "$kgdb"; then @@ -57,7 +55,7 @@ ${gdb} \ -q \\ -ex 'add-auto-load-safe-path $(pwd)' \\ -ex 'file vmlinux' \\ --ex 'target remote localhost:${port}' \\ +-ex 'target remote localhost:${common_gdb_port}' \\ " else # ## lx-symbols @@ -84,11 +82,11 @@ ${gdb} \ -q \\ -ex 'add-auto-load-safe-path $(pwd)' \\ -ex 'file vmlinux' \\ --ex 'target remote localhost:${port}' \\ +-ex 'target remote localhost:${common_gdb_port}' \\ ${brk} \ -ex 'continue' \\ ${lx_symbols} \ " fi "${root_dir}/eeval" "cd '${common_linux_variant_dir}' && \\ -$cmd $after" "${common_out_run_dir}/rungdb.sh" +$cmd $after" "${common_run_dir}/rungdb.sh" diff --git a/rungdbserver b/rungdbserver index 5e26bda..58dafe1 100755 --- a/rungdbserver +++ b/rungdbserver @@ -18,7 +18,7 @@ set_common_vars "$arch" "$gem5" "${host_dir}/usr/bin/${arch}-linux-gdb" \ -q \ -ex "set sysroot ${buildroot_out_dir}/staging" \ - -ex 'target remote localhost:45455' \ + -ex "target remote localhost:${common_qemu_hostfwd_generic_port}" \ -ex 'tb main' \ -ex 'c' \ "${build_dir}/${executable}" \ diff --git a/trace-boot b/trace-boot index ee94389..d3b84b8 100755 --- a/trace-boot +++ b/trace-boot @@ -26,6 +26,6 @@ else echo "instructions $(wc -l "${common_trace_txt_file}" | cut -d' ' -f1)" entry_addr=$("${root_dir}/runtc" readelf -h "${build_dir}/linux-custom/vmlinux" | grep 'Entry point address' | sed -E 's/.*: *//') echo "entry_address ${entry_addr}" - sed "/${entry_addr}/q" "${common_trace_txt_file}" >"${qemu_out_dir}/trace-boot.txt" - echo "instructions_firmware $(wc -l "${qemu_out_dir}/trace-boot.txt" | cut -d' ' -f1)" + sed "/${entry_addr}/q" "${common_trace_txt_file}" >"${common_qemu_run_dir}/trace-boot.txt" + echo "instructions_firmware $(wc -l "${common_qemu_run_dir}/trace-boot.txt" | cut -d' ' -f1)" fi diff --git a/trace2line b/trace2line index 84fe21e..f635e73 100755 --- a/trace2line +++ b/trace2line @@ -24,4 +24,4 @@ kernel_dir="${build_dir}/linux-custom" xargs "${host_dir}/bin/${arch}-linux-addr2line" -e "${kernel_dir}/vmlinux" -fp | \ sed -E "s|at ${kernel_dir}/(\./\|)||" | \ uniq -c \ -> "${common_out_run_dir}/trace-lines.txt" +> "${common_run_dir}/trace-lines.txt"