getting started: hide the initial build under ./build

make build awesomer and more generic, convert to python

rename ./configure to ./download-dependencies, since it wasn't configuring
anything

.travis.yml

@@ -4,14 +4,14 @@ sudo: required
 
 install: |
   cd "$TRAVIS_BUILD_DIR"
-  bash -x ./configure --travis
+  bash -x ./download-dependencies --travis
 
 script: |
   cd "$TRAVIS_BUILD_DIR"
   # --nproc: I'm unable to install nproc on Travis.
   # TODO why? Is part of coreutils in Ubuntu 16.04:
   # http://manpages.ubuntu.com/manpages/trusty/man1/nproc.1.html
-  # which ./configure is installing.
+  # which ./download-dependencies is installing.
   #
   # awk: without it, too much stdout (4Mb max)
   # If we ignore stdout: Travis kills job because it spent

README.adoc

@@ -72,12 +72,8 @@ Reserve 12Gb of disk and run:
 ....
 git clone https://github.com/cirosantilli/linux-kernel-module-cheat
 cd linux-kernel-module-cheat
-./configure --qemu && \
+./download-dependencies
-./build-qemu       && \
+./build
-./build-linux      && \
-./build-modules    && \
-./build-userland   && \
-./build-buildroot  && \
 ./run
 ....
 
@@ -95,7 +91,7 @@ If you don't want to wait, you could also try the following faster but much more
 
 but you will soon find that they are simply not enough if you anywhere near serious about systems programming.
 
-If `./configure` fails with:
+If `./download-dependencies` fails with:
 
 ....
 E: You must put some 'source' URIs in your sources.list
@@ -103,15 +99,7 @@ E: You must put some 'source' URIs in your sources.list
 
 see this: https://askubuntu.com/questions/496549/error-you-must-put-some-source-uris-in-your-sources-list/857433#857433 I don't know how to automate this step. Why, Ubuntu, why.
 
-It does not work if you just download the `.zip` from GitHub because we use link:.gitmodules[Git submodules], you must clone this repo. `./configure` then fetches only the required submodules for you.
+It does not work if you just download the `.zip` from GitHub because we use link:.gitmodules[Git submodules], you must clone this repo. `./download-dependencies` then fetches only the required submodules for you.
-
-The order of build commands matters:
-
-* `./build-linux` must come before `./build-modules` because the kernel modules depend on the Linux kernel build. We could lessen this need by calling `make modules_prepare` on the kernel tree, which does not require a full build, but this is not currently done
-* `./build-modules` and `./build-userland` must come before `./build-buildroot` because generate files that will be placed in the root filesystem. If you don't call them before, the generated files will not be in the root filesystem.
-* `build-qemu` must come before `./build-buildroot` because it builds the `qemu-img` tool that we use to convert the raw disk image into link:https://en.wikipedia.org/wiki/Qcow[qcow2] format that QEMU boots from in our setup
-
-If you mess up the order, just build things again in the right order and you will be fine.
 
 After `./run`, QEMU opens up and you can start playing with the kernel modules inside the simulated system:
 
@@ -243,6 +231,16 @@ Then rebuild the Linux kernel, quit QEMU and reboot the modified kernel:
 
 and, surely enough, your message has appeared at the beginning of the boot.
 
+We could have used just `./build` as in the initial build, but doing just `./build-linux` will save us a bit of time.
+
+The link:build[`./build`] script is just a lightweight wrapper, but when you start modifying components such as the Linux kernel, it is better to run individual steps directly.
+
+You can see that `./build` does `./build-linux` by running:
+
+....
+./build --dry-run
+....
+
 So you are now officially a Linux kernel hacker, way to go!
 
 ===== Your first kernel module hack
@@ -285,7 +283,7 @@ The safe way, is to fist quit QEMU, then rebuild the modules, root filesystem, a
 
 `./build-buildroot` generates the root filesystem with the modules that we compiled at `./build-modules`.
 
-`--eval-busybox` is optional: you could just type `insmod /hello.ko` in the terminal, but this makes it run automatically at the end of boot.
+`--eval-busybox` is optional: you could just type `insmod /hello.ko` in the terminal, but this makes it run automatically at the end of boot, and then drops you into a shell.
 
 If the guest and host are the same arch, typically x86_64, you can speed up boot further with <<kvm>>:
 
@@ -371,8 +369,6 @@ All of this makes QEMU the natural choice of default system simulator.
 
 === gem5 Buildroot setup
 
-TODO document kernel modules don't work on 9p
-
 ==== About the gem5 Buildroot setup
 
 This setup is like the <<qemu-buildroot-setup>>, but it uses link:http://gem5.org/[gem5] instead of QEMU as a system simulator.
@@ -393,16 +389,26 @@ See <<gem5-vs-qemu>> for a more thorough comparison.
 
 ==== gem5 Buildroot setup getting started
 
-For the most part, if you just add the `--gem5` option or `*-gem5` suffix to all commands and everything should magically work:
+For the most part, if you just add the `--gem5` option or `*-gem5` suffix to all commands and everything should magically work.
+
+If you haven't built Buildroot yet for <<qemu-buildroot-setup>>, you can build from the beginning with:
 
 ....
-./configure --gem5
+./download-dependencies --gem5
-./build-gem5
+./build --gem5 --no-qemu
 ./build-buildroot --gem5
 ./run --gem5
 ....
 
-If you have already built <<qemu-buildroot-setup>> previously, don't be afraid, gem5 and QEMU use almost the same root filesystem and kernel, so `./build-buildroot --gem` will be fast. It is currently only needed for the <<m5>> tool.
+`--no-qemu` is optional, but it makes the 
+....
+./download-dependencies --gem5
+./build-gem5 --gem5
+./build-buildroot --gem5
+./run --gem5
+....
+
+If you have already built  previously, don't be afraid: gem5 and QEMU use almost the same root filesystem and kernel, so `./build-buildroot --gem` will be fast. It is currently only needed for the <<m5>> tool.
 
 To get a terminal, either open a new shell and run:
 
@@ -802,7 +808,7 @@ Our C bare-metal compiler is built with link:https://github.com/crosstool-ng/cro
 QEMU:
 
 ....
-./configure --baremetal --qemu
+./download-dependencies --baremetal --qemu
 ./build-qemu --arch arm
 ./build-crosstool-ng --arch arm
 ./build-baremetal --arch arm
@@ -857,7 +863,7 @@ To use gem5 instead of QEMU do:
 
 ....
 patch -d "$(./getvar gem5_src_dir)" -p 1 < patches/manual/gem5-semihost.patch
-./configure --baremetal --gem5
+./download-dependencies --baremetal --gem5
 ./build-gem5 --arch arm
 ./build-crosstool-ng --arch arm
 ./build-baremetal --arch arm --gem5
@@ -8007,7 +8013,7 @@ less "$(./getvar -a "$arch" run_dir)/trace.txt"
 
 This functionality relies on the following setup:
 
-* `./configure --enable-trace-backends=simple`. This logs in a binary format to the trace file.
+* `./download-dependencies --enable-trace-backends=simple`. This logs in a binary format to the trace file.
 +
 It makes 3x execution faster than the default trace backend which logs human readable data to stdout.
 +
@@ -8747,7 +8753,7 @@ There are two ways to run PARSEC with this repo:
 ====== PARSEC benchmark without parsecmgmt
 
 ....
-./configure --gem5 --parsec-benchmark
+./download-dependencies --gem5 --parsec-benchmark
 ./build-buildroot --arch arm --buildroot-config 'BR2_PACKAGE_PARSEC_BENCHMARK=y' --gem5
 ./run --arch arm --gem5
 ....
@@ -10189,7 +10195,7 @@ CT_GDB_CROSS_SIM=y
 which by grepping crosstool-NG we can see does on GDB:
 
 ....
-./configure --enable-sim
+./download-dependencies --enable-sim
 ....
 
 Those are not set by default on `gdb-multiarch` in Ubuntu 16.04.
@@ -10384,7 +10390,7 @@ We tried to automate it on Travis with link:.travis.yml[] but it hits the curren
 Benchmark all:
 
 ....
-./build-all
+./build --all-linux-components
 ./bench-boot
 cat "$(./getvar bench_boot)"
 ....
@@ -10464,7 +10470,7 @@ Kernel panic - not syncing: Attempted to kill the idle task!
 
 ==== Benchmark builds
 
-The build times are calculated after doing `./configure` and link:https://buildroot.org/downloads/manual/manual.html#_offline_builds[`make source`], which downloads the sources, and basically benchmarks the <<benchmark-internets,Internet>>.
+The build times are calculated after doing `./download-dependencies` and link:https://buildroot.org/downloads/manual/manual.html#_offline_builds[`make source`], which downloads the sources, and basically benchmarks the <<benchmark-internets,Internet>>.
 
 Sample build time at 2c12b21b304178a81c9912817b782ead0286d282: 28 minutes, 15 with full ccache hits. Breakdown: 19% GCC, 13% Linux kernel, 7% uclibc, 6% host-python, 5% host-qemu, 5% host-gdb, 2% host-binutils
 
@@ -11127,7 +11133,7 @@ It takes too much time to be feasible for every patch, but it should be done for
 ==== Automated tests
 
 ....
-./build-all
+./build --all-linux-components
 ./test --size 3
 echo $?
 ....
@@ -11136,7 +11142,7 @@ should output 0.
 
 Sources:
 
-* link:build-all[]
+* link:build[]
 * link:test[]
 
 Test just the kernel modules:
@@ -11307,10 +11313,10 @@ TODO also run tests and only release if they pass.
 
 ==== release-zip
 
-Create a zip containing all files required for <<prebuilt>>
+Create a zip containing all files required for <<prebuilt>>:
 
 ....
-./build-all -G
+./build --all-linux-components
 ./release-zip
 ....
 

bench-all

@@ -116,7 +116,7 @@ fi
 
 if "$bench_linux_boot"; then
   cd "${root_dir}"
-  "${root_dir}/build-all"
+  "${root_dir}/build" --all
   "${root_dir}/bench-boot" --size 3
   cp "$(${root_dir}/getvar bench_boot)" "$new_dir"
 fi

build

@@ -0,0 +1,172 @@
+#!/usr/bin/env python3
+
+import argparse
+import collections
+import os
+
+import common
+
+class Component():
+    def __init__(self, default_selected, build_callback):
+        self.default_selected = default_selected
+        self.build_callback = build_callback
+    def build(self, arch, dry_run):
+        self.build_callback(arch, dry_run)
+
+class BaremetalComponent(Component):
+    def __init__(self, default_selected):
+        self.default_selected = default_selected
+    def build(self, arch, dry_run):
+        common.run_cmd(['build-crosstool-ng', '--arch', arch], dry_run=dry_run)
+        common.run_cmd(['build-baremetal', '--arch', arch], dry_run=dry_run)
+        common.run_cmd(['build-baremetal', '--arch', arch, '--gem5'], dry_run=dry_run)
+        common.run_cmd(['build-baremetal', '--arch', arch, '--gem5', '--machine', 'RealViewPBX'], dry_run=dry_run)
+
+def add_bool_arg(parser, name, default=False):
+    group = parser.add_mutually_exclusive_group(required=False)
+    group.add_argument('--' + name, default=False, action='store_true')
+    group.add_argument('--no-' + name, default=False, action='store_true')
+
+def run_cmd(cmd, dry_run):
+    cmd_abs = cmd.copy()
+    cmd_abs[0] = os.path.join(common.root_dir, cmd[0])
+    common.run_cmd(cmd_abs, dry_run=dry_run)
+
+# Topological sorted on build dependencies.
+name_to_component_map = collections.OrderedDict([
+    ('baremetal', BaremetalComponent(False)),
+    ('gem5', Component(
+        False,
+        lambda arch, dry_run: run_cmd(['build-gem5', '--arch', arch], dry_run)
+    )),
+    ('qemu', Component(
+        True,
+        lambda arch, dry_run: run_cmd(['build-qemu', '--arch', arch], dry_run)
+    )),
+    ('linux', Component(
+        True,
+        lambda arch, dry_run: run_cmd(['build-linux', '--arch', arch], dry_run=dry_run)
+    )),
+    ('modules', Component(
+        True,
+        lambda arch, dry_run: run_cmd(['build-modules', '--arch', arch], dry_run=dry_run)
+    )),
+    ('userland', Component(
+        True,
+        lambda arch, dry_run: run_cmd(['build-userland', '--arch', arch], dry_run=dry_run),
+    )),
+    ('buildroot', Component(
+        True,
+        lambda arch, dry_run: run_cmd(['build-buildroot', '--arch', arch, '--gem5'], dry_run=dry_run),
+    )),
+])
+component_names = name_to_component_map.keys()
+linux_component_names = {
+    'gem5',
+    'qemu',
+    'linux',
+    'modules',
+    'userland',
+    'buildroot',
+}
+
+parser = argparse.ArgumentParser(
+    description= '''
+Shallow helper to build everything, or a subset of everything conveniently.
+
+While developing something however, you will likely want to just run the
+required sub-build commands manually to speed things up and better understand
+what is going on.
+
+Without any args, build only what is necessary for
+https://github.com/cirosantilli/linux-kernel-module-cheat#qemu-buildroot-setup
+
+....
+./%(prog)s
+....
+
+This includes:
+
+* QEMU
+* Linux kernel
+* kernel modules and userland tools
+* Buildroot
+
+just for x86_64.
+
+To build EVERYTHING:
+
+This will build QEMU, gem5, Buildroot, Linux, etc.
+for x86_64, arm and aarch64.
+
+With --archs, build everything for just the given archs:
+
+....
+./%(prog)s --archs 'arm aarch64'
+....
+
+Other options make this script build only the given components. E.g., to build
+just Linux and QEMU for all archs, but not gem5, Buildroot, etc.:
+
+....
+./%(prog)s --linux --qemu
+....
+
+this is useful to while developing those components to prepare to quickly
+''',
+    formatter_class=argparse.RawTextHelpFormatter,
+)
+parser.add_argument('--all', default=False, action='store_true', help='''\
+Build absolutely everything.
+''')
+parser.add_argument('--all-components', default=False, action='store_true', help='''\
+Build all components within the selected archs.
+''')
+parser.add_argument('--all-linux-components', default=False, action='store_true', help='''\
+Build all Linux-releated components within the selected archs.
+Excludes for example baremetal examples.
+''')
+group = parser.add_mutually_exclusive_group(required=False)
+group.add_argument('--all-archs', default=False, action='store_true', help='''\
+Build the selected components for all archs.
+''')
+group.add_argument('--arch', choices=common.arch_choices, default=[], action='append', help='''\
+Build the selected components for this arch. Select multiple arches by
+passing this option multiple times. Default: [{}]
+'''.format(common.default_arch))
+for component in component_names:
+    add_bool_arg(parser, component)
+parser.add_argument('--dry-run', default=False, action='store_true', help='''\
+Print the commands that would be run, but don't run them.
+''')
+args = parser.parse_args()
+
+# Decide archs.
+if args.arch == []:
+    if args.all or args.all_archs:
+        archs = common.all_archs.copy()
+    else:
+        archs = set([common.default_arch])
+else:
+    archs = set()
+    for arch in args.arch:
+        if arch in common.arch_short_to_long_dict:
+            arch = common.arch_short_to_long_dict[arch]
+        archs.add(arch)
+
+# Decide components.
+selected_component_names = []
+for name in component_names:
+    component = name_to_component_map[name]
+    if (
+        args.all or
+        args.all_components or
+        (args.all_linux_components and name in linux_component_names) or
+        getattr(args, name) or
+        component.default_selected
+    ):
+        selected_component_names.append(name)
+
+for arch in archs:
+    for name in selected_component_names:
+        name_to_component_map[name].build(arch, args.dry_run)

build-all

@@ -1,111 +0,0 @@
-#!/usr/bin/env bash
-
-# Shallow helper to build everything, or a subset of everything conveniently.
-#
-# While developing something however, you will likely want to just run the
-# required sub-build commands manually to speed things up and understand what
-# is going on.
-#
-# Without any args, build everything for all archs:
-#
-#     ./build-all
-#
-# This will build QEMU, gem5, Buildroot, Linux, etc.
-# for x86_64, arm and aarch64.
-#
-# With --archs, build everything for just the given archs:
-#
-#    ./build-all --archs 'arm aarch64'
-#
-# Other options make this script build only the given components. E.g., to build
-# just Linux and QEMU for all archs, but not gem5, Buildroot, etc.:
-#
-#    ./build-all --linux --qemu
-#
-# this is useful to while developing those components to prepare to quickly
-
-set -eu
-archs='x86_64 arm aarch64'
-baremetal=false
-buildroot=false
-gem5=false
-linux=false
-modules=false
-qemu=false
-userland=false
-while [ $# -gt 0 ]; do
-  case "$1" in
-    --archs)
-      archs="$2"
-      shift 2
-      ;;
-    --baremetal)
-      baremetal=true
-      shift
-    ;;
-    --buildroot)
-      buildroot=true
-      shift
-    ;;
-    --gem5)
-      gem5=true
-      shift
-    ;;
-    --linux)
-      linux=true
-      shift
-    ;;
-    --modules)
-      modules=true
-      shift
-    ;;
-    --qemu)
-      qemu=true
-      shift
-    ;;
-    --userland)
-      userland=true
-      shift
-    ;;
-  esac
-done
-if ! (
-    "$baremetal" || \
-    "$buildroot" || \
-    "$gem5" || \
-    "$linux" || \
-    "$qemu"
-  )
-then
-  baremetal=true
-  buildroot=true
-  gem5=true
-  linux=true
-  qemu=true
-fi
-for arch in $archs; do
-  if "$qemu"; then
-    ./build-qemu --arch "$arch"
-  fi
-  if "$linux"; then
-    ./build-linux --arch "$arch"
-  fi
-  if "$modules"; then
-    ./build-modules
-  fi
-  if "$userland"; then
-    ./build-userland
-  fi
-  if "$baremetal" && [ ! "$arch" = x86_64 ]; then
-    ./build-crosstool-ng --arch "$arch"
-    ./build-baremetal --arch "$arch"
-    ./build-baremetal --arch "$arch" --gem5
-    ./build-baremetal --arch "$arch" --gem5 --machine RealViewPBX
-  fi
-  if "$buildroot"; then
-    ./build-buildroot --arch "$arch" --gem5 --kernel-modules "$@"
-  fi
-  if "$gem5"; then
-    ./build-gem5 --arch "$arch"
-  fi
-done

build-buildroot

@@ -32,7 +32,6 @@ defaults = {
 def get_argparse():
     parser = common.get_argparse(argparse_args={'description':'Run Linux on an emulator'})
     common.add_build_arguments(parser)
-    kernel_module_group = parser.add_mutually_exclusive_group()
     parser.add_argument(
         '-B', '--buildroot-config', default=defaults['buildroot_config'], action='append',
         help='''Add a single Buildroot config to the current build.
@@ -93,7 +92,7 @@ Ignore all default kernel configurations and use this file instead.
 Still uses options explicitly passed with `-C` and `-c` on top of it.
 '''
     )
-    kernel_module_group.add_argument(
+    parser.add_argument(
         '-k', '--kernel-modules', default=defaults['kernel_modules'], action='store_true',
         help='''Reconfigure and rebuild the kernel modules package.
 Force --build-linux to true, since building the modules requires building Linux.

common.py

@@ -2,6 +2,7 @@
 
 import argparse
 import base64
+import collections
 import copy
 import datetime
 import distutils.file_util
@@ -44,12 +45,17 @@ qemu_src_dir = os.path.join(submodules_dir, 'qemu')
 parsec_benchmark_src_dir = os.path.join(submodules_dir, 'parsec-benchmark')
 ccache_dir = os.path.join('/usr', 'lib', 'ccache')
 default_build_id = 'default'
-arch_map = {
+arch_short_to_long_dict = collections.OrderedDict([
-    'a': 'arm',
+    ('x', 'x86_64'),
-    'A': 'aarch64',
+    ('a', 'arm'),
-    'x': 'x86_64',
+    ('A', 'aarch64'),
-}
+])
-arches = [arch_map[k] for k in arch_map]
+all_archs = [arch_short_to_long_dict[k] for k in arch_short_to_long_dict]
+arch_choices = []
+for key in this.arch_short_to_long_dict:
+    arch_choices.append(key)
+    arch_choices.append(this.arch_short_to_long_dict[key])
+default_arch = 'x86_64'
 gem5_cpt_prefix = '^cpt\.'
 sha = subprocess.check_output(['git', '-C', root_dir, 'log', '-1', '--format=%H']).decode().rstrip()
 release_dir = os.path.join(this.out_dir, 'release')
@@ -112,16 +118,12 @@ def get_argparse(default_args=None, argparse_args=None):
         default_args = {}
     if argparse_args is None:
         argparse_args = {}
-    arch_choices = []
-    for key in this.arch_map:
-        arch_choices.append(key)
-        arch_choices.append(this.arch_map[key])
     parser = argparse.ArgumentParser(
         formatter_class=argparse.RawTextHelpFormatter,
         **argparse_args
     )
     parser.add_argument(
-        '-a', '--arch', choices=arch_choices, default='x86_64',
+        '-a', '--arch', choices=this.arch_choices, default=this.default_arch,
         help='CPU architecture. Default: %(default)s'
     )
     parser.add_argument(
@@ -166,8 +168,8 @@ See the documentation for other values known to work.
 '''
     )
     parser.add_argument(
-        '-M', '--gem5-build-id', default=default_build_id,
+        '-M', '--gem5-build-id',
-        help='gem5 build ID. Allows you to keep multiple separate gem5 builds. Default: %(default)s'
+        help='gem5 build ID. Allows you to keep multiple separate gem5 builds. Default: %(default)s'.format(default_build_id)
     )
     parser.add_argument(
         '-N', '--gem5-worktree',
@@ -341,7 +343,7 @@ def make_build_dirs():
 
 def make_run_dirs():
     '''
-    Make directories rquired for the run.
+    Make directories required for the run.
     The user could nuke those anytime between runs to try and clean things up.
     '''
     global this
@@ -435,6 +437,7 @@ def run_cmd(
         extra_env=None,
         extra_paths=None,
         delete_env=None,
+        dry_run=False,
         **kwargs
     ):
     '''
@@ -456,6 +459,9 @@ def run_cmd(
 
     :param extra_env: extra environment variables to add when running the command
     :type extra_env: Dict[str,str]
+
+    :param dry_run: don't run the commands, just potentially print them. Debug aid.
+    :type dry_run: Bool
     '''
     if out_file is not None:
         stdout = subprocess.PIPE
@@ -501,27 +507,30 @@ def run_cmd(
     #sigpipe_old = signal.getsignal(signal.SIGPIPE)
     #signal.signal(signal.SIGPIPE, signal.SIG_DFL)
 
-    # https://stackoverflow.com/questions/15535240/python-popen-write-to-stdout-and-log-file-simultaneously/52090802#52090802
+    if not dry_run:
-    with subprocess.Popen(cmd, stdout=stdout, stderr=stderr, env=env, **kwargs) as proc:
+        # https://stackoverflow.com/questions/15535240/python-popen-write-to-stdout-and-log-file-simultaneously/52090802#52090802
-        if out_file is not None:
+        with subprocess.Popen(cmd, stdout=stdout, stderr=stderr, env=env, **kwargs) as proc:
-            os.makedirs(os.path.split(os.path.abspath(out_file))[0], exist_ok=True)
+            if out_file is not None:
-            with open(out_file, 'bw') as logfile:
+                os.makedirs(os.path.split(os.path.abspath(out_file))[0], exist_ok=True)
-                while True:
+                with open(out_file, 'bw') as logfile:
-                    byte = proc.stdout.read(1)
+                    while True:
-                    if byte:
+                        byte = proc.stdout.read(1)
-                        if show_stdout:
+                        if byte:
-                            sys.stdout.buffer.write(byte)
+                            if show_stdout:
-                            try:
+                                sys.stdout.buffer.write(byte)
-                                sys.stdout.flush()
+                                try:
-                            except BlockingIOError:
+                                    sys.stdout.flush()
-                                # TODO understand. Why, Python, why.
+                                except BlockingIOError:
-                                pass
+                                    # TODO understand. Why, Python, why.
-                        logfile.write(byte)
+                                    pass
-                    else:
+                            logfile.write(byte)
-                        break
+                        else:
-    signal.signal(signal.SIGINT, sigint_old)
+                            break
-    #signal.signal(signal.SIGPIPE, sigpipe_old)
+        signal.signal(signal.SIGINT, sigint_old)
-    return proc.returncode
+        #signal.signal(signal.SIGPIPE, sigpipe_old)
+        return proc.returncode
+    else:
+        return 0
 
 def setup(parser):
     '''
@@ -530,11 +539,14 @@ def setup(parser):
     '''
     global this
     args = parser.parse_args()
-    if args.arch in this.arch_map:
+    if args.arch in this.arch_short_to_long_dict:
-        args.arch = this.arch_map[args.arch]
+        args.arch = this.arch_short_to_long_dict[args.arch]
-    # Because argparse sucks:
+    if args.gem5_build_id is None:
-    # https://stackoverflow.com/questions/30487767/check-if-argparse-optional-argument-is-set-or-not
+        args.gem5_build_id = default_build_id
-    if args.gem5_worktree is not None and args.gem5_build_id == default_build_id:
+        gem5_build_id_given = False
+    else:
+        gem5_build_id_given = True
+    if args.gem5_worktree is not None and not gem5_build_id_given:
         args.gem5_build_id = args.gem5_worktree
     this.machine = args.machine
     if args.arch == 'arm':

download-dependencies

@@ -1,4 +1,12 @@
 #!/usr/bin/env bash
+
+# Download build dependencies for the build, notably:
+#
+# - required git submodules
+# - host packages
+#
+# Only needs to be run once before each type of build per-system.
+
 set -eux
 all=false
 apt_get=true

release

@@ -24,7 +24,7 @@ start_time = time.time()
 # know what the smallest root filesystem size is and use it either...
 # https://stackoverflow.com/questions/47320800/how-to-clean-only-target-in-buildroot
 subprocess.check_call([os.path.join(common.root_dir, 'configure'), '--all'])
-subprocess.check_call([os.path.join(common.root_dir, 'build-all')])
+subprocess.check_call([os.path.join(common.root_dir, 'build'), '--all-linux-components'])
 release_zip.main()
 subprocess.check_call(['git', 'push'])
 release_upload.main()

release-zip

@@ -15,7 +15,7 @@ def main():
     if os.path.exists(common.release_zip_file):
         os.unlink(common.release_zip_file)
     zipf = zipfile.ZipFile(common.release_zip_file, 'w', zipfile.ZIP_DEFLATED)
-    for arch in common.arches:
+    for arch in common.all_archs:
         common.setup(common.get_argparse(default_args={'arch': arch}))
         zipf.write(common.qcow2_file, arcname=os.path.relpath(common.qcow2_file, common.root_dir))
         zipf.write(common.linux_image, arcname=os.path.relpath(common.linux_image, common.root_dir))