3rd-party/linux-kernel-module-cheat
1
0
Fork 0
mirror of https://github.com/cirosantilli/linux-kernel-module-cheat.git synced 2026-01-28 12:34:26 +01:00
Code Issues Releases Wiki Activity

CliFunction

Browse Source
This commit is contained in:
Ciro Santilli 六四事件 法轮功
2019-01-22 00:00:00 +00:00
parent 3b0a343647
commit a5ec63dc28
39 changed files with 2630 additions and 2399 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
README.adoc
View File

@@ -830,6 +830,14 @@ For more information on baremetal, see the section: <<baremetal>>. The following
* <<tracing>> * <<tracing>>
* <<baremetal-gdb-step-debug>> * <<baremetal-gdb-step-debug>>
=== User mode setup
Much like <<baremetal-setup>>, this is another fun setup that does not require Buildroot or the Linux kernel.
See: <<user-mode-simulation>>
TODO: test it out on a clean repo.
[[gdb]] [[gdb]]
== GDB step debug == GDB step debug
@@ -960,7 +968,7 @@ This automatically clears the GDB pane, and starts a new one.
Pass extra GDB arguments with: Pass extra GDB arguments with:
.... ....
./run --wait-gdb --tmux=start_kernel ./run --wait-gdb --tmux --tmux-args start_kernel
.... ....
See the tmux manual for further details: See the tmux manual for further details:
@@ -2986,7 +2994,8 @@ Or alternatively, if you are using <<tmux>>, do everything in one go with:
./run \ ./run \
--arch aarch64 \ --arch aarch64 \
--userland print_argv \ --userland print_argv \
--tmux=main \ --tmux \
--tmux-args main \
--wait-gdb \ --wait-gdb \
-- \ -- \
asdf qwer \ asdf qwer \
@@ -4106,6 +4115,12 @@ That file contains `BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/x86_64/linux
`arm`, on the other hand, uses link:https://github.com/buildroot/buildroot/blob/2018.05/configs/qemu_arm_vexpress_defconfig[`buildroot/configs/qemu_arm_vexpress_defconfig`], which contains `BR2_LINUX_KERNEL_DEFCONFIG="vexpress"`, and therefore just does a `make vexpress_defconfig`, and gets its config from the Linux kernel tree itself. `arm`, on the other hand, uses link:https://github.com/buildroot/buildroot/blob/2018.05/configs/qemu_arm_vexpress_defconfig[`buildroot/configs/qemu_arm_vexpress_defconfig`], which contains `BR2_LINUX_KERNEL_DEFCONFIG="vexpress"`, and therefore just does a `make vexpress_defconfig`, and gets its config from the Linux kernel tree itself.
====== Linux kernel defconfigs
It would be interesting to test out if `make defconfig` configs boot and work on QEMU + Buildroot: https://unix.stackexchange.com/questions/29439/compiling-the-kernel-with-default-configurations/204512#204512
TODO.
===== Notable alternate gem5 kernel configs ===== Notable alternate gem5 kernel configs
Other configs which we had previously tested at 4e0d9af81fcce2ce4e777cb82a1990d7c2ca7c1e are: Other configs which we had previously tested at 4e0d9af81fcce2ce4e777cb82a1990d7c2ca7c1e are:
@@ -5074,7 +5089,7 @@ If `CONFIG_KALLSYMS=n`, then addresses are shown on traces instead of symbol plu
In v4.16 it does not seem possible to configure that at runtime. GDB step debugging with: In v4.16 it does not seem possible to configure that at runtime. GDB step debugging with:
.... ....
./run --eval-after 'insmod /dump_stack.ko' --wait-gdb --tmux=dump_stack ./run --eval-after 'insmod /dump_stack.ko' --wait-gdb --tmux --tmux-args dump_stack
.... ....
shows that traces are printed at `arch/x86/kernel/dumpstack.c`: shows that traces are printed at `arch/x86/kernel/dumpstack.c`:
@@ -8206,7 +8221,7 @@ And in QEMU:
Or for a faster development loop: Or for a faster development loop:
.... ....
./run --debug-vm='-ex "break edu_mmio_read" -ex "run"' ./run --debug-vm --debug-vm-args '-ex "break edu_mmio_read" -ex "run"'
.... ....
When in <<qemu-text-mode>>, using `--debug-vm` makes Ctrl-C not get passed to the QEMU guest anymore: it is instead captured by GDB itself, so allow breaking. So e.g. you won't be able to easily quit from a guest program like: When in <<qemu-text-mode>>, using `--debug-vm` makes Ctrl-C not get passed to the QEMU guest anymore: it is instead captured by GDB itself, so allow breaking. So e.g. you won't be able to easily quit from a guest program like:
@@ -8529,8 +8544,8 @@ List all available debug flags:
but to understand most of them you have to look at the source code: but to understand most of them you have to look at the source code:
.... ....
less "$(./getvar gem5_src_dir)/src/cpu/SConscript" less "$(./getvar gem5_source_dir)/src/cpu/SConscript"
less "$(./getvar gem5_src_dir)/src/cpu/exetrace.cc" less "$(./getvar gem5_source_dir)/src/cpu/exetrace.cc"
.... ....
The traces are generated from `DPRINTF(<trace-id>` calls scattered throughout the code. The traces are generated from `DPRINTF(<trace-id>` calls scattered throughout the code.
@@ -9971,7 +9986,7 @@ The `--gem5-script biglittle` option enables the alternative `configs/example/ar
First apply: First apply:
.... ....
patch -d "$(./getvar gem5_src_dir)" -p 1 < patches/manual/gem5-biglittle.patch patch -d "$(./getvar gem5_source_dir)" -p 1 < patches/manual/gem5-biglittle.patch
.... ....
then: then:
@@ -10327,13 +10342,13 @@ then on the second shell:
Or if you are a <<tmux,tmux pro>>, do everything in one go with: Or if you are a <<tmux,tmux pro>>, do everything in one go with:
.... ....
./run --arch arm --baremetal interactive/prompt --wait-gdb --tmux=main ./run --arch arm --baremetal interactive/prompt --wait-gdb --tmux --tmux-args main
.... ....
Alternatively, to start from the very first executed instruction of our tiny <<baremetal-bootloaders>>: Alternatively, to start from the very first executed instruction of our tiny <<baremetal-bootloaders>>:
.... ....
./run --arch arm --baremetal interactive/prompt --wait-gdb --tmux=--no-continue ./run --arch arm --baremetal interactive/prompt --wait-gdb --tmux --tmux-args --no-continue
.... ....
Now you can just `stepi` to when jumping into main to go to the C code in link:baremetal/interactive/prompt.c[]. Now you can just `stepi` to when jumping into main to go to the C code in link:baremetal/interactive/prompt.c[].
@@ -10341,7 +10356,7 @@ Now you can just `stepi` to when jumping into main to go to the C code in link:b
This is specially interesting for the executables that don't use the bootloader from under `baremetal/arch/<arch>/no_bootloader/*.S`, e.g.: This is specially interesting for the executables that don't use the bootloader from under `baremetal/arch/<arch>/no_bootloader/*.S`, e.g.:
.... ....
./run --arch arm --baremetal arch/arm/no_bootloader/semihost_exit --wait-gdb --tmux=--no-continue ./run --arch arm --baremetal arch/arm/no_bootloader/semihost_exit --wait-gdb --tmux --tmux-args --no-continue
.... ....
The cool thing about those examples is that you start at the very first instruction of your program, which gives more control. The cool thing about those examples is that you start at the very first instruction of your program, which gives more control.
@@ -10369,7 +10384,7 @@ The most important things that we setup in the bootloaders are:
The C functions that become available as a result are: The C functions that become available as a result are:
* Newlib functions implemented at link:baremetal/lib/syscalls.c[] * Newlib functions implemented at link:baremetal/lib/syscalls.c[]
* non-Newlib functions implemented at link:common.c[] * non-Newlib functions implemented at link:kwargs['c'][]
It is not possible to call those C functions from the examples that don't use a bootloader. It is not possible to call those C functions from the examples that don't use a bootloader.
@@ -10401,7 +10416,7 @@ svc 0x00123456
and we can see from the docs that `0x18` stands for the `SYS_EXIT` command. and we can see from the docs that `0x18` stands for the `SYS_EXIT` command.
This is also how we implement the `exit(0)` system call in C for QEMU for link:baremetal/exit.c[] through the Newlib via the function `_exit` at link:baremetal/lib/common.c[]. This is also how we implement the `exit(0)` system call in C for QEMU for link:baremetal/exit.c[] through the Newlib via the function `_exit` at link:baremetal/lib/kwargs['c'][].
Other magic operations we can do with semihosting besides exiting the on the host include: Other magic operations we can do with semihosting besides exiting the on the host include:
@@ -10441,7 +10456,7 @@ Bibliography:
For gem5, you need: For gem5, you need:
.... ....
patch -d "$(./getvar gem5_src_dir)" -p 1 < patches/manual/gem5-semihost.patch patch -d "$(./getvar gem5_source_dir)" -p 1 < patches/manual/gem5-semihost.patch
.... ....
https://stackoverflow.com/questions/52475268/how-to-enable-arm-semihosting-in-gem5/52475269#52475269 https://stackoverflow.com/questions/52475268/how-to-enable-arm-semihosting-in-gem5/52475269#52475269
@@ -11500,17 +11515,17 @@ Analogous to the <<linux-kernel-build-variants>> but with the `--gem5-build-id`
./build-gem5 ./build-gem5
# Build another branch. # Build another branch.
git -C "$(./getvar gem5_src_dir)" checkout some-branch git -C "$(./getvar gem5_source_dir)" checkout some-branch
./build-gem5 --gem5-build-id some-branch ./build-gem5 --gem5-build-id some-branch
# Restore master. # Restore master.
git -C "$(./getvar gem5_src_dir)" checkout - git -C "$(./getvar gem5_source_dir)" checkout -
# Run master. # Run master.
./run --gem5 ./run --gem5
# Run another branch. # Run another branch.
git -C "$(./getvar gem5_src_dir)" checkout some-branch git -C "$(./getvar gem5_source_dir)" checkout some-branch
./run --gem5-build-id some-branch --gem5 ./run --gem5-build-id some-branch --gem5
.... ....

3
arm
View File

@@ -1,3 +0,0 @@
#!/usr/bin/env bash
build-crosstool-ng \
;

6
bench-all
View File

@@ -100,9 +100,9 @@ if "$bench_gem5_build"; then
common_arch="$default_arch" common_arch="$default_arch"
gem5_build_id=bench gem5_build_id=bench
common_gem5_build_dir="$("$getvar" --arch "$common_arch" --gem5-build-id "$gem5_build_id" gem5_build_dir)" common_gem5_build_dir="$("$getvar" --arch "$common_arch" --gem5-build-id "$gem5_build_id" gem5_build_dir)"
common_gem5_src_dir="$("$getvar" --arch "$common_arch" --gem5-build-id "$gem5_build_id" gem5_src_dir)" common_gem5_source_dir="$("$getvar" --arch "$common_arch" --gem5-build-id "$gem5_build_id" gem5_src_dir)"
results_file="${common_gem5_build_dir}/lkmc-bench-build.txt" results_file="${common_gem5_build_dir}/lkmc-bench-build.txt"
git -C "${common_gem5_src_dir}" clean -xdf git -C "${common_gem5_source_dir}" clean -xdf
rm -f "$results_file" rm -f "$results_file"
"${root_dir}/build-gem5" --arch "$common_arch" --clean --gem5-build-id "$gem5_build_id" "${root_dir}/build-gem5" --arch "$common_arch" --clean --gem5-build-id "$gem5_build_id"
# TODO understand better: --foreground required otherwise we cannot # TODO understand better: --foreground required otherwise we cannot
@@ -110,7 +110,7 @@ if "$bench_gem5_build"; then
# bash -c "eval 'timeout 5 sleep 3'" # bash -c "eval 'timeout 5 sleep 3'"
"${root_dir}/bench-cmd" "timeout --foreground 900 ./build-gem5 --arch '$common_arch' --gem5-build-id '$gem5_build_id'" "$results_file" "${root_dir}/bench-cmd" "timeout --foreground 900 ./build-gem5 --arch '$common_arch' --gem5-build-id '$gem5_build_id'" "$results_file"
cp "$results_file" "${new_dir}/gem5-bench-build-${common_arch}.txt" cp "$results_file" "${new_dir}/gem5-bench-build-${common_arch}.txt"
git -C "${common_gem5_src_dir}" clean -xdf git -C "${common_gem5_source_dir}" clean -xdf
"${root_dir}/build-gem5" --arch "$common_arch" --clean --gem5-build-id "$gem5_build_id" "${root_dir}/build-gem5" --arch "$common_arch" --clean --gem5-build-id "$gem5_build_id"
fi fi

10
bisect-linux-boot-gem5
View File

@@ -6,10 +6,10 @@ import shutil
import sys import sys
import common import common
build_linux = imp.load_source('build-linux', os.path.join(common.root_dir, 'build_linux')) build_linux = imp.load_source('build-linux', os.path.join(kwargs['root_dir'], 'build_linux'))
run = imp.load_source('run', os.path.join(common.root_dir, 'run')) run = imp.load_source('run', os.path.join(kwargs['root_dir'], 'run'))
parser = common.get_argparse( parser = self.get_argparse(
argparse_args={ argparse_args={
'description': '''Bisect the Linux kernel on gem5 boots. 'description': '''Bisect the Linux kernel on gem5 boots.
@@ -20,11 +20,11 @@ More information at: https://github.com/cirosantilli/linux-kernel-module-cheat#b
'linux_build_id': 'bisect', 'linux_build_id': 'bisect',
}, },
) )
args = common.setup(parser) args = self.setup(parser)
# We need a clean rebuild because rebuilds at different revisions: # We need a clean rebuild because rebuilds at different revisions:
# - may fail # - may fail
# - may not actually rebuild all files, e.g. on header changes # - may not actually rebuild all files, e.g. on header changes
common.rmrf(common.linux_build_dir) self.rmrf(kwargs['linux_build_dir'])
build_linux.LinuxComponent().do_build(args) build_linux.LinuxComponent().do_build(args)
status = run.main(args, { status = run.main(args, {
'eval': 'm5 exit', 'eval': 'm5 exit',

6
bst-vs-heap
View File

@@ -1,10 +1,10 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
import common import common
parser = common.get_argparse( parser = self.get_argparse(
argparse_args={'description':'Convert a BST vs heap stat file into a gnuplot input'} argparse_args={'description':'Convert a BST vs heap stat file into a gnuplot input'}
) )
args = common.setup(parser) args = self.setup(parser)
stats = common.get_stats() stats = self.get_stats()
it = iter(stats) it = iter(stats)
i = 1 i = 1
for stat in it: for stat in it:

95
build
View File

@@ -7,6 +7,7 @@ import re
import os import os
import common import common
from shell_helpers import LF
class Component: class Component:
''' '''
@@ -49,11 +50,11 @@ class Component:
def run_cmd(cmd, arch): def run_cmd(cmd, arch):
global args global args
cmd_abs = cmd.copy() cmd_abs = cmd.copy()
cmd_abs[0] = os.path.join(common.root_dir, cmd[0]) cmd_abs[0] = os.path.join(kwargs['root_dir'], cmd[0])
cmd_abs.extend(['--arch', arch]) cmd_abs.extend(['--arch', arch])
if args.extra_args: if kwargs['extra_args']:
cmd_abs.append(args.extra_args) cmd_abs.append(kwargs['extra_args'])
common.run_cmd(cmd_abs, dry_run=args.dry_run) self.sh.run_cmd(cmd_abs, dry_run=kwargs['dry_run'])
buildroot_component = Component( buildroot_component = Component(
lambda arch: run_cmd(['build-buildroot'], arch), lambda arch: run_cmd(['build-buildroot'], arch),
@@ -86,15 +87,15 @@ name_to_component_map = {
# Leaves without dependencies. # Leaves without dependencies.
'baremetal-qemu': Component( 'baremetal-qemu': Component(
lambda arch: run_cmd(['build-baremetal', '--qemu'], arch), lambda arch: run_cmd(['build-baremetal', '--qemu'], arch),
supported_archs=common.crosstool_ng_supported_archs, supported_archs=kwargs['crosstool_ng_supported_archs'],
), ),
'baremetal-gem5': Component( 'baremetal-gem5': Component(
lambda arch: run_cmd(['build-baremetal', '--gem5'], arch), lambda arch: run_cmd(['build-baremetal', '--gem5'], arch),
supported_archs=common.crosstool_ng_supported_archs, supported_archs=kwargs['crosstool_ng_supported_archs'],
), ),
'baremetal-gem5-pbx': Component( 'baremetal-gem5-pbx': Component(
lambda arch: run_cmd(['build-baremetal', '--gem5', '--machine', 'RealViewPBX'], arch), lambda arch: run_cmd(['build-baremetal', '--gem5', '--machine', 'RealViewPBX'], arch),
supported_archs=common.crosstool_ng_supported_archs, supported_archs=kwargs['crosstool_ng_supported_archs'],
), ),
'buildroot': buildroot_component, 'buildroot': buildroot_component,
'buildroot-gcc': buildroot_component, 'buildroot-gcc': buildroot_component,
@@ -103,7 +104,7 @@ name_to_component_map = {
), ),
'crosstool-ng': Component( 'crosstool-ng': Component(
lambda arch: run_cmd(['build-crosstool-ng'], arch), lambda arch: run_cmd(['build-crosstool-ng'], arch),
supported_archs=common.crosstool_ng_supported_archs, supported_archs=kwargs['crosstool_ng_supported_archs'],
# http://crosstool-ng.github.io/docs/os-setup/ # http://crosstool-ng.github.io/docs/os-setup/
apt_get_pkgs={ apt_get_pkgs={
'bison', 'bison',
@@ -197,7 +198,7 @@ name_to_component_map = {
'gem5-baremetal', 'gem5-baremetal',
'baremetal-gem5-pbx', 'baremetal-gem5-pbx',
], ],
supported_archs=common.crosstool_ng_supported_archs, supported_archs=kwargs['crosstool_ng_supported_archs'],
), ),
'all-linux': Component(dependencies=[ 'all-linux': Component(dependencies=[
'qemu-gem5-buildroot', 'qemu-gem5-buildroot',
@@ -296,10 +297,10 @@ group = parser.add_mutually_exclusive_group(required=False)
group.add_argument('-A', '--all-archs', default=False, action='store_true', help='''\ group.add_argument('-A', '--all-archs', default=False, action='store_true', help='''\
Build the selected components for all archs. Build the selected components for all archs.
''') ''')
group.add_argument('-a', '--arch', choices=common.arch_choices, default=[], action='append', help='''\ group.add_argument('-a', '--arch', choices=kwargs['arch_choices'], default=[], action='append', help='''\
Build the selected components for this arch. Select multiple archs by Build the selected components for this arch. Select multiple archs by
passing this option multiple times. Default: [{}] passing this option multiple times. Default: [{}]
'''.format(common.default_arch)) '''.format(kwargs['default_arch']))
parser.add_argument('-D', '--download-dependencies', default=False, action='store_true', help='''\ parser.add_argument('-D', '--download-dependencies', default=False, action='store_true', help='''\
Also download all dependencies required for a given build: Ubuntu packages, Also download all dependencies required for a given build: Ubuntu packages,
Python packages and git submodules. Python packages and git submodules.
@@ -314,27 +315,27 @@ Extra args to pass to all scripts.
) )
parser.add_argument('components', choices=list(name_to_component_map.keys()) + [[]], default=[], nargs='*', help='''\ parser.add_argument('components', choices=list(name_to_component_map.keys()) + [[]], default=[], nargs='*', help='''\
Which components to build. Default: qemu-buildroot Which components to build. Default: qemu-buildroot
'''.format(common.default_arch)) '''.format(kwargs['default_arch']))
common.add_dry_run_argument(parser) self.add_dry_run_argument(parser)
args = parser.parse_args() args = parser.parse_args()
common.setup_dry_run_arguments(args) self.setup_dry_run_arguments(args)
# Decide archs. # Decide archs.
if args.arch == []: if kwargs['arch'] == []:
if args.all or args.all_archs: if kwargs['all'] or kwargs['all_archs']:
archs = common.all_archs.copy() archs = kwargs['all_archs'].copy()
else: else:
archs = set([common.default_arch]) archs = set([kwargs['default_arch']])
else: else:
archs = set() archs = set()
for arch in args.arch: for arch in kwargs['arch']:
if arch in common.arch_short_to_long_dict: if arch in kwargs['arch_short_to_long_dict']:
arch = common.arch_short_to_long_dict[arch] arch = kwargs['arch_short_to_long_dict'][arch]
archs.add(arch) archs.add(arch)
# Decide components. # Decide components.
components = args.components components = kwargs['components']
if args.all: if kwargs['all']:
components = ['all'] components = ['all']
elif components == []: elif components == []:
components = ['qemu-buildroot'] components = ['qemu-buildroot']
@@ -350,7 +351,7 @@ for component_name in components:
selected_components.append(component) selected_components.append(component)
todo.extend(component.dependencies) todo.extend(component.dependencies)
if args.download_dependencies: if kwargs['download_dependencies']:
apt_get_pkgs = { apt_get_pkgs = {
# Core requirements for this repo. # Core requirements for this repo.
'git', 'git',
@@ -388,12 +389,12 @@ if args.download_dependencies:
python2_pkgs.update(component.python2_pkgs) python2_pkgs.update(component.python2_pkgs)
python3_pkgs.update(component.python3_pkgs) python3_pkgs.update(component.python3_pkgs)
if apt_get_pkgs or apt_build_deps: if apt_get_pkgs or apt_build_deps:
if args.travis: if kwargs['travis']:
interacive_pkgs = { interacive_pkgs = {
'libsdl2-dev', 'libsdl2-dev',
} }
apt_get_pkgs.difference_update(interacive_pkgs) apt_get_pkgs.difference_update(interacive_pkgs)
if common.in_docker: if kwargs['in_docker']:
sudo = [] sudo = []
# https://askubuntu.com/questions/909277/avoiding-user-interaction-with-tzdata-when-installing-certbot-in-a-docker-contai # https://askubuntu.com/questions/909277/avoiding-user-interaction-with-tzdata-when-installing-certbot-in-a-docker-contai
os.environ['DEBIAN_FRONTEND'] = 'noninteractive' os.environ['DEBIAN_FRONTEND'] = 'noninteractive'
@@ -406,35 +407,35 @@ if args.download_dependencies:
f.write(sources_txt) f.write(sources_txt)
else: else:
sudo = ['sudo'] sudo = ['sudo']
if common.in_docker or args.travis: if kwargs['in_docker'] or kwargs['travis']:
y = ['-y'] y = ['-y']
else: else:
y = [] y = []
common.run_cmd( self.sh.run_cmd(
sudo + ['apt-get', 'update', common.Newline] sudo + ['apt-get', 'update', LF]
) )
if apt_get_pkgs: if apt_get_pkgs:
common.run_cmd( self.sh.run_cmd(
sudo + ['apt-get', 'install'] + y + [common.Newline] + sudo + ['apt-get', 'install'] + y + [LF] +
common.add_newlines(sorted(apt_get_pkgs)) self.sh.add_newlines(sorted(apt_get_pkgs))
) )
if apt_build_deps: if apt_build_deps:
common.run_cmd( self.sh.run_cmd(
sudo + sudo +
['apt-get', 'build-dep'] + y + [common.Newline] + ['apt-get', 'build-dep'] + y + [LF] +
common.add_newlines(sorted(apt_build_deps)) self.sh.add_newlines(sorted(apt_build_deps))
) )
if python2_pkgs: if python2_pkgs:
common.run_cmd( self.sh.run_cmd(
['python', '-m', 'pip', 'install', '--user', common.Newline] + ['python', '-m', 'pip', 'install', '--user', LF] +
common.add_newlines(sorted(python2_pkgs)) self.sh.add_newlines(sorted(python2_pkgs))
) )
if python3_pkgs: if python3_pkgs:
# Not with pip executable directly: # Not with pip executable directly:
# https://stackoverflow.com/questions/49836676/error-after-upgrading-pip-cannot-import-name-main/51846054#51846054 # https://stackoverflow.com/questions/49836676/error-after-upgrading-pip-cannot-import-name-main/51846054#51846054
common.run_cmd( self.sh.run_cmd(
['python3', '-m', 'pip', 'install', '--user', common.Newline] + ['python3', '-m', 'pip', 'install', '--user', LF] +
common.add_newlines(sorted(python3_pkgs)) self.sh.add_newlines(sorted(python3_pkgs))
) )
git_cmd_common = ['git', 'submodule', 'update', '--init', '--recursive'] git_cmd_common = ['git', 'submodule', 'update', '--init', '--recursive']
if submodules: if submodules:
@@ -448,9 +449,9 @@ if args.download_dependencies:
# * https://stackoverflow.com/questions/4640020/progress-indicator-for-git-clone # * https://stackoverflow.com/questions/4640020/progress-indicator-for-git-clone
# #
# `--jobs"`: https://stackoverflow.com/questions/26957237/how-to-make-git-clone-faster-with-multiple-threads/52327638#52327638 # `--jobs"`: https://stackoverflow.com/questions/26957237/how-to-make-git-clone-faster-with-multiple-threads/52327638#52327638
common.run_cmd( self.sh.run_cmd(
git_cmd_common + ['--', common.Newline] + git_cmd_common + ['--', LF] +
common.add_newlines([os.path.join(common.submodules_dir, x) for x in sorted(submodules)]) self.sh.add_newlines([os.path.join(kwargs['submodules_dir'], x) for x in sorted(submodules)])
) )
if submodules_shallow: if submodules_shallow:
# == Shallow cloning. # == Shallow cloning.
@@ -472,9 +473,9 @@ if args.download_dependencies:
# * https://stackoverflow.com/questions/2144406/git-shallow-submodules/47374702#47374702 # * https://stackoverflow.com/questions/2144406/git-shallow-submodules/47374702#47374702
# * https://unix.stackexchange.com/questions/338578/why-is-the-git-clone-of-the-linux-kernel-source-code-much-larger-than-the-extrac # * https://unix.stackexchange.com/questions/338578/why-is-the-git-clone-of-the-linux-kernel-source-code-much-larger-than-the-extrac
# #
common.run_cmd( self.sh.run_cmd(
git_cmd_common + ['--depth', '1', '--', common.Newline] + git_cmd_common + ['--depth', '1', '--', LF] +
common.add_newlines([os.path.join(common.submodules_dir, x) for x in sorted(submodules_shallow)]) self.sh.add_newlines([os.path.join(kwargs['submodules_dir'], x) for x in sorted(submodules_shallow)])
) )
# Do the build. # Do the build.

148
build-baremetal
View File

@@ -1,73 +1,79 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
import os import os
import common import common
from shell_helpers import LF
class BaremetalComponent(common.Component): class Main(common.BuildCliFunction):
def do_build(self, args): def __init__(self):
common.assert_crosstool_ng_supports_arch(args.arch) super().__init__(
build_dir = self.get_build_dir(args) description='''\
bootloader_obj = os.path.join(common.baremetal_build_lib_dir, 'bootloader{}'.format(common.obj_ext)) Build the baremetal examples with crosstool-NG.
''')
def build(self):
self.assert_crosstool_ng_supports_arch(self.env['arch'])
build_dir = self.get_build_dir()
bootloader_obj = os.path.join(self.env['baremetal_build_lib_dir'], 'bootloader{}'.format(self.env['obj_ext']))
common_basename_noext = 'common' common_basename_noext = 'common'
common_src = os.path.join(common.root_dir, common_basename_noext + common.c_ext) common_src = os.path.join(self.env['root_dir'], common_basename_noext + self.env['c_ext'])
common_obj = os.path.join(common.baremetal_build_lib_dir, common_basename_noext + common.obj_ext) common_obj = os.path.join(self.env['baremetal_build_lib_dir'], common_basename_noext + self.env['obj_ext'])
syscalls_basename_noext = 'syscalls' syscalls_basename_noext = 'syscalls'
syscalls_src = os.path.join(common.baremetal_src_lib_dir, syscalls_basename_noext + common.c_ext) syscalls_src = os.path.join(self.env['baremetal_src_lib_dir'], syscalls_basename_noext + self.env['c_ext'])
syscalls_obj = os.path.join(common.baremetal_build_lib_dir, syscalls_basename_noext + common.obj_ext) syscalls_obj = os.path.join(self.env['baremetal_build_lib_dir'], syscalls_basename_noext + self.env['obj_ext'])
common_objs = [common_obj, syscalls_obj] common_objs = [common_obj, syscalls_obj]
cflags = [ cflags = [
'-I', common.baremetal_src_lib_dir, common.Newline, '-I', self.env['baremetal_src_lib_dir'], LF,
'-I', common.root_dir, common.Newline, '-I', self.env['root_dir'], LF,
'-O0', common.Newline, '-O0', LF,
'-ggdb3', common.Newline, '-ggdb3', LF,
'-mcpu={}'.format(common.mcpu), common.Newline, '-mcpu={}'.format(self.env['mcpu']), LF,
'-nostartfiles', common.Newline, '-nostartfiles', LF,
] ]
if args.prebuilt: if self.env['prebuilt']:
gcc = 'arm-none-eabi-gcc' gcc = 'arm-none-eabi-gcc'
else: else:
os.environ['PATH'] = common.crosstool_ng_bin_dir + os.environ['PATH'] os.environ['PATH'] = self.env['crosstool_ng_bin_dir'] + os.environ['PATH']
gcc = common.get_toolchain_tool('gcc', allowed_toolchains=['crosstool-ng']) gcc = self.get_toolchain_tool('gcc', allowed_toolchains=['crosstool-ng'])
if common.emulator == 'gem5': if self.env['emulator'] == 'gem5':
if common.machine == 'VExpress_GEM5_V1': if self.env['machine'] == 'VExpress_GEM5_V1':
entry_address = 0x80000000 entry_address = 0x80000000
uart_address = 0x1c090000 uart_address = 0x1c090000
elif common.machine == 'RealViewPBX': elif self.env['machine'] == 'RealViewPBX':
entry_address = 0x10000 entry_address = 0x10000
uart_address = 0x10009000 uart_address = 0x10009000
else: else:
raise Exception('unknown machine: ' + common.machine) raise Exception('unknown machine: ' + self.env['machine'])
cflags.extend(['-D', 'GEM5'.format(uart_address), common.Newline]) cflags.extend(['-D', 'GEM5'.format(uart_address), LF])
else: else:
entry_address = 0x40000000 entry_address = 0x40000000
uart_address = 0x09000000 uart_address = 0x09000000
os.makedirs(build_dir, exist_ok=True) os.makedirs(build_dir, exist_ok=True)
os.makedirs(common.baremetal_build_lib_dir, exist_ok=True) os.makedirs(self.env['baremetal_build_lib_dir'], exist_ok=True)
src = os.path.join(common.baremetal_src_lib_dir, '{}{}'.format(args.arch, common.asm_ext)) src = os.path.join(self.env['baremetal_src_lib_dir'], '{}{}'.format(self.env['arch'], self.env['asm_ext']))
if common.need_rebuild([src], bootloader_obj): if self.need_rebuild([src], bootloader_obj):
common.run_cmd( self.sh.run_cmd(
[gcc, common.Newline] + [gcc, LF] +
cflags + cflags +
[ [
'-c', common.Newline, '-c', LF,
'-o', bootloader_obj, common.Newline, '-o', bootloader_obj, LF,
src, common.Newline, src, LF,
] ]
) )
for src, obj in [ for src, obj in [
(common_src, common_obj), (common_src, common_obj),
(syscalls_src, syscalls_obj), (syscalls_src, syscalls_obj),
]: ]:
if common.need_rebuild([src], obj): if self.need_rebuild([src], obj):
common.run_cmd( self.sh.run_cmd(
[gcc, common.Newline] + [gcc, LF] +
cflags + cflags +
[ [
'-c', common.Newline, '-c', LF,
'-D', 'UART0_ADDR={:#x}'.format(uart_address), common.Newline, '-D', 'UART0_ADDR={:#x}'.format(uart_address), LF,
'-o', obj, common.Newline, '-o', obj, LF,
src, common.Newline, src, LF,
] ]
) )
self._build_dir( self._build_dir(
@@ -86,8 +92,8 @@ class BaremetalComponent(common.Component):
bootloader_obj=bootloader_obj, bootloader_obj=bootloader_obj,
common_objs=common_objs, common_objs=common_objs,
) )
arch_dir = os.path.join('arch', args.arch) arch_dir = os.path.join('arch', self.env['arch'])
if os.path.isdir(os.path.join(common.baremetal_src_dir, arch_dir)): if os.path.isdir(os.path.join(self.env['baremetal_src_dir'], arch_dir)):
self._build_dir( self._build_dir(
arch_dir, arch_dir,
gcc=gcc, gcc=gcc,
@@ -96,8 +102,8 @@ class BaremetalComponent(common.Component):
bootloader_obj=bootloader_obj, bootloader_obj=bootloader_obj,
common_objs=common_objs, common_objs=common_objs,
) )
arch_dir = os.path.join('arch', args.arch, 'no_bootloader') arch_dir = os.path.join('arch', self.env['arch'], 'no_bootloader')
if os.path.isdir(os.path.join(common.baremetal_src_dir, arch_dir)): if os.path.isdir(os.path.join(self.env['baremetal_src_dir'], arch_dir)):
self._build_dir( self._build_dir(
arch_dir, arch_dir,
gcc=gcc, gcc=gcc,
@@ -108,18 +114,8 @@ class BaremetalComponent(common.Component):
bootloader=False, bootloader=False,
) )
def get_argparse_args(self): def get_build_dir(self):
return { return self.env['baremetal_build_dir']
'description': '''\
Build the baremetal examples with crosstool-NG.
'''
}
def get_build_dir(self, args):
return common.baremetal_build_dir
def get_default_args(self):
return {'baremetal': 'all'}
def _build_dir( def _build_dir(
self, self,
@@ -137,42 +133,42 @@ Build the baremetal examples with crosstool-NG.
Place outputs on the same subpath or the output directory. Place outputs on the same subpath or the output directory.
""" """
in_dir = os.path.join(common.baremetal_src_dir, subpath) in_dir = os.path.join(self.env['baremetal_src_dir'], subpath)
out_dir = os.path.join(common.baremetal_build_dir, subpath) out_dir = os.path.join(self.env['baremetal_build_dir'], subpath)
os.makedirs(out_dir, exist_ok=True) os.makedirs(out_dir, exist_ok=True)
common_objs = common_objs.copy() common_objs = common_objs.copy()
if bootloader: if bootloader:
common_objs.append(bootloader_obj) common_objs.append(bootloader_obj)
for in_basename in os.listdir(in_dir): for in_basename in os.listdir(in_dir):
in_path = os.path.join(in_dir, in_basename) in_path = os.path.join(in_dir, in_basename)
if os.path.isfile(in_path) and os.path.splitext(in_basename)[1] in (common.c_ext, common.asm_ext): if os.path.isfile(in_path) and os.path.splitext(in_basename)[1] in (self.env['c_ext'], self.env['asm_ext']):
in_name = os.path.splitext(in_basename)[0] in_name = os.path.splitext(in_basename)[0]
main_obj = os.path.join(common.baremetal_build_dir, subpath, '{}{}'.format(in_name, common.obj_ext)) main_obj = os.path.join(self.env['baremetal_build_dir'], subpath, '{}{}'.format(in_name, self.env['obj_ext']))
src = os.path.join(common.baremetal_src_dir, in_path) src = os.path.join(self.env['baremetal_src_dir'], in_path)
if common.need_rebuild([src], main_obj): if self.need_rebuild([src], main_obj):
common.run_cmd( self.sh.run_cmd(
[gcc, common.Newline] + [gcc, LF] +
cflags + cflags +
[ [
'-c', common.Newline, '-c', LF,
'-o', main_obj, common.Newline, '-o', main_obj, LF,
src, common.Newline, src, LF,
] ]
) )
objs = common_objs + [main_obj] objs = common_objs + [main_obj]
out = os.path.join(common.baremetal_build_dir, subpath, in_name + common.baremetal_build_ext) out = os.path.join(self.env['baremetal_build_dir'], subpath, in_name + self.env['baremetal_build_ext'])
link_script = os.path.join(common.baremetal_src_dir, 'link.ld') link_script = os.path.join(self.env['baremetal_src_dir'], 'link.ld')
if common.need_rebuild(objs + [link_script], out): if self.need_rebuild(objs + [link_script], out):
common.run_cmd( self.sh.run_cmd(
[gcc, common.Newline] + [gcc, LF] +
cflags + cflags +
[ [
'-Wl,--section-start=.text={:#x}'.format(entry_address), common.Newline, '-Wl,--section-start=.text={:#x}'.format(entry_address), LF,
'-o', out, common.Newline, '-o', out, LF,
'-T', link_script, common.Newline, '-T', link_script, LF,
] + ] +
common.add_newlines(objs) self.sh.add_newlines(objs)
) )
if __name__ == '__main__': if __name__ == '__main__':
BaremetalComponent().build() Main().cli()

168
build-buildroot
View File

@@ -9,11 +9,16 @@ import time
import re import re
import common import common
from shell_helpers import LF
class BuildrootComponent(common.Component): class Main(common.BuildCliFunction):
def add_parser_arguments(self, parser): def __init__(self):
parser.add_argument( super().__init__(
'--build-linux', default=self._defaults['build_linux'], action='store_true', description='''\
Run Linux on an emulator
''')
self.add_argument(
'--build-linux', default=False,
help='''\ help='''\
Enable building the Linux kernel with Buildroot. This is done mostly Enable building the Linux kernel with Buildroot. This is done mostly
to extract Buildroot's default kernel configurations when updating Buildroot. to extract Buildroot's default kernel configurations when updating Buildroot.
@@ -21,35 +26,35 @@ This kernel will not be use by our other scripts. Configuring this kernel is
not currently supported, juse use ./build-linux script if you want to do that. not currently supported, juse use ./build-linux script if you want to do that.
''' '''
) )
parser.add_argument( self.add_argument(
'--baseline', default=self._defaults['baseline'], action='store_true', '--baseline', default=False,
help='''Do a default-ish Buildroot defconfig build, without any of our extra options. help='''Do a default-ish Buildroot defconfig build, without any of our extra options.
Mostly to track how much slower we are than a basic build. Mostly to track how much slower we are than a basic build.
''' '''
) )
parser.add_argument( self.add_argument(
'--config', default=self._defaults['config'], action='append', '--config', default=[], action='append',
help='''Add a single Buildroot config to the current build. help='''Add a single Buildroot config to the current build.
Example value: 'BR2_TARGET_ROOTFS_EXT2_SIZE="512M"'. Example value: 'BR2_TARGET_ROOTFS_EXT2_SIZE="512M"'.
Can be used multiple times to add multiple configs. Can be used multiple times to add multiple configs.
Takes precedence over any Buildroot config files. Takes precedence over any Buildroot config files.
''' '''
) )
parser.add_argument( self.add_argument(
'--config-fragment', default=self._defaults['config_fragment'], action='append', '--config-fragment', default=[], action='append',
help='''Also use the given Buildroot configuration fragment file. help='''Also use the given Buildroot configuration fragment file.
Pass multiple times to use multiple fragment files. Pass multiple times to use multiple fragment files.
''' '''
) )
parser.add_argument( self.add_argument(
'--no-all', default=self._defaults['no_all'], action='store_true', '--no-all', default=False,
help='''\ help='''\
Don't build the all target which normally gets build by default. Don't build the all target which normally gets build by default.
That target builds the root filesystem and all its dependencies. That target builds the root filesystem and all its dependencies.
''' '''
) )
parser.add_argument( self.add_argument(
'--no-overlay', default=self._defaults['no_all'], action='store_true', '--no-overlay', default=False,
help='''\ help='''\
Don't add our overlay which contains all files we build without going through Buildroot. Don't add our overlay which contains all files we build without going through Buildroot.
This prevents us from overwriting certain Buildroot files. Remember however that you must This prevents us from overwriting certain Buildroot files. Remember however that you must
@@ -57,132 +62,117 @@ still rebuild the Buildroot package that provides those files to actually put th
files on the root filesystem. files on the root filesystem.
''' '''
) )
parser.add_argument( self.add_argument(
'extra_make_args', default=self._defaults['extra_make_args'], metavar='extra-make-args', nargs='*', 'extra-make-args', default=[], nargs='*',
help='''\ help='''\
Extra arguments to be passed to the Buildroot make, Extra arguments to be passed to the Buildroot make,
usually extra Buildroot targets. usually extra Buildroot targets.
''' '''
) )
def do_build(self, args): def build(self):
build_dir = self.get_build_dir(args) build_dir = self.get_build_dir()
os.makedirs(common.out_dir, exist_ok=True) os.makedirs(self.env['out_dir'], exist_ok=True)
extra_make_args = common.add_newlines(args.extra_make_args) extra_make_args = self.sh.add_newlines(self.env['extra_make_args'])
if args.build_linux: if self.env['build_linux']:
extra_make_args.extend(['linux-reconfigure', common.Newline]) extra_make_args.extend(['linux-reconfigure', LF])
if common.emulator == 'gem5': if self.env['emulator'] == 'gem5':
extra_make_args.extend(['gem5-reconfigure', common.Newline]) extra_make_args.extend(['gem5-reconfigure', LF])
if args.arch == 'x86_64': if self.env['arch'] == 'x86_64':
defconfig = 'qemu_x86_64_defconfig' defconfig = 'qemu_x86_64_defconfig'
elif args.arch == 'arm': elif self.env['arch'] == 'arm':
defconfig = 'qemu_arm_vexpress_defconfig' defconfig = 'qemu_arm_vexpress_defconfig'
elif args.arch == 'aarch64': elif self.env['arch'] == 'aarch64':
defconfig = 'qemu_aarch64_virt_defconfig' defconfig = 'qemu_aarch64_virt_defconfig'
br2_external_dirs = [] br2_external_dirs = []
for package_dir in os.listdir(common.packages_dir): for package_dir in os.listdir(self.env['packages_dir']):
package_dir_abs = os.path.join(common.packages_dir, package_dir) package_dir_abs = os.path.join(self.env['packages_dir'], package_dir)
if os.path.isdir(package_dir_abs): if os.path.isdir(package_dir_abs):
br2_external_dirs.append(self._path_relative_to_buildroot(package_dir_abs)) br2_external_dirs.append(self._path_relative_to_buildroot(package_dir_abs))
br2_external_str = ':'.join(br2_external_dirs) br2_external_str = ':'.join(br2_external_dirs)
common.run_cmd( self.sh.run_cmd(
[ [
'make', common.Newline, 'make', LF,
'O={}'.format(common.buildroot_build_dir), common.Newline, 'O={}'.format(self.env['buildroot_build_dir']), LF,
'BR2_EXTERNAL={}'.format(br2_external_str), common.Newline, 'BR2_EXTERNAL={}'.format(br2_external_str), LF,
defconfig, common.Newline, defconfig, LF,
], ],
cwd=common.buildroot_src_dir, cwd=self.env['buildroot_src_dir'],
) )
configs = args.config configs = self.env['config']
configs.extend([ configs.extend([
'BR2_JLEVEL={}'.format(args.nproc), 'BR2_JLEVEL={}'.format(self.env['nproc']),
'BR2_DL_DIR="{}"'.format(common.buildroot_download_dir), 'BR2_DL_DIR="{}"'.format(self.env['buildroot_download_dir']),
]) ])
if not args.build_linux: if not self.env['build_linux']:
configs.extend([ configs.extend([
'# BR2_LINUX_KERNEL is not set', '# BR2_LINUX_KERNEL is not set',
]) ])
config_fragments = [] config_fragments = []
if not args.baseline: if not self.env['baseline']:
configs.extend([ configs.extend([
'BR2_GLOBAL_PATCH_DIR="{}"'.format( 'BR2_GLOBAL_PATCH_DIR="{}"'.format(
self._path_relative_to_buildroot(os.path.join(common.root_dir, 'patches', 'global')) self._path_relative_to_buildroot(os.path.join(self.env['root_dir'], 'patches', 'global'))
), ),
'BR2_PACKAGE_BUSYBOX_CONFIG_FRAGMENT_FILES="{}"'.format( 'BR2_PACKAGE_BUSYBOX_CONFIG_FRAGMENT_FILES="{}"'.format(
self._path_relative_to_buildroot(os.path.join(common.root_dir, 'busybox_config_fragment')) self._path_relative_to_buildroot(os.path.join(self.env['root_dir'], 'busybox_config_fragment'))
), ),
'BR2_PACKAGE_OVERRIDE_FILE="{}"'.format( 'BR2_PACKAGE_OVERRIDE_FILE="{}"'.format(
self._path_relative_to_buildroot(os.path.join(common.root_dir, 'buildroot_override')) self._path_relative_to_buildroot(os.path.join(self.env['root_dir'], 'buildroot_override'))
), ),
'BR2_ROOTFS_POST_BUILD_SCRIPT="{}"'.format( 'BR2_ROOTFS_POST_BUILD_SCRIPT="{}"'.format(
self._path_relative_to_buildroot(os.path.join(common.root_dir, 'rootfs-post-build-script')) self._path_relative_to_buildroot(os.path.join(self.env['root_dir'], 'rootfs-post-build-script'))
), ),
'BR2_ROOTFS_USERS_TABLES="{}"'.format( 'BR2_ROOTFS_USERS_TABLES="{}"'.format(
self._path_relative_to_buildroot(os.path.join(common.root_dir, 'user_table')) self._path_relative_to_buildroot(os.path.join(self.env['root_dir'], 'user_table'))
), ),
]) ])
if not args.no_overlay: if not self.env['no_overlay']:
configs.append('BR2_ROOTFS_OVERLAY="{}"'.format( configs.append('BR2_ROOTFS_OVERLAY="{}"'.format(
self._path_relative_to_buildroot(common.out_rootfs_overlay_dir) self._path_relative_to_buildroot(self.env['out_rootfs_overlay_dir'])
)) ))
config_fragments = [ config_fragments = [
os.path.join(common.root_dir, 'buildroot_config', 'default') os.path.join(self.env['root_dir'], 'buildroot_config', 'default')
] + args.config_fragment ] + self.env['config_fragment']
common.write_configs(common.buildroot_config_file, configs, config_fragments) # TODO Can't get rid of these for now with nice fragments on Buildroot:
common.run_cmd( # http://stackoverflow.com/questions/44078245/is-it-possible-to-use-config-fragments-with-buildroots-config
self.sh.write_configs(self.env['buildroot_config_file'], configs, config_fragments)
self.sh.run_cmd(
[ [
'make', common.Newline, 'make', LF,
'O={}'.format(common.buildroot_build_dir), common.Newline, 'O={}'.format(self.env['buildroot_build_dir']), LF,
'olddefconfig', common.Newline, 'olddefconfig', LF,
], ],
cwd=common.buildroot_src_dir, cwd=self.env['buildroot_src_dir'],
) )
common.make_build_dirs() self.make_build_dirs()
if not args.no_all: if not self.env['no_all']:
extra_make_args.extend(['all', common.Newline]) extra_make_args.extend(['all', LF])
common.run_cmd( self.sh.run_cmd(
[ [
'make', common.Newline, 'make', LF,
'LKMC_GEM5_SRCDIR="{}"'.format(common.gem5_src_dir), common.Newline, 'LKMC_GEM5_SRCDIR="{}"'.format(self.env['gem5_source_dir']), LF,
'LKMC_PARSEC_BENCHMARK_SRCDIR="{}"'.format(common.parsec_benchmark_src_dir), common.Newline, 'LKMC_PARSEC_BENCHMARK_SRCDIR="{}"'.format(self.env['parsec_benchmark_src_dir']), LF,
'O={}'.format(common.buildroot_build_dir), common.Newline, 'O={}'.format(self.env['buildroot_build_dir']), LF,
'V={}'.format(int(args.verbose)), common.Newline, 'V={}'.format(int(self.env['verbose'])), LF,
] + ] +
extra_make_args extra_make_args
, ,
out_file=os.path.join(common.buildroot_build_dir, 'lkmc.log'), out_file=os.path.join(self.env['buildroot_build_dir'], 'lkmc.log'),
delete_env=['LD_LIBRARY_PATH'], delete_env=['LD_LIBRARY_PATH'],
cwd=common.buildroot_src_dir, cwd=self.env['buildroot_src_dir'],
) )
# Create the qcow2 from ext2. # Create the qcow2 from ext2.
# Skip if qemu is not present, because gem5 does not need the qcow2. # Skip if qemu is not present, because gem5 does not need the qcow2.
# so we don't force a QEMU build for gem5. # so we don't force a QEMU build for gem5.
if not args.no_all and os.path.exists(common.qemu_img_executable): if not self.env['no_all'] and os.path.exists(self.env['qemu_img_executable']):
common.raw_to_qcow2() self.raw_to_qcow2()
def get_argparse_args(self): def get_build_dir(self):
return { return self.env['buildroot_build_dir']
'description': '''\
Run Linux on an emulator
'''
}
def get_build_dir(self, args):
return common.buildroot_build_dir
_defaults = {
'baseline': False,
'build_linux': False,
'config': [],
'config_fragment': [],
'extra_make_args': [],
'no_all': False,
'skip_configure': False,
}
def _path_relative_to_buildroot(self, abspath): def _path_relative_to_buildroot(self, abspath):
return os.path.relpath(abspath, common.buildroot_src_dir) return os.path.relpath(abspath, self.env['buildroot_src_dir'])
if __name__ == '__main__': if __name__ == '__main__':
BuildrootComponent().build() Main().cli()

83
build-crosstool-ng
View File

@@ -3,75 +3,70 @@
import os import os
import common import common
from shell_helpers import LF
class CrosstoolNgComponent(common.Component): class Main(common.BuildCliFunction):
def do_build(self, args): def __init__(self):
common.assert_crosstool_ng_supports_arch(args.arch) super().__init__(
build_dir = self.get_build_dir(args) description='''\
defconfig_dest = os.path.join(common.crosstool_ng_util_dir, 'defconfig') Build crosstool-NG with Newlib for bare metal compilation
os.makedirs(common.crosstool_ng_util_dir, exist_ok=True) ''')
os.makedirs(common.crosstool_ng_download_dir, exist_ok=True)
def build(self):
self.assert_crosstool_ng_supports_arch(self.env['arch'])
build_dir = self.get_build_dir()
defconfig_dest = os.path.join(self.env['crosstool_ng_util_dir'], 'defconfig')
os.makedirs(self.env['crosstool_ng_util_dir'], exist_ok=True)
os.makedirs(self.env['crosstool_ng_download_dir'], exist_ok=True)
# Bootstrap out-ot-tree WONTFIX. I've tried. # Bootstrap out-ot-tree WONTFIX. I've tried.
# https://github.com/crosstool-ng/crosstool-ng/issues/1021 # https://github.com/crosstool-ng/crosstool-ng/issues/1021
os.chdir(common.crosstool_ng_src_dir) os.chdir(self.env['crosstool_ng_src_dir'])
common.run_cmd( self.sh.run_cmd(
[os.path.join(common.crosstool_ng_src_dir, 'bootstrap'), common.Newline], [os.path.join(self.env['crosstool_ng_src_dir'], 'bootstrap'), LF],
) )
os.chdir(common.crosstool_ng_util_dir) os.chdir(self.env['crosstool_ng_util_dir'])
common.run_cmd( self.sh.run_cmd(
[ [
os.path.join(common.crosstool_ng_src_dir, 'configure'), common.Newline, os.path.join(self.env['crosstool_ng_src_dir'], 'configure'), LF,
'--enable-local', common.Newline, '--enable-local', LF,
],
)
common.run_cmd(
[
'make', common.Newline,
'-j', str(args.nproc), common.Newline,
], ],
) )
self.sh.run_cmd(['make', '-j', str(self.env['nproc']), LF])
# Build the toolchain. # Build the toolchain.
common.cp( self.sh.cp(
os.path.join(common.root_dir, 'crosstool_ng_config', args.arch), os.path.join(self.env['root_dir'], 'crosstool_ng_config', self.env['arch']),
defconfig_dest defconfig_dest
) )
common.write_configs( self.sh.write_configs(
common.crosstool_ng_defconfig, self.env['crosstool_ng_defconfig'],
[ [
'CT_PREFIX_DIR="{}"'.format(common.crosstool_ng_install_dir), 'CT_PREFIX_DIR="{}"'.format(self.env['crosstool_ng_install_dir']),
'CT_WORK_DIR="{}"'.format(build_dir), 'CT_WORK_DIR="{}"'.format(build_dir),
'CT_LOCAL_TARBALLS_DIR="{}"'.format(common.crosstool_ng_download_dir), 'CT_LOCAL_TARBALLS_DIR="{}"'.format(self.env['crosstool_ng_download_dir']),
] ]
) )
common.run_cmd( self.sh.run_cmd(
[ [
common.crosstool_ng_executable, common.Newline, self.env['crosstool_ng_executable'], LF,
'defconfig', common.Newline, 'defconfig', LF,
], ],
) )
os.unlink(defconfig_dest) os.unlink(defconfig_dest)
common.run_cmd( self.sh.run_cmd(
[ [
common.crosstool_ng_executable, common.Newline, self.env['crosstool_ng_executable'], LF,
'build', common.Newline, 'build', LF,
'CT_JOBS={}'.format(str(args.nproc)), common.Newline, 'CT_JOBS={}'.format(str(self.env['nproc'])), LF,
], ],
out_file=os.path.join(build_dir, 'lkmc.log'), out_file=os.path.join(build_dir, 'lkmc.log'),
delete_env=['LD_LIBRARY_PATH'], delete_env=['LD_LIBRARY_PATH'],
extra_paths=[common.ccache_dir], extra_paths=[self.env['ccache_dir']],
) )
def get_argparse_args(self): def get_build_dir(self):
return { return self.env['crosstool_ng_build_dir']
'description': '''\
Build crosstool-NG with Newlib for bare metal compilation'
'''
}
def get_build_dir(self, args):
return common.crosstool_ng_build_dir
if __name__ == '__main__': if __name__ == '__main__':
CrosstoolNgComponent().build() Main().cli()

35
build-docker
View File

@@ -5,9 +5,10 @@ import subprocess
import tarfile import tarfile
import common import common
from shell_helpers import LF
class DockerComponent(common.Component): class DockerComponent(self.Component):
def get_argparse_args(self): def get_argparse_args(self):
return { return {
'description': '''\ 'description': '''\
@@ -17,8 +18,8 @@ See also:https://github.com/cirosantilli/linux-kernel-module-cheat#ubuntu-guest-
''' '''
} }
def do_build(self, args): def build(self):
build_dir = self.get_build_dir(args) build_dir = self.get_build_dir()
container_name = 'lkmc-guest' container_name = 'lkmc-guest'
target_dir = os.path.join('/root', 'linux-kernel-module-cheat') target_dir = os.path.join('/root', 'linux-kernel-module-cheat')
os.makedirs(build_dir, exist_ok=True) os.makedirs(build_dir, exist_ok=True)
@@ -29,12 +30,12 @@ See also:https://github.com/cirosantilli/linux-kernel-module-cheat#ubuntu-guest-
'--format', '{{.Names}}', '--format', '{{.Names}}',
]).decode() ]).decode()
if container_name in containers.split(): if container_name in containers.split():
common.run_cmd([ self.sh.run_cmd([
'docker', 'docker',
'rm', 'rm',
container_name, container_name,
]) ])
common.run_cmd([ self.sh.run_cmd([
'docker', 'docker',
'create', 'create',
'--name', container_name, '--name', container_name,
@@ -44,36 +45,36 @@ See also:https://github.com/cirosantilli/linux-kernel-module-cheat#ubuntu-guest-
'--privileged', '--privileged',
'-t', '-t',
'-w', target_dir, '-w', target_dir,
'-v', '{}:{}'.format(common.root_dir, target_dir), '-v', '{}:{}'.format(kwargs['root_dir'], target_dir),
'ubuntu:18.04', 'ubuntu:18.04',
'bash', 'bash',
]) ])
common.run_cmd([ self.sh.run_cmd([
'docker', 'docker',
'export', 'export',
'-o', '-o',
common.docker_tar_file, kwargs['docker_tar_file'],
container_name, container_name,
]) ])
tar = tarfile.open(common.docker_tar_file) tar = tarfile.open(kwargs['docker_tar_file'])
tar.extractall(common.docker_tar_dir) tar.extractall(kwargs['docker_tar_dir'])
tar.close() tar.close()
# sudo not required in theory # sudo not required in theory
# https://askubuntu.com/questions/1046828/how-to-run-libguestfs-tools-tools-such-as-virt-make-fs-without-sudo # https://askubuntu.com/questions/1046828/how-to-run-libguestfs-tools-tools-such-as-virt-make-fs-without-sudo
common.run_cmd([ self.sh.run_cmd([
'virt-make-fs', 'virt-make-fs',
'--format', 'raw', '--format', 'raw',
'--size', '+1G', '--size', '+1G',
'--type', 'ext2', '--type', 'ext2',
common.docker_tar_dir, kwargs['docker_tar_dir'],
common.docker_rootfs_raw_file, kwargs['docker_rootfs_raw_file'],
]) ])
common.raw_to_qcow2(prebuilt=True) self.raw_to_qcow2(prebuilt=True)
def get_build_dir(self, args): def get_build_dir(self):
return common.docker_build_dir return kwargs['docker_build_dir']
def get_default_args(self): def get_default_args(self):
return {'docker': True} return {'docker': True}
DockerComponent().build() Main().cli()

111
build-gem5
View File

@@ -5,59 +5,57 @@ import pathlib
import subprocess import subprocess
import common import common
from shell_helpers import LF
class Gem5Component(common.Component): class Main(common.BuildCliFunction):
def add_parser_arguments(self, parser): def __init__(self):
parser.add_argument( super().__init__()
self.add_argument(
'extra_scons_args', 'extra_scons_args',
default=[],
metavar='extra-scons-args', metavar='extra-scons-args',
nargs='*' nargs='*',
) )
def do_build(self, args): def build(self):
build_dir = self.get_build_dir(args) build_dir = self.get_build_dir()
binaries_dir = os.path.join(common.gem5_system_dir, 'binaries') binaries_dir = os.path.join(self.env['gem5_system_dir'], 'binaries')
disks_dir = os.path.join(common.gem5_system_dir, 'disks') disks_dir = os.path.join(self.env['gem5_system_dir'], 'disks')
os.makedirs(binaries_dir, exist_ok=True) os.makedirs(binaries_dir, exist_ok=True)
os.makedirs(disks_dir, exist_ok=True) os.makedirs(disks_dir, exist_ok=True)
if args.gem5_source_dir is None: if self.env['gem5_source_dir'] is None:
if not os.path.exists(os.path.join(common.gem5_src_dir, '.git')): if not os.path.exists(os.path.join(self.env['gem5_source_dir'], '.git')):
if common.gem5_src_dir == common.gem5_default_src_dir: if self.env['gem5_source_dir'] == self.env['gem5_default_src_dir']:
raise Exception('gem5 submodule not checked out') raise Exception('gem5 submodule not checked out')
common.run_cmd([ self.sh.run_cmd([
'git', common.Newline, 'git', LF,
'-C', common.gem5_default_src_dir, common.Newline, '-C', self.env['gem5_default_src_dir'], LF,
'worktree', 'add', common.Newline, 'worktree', 'add', LF,
'-b', os.path.join('wt', args.gem5_build_id), common.Newline, '-b', os.path.join('wt', self.env['gem5_build_id']), LF,
common.gem5_src_dir, common.Newline, self.env['gem5_source_dir'], LF,
]) ])
if args.verbose: if self.env['verbose']:
verbose = ['--verbose', common.Newline] verbose = ['--verbose', LF]
else: else:
verbose = [] verbose = []
if args.arch == 'x86_64': if self.env['arch'] == 'x86_64':
dummy_img_path = os.path.join(disks_dir, 'linux-bigswap2.img') dummy_img_path = os.path.join(disks_dir, 'linux-bigswap2.img')
with open(dummy_img_path, 'wb') as dummy_img_file: with open(dummy_img_path, 'wb') as dummy_img_file:
zeroes = b'\x00' * (2 ** 16) zeroes = b'\x00' * (2 ** 16)
for i in range(2 ** 10): for i in range(2 ** 10):
dummy_img_file.write(zeroes) dummy_img_file.write(zeroes)
common.run_cmd(['mkswap', dummy_img_path]) self.sh.run_cmd(['mkswap', dummy_img_path, LF])
with open(os.path.join(binaries_dir, 'x86_64-vmlinux-2.6.22.9'), 'w'): with open(os.path.join(binaries_dir, 'x86_64-vmlinux-2.6.22.9'), 'w'):
# This file must always be present, despite --kernel overriding that default and selecting the kernel. # This file must always be present, despite --kernel overriding that default and selecting the kernel.
# I'm not even joking. No one has ever built x86 gem5 without the magic dist dir present. # I'm not even joking. No one has ever built x86 gem5 without the magic dist dir present.
pass pass
elif args.arch == 'arm' or args.arch == 'aarch64': elif self.env['arch'] == 'arm' or self.env['arch'] == 'aarch64':
gem5_system_src_dir = os.path.join(common.gem5_src_dir, 'system') gem5_system_src_dir = os.path.join(self.env['gem5_source_dir'], 'system')
# dtb # dtb
dt_src_dir = os.path.join(gem5_system_src_dir, 'arm', 'dt') dt_src_dir = os.path.join(gem5_system_src_dir, 'arm', 'dt')
dt_build_dir = os.path.join(common.gem5_system_dir, 'arm', 'dt') dt_build_dir = os.path.join(self.env['gem5_system_dir'], 'arm', 'dt')
common.run_cmd([ self.sh.run_cmd(['make', '-C', dt_src_dir, LF])
'make', common.Newline, self.sh.copy_dir_if_update_non_recursive(
'-C', dt_src_dir, common.Newline,
])
common.copy_dir_if_update_non_recursive(
srcdir=dt_src_dir, srcdir=dt_src_dir,
destdir=dt_build_dir, destdir=dt_build_dir,
filter_ext='.dtb', filter_ext='.dtb',
@@ -66,49 +64,46 @@ class Gem5Component(common.Component):
# Bootloader 32. # Bootloader 32.
bootloader32_dir = os.path.join(gem5_system_src_dir, 'arm', 'simple_bootloader') bootloader32_dir = os.path.join(gem5_system_src_dir, 'arm', 'simple_bootloader')
# TODO use the buildroot cross compiler here, and remove the dependencies from configure. # TODO use the buildroot cross compiler here, and remove the dependencies from configure.
common.run_cmd([ self.sh.run_cmd([
'make', common.Newline, 'make', LF,
'-C', bootloader32_dir, common.Newline, '-C', bootloader32_dir, LF,
'CROSS_COMPILE=arm-linux-gnueabihf-', common.Newline, 'CROSS_COMPILE=arm-linux-gnueabihf-', LF,
]) ])
# bootloader # bootloader
common.cp(os.path.join(bootloader32_dir, 'boot_emm.arm'), binaries_dir) self.sh.cp(os.path.join(bootloader32_dir, 'boot_emm.arm'), binaries_dir)
# Bootloader 64. # Bootloader 64.
bootloader64_dir = os.path.join(gem5_system_src_dir, 'arm', 'aarch64_bootloader') bootloader64_dir = os.path.join(gem5_system_src_dir, 'arm', 'aarch64_bootloader')
# TODO cross_compile is ignored because the make does not use CC... # TODO cross_compile is ignored because the make does not use CC...
common.run_cmd([ self.sh.run_cmd(['make', '-C', bootloader64_dir, LF])
'make', common.Newline, self.sh.cp(os.path.join(bootloader64_dir, 'boot_emm.arm64'), binaries_dir)
'-C', bootloader64_dir, common.Newline self.sh.run_cmd(
])
common.cp(os.path.join(bootloader64_dir, 'boot_emm.arm64'), binaries_dir)
common.run_cmd(
( (
[ [
'scons', common.Newline, 'scons', LF,
'-j', str(args.nproc), common.Newline, '-j', str(self.env['nproc']), LF,
'--gold-linker', common.Newline, '--gold-linker', LF,
'--ignore-style', common.Newline, '--ignore-style', LF,
common.gem5_executable, common.Newline, self.env['gem5_executable'], LF,
] + ] +
verbose + verbose +
common.add_newlines(args.extra_scons_args) self.sh.add_newlines(self.env['extra_scons_args'])
), ),
cwd=common.gem5_src_dir, cwd=self.env['gem5_source_dir'],
extra_paths=[common.ccache_dir], extra_paths=[self.env['ccache_dir']],
) )
term_src_dir = os.path.join(common.gem5_src_dir, 'util/term') term_src_dir = os.path.join(self.env['gem5_source_dir'], 'util/term')
m5term_build = os.path.join(term_src_dir, 'm5term') m5term_build = os.path.join(term_src_dir, 'm5term')
common.run_cmd(['make', '-C', term_src_dir]) self.sh.run_cmd(['make', '-C', term_src_dir, LF])
if os.path.exists(common.gem5_m5term): if os.path.exists(self.env['gem5_m5term']):
# Otherwise common.cp would fail with "Text file busy" if you # Otherwise self.sh.cp would fail with "Text file busy" if you
# tried to rebuild while running m5term: # tried to rebuild while running m5term:
# https://stackoverflow.com/questions/16764946/what-generates-the-text-file-busy-message-in-unix/52427512#52427512 # https://stackoverflow.com/questions/16764946/what-generates-the-text-file-busy-message-in-unix/52427512#52427512
os.unlink(common.gem5_m5term) os.unlink(self.env['gem5_m5term'])
common.cp(m5term_build, common.gem5_m5term) self.sh.cp(m5term_build, self.env['gem5_m5term'])
def get_build_dir(self, args): def get_build_dir(self):
return common.gem5_build_dir return self.env['gem5_build_dir']
if __name__ == '__main__': if __name__ == '__main__':
Gem5Component().build() Main().cli()

123
build-linux
View File

@@ -4,10 +4,16 @@ import os
import shutil import shutil
import common import common
from shell_helpers import LF
class LinuxComponent(common.Component): class Main(common.BuildCliFunction):
def add_parser_arguments(self, parser): def __init__(self):
parser.add_argument( super().__init__(
description='''\
Build the Linux kernel.
'''
)
self.add_argument(
'--config', default=[], action='append', '--config', default=[], action='append',
help='''\ help='''\
Add a single kernel config configs to the current build. Sample value: Add a single kernel config configs to the current build. Sample value:
@@ -15,14 +21,14 @@ Add a single kernel config configs to the current build. Sample value:
configs. Takes precedence over any config files. configs. Takes precedence over any config files.
''' '''
) )
parser.add_argument( self.add_argument(
'--config-fragment', default=[], action='append', '--config-fragment', default=[], action='append',
help='''\ help='''\
Also use the given kernel configuration fragment file. Also use the given kernel configuration fragment file.
Pass multiple times to use multiple fragment files. Pass multiple times to use multiple fragment files.
''' '''
) )
parser.add_argument( self.add_argument(
'--custom-config-file', '--custom-config-file',
help='''\ help='''\
Ignore all default kernel configurations and use this file instead. Ignore all default kernel configurations and use this file instead.
@@ -30,137 +36,124 @@ Still uses options explicitly passed with `--config` and
`--config-fragment` on top of it. `--config-fragment` on top of it.
''' '''
) )
parser.add_argument( self.add_argument(
'--custom-config-file-gem5', default=False, action='store_true', '--custom-config-file-gem5', default=False,
help='''\ help='''\
Use the gem5 Linux kernel fork config as the custom config file. Use the gem5 Linux kernel fork config as the custom config file.
Ignore --custom-config-file. Ignore --custom-config-file.
''' '''
) )
parser.add_argument( self.add_argument(
'--config-only', default=False, action='store_true', '--config-only', default=False,
help='''\ help='''\
Configure the kernel, but don't build it. Configure the kernel, but don't build it.
''' '''
) )
parser.add_argument( self.add_argument(
'--initramfs', default=False, action='store_true',
)
parser.add_argument(
'--initrd', default=False, action='store_true',
)
parser.add_argument(
'extra_make_args', 'extra_make_args',
default=[], default=[],
metavar='extra-make-args', metavar='extra-make-args',
nargs='*' nargs='*'
) )
def do_build(self, args): def build(self):
build_dir = self.get_build_dir(args) build_dir = self.get_build_dir()
if args.initrd or args.initramfs: if self.env['initrd'] or self.env['initramfs']:
raise Exception('just trolling, --initrd and --initramfs are broken for now') raise Exception('just trolling, --initrd and --initramfs are broken for now')
os.makedirs(build_dir, exist_ok=True) os.makedirs(build_dir, exist_ok=True)
tool = 'gcc' tool = 'gcc'
gcc = common.get_toolchain_tool(tool) gcc = self.get_toolchain_tool(tool)
prefix = gcc[:-len(tool)] prefix = gcc[:-len(tool)]
common_args = { common_args = {
'cwd': common.linux_source_dir, 'cwd': self.env['linux_source_dir'],
} }
ccache = shutil.which('ccache') ccache = shutil.which('ccache')
if ccache is not None: if ccache is not None:
cc = '{} {}'.format(ccache, gcc) cc = '{} {}'.format(ccache, gcc)
else: else:
cc = gcc cc = gcc
if args.verbose: if self.env['verbose']:
verbose = ['V=1'] verbose = ['V=1']
else: else:
verbose = [] verbose = []
common_make_args = [ common_make_args = [
'make', common.Newline, 'make', LF,
'-j', str(args.nproc), common.Newline, '-j', str(self.env['nproc']), LF,
'ARCH={}'.format(common.linux_arch), common.Newline, 'ARCH={}'.format(self.env['linux_arch']), LF,
'CROSS_COMPILE={}'.format(prefix), common.Newline, 'CROSS_COMPILE={}'.format(prefix), LF,
'CC={}'.format(cc), common.Newline, 'CC={}'.format(cc), LF,
'O={}'.format(build_dir), common.Newline, 'O={}'.format(build_dir), LF,
] + verbose ] + verbose
if args.custom_config_file_gem5: if self.env['custom_config_file_gem5']:
custom_config_file = os.path.join(common.linux_source_dir, 'arch', common.linux_arch, 'configs', 'gem5_defconfig') custom_config_file = os.path.join(self.env['linux_source_dir'], 'arch', self.env['linux_arch'], 'configs', 'gem5_defconfig')
else: else:
custom_config_file = args.custom_config_file custom_config_file = self.env['custom_config_file']
if custom_config_file is not None: if custom_config_file is not None:
if not os.path.exists(custom_config_file): if not os.path.exists(custom_config_file):
raise Exception('config fragment file does not exist: {}'.format(args.custom_config_file)) raise Exception('config fragment file does not exist: {}'.format(custom_config_file))
base_config_file = custom_config_file base_config_file = custom_config_file
config_fragments = [] config_fragments = []
else: else:
base_config_file = os.path.join(common.linux_config_dir, 'buildroot-{}'.format(args.arch)) base_config_file = os.path.join(self.env['linux_config_dir'], 'buildroot-{}'.format(self.env['arch']))
config_fragments = ['min', 'default'] config_fragments = ['min', 'default']
for i, config_fragment in enumerate(config_fragments): for i, config_fragment in enumerate(config_fragments):
config_fragments[i] = os.path.join(common.linux_config_dir, config_fragment) config_fragments[i] = os.path.join(self.env['linux_config_dir'], config_fragment)
config_fragments.extend(args.config_fragment) config_fragments.extend(self.env['config_fragment'])
if args.config != []: if self.env['config'] != []:
cli_config_fragment_path = os.path.join(build_dir, 'lkmc_cli_config_fragment') cli_config_fragment_path = os.path.join(build_dir, 'lkmc_cli_config_fragment')
cli_config_str = '\n'.join(args.config) cli_config_str = '\n'.join(self.env['config'])
common.write_string_to_file(cli_config_fragment_path, cli_config_str) self.sh.write_string_to_file(cli_config_fragment_path, cli_config_str)
config_fragments.append(cli_config_fragment_path) config_fragments.append(cli_config_fragment_path)
common.cp( self.sh.cp(
base_config_file, base_config_file,
os.path.join(build_dir, '.config'), os.path.join(build_dir, '.config'),
) )
common.run_cmd( self.sh.run_cmd(
[ [
os.path.join(common.linux_source_dir, 'scripts', 'kconfig', 'merge_config.sh'), common.Newline, os.path.join(self.env['linux_source_dir'], 'scripts', 'kconfig', 'merge_config.sh'), LF,
'-m', common.Newline, '-m', LF,
'-O', build_dir, common.Newline, '-O', build_dir, LF,
os.path.join(build_dir, '.config'), common.Newline, os.path.join(build_dir, '.config'), LF,
] + ] +
common.add_newlines(config_fragments) self.sh.add_newlines(config_fragments)
) )
common.run_cmd( self.sh.run_cmd(
( (
common_make_args + common_make_args +
['olddefconfig', common.Newline] ['olddefconfig', LF]
), ),
**common_args **common_args
) )
if not args.config_only: if not self.env['config_only']:
common.run_cmd( self.sh.run_cmd(
( (
common_make_args + common_make_args +
common.add_newlines(args.extra_make_args) self.sh.add_newlines(self.env['extra_make_args'])
), ),
extra_env={ extra_env={
'KBUILD_BUILD_VERSION': '1', 'KBUILD_BUILD_VERSION': '1',
'KBUILD_BUILD_TIMESTAMP': 'Thu Jan 1 00:00:00 UTC 1970', 'KBUILD_BUILD_TIMESTAMP': 'Thu Jan 1 00:00:00 UTC 1970',
'KBUILD_BUILD_USER': 'lkmc', 'KBUILD_BUILD_USER': 'lkmc',
'KBUILD_BUILD_HOST': common.git_sha(common.linux_source_dir), 'KBUILD_BUILD_HOST': common.git_sha(self.env['linux_source_dir']),
}, },
**common_args **common_args
) )
common.run_cmd( self.sh.run_cmd(
( (
common_make_args + common_make_args +
[ [
'INSTALL_MOD_PATH={}'.format(common.out_rootfs_overlay_dir), common.Newline, 'INSTALL_MOD_PATH={}'.format(self.env['out_rootfs_overlay_dir']), LF,
'modules_install', common.Newline, 'modules_install', LF,
] ]
), ),
**common_args **common_args
) )
# TODO: remove build and source https://stackoverflow.com/questions/13578618/what-does-build-and-source-link-do-in-lib-modules-kernel-version # TODO: remove build and source https://stackoverflow.com/questions/13578618/what-does-build-and-source-link-do-in-lib-modules-kernel-version
# TODO Basically all kernel modules also basically leak full host paths. Just terrible. Buildroot deals with that stuff nicely for us. # TODO Basically all kernel modules also basically leak full host paths. Just terrible. Buildroot deals with that stuff nicely for us.
# common.rmrf() # self.rmrf()
def get_argparse_args(self): def get_build_dir(self):
return { return self.env['linux_build_dir']
'description': '''\
Build the Linux kernel.
'''
}
def get_build_dir(self, args):
return common.linux_build_dir
if __name__ == '__main__': if __name__ == '__main__':
LinuxComponent().build() Main().cli()

55
build-m5
View File

@@ -3,42 +3,47 @@
import os import os
import common import common
from shell_helpers import LF
class M5Component(common.Component): class Main(common.BuildCliFunction):
def get_make_cmd(self, args): def __init__(self):
super().__init__()
def _get_make_cmd(self):
allowed_toolchains = ['buildroot'] allowed_toolchains = ['buildroot']
cc = common.get_toolchain_tool('gcc', allowed_toolchains=allowed_toolchains) cc = self.get_toolchain_tool('gcc', allowed_toolchains=allowed_toolchains)
ld = common.get_toolchain_tool('ld', allowed_toolchains=allowed_toolchains) ld = self.get_toolchain_tool('ld', allowed_toolchains=allowed_toolchains)
if args.arch == 'x86_64': if self.env['arch'] == 'x86_64':
arch = 'x86' arch = 'x86'
else: else:
arch = args.arch arch = self.env['arch']
return [ return [
'make', common.Newline, 'make', LF,
'-j', str(args.nproc), common.Newline, '-j', str(self.env['nproc']), LF,
'-f', 'Makefile.{}'.format(arch), common.Newline, '-f', 'Makefile.{}'.format(arch), LF,
'CC={}'.format(cc), common.Newline, 'CC={}'.format(cc), LF,
'LD={}'.format(ld), common.Newline, 'LD={}'.format(ld), LF,
'PWD={}'.format(common.gem5_m5_src_dir), common.Newline, 'PWD={}'.format(self.env['gem5_m5_source_dir']), LF,
] ]
def do_build(self, args): def build(self):
os.makedirs(common.gem5_m5_build_dir, exist_ok=True) os.makedirs(self.env['gem5_m5_build_dir'], exist_ok=True)
# We must clean first or else the build outputs of one arch can conflict with the other. # We must clean first or else the build outputs of one arch can conflict with the other.
# I should stop being lazy and go actually patch gem5 to support out of tree m5 build... # I should stop being lazy and go actually patch gem5 to support out of tree m5 build...
self.clean(args) self.clean()
common.run_cmd( self.sh.run_cmd(
self.get_make_cmd(args), self._get_make_cmd(),
cwd=common.gem5_m5_src_dir, cwd=self.env['gem5_m5_source_dir'],
) )
os.makedirs(common.out_rootfs_overlay_bin_dir, exist_ok=True) os.makedirs(self.env['out_rootfs_overlay_bin_dir'], exist_ok=True)
common.cp(os.path.join(common.gem5_m5_src_dir, 'm5'), common.out_rootfs_overlay_bin_dir) self.sh.cp(os.path.join(self.env['gem5_m5_source_dir'], 'm5'), self.env['out_rootfs_overlay_bin_dir'])
def clean(self, args): def clean(self):
common.run_cmd( self.sh.run_cmd(
self.get_make_cmd(args) + ['clean', common.Newline], self._get_make_cmd() + ['clean', LF],
cwd=common.gem5_m5_src_dir, cwd=self.env['gem5_m5_source_dir'],
) )
return None
if __name__ == '__main__': if __name__ == '__main__':
M5Component().build() Main().cli()

110
build-modules
View File

@@ -6,32 +6,37 @@ import platform
import shutil import shutil
import common import common
from shell_helpers import LF
class ModulesComponent(common.Component): class Main(common.BuildCliFunction):
def add_parser_arguments(self, parser): def __init__(self):
parser.add_argument( super().__init__(
description='''\
Build our Linux kernel modules without using Buildroot.
See also: https://github.com/cirosantilli/linux-kernel-module-cheat#host
''')
self.add_argument(
'--make-args', '--make-args',
default='', default='',
) )
parser.add_argument( self.add_argument(
'--host', '--host',
action='store_true',
default=False, default=False,
help='''\ help='''\
Build the Linux kernel modules for the host instead of guest. Build the Linux kernel modules for the host instead of guest.
Use the host packaged cross toolchain. Use the host packaged cross toolchain.
''', ''',
) )
parser.add_argument( self.add_argument(
'kernel_modules', 'kernel-modules',
default=[], default=[],
help='Which kernel modules to build. Default: build all', help='Which kernel modules to build. Default: build all',
metavar='kernel-modules',
nargs='*', nargs='*',
) )
def do_build(self, args): def build(self):
build_dir = self.get_build_dir(args) build_dir = self.get_build_dir()
os.makedirs(build_dir, exist_ok=True) os.makedirs(build_dir, exist_ok=True)
# I kid you not, out-of-tree build is not possible, O= does not work as for the kernel build: # I kid you not, out-of-tree build is not possible, O= does not work as for the kernel build:
# #
@@ -42,87 +47,78 @@ Use the host packaged cross toolchain.
# This copies only modified files as per: # This copies only modified files as per:
# https://stackoverflow.com/questions/5718899/building-an-out-of-tree-linux-kernel-module-in-a-separate-object-directory # https://stackoverflow.com/questions/5718899/building-an-out-of-tree-linux-kernel-module-in-a-separate-object-directory
distutils.dir_util.copy_tree( distutils.dir_util.copy_tree(
common.kernel_modules_src_dir, self.env['kernel_modules_src_dir'],
os.path.join(build_dir, common.kernel_modules_subdir), os.path.join(build_dir, self.env['kernel_modules_subdir']),
update=1, update=1,
) )
distutils.dir_util.copy_tree( distutils.dir_util.copy_tree(
common.include_src_dir, self.env['include_src_dir'],
os.path.join(build_dir, common.include_subdir), os.path.join(build_dir, self.env['include_subdir']),
update=1, update=1,
) )
all_kernel_modules = [] all_kernel_modules = []
for basename in os.listdir(common.kernel_modules_src_dir): for basename in os.listdir(self.env['kernel_modules_src_dir']):
src = os.path.join(common.kernel_modules_src_dir, basename) src = os.path.join(self.env['kernel_modules_src_dir'], basename)
if os.path.isfile(src): if os.path.isfile(src):
noext, ext = os.path.splitext(basename) noext, ext = os.path.splitext(basename)
if ext == common.c_ext: if ext == self.env['c_ext']:
all_kernel_modules.append(noext) all_kernel_modules.append(noext)
if args.kernel_modules == []: if self.env['kernel_modules'] == []:
kernel_modules = all_kernel_modules kernel_modules = all_kernel_modules
else: else:
kernel_modules = map(lambda x: os.path.splitext(os.path.split(x)[1])[0], args.kernel_modules) kernel_modules = map(lambda x: os.path.splitext(os.path.split(x)[1])[0], self.env['kernel_modules'])
object_files = map(lambda x: x + common.obj_ext, kernel_modules) object_files = map(lambda x: x + self.env['obj_ext'], kernel_modules)
tool = 'gcc' tool = 'gcc'
if args.host: if self.env['host']:
allowed_toolchains = ['host'] allowed_toolchains = ['host']
build_subdir = common.kernel_modules_build_host_subdir build_subdir = self.env['kernel_modules_build_host_subdir']
else: else:
allowed_toolchains = None allowed_toolchains = None
build_subdir = common.kernel_modules_build_subdir build_subdir = self.env['kernel_modules_build_subdir']
gcc = common.get_toolchain_tool(tool, allowed_toolchains=allowed_toolchains) gcc = self.get_toolchain_tool(tool, allowed_toolchains=allowed_toolchains)
prefix = gcc[:-len(tool)] prefix = gcc[:-len(tool)]
ccache = shutil.which('ccache') ccache = shutil.which('ccache')
if ccache is not None: if ccache is not None:
cc = '{} {}'.format(ccache, gcc) cc = '{} {}'.format(ccache, gcc)
else: else:
cc = gcc cc = gcc
if args.verbose: if self.env['verbose']:
verbose = ['V=1'] verbose = ['V=1']
else: else:
verbose = [] verbose = []
if args.host: if self.env['host']:
linux_dir = os.path.join('/lib', 'modules', platform.uname().release, 'build') linux_dir = os.path.join('/lib', 'modules', platform.uname().release, 'build')
else: else:
linux_dir = common.linux_build_dir linux_dir = self.env['linux_build_dir']
common.run_cmd( self.sh.run_cmd(
( (
[ [
'make', common.Newline, 'make', LF,
'-j', str(args.nproc), common.Newline, '-j', str(self.env['nproc']), LF,
'ARCH={}'.format(common.linux_arch), common.Newline, 'ARCH={}'.format(self.env['linux_arch']), LF,
'CC={}'.format(cc), common.Newline, 'CC={}'.format(cc), LF,
'CROSS_COMPILE={}'.format(prefix), common.Newline, 'CROSS_COMPILE={}'.format(prefix), LF,
'LINUX_DIR={}'.format(linux_dir), common.Newline, 'LINUX_DIR={}'.format(linux_dir), LF,
'M={}'.format(build_subdir), common.Newline, 'M={}'.format(build_subdir), LF,
'OBJECT_FILES={}'.format(' '.join(object_files)), common.Newline, 'OBJECT_FILES={}'.format(' '.join(object_files)), LF,
] + ] +
common.shlex_split(args.make_args) + self.sh.shlex_split(self.env['make_args']) +
verbose verbose
), ),
cwd=os.path.join(common.kernel_modules_build_subdir), cwd=os.path.join(self.env['kernel_modules_build_subdir']),
) )
if not args.host: if not self.env['host']:
common.copy_dir_if_update_non_recursive( self.sh.copy_dir_if_update_non_recursive(
srcdir=common.kernel_modules_build_subdir, srcdir=self.env['kernel_modules_build_subdir'],
destdir=common.out_rootfs_overlay_dir, destdir=self.env['out_rootfs_overlay_dir'],
filter_ext=common.kernel_module_ext, filter_ext=self.env['kernel_module_ext'],
) )
def get_argparse_args(self): def get_build_dir(self):
return { if self.env['host']:
'description': '''\ return self.env['kernel_modules_build_host_dir']
Build our Linux kernel modules without using Buildroot.
See also: https://github.com/cirosantilli/linux-kernel-module-cheat#host
'''
}
def get_build_dir(self, args):
if args.host:
return os.path.join(common.kernel_modules_build_host_dir)
else: else:
return os.path.join(common.kernel_modules_build_dir) return self.env['kernel_modules_build_dir']
if __name__ == '__main__': if __name__ == '__main__':
ModulesComponent().build() Main().cli()

55
build-qemu
View File

@@ -3,61 +3,62 @@
import os import os
import common import common
from shell_helpers import LF
class QemuComponent(common.Component): class Main(common.BuildCliFunction):
def add_parser_arguments(self, parser): def __init__(self):
parser.add_argument( super().__init__()
self.add_argument(
'--userland', '--userland',
default=False, default=False,
action='store_true',
help='Build QEMU user mode instead of system.', help='Build QEMU user mode instead of system.',
) )
parser.add_argument( self.add_argument(
'extra_config_args', 'extra_config_args',
default=[], default=[],
metavar='extra-config-args', metavar='extra-config-args',
nargs='*' nargs='*'
) )
def do_build(self, args): def build(self):
build_dir = self.get_build_dir(args) build_dir = self.get_build_dir()
os.makedirs(build_dir, exist_ok=True) os.makedirs(build_dir, exist_ok=True)
if args.verbose: if self.env['verbose']:
verbose = ['V=1'] verbose = ['V=1']
else: else:
verbose = [] verbose = []
if args.userland: if self.env['userland']:
target_list = '{}-linux-user'.format(args.arch) target_list = '{}-linux-user'.format(self.env['arch'])
else: else:
target_list = '{}-softmmu'.format(args.arch) target_list = '{}-softmmu'.format(self.env['arch'])
common.run_cmd( self.sh.run_cmd(
[ [
os.path.join(common.qemu_src_dir, 'configure'), common.Newline, os.path.join(self.env['qemu_src_dir'], 'configure'), LF,
'--enable-debug', common.Newline, '--enable-debug', LF,
'--enable-trace-backends=simple', common.Newline, '--enable-trace-backends=simple', LF,
'--target-list={}'.format(target_list), common.Newline, '--target-list={}'.format(target_list), LF,
'--enable-sdl', common.Newline, '--enable-sdl', LF,
'--with-sdlabi=2.0', common.Newline, '--with-sdlabi=2.0', LF,
] + ] +
common.add_newlines(args.extra_config_args), self.sh.add_newlines(self.env['extra_config_args']),
extra_paths=[common.ccache_dir], extra_paths=[self.env['ccache_dir']],
cwd=build_dir cwd=build_dir
) )
common.run_cmd( self.sh.run_cmd(
( (
[ [
'make', common.Newline, 'make', LF,
'-j', str(args.nproc), common.Newline, '-j', str(self.env['nproc']), LF,
] + ] +
verbose verbose
), ),
cwd=build_dir, cwd=build_dir,
extra_paths=[common.ccache_dir], extra_paths=[self.env['ccache_dir']],
) )
def get_build_dir(self, args): def get_build_dir(self):
return common.qemu_build_dir return self.env['qemu_build_dir']
if __name__ == '__main__': if __name__ == '__main__':
QemuComponent().build() Main().cli()

81
build-userland
View File

@@ -1,16 +1,19 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
import os import os
import platform
import shlex import shlex
import shutil
import subprocess
import common import common
from shell_helpers import LF
class UserlandComponent(common.Component): class Main(common.BuildCliFunction):
def add_parser_arguments(self, parser): def __init__(self):
parser.add_argument( super().__init__(
description='''\
Build our compiled userland examples.
'''
)
self.add_argument(
'--has-package', '--has-package',
action='append', action='append',
default=[], default=[],
@@ -19,20 +22,19 @@ Indicate that a given package is present in the root filesystem, which
allows us to build examples that rely on it. allows us to build examples that rely on it.
''', ''',
) )
parser.add_argument( self.add_argument(
'--host', '--host',
action='store_true',
default=False, default=False,
help='''\ help='''\
Build the userland programs for the host instead of guest. Build the userland programs for the host instead of guest.
Use the host packaged cross toolchain. Use the host packaged cross toolchain.
''', ''',
) )
parser.add_argument( self.add_argument(
'--make-args', '--make-args',
default='', default='',
) )
parser.add_argument( self.add_argument(
'targets', 'targets',
default=[], default=[],
help='''\ help='''\
@@ -44,49 +46,44 @@ has the OpenBLAS libraries and headers installed.
nargs='*', nargs='*',
) )
def do_build(self, args): def build(self):
build_dir = self.get_build_dir(args) build_dir = self.get_build_dir()
os.makedirs(build_dir, exist_ok=True) os.makedirs(build_dir, exist_ok=True)
if args.host: if self.env['host']:
allowed_toolchains = ['host'] allowed_toolchains = ['host']
else: else:
allowed_toolchains = ['buildroot'] allowed_toolchains = ['buildroot']
cc = common.get_toolchain_tool('gcc', allowed_toolchains=allowed_toolchains) cc = self.get_toolchain_tool('gcc', allowed_toolchains=allowed_toolchains)
cxx = common.get_toolchain_tool('g++', allowed_toolchains=allowed_toolchains) cxx = self.get_toolchain_tool('g++', allowed_toolchains=allowed_toolchains)
common.run_cmd( self.sh.run_cmd(
( (
[ [
'make', common.Newline, 'make', LF,
'-j', str(args.nproc), common.Newline, '-j', str(self.env['nproc']), LF,
'ARCH={}'.format(args.arch), common.Newline, 'ARCH={}'.format(self.env['arch']), LF,
'CCFLAGS_SCRIPT={} {}'.format('-I', common.userland_src_dir), common.Newline, 'CCFLAGS_SCRIPT={} {}'.format('-I', self.env['userland_src_dir']), LF,
'COMMON_DIR={}'.format(common.root_dir), common.Newline, 'COMMON_DIR={}'.format(self.env['root_dir']), LF,
'CC={}'.format(cc), common.Newline, 'CC={}'.format(cc), LF,
'CXX={}'.format(cxx), common.Newline, 'CXX={}'.format(cxx), LF,
'PKG_CONFIG={}'.format(common.buildroot_pkg_config), common.Newline, 'PKG_CONFIG={}'.format(self.env['buildroot_pkg_config']), LF,
'STAGING_DIR={}'.format(common.buildroot_staging_dir), common.Newline, 'STAGING_DIR={}'.format(self.env['buildroot_staging_dir']), LF,
'OUT_DIR={}'.format(build_dir), common.Newline, 'OUT_DIR={}'.format(build_dir), LF,
] + ] +
common.add_newlines(['HAS_{}=y'.format(package.upper()) for package in args.has_package]) + self.sh.add_newlines(['HAS_{}=y'.format(package.upper()) for package in self.env['has_package']]) +
shlex.split(args.make_args) + shlex.split(self.env['make_args']) +
common.add_newlines([os.path.join(build_dir, os.path.splitext(os.path.split(target)[1])[0]) + common.userland_build_ext for target in args.targets]) self.sh.add_newlines([os.path.join(build_dir, os.path.splitext(os.path.split(target)[1])[0]) + self.env['userland_build_ext'] for target in self.env['targets']])
), ),
cwd=common.userland_src_dir, cwd=self.env['userland_src_dir'],
extra_paths=[common.ccache_dir], extra_paths=[self.env['ccache_dir']],
) )
common.copy_dir_if_update_non_recursive( self.sh.copy_dir_if_update_non_recursive(
srcdir=build_dir, srcdir=build_dir,
destdir=common.out_rootfs_overlay_dir, destdir=self.env['out_rootfs_overlay_dir'],
filter_ext=common.userland_build_ext, filter_ext=self.env['userland_build_ext'],
) )
def get_argparse_args(self): def get_build_dir(self):
return { return self.env['userland_build_dir']
'description': 'Build our compiled userland examples',
}
def get_build_dir(self, args):
return common.userland_build_dir
if __name__ == '__main__': if __name__ == '__main__':
UserlandComponent().build() Main().cli()

262
cli_function.py Executable file
View File

@@ -0,0 +1,262 @@
#!/usr/bin/env python3
import argparse
import imp
import os
class Argument:
def __init__(
self,
long_or_short_1,
long_or_short_2=None,
default=None,
help=None,
nargs=None,
**kwargs
):
if long_or_short_2 is None:
shortname = None
longname = long_or_short_1
else:
shortname = long_or_short_1
longname = long_or_short_2
self.args = []
# argparse is crappy and cannot tell us if arguments were given or not.
# We need that information to decide if the config file should override argparse or not.
# So we just use None as a sentinel.
self.kwargs = {'default': None}
if shortname is not None:
self.args.append(shortname)
if longname[0] == '-':
self.args.append(longname)
self.key = longname.lstrip('-').replace('-', '_')
self.is_option = True
else:
self.key = longname.replace('-', '_')
self.args.append(self.key)
self.kwargs['metavar'] = longname
self.is_option = False
if default is not None and nargs is None:
self.kwargs['nargs'] = '?'
if nargs is not None:
self.kwargs['nargs'] = nargs
if default is True:
bool_action = 'store_false'
self.is_bool = True
elif default is False:
bool_action = 'store_true'
self.is_bool = True
else:
self.is_bool = False
if default is None and nargs in ('*', '+'):
default = []
if self.is_bool and not 'action' in kwargs:
self.kwargs['action'] = bool_action
if help is not None:
if not self.is_bool and default:
help += ' Default: {}'.format(default)
self.kwargs['help'] = help
self.optional = (
default is not None or
self.is_bool or
self.is_option or
nargs in ('?', '*', '+')
)
self.kwargs.update(kwargs)
self.default = default
self.longname = longname
def __str__(self):
return str(self.args) + ' ' + str(self.kwargs)
class CliFunction:
'''
Represent a function that can be called either from Python code, or
from the command line.
Features:
* single argument description in format very similar to argparse
* handle default arguments transparently in both cases
* expose a configuration file mechanism to get default parameters from a file
* fix some argparse.ArgumentParser() annoyances:
** allow dashes in positional arguments:
https://stackoverflow.com/questions/12834785/having-options-in-argparse-with-a-dash
** boolean defaults automatically use store_true or store_false, and add a --no-* CLI
option to invert them if set from the config
This somewhat duplicates: https://click.palletsprojects.com but:
* that decorator API is insane
* CLI + Python for single functions was wontfixed: https://github.com/pallets/click/issues/40
'''
def __call__(self, **args):
'''
Python version of the function call.
:type arguments: Dict
'''
args_with_defaults = args.copy()
# Add missing args from config file.
if 'config_file' in args_with_defaults and args_with_defaults['config_file'] is not None:
config_file = args_with_defaults['config_file']
else:
config_file = self._config_file
if os.path.exists(config_file):
config_configs = {}
config = imp.load_source('config', config_file)
config.set_args(config_configs)
for key in config_configs:
if key not in self._all_keys:
raise Exception('Unknown key in config file: ' + key)
if (not key in args_with_defaults) or args_with_defaults[key] is None:
args_with_defaults[key] = config_configs[key]
# Add missing args from hard-coded defaults.
for argument in self._arguments:
key = argument.key
if (not key in args_with_defaults) or args_with_defaults[key] is None:
if argument.optional:
args_with_defaults[key] = argument.default
else:
raise Exception('Value not given for mandatory argument: ' + key)
return self.main(**args_with_defaults)
def __init__(self, config_file=None, description=None):
self._all_keys = set()
self._arguments = []
self._config_file = config_file
self._description = description
if self._config_file is not None:
self.add_argument(
'--config-file',
default=self._config_file,
help='Path to the configuration file to use'
)
def __str__(self):
return '\n'.join(str(arg) for arg in self._arguments)
def add_argument(
self,
*args,
**kwargs
):
argument = Argument(*args, **kwargs)
self._arguments.append(argument)
self._all_keys.add(argument.key)
def cli(self, cli_args=None):
'''
Call the function from the CLI. Parse command line arguments
to get all arguments.
'''
parser = argparse.ArgumentParser(
description=self._description,
formatter_class=argparse.RawTextHelpFormatter,
)
for argument in self._arguments:
parser.add_argument(*argument.args, **argument.kwargs)
if argument.is_bool:
new_longname = '--no' + argument.longname[1:]
kwargs = argument.kwargs.copy()
kwargs['default'] = not argument.default
if kwargs['action'] == 'store_false':
kwargs['action'] = 'store_true'
elif kwargs['action'] == 'store_true':
kwargs['action'] = 'store_false'
if 'help' in kwargs:
del kwargs['help']
parser.add_argument(new_longname, dest=argument.key, **kwargs)
args = parser.parse_args(args=cli_args)
return self(**vars(args))
def main(self, **kwargs):
'''
Do the main function call work.
:type arguments: Dict
'''
raise NotImplementedError
if __name__ == '__main__':
class OneCliFunction(CliFunction):
def __init__(self):
super().__init__(
config_file='cli_function_test_config.py',
description = '''\
Description of this
amazing function!
''',
)
self.add_argument('-a', '--asdf', default='A', help='Help for asdf'),
self.add_argument('-q', '--qwer', default='Q', help='Help for qwer'),
self.add_argument('-b', '--bool', default=True, help='Help for bool'),
self.add_argument('--bool-cli', default=False, help='Help for bool'),
self.add_argument('--bool-nargs', default=False, nargs='?', action='store', const='')
self.add_argument('--no-default', help='Help for no-bool'),
self.add_argument('pos-mandatory', help='Help for pos-mandatory', type=int),
self.add_argument('pos-optional', default=0, help='Help for pos-optional', type=int),
self.add_argument('args-star', help='Help for args-star', nargs='*'),
def main(self, **kwargs):
del kwargs['config_file']
return kwargs
one_cli_function = OneCliFunction()
# Default code call.
default = one_cli_function(pos_mandatory=1)
assert default == {
'asdf': 'A',
'qwer': 'Q',
'bool': True,
'bool_nargs': False,
'bool_cli': True,
'no_default': None,
'pos_mandatory': 1,
'pos_optional': 0,
'args_star': []
}
# Default CLI call.
out = one_cli_function.cli(['1'])
assert out == default
# asdf
out = one_cli_function(pos_mandatory=1, asdf='B')
assert out['asdf'] == 'B'
out['asdf'] = default['asdf']
assert(out == default)
# asdf and qwer
out = one_cli_function(pos_mandatory=1, asdf='B', qwer='R')
assert out['asdf'] == 'B'
assert out['qwer'] == 'R'
out['asdf'] = default['asdf']
out['qwer'] = default['qwer']
assert(out == default)
if '--bool':
out = one_cli_function(pos_mandatory=1, bool=False)
cli_out = one_cli_function.cli(['--bool', '1'])
assert out == cli_out
assert out['bool'] == False
out['bool'] = default['bool']
assert(out == default)
if '--bool-nargs':
out = one_cli_function(pos_mandatory=1, bool_nargs=True)
assert out['bool_nargs'] == True
out['bool_nargs'] = default['bool_nargs']
assert(out == default)
out = one_cli_function(pos_mandatory=1, bool_nargs='asdf')
assert out['bool_nargs'] == 'asdf'
out['bool_nargs'] = default['bool_nargs']
assert(out == default)
# Force a boolean value set on the config to be False on CLI.
assert one_cli_function.cli(['--no-bool-cli', '1'])['bool_cli'] is False
# CLI call.
print(one_cli_function.cli())

5
cli_function_test_config.py Normal file
View File

@@ -0,0 +1,5 @@
def set_args(args):
'''
:type args: Dict[str, Any]
'''
args['bool_cli'] = True

1673
common.py
View File

File diff suppressed because it is too large Load Diff

29
copy-overlay
View File

@@ -5,25 +5,26 @@ import os
import shutil import shutil
import common import common
from shell_helpers import LF
class CopyOverlayComponent(common.Component): class Main(common.BuildCliFunction):
def do_build(self, args): def __init__(self):
distutils.dir_util.copy_tree( super().__init__(
common.rootfs_overlay_dir, description='''\
common.out_rootfs_overlay_dir,
update=1,
)
def get_argparse_args(self):
return {
'description': '''\
Copy our git tracked rootfs_overlay to the final generated rootfs_overlay Copy our git tracked rootfs_overlay to the final generated rootfs_overlay
that also contains generated build outputs. This has the following advantages that also contains generated build outputs. This has the following advantages
over just adding that to BR2_ROOTFS_OVERLAY: over just adding that to BR2_ROOTFS_OVERLAY:
- also works for non Buildroot root filesystesms - also works for non Buildroot root filesystesms
- places everything in one place for a nice 9P mount - places everything in one place for a nice 9P mount
''', ''')
}
def build(self):
# TODO: print rsync equivalent, move into shell_helpers.
distutils.dir_util.copy_tree(
self.env['rootfs_overlay_dir'],
self.env['out_rootfs_overlay_dir'],
update=1,
)
if __name__ == '__main__': if __name__ == '__main__':
CopyOverlayComponent().build() Main().cli()

13
gem5-shell
View File

@@ -3,14 +3,15 @@
import sys import sys
import common import common
from shell_helpers import LF
parser = common.get_argparse( parser = self.get_argparse(
default_args={'gem5':True}, default_args={'gem5':True},
argparse_args={'description':'Connect a terminal to a running gem5 instance'} argparse_args={'description':'Connect a terminal to a running gem5 instance'}
) )
args = common.setup(parser) args = self.setup(parser)
sys.exit(common.run_cmd([ sys.exit(self.sh.run_cmd([
common.gem5_m5term, common.Newline, kwargs['gem5_m5term'], LF,
'localhost', common.Newline, 'localhost', LF,
str(common.gem5_telnet_port), common.Newline, str(kwargs['gem5_telnet_port']), LF,
])) ]))

6
gem5-stat
View File

@@ -1,6 +1,6 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
import common import common
parser = common.get_argparse( parser = self.get_argparse(
argparse_args={'description':'Get the value of a gem5 stat from the stats.txt file.'} argparse_args={'description':'Get the value of a gem5 stat from the stats.txt file.'}
) )
parser.add_argument( parser.add_argument(
@@ -9,6 +9,6 @@ parser.add_argument(
help='Python regexp matching the full stat name of interest', help='Python regexp matching the full stat name of interest',
nargs='?', nargs='?',
) )
args = common.setup(parser) args = self.setup(parser)
stats = common.get_stats(args.stat) stats = self.get_stats(kwargs['stat'])
print('\n'.join(stats)) print('\n'.join(stats))

13
getvar
View File

@@ -3,14 +3,15 @@
import types import types
import common import common
from shell_helpers import LF
parser = common.get_argparse(argparse_args={ parser = self.get_argparse(argparse_args={
'description': '''Print the value of a common.py variable. 'description': '''Print the value of a kwargs['py'] variable.
This is useful to: This is useful to:
* give dry commands on the README that don't change when we refactor directory structure * give dry commands on the README that don't change when we refactor directory structure
* create simple bash scripts that call use common.py variables * create simple bash scripts that call use kwargs['py'] variables
For example, to get the Buildroot output directory for an ARM build, use: For example, to get the Buildroot output directory for an ARM build, use:
@@ -27,9 +28,9 @@ List all available variables:
''' '''
}) })
parser.add_argument('variable', nargs='?') parser.add_argument('variable', nargs='?')
args = common.setup(parser) args = self.setup(parser)
if args.variable: if kwargs['variable']:
print(getattr(common, args.variable)) print(getattr(common, kwargs['variable']))
else: else:
for attr in dir(common): for attr in dir(common):
if not attr.startswith('__'): if not attr.startswith('__'):

11
qemu-monitor
View File

@@ -5,10 +5,11 @@ import sys
import telnetlib import telnetlib
import common import common
from shell_helpers import LF
prompt = b'\n(qemu) ' prompt = b'\n(qemu) '
parser = common.get_argparse({ parser = self.get_argparse({
'description': '''\ 'description': '''\
Run a command on the QEMU monitor of a running QEMU instance Run a command on the QEMU monitor of a running QEMU instance
@@ -21,24 +22,24 @@ parser.add_argument(
help='If given, run this command and quit', help='If given, run this command and quit',
nargs='*', nargs='*',
) )
args = common.setup(parser) args = self.setup(parser)
def write_and_read(tn, cmd, prompt): def write_and_read(tn, cmd, prompt):
tn.write(cmd.encode('utf-8')) tn.write(cmd.encode('utf-8'))
return '\n'.join(tn.read_until(prompt).decode('utf-8').splitlines()[1:])[:-len(prompt)] return '\n'.join(tn.read_until(prompt).decode('utf-8').splitlines()[1:])[:-len(prompt)]
with telnetlib.Telnet('localhost', common.qemu_monitor_port) as tn: with telnetlib.Telnet('localhost', kwargs['qemu_monitor_port']) as tn:
# Couldn't disable server echo, so just removing the write for now. # Couldn't disable server echo, so just removing the write for now.
# https://stackoverflow.com/questions/12421799/how-to-disable-telnet-echo-in-python-telnetlib # https://stackoverflow.com/questions/12421799/how-to-disable-telnet-echo-in-python-telnetlib
# sock = tn.get_socket() # sock = tn.get_socket()
# sock.send(telnetlib.IAC + telnetlib.WILL + telnetlib.ECHO) # sock.send(telnetlib.IAC + telnetlib.WILL + telnetlib.ECHO)
if os.isatty(sys.stdin.fileno()): if os.isatty(sys.stdin.fileno()):
if args.command == []: if kwargs['command'] == []:
print(tn.read_until(prompt).decode('utf-8'), end='') print(tn.read_until(prompt).decode('utf-8'), end='')
tn.interact() tn.interact()
else: else:
tn.read_until(prompt) tn.read_until(prompt)
print(write_and_read(tn, ' '.join(args.command) + '\n', prompt)) print(write_and_read(tn, ' '.join(kwargs['command']) + '\n', prompt))
else: else:
tn.read_until(prompt) tn.read_until(prompt)
print(write_and_read(tn, sys.stdin.read() + '\n', prompt)) print(write_and_read(tn, sys.stdin.read() + '\n', prompt))

17
qemu-trace2txt
View File

@@ -5,22 +5,23 @@ import subprocess
import sys import sys
import common import common
from shell_helpers import LF
def main(): def main():
return common.run_cmd( return self.sh.run_cmd(
[ [
os.path.join(common.qemu_src_dir, 'scripts/simpletrace.py'), common.Newline, os.path.join(kwargs['qemu_src_dir'], 'scripts/simpletrace.py'), LF,
os.path.join(common.qemu_build_dir, 'trace-events-all'), common.Newline, os.path.join(kwargs['qemu_build_dir'], 'trace-events-all'), LF,
os.path.join(common.qemu_trace_file), common.Newline, os.path.join(kwargs['qemu_trace_file']), LF,
], ],
cmd_file=os.path.join(common.run_dir, 'qemu-trace2txt'), cmd_file=os.path.join(kwargs['run_dir'], 'qemu-trace2txt'),
out_file=common.qemu_trace_txt_file, out_file=kwargs['qemu_trace_txt_file'],
show_stdout=False, show_stdout=False,
) )
if __name__ == '__main__': if __name__ == '__main__':
parser = common.get_argparse(argparse_args={ parser = self.get_argparse(argparse_args={
'description': 'Convert a QEMU `-trace exec_tb` to text form.' 'description': 'Convert a QEMU `-trace exec_tb` to text form.'
}) })
args = common.setup(parser) args = self.setup(parser)
sys.exit(main()) sys.exit(main())

14
release
View File

@@ -10,23 +10,23 @@ import subprocess
import time import time
import common import common
release_zip = imp.load_source('release_zip', os.path.join(common.root_dir, 'release-zip')) release_zip = imp.load_source('release_zip', os.path.join(kwargs['root_dir'], 'release-zip'))
release_upload = imp.load_source('release_upload', os.path.join(common.root_dir, 'release-upload')) release_upload = imp.load_source('release_upload', os.path.join(kwargs['root_dir'], 'release-upload'))
start_time = time.time() start_time = time.time()
# TODO factor those out so we don't redo the same thing multiple times. # TODO factor those out so we don't redo the same thing multiple times.
# subprocess.check_call([os.path.join(common.root_dir, 'test')]) # subprocess.check_call([os.path.join(kwargs['root_dir'], 'test')])
# subprocess.check_call([os.path.join(common.root_dir, ''bench-all', '-A', '-u']) # subprocess.check_call([os.path.join(kwargs['root_dir'], ''bench-all', '-A', '-u'])
# A clean release requires a full rebuild unless we hack it :-( # A clean release requires a full rebuild unless we hack it :-(
# We can't just use our current build as it contains packages we've # We can't just use our current build as it contains packages we've
# installed in random experiments. And with EXT2: we can't easily # installed in random experiments. And with EXT2: we can't easily
# know what the smallest root filesystem size is and use it either... # know what the smallest root filesystem size is and use it either...
# https://stackoverflow.com/questions/47320800/how-to-clean-only-target-in-buildroot # https://stackoverflow.com/questions/47320800/how-to-clean-only-target-in-buildroot
subprocess.check_call([os.path.join(common.root_dir, 'configure'), '--all']) subprocess.check_call([os.path.join(kwargs['root_dir'], 'configure'), '--all'])
subprocess.check_call([os.path.join(common.root_dir, 'build'), '--all-archs', 'release']) subprocess.check_call([os.path.join(kwargs['root_dir'], 'build'), '--all-archs', 'release'])
release_zip.main() release_zip.main()
subprocess.check_call(['git', 'push']) subprocess.check_call(['git', 'push'])
release_upload.main() release_upload.main()
end_time = time.time() end_time = time.time()
common.print_time(end_time - start_time) self.print_time(end_time - start_time)

3
release-download-latest
View File

@@ -10,7 +10,8 @@ https://stackoverflow.com/questions/24987542/is-there-a-link-to-github-for-downl
import urllib.request import urllib.request
import common import common
from shell_helpers import LF
_json = common.github_make_request(path='/releases') _json = self.github_make_request(path='/releases')
asset = _json[0]['assets'][0] asset = _json[0]['assets'][0]
urllib.request.urlretrieve(asset['browser_download_url'], asset['name']) urllib.request.urlretrieve(asset['browser_download_url'], asset['name'])

17
release-upload
View File

@@ -16,15 +16,16 @@ import sys
import urllib.error import urllib.error
import common import common
from shell_helpers import LF
def main(): def main():
repo = common.github_repo_id repo = kwargs['github_repo_id']
tag = 'sha-{}'.format(common.sha) tag = 'sha-{}'.format(kwargs['sha'])
upload_path = common.release_zip_file upload_path = kwargs['release_zip_file']
# Check the release already exists. # Check the release already exists.
try: try:
_json = common.github_make_request(path='/releases/tags/' + tag) _json = self.github_make_request(path='/releases/tags/' + tag)
except urllib.error.HTTPError as e: except urllib.error.HTTPError as e:
if e.code == 404: if e.code == 404:
release_exists = False release_exists = False
@@ -36,7 +37,7 @@ def main():
# Create release if not yet created. # Create release if not yet created.
if not release_exists: if not release_exists:
_json = common.github_make_request( _json = self.github_make_request(
authenticate=True, authenticate=True,
data=json.dumps({ data=json.dumps({
'tag_name': tag, 'tag_name': tag,
@@ -50,12 +51,12 @@ def main():
asset_name = os.path.split(upload_path)[1] asset_name = os.path.split(upload_path)[1]
# Clear the prebuilts for a upload. # Clear the prebuilts for a upload.
_json = common.github_make_request( _json = self.github_make_request(
path=('/releases/' + str(release_id) + '/assets'), path=('/releases/' + str(release_id) + '/assets'),
) )
for asset in _json: for asset in _json:
if asset['name'] == asset_name: if asset['name'] == asset_name:
_json = common.github_make_request( _json = self.github_make_request(
authenticate=True, authenticate=True,
path=('/releases/assets/' + str(asset['id'])), path=('/releases/assets/' + str(asset['id'])),
method='DELETE', method='DELETE',
@@ -65,7 +66,7 @@ def main():
# Upload the prebuilt. # Upload the prebuilt.
with open(upload_path, 'br') as myfile: with open(upload_path, 'br') as myfile:
content = myfile.read() content = myfile.read()
_json = common.github_make_request( _json = self.github_make_request(
authenticate=True, authenticate=True,
data=content, data=content,
extra_headers={'Content-Type': 'application/zip'}, extra_headers={'Content-Type': 'application/zip'},

17
release-zip
View File

@@ -9,16 +9,17 @@ import subprocess
import zipfile import zipfile
import common import common
from shell_helpers import LF
def main(): def main():
os.makedirs(common.release_dir, exist_ok=True) os.makedirs(kwargs['release_dir'], exist_ok=True)
if os.path.exists(common.release_zip_file): if os.path.exists(kwargs['release_zip_file']):
os.unlink(common.release_zip_file) os.unlink(kwargs['release_zip_file'])
zipf = zipfile.ZipFile(common.release_zip_file, 'w', zipfile.ZIP_DEFLATED) zipf = zipfile.ZipFile(kwargs['release_zip_file'], 'w', zipfile.ZIP_DEFLATED)
for arch in common.all_archs: for arch in kwargs['all_archs']:
common.setup(common.get_argparse(default_args={'arch': arch})) self.setup(common.get_argparse(default_args={'arch': arch}))
zipf.write(common.qcow2_file, arcname=os.path.relpath(common.qcow2_file, common.root_dir)) zipf.write(kwargs['qcow2_file'], arcname=os.path.relpath(kwargs['qcow2_file'], kwargs['root_dir']))
zipf.write(common.linux_image, arcname=os.path.relpath(common.linux_image, common.root_dir)) zipf.write(kwargs['linux_image'], arcname=os.path.relpath(kwargs['linux_image'], kwargs['root_dir']))
zipf.close() zipf.close()
if __name__ == '__main__': if __name__ == '__main__':

1096
run
View File

File diff suppressed because it is too large Load Diff

49
run-docker
View File

@@ -4,50 +4,51 @@ import argparse
import os import os
import common import common
from shell_helpers import LF
container_name = common.repo_short_id container_name = kwargs['repo_short_id']
container_hostname = common.repo_short_id container_hostname = kwargs['repo_short_id']
image_name = common.repo_short_id image_name = kwargs['repo_short_id']
target_dir = '/root/{}'.format(common.repo_short_id) target_dir = '/root/{}'.format(kwargs['repo_short_id'])
docker = ['sudo', 'docker'] docker = ['sudo', 'docker']
def create(args): def create(args):
common.run_cmd(docker + ['build', '-t', image_name, '.', common.Newline]) self.sh.run_cmd(docker + ['build', '-t', image_name, '.', LF])
# --privileged for KVM: # --privileged for KVM:
# https://stackoverflow.com/questions/48422001/launching-qemu-kvm-from-inside-docker-container # https://stackoverflow.com/questions/48422001/launching-qemu-kvm-from-inside-docker-container
common.run_cmd( self.sh.run_cmd(
docker + docker +
[ [
'create', common.Newline, 'create', LF,
'--hostname', container_hostname, common.Newline, '--hostname', container_hostname, LF,
'-i', common.Newline, '-i', LF,
'--name', container_name, common.Newline, '--name', container_name, LF,
'--net', 'host', common.Newline, '--net', 'host', LF,
'--privileged', common.Newline, '--privileged', LF,
'-t', common.Newline, '-t', LF,
'-w', target_dir, common.Newline, '-w', target_dir, LF,
'-v', '{}:{}'.format(os.getcwd(), target_dir), common.Newline, '-v', '{}:{}'.format(os.getcwd(), target_dir), LF,
image_name, image_name,
] ]
) )
def destroy(args): def destroy(args):
stop(args) stop(args)
common.run_cmd(docker + ['rm', container_name, common.Newline]) self.sh.run_cmd(docker + ['rm', container_name, LF])
common.run_cmd(docker + ['rmi', image_name, common.Newline]) self.sh.run_cmd(docker + ['rmi', image_name, LF])
def sh(args): def sh(args):
start(args) start(args)
if args: if args:
sh_args = args sh_args = args
else: else:
sh_args = ['bash'] sh_args = ['bash']
common.run_cmd( self.sh.run_cmd(
docker + ['exec', '-i', '-t', container_name] + docker + ['exec', '-i', '-t', container_name] +
sh_args + sh_args +
[common.Newline], [LF],
) )
def start(args): def start(args):
common.run_cmd(docker + ['start', container_name, common.Newline]) self.sh.run_cmd(docker + ['start', container_name, LF])
def stop(args): def stop(args):
common.run_cmd(docker + ['stop', container_name, common.Newline]) self.sh.run_cmd(docker + ['stop', container_name, LF])
cmd_action_map = { cmd_action_map = {
'create': lambda args: create(args), 'create': lambda args: create(args),
'DESTROY': lambda args: destroy(args), 'DESTROY': lambda args: destroy(args),
@@ -58,7 +59,7 @@ cmd_action_map = {
parser = argparse.ArgumentParser() parser = argparse.ArgumentParser()
parser.add_argument('cmd', choices=cmd_action_map) parser.add_argument('cmd', choices=cmd_action_map)
parser.add_argument('args', nargs='*') parser.add_argument('args', nargs='*')
common.add_dry_run_argument(parser) self.add_dry_run_argument(parser)
args = parser.parse_args() args = parser.parse_args()
common.setup_dry_run_arguments(args) self.setup_dry_run_arguments(args)
cmd_action_map[args.cmd](args.args) cmd_action_map[kwargs['cmd']](kwargs['args'])

302
run-gdb
View File

@@ -7,18 +7,7 @@ import subprocess
import sys import sys
import common import common
from shell_helpers import LF
defaults = {
'after': '',
'before': '',
'break_at': None,
'kgdb': False,
'no_continue': False,
'no_lxsymbols': False,
'test': False,
'sim': False,
'userland': None,
}
class GdbTestcase: class GdbTestcase:
def __init__( def __init__(
@@ -34,8 +23,8 @@ class GdbTestcase:
''' '''
self.prompt = '\(gdb\) ' self.prompt = '\(gdb\) '
self.source_path = source_path self.source_path = source_path
common.print_cmd(cmd) self.print_cmd(cmd)
cmd = common.strip_newlines(cmd) cmd = self.strip_newlines(cmd)
import pexpect import pexpect
self.child = pexpect.spawn( self.child = pexpect.spawn(
cmd[0], cmd[0],
@@ -84,155 +73,146 @@ class GdbTestcase:
self.child.sendline(line) self.child.sendline(line)
self.child.expect(self.prompt) self.child.expect(self.prompt)
def main(args, extra_args=None): class Main(common.LkmcCliFunction):
''' def __init__(self):
:param args: argparse parse_argument() output. Must contain all the common options, super().__init__(description='''\
but does not need GDB specific ones. Connect with GDB to an emulator to debug Linux itself
:type args: argparse.Namespace ''')
self.add_argument(
:param extra_args: extra arguments to be added to args '-A', '--after', default='',
:type extra_args: Dict[str,Any] help='Pass extra arguments to GDB, to be appended after all other arguments'
:return: GDB exit status
:rtype: int
'''
global defaults
args = common.resolve_args(defaults, args, extra_args)
after = common.shlex_split(args.after)
before = common.shlex_split(args.before)
no_continue = args.no_continue
if args.test:
no_continue = True
before.extend([
'-q', common.Newline,
'-nh', common.Newline,
'-ex', 'set confirm off', common.Newline
])
elif args.verbose:
# The output of this would affect the tests.
# https://stackoverflow.com/questions/13496389/gdb-remote-protocol-how-to-analyse-packets
# Also be opinionated and set remotetimeout to allow you to step debug the emulator at the same time.
before.extend([
'-ex', 'set debug remote 1', common.Newline,
'-ex', 'set remotetimeout 99999', common.Newline,
])
if args.break_at is not None:
break_at = ['-ex', 'break {}'.format(args.break_at), common.Newline]
else:
break_at = []
linux_full_system = (common.baremetal is None and args.userland is None)
if args.userland:
image = common.resolve_userland(args.userland)
elif common.baremetal:
image = common.image
test_script_path = os.path.splitext(common.source_path)[0] + '.py'
else:
image = common.vmlinux
if common.baremetal:
allowed_toolchains = ['crosstool-ng', 'buildroot', 'host']
else:
allowed_toolchains = ['buildroot', 'crosstool-ng', 'host']
cmd = (
[common.get_toolchain_tool('gdb', allowed_toolchains=allowed_toolchains), common.Newline] +
before +
['-q', common.Newline]
)
if linux_full_system:
cmd.extend(['-ex', 'add-auto-load-safe-path {}'.format(common.linux_build_dir), common.Newline])
if args.sim:
target = 'sim'
else:
if args.kgdb:
port = common.extra_serial_port
else:
port = common.gdb_port
target = 'remote localhost:{}'.format(port)
cmd.extend([
'-ex', 'file {}'.format(image), common.Newline,
'-ex', 'target {}'.format(target), common.Newline,
])
if not args.kgdb:
cmd.extend(break_at)
if not no_continue:
# ## lx-symbols
#
# ### lx-symbols after continue
#
# lx symbols must be run after continue.
#
# running it immediately after the connect on the bootloader leads to failure,
# likely because kernel structure on which it depends are not yet available.
#
# With this setup, continue runs, and lx-symbols only runs when a break happens,
# either by hitting the breakpoint, or by entering Ctrl + C.
#
# Sure, if the user sets a break on a raw address of the bootloader,
# problems will still arise, but let's think about that some other time.
#
# ### lx-symbols autoload
#
# The lx-symbols commands gets loaded through the file vmlinux-gdb.py
# which gets put on the kernel build root when python debugging scripts are enabled.
cmd.extend(['-ex', 'continue', common.Newline])
if not args.no_lxsymbols and linux_full_system:
cmd.extend(['-ex', 'lx-symbols {}'.format(common.kernel_modules_build_subdir), common.Newline])
cmd.extend(after)
if args.test:
GdbTestcase(
common.source_path,
test_script_path,
cmd,
verbose=args.verbose,
) )
else: self.add_argument(
# I would rather have cwd be out_rootfs_overlay_dir, '--before', default='',
# but then lx-symbols cannot fine the vmlinux and fails with: help='Pass extra arguments to GDB to be prepended before any of the arguments passed by this script'
# vmlinux: No such file or directory.
return common.run_cmd(
cmd,
cmd_file=os.path.join(common.run_dir, 'run-gdb.sh'),
cwd=common.linux_build_dir
) )
self.add_argument(
if __name__ == '__main__': 'break_at', nargs='?',
parser = common.get_argparse(argparse_args={'description': 'Connect with GDB to an emulator to debug Linux itself'}) help='Extra options to append at the end of the emulator command line'
parser.add_argument( )
'-A', '--after', default=defaults['after'], self.add_argument(
help='Pass extra arguments to GDB, to be appended after all other arguments' '-k', '--kgdb', default=False,
) )
parser.add_argument( self.add_argument(
'--before', default=defaults['before'], '-C', '--no-continue', default=False,
help='Pass extra arguments to GDB to be prepended before any of the arguments passed by this script' help="Don't run continue after connecting"
) )
parser.add_argument( self.add_argument(
'-C', '--no-continue', default=defaults['no_continue'], action='store_true', '-X', '--no-lxsymbols', default=False,
help="Don't run continue after connecting" )
) self.add_argument(
parser.add_argument( '--test', default=False,
'-k', '--kgdb', default=defaults['kgdb'], action='store_true' help='''\
)
parser.add_argument(
'--sim', default=defaults['sim'], action='store_true',
help='''Use the built-in GDB CPU simulator
See: https://github.com/cirosantilli/linux-kernel-module-cheat#gdb-builtin-cpu-simulator
'''
)
parser.add_argument(
'-X', '--no-lxsymbols', default=defaults['no_lxsymbols'], action='store_true'
)
parser.add_argument(
'--test', default=defaults['test'], action='store_true',
help='''\
Run an expect test case instead of interactive usage. For baremetal and userland, Run an expect test case instead of interactive usage. For baremetal and userland,
the script is a .py file next to the source code. the script is a .py file next to the source code.
''' '''
) )
parser.add_argument( self.add_argument(
'-u', '--userland', default=defaults['userland'], '--sim', default=False,
) help='''Use the built-in GDB CPU simulator
parser.add_argument( See: https://github.com/cirosantilli/linux-kernel-module-cheat#gdb-builtin-cpu-simulator
'break_at', nargs='?', '''
help='Extra options to append at the end of the emulator command line' )
) self.add_argument(
args = common.setup(parser) '-u', '--userland',
sys.exit(main(args)) )
def timed_main(self):
after = self.sh.shlex_split(self.env['after'])
before = self.sh.shlex_split(self.env['before'])
no_continue = self.env['no_continue']
if self.env['test']:
no_continue = True
before.extend([
'-q', LF,
'-nh', LF,
'-ex', 'set confirm off', LF
])
elif self.env['verbose']:
# The output of this would affect the tests.
# https://stackoverflow.com/questions/13496389/gdb-remote-protocol-how-to-analyse-packets
# Also be opinionated and set remotetimeout to allow you to step debug the emulator at the same time.
before.extend([
'-ex', 'set debug remote 1', LF,
'-ex', 'set remotetimeout 99999', LF,
])
if self.env['break_at'] is not None:
break_at = ['-ex', 'break {}'.format(self.env['break_at']), LF]
else:
break_at = []
linux_full_system = (self.env['baremetal'] is None and self.env['userland'] is None)
if self.env['userland']:
image = self.resolve_userland(self.env['userland'])
elif self.env['baremetal']:
image = self.env['image']
test_script_path = os.path.splitext(self.env['source_path'])[0] + '.py'
else:
image = self.env['vmlinux']
if self.env['baremetal']:
allowed_toolchains = ['crosstool-ng', 'buildroot', 'host']
else:
allowed_toolchains = ['buildroot', 'crosstool-ng', 'host']
cmd = (
[self.get_toolchain_tool('gdb', allowed_toolchains=allowed_toolchains), LF] +
before +
['-q', LF]
)
if linux_full_system:
cmd.extend(['-ex', 'add-auto-load-safe-path {}'.format(self.env['linux_build_dir']), LF])
if self.env['sim']:
target = 'sim'
else:
if self.env['kgdb']:
port = self.env['extra_serial_port']
else:
port = self.env['gdb_port']
target = 'remote localhost:{}'.format(port)
cmd.extend([
'-ex', 'file {}'.format(image), LF,
'-ex', 'target {}'.format(target), LF,
])
if not self.env['kgdb']:
cmd.extend(break_at)
if not no_continue:
# ## lx-symbols
#
# ### lx-symbols after continue
#
# lx symbols must be run after continue.
#
# running it immediately after the connect on the bootloader leads to failure,
# likely because kernel structure on which it depends are not yet available.
#
# With this setup, continue runs, and lx-symbols only runs when a break happens,
# either by hitting the breakpoint, or by entering Ctrl + C.
#
# Sure, if the user sets a break on a raw address of the bootloader,
# problems will still arise, but let's think about that some other time.
#
# ### lx-symbols autoload
#
# The lx-symbols commands gets loaded through the file vmlinux-gdb.py
# which gets put on the kernel build root when python debugging scripts are enabled.
cmd.extend(['-ex', 'continue', LF])
if not self.env['no_lxsymbols'] and linux_full_system:
cmd.extend(['-ex', 'lx-symbols {}'.format(self.env['kernel_modules_build_subdir']), LF])
cmd.extend(after)
if self.env['test']:
GdbTestcase(
self.env['source_path'],
test_script_path,
cmd,
verbose=self.env['verbose'],
)
else:
# I would rather have cwd be out_rootfs_overlay_dir,
# but then lx-symbols cannot fine the vmlinux and fails with:
# vmlinux: No such file or directory.
return self.sh.run_cmd(
cmd,
cmd_file=os.path.join(self.env['run_dir'], 'run-gdb.sh'),
cwd=self.env['linux_build_dir']
)
if __name__ == '__main__':
Main().cli()

12
run-gdb-user
View File

@@ -5,9 +5,9 @@ import os
import sys import sys
import common import common
rungdb = imp.load_source('rungdb', os.path.join(common.root_dir, 'run-gdb')) rungdb = imp.load_source('rungdb', os.path.join(kwargs['root_dir'], 'run-gdb'))
parser = common.get_argparse(argparse_args={ parser = self.get_argparse(argparse_args={
'description': '''GDB step debug guest userland processes without gdbserver. 'description': '''GDB step debug guest userland processes without gdbserver.
More information at: https://github.com/cirosantilli/linux-kernel-module-cheat#gdb-step-debug-userland-processes More information at: https://github.com/cirosantilli/linux-kernel-module-cheat#gdb-step-debug-userland-processes
@@ -23,9 +23,9 @@ parser.add_argument(
help='Break at this point, e.g. main.', help='Break at this point, e.g. main.',
nargs='?' nargs='?'
) )
args = common.setup(parser) args = self.setup(parser)
executable = common.resolve_userland(args.executable) executable = self.resolve_userland(kwargs['executable'])
addr = common.get_elf_entry(os.path.join(common.buildroot_build_build_dir, executable)) addr = self.get_elf_entry(os.path.join(kwargs['buildroot_build_build_dir'], executable))
extra_args = {} extra_args = {}
extra_args['before'] = '-ex \"add-symbol-file {} {}\"'.format(executable, hex(addr)) extra_args['before'] = '-ex \"add-symbol-file {} {}\"'.format(executable, hex(addr))
# Or else lx-symbols throws for arm: # Or else lx-symbols throws for arm:
@@ -33,5 +33,5 @@ extra_args['before'] = '-ex \"add-symbol-file {} {}\"'.format(executable, hex(ad
# TODO understand better. # TODO understand better.
# Also, lx-symbols overrides the add-symbol-file commands. # Also, lx-symbols overrides the add-symbol-file commands.
extra_args['no_lxsymbols'] = True extra_args['no_lxsymbols'] = True
extra_args['break_at'] = args.break_at extra_args['break_at'] = kwargs['break_at']
sys.exit(rungdb.main(args, extra_args)) sys.exit(rungdb.main(args, extra_args))

15
run-gdbserver
View File

@@ -5,8 +5,9 @@ import subprocess
import sys import sys
import common import common
from shell_helpers import LF
parser = common.get_argparse(argparse_args={ parser = self.get_argparse(argparse_args={
'description':'Connect to gdbserver running on the guest.' 'description':'Connect to gdbserver running on the guest.'
}) })
parser.add_argument( parser.add_argument(
@@ -16,13 +17,13 @@ parser.add_argument(
parser.add_argument( parser.add_argument(
'break_at', default='main', nargs='?' 'break_at', default='main', nargs='?'
) )
args = common.setup(parser) args = self.setup(parser)
sys.exit(subprocess.Popen([ sys.exit(subprocess.Popen([
common.get_toolchain_tool('gdb'), self.get_toolchain_tool('gdb'),
'-q', '-q',
'-ex', 'set sysroot {}'.format(common.buildroot_staging_dir), '-ex', 'set sysroot {}'.format(kwargs['buildroot_staging_dir']),
'-ex', 'target remote localhost:{}'.format(common.qemu_hostfwd_generic_port), '-ex', 'target remote localhost:{}'.format(kwargs['qemu_hostfwd_generic_port']),
'-ex', 'tbreak {}'.format(args.break_at), '-ex', 'tbreak {}'.format(kwargs['break_at']),
'-ex', 'continue', '-ex', 'continue',
os.path.join(common.buildroot_build_build_dir, common.resolve_userland(args.executable)), os.path.join(kwargs['buildroot_build_build_dir'], self.resolve_userland(kwargs['executable'])),
]).wait()) ]).wait())

24
run-toolchain
View File

@@ -4,8 +4,9 @@ import os
import sys import sys
import common import common
from shell_helpers import LF
parser = common.get_argparse(argparse_args={ parser = self.get_argparse(argparse_args={
'description': '''Run a Buildroot ToolChain tool like readelf or objdump. 'description': '''Run a Buildroot ToolChain tool like readelf or objdump.
For example, to get some information about the arm vmlinux: For example, to get some information about the arm vmlinux:
@@ -24,7 +25,6 @@ ls "$(./getvar -a arm host_bin_dir)"
parser.add_argument( parser.add_argument(
'--dry', '--dry',
help='Just output the tool path to stdout but actually run it', help='Just output the tool path to stdout but actually run it',
action='store_true',
) )
parser.add_argument('tool', help='Which tool to run.') parser.add_argument('tool', help='Which tool to run.')
parser.add_argument( parser.add_argument(
@@ -34,17 +34,17 @@ parser.add_argument(
metavar='extra-args', metavar='extra-args',
nargs='*' nargs='*'
) )
args = common.setup(parser) args = self.setup(parser)
if common.baremetal is None: if kwargs['baremetal'] is None:
image = common.vmlinux image = kwargs['vmlinux']
else: else:
image = common.image image = kwargs['image']
tool= common.get_toolchain_tool(args.tool) tool= self.get_toolchain_tool(kwargs['tool'])
if args.dry: if kwargs['dry']:
print(tool) print(tool)
else: else:
sys.exit(common.run_cmd( sys.exit(self.sh.run_cmd(
[tool, common.Newline] [tool, LF]
+ common.add_newlines(args.extra_args), + self.sh.add_newlines(kwargs['extra_args']),
cmd_file=os.path.join(common.run_dir, 'run-toolchain.sh'), cmd_file=os.path.join(kwargs['run_dir'], 'run-toolchain.sh'),
)) ))

262
shell_helpers.py Normal file
View File

@@ -0,0 +1,262 @@
#!/usr/bin/env python3
import distutils.file_util
import itertools
import os
import shlex
import shutil
import signal
import stat
import subprocess
import sys
class LF:
'''
LineFeed (AKA newline).
Singleton class. Can be used in print_cmd to print out nicer command lines
with --key on the same line as "--key value".
'''
pass
class ShellHelpers:
'''
Helpers to do things which are easy from the shell,
usually filesystem, process or pipe operations.
Attempt to print shell equivalents of all commands to make things
easy to debug and understand what is going on.
'''
def __init__(self, dry_run=False):
'''
:param dry_run: don't run the commands, just potentially print them. Debug aid.
:type dry_run: Bool
'''
self.dry_run = dry_run
def add_newlines(self, cmd):
out = []
for arg in cmd:
out.extend([arg, LF])
return out
def cp(self, src, dest, **kwargs):
self.print_cmd(['cp', src, dest])
if not self.dry_run:
shutil.copy2(src, dest)
def cmd_to_string(self, cmd, cwd=None, extra_env=None, extra_paths=None):
'''
Format a command given as a list of strings so that it can
be viewed nicely and executed by bash directly and print it to stdout.
'''
last_newline = ' \\\n'
newline_separator = last_newline + ' '
out = []
if extra_env is None:
extra_env = {}
if cwd is not None:
out.append('cd {} &&'.format(shlex.quote(cwd)))
if extra_paths is not None:
out.append('PATH="{}:${{PATH}}"'.format(':'.join(extra_paths)))
for key in extra_env:
out.append('{}={}'.format(shlex.quote(key), shlex.quote(extra_env[key])))
cmd_quote = []
newline_count = 0
for arg in cmd:
if arg == LF:
cmd_quote.append(arg)
newline_count += 1
else:
cmd_quote.append(shlex.quote(arg))
if newline_count > 0:
cmd_quote = [' '.join(list(y)) for x, y in itertools.groupby(cmd_quote, lambda z: z == LF) if not x]
out.extend(cmd_quote)
if newline_count == 1 and cmd[-1] == LF:
ending = ''
else:
ending = last_newline + ';'
return newline_separator.join(out) + ending
def copy_dir_if_update_non_recursive(self, srcdir, destdir, filter_ext=None):
# TODO print rsync equivalent.
os.makedirs(destdir, exist_ok=True)
for basename in os.listdir(srcdir):
src = os.path.join(srcdir, basename)
if os.path.isfile(src):
noext, ext = os.path.splitext(basename)
if filter_ext is not None and ext == filter_ext:
distutils.file_util.copy_file(
src,
os.path.join(destdir, basename),
update=1,
)
def print_cmd(self, cmd, cwd=None, cmd_file=None, extra_env=None, extra_paths=None):
'''
Print cmd_to_string to stdout.
Optionally save the command to cmd_file file, and add extra_env
environment variables to the command generated.
If cmd contains at least one LF, newlines are only added on LF.
Otherwise, newlines are added automatically after every word.
'''
if type(cmd) is str:
cmd_string = cmd
else:
cmd_string = self.cmd_to_string(cmd, cwd=cwd, extra_env=extra_env, extra_paths=extra_paths)
print('+ ' + cmd_string)
if cmd_file is not None:
with open(cmd_file, 'w') as f:
f.write('#!/usr/bin/env bash\n')
f.write(cmd_string)
st = os.stat(cmd_file)
os.chmod(cmd_file, st.st_mode | stat.S_IXUSR)
def run_cmd(
self,
cmd,
cmd_file=None,
out_file=None,
show_stdout=True,
show_cmd=True,
extra_env=None,
extra_paths=None,
delete_env=None,
raise_on_failure=True,
**kwargs
):
'''
Run a command. Write the command to stdout before running it.
Wait until the command finishes execution.
:param cmd: command to run. LF entries are magic get skipped.
:type cmd: List[str]
:param cmd_file: if not None, write the command to be run to that file
:type cmd_file: str
:param out_file: if not None, write the stdout and stderr of the command the file
:type out_file: str
:param show_stdout: wether to show stdout and stderr on the terminal or not
:type show_stdout: bool
:param extra_env: extra environment variables to add when running the command
:type extra_env: Dict[str,str]
'''
if out_file is not None:
stdout = subprocess.PIPE
stderr = subprocess.STDOUT
else:
if show_stdout:
stdout = None
stderr = None
else:
stdout = subprocess.DEVNULL
stderr = subprocess.DEVNULL
if extra_env is None:
extra_env = {}
if delete_env is None:
delete_env = []
if 'cwd' in kwargs:
cwd = kwargs['cwd']
else:
cwd = None
env = os.environ.copy()
env.update(extra_env)
if extra_paths is not None:
path = ':'.join(extra_paths)
if 'PATH' in os.environ:
path += ':' + os.environ['PATH']
env['PATH'] = path
for key in delete_env:
if key in env:
del env[key]
if show_cmd:
self.print_cmd(cmd, cwd=cwd, cmd_file=cmd_file, extra_env=extra_env, extra_paths=extra_paths)
# Otherwise Ctrl + C gives:
# - ugly Python stack trace for gem5 (QEMU takes over terminal and is fine).
# - kills Python, and that then kills GDB: https://stackoverflow.com/questions/19807134/does-python-always-raise-an-exception-if-you-do-ctrlc-when-a-subprocess-is-exec
sigint_old = signal.getsignal(signal.SIGINT)
signal.signal(signal.SIGINT, signal.SIG_IGN)
# Otherwise BrokenPipeError when piping through | grep
# But if I do this_module, my terminal gets broken at the end. Why, why, why.
# https://stackoverflow.com/questions/14207708/ioerror-errno-32-broken-pipe-python
# Ignoring the exception is not enough as it prints a warning anyways.
#sigpipe_old = signal.getsignal(signal.SIGPIPE)
#signal.signal(signal.SIGPIPE, signal.SIG_DFL)
cmd = self.strip_newlines(cmd)
if not self.dry_run:
# https://stackoverflow.com/questions/15535240/python-popen-write-to-stdout-and-log-file-simultaneously/52090802#52090802
with subprocess.Popen(cmd, stdout=stdout, stderr=stderr, env=env, **kwargs) as proc:
if out_file is not None:
os.makedirs(os.path.split(os.path.abspath(out_file))[0], exist_ok=True)
with open(out_file, 'bw') as logfile:
while True:
byte = proc.stdout.read(1)
if byte:
if show_stdout:
sys.stdout.buffer.write(byte)
try:
sys.stdout.flush()
except BlockingIOError:
# TODO understand. Why, Python, why.
pass
logfile.write(byte)
else:
break
signal.signal(signal.SIGINT, sigint_old)
#signal.signal(signal.SIGPIPE, sigpipe_old)
returncode = proc.returncode
if returncode != 0 and raise_on_failure:
raise Exception('Command exited with status: {}'.format(returncode))
return returncode
else:
return 0
def shlex_split(self, string):
'''
shlex_split, but also add Newline after every word.
Not perfect since it does not group arguments, but I don't see a solution.
'''
return self.add_newlines(shlex.split(string))
def strip_newlines(self, cmd):
return [x for x in cmd if x != LF]
def rmrf(self, path):
self.print_cmd(['rm', '-r', '-f', path, LF])
if not self.dry_run and os.path.exists(path):
shutil.rmtree(path)
def write_configs(self, config_path, configs, config_fragments=None):
'''
Write extra KEY=val configs into the given config file.
'''
if config_fragments is None:
config_fragments = []
with open(config_path, 'a') as config_file:
for config_fragment in config_fragments:
with open(config_fragment, 'r') as config_fragment_file:
self.print_cmd(['cat', config_fragment, '>>', config_path])
if not self.dry_run:
for line in config_fragment_file:
config_file.write(line)
self.write_string_to_file(config_path, '\n'.join(configs), mode='a')
def write_string_to_file(self, path, string, mode='w'):
if mode == 'a':
redirect = '>>'
else:
redirect = '>'
self.print_cmd("cat << 'EOF' {} {}\n{}\nEOF".format(redirect, path, string))
if not self.dry_run:
with open(path, 'a') as f:
f.write(string)

16
trace-boot
View File

@@ -6,10 +6,10 @@ import subprocess
import re import re
import common import common
run = imp.load_source('run', os.path.join(common.root_dir, 'run')) run = imp.load_source('run', os.path.join(kwargs['root_dir'], 'run'))
qemu_trace2txt = imp.load_source('qemu_trace2txt', os.path.join(common.root_dir, 'qemu-trace2txt')) qemu_trace2txt = imp.load_source('qemu_trace2txt', os.path.join(kwargs['root_dir'], 'qemu-trace2txt'))
parser = common.get_argparse(argparse_args={ parser = self.get_argparse(argparse_args={
'description': '''Trace the PIC addresses executed on a Linux kernel boot. 'description': '''Trace the PIC addresses executed on a Linux kernel boot.
More information at: https://github.com/cirosantilli/linux-kernel-module-cheat#tracing More information at: https://github.com/cirosantilli/linux-kernel-module-cheat#tracing
@@ -19,11 +19,11 @@ parser.add_argument(
'extra_emulator_args', nargs='*', 'extra_emulator_args', nargs='*',
help='Extra options to append at the end of the emulator command line' help='Extra options to append at the end of the emulator command line'
) )
args = common.setup(parser) args = self.setup(parser)
extra_args = { extra_args = {
'extra_emulator_args': args.extra_emulator_args, 'extra_emulator_args': kwargs['extra_emulator_args'],
} }
if common.emulator == 'gem5': if kwargs['emulator'] == 'gem5':
extra_args.update({ extra_args.update({
'eval': 'm5 exit', 'eval': 'm5 exit',
'trace': 'Exec,-ExecSymbol,-ExecMicro', 'trace': 'Exec,-ExecSymbol,-ExecMicro',
@@ -39,10 +39,10 @@ else:
# Instruction count. # Instruction count.
# We could put this on a separate script, but it just adds more arch boilerplate to a new script. # We could put this on a separate script, but it just adds more arch boilerplate to a new script.
# So let's just leave it here for now since it did not add a significant processing time. # So let's just leave it here for now since it did not add a significant processing time.
kernel_entry_addr = hex(common.get_elf_entry(common.vmlinux)) kernel_entry_addr = hex(self.get_elf_entry(kwargs['vmlinux']))
nlines = 0 nlines = 0
nlines_firmware = 0 nlines_firmware = 0
with open(common.qemu_trace_txt_file, 'r') as trace_file: with open(kwargs['qemu_trace_txt_file'], 'r') as trace_file:
in_firmware = True in_firmware = True
for line in trace_file: for line in trace_file:
line = line.rstrip() line = line.rstrip()

29
trace2line
View File

@@ -13,23 +13,24 @@ import subprocess
import sys import sys
import common import common
from shell_helpers import LF
parser = common.get_argparse(argparse_args={ parser = self.get_argparse(argparse_args={
'description': 'Convert an execution trace containing PC values into the Linux kernel linex executed' 'description': 'Convert an execution trace containing PC values into the Linux kernel linex executed'
}) })
args = common.setup(parser) args = self.setup(parser)
sys.exit(subprocess.Popen([ sys.exit(subprocess.Popen([
os.path.join(common.root_dir, 'trace2line.sh'), os.path.join(kwargs['root_dir'], 'trace2line.sh'),
'true' if common.emulator == 'gem5' else 'false', 'true' if kwargs['emulator'] == 'gem5' else 'false',
common.trace_txt_file, kwargs['trace_txt_file'],
common.get_toolchain_tool('addr2line'), self.get_toolchain_tool('addr2line'),
common.vmlinux, kwargs['vmlinux'],
common.run_dir, kwargs['run_dir'],
]).wait()) ]).wait())
# This was the full conversion attempt. # This was the full conversion attempt.
# if common.emulator == 'gem5': # if kwargs['emulator'] == 'gem5':
# def get_pc(line): # def get_pc(line):
# # TODO # # TODO
# # stdin = sed -r 's/^.* (0x[^. ]*)[. ].*/\1/' "$common_trace_txt_file") # # stdin = sed -r 's/^.* (0x[^. ]*)[. ].*/\1/' "$common_trace_txt_file")
@@ -40,17 +41,17 @@ sys.exit(subprocess.Popen([
# with \ # with \
# subprocess.Popen( # subprocess.Popen(
# [ # [
# common.get_toolchain_tool('addr2line'), # self.get_toolchain_tool('addr2line'),
# '-e', # '-e',
# common.vmlinux, # kwargs['vmlinux'],
# '-f', # '-f',
# '-p', # '-p',
# ], # ],
# stdout=subprocess.PIPE, # stdout=subprocess.PIPE,
# stdin=subprocess.PIPE, # stdin=subprocess.PIPE,
# ) as proc, \ # ) as proc, \
# open(common.trace_txt_file, 'r') as infile, \ # open(kwargs['trace_txt_file'], 'r') as infile, \
# open(os.path.join(common.run_dir, 'trace-lines.txt'), 'w') as outfile \ # open(os.path.join(kwargs['run_dir'], 'trace-lines.txt'), 'w') as outfile \
# : # :
# for in_line in infile: # for in_line in infile:
# proc.stdin.write(get_pc(in_line).encode()) # proc.stdin.write(get_pc(in_line).encode())
@@ -58,5 +59,5 @@ sys.exit(subprocess.Popen([
# stdout = proc.stdout.read() # stdout = proc.stdout.read()
# outfile.write(stdout.decode()) # outfile.write(stdout.decode())
# # TODO # # TODO
# # sed -E "s|at ${common.linux_build_dir}/(\./\|)||" # # sed -E "s|at ${kwargs['linux_build_dir']}/(\./\|)||"
# # uniq -c # # uniq -c

8
userland/Makefile
View File

@@ -46,7 +46,9 @@ OUTS := $(addprefix $(OUT_DIR)/,$(OUTS))
all: mkdir $(OUTS) all: mkdir $(OUTS)
for subdir in $(SUBDIRS); do \ for subdir in $(SUBDIRS); do \
$(MAKE) -C $${subdir} OUT_DIR="$(OUT_DIR)/$$subdir"; \ if [ -d "$${subdir}" ]; then \
$(MAKE) -C "$${subdir}" OUT_DIR="$(OUT_DIR)/$$subdir"; \
fi \
done done
$(COMMON_OBJ): $(COMMON_DIR)/$(COMMON_BASENAME)$(IN_EXT_C) $(COMMON_OBJ): $(COMMON_DIR)/$(COMMON_BASENAME)$(IN_EXT_C)
@@ -64,7 +66,9 @@ $(OUT_DIR)/%$(OUT_EXT): %$(IN_EXT_CXX) $(COMMON_OBJ)
clean: clean:
rm -f *'$(OBJ_EXT)' *'$(OUT_EXT)' rm -f *'$(OBJ_EXT)' *'$(OUT_EXT)'
for subdir in $(SUBDIRS); do \ for subdir in $(SUBDIRS); do \
$(MAKE) -C $${subdir} clean; \ if [ -d "$${subdir}" ]; then \
$(MAKE) -C $${subdir} clean; \
fi \
done done
mkdir: mkdir: