3rd-party/linux-kernel-module-cheat
1
0
Fork 0
mirror of https://github.com/cirosantilli/linux-kernel-module-cheat.git synced 2026-01-28 04:24:26 +01:00
Code Issues Releases Wiki Activity

run kind of runs

Browse Source
This commit is contained in:
Ciro Santilli 六四事件 法轮功
2018-12-09 00:00:01 +00:00
parent 5e20ba833b
commit fa1e4ffa7d
34 changed files with 848 additions and 838 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
README.adoc
View File

@@ -960,7 +960,7 @@ This automatically clears the GDB pane, and starts a new one.
Pass extra GDB arguments with: Pass extra GDB arguments with:
.... ....
./run --wait-gdb --tmux=start_kernel ./run --wait-gdb --tmux --tmux-args start_kernel
.... ....
See the tmux manual for further details: See the tmux manual for further details:
@@ -2986,7 +2986,8 @@ Or alternatively, if you are using <<tmux>>, do everything in one go with:
./run \ ./run \
--arch aarch64 \ --arch aarch64 \
--userland print_argv \ --userland print_argv \
--tmux=main \ --tmux \
--tmux-args main \
--wait-gdb \ --wait-gdb \
-- \ -- \
asdf qwer \ asdf qwer \
@@ -5056,7 +5057,7 @@ If `CONFIG_KALLSYMS=n`, then addresses are shown on traces instead of symbol plu
In v4.16 it does not seem possible to configure that at runtime. GDB step debugging with: In v4.16 it does not seem possible to configure that at runtime. GDB step debugging with:
.... ....
./run --eval-after 'insmod /dump_stack.ko' --wait-gdb --tmux=dump_stack ./run --eval-after 'insmod /dump_stack.ko' --wait-gdb --tmux --tmux-args dump_stack
.... ....
shows that traces are printed at `arch/x86/kernel/dumpstack.c`: shows that traces are printed at `arch/x86/kernel/dumpstack.c`:
@@ -8168,7 +8169,7 @@ And in QEMU:
Or for a faster development loop: Or for a faster development loop:
.... ....
./run --debug-vm='-ex "break edu_mmio_read" -ex "run"' ./run --debug-vm --debug-vm-args '-ex "break edu_mmio_read" -ex "run"'
.... ....
When in <<qemu-text-mode>>, using `--debug-vm` makes Ctrl-C not get passed to the QEMU guest anymore: it is instead captured by GDB itself, so allow breaking. So e.g. you won't be able to easily quit from a guest program like: When in <<qemu-text-mode>>, using `--debug-vm` makes Ctrl-C not get passed to the QEMU guest anymore: it is instead captured by GDB itself, so allow breaking. So e.g. you won't be able to easily quit from a guest program like:
@@ -10259,13 +10260,13 @@ then on the second shell:
Or if you are a <<tmux,tmux pro>>, do everything in one go with: Or if you are a <<tmux,tmux pro>>, do everything in one go with:
.... ....
./run --arch arm --baremetal interactive/prompt --wait-gdb --tmux=main ./run --arch arm --baremetal interactive/prompt --wait-gdb --tmux --tmux-args main
.... ....
Alternatively, to start from the very first executed instruction of our tiny <<baremetal-bootloaders>>: Alternatively, to start from the very first executed instruction of our tiny <<baremetal-bootloaders>>:
.... ....
./run --arch arm --baremetal interactive/prompt --wait-gdb --tmux=--no-continue ./run --arch arm --baremetal interactive/prompt --wait-gdb --tmux --tmux-args --no-continue
.... ....
Now you can just `stepi` to when jumping into main to go to the C code in link:baremetal/interactive/prompt.c[]. Now you can just `stepi` to when jumping into main to go to the C code in link:baremetal/interactive/prompt.c[].
@@ -10273,7 +10274,7 @@ Now you can just `stepi` to when jumping into main to go to the C code in link:b
This is specially interesting for the executables that don't use the bootloader from under `baremetal/arch/<arch>/no_bootloader/*.S`, e.g.: This is specially interesting for the executables that don't use the bootloader from under `baremetal/arch/<arch>/no_bootloader/*.S`, e.g.:
.... ....
./run --arch arm --baremetal arch/arm/no_bootloader/semihost_exit --wait-gdb --tmux=--no-continue ./run --arch arm --baremetal arch/arm/no_bootloader/semihost_exit --wait-gdb --tmux --tmux-args --no-continue
.... ....
The cool thing about those examples is that you start at the very first instruction of your program, which gives more control. The cool thing about those examples is that you start at the very first instruction of your program, which gives more control.

6
bisect-linux-boot-gem5
View File

@@ -9,7 +9,7 @@ import common
build_linux = imp.load_source('build-linux', os.path.join(kwargs['root_dir'], 'build_linux')) build_linux = imp.load_source('build-linux', os.path.join(kwargs['root_dir'], 'build_linux'))
run = imp.load_source('run', os.path.join(kwargs['root_dir'], 'run')) run = imp.load_source('run', os.path.join(kwargs['root_dir'], 'run'))
parser = common.get_argparse( parser = self.get_argparse(
argparse_args={ argparse_args={
'description': '''Bisect the Linux kernel on gem5 boots. 'description': '''Bisect the Linux kernel on gem5 boots.
@@ -20,11 +20,11 @@ More information at: https://github.com/cirosantilli/linux-kernel-module-cheat#b
'linux_build_id': 'bisect', 'linux_build_id': 'bisect',
}, },
) )
args = common.setup(parser) args = self.setup(parser)
# We need a clean rebuild because rebuilds at different revisions: # We need a clean rebuild because rebuilds at different revisions:
# - may fail # - may fail
# - may not actually rebuild all files, e.g. on header changes # - may not actually rebuild all files, e.g. on header changes
common.rmrf(kwargs['linux_build_dir']) self.rmrf(kwargs['linux_build_dir'])
build_linux.LinuxComponent().do_build(args) build_linux.LinuxComponent().do_build(args)
status = run.main(args, { status = run.main(args, {
'eval': 'm5 exit', 'eval': 'm5 exit',

6
bst-vs-heap
View File

@@ -1,10 +1,10 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
import common import common
parser = common.get_argparse( parser = self.get_argparse(
argparse_args={'description':'Convert a BST vs heap stat file into a gnuplot input'} argparse_args={'description':'Convert a BST vs heap stat file into a gnuplot input'}
) )
args = common.setup(parser) args = self.setup(parser)
stats = common.get_stats() stats = self.get_stats()
it = iter(stats) it = iter(stats)
i = 1 i = 1
for stat in it: for stat in it:

4
build
View File

@@ -315,9 +315,9 @@ Extra args to pass to all scripts.
parser.add_argument('components', choices=list(name_to_component_map.keys()) + [[]], default=[], nargs='*', help='''\ parser.add_argument('components', choices=list(name_to_component_map.keys()) + [[]], default=[], nargs='*', help='''\
Which components to build. Default: qemu-buildroot Which components to build. Default: qemu-buildroot
'''.format(kwargs['default_arch'])) '''.format(kwargs['default_arch']))
common.add_dry_run_argument(parser) self.add_dry_run_argument(parser)
args = parser.parse_args() args = parser.parse_args()
common.setup_dry_run_arguments(args) self.setup_dry_run_arguments(args)
# Decide archs. # Decide archs.
if kwargs['arch'] == []: if kwargs['arch'] == []:

14
build-baremetal
View File

@@ -4,9 +4,9 @@ import os
import common import common
class BaremetalComponent(common.Component): class BaremetalComponent(self.Component):
def do_build(self, args): def do_build(self, args):
common.assert_crosstool_ng_supports_arch(kwargs['arch']) self.assert_crosstool_ng_supports_arch(kwargs['arch'])
build_dir = self.get_build_dir(args) build_dir = self.get_build_dir(args)
bootloader_obj = os.path.join(kwargs['baremetal_build_lib_dir'], 'bootloader{}'.format(kwargs['obj_ext'])) bootloader_obj = os.path.join(kwargs['baremetal_build_lib_dir'], 'bootloader{}'.format(kwargs['obj_ext']))
common_basename_noext = 'common' common_basename_noext = 'common'
@@ -28,7 +28,7 @@ class BaremetalComponent(common.Component):
gcc = 'arm-none-eabi-gcc' gcc = 'arm-none-eabi-gcc'
else: else:
os.environ['PATH'] = kwargs['crosstool_ng_bin_dir'] + os.environ['PATH'] os.environ['PATH'] = kwargs['crosstool_ng_bin_dir'] + os.environ['PATH']
gcc = common.get_toolchain_tool('gcc', allowed_toolchains=['crosstool-ng']) gcc = self.get_toolchain_tool('gcc', allowed_toolchains=['crosstool-ng'])
if kwargs['emulator'] == 'gem5': if kwargs['emulator'] == 'gem5':
if kwargs['machine'] == 'VExpress_GEM5_V1': if kwargs['machine'] == 'VExpress_GEM5_V1':
entry_address = 0x80000000 entry_address = 0x80000000
@@ -45,7 +45,7 @@ class BaremetalComponent(common.Component):
os.makedirs(build_dir, exist_ok=True) os.makedirs(build_dir, exist_ok=True)
os.makedirs(kwargs['baremetal_build_lib_dir'], exist_ok=True) os.makedirs(kwargs['baremetal_build_lib_dir'], exist_ok=True)
src = os.path.join(kwargs['baremetal_src_lib_dir'], '{}{}'.format(kwargs['arch'], kwargs['asm_ext'])) src = os.path.join(kwargs['baremetal_src_lib_dir'], '{}{}'.format(kwargs['arch'], kwargs['asm_ext']))
if common.need_rebuild([src], bootloader_obj): if self.need_rebuild([src], bootloader_obj):
self.sh.run_cmd( self.sh.run_cmd(
[gcc, LF] + [gcc, LF] +
cflags + cflags +
@@ -59,7 +59,7 @@ class BaremetalComponent(common.Component):
(common_src, common_obj), (common_src, common_obj),
(syscalls_src, syscalls_obj), (syscalls_src, syscalls_obj),
]: ]:
if common.need_rebuild([src], obj): if self.need_rebuild([src], obj):
self.sh.run_cmd( self.sh.run_cmd(
[gcc, LF] + [gcc, LF] +
cflags + cflags +
@@ -149,7 +149,7 @@ Build the baremetal examples with crosstool-NG.
in_name = os.path.splitext(in_basename)[0] in_name = os.path.splitext(in_basename)[0]
main_obj = os.path.join(kwargs['baremetal_build_dir'], subpath, '{}{}'.format(in_name, kwargs['obj_ext'])) main_obj = os.path.join(kwargs['baremetal_build_dir'], subpath, '{}{}'.format(in_name, kwargs['obj_ext']))
src = os.path.join(kwargs['baremetal_src_dir'], in_path) src = os.path.join(kwargs['baremetal_src_dir'], in_path)
if common.need_rebuild([src], main_obj): if self.need_rebuild([src], main_obj):
self.sh.run_cmd( self.sh.run_cmd(
[gcc, LF] + [gcc, LF] +
cflags + cflags +
@@ -162,7 +162,7 @@ Build the baremetal examples with crosstool-NG.
objs = common_objs + [main_obj] objs = common_objs + [main_obj]
out = os.path.join(kwargs['baremetal_build_dir'], subpath, in_name + kwargs['baremetal_build_ext']) out = os.path.join(kwargs['baremetal_build_dir'], subpath, in_name + kwargs['baremetal_build_ext'])
link_script = os.path.join(kwargs['baremetal_src_dir'], 'link.ld') link_script = os.path.join(kwargs['baremetal_src_dir'], 'link.ld')
if common.need_rebuild(objs + [link_script], out): if self.need_rebuild(objs + [link_script], out):
self.sh.run_cmd( self.sh.run_cmd(
[gcc, LF] + [gcc, LF] +
cflags + cflags +

14
build-buildroot
View File

@@ -10,10 +10,10 @@ import re
import common import common
class BuildrootComponent(common.Component): class BuildrootComponent(self.Component):
def add_parser_arguments(self, parser): def add_parser_arguments(self, parser):
parser.add_argument( parser.add_argument(
'--build-linux', default=self._defaults['build_linux'], action='store_true', '--build-linux', default=self._defaults['build_linux'],
help='''\ help='''\
Enable building the Linux kernel with Buildroot. This is done mostly Enable building the Linux kernel with Buildroot. This is done mostly
to extract Buildroot's default kernel configurations when updating Buildroot. to extract Buildroot's default kernel configurations when updating Buildroot.
@@ -22,7 +22,7 @@ not currently supported, juse use ./build-linux script if you want to do that.
''' '''
) )
parser.add_argument( parser.add_argument(
'--baseline', default=self._defaults['baseline'], action='store_true', '--baseline', default=self._defaults['baseline'],
help='''Do a default-ish Buildroot defconfig build, without any of our extra options. help='''Do a default-ish Buildroot defconfig build, without any of our extra options.
Mostly to track how much slower we are than a basic build. Mostly to track how much slower we are than a basic build.
''' '''
@@ -42,14 +42,14 @@ Pass multiple times to use multiple fragment files.
''' '''
) )
parser.add_argument( parser.add_argument(
'--no-all', default=self._defaults['no_all'], action='store_true', '--no-all', default=self._defaults['no_all'],
help='''\ help='''\
Don't build the all target which normally gets build by default. Don't build the all target which normally gets build by default.
That target builds the root filesystem and all its dependencies. That target builds the root filesystem and all its dependencies.
''' '''
) )
parser.add_argument( parser.add_argument(
'--no-overlay', default=self._defaults['no_all'], action='store_true', '--no-overlay', default=self._defaults['no_all'],
help='''\ help='''\
Don't add our overlay which contains all files we build without going through Buildroot. Don't add our overlay which contains all files we build without going through Buildroot.
This prevents us from overwriting certain Buildroot files. Remember however that you must This prevents us from overwriting certain Buildroot files. Remember however that you must
@@ -140,7 +140,7 @@ usually extra Buildroot targets.
], ],
cwd=kwargs['buildroot_src_dir'], cwd=kwargs['buildroot_src_dir'],
) )
common.make_build_dirs() self.make_build_dirs()
if not kwargs['no_all']: if not kwargs['no_all']:
extra_make_args.extend(['all', LF]) extra_make_args.extend(['all', LF])
self.sh.run_cmd( self.sh.run_cmd(
@@ -161,7 +161,7 @@ usually extra Buildroot targets.
# Skip if qemu is not present, because gem5 does not need the qcow2. # Skip if qemu is not present, because gem5 does not need the qcow2.
# so we don't force a QEMU build for gem5. # so we don't force a QEMU build for gem5.
if not kwargs['no_all'] and os.path.exists(kwargs['qemu_img_executable']): if not kwargs['no_all'] and os.path.exists(kwargs['qemu_img_executable']):
common.raw_to_qcow2() self.raw_to_qcow2()
def get_argparse_args(self): def get_argparse_args(self):
return { return {

6
build-crosstool-ng
View File

@@ -4,9 +4,9 @@ import os
import common import common
class CrosstoolNgComponent(common.Component): class CrosstoolNgComponent(self.Component):
def do_build(self, args): def do_build(self, args):
common.assert_crosstool_ng_supports_arch(kwargs['arch']) self.assert_crosstool_ng_supports_arch(kwargs['arch'])
build_dir = self.get_build_dir(args) build_dir = self.get_build_dir(args)
defconfig_dest = os.path.join(kwargs['crosstool_ng_util_dir'], 'defconfig') defconfig_dest = os.path.join(kwargs['crosstool_ng_util_dir'], 'defconfig')
os.makedirs(kwargs['crosstool_ng_util_dir'], exist_ok=True) os.makedirs(kwargs['crosstool_ng_util_dir'], exist_ok=True)
@@ -37,7 +37,7 @@ class CrosstoolNgComponent(common.Component):
os.path.join(kwargs['root_dir'], 'crosstool_ng_config', kwargs['arch']), os.path.join(kwargs['root_dir'], 'crosstool_ng_config', kwargs['arch']),
defconfig_dest defconfig_dest
) )
common.write_configs( self.write_configs(
kwargs['crosstool_ng_defconfig'], kwargs['crosstool_ng_defconfig'],
[ [
'CT_PREFIX_DIR="{}"'.format(kwargs['crosstool_ng_install_dir']), 'CT_PREFIX_DIR="{}"'.format(kwargs['crosstool_ng_install_dir']),

4
build-docker
View File

@@ -7,7 +7,7 @@ import tarfile
import common import common
class DockerComponent(common.Component): class DockerComponent(self.Component):
def get_argparse_args(self): def get_argparse_args(self):
return { return {
'description': '''\ 'description': '''\
@@ -68,7 +68,7 @@ See also:https://github.com/cirosantilli/linux-kernel-module-cheat#ubuntu-guest-
kwargs['docker_tar_dir'], kwargs['docker_tar_dir'],
kwargs['docker_rootfs_raw_file'], kwargs['docker_rootfs_raw_file'],
]) ])
common.raw_to_qcow2(prebuilt=True) self.raw_to_qcow2(prebuilt=True)
def get_build_dir(self, args): def get_build_dir(self, args):
return kwargs['docker_build_dir'] return kwargs['docker_build_dir']

6
build-linux
View File

@@ -37,7 +37,7 @@ Still uses options explicitly passed with `--config` and
''' '''
) )
self.add_argument( self.add_argument(
'--config-only', default=False, action='store_true', '--config-only', default=False,
help='''\ help='''\
Configure the kernel, but don't build it. Configure the kernel, but don't build it.
''' '''
@@ -91,7 +91,7 @@ Configure the kernel, but don't build it.
if self.env['config'] != []: if self.env['config'] != []:
cli_config_fragment_path = os.path.join(build_dir, 'lkmc_cli_config_fragment') cli_config_fragment_path = os.path.join(build_dir, 'lkmc_cli_config_fragment')
cli_config_str = '\n'.join(self.env['config']) cli_config_str = '\n'.join(self.env['config'])
common.write_string_to_file(cli_config_fragment_path, cli_config_str) self.write_string_to_file(cli_config_fragment_path, cli_config_str)
config_fragments.append(cli_config_fragment_path) config_fragments.append(cli_config_fragment_path)
self.sh.cp( self.sh.cp(
base_config_file, base_config_file,
@@ -133,7 +133,7 @@ Configure the kernel, but don't build it.
) )
# TODO: remove build and source https://stackoverflow.com/questions/13578618/what-does-build-and-source-link-do-in-lib-modules-kernel-version # TODO: remove build and source https://stackoverflow.com/questions/13578618/what-does-build-and-source-link-do-in-lib-modules-kernel-version
# TODO Basically all kernel modules also basically leak full host paths. Just terrible. Buildroot deals with that stuff nicely for us. # TODO Basically all kernel modules also basically leak full host paths. Just terrible. Buildroot deals with that stuff nicely for us.
# common.rmrf() # self.rmrf()
def get_build_dir(self): def get_build_dir(self):
return self.env['linux_build_dir'] return self.env['linux_build_dir']

6
build-m5
View File

@@ -4,11 +4,11 @@ import os
import common import common
class M5Component(common.Component): class M5Component(self.Component):
def get_make_cmd(self, args): def get_make_cmd(self, args):
allowed_toolchains = ['buildroot'] allowed_toolchains = ['buildroot']
cc = common.get_toolchain_tool('gcc', allowed_toolchains=allowed_toolchains) cc = self.get_toolchain_tool('gcc', allowed_toolchains=allowed_toolchains)
ld = common.get_toolchain_tool('ld', allowed_toolchains=allowed_toolchains) ld = self.get_toolchain_tool('ld', allowed_toolchains=allowed_toolchains)
if kwargs['arch'] == 'x86_64': if kwargs['arch'] == 'x86_64':
arch = 'x86' arch = 'x86'
else: else:

9
build-modules
View File

@@ -7,7 +7,7 @@ import shutil
import common import common
class ModulesComponent(common.Component): class ModulesComponent(self.Component):
def add_parser_arguments(self, parser): def add_parser_arguments(self, parser):
parser.add_argument( parser.add_argument(
'--make-args', '--make-args',
@@ -15,7 +15,6 @@ class ModulesComponent(common.Component):
) )
parser.add_argument( parser.add_argument(
'--host', '--host',
action='store_true',
default=False, default=False,
help='''\ help='''\
Build the Linux kernel modules for the host instead of guest. Build the Linux kernel modules for the host instead of guest.
@@ -70,7 +69,7 @@ Use the host packaged cross toolchain.
else: else:
allowed_toolchains = None allowed_toolchains = None
build_subdir = kwargs['kernel_modules_build_subdir'] build_subdir = kwargs['kernel_modules_build_subdir']
gcc = common.get_toolchain_tool(tool, allowed_toolchains=allowed_toolchains) gcc = self.get_toolchain_tool(tool, allowed_toolchains=allowed_toolchains)
prefix = gcc[:-len(tool)] prefix = gcc[:-len(tool)]
ccache = shutil.which('ccache') ccache = shutil.which('ccache')
if ccache is not None: if ccache is not None:
@@ -97,13 +96,13 @@ Use the host packaged cross toolchain.
'M={}'.format(build_subdir), LF, 'M={}'.format(build_subdir), LF,
'OBJECT_FILES={}'.format(' '.join(object_files)), LF, 'OBJECT_FILES={}'.format(' '.join(object_files)), LF,
] + ] +
common.shlex_split(kwargs['make_args']) + self.sh.shlex_split(kwargs['make_args']) +
verbose verbose
), ),
cwd=os.path.join(kwargs['kernel_modules_build_subdir']), cwd=os.path.join(kwargs['kernel_modules_build_subdir']),
) )
if not kwargs['host']: if not kwargs['host']:
common.copy_dir_if_update_non_recursive( self.copy_dir_if_update_non_recursive(
srcdir=kwargs['kernel_modules_build_subdir'], srcdir=kwargs['kernel_modules_build_subdir'],
destdir=kwargs['out_rootfs_overlay_dir'], destdir=kwargs['out_rootfs_overlay_dir'],
filter_ext=kwargs['kernel_module_ext'], filter_ext=kwargs['kernel_module_ext'],

3
build-qemu
View File

@@ -4,12 +4,11 @@ import os
import common import common
class QemuComponent(common.Component): class QemuComponent(self.Component):
def add_parser_arguments(self, parser): def add_parser_arguments(self, parser):
parser.add_argument( parser.add_argument(
'--userland', '--userland',
default=False, default=False,
action='store_true',
help='Build QEMU user mode instead of system.', help='Build QEMU user mode instead of system.',
) )
parser.add_argument( parser.add_argument(

9
build-userland
View File

@@ -8,7 +8,7 @@ import subprocess
import common import common
class UserlandComponent(common.Component): class UserlandComponent(self.Component):
def add_parser_arguments(self, parser): def add_parser_arguments(self, parser):
parser.add_argument( parser.add_argument(
'--has-package', '--has-package',
@@ -21,7 +21,6 @@ allows us to build examples that rely on it.
) )
parser.add_argument( parser.add_argument(
'--host', '--host',
action='store_true',
default=False, default=False,
help='''\ help='''\
Build the userland programs for the host instead of guest. Build the userland programs for the host instead of guest.
@@ -51,8 +50,8 @@ has the OpenBLAS libraries and headers installed.
allowed_toolchains = ['host'] allowed_toolchains = ['host']
else: else:
allowed_toolchains = ['buildroot'] allowed_toolchains = ['buildroot']
cc = common.get_toolchain_tool('gcc', allowed_toolchains=allowed_toolchains) cc = self.get_toolchain_tool('gcc', allowed_toolchains=allowed_toolchains)
cxx = common.get_toolchain_tool('g++', allowed_toolchains=allowed_toolchains) cxx = self.get_toolchain_tool('g++', allowed_toolchains=allowed_toolchains)
self.sh.run_cmd( self.sh.run_cmd(
( (
[ [
@@ -74,7 +73,7 @@ has the OpenBLAS libraries and headers installed.
cwd=kwargs['userland_src_dir'], cwd=kwargs['userland_src_dir'],
extra_paths=[kwargs['ccache_dir']], extra_paths=[kwargs['ccache_dir']],
) )
common.copy_dir_if_update_non_recursive( self.copy_dir_if_update_non_recursive(
srcdir=build_dir, srcdir=build_dir,
destdir=kwargs['out_rootfs_overlay_dir'], destdir=kwargs['out_rootfs_overlay_dir'],
filter_ext=kwargs['userland_build_ext'], filter_ext=kwargs['userland_build_ext'],

67
cli_function.py Normal file → Executable file
View File

@@ -41,15 +41,17 @@ class Argument:
if nargs is not None: if nargs is not None:
self.kwargs['nargs'] = nargs self.kwargs['nargs'] = nargs
if default is True: if default is True:
self.kwargs['action'] = 'store_false' bool_action = 'store_false'
self.is_bool = True self.is_bool = True
elif default is False: elif default is False:
self.kwargs['action'] = 'store_true' bool_action = 'store_true'
self.is_bool = True self.is_bool = True
else: else:
self.is_bool = False self.is_bool = False
if default is None and nargs in ('*', '+'): if default is None and nargs in ('*', '+'):
default = [] default = []
if self.is_bool and not 'action' in kwargs:
self.kwargs['action'] = bool_action
if help is not None: if help is not None:
if not self.is_bool and default: if not self.is_bool and default:
help += ' Default: {}'.format(default) help += ' Default: {}'.format(default)
@@ -131,6 +133,9 @@ class CliFunction:
help='Path to the configuration file to use' help='Path to the configuration file to use'
) )
def __str__(self):
return '\n'.join(str(arg) for arg in self._arguments)
def add_argument( def add_argument(
self, self,
*args, *args,
@@ -155,11 +160,10 @@ class CliFunction:
new_longname = '--no' + argument.longname[1:] new_longname = '--no' + argument.longname[1:]
kwargs = argument.kwargs.copy() kwargs = argument.kwargs.copy()
kwargs['default'] = not argument.default kwargs['default'] = not argument.default
if argument.default: if kwargs['action'] == 'store_false':
action = 'store_true' kwargs['action'] = 'store_true'
else: elif kwargs['action'] == 'store_true':
action = 'store_false' kwargs['action'] = 'store_false'
kwargs['action'] = action
if 'help' in kwargs: if 'help' in kwargs:
del kwargs['help'] del kwargs['help']
parser.add_argument(new_longname, dest=argument.key, **kwargs) parser.add_argument(new_longname, dest=argument.key, **kwargs)
@@ -188,6 +192,7 @@ amazing function!
self.add_argument('-q', '--qwer', default='Q', help='Help for qwer'), self.add_argument('-q', '--qwer', default='Q', help='Help for qwer'),
self.add_argument('-b', '--bool', default=True, help='Help for bool'), self.add_argument('-b', '--bool', default=True, help='Help for bool'),
self.add_argument('--bool-cli', default=False, help='Help for bool'), self.add_argument('--bool-cli', default=False, help='Help for bool'),
self.add_argument('--bool-nargs', default=False, nargs='?', action='store', const='')
self.add_argument('--no-default', help='Help for no-bool'), self.add_argument('--no-default', help='Help for no-bool'),
self.add_argument('pos-mandatory', help='Help for pos-mandatory', type=int), self.add_argument('pos-mandatory', help='Help for pos-mandatory', type=int),
self.add_argument('pos-optional', default=0, help='Help for pos-optional', type=int), self.add_argument('pos-optional', default=0, help='Help for pos-optional', type=int),
@@ -196,32 +201,62 @@ amazing function!
del kwargs['config_file'] del kwargs['config_file']
return kwargs return kwargs
# Code calls. one_cli_function = OneCliFunction()
default = OneCliFunction()(pos_mandatory=1)
assert default == {'asdf': 'A', 'qwer': 'Q', 'bool': True, 'bool_cli': True, 'no_default': None, 'pos_mandatory': 1, 'pos_optional': 0, 'args_star': []} # Default code call.
default = one_cli_function(pos_mandatory=1)
assert default == {
'asdf': 'A',
'qwer': 'Q',
'bool': True,
'bool_nargs': False,
'bool_cli': True,
'no_default': None,
'pos_mandatory': 1,
'pos_optional': 0,
'args_star': []
}
# Default CLI call.
out = one_cli_function.cli(['1'])
assert out == default
# asdf # asdf
out = OneCliFunction()(pos_mandatory=1, asdf='B') out = one_cli_function(pos_mandatory=1, asdf='B')
assert out['asdf'] == 'B' assert out['asdf'] == 'B'
out['asdf'] = default['asdf'] out['asdf'] = default['asdf']
assert(out == default) assert(out == default)
# asdf and qwer # asdf and qwer
out = OneCliFunction()(pos_mandatory=1, asdf='B', qwer='R') out = one_cli_function(pos_mandatory=1, asdf='B', qwer='R')
assert out['asdf'] == 'B' assert out['asdf'] == 'B'
assert out['qwer'] == 'R' assert out['qwer'] == 'R'
out['asdf'] = default['asdf'] out['asdf'] = default['asdf']
out['qwer'] = default['qwer'] out['qwer'] = default['qwer']
assert(out == default) assert(out == default)
# bool if '--bool':
out = OneCliFunction()(pos_mandatory=1, bool=False) out = one_cli_function(pos_mandatory=1, bool=False)
cli_out = one_cli_function.cli(['--bool', '1'])
assert out == cli_out
assert out['bool'] == False assert out['bool'] == False
out['bool'] = default['bool'] out['bool'] = default['bool']
assert(out == default) assert(out == default)
if '--bool-nargs':
out = one_cli_function(pos_mandatory=1, bool_nargs=True)
assert out['bool_nargs'] == True
out['bool_nargs'] = default['bool_nargs']
assert(out == default)
out = one_cli_function(pos_mandatory=1, bool_nargs='asdf')
assert out['bool_nargs'] == 'asdf'
out['bool_nargs'] = default['bool_nargs']
assert(out == default)
# Force a boolean value set on the config to be False on CLI. # Force a boolean value set on the config to be False on CLI.
assert OneCliFunction().cli(['--no-bool-cli', '1'])['bool_cli'] is False assert one_cli_function.cli(['--no-bool-cli', '1'])['bool_cli'] is False
# CLI call. # CLI call.
print(OneCliFunction().cli()) print(one_cli_function.cli())

349
common.py
View File

@@ -13,7 +13,6 @@ import os
import re import re
import shutil import shutil
import signal import signal
import stat
import subprocess import subprocess
import sys import sys
import time import time
@@ -120,7 +119,7 @@ mkdir are generally omitted since those are obvious
''' '''
) )
self.add_argument( self.add_argument(
'-v', '--verbose', default=False, action='store_true', '-v', '--verbose', default=False,
help='Show full compilation commands when they are not shown by default.' help='Show full compilation commands when they are not shown by default.'
) )
@@ -163,10 +162,10 @@ Linux build ID. Allows you to keep multiple separate Linux builds.
''' '''
) )
self.add_argument( self.add_argument(
'--initramfs', default=False, action='store_true', '--initramfs', default=False,
) )
self.add_argument( self.add_argument(
'--initrd', default=False, action='store_true', '--initrd', default=False,
) )
# Baremetal. # Baremetal.
@@ -189,7 +188,7 @@ inside baremetal/ and then try to use corresponding executable.
help='Buildroot build ID. Allows you to keep multiple separate gem5 builds.' help='Buildroot build ID. Allows you to keep multiple separate gem5 builds.'
) )
self.add_argument( self.add_argument(
'--buildroot-linux', default=False, action='store_true', '--buildroot-linux', default=False,
help='Boot with the Buildroot Linux kernel instead of our custom built one. Mostly for sanity checks.' help='Boot with the Buildroot Linux kernel instead of our custom built one. Mostly for sanity checks.'
) )
@@ -199,7 +198,7 @@ inside baremetal/ and then try to use corresponding executable.
help='Crosstool-NG build ID. Allows you to keep multiple separate crosstool-NG builds.' help='Crosstool-NG build ID. Allows you to keep multiple separate crosstool-NG builds.'
) )
self.add_argument( self.add_argument(
'--docker', default=False, action='store_true', '--docker', default=False,
help='''\ help='''\
Use the docker download Ubuntu root filesystem instead of the default Buildroot one. Use the docker download Ubuntu root filesystem instead of the default Buildroot one.
''' '''
@@ -234,7 +233,7 @@ and then inspect separate outputs later in different output directories.
''' '''
) )
self.add_argument( self.add_argument(
'-P', '--prebuilt', default=False, action='store_true', '-P', '--prebuilt', default=False,
help='''\ help='''\
Use prebuilt packaged host utilities as much as possible instead Use prebuilt packaged host utilities as much as possible instead
of the ones we built ourselves. Saves build time, but decreases of the ones we built ourselves. Saves build time, but decreases
@@ -251,11 +250,11 @@ instances in parallel. Default: the run ID (-n) if that is an integer, otherwise
# Misc. # Misc.
self.add_argument( self.add_argument(
'-g', '--gem5', default=False, action='store_true', '-g', '--gem5', default=False,
help='Use gem5 instead of QEMU.' help='Use gem5 instead of QEMU.'
) )
self.add_argument( self.add_argument(
'--qemu', default=False, action='store_true', '--qemu', default=False,
help='''\ help='''\
Use QEMU as the emulator. This option exists in addition to --gem5 Use QEMU as the emulator. This option exists in addition to --gem5
to allow overriding configs from the CLI. to allow overriding configs from the CLI.
@@ -426,6 +425,7 @@ to allow overriding configs from the CLI.
env['executable'] = env['qemu_executable'] env['executable'] = env['qemu_executable']
env['run_dir'] = env['qemu_run_dir'] env['run_dir'] = env['qemu_run_dir']
env['termout_file'] = env['qemu_termout_file'] env['termout_file'] = env['qemu_termout_file']
env['guest_terminal_file'] = env['qemu_termout_file']
env['trace_txt_file'] = env['qemu_trace_txt_file'] env['trace_txt_file'] = env['qemu_trace_txt_file']
env['run_cmd_file'] = join(env['run_dir'], 'run.sh') env['run_cmd_file'] = join(env['run_dir'], 'run.sh')
@@ -497,7 +497,7 @@ to allow overriding configs from the CLI.
if env['baremetal'] == 'all': if env['baremetal'] == 'all':
path = env['baremetal'] path = env['baremetal']
else: else:
path = resolve_executable( path = self.resolve_executable(
env['baremetal'], env['baremetal'],
env['baremetal_src_dir'], env['baremetal_src_dir'],
env['baremetal_build_dir'], env['baremetal_build_dir'],
@@ -515,6 +515,23 @@ to allow overriding configs from the CLI.
env['image'] = path env['image'] = path
self.env = env self.env = env
def assert_crosstool_ng_supports_arch(self, arch):
if arch not in self.env['crosstool_ng_supported_archs']:
raise Exception('arch not yet supported: ' + arch)
@staticmethod
def base64_encode(string):
return base64.b64encode(string.encode()).decode()
def gem_list_checkpoint_dirs(self):
'''
List checkpoint directory, oldest first.
'''
prefix_re = re.compile(self.env['gem5_cpt_prefix'])
files = list(filter(lambda x: os.path.isdir(os.path.join(self.env['m5out_dir'], x)) and prefix_re.search(x), os.listdir(self.env['m5out_dir'])))
files.sort(key=lambda x: os.path.getmtime(os.path.join(self.env['m5out_dir'], x)))
return files
def get_elf_entry(self, elf_file_path): def get_elf_entry(self, elf_file_path):
readelf_header = subprocess.check_output([ readelf_header = subprocess.check_output([
self.get_toolchain_tool('readelf'), self.get_toolchain_tool('readelf'),
@@ -528,6 +545,21 @@ to allow overriding configs from the CLI.
break break
return int(addr, 0) return int(addr, 0)
def get_stats(self, stat_re=None, stats_file=None):
if stat_re is None:
stat_re = '^system.cpu[0-9]*.numCycles$'
if stats_file is None:
stats_file = self.env['stats_file']
stat_re = re.compile(stat_re)
ret = []
with open(stats_file, 'r') as statfile:
for line in statfile:
if line[0] != '-':
cols = line.split()
if len(cols) > 1 and stat_re.search(cols[0]):
ret.append(cols[1])
return ret
def get_toolchain_prefix(self, tool, allowed_toolchains=None): def get_toolchain_prefix(self, tool, allowed_toolchains=None):
buildroot_full_prefix = os.path.join(self.env['host_bin_dir'], self.env['buildroot_toolchain_prefix']) buildroot_full_prefix = os.path.join(self.env['host_bin_dir'], self.env['buildroot_toolchain_prefix'])
buildroot_exists = os.path.exists('{}-{}'.format(buildroot_full_prefix, tool)) buildroot_exists = os.path.exists('{}-{}'.format(buildroot_full_prefix, tool))
@@ -562,6 +594,41 @@ to allow overriding configs from the CLI.
def get_toolchain_tool(self, tool, allowed_toolchains=None): def get_toolchain_tool(self, tool, allowed_toolchains=None):
return '{}-{}'.format(self.get_toolchain_prefix(tool, allowed_toolchains), tool) return '{}-{}'.format(self.get_toolchain_prefix(tool, allowed_toolchains), tool)
@staticmethod
def github_make_request(
authenticate=False,
data=None,
extra_headers=None,
path='',
subdomain='api',
url_params=None,
**extra_request_args
):
if extra_headers is None:
extra_headers = {}
headers = {'Accept': 'application/vnd.github.v3+json'}
headers.update(extra_headers)
if authenticate:
headers['Authorization'] = 'token ' + os.environ['LKMC_GITHUB_TOKEN']
if url_params is not None:
path += '?' + urllib.parse.urlencode(url_params)
request = urllib.request.Request(
'https://' + subdomain + '.github.com/repos/' + github_repo_id + path,
headers=headers,
data=data,
**extra_request_args
)
response_body = urllib.request.urlopen(request).read().decode()
if response_body:
_json = json.loads(response_body)
else:
_json = {}
return _json
@staticmethod
def log_error(msg):
print('error: {}'.format(msg), file=sys.stderr)
def main(self, **kwargs): def main(self, **kwargs):
''' '''
Time the main of the derived class. Time the main of the derived class.
@@ -571,13 +638,107 @@ to allow overriding configs from the CLI.
kwargs.update(consts) kwargs.update(consts)
self._init_env(kwargs) self._init_env(kwargs)
self.sh = shell_helpers.ShellHelpers(dry_run=self.env['dry_run']) self.sh = shell_helpers.ShellHelpers(dry_run=self.env['dry_run'])
self.timed_main() ret = self.timed_main()
if not kwargs['dry_run']: if not kwargs['dry_run']:
end_time = time.time() end_time = time.time()
print_time(end_time - start_time) self.print_time(end_time - start_time)
return ret
def run_cmd(self, *args, **kwargs): def make_build_dirs(self):
self.sh.run_cmd(*args, **kwargs) os.makedirs(self.env['buildroot_build_build_dir'], exist_ok=True)
os.makedirs(self.env['gem5_build_dir'], exist_ok=True)
os.makedirs(self.env['out_rootfs_overlay_dir'], exist_ok=True)
def make_run_dirs(self):
'''
Make directories required for the run.
The user could nuke those anytime between runs to try and clean things up.
'''
os.makedirs(self.env['gem5_run_dir'], exist_ok=True)
os.makedirs(self.env['p9_dir'], exist_ok=True)
os.makedirs(self.env['qemu_run_dir'], exist_ok=True)
@staticmethod
def need_rebuild(srcs, dst):
if not os.path.exists(dst):
return True
for src in srcs:
if os.path.getmtime(src) > os.path.getmtime(dst):
return True
return False
@staticmethod
def print_time(ellapsed_seconds):
hours, rem = divmod(ellapsed_seconds, 3600)
minutes, seconds = divmod(rem, 60)
print("time {:02}:{:02}:{:02}".format(int(hours), int(minutes), int(seconds)))
def raw_to_qcow2(eslf, prebuilt=False, reverse=False):
if prebuilt or not os.path.exists(self.env['qemu_img_executable']):
disable_trace = []
qemu_img_executable = self.env['qemu_img_basename']
else:
# Prevent qemu-img from generating trace files like QEMU. Disgusting.
disable_trace = ['-T', 'pr_manager_run,file=/dev/null', LF,]
qemu_img_executable = self.env['qemu_img_executable']
infmt = 'raw'
outfmt = 'qcow2'
infile = self.env['rootfs_raw_file']
outfile = self.env['qcow2_file']
if reverse:
tmp = infmt
infmt = outfmt
outfmt = tmp
tmp = infile
infile = outfile
outfile = tmp
self.sh.run_cmd(
[
qemu_img_executable, LF,
] +
disable_trace +
[
'convert', LF,
'-f', infmt, LF,
'-O', outfmt, LF,
infile, LF,
outfile, LF,
]
)
@staticmethod
def resolve_args(defaults, args, extra_args):
if extra_args is None:
extra_args = {}
argcopy = copy.copy(args)
argcopy.__dict__ = dict(list(defaults.items()) + list(argcopy.__dict__.items()) + list(extra_args.items()))
return argcopy
@staticmethod
def resolve_executable(in_path, magic_in_dir, magic_out_dir, out_ext):
if os.path.isabs(in_path):
return in_path
else:
paths = [
os.path.join(magic_out_dir, in_path),
os.path.join(
magic_out_dir,
os.path.relpath(in_path, magic_in_dir),
)
]
paths[:] = [os.path.splitext(path)[0] + out_ext for path in paths]
for path in paths:
if os.path.exists(path):
return path
raise Exception('Executable file not found. Tried:\n' + '\n'.join(paths))
def resolve_userland(self, path):
return self.resolve_executable(
path,
self.env['userland_src_dir'],
self.env['userland_build_dir'],
self.env['userland_build_ext'],
)
def timed_main(self): def timed_main(self):
''' '''
@@ -628,162 +789,6 @@ class BuildCliFunction(LkmcCliFunction):
The actual build work is done by do_build in implementing classes. The actual build work is done by do_build in implementing classes.
''' '''
if self.env['clean']: if self.env['clean']:
self.clean() return self.clean()
else: else:
self.build() return self.build()
def assert_crosstool_ng_supports_arch(arch):
if arch not in kwargs['crosstool_ng_supported_archs']:
raise Exception('arch not yet supported: ' + arch)
def base64_encode(string):
return base64.b64encode(string.encode()).decode()
def gem_list_checkpoint_dirs():
'''
List checkpoint directory, oldest first.
'''
prefix_re = re.compile(kwargs['gem5_cpt_prefix'])
files = list(filter(lambda x: os.path.isdir(os.path.join(kwargs['m5out_dir'], x)) and prefix_re.search(x), os.listdir(kwargs['m5out_dir'])))
files.sort(key=lambda x: os.path.getmtime(os.path.join(kwargs['m5out_dir'], x)))
return files
def get_stats(stat_re=None, stats_file=None):
if stat_re is None:
stat_re = '^system.cpu[0-9]*.numCycles$'
if stats_file is None:
stats_file = kwargs['stats_file']
stat_re = re.compile(stat_re)
ret = []
with open(stats_file, 'r') as statfile:
for line in statfile:
if line[0] != '-':
cols = line.split()
if len(cols) > 1 and stat_re.search(cols[0]):
ret.append(cols[1])
return ret
def github_make_request(
authenticate=False,
data=None,
extra_headers=None,
path='',
subdomain='api',
url_params=None,
**extra_request_args
):
if extra_headers is None:
extra_headers = {}
headers = {'Accept': 'application/vnd.github.v3+json'}
headers.update(extra_headers)
if authenticate:
headers['Authorization'] = 'token ' + os.environ['LKMC_GITHUB_TOKEN']
if url_params is not None:
path += '?' + urllib.parse.urlencode(url_params)
request = urllib.request.Request(
'https://' + subdomain + '.github.com/repos/' + github_repo_id + path,
headers=headers,
data=data,
**extra_request_args
)
response_body = urllib.request.urlopen(request).read().decode()
if response_body:
_json = json.loads(response_body)
else:
_json = {}
return _json
def log_error(msg):
print('error: {}'.format(msg), file=sys.stderr)
def make_build_dirs():
os.makedirs(kwargs['buildroot_build_build_dir'], exist_ok=True)
os.makedirs(kwargs['gem5_build_dir'], exist_ok=True)
os.makedirs(kwargs['out_rootfs_overlay_dir'], exist_ok=True)
def make_run_dirs():
'''
Make directories required for the run.
The user could nuke those anytime between runs to try and clean things up.
'''
os.makedirs(kwargs['gem5_run_dir'], exist_ok=True)
os.makedirs(kwargs['p9_dir'], exist_ok=True)
os.makedirs(kwargs['qemu_run_dir'], exist_ok=True)
def need_rebuild(srcs, dst):
if not os.path.exists(dst):
return True
for src in srcs:
if os.path.getmtime(src) > os.path.getmtime(dst):
return True
return False
def print_time(ellapsed_seconds):
hours, rem = divmod(ellapsed_seconds, 3600)
minutes, seconds = divmod(rem, 60)
print("time {:02}:{:02}:{:02}".format(int(hours), int(minutes), int(seconds)))
def raw_to_qcow2(prebuilt=False, reverse=False):
if prebuilt or not os.path.exists(kwargs['qemu_img_executable']):
disable_trace = []
qemu_img_executable = kwargs['qemu_img_basename']
else:
# Prevent qemu-img from generating trace files like QEMU. Disgusting.
disable_trace = ['-T', 'pr_manager_run,file=/dev/null', LF,]
qemu_img_executable = kwargs['qemu_img_executable']
infmt = 'raw'
outfmt = 'qcow2'
infile = kwargs['rootfs_raw_file']
outfile = kwargs['qcow2_file']
if reverse:
tmp = infmt
infmt = outfmt
outfmt = tmp
tmp = infile
infile = outfile
outfile = tmp
self.sh.run_cmd(
[
qemu_img_executable, LF,
] +
disable_trace +
[
'convert', LF,
'-f', infmt, LF,
'-O', outfmt, LF,
infile, LF,
outfile, LF,
]
)
def resolve_args(defaults, args, extra_args):
if extra_args is None:
extra_args = {}
argcopy = copy.copy(args)
argcopy.__dict__ = dict(list(defaults.items()) + list(argcopy.__dict__.items()) + list(extra_args.items()))
return argcopy
def resolve_executable(in_path, magic_in_dir, magic_out_dir, out_ext):
if os.path.isabs(in_path):
return in_path
else:
paths = [
os.path.join(magic_out_dir, in_path),
os.path.join(
magic_out_dir,
os.path.relpath(in_path, magic_in_dir),
)
]
paths[:] = [os.path.splitext(path)[0] + out_ext for path in paths]
for path in paths:
if os.path.exists(path):
return path
raise Exception('Executable file not found. Tried:\n' + '\n'.join(paths))
def resolve_userland(path):
return resolve_executable(
path,
kwargs['userland_src_dir'],
kwargs['userland_build_dir'],
kwargs['userland_build_ext'],
)

2
copy-overlay
View File

@@ -6,7 +6,7 @@ import shutil
import common import common
class CopyOverlayComponent(common.Component): class CopyOverlayComponent(self.Component):
def do_build(self, args): def do_build(self, args):
distutils.dir_util.copy_tree( distutils.dir_util.copy_tree(
kwargs['rootfs_overlay_dir'], kwargs['rootfs_overlay_dir'],

4
gem5-shell
View File

@@ -4,11 +4,11 @@ import sys
import common import common
parser = common.get_argparse( parser = self.get_argparse(
default_args={'gem5':True}, default_args={'gem5':True},
argparse_args={'description':'Connect a terminal to a running gem5 instance'} argparse_args={'description':'Connect a terminal to a running gem5 instance'}
) )
args = common.setup(parser) args = self.setup(parser)
sys.exit(self.sh.run_cmd([ sys.exit(self.sh.run_cmd([
kwargs['gem5_m5term'], LF, kwargs['gem5_m5term'], LF,
'localhost', LF, 'localhost', LF,

6
gem5-stat
View File

@@ -1,6 +1,6 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
import common import common
parser = common.get_argparse( parser = self.get_argparse(
argparse_args={'description':'Get the value of a gem5 stat from the stats.txt file.'} argparse_args={'description':'Get the value of a gem5 stat from the stats.txt file.'}
) )
parser.add_argument( parser.add_argument(
@@ -9,6 +9,6 @@ parser.add_argument(
help='Python regexp matching the full stat name of interest', help='Python regexp matching the full stat name of interest',
nargs='?', nargs='?',
) )
args = common.setup(parser) args = self.setup(parser)
stats = common.get_stats(kwargs['stat']) stats = self.get_stats(kwargs['stat'])
print('\n'.join(stats)) print('\n'.join(stats))

4
getvar
View File

@@ -4,7 +4,7 @@ import types
import common import common
parser = common.get_argparse(argparse_args={ parser = self.get_argparse(argparse_args={
'description': '''Print the value of a kwargs['py'] variable. 'description': '''Print the value of a kwargs['py'] variable.
This is useful to: This is useful to:
@@ -27,7 +27,7 @@ List all available variables:
''' '''
}) })
parser.add_argument('variable', nargs='?') parser.add_argument('variable', nargs='?')
args = common.setup(parser) args = self.setup(parser)
if kwargs['variable']: if kwargs['variable']:
print(getattr(common, kwargs['variable'])) print(getattr(common, kwargs['variable']))
else: else:

4
qemu-monitor
View File

@@ -8,7 +8,7 @@ import common
prompt = b'\n(qemu) ' prompt = b'\n(qemu) '
parser = common.get_argparse({ parser = self.get_argparse({
'description': '''\ 'description': '''\
Run a command on the QEMU monitor of a running QEMU instance Run a command on the QEMU monitor of a running QEMU instance
@@ -21,7 +21,7 @@ parser.add_argument(
help='If given, run this command and quit', help='If given, run this command and quit',
nargs='*', nargs='*',
) )
args = common.setup(parser) args = self.setup(parser)
def write_and_read(tn, cmd, prompt): def write_and_read(tn, cmd, prompt):
tn.write(cmd.encode('utf-8')) tn.write(cmd.encode('utf-8'))

4
qemu-trace2txt
View File

@@ -19,8 +19,8 @@ def main():
) )
if __name__ == '__main__': if __name__ == '__main__':
parser = common.get_argparse(argparse_args={ parser = self.get_argparse(argparse_args={
'description': 'Convert a QEMU `-trace exec_tb` to text form.' 'description': 'Convert a QEMU `-trace exec_tb` to text form.'
}) })
args = common.setup(parser) args = self.setup(parser)
sys.exit(main()) sys.exit(main())

2
release
View File

@@ -29,4 +29,4 @@ release_zip.main()
subprocess.check_call(['git', 'push']) subprocess.check_call(['git', 'push'])
release_upload.main() release_upload.main()
end_time = time.time() end_time = time.time()
common.print_time(end_time - start_time) self.print_time(end_time - start_time)

2
release-download-latest
View File

@@ -11,6 +11,6 @@ import urllib.request
import common import common
_json = common.github_make_request(path='/releases') _json = self.github_make_request(path='/releases')
asset = _json[0]['assets'][0] asset = _json[0]['assets'][0]
urllib.request.urlretrieve(asset['browser_download_url'], asset['name']) urllib.request.urlretrieve(asset['browser_download_url'], asset['name'])

10
release-upload
View File

@@ -24,7 +24,7 @@ def main():
# Check the release already exists. # Check the release already exists.
try: try:
_json = common.github_make_request(path='/releases/tags/' + tag) _json = self.github_make_request(path='/releases/tags/' + tag)
except urllib.error.HTTPError as e: except urllib.error.HTTPError as e:
if e.code == 404: if e.code == 404:
release_exists = False release_exists = False
@@ -36,7 +36,7 @@ def main():
# Create release if not yet created. # Create release if not yet created.
if not release_exists: if not release_exists:
_json = common.github_make_request( _json = self.github_make_request(
authenticate=True, authenticate=True,
data=json.dumps({ data=json.dumps({
'tag_name': tag, 'tag_name': tag,
@@ -50,12 +50,12 @@ def main():
asset_name = os.path.split(upload_path)[1] asset_name = os.path.split(upload_path)[1]
# Clear the prebuilts for a upload. # Clear the prebuilts for a upload.
_json = common.github_make_request( _json = self.github_make_request(
path=('/releases/' + str(release_id) + '/assets'), path=('/releases/' + str(release_id) + '/assets'),
) )
for asset in _json: for asset in _json:
if asset['name'] == asset_name: if asset['name'] == asset_name:
_json = common.github_make_request( _json = self.github_make_request(
authenticate=True, authenticate=True,
path=('/releases/assets/' + str(asset['id'])), path=('/releases/assets/' + str(asset['id'])),
method='DELETE', method='DELETE',
@@ -65,7 +65,7 @@ def main():
# Upload the prebuilt. # Upload the prebuilt.
with open(upload_path, 'br') as myfile: with open(upload_path, 'br') as myfile:
content = myfile.read() content = myfile.read()
_json = common.github_make_request( _json = self.github_make_request(
authenticate=True, authenticate=True,
data=content, data=content,
extra_headers={'Content-Type': 'application/zip'}, extra_headers={'Content-Type': 'application/zip'},

2
release-zip
View File

@@ -16,7 +16,7 @@ def main():
os.unlink(kwargs['release_zip_file']) os.unlink(kwargs['release_zip_file'])
zipf = zipfile.ZipFile(kwargs['release_zip_file'], 'w', zipfile.ZIP_DEFLATED) zipf = zipfile.ZipFile(kwargs['release_zip_file'], 'w', zipfile.ZIP_DEFLATED)
for arch in kwargs['all_archs']: for arch in kwargs['all_archs']:
common.setup(common.get_argparse(default_args={'arch': arch})) self.setup(common.get_argparse(default_args={'arch': arch}))
zipf.write(kwargs['qcow2_file'], arcname=os.path.relpath(kwargs['qcow2_file'], kwargs['root_dir'])) zipf.write(kwargs['qcow2_file'], arcname=os.path.relpath(kwargs['qcow2_file'], kwargs['root_dir']))
zipf.write(kwargs['linux_image'], arcname=os.path.relpath(kwargs['linux_image'], kwargs['root_dir'])) zipf.write(kwargs['linux_image'], arcname=os.path.relpath(kwargs['linux_image'], kwargs['root_dir']))
zipf.close() zipf.close()

715
run
View File

@@ -8,196 +8,362 @@ import sys
import time import time
import common import common
from shell_helpers import LF
defaults = { class Main(common.LkmcCliFunction):
'background': False, def __init__(self):
'cpus': 1, super().__init__(
'wait_gdb': False, description='''\
'debug_vm': None, Run some content on an emulator.
'eval': None, '''
'extra_emulator_args': [], )
'gem5_exe_args': '', self.add_argument(
'gem5_script': 'fs', '--background', default=False,
'gem5_readfile': '', help='''\
'gem5_restore': None, Send QEMU output to a file instead of the terminal so it does not require a
'graphic': False, terminal attached to run on the background. Interactive input cannot be given.
'kernel_cli': None, TODO: use a port instead. If only there was a way to redirect a serial to multiple
'kernel_cli_after_dash': None, places, both to a port and a file? We use the file currently to be able to have
'eval_after': None, any output at all.
'kgdb': False, https://superuser.com/questions/1373226/how-to-redirect-qemu-serial-output-to-both-a-file-and-the-terminal-or-a-port
'kdb': False, '''
'kvm': False, )
'memory': '256M', self.add_argument(
'record': False, '-c', '--cpus', default=1, type=int,
'replay': False, help='Number of guest CPUs to emulate. Default: %(default)s'
'terminal': False, )
'tmux': None, self.add_argument(
'trace': None, '-D', '--debug-vm', default=False,
'trace_stdout': False, help='Run GDB on the emulator itself.'
'userland': None, )
'userland_before': '', self.add_argument(
'vnc': False, '--debug-vm-args', default='',
} help='Pass arguments to GDB.'
)
self.add_argument(
'-E', '--eval',
help='''\
Replace the normal init with a minimal init that just evals the given string.
See: https://github.com/cirosantilli/linux-kernel-module-cheat#replace-init
'''
)
self.add_argument(
'-e', '--kernel-cli',
help='''\
Pass an extra Linux kernel command line options, and place them before
the dash separator `-`. Only options that come before the `-`, i.e.
"standard" options, should be passed with this option.
Example: `./run -a arm -e 'init=/poweroff.out'`
'''
)
self.add_argument(
'-F', '--eval-after',
help='''\
Pass a base64 encoded command line parameter that gets evalled at the end of
the normal init.
See: https://github.com/cirosantilli/linux-kernel-module-cheat#init-busybox
'''
)
self.add_argument(
'-f', '--kernel-cli-after-dash',
help='''\
Pass an extra Linux kernel command line options, add a dash `-`
separator, and place the options after the dash. Intended for custom
options understood by our `init` scripts, most of which are prefixed
by `lkmc_`.
Example: `./run --kernel-cli-after-dash 'lkmc_eval="wget google.com" lkmc_lala=y'`
Mnenomic: `-f` comes after `-e`.
'''
)
self.add_argument(
'-G', '--gem5-exe-args', default='',
help='''\
Pass extra options to the gem5 executable.
Do not confuse with the arguments passed to config scripts,
like `fs.py`. Example:
./run -G '--debug-flags=Exec --debug' --gem5 -- --cpu-type=HPI --caches
will run:
gem.op5 --debug-flags=Exec fs.py --cpu-type=HPI --caches
'''
)
self.add_argument(
'--gem5-script', default='fs', choices=['fs', 'biglittle'],
help='Which gem5 script to use'
)
self.add_argument(
'--gem5-readfile', default='',
help='Set the contents of m5 readfile to this string.'
)
self.add_argument(
'-K', '--kvm', default=False,
help='Use KVM. Only works if guest arch == host arch'
)
self.add_argument(
'--kgdb', default=False,
)
self.add_argument(
'--kdb', default=False,
)
self.add_argument(
'-l', '--gem5-restore', type=int,
help='''\
Restore the nth most recently taken gem5 checkpoint according to directory
timestamps.
'''
)
self.add_argument(
'-m', '--memory', default='256M',
help='''\
Set the memory size of the guest. E.g.: `-m 512M`. We try to keep the default
at the minimal ammount amount that boots all archs. Anything lower could lead
some arch to fail to boot.
Default: %(default)s
'''
)
self.add_argument(
'-R', '--replay', default=False,
help='Replay a QEMU run record deterministically'
)
self.add_argument(
'-r', '--record', default=False,
help='Record a QEMU run record for later replay with `-R`'
)
self.add_argument(
'-T', '--trace',
help='''\
Set trace events to be enabled. If not given, gem5 tracing is completely
disabled, while QEMU tracing is enabled but uses default traces that are very
rare and don't affect performance, because `./configure
--enable-trace-backends=simple` seems to enable some traces by default, e.g.
`pr_manager_run`, and I don't know how to get rid of them.
'''
)
self.add_argument(
'--trace-stdout', default=False,
help='''\
Output trace to stdout instead of a file. Only works for gem5 currently.
'''
)
self.add_argument(
'--terminal', default=False,
help='''\
Output to the terminal, don't pipe to tee as the default.
Does not save the output to a file, but allows you to use debuggers.
Set automatically by --debug-vm, but you still need this option to debug
gem5 Python scripts.
'''
)
self.add_argument(
'-t', '--tmux', default=False,
help='''\
Create a tmux split the window. You must already be inside of a `tmux` session
to use this option:
* on the main window, run the emulator as usual
* on the split:
** if on QEMU and `-d` is given, GDB
** if on gem5, the gem5 terminal
'''
)
self.add_argument(
'--tmux-args',
help='''\
Parameters to pass to the program running on the tmux split.
'''
)
self.add_argument(
'-u', '--userland',
help='''\
Run the given userland executable in user mode instead of booting the Linux kernel
in full system mode. In gem5, user mode is called Syscall Emulation (SE) mode and
uses se.py.
Path resolution is similar to --baremetal.
'''
)
self.add_argument(
'--userland-before', default='',
help='''\
Pass these arguments to the QEMU user mode CLI before the program to execute.
This is required with --userland since arguments that come at the end are interpreted
as command line arguments to that executable.
'''
)
self.add_argument(
'-w', '--wait-gdb', default=False,
help='Wait for GDB to connect before starting execution'
)
self.add_argument(
'-x', '--graphic', default=False,
help='Run in graphic mode. Mnemonic: X11'
)
self.add_argument(
'-V', '--vnc', default=False,
help='''\
Run QEMU with VNC instead of the default SDL. Connect to it with:
`vinagre localhost:5900`.
'''
)
self.add_argument(
'extra_emulator_args', nargs='*', default=[],
help='Extra options to append at the end of the emulator command line'
)
def main(args, extra_args=None): def timed_main(self):
global defaults
args = common.resolve_args(defaults, args, extra_args)
# Common qemu / gem5 logic. # Common qemu / gem5 logic.
# nokaslr: # nokaslr:
# * https://unix.stackexchange.com/questions/397939/turning-off-kaslr-to-debug-linux-kernel-using-qemu-and-gdb # * https://unix.stackexchange.com/questions/397939/turning-off-kaslr-to-debug-linux-kernel-using-qemu-and-gdb
# * https://stackoverflow.com/questions/44612822/unable-to-debug-kernel-with-qemu-gdb/49840927#49840927 # * https://stackoverflow.com/questions/44612822/unable-to-debug-kernel-with-qemu-gdb/49840927#49840927
# Turned on by default since v4.12 # Turned on by default since v4.12
kernel_cli = 'console_msg_format=syslog nokaslr norandmaps panic=-1 printk.devkmsg=on printk.time=y rw' kernel_cli = 'console_msg_format=syslog nokaslr norandmaps panic=-1 printk.devkmsg=on printk.time=y rw'
if kwargs['kernel_cli'] is not None: if self.env['kernel_cli'] is not None:
kernel_cli += ' {}'.format(kwargs['kernel_cli']) kernel_cli += ' {}'.format(self.env['kernel_cli'])
kernel_cli_after_dash = '' kernel_cli_after_dash = ''
extra_emulator_args = [] extra_emulator_args = []
extra_qemu_args = [] extra_qemu_args = []
if kwargs['debug_vm'] is not None: if self.env['debug_vm'] is not None:
debug_vm = ['gdb', LF, '-q', common.Newline] + common.shlex_split(kwargs['debug_vm']) + ['--args', common.Newline] debug_vm = ['gdb', LF, '-q', LF] + self.sh.shlex_split(self.env['debug_vm_args']) + ['--args', LF]
else: else:
debug_vm = [] debug_vm = []
if kwargs['wait_gdb']: if self.env['wait_gdb']:
extra_qemu_args.extend(['-S', LF]) extra_qemu_args.extend(['-S', LF])
if kwargs['eval_after'] is not None: if self.env['eval_after'] is not None:
kernel_cli_after_dash += ' lkmc_eval_base64="{}"'.format(common.base64_encode(kwargs['eval_after'])) kernel_cli_after_dash += ' lkmc_eval_base64="{}"'.format(self.base64_encode(self.env['eval_after']))
if kwargs['kernel_cli_after_dash'] is not None: if self.env['kernel_cli_after_dash'] is not None:
kernel_cli_after_dash += ' {}'.format(kwargs['kernel_cli_after_dash']) kernel_cli_after_dash += ' {}'.format(self.env['kernel_cli_after_dash'])
if kwargs['vnc']: if self.env['vnc']:
vnc = ['-vnc', ':0', LF] vnc = ['-vnc', ':0', LF]
else: else:
vnc = [] vnc = []
if kwargs['initrd'] or kwargs['initramfs']: if self.env['initrd'] or self.env['initramfs']:
ramfs = True ramfs = True
else: else:
ramfs = False ramfs = False
if kwargs['eval'] is not None: if self.env['eval'] is not None:
if ramfs: if ramfs:
initarg = 'rdinit' initarg = 'rdinit'
else: else:
initarg = 'init' initarg = 'init'
kernel_cli += ' {}=/eval_base64.sh'.format(initarg) kernel_cli += ' {}=/eval_base64.sh'.format(initarg)
kernel_cli_after_dash += ' lkmc_eval="{}"'.format(common.base64_encode(kwargs['eval'])) kernel_cli_after_dash += ' lkmc_eval="{}"'.format(self.base64_encode(self.env['eval']))
if not kwargs['graphic']: if not self.env['graphic']:
extra_qemu_args.extend(['-nographic', LF]) extra_qemu_args.extend(['-nographic', LF])
console = None console = None
console_type = None console_type = None
console_count = 0 console_count = 0
if kwargs['arch'] == 'x86_64': if self.env['arch'] == 'x86_64':
console_type = 'ttyS' console_type = 'ttyS'
elif kwargs['is_arm']: elif self.env['is_arm']:
console_type = 'ttyAMA' console_type = 'ttyAMA'
console = '{}{}'.format(console_type, console_count) console = '{}{}'.format(console_type, console_count)
console_count += 1 console_count += 1
if not (kwargs['arch'] == 'x86_64' and kwargs['graphic']): if not (self.env['arch'] == 'x86_64' and self.env['graphic']):
kernel_cli += ' console={}'.format(console) kernel_cli += ' console={}'.format(console)
extra_console = '{}{}'.format(console_type, console_count) extra_console = '{}{}'.format(console_type, console_count)
console_count += 1 console_count += 1
if kwargs['kdb'] or kwargs['kgdb']: if self.env['kdb'] or self.env['kgdb']:
kernel_cli += ' kgdbwait' kernel_cli += ' kgdbwait'
if kwargs['kdb']: if self.env['kdb']:
if kwargs['graphic']: if self.env['graphic']:
kdb_cmd = 'kbd,' kdb_cmd = 'kbd,'
else: else:
kdb_cmd = '' kdb_cmd = ''
kernel_cli += ' kgdboc={}{},115200'.format(kdb_cmd, console) kernel_cli += ' kgdboc={}{},115200'.format(kdb_cmd, console)
if kwargs['kgdb']: if self.env['kgdb']:
kernel_cli += ' kgdboc={},115200'.format(extra_console) kernel_cli += ' kgdboc={},115200'.format(extra_console)
if kernel_cli_after_dash: if kernel_cli_after_dash:
kernel_cli += " -{}".format(kernel_cli_after_dash) kernel_cli += " -{}".format(kernel_cli_after_dash)
extra_env = {} extra_env = {}
if kwargs['trace'] is None: if self.env['trace'] is None:
do_trace = False do_trace = False
# A dummy value that is already turned on by default and does not produce large output, # A dummy value that is already turned on by default and does not produce large output,
# just to prevent QEMU from emitting a warning that '' is not valid. # just to prevent QEMU from emitting a warning that '' is not valid.
trace_type = 'load_file' trace_type = 'load_file'
else: else:
do_trace = True do_trace = True
trace_type = kwargs['trace'] trace_type = self.env['trace']
def raise_rootfs_not_found(): def raise_rootfs_not_found():
if not kwargs['dry_run']: if not self.env['dry_run']:
raise Exception('Root filesystem not found. Did you build it?\n' \ raise Exception('Root filesystem not found. Did you build it?\n' \
'Tried to use: ' + kwargs['disk_image']) 'Tried to use: ' + self.env['disk_image'])
def raise_image_not_found(): def raise_image_not_found():
if not kwargs['dry_run']: if not self.env['dry_run']:
raise Exception('Executable image not found. Did you build it?\n' \ raise Exception('Executable image not found. Did you build it?\n' \
'Tried to use: ' + kwargs['image']) 'Tried to use: ' + self.env['image'])
if kwargs['image'] is None: if self.env['image'] is None:
raise Exception('Baremetal ELF file not found. Tried:\n' + '\n'.join(paths)) raise Exception('Baremetal ELF file not found. Tried:\n' + '\n'.join(paths))
cmd = debug_vm.copy() cmd = debug_vm.copy()
if kwargs['emulator'] == 'gem5': if self.env['emulator'] == 'gem5':
if kwargs['baremetal'] is None: if self.env['baremetal'] is None:
if not os.path.exists(kwargs['rootfs_raw_file']): if not os.path.exists(self.env['rootfs_raw_file']):
if not os.path.exists(kwargs['qcow2_file']): if not os.path.exists(self.env['qcow2_file']):
raise_rootfs_not_found() raise_rootfs_not_found()
common.raw_to_qcow2(prebuilt=kwargs['prebuilt'], reverse=True) self.raw_to_qcow2(prebuilt=self.env['prebuilt'], reverse=True)
else: else:
if not os.path.exists(kwargs['gem5_fake_iso']): if not os.path.exists(self.env['gem5_fake_iso']):
os.makedirs(os.path.dirname(kwargs['gem5_fake_iso']), exist_ok=True) os.makedirs(os.path.dirname(self.env['gem5_fake_iso']), exist_ok=True)
common.write_string_to_file(kwargs['gem5_fake_iso'], 'a' * 512) self.write_string_to_file(self.env['gem5_fake_iso'], 'a' * 512)
if not os.path.exists(kwargs['image']): if not os.path.exists(self.env['image']):
# This is to run gem5 from a prebuilt download. # This is to run gem5 from a prebuilt download.
if (not kwargs['baremetal'] is None) or (not os.path.exists(kwargs['linux_image'])): if (not self.env['baremetal'] is None) or (not os.path.exists(self.env['linux_image'])):
raise_image_not_found() raise_image_not_found()
self.sh.run_cmd([os.path.join(kwargs['extract_vmlinux'], kwargs['linux_image'])]) self.sh.run_cmd([os.path.join(self.env['extract_vmlinux'], self.env['linux_image'])])
os.makedirs(os.path.dirname(kwargs['gem5_readfile']), exist_ok=True) os.makedirs(os.path.dirname(self.env['gem5_readfile']), exist_ok=True)
common.write_string_to_file(kwargs['gem5_readfile'], kwargs['gem5_readfile']) self.write_string_to_file(self.env['gem5_readfile'], self.env['gem5_readfile'])
memory = '{}B'.format(kwargs['memory']) memory = '{}B'.format(self.env['memory'])
gem5_exe_args = common.shlex_split(kwargs['gem5_exe_args']) gem5_exe_args = self.sh.shlex_split(self.env['gem5_exe_args'])
if do_trace: if do_trace:
gem5_exe_args.extend(['--debug-flags={}'.format(trace_type), LF]) gem5_exe_args.extend(['--debug-flags={}'.format(trace_type), LF])
extra_env['M5_PATH'] = kwargs['gem5_system_dir'] extra_env['M5_PATH'] = self.env['gem5_system_dir']
# https://stackoverflow.com/questions/52312070/how-to-modify-a-file-under-src-python-and-run-it-without-rebuilding-in-gem5/52312071#52312071 # https://stackoverflow.com/questions/52312070/how-to-modify-a-file-under-src-python-and-run-it-without-rebuilding-in-gem5/52312071#52312071
extra_env['M5_OVERRIDE_PY_SOURCE'] = 'true' extra_env['M5_OVERRIDE_PY_SOURCE'] = 'true'
if kwargs['trace_stdout']: if self.env['trace_stdout']:
debug_file = 'cout' debug_file = 'cout'
else: else:
debug_file = 'trace.txt' debug_file = 'trace.txt'
cmd.extend( cmd.extend(
[ [
kwargs['executable'], LF, self.env['executable'], LF,
'--debug-file', debug_file, LF, '--debug-file', debug_file, LF,
'--listener-mode', 'on', LF, '--listener-mode', 'on', LF,
'--outdir', kwargs['m5out_dir'], LF, '--outdir', self.env['m5out_dir'], LF,
] + ] +
gem5_exe_args gem5_exe_args
) )
if kwargs['userland'] is not None: if self.env['userland'] is not None:
cmd.extend([ cmd.extend([
kwargs['gem5_se_file'], LF, self.env['gem5_se_file'], LF,
'-c', common.resolve_userland(kwargs['userland']), LF, '-c', self.resolve_userland(self.env['userland']), LF,
]) ])
else: else:
if kwargs['gem5_script'] == 'fs': if self.env['gem5_script'] == 'fs':
# TODO port # TODO port
if kwargs['gem5_restore'] is not None: if self.env['gem5_restore'] is not None:
cpt_dirs = common.gem_list_checkpoint_dirs() cpt_dirs = self.gem_list_checkpoint_dirs()
cpt_dir = cpt_dirs[-kwargs['gem5_restore']] cpt_dir = cpt_dirs[-self.env['gem5_restore']]
extra_emulator_args.extend(['-r', str(sorted(cpt_dirs).index(cpt_dir) + 1)]) extra_emulator_args.extend(['-r', str(sorted(cpt_dirs).index(cpt_dir) + 1)])
cmd.extend([ cmd.extend([
kwargs['gem5_fs_file'], LF, self.env['gem5_fs_file'], LF,
'--disk-image', kwargs['disk_image'], LF, '--disk-image', self.env['disk_image'], LF,
'--kernel', kwargs['image'], LF, '--kernel', self.env['image'], LF,
'--mem-size', memory, LF, '--mem-size', memory, LF,
'--num-cpus', str(kwargs['cpus']), LF, '--num-cpus', str(self.env['cpus']), LF,
'--script', kwargs['gem5_readfile'], LF, '--script', self.env['gem5_readfile'], LF,
]) ])
if kwargs['arch'] == 'x86_64': if self.env['arch'] == 'x86_64':
if kwargs['kvm']: if self.env['kvm']:
cmd.extend(['--cpu-type', 'X86KvmCPU', LF]) cmd.extend(['--cpu-type', 'X86KvmCPU', LF])
cmd.extend(['--command-line', 'earlyprintk={} lpj=7999923 root=/dev/sda {}'.format(console, kernel_cli), LF]) cmd.extend(['--command-line', 'earlyprintk={} lpj=7999923 root=/dev/sda {}'.format(console, kernel_cli), LF])
elif kwargs['is_arm']: elif self.env['is_arm']:
if kwargs['kvm']: if self.env['kvm']:
cmd.extend(['--cpu-type', 'ArmV8KvmCPU', LF]) cmd.extend(['--cpu-type', 'ArmV8KvmCPU', LF])
cmd.extend([ cmd.extend([
# TODO why is it mandatory to pass mem= here? Not true for QEMU. # TODO why is it mandatory to pass mem= here? Not true for QEMU.
# Anything smaller than physical blows up as expected, but why can't it auto-detect the right value? # Anything smaller than physical blows up as expected, but why can't it auto-detect the right value?
'--command-line', 'earlyprintk=pl011,0x1c090000 lpj=19988480 rw loglevel=8 mem={} root=/dev/sda {}'.format(memory, kernel_cli), LF, '--command-line', 'earlyprintk=pl011,0x1c090000 lpj=19988480 rw loglevel=8 mem={} root=/dev/sda {}'.format(memory, kernel_cli), LF,
'--dtb-filename', os.path.join(kwargs['gem5_system_dir'], 'arm', 'dt', 'armv{}_gem5_v1_{}cpu.dtb'.format(kwargs['armv'], kwargs['cpus'])), LF, '--dtb-filename', os.path.join(self.env['gem5_system_dir'], 'arm', 'dt', 'armv{}_gem5_v1_{}cpu.dtb'.format(self.env['armv'], self.env['cpus'])), LF,
'--machine-type', kwargs['machine'], LF, '--machine-type', self.env['machine'], LF,
]) ])
if kwargs['baremetal'] is None: if self.env['baremetal'] is None:
cmd.extend([ cmd.extend([
'--param', 'system.panic_on_panic = True', LF]) '--param', 'system.panic_on_panic = True', LF])
else: else:
@@ -205,80 +371,80 @@ def main(args, extra_args=None):
'--bare-metal', LF, '--bare-metal', LF,
'--param', 'system.auto_reset_addr = True', LF, '--param', 'system.auto_reset_addr = True', LF,
]) ])
if kwargs['arch'] == 'aarch64': if self.env['arch'] == 'aarch64':
# https://stackoverflow.com/questions/43682311/uart-communication-in-gem5-with-arm-bare-metal/50983650#50983650 # https://stackoverflow.com/questions/43682311/uart-communication-in-gem5-with-arm-bare-metal/50983650#50983650
cmd.extend(['--param', 'system.highest_el_is_64 = True', LF]) cmd.extend(['--param', 'system.highest_el_is_64 = True', LF])
elif kwargs['gem5_script'] == 'biglittle': elif self.env['gem5_script'] == 'biglittle':
if kwargs['kvm']: if self.env['kvm']:
cpu_type = 'kvm' cpu_type = 'kvm'
else: else:
cpu_type = 'atomic' cpu_type = 'atomic'
if kwargs['gem5_restore'] is not None: if self.env['gem5_restore'] is not None:
cpt_dir = common.gem_list_checkpoint_dirs()[-kwargs['gem5_restore']] cpt_dir = self.gem_list_checkpoint_dirs()[-self.env['gem5_restore']]
extra_emulator_args.extend(['--restore-from', os.path.join(kwargs['m5out_dir'], cpt_dir)]) extra_emulator_args.extend(['--restore-from', os.path.join(self.env['m5out_dir'], cpt_dir)])
cmd.extend([ cmd.extend([
os.path.join(kwargs['gem5_source_dir'], 'configs', 'example', 'arm', 'fs_bigLITTLE.py'), LF, os.path.join(self.env['gem5_source_dir'], 'configs', 'example', 'arm', 'fs_bigLITTLE.py'), LF,
'--big-cpus', '2', LF, '--big-cpus', '2', LF,
'--cpu-type', cpu_type, LF, '--cpu-type', cpu_type, LF,
'--disk', kwargs['disk_image'], LF, '--disk', self.env['disk_image'], LF,
'--dtb', os.path.join(kwargs['gem5_system_dir'], 'arm', 'dt', 'armv8_gem5_v1_big_little_2_2.dtb'), LF, '--dtb', os.path.join(self.env['gem5_system_dir'], 'arm', 'dt', 'armv8_gem5_v1_big_little_2_2.dtb'), LF,
'--kernel', kwargs['image'], LF, '--kernel', self.env['image'], LF,
'--little-cpus', '2', LF, '--little-cpus', '2', LF,
]) ])
if kwargs['wait_gdb']: if self.env['wait_gdb']:
# https://stackoverflow.com/questions/49296092/how-to-make-gem5-wait-for-gdb-to-connect-to-reliably-break-at-start-kernel-of-th # https://stackoverflow.com/questions/49296092/how-to-make-gem5-wait-for-gdb-to-connect-to-reliably-break-at-start-kernel-of-th
cmd.extend(['--param', 'system.cpu[0].wait_for_remote_gdb = True', LF]) cmd.extend(['--param', 'system.cpu[0].wait_for_remote_gdb = True', LF])
else: else:
qemu_user_and_system_options = [ qemu_user_and_system_options = [
'-trace', 'enable={},file={}'.format(trace_type, kwargs['qemu_trace_file']), LF, '-trace', 'enable={},file={}'.format(trace_type, self.env['qemu_trace_file']), LF,
] ]
if kwargs['userland'] is not None: if self.env['userland'] is not None:
if kwargs['wait_gdb']: if self.env['wait_gdb']:
debug_args = ['-g', str(kwargs['gdb_port']), LF] debug_args = ['-g', str(self.env['gdb_port']), LF]
else: else:
debug_args = [] debug_args = []
cmd.extend( cmd.extend(
[ [
os.path.join(kwargs['qemu_build_dir'], '{}-linux-user'.format(kwargs['arch']), 'qemu-{}'.format(kwargs['arch'])), LF, os.path.join(self.env['qemu_build_dir'], '{}-linux-user'.format(self.env['arch']), 'qemu-{}'.format(self.env['arch'])), LF,
'-L', kwargs['target_dir'], LF '-L', self.env['target_dir'], LF
] + ] +
qemu_user_and_system_options + qemu_user_and_system_options +
common.shlex_split(kwargs['userland_before']) + self.sh.shlex_split(self.env['userland_before']) +
debug_args + debug_args +
[ [
common.resolve_userland(kwargs['userland']), LF self.resolve_userland(self.env['userland']), LF
] ]
) )
else: else:
if not os.path.exists(kwargs['image']): if not os.path.exists(self.env['image']):
raise_image_not_found() raise_image_not_found()
extra_emulator_args.extend(extra_qemu_args) extra_emulator_args.extend(extra_qemu_args)
common.make_run_dirs() self.make_run_dirs()
if kwargs['prebuilt'] or not os.path.exists(kwargs['qemu_executable']): if self.env['prebuilt'] or not os.path.exists(self.env['qemu_executable']):
qemu_executable = kwargs['qemu_executable_basename'] qemu_executable = self.env['qemu_executable_basename']
qemu_executable_prebuilt = True qemu_executable_prebuilt = True
else: else:
qemu_executable = kwargs['qemu_executable'] qemu_executable = self.env['qemu_executable']
qemu_executable_prebuilt = False qemu_executable_prebuilt = False
qemu_executable = shutil.which(qemu_executable) qemu_executable = shutil.which(qemu_executable)
if qemu_executable is None: if qemu_executable is None:
raise Exception('QEMU executable not found, did you forget to build or install it?\n' \ raise Exception('QEMU executable not found, did you forget to build or install it?\n' \
'Tried to use: ' + qemu_executable) 'Tried to use: ' + qemu_executable)
if kwargs['debug_vm']: if self.env['debug_vm']:
serial_monitor = [] serial_monitor = []
else: else:
if kwargs['background']: if self.env['background']:
serial_monitor = ['-serial', 'file:{}'.format(kwargs['qemu_background_serial_file']), LF] serial_monitor = ['-serial', 'file:{}'.format(self.env['qemu_background_serial_file']), LF]
else: else:
serial_monitor = ['-serial', 'mon:stdio', LF] serial_monitor = ['-serial', 'mon:stdio', LF]
if kwargs['kvm']: if self.env['kvm']:
extra_emulator_args.extend(['-enable-kvm', LF]) extra_emulator_args.extend(['-enable-kvm', LF])
extra_emulator_args.extend(['-serial', 'tcp::{},server,nowait'.format(kwargs['extra_serial_port']), LF]) extra_emulator_args.extend(['-serial', 'tcp::{},server,nowait'.format(self.env['extra_serial_port']), LF])
virtfs_data = [ virtfs_data = [
(kwargs['p9_dir'], 'host_data'), (self.env['p9_dir'], 'host_data'),
(kwargs['out_dir'], 'host_out'), (self.env['out_dir'], 'host_out'),
(kwargs['out_rootfs_overlay_dir'], 'host_out_rootfs_overlay'), (self.env['out_rootfs_overlay_dir'], 'host_out_rootfs_overlay'),
(kwargs['rootfs_overlay_dir'], 'host_rootfs_overlay'), (self.env['rootfs_overlay_dir'], 'host_rootfs_overlay'),
] ]
virtfs_cmd = [] virtfs_cmd = []
for virtfs_dir, virtfs_tag in virtfs_data: for virtfs_dir, virtfs_tag in virtfs_data:
@@ -293,13 +459,13 @@ def main(args, extra_args=None):
[ [
qemu_executable, LF, qemu_executable, LF,
'-device', 'rtl8139,netdev=net0', LF, '-device', 'rtl8139,netdev=net0', LF,
'-gdb', 'tcp::{}'.format(kwargs['gdb_port']), LF, '-gdb', 'tcp::{}'.format(self.env['gdb_port']), LF,
'-kernel', kwargs['image'], LF, '-kernel', self.env['image'], LF,
'-m', kwargs['memory'], LF, '-m', self.env['memory'], LF,
'-monitor', 'telnet::{},server,nowait'.format(kwargs['qemu_monitor_port']), LF, '-monitor', 'telnet::{},server,nowait'.format(self.env['qemu_monitor_port']), LF,
'-netdev', 'user,hostfwd=tcp::{}-:{},hostfwd=tcp::{}-:22,id=net0'.format(kwargs['qemu_hostfwd_generic_port'], kwargs['qemu_hostfwd_generic_port'], kwargs['qemu_hostfwd_ssh_port']), LF, '-netdev', 'user,hostfwd=tcp::{}-:{},hostfwd=tcp::{}-:22,id=net0'.format(self.env['qemu_hostfwd_generic_port'], self.env['qemu_hostfwd_generic_port'], self.env['qemu_hostfwd_ssh_port']), LF,
'-no-reboot', LF, '-no-reboot', LF,
'-smp', str(kwargs['cpus']), LF, '-smp', str(self.env['cpus']), LF,
] + ] +
virtfs_cmd + virtfs_cmd +
serial_monitor + serial_monitor +
@@ -307,9 +473,9 @@ def main(args, extra_args=None):
) )
if not qemu_executable_prebuilt: if not qemu_executable_prebuilt:
cmd.extend(qemu_user_and_system_options) cmd.extend(qemu_user_and_system_options)
if kwargs['initrd']: if self.env['initrd']:
extra_emulator_args.extend(['-initrd', os.path.join(kwargs['buildroot_images_dir'], 'rootfs.cpio')]) extra_emulator_args.extend(['-initrd', os.path.join(self.env['buildroot_images_dir'], 'rootfs.cpio')])
rr = kwargs['record'] or kwargs['replay'] rr = self.env['record'] or self.env['replay']
if ramfs: if ramfs:
# TODO why is this needed, and why any string works. # TODO why is this needed, and why any string works.
root = 'root=/dev/anything' root = 'root=/dev/anything'
@@ -324,14 +490,14 @@ def main(args, extra_args=None):
root = 'root=/dev/vda' root = 'root=/dev/vda'
rrid = '' rrid = ''
snapshot = ',snapshot' snapshot = ',snapshot'
if kwargs['baremetal'] is None: if self.env['baremetal'] is None:
if not os.path.exists(kwargs['qcow2_file']): if not os.path.exists(self.env['qcow2_file']):
if not os.path.exists(kwargs['rootfs_raw_file']): if not os.path.exists(self.env['rootfs_raw_file']):
raise_rootfs_not_found() raise_rootfs_not_found()
common.raw_to_qcow2(prebuilt=kwargs['prebuilt']) self.raw_to_qcow2(prebuilt=self.env['prebuilt'])
extra_emulator_args.extend([ extra_emulator_args.extend([
'-drive', '-drive',
'file={},format=qcow2,if={}{}{}'.format(kwargs['disk_image'], driveif, snapshot, rrid), 'file={},format=qcow2,if={}{}{}'.format(self.env['disk_image'], driveif, snapshot, rrid),
LF, LF,
]) ])
if rr: if rr:
@@ -342,20 +508,20 @@ def main(args, extra_args=None):
if rr: if rr:
extra_emulator_args.extend([ extra_emulator_args.extend([
'-object', 'filter-replay,id=replay,netdev=net0', '-object', 'filter-replay,id=replay,netdev=net0',
'-icount', 'shift=7,rr={},rrfile={}'.format('record' if kwargs['record'] else 'replay', kwargs['qemu_rrfile']), '-icount', 'shift=7,rr={},rrfile={}'.format('record' if self.env['record'] else 'replay', self.env['qemu_rrfile']),
]) ])
virtio_gpu_pci = [] virtio_gpu_pci = []
else: else:
virtio_gpu_pci = ['-device', 'virtio-gpu-pci', LF] virtio_gpu_pci = ['-device', 'virtio-gpu-pci', LF]
if kwargs['arch'] == 'x86_64': if self.env['arch'] == 'x86_64':
append = ['-append', '{} nopat {}'.format(root, kernel_cli), LF] append = ['-append', '{} nopat {}'.format(root, kernel_cli), LF]
cmd.extend([ cmd.extend([
'-M', kwargs['machine'], LF, '-M', self.env['machine'], LF,
'-device', 'edu', LF, '-device', 'edu', LF,
]) ])
elif kwargs['is_arm']: elif self.env['is_arm']:
extra_emulator_args.extend(['-semihosting', LF]) extra_emulator_args.extend(['-semihosting', LF])
if kwargs['arch'] == 'arm': if self.env['arch'] == 'arm':
cpu = 'cortex-a15' cpu = 'cortex-a15'
else: else:
cpu = 'cortex-a57' cpu = 'cortex-a57'
@@ -364,46 +530,46 @@ def main(args, extra_args=None):
[ [
# highmem=off needed since v3.0.0 due to: # highmem=off needed since v3.0.0 due to:
# http://lists.nongnu.org/archive/html/qemu-discuss/2018-08/msg00034.html # http://lists.nongnu.org/archive/html/qemu-discuss/2018-08/msg00034.html
'-M', '{},highmem=off'.format(kwargs['machine']), LF, '-M', '{},highmem=off'.format(self.env['machine']), LF,
'-cpu', cpu, LF, '-cpu', cpu, LF,
] + ] +
virtio_gpu_pci virtio_gpu_pci
) )
if kwargs['baremetal'] is None: if self.env['baremetal'] is None:
cmd.extend(append) cmd.extend(append)
if kwargs['tmux'] is not None: if self.env['tmux']:
tmux_args = '--run-id {}'.format(kwargs['run_id']) tmux_args = '--run-id {}'.format(self.env['run_id'])
if kwargs['emulator'] == 'gem5': if self.env['emulator'] == 'gem5':
tmux_cmd = './gem5-shell' tmux_cmd = './gem5-shell'
elif kwargs['wait_gdb']: else:
tmux_cmd = './run-gdb' tmux_cmd = './run-gdb'
# TODO find a nicer way to forward all those args automatically. # TODO find a nicer way to forward all those args automatically.
# Part of me wants to: https://github.com/jonathanslenders/pymux # Part of me wants to: https://github.com/jonathanslenders/pymux
# but it cannot be used as a library properly it seems, and it is # but it cannot be used as a library properly it seems, and it is
# slower than tmux. # slower than tmux.
tmux_args += " --arch {} --linux-build-id '{}' --run-id '{}'".format( tmux_args += " --arch {} --linux-build-id '{}' --run-id '{}'".format(
kwargs['arch'], self.env['arch'],
kwargs['linux_build_id'], self.env['linux_build_id'],
kwargs['run_id'], self.env['run_id'],
) )
if kwargs['baremetal']: if self.env['baremetal']:
tmux_args += " --baremetal '{}'".format(kwargs['baremetal']) tmux_args += " --baremetal '{}'".format(self.env['baremetal'])
if kwargs['userland']: if self.env['userland']:
tmux_args += " --userland '{}'".format(kwargs['userland']) tmux_args += " --userland '{}'".format(self.env['userland'])
tmux_args += ' {}'.format(kwargs['tmux']) tmux_args += ' {}'.format(self.env['tmux'])
subprocess.Popen([ subprocess.Popen([
os.path.join(kwargs['root_dir'], 'tmu'), os.path.join(self.env['root_dir'], 'tmu'),
"sleep 2;{} {}".format(tmux_cmd, tmux_args) "sleep 2;{} {}".format(tmux_cmd, tmux_args)
]) ])
cmd.extend(extra_emulator_args) cmd.extend(extra_emulator_args)
cmd.extend(kwargs['extra_emulator_args']) cmd.extend(self.env['extra_emulator_args'])
if debug_vm or kwargs['terminal']: if debug_vm or self.env['terminal']:
out_file = None out_file = None
else: else:
out_file = kwargs['termout_file'] out_file = self.env['termout_file']
self.sh.run_cmd(cmd, cmd_file=kwargs['run_cmd_file'], out_file=out_file, extra_env=extra_env) self.sh.run_cmd(cmd, cmd_file=self.env['run_cmd_file'], out_file=out_file, extra_env=extra_env)
# Check if guest panicked. # Check if guest panicked.
if kwargs['emulator'] == 'gem5': if self.env['emulator'] == 'gem5':
# We have to do some parsing here because gem5 exits with status 0 even when panic happens. # We have to do some parsing here because gem5 exits with status 0 even when panic happens.
# Grepping for '^panic: ' does not work because some errors don't show that message. # Grepping for '^panic: ' does not work because some errors don't show that message.
panic_msg = b'--- BEGIN LIBC BACKTRACE ---$' panic_msg = b'--- BEGIN LIBC BACKTRACE ---$'
@@ -411,217 +577,22 @@ def main(args, extra_args=None):
panic_msg = b'Kernel panic - not syncing' panic_msg = b'Kernel panic - not syncing'
panic_re = re.compile(panic_msg) panic_re = re.compile(panic_msg)
error_string_found = False error_string_found = False
if out_file is not None and not kwargs['dry_run']: if out_file is not None and not self.env['dry_run']:
with open(kwargs['termout_file'], 'br') as logfile: with open(self.env['termout_file'], 'br') as logfile:
for line in logfile: for line in logfile:
if panic_re.search(line): if panic_re.search(line):
error_string_found = True error_string_found = True
if os.path.exists(kwargs['guest_terminal_file']): if os.path.exists(self.env['guest_terminal_file']):
with open(kwargs['guest_terminal_file'], 'br') as logfile: with open(self.env['guest_terminal_file'], 'br') as logfile:
lines = logfile.readlines() lines = logfile.readlines()
if lines: if lines:
last_line = lines[-1] last_line = lines[-1]
if last_line.rstrip() == kwargs['magic_fail_string']: if last_line.rstrip() == self.env['magic_fail_string']:
error_string_found = True error_string_found = True
if error_string_found: if error_string_found:
common.log_error('simulation error detected by parsing logs') self.log_error('simulation error detected by parsing logs')
return 1 return 1
return 0 return 0
def get_argparse():
parser = common.get_argparse(argparse_args={'description':'Run Linux on an emulator'})
init_group = parser.add_mutually_exclusive_group()
kvm_group = parser.add_mutually_exclusive_group()
parser.add_argument(
'--background', default=defaults['background'], action='store_true',
help='''\
Send QEMU output to a file instead of the terminal so it does not require a
terminal attached to run on the background. Interactive input cannot be given.
TODO: use a port instead. If only there was a way to redirect a serial to multiple
places, both to a port and a file? We use the file currently to be able to have
any output at all.
https://superuser.com/questions/1373226/how-to-redirect-qemu-serial-output-to-both-a-file-and-the-terminal-or-a-port
'''
)
parser.add_argument(
'-c', '--cpus', default=defaults['cpus'], type=int,
help='Number of guest CPUs to emulate. Default: %(default)s'
)
parser.add_argument(
'-D', '--debug-vm', default=defaults['debug_vm'], nargs='?', action='store', const='',
help='Run GDB on the emulator itself.'
)
parser.add_argument(
'-E', '--eval',
help='''\
Replace the normal init with a minimal init that just evals the given string.
See: https://github.com/cirosantilli/linux-kernel-module-cheat#replace-init
'''
)
parser.add_argument(
'-e', '--kernel-cli',
help='''\
Pass an extra Linux kernel command line options, and place them before
the dash separator `-`. Only options that come before the `-`, i.e.
"standard" options, should be passed with this option.
Example: `./run -a arm -e 'init=/poweroff.out'`
'''
)
parser.add_argument(
'-F', '--eval-after',
help='''\
Pass a base64 encoded command line parameter that gets evalled at the end of
the normal init.
See: https://github.com/cirosantilli/linux-kernel-module-cheat#init-busybox
'''
)
parser.add_argument(
'-f', '--kernel-cli-after-dash',
help='''\
Pass an extra Linux kernel command line options, add a dash `-`
separator, and place the options after the dash. Intended for custom
options understood by our `init` scripts, most of which are prefixed
by `lkmc_`.
Example: `./run --kernel-cli-after-dash 'lkmc_eval="wget google.com" lkmc_lala=y'`
Mnenomic: `-f` comes after `-e`.
'''
)
parser.add_argument(
'-G', '--gem5-exe-args', default=defaults['gem5_exe_args'],
help='''\
Pass extra options to the gem5 executable.
Do not confuse with the arguments passed to config scripts,
like `fs.py`. Example:
./run -G '--debug-flags=Exec --debug' --gem5 -- --cpu-type=HPI --caches
will run:
gem.op5 --debug-flags=Exec fs.py --cpu-type=HPI --caches
'''
)
parser.add_argument(
'--gem5-script', default=defaults['gem5_script'], choices=['fs', 'biglittle'],
help='Which gem5 script to use'
)
parser.add_argument(
'--gem5-readfile', default=defaults['gem5_readfile'],
help='Set the contents of m5 readfile to this string.'
)
kvm_group.add_argument(
'-K', '--kvm', default=defaults['kvm'], action='store_true',
help='Use KVM. Only works if guest arch == host arch'
)
parser.add_argument(
'--kgdb', default=defaults['kgdb'], action='store_true'
)
parser.add_argument(
'--kdb', default=defaults['kdb'], action='store_true'
)
parser.add_argument(
'-l', '--gem5-restore', type=int,
help='''\
Restore the nth most recently taken gem5 checkpoint according to directory
timestamps.
'''
)
parser.add_argument(
'-m', '--memory', default=defaults['memory'],
help='''\
Set the memory size of the guest. E.g.: `-m 512M`. We try to keep the default
at the minimal ammount amount that boots all archs. Anything lower could lead
some arch to fail to boot.
Default: %(default)s
'''
)
group = parser.add_mutually_exclusive_group()
group.add_argument(
'-R', '--replay', default=defaults['replay'], action='store_true',
help='Replay a QEMU run record deterministically'
)
group.add_argument(
'-r', '--record', default=defaults['record'], action='store_true',
help='Record a QEMU run record for later replay with `-R`'
)
parser.add_argument(
'-T', '--trace',
help='''\
Set trace events to be enabled. If not given, gem5 tracing is completely
disabled, while QEMU tracing is enabled but uses default traces that are very
rare and don't affect performance, because `./configure
--enable-trace-backends=simple` seems to enable some traces by default, e.g.
`pr_manager_run`, and I don't know how to get rid of them.
'''
)
parser.add_argument(
'--trace-stdout', default=defaults['trace_stdout'], action='store_true',
help='''\
Output trace to stdout instead of a file. Only works for gem5 currently.
'''
)
init_group.add_argument(
'--terminal', default=defaults['terminal'], action='store_true',
help='''\
Output to the terminal, don't pipe to tee as the default.
Does not save the output to a file, but allows you to use debuggers.
Set automatically by --debug-vm, but you still need this option to debug
gem5 Python scripts.
'''
)
parser.add_argument(
'-t', '--tmux', default=defaults['tmux'], nargs='?', action='store', const='',
help='''\
Create a tmux split the window. You must already be inside of a `tmux` session
to use this option:
* on the main window, run the emulator as usual
* on the split:
** if on QEMU and `-d` is given, GDB
** if on gem5, the gem5 terminal
If values are given to this option, pass those as parameters
to the program running on the split.
'''
)
parser.add_argument(
'-u', '--userland', default=defaults['userland'],
help='''\
Run the given userland executable in user mode instead of booting the Linux kernel
in full system mode. In gem5, user mode is called Syscall Emulation (SE) mode and
uses se.py.
Path resolution is similar to --baremetal.
'''
)
parser.add_argument(
'--userland-before', default=defaults['userland_before'],
help='''\
Pass these arguments to the QEMU user mode CLI before the program to execute.
This is required with --userland since arguments that come at the end are interpreted
as command line arguments to that executable.
'''
)
kvm_group.add_argument(
'-w', '--wait-gdb', default=defaults['wait_gdb'], action='store_true',
help='Wait for GDB to connect before starting execution'
)
parser.add_argument(
'-x', '--graphic', default=defaults['graphic'], action='store_true',
help='Run in graphic mode. Mnemonic: X11'
)
parser.add_argument(
'-V', '--vnc', default=defaults['vnc'], action='store_true',
help='''\
Run QEMU with VNC instead of the default SDL. Connect to it with:
`vinagre localhost:5900`.
'''
)
parser.add_argument(
'extra_emulator_args', nargs='*', default=defaults['extra_emulator_args'],
help='Extra options to append at the end of the emulator command line'
)
return parser
if __name__ == '__main__': if __name__ == '__main__':
parser = get_argparse() Main().cli()
args = common.setup(parser)
start_time = time.time()
exit_status = main(args)
end_time = time.time()
common.print_time(end_time - start_time)
sys.exit(exit_status)

4
run-docker
View File

@@ -58,7 +58,7 @@ cmd_action_map = {
parser = argparse.ArgumentParser() parser = argparse.ArgumentParser()
parser.add_argument('cmd', choices=cmd_action_map) parser.add_argument('cmd', choices=cmd_action_map)
parser.add_argument('args', nargs='*') parser.add_argument('args', nargs='*')
common.add_dry_run_argument(parser) self.add_dry_run_argument(parser)
args = parser.parse_args() args = parser.parse_args()
common.setup_dry_run_arguments(args) self.setup_dry_run_arguments(args)
cmd_action_map[kwargs['cmd']](kwargs['args']) cmd_action_map[kwargs['cmd']](kwargs['args'])

28
run-gdb
View File

@@ -34,8 +34,8 @@ class GdbTestcase:
''' '''
self.prompt = '\(gdb\) ' self.prompt = '\(gdb\) '
self.source_path = source_path self.source_path = source_path
common.print_cmd(cmd) self.print_cmd(cmd)
cmd = common.strip_newlines(cmd) cmd = self.strip_newlines(cmd)
import pexpect import pexpect
self.child = pexpect.spawn( self.child = pexpect.spawn(
cmd[0], cmd[0],
@@ -97,9 +97,9 @@ def main(args, extra_args=None):
:rtype: int :rtype: int
''' '''
global defaults global defaults
args = common.resolve_args(defaults, args, extra_args) args = self.resolve_args(defaults, args, extra_args)
after = common.shlex_split(kwargs['after']) after = self.sh.shlex_split(kwargs['after'])
before = common.shlex_split(kwargs['before']) before = self.sh.shlex_split(kwargs['before'])
no_continue = kwargs['no_continue'] no_continue = kwargs['no_continue']
if kwargs['test']: if kwargs['test']:
no_continue = True no_continue = True
@@ -122,7 +122,7 @@ def main(args, extra_args=None):
break_at = [] break_at = []
linux_full_system = (kwargs['baremetal'] is None and kwargs['userland'] is None) linux_full_system = (kwargs['baremetal'] is None and kwargs['userland'] is None)
if kwargs['userland']: if kwargs['userland']:
image = common.resolve_userland(kwargs['userland']) image = self.resolve_userland(kwargs['userland'])
elif kwargs['baremetal']: elif kwargs['baremetal']:
image = kwargs['image'] image = kwargs['image']
test_script_path = os.path.splitext(kwargs['source_path'])[0] + '.py' test_script_path = os.path.splitext(kwargs['source_path'])[0] + '.py'
@@ -133,7 +133,7 @@ def main(args, extra_args=None):
else: else:
allowed_toolchains = ['buildroot', 'crosstool-ng', 'host'] allowed_toolchains = ['buildroot', 'crosstool-ng', 'host']
cmd = ( cmd = (
[common.get_toolchain_tool('gdb', allowed_toolchains=allowed_toolchains), LF] + [self.get_toolchain_tool('gdb', allowed_toolchains=allowed_toolchains), LF] +
before + before +
['-q', LF] ['-q', LF]
) )
@@ -195,7 +195,7 @@ def main(args, extra_args=None):
) )
if __name__ == '__main__': if __name__ == '__main__':
parser = common.get_argparse(argparse_args={'description': 'Connect with GDB to an emulator to debug Linux itself'}) parser = self.get_argparse(argparse_args={'description': 'Connect with GDB to an emulator to debug Linux itself'})
parser.add_argument( parser.add_argument(
'-A', '--after', default=defaults['after'], '-A', '--after', default=defaults['after'],
help='Pass extra arguments to GDB, to be appended after all other arguments' help='Pass extra arguments to GDB, to be appended after all other arguments'
@@ -205,23 +205,23 @@ if __name__ == '__main__':
help='Pass extra arguments to GDB to be prepended before any of the arguments passed by this script' help='Pass extra arguments to GDB to be prepended before any of the arguments passed by this script'
) )
parser.add_argument( parser.add_argument(
'-C', '--no-continue', default=defaults['no_continue'], action='store_true', '-C', '--no-continue', default=defaults['no_continue'],
help="Don't run continue after connecting" help="Don't run continue after connecting"
) )
parser.add_argument( parser.add_argument(
'-k', '--kgdb', default=defaults['kgdb'], action='store_true' '-k', '--kgdb', default=defaults['kgdb'],
) )
parser.add_argument( parser.add_argument(
'--sim', default=defaults['sim'], action='store_true', '--sim', default=defaults['sim'],
help='''Use the built-in GDB CPU simulator help='''Use the built-in GDB CPU simulator
See: https://github.com/cirosantilli/linux-kernel-module-cheat#gdb-builtin-cpu-simulator See: https://github.com/cirosantilli/linux-kernel-module-cheat#gdb-builtin-cpu-simulator
''' '''
) )
parser.add_argument( parser.add_argument(
'-X', '--no-lxsymbols', default=defaults['no_lxsymbols'], action='store_true' '-X', '--no-lxsymbols', default=defaults['no_lxsymbols'],
) )
parser.add_argument( parser.add_argument(
'--test', default=defaults['test'], action='store_true', '--test', default=defaults['test'],
help='''\ help='''\
Run an expect test case instead of interactive usage. For baremetal and userland, Run an expect test case instead of interactive usage. For baremetal and userland,
the script is a .py file next to the source code. the script is a .py file next to the source code.
@@ -234,5 +234,5 @@ the script is a .py file next to the source code.
'break_at', nargs='?', 'break_at', nargs='?',
help='Extra options to append at the end of the emulator command line' help='Extra options to append at the end of the emulator command line'
) )
args = common.setup(parser) args = self.setup(parser)
sys.exit(main(args)) sys.exit(main(args))

8
run-gdb-user
View File

@@ -7,7 +7,7 @@ import sys
import common import common
rungdb = imp.load_source('rungdb', os.path.join(kwargs['root_dir'], 'run-gdb')) rungdb = imp.load_source('rungdb', os.path.join(kwargs['root_dir'], 'run-gdb'))
parser = common.get_argparse(argparse_args={ parser = self.get_argparse(argparse_args={
'description': '''GDB step debug guest userland processes without gdbserver. 'description': '''GDB step debug guest userland processes without gdbserver.
More information at: https://github.com/cirosantilli/linux-kernel-module-cheat#gdb-step-debug-userland-processes More information at: https://github.com/cirosantilli/linux-kernel-module-cheat#gdb-step-debug-userland-processes
@@ -23,9 +23,9 @@ parser.add_argument(
help='Break at this point, e.g. main.', help='Break at this point, e.g. main.',
nargs='?' nargs='?'
) )
args = common.setup(parser) args = self.setup(parser)
executable = common.resolve_userland(kwargs['executable']) executable = self.resolve_userland(kwargs['executable'])
addr = common.get_elf_entry(os.path.join(kwargs['buildroot_build_build_dir'], executable)) addr = self.get_elf_entry(os.path.join(kwargs['buildroot_build_build_dir'], executable))
extra_args = {} extra_args = {}
extra_args['before'] = '-ex \"add-symbol-file {} {}\"'.format(executable, hex(addr)) extra_args['before'] = '-ex \"add-symbol-file {} {}\"'.format(executable, hex(addr))
# Or else lx-symbols throws for arm: # Or else lx-symbols throws for arm:

8
run-gdbserver
View File

@@ -6,7 +6,7 @@ import sys
import common import common
parser = common.get_argparse(argparse_args={ parser = self.get_argparse(argparse_args={
'description':'Connect to gdbserver running on the guest.' 'description':'Connect to gdbserver running on the guest.'
}) })
parser.add_argument( parser.add_argument(
@@ -16,13 +16,13 @@ parser.add_argument(
parser.add_argument( parser.add_argument(
'break_at', default='main', nargs='?' 'break_at', default='main', nargs='?'
) )
args = common.setup(parser) args = self.setup(parser)
sys.exit(subprocess.Popen([ sys.exit(subprocess.Popen([
common.get_toolchain_tool('gdb'), self.get_toolchain_tool('gdb'),
'-q', '-q',
'-ex', 'set sysroot {}'.format(kwargs['buildroot_staging_dir']), '-ex', 'set sysroot {}'.format(kwargs['buildroot_staging_dir']),
'-ex', 'target remote localhost:{}'.format(kwargs['qemu_hostfwd_generic_port']), '-ex', 'target remote localhost:{}'.format(kwargs['qemu_hostfwd_generic_port']),
'-ex', 'tbreak {}'.format(kwargs['break_at']), '-ex', 'tbreak {}'.format(kwargs['break_at']),
'-ex', 'continue', '-ex', 'continue',
os.path.join(kwargs['buildroot_build_build_dir'], common.resolve_userland(kwargs['executable'])), os.path.join(kwargs['buildroot_build_build_dir'], self.resolve_userland(kwargs['executable'])),
]).wait()) ]).wait())

7
run-toolchain
View File

@@ -5,7 +5,7 @@ import sys
import common import common
parser = common.get_argparse(argparse_args={ parser = self.get_argparse(argparse_args={
'description': '''Run a Buildroot ToolChain tool like readelf or objdump. 'description': '''Run a Buildroot ToolChain tool like readelf or objdump.
For example, to get some information about the arm vmlinux: For example, to get some information about the arm vmlinux:
@@ -24,7 +24,6 @@ ls "$(./getvar -a arm host_bin_dir)"
parser.add_argument( parser.add_argument(
'--dry', '--dry',
help='Just output the tool path to stdout but actually run it', help='Just output the tool path to stdout but actually run it',
action='store_true',
) )
parser.add_argument('tool', help='Which tool to run.') parser.add_argument('tool', help='Which tool to run.')
parser.add_argument( parser.add_argument(
@@ -34,12 +33,12 @@ parser.add_argument(
metavar='extra-args', metavar='extra-args',
nargs='*' nargs='*'
) )
args = common.setup(parser) args = self.setup(parser)
if kwargs['baremetal'] is None: if kwargs['baremetal'] is None:
image = kwargs['vmlinux'] image = kwargs['vmlinux']
else: else:
image = kwargs['image'] image = kwargs['image']
tool= common.get_toolchain_tool(kwargs['tool']) tool= self.get_toolchain_tool(kwargs['tool'])
if kwargs['dry']: if kwargs['dry']:
print(tool) print(tool)
else: else:

2
shell_helpers.py
View File

@@ -6,7 +6,9 @@ import os
import shlex import shlex
import shutil import shutil
import signal import signal
import stat
import subprocess import subprocess
import sys
class LF: class LF:
''' '''

6
trace-boot
View File

@@ -9,7 +9,7 @@ import common
run = imp.load_source('run', os.path.join(kwargs['root_dir'], 'run')) run = imp.load_source('run', os.path.join(kwargs['root_dir'], 'run'))
qemu_trace2txt = imp.load_source('qemu_trace2txt', os.path.join(kwargs['root_dir'], 'qemu-trace2txt')) qemu_trace2txt = imp.load_source('qemu_trace2txt', os.path.join(kwargs['root_dir'], 'qemu-trace2txt'))
parser = common.get_argparse(argparse_args={ parser = self.get_argparse(argparse_args={
'description': '''Trace the PIC addresses executed on a Linux kernel boot. 'description': '''Trace the PIC addresses executed on a Linux kernel boot.
More information at: https://github.com/cirosantilli/linux-kernel-module-cheat#tracing More information at: https://github.com/cirosantilli/linux-kernel-module-cheat#tracing
@@ -19,7 +19,7 @@ parser.add_argument(
'extra_emulator_args', nargs='*', 'extra_emulator_args', nargs='*',
help='Extra options to append at the end of the emulator command line' help='Extra options to append at the end of the emulator command line'
) )
args = common.setup(parser) args = self.setup(parser)
extra_args = { extra_args = {
'extra_emulator_args': kwargs['extra_emulator_args'], 'extra_emulator_args': kwargs['extra_emulator_args'],
} }
@@ -39,7 +39,7 @@ else:
# Instruction count. # Instruction count.
# We could put this on a separate script, but it just adds more arch boilerplate to a new script. # We could put this on a separate script, but it just adds more arch boilerplate to a new script.
# So let's just leave it here for now since it did not add a significant processing time. # So let's just leave it here for now since it did not add a significant processing time.
kernel_entry_addr = hex(common.get_elf_entry(kwargs['vmlinux'])) kernel_entry_addr = hex(self.get_elf_entry(kwargs['vmlinux']))
nlines = 0 nlines = 0
nlines_firmware = 0 nlines_firmware = 0
with open(kwargs['qemu_trace_txt_file'], 'r') as trace_file: with open(kwargs['qemu_trace_txt_file'], 'r') as trace_file:

8
trace2line
View File

@@ -14,15 +14,15 @@ import sys
import common import common
parser = common.get_argparse(argparse_args={ parser = self.get_argparse(argparse_args={
'description': 'Convert an execution trace containing PC values into the Linux kernel linex executed' 'description': 'Convert an execution trace containing PC values into the Linux kernel linex executed'
}) })
args = common.setup(parser) args = self.setup(parser)
sys.exit(subprocess.Popen([ sys.exit(subprocess.Popen([
os.path.join(kwargs['root_dir'], 'trace2line.sh'), os.path.join(kwargs['root_dir'], 'trace2line.sh'),
'true' if kwargs['emulator'] == 'gem5' else 'false', 'true' if kwargs['emulator'] == 'gem5' else 'false',
kwargs['trace_txt_file'], kwargs['trace_txt_file'],
common.get_toolchain_tool('addr2line'), self.get_toolchain_tool('addr2line'),
kwargs['vmlinux'], kwargs['vmlinux'],
kwargs['run_dir'], kwargs['run_dir'],
]).wait()) ]).wait())
@@ -40,7 +40,7 @@ sys.exit(subprocess.Popen([
# with \ # with \
# subprocess.Popen( # subprocess.Popen(
# [ # [
# common.get_toolchain_tool('addr2line'), # self.get_toolchain_tool('addr2line'),
# '-e', # '-e',
# kwargs['vmlinux'], # kwargs['vmlinux'],
# '-f', # '-f',