3rd-party/linux-kernel-module-cheat
1
0
Fork 0
mirror of https://github.com/cirosantilli/linux-kernel-module-cheat.git synced 2026-01-22 17:55:57 +01:00
Code Issues Releases Wiki Activity

Build the Linux kernel independently from Buildroot

Browse Source 
This will allow for other types of root filesystems that don't rely on Buildroot
to be added and used in the future.

Propagate --verbose on all build scripts to see full GCC commands.

build-all: allow for neat subsets

also 9p share rootfs_overlay. TODO document.
This commit is contained in:
Ciro Santilli 六四事件 法轮功
2018-09-28 08:31:38 +01:00
parent e8cd0caa9e
commit bc73cebff1
27 changed files with 942 additions and 214 deletions
Download Patch File Download Diff File Expand all files Collapse all files

97
README.adoc
View File

@@ -75,6 +75,8 @@ cd linux-kernel-module-cheat
./configure --qemu && \
./build-qemu &&
./build-buildroot && \
./build-linux && \
./build-modules && \
./run && \
:;
....
@@ -98,7 +100,7 @@ see this: https://askubuntu.com/questions/496549/error-you-must-put-some-source-
It does not work if you just download the `.zip` from GitHub because we use link:.gitmodules[Git submodules], you must clone this repo. `./configure` then fetches only the required submodules for you.
QEMU opens up and you can start playing with the kernel modules inside the simulated system:
QEMU opens up and you can start playing with the kernel modules inside the simulated system: TODO fix path to 9p:
....
insmod /hello.ko
@@ -210,10 +212,12 @@ pr_info("hello init hacked\n");
and then rebuild the kernel modules and re-run to see it take effect:
....
./build-buildroot --kernel-modules
./build-modules
./run --eval-busybox 'insmod /hello.ko'
....
TODO mention 9p
Congratulations, you are now officially a kernel module hacker!
We use `./build-buildroot` because the kernel modules go inside the root filesystem, and it is Buildroot that generates and updates our root filesystem. The kernel modules are link:packages/kernel_modules[inside a Buildroot package].
@@ -245,7 +249,7 @@ pr_info("I'VE HACKED THE LINUX KERNEL!!!");
Then rebuild the Linux kernel and reboot:
....
./build-buildroot && ./run
./build-linux && ./run
....
and, surely enough, your message has appeared at the beginning of the boot.
@@ -321,6 +325,8 @@ All of this makes QEMU the natural choice of default system simulator.
=== gem5 Buildroot setup
TODO document kernel modules don't work on 9p
==== About the gem5 Buildroot setup
This setup is like the <<qemu-buildroot-setup>>, but it uses link:http://gem5.org/[gem5] instead of QEMU as a system simulator.
@@ -535,7 +541,6 @@ The limitations are severe however:
* can't <<gdb,GDB step debug the kernel>>, since the source and cross toolchain with GDB are not available. Buildroot cannot easily use a host toolchain: <<prebuilt-toolchain>>.
+
Maybe we could work around this by just downloading the kernel source somehow, and using a host prebuilt GDB, but we felt that it would be too messy and unreliable.
* can't create new modules or modify the existing ones, since no cross toolchain
* you won't get the latest version of this repository. Our <<travis>> attempt to automate builds failed, and storing a release for every commit would likely make GitHub mad at us.
* <<gem5>> is not currently supported, although it should not be too hard to do. Annoyances:
+
@@ -583,6 +588,23 @@ then do whatever that checked out README says.
If you are curious to see what the releases contain in detail, have a look at our <<release,release procedure>>.
To build the kernel modules, simply do:
....
./build-linux -- modules_prepare
./build-modules
./run
....
`modules_prepare` does the minimal build procedure required on the kernel for us to be able to compile the kernel modules, and is way faster than doing a full kernel build. A full kernel build would also work however.
To modify the Linux kernel, build and use it as usual:
....
./build-linux
./run
....
////
For gem5, do:
@@ -601,11 +623,11 @@ The Linux kernel is required for `extract-vmlinux` to convert the compressed ker
[[host]]
=== Host kernel module setup
**THIS IS DANGEROUS, YOU HAVE BEEN WARNED**
**THIS IS DANGEROUS (AND FUN), YOU HAVE BEEN WARNED**
This method runs the kernel modules directly on your host computer without a VM, and saves you the compilation time and disk usage of the virtual machine method.
It has however severe limitations, and you will soon see that the compilation time and disk usage are well worth it:
It has however severe limitations:
* can't control which kernel version and build options to use. So some of the modules will likely not compile because of kernel API changes, since https://stackoverflow.com/questions/37098482/how-to-build-a-linux-kernel-module-so-that-it-is-compatible-with-all-kernel-rele/45429681#45429681[the Linux kernel does not have a stable kernel module API].
* bugs can easily break you system. E.g.:
@@ -618,20 +640,34 @@ It has however severe limitations, and you will soon see that the compilation ti
Still interested?
....
cd packages/kernel_modules
./make-host.sh
./build-modules --host
....
If the compilation of any of the C files fails because of kernel or toolchain differences that we don't control on the host, just rename it to remove the `.c` extension and try again:
Compilation will likely fail for some modules because of kernel or toolchain differences that we can't control on the host.
The best solution is to compile just your modules with:
....
./build-modules --host -- hello hello2
....
which is equivalent to:
....
./build-modules --host -- packages/kernel/modules/hello.c packages/kernel/modules/hello2.c
....
Or just remove the `.c` extension from the failing files and try again:
....
cd "$(./getvar kernel_modules_src_dir)"
mv broken.c broken.c~
./build_host
....
Once you manage to compile, and have come to terms with the fact that this may blow up your host, try it out with:
....
cd "$(./getvar kernel_modules_build_host_dir)"
sudo insmod hello.ko
# Our module is there.
@@ -649,15 +685,6 @@ dmesg -T
sudo lsmod | grep hello
....
Once you are done with this method, you must clean up the in-tree build objects before you decide to do the right thing and move on to the superior `./build` Buildroot method:
....
cd packages/kernel_modules
./make-host.sh clean
....
otherwise they will cause problems.
==== Hello host
Minimal host build system example:
@@ -3362,6 +3389,8 @@ although this can be useful when someone gives you a random image.
[[kernel-configs-about]]
==== About our Linux kernel configs
TODO: explain link:update-buildroot-kernel-config[]
We have managed to come up with minimalistic kernel configs that work for both QEMU and gem5 (oh, the hours of bisection).
Our configs are all based on Buildroot's configs, which were designed for QEMU, and then on top of those we also add:
@@ -7276,7 +7305,7 @@ Bibliography:
==== 9P gem5
Seems possible! Lets do it:
TODO seems possible! Lets do it:
* http://gem5.org/wiki/images/b/b8/Summit2017_wa_devlib.pdf
* http://gem5.org/WA-gem5
@@ -10853,23 +10882,21 @@ We use it for:
+
C files for example need compilation, and must go through the regular package system, e.g. through link:packages/kernel_modules/user[].
=== CONTRIBUTING
=== Test this repo
==== Testing
This section describes how to run the most complete set of tests possible.
Testing that should be done for every functional patch.
It takes too much time to be feasible for every patch, but it should be done for every release.
===== Guest testing
Run all tests:
==== Automated tests
....
./build-all
./test
./test --size 3
echo $?
....
Should output 0.
should output 0.
Sources:
@@ -10893,7 +10920,7 @@ Test that the Internet works:
Source: link:rootfs_overlay/test_all.sh[].
===== Host testing
===== Test GDB
Shell 1:
@@ -10914,7 +10941,7 @@ Then proceed to do the following tests:
* `/count.sh` and `b __x64_sys_write`
* `insmod /timer.ko` and `b lkmc_timer_callback`
==== Bisection
=== Bisection
When updating the Linux kernel, QEMU and gem5, things sometimes break.
@@ -10953,7 +10980,7 @@ git submodule update
An example of Linux kernel commit bisection on gem5 boots can be found at: link:bisect-linux-boot-gem5[].
==== Update a forked submodule
=== Update a forked submodule
This is a template update procedure for submodules for which we have some patches on on top of mainline.
@@ -10979,7 +11006,7 @@ git rebase --onto "$next_mainline_revision" "$last_mainline_revision"
git commit -m "linux: update to ${next_mainline_revision}"
....
==== Sanity checks
=== Sanity checks
Basic C and C++ hello worlds:
@@ -11000,7 +11027,7 @@ Sources:
* link:packages/kernel_modules/user/hello.c[]
* link:packages/kernel_modules/user/hello_cpp.c[]
===== rand_check.out
==== rand_check.out
Print out several parameters that normally change randomly from boot to boot:
@@ -11015,7 +11042,7 @@ This can be used to check the determinism of:
* <<norandmaps>>
* <<qemu-record-and-replay>>
==== Release
=== Release
Create a release:
@@ -11042,7 +11069,7 @@ This should in particular enable to easily update <<prebuilt>>.
TODO also run tests and only release if they pass.
===== release-zip
==== release-zip
Create a zip containing all files required for <<prebuilt>>

2
bench-all
View File

@@ -117,7 +117,7 @@ fi
if "$bench_linux_boot"; then
cd "${root_dir}"
"${root_dir}/build-all"
"${root_dir}/bench-boot" -t 3
"${root_dir}/bench-boot" --size 3
cp "$(${root_dir}/getvar bench_boot)" "$new_dir"
fi

15
bench-boot
View File

@@ -2,21 +2,18 @@
set -eu
root_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)"
test_size=1
OPTIND=1
while getopts t: OPT; do
case "$OPT" in
t)
while [ $# -gt 0 ]; do
case "$1" in
--size)
# 1: a few seconds and important
# 2: < 5 minutes and important or a few seconds and not too important
# 3: all
test_size="$OPTARG"
;;
?)
exit 2
test_size="$2"
shift 2
;;
esac
done
shift "$(($OPTIND - 1))"
if [ $# -gt 1 ]; then
extra_args=" $*"
else

2
bisect-linux-boot-gem5
View File

@@ -24,7 +24,7 @@ args = common.setup(parser)
# We need a clean rebuild because rebuilds at different revisions:
# - may fail
# - may not actually rebuild all files, e.g. on header changes
common.rmrf(common.linux_variant_dir)
common.rmrf(common.linux_build_dir)
assert build.main(args) == 0
status = run.main(args, {
'eval': 'm5 exit',

85
build-all
View File

@@ -1,28 +1,89 @@
#!/usr/bin/env bash
# Build everything, or a subset of everything.
#
# Without any args, build everything for all archs:
#
# ./build-all
#
# This will build QEMU, gem5, Buildroot, Linux, etc.
# for x86_64, arm and aarch64.
#
# With --archs, build everything for just the given archs:
#
# ./build-all --archs 'arm aarch64'
#
# Other options make this script build only the given projects. E.g., to build
# just Linux and QEMU for all archs, but not gem5, Buildroot, etc.:
#
# ./build-all --linux --qemu
set -eu
archs='x86_64 arm aarch64'
gem5=true
while getopts A:G OPT; do
case "$OPT" in
A)
archs="$OPTARG"
baremetal=false
buildroot=false
gem5=false
linux=false
qemu=false
while [ $# -gt 0 ]; do
case "$1" in
--archs)
archs="$2"
shift 2
;;
G)
gem5=false
--baremetal)
baremetal=true
shift
;;
--buildroot)
buildroot=true
shift
;;
--gem5)
gem5=true
shift
;;
--linux)
linux=true
shift
;;
--qemu)
qemu=true
shift
;;
esac
done
shift "$(($OPTIND - 1))"
if ! (
"$baremetal" || \
"$buildroot" || \
"$gem5" || \
"$linux" || \
"$qemu"
)
then
baremetal=true
buildroot=true
gem5=true
linux=true
qemu=true
fi
for arch in $archs; do
if "$qemu"; then
./build-qemu --arch "$arch"
if "$gem5"; then
./build-gem5 --arch "$arch"
fi
./build-buildroot --arch "$arch" --gem5 --kernel-modules -l "$@"
if [ ! "$arch" = x86_64 ]; then
if "$baremetal" && [ ! "$arch" = x86_64 ]; then
./build-crosstool-ng --arch "$arch"
./build-baremetal --arch "$arch"
./build-baremetal --arch "$arch" --gem5
./build-baremetal --arch "$arch" --gem5 --machine RealViewPBX
fi
if "$buildroot"; then
./build-buildroot --arch "$arch" --gem5 --kernel-modules "$@"
fi
if "$gem5"; then
./build-gem5 --arch "$arch"
fi
if "$linux"; then
./build-linux --arch "$arch"
fi
done

2
build-baremetal
View File

@@ -76,7 +76,7 @@ def main(args, extra_args=None):
gcc = 'arm-none-eabi-gcc'
else:
os.environ['PATH'] = common.crosstool_ng_bin_dir + os.environ['PATH']
gcc = common.get_toolchain_tool('gcc')
gcc = common.get_toolchain_tool('gcc', allowed_toolchains=['crosstool-ng'])
if args.gem5:
if common.machine == 'VExpress_GEM5_V1':
entry_address = 0x80000000

86
build-buildroot
View File

@@ -13,7 +13,7 @@ import common
defaults = {
'baseline': False,
'buildroot_bare_kernel': False,
'build_linux': False,
'buildroot_config': [],
'buildroot_config_fragment': [],
'initramfs': False,
@@ -27,7 +27,6 @@ defaults = {
'no_all': False,
'nproc': None,
'skip_configure': False,
'verbose': False,
'extra_make_args': [],
}
@@ -47,15 +46,25 @@ Takes precedence over any Buildroot config files.
'-b', '--buildroot-config-fragment', default=defaults['buildroot_config_fragment'], action='append',
help='''Also use the given Buildroot configuration fragment file.
Pass multiple times to use multiple fragment files.'''
)
parser.add_argument(
'--build-linux', default=defaults['build_linux'], action='store_true',
help='''\
Enable building the Linux kernel with Buildroot. This is done mostly
to extract Buildroot's default kernel configurations when updating Buildroot.
That kernel will not be use by our scripts.
'''
)
parser.add_argument(
'--baseline', default=defaults['baseline'], action='store_true',
help='''Do a default-ish Buildroot defconfig build, without any of our extra options.
Mostly to track how much slower we are than a basic build.'''
Mostly to track how much slower we are than a basic build.
'''
)
parser.add_argument(
'-C', '--kernel-config', default=defaults['kernel_config'], action='append',
help='''Add a single kernel config configs to the current build.
help='''\
Add a single kernel config configs to the current build.
Example value: 'CONFIG_FORTIFY_SOURCE=y'.
Can be used multiple times to add multiple configs.
Takes precedence over any Buildroot config files.
@@ -63,8 +72,10 @@ Takes precedence over any Buildroot config files.
)
parser.add_argument(
'-c', '--kernel-config-fragment', default=defaults['kernel_config_fragment'], action='append',
help='''Also use the given kernel configuration fragment file.
Pass multiple times to use multiple fragment files.'''
help='''\
Also use the given kernel configuration fragment file.
Pass multiple times to use multiple fragment files.
'''
)
parser.add_argument(
'-I', '--initramfs', default=defaults['initramfs'], action='store_true',
@@ -78,41 +89,35 @@ Pass multiple times to use multiple fragment files.'''
)
parser.add_argument(
'-K', '--kernel-custom-config-file', default=defaults['kernel_custom_config_file'],
help='''Ignore all default kernel configurations and use this file instead.
Still uses options explicitly passed with `-C` and `-c` on top of it.'''
help='''\
Ignore all default kernel configurations and use this file instead.
Still uses options explicitly passed with `-C` and `-c` on top of it.
'''
)
kernel_module_group.add_argument(
'-k', '--kernel-modules', default=defaults['kernel_modules'], action='store_true',
help='Reconfigure and rebuild the kernel modules package'
)
parser.add_argument(
'-l', '--linux-reconfigure', default=defaults['linux_reconfigure'], action='store_true',
help='''Reconfigure and rebuild the Linux kernel.
Touches kernel configuration files to overcome:
https://stackoverflow.com/questions/49260466/why-when-i-change-br2-linux-kernel-custom-config-file-and-run-make-linux-reconfi'''
)
parser.add_argument(
'--no-all', default=defaults['no_all'], action='store_true',
help='''Don't build the all target which normally gets build by default.
That target builds the root filesystem and all its dependencies.'''
)
kernel_module_group.add_argument(
'--no-kernel-modules', default=defaults['no_kernel_modules'], action='store_true',
help="Don't build the kernel modules package"
help='''\
Don't build the all target which normally gets build by default.
That target builds the root filesystem and all its dependencies.
'''
)
parser.add_argument(
'--skip-configure', default=defaults['skip_configure'], action='store_true',
help='''Skip the Buildroot configuration. Saves a few seconds,
but requires you to know what you are doing :-)'''
)
parser.add_argument(
'-v', '--verbose', default=defaults['verbose'], action='store_true',
help='Do a verbose build'
help='''\
Skip the Buildroot configuration. Saves a few seconds,
but requires you to know what you are doing :-)
'''
)
parser.add_argument(
'extra_make_args', default=defaults['extra_make_args'], metavar='extra-make-args', nargs='*',
help='''Extra arguments to be passed to the Buildroot make,
usually extra Buildroot targets.'''
help='''\
Extra arguments to be passed to the Buildroot make,
usually extra Buildroot targets.
'''
)
return parser
@@ -125,6 +130,7 @@ def main(args, extra_args=None):
os.makedirs(common.out_dir, exist_ok=True)
extra_make_args = args.extra_make_args.copy()
if args.kernel_modules:
assert(args.build_linux)
extra_make_args.append('kernel_modules-reconfigure')
if args.linux_reconfigure:
extra_make_args.append('linux-reconfigure')
@@ -171,6 +177,10 @@ def main(args, extra_args=None):
'BR2_DL_DIR="{}"'.format(common.buildroot_download_dir),
])
common.write_configs(common.buildroot_config_file, buildroot_configs)
if args.build_linux:
buildroot_configs.extend([
'# BR2_LINUX_KERNEL is not set',
])
if not args.baseline:
buildroot_configs.extend([
'BR2_GLOBAL_PATCH_DIR="{}"'.format(
@@ -180,13 +190,13 @@ def main(args, extra_args=None):
'BR2_PACKAGE_OVERRIDE_FILE="{}"'.format(
path_relative_to_buildroot(os.path.join(common.root_dir, 'buildroot_override'))),
'BR2_ROOTFS_OVERLAY="{}"'.format(
path_relative_to_buildroot(os.path.join(common.root_dir, 'rootfs_overlay'))),
path_relative_to_buildroot(common.rootfs_overlay_dir)),
'BR2_ROOTFS_POST_BUILD_SCRIPT="{}"'.format(
path_relative_to_buildroot(os.path.join(common.root_dir, 'rootfs-post-build-script'))),
'BR2_ROOTFS_USERS_TABLES="{}"'.format(
path_relative_to_buildroot(os.path.join(common.root_dir, 'user_table'))),
])
if not args.no_kernel_modules:
if args.kernel_modules:
buildroot_configs.append('BR2_PACKAGE_KERNEL_MODULES=y')
if args.gem5:
buildroot_configs.append('BR2_PACKAGE_GEM5=y')
@@ -241,13 +251,12 @@ def main(args, extra_args=None):
raise Exception('Kernel config fragment file does not exist: {}'.format(args.kernel_custom_config_file))
default_kernel_config_fragments = []
else:
kernel_config_fragment_dir = os.path.join(common.root_dir, 'kernel_config')
default_kernel_config_fragments = ['min', 'default']
if args.linux_reconfigure:
# https://stackoverflow.com/questions/49260466/why-when-i-change-br2-linux-kernel-custom-config-file-and-run-make-linux-reconfi
pathlib.Path(os.path.join(kernel_config_fragment_dir, 'min')).touch()
pathlib.Path(os.path.join(common.linux_config_dir, 'min')).touch()
for i, default_kernel_config_fragment in enumerate(default_kernel_config_fragments):
default_kernel_config_fragments[i] = os.path.join(kernel_config_fragment_dir, default_kernel_config_fragment)
default_kernel_config_fragments[i] = os.path.join(common.linux_config_dir, default_kernel_config_fragment)
kernel_config_fragments.extend(default_kernel_config_fragments)
for i, frag in enumerate(kernel_config_fragments):
kernel_config_fragments[i] = path_relative_to_buildroot(frag)
@@ -263,17 +272,6 @@ def main(args, extra_args=None):
cwd=common.buildroot_src_dir,
) == 0
# Manage Linux kernel and QEMU variants.
def symlink_buildroot_variant(custom_dir, variant_dir):
if os.path.islink(custom_dir):
os.unlink(custom_dir)
elif os.path.isdir(custom_dir):
# Migration for existing builds.
shutil.move(custom_dir, variant_dir)
os.makedirs(variant_dir, exist_ok=True)
os.symlink(variant_dir, custom_dir)
symlink_buildroot_variant(common.linux_build_dir, common.linux_variant_dir)
# Do the actual build.
common.mkdir()
if not args.no_all:

10
build-gem5
View File

@@ -38,6 +38,10 @@ else:
'-b', os.path.join('wt', args.gem5_build_id),
common.gem5_src_dir
]) == 0
if args.verbose:
verbose = ['--verbose']
else:
verbose = []
if args.arch == 'x86_64':
dummy_img_path = os.path.join(disks_dir, 'linux-bigswap2.img')
with open(dummy_img_path, 'wb') as dummy_img_file:
@@ -73,14 +77,16 @@ else:
assert common.run_cmd(['make', '-C', bootloader64_dir]) == 0
shutil.copy2(os.path.join(bootloader64_dir, 'boot_emm.arm64'), binaries_dir)
assert common.run_cmd(
(
[
'scons',
# TODO factor with build.
'-j', str(multiprocessing.cpu_count()),
'--ignore-style',
common.gem5_executable
] +
args.extra_scons_args,
verbose +
args.extra_scons_args
),
cwd=common.gem5_src_dir,
extra_paths=[common.ccache_dir],
) == 0

86
build-linux Executable file
View File

@@ -0,0 +1,86 @@
#!/usr/bin/env python3
import multiprocessing
import os
import shutil
import subprocess
import time
import common
parser = common.get_argparse()
common.add_build_arguments(parser)
parser.add_argument(
'extra_make_args',
default=[],
metavar='extra-make-args',
nargs='*'
)
args = common.setup(parser)
if args.clean:
common.rmrf(common.linux_build_dir)
else:
start_time = time.time()
os.makedirs(common.linux_build_dir, exist_ok=True)
shutil.copy2(
os.path.join(common.linux_config_dir, 'buildroot-{}'.format(args.arch)),
os.path.join(common.linux_build_dir, '.config'),
)
tool = 'gcc'
gcc = common.get_toolchain_tool(tool)
prefix = gcc[:-len(tool)]
common_args = {
'cwd': common.linux_src_dir,
}
ccache = shutil.which('ccache')
if ccache is not None:
cc = '{} {}'.format(ccache, gcc)
else:
cc = gcc
common_make_args = [
'ARCH={}'.format(common.linux_arch),
'CROSS_COMPILE={}'.format(prefix),
'CC={}'.format(cc),
'O={}'.format(common.linux_build_dir),
]
if args.verbose:
verbose = ['V=1']
else:
verbose = []
assert common.run_cmd(
[
os.path.join(common.linux_src_dir, 'scripts', 'kconfig', 'merge_config.sh'),
'-m',
'-O', common.linux_build_dir,
os.path.join(common.linux_build_dir, '.config'),
os.path.join(common.linux_config_dir, 'min'),
os.path.join(common.linux_config_dir, 'default'),
],
) == 0
assert common.run_cmd(
(
[
'make',
'-j', str(multiprocessing.cpu_count()),
] +
common_make_args +
[
'olddefconfig',
]
),
**common_args,
) == 0
assert common.run_cmd(
(
[
'make',
'-j', str(multiprocessing.cpu_count()),
] +
common_make_args +
verbose +
args.extra_make_args
),
**common_args,
) == 0
end_time = time.time()
common.print_time(end_time - start_time)

122
build-modules Executable file
View File

@@ -0,0 +1,122 @@
#!/usr/bin/env python3
import distutils.file_util
import multiprocessing
import os
import platform
import shutil
import subprocess
import time
import common
parser = common.get_argparse()
common.add_build_arguments(parser)
parser.add_argument(
'--host',
action='store_true',
default=False,
help='Build the modules for the host instead of guest',
)
parser.add_argument(
'kernel_modules',
default=[],
help='Which kernel modules to build. Default: build all',
metavar='kernel-modules',
nargs='*',
)
args = common.setup(parser)
if args.host:
build_dir = os.path.join(common.kernel_modules_build_host_dir)
else:
build_dir = os.path.join(common.kernel_modules_build_dir)
if args.clean:
common.rmrf(build_dir)
else:
start_time = time.time()
os.makedirs(build_dir, exist_ok=True)
# Build kernel modules.
#
# I kid you not, out-of-tree build is not possible, O= does not work as for the kernel build:
#
# * https://stackoverflow.com/questions/5718899/building-an-out-of-tree-linux-kernel-module-in-a-separate-object-directory
# * https://stackoverflow.com/questions/12244979/build-kernel-module-into-a-specific-directory
# * https://stackoverflow.com/questions/18386182/out-of-tree-kernel-modules-multiple-module-single-makefile-same-source-file
#
# This copies only modified files as per:
# https://stackoverflow.com/questions/5718899/building-an-out-of-tree-linux-kernel-module-in-a-separate-object-directory
all_kernel_modules = []
for basename in os.listdir(common.kernel_modules_src_dir):
src = os.path.join(common.kernel_modules_src_dir, basename)
if os.path.isfile(src):
distutils.file_util.copy_file(
src,
os.path.join(build_dir, basename),
update=1,
)
noext, ext = os.path.splitext(basename)
if ext == common.c_ext:
all_kernel_modules.append(noext)
if args.kernel_modules == []:
kernel_modules = all_kernel_modules
else:
kernel_modules = map(lambda x: os.path.splitext(os.path.split(x)[1])[0], args.kernel_modules)
object_files = map(lambda x: x + common.obj_ext, kernel_modules)
tool = 'gcc'
if args.host:
allowed_toolchains = ['host']
else:
allowed_toolchains = None
gcc = common.get_toolchain_tool(tool, allowed_toolchains=allowed_toolchains)
prefix = gcc[:-len(tool)]
ccache = shutil.which('ccache')
if ccache is not None:
cc = '{} {}'.format(ccache, gcc)
else:
cc = gcc
if args.verbose:
verbose = ['V=1']
else:
verbose = []
if args.host:
linux_dir = os.path.join('/lib', 'modules', platform.uname().release, 'build')
else:
linux_dir = common.linux_build_dir
assert common.run_cmd(
(
[
'make',
'-j', str(multiprocessing.cpu_count()),
'ARCH={}'.format(common.linux_arch),
'CC={}'.format(cc),
'CROSS_COMPILE={}'.format(prefix),
'LINUX_DIR={}'.format(linux_dir),
'M={}'.format(build_dir),
'OBJECT_FILES={}'.format(' '.join(object_files)),
] +
verbose
),
cwd=common.kernel_modules_src_dir,
) == 0
# Build userland tools.
if args.host:
allowed_toolchains = ['host']
else:
allowed_toolchains = ['buildroot']
cc = common.get_toolchain_tool('gcc', allowed_toolchains=allowed_toolchains)
cxx = common.get_toolchain_tool('g++', allowed_toolchains=allowed_toolchains)
assert common.run_cmd(
[
'make',
'-j', str(multiprocessing.cpu_count()),
'CC={}'.format(cc),
'CXX={}'.format(cxx),
'OUT_DIR={}'.format(os.path.join(build_dir, 'user')),
],
cwd=os.path.join(common.kernel_modules_src_dir, 'user'),
extra_paths=[common.ccache_dir],
) == 0
end_time = time.time()
common.print_time(end_time - start_time)

11
build-qemu
View File

@@ -21,6 +21,10 @@ if args.clean:
else:
start_time = time.time()
os.makedirs(common.qemu_build_dir, exist_ok=True)
if args.verbose:
verbose = ['V=1']
else:
verbose = []
assert common.run_cmd(
[
os.path.join(common.qemu_src_dir, 'configure'),
@@ -35,11 +39,14 @@ else:
cwd=common.qemu_build_dir
) == 0
assert common.run_cmd(
(
[
'make',
# TODO factor with build.
'-j', str(multiprocessing.cpu_count()),
],
] +
verbose
),
cwd=common.qemu_build_dir,
extra_paths=[common.ccache_dir],
) == 0

115
common.py
View File

@@ -32,6 +32,8 @@ submodules_dir = os.path.join(root_dir, 'submodules')
buildroot_src_dir = os.path.join(submodules_dir, 'buildroot')
crosstool_ng_src_dir = os.path.join(submodules_dir, 'crosstool-ng')
linux_src_dir = os.path.join(submodules_dir, 'linux')
linux_config_dir = os.path.join(this.root_dir, 'kernel_config')
rootfs_overlay_dir = os.path.join(this.root_dir, 'rootfs_overlay')
extract_vmlinux = os.path.join(linux_src_dir, 'scripts', 'extract-vmlinux')
qemu_src_dir = os.path.join(submodules_dir, 'qemu')
parsec_benchmark_src_dir = os.path.join(submodules_dir, 'parsec-benchmark')
@@ -47,6 +49,10 @@ sha = subprocess.check_output(['git', '-C', root_dir, 'log', '-1', '--format=%H'
release_dir = os.path.join(this.out_dir, 'release')
release_zip_file = os.path.join(this.release_dir, 'lkmc-{}.zip'.format(this.sha))
github_repo_id = 'cirosantilli/linux-kernel-module-cheat'
asm_ext = '.S'
c_ext = '.c'
kernel_module_ext = '.ko'
obj_ext = '.o'
config_file = os.path.join(data_dir, 'config')
if os.path.exists(config_file):
config = imp.load_source('config', config_file)
@@ -101,6 +107,11 @@ def get_argparse(default_args=None, argparse_args=None):
'--baremetal',
help='Use Baremetal examples instead of Linux kernel ones'
)
parser.add_argument(
'--buildroot-build-id',
default=default_build_id,
help='Buildroot build ID. Allows you to keep multiple separate gem5 builds. Default: %(default)s'
)
parser.add_argument(
'--crosstool-ng-build-id', default=default_build_id,
help='Crosstool-NG build ID. Allows you to keep multiple separate crosstool-NG builds. Default: %(default)s'
@@ -109,6 +120,12 @@ def get_argparse(default_args=None, argparse_args=None):
'-g', '--gem5', default=False, action='store_true',
help='Use gem5 instead of QEMU'
)
parser.add_argument(
'--gem5-src',
help='''\
Use the given directory as the gem5 source tree. Ignore `--gem5-worktree`.
'''
)
parser.add_argument(
'-L', '--linux-build-id', default=default_build_id,
help='Linux build ID. Allows you to keep multiple separate Linux builds. Default: %(default)s'
@@ -173,15 +190,14 @@ Default: the run ID (-n) if that is an integer, otherwise 0.
'-Q', '--qemu-build-id', default=default_build_id,
help='QEMU build ID. Allows you to keep multiple separate QEMU builds. Default: %(default)s'
)
parser.add_argument(
'--buildroot-build-id',
default=default_build_id,
help='Buildroot build ID. Allows you to keep multiple separate gem5 builds. Default: %(default)s'
)
parser.add_argument(
'-t', '--gem5-build-type', default='opt',
help='gem5 build type, most often used for "debug" builds. Default: %(default)s'
)
parser.add_argument(
'-v', '--verbose', default=False, action='store_true',
help='Show full compilation commands when they are not shown by default.'
)
if hasattr(this, 'configs'):
defaults = this.configs.copy()
else:
@@ -224,12 +240,39 @@ def get_stats(stat_re=None, stats_file=None):
ret.append(cols[1])
return ret
def get_toolchain_tool(tool):
global this
if this.baremetal is None:
return glob.glob(os.path.join(this.host_bin_dir, '*-buildroot-*-{}'.format(tool)))[0]
def get_toolchain_prefix(tool, allowed_toolchains=None):
buildroot_full_prefix = os.path.join(this.host_bin_dir, this.buildroot_toolchain_prefix)
buildroot_exists = os.path.exists('{}-{}'.format(buildroot_full_prefix, tool))
crosstool_ng_full_prefix = os.path.join(this.crosstool_ng_bin_dir, this.crosstool_ng_toolchain_prefix)
crosstool_ng_exists = os.path.exists('{}-{}'.format(crosstool_ng_full_prefix, tool))
host_tool = '{}-{}'.format(this.ubuntu_toolchain_prefix, tool)
host_path = shutil.which(host_tool)
if host_path is not None:
host_exists = True
host_full_prefix = host_path[:-(len(tool)+1)]
else:
return os.path.join(this.crosstool_ng_bin_dir, '{}-{}'.format(this.crosstool_ng_prefix, tool))
host_exists = False
host_full_prefix = None
known_toolchains = {
'crosstool-ng': (crosstool_ng_exists, crosstool_ng_full_prefix),
'buildroot': (buildroot_exists, buildroot_full_prefix),
'host': (host_exists, host_full_prefix),
}
if allowed_toolchains is None:
if this.baremetal is None:
allowed_toolchains = ['buildroot', 'crosstool-ng', 'host']
else:
allowed_toolchains = ['crosstool-ng', 'buildroot', 'host']
tried = []
for toolchain in allowed_toolchains:
exists, prefix = known_toolchains[toolchain]
tried.append('{}-{}'.format(prefix, tool))
if exists:
return prefix
raise Exception('Tool not found. Tried:\n' + '\n'.join(tried))
def get_toolchain_tool(tool, allowed_toolchains=None):
return '{}-{}'.format(this.get_toolchain_prefix(tool, allowed_toolchains), tool)
def github_make_request(
authenticate=False,
@@ -273,7 +316,7 @@ def mkdir():
os.makedirs(this.qemu_run_dir, exist_ok=True)
os.makedirs(this.p9_dir, exist_ok=True)
def print_cmd(cmd, cwd, cmd_file=None, extra_env=None, extra_paths=None):
def print_cmd(cmd, cwd=None, cmd_file=None, extra_env=None, extra_paths=None):
'''
Format a command given as a list of strings so that it can
be viewed nicely and executed by bash directly and print it to stdout.
@@ -283,6 +326,7 @@ def print_cmd(cmd, cwd, cmd_file=None, extra_env=None, extra_paths=None):
'''
newline_separator = ' \\\n'
out = []
if cwd is not None:
out.append('cd {} &&{}'.format(shlex.quote(cwd), newline_separator))
if extra_paths is not None:
out.append('PATH="{}:${{PATH}}"'.format(':'.join(extra_paths)) + newline_separator)
@@ -395,7 +439,7 @@ def run_cmd(
if 'cwd' in kwargs:
cwd = kwargs['cwd']
else:
cwd = os.getcwd()
cwd = None
env = os.environ.copy()
env.update(extra_env)
if extra_paths is not None:
@@ -458,7 +502,9 @@ def setup(parser):
this.armv = 7
this.gem5_arch = 'ARM'
this.mcpu = 'cortex-a15'
this.crosstool_ng_prefix = 'arm-unknown-eabi'
this.buildroot_toolchain_prefix = 'arm-buildroot-linux-uclibcgnueabihf'
this.crosstool_ng_toolchain_prefix = 'arm-unknown-eabi'
this.ubuntu_toolchain_prefix = 'arm-linux-gnueabihf'
if args.gem5:
if this.machine is None:
this.machine = 'VExpress_GEM5_V1'
@@ -469,7 +515,9 @@ def setup(parser):
this.armv = 8
this.gem5_arch = 'ARM'
this.mcpu = 'cortex-a57'
this.crosstool_ng_prefix = 'aarch64-unknown-elf'
this.buildroot_toolchain_prefix = 'aarch64-buildroot-linux-uclibc'
this.crosstool_ng_toolchain_prefix = 'aarch64-unknown-elf'
this.ubuntu_toolchain_prefix = 'aarch64-linux-gnu'
if args.gem5:
if this.machine is None:
this.machine = 'VExpress_GEM5_V1'
@@ -477,8 +525,10 @@ def setup(parser):
if this.machine is None:
this.machine = 'virt'
elif args.arch == 'x86_64':
this.crosstool_ng_prefix = 'TODO'
this.crosstool_ng_toolchain_prefix = 'x86_64-unknown-elf'
this.gem5_arch = 'X86'
this.buildroot_toolchain_prefix = 'x86_64-buildroot-linux-uclibc'
this.ubuntu_toolchain_prefix = 'x86_64-linux-gnu'
if args.gem5:
if this.machine is None:
this.machine = 'TODO'
@@ -490,9 +540,6 @@ def setup(parser):
this.buildroot_download_dir = os.path.join(this.buildroot_out_dir, 'download')
this.buildroot_config_file = os.path.join(this.buildroot_build_dir, '.config')
this.build_dir = os.path.join(this.buildroot_build_dir, 'build')
this.linux_build_dir = os.path.join(this.build_dir, 'linux-custom')
this.linux_variant_dir = '{}.{}'.format(this.linux_build_dir, args.linux_build_id)
this.vmlinux = os.path.join(this.linux_variant_dir, "vmlinux")
this.qemu_build_dir = os.path.join(this.out_dir, 'qemu', args.qemu_build_id)
this.qemu_executable_basename = 'qemu-system-{}'.format(args.arch)
this.qemu_executable = os.path.join(this.qemu_build_dir, '{}-softmmu'.format(args.arch), this.qemu_executable_basename)
@@ -560,15 +607,28 @@ def setup(parser):
this.gem5_se_file = os.path.join(this.gem5_config_dir, 'example', 'se.py')
this.gem5_fs_file = os.path.join(this.gem5_config_dir, 'example', 'fs.py')
this.run_cmd_file = os.path.join(this.run_dir, 'run.sh')
if args.arch == 'arm':
this.linux_image = os.path.join('arch', 'arm', 'boot', 'zImage')
elif args.arch == 'aarch64':
this.linux_image = os.path.join('arch', 'arm64', 'boot', 'Image')
elif args.arch == 'x86_64':
this.linux_image = os.path.join('arch', 'x86', 'boot', 'bzImage')
this.linux_image = os.path.join(this.linux_variant_dir, linux_image)
# Ports.
# Linux
this.linux_buildroot_build_dir = os.path.join(this.build_dir, 'linux-custom')
this.linux_build_dir = os.path.join(this.out_dir, 'linux', args.linux_build_id, args.arch)
this.vmlinux = os.path.join(this.linux_build_dir, "vmlinux")
if args.arch == 'arm':
this.linux_arch = 'arm'
this.linux_image = os.path.join('arch', this.linux_arch, 'boot', 'zImage')
elif args.arch == 'aarch64':
this.linux_arch = 'arm64'
this.linux_image = os.path.join('arch', this.linux_arch, 'boot', 'Image')
elif args.arch == 'x86_64':
this.linux_arch = 'x86'
this.linux_image = os.path.join('arch', this.linux_arch, 'boot', 'bzImage')
this.linux_image = os.path.join(this.linux_build_dir, linux_image)
# Kernel modules.
this.kernel_modules_build_base_dir = os.path.join(this.out_dir, 'kernel_modules')
this.kernel_modules_build_dir = os.path.join(this.kernel_modules_build_base_dir, args.arch)
this.kernel_modules_build_host_dir = os.path.join(this.kernel_modules_build_base_dir, 'host')
# Ports
if args.port_offset is None:
try:
args.port_offset = int(args.run_id)
@@ -590,9 +650,6 @@ def setup(parser):
this.baremetal_lib_basename = 'lib'
this.baremetal_src_dir = os.path.join(this.root_dir, 'baremetal')
this.baremetal_src_lib_dir = os.path.join(this.baremetal_src_dir, this.baremetal_lib_basename)
this.c_ext = '.c'
this.asm_ext = '.S'
this.obj_ext = '.o'
if args.gem5:
this.simulator_name = 'gem5'
else:

4
configure vendored
View File

@@ -82,6 +82,8 @@ build-essential \
coreutils \
cpio \
expect \
gcc-aarch64-linux-gnu \
gcc-arm-linux-gnueabi \
git \
moreutils \
rsync \
@@ -93,8 +95,6 @@ wget \
if "$gem5"; then
pkgs="${pkgs} \
ccache \
gcc-aarch64-linux-gnu \
gcc-arm-linux-gnueabi \
libgoogle-perftools-dev \
protobuf-compiler \
python-dev \

98
kernel_config/buildroot-aarch64 Normal file
View File

@@ -0,0 +1,98 @@
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_NO_HZ_IDLE=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_TASKSTATS=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_SCHED_AUTOGROUP=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_KALLSYMS_ALL=y
CONFIG_PROFILING=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODVERSIONS=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_BLK_DEV_BSGLIB=y
CONFIG_ARCH_VEXPRESS=y
CONFIG_PCI=y
CONFIG_PCI_HOST_GENERIC=y
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_BINFMT_MISC=y
CONFIG_COMPAT=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_PACKET_DIAG=y
CONFIG_UNIX=y
CONFIG_NET_KEY=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_BRIDGE=m
CONFIG_NET_SCHED=y
CONFIG_VSOCKETS=y
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_NET_9P_DEBUG=y
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_CONNECTOR=y
CONFIG_VIRTIO_BLK=y
CONFIG_DUMMY_IRQ=m
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_SG=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SCAN_ASYNC=y
CONFIG_SCSI_VIRTIO=y
CONFIG_ATA=y
CONFIG_ATA_PIIX=y
CONFIG_NETDEVICES=y
CONFIG_DUMMY=y
CONFIG_MACVLAN=y
CONFIG_VIRTIO_NET=y
CONFIG_NLMON=y
CONFIG_8139CP=y
# CONFIG_INPUT_MOUSE is not set
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_SERIAL_AMBA_PL011=y
CONFIG_SERIAL_AMBA_PL011_CONSOLE=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_FB=y
CONFIG_LOGO=y
CONFIG_RTC_CLASS=y
CONFIG_UIO=m
CONFIG_UIO_PDRV_GENIRQ=m
CONFIG_UIO_DMEM_GENIRQ=m
CONFIG_UIO_PCI_GENERIC=m
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_MMIO=y
CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
CONFIG_MAILBOX=y
CONFIG_PL320_MBOX=y
CONFIG_EXT4_FS=y
CONFIG_OVERLAY_FS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_CONFIGFS_FS=y
CONFIG_SQUASHFS=y
CONFIG_9P_FS=y
CONFIG_9P_FS_POSIX_ACL=y
CONFIG_9P_FS_SECURITY=y
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DEBUG_INFO=y
CONFIG_GDB_SCRIPTS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_IRQSOFF_TRACER=y
CONFIG_SCHED_TRACER=y
CONFIG_HWLAT_TRACER=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_STACK_TRACER=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_KGDB=y
CONFIG_KGDB_TESTS=y
CONFIG_KGDB_KDB=y
CONFIG_KDB_KEYBOARD=y
# CONFIG_STRICT_DEVMEM is not set

180
kernel_config/buildroot-arm Normal file
View File

@@ -0,0 +1,180 @@
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_SYSVIPC=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=14
CONFIG_CGROUPS=y
CONFIG_CPUSETS=y
# CONFIG_UTS_NS is not set
# CONFIG_IPC_NS is not set
# CONFIG_PID_NS is not set
# CONFIG_NET_NS is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_KALLSYMS_ALL=y
CONFIG_PROFILING=y
CONFIG_OPROFILE=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODVERSIONS=y
CONFIG_MODULE_SRCVERSION_ALL=y
# CONFIG_LBDAF is not set
# CONFIG_BLK_DEV_BSG is not set
# CONFIG_IOSCHED_DEADLINE is not set
# CONFIG_IOSCHED_CFQ is not set
CONFIG_ARCH_VEXPRESS=y
CONFIG_ARCH_VEXPRESS_DCSCB=y
CONFIG_ARCH_VEXPRESS_TC2_PM=y
CONFIG_PCI=y
CONFIG_PCI_HOST_GENERIC=y
CONFIG_SMP=y
CONFIG_HAVE_ARM_ARCH_TIMER=y
CONFIG_MCPM=y
CONFIG_VMSPLIT_2G=y
CONFIG_NR_CPUS=8
CONFIG_ARM_PSCI=y
CONFIG_CMA=y
CONFIG_ZBOOT_ROM_TEXT=0x0
CONFIG_ZBOOT_ROM_BSS=0x0
CONFIG_CMDLINE="console=ttyAMA0"
CONFIG_CPU_IDLE=y
CONFIG_VFP=y
CONFIG_NEON=y
# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_INET=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
CONFIG_IP_PNP_BOOTP=y
# CONFIG_IPV6 is not set
# CONFIG_WIRELESS is not set
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_NET_9P_DEBUG=y
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_CONNECTOR=y
CONFIG_MTD=y
CONFIG_MTD_CMDLINE_PARTS=y
CONFIG_MTD_BLOCK=y
CONFIG_MTD_CFI=y
CONFIG_MTD_CFI_INTELEXT=y
CONFIG_MTD_CFI_AMDSTD=y
CONFIG_MTD_PHYSMAP=y
CONFIG_MTD_PHYSMAP_OF=y
CONFIG_MTD_PLATRAM=y
CONFIG_MTD_UBI=y
CONFIG_VIRTIO_BLK=y
CONFIG_DUMMY_IRQ=m
# CONFIG_SCSI_PROC_FS is not set
CONFIG_BLK_DEV_SD=y
CONFIG_SCSI_VIRTIO=y
CONFIG_ATA=y
# CONFIG_SATA_PMP is not set
CONFIG_ATA_PIIX=y
CONFIG_NETDEVICES=y
CONFIG_VIRTIO_NET=y
CONFIG_8139CP=y
CONFIG_SMC91X=y
CONFIG_SMSC911X=y
# CONFIG_WLAN is not set
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_MOUSE is not set
# CONFIG_SERIO_SERPORT is not set
CONFIG_SERIO_AMBAKMI=y
CONFIG_LEGACY_PTY_COUNT=16
CONFIG_SERIAL_AMBA_PL011=y
CONFIG_SERIAL_AMBA_PL011_CONSOLE=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_I2C=y
CONFIG_I2C_VERSATILE=y
CONFIG_SENSORS_VEXPRESS=y
CONFIG_REGULATOR_VEXPRESS=y
CONFIG_FB=y
CONFIG_FB_ARMCLCD=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set
CONFIG_SOUND=y
CONFIG_SND=y
# CONFIG_SND_DRIVERS is not set
CONFIG_SND_ARMAACI=y
CONFIG_HID_DRAGONRISE=y
CONFIG_HID_GYRATION=y
CONFIG_HID_TWINHAN=y
CONFIG_HID_NTRIG=y
CONFIG_HID_PANTHERLORD=y
CONFIG_HID_PETALYNX=y
CONFIG_HID_SAMSUNG=y
CONFIG_HID_SONY=y
CONFIG_HID_SUNPLUS=y
CONFIG_HID_GREENASIA=y
CONFIG_HID_SMARTJOYPLUS=y
CONFIG_HID_TOPSEED=y
CONFIG_HID_THRUSTMASTER=y
CONFIG_HID_ZEROPLUS=y
CONFIG_USB=y
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y
CONFIG_USB_MON=y
CONFIG_USB_STORAGE=y
CONFIG_USB_ISP1760=y
CONFIG_MMC=y
CONFIG_MMC_ARMMMCI=y
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y
CONFIG_LEDS_GPIO=y
CONFIG_LEDS_TRIGGERS=y
CONFIG_LEDS_TRIGGER_HEARTBEAT=y
CONFIG_LEDS_TRIGGER_CPU=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_DRV_PL031=y
CONFIG_UIO=m
CONFIG_UIO_PDRV_GENIRQ=m
CONFIG_UIO_DMEM_GENIRQ=m
CONFIG_UIO_PCI_GENERIC=m
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_MMIO=y
CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
CONFIG_EXT2_FS=y
CONFIG_EXT3_FS=y
CONFIG_OVERLAY_FS=y
CONFIG_VFAT_FS=y
CONFIG_TMPFS=y
CONFIG_CONFIGFS_FS=y
CONFIG_JFFS2_FS=y
CONFIG_UBIFS_FS=y
CONFIG_CRAMFS=y
CONFIG_SQUASHFS=y
CONFIG_SQUASHFS_LZO=y
CONFIG_NFS_FS=y
CONFIG_ROOT_NFS=y
CONFIG_9P_FS=y
CONFIG_9P_FS_POSIX_ACL=y
CONFIG_9P_FS_SECURITY=y
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ISO8859_1=y
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DEBUG_INFO=y
CONFIG_GDB_SCRIPTS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DETECT_HUNG_TASK=y
# CONFIG_SCHED_DEBUG is not set
CONFIG_IRQSOFF_TRACER=y
CONFIG_SCHED_TRACER=y
CONFIG_HWLAT_TRACER=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_STACK_TRACER=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_KGDB=y
CONFIG_KGDB_TESTS=y
CONFIG_KGDB_KDB=y
CONFIG_KDB_KEYBOARD=y
CONFIG_DEBUG_USER=y
# CONFIG_CRYPTO_HW is not set

93
kernel_config/buildroot-x86_64 Normal file
View File

@@ -0,0 +1,93 @@
CONFIG_SYSVIPC=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_CGROUPS=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_KALLSYMS_ALL=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODVERSIONS=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_SMP=y
CONFIG_HYPERVISOR_GUEST=y
CONFIG_PARAVIRT=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_INET=y
# CONFIG_WIRELESS is not set
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_NET_9P_DEBUG=y
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_CONNECTOR=y
CONFIG_VIRTIO_BLK=y
CONFIG_DUMMY_IRQ=m
CONFIG_BLK_DEV_SD=y
CONFIG_SCSI_VIRTIO=y
CONFIG_ATA=y
CONFIG_ATA_PIIX=y
CONFIG_NETDEVICES=y
CONFIG_VIRTIO_NET=y
CONFIG_NE2K_PCI=y
CONFIG_8139CP=y
# CONFIG_WLAN is not set
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_MOUSE is not set
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_HW_RANDOM_VIRTIO=m
CONFIG_DRM=y
CONFIG_DRM_QXL=y
CONFIG_DRM_BOCHS=y
CONFIG_DRM_VIRTIO_GPU=y
CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y
CONFIG_LOGO=y
CONFIG_SOUND=y
CONFIG_SND=y
CONFIG_SND_HDA_INTEL=y
CONFIG_SND_HDA_GENERIC=y
CONFIG_USB=y
CONFIG_USB_XHCI_HCD=y
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_UHCI_HCD=y
CONFIG_USB_STORAGE=y
CONFIG_UIO=m
CONFIG_UIO_PDRV_GENIRQ=m
CONFIG_UIO_DMEM_GENIRQ=m
CONFIG_UIO_PCI_GENERIC=m
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_INPUT=y
CONFIG_VIRTIO_MMIO=y
CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
CONFIG_EXT4_FS=y
CONFIG_AUTOFS4_FS=y
CONFIG_OVERLAY_FS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_CONFIGFS_FS=y
CONFIG_SQUASHFS=y
CONFIG_9P_FS=y
CONFIG_9P_FS_POSIX_ACL=y
CONFIG_9P_FS_SECURITY=y
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DEBUG_INFO=y
CONFIG_GDB_SCRIPTS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_IRQSOFF_TRACER=y
CONFIG_SCHED_TRACER=y
CONFIG_HWLAT_TRACER=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_STACK_TRACER=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_KGDB=y
CONFIG_KGDB_TESTS=y
CONFIG_KGDB_LOW_LEVEL_TRAP=y
CONFIG_KGDB_KDB=y
CONFIG_KDB_KEYBOARD=y
# CONFIG_STRICT_DEVMEM is not set
CONFIG_X86_PTDUMP=y
CONFIG_UNWINDER_FRAME_POINTER=y

11
kernel_config/default
View File

@@ -2,7 +2,7 @@
CONFIG_BLK_DEV_INITRD=y
CONFIG_DYNAMIC_DEBUG=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_STRICT_DEVMEM=n
# CONFIG_STRICT_DEVMEM is not set
# Filesystems.
CONFIG_DEBUG_FS=y
@@ -24,10 +24,10 @@ CONFIG_KGDB_KDB=y
CONFIG_KGDB_LOW_LEVEL_TRAP=y
CONFIG_KGDB_SERIAL_CONSOLE=y
CONFIG_KGDB_TESTS=y
CONFIG_KGDB_TESTS_ON_BOOT=n
# CONFIG_KGDB_TESTS_ON_BOOT is not set
CONFIG_MAGIC_SYSRQ=y
CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1
CONFIG_SERIAL_KGDB_NMI=n
# CONFIG_SERIAL_KGDB_NMI is not set
# Non-static variables show up on /proc/kallsyms
# https://stackoverflow.com/questions/20196636/does-kallsyms-have-all-the-symbol-of-kernel-functions/44614878#44614878
@@ -89,10 +89,6 @@ CONFIG_LOGO=y
## Networking
# Will everything blow up?
# https://superuser.com/questions/684005/how-does-one-permanently-disable-gnu-linux-networking/1255015#1255015
#CONFIG_NET=n
# If given, we can use QEMU 2.9.0 default x86 networking without any -net or -netdev options,
# since E1000 is the default networking device as mentioned at:
# https://en.wikibooks.org/w/index.php?title=QEMU/Networking&oldid=3268753
@@ -121,7 +117,6 @@ CONFIG_UIO_PCI_GENERIC=m
## ARM
# Like CONFIG_X86_PTDUMP for ARM.
CONFIG_ARM64_PTDUMP=y
# For record and replay.

9
packages/kernel_modules/Makefile
View File

@@ -1,10 +1,7 @@
obj-m += $(addsuffix .o, $(notdir $(basename $(filter-out %.mod.c, $(wildcard $(BR2_EXTERNAL_KERNEL_MODULES_PATH)/*.c)))))
obj-m += $(OBJECT_FILES)
ccflags-y := -DDEBUG -g -std=gnu99 -Werror -Wno-declaration-after-statement -Wframe-larger-than=1000000000
.PHONY: all clean
.PHONY: all
all:
$(MAKE) -C '/lib/modules/$(shell uname -r)/build' M='$(PWD)' modules
clean:
$(MAKE) -C '$(LINUX_DIR)' M='$(PWD)' clean
$(MAKE) -C '$(LINUX_DIR)' M='$(M)'

24
packages/kernel_modules/make-host.sh
View File

@@ -1,24 +0,0 @@
#!/usr/bin/env bash
# We can almost do everything from the Makefile itself by using default values for
#
# LINUX_DIR ?= "/lib/modules/$(uname -r)/build"
# BR2_EXTERNAL_KERNEL_MODULES_PATH="$(pwd)"
#
# The problem with that is that if you define those variables in your environment,
# the build breaks, so this is more portable.
#
# Trying to add `-i` to overcome incompatible modules will fail,
# because any build failure prevents the generation of all `.mod.c` files.
j="$(($(nproc) - 2))"
while getopts j: OPT; do
case "$OPT" in
'j')
j="$OPTARG"
;;
esac
done
shift $(($OPTIND - 1))
make -j "$j" KERNEL_MODULES_PATH="$(pwd)" LINUX_DIR="/lib/modules/$(uname -r)/build" "$@"
make -C user/ -j "$j" "$@"

15
packages/kernel_modules/user/Makefile
View File

@@ -1,12 +1,13 @@
.PHONY: all clean
.PHONY: all clean mkdir
CFLAGS_EXTRA = -fopenmp -std=c99
CXXFLAGS_EXTRA = -std=c++17
CCFLAGS_EXTRA = -Wall -Werror -Wextra
IN_EXT_C = .c
IN_EXT_CXX = .cpp
LIBS =
LIBS = -lm
OUT_EXT = .out
OUT_DIR = .
OUTS := $(foreach IN_EXT,$(IN_EXT_C) $(IN_EXT_CXX),$(addsuffix $(OUT_EXT), $(basename $(wildcard *$(IN_EXT)))))
ifeq ($(BR2_PACKAGE_EIGEN),y)
@@ -30,14 +31,18 @@ ifeq ($(BR2_PACKAGE_OPENBLAS),y)
else
OUTS := $(filter-out openblas$(OUT_EXT),$(OUTS))
endif
OUTS := $(addprefix $(OUT_DIR)/,$(OUTS))
all: $(OUTS)
all: mkdir $(OUTS)
%$(OUT_EXT): %$(IN_EXT_C)
$(OUT_DIR)/%$(OUT_EXT): %$(IN_EXT_C)
$(CC) $(CFLAGS) $(CFLAGS_EXTRA) -o '$@' '$<' $(LIBS)
%$(OUT_EXT): %$(IN_EXT_CXX)
$(OUT_DIR)/%$(OUT_EXT): %$(IN_EXT_CXX)
$(CXX) $(CXXFLAGS) $(CXXFLAGS_EXTRA) -o '$@' '$<' $(LIBS)
clean:
rm -f *'$(OUT_EXT)'
mkdir:
mkdir -p '$(OUT_DIR)'

5
rootfs-post-build-script
View File

@@ -4,8 +4,9 @@
# which gets called by the default target.
target_dir="$1"
mkdir -p \
"${target_dir}/mnt/9p" \
"${target_dir}/mnt/out" \
"${target_dir}/mnt/9p/data" \
"${target_dir}/mnt/9p/out" \
"${target_dir}/mnt/9p/rootfs_overlay" \
;
# Maybe there is a cleaner way to get rid of those files,
# like disabling some Buildroot packages, but no patience.

5
rootfs_overlay/etc/fstab
View File

@@ -7,5 +7,6 @@ tmpfs /tmp tmpfs mode=1777 0 0
tmpfs /run tmpfs mode=0755,nosuid,nodev 0 0
sysfs /sys sysfs defaults 0 0
debugfs /sys/kernel/debug debugfs defaults 0 0
host_scratch /mnt/9p 9p trans=virtio,version=9p2000.L 0 0
host_out /mnt/out 9p trans=virtio,version=9p2000.L 0 0
host_data /mnt/9p/data 9p trans=virtio,version=9p2000.L 0 0
host_out /mnt/9p/out 9p trans=virtio,version=9p2000.L 0 0
host_rootfs_overlay /mnt/9p/rootfs_overlay 9p trans=virtio,version=9p2000.L 0 0

5
run
View File

@@ -217,8 +217,9 @@ def main(args, extra_args=None):
'-no-reboot',
'-smp', str(args.cpus),
'-trace', 'enable={},file={}'.format(trace_type, common.qemu_trace_file),
'-virtfs', 'local,path={},mount_tag=host_scratch,security_model=mapped,id=host_scratch'.format(common.p9_dir),
'-virtfs', 'local,path={},mount_tag=host_out,security_model=mapped,id=host_out'.format(common.build_dir),
'-virtfs', 'local,path={},mount_tag=host_data,security_model=mapped,id=host_data'.format(common.p9_dir),
'-virtfs', 'local,path={},mount_tag=host_out,security_model=mapped,id=host_out'.format(common.out_dir),
'-virtfs', 'local,path={},mount_tag=host_rootfs_overlay,security_model=mapped,id=host_rootfs_overlay'.format(common.rootfs_overlay_dir),
] +
serial_monitor +
vnc

8
run-gdb
View File

@@ -44,15 +44,17 @@ def main(args, extra_args=None):
break_at = []
if args.baremetal is None:
image = common.vmlinux
allowed_toolchains = ['buildroot', 'crosstool-ng', 'host']
else:
image = common.image
allowed_toolchains = ['crosstool-ng', 'buildroot', 'host']
cmd = (
[common.get_toolchain_tool('gdb')] +
[common.get_toolchain_tool('gdb', allowed_toolchains=allowed_toolchains)] +
before +
['-q']
)
if args.baremetal is None:
cmd.extend(['-ex', 'add-auto-load-safe-path {}'.format(common.linux_variant_dir)])
cmd.extend(['-ex', 'add-auto-load-safe-path {}'.format(common.linux_build_dir)])
if args.sim:
target = 'sim'
else:
@@ -85,7 +87,7 @@ def main(args, extra_args=None):
# which gets put on the kernel build root when python debugging scripts are enabled.
cmd.extend(['-ex', 'continue'] + lx_symbols)
cmd.extend(after)
return common.run_cmd(cmd, cmd_file=os.path.join(common.run_dir, 'run-gdb.sh'), cwd=common.linux_variant_dir)
return common.run_cmd(cmd, cmd_file=os.path.join(common.run_dir, 'run-gdb.sh'), cwd=common.linux_build_dir)
if __name__ == '__main__':
parser = common.get_argparse(argparse_args={'description': 'Connect with GDB to an emulator to debug Linux itself'})

9
run-toolchain
View File

@@ -36,10 +36,15 @@ parser.add_argument(
nargs='*'
)
args = common.setup(parser)
if args.baremetal is None:
image = common.vmlinux
else:
image = common.image
tool= common.get_toolchain_tool(args.tool, allowed_toolchains=allowed_toolchains)
if args.dry:
print(common.get_toolchain_tool(args.tool))
print(tool)
else:
sys.exit(common.run_cmd(
[common.get_toolchain_tool(args.tool)] + args.extra_args,
[tool] + args.extra_args,
cmd_file=os.path.join(common.run_dir, 'run-toolchain.sh'),
))

2
test
View File

@@ -1,4 +1,4 @@
#!/usr/bin/env bash
set -eu
./bench-boot -t "${1:-1}"
./bench-boot --size "${1:-1}"
./test-kernel-modules

13
update-buildroot-kernel-configs Executable file
View File

@@ -0,0 +1,13 @@
#!/usr/bin/env bash
set -eux
root_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)"
getvar="${root_dir}/getvar"
for arch in x86_64 arm aarch64; do
linux_arch="$("${getvar}" --arch "$arch" linux_arch)"
linux_buildroot_build_dir="$("${getvar}" --arch "$arch" linux_buildroot_build_dir)"
linux_config_dir="$("${getvar}" --arch "$arch" linux_config_dir)"
"${root_dir}/build-buildroot" --baseline --build-linux --no-all -- linux-configure
cd "$linux_build_dir"
make ARCH="$linux_arch" savedefconfig
cp defconfig "${linux_config_dir}/buildroot-${arch}"
done