3rd-party/linux-kernel-module-cheat
1
0
Fork 0
mirror of https://github.com/cirosantilli/linux-kernel-module-cheat.git synced 2026-01-25 19:21:35 +01:00
Code Issues Releases Wiki Activity

userland: try to make userland executable selection saner

Browse Source 
Only allow existing files to be built, stop extension expansion madness.

cli_function: get_cli print booleans properly, was printing without --no-
for negations.
This commit is contained in:
Ciro Santilli 六四事件 法轮功
2019-05-05 00:00:00 +00:00
parent f1c3b64a55
commit eba97f9cef
11 changed files with 570 additions and 459 deletions
Download Patch File Download Diff File Expand all files Collapse all files

190
README.adoc
View File

@@ -350,7 +350,7 @@ Lol!
We can also test our hacked glibc on <<user-mode-simulation>> with: We can also test our hacked glibc on <<user-mode-simulation>> with:
.... ....
./run --userland c/hello ./run --userland userland/c/hello.c
.... ....
I just noticed that this is actually a good way to develop glibc for other archs. I just noticed that this is actually a good way to develop glibc for other archs.
@@ -427,7 +427,7 @@ Finally, rebuild Binutils, userland and test our program with <<user-mode-simula
.... ....
./build-buildroot -- host-binutils-rebuild ./build-buildroot -- host-binutils-rebuild
./build-userland --static ./build-userland --static
./run --static --userland arch/x86_64/binutils_hack ./run --static --userland userland/arch/x86_64/binutils_hack.c
.... ....
and we se that `myinc` worked since the assert did not fail! and we se that `myinc` worked since the assert did not fail!
@@ -444,7 +444,7 @@ If we run the program link:userland/c/gcc_hack.c[]:
.... ....
./build-userland --static ./build-userland --static
./run --static --userland c/gcc_hack ./run --static --userland userland/c/gcc_hack.c
.... ....
it produces the normal boring output: it produces the normal boring output:
@@ -496,7 +496,7 @@ Now rebuild GCC, the program and re-run it:
.... ....
./build-buildroot -- host-gcc-final-rebuild ./build-buildroot -- host-gcc-final-rebuild
./build-userland --static ./build-userland --static
./run --static --userland c/gcc_hack ./run --static --userland userland/c/gcc_hack.c
.... ....
and the new ouptut is now: and the new ouptut is now:
@@ -1002,8 +1002,6 @@ cd userland/c
./test ./test
.... ....
Note however that if you run this from link:userland/[] toplevel, it would try to build the link:userland/libs/[] folder, which depends on certain libraries being installed on the host, e.g. <<blas>>.
You can install those libraries and do the build in one go with: You can install those libraries and do the build in one go with:
.... ....
@@ -1043,7 +1041,7 @@ The `build` scripts inside link:userland/[] are just symlinks to link:build-user
So you can use any option supported by `build-userland` script freely with `build-userland-in-tree` and `build`. So you can use any option supported by `build-userland` script freely with `build-userland-in-tree` and `build`.
The situation is analogous for link:userland/test[], link:test-user-mode-in-tree[] and link:test-user-mode[]. The situation is analogous for link:userland/test[], link:test-user-mode-in-tree[] and link:test-user-mode[], which are further documented at: <<user-mode-tests>>.
Do a more clean out-of-tree build instead and run the program: Do a more clean out-of-tree build instead and run the program:
@@ -1123,7 +1121,7 @@ For example, to run link:baremetal/hello.c[] in QEMU do:
.... ....
./build --arch aarch64 --download-dependencies qemu-baremetal ./build --arch aarch64 --download-dependencies qemu-baremetal
./run --arch aarch64 --baremetal hello ./run --arch aarch64 --baremetal baremetal/hello.c
.... ....
The terminal prints: The terminal prints:
@@ -1135,7 +1133,7 @@ hello
Now let's run link:baremetal/arch/aarch64/add.S[]: Now let's run link:baremetal/arch/aarch64/add.S[]:
.... ....
./run --arch aarch64 --baremetal arch/aarch64/add ./run --arch aarch64 --baremetal baremetal/arch/aarch64/add.S
.... ....
This time, the terminal does not print anything, which indicates success. This time, the terminal does not print anything, which indicates success.
@@ -1145,7 +1143,7 @@ If you look into the source, you will see that we just have an assertion there.
You can see a sample assertion fail in link:baremetal/interactive/assert_fail.c[]: You can see a sample assertion fail in link:baremetal/interactive/assert_fail.c[]:
.... ....
./run --arch aarch64 --baremetal interactive/assert_fail ./run --arch aarch64 --baremetal baremetal/interactive/assert_fail.c
.... ....
and the terminal contains: and the terminal contains:
@@ -1161,7 +1159,7 @@ and the exit status of our script is 1:
echo $? echo $?
.... ....
To modify a baremetal program, simply edit the file, .g. To modify a baremetal program, simply edit the file, e.g.
.... ....
vim baremetal/hello.c vim baremetal/hello.c
@@ -1170,11 +1168,11 @@ vim baremetal/hello.c
and rebuild: and rebuild:
.... ....
./build --arch aarch64 --download-dependencies qemu-baremetal ./build-baremetal --arch aarch64
./run --arch aarch64 --baremetal hello ./run --arch aarch64 --baremetal baremetal/hello.c
.... ....
`./build qemu-baremetal` had called link:build-baremetal[] for us previously, in addition to its requirements. `./build qemu-baremetal` that we run previously is only needed for the initial build. That script calls link:build-baremetal[] for us, in addition to building prerequisites such as QEMU and crosstool-NG.
`./build-baremetal` uses crosstool-NG, and so it must be preceded by link:build-crosstool-ng[], which `./build qemu-baremetal` also calls. `./build-baremetal` uses crosstool-NG, and so it must be preceded by link:build-crosstool-ng[], which `./build qemu-baremetal` also calls.
@@ -1188,14 +1186,14 @@ Alternatively, for the sake of tab completion, we also accept relative paths ins
Absolute paths however are used as is and must point to the actual executable: Absolute paths however are used as is and must point to the actual executable:
.... ....
./run --arch aarch64 --baremetal "$(./getvar --arch aarch64 baremetal_build_dir)/exit.elf" ./run --arch aarch64 --baremetal "$(./getvar --arch aarch64 baremetal_build_dir)/hello.elf"
.... ....
To use gem5 instead of QEMU do: To use gem5 instead of QEMU do:
.... ....
./build --download-dependencies gem5-baremetal ./build --download-dependencies gem5-baremetal
./run --arch aarch64 --baremetal interactive/prompt --emulator gem5 ./run --arch aarch64 --baremetal baremetal/hello.c --emulator gem5
.... ....
and then <<qemu-buildroot-setup,as usual>> open a shell with: and then <<qemu-buildroot-setup,as usual>> open a shell with:
@@ -1207,7 +1205,7 @@ and then <<qemu-buildroot-setup,as usual>> open a shell with:
Or as usual, <<tmux>> users can do both in one go with: Or as usual, <<tmux>> users can do both in one go with:
.... ....
./run --arch aarch64 --baremetal interactive/prompt --emulator gem5 --tmux ./run --arch aarch64 --baremetal baremetal/hello.c --emulator gem5 --tmux
.... ....
TODO: the carriage returns are a bit different than in QEMU, see: <<gem5-baremetal-carriage-return>>. TODO: the carriage returns are a bit different than in QEMU, see: <<gem5-baremetal-carriage-return>>.
@@ -1215,8 +1213,8 @@ TODO: the carriage returns are a bit different than in QEMU, see: <<gem5-baremet
Note that `./build-baremetal` requires the `--emulator gem5` option, and generates separate executable images for both, as can be seen from: Note that `./build-baremetal` requires the `--emulator gem5` option, and generates separate executable images for both, as can be seen from:
.... ....
echo "$(./getvar --arch aarch64 --baremetal interactive/prompt --emulator qemu image)" echo "$(./getvar --arch aarch64 --baremetal baremetal/hello.c --emulator qemu image)"
echo "$(./getvar --arch aarch64 --baremetal interactive/prompt --emulator gem5 image)" echo "$(./getvar --arch aarch64 --baremetal baremetal/hello.c --emulator gem5 image)"
.... ....
This is unlike the Linux kernel that has a single image for both QEMU and gem5: This is unlike the Linux kernel that has a single image for both QEMU and gem5:
@@ -1232,14 +1230,14 @@ The reason for that is that on baremetal we don't parse the <<device-tree,device
.... ....
./build-baremetal --arch aarch64 --emulator gem5 --machine RealViewPBX ./build-baremetal --arch aarch64 --emulator gem5 --machine RealViewPBX
./run --arch aarch64 --baremetal interactive/prompt --emulator gem5 --machine RealViewPBX ./run --arch aarch64 --baremetal baremetal/hello.c --emulator gem5 --machine RealViewPBX
.... ....
This generates yet new separate images with new magic constants: This generates yet new separate images with new magic constants:
.... ....
echo "$(./getvar --arch aarch64 --baremetal interactive/prompt --emulator gem5 --machine VExpress_GEM5_V1 image)" echo "$(./getvar --arch aarch64 --baremetal baremetal/hello.c --emulator gem5 --machine VExpress_GEM5_V1 image)"
echo "$(./getvar --arch aarch64 --baremetal interactive/prompt --emulator gem5 --machine RealViewPBX image)" echo "$(./getvar --arch aarch64 --baremetal baremetal/hello.c --emulator gem5 --machine RealViewPBX image)"
.... ....
But just stick to newer and better `VExpress_GEM5_V1` unless you have a good reason to use `RealViewPBX`. But just stick to newer and better `VExpress_GEM5_V1` unless you have a good reason to use `RealViewPBX`.
@@ -3454,7 +3452,7 @@ Let's run link:userland/c/print_argv.c[] built with the Buildroot toolchain on Q
.... ....
./build user-mode-qemu ./build user-mode-qemu
./run \ ./run \
--userland c/print_argv \ --userland userland/c/print_argv.c \
--userland-args='asdf "qw er"' \ --userland-args='asdf "qw er"' \
; ;
.... ....
@@ -3485,7 +3483,7 @@ It's nice when <<gdb,the obvious>> just works, right?
./run \ ./run \
--arch aarch64 \ --arch aarch64 \
--gdb-wait \ --gdb-wait \
--userland c/print_argv \ --userland userland/c/print_argv.c \
--userland-args 'asdf "qw er"' \ --userland-args 'asdf "qw er"' \
; ;
.... ....
@@ -3495,7 +3493,7 @@ and on another shell:
.... ....
./run-gdb \ ./run-gdb \
--arch aarch64 \ --arch aarch64 \
--userland c/print_argv \ --userland userland/c/print_argv.c \
main \ main \
; ;
.... ....
@@ -3506,7 +3504,7 @@ Or alternatively, if you are using <<tmux>>, do everything in one go with:
./run \ ./run \
--arch aarch64 \ --arch aarch64 \
--gdb \ --gdb \
--userland c/print_argv \ --userland userland/c/print_argv.c \
--userland-args 'asdf "qw er"' \ --userland-args 'asdf "qw er"' \
; ;
.... ....
@@ -3538,6 +3536,8 @@ This script skips a manually configured list of tests, notably:
* tests that take perceptible ammounts of time * tests that take perceptible ammounts of time
* known bugs we didn't have time to fix ;-) * known bugs we didn't have time to fix ;-)
Tests under link:userland/libs/[] depend on certain libraries being available on the target, e.g. <<blas>> for link:userland/libs/blas[]. They are not run by default, but can be enabled with `--has-package` and `--has-all-packages`.
The gem5 tests require building statically with build id `static`, see also: <<gem5-syscall-emulation-mode>>. TODO automate this better. The gem5 tests require building statically with build id `static`, see also: <<gem5-syscall-emulation-mode>>. TODO automate this better.
See: <<test-this-repo>> for more useful testing tips. See: <<test-this-repo>> for more useful testing tips.
@@ -3570,7 +3570,7 @@ sudo apt-get install gcc-aarch64-linux-gnu qemu-system-aarch64
--arch aarch64 \ --arch aarch64 \
--qemu-which host --qemu-which host
--userland-build-id host \ --userland-build-id host \
--userland c/print_argv \ --userland userland/c/print_argv.c \
--userland-args 'asdf "qw er"' \ --userland-args 'asdf "qw er"' \
; ;
.... ....
@@ -3603,7 +3603,7 @@ We don't have this failure for QEMU on an 18.04 host, only gem5.
QEMU by default copies the host `uname` value. However, our scripts set it by default to our the latest Buildroot kernel version with QEMU's `-r` option, which is exposed as `--kernel-version`: QEMU by default copies the host `uname` value. However, our scripts set it by default to our the latest Buildroot kernel version with QEMU's `-r` option, which is exposed as `--kernel-version`:
.... ....
./run --arch aarch64 --kernel-version 4.18 --userland ./posix/uname ./run --arch aarch64 --kernel-version 4.18 --userland userland/posix/uname.c
.... ....
Source: link:userland/posix/uname.c[]. Source: link:userland/posix/uname.c[].
@@ -3633,8 +3633,8 @@ Reproduction:
.... ....
rm -f "$(./getvar buildroot_target_dir)/etc/ld.so.cache" rm -f "$(./getvar buildroot_target_dir)/etc/ld.so.cache"
./run --userland c/hello ./run --userland userland/c/hello.c
./run --userland c/hello --qemu-which host ./run --userland userland/c/hello.c --qemu-which host
.... ....
Outcome: Outcome:
@@ -3672,7 +3672,7 @@ Example:
./run \ ./run \
--arch aarch64 \ --arch aarch64 \
--static \ --static \
--userland c/print_argv \ --userland userland/c/print_argv.c \
--userland-args 'asdf "qw er"' \ --userland-args 'asdf "qw er"' \
; ;
.... ....
@@ -3734,7 +3734,7 @@ So let's just play with some static ones:
./run \ ./run \
--arch aarch64 \ --arch aarch64 \
--emulator gem5 \ --emulator gem5 \
--userland c/print_argv \ --userland userland/c/print_argv.c \
--userland-args 'asdf "qw er"' \ --userland-args 'asdf "qw er"' \
; ;
.... ....
@@ -3749,14 +3749,14 @@ TODO: how to escape spaces on the command line arguments?
--emulator gem5 \ --emulator gem5 \
--gdb-wait \ --gdb-wait \
--static \ --static \
--userland c/print_argv \ --userland userland/c/print_argv.c \
--userland-args 'asdf "qw er"' \ --userland-args 'asdf "qw er"' \
; ;
./run-gdb \ ./run-gdb \
--arch aarch64 \ --arch aarch64 \
--emulator gem5 \ --emulator gem5 \
--static \ --static \
--userland c/print_argv \ --userland userland/c/print_argv.c \
main \ main \
; ;
.... ....
@@ -3766,7 +3766,7 @@ TODO: how to escape spaces on the command line arguments?
As of gem5 7fa4c946386e7207ad5859e8ade0bbfc14000d91, the crappy `se.py` script does not forward the exit status of syscall emulation mode, you can test it with: As of gem5 7fa4c946386e7207ad5859e8ade0bbfc14000d91, the crappy `se.py` script does not forward the exit status of syscall emulation mode, you can test it with:
.... ....
./run --dry-run --emulator gem5 --static --userland c/false ./run --dry-run --emulator gem5 --static --userland userland/c/false.c
.... ....
Source: link:userland/c/false.c[]. Source: link:userland/c/false.c[].
@@ -3890,6 +3890,28 @@ The same can be reproduced by copying the raw QEMU command and piping it through
TODO: investigate further and then possibly post on QEMU mailing list. TODO: investigate further and then possibly post on QEMU mailing list.
===== QEMU user mode does not show errors
Similarly to <<qemu-user-mode-does-not-show-stdout-immediately>>, QEMU error messages do not show at all through pipes.
In particular, it does not say anything if you pass it a non-existing executable:
....
./build-userland --clean userland/c/hello.c
./run --userland userland/c/hello.c
echo $?
....
does not output anything, except for the `1` exit status.
If you run however the raw command without a pipe manually, it shows a helpful error message:
....
Error while loading /path/to/linux-kernel-module-cheat/out/userland/default/x86_64/c/hello.out: No such file or directory
....
Tested in de77c62c091f6418e73b64e8a0a19639c587a103 + 1.
== Kernel module utilities == Kernel module utilities
=== insmod === insmod
@@ -10017,7 +10039,7 @@ Usage:
--arch aarch64 \ --arch aarch64 \
--emulator gem5 \ --emulator gem5 \
--static \ --static \
--userland cpp/bst_vs_heap \ --userland userland/cpp/bst_vs_heap.cpp \
--userland-args='1000' \ --userland-args='1000' \
; ;
./bst-vs-heap --arch aarch64 > bst_vs_heap.dat ./bst-vs-heap --arch aarch64 > bst_vs_heap.dat
@@ -11512,13 +11534,13 @@ This section will document ISA generic ideas. ISA specifics are documented on th
The first example that you want to run for each arch is: The first example that you want to run for each arch is:
.... ....
./run --userland arch/<arch>/add ./run --userland userland/arch/<arch>/add.S
.... ....
e.g.: e.g.:
.... ....
./run --userland arch/x86_64/add ./run --userland userland/arch/x86_64/add.S
.... ....
Sources: Sources:
@@ -11541,7 +11563,7 @@ And then watch the assertion fail:
.... ....
./build-userland ./build-userland
./run --userland arch/x86_64/add ./run --userland userland/arch/x86_64/add.S
.... ....
with error message: with error message:
@@ -11598,8 +11620,8 @@ Such executables are called freestanding because they don't execute the glibc in
In order to GDB step debug those executables, you will want to use `--no-continue`, e.g.: In order to GDB step debug those executables, you will want to use `--no-continue`, e.g.:
.... ....
./run --arch aarch64 --userland arch/aarch64/freestanding/hello --gdb-wait ./run --arch aarch64 --userland userland/arch/aarch64/freestanding/linux/hello.S --gdb-wait
./run-gdb --arch aarch64 --no-continue --userland arch/aarch64/freestanding/hello ./run-gdb --arch aarch64 --no-continue --userland userland/arch/aarch64/freestanding/linux/hello.S
.... ....
You are now left on the very first instruction of our tiny executable! You are now left on the very first instruction of our tiny executable!
@@ -11774,24 +11796,26 @@ Getting started at: <<baremetal-setup>>
=== Baremetal GDB step debug === Baremetal GDB step debug
GDB step debug works on baremetal exactly as it does on the Linux kernel, except that is is even cooler here since we can easily control and understand every single instruction that is being run! GDB step debug works on baremetal exactly as it does on the Linux kernel: <<gdb>>.
Except that is is even cooler here since we can easily control and understand every single instruction that is being run!
For example, on the first shell: For example, on the first shell:
.... ....
./run --arch arm --baremetal interactive/prompt --gdb-wait ./run --arch arm --baremetal baremetal/hello.c --gdb-wait
.... ....
then on the second shell: then on the second shell:
.... ....
./run-gdb --arch arm --baremetal interactive/prompt -- main ./run-gdb --arch arm --baremetal baremetal/hello.c -- main
.... ....
Or if you are a <<tmux,tmux pro>>, do everything in one go with: Or if you are a <<tmux,tmux pro>>, do everything in one go with:
.... ....
./run --arch arm --baremetal interactive/prompt --gdb-wait --tmux-args main ./run --arch arm --baremetal baremetal/hello.c --gdb
.... ....
Alternatively, to start from the very first executed instruction of our tiny <<baremetal-bootloaders>>: Alternatively, to start from the very first executed instruction of our tiny <<baremetal-bootloaders>>:
@@ -11799,20 +11823,20 @@ Alternatively, to start from the very first executed instruction of our tiny <<b
.... ....
./run \ ./run \
--arch arm \ --arch arm \
--baremetal interactive/prompt \ --baremetal baremetal/hello.c \
--gdb-wait \ --gdb-wait \
--tmux-args=--no-continue \ --tmux-args=--no-continue \
; ;
.... ....
Now you can just `stepi` to when jumping into main to go to the C code in link:baremetal/interactive/prompt.c[]. Now you can just `stepi` to when jumping into main to go to the C code in link:baremetal/hello.c[].
This is specially interesting for the executables that don't use the bootloader from under `baremetal/arch/<arch>/no_bootloader/*.S`, e.g.: This is specially interesting for the executables that don't use the bootloader from under `baremetal/arch/<arch>/no_bootloader/*.S`, e.g.:
.... ....
./run \ ./run \
--arch arm \ --arch arm \
--baremetal arch/arm/no_bootloader/semihost_exit \ --baremetal baremetal/arch/arm/no_bootloader/semihost_exit.S \
--gdb-wait \ --gdb-wait \
--tmux-args=--no-continue \ --tmux-args=--no-continue \
; ;
@@ -11858,7 +11882,7 @@ It is documented at: https://developer.arm.com/docs/100863/latest/introduction
For example, the following code makes QEMU exit: For example, the following code makes QEMU exit:
.... ....
./run --arch arm --baremetal arch/arm/semihost_exit ./run --arch arm --baremetal baremetal/arch/arm/semihost_exit.S
.... ....
Source: link:baremetal/arch/arm/no_bootloader/semihost_exit.S[] Source: link:baremetal/arch/arm/no_bootloader/semihost_exit.S[]
@@ -11942,7 +11966,7 @@ For `arm`, some baremetal examples compile fine with:
.... ....
sudo apt-get install gcc-arm-none-eabi qemu-system-arm sudo apt-get install gcc-arm-none-eabi qemu-system-arm
./build-baremetal --arch arm --gcc-which host-baremetal ./build-baremetal --arch arm --gcc-which host-baremetal
./run --arch arm --baremetal interactive/prompt --qemu-which host ./run --arch arm --baremetal baremetal/hello.c --qemu-which host
.... ....
However, there are as usual limitations to using prebuilts: However, there are as usual limitations to using prebuilts:
@@ -11971,7 +11995,7 @@ TODO: any advantage over QEMU? I doubt it, mostly using it as as toy for now:
Without running `./run`, do directly: Without running `./run`, do directly:
.... ....
./run-gdb --arch arm --baremetal interactive/prompt --sim ./run-gdb --arch arm --baremetal baremetal/hello.c --sim
.... ....
Then inside GDB: Then inside GDB:
@@ -12039,8 +12063,8 @@ ARM exception levels are analogous to x86 <<ring0,rings>>.
Print the EL at the beginning of a baremetal simulation: Print the EL at the beginning of a baremetal simulation:
.... ....
./run --arch arm --baremetal arch/arm/el ./run --arch arm --baremetal baremetal/arch/arm/el.c
./run --arch aarch64 --baremetal arch/aarch64/el ./run --arch aarch64 --baremetal baremetal/arch/aarch64/el.c
.... ....
Sources: Sources:
@@ -12055,12 +12079,12 @@ The lower ELs are not mandated by the architecture, and can be controlled throug
In QEMU, you can configure the lowest EL as explained at https://stackoverflow.com/questions/42824706/qemu-system-aarch64-entering-el1-when-emulating-a53-power-up In QEMU, you can configure the lowest EL as explained at https://stackoverflow.com/questions/42824706/qemu-system-aarch64-entering-el1-when-emulating-a53-power-up
.... ....
./run --arch arm --baremetal arch/arm/el ./run --arch arm --baremetal baremetal/arch/arm/el.c
./run --arch arm --baremetal arch/arm/el -- -machine virtualization=on ./run --arch arm --baremetal baremetal/arch/arm/el.c -- -machine virtualization=on
./run --arch arm --baremetal arch/arm/el -- -machine secure=on ./run --arch arm --baremetal baremetal/arch/arm/el.c -- -machine secure=on
./run --arch aarch64 --baremetal arch/aarch64/el ./run --arch aarch64 --baremetal baremetal/arch/aarch64/el.c
./run --arch aarch64 --baremetal arch/aarch64/el -- -machine virtualization=on ./run --arch aarch64 --baremetal baremetal/arch/aarch64/el.c -- -machine virtualization=on
./run --arch aarch64 --baremetal arch/aarch64/el -- -machine secure=on ./run --arch aarch64 --baremetal baremetal/arch/aarch64/el.c -- -machine secure=on
.... ....
outputs respectively: outputs respectively:
@@ -12079,17 +12103,17 @@ TODO: why is `arm` stuck at `19` which equals Supervisor mode?
In gem5, you can configure the lowest EL with: In gem5, you can configure the lowest EL with:
.... ....
./run --arch arm --baremetal arch/arm/el --emulator gem5 ./run --arch arm --baremetal baremeta/arch/arm/el.c --emulator gem5
cat "$(./getvar --arch arm --emulator gem5 gem5_guest_terminal_file)" cat "$(./getvar --arch arm --emulator gem5 gem5_guest_terminal_file)"
./run --arch arm --baremetal arch/arm/el --emulator gem5 -- --param 'system.have_virtualization = True' ./run --arch arm --baremetal baremetal/arch/arm/el.c --emulator gem5 -- --param 'system.have_virtualization = True'
cat "$(./getvar --arch arm --emulator gem5 gem5_guest_terminal_file)" cat "$(./getvar --arch arm --emulator gem5 gem5_guest_terminal_file)"
./run --arch arm --baremetal arch/arm/el --emulator gem5 -- --param 'system.have_security = True' ./run --arch arm --baremetal baremetal/arch/arm/el.c --emulator gem5 -- --param 'system.have_security = True'
cat "$(./getvar --arch arm --emulator gem5 gem5_guest_terminal_file)" cat "$(./getvar --arch arm --emulator gem5 gem5_guest_terminal_file)"
./run --arch aarch64 --baremetal arch/aarch64/el --emulator gem5 ./run --arch aarch64 --baremetal baremetal/arch/aarch64/el.c --emulator gem5
cat "$(./getvar --arch aarch64 --emulator gem5 gem5_guest_terminal_file)" cat "$(./getvar --arch aarch64 --emulator gem5 gem5_guest_terminal_file)"
./run --arch aarch64 --baremetal arch/aarch64/el --emulator gem5 -- --param 'system.have_virtualization = True' ./run --arch aarch64 --baremetal baremetal/arch/aarch64/el.c --emulator gem5 -- --param 'system.have_virtualization = True'
cat "$(./getvar --arch aarch64 --emulator gem5 gem5_guest_terminal_file)" cat "$(./getvar --arch aarch64 --emulator gem5 gem5_guest_terminal_file)"
./run --arch aarch64 --baremetal arch/aarch64/el --emulator gem5 -- --param 'system.have_security = True' ./run --arch aarch64 --baremetal baremetal/arch/aarch64/el.c --emulator gem5 -- --param 'system.have_security = True'
cat "$(./getvar --arch aarch64 --emulator gem5 gem5_guest_terminal_file)" cat "$(./getvar --arch aarch64 --emulator gem5 gem5_guest_terminal_file)"
.... ....
@@ -12111,14 +12135,14 @@ This is the most basic example of exception handling we have.
We a handler for `svc`, do an `svc`, and observe that the handler got called and returned from C and assembly: We a handler for `svc`, do an `svc`, and observe that the handler got called and returned from C and assembly:
.... ....
./run --arch aarch64 --baremetal arch/aarch64/svc ./run --arch aarch64 --baremetal baremetal/arch/aarch64/svc.c
./run --arch aarch64 --baremetal arch/aarch64/svc_asm ./run --arch aarch64 --baremetal baremetal/arch/aarch64/svc_asm.S
.... ....
Sources: Sources:
* link:baremetal/arch/aarch64/svc_asm.S[]
* link:baremetal/arch/aarch64/svc.c[] * link:baremetal/arch/aarch64/svc.c[]
* link:baremetal/arch/aarch64/svc_asm.S[]
Sample output for the C one: Sample output for the C one:
@@ -12171,7 +12195,7 @@ Both QEMU and gem5 are able to trace interrupts in addition to instructions, and
.... ....
./run \ ./run \
--arch aarch64 \ --arch aarch64 \
--baremetal arch/aarch64/svc_asm --baremetal baremetal/arch/aarch64/svc_asm.S
-- -d in_asm,int \ -- -d in_asm,int \
; ;
.... ....
@@ -12198,7 +12222,7 @@ and:
.... ....
./run \ ./run \
--arch aarch64 \ --arch aarch64 \
--baremetal arch/aarch64/svc_asm \ --baremetal baremetal/arch/aarch64/svc_asm.S \
--trace ExecAll,Faults \ --trace ExecAll,Faults \
--trace-stdout \ --trace-stdout \
; ;
@@ -12270,10 +12294,10 @@ Bibliography:
==== ARM multicore ==== ARM multicore
.... ....
./run --arch aarch64 --baremetal arch/aarch64/multicore --cpus 2 ./run --arch aarch64 --baremetal baremetal/arch/aarch64/multicore.S --cpus 2
./run --arch aarch64 --baremetal arch/aarch64/multicore --cpus 2 --emulator gem5 ./run --arch aarch64 --baremetal baremetal/arch/aarch64/multicore.S --cpus 2 --emulator gem5
./run --arch arm --baremetal arch/aarch64/multicore --cpus 2 ./run --arch arm --baremetal baremetal/arch/aarch64/multicore.S --cpus 2
./run --arch arm --baremetal arch/aarch64/multicore --cpus 2 --emulator gem5 ./run --arch arm --baremetal baremetal/arch/aarch64/multicore.S --cpus 2 --emulator gem5
.... ....
Sources: Sources:
@@ -12288,7 +12312,7 @@ So, we need CPU 1 to come to the rescue and set that memory address to `1`, othe
Don't believe me? Then try: Don't believe me? Then try:
.... ....
./run --arch aarch64 --baremetal arch/aarch64/multicore --cpus 1 ./run --arch aarch64 --baremetal baremetal/arch/aarch64/multicore.S --cpus 1
.... ....
and watch it hang forever. and watch it hang forever.
@@ -12296,7 +12320,7 @@ and watch it hang forever.
Note that if you try the same thing on gem5: Note that if you try the same thing on gem5:
.... ....
./run --arch aarch64 --baremetal arch/aarch64/multicore --cpus 1 --emulator gem5 ./run --arch aarch64 --baremetal baremetal/arch/aarch64/multicore.S --cpus 1 --emulator gem5
.... ....
then the gem5 actually exits, but with a different message: then the gem5 actually exits, but with a different message:
@@ -13888,14 +13912,14 @@ Sources:
If a test fails, re-run the test commands manually and use `--verbose` to understand what happened: If a test fails, re-run the test commands manually and use `--verbose` to understand what happened:
.... ....
./run --arch arm --background --baremetal add --gdb-wait & ./run --arch arm --background --baremetal baremetal/add.c --gdb-wait &
./run-gdb --arch arm --baremetal add --verbose -- main ./run-gdb --arch arm --baremetal baremetal/add.c --verbose -- main
.... ....
and possibly repeat the GDB steps manually with the usual: and possibly repeat the GDB steps manually with the usual:
.... ....
./run-gdb --arch arm --baremetal add --no-continue --verbose ./run-gdb --arch arm --baremetal baremetal/add.c --no-continue --verbose
.... ....
To debug GDB problems on gem5, you might want to enable the following <<gem5-tracing,tracing>> options: To debug GDB problems on gem5, you might want to enable the following <<gem5-tracing,tracing>> options:
@@ -13903,7 +13927,7 @@ To debug GDB problems on gem5, you might want to enable the following <<gem5-tra
.... ....
./run \ ./run \
--arch arm \ --arch arm \
--baremetal add \ --baremetal baremetal/add.c \
--gdb-wait \ --gdb-wait \
--trace GDBRecv,GDBSend \ --trace GDBRecv,GDBSend \
--trace-stdout \ --trace-stdout \
@@ -13960,6 +13984,12 @@ You should also test that the Internet works:
./run --arch x86_64 --kernel-cli '- lkmc_eval="ifup -a;wget -S google.com;poweroff;"' ./run --arch x86_64 --kernel-cli '- lkmc_eval="ifup -a;wget -S google.com;poweroff;"'
.... ....
===== test-build-userland
`build-userland` has a wide variety of target selection modes, and it was hard to keep them all working without a test.
So we've created the simple: link:test-build-userland[] to ensure that at least none of the build modes blows up.
=== Bisection === Bisection
When updating the Linux kernel, QEMU and gem5, things sometimes break. When updating the Linux kernel, QEMU and gem5, things sometimes break.

267
build-userland
View File

@@ -8,6 +8,7 @@ import threading
from shell_helpers import LF from shell_helpers import LF
import common import common
from thread_pool import ThreadPool from thread_pool import ThreadPool
import path_properties
class Main(common.BuildCliFunction): class Main(common.BuildCliFunction):
def __init__(self, *args, **kwargs): def __init__(self, *args, **kwargs):
@@ -16,12 +17,9 @@ class Main(common.BuildCliFunction):
Build our compiled userland examples. Build our compiled userland examples.
''' '''
super().__init__(*args, **kwargs) super().__init__(*args, **kwargs)
self.default_cstd = 'c11'
self.default_cxxstd = 'c++17'
self.add_argument( self.add_argument(
'--has-package', '--has-package',
action='append', action='append',
default=[],
help='''\ help='''\
Indicate that a given package is present in the root filesystem, which Indicate that a given package is present in the root filesystem, which
allows us to build examples that rely on it. allows us to build examples that rely on it.
@@ -32,26 +30,26 @@ allows us to build examples that rely on it.
action='store_true', action='store_true',
help='''\ help='''\
Indicate that all packages from --has-package are available. Indicate that all packages from --has-package are available.
''',
)
self.add_argument(
'--in-tree',
default=False,
help='''\
Place build output inside soure tree to conveniently run it, especially when
building with the host toolchain.
''', ''',
) )
self.add_argument( self.add_argument(
'targets', 'targets',
default=[], default=[],
help='''\ help='''\
Build only the given userland programs or all programs in the given directories. Select to build only the given userland programs, or all programs under
the given directories.
Default: build all.
Must point to either sources or directories under userland/, or to LKMC
toplevel which is a synonym for userland/.
Default: build all examples that have their package dependencies met, e.g.: Default: build all examples that have their package dependencies met, e.g.:
- userland/arch/ programs only build if the target arch matches - userland/arch/ programs only build if the target arch matches
- an OpenBLAS example can only be built if the target root filesystem - an OpenBLAS example can only be built if the target root filesystem
has the OpenBLAS libraries and headers installed, which you must inform with --has-package has the OpenBLAS libraries and headers installed, which you must inform
with --has-package
''', ''',
nargs='*', nargs='*',
) )
@@ -63,10 +61,10 @@ Default: build all examples that have their package dependencies met, e.g.:
self, self,
in_path, in_path,
out_path, out_path,
ccflags, cc_flags,
ccflags_after=None, cc_flags_after=None,
cstd=None, c_std=None,
cxxstd=None, cxx_std=None,
extra_deps=None, extra_deps=None,
extra_objs=None, extra_objs=None,
link=True, link=True,
@@ -75,61 +73,57 @@ Default: build all examples that have their package dependencies met, e.g.:
extra_deps = [] extra_deps = []
if extra_objs is None: if extra_objs is None:
extra_objs = [] extra_objs = []
if ccflags_after is None: if cc_flags_after is None:
ccflags_after = [] cc_flags_after = []
ret = 0 ret = 0
if self.need_rebuild([in_path] + extra_objs + extra_deps, out_path): if self.need_rebuild([in_path] + extra_objs + extra_deps, out_path):
ccflags = ccflags.copy() cc_flags = cc_flags.copy()
if not link: if not link:
ccflags.extend(['-c', LF]) cc_flags.extend(['-c', LF])
in_ext = os.path.splitext(in_path)[1] in_ext = os.path.splitext(in_path)[1]
do_compile = True
if in_ext in (self.env['c_ext'], self.env['asm_ext']): if in_ext in (self.env['c_ext'], self.env['asm_ext']):
cc = self.env['gcc'] cc = self.env['gcc']
if cstd is None: if c_std is None:
std = self.default_cstd std = path_properties.default_c_std
else: else:
std = cstd std = c_std
ccflags.extend([ cc_flags.extend([
'-fopenmp', LF, '-fopenmp', LF,
]) ])
elif in_ext == self.env['cxx_ext']: elif in_ext == self.env['cxx_ext']:
cc = self.env['gxx'] cc = self.env['gxx']
if cxxstd is None: if cxx_std is None:
std = self.default_cxxstd std = path_properties.default_cxx_std
else: else:
std = cxxstd std = cxx_std
else: os.makedirs(os.path.dirname(out_path), exist_ok=True)
do_compile = False ret = self.sh.run_cmd(
if do_compile: (
os.makedirs(os.path.dirname(out_path), exist_ok=True) [
ret = self.sh.run_cmd( cc, LF,
( ] +
[ cc_flags +
cc, LF, [
] + '-std={}'.format(std), LF,
ccflags + '-o', out_path, LF,
[ in_path, LF,
'-std={}'.format(std), LF, ] +
'-o', out_path, LF, self.sh.add_newlines(extra_objs) +
in_path, LF, [
] + '-lm', LF,
self.sh.add_newlines(extra_objs) + '-pthread', LF,
[ ] +
'-lm', LF, cc_flags_after
'-pthread', LF, ),
] + extra_paths=[self.env['ccache_dir']],
ccflags_after )
),
extra_paths=[self.env['ccache_dir']],
)
return ret return ret
def build(self): def build(self):
build_dir = self.get_build_dir() build_dir = self.get_build_dir()
has_packages = set(self.env['has_package']) has_packages = set(self.env['has_package'])
has_all_packages = self.env['has_all_packages'] has_all_packages = self.env['has_all_packages']
ccflags = [ cc_flags = [
'-I', self.env['root_dir'], LF, '-I', self.env['root_dir'], LF,
'-O{}'.format(self.env['optimization_level']), LF, '-O{}'.format(self.env['optimization_level']), LF,
'-Wall', LF, '-Wall', LF,
@@ -139,7 +133,7 @@ Default: build all examples that have their package dependencies met, e.g.:
'-ggdb3', LF, '-ggdb3', LF,
] + self.sh.shlex_split(self.env['ccflags']) ] + self.sh.shlex_split(self.env['ccflags'])
if self.env['static']: if self.env['static']:
ccflags.extend(['-static', LF]) cc_flags.extend(['-static', LF])
common_obj = os.path.join( common_obj = os.path.join(
build_dir, build_dir,
self.env['common_basename_noext'] + self.env['obj_ext'] self.env['common_basename_noext'] + self.env['obj_ext']
@@ -147,7 +141,7 @@ Default: build all examples that have their package dependencies met, e.g.:
self._build_one( self._build_one(
in_path=self.env['common_c'], in_path=self.env['common_c'],
out_path=common_obj, out_path=common_obj,
ccflags=ccflags, cc_flags=cc_flags,
extra_deps=[self.env['common_h']], extra_deps=[self.env['common_h']],
link=False, link=False,
) )
@@ -166,7 +160,7 @@ Default: build all examples that have their package dependencies met, e.g.:
common_obj_asm_relpath common_obj_asm_relpath
), ),
out_path=common_obj_asm, out_path=common_obj_asm,
ccflags=ccflags, cc_flags=cc_flags,
extra_deps=[self.env['common_h']], extra_deps=[self.env['common_h']],
link=False, link=False,
) )
@@ -180,7 +174,7 @@ Default: build all examples that have their package dependencies met, e.g.:
# fatal error: Eigen/Dense: No such file or directory as of # fatal error: Eigen/Dense: No such file or directory as of
# 975ce0723ee3fa1fea1766e6683e2f3acb8558d6 # 975ce0723ee3fa1fea1766e6683e2f3acb8558d6
# http://lists.busybox.net/pipermail/buildroot/2018-June/222914.html # http://lists.busybox.net/pipermail/buildroot/2018-June/222914.html
'ccflags': [ 'cc_flags': [
'-I', '-I',
os.path.join( os.path.join(
eigen_root, eigen_root,
@@ -191,7 +185,7 @@ Default: build all examples that have their package dependencies met, e.g.:
LF LF
], ],
# Header only. # Header only.
'ccflags_after': [], 'cc_flags_after': [],
}, },
'libdrm': {}, 'libdrm': {},
'openblas': {}, 'openblas': {},
@@ -202,46 +196,30 @@ Default: build all examples that have their package dependencies met, e.g.:
nthreads=self.env['nproc'], nthreads=self.env['nproc'],
) as thread_pool: ) as thread_pool:
try: try:
for path, in_dirnames, in_filenames in self.walk_source_targets( for target in self.env['targets']:
self.env['targets'], for path, in_dirnames, in_filenames in self.sh.walk(target):
self.env['userland_in_exts'] path_abs = os.path.abspath(path)
): dirpath_relative_root = path_abs[rootdir_abs_len + 1:]
path_abs = os.path.abspath(path) dirpath_relative_root_components = dirpath_relative_root.split(os.sep)
dirpath_relative_root = path_abs[rootdir_abs_len + 1:] dirpath_relative_root_components_len = len(dirpath_relative_root_components)
dirpath_relative_root_components = dirpath_relative_root.split(os.sep) out_dir = os.path.join(
dirpath_relative_root_components_len = len(dirpath_relative_root_components) build_dir,
out_dir = os.path.join( dirpath_relative_root
build_dir, )
dirpath_relative_root common_objs_dir = [common_obj]
) cc_flags_after = []
common_objs_dir = [common_obj] cc_flags_dir = cc_flags.copy()
ccflags_after = [] if dirpath_relative_root_components_len > 0:
ccflags_dir = ccflags.copy()
if dirpath_relative_root_components_len > 0:
if dirpath_relative_root_components[0] in (
'gcc',
'kernel_modules',
'linux',
):
cstd = 'gnu11'
cxxstd = 'gnu++17'
else:
cstd = self.default_cstd
cxxstd = self.default_cxxstd
# -pedantic complains even if we use -std=gnu11.
ccflags_dir.extend(['-pedantic', LF])
if dirpath_relative_root_components[0] == 'arch': if dirpath_relative_root_components[0] == 'arch':
if dirpath_relative_root_components_len > 1: if dirpath_relative_root_components_len > 1:
if dirpath_relative_root_components[1] == self.env['arch']: if dirpath_relative_root_components[1] == self.env['arch']:
ccflags_dir.extend([ cc_flags_dir.extend([
'-I', os.path.join(self.env['userland_source_arch_arch_dir']), LF, '-I', os.path.join(self.env['userland_source_arch_arch_dir']), LF,
'-I', os.path.join(self.env['userland_source_arch_dir']), LF, '-I', os.path.join(self.env['userland_source_arch_dir']), LF,
'-fno-pie', LF,
'-no-pie', LF,
]) ])
if 'freestanding' in dirpath_relative_root_components: if 'freestanding' in dirpath_relative_root_components:
common_objs_dir = [] common_objs_dir = []
ccflags_dir.extend([ cc_flags_dir.extend([
'-ffreestanding', LF, '-ffreestanding', LF,
'-nostdlib', LF, '-nostdlib', LF,
'-static', LF, '-static', LF,
@@ -251,24 +229,6 @@ Default: build all examples that have their package dependencies met, e.g.:
common_objs_dir = [] common_objs_dir = []
else: else:
common_objs_dir = [common_obj_asm] common_objs_dir = [common_obj_asm]
if self.env['arch'] == 'arm':
ccflags_dir.extend([
'-Xassembler', '-mcpu=cortex-a72', LF,
# To prevent:
# > vfp.S: Error: selected processor does not support <FPU instruction> in ARM mode
# https://stackoverflow.com/questions/41131432/cross-compiling-error-selected-processor-does-not-support-fmrx-r3-fpexc-in/52875732#52875732
# We aim to take the most extended mode currently available that works on QEMU.
'-Xassembler', '-mfpu=crypto-neon-fp-armv8.1', LF,
'-Xassembler', '-meabi=5', LF,
# Treat inline assembly as arm instead of thumb
# The opposite of -mthumb.
'-marm', LF,
# Make gcc generate .syntax unified for inline assembly.
# However, it gets ignored if -marm is given, which a GCC bug that was recently fixed:
# https://stackoverflow.com/questions/54078112/how-to-write-syntax-unified-ual-armv7-inline-assembly-in-gcc/54132097#54132097
# So we just write divided inline assembly for now.
'-masm-syntax-unified', LF,
])
else: else:
continue continue
else: else:
@@ -279,45 +239,48 @@ Default: build all examples that have their package dependencies met, e.g.:
if not (has_all_packages or pkg_key in has_packages): if not (has_all_packages or pkg_key in has_packages):
continue continue
pkg = pkgs[pkg_key] pkg = pkgs[pkg_key]
if 'ccflags' in pkg: if 'cc_flags' in pkg:
ccflags_dir.extend(pkg['ccflags']) cc_flags_dir.extend(pkg['cc_flags'])
else: else:
pkg_config_output = subprocess.check_output([ pkg_config_output = subprocess.check_output([
self.env['pkg_config'], self.env['pkg_config'],
'--cflags', '--cflags',
pkg_key pkg_key
]).decode() ]).decode()
ccflags_dir.extend(self.sh.shlex_split(pkg_config_output)) cc_flags_dir.extend(self.sh.shlex_split(pkg_config_output))
if 'ccflags_after' in pkg: if 'cc_flags_after' in pkg:
ccflags_dir.extend(pkg['ccflags_after']) cc_flags_dir.extend(pkg['cc_flags_after'])
else: else:
pkg_config_output = subprocess.check_output([ pkg_config_output = subprocess.check_output([
self.env['pkg_config'], self.env['pkg_config'],
'--libs', '--libs',
pkg_key pkg_key
]).decode() ]).decode()
ccflags_after.extend(self.sh.shlex_split(pkg_config_output)) cc_flags_after.extend(self.sh.shlex_split(pkg_config_output))
for in_filename in in_filenames: for in_filename in in_filenames:
path_relative_root = os.path.join(dirpath_relative_root, in_filename) in_path = os.path.join(path, in_filename)
if path_relative_root == common_obj_asm_relpath: cc_flags_file = cc_flags_dir.copy()
continue in_ext = os.path.splitext(in_filename)[1]
in_path = os.path.join(path, in_filename) if not in_ext in self.env['userland_in_exts']:
in_name, in_ext = os.path.splitext(in_filename) continue
out_path = os.path.join( my_path_properties = path_properties.get(os.path.join(
out_dir, self.env['userland_subdir'],
in_name + self.env['userland_build_ext'] dirpath_relative_root,
) in_filename
error = thread_pool.submit({ ))
'in_path': in_path, if my_path_properties['pedantic']:
'out_path': out_path, cc_flags_file.extend(['-pedantic', LF])
'ccflags': ccflags_dir, error = thread_pool.submit({
'cstd': cstd, 'c_std': my_path_properties['c_std'],
'cxxstd': cxxstd, 'cc_flags': cc_flags_file + my_path_properties['cc_flags'],
'extra_objs': common_objs_dir, 'cc_flags_after': cc_flags_after,
'ccflags_after': ccflags_after, 'cxx_std': my_path_properties['cxx_std'],
}) 'extra_objs': common_objs_dir,
if error is not None: 'in_path': in_path,
raise common.ExitLoop() 'out_path': self.resolve_userland_executable(in_path),
})
if error is not None:
raise common.ExitLoop()
except common.ExitLoop: except common.ExitLoop:
pass pass
error = thread_pool.get_error() error = thread_pool.get_error()
@@ -328,27 +291,33 @@ Default: build all examples that have their package dependencies met, e.g.:
self.sh.copy_dir_if_update( self.sh.copy_dir_if_update(
srcdir=build_dir, srcdir=build_dir,
destdir=self.env['out_rootfs_overlay_lkmc_dir'], destdir=self.env['out_rootfs_overlay_lkmc_dir'],
filter_ext=self.env['userland_build_ext'], filter_ext=self.env['userland_executable_ext'],
) )
return 0 return 0
def clean(self): def clean(self):
if self.env['in_tree']: if self.env['in_tree']:
for path, dirnames, filenames in self.walk_source_targets( for target in self.env['targets']:
self.env['targets'], if os.path.exists(target):
self.env['userland_out_exts'], if os.path.isfile(target):
empty_ok=True self.sh.rmrf(self.resolve_userland_executable(target))
): else:
for filename in filenames: for path, dirnames, filenames in self.sh.walk(target):
self.sh.rmrf(os.path.join(path, filename)) for filename in filenames:
if os.path.splitext(filename)[1] in self.env['userland_out_exts']:
self.sh.rmrf(os.path.join(path, filename))
else: else:
self.sh.rmrf(self.get_build_dir()) for target in self.env['targets']:
self.sh.rmrf(self.resolve_userland_executable(target))
def get_build_dir(self): def get_build_dir(self):
if self.env['in_tree']: return self.env['userland_build_dir']
return self.env['userland_source_dir']
else: def setup_one(self):
return self.env['userland_build_dir'] self.env['targets'] = self.resolve_targets(
self.env['userland_source_dir'],
self.env['targets']
)
if __name__ == '__main__': if __name__ == '__main__':
Main().cli() Main().cli()

1
build-userland-in-tree
View File

@@ -22,7 +22,6 @@ https://github.com/cirosantilli/linux-kernel-module-cheat#userland-setup-getting
''', ''',
defaults={ defaults={
'gcc_which': 'host', 'gcc_which': 'host',
'has_all_packages': True,
'in_tree': True, 'in_tree': True,
'targets': ['.'], 'targets': ['.'],
} }

8
cli_function.py
View File

@@ -290,7 +290,10 @@ class CliFunction:
if value != default: if value != default:
if argument.is_option: if argument.is_option:
if argument.is_bool: if argument.is_bool:
vals = [(argument.longname,)] if value:
vals = [(argument.longname,)]
else:
vals = [('--no-' + argument.longname[2:],)]
elif 'action' in argument.kwargs and argument.kwargs['action'] == 'append': elif 'action' in argument.kwargs and argument.kwargs['action'] == 'append':
vals = [(argument.longname, str(val)) for val in value] vals = [(argument.longname, str(val)) for val in value]
else: else:
@@ -454,7 +457,8 @@ amazing function!
# get_cli # get_cli
assert one_cli_function.get_cli(pos_mandatory=1, asdf='B') == [('--asdf', 'B'), ('--bool-cli',), ('1',)] assert one_cli_function.get_cli(pos_mandatory=1, asdf='B') == [('--asdf', 'B'), ('--bool-cli',), ('1',)]
assert one_cli_function.get_cli(pos_mandatory=1, asdf='B', qwer='R') == [('--asdf', 'B'), ('--bool-cli',), ('--qwer', 'R'), ('1',)] assert one_cli_function.get_cli(pos_mandatory=1, asdf='B', qwer='R') == [('--asdf', 'B'), ('--bool-cli',), ('--qwer', 'R'), ('1',)]
assert one_cli_function.get_cli(pos_mandatory=1, bool_true=False) == [('--bool-cli',), ('--bool-true',), ('1',)] assert one_cli_function.get_cli(pos_mandatory=1, bool_true=False) == [('--bool-cli',), ('--no-bool-true',), ('1',)]
assert one_cli_function.get_cli(pos_mandatory=1, bool_false=True) == [('--bool-cli',), ('--bool-false',), ('1',)]
assert one_cli_function.get_cli(pos_mandatory=1, pos_optional=2, args_star=['asdf', 'qwer']) == [('--bool-cli',), ('1',), ('2',), ('asdf',), ('qwer',)] assert one_cli_function.get_cli(pos_mandatory=1, pos_optional=2, args_star=['asdf', 'qwer']) == [('--bool-cli',), ('1',), ('2',), ('asdf',), ('qwer',)]
assert one_cli_function.get_cli(pos_mandatory=1, append=['2', '3']) == [('--append', '2'), ('--append', '3',), ('--bool-cli',), ('1',)] assert one_cli_function.get_cli(pos_mandatory=1, append=['2', '3']) == [('--append', '2'), ('--append', '3',), ('--bool-cli',), ('1',)]

190
common.py
View File

@@ -62,7 +62,7 @@ consts['kernel_modules_source_dir'] = os.path.join(consts['root_dir'], consts['k
consts['userland_subdir'] = 'userland' consts['userland_subdir'] = 'userland'
consts['userland_source_dir'] = os.path.join(consts['root_dir'], consts['userland_subdir']) consts['userland_source_dir'] = os.path.join(consts['root_dir'], consts['userland_subdir'])
consts['userland_source_arch_dir'] = os.path.join(consts['userland_source_dir'], 'arch') consts['userland_source_arch_dir'] = os.path.join(consts['userland_source_dir'], 'arch')
consts['userland_build_ext'] = '.out' consts['userland_executable_ext'] = '.out'
consts['include_subdir'] = 'lkmc' consts['include_subdir'] = 'lkmc'
consts['include_source_dir'] = os.path.join(consts['root_dir'], consts['include_subdir']) consts['include_source_dir'] = os.path.join(consts['root_dir'], consts['include_subdir'])
consts['submodules_dir'] = os.path.join(consts['root_dir'], 'submodules') consts['submodules_dir'] = os.path.join(consts['root_dir'], 'submodules')
@@ -110,7 +110,7 @@ consts['userland_in_exts'] = [
consts['cxx_ext'], consts['cxx_ext'],
] ]
consts['userland_out_exts'] = [ consts['userland_out_exts'] = [
consts['userland_build_ext'], consts['userland_executable_ext'],
consts['obj_ext'], consts['obj_ext'],
] ]
consts['config_file'] = os.path.join(consts['data_dir'], 'config.py') consts['config_file'] = os.path.join(consts['data_dir'], 'config.py')
@@ -358,10 +358,8 @@ See: https://github.com/cirosantilli/linux-kernel-module-cheat#initrd
help='''\ help='''\
Use the given baremetal executable instead of the Linux kernel. Use the given baremetal executable instead of the Linux kernel.
If the path is absolute, it is used as is. If the path points to a source code inside baremetal/, then the
corresponding executable is automatically found.
If the path is relative, we assume that it points to a source code
inside baremetal/ and then try to use corresponding executable.
''' '''
) )
@@ -467,6 +465,17 @@ CLI arguments to pass to the userland executable.
) )
# Run. # Run.
self.add_argument(
'--in-tree',
default=False,
help='''\
Place build output inside source tree to conveniently run it, especially when
building with the host native toolchain. Currently only supported by ./build-userland.
When running, prefer in-tree executables instead of out-of-tree ones, e.g.:
userland/c/hello resolves userland/c/hello.out instead of the out-of-tree one.
''',
)
self.add_argument( self.add_argument(
'--port-offset', '--port-offset',
type=int, type=int,
@@ -759,7 +768,10 @@ Incompatible archs are skipped.
# Userland # Userland
env['userland_source_arch_arch_dir'] = join(env['userland_source_arch_dir'], env['arch']) env['userland_source_arch_arch_dir'] = join(env['userland_source_arch_dir'], env['arch'])
env['userland_build_dir'] = join(env['out_dir'], 'userland', env['userland_build_id'], env['arch']) if env['in_tree']:
env['userland_build_dir'] = self.env['userland_source_dir']
else:
env['userland_build_dir'] = join(env['out_dir'], 'userland', env['userland_build_id'], env['arch'])
# Kernel modules. # Kernel modules.
env['kernel_modules_build_dir'] = join(env['kernel_modules_build_base_dir'], env['arch']) env['kernel_modules_build_dir'] = join(env['kernel_modules_build_base_dir'], env['arch'])
@@ -821,7 +833,7 @@ Incompatible archs are skipped.
env['baremetal'], env['baremetal'],
env['baremetal_source_dir'], env['baremetal_source_dir'],
env['baremetal_build_dir'], env['baremetal_build_dir'],
[env['baremetal_build_ext']], env['baremetal_build_ext'],
) )
source_path_noext = os.path.splitext(join( source_path_noext = os.path.splitext(join(
env['baremetal_source_dir'], env['baremetal_source_dir'],
@@ -905,6 +917,15 @@ lunch aosp_{}-eng
self._common_args.add(key) self._common_args.add(key)
super().add_argument(*args, **kwargs) super().add_argument(*args, **kwargs)
def assert_is_subpath(self, subpath, parent):
if not self.is_subpath(subpath, parent):
raise Exception(
'Can only accept targets inside {}, given: {}'.format(
parent,
subpath
)
)
def get_elf_entry(self, elf_file_path): def get_elf_entry(self, elf_file_path):
readelf_header = subprocess.check_output([ readelf_header = subprocess.check_output([
self.get_toolchain_tool('readelf'), self.get_toolchain_tool('readelf'),
@@ -1028,6 +1049,12 @@ lunch aosp_{}-eng
if flush: if flush:
sys.stdout.flush() sys.stdout.flush()
def is_subpath(self, subpath, parent):
'''
https://stackoverflow.com/questions/3812849/how-to-check-whether-a-directory-is-a-sub-directory-of-another-directory
'''
return os.path.abspath(subpath).startswith(os.path.abspath(parent))
def main(self, *args, **kwargs): def main(self, *args, **kwargs):
''' '''
Run timed_main across all selected archs and emulators. Run timed_main across all selected archs and emulators.
@@ -1049,10 +1076,6 @@ lunch aosp_{}-eng
real_emulators = env['emulators'] real_emulators = env['emulators']
return_value = 0 return_value = 0
try: try:
ret = self.setup()
if ret is not None and ret != 0:
return_value = ret
raise ExitLoop()
for emulator in real_emulators: for emulator in real_emulators:
for arch in real_archs: for arch in real_archs:
if arch in env['arch_short_to_long_dict']: if arch in env['arch_short_to_long_dict']:
@@ -1079,6 +1102,7 @@ lunch aosp_{}-eng
dry_run=self.env['dry_run'], dry_run=self.env['dry_run'],
quiet=self.env['quiet'], quiet=self.env['quiet'],
) )
self.setup_one()
ret = self.timed_main() ret = self.timed_main()
if not env['dry_run']: if not env['dry_run']:
end_time = time.time() end_time = time.time()
@@ -1163,92 +1187,70 @@ lunch aosp_{}-eng
] ]
) )
def resolve_source_tree(self, in_path, exts, source_tree_root, empty_ok=False): def resolve_executable(
self,
in_path,
magic_in_dir,
magic_out_dir,
executable_ext
):
''' '''
Convert a convenient shorthand user input string to paths of existing files Resolve the path of an userland or baremetal executable.
in the source tree.
Ensure that the path lies inside source_tree_root. If it is in tree, resolve source paths to their corresponding executables.
Multiple matches may happen if multiple multiple exts files exist. If it is out of tree, return the same exact path as input.
E.g., after an in-tree build, in_path='hello' and exts=['.c', '.out']
would match both:
- userland/hello.c If the input path is a file, add the executable extension automatically.
- userland/hello.out
If you also want directories to be matched, just add an empty string
`''` to exts, which leads all of the following to match the arch directory:
- arch
- arch.c
- userland/arch
- /full/path/to/userland/arch
Note however that this potentially prevents differentiation between
files and directories: e.g. if you had both a file arch.c and a directory arch/,
and exts=['', '.c'], then both would get matched.
''' '''
in_path = os.path.abspath(in_path) in_path_abs = os.path.abspath(in_path)
source_tree_root = os.path.abspath(source_tree_root) magic_in_dir_abs = os.path.abspath(magic_in_dir)
if not in_path.startswith(source_tree_root): magic_out_dir_abs = os.path.abspath(magic_out_dir)
raise Exception( if self.is_subpath(in_path_abs, magic_in_dir_abs):
'The input path {} is not inside the source directory {}'.format( out = os.path.abspath(os.path.join(
in_path, magic_out_dir_abs,
source_tree_root os.path.relpath(
) os.path.splitext(in_path_abs)[0],
os.path.abspath(magic_in_dir_abs)
)),
) )
result = [] if os.path.isfile(in_path):
name, ext = os.path.splitext(in_path) out += executable_ext
for try_ext in exts: return out
try_path = name + try_ext
if os.path.exists(try_path):
result.append(try_path)
if not result and not empty_ok:
raise Exception('No file not found for input: ' + in_path)
return result
def resolve_executable(self, in_path, magic_in_dir, magic_out_dir, out_exts):
if os.path.isabs(in_path):
return in_path
else: else:
paths = [ return in_path_abs
os.path.join(magic_out_dir, in_path),
os.path.join( def resolve_targets(self, source_dir, targets):
magic_out_dir, if not targets:
os.path.relpath(in_path, magic_in_dir), targets = [source_dir]
) new_targets = []
] for target in targets:
for path in paths: target = self.toplevel_to_source_dir(target, source_dir)
for out_ext in out_exts: self.assert_is_subpath(target, source_dir)
path = os.path.splitext(path)[0] + out_ext new_targets.append(target)
if os.path.exists(path): return new_targets
return path
if not self.env['dry_run']:
raise Exception('Executable file not found. Tried:\n' + '\n'.join(paths))
def resolve_userland_executable(self, path): def resolve_userland_executable(self, path):
'''
Convert an userland source path-like string to an
absolute userland build output path.
'''
return self.resolve_executable( return self.resolve_executable(
path, path,
self.env['userland_source_dir'], self.env['userland_source_dir'],
self.env['userland_build_dir'], self.env['userland_build_dir'],
[self.env['userland_build_ext']], self.env['userland_executable_ext'],
) )
def setup(self): def setup_one(self):
''' '''
Similar to timed_main, but gets run only once for all --arch and --emulator, Run just before timed_main, after _init_env.
before timed_main.
Different from __init__, since at this point env has already been calculated,
so variables that don't depend on --arch or --emulator can be used.
''' '''
pass pass
def toplevel_to_source_dir(self, path, source_dir):
path = os.path.abspath(path)
if path == self.env['root_dir']:
return source_dir
else:
return path
def timed_main(self): def timed_main(self):
''' '''
Main action of the derived class. Main action of the derived class.
@@ -1259,38 +1261,10 @@ lunch aosp_{}-eng
def teardown(self): def teardown(self):
''' '''
Similar to setup, but run after timed_main. Similar to setup, but run once after all timed_main are called.
''' '''
pass pass
def walk_source_targets(self, targets, exts, empty_ok=False):
'''
Resolve userland or baremetal source tree targets, and walk them.
Ignore the input extension of targets, and select only files
with the given extensions exts.
'''
if targets:
targets = targets
else:
targets = [self.env['userland_source_dir']]
for target in targets:
resolved_targets = self.resolve_source_tree(
target,
exts + [''],
self.env['userland_source_dir'],
empty_ok=empty_ok,
)
for resolved_target in resolved_targets:
for path, dirnames, filenames in self.sh.walk(resolved_target):
dirnames.sort()
filenames = [
filename for filename in filenames
if os.path.splitext(filename)[1] in exts
]
filenames.sort()
yield path, dirnames, filenames
class BuildCliFunction(LkmcCliFunction): class BuildCliFunction(LkmcCliFunction):
''' '''
A CLI function with common facilities to build stuff, e.g.: A CLI function with common facilities to build stuff, e.g.:

185
path_properties.py
View File

@@ -2,6 +2,8 @@
import os import os
from shell_helpers import LF
class PathProperties: class PathProperties:
''' '''
Encodes properties of userland and baremetal paths. Encodes properties of userland and baremetal paths.
@@ -12,30 +14,42 @@ class PathProperties:
self, self,
**kwargs **kwargs
): ):
self.properties = { property_keys = {
'allowed_archs': None, 'allowed_archs',
'exit_status': 0, 'c_std',
'interactive': False, 'cc_flags',
'more_than_1s': False, 'cc_pedantic',
'cxx_std',
'exit_status',
'interactive',
'skip_run_unclassified',
'more_than_1s',
# The path does not generate an executable in itself, e.g. # The path does not generate an executable in itself, e.g.
# it only generates intermediate object files. # it only generates intermediate object files.
'no_executable': False, 'no_executable',
'pedantic',
# the test receives a signal. We skip those tests for now, # the test receives a signal. We skip those tests for now,
# on userland because we are lazy to figure out the exact semantics # on userland because we are lazy to figure out the exact semantics
# of how Python + QEMU + gem5 determine the exit status of signals. # of how Python + QEMU + gem5 determine the exit status of signals.
'receives_signal': False, 'receives_signal',
'requires_kernel_modules': False, 'requires_kernel_modules',
} }
for key in kwargs: for key in kwargs:
if not key in self.properties: if not key in property_keys:
raise ValueError('Unknown key: {}'.format(key)) raise ValueError('Unknown key: {}'.format(key))
self.properties.update(kwargs) self.properties = kwargs
def __getitem__(self, key): def __getitem__(self, key):
return self.properties[key] return self.properties[key]
def __repr__(self):
return str(self.properties)
def update(self, other): def update(self, other):
return self.properties.update(other.properties) other_tmp_properties = other.properties.copy()
if 'cc_flags' in self.properties and 'cc_flags' in other_tmp_properties:
other_tmp_properties['cc_flags'] = self.properties['cc_flags'] + other_tmp_properties['cc_flags']
return self.properties.update(other_tmp_properties)
def should_be_tested(self, arch): def should_be_tested(self, arch):
return \ return \
@@ -44,57 +58,138 @@ class PathProperties:
not self['no_executable'] and \ not self['no_executable'] and \
not self['receives_signal'] and \ not self['receives_signal'] and \
not self['requires_kernel_modules'] and \ not self['requires_kernel_modules'] and \
not self['skip_run_unclassified'] and \
( (
self['allowed_archs'] is None or self['allowed_archs'] is None or
arch in self['allowed_archs'] arch in self['allowed_archs']
) )
class PrefixTree: class PrefixTree:
def __init__(self, children=None, value=None): def __init__(self, path_properties_dict=None, children=None):
if children == None: if children is None:
children = {} children = {}
if path_properties_dict is None:
path_properties_dict = {}
self.children = children self.children = children
self.value = value self.path_properties = PathProperties(**path_properties_dict)
path_properties_tree = PrefixTree({ default_c_std = 'c11'
'arch': PrefixTree({ default_cxx_std = 'c++17'
'x86_64': PrefixTree( gnu_extensions = {
'c_std': 'gnu11',
'cc_pedantic': False,
'cxx_std': 'gnu++17'
}
path_properties_tree = PrefixTree(
{
'c_std': default_c_std,
'cxx_std': default_cxx_std,
'pedantic': True,
'allowed_archs': None,
'c_std': None,
'cc_flags': [],
'cc_pedantic': True,
'cxx_std': None,
'exit_status': 0,
'interactive': False,
'skip_run_unclassified': False,
'more_than_1s': False,
# The path does not generate an executable in itself, e.g.
# it only generates intermediate object files.
'no_executable': False,
'pedantic': False,
# the test receives a signal. We skip those tests for now,
# on userland because we are lazy to figure out the exact semantics
# of how Python + QEMU + gem5 determine the exit status of signals.
'receives_signal': False,
'requires_kernel_modules': False,
},
{
'userland': PrefixTree(
{ {
'c': PrefixTree({
'ring0.c': PrefixTree(value=PathProperties(receives_signal=True))
})
}, },
PathProperties(allowed_archs={'x86_64'}), {
), 'arch': PrefixTree(
'arm': PrefixTree(value=PathProperties(allowed_archs={'arm'})), {
'aarch64': PrefixTree(value=PathProperties(allowed_archs={'aarch64'})), 'cc_flags': [
'empty.S': PrefixTree(value=PathProperties(no_executable=True)), '-fno-pie', LF,
'fail.S': PrefixTree(value=PathProperties(no_executable=True)), '-no-pie', LF,
'main.c': PrefixTree(value=PathProperties(no_executable=True)), ]
}), },
'c': PrefixTree({ {
'assert_fail.c': PrefixTree(value=PathProperties(exit_status=1)), 'arm': PrefixTree(
'false.c': PrefixTree(value=PathProperties(exit_status=1)), {
'getchar.c': PrefixTree(value=PathProperties(interactive=True)), 'allowed_archs': {'arm'},
'infinite_loop.c': PrefixTree(value=PathProperties(more_than_1s=True)), 'cc_flags': [
}), '-Xassembler', '-mcpu=cortex-a72', LF,
'kernel_modules': PrefixTree(value=PathProperties(requires_kernel_modules=True)), # To prevent:
'linux': PrefixTree(value=PathProperties(requires_kernel_modules=True)), # > vfp.S: Error: selected processor does not support <FPU instruction> in ARM mode
'posix': PrefixTree({ # https://stackoverflow.com/questions/41131432/cross-compiling-error-selected-processor-does-not-support-fmrx-r3-fpexc-in/52875732#52875732
'count.c': PrefixTree(value=PathProperties(more_than_1s=True)), # We aim to take the most extended mode currently available that works on QEMU.
'sleep_forever.c': PrefixTree(value=PathProperties(more_than_1s=True)), '-Xassembler', '-mfpu=crypto-neon-fp-armv8.1', LF,
'virt_to_phys_test.c': PrefixTree(value=PathProperties(more_than_1s=True)), '-Xassembler', '-meabi=5', LF,
}) # Treat inline assembly as arm instead of thumb
}) # The opposite of -mthumb.
'-marm', LF,
# Make gcc generate .syntax unified for inline assembly.
# However, it gets ignored if -marm is given, which a GCC bug that was recently fixed:
# https://stackoverflow.com/questions/54078112/how-to-write-syntax-unified-ual-armv7-inline-assembly-in-gcc/54132097#54132097
# So we just write divided inline assembly for now.
'-masm-syntax-unified', LF,
]
}
),
'aarch64': PrefixTree({'allowed_archs': {'aarch64'}}),
'empty.S': PrefixTree({'no_executable': True}),
'fail.S': PrefixTree({'no_executable': True}),
'main.c': PrefixTree({'no_executable': True}),
'x86_64': PrefixTree(
{'allowed_archs': {'x86_64'}},
{
'c': PrefixTree(
{},
{
'ring0.c': PrefixTree({'receives_signal': True})
}
),
}
),
}
),
'c': PrefixTree(
{},
{
'assert_fail.c': PrefixTree({'exit_status': 1}),
'false.c': PrefixTree({'exit_status': 1}),
'getchar.c': PrefixTree({'interactive': True}),
'infinite_loop.c': PrefixTree({'more_than_1s': True}),
}
),
'gcc': PrefixTree(gnu_extensions),
'kernel_modules': PrefixTree({**gnu_extensions, **{'requires_kernel_modules': True}}),
'linux': PrefixTree(
{**gnu_extensions, **{'skip_run_unclassified': True}},
),
'posix': PrefixTree(
{},
{
'count.c': PrefixTree({'more_than_1s': True}),
'sleep_forever.c': PrefixTree({'more_than_1s': True}),
'virt_to_phys_test.c': PrefixTree({'more_than_1s': True}),
}
)
}
)
}
)
def get(test_path): def get(test_path):
cur_node = path_properties_tree cur_node = path_properties_tree
path_properties = PathProperties() path_properties = PathProperties(**cur_node.path_properties.properties)
for path_component in test_path.split(os.sep): for path_component in test_path.split(os.sep):
if path_component in cur_node.children: if path_component in cur_node.children:
cur_node = cur_node.children[path_component] cur_node = cur_node.children[path_component]
if cur_node.value is not None: path_properties.update(cur_node.path_properties)
path_properties.update(cur_node.value)
else: else:
break break
return path_properties return path_properties

29
run
View File

@@ -18,7 +18,8 @@ Run some content on an emulator.
''' '''
) )
self.add_argument( self.add_argument(
'--background', default=False, '--background',
default=False,
help='''\ help='''\
Send QEMU serial output to a file instead of the terminal so it does not require a Send QEMU serial output to a file instead of the terminal so it does not require a
terminal attached to run on the background. Interactive input cannot be given. terminal attached to run on the background. Interactive input cannot be given.
@@ -342,10 +343,10 @@ Extra options to append at the end of the emulator command line.
if not self.env['_args_given']['gdb_wait']: if not self.env['_args_given']['gdb_wait']:
self.env['gdb_wait'] = True self.env['gdb_wait'] = True
if not self.env['_args_given']['tmux_args']: if not self.env['_args_given']['tmux_args']:
if self.env['userland'] is not None: if self.env['userland'] is None and self.env['baremetal'] is None:
self.env['tmux_args'] = 'main'
else:
self.env['tmux_args'] = 'start_kernel' self.env['tmux_args'] = 'start_kernel'
else:
self.env['tmux_args'] = 'main'
if not self.env['_args_given']['tmux_program']: if not self.env['_args_given']['tmux_program']:
self.env['tmux_program'] = 'gdb' self.env['tmux_program'] = 'gdb'
if self.env['tmux_args'] is not None or self.env['_args_given']['tmux_program']: if self.env['tmux_args'] is not None or self.env['_args_given']['tmux_program']:
@@ -621,7 +622,11 @@ Extra options to append at the end of the emulator command line.
'-kernel', self.env['image'], LF, '-kernel', self.env['image'], LF,
'-m', self.env['memory'], LF, '-m', self.env['memory'], LF,
'-monitor', 'telnet::{},server,nowait'.format(self.env['qemu_monitor_port']), LF, '-monitor', 'telnet::{},server,nowait'.format(self.env['qemu_monitor_port']), LF,
'-netdev', 'user,hostfwd=tcp::{}-:{},hostfwd=tcp::{}-:22,id=net0'.format(self.env['qemu_hostfwd_generic_port'], self.env['qemu_hostfwd_generic_port'], self.env['qemu_hostfwd_ssh_port']), LF, '-netdev', 'user,hostfwd=tcp::{}-:{},hostfwd=tcp::{}-:22,id=net0'.format(
self.env['qemu_hostfwd_generic_port'],
self.env['qemu_hostfwd_generic_port'],
self.env['qemu_hostfwd_ssh_port']
), LF,
'-no-reboot', LF, '-no-reboot', LF,
'-smp', str(self.env['cpus']), LF, '-smp', str(self.env['cpus']), LF,
] + ] +
@@ -657,7 +662,12 @@ Extra options to append at the end of the emulator command line.
self.raw_to_qcow2(qemu_which=self.env['qemu_which']) self.raw_to_qcow2(qemu_which=self.env['qemu_which'])
extra_emulator_args.extend([ extra_emulator_args.extend([
'-drive', '-drive',
'file={},format=qcow2,if={}{}{}'.format(self.env['disk_image'], driveif, snapshot, rrid), 'file={},format=qcow2,if={}{}{}'.format(
self.env['disk_image'],
driveif,
snapshot,
rrid
),
LF, LF,
]) ])
if rr: if rr:
@@ -668,7 +678,10 @@ Extra options to append at the end of the emulator command line.
if rr: if rr:
extra_emulator_args.extend([ extra_emulator_args.extend([
'-object', 'filter-replay,id=replay,netdev=net0', '-object', 'filter-replay,id=replay,netdev=net0',
'-icount', 'shift=7,rr={},rrfile={}'.format('record' if self.env['record'] else 'replay', self.env['qemu_rrfile']), '-icount', 'shift=7,rr={},rrfile={}'.format(
'record' if self.env['record'] else 'replay',
self.env['qemu_rrfile']
),
]) ])
virtio_gpu_pci = [] virtio_gpu_pci = []
else: else:
@@ -717,6 +730,8 @@ Extra options to append at the end of the emulator command line.
tmux_args += " --baremetal '{}'".format(self.env['baremetal']) tmux_args += " --baremetal '{}'".format(self.env['baremetal'])
if self.env['userland']: if self.env['userland']:
tmux_args += " --userland '{}'".format(self.env['userland']) tmux_args += " --userland '{}'".format(self.env['userland'])
if self.env['in_tree']:
tmux_args += ' --in-tree'
if self.env['tmux_args'] is not None: if self.env['tmux_args'] is not None:
tmux_args += ' {}'.format(self.env['tmux_args']) tmux_args += ' {}'.format(self.env['tmux_args'])
subprocess.Popen([ subprocess.Popen([

5
shell_helpers.py
View File

@@ -136,8 +136,7 @@ class ShellHelpers:
self.copy_dir_if_update_non_recursive(srcdir, destdir, filter_ext) self.copy_dir_if_update_non_recursive(srcdir, destdir, filter_ext)
srcdir_abs = os.path.abspath(srcdir) srcdir_abs = os.path.abspath(srcdir)
srcdir_abs_len = len(srcdir_abs) srcdir_abs_len = len(srcdir_abs)
for path, dirnames, filenames in os.walk(srcdir_abs): for path, dirnames, filenames in self.walk(srcdir_abs):
dirnames.sort()
for dirname in dirnames: for dirname in dirnames:
dirpath = os.path.join(path, dirname) dirpath = os.path.join(path, dirname)
dirpath_relative_root = dirpath[srcdir_abs_len + 1:] dirpath_relative_root = dirpath[srcdir_abs_len + 1:]
@@ -329,6 +328,8 @@ class ShellHelpers:
yield dirname, [], [basename] yield dirname, [], [basename]
else: else:
for path, dirnames, filenames in os.walk(root): for path, dirnames, filenames in os.walk(root):
dirnames.sort()
filenames.sort()
yield path, dirnames, filenames yield path, dirnames, filenames
def wget(self, url, download_path): def wget(self, url, download_path):

2
test-baremetal
View File

@@ -43,7 +43,7 @@ If given, run only the given tests. Otherwise, run all tests.
self.env['emulator'] == 'gem5' and os.path.basename(path).startswith('semihost_') or self.env['emulator'] == 'gem5' and os.path.basename(path).startswith('semihost_') or
self.env['emulator'] == 'qemu' and os.path.basename(path).startswith('gem5_') self.env['emulator'] == 'qemu' and os.path.basename(path).startswith('gem5_')
): ):
sources.append(os.path.relpath(path, self.env['baremetal_source_dir'])) sources.append(os.path.relpath(path, self.env['root_dir']))
else: else:
sources = self.env['tests'] sources = self.env['tests']
for source in sources: for source in sources:

97
test-build-userland
View File

@@ -1,52 +1,71 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Quick sanity check that userland target resoltion at least does not blow up. # Quick sanity check that userland target resolution works.
set -eux set -eux
./build-userland for in_tree in '' --in-tree; do
./build-userland --clean userland_build_dir="$(./getvar $in_tree userland_build_dir)"
./build-userland userland/c # Toplevel.
./build-userland --clean userland/c ./build-userland $in_tree
./build-userland userland/c/hello [ -f "${userland_build_dir}/c/hello.out" ]
./build-userland --clean userland/c/hello ./build-userland $in_tree --clean
./build-userland userland/c/hello. ! [ -f "${userland_build_dir}/c/hello.out" ]
./build-userland --clean userland/c/hello.
./build-userland userland/c/hello.c
./build-userland --clean userland/c/hello.c
./build-userland userland/c/hello.out
./build-userland --clean userland/c/hello.out
./build-userland "$(pwd)/userland/c/hello.out"
./build-userland --clean "$(pwd)/userland/c/hello.out"
./build-userland --in-tree # Toplevel explicit.
./build-userland --in-tree --clean ./build-userland $in_tree userland/
./build-userland --in-tree userland/c [ -f "${userland_build_dir}/c/hello.out" ]
./build-userland --in-tree --clean userland/c ./build-userland $in_tree --clean
./build-userland --in-tree userland/c/hello ! [ -f "${userland_build_dir}/c/hello.out" ]
./build-userland --in-tree --clean userland/c/hello
./build-userland --in-tree userland/c/hello. # Toplevel root dir.
./build-userland --in-tree --clean userland/c/hello. ./build-userland $in_tree .
./build-userland --in-tree userland/c/hello.c [ -f "${userland_build_dir}/c/hello.out" ]
./build-userland --in-tree --clean userland/c/hello.c ./build-userland $in_tree --clean
./build-userland --in-tree userland/c/hello.out ! [ -f "${userland_build_dir}/c/hello.out" ]
./build-userland --in-tree --clean userland/c/hello.out
./build-userland --in-tree "$(pwd)/userland/c/hello.out" # Subdirectory.
./build-userland --in-tree --clean "$(pwd)/userland/c/hello.out" ./build-userland $in_tree userland/c
./build-userland --in-tree --clean [ -f "${userland_build_dir}/c/hello.out" ]
./build-userland $in_tree --clean userland/c
! [ -f "${userland_build_dir}/c/hello.out" ]
# One program.
./build-userland $in_tree userland/c/hello.c
[ -f "${userland_build_dir}/c/hello.out" ]
./build-userland $in_tree --clean userland/c/hello.c
! [ -f "${userland_build_dir}/c/hello.out" ]
# Things that don't work: building:
# - non-existent files
# - paths outside of tree
! ./build-userland $in_tree userland/c/hello
! ./build-userland $in_tree userland/c/hello.
! ./build-userland $in_tree "${userland_build_dir}/c/hello.out"
tmpfile="$(mktemp)"
! ./build-userland $in_tree "$tmpfile"
rm "$tmpfile"
! ./build-userland $in_tree ..
! ./build-userland $in_tree kernel_modules
./build-userland --clean $in_tree
# Clean is however more forgiving and accepts paths that don't exist.
./build-userland --clean $in_tree userland/does_not_exist
done
./build-userland-in-tree
[ -f userland/c/hello.out ]
./build-userland-in-tree --clean
! [ -f userland/c/hello.out ]
cd userland cd userland
./build ./build
[ -f c/hello.out ]
./build --clean ./build --clean
! [ -f c/hello.out ]
./build c ./build c
[ -f c/hello.out ]
./build --clean c ./build --clean c
./build c/hello ! [ -f c/hello.out ]
./build --clean c/hello
./build c/hello.
./build --clean c/hello.
./build c/hello.c
./build --clean c/hello.c ./build --clean c/hello.c
./build c/hello.out ! [ -f c/hello.out ]
./build --clean c/hello.out
./build "$(pwd)/c/hello.out"
./build --clean "$(pwd)/c/hello.out"

55
test-user-mode
View File

@@ -27,6 +27,12 @@ If given, run only the given tests. Otherwise, run all tests.
''' '''
) )
def setup_one(self):
self.env['tests'] = self.resolve_targets(
self.env['userland_source_dir'],
self.env['tests']
)
def timed_main(self): def timed_main(self):
run_args = self.get_common_args() run_args = self.get_common_args()
run_args['ctrl_c_host'] = True run_args['ctrl_c_host'] = True
@@ -35,37 +41,36 @@ If given, run only the given tests. Otherwise, run all tests.
if self.env['emulator'] == 'gem5': if self.env['emulator'] == 'gem5':
run_args['userland_build_id'] = 'static' run_args['userland_build_id'] = 'static'
had_failure = False had_failure = False
rootdir_abs_len = len(self.env['userland_source_dir']) rootdir_abs_len = len(self.env['root_dir'])
with ThreadPool( with ThreadPool(
self.run_test, self.run_test,
nthreads=self.env['nproc'], nthreads=self.env['nproc'],
thread_id_arg='thread_id', thread_id_arg='thread_id',
) as thread_pool: ) as thread_pool:
try: try:
for path, in_dirnames, in_filenames in self.walk_source_targets( for test in self.env['tests']:
self.env['tests'], for path, in_dirnames, in_filenames in self.sh.walk(test):
self.env['userland_in_exts'] path_abs = os.path.abspath(path)
): dirpath_relative_root = path_abs[rootdir_abs_len + 1:]
path_abs = os.path.abspath(path) for in_filename in in_filenames:
dirpath_relative_root = path_abs[rootdir_abs_len + 1:] if os.path.splitext(in_filename)[1] in self.env['userland_in_exts']:
for in_filename in in_filenames: path_relative_root = os.path.join(dirpath_relative_root, in_filename)
path_relative_root = os.path.join(dirpath_relative_root, in_filename) my_path_properties = path_properties.get(path_relative_root)
my_path_properties = path_properties.get(path_relative_root) if my_path_properties.should_be_tested(self.env['arch']):
if my_path_properties.should_be_tested(self.env['arch']): cur_run_args = run_args.copy()
cur_run_args = run_args.copy() cur_run_args.update({
cur_run_args.update({ 'background': True,
'background': True, 'userland': os.path.relpath(os.path.join(path_abs, in_filename), os.getcwd()),
'userland': path_relative_root, })
}) error = thread_pool.submit({
error = thread_pool.submit({ 'expected_exit_status': my_path_properties['exit_status'],
'expected_exit_status': my_path_properties['exit_status'], 'run_args': cur_run_args,
'run_args': cur_run_args, 'run_obj': self.import_path_main('run'),
'run_obj': self.import_path_main('run'), 'test_id': path_relative_root,
'test_id': path_relative_root, })
}) if error is not None:
if error is not None: if self.env['quit_on_fail']:
if self.env['quit_on_fail']: raise common.ExitLoop()
raise common.ExitLoop()
except common.ExitLoop: except common.ExitLoop:
pass pass
error = thread_pool.get_error() error = thread_pool.get_error()