3rd-party/linux-kernel-module-cheat
1
0
Fork 0
mirror of https://github.com/cirosantilli/linux-kernel-module-cheat.git synced 2026-01-26 11:41:35 +01:00
Code Issues Releases Wiki Activity

rename build to build-buildroot

Browse Source
This commit is contained in:
Ciro Santilli
2018-09-13 05:20:51 +01:00
parent 7189742a29
commit 3f4ae17017
9 changed files with 455 additions and 412 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.travis.yml
View File

@@ -21,5 +21,6 @@ script: |
# The following packages have unmet dependencies: # The following packages have unmet dependencies:
# libsdl2-dev : Depends: libegl1-mesa-dev # libsdl2-dev : Depends: libegl1-mesa-dev
# Depends: libgles2-mesa-dev # Depends: libgles2-mesa-dev
bash -x ./build-qemu -j 16 -S |& awk 'NR % 1000 == 0'
bash -x ./build -j 16 -S |& awk 'NR % 1000 == 0' bash -x ./build -j 16 -S |& awk 'NR % 1000 == 0'
bash -x ./run -e 'init=/poweroff.out' bash -x ./run -e 'init=/poweroff.out'

179
README.adoc
View File

@@ -98,7 +98,7 @@ git clone https://github.com/cirosantilli/linux-kernel-module-cheat
cd linux-kernel-module-cheat cd linux-kernel-module-cheat
./configure && \ ./configure && \
./build-qemu && ./build-qemu &&
./build && \ ./build-buildroot && \
./run && \ ./run && \
:; :;
.... ....
@@ -111,7 +111,7 @@ It is super easy to build for different CPU architectures, just use the `--arch`
.... ....
./build-qemu --arch arm && \ ./build-qemu --arch arm && \
./build --arch arm && \ ./build-buildroot --arch arm && \
./run --arch arm && \ ./run --arch arm && \
:; :;
.... ....
@@ -221,7 +221,7 @@ pr_info("hello init hacked\n");
and then rebuild the kernel modules and re-run to see it take effect: and then rebuild the kernel modules and re-run to see it take effect:
.... ....
./build --kernel-modules ./build-buildroot --kernel-modules
./run -F 'insmod /hello.ko' ./run -F 'insmod /hello.ko'
.... ....
@@ -246,7 +246,7 @@ pr_info("I'VE HACKED THE LINUX KERNEL!!!");
Then rebuild the Linux kernel and reboot: Then rebuild the Linux kernel and reboot:
.... ....
./build && ./run ./build-buildroot && ./run
.... ....
and, surely enough, your message has appeared at the beginning of the boot. and, surely enough, your message has appeared at the beginning of the boot.
@@ -281,7 +281,7 @@ and once again, there is your message: QEMU communicated it to the Linux kernel,
You have now gone from newb to hardware hacker in a mere 15 minutes, your rate of progress is truly astounding!!! You have now gone from newb to hardware hacker in a mere 15 minutes, your rate of progress is truly astounding!!!
BTW: if you want to be a real hardware hacker, it just can't be done with open source tools. The root obstacle is that link:https://en.wikipedia.org/wiki/Semiconductor_fabrication_plant[Silicon fabs] don't reveal their design rules, which implies that there are no decent standard cell libraries, which implies that you can't get decent power performance and area estimates, see: https://www.quora.com/Are-there-good-open-source-standard-cell-libraries-to-learn-IC-synthesis-with-EDA-tools/answer/Ciro-Santilli The only thing you can do is purely functional designs with Verilator, but you will never know if it can be actually produced and how efficient it can be. BTW: if you want to be a real hardware hacker, it just can't be done with open source tools as of 2018. The root obstacle is that link:https://en.wikipedia.org/wiki/Semiconductor_fabrication_plant[Silicon fabs] don't publish reveal their link:https://en.wikipedia.org/wiki/Design_rule_checking[design rules], which implies that there are no decent standard cell libraries, which implies that you can't get decent power performance and area estimates, see: https://www.quora.com/Are-there-good-open-source-standard-cell-libraries-to-learn-IC-synthesis-with-EDA-tools/answer/Ciro-Santilli The only thing you can do with open source is purely functional designs with Verilator, but you will never know if it can be actually produced and how efficient it can be.
==== About the QEMU Buildroot setup ==== About the QEMU Buildroot setup
@@ -338,12 +338,12 @@ For the most part, if you just add the `--gem5` option or `-gem5` suffix to all
.... ....
./configure --gem5 && \ ./configure --gem5 && \
./build-gem5 --arch aarch64 && \ ./build-gem5 --arch aarch64 && \
./build --arch aarch64 --gem5 && \ ./build-buildroot --arch aarch64 --gem5 && \
./run --arch aarch64 --gem5 &&\ ./run --arch aarch64 --gem5 &&\
:; :;
.... ....
If you have already built <<qemu-buildroot-setup>> previously, don't be afraid, gem5 and QEMU use almost the same root filesystem and kernel, so `./build --gem` will be fast. It is currently only needed for the <<m5>> tool. If you have already built <<qemu-buildroot-setup>> previously, don't be afraid, gem5 and QEMU use almost the same root filesystem and kernel, so `./build-buildroot --gem` will be fast. It is currently only needed for the <<m5>> tool.
To get a terminal, either open a new shell and run: To get a terminal, either open a new shell and run:
@@ -646,7 +646,7 @@ sync
When you do: When you do:
.... ....
./build ./build-buildroot
.... ....
the disk image gets overwritten by a fresh filesystem and you lose all changes. the disk image gets overwritten by a fresh filesystem and you lose all changes.
@@ -2106,7 +2106,7 @@ The portability of the kernel and toolchains is amazing: change an option and mo
To use `arm` instead of x86 for example: To use `arm` instead of x86 for example:
.... ....
./build --arch arm ./build-buildroot --arch arm
./run --arch arm ./run --arch arm
.... ....
@@ -2290,7 +2290,7 @@ insmod /hello.ko
/poweroff.out /poweroff.out
' > rootfs_overlay/gitignore.sh ' > rootfs_overlay/gitignore.sh
chmod +x rootfs_overlay/gitignore.sh chmod +x rootfs_overlay/gitignore.sh
./build ./build-buildroot
./run --kernel-cli 'init=/gitignore.sh' ./run --kernel-cli 'init=/gitignore.sh'
.... ....
@@ -2383,7 +2383,7 @@ Alternatively, you can also add the comamdns to run to a new `init.d` entry to r
.... ....
cp rootfs_overlay/etc/init.d/S98 rootfs_overlay/etc/init.d/S99.gitignore cp rootfs_overlay/etc/init.d/S98 rootfs_overlay/etc/init.d/S99.gitignore
vim rootfs_overlay/etc/init.d/S99.gitignore vim rootfs_overlay/etc/init.d/S99.gitignore
./build ./build-buildroot
./run ./run
.... ....
@@ -2518,7 +2518,7 @@ With this setup, you don't even need to give a root filesystem to the kernel, it
To enable initrd instead of the default ext2 disk image, do: To enable initrd instead of the default ext2 disk image, do:
.... ....
./build --initrd ./build-buildroot --initrd
./run --initrd ./run --initrd
.... ....
@@ -2588,13 +2588,13 @@ So the only argument that QEMU needs is the `-kernel`, no `-drive` not even `-in
Try it out with: Try it out with:
.... ....
./build --initramfs -l && ./run --initramfs ./build-buildroot --initramfs -l && ./run --initramfs
.... ....
The `-l` (ell) should only be used the first time you move to / from a different root filesystem method (ext2 or cpio) to initramfs to overcome: https://stackoverflow.com/questions/49260466/why-when-i-change-br2-linux-kernel-custom-config-file-and-run-make-linux-reconfi The `-l` (ell) should only be used the first time you move to / from a different root filesystem method (ext2 or cpio) to initramfs to overcome: https://stackoverflow.com/questions/49260466/why-when-i-change-br2-linux-kernel-custom-config-file-and-run-make-linux-reconfi
.... ....
./build --initramfs && ./run --initramfs ./build-buildroot --initramfs && ./run --initramfs
.... ....
It is interesting to see how this increases the size of the kernel image if you do a: It is interesting to see how this increases the size of the kernel image if you do a:
@@ -2996,7 +2996,7 @@ More concretely:
.... ....
git -C "$(./getvar linux_src_dir)" checkout gem5/v4.15 git -C "$(./getvar linux_src_dir)" checkout gem5/v4.15
./build \ ./build-buildroot \
--arch arm \ --arch arm \
--gem5 \ --gem5 \
-l \ -l \
@@ -3066,7 +3066,7 @@ For `aarch64` we also need `-c kernel_config_fragment/display`:
.... ....
git -C "$(./getvar linux_src_dir)" checkout gem5/v4.15 git -C "$(./getvar linux_src_dir)" checkout gem5/v4.15
./build \ ./build-buildroot \
--arch aarch64 \ --arch aarch64 \
-c kernel_config_fragment/display \ -c kernel_config_fragment/display \
--gem5 \ --gem5 \
@@ -3144,7 +3144,7 @@ Once you've seen the `CONFIG_LOGO` penguin as a sanity check, you can try to go
Build and run: Build and run:
.... ....
./build --buildroot-config-fragment br2/x11 ./build-buildroot --buildroot-config-fragment br2/x11
./run --graphic ./run --graphic
.... ....
@@ -3174,7 +3174,7 @@ Not sure how well that graphics stack represents real systems, but if it does it
To x11 packages have an `xserver` prefix as in: To x11 packages have an `xserver` prefix as in:
.... ....
./build --buildroot-config-fragment br2/x11 -- xserver_xorg-server-reconfigure ./build-buildroot --buildroot-config-fragment br2/x11 -- xserver_xorg-server-reconfigure
.... ....
the easiest way to find them out is to just list `"$(./getvar build_dir)/x*`. the easiest way to find them out is to just list `"$(./getvar build_dir)/x*`.
@@ -3266,7 +3266,7 @@ By default, we use a `.config` that is a mixture of:
Use just your own exact `.config` instead: Use just your own exact `.config` instead:
.... ....
./build -K data/myconfig -l ./build-buildroot -K data/myconfig -l
.... ....
Beware that Buildroot can `sed` override some of the configurations we make no matter what, e.g. it forces `CONFIG_BLK_DEV_INITRD=y` when `BR2_TARGET_ROOTFS_CPIO` is on, so you might want to double check as explained at <<find-the-kernel-config>>. TODO check if there is a way to prevent that patching and maybe patch Buildroot for it, it is too fuzzy. People should be able to just build with whatever `.config` they want. Beware that Buildroot can `sed` override some of the configurations we make no matter what, e.g. it forces `CONFIG_BLK_DEV_INITRD=y` when `BR2_TARGET_ROOTFS_CPIO` is on, so you might want to double check as explained at <<find-the-kernel-config>>. TODO check if there is a way to prevent that patching and maybe patch Buildroot for it, it is too fuzzy. People should be able to just build with whatever `.config` they want.
@@ -3274,7 +3274,7 @@ Beware that Buildroot can `sed` override some of the configurations we make no m
Modify a single option: Modify a single option:
.... ....
./build -C 'CONFIG_FORTIFY_SOURCE=y' -l ./build-buildroot -C 'CONFIG_FORTIFY_SOURCE=y' -l
.... ....
Use an extra kernel config fragment file: Use an extra kernel config fragment file:
@@ -3284,7 +3284,7 @@ printf '
CONFIG_IKCONFIG=y CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y CONFIG_IKCONFIG_PROC=y
' > myconfig ' > myconfig
./build -c 'myconfig' -l ./build-buildroot -c 'myconfig' -l
.... ....
`-K`, `-c`, `-C` can all be used at the same time. Options passed via `-C` take precedence over `-c`, which takes precedence over `-K`. `-K`, `-c`, `-C` can all be used at the same time. Options passed via `-C` take precedence over `-c`, which takes precedence over `-K`.
@@ -5625,7 +5625,7 @@ TODO: can you get function arguments? https://stackoverflow.com/questions/276087
kprobes is an instrumentation mechanism that injects arbitrary code at a given address in a trap instruction, much like GDB. Oh, the good old kernel. :-) kprobes is an instrumentation mechanism that injects arbitrary code at a given address in a trap instruction, much like GDB. Oh, the good old kernel. :-)
.... ....
./build -C 'CONFIG_KPROBES=y' ./build-buildroot -C 'CONFIG_KPROBES=y'
.... ....
Then on guest: Then on guest:
@@ -5786,7 +5786,7 @@ Make it harder to get hacked and easier to notice that you were, at the cost of
Detects buffer overflows for us: Detects buffer overflows for us:
.... ....
./build -C 'CONFIG_FORTIFY_SOURCE=y' --linux-build-id fortify --kernel-modules ./build-buildroot -C 'CONFIG_FORTIFY_SOURCE=y' --linux-build-id fortify --kernel-modules
./run --eval-busybox 'insmod /strlen_overflow.ko' --linux-build-id fortify ./run --eval-busybox 'insmod /strlen_overflow.ko' --linux-build-id fortify
.... ....
@@ -6089,7 +6089,7 @@ tty63::respawn:-/bin/sh
# Leave one serial empty. # Leave one serial empty.
#::respawn:/sbin/getty -L ttyS3 0 vt100 #::respawn:/sbin/getty -L ttyS3 0 vt100
' >> rootfs_overlay/etc/inittab ' >> rootfs_overlay/etc/inittab
./build ./build-buildroot
./run --graphic -- \ ./run --graphic -- \
-serial telnet::1235,server,nowait \ -serial telnet::1235,server,nowait \
-serial vc:800x600 \ -serial vc:800x600 \
@@ -6291,7 +6291,7 @@ Looks like a recompile is needed to modify the image...
DRM / DRI is the new interface that supersedes `fbdev`: DRM / DRI is the new interface that supersedes `fbdev`:
.... ....
./build --buildroot-config 'BR2_PACKAGE_LIBDRM=y' --kernel-modules ./build-buildroot --buildroot-config 'BR2_PACKAGE_LIBDRM=y' --kernel-modules
./run --eval-busybox '/libdrm_modeset.out' --graphic ./run --eval-busybox '/libdrm_modeset.out' --graphic
.... ....
@@ -6302,7 +6302,7 @@ Outcome: for a few seconds, the screen that contains the terminal gets taken ove
TODO not working for `aarch64`, it takes over the screen for a few seconds and the kernel messages disappear, but the screen stays black all the time. TODO not working for `aarch64`, it takes over the screen for a few seconds and the kernel messages disappear, but the screen stays black all the time.
.... ....
./build --buildroot-config 'BR2_PACKAGE_LIBDRM=y' --kernel-modules ./build-buildroot --buildroot-config 'BR2_PACKAGE_LIBDRM=y' --kernel-modules
./run --eval-busybox '/libdrm_modeset.out' --graphic ./run --eval-busybox '/libdrm_modeset.out' --graphic
.... ....
@@ -6343,7 +6343,7 @@ Tested on: link:http://github.com/cirosantilli/linux-kernel-module-cheat/commit/
==== kmscube ==== kmscube
.... ....
./build --buildroot-config-fragment br2/kmscube ./build-buildroot --buildroot-config-fragment br2/kmscube
.... ....
Outcome: a colored spinning cube coded in OpenGL + EGL takes over your display and spins forever: https://www.youtube.com/watch?v=CqgJMgfxjsk Outcome: a colored spinning cube coded in OpenGL + EGL takes over your display and spins forever: https://www.youtube.com/watch?v=CqgJMgfxjsk
@@ -6410,9 +6410,9 @@ TODO get working.
Looks like a more raw alternative to libdrm: Looks like a more raw alternative to libdrm:
.... ....
./build --buildroot-config 'BR2_PACKABE_LIBDRI2=y' ./build-buildroot --buildroot-config 'BR2_PACKABE_LIBDRI2=y'
wget -O packages/kernel_modules/user/dri2test.c https://raw.githubusercontent.com/robclark/libdri2/master/test/dri2test.c wget -O packages/kernel_modules/user/dri2test.c https://raw.githubusercontent.com/robclark/libdri2/master/test/dri2test.c
./build --kernel-modules ./build-buildroot --kernel-modules
.... ....
but then I noticed that that example requires multiple files, and I don't feel like integrating it into our build. but then I noticed that that example requires multiple files, and I don't feel like integrating it into our build.
@@ -6434,7 +6434,7 @@ C userland test suite.
Buildroot already has a package, so it is trivial to build it: Buildroot already has a package, so it is trivial to build it:
.... ....
./build --buildroot-config 'BR2_PACKAGE_LTP_TESTSUITE=y' ./build-buildroot --buildroot-config 'BR2_PACKAGE_LTP_TESTSUITE=y'
.... ....
Then try it out with: Then try it out with:
@@ -6453,8 +6453,8 @@ TODO a large chunk of tests, the Open POSIX test suite, is disabled with a comme
POSIX userland stress. Two versions: POSIX userland stress. Two versions:
.... ....
./build --buildroot-config 'BR2_PACKAGE_STRESS=y' ./build-buildroot --buildroot-config 'BR2_PACKAGE_STRESS=y'
./build --buildroot-config 'BR2_PACKAGE_STRESS_NG=y' ./build-buildroot --buildroot-config 'BR2_PACKAGE_STRESS_NG=y'
.... ....
Websites: Websites:
@@ -6908,7 +6908,7 @@ where link:http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0190b/
Then compile with: Then compile with:
.... ....
./build --arch arm --buildroot-config-fragment br2/gpio -c kernel_config_fragment/gpio -l ./build-buildroot --arch arm --buildroot-config-fragment br2/gpio -c kernel_config_fragment/gpio -l
.... ....
then test it out with: then test it out with:
@@ -6941,7 +6941,7 @@ static void arm_sysctl_write(void *opaque, hwaddr offset,
and then rebuild with: and then rebuild with:
.... ....
./build --arch arm -c kernel_config_fragment/leds -lq ./build-buildroot --arch arm -c kernel_config_fragment/leds -lq
.... ....
But beware that one of the LEDs has a heartbeat trigger by default (specified on dts), so it will produce a lot of output. But beware that one of the LEDs has a heartbeat trigger by default (specified on dts), so it will produce a lot of output.
@@ -7160,7 +7160,7 @@ We us this exact procedure to connect to <<gdbserver>>.
Not enabled by default due to the build / runtime overhead. To enable, build with: Not enabled by default due to the build / runtime overhead. To enable, build with:
.... ....
./build --buildroot-config 'BR2_PACKAGE_OPENSSH=y' ./build-buildroot --buildroot-config 'BR2_PACKAGE_OPENSSH=y'
.... ....
Then inside the guest turn on sshd: Then inside the guest turn on sshd:
@@ -7215,7 +7215,7 @@ This has nothing to do with the Linux kernel, but it is cool:
.... ....
sudo apt-get install qemu-user sudo apt-get install qemu-user
./build --arch arm ./build-buildroot --arch arm
cd "$(./getvar target_dir)" cd "$(./getvar target_dir)"
qemu-arm -L . bin/ls qemu-arm -L . bin/ls
.... ....
@@ -7429,7 +7429,7 @@ Peter Maydell said potentially not possible nicely as of August 2018: https://st
It is also worth looking into the QEMU Guest Agent tool `qemu-gq` that can be enabled with: It is also worth looking into the QEMU Guest Agent tool `qemu-gq` that can be enabled with:
.... ....
./build --buildroot-config 'BR2_PACKAGE_QEMU=y' ./build-buildroot --buildroot-config 'BR2_PACKAGE_QEMU=y'
.... ....
See also: https://superuser.com/questions/930588/how-to-pass-commands-noninteractively-to-running-qemu-from-the-guest-qmp-via-te See also: https://superuser.com/questions/930588/how-to-pass-commands-noninteractively-to-running-qemu-from-the-guest-qmp-via-te
@@ -7678,7 +7678,7 @@ TODO `arm` and `aarch64` only seem to work with initrd since I cannot plug a wor
Then, when I tried with <<initrd>> and no disk: Then, when I tried with <<initrd>> and no disk:
.... ....
./build --arch aarch64 -i ./build-buildroot --arch aarch64 -i
./qemurr --arch aarch64 --eval-busybox '/rand_check.out;/poweroff.out;' -i ./qemurr --arch aarch64 --eval-busybox '/rand_check.out;/poweroff.out;' -i
.... ....
@@ -8203,7 +8203,7 @@ Source: link:packages/kernel_modules/user/openmp.c[]
Buildroot supports it, which makes everything just trivial: Buildroot supports it, which makes everything just trivial:
.... ....
./build --buildroot-config 'BR2_PACKAGE_OPENBLAS=y' --kernel-modules ./build-buildroot --buildroot-config 'BR2_PACKAGE_OPENBLAS=y' --kernel-modules
./run --eval-busybox '/openblas.out; echo $?' ./run --eval-busybox '/openblas.out; echo $?'
.... ....
@@ -8242,7 +8242,7 @@ cblas_dgemm( CblasColMajor, CblasNoTrans, CblasTrans,3,3,2 ,1, A,3, B,
Header only linear algebra library with a mainline Buildroot package: Header only linear algebra library with a mainline Buildroot package:
.... ....
./build --buildroot-config 'BR2_PACKAGE_EIGEN=y' --kernel-modules ./build-buildroot --buildroot-config 'BR2_PACKAGE_EIGEN=y' --kernel-modules
.... ....
Just create an array and print it: Just create an array and print it:
@@ -8277,7 +8277,7 @@ There are two ways to run PARSEC with this repo:
.... ....
./configure -gpq && \ ./configure -gpq && \
./build --arch arm --buildroot-config 'BR2_PACKAGE_PARSEC_BENCHMARK=y' --gem5 && \ ./build-buildroot --arch arm --buildroot-config 'BR2_PACKAGE_PARSEC_BENCHMARK=y' --gem5 && \
./run --arch arm --gem5 && \ ./run --arch arm --gem5 && \
:; :;
.... ....
@@ -8305,7 +8305,7 @@ but this might be a bit time consuming in gem5.
Running a benchmark of a size different than `test`, e.g. `simsmall`, requires a rebuild with: Running a benchmark of a size different than `test`, e.g. `simsmall`, requires a rebuild with:
.... ....
./build \ ./build-buildroot \
--arch arm \ --arch arm \
--buildroot-config 'BR2_PACKAGE_PARSEC_BENCHMARK=y' \ --buildroot-config 'BR2_PACKAGE_PARSEC_BENCHMARK=y' \
--buildroot-config 'BR2_PACKAGE_PARSEC_BENCHMARK_INPUT_SIZE="simsmall"' \ --buildroot-config 'BR2_PACKAGE_PARSEC_BENCHMARK_INPUT_SIZE="simsmall"' \
@@ -8363,7 +8363,7 @@ but it simply is not feasible in gem5 because it takes too long.
If you still want to run this, try it out with: If you still want to run this, try it out with:
.... ....
./build \ ./build-buildroot \
--arch aarch64 \ --arch aarch64 \
--buildroot-config 'BR2_PACKAGE_PARSEC_BENCHMARK=y' \ --buildroot-config 'BR2_PACKAGE_PARSEC_BENCHMARK=y' \
--buildroot-config 'BR2_PACKAGE_PARSEC_BENCHMARK_PARSECMGMT=y' \ --buildroot-config 'BR2_PACKAGE_PARSEC_BENCHMARK_PARSECMGMT=y' \
@@ -8394,7 +8394,7 @@ rm -rf \
"$(./getvar buildroot_build_dir)"/images/rootfs.* \ "$(./getvar buildroot_build_dir)"/images/rootfs.* \
"$(./getvar buildroot_build_dir)"/target/parsec-* \ "$(./getvar buildroot_build_dir)"/target/parsec-* \
; ;
./build --arch arm --gem5 ./build-buildroot --arch arm --gem5
.... ....
====== PARSEC benchmark hacking ====== PARSEC benchmark hacking
@@ -8420,7 +8420,7 @@ before going for the cross compile build.
Don't forget to explicitly rebuild PARSEC with: Don't forget to explicitly rebuild PARSEC with:
+ +
.... ....
./build \ ./build-buildroot \
--arch arm \ --arch arm \
--buildroot-config 'BR2_PACKAGE_PARSEC_BENCHMARK=y' \ --buildroot-config 'BR2_PACKAGE_PARSEC_BENCHMARK=y' \
--gem5 \ --gem5 \
@@ -9026,7 +9026,7 @@ Lets try to understand some stats better.
link:https://en.wikipedia.org/wiki/Time_Stamp_Counter[x86 instruction] that returns the cycle count since reset: link:https://en.wikipedia.org/wiki/Time_Stamp_Counter[x86 instruction] that returns the cycle count since reset:
.... ....
./build --gem5 --kernel-modules && \ ./build-buildroot --gem5 --kernel-modules && \
./run --eval '/rdtsc.out;m5 exit;' --gem5 && \ ./run --eval '/rdtsc.out;m5 exit;' --gem5 && \
./gem5-stat && \ ./gem5-stat && \
:; :;
@@ -9124,7 +9124,7 @@ After making changes to a Buildroot package, you must explicitly request it to b
For example, if you modify the kernel modules, which is a Buildroot package, you must rebuild with: For example, if you modify the kernel modules, which is a Buildroot package, you must rebuild with:
.... ....
./build -- kernel_modules-reconfigure ./build-buildroot -- kernel_modules-reconfigure
.... ....
where `kernel_modules` is the name of our Buildroot package that contains the kernel modules. where `kernel_modules` is the name of our Buildroot package that contains the kernel modules.
@@ -9132,20 +9132,20 @@ where `kernel_modules` is the name of our Buildroot package that contains the ke
Since rebuilding this package is such a common case, we have a shortcut for it: Since rebuilding this package is such a common case, we have a shortcut for it:
.... ....
./build --kernel-modules ./build-buildroot --kernel-modules
.... ....
==== Rebuild a Buildroot package with different build options ==== Rebuild a Buildroot package with different build options
We provide the following mechanisms: We provide the following mechanisms:
* `./build --buildroot-config-fragment data/br2`: append the Buildroot configuration file `data/br2` to a single build. Must be passed every time you run `./build`. The format is the same as link:br2/default[]. * `./build-buildroot --buildroot-config-fragment data/br2`: append the Buildroot configuration file `data/br2` to a single build. Must be passed every time you run `./build`. The format is the same as link:br2/default[].
* `./build --buildroot-config 'BR2_SOME_OPTION="myval"'`: append a single option to a single build. * `./build-buildroot --buildroot-config 'BR2_SOME_OPTION="myval"'`: append a single option to a single build.
For example, if you decide to <<enable-buildroot-compiler-optimizations>> after an initial build is finished, you must first clean the build before rebuilding: For example, if you decide to <<enable-buildroot-compiler-optimizations>> after an initial build is finished, you must first clean the build before rebuilding:
.... ....
./build --buildroot-config 'BR2_OPTIMIZE_3=y' kernel_modules-dirclean kernel_modules-reconfigure ./build-buildroot --buildroot-config 'BR2_OPTIMIZE_3=y' kernel_modules-dirclean kernel_modules-reconfigure
.... ....
as explained at: https://buildroot.org/downloads/manual/manual.html#rebuild-pkg as explained at: https://buildroot.org/downloads/manual/manual.html#rebuild-pkg
@@ -9179,7 +9179,7 @@ Then, you have two choices:
* if you already have a full `-O0` build, you can choose to rebuild just your package of interest to save some time as described at: <<rebuild-a-package-with-different-build-options>> * if you already have a full `-O0` build, you can choose to rebuild just your package of interest to save some time as described at: <<rebuild-a-package-with-different-build-options>>
+ +
.... ....
./build \ ./build-buildroot \
--buildroot-config 'BR2_OPTIMIZE_3=y' \ --buildroot-config 'BR2_OPTIMIZE_3=y' \
-- \ -- \
kernel_modules-dirclean \ kernel_modules-dirclean \
@@ -9196,7 +9196,7 @@ Kernel-wise it should be fine though due to: <<kernel-o0>>
+ +
.... ....
mv out out~ mv out out~
./build --buildroot-config 'BR2_OPTIMIZE_3=y' ./build-buildroot --buildroot-config 'BR2_OPTIMIZE_3=y'
.... ....
=== Find Buildroot options with make menuconfig === Find Buildroot options with make menuconfig
@@ -9304,7 +9304,7 @@ If none of those methods are flexible enough for you, create a new package as fo
* don't forget to rebuild with: * don't forget to rebuild with:
+ +
.... ....
./build -- sample_package-reconfigure ./build-buildroot -- sample_package-reconfigure
./run --eval-busybox '/sample_package.out' ./run --eval-busybox '/sample_package.out'
.... ....
+ +
@@ -9322,11 +9322,11 @@ If you want to have multiple versions of the GCC toolchain or root filesystem, t
.... ....
# Build master. # Build master.
./build ./build-buildroot
# Build another branch. # Build another branch.
git -C "$(./getvar buildroot_src_dir)" checkout 2018.05 git -C "$(./getvar buildroot_src_dir)" checkout 2018.05
./build --buildroot-build-id 2018.05 ./build-buildroot --buildroot-build-id 2018.05
# Restore master. # Restore master.
git -C "$(./getvar buildroot_src_dir)" checkout - git -C "$(./getvar buildroot_src_dir)" checkout -
@@ -9345,9 +9345,9 @@ Since the Linux kernel is so important to us, we have created a convenient dedic
For example, if you want to keep two builds around, one for the latest Linux version, and the other for Linux `v4.16`: For example, if you want to keep two builds around, one for the latest Linux version, and the other for Linux `v4.16`:
.... ....
./build ./build-buildroot
git -C "$(./getvar linux_src_dir)" checkout v4.16 git -C "$(./getvar linux_src_dir)" checkout v4.16
./build --linux-build-id v4.16 ./build-buildroot --linux-build-id v4.16
git -C "$(./getvar linux_src_dir)" checkout - git -C "$(./getvar linux_src_dir)" checkout -
./run ./run
./run --linux-build-id v4.16 ./run --linux-build-id v4.16
@@ -9446,7 +9446,7 @@ Therefore, future builds for `worktree-id` will not automatically modify the rev
.... ....
git -C data/gem5/some-branch checkout some-branch-v2 git -C data/gem5/some-branch checkout some-branch-v2
./build --gem5 --gem5-build-id some-branch --gem5-worktree some-branch ./build-buildroot --gem5 --gem5-build-id some-branch --gem5-worktree some-branch
.... ....
`--gem5-worktree` is only required if you have multiple gem5 checkouts, e.g. it would not be required for multiple builds of the same tree, e.g. a <<gem5-debug-build>> and a non-debug one. `--gem5-worktree` is only required if you have multiple gem5 checkouts, e.g. it would not be required for multiple builds of the same tree, e.g. a <<gem5-debug-build>> and a non-debug one.
@@ -9476,7 +9476,7 @@ Therefore the performance different is very big, making debug mode almost unusab
This hack-ish technique allows us to rebuild just one package at a time: This hack-ish technique allows us to rebuild just one package at a time:
.... ....
./build KERNEL_MODULE_VERSION=mybranch ./build-buildroot KERNEL_MODULE_VERSION=mybranch
.... ....
and now you can see that a new version of `kernel_modules` was built and put inside the image: and now you can see that a new version of `kernel_modules` was built and put inside the image:
@@ -9490,7 +9490,7 @@ Unfortunately we don't have a nice runtime selection with `./run` implemented cu
TODO: is there a way to do it nicely for `*_OVERRIDE_SRCDIR` packages from link:buildroot_override[]? I tried: TODO: is there a way to do it nicely for `*_OVERRIDE_SRCDIR` packages from link:buildroot_override[]? I tried:
.... ....
./build -l LINUX_VERSION=mybranch ./build-buildroot -l LINUX_VERSION=mybranch
.... ....
but it fails with: but it fails with:
@@ -9502,7 +9502,7 @@ linux/linux.mk:492: *** LINUX_SITE cannot be empty when LINUX_SOURCE is not. St
and theI tried: and theI tried:
.... ....
./build -l LINUX_VERSION=mybranch LINUX_SITE="$(pwd)/linux" ./build-buildroot -l LINUX_VERSION=mybranch LINUX_SITE="$(pwd)/linux"
.... ....
but it feels hackish, and the build was slower than normal, looks like the build was single threaded? but it feels hackish, and the build was slower than normal, looks like the build was single threaded?
@@ -9518,7 +9518,7 @@ Maybe you need to increase the filesystem size (BR2_TARGET_ROOTFS_EXT2_SIZE)
The solution is to simply add: The solution is to simply add:
.... ....
./build --buildroot-config 'BR2_TARGET_ROOTFS_EXT2_SIZE="512M"' ./build-buildroot --buildroot-config 'BR2_TARGET_ROOTFS_EXT2_SIZE="512M"'
.... ....
where 512Mb is "large enough". where 512Mb is "large enough".
@@ -9555,14 +9555,14 @@ which contributed to a large part of the slowness.
Test how Buildroot deals with many files with: Test how Buildroot deals with many files with:
.... ....
./build \ ./build-buildroot \
--buildroot-config 'BR2_PACKAGE_LKMC_MANY_FILES=y' \ --buildroot-config 'BR2_PACKAGE_LKMC_MANY_FILES=y' \
-- \ -- \
lkmc_many_files-reconfigure \ lkmc_many_files-reconfigure \
|& \ |& \
ts -i '%.s' \ ts -i '%.s' \
; ;
./build |& ts -i '%.s' ./build-buildroot |& ts -i '%.s'
.... ....
and notice how the second build, which does not rebuilt the package at all, still gets stuck in the `RPATH` check forever without our Buildroot patch. and notice how the second build, which does not rebuilt the package at all, still gets stuck in the `RPATH` check forever without our Buildroot patch.
@@ -9736,7 +9736,7 @@ The build times are calculated after doing `./configure` and link:https://buildr
Sample build time at 2c12b21b304178a81c9912817b782ead0286d282: 28 minutes, 15 with full ccache hits. Breakdown: 19% GCC, 13% Linux kernel, 7% uclibc, 6% host-python, 5% host-qemu, 5% host-gdb, 2% host-binutils Sample build time at 2c12b21b304178a81c9912817b782ead0286d282: 28 minutes, 15 with full ccache hits. Breakdown: 19% GCC, 13% Linux kernel, 7% uclibc, 6% host-python, 5% host-qemu, 5% host-gdb, 2% host-binutils
Single file change on `./build kernel_modules-reconfigure`: 7 seconds. Single file change on `./build-buildroot kernel_modules-reconfigure`: 7 seconds.
Buildroot automatically stores build timestamps as milliseconds since Epoch. Convert to minutes: Buildroot automatically stores build timestamps as milliseconds since Epoch. Convert to minutes:
@@ -9754,7 +9754,7 @@ cat ../linux-kernel-module-cheat-regression/*/build-time.log
===== Find which packages are making the build slow and big ===== Find which packages are making the build slow and big
.... ....
./build --skip-configure -- graph-build graph-size graph-depends ./build-buildroot --skip-configure -- graph-build graph-size graph-depends
cd "$(./getvar buildroot_build_dir)/graphs" cd "$(./getvar buildroot_build_dir)/graphs"
xdg-open build.pie-packages.pdf xdg-open build.pie-packages.pdf
xdg-open graph-depends.pdf xdg-open graph-depends.pdf
@@ -9890,22 +9890,43 @@ All our build outputs are stored under `out/`, so the coarsest and most effectiv
rm -rf out rm -rf out
.... ....
This implies a full rebuild for all archs however, so you might first want to explore finer grained cleans. This implies a full rebuild for all archs however, so you might first want to explore finer grained cleans first.
To only nuke one architecture, do: All our individual `build-*` scripts have a `--clean` option to completely nuke their builds:
.... ....
rm -rf "$(./getvar buildroot_build_dir)" ./build-gem5 --clean
./build-qemu --clean
./build-buildroot --clean
.... ....
Only nuke one one package: Verify with:
.... ....
rm -rf "$(./getvar buildroot_build_dir)/build/host-qemu-custom" ls "$(./getvar qemu_build_dir)"
./build ls "$(./getvar gem5_build_dir)"
ls "$(./getvar buildroot_build_dir)"
.... ....
This is sometimes necessary when changing the version of the submodules, and then builds fail. We should try to understand why and report bugs. Note that host tools like QEMU and gem5 store all archs in a single directory to factor out build objects, so cleaning one arch will clean all of them.
To only nuke one one Buildroot package, we can use the link:https://buildroot.org/downloads/manual/manual.html#pkg-build-steps[]`-dirclean`] Buildroot target:
....
./build-buildroot --no-all -- <package-name>-dirclean
....
e.g.:
....
./build-buildroot --no-all -- kernel_modules-dirclean
....
Verify with:
....
ls "$(./getvar build_dir)"
....
=== Directory structure === Directory structure
@@ -9945,7 +9966,7 @@ Every directory inside it is a Buildroot package.
Those packages get automatically added to Buildroot's `BR2_EXTERNAL`, so all you need to do is to turn them on during build, e.g.: Those packages get automatically added to Buildroot's `BR2_EXTERNAL`, so all you need to do is to turn them on during build, e.g.:
.... ....
./build --buildroot-config BR2_SAMPLE_PACKAGE=y ./build-buildroot --buildroot-config BR2_SAMPLE_PACKAGE=y
.... ....
If you want to add something to the root filesystem, possibly involving cross-compilation, then packages are the way to go. If you want to add something to the root filesystem, possibly involving cross-compilation, then packages are the way to go.
@@ -10052,18 +10073,19 @@ For example, when updating from QEMU `v2.12.0` to `v3.0.0-rc3`, the Linux kernel
We then bisected it as explained at: https://stackoverflow.com/questions/4713088/how-to-use-git-bisect/22592593#22592593 with the link:qemu-bisect-boot[] script: We then bisected it as explained at: https://stackoverflow.com/questions/4713088/how-to-use-git-bisect/22592593#22592593 with the link:qemu-bisect-boot[] script:
.... ....
root_dir="$(pwd)"
cd "$(./getvar qemu_src_dir)" cd "$(./getvar qemu_src_dir)"
git bisect start git bisect start
# Check that our test script fails on v3.0.0-rc3 as expected, and mark it as bad. # Check that our test script fails on v3.0.0-rc3 as expected, and mark it as bad.
../../qemu-bisect-boot "${root_dir}/qemu-bisect-boot"
# Should output 1. # Should output 1.
echo #? echo #?
git bisect bad git bisect bad
# Same for the good end. # Same for the good end.
git checkout v2.12.0 git checkout v2.12.0
../qemu-bisect-boot "${root_dir}/qemu-bisect-boot"
# Should output 0. # Should output 0.
echo #? echo #?
git bisect good git bisect good
@@ -10073,9 +10095,8 @@ git bisect run ../biset-qemu-linux-boot
# Clean up after the bisection. # Clean up after the bisection.
git bisect reset git bisect reset
cd ..
git submodule update git submodule update
rm -rf "$(./getvar --arch arm build_dir)/host-qemu-custom.bisect" "${root_dir}/build-qemu" --clean --qemu-build-id bisect
.... ....
An example of Linux kernel commit bisection on gem5 boots can be found at: link:bisect-linux-boot-gem5[]. An example of Linux kernel commit bisection on gem5 boots can be found at: link:bisect-linux-boot-gem5[].

68
bench-all
View File

@@ -2,8 +2,8 @@
set -eux set -eux
root_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)" root_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)"
bench_all=false bench_all=false
bench_build=false bench_buildroot_build=false
bench_buildroot_baseline=false bench_buildroot_baseline_build=false
bench_gem5_build=false bench_gem5_build=false
bench_linux_boot=false bench_linux_boot=false
default_arch=x86_64 default_arch=x86_64
@@ -17,10 +17,10 @@ while getopts Aa:Bbglu OPT; do
default_arch="$OPTARG" default_arch="$OPTARG"
;; ;;
b) b)
bench_build=true bench_buildroot_build=true
;; ;;
B) B)
bench_buildroot_baseline=true bench_buildroot_baseline_build=true
;; ;;
g) g)
bench_gem5_build=true bench_gem5_build=true
@@ -39,16 +39,16 @@ done
shift "$(($OPTIND - 1))" shift "$(($OPTIND - 1))"
comment="${1:-}" comment="${1:-}"
if \ if \
! "$bench_build" && \ ! "$bench_buildroot_build" && \
! "$bench_buildroot_baseline" && \ ! "$bench_buildroot_baseline_build" && \
! "$bench_gem5_build" && \ ! "$bench_gem5_build" && \
! "$bench_linux_boot" \ ! "$bench_linux_boot" \
; then ; then
bench_all=true bench_all=true
fi fi
if "$bench_all"; then if "$bench_all"; then
bench_build=true bench_buildroot_build=true
bench_buildroot_baseline=true bench_buildroot_baseline_build=true
bench_gem5_build=true bench_gem5_build=true
bench_linux_boot=true bench_linux_boot=true
fi fi
@@ -68,33 +68,31 @@ dir_basename="${seq_id}_$("$getvar" sha)"
new_dir="${benchmark_repo}/${dir_basename}" new_dir="${benchmark_repo}/${dir_basename}"
mkdir "$new_dir" mkdir "$new_dir"
if "$bench_build"; then do_bench_buildroot_build() (
common_arch="$default_arch" arch="$default_arch"
common_suffix=bench build_id=bench
common_buildroot_build_dir="$("$getvar" --arch "$common_arch" --buildroot-build-id "$common_suffix" buildroot_build_dir)" if [ "$1" = baseline ]; then
common_build_dir="$("$getvar" --arch "$common_arch" --buildroot-build-id "$common_suffix" build_dir)" baseline=--baseline
rm -rf "$common_buildroot_build_dir" baseline_suffix=-baseline
./build --arch "$common_arch" --buildroot-config 'BR2_CCACHE=n' --buildroot-build-id "$common_suffix" else
cp "${common_build_dir}/build-time.log" "${new_dir}/build-time-${common_arch}.log" baseline=
rm -rf "$common_buildroot_build_dir" baseline_suffix=
fi
common_build_dir="$("$getvar" --arch "$arch" --buildroot-build-id "$build_id" build_dir)"
common_images_dir="$("$getvar" --arch "$arch" --buildroot-build-id "$build_id" images_dir)"
"${root_dir}/build-buildroot" --arch "$arch" $baseline --buildroot-build-id "$build_id" --clean
"${root_dir}/build-buildroot" --arch "$arch" $baseline --buildroot-build-id "$build_id" --buildroot-config 'BR2_CCACHE=n'
cp "${common_build_dir}/build-time.log" "${new_dir}/buildroot-build-time-${baseline_suffix}${arch}.log"
wc -c "${images_dir}/"* > "${new_dir}/buildroot-image-size-${baseline_suffix}${arch}.log"
"${root_dir}/build-buildroot" --arch "$arch" $baseline --buildroot-build-id "$build_id" --clean
)
if "$bench_buildroot_build"; then
do_bench_buildroot_build
fi fi
if "$bench_buildroot_baseline"; then if "$bench_buildroot_baseline_build"; then
common_buildroot_src_dir="$("$getvar" --arch "$default_arch" build_dir)" do_bench_buildroot_build baseline
cd "${common_buildroot_src_dir}"
git clean -xdf
make "qemu_${default_arch}_defconfig"
printf '
BR2_CCACHE=y
BR2_TARGET_ROOTFS_CPIO=y
BR2_TARGET_ROOTFS_EXT2=n
' >>.config
make olddefconfig
make source
time env -u LD_LIBRARY_PATH make BR2_JLEVEL="$(nproc)"
cp output/build/build-time.log "${new_dir}/baseline-build-time-${default_arch}.log"
wc -c output/images/* > "${new_dir}/baseline-image-size-${default_arch}.log"
git clean -xdf
fi fi
if "$bench_gem5_build"; then if "$bench_gem5_build"; then
@@ -117,8 +115,8 @@ fi
if "$bench_linux_boot"; then if "$bench_linux_boot"; then
cd "${root_dir}" cd "${root_dir}"
./build-all "${root_dir}/build-all"
./bench-boot -t 3 "${root_dir}/bench-boot" -t 3
cp "$(${root_dir}/getvar bench_boot)" "$new_dir" cp "$(${root_dir}/getvar bench_boot)" "$new_dir"
fi fi

4
bisect-linux-boot-gem5
View File

@@ -21,10 +21,10 @@ More information at: https://github.com/cirosantilli/linux-kernel-module-cheat#b
}, },
) )
args = common.setup(parser) args = common.setup(parser)
# We need a clean rebuild becuase rebuilds at different revisions: # We need a clean rebuild because rebuilds at different revisions:
# - may fail # - may fail
# - may not actually rebuild all files, e.g. on header changes # - may not actually rebuild all files, e.g. on header changes
shutil.rmtree(common.linux_variant_dir) common.rmrf(common.linux_variant_dir)
assert build.main(args) == 0 assert build.main(args) == 0
status = run.main(args, { status = run.main(args, {
'eval': 'm5 exit', 'eval': 'm5 exit',

294
build
View File

@@ -1,294 +0,0 @@
#!/usr/bin/env python3
import multiprocessing
import os
import pathlib
import shutil
import subprocess
import sys
import re
import common
defaults = {
'buildroot_bare_kernel': False,
'buildroot_config': [],
'buildroot_config_fragment': [],
'initramfs': False,
'initrd': False,
'kernel_config': [],
'kernel_config_fragment': [],
'kernel_custom_config_file': None,
'kernel_modules': False,
'linux_reconfigure': False,
'nproc': None,
'skip_configure': False,
'verbose': False,
'extra_make_args': [],
}
def path_relative_to_buildroot(abspath):
return os.path.relpath(abspath, common.buildroot_src_dir)
def main(args, extra_args=None):
global defaults
args = common.resolve_args(defaults, args, extra_args)
os.makedirs(common.out_dir, exist_ok=True)
extra_make_args = args.extra_make_args.copy()
if args.kernel_modules:
extra_make_args.append('kernel_modules-reconfigure')
if args.linux_reconfigure:
extra_make_args.append('linux-reconfigure')
if args.gem5:
extra_make_args.append('gem5-reconfigure')
if args.nproc is None:
nproc = multiprocessing.cpu_count()
else:
nproc = args.nproc
if args.arch == 'x86_64':
defconfig = 'qemu_x86_64_defconfig'
elif args.arch == 'arm':
defconfig = 'qemu_arm_vexpress_defconfig'
elif args.arch == 'aarch64':
defconfig = 'qemu_aarch64_virt_defconfig'
# Configure.
if not args.skip_configure:
# Initial make configure.
#cd "${common_buildroot_src_dir}"
#for p in $(find "${common_root_dir}/patches/buildroot/" -maxdepth 1 -name '*.patch' -print); do
# patch -N -r - -p 1 < "$p" || :
#done
br2_external_dirs = []
packages_dir = os.path.join(common.root_dir, 'packages')
for package_dir in os.listdir(packages_dir):
package_dir_abs = os.path.join(packages_dir, package_dir)
if os.path.isdir(package_dir_abs):
br2_external_dirs.append(path_relative_to_buildroot(package_dir_abs))
br2_external_str = ':'.join(br2_external_dirs)
subprocess.check_call(
[
'make',
'O={}'.format(common.buildroot_build_dir),
'BR2_EXTERNAL={}'.format(br2_external_str),
defconfig,
],
cwd=common.buildroot_src_dir,
)
# Extra buildroot configs.
buildroot_configs = args.buildroot_config
buildroot_configs.extend([
'BR2_JLEVEL={}'.format(nproc),
'BR2_DL_DIR="{}"'.format(common.dl_dir),
'BR2_GLOBAL_PATCH_DIR="{}"'.format(
path_relative_to_buildroot(os.path.join(common.root_dir, 'patches', 'global'))),
'BR2_PACKAGE_BUSYBOX_CONFIG_FRAGMENT_FILES="{}"'.format(
path_relative_to_buildroot(os.path.join(common.root_dir, 'busybox_config_fragment'))),
'BR2_PACKAGE_OVERRIDE_FILE="{}"'.format(
path_relative_to_buildroot(os.path.join(common.root_dir, 'buildroot_override'))),
'BR2_ROOTFS_OVERLAY="{}"'.format(
path_relative_to_buildroot(os.path.join(common.root_dir, 'rootfs_overlay'))),
'BR2_ROOTFS_POST_BUILD_SCRIPT="{}"'.format(
path_relative_to_buildroot(os.path.join(common.root_dir, 'rootfs_post_build_script'))),
'BR2_ROOTFS_USERS_TABLES="{}"'.format(
path_relative_to_buildroot(os.path.join(common.root_dir, 'user_table'))),
])
if args.gem5:
buildroot_configs.append('BR2_PACKAGE_GEM5=y')
if args.initramfs:
buildroot_configs.extend([
'BR2_TARGET_ROOTFS_CPIO=n',
'BR2_TARGET_ROOTFS_EXT2=n',
'BR2_TARGET_ROOTFS_INITRAMFS=y',
])
if args.initrd:
buildroot_configs.extend([
'BR2_TARGET_ROOTFS_CPIO=y',
'BR2_TARGET_ROOTFS_EXT2=n'
'BR2_TARGET_ROOTFS_INITRAMFS=n',
])
buildroot_config_fragments = [
os.path.join(common.root_dir, 'buildroot_config', 'default')
] + args.buildroot_config_fragment
# Decide kernel configuration.
kernel_config_fragments = []
if True:
# CLI kernel configurations.
kernel_config_fragment_cli_path = os.path.join(common.buildroot_build_dir, 'lkmc_kernel_config_fragment_cli')
kernel_config_cli_str = '\n'.join(args.kernel_config)
if os.path.exists(kernel_config_fragment_cli_path):
with open(kernel_config_fragment_cli_path, 'r') as kernel_config_fragment_cli_file:
kernel_config_cli_str_old = kernel_config_fragment_cli_file.read()
if kernel_config_cli_str != kernel_config_cli_str_old:
# Only update if modified, otherwise Buildroot rebuilds the
# kernel always rebuilds kernel every time.
do_write = True
else:
do_write = True
with open(kernel_config_fragment_cli_path, 'w') as kernel_config_fragment_cli_file:
kernel_config_fragment_cli_file.write(kernel_config_cli_str)
kernel_config_fragments.append(os.path.join(kernel_config_fragment_cli_path))
if True:
# Kernel configuration fragments.
if args.kernel_custom_config_file is not None:
if os.path.exists(args.kernel_custom_config_file):
buildroot_configs.extend([
'BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y',
'BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE=\"{}\"'.format(args.kernel_custom_config_file),
])
if args.linux_reconfigure:
pathlib.Path(args.kernel_custom_config_file).touch()
else:
raise Exception('Kernel config fragment file does not exist: {}'.format(args.kernel_custom_config_file))
default_kernel_config_fragments = []
else:
kernel_config_fragment_dir = os.path.join(common.root_dir, 'kernel_config')
default_kernel_config_fragments = ['min', 'default']
if args.linux_reconfigure:
# https://stackoverflow.com/questions/49260466/why-when-i-change-br2-linux-kernel-custom-config-file-and-run-make-linux-reconfi
pathlib.Path(os.path.join(kernel_config_fragment_dir, 'min')).touch()
for i, default_kernel_config_fragment in enumerate(default_kernel_config_fragments):
default_kernel_config_fragments[i] = os.path.join(kernel_config_fragment_dir, default_kernel_config_fragment)
kernel_config_fragments.extend(default_kernel_config_fragments)
for i, frag in enumerate(kernel_config_fragments):
kernel_config_fragments[i] = path_relative_to_buildroot(frag)
buildroot_kernel_config_fragment_str = 'BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES="{}"'.format(' '.join(kernel_config_fragments))
buildroot_configs.append(buildroot_kernel_config_fragment_str)
# Write extra configs into the Buildroot config file.
# TODO Can't get rid of these for now with nice fragments:
# http://stackoverflow.com/questions/44078245/is-it-possible-to-use-config-fragments-with-buildroots-config
with open(common.buildroot_config_file, 'a') as br2_config_file:
for buildroot_config_fragment in buildroot_config_fragments:
with open(buildroot_config_fragment, 'r') as br2_config_fragment:
for line in br2_config_fragment:
br2_config_file.write(line)
for buildroot_config in buildroot_configs:
br2_config_file.write(buildroot_config + '\n')
subprocess.check_call(
[
'make',
'O={}'.format(common.buildroot_build_dir),
'olddefconfig',
],
cwd=common.buildroot_src_dir,
)
common.mkdir()
# Manage Linux kernel and QEMU variants.
def symlink_buildroot_variant(custom_dir, variant_dir):
if os.path.islink(custom_dir):
os.unlink(custom_dir)
elif os.path.isdir(custom_dir):
# Migration for existing builds.
shutil.move(custom_dir, variant_dir)
os.makedirs(variant_dir, exist_ok=True)
os.symlink(variant_dir, custom_dir)
symlink_buildroot_variant(common.linux_build_dir, common.linux_variant_dir)
# Do the actual build.
assert common.run_cmd(
[
'make',
'GEM5_LKMC_SRCDIR="{}"'.format(common.gem5_src_dir),
'O={}'.format(common.buildroot_build_dir),
'V={}'.format(int(args.verbose)),
] +
extra_make_args +
['all']
,
out_file=os.path.join(common.buildroot_build_dir, 'lkmc.log'),
delete_env=['LD_LIBRARY_PATH'],
cwd=common.buildroot_src_dir,
) == 0
# Create the qcow2 from ext2. This is optional, because gem5
# does not need the qcow2.
if os.path.exists(common.qemu_img_executable):
assert common.run_cmd([
common.qemu_img_executable,
'-T', 'pr_manager_run,file=/dev/null',
'convert',
'-f', 'raw',
'-O', 'qcow2',
common.ext2_file,
common.qcow2_file,
]) == 0
return 0
def get_argparse():
parser = common.get_argparse(argparse_args={'description':'Run Linux on an emulator'})
parser.add_argument(
'-B', '--buildroot-config', default=defaults['buildroot_config'], action='append',
help='''Add a single Buildroot config to the current build.
Example value: 'BR2_TARGET_ROOTFS_EXT2_SIZE="512M"'.
Can be used multiple times to add multiple configs.
Takes precedence over any Buildroot config files.
'''
)
parser.add_argument(
'-b', '--buildroot-config-fragment', default=defaults['buildroot_config_fragment'], action='append',
help='''Also use the given Buildroot configuration fragment file.
Pass multiple times to use multiple fragment files.'''
)
parser.add_argument(
'-C', '--kernel-config', default=defaults['kernel_config'], action='append',
help='''Add a single kernel config configs to the current build.
Example value: 'CONFIG_FORTIFY_SOURCE=y'.
Can be used multiple times to add multiple configs.
Takes precedence over any Buildroot config files.
'''
)
parser.add_argument(
'-c', '--kernel-config-fragment', default=defaults['kernel_config_fragment'], action='append',
help='''Also use the given kernel configuration fragment file.
Pass multiple times to use multiple fragment files.'''
)
parser.add_argument(
'-I', '--initramfs', default=defaults['initramfs'], action='store_true',
)
parser.add_argument(
'-i', '--initrd', default=defaults['initrd'], action='store_true',
)
parser.add_argument(
'-j', '--nproc', default=defaults['nproc'], type=int,
help='Number of processors to use for the build. Default: all.'
)
parser.add_argument(
'--skip-configure', default=defaults['skip_configure'], action='store_true',
help='''Skip the Buildroot configuration. Saves a few seconds,
but requires you to know what you are doing :-)'''
)
parser.add_argument(
'-K', '--kernel-custom-config-file', default=defaults['kernel_custom_config_file'],
help='''Ignore all default kernel configurations and use this file instead.
Still uses options explicitly passed with `-C` and `-c` on top of it.'''
)
parser.add_argument(
'-k', '--kernel-modules', default=defaults['kernel_modules'], action='store_true',
help='Reconfigure and rebuild the kernel modules'
)
parser.add_argument(
'-l', '--linux-reconfigure', default=defaults['linux_reconfigure'], action='store_true',
help='''Reconfigure and rebuild the Linux kernel.
Touches kernel configuration files to overcome:
https://stackoverflow.com/questions/49260466/why-when-i-change-br2-linux-kernel-custom-config-file-and-run-make-linux-reconfi'''
)
parser.add_argument(
'-v', '--verbose', default=defaults['verbose'], action='store_true',
help='Do a verbose build'
)
parser.add_argument(
'extra_make_args', default=defaults['extra_make_args'], metavar='extra-make-args', nargs='*'
)
return parser
if __name__ == '__main__':
parser = get_argparse()
args = common.setup(parser)
sys.exit(main(args))

313
build-buildroot Executable file
View File

@@ -0,0 +1,313 @@
#!/usr/bin/env python3
import multiprocessing
import os
import pathlib
import shutil
import subprocess
import sys
import re
import common
defaults = {
'baseline': False,
'buildroot_bare_kernel': False,
'buildroot_config': [],
'buildroot_config_fragment': [],
'initramfs': False,
'initrd': False,
'kernel_config': [],
'kernel_config_fragment': [],
'kernel_custom_config_file': None,
'kernel_modules': False,
'linux_reconfigure': False,
'no_all': False,
'nproc': None,
'skip_configure': False,
'verbose': False,
'extra_make_args': [],
}
def path_relative_to_buildroot(abspath):
return os.path.relpath(abspath, common.buildroot_src_dir)
def main(args, extra_args=None):
global defaults
args = common.resolve_args(defaults, args, extra_args)
if args.clean:
common.rmrf(common.buildroot_build_dir)
else:
os.makedirs(common.out_dir, exist_ok=True)
extra_make_args = args.extra_make_args.copy()
if args.kernel_modules:
extra_make_args.append('kernel_modules-reconfigure')
if args.linux_reconfigure:
extra_make_args.append('linux-reconfigure')
if args.gem5:
extra_make_args.append('gem5-reconfigure')
if args.nproc is None:
nproc = multiprocessing.cpu_count()
else:
nproc = args.nproc
if args.arch == 'x86_64':
defconfig = 'qemu_x86_64_defconfig'
elif args.arch == 'arm':
defconfig = 'qemu_arm_vexpress_defconfig'
elif args.arch == 'aarch64':
defconfig = 'qemu_aarch64_virt_defconfig'
# Configure.
if not args.skip_configure:
# Initial make configure.
# TODO port and test.
#cd "${common_buildroot_src_dir}"
#for p in $(find "${common_root_dir}/patches/buildroot/" -maxdepth 1 -name '*.patch' -print); do
# patch -N -r - -p 1 < "$p" || :
#done
br2_external_dirs = []
packages_dir = os.path.join(common.root_dir, 'packages')
for package_dir in os.listdir(packages_dir):
package_dir_abs = os.path.join(packages_dir, package_dir)
if os.path.isdir(package_dir_abs):
br2_external_dirs.append(path_relative_to_buildroot(package_dir_abs))
br2_external_str = ':'.join(br2_external_dirs)
subprocess.check_call(
[
'make',
'O={}'.format(common.buildroot_build_dir),
'BR2_EXTERNAL={}'.format(br2_external_str),
defconfig,
],
cwd=common.buildroot_src_dir,
)
if not args.baseline:
# Extra buildroot configs.
buildroot_configs = args.buildroot_config
buildroot_configs.extend([
'BR2_JLEVEL={}'.format(nproc),
'BR2_DL_DIR="{}"'.format(common.dl_dir),
'BR2_GLOBAL_PATCH_DIR="{}"'.format(
path_relative_to_buildroot(os.path.join(common.root_dir, 'patches', 'global'))),
'BR2_PACKAGE_BUSYBOX_CONFIG_FRAGMENT_FILES="{}"'.format(
path_relative_to_buildroot(os.path.join(common.root_dir, 'busybox_config_fragment'))),
'BR2_PACKAGE_OVERRIDE_FILE="{}"'.format(
path_relative_to_buildroot(os.path.join(common.root_dir, 'buildroot_override'))),
'BR2_ROOTFS_OVERLAY="{}"'.format(
path_relative_to_buildroot(os.path.join(common.root_dir, 'rootfs_overlay'))),
'BR2_ROOTFS_POST_BUILD_SCRIPT="{}"'.format(
path_relative_to_buildroot(os.path.join(common.root_dir, 'rootfs_post_build_script'))),
'BR2_ROOTFS_USERS_TABLES="{}"'.format(
path_relative_to_buildroot(os.path.join(common.root_dir, 'user_table'))),
])
if args.gem5:
buildroot_configs.append('BR2_PACKAGE_GEM5=y')
if args.initramfs:
buildroot_configs.extend([
'BR2_TARGET_ROOTFS_CPIO=n',
'BR2_TARGET_ROOTFS_EXT2=n',
'BR2_TARGET_ROOTFS_INITRAMFS=y',
])
if args.initrd:
buildroot_configs.extend([
'BR2_TARGET_ROOTFS_CPIO=y',
'BR2_TARGET_ROOTFS_EXT2=n'
'BR2_TARGET_ROOTFS_INITRAMFS=n',
])
buildroot_config_fragments = [
os.path.join(common.root_dir, 'buildroot_config', 'default')
] + args.buildroot_config_fragment
# Decide kernel configuration.
kernel_config_fragments = []
if True:
# CLI kernel configurations.
kernel_config_fragment_cli_path = os.path.join(common.buildroot_build_dir, 'lkmc_kernel_config_fragment_cli')
kernel_config_cli_str = '\n'.join(args.kernel_config)
if os.path.exists(kernel_config_fragment_cli_path):
with open(kernel_config_fragment_cli_path, 'r') as kernel_config_fragment_cli_file:
kernel_config_cli_str_old = kernel_config_fragment_cli_file.read()
if kernel_config_cli_str != kernel_config_cli_str_old:
# Only update if modified, otherwise Buildroot rebuilds the
# kernel always rebuilds kernel every time.
do_write = True
else:
do_write = True
with open(kernel_config_fragment_cli_path, 'w') as kernel_config_fragment_cli_file:
kernel_config_fragment_cli_file.write(kernel_config_cli_str)
kernel_config_fragments.append(os.path.join(kernel_config_fragment_cli_path))
if True:
# Kernel configuration fragments.
if args.kernel_custom_config_file is not None:
if os.path.exists(args.kernel_custom_config_file):
buildroot_configs.extend([
'BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y',
'BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE=\"{}\"'.format(args.kernel_custom_config_file),
])
if args.linux_reconfigure:
pathlib.Path(args.kernel_custom_config_file).touch()
else:
raise Exception('Kernel config fragment file does not exist: {}'.format(args.kernel_custom_config_file))
default_kernel_config_fragments = []
else:
kernel_config_fragment_dir = os.path.join(common.root_dir, 'kernel_config')
default_kernel_config_fragments = ['min', 'default']
if args.linux_reconfigure:
# https://stackoverflow.com/questions/49260466/why-when-i-change-br2-linux-kernel-custom-config-file-and-run-make-linux-reconfi
pathlib.Path(os.path.join(kernel_config_fragment_dir, 'min')).touch()
for i, default_kernel_config_fragment in enumerate(default_kernel_config_fragments):
default_kernel_config_fragments[i] = os.path.join(kernel_config_fragment_dir, default_kernel_config_fragment)
kernel_config_fragments.extend(default_kernel_config_fragments)
for i, frag in enumerate(kernel_config_fragments):
kernel_config_fragments[i] = path_relative_to_buildroot(frag)
buildroot_kernel_config_fragment_str = 'BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES="{}"'.format(' '.join(kernel_config_fragments))
buildroot_configs.append(buildroot_kernel_config_fragment_str)
# Write extra configs into the Buildroot config file.
# TODO Can't get rid of these for now with nice fragments:
# http://stackoverflow.com/questions/44078245/is-it-possible-to-use-config-fragments-with-buildroots-config
with open(common.buildroot_config_file, 'a') as br2_config_file:
for buildroot_config_fragment in buildroot_config_fragments:
with open(buildroot_config_fragment, 'r') as br2_config_fragment:
for line in br2_config_fragment:
br2_config_file.write(line)
for buildroot_config in buildroot_configs:
br2_config_file.write(buildroot_config + '\n')
subprocess.check_call(
[
'make',
'O={}'.format(common.buildroot_build_dir),
'olddefconfig',
],
cwd=common.buildroot_src_dir,
)
common.mkdir()
# Manage Linux kernel and QEMU variants.
def symlink_buildroot_variant(custom_dir, variant_dir):
if os.path.islink(custom_dir):
os.unlink(custom_dir)
elif os.path.isdir(custom_dir):
# Migration for existing builds.
shutil.move(custom_dir, variant_dir)
os.makedirs(variant_dir, exist_ok=True)
os.symlink(variant_dir, custom_dir)
symlink_buildroot_variant(common.linux_build_dir, common.linux_variant_dir)
# Do the actual build.
if not args.no_all:
extra_make_args.append('all')
assert common.run_cmd(
[
'make',
'GEM5_LKMC_SRCDIR="{}"'.format(common.gem5_src_dir),
'O={}'.format(common.buildroot_build_dir),
'V={}'.format(int(args.verbose)),
] +
extra_make_args
,
out_file=os.path.join(common.buildroot_build_dir, 'lkmc.log'),
delete_env=['LD_LIBRARY_PATH'],
cwd=common.buildroot_src_dir,
) == 0
# Create the qcow2 from ext2. This is optional, because gem5
# does not need the qcow2.
if os.path.exists(common.qemu_img_executable):
assert common.run_cmd([
common.qemu_img_executable,
'-T', 'pr_manager_run,file=/dev/null',
'convert',
'-f', 'raw',
'-O', 'qcow2',
common.ext2_file,
common.qcow2_file,
]) == 0
return 0
def get_argparse():
parser = common.get_argparse(argparse_args={'description':'Run Linux on an emulator'})
common.add_build_arguments(parser)
parser.add_argument(
'-B', '--buildroot-config', default=defaults['buildroot_config'], action='append',
help='''Add a single Buildroot config to the current build.
Example value: 'BR2_TARGET_ROOTFS_EXT2_SIZE="512M"'.
Can be used multiple times to add multiple configs.
Takes precedence over any Buildroot config files.
'''
)
parser.add_argument(
'-b', '--buildroot-config-fragment', default=defaults['buildroot_config_fragment'], action='append',
help='''Also use the given Buildroot configuration fragment file.
Pass multiple times to use multiple fragment files.'''
)
parser.add_argument(
'--baseline', default=defaults['baseline'], action='store_true',
help='''Do a default-ish Buildroot defconfig build, without any of our extra options.
Mostly to track how much slower we are than a basic build.'''
)
parser.add_argument(
'-C', '--kernel-config', default=defaults['kernel_config'], action='append',
help='''Add a single kernel config configs to the current build.
Example value: 'CONFIG_FORTIFY_SOURCE=y'.
Can be used multiple times to add multiple configs.
Takes precedence over any Buildroot config files.
'''
)
parser.add_argument(
'-c', '--kernel-config-fragment', default=defaults['kernel_config_fragment'], action='append',
help='''Also use the given kernel configuration fragment file.
Pass multiple times to use multiple fragment files.'''
)
parser.add_argument(
'-I', '--initramfs', default=defaults['initramfs'], action='store_true',
)
parser.add_argument(
'-i', '--initrd', default=defaults['initrd'], action='store_true',
)
parser.add_argument(
'-j', '--nproc', default=defaults['nproc'], type=int,
help='Number of processors to use for the build. Default: all.'
)
parser.add_argument(
'-K', '--kernel-custom-config-file', default=defaults['kernel_custom_config_file'],
help='''Ignore all default kernel configurations and use this file instead.
Still uses options explicitly passed with `-C` and `-c` on top of it.'''
)
parser.add_argument(
'-k', '--kernel-modules', default=defaults['kernel_modules'], action='store_true',
help='Reconfigure and rebuild the kernel modules'
)
parser.add_argument(
'-l', '--linux-reconfigure', default=defaults['linux_reconfigure'], action='store_true',
help='''Reconfigure and rebuild the Linux kernel.
Touches kernel configuration files to overcome:
https://stackoverflow.com/questions/49260466/why-when-i-change-br2-linux-kernel-custom-config-file-and-run-make-linux-reconfi'''
)
parser.add_argument(
'--no-all', default=defaults['no_all'], action='store_true',
help='''Don't build the all target which normally gets build by default.
That target builds the root filesystem and all its dependencies.'''
)
parser.add_argument(
'--skip-configure', default=defaults['skip_configure'], action='store_true',
help='''Skip the Buildroot configuration. Saves a few seconds,
but requires you to know what you are doing :-)'''
)
parser.add_argument(
'-v', '--verbose', default=defaults['verbose'], action='store_true',
help='Do a verbose build'
)
parser.add_argument(
'extra_make_args', default=defaults['extra_make_args'], metavar='extra-make-args', nargs='*'
)
return parser
if __name__ == '__main__':
parser = get_argparse()
args = common.setup(parser)
sys.exit(main(args))

2
build-gem5
View File

@@ -21,7 +21,7 @@ args = common.setup(parser)
binaries_dir = os.path.join(common.gem5_system_dir, 'binaries') binaries_dir = os.path.join(common.gem5_system_dir, 'binaries')
disks_dir = os.path.join(common.gem5_system_dir, 'disks') disks_dir = os.path.join(common.gem5_system_dir, 'disks')
if args.clean: if args.clean:
shutil.rmtree(common.gem5_build_dir) common.rmrf(common.gem5_build_dir)
else: else:
os.makedirs(binaries_dir, exist_ok=True) os.makedirs(binaries_dir, exist_ok=True)
os.makedirs(disks_dir, exist_ok=True) os.makedirs(disks_dir, exist_ok=True)

2
build-qemu
View File

@@ -17,7 +17,7 @@ parser.add_argument(
) )
args = common.setup(parser) args = common.setup(parser)
if args.clean: if args.clean:
shutil.rmtree(common.qemu_build_dir) common.rmrf(common.qemu_build_dir)
else: else:
os.makedirs(common.qemu_build_dir, exist_ok=True) os.makedirs(common.qemu_build_dir, exist_ok=True)
subprocess.check_call( subprocess.check_call(

4
common.py
View File

@@ -217,6 +217,10 @@ def resolve_args(defaults, args, extra_args):
argcopy.__dict__ = dict(list(defaults.items()) + list(argcopy.__dict__.items()) + list(extra_args.items())) argcopy.__dict__ = dict(list(defaults.items()) + list(argcopy.__dict__.items()) + list(extra_args.items()))
return argcopy return argcopy
def rmrf(path):
if os.path.exists(path):
shutil.rmtree(path)
def run_cmd( def run_cmd(
cmd, cmd,
cmd_file=None, cmd_file=None,