./run --arch arm --cpus 2 --emulator gem5@@ -17631,7 +17648,7 @@ getconf _NPROCESSORS_CONF
If you don’t know which one to go for, start with QEMU Buildroot setup getting started.
Design goals of this project are documented at: Section 29.18.1, “Design goals”.
+Design goals of this project are documented at: Section 30.18.1, “Design goals”.
This setup has been mostly tested on Ubuntu. For other host operating systems see: Section 29.1, “Supported hosts”. For greater stability, consider using the latest release instead of master: https://github.com/cirosantilli/linux-kernel-module-cheat/releases
+This setup has been mostly tested on Ubuntu. For other host operating systems see: Section 30.1, “Supported hosts”. For greater stability, consider using the latest release instead of master: https://github.com/cirosantilli/linux-kernel-module-cheat/releases
Reserve 12Gb of disk and run:
@@ -1882,7 +1885,7 @@ cd linux-kernel-module-cheatYou don’t need to clone recursively even though we have .git submodules: download-dependencies fetches just the submodules that you need for this build to save time.
If something goes wrong, see: Section 29.2, “Common build issues” and use our issue tracker: https://github.com/cirosantilli/linux-kernel-module-cheat/issues
+If something goes wrong, see: Section 30.2, “Common build issues” and use our issue tracker: https://github.com/cirosantilli/linux-kernel-module-cheat/issues
The initial build will take a while (30 minutes to 2 hours) to clone and build, see Benchmark builds for more details.
@@ -1965,7 +1968,7 @@ hello2 cleanupTo avoid typing --arch aarch64 many times, you can set the default arch as explained at: Section 29.4, “Default command line arguments”
To avoid typing --arch aarch64 many times, you can set the default arch as explained at: Section 30.4, “Default command line arguments”
I now urge you to read the following sections which contain widely applicable information:
@@ -2269,10 +2272,10 @@ hello /root/.profileIf you really want to develop semiconductors, your only choice is to join an university or a semiconductor company that has the EDA licenses.
See also: Section 29.19.2, “Should you waste your life with systems programming?”.
+See also: Section 30.19.2, “Should you waste your life with systems programming?”.
While hacking QEMU, you will likely want to GDB step its source. That is trivial since QEMU is just another userland program like any other, but our setup has a shortcut to make it even more convenient, see: Section 17.7, “Debug the emulator”.
+While hacking QEMU, you will likely want to GDB step its source. That is trivial since QEMU is just another userland program like any other, but our setup has a shortcut to make it even more convenient, see: Section 18.7, “Debug the emulator”.
and can therefore be used to estimate system performance, see: Section 18.2, “gem5 run benchmark” for an example.
+and can therefore be used to estimate system performance, see: Section 19.2, “gem5 run benchmark” for an example.
The downside of gem5 much slower than QEMU because of the greater simulation detail.
@@ -2682,7 +2685,7 @@ j = 0More gem5 information is present at: Section 18, “gem5”
+More gem5 information is present at: Section 19, “gem5”
Good next steps are:
@@ -2708,7 +2711,7 @@ j = 0This repository has been tested inside clean Docker containers.
This is a good option if you are on a Linux host, but the native setup failed due to your weird host distribution, and you have better things to do with your life than to debug it. See also: Section 29.1, “Supported hosts”.
+This is a good option if you are on a Linux host, but the native setup failed due to your weird host distribution, and you have better things to do with your life than to debug it. See also: Section 30.1, “Supported hosts”.
For example, to do a QEMU Buildroot setup inside Docker, run:
@@ -2896,7 +2899,7 @@ j = 0can’t GDB step debug the kernel, since the source and cross toolchain with GDB are not available. Buildroot cannot easily use a host toolchain: Section 28.2.2.1.1, “Buildroot use prebuilt host toolchain”.
+can’t GDB step debug the kernel, since the source and cross toolchain with GDB are not available. Buildroot cannot easily use a host toolchain: Section 29.2.2.1.1, “Buildroot use prebuilt host toolchain”.
Maybe we could work around this by just downloading the kernel source somehow, and using a host prebuilt GDB, but we felt that it would be too messy and unreliable.
as shown at: Section 17.7, “Debug the emulator”, although direct GDB host usage works as well of course.
+as shown at: Section 18.7, “Debug the emulator”, although direct GDB host usage works as well of course.
TODO: the carriage returns are a bit different than in QEMU, see: Section 26.4, “gem5 baremetal carriage return”.
+TODO: the carriage returns are a bit different than in QEMU, see: Section 27.4, “gem5 baremetal carriage return”.
Note that ./build-baremetal requires the --emulator gem5 option, and generates separate executable images for both, as can be seen from:
see also: Section 18.18, “gem5 ARM platforms”.
+see also: Section 19.18, “gem5 ARM platforms”.
This generates yet new separate images with new magic constants:
@@ -3718,10 +3721,10 @@ echo "$(./getvar --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 -But just stick to newer and better VExpress_GEM5_V1 unless you have a good reason to use RealViewPBX.
When doing baremetal programming, it is likely that you will want to learn userland assembly first, see: Section 21, “Userland assembly”.
+When doing baremetal programming, it is likely that you will want to learn userland assembly first, see: Section 22, “Userland assembly”.
For more information on baremetal, see the section: Section 26, “Baremetal”.
+For more information on baremetal, see the section: Section 27, “Baremetal”.
The following subjects are particularly important:
@@ -3786,7 +3789,7 @@ xdg-open README.htmlMore information about our documentation internals can be found at: Section 29.5, “Documentation”
+More information about our documentation internals can be found at: Section 30.5, “Documentation”
For a more minimal baremetal multicore setup, see: Section 26.8.3, “ARM multicore”.
+For a more minimal baremetal multicore setup, see: Section 27.8.3, “ARM multicore”.
We can set and get which cores the Linux kernel allows a program to run on with sched_getaffinity and sched_setaffinity:
The number of cores is modified as explained at: Section 18.2.2.1, “Number of cores”
+The number of cores is modified as explained at: Section 19.2.2.1, “Number of cores”
taskset from the util-linux package sets the initial core affinity of a program:
which can be good for automated tests, as it ensures that you are using a pristine unmodified system image every time.
Not however that we already disable disk persistency by default on ext2 filesystems even without --initrd: Section 17.2, “Disk persistency”.
Not however that we already disable disk persistency by default on ext2 filesystems even without --initrd: Section 18.2, “Disk persistency”.
One downside of this method is that it has to put the entire filesystem into memory, and could lead to a panic:
@@ -7104,7 +7107,7 @@ qw erTo stop at the very first instruction of a freestanding program, just use --no-continue. A good example of this is shown at: Section 21.5.1, “Freestanding programs”.
To stop at the very first instruction of a freestanding program, just use --no-continue. A good example of this is shown at: Section 22.5.1, “Freestanding programs”.
The gem5 tests require building statically with build id static, see also: Section 10.6, “gem5 syscall emulation mode”. TODO automate this better.
See: Section 29.13, “Test this repo” for more useful testing tips.
+See: Section 30.13, “Test this repo” for more useful testing tips.
modules built with Buildroot, see: Section 29.12.2.1, “kernel_modules buildroot package”
+modules built with Buildroot, see: Section 30.12.2.1, “kernel_modules buildroot package”
modules built from the kernel tree itself, see: Section 15.12.2, “dummy-irq”
@@ -7959,7 +7962,7 @@ hellono need to regenerate the root filesystem at all and reboot
overcomes the check_bin_arch problem as shown at: Section 19.8, “Buildroot rebuild is slow when the root filesystem is large”
overcomes the check_bin_arch problem as shown at: Section 20.8, “Buildroot rebuild is slow when the root filesystem is large”
We disable networking by default because it starts an userland process, and we want to keep the number of userland processes to a minimum to make the system more understandable as explained at: Section 29.18.3, “Resource tradeoff guidelines”
+We disable networking by default because it starts an userland process, and we want to keep the number of userland processes to a minimum to make the system more understandable as explained at: Section 30.18.3, “Resource tradeoff guidelines”
To enable networking on Buildroot, simply run:
@@ -9513,7 +9516,7 @@ CONFIG_IKCONFIG_PROC=yarm and aarch64 configs present in the official ARM gem5 Linux kernel fork as described at: Section 18.8, “gem5 arm Linux kernel patches”. Some of the configs present there are added by the patches.
arm and aarch64 configs present in the official ARM gem5 Linux kernel fork as described at: Section 19.8, “gem5 arm Linux kernel patches”. Some of the configs present there are added by the patches.
Jason’s magic x86_64 config: http://web.archive.org/web/20171229121642/http://www.lowepower.com/jason/files/config which is referenced at: http://web.archive.org/web/20171229121525/http://www.lowepower.com/jason/setting-up-gem5-full-system.html. QEMU boots with that by removing # CONFIG_VIRTIO_PCI is not set.
This also makes this repo the perfect setup to develop the Linux kernel.
In case something breaks while updating the Linux kernel, you can try to bisect it to understand the root cause, see: Section 29.14, “Bisection”.
+In case something breaks while updating the Linux kernel, you can try to bisect it to understand the root cause, see: Section 30.14, “Bisection”.
First, use use the branching procedure described at: Section 29.16, “Update a forked submodule”
+First, use use the branching procedure described at: Section 30.16, “Update a forked submodule”
Because the kernel is so central to this repository, almost all tests must be re-run, so basically just follow the full testing procedure described at: Section 29.13, “Test this repo”. The only tests that can be skipped are essentially the Baremetal tests.
+Because the kernel is so central to this repository, almost all tests must be re-run, so basically just follow the full testing procedure described at: Section 30.13, “Test this repo”. The only tests that can be skipped are essentially the Baremetal tests.
Before comitting, don’t forget to update:
@@ -13996,7 +13999,7 @@ detected buffer overflow in strlenSELinux requires glibc as mentioned at: Section 19.10, “libc choice”.
+SELinux requires glibc as mentioned at: Section 20.10, “libc choice”.
STRESS_NG is likely the best, but it requires glibc, see: Section 19.10, “libc choice”.
STRESS_NG is likely the best, but it requires glibc, see: Section 20.10, “libc choice”.
Websites:
@@ -15266,10 +15269,24 @@ psU-Boot is a popular bootloader.
+It can read disk filesystems, and Buildroot supports it, so we could in theory put it into memory, and let it find a kernel image from the root filesystem and boot that, but I didn’t manage to get it working yet: https://stackoverflow.com/questions/58028789/how-to-boot-linux-aarch64-with-u-boot-with-buildroot-on-qemu
+QEMU is a system simulator: it simulates a CPU and devices such as interrupt handlers, timers, UART, screen, keyboard, etc.
We disable disk persistency for both QEMU and gem5 by default, to prevent the emulator from putting the image in an unknown state.
Disk persistency is useful to re-run shell commands from the history of a previous session with Ctrl-R, but we felt that the loss of determinism was not worth it.
TODO how to make gem5 disk writes persistent?
qcow2 does not appear supported, there are not hits in the source tree, and there is a mention on Nate’s 2009 wishlist: http://gem5.org/Nate%27s_Wish_List
QEMU allows us to take snapshots at any time through the monitor.
Bibliography: https://stackoverflow.com/questions/40227651/does-qemu-emulator-have-checkpoint-function/48724371#48724371
Snapshots are stored inside the .qcow2 images themselves.
This section documents:
Only tested in x86.
PCI driver for our minimal pci_min.c QEMU fork device:
Small upstream educational PCI device:
In this section we will try to interact with PCI devices directly from userland without kernel modules.
There are two versions of setpci and lspci:
The PCI standard is non-free, obviously like everything in low level: https://pcisig.com/specifications but Google gives several illegal PDF hits :-)
lspci -k shows something like:
TODO: broken. Was working before we moved arm from -M versatilepb to -M virt around af210a76711b7fa4554dcc2abd0ddacfc810dfd4. Either make it work on -M virt if that is possible, or document precisely how to make it work with versatilepb, or hopefully vexpress which is newer.
TODO: broken when arm moved to -M virt, same as GPIO.
Minimal platform device example coded into the -M versatilepb SoC of our QEMU fork.
TODO get some working!
The QEMU monitor is a magic terminal that allows you to send text commands to the QEMU VM itself: https://en.wikibooks.org/wiki/QEMU/Monitor
Peter Maydell said potentially not possible nicely as of August 2018: https://stackoverflow.com/questions/51747744/how-to-run-a-qemu-monitor-command-from-inside-the-guest/51764110#51764110
When doing GDB step debug it is possible to send QEMU monitor commands through the GDB monitor command, which saves you the trouble of opening yet another shell.
When you start hacking QEMU or gem5, it is useful to see what is going on inside the emulator themselves.
The build outputs are automatically stored in a different directories for optimized and debug builds, which prevents debug files from overwriting opt ones. Therefore, --gem5-build-id is not required:
The price to pay for debuggability is high however: a Linux kernel boot was about 3x slower in QEMU and 14 times slower in gem5 debug compared to opt, see benchmarks at: Section 28.2.1, “Benchmark Linux kernel boot”
+The price to pay for debuggability is high however: a Linux kernel boot was about 3x slower in QEMU and 14 times slower in gem5 debug compared to opt, see benchmarks at: Section 29.2.1, “Benchmark Linux kernel boot”
When in QEMU text mode, using --debug-vm makes Ctrl-C not get passed to the QEMU guest anymore: it is instead captured by GDB itself, so allow breaking. So e.g. you won’t be able to easily quit from a guest program like:
You can still send key presses to QEMU however even without the mouse capture, just either click on the title bar, or alt tab to give it focus.
While step debugging any complext program, you always end up feeling the need to step in reverse to reach the last call to some function that was called before the failure point, in order to trace back the problem to the actual bug source.
Start pdb at the first instruction:
QEMU can log several different events.
QEMU also has a second trace mechanism in addition to -trace, find out the events with:
TODO: is it possible to show the register values for each instruction?
PANDA can list memory addresses, so I bet it can also decode the instructions: https://github.com/panda-re/panda/blob/883c85fa35f35e84a323ed3d464ff40030f06bd6/panda/docs/LINE_Censorship.md I wonder why they don’t just upstream those things to QEMU’s tracing: https://github.com/panda-re/panda/issues/290
gem5 can do it as shown at: Section 17.8.6, “gem5 tracing”.
+gem5 can do it as shown at: Section 18.8.6, “gem5 tracing”.
We can further use Binutils' addr2line to get the line that corresponds to each address:
QEMU runs, unlike gem5, are not deterministic by default, however it does support a record and replay mechanism that allows you to replay a previous run deterministically.
Solved on unmerged c42634d8e3428cfa60672c3ba89cabefc720cde9 from https://github.com/ispras/qemu/tree/rr-180725
TODO get working.
TODO: is there any way to distinguish which instruction runs on each core? Doing:
gem5 provides also provides a tracing mechanism documented at: http://www.gem5.org/Trace_Based_Debugging:
TODO: 7452d399290c9c1fc6366cdad129ef442f323564 ./trace2line this is too slow and takes hours. QEMU’s processing of 170k events takes 7 seconds. gem5’s processing is analogous, but there are 140M events, so it should take 7000 seconds ~ 2 hours which seems consistent with what I observe, so maybe there is no way to speed this up… The workaround is to just use gem5’s ExecSymbol to get function granularity, and then GDB individually if line detail is needed?
This debug flag traces all instructions.
25007500: time count in some unit. Note how the microops execute at further timestamps.
system.cpu: distinguishes between CPUs when there are more than one. For example, running Section 26.8.3, “ARM multicore” with two cores produces system.cpu0 and system.cpu1
system.cpu: distinguishes between CPUs when there are more than one. For example, running Section 27.8.3, “ARM multicore” with two cores produces system.cpu0 and system.cpu1
T0: thread number. TODO: hyperthread? How to play with it?
This flag shows a more detailed register usage than gem5 ExecAll trace format.
As of gem5 16eeee5356585441a49d05c78abc328ef09f7ace the default tracer is ExeTracer. It is set at:
Sometimes in Ubuntu 14.04, after the QEMU SDL GUI starts, it does not get updated after keyboard strokes, and there are artifacts like disappearing text.
Getting started at: Section 1.2, “gem5 Buildroot setup”.
runs are deterministic by default, unlike QEMU which has a special QEMU record and replay mode, that requires first playing the content once and then replaying
gem5 ARM at least appears to implement more low level CPU functionality than QEMU, e.g. QEMU only added EL2 in 2018: https://stackoverflow.com/questions/42824706/qemu-system-aarch64-entering-el1-when-emulating-a53-power-up See also: Section 26.8.1, “ARM exception levels”
+gem5 ARM at least appears to implement more low level CPU functionality than QEMU, e.g. QEMU only added EL2 in 2018: https://stackoverflow.com/questions/42824706/qemu-system-aarch64-entering-el1-when-emulating-a53-power-up See also: Section 27.8.1, “ARM exception levels”
slower than QEMU, see: Section 28.2.1, “Benchmark Linux kernel boot”
+slower than QEMU, see: Section 29.2.1, “Benchmark Linux kernel boot”
This implies that the user base is much smaller, since no Android devs.
OK, this is why we used gem5 in the first place, performance measurements!
but the problem is that this method does not allow to easily run a different script without running the boot again. The ./gem5.sh script works around that by using m5 readfile as explained further at: Section 18.5.2, “gem5 checkpoint restore and run a different script”.
but the problem is that this method does not allow to easily run a different script without running the boot again. The ./gem5.sh script works around that by using m5 readfile as explained further at: Section 19.5.2, “gem5 checkpoint restore and run a different script”.
Now you can play a fun little game with your friends:
@@ -17572,7 +17589,7 @@ cat out/gem5-bench-dhrystone.txtTo find out why your program is slow, a good first step is to have a look at the gem5 m5out/stats.txt file.
A few imperfections of our benchmarking method are:
Besides optimizing a program for a given CPU setup, chip developers can also do the inverse, and optimize the chip for a given benchmark!
The rabbit hole is likely deep, but let’s scratch a bit of the surface.
./run --arch arm --cpus 2 --emulator gem5@@ -17631,7 +17648,7 @@ getconf _NPROCESSORS_CONF
TODO why in User mode simulation QEMU always shows the number of cores of the host. E.g., both of the following output the same as nproc on the host:
gem5 user mode multithreading has been particularly flaky compared to QEMU’s.
See bug report at: https://github.com/cirosantilli/linux-kernel-module-cheat/issues/81
TODO These look promising:
./run --memory 512M@@ -18042,7 +18059,7 @@ get_avphys_pages() * sysconf(_SC_PAGESIZE) = 0x1D178000
TODO These look promising:
Clock frequency: TODO how does it affect performance in benchmarks?
Buildroot built-in libraries, mostly under Libraries > Other:
There are not yet enabled, but it should be easy to so, see: Section 19.5, “Add new Buildroot packages”
+There are not yet enabled, but it should be easy to so, see: Section 20.5, “Add new Buildroot packages”
The following benchmark setup works both:
The cache sizes were chosen to match the host P51 to improve the comparison. Ideally we should also use the same standard library.
Note that this will take a long time, and will produce a humongous ~40Gb stats file as explained at: Section 18.9.2.1, “gem5 only dump selected stats”
+Note that this will take a long time, and will produce a humongous ~40Gb stats file as explained at: Section 19.9.2.1, “gem5 only dump selected stats”
Sources:
@@ -18244,7 +18261,7 @@ xdg-open bst_vs_heap_vs_hashmap_gem5.tmp.pngBuildroot supports it, which makes everything just trivial:
Header only linear algebra library with a mainline Buildroot package:
We have ported parts of the PARSEC benchmark for cross compilation at: https://github.com/cirosantilli/parsec-benchmark See the documentation on that repo to find out which benchmarks have been ported. Some of the benchmarks were are segfaulting, they are documented in that repo.
./build --arch arm --download-dependencies gem5-buildroot parsec-benchmark @@ -18386,7 +18403,7 @@ cblas_dgemm( CblasColMajor, CblasNoTrans, CblasTrans,3,3,2 ,1, A,3, B,
Running a benchmark of a size different than test, e.g. simsmall, requires a rebuild with:
Most users won’t want to use this method because:
If you want to remove PARSEC later, Buildroot doesn’t provide an automated package removal mechanism as mentioned at: Section 19.6, “Remove Buildroot packages”, but the following procedure should be satisfactory:
+If you want to remove PARSEC later, Buildroot doesn’t provide an automated package removal mechanism as mentioned at: Section 20.6, “Remove Buildroot packages”, but the following procedure should be satisfactory:
If you end up going inside submodules/parsec-benchmark to hack up the benchmark (you will!), these tips will be helpful.
Analogous to QEMU:
Analogous to QEMU, on the first shell:
We are unable to use gdbserver because of networking as mentioned at: Section 14.3.1.3, “gem5 host to guest networking”
Analogous to QEMU’s Snapshot, but better since it can be started from inside the guest, so we can easily checkpoint after a specific guest event, e.g. just before init is done.
since boot has already happened, and the parameters are already in the RAM of the snapshot.
Checkpoints are stored inside the m5out directory at:
You want to automate running several tests from a single pristine post-boot state.
gem5 can switch to a different CPU model when restoring a checkpoint.
Remember that in the gem5 command line, we can either pass options to the script being run as in:
m5ops are magic instructions which lead gem5 to do magic things, like quitting or dumping stats.
m5 is a guest command line utility that is installed and run on the guest, that serves as a CLI front-end for the m5ops
It is possible to guess what most tools do from the corresponding m5ops, but let’s at least document the less obvious ones here.
End the simulation with a failure exit event:
Send a guest file to the host. 9P is a more advanced alternative.
Read a host file pointed to by the fs.py --script option to stdout.
Ermm, just another m5 readfile that only takes integers and only from CLI options? Is this software so redundant?
Trivial combination of m5 readfile + execute the script.
gem5 allocates some magic instructions on unused instruction encodings for convenient guest instrumentation.
Let’s study how m5 uses them:
include/gem5/asm/generic/m5ops.h also describes some annotation instructions.
https://gem5.googlesource.com/arm/linux/ contains an ARM Linux kernel forks with a few gem5 specific Linux kernel patches on top of mainline created by ARM Holdings on top of a few upstream kernel releases.
drm: Add component-aware simple encoder allows you to see images through VNC, see: Section 13.3, “gem5 graphic mode”
gem5: Add support for gem5’s extended GIC mode adds support for more than 8 cores, see: Section 18.2.2.1.4, “gem5 ARM full system with more than 8 cores”
gem5: Add support for gem5’s extended GIC mode adds support for more than 8 cores, see: Section 19.2.2.1.4, “gem5 ARM full system with more than 8 cores”
Tested on 649d06d6758cefd080d04dc47fd6a5a26a620874 + 1.
We have observed that with the kernel patches, boot is 2x faster, falling from 1m40s to 50s.
When you run gem5, it generates an m5out directory at:
The files in that directory contains some very important information about the run, and you should become familiar with every one of them.
Contains UART output, both from the Linux kernel or from the baremetal system.
This file contains important statistics about the run:
For x86, it is interesting to try and correlate numCycles with:
TODO
The m5out/config.ini file, contains a very good high level description of the system:
We use the m5term in-tree executable to connect to the terminal instead of a direct telnet.
We have made a crazy setup that allows you to just cd into submodules/gem5, and edit Python scripts directly there.
By default, we use configs/example/fs.py script.
This error happens when the following instruction limits are reached:
In order to use different build options, you might also want to use gem5 build variants to keep the build outputs separate from one another.
Explained at: Section 17.7, “Debug the emulator”.
+Explained at: Section 18.7, “Debug the emulator”.
TODO test properly, benchmark vs GCC.
If there gem5 appears to have a C++ undefined behaviour bug, which is often very difficult to track down, you can try to build it with the following extra SCons options:
Ruby is a system that includes the SLICC domain specific language to describe memory systems: http://gem5.org/Ruby
Python 3 support was mostly added in 2019 Q3 at arounda347a1a68b8a6e370334be3a1d2d66675891e0f1 but remained buggy for some time afterwards.
gem5 has a few in tree CPU models for different purposes. In fs.py and se.py, those are selectable with the --cpu-type option. Here is an overview of the most interesting ones:
The gem5 platform is selectable with the --machine option, which is named after the analogous QEMU -machine option, and which sets the --machine-type.
The interaction uses the Python C extension interface https://docs.python.org/2/extending/extending.html interface through the pybind11 helper library: https://github.com/pybind/pybind11
Buildroot is a set of Make scripts that download and compile from source compatible versions of:
C standard library: Buildroot supports several implementations, see: Section 19.10, “libc choice”
+C standard library: Buildroot supports several implementations, see: Section 20.10, “libc choice”
BusyBox: provides the shell and basic command line utilities
@@ -20454,7 +20471,7 @@ qemu-system-aarch64 -M virt -cpu cortex-a57 -nographic -smp 1 -kernel output/imaWe provide the following mechanisms:
The clean is necessary because the source files didn’t change, so make would just check the timestamps and not build anything.
You will then likely want to make those more permanent as explained at: Section 29.4, “Default command line arguments”.
+You will then likely want to make those more permanent as explained at: Section 30.4, “Default command line arguments”.
If you are benchmarking compiled programs instead of hand written assembly, remember that we configure Buildroot to disable optimizations by default with:
if you already have a full -O0 build, you can choose to rebuild just your package of interest to save some time as described at: Section 19.2, “Custom Buildroot configs”
if you already have a full -O0 build, you can choose to rebuild just your package of interest to save some time as described at: Section 20.2, “Custom Buildroot configs”
./build-buildroot \ @@ -20560,7 +20577,7 @@ qemu-system-aarch64 -M virt -cpu cortex-a57 -nographic -smp 1 -kernel output/ima
make menuconfig is a convenient way to find Buildroot configurations:
At startup, we login automatically as the root user.
Replace on inittab:
First, see if you can’t get away without actually adding a new package, for example:
if you have a standalone C file with no dependencies besides the C standard library to be compiled with GCC, just add a new file under buildroot_packages/sample_package and you are done
if you have a dependency on a library, first check if Buildroot doesn’t have a package for it already with ls buildroot/package. If yes, just enable that package as explained at: Section 19.2, “Custom Buildroot configs”
if you have a dependency on a library, first check if Buildroot doesn’t have a package for it already with ls buildroot/package. If yes, just enable that package as explained at: Section 20.2, “Custom Buildroot configs”
If none of those methods are flexible enough for you, you can just fork or hack up buildroot_packages/sample_package the sample package to do what you want.
For how to use that package, see: Section 29.12.2, “buildroot_packages directory”.
+For how to use that package, see: Section 30.12.2, “buildroot_packages directory”.
Then iterate trying to do what you want and reading the manual until it works: https://buildroot.org/downloads/manual/manual.html
Once you’ve built a package in to the image, there is no easy way to remove it.
Also mentioned at: https://stackoverflow.com/questions/47320800/how-to-clean-only-target-in-buildroot
See this for a sample manual workaround: Section 18.2.3.4.4, “PARSEC uninstall”.
+See this for a sample manual workaround: Section 19.2.3.4.4, “PARSEC uninstall”.
When adding new large package to the Buildroot root filesystem, it may fail with the message:
libguestfs: https://serverfault.com/questions/246835/convert-directory-to-qemu-kvm-virtual-disk-image/916697#916697, in particular vfs-minimum-size
use methods described at: Section 18.5.2, “gem5 checkpoint restore and run a different script” instead of putting builds on the root filesystem
+use methods described at: Section 19.5.2, “gem5 checkpoint restore and run a different script” instead of putting builds on the root filesystem
Bibliography: https://stackoverflow.com/questions/49211241/is-there-a-way-to-automatically-detect-the-minimum-required-br2-target-rootfs-ex
SquashFS creation with mksquashfs does not take fixed sizes, and I have successfully booted from it, but it is readonly, which is unacceptable.
Buildroot is not designed for large root filesystem images, and the rebuild becomes very slow when we add a large package to it.
When asking for help on upstream repositories outside of this repository, you will need to provide the commands that you are running in detail without referencing our scripts.
Then, you will also want to do a Bisection to pinpoint the exact commit to blame, and CC that developer.
Finally, give the images you used save upstream developers' time as shown at: Section 29.17.2, “release-zip”.
+Finally, give the images you used save upstream developers' time as shown at: Section 30.17.2, “release-zip”.
For Buildroot problems, you should wither provide the config you have:
@@ -20864,7 +20881,7 @@ git -C "$(./getvar qemu_source_dir)" checkout -Buildroot supports several libc implementations, including:
One "downside" of glibc is that it exercises much more kernel functionality on its more bloated pre-main init, which breaks user mode C hello worlds more often, see: Section 10.4, “User mode simulation with glibc”. I quote "downside" because glibc is actually exposing emulator bugs which we should actually go and fix.
This repo doesn’t do much more other than setting a bunch of Buildroot configurations and building it, and the minimal work you have to do to get QEMU to boot Buildroot from scratch is tiny if you want to quickly test Buildroot specifics, for example:
+PPC https://stackoverflow.com/questions/48021127/build-powerpc-kernel-and-boot-powerpc-kernel-on-qemu/49349262#49349262 just work commands for PPC, comment on how to replace kernel
+This section contains userland content, such as C, C++ and POSIX examples.
@@ -20923,7 +20959,7 @@ git -C "$(./getvar qemu_source_dir)" checkout -Getting started at: Section 1.6, “Userland setup”
Userland assembly content is located at: Section 21, “Userland assembly”. It was split from this section basically because we were hitting the HTML h6 limit, stupid web :-)
Userland assembly content is located at: Section 22, “Userland assembly”. It was split from this section basically because we were hitting the HTML h6 limit, stupid web :-)
This content makes up the bulk of the userland/ directory.
@@ -20935,7 +20971,7 @@ git -C "$(./getvar qemu_source_dir)" checkout -This section was originally moved in here from: https://github.com/cirosantilli/cpp-cheat
Programs under userland/c/ are examples of ANSI C programming:
Allocate memory! Vs using the stack: https://stackoverflow.com/questions/4584089/what-is-the-function-of-the-push-pop-instructions-used-on-registers-in-x86-ass/33583134#33583134
malloc leads to the infinite joys of Memory leaks.
TODO: the exact answer is going to be hard.
General overview at: https://stackoverflow.com/questions/2798330/maximum-memory-which-malloc-can-allocate
If we start using the pages, the OOM killer would sooner or later step in and kill our process: Linux out-of-memory killer.
We can observe the OOM in LKMC 1e969e832f66cb5a72d12d57c53fb09e9721d589 which defaults to 256MiB of memory with:
Added in C11!
Example: userland/gcc/empty_struct.c
GCC implements the OpenMP threading implementation: https://stackoverflow.com/questions/3949901/pthreads-vs-openmp
Programs under userland/cpp/ are examples of ISO C programming.
Like for C, you have to pay for the standards… insane. So we just use the closest free drafts instead.
https://stackoverflow.com/questions/81656/where-do-i-find-the-current-c-or-c-standard-documents
Programs under userland/posix/ are examples of POSIX C programming.
POSIX' multithreading API. This was for a looong time the only "portable" multithreading alternative, until C++11 finally added threads, thus also extending the portability to Windows.
The mmap system call allows advanced memory operations.
Linux adds has several POSIX extension flags to it.
Basic mmap example, do the same as userland/c/malloc.c, but with mmap.
Memory mapped file example: userland/posix/mmap_file.c
A bit like read and write, but from / to the Internet!
userland/posix/wget.c tiny wget re-implementation. See: https://stackoverflow.com/questions/11208299/how-to-make-an-http-get-request-in-c-without-libcurl/35680609#35680609
The following sections are related to multithreading in userland:
Let’s group the hard-to-debug undefined-behaviour-like stuff found in C / C+ here and how to tackle those problems.
Programs under userland/arch/<arch>/ are examples of userland assembly programming.
registers, see: Section 21.1, “Assembly registers”
+registers, see: Section 22.1, “Assembly registers”
jumping:
@@ -21817,14 +21866,14 @@ error: asm_main returned 1 at line 8After seeing an ADD hello world, you need to learn the general registers:
x86, see: Section 22.1, “x86 registers”
+x86, see: Section 23.1, “x86 registers”
arm
@@ -21855,7 +21904,7 @@ error: asm_main returned 1 at line 8Bibliography: ARMv7 architecture reference manual A2.3 "ARM core registers".
Example: userland/arch/aarch64/x31.S
Keep in mind that many ISAs started floating point as an optional thing, and it later got better integrated into the main CPU, side by side with SIMD.
Much like ADD for non-SIMD, start learning SIMD instructions by looking at the integer and floating point SIMD ADD instructions of each ISA:
Bibliography: https://stackoverflow.com/questions/1389712/getting-started-with-intel-x86-sse-simd-instructions/56409539#56409539
Fused multiply add:
By "userland assembly", we mean "the parts of the ISA which can be freely used from userland".
One big difference between both is that we can run userland assembly on Userland setup, which is easier to get running and debug.
In particular, most userland assembly examples link to the C standard library, see: Section 21.5, “Userland assembly C standard library”.
+In particular, most userland assembly examples link to the C standard library, see: Section 22.5, “Userland assembly C standard library”.
Userland assembly is generally simpler, and a pre-requisite for Baremetal setup.
@@ -22142,7 +22191,7 @@ When instructions do not interpret this operand encoding as the zero register, uAll examples except the Freestanding programs link to the C standard library.
Unlike most our other assembly examples, which use the C standard library for portability, examples under freestanding/ directories don’t link to the C standard library:
This is analogous to step debugging baremetal examples.
Assembly examples under nostartfiles directories can use the standard library, but they don’t use the pre-main boilerplate and start directly at our explicitly given _start:
Examples under arch/<arch>/c/ directories show to how use inline assembly from higher level languages such as C:
Used notably in some of the Linux system calls setups:
In arm, it is the only way to achieve this effect: https://stackoverflow.com/questions/10831792/how-to-use-specific-register-in-arm-inline-assembler
This feature notably useful for making system calls from C, see: Section 21.7, “Linux system calls”.
+This feature notably useful for making system calls from C, see: Section 22.7, “Linux system calls”.
How to use temporary registers in inline assembly:
An example of using the & early-clobber modifier: link:userland/arch/aarch64/earlyclobber.c
Not documented as of GCC 8.2, but possible: https://stackoverflow.com/questions/53960240/armv8-floating-point-output-inline-assembly
Pre-existing C wrappers using inline assembly, this is what production programs should use instead of inline assembly for SIMD:
Good official cheatsheet with all intrinsics and what they expand to: https://software.intel.com/sites/landingpage/IntrinsicsGuide
The following Userland setup programs illustrate how to make system calls:
A summary of results is shown at: Table 3, “Summary of Linux calling conventions for several architectures”.
Examples:
Call C standard library functions from assembly and vice versa.
GNU GAS is the default assembler used by GDB, and therefore it completely dominates in Linux.
The Linux kernel in particular uses GNU GAS assembly extensively for the arch specific parts under arch/.
In this tutorial, we use exclusively C Preprocessor /**/ comments because:
Summary:
Let’s see how many bytes go into each data type:
There are two types of ARMv7 assemblies:
cannot have implicit destination with shift, see: Section 23.4.4.1, “ARM shift suffixes”
+cannot have implicit destination with shift, see: Section 24.4.4.1, “ARM shift suffixes”
When reading disassembly, many instructions have either a .n or .w suffix.
Arch agnostic infrastructure getting started at: Section 21, “Userland assembly”.
+Arch agnostic infrastructure getting started at: Section 22, “Userland assembly”.
link:userland/arch/x86_64/registers.S
Example: userland/arch/x86_64/address_modes.S
5.1.1 "Data Transfer Instructions"
Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 7.3.1.2 "Exchange Instructions":
TODO: concrete multi-thread GCC inline assembly examples of how all those instructions are normally used as synchronization primitives.
Examples:
This is partly why the ternary ? C operator exists: https://stackoverflow.com/questions/3565368/ternary-operator-vs-if-else
It is interesting to compare this with ARMv7 conditional execution: which is available for all instructions, as shown at: Section 23.2.5, “ARM conditional execution”.
+It is interesting to compare this with ARMv7 conditional execution: which is available for all instructions, as shown at: Section 24.2.5, “ARM conditional execution”.
Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.1.2 "Binary Arithmetic Instructions":
Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.1.4 "Logical Instructions"
Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.1.5 "Shift and Rotate Instructions"
Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.1.6 "Bit and Byte Instructions"
Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.1.7 "Control Transfer Instructions"
Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.1.8 "String Instructions"
However, as computer architecture evolved, those instructions might not offer considerable speedups anymore, and modern glibc such as 2.29 just uses x86 SIMD operations instead:, see also: https://stackoverflow.com/questions/33480999/how-can-the-rep-stosb-instruction-execute-faster-than-the-equivalent-loop
Example: userland/arch/x86_64/rep.S
Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.1.13 "Miscellaneous Instructions"
Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.1.15 Random Number Generator Instructions
RDRAND sets the carry flag when data is ready so we must loop if the carry flag isn’t set.
Example: userland/arch/x86_64/cpuid.S
Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.2 "X87 FPU INSTRUCTIONS"
Parent section: Section 21.3, “SIMD assembly”
+Parent section: Section 22.3, “SIMD assembly”
History:
@@ -24168,12 +24217,12 @@ pop %rbpIntel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.5 "SSE INSTRUCTIONS"
Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.5.1.1 "SSE Data Transfer Instructions"
Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.5.1.2 "SSE Packed Arithmetic Instructions"
Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.5.1.6 "SSE Conversion Instructions"
Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.6 "SSE2 INSTRUCTIONS"
userland/arch/x86_64/paddq.S: PADDQ, PADDL, PADDW, PADDB
Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.15 "FUSED-MULTIPLY-ADD (FMA)"
Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.20 "SYSTEM INSTRUCTIONS"
Sources:
RDTSCP is like RDTSP, but it also stores the CPU ID into ECX: this is convenient because the value of RDTSC depends on which core we are currently on, so you often also want the core ID when you want the RDTSC.
TODO We didn’t manage to find a working ARM analogue to x86 RDTSC instruction: kernel_modules/pmccntr.c is oopsing, and even it if weren’t, it likely won’t give the cycle count since boot since it needs to be activate before it starts counting anything:
Inline assembly example at: userland/cpp/atomic.cpp
We are using the May 2019 version unless otherwise noted.
Also I can’t find older versions on the website easily, so I just web archive everything.
Arch general getting started at: Section 21, “Userland assembly”.
+Arch general getting started at: Section 22, “Userland assembly”.
Instructions here loosely grouped based on that of the ARMv7 architecture reference manual Chapter A4 "The Instruction Sets".
@@ -24487,7 +24536,7 @@ taskset -c 1 ./userland/arch/x86_64/rdtscp.out | tail -n 1We cover here mostly ARMv7, and then treat aarch64 differentially, since much of the ARMv7 userland is the same in aarch32.
The ARM architecture is has been used on the vast majority of mobile phones in the 2010’s, and on a large fraction of micro controllers.
ARM Holdings was bought by the Japanese giant SoftBank in 2016.
ARMv7 is the older architecture described at: ARMv7 architecture reference manual.
They are described at: ARMv8 architecture reference manual A1.7 "ARMv8 architecture extensions".
32-bit mode of operation of ARMv8.
A great summary of differences can be found at: https://en.wikipedia.org/wiki/ARM_architecture#AArch64_features
aarch32 has two encodings: Thumb and ARM: Section 23.1.3, “ARM instruction encodings”
+aarch32 has two encodings: Thumb and ARM: Section 24.1.3, “ARM instruction encodings”
in ARMv8, the stack can be enforced to 16-byte alignment: Section 23.3.2.2.1, “ARMV8 aarch64 stack alignment”
+in ARMv8, the stack can be enforced to 16-byte alignment: Section 24.3.2.2.1, “ARMV8 aarch64 stack alignment”
The ARM instruction set is itself protected by patents / copyright / whatever, and you have to pay ARM Holdings a licence to implement it, even if you are creating your own custom Verilog code.
Understanding the basics of instruction encodings is fundamental to help you to remember what instructions do and why some things are possible or not, notably the ARM LDR pseudo-instruction and the ADRP instruction.
Thumb examples are available at:
ARM can switch between big and little endian mode on the fly!
Unconditional branch.
Branch if equal based on the status registers.
Branch with link, i.e. branch and store the return address on the RL register.
Example: userland/arch/aarch64/ret.S
Compare and branch if zero.
Weirdly, ARM B instruction and family are not the only instructions that can execute conditionally on the flags: the same also applies to most instructions, e.g. ADD.
In ARM, there are only two instruction families that do memory access:
LDR can be either a regular instruction that loads stuff into memory, or also a pseudo-instruction (assembler magic): http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0041c/Babbfdih.html
Example: userland/arch/arm/address_modes.S
As an application of the post-indexed addressing mode, let’s increment an array.
There are LDR variants that load less than full 4 bytes:
Store from memory into registers.
PC-relative STR is not possible in aarch64.
Push a pair of registers to the stack.
In ARMv8, the stack can be enforced to 16-byte alignment.
Pop values form stack into the register and optionally update the address register.
Arithmetic:
Example: userland/arch/aarch64/cset.S
Bitwise Bit Clear: clear some bits.
Unsigned Bitfield Move.
TODO: explain full behaviour. Very complicated. Has several simpler to understand aliases.
Alias for:
TODO: explain. Similar to UBFM but leave untouched bits unmodified.
Examples:
Move an immediate to a register, or a register to another register.
Cannot load from or to memory, since only the LDR and STR instruction families can do that in ARM as mentioned at: Section 23.3, “ARM load and store instructions”.
+Cannot load from or to memory, since only the LDR and STR instruction families can do that in ARM as mentioned at: Section 24.3, “ARM load and store instructions”.
Example: userland/arch/arm/mov.S
@@ -25590,7 +25639,7 @@ ldmia sp!, reglistAssemblers however support magic memory allocations which may hide what is truly going on: https://stackoverflow.com/questions/14046686/why-use-ldr-over-mov-or-vice-versa-in-arm-assembly Always ask your friendly disassembly for a good confirmation.
Set the higher or lower 16 bits of a register to an immediate in one go.
Fill a 64 bit register with 4 16-bit instructions one at a time.
Set 16-bits negated and the rest to 1.
Most data processing instructions can also optionally shift the second register operand.
Example: userland/arch/arm/s_suffix.S
Similar rationale to the ARM LDR pseudo-instruction, allowing to easily store a PC-relative reachable address into a register in one go, to overcome the 4-byte fixed instruction size.
Parent section: Section 21.10, “NOP instructions”
+Parent section: Section 22.10, “NOP instructions”
There are a few different ways to encode NOP, notably MOV a register into itself, and a dedicated miscellaneous instruction.
@@ -25730,7 +25779,7 @@ ldmia sp!, reglistGuaranteed undefined! Therefore raise illegal instruction signal. Used by GCC __builtin_trap apparently: https://stackoverflow.com/questions/16081618/programmatically-cause-undefined-instruction-exception
Parent section: Section 21.3, “SIMD assembly”
+Parent section: Section 22.3, “SIMD assembly”
The name for the ARMv7 and AArch32 floating point and SIMD instructions / registers.
TODO example
userland/arch/arm/vadd_scalar.S: see also: Section 21.2, “Floating point assembly”
+userland/arch/arm/vadd_scalar.S: see also: Section 22.2, “Floating point assembly”
userland/arch/arm/vadd_vector.S: see also: Section 21.3, “SIMD assembly”
+userland/arch/arm/vadd_vector.S: see also: Section 22.3, “SIMD assembly”
Example: userland/arch/arm/vcvt.S
Example: userland/arch/arm/vcvtr.S
Example: userland/arch/arm/vcvt.S
The ARMv8 architecture reference manual specifies floating point and SIMD support in the main architecture at A1.5 "Advanced SIMD and floating-point support".
The feature is often refered to simply as "SIMD&FP" throughout the manual.
The Linux kernel shows /proc/cpuinfo compatibility as neon, which is yet another intermediate name that came up at some point, see: Section 23.6.2.2, “ARM NEON”.
The Linux kernel shows /proc/cpuinfo compatibility as neon, which is yet another intermediate name that came up at some point, see: Section 24.6.2.2, “ARM NEON”.
Vs ARM VFP: https://stackoverflow.com/questions/4097034/arm-cortex-a8-whats-the-difference-between-vfp-and-neon
Support is semi-mandatory. ARMv8 architecture reference manual A1.5 "Advanced SIMD and floating-point support":
Just an informal name for the "Advanced SIMD instructions"? Very confusing.
TODO example.
userland/arch/aarch64/fadd_vector.S: see also: Section 21.3, “SIMD assembly”
+userland/arch/aarch64/fadd_vector.S: see also: Section 22.3, “SIMD assembly”
userland/arch/aarch64/fadd_scalar.S: see also: Section 21.2, “Floating point assembly”
+userland/arch/aarch64/fadd_scalar.S: see also: Section 22.2, “Floating point assembly”
It is very confusing, but FADDS and FADDD in Aarch32 are pre-UAL for vadd.f32 and vadd.f64 which we use in this tutorial, see: Section 23.6.1.2, “ARM VADD instruction”
It is very confusing, but FADDS and FADDD in Aarch32 are pre-UAL for vadd.f32 and vadd.f64 which we use in this tutorial, see: Section 24.6.1.2, “ARM VADD instruction”
The same goes for most ARMv7 mnemonics: f* is old, and v* is the newer better syntax.
Also keep in mind that fused multiply add is FMADD.
Examples at: Section 21.3, “SIMD assembly”
+Examples at: Section 22.3, “SIMD assembly”
Example: userland/arch/aarch64/ld2.S
Scalable Vector Extension.
Using SVE normally requires setting the CPACR_EL1.FPEN and ZEN bits, which as as of lkmc 29fd625f3fda79f5e0ee6cac43517ba74340d513 + 1 we also enable in our Baremetal bootloaders, see also: aarch64 baremetal NEON setup.
ARMv8 architecture reference manual A1.7 "ARMv8 architecture extensions" says:
ARMv8 architecture reference manual db A1.7.3 "The ARMv8.1 architecture extension"
ARMv8 architecture reference manual db "ARMv8.1-LSE, ARMv8.1 Large System Extensions"
Good getting started tutorials:
The official manuals were stored in http://infocenter.arm.com but as of 2017 they started to slowly move to https://developer.arm.com.
Bibliography: https://www.quora.com/Where-can-I-find-the-official-documentation-of-ARM-instruction-set-architectures-ISAs
ARM also releases documentation specific to each given processor.
This adds extra details to the more portable ARMv8 architecture reference manual ISA documentation.
https://en.wikipedia.org/wiki/Executable_and_Linkable_Format
@@ -26424,7 +26473,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.Getting started at: Section 1.7, “Baremetal setup”
GDB step debug works on baremetal exactly as it does on the Linux kernel, which is described at: Section 2, “GDB step debug”.
As can be seen from Baremetal GDB step debug, all examples under baremetal/, with the exception of baremetal/arch/<arch>/no_bootloader, start from our tiny bootloaders:
the stack pointer
TODO: we don’t do this currently but maybe we should setup BSS
@@ -26595,7 +26644,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.Semihosting is a publicly documented interface specified by ARM Holdings that allows us to do some magic operations very useful in development.
For gem5, you need:
TODO: our example is printing newlines without automatic carriage return \r as in:
For arm, some baremetal examples compile fine with:
TODO not working as of 8825222579767f2ee7e46ffd8204b9e509440759 + 1. Not yet properly researched / reported upstream yet.
It is incredible, but GDB also has a CPU simulator inside of it as documented at: https://sourceware.org/gdb/onlinedocs/gdb/Target-Commands.html
Since I had this compiled, I also decided to try it out on userland.
In this section we will focus on learning ARM architecture concepts that can only learnt on baremetal setups.
Userland information can be found at: https://github.com/cirosantilli/arm-assembly-cheat
ARM exception levels are analogous to x86 rings.
According to ARMv7 architecture reference manual, access to that register is controlled by other registers NSACR.{CP11, CP10} and HCPTR so those must be turned off, but I’m lazy to investigate now, even just trying to dump those registers in userland/arch/arm/dump_regs.c also leads to exceptions…
TODO. Create a minimal runnable example of going into EL0 and jumping to EL1.
See ARMv8 architecture reference manual db D1.6.2 "The stack pointer registers".
This is the most basic example of exception handling we have.
The vector table format is described on ARMv8 architecture reference manual Table D1-7 "Vector offsets from vector table base address".
Exception Syndrome Register.
See example at: Section 26.8.2, “ARM SVC instruction”
+See example at: Section 27.8.2, “ARM SVC instruction”
Documentation: ARMv8 architecture reference manual db D12.2.36 "ESR_EL1, Exception Syndrome Register (EL1)".
Exception Link Register.
See the example at: Section 26.8.2, “ARM SVC instruction”
+See the example at: Section 27.8.2, “ARM SVC instruction”
Examples:
Bibliography: https://stackoverflow.com/questions/980999/what-does-multicore-assembly-language-look-like/33651438#33651438
The WFE and SEV instructions are just hints: a compliant implementation can treat them as NOPs.
In QEMU, CPU 1 starts in a halted state. This can be observed from GDB, where:
TODO: create and study a minimal examples in gem5 where the DMB instruction leads to less cycles: https://stackoverflow.com/questions/15491751/real-life-use-cases-of-barriers-dsb-dmb-isb-in-arm
The ARM timer is the simplest way to generate hardware interrupts periodically, and therefore serves as the simples example of ARM GIC usage.
Generic Interrupt Controller.
TODO create a minimal working aarch64 example analogous to the x86 one at: https://github.com/cirosantilli/x86-bare-metal-examples/blob/6dc9a73830fc05358d8d66128f740ef9906f7677/paging.S
First, also consider the userland bibliography: Section 23.8, “ARM assembly bibliography”.
+First, also consider the userland bibliography: Section 24.8, “ARM assembly bibliography”.
The most useful ARM baremetal example sets we’ve seen so far are:
@@ -28077,7 +28126,7 @@ cntvct_el0 0x3CF516FIt is nice when thing just work.
But you can also learn a thing or two from how I actually made them work in the first place.
Enter the QEMU console:
Inside baremetal/lib/aarch64.S there is a chunk of code that enables floating point operations:
Baremetal tests work exactly like User mode tests, except that you have to add the --mode baremetal option, for example:
In baremetal, we detect if tests failed by parsing logs for the Magic failure string.
See: Section 29.13, “Test this repo” for more useful testing tips.
+See: Section 30.13, “Test this repo” for more useful testing tips.
Remember: Android AOSP is a huge undocumented piece of bloatware. It’s integration into this repo will likely never be super good.
@@ -28413,7 +28462,7 @@ ISBTested on: 8.1.0_r60.
Tested on: 8.1.0_r60.
From mount, we can see that some of the mounted images are ro.
When I install an app like F-Droid, it goes under /data according to:
I don’t know how to download files from the web on Vanilla android, the default browser does not download anything, and there is no wget:
For Linux in general, see: Section 6, “init”.
TODO: didn’t fully port during refactor after 3b0a343647bed577586989fb702b760bd280844a. Reimplementing should not be hard.
@@ -28759,7 +28808,7 @@ cd -We have exploreed a few Continuous integration solutions.
We haven’t setup any of them yet.
We tried to automate it on Travis with .travis.yml but it hits the current 50 minute job timeout: https://travis-ci.org/cirosantilli/linux-kernel-module-cheat/builds/296454523 And I bet it would likely hit a disk maxout either way if it went on.
This setup sucessfully built gem5 on every commit: .circleci/config.yml
Run all kernel boot benchmarks for one arch:
TODO: aarch64 gem5 and QEMU use the same kernel, so why is the gem5 instruction count so much much higher?
TODO 62f6870e4e0b384c4bd2d514116247e81b241251 takes 33 minutes to finish at 62f6870e4e0b384c4bd2d514116247e81b241251:
The build times are calculated after doing ./configure and make source, which downloads the sources, and basically benchmarks the Internet.
./build-buildroot -- graph-build graph-size graph-depends @@ -28963,14 +29012,14 @@ xdg-open graph-size.pdf
The biggest build time hog is always GCC, and it does not look like we can use a precompiled one: https://stackoverflow.com/questions/10833672/buildroot-environment-with-host-toolchain
This is the minimal build we could expect to get away with.
How long it takes to build gem5 itself.
This is the critical development parameter, and is dominated by the link time of huge binaries.
Lenovo ThinkPad P51 laptop:
2c12b21b304178a81c9912817b782ead0286d282:
gem5:
The host requirements depend a lot on which examples you want to run.
If ./build --download-dependencies fails with:
It does not work if you just download the .zip with the sources for this repository from GitHub because we use Git submodules, you must clone this repo.
If you just want to run a command after boot ends without thinking much about it, just use the --eval-after option, e.g.:
It gets annoying to retype --arch aarch64 for every single command, or to remember --config setups.
To learn how to build the documentation see: Section 1.8, “Build the documentation”.
When running build-doc, we do the following checks:
The scripts prints what you have to fix and exits with an error status if there are any errors.
Documentation for asciidoctor/extract-link-targets
Documentation for asciidoctor/extract-header-ids
The Asciidoctor extension scripts:
As mentioned before the TOC, we have to push this README to GitHub pages due to: https://github.com/isaacs/github/issues/1610
You did something crazy, and nothing seems to work anymore?
ccache might save you a lot of re-build when you decide to Clean the build or create a new build variant.
It is not possible to rebuild the root filesystem while running QEMU because QEMU holds the file qcow2 file:
When doing long simulations sweeping across multiple system parameters, it becomes fundamental to do multiple simulations in parallel.
To run multiple gem5 checkouts, see: Section 29.11.3.1, “gem5 worktree”.
+To run multiple gem5 checkouts, see: Section 30.11.3.1, “gem5 worktree”.
Implementation note: we create multiple namespaces for two things:
@@ -29807,7 +29856,7 @@ less "$(./getvar --arch aarch64 --emulator gem5 --run-id 1 termout_file)"It often happens that you are comparing two versions of the build, a good and a bad one, and trying to figure out why the bad one is bad.
Our build variants system allows you to keep multiple built versions of all major components, so that you can easily switching between running one or the other.
If you want to keep two builds around, one for the latest Linux version, and the other for Linux v4.16:
To run both kernels simultaneously, one on each QEMU instance, see: Section 29.10, “Simultaneous runs”.
+To run both kernels simultaneously, one on each QEMU instance, see: Section 30.10, “Simultaneous runs”.
Analogous to the Linux kernel build variants but with the --qemu-build-id option instead:
Analogous to the Linux kernel build variants but with the --gem5-build-id option instead:
Therefore, you can’t forget to checkout to the sources to that of the corresponding build before running, unless you explicitly tell gem5 to use a non-default source tree with gem5 worktree. This becomes inevitable when you want to launch multiple simultaneous runs at different checkouts.
--gem5-build-id goes a long way, but if you want to seamlessly switch between two gem5 tress without checking out multiple times, then --gem5-worktree is for you.
Suppose that you are working on a private fork of gem5, but you want to use this repository to develop it as well.
Allows you to have multiple versions of the GCC toolchain or root filesystem.
lkmc/ contains sources and headers that are shared across kernel modules, userland and baremetal examples.
Another option would have been to name it as includes/lkmc, but that would make paths longer, and we might want to store source code in that directory as well in the future.
When factoring out functionality across userland examples, there are two main options:
Source: buildroot_packages/
A custom build script can give you more flexibility: e.g. the package can be made work with other root filesystems more easily, have better 9P support, and rebuild faster as it evades some Buildroot boilerplate.
Has the following structure:
Patches in this directory are never applied automatically: it is up to users to manually apply them before usage following the instructions in this documentation.
We use this directory for:
The files:
Print out several parameters that normally change randomly from boot to boot:
lkmc_home refers to the target base directory in which we put all our custom built stuff, such as userland executables and kernel modules.
Run almost all tests:
test does not all possible tests, because there are too many possible variations and that would take forever. The rationale is the same as for ./build all and is explained in ./build --help.
You can select multiple archs and emulators of interest, as for an other command, with:
By default, continue running even after the first failure happens, and they show a summary at the end.
TODO: we really need a mechanism to automatically generate the test list automatically e.g. based on path_properties, currently there are many tests missing, and we have to add everything manually which is very annoying.
We have some pexpect automated tests for GDB for both userland and baremetal programs!
We do not know of any way to set the emulator exit status in QEMU arm full system.
For the Linux kernel, do the following manual tests for now.
You should also test that the Internet works:
build-userland and test-executables have a wide variety of target selection modes, and it was hard to keep them all working without some tests:
When updating the Linux kernel, QEMU and gem5, things sometimes break.
In order to build and run each userland and baremetal example properly, we need per-file metadata such as compiler flags and required number of cores.
This is a template update procedure for submodules for which we have some patches on on top of mainline.
Ensure that the Automated tests are passing on a clean build:
The ./build-test command builds a superset of what will be downloaded which also tests other things we would like to be working on the release. For the minimal build to generate the files to be uploaded, see: Section 29.17.2, “release-zip”
The ./build-test command builds a superset of what will be downloaded which also tests other things we would like to be working on the release. For the minimal build to generate the files to be uploaded, see: Section 30.17.2, “release-zip”
The clean build is necessary as it generates clean images since it is not possible to remove Buildroot packages
@@ -30943,7 +30992,7 @@ git push --follow-tagsCreate a zip containing all files required for Prebuilt setup:
After:
This project was created to help me understand, modify and test low level system components by using system simulators.
The trade-offs between the different setups are basically a balance between:
compatibility: how likely is is that all the components will work well together: emulator, compiler, kernel, standard library, …
guest software availability: how wide is your choice of easily installed guest software packages? See also: Section 29.18.4, “Linux distro choice”
+guest software availability: how wide is your choice of easily installed guest software packages? See also: Section 30.18.4, “Linux distro choice”
Choosing which features go into our default builds means making tradeoffs, here are our guidelines:
In order to learn how to measure some of those aspects, see: Section 28, “Benchmark this repo”.
+In order to learn how to measure some of those aspects, see: Section 29, “Benchmark this repo”.
We haven’t found the ultimate distro yet, here is a summary table of trade-offs that we care about: Table 7, “Comparison of Linux distros for usage in this repository”.
@@ -31312,7 +31361,7 @@ git push --follow-tags
Being the hardcore person who fully understands an important complex system such as a computer, it does have a nice ring to it doesn’t it?