diff --git a/index.html b/index.html index 9e8ad6f..64d5a65 100644 --- a/index.html +++ b/index.html @@ -556,7 +556,11 @@ pre{ white-space:pre }
  • 2.5. GDB step debug early boot @@ -805,1503 +809,1507 @@ pre{ white-space:pre }
    • -
  • 15. Linux kernel +
  • 15. Operating systems
    • +
  • 16. Linux kernel
    • -
  • 16. Xen
    • -
  • 17. U-Boot
    • -
  • 18. QEMU +
  • 17. FreeBSD
    • +
  • 18. RTOS -
    • -
  • 19. gem5 +
  • 19. Xen
    • +
  • 20. U-Boot
    • +
  • 21. Emulators
    • +
  • 22. QEMU -
    • -
  • 20. Gensim
    • -
  • 21. Buildroot +
  • 23. gem5 +
    • +
  • 24. Gensim
    • +
  • 25. Buildroot -
    • -
  • 23. Userland assembly +
  • 26. Userland content +
    • +
  • 27. Userland assembly
    • -
  • 25. ARM userland assembly +
  • 28. x86 userland assembly -
    • -
  • 26. ELF
    • -
  • 27. IEEE 754
    • -
  • 28. Baremetal +
  • 29. ARM userland assembly -
    • -
  • 29. Android - -
    • -
  • 30. Benchmark this repo - -
    • -
  • 31. RTOS - -
    • -
  • 32. Compilers - -
    • -
  • 33. Computer architecture - +
    • +
  • 30. ELF
    • +
  • 31. IEEE 754
    • +
  • 32. Baremetal
    -

    and can therefore be used to estimate system performance, see: Section 19.2, “gem5 run benchmark” for an example.

    +

    and can therefore be used to estimate system performance, see: Section 23.2, “gem5 run benchmark” for an example.

    The downside of gem5 much slower than QEMU because of the greater simulation detail.

    @@ -3327,7 +3335,7 @@ j = 0
    -

    More gem5 information is present at: Section 19, “gem5”

    +

    More gem5 information is present at: Section 23, “gem5”

    Good next steps are:

    @@ -3356,7 +3364,7 @@ j = 0

    This repository has been tested inside clean Docker containers.

    -

    This is a good option if you are on a Linux host, but the native setup failed due to your weird host distribution, and you have better things to do with your life than to debug it. See also: Section 34.1, “Supported hosts”.

    +

    This is a good option if you are on a Linux host, but the native setup failed due to your weird host distribution, and you have better things to do with your life than to debug it. See also: Section 37.1, “Supported hosts”.

    For example, to do a QEMU Buildroot setup inside Docker, run:

    @@ -3544,7 +3552,7 @@ j = 0
    -

    as shown at: Section 18.8, “Debug the emulator”, although direct GDB host usage works as well of course.

    +

    as shown at: Section 22.8, “Debug the emulator”, although direct GDB host usage works as well of course.

    @@ -4318,7 +4326,7 @@ error: simulation error detected by parsing logs
    -

    TODO: the carriage returns are a bit different than in QEMU, see: Section 28.6, “gem5 baremetal carriage return”.

    +

    TODO: the carriage returns are a bit different than in QEMU, see: Section 32.6, “gem5 baremetal carriage return”.

    Note that ./build-baremetal requires the --emulator gem5 option, and generates separate executable images for both, as can be seen from:

    @@ -4351,7 +4359,7 @@ echo "$(./getvar --arch aarch64 --emulator gem5 image)"
    -

    see also: Section 19.18, “gem5 ARM platforms”.

    +

    see also: Section 23.18, “gem5 ARM platforms”.

    This generates yet new separate images with new magic constants:

    @@ -4366,10 +4374,10 @@ echo "$(./getvar --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 -

    But just stick to newer and better VExpress_GEM5_V1 unless you have a good reason to use RealViewPBX.

    -

    When doing baremetal programming, it is likely that you will want to learn userland assembly first, see: Section 23, “Userland assembly”.

    +

    When doing baremetal programming, it is likely that you will want to learn userland assembly first, see: Section 27, “Userland assembly”.

    -

    For more information on baremetal, see the section: Section 28, “Baremetal”.

    +

    For more information on baremetal, see the section: Section 32, “Baremetal”.

    The following subjects are particularly important:

    @@ -4434,7 +4442,7 @@ xdg-open README.html
    -

    More information about our documentation internals can be found at: Section 34.5, “Documentation”

    +

    More information about our documentation internals can be found at: Section 37.5, “Documentation”

    @@ -5220,6 +5228,52 @@ echo 'file kernel/module.c +p' > /sys/kernel/debug/dynamic_debug/control

    Further discussion at: Linux kernel entry point.

    +
    +

    In the specific case of gem5 aarch64 at least:

    +
    +
    + +

    2.5.1. Linux kernel entry point

    @@ -5241,9 +5295,6 @@ echo 'file kernel/module.c +p' > /sys/kernel/debug/dynamic_debug/control

    gem5 ExecAll trace format>> however does show the right symbols however! This could be because gem5 uses vmlinux to boot, which QEMU uses the compressed version, and as mentioned on the Stack Overflow answer, the entry point is actually a tiny decompresser routine.

    -

    In gem5 aarch64 Linux v4.18, experimentally the entry point of secondary CPUs seems to be secondary_holding_pen as shown at https://gist.github.com/cirosantilli2/34a7bc450fcb6c1c1a910369be1fdd90

    -
    -

    I also tried to hack run-gdb with:

    @@ -5263,7 +5314,7 @@ echo 'file kernel/module.c +p' > /sys/kernel/debug/dynamic_debug/control

    and no I do have the symbols from arch/arm/boot/compressed/vmlinux', but the breaks still don’t work.

    -

    v4.19 also added a CONFIG_HAVE_KERNEL_UNCOMPRESSED=y option for having the kernel uncompressed which could make following the startup easier, but it is only available on s390. aarch64 however is already uncompressed by default, so might be the easiest one. See also: Section 15.20.1, “vmlinux vs bzImage vs zImage vs Image”.

    +

    v4.19 also added a CONFIG_HAVE_KERNEL_UNCOMPRESSED=y option for having the kernel uncompressed which could make following the startup easier, but it is only available on s390. aarch64 however is already uncompressed by default, so might be the easiest one. See also: Section 16.20.1, “vmlinux vs bzImage vs zImage vs Image”.

    You then need the associated KERNEL_UNCOMPRESSED to enable it if available:

    @@ -5275,6 +5326,92 @@ echo 'file kernel/module.c +p' > /sys/kernel/debug/dynamic_debug/control depends on HAVE_KERNEL_UNCOMPRESSED
    +
    +
    2.5.1.1. arm64 secondary CPU entry point
    +
    +

    In gem5 aarch64 Linux v4.18, experimentally the entry point of secondary CPUs seems to be secondary_holding_pen as shown at https://gist.github.com/cirosantilli2/34a7bc450fcb6c1c1a910369be1fdd90

    +
    +
    +

    What happens is that:

    +
    +
    +
      +
    • +

      the bootloader goes in in WFE

      +
      • +
    • +

      the kernel writes the entry point to the secondary CPU (the address of secondary_holding_pen) with CPU0 at the address given to the kernel in the cpu-release-addr of the DTB

      +
      • +
    • +

      the kernel wakes up the bootloader with a SEV, and the bootloader boots to the address the kernel told it

      +
      • +
    +
    +
    +

    The CPU0 action happens at: https://github.com/cirosantilli/linux/blob/v5.7/arch/arm64/kernel/smp_spin_table.c:

    +
    +
    +

    Here’s the code that writes the address and does SEV:

    +
    +
    +
    +
    static int smp_spin_table_cpu_prepare(unsigned int cpu)
+{
+	__le64 __iomem *release_addr;
+
+	if (!cpu_release_addr[cpu])
+		return -ENODEV;
+
+	/*
+	 * The cpu-release-addr may or may not be inside the linear mapping.
+	 * As ioremap_cache will either give us a new mapping or reuse the
+	 * existing linear mapping, we can use it to cover both cases. In
+	 * either case the memory will be MT_NORMAL.
+	 */
+	release_addr = ioremap_cache(cpu_release_addr[cpu],
+				     sizeof(*release_addr));
+	if (!release_addr)
+		return -ENOMEM;
+
+	/*
+	 * We write the release address as LE regardless of the native
+	 * endianess of the kernel. Therefore, any boot-loaders that
+	 * read this address need to convert this address to the
+	 * boot-loader's endianess before jumping. This is mandated by
+	 * the boot protocol.
+	 */
+	writeq_relaxed(__pa_symbol(secondary_holding_pen), release_addr);
+	__flush_dcache_area((__force void *)release_addr,
+			    sizeof(*release_addr));
+
+	/*
+	 * Send an event to wake up the secondary CPU.
+	 */
+	sev();
    +
    +
    +
    +

    and here’s the code that reads the value from the DTB:

    +
    +
    +
    +
    static int smp_spin_table_cpu_init(unsigned int cpu)
+{
+	struct device_node *dn;
+	int ret;
+
+	dn = of_get_cpu_node(cpu, NULL);
+	if (!dn)
+		return -ENODEV;
+
+	/*
+	 * Determine the address from which the CPU is polling.
+	 */
+	ret = of_property_read_u64(dn, "cpu-release-addr",
+				   &cpu_release_addr[cpu]);
    +
    +
    +

    2.5.2. Linux kernel arch-agnostic entry point

    @@ -5680,7 +5817,7 @@ Breakpoint 3 at 0xffffffff811615e3: fdget_pos. (9 locations)

    2.9. GDB step debug multicore userland

    -

    For a more minimal baremetal multicore setup, see: Section 28.10.3, “ARM baremetal multicore”.

    +

    For a more minimal baremetal multicore setup, see: Section 32.10.3, “ARM baremetal multicore”.

    We can set and get which cores the Linux kernel allows a program to run on with sched_getaffinity and sched_setaffinity:

    @@ -5728,7 +5865,7 @@ sched_getcpu = 0
    -

    The number of cores is modified as explained at: Section 19.3.1, “Number of cores”

    +

    The number of cores is modified as explained at: Section 23.3.1, “Number of cores”

    taskset from the util-linux package sets the initial core affinity of a program:

    @@ -6863,7 +7000,7 @@ cr3 = 0xFFFFF0DCDC000

    Source: rootfs_overlay/lkmc/eval_base64.sh.

    -

    This allows quoting and newlines by base64 encoding on host, and decoding on guest, see: Section 15.3.1, “Kernel command line parameters escaping”.

    +

    This allows quoting and newlines by base64 encoding on host, and decoding on guest, see: Section 16.3.1, “Kernel command line parameters escaping”.

    It also automatically chooses between init= and rcinit= for you, see: Section 6.3, “Path to init”

    @@ -7337,7 +7474,7 @@ cat f

    which can be good for automated tests, as it ensures that you are using a pristine unmodified system image every time.

    -

    Not however that we already disable disk persistency by default on ext2 filesystems even without --initrd: Section 18.3, “Disk persistency”.

    +

    Not however that we already disable disk persistency by default on ext2 filesystems even without --initrd: Section 22.3, “Disk persistency”.

    One downside of this method is that it has to put the entire filesystem into memory, and could lead to a panic:

    @@ -7535,7 +7672,7 @@ cat f
    -

    We think that this might be because gem5 boots directly vmlinux, and not from the final compressed images that contain the attached rootfs such as bzImage, which is what QEMU does, see also: Section 15.20.1, “vmlinux vs bzImage vs zImage vs Image”.

    +

    We think that this might be because gem5 boots directly vmlinux, and not from the final compressed images that contain the attached rootfs such as bzImage, which is what QEMU does, see also: Section 16.20.1, “vmlinux vs bzImage vs zImage vs Image”.

    To do this failed test, we automatically pass a dummy disk image as of gem5 7fa4c946386e7207ad5859e8ade0bbfc14000d91 since the scripts don’t handle a missing --disk-image well, much like is currently done for Baremetal.

    @@ -8060,7 +8197,7 @@ qw er
    -

    To stop at the very first instruction of a freestanding program, just use --no-continue. A good example of this is shown at: Section 23.5.1, “Freestanding programs”.

    +

    To stop at the very first instruction of a freestanding program, just use --no-continue. A good example of this is shown at: Section 27.5.1, “Freestanding programs”.

    @@ -8113,7 +8250,7 @@ qw er

    The gem5 tests require building statically with build id static, see also: Section 10.7, “gem5 syscall emulation mode”. TODO automate this better.

    -

    See: Section 34.16, “Test this repo” for more useful testing tips.

    +

    See: Section 37.16, “Test this repo” for more useful testing tips.

    @@ -8154,7 +8291,7 @@ qw er
    -

    Here is an interesting examples of this: Section 15.19.1, “Linux Test Project”

    +

    Here is an interesting examples of this: Section 16.19.1, “Linux Test Project”

    @@ -8528,7 +8665,7 @@ qemu: uncaught target signal 6 (Aborted) - core dumped

    Support for dynamic linking was added in November 2019: https://stackoverflow.com/questions/50542222/how-to-run-a-dynamically-linked-executable-syscall-emulation-mode-se-py-in-gem5/50696098#50696098

    -

    Note that as shown at Section 30.2.2, “Benchmark emulators on userland executables”, the dynamic version runs 200x more instructions, which might have an impact on smaller simulations in detailed CPUs.

    +

    Note that as shown at Section 34.2.2, “Benchmark emulators on userland executables”, the dynamic version runs 200x more instructions, which might have an impact on smaller simulations in detailed CPUs.

    @@ -8965,10 +9102,10 @@ Program aborted at tick 0
    @@ -9089,7 +9226,7 @@ Program aborted at tick 0

    no need to regenerate the root filesystem at all and reboot

  • -

    overcomes the check_bin_arch problem as shown at: Section 21.8, “Buildroot rebuild is slow when the root filesystem is large”

    +

    overcomes the check_bin_arch problem as shown at: Section 25.8, “Buildroot rebuild is slow when the root filesystem is large”

    • @@ -9872,7 +10009,7 @@ xeyes

    14.1. Enable networking

    -

    We disable networking by default because it starts an userland process, and we want to keep the number of userland processes to a minimum to make the system more understandable as explained at: Section 34.20.3, “Resource tradeoff guidelines”

    +

    We disable networking by default because it starts an userland process, and we want to keep the number of userland processes to a minimum to make the system more understandable as explained at: Section 37.20.3, “Resource tradeoff guidelines”

    To enable networking on Buildroot, simply run:

    @@ -10212,27 +10349,87 @@ mount -t 9p -o trans=virtio,version=9p2000.L host0 /mnt/my9p

    14.4.3. 9P gem5

    -

    TODO seems possible! Lets do it:

    -
    -
    - +

    Is possible on aarch64 as shown at: https://gem5-review.googlesource.com/c/public/gem5/+/22831, and it is just a matter of exposing to X86 for those that want it.

    -

    From the source, there is just one exported tag named gem5, so we could try on the guest:

    +

    Enable it by passing the --vio-9p option on the fs.py gem5 command line:

    +
    +
    +
    +
    ./run --arch aarch64 --emulator gem5 -- --vio-9p
    +
    +
    +
    +

    Then on the guest:

    mkdir -p /mnt/9p/gem5
-mount -t 9p -o trans=virtio,version=9p2000.L gem5 /mnt/9p/data
    +mount -t 9p -o trans=virtio,version=9p2000.L,aname=/path/to/linux-kernel-module-cheat/out/run/gem5/aarch64/0/m5out/9p/share gem5 /mnt/9p/gem5 +echo asdf > /mnt/9p/gem5/qwer
    +
    +

    Yes, you have to pass the full path to the directory on the host. Yes, this is horrible.

    +
    +
    +

    The shared directory is:

    +
    +
    +
    +
    out/run/gem5/aarch64/0/m5out/9p/share
    +
    +
    +
    +

    so we can observe the file the guest wrote from the host with:

    +
    +
    +
    +
    out/run/gem5/aarch64/0/m5out/9p/share/qwer
    +
    +
    +
    +

    and vice versa:

    +
    +
    +
    +
    echo zxvc > out/run/gem5/aarch64/0/m5out/9p/share/qwer
    +
    +
    +
    +

    is now visible from the guest:

    +
    +
    +
    +
    cat /mnt/9p/gem5/qwer
    +
    +
    +
    +

    Checkpoint restore with an open mount will likely fail because gem5 uses an ugly external executable to implement diod. The protocol is not very complex, and QEMU implements it in-tree, which is what gem5 should do as well at some point.

    +
    +
    +

    Also checkpoint without --vio-9p and restore with --vio-9p did not work either, the mount fails.

    +
    +
    +

    However, this did work, on guest:

    +
    +
    +
    +
    unmount /mnt/9p/gem5
+m5 checkpoint
    +
    +
    +
    +

    then restore with the detalied CPU of interest e.g.

    +
    +
    +
    +
    ./run --arch aarch64 --emulator gem5 -- --vio-9p --cpu-type DerivO3CPU --caches
    +
    +
    +
    +

    Tested on gem5 b2847f43c91e27f43bd4ac08abd528efcf00f2fd, LKMC 52a5fdd7c1d6eadc5900fc76e128995d4849aada.

    +

    14.4.4. NFS

    @@ -10310,12 +10507,42 @@ mount -t nfs 10.0.2.2:/tmp /mnt/nfs
    -

    15. Linux kernel

    +

    15. Operating systems

    +
    +

    https://en.wikipedia.org/wiki/Operating_system

    +
    +
    + +
    +
    +
    +
    +

    16. Linux kernel

    +
    +
    +

    https://en.wikipedia.org/wiki/Linux_kernel

    +
    -

    15.1. Linux kernel configuration

    +

    16.1. Linux kernel configuration

    -

    15.1.1. Modify kernel config

    +

    16.1.1. Modify kernel config

    To modify a single option on top of our default kernel configs, do:

    @@ -10360,7 +10587,7 @@ CONFIG_IKCONFIG_PROC=y
    -

    There is also a shortcut --custom-config-file to use the gem5 arm Linux kernel patches.

    +

    There is also a shortcut --custom-config-file-gem5 to use the gem5 arm Linux kernel patches.

    The following options can all be used together, sorted by decreasing config setting power precedence:

    @@ -10406,11 +10633,11 @@ cp "$(./getvar linux_build_dir)/defconfig" data/myconfig
    -

    You can also use other config generating targets such as defconfig with the same method as shown at: Section 15.1.3.1.1, “Linux kernel defconfig”.

    +

    You can also use other config generating targets such as defconfig with the same method as shown at: Section 16.1.3.1.1, “Linux kernel defconfig”.

    -

    15.1.2. Find the kernel config

    +

    16.1.2. Find the kernel config

    Get the build config in guest:

    @@ -10468,14 +10695,14 @@ CONFIG_IKCONFIG_PROC=y
    -

    15.1.3. About our Linux kernel configs

    +

    16.1.3. About our Linux kernel configs

    By default, build-linux generates a .config that is a mixture of:

    -
    15.1.3.1.2. Linux kernel min config
    +
    16.1.3.1.2. Linux kernel min config

    linux_config/min contains minimal tweaks required to boot gem5 or for using our slightly different QEMU command line options than Buildroot on all archs.

    -

    It is one of the default config fragments we use, as explained at: Section 15.1.3, “About our Linux kernel configs”>.

    +

    It is one of the default config fragments we use, as explained at: Section 16.1.3, “About our Linux kernel configs”>.

    Having the same config working for both QEMU and gem5 (oh, the hours of bisection) means that you can deal with functional matters in QEMU, which runs much faster, and switch to gem5 only for performance issues.

    @@ -10661,14 +10888,14 @@ CONFIG_IKCONFIG_PROC=y
    -
    15.1.3.2. Notable alternate gem5 kernel configs
    +
    16.1.3.2. Notable alternate gem5 kernel configs

    Other configs which we had previously tested at 4e0d9af81fcce2ce4e777cb82a1990d7c2ca7c1e are:

    -

    15.2. Kernel version

    +

    16.2. Kernel version

    -

    15.2.1. Find the kernel version

    +

    16.2.1. Find the kernel version

    We try to use the latest possible kernel major release version.

    @@ -10704,7 +10931,7 @@ git log | grep -E ' Linux [0-9]+\.' | head
    -

    15.2.2. Update the Linux kernel

    +

    16.2.2. Update the Linux kernel

    During update all you kernel modules may break since the kernel API is not stable.

    @@ -10721,15 +10948,15 @@ git log | grep -E ' Linux [0-9]+\.' | head

    This also makes this repo the perfect setup to develop the Linux kernel.

    -

    In case something breaks while updating the Linux kernel, you can try to bisect it to understand the root cause, see: Section 34.17, “Bisection”.

    +

    In case something breaks while updating the Linux kernel, you can try to bisect it to understand the root cause, see: Section 37.17, “Bisection”.

    -
    15.2.2.1. Update the Linux kernel LKMC procedure
    +
    16.2.2.1. Update the Linux kernel LKMC procedure
    -

    First, use use the branching procedure described at: Section 34.18, “Update a forked submodule”

    +

    First, use use the branching procedure described at: Section 37.18, “Update a forked submodule”

    -

    Because the kernel is so central to this repository, almost all tests must be re-run, so basically just follow the full testing procedure described at: Section 34.16, “Test this repo”. The only tests that can be skipped are essentially the Baremetal tests.

    +

    Because the kernel is so central to this repository, almost all tests must be re-run, so basically just follow the full testing procedure described at: Section 37.16, “Test this repo”. The only tests that can be skipped are essentially the Baremetal tests.

    Before comitting, don’t forget to update:

    @@ -10757,7 +10984,7 @@ git log | grep -E ' Linux [0-9]+\.' | head
    -

    15.2.3. Downgrade the Linux kernel

    +

    16.2.3. Downgrade the Linux kernel

    The kernel is not forward compatible, however, so downgrading the Linux kernel requires downgrading the userland too to the latest Buildroot branch that supports it.

    @@ -10833,7 +11060,7 @@ git log | grep -E ' Linux [0-9]+\.' | head
    -

    15.3. Kernel command line parameters

    +

    16.3. Kernel command line parameters

    Bootloaders can pass a string as input to the Linux kernel when it is booting to control its behaviour, much like the execve system call does to userland processes.

    @@ -10894,7 +11121,7 @@ git log | grep -E ' Linux [0-9]+\.' | head
    -

    15.3.1. Kernel command line parameters escaping

    +

    16.3.1. Kernel command line parameters escaping

    Double quotes can be used to escape spaces as in opt="a b", but double quotes themselves cannot be escaped, e.g. opt"a\"b"

    @@ -10903,7 +11130,7 @@ git log | grep -E ' Linux [0-9]+\.' | head
    -

    15.3.2. Kernel command line parameters definition points

    +

    16.3.2. Kernel command line parameters definition points

    There are two methods:

    @@ -10946,7 +11173,7 @@ git log | grep -E ' Linux [0-9]+\.' | head
    -

    15.3.3. rw

    +

    16.3.3. rw

    By default, the Linux kernel mounts the root filesystem as readonly. TODO rationale?

    @@ -11018,7 +11245,7 @@ mount
    -

    15.3.4. norandmaps

    +

    16.3.4. norandmaps

    Disable userland address space randomization. Test it out by running rand_check.out twice:

    @@ -11042,7 +11269,7 @@ mount
    -

    15.4. printk

    +

    16.4. printk

    printk is the most simple and widely used way of getting information from the kernel, so you should familiarize yourself with its basic configuration.

    @@ -11122,10 +11349,10 @@ mount
    -

    The debug highest level is a bit more magic, see: Section 15.4.3, “pr_debug” for more info.

    +

    The debug highest level is a bit more magic, see: Section 16.4.3, “pr_debug” for more info.

    -

    15.4.1. /proc/sys/kernel/printk

    +

    16.4.1. /proc/sys/kernel/printk

    The current printk level can be obtained with:

    @@ -11312,7 +11539,7 @@ early_param("quiet", quiet_kernel);
    -

    15.4.2. ignore_loglevel

    +

    16.4.2. ignore_loglevel

    ./run --kernel-cli 'ignore_loglevel'
    @@ -11331,7 +11558,7 @@ early_param("quiet", quiet_kernel);
    -

    15.4.3. pr_debug

    +

    16.4.3. pr_debug

    https://stackoverflow.com/questions/28936199/why-is-pr-debug-of-the-linux-kernel-not-giving-any-output/49835405#49835405

    @@ -11429,7 +11656,7 @@ insmod myprintk.ko

    Get ready for the noisiest boot ever, I think it overflows the printk buffer and funny things happen.

    -
    15.4.3.1. pr_debug != printk(KERN_DEBUG
    +
    16.4.3.1. pr_debug != printk(KERN_DEBUG

    When CONFIG_DYNAMIC_DEBUG is set, printk(KERN_DEBUG is not the exact same as pr_debug( since printk(KERN_DEBUG messages are visible with:

    @@ -11478,9 +11705,9 @@ insmod myprintk.ko
    -

    15.5. Kernel module APIs

    +

    16.5. Kernel module APIs

    -

    15.5.1. Kernel module parameters

    +

    16.5.1. Kernel module parameters

    The Linux kernel allows passing module parameters at insertion time through the init_module and finit_module system calls.

    @@ -11553,7 +11780,7 @@ parm: i:my favorite int
    -
    15.5.1.1. modprobe.conf
    +
    16.5.1.1. modprobe.conf

    modprobe insertion can also set default parameters via the /etc/modprobe.conf file:

    @@ -11580,7 +11807,7 @@ cat /sys/kernel/debug/lkmc_params
    -

    15.5.2. Kernel module dependencies

    +

    16.5.2. Kernel module dependencies

    One module can depend on symbols of another module that are exported with EXPORT_SYMBOL:

    @@ -11693,7 +11920,7 @@ extra/dep.ko:

    TODO: what for, and at which point point does Buildroot / BusyBox generate that file?

    -
    15.5.2.1. Kernel module dependencies with modprobe
    +
    16.5.2.1. Kernel module dependencies with modprobe

    Unlike insmod, modprobe deals with kernel module dependencies for us.

    @@ -11779,7 +12006,7 @@ buildroot_dep 16384 1 buildroot_dep2
    -

    15.5.3. MODULE_INFO

    +

    16.5.3. MODULE_INFO

    Module metadata is stored on module files at compile time. Some of the fields can be retrieved through the THIS_MODULE struct module:

    @@ -11899,7 +12126,7 @@ vermagic: 4.17.0 SMP mod_unload modversions
    -

    15.5.4. vermagic

    +

    16.5.4. vermagic

    Vermagic is a magic string present in the kernel and on MODULE_INFO of kernel modules. It is used to verify that the kernel module was compiled against a compatible kernel version and relevant configuration:

    @@ -11978,7 +12205,7 @@ vermagic: 4.17.0 SMP mod_unload modversions
    -

    15.5.5. init_module

    +

    16.5.5. init_module

    init_module and cleanup_module are an older alternative to the module_init and module_exit macros:

    @@ -12005,7 +12232,7 @@ cleanup_module
    -

    15.5.6. Floating point in kernel modules

    +

    16.5.6. Floating point in kernel modules

    It is generally hard / impossible to use floating point operations in the kernel. TODO understand details.

    @@ -12076,7 +12303,7 @@ cleanup_module
    -

    15.6. Kernel panic and oops

    +

    16.6. Kernel panic and oops

    To test out kernel panics and oops in controlled circumstances, try out the modules:

    @@ -12140,7 +12367,7 @@ insmod oops.ko
    -

    15.6.1. Kernel panic

    +

    16.6.1. Kernel panic

    On panic, the kernel dies, and so does our terminal.

    @@ -12190,7 +12417,7 @@ Kernel Offset: disabled
    -
    15.6.1.1. Kernel module stack trace to source line
    +
    16.6.1.1. Kernel module stack trace to source line

    The log shows which module each symbol belongs to if any, e.g.:

    @@ -12256,25 +12483,25 @@ Kernel Offset: disabled
    -
    15.6.1.2. BUG_ON
    +
    16.6.1.2. BUG_ON

    Basically just calls panic("BUG!") for most archs.

    -
    15.6.1.3. Exit emulator on panic
    +
    16.6.1.3. Exit emulator on panic

    For testing purposes, it is very useful to quit the emulator automatically with exit status non zero in case of kernel panic, instead of just hanging forever.

    -
    15.6.1.3.1. Exit QEMU on panic
    +
    16.6.1.3.1. Exit QEMU on panic

    Enabled by default with:

    -
    15.6.1.3.2. Exit gem5 on panic
    +
    16.6.1.3.2. Exit gem5 on panic

    gem5 9048ef0ffbf21bedb803b785fb68f83e95c04db8 (January 2019) can detect panics automatically if the option system.panic_on_panic is on.

    @@ -12353,7 +12580,7 @@ Kernel Offset: disabled
    -
    15.6.1.4. Reboot on panic
    +
    16.6.1.4. Reboot on panic

    Make the kernel reboot after n seconds after panic:

    @@ -12383,7 +12610,7 @@ Kernel Offset: disabled
    -
    15.6.1.5. Panic trace show addresses instead of symbols
    +
    16.6.1.5. Panic trace show addresses instead of symbols

    If CONFIG_KALLSYMS=n, then addresses are shown on traces instead of symbol plus offset.

    @@ -12422,7 +12649,7 @@ Kernel Offset: disabled
    -

    15.6.2. Kernel oops

    +

    16.6.2. Kernel oops

    On oops, the shell still lives after.

    @@ -12539,7 +12766,7 @@ CR2: 0000000000000000
    -

    15.6.3. dump_stack

    +

    16.6.3. dump_stack

    The dump_stack function produces a stack trace much like panic and oops, but causes no problems and we return to the normal control flow, and can cleanly remove the module afterwards:

    @@ -12553,7 +12780,7 @@ CR2: 0000000000000000
    -

    15.6.4. WARN_ON

    +

    16.6.4. WARN_ON

    The WARN_ON macro basically just calls dump_stack.

    @@ -12574,7 +12801,7 @@ insmod warn_on.ko
    -

    15.6.5. not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

    +

    16.6.5. not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

    Let’s learn how to diagnose problems with the root filesystem not being found. TODO add a sample panic error message for each error type:

    @@ -12725,7 +12952,7 @@ CONFIG_VIRTIO_PCI=y
    -

    15.7. Pseudo filesystems

    +

    16.7. Pseudo filesystems

    Pseudo filesystems are filesystems that don’t represent actual files in a hard disk, but rather allow us to do special operations on filesystem-related system calls.

    @@ -12746,7 +12973,7 @@ CONFIG_VIRTIO_PCI=y
    -

    15.7.1. debugfs

    +

    16.7.1. debugfs

    Debugfs is the simplest pseudo filesystem to play around with:

    @@ -12810,7 +13037,7 @@ echo $?
    -

    15.7.2. procfs

    +

    16.7.2. procfs

    Procfs is just another fops entry point:

    @@ -12861,7 +13088,7 @@ echo $?
    -
    15.7.2.1. /proc/version
    +
    16.7.2.1. /proc/version

    Its data is shared with uname(), which is a POSIX C function and has a Linux syscall to back it up.

    @@ -12902,7 +13129,7 @@ echo $?
    -

    15.7.3. sysfs

    +

    16.7.3. sysfs

    Sysfs is more restricted than procfs, as it does not take an arbitrary file_operations:

    @@ -12982,7 +13209,7 @@ echo $?
    -

    15.7.4. Character devices

    +

    16.7.4. Character devices

    Character devices can have arbitrary File operations associated to them:

    @@ -13076,7 +13303,7 @@ echo $?

    Bibliography: https://unix.stackexchange.com/questions/37829/understanding-character-device-or-character-special-files/371758#371758

    -
    15.7.4.1. Automatically create character device file on insmod
    +
    16.7.4.1. Automatically create character device file on insmod

    And also destroy it on rmmod:

    @@ -13114,9 +13341,9 @@ echo $?
    -

    15.8. Pseudo files

    +

    16.8. Pseudo files

    -

    15.8.1. File operations

    +

    16.8.1. File operations

    File operations are the main method of userland driver communication.

    @@ -13169,7 +13396,7 @@ echo $?
    -

    15.8.2. seq_file

    +

    16.8.2. seq_file

    Writing trivial read File operations is repetitive and error prone. The seq_file API makes the process much easier for those trivial cases:

    @@ -13230,7 +13457,7 @@ echo $?
    -
    15.8.2.1. seq_file single_open
    +
    16.8.2.1. seq_file single_open

    If you have the entire read output upfront, single_open is an even more convenient version of seq_file:

    @@ -13273,7 +13500,7 @@ cd
    -

    15.8.3. poll

    +

    16.8.3. poll

    The poll system call allows an user process to do a non-busy wait on a kernel event.

    @@ -13378,7 +13605,7 @@ POLLIN n=10 buf=4294893839
    -

    15.8.4. ioctl

    +

    16.8.4. ioctl

    The ioctl system call is the best way to pass an arbitrary number of parameters to the kernel in a single go:

    @@ -13475,7 +13702,7 @@ echo $?
    -

    15.8.5. mmap

    +

    16.8.5. mmap

    The mmap system call allows us to share memory between user and kernel space without copying:

    @@ -13545,7 +13772,7 @@ echo $?
    -

    15.8.6. Anonymous inode

    +

    16.8.6. Anonymous inode

    Anonymous inodes allow getting multiple file descriptors from a single filesystem entry, which reduces namespace pollution compared to creating multiple device files:

    @@ -13593,7 +13820,7 @@ echo $?
    - +

    Netlink sockets offer a socket API for kernel / userland communication:

    @@ -13658,7 +13885,7 @@ for i in `seq 16`; do ./netlink.out & done
    -

    15.9. kthread

    +

    16.9. kthread

    Kernel threads are managed exactly like userland threads; they also have a backing task_struct, and are scheduled with the same mechanism:

    @@ -13696,7 +13923,7 @@ for i in `seq 16`; do ./netlink.out & done
    -

    The sleep is done with usleep_range, see: Section 15.9.2, “sleep”.

    +

    The sleep is done with usleep_range, see: Section 16.9.2, “sleep”.

    Bibliography:

    @@ -13712,7 +13939,7 @@ for i in `seq 16`; do ./netlink.out & done
    -

    15.9.1. kthreads

    +

    16.9.1. kthreads

    Let’s launch two threads and see if they actually run in parallel:

    @@ -13755,7 +13982,7 @@ for i in `seq 16`; do ./netlink.out & done
    -

    15.9.2. sleep

    +

    16.9.2. sleep

    Count to dmesg every one second from 0 up to n - 1:

    @@ -13785,7 +14012,7 @@ for i in `seq 16`; do ./netlink.out & done
    -

    15.9.3. Workqueues

    +

    16.9.3. Workqueues

    A more convenient front-end for kthread:

    @@ -13818,7 +14045,7 @@ for i in `seq 16`; do ./netlink.out & done

    Bibliography: https://github.com/torvalds/linux/blob/v4.17/Documentation/core-api/workqueue.rst

    -
    15.9.3.1. Workqueue from workqueue
    +
    16.9.3.1. Workqueue from workqueue

    Count from 0 to 9 every second infinitely many times by scheduling a new work item from a work item:

    @@ -13844,7 +14071,7 @@ for i in `seq 16`; do ./netlink.out & done
    -

    15.9.4. schedule

    +

    16.9.4. schedule

    Let’s block the entire kernel! Yay:

    @@ -13891,7 +14118,7 @@ for i in `seq 16`; do ./netlink.out & done
    -

    15.9.5. Wait queues

    +

    16.9.5. Wait queues

    Wait queues are a way to make a thread sleep until an event happens on the queue:

    @@ -13955,7 +14182,7 @@ for i in `seq 16`; do ./netlink.out & done
    -

    15.10. Timers

    +

    16.10. Timers

    Count from 0 to 9 infinitely many times in 1 second intervals using timers:

    @@ -13996,9 +14223,9 @@ for i in `seq 16`; do ./netlink.out & done
    -

    15.11. IRQ

    +

    16.11. IRQ

    -

    15.11.1. irq.ko

    +

    16.11.1. irq.ko

    Brute force monitor every shared interrupt that will accept us:

    @@ -14104,7 +14331,7 @@ request_irq irq = 1 ret = 0
    -

    15.11.2. dummy-irq

    +

    16.11.2. dummy-irq

    The Linux kernel v4.16 mainline also has a dummy-irq module at drivers/misc/dummy-irq.c for monitoring a single IRQ.

    @@ -14162,7 +14389,7 @@ request_irq irq = 1 ret = 0
    -

    15.11.3. /proc/interrupts

    +

    16.11.3. /proc/interrupts

    In the guest with QEMU graphic mode:

    @@ -14196,12 +14423,12 @@ request_irq irq = 1 ret = 0
    -

    15.12. Kernel utility functions

    +

    16.12. Kernel utility functions

    https://github.com/torvalds/linux/blob/v4.17/Documentation/core-api/kernel-api.rst

    -

    15.12.1. kstrto

    +

    16.12.1. kstrto

    Convert a string to an integer:

    @@ -14237,7 +14464,7 @@ echo $?
    -

    15.12.2. virt_to_phys

    +

    16.12.2. virt_to_phys

    Convert a virtual address to physical:

    @@ -14307,7 +14534,7 @@ virt_to_phys(&static_var) = 0x40002308
    -
    15.12.2.1. Userland physical address experiments
    +
    16.12.2.1. Userland physical address experiments

    Only tested in x86_64.

    @@ -14428,7 +14655,7 @@ pid 110
    -
    15.12.2.1.1. QEMU xp
    +
    16.12.2.1.1. QEMU xp

    The xp QEMU monitor command reads memory at a given physical address.

    @@ -14459,7 +14686,7 @@ pid 110
    -
    15.12.2.1.2. /dev/mem
    +
    16.12.2.1.2. /dev/mem

    /dev/mem exposes access to physical addresses, and we use it through the convenient devmem BusyBox utility.

    @@ -14535,7 +14762,7 @@ Value at address 0X7C7B800 (0x7ff7dbe01800): 0x12345678
    -
    15.12.2.1.3. pagemap_dump.out
    +
    16.12.2.1.3. pagemap_dump.out

    Dump the physical address of all pages mapped to a given process using /proc/<pid>/maps and /proc/<pid>/pagemap.

    @@ -14706,7 +14933,7 @@ pid 63
    -

    15.13. Linux kernel tracing

    +

    16.13. Linux kernel tracing

    Good overviews:

    @@ -14724,7 +14951,7 @@ pid 63

    I hope to have examples of all methods some day, since I’m obsessed with visibility.

    -

    15.13.1. CONFIG_PROC_EVENTS

    +

    16.13.1. CONFIG_PROC_EVENTS

    Logs proc events such as process creation to a netlink socket.

    @@ -14791,7 +15018,7 @@ a
    -
    15.13.1.1. CONFIG_PROC_EVENTS aarch64
    +
    16.13.1.1. CONFIG_PROC_EVENTS aarch64

    0111ca406bdfa6fd65a2605d353583b4c4051781 was failing with:

    @@ -14861,7 +15088,7 @@ make: *** [_all] Error 2
    -

    15.13.2. ftrace

    +

    16.13.2. ftrace

    Trace a single function:

    @@ -14969,13 +15196,13 @@ echo function_graph > current_tracer

    TODO: can you get function arguments? https://stackoverflow.com/questions/27608752/does-ftrace-allow-capture-of-system-call-arguments-to-the-linux-kernel-or-only

    -
    15.13.2.1. ftrace system calls
    +
    16.13.2.1. ftrace system calls

    https://stackoverflow.com/questions/29840213/how-do-i-trace-a-system-call-in-linux/51856306#51856306

    -
    15.13.2.2. trace-cmd
    +
    16.13.2.2. trace-cmd

    TODO example:

    @@ -14987,7 +15214,7 @@ echo function_graph > current_tracer
    -

    15.13.3. Kprobes

    +

    16.13.3. Kprobes

    kprobes is an instrumentation mechanism that injects arbitrary code at a given address in a trap instruction, much like GDB. Oh, the good old kernel. :-)

    @@ -15052,7 +15279,7 @@ sleep 4 & sleep 4 &
    -

    15.13.4. Count boot instructions

    +

    16.13.4. Count boot instructions

    TODO: didn’t port during refactor after 3b0a343647bed577586989fb702b760bd280844a. Reimplementing should not be hard.

    @@ -15244,12 +15471,12 @@ instructions_firmware 20708
    -

    15.14. Linux kernel hardening

    +

    16.14. Linux kernel hardening

    Make it harder to get hacked and easier to notice that you were, at the cost of some (small?) runtime overhead.

    -

    15.14.1. CONFIG_FORTIFY_SOURCE

    +

    16.14.1. CONFIG_FORTIFY_SOURCE

    Detects buffer overflows for us:

    @@ -15294,12 +15521,12 @@ detected buffer overflow in strlen
    -

    15.14.2. Linux security modules

    +

    16.14.2. Linux security modules

    https://en.wikipedia.org/wiki/Linux_Security_Modules

    -
    15.14.2.1. SELinux
    +
    16.14.2.1. SELinux

    TODO get a hello world permission control working:

    @@ -15390,13 +15617,13 @@ detected buffer overflow in strlen
    -

    SELinux requires glibc as mentioned at: Section 21.10, “libc choice”.

    +

    SELinux requires glibc as mentioned at: Section 25.10, “libc choice”.

    -

    15.15. User mode Linux

    +

    16.15. User mode Linux

    I once got UML running on a minimal Buildroot setup at: https://unix.stackexchange.com/questions/73203/how-to-create-rootfs-for-user-mode-linux-on-fedora-18/372207#372207

    @@ -15408,7 +15635,7 @@ detected buffer overflow in strlen
    -

    15.16. UIO

    +

    16.16. UIO

    UIO is a kernel subsystem that allows to do certain types of driver operations from userland.

    @@ -15516,9 +15743,9 @@ detected buffer overflow in strlen
    -

    15.17. Linux kernel interactive stuff

    +

    16.17. Linux kernel interactive stuff

    -

    15.17.1. Linux kernel console fun

    +

    16.17.1. Linux kernel console fun

    Requires Graphics.

    @@ -15562,7 +15789,7 @@ detected buffer overflow in strlen
    -

    15.17.2. Linux kernel magic keys

    +

    16.17.2. Linux kernel magic keys

    Requires Graphics.

    @@ -15599,7 +15826,7 @@ sendkey shift-pgdown
    -
    15.17.2.1. Ctrl Alt Del
    +
    16.17.2.1. Ctrl Alt Del

    If you run in QEMU graphic mode:

    @@ -15633,7 +15860,7 @@ sendkey shift-pgdown
    -

    This leads Linux to try to reboot, and QEMU shutdowns due to the -no-reboot option which we set by default for, see: Section 15.6.1.3, “Exit emulator on panic”.

    +

    This leads Linux to try to reboot, and QEMU shutdowns due to the -no-reboot option which we set by default for, see: Section 16.6.1.3, “Exit emulator on panic”.

    Here is a minimal example of Ctrl Alt Del:

    @@ -15817,7 +16044,7 @@ static void halt_reboot_pwoff(int sig)
    -
    15.17.2.2. SysRq
    +
    16.17.2.2. SysRq

    We cannot test these actual shortcuts on QEMU since the host captures them at a lower level, but from:

    @@ -15888,7 +16115,7 @@ static void halt_reboot_pwoff(int sig)
    -

    15.17.3. TTY

    +

    16.17.3. TTY

    In order to play with TTYs, do this:

    @@ -16127,7 +16354,7 @@ tty63::respawn:-/bin/sh
    -
    15.17.3.1. Start a getty from outside of init
    +
    16.17.3.1. Start a getty from outside of init

    TODO: https://unix.stackexchange.com/questions/196704/getty-start-from-command-line

    @@ -16184,7 +16411,7 @@ tty63::respawn:-/bin/sh
    -
    15.17.3.2. console kernel boot parameter
    +
    16.17.3.2. console kernel boot parameter

    Take the command described at TTY and try adding the following:

    @@ -16220,7 +16447,7 @@ tty63::respawn:-/bin/sh
    - +

    If you run in Graphics, then you get a Penguin image for every core above the console! https://askubuntu.com/questions/80938/is-it-possible-to-get-the-tux-logo-on-the-text-based-boot

    @@ -16264,7 +16491,7 @@ tty63::respawn:-/bin/sh
    -

    15.18. DRM

    +

    16.18. DRM

    DRM / DRI is the new interface that supersedes fbdev:

    @@ -16347,7 +16574,7 @@ crw------- 1 root root 226, 0 May 28 09:41 card0

    Tested on: 93e383902ebcc03d8a7ac0d65961c0e62af9612b

    -

    15.18.1. kmscube

    +

    16.18.1. kmscube

    ./build-buildroot --config-fragment buildroot_config/kmscube
    @@ -16409,7 +16636,7 @@ failed to initialize legacy DRM
    -

    15.18.2. kmscon

    +

    16.18.2. kmscon

    TODO get working.

    @@ -16430,7 +16657,7 @@ failed to initialize legacy DRM
    -

    15.18.3. libdri2

    +

    16.18.3. libdri2

    TODO get working.

    @@ -16456,12 +16683,12 @@ wget \
    -

    15.19. Linux kernel testing

    +

    16.19. Linux kernel testing

    Bibliography: https://stackoverflow.com/questions/3177338/how-is-the-linux-kernel-tested

    -

    15.19.1. Linux Test Project

    +

    16.19.1. Linux Test Project

    https://github.com/linux-test-project/ltp

    @@ -16508,7 +16735,7 @@ wget \
    -

    15.19.2. stress

    +

    16.19.2. stress

    POSIX userland stress. Two versions:

    @@ -16521,7 +16748,7 @@ wget \
    -

    STRESS_NG is likely the best, but it requires glibc, see: Section 21.10, “libc choice”.

    +

    STRESS_NG is likely the best, but it requires glibc, see: Section 25.10, “libc choice”.

    Websites:

    @@ -16563,9 +16790,9 @@ ps
    -

    15.20. Linux kernel build system

    +

    16.20. Linux kernel build system

    -

    15.20.1. vmlinux vs bzImage vs zImage vs Image

    +

    16.20.1. vmlinux vs bzImage vs zImage vs Image

    Between all archs on QEMU and gem5 we touch all of those kernel built output files.

    @@ -16581,7 +16808,7 @@ ps
    -

    15.21. Virtio

    +

    16.21. Virtio

    https://www.linux-kvm.org/page/Virtio

    @@ -16599,9 +16826,9 @@ ps
    -

    15.22. Kernel modules

    +

    16.22. Kernel modules

    -

    15.22.1. dump_regs

    +

    16.22.1. dump_regs

    The following kernel modules and Baremetal executables dump and disassemble various registers which cannot be observed from userland (usually "system registers", "control registers"):

    @@ -16679,9 +16906,86 @@ ps
    -

    16. Xen

    +

    17. FreeBSD

    +

    https://en.wikipedia.org/wiki/FreeBSD

    +
    +
    +

    Prebuilt on Ubuntu 20.04 worked: https://stackoverflow.com/questions/49656395/how-to-boot-freebsd-image-under-qemu/64027161#64027161

    +
    +
    +

    TODO minimal build + boot on QEMU example anywhere???

    +
    +
    +
    +
    +

    18. RTOS

    +
    +
    +

    https://en.wikipedia.org/wiki/Real-time_operating_system

    +
    +
    +

    18.1. Zephyr

    +
    +

    https://en.wikipedia.org/wiki/Zephyr_(operating_system)

    +
    +
    +

    Zephyr is an RTOS that has POSIX support. I think it works much like our Baremetal setup which uses Newlib and generates individual ELF files that contain both our C program’s code, and the Zephyr libraries.

    +
    +
    +

    TODO get a hello world working, and then consider further integration in this repo, e.g. being able to run all C userland content on it.

    +
    +
    +

    TODO: Cortex-A CPUs are not currently supported, there are some qemu_cortex_m0 boards, but can’t find a QEMU Cortex-A. There is an x86_64 qemu board, but we don’t currently have an x86 baremetal toolchain. For this reason, we won’t touch this further for now.

    +
    +
    +

    However, unlike Newlib, Zephyr must be setting up a simple pre-main runtime to be able to handle threads.

    +
    +
    +

    Failed attempt:

    +
    +
    +
    +
    # https://askubuntu.com/questions/952429/is-there-a-good-ppa-for-cmake-backports
+wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | sudo apt-key add -
+sudo apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic-rc main'
+sudo apt-get update
+sudo apt-get install cmake
+git clone https://github.com/zephyrproject-rtos/zephyr
+pip3 install --user -U west packaging
+cd zephyr
+git checkout v1.14.1
+west init zephyrproject
+west update
+export ZEPHYR_TOOLCHAIN_VARIANT=xtools
+export XTOOLS_TOOLCHAIN_PATH="$(pwd)/out/crosstool-ng/build/default/install/aarch64/bin/"
+source zephyr-env.sh
+west build -b qemu_aarch64 samples/hello_world
    +
    +
    +
    +

    The build system of that project is a bit excessive / wonky. You need an edge CMake not present in Ubuntu 18.04, which I don’t want to install right now, and it uses the weird custom west build tool frontend.

    +
    +
    +
    +

    18.2. ARM Mbed

    +
    +

    https://en.wikipedia.org/wiki/Mbed

    +
    +
    +

    TODO minimal setup to run it on QEMU? Possible?

    +
    +
    +
    +
    +
    +

    19. Xen

    +
    +
    +

    https://en.wikipedia.org/wiki/Xen

    +
    +

    TODO: get prototype working and then properly integrate:

    @@ -16756,7 +17060,7 @@ ps
    -

    17. U-Boot

    +

    20. U-Boot

    https://en.wikipedia.org/wiki/Das_U-Boot

    @@ -16770,10 +17074,31 @@ ps
    -

    18. QEMU

    +

    21. Emulators

    +
    +
    +

    https://en.wikipedia.org/wiki/Emulator

    +
    +
    + +
    +
    +
    +
    +

    22. QEMU

    -

    18.1. Introduction to QEMU

    +

    22.1. Introduction to QEMU

    QEMU is a system simulator: it simulates a CPU and devices such as interrupt handlers, timers, UART, screen, keyboard, etc.

    @@ -16803,7 +17128,7 @@ ps
    -

    18.2. Binary translation

    +

    22.2. Binary translation

    https://en.wikipedia.org/wiki/Binary_translation

    @@ -16812,7 +17137,7 @@ ps
    -

    18.3. Disk persistency

    +

    22.3. Disk persistency

    We disable disk persistency for both QEMU and gem5 by default, to prevent the emulator from putting the image in an unknown state.

    @@ -16867,7 +17192,7 @@ ps

    Disk persistency is useful to re-run shell commands from the history of a previous session with Ctrl-R, but we felt that the loss of determinism was not worth it.

    -

    18.3.1. gem5 disk persistency

    +

    22.3.1. gem5 disk persistency

    TODO how to make gem5 disk writes persistent?

    @@ -16897,7 +17222,7 @@ index 17498c42b..76b8b351d 100644
    -

    18.4. gem5 qcow2

    +

    22.4. gem5 qcow2

    qcow2 does not appear supported, there are not hits in the source tree, and there is a mention on Nate’s 2009 wishlist: http://gem5.org/Nate%27s_Wish_List

    @@ -16906,7 +17231,7 @@ index 17498c42b..76b8b351d 100644
    -

    18.5. Snapshot

    +

    22.5. Snapshot

    QEMU allows us to take snapshots at any time through the monitor.

    @@ -17004,7 +17329,7 @@ index 17498c42b..76b8b351d 100644

    Bibliography: https://stackoverflow.com/questions/40227651/does-qemu-emulator-have-checkpoint-function/48724371#48724371

    -

    18.5.1. Snapshot internals

    +

    22.5.1. Snapshot internals

    Snapshots are stored inside the .qcow2 images themselves.

    @@ -17053,7 +17378,7 @@ Format specific information:
    -

    18.6. Device models

    +

    22.6. Device models

    This section documents:

    @@ -17098,12 +17423,12 @@ Format specific information:
    -

    18.6.1. PCI

    +

    22.6.1. PCI

    Only tested in x86.

    -
    18.6.1.1. pci_min
    +
    22.6.1.1. pci_min

    PCI driver for our minimal pci_min.c QEMU fork device:

    @@ -17173,7 +17498,7 @@ lkmc_pci_min mmio_write addr = 4 val = 0 size = 4
    -
    18.6.1.2. QEMU edu PCI device
    +
    22.6.1.2. QEMU edu PCI device

    Small upstream educational PCI device:

    @@ -17243,7 +17568,7 @@ lkmc_pci_min mmio_write addr = 4 val = 0 size = 4
    -
    18.6.1.3. Manipulate PCI registers directly
    +
    22.6.1.3. Manipulate PCI registers directly

    In this section we will try to interact with PCI devices directly from userland without kernel modules.

    @@ -17389,7 +17714,7 @@ devmem 0xfeb54000 w 0x12345678
    -
    18.6.1.4. pciutils
    +
    22.6.1.4. pciutils

    There are two versions of setpci and lspci:

    @@ -17405,7 +17730,7 @@ devmem 0xfeb54000 w 0x12345678
    -
    18.6.1.5. Introduction to PCI
    +
    22.6.1.5. Introduction to PCI

    The PCI standard is non-free, obviously like everything in low level: https://pcisig.com/specifications but Google gives several illegal PDF hits :-)

    @@ -17465,7 +17790,7 @@ devmem 0xfeb54000 w 0x12345678
    -
    18.6.1.6. PCI BFD
    +
    22.6.1.6. PCI BFD

    lspci -k shows something like:

    @@ -17519,7 +17844,7 @@ devmem 0xfeb54000 w 0x12345678
    -
    18.6.1.7. PCI BAR
    +
    22.6.1.7. PCI BAR

    https://stackoverflow.com/questions/30190050/what-is-base-address-register-bar-in-pcie/44716618#44716618

    @@ -17561,7 +17886,7 @@ pci_register_bar(pdev, 0, PCI_BASE_ADDRESS_SPACE_MEMORY, &edu->mmio);
    -

    18.6.2. GPIO

    +

    22.6.2. GPIO

    TODO: broken. Was working before we moved arm from -M versatilepb to -M virt around af210a76711b7fa4554dcc2abd0ddacfc810dfd4. Either make it work on -M virt if that is possible, or document precisely how to make it work with versatilepb, or hopefully vexpress which is newer.

    @@ -17604,7 +17929,7 @@ pci_register_bar(pdev, 0, PCI_BASE_ADDRESS_SPACE_MEMORY, &edu->mmio);
    -

    18.6.3. LEDs

    +

    22.6.3. LEDs

    TODO: broken when arm moved to -M virt, same as GPIO.

    @@ -17676,7 +18001,7 @@ echo 255 >brightness
    -

    18.6.4. platform_device

    +

    22.6.4. platform_device

    Minimal platform device example coded into the -M versatilepb SoC of our QEMU fork.

    @@ -17754,7 +18079,7 @@ insmod platform_device.ko
    -

    18.6.5. gem5 educational hardware models

    +

    22.6.5. gem5 educational hardware models

    TODO get some working!

    @@ -17764,7 +18089,7 @@ insmod platform_device.ko
    -

    18.7. QEMU monitor

    +

    22.7. QEMU monitor

    The QEMU monitor is a magic terminal that allows you to send text commands to the QEMU VM itself: https://en.wikibooks.org/wiki/QEMU/Monitor

    @@ -17884,7 +18209,7 @@ insmod platform_device.ko
    -

    18.7.1. QEMU monitor from guest

    +

    22.7.1. QEMU monitor from guest

    Peter Maydell said potentially not possible nicely as of August 2018: https://stackoverflow.com/questions/51747744/how-to-run-a-qemu-monitor-command-from-inside-the-guest/51764110#51764110

    @@ -17901,7 +18226,7 @@ insmod platform_device.ko
    -

    18.7.2. QEMU monitor from GDB

    +

    22.7.2. QEMU monitor from GDB

    When doing GDB step debug it is possible to send QEMU monitor commands through the GDB monitor command, which saves you the trouble of opening yet another shell.

    @@ -17917,7 +18242,7 @@ monitor info qtree
    -

    18.8. Debug the emulator

    +

    22.8. Debug the emulator

    When you start hacking QEMU or gem5, it is useful to see what is going on inside the emulator themselves.

    @@ -17977,10 +18302,10 @@ run

    The build outputs are automatically stored in a different directories for optimized and debug builds, which prevents debug files from overwriting opt ones. Therefore, --gem5-build-id is not required.

    -

    The price to pay for debuggability is high however: a Linux kernel boot was about 3x slower in QEMU and 14 times slower in gem5 debug compared to opt, see benchmarks at: Section 30.2.1, “Benchmark Linux kernel boot”.

    +

    The price to pay for debuggability is high however: a Linux kernel boot was about 3x slower in QEMU and 14 times slower in gem5 debug compared to opt, see benchmarks at: Section 34.2.1, “Benchmark Linux kernel boot”.

    -

    Similar slowdowns can be observed at: Section 30.2.2, “Benchmark emulators on userland executables”.

    +

    Similar slowdowns can be observed at: Section 34.2.2, “Benchmark emulators on userland executables”.

    When in QEMU text mode, using --debug-vm makes Ctrl-C not get passed to the QEMU guest anymore: it is instead captured by GDB itself, so allow breaking. So e.g. you won’t be able to easily quit from a guest program like:

    @@ -17997,7 +18322,7 @@ run

    You can still send key presses to QEMU however even without the mouse capture, just either click on the title bar, or alt tab to give it focus.

    -

    18.8.1. Reverse debug the emulator

    +

    22.8.1. Reverse debug the emulator

    While step debugging any complex program, you always end up feeling the need to step in reverse to reach the last call to some function that was called before the failure point, in order to trace back the problem to the actual bug source.

    @@ -18086,7 +18411,7 @@ reverse-next
    -

    18.8.2. Debug gem5 Python scripts

    +

    22.8.2. Debug gem5 Python scripts

    Start pdb at the first instruction:

    @@ -18120,7 +18445,7 @@ reverse-next
    -

    18.9. Tracing

    +

    22.9. Tracing

    QEMU can log several different events.

    @@ -18211,7 +18536,7 @@ Call Trace:
    -

    18.9.1. QEMU -d tracing

    +

    22.9.1. QEMU -d tracing

    QEMU also has a second trace mechanism in addition to -trace, find out the events with:

    @@ -18252,7 +18577,7 @@ IN:
    -

    18.9.2. QEMU trace register values

    +

    22.9.2. QEMU trace register values

    TODO: is it possible to show the register values for each instruction?

    @@ -18282,11 +18607,11 @@ IN:

    PANDA can list memory addresses, so I bet it can also decode the instructions: https://github.com/panda-re/panda/blob/883c85fa35f35e84a323ed3d464ff40030f06bd6/panda/docs/LINE_Censorship.md I wonder why they don’t just upstream those things to QEMU’s tracing: https://github.com/panda-re/panda/issues/290

    -

    gem5 can do it as shown at: Section 18.9.8, “gem5 tracing”.

    +

    gem5 can do it as shown at: Section 22.9.8, “gem5 tracing”.

    -

    18.9.3. QEMU trace memory accesses

    +

    22.9.3. QEMU trace memory accesses

    Not possible apparently, not even with the memory_region_ops_read and memory_region_ops_write trace events, Peter comments https://lists.gnu.org/archive/html/qemu-devel/2015-06/msg07482.html

    @@ -18305,7 +18630,7 @@ of guest operations.

    -

    18.9.4. Trace source lines

    +

    22.9.4. Trace source lines

    We can further use Binutils' addr2line to get the line that corresponds to each address:

    @@ -18361,7 +18686,7 @@ less "$(./getvar --arch x86_64 run_dir)/trace-lines.txt"
    -

    18.9.5. QEMU record and replay

    +

    22.9.5. QEMU record and replay

    QEMU runs, unlike gem5, are not deterministic by default, however it does support a record and replay mechanism that allows you to replay a previous run deterministically.

    @@ -18468,7 +18793,7 @@ less "$(./getvar --arch x86_64 run_dir)/trace-lines.txt"

    Solved on unmerged c42634d8e3428cfa60672c3ba89cabefc720cde9 from https://github.com/ispras/qemu/tree/rr-180725

    -
    18.9.5.1. QEMU reverse debugging
    +
    22.9.5.1. QEMU reverse debugging

    TODO get working.

    @@ -18507,7 +18832,7 @@ reverse-continue
    -

    18.9.6. QEMU trace multicore

    +

    22.9.6. QEMU trace multicore

    TODO: is there any way to distinguish which instruction runs on each core? Doing:

    @@ -18522,13 +18847,13 @@ reverse-continue
    -

    18.9.7. QEMU get guest instruction count

    +

    22.9.7. QEMU get guest instruction count

    TODO: https://stackoverflow.com/questions/58766571/how-to-count-the-number-of-guest-instructions-qemu-executed-from-the-beginning-t

    -

    18.9.8. gem5 tracing

    +

    22.9.8. gem5 tracing

    gem5 provides also provides a tracing mechanism documented at: http://www.gem5.org/Trace_Based_Debugging:

    @@ -18639,7 +18964,7 @@ less "$(./getvar --arch aarch64 run_dir)/trace-lines.txt"

    TODO: 7452d399290c9c1fc6366cdad129ef442f323564 ./trace2line this is too slow and takes hours. QEMU’s processing of 170k events takes 7 seconds. gem5’s processing is analogous, but there are 140M events, so it should take 7000 seconds ~ 2 hours which seems consistent with what I observe, so maybe there is no way to speed this up…​ The workaround is to just use gem5’s ExecSymbol to get function granularity, and then GDB individually if line detail is needed?

    -
    18.9.8.1. gem5 trace internals
    +
    22.9.8.1. gem5 trace internals

    gem5 traces are generated from DPRINTF(<trace-id> calls scattered throughout the code, except for ExecAll instruction traces, which uses Debug::ExecEnable directly..

    @@ -18676,7 +19001,7 @@ extern SimpleFlag ExecEnable;
    -
    18.9.8.2. gem5 ExecAll trace format
    +
    22.9.8.2. gem5 ExecAll trace format

    This debug flag traces all instructions.

    @@ -18714,7 +19039,7 @@ extern SimpleFlag ExecEnable;

    25007500: time count in some unit. Note how the microops execute at further timestamps.

  • -

    system.cpu: distinguishes between CPUs when there are more than one. For example, running Section 28.10.3, “ARM baremetal multicore” with two cores produces system.cpu0 and system.cpu1

    +

    system.cpu: distinguishes between CPUs when there are more than one. For example, running Section 32.10.3, “ARM baremetal multicore” with two cores produces system.cpu0 and system.cpu1

  • T0: thread number. TODO: hyperthread? How to play with it?

    @@ -18759,7 +19084,7 @@ extern SimpleFlag ExecEnable;
    • -
    18.9.8.3. gem5 Registers trace format
    +
    22.9.8.3. gem5 Registers trace format

    This flag shows a more detailed register usage than gem5 ExecAll trace format.

    @@ -18814,13 +19139,13 @@ add x1, x0, 2
    -
    18.9.8.4. gem5 TARMAC traces
    +
    22.9.8.4. gem5 TARMAC traces

    https://stackoverflow.com/questions/54882466/how-to-use-the-tarmac-tracer-with-gem5

    -
    18.9.8.5. gem5 tracing internals
    +
    22.9.8.5. gem5 tracing internals

    As of gem5 16eeee5356585441a49d05c78abc328ef09f7ace the default tracer is ExeTracer. It is set at:

    @@ -18893,7 +19218,7 @@ src/arch/x86/nativetrace.hh:41:class X86NativeTrace : public NativeTrace
    -

    18.10. QEMU GUI is unresponsive

    +

    22.10. QEMU GUI is unresponsive

    Sometimes in Ubuntu 14.04, after the QEMU SDL GUI starts, it does not get updated after keyboard strokes, and there are artifacts like disappearing text.

    @@ -18917,7 +19242,7 @@ root
    -

    19. gem5

    +

    23. gem5

    Getting started at: Section 1.4, “gem5 Buildroot setup”.

    @@ -18936,7 +19261,7 @@ root
    -

    19.1. gem5 vs QEMU

    +

    23.1. gem5 vs QEMU

    -

    19.2. gem5 run benchmark

    +

    23.2. gem5 run benchmark

    OK, this is why we used gem5 in the first place, performance measurements!

    @@ -19211,7 +19536,7 @@ cat out/gem5-bench-dhrystone.txt
    -

    but the problem is that this method does not allow to easily run a different script without running the boot again. The ./gem5.sh script works around that by using m5 readfile as explained further at: Section 19.6.3, “gem5 checkpoint restore and run a different script”.

    +

    but the problem is that this method does not allow to easily run a different script without running the boot again. The ./gem5.sh script works around that by using m5 readfile as explained further at: Section 23.6.3, “gem5 checkpoint restore and run a different script”.

    Now you can play a fun little game with your friends:

    @@ -19246,7 +19571,7 @@ cat out/gem5-bench-dhrystone.txt

    To find out why your program is slow, a good first step is to have a look at the gem5 m5out/stats.txt file.

    -

    19.2.1. Skip extra benchmark instructions

    +

    23.2.1. Skip extra benchmark instructions

    A few imperfections of our benchmarking method are:

    @@ -19282,7 +19607,7 @@ cat out/gem5-bench-dhrystone.txt
    -

    19.3. gem5 system parameters

    +

    23.3. gem5 system parameters

    Besides optimizing a program for a given CPU setup, chip developers can also do the inverse, and optimize the chip for a given benchmark!

    @@ -19290,7 +19615,7 @@ cat out/gem5-bench-dhrystone.txt

    The rabbit hole is likely deep, but let’s scratch a bit of the surface.

    -

    19.3.1. Number of cores

    +

    23.3.1. Number of cores

    ./run --arch arm --cpus 2 --emulator gem5
    @@ -19337,7 +19662,7 @@ getconf _NPROCESSORS_CONF
    -
    19.3.1.1. QEMU user mode multithreading
    +
    23.3.1.1. QEMU user mode multithreading

    User mode simulation QEMU v4.0.0 always shows the number of cores of the host, presumably because the thread switching uses host threads directly which would make that harder to implement.

    @@ -19374,7 +19699,7 @@ ps Haux | grep qemu | wc
    -
    19.3.1.2. gem5 ARM full system with more than 8 cores
    +
    23.3.1.2. gem5 ARM full system with more than 8 cores

    https://stackoverflow.com/questions/50248067/how-to-run-a-gem5-arm-aarch64-full-system-simulation-with-fs-py-with-more-than-8

    @@ -19416,7 +19741,7 @@ ps Haux | grep qemu | wc
    -

    19.3.2. gem5 cache size

    +

    23.3.2. gem5 cache size

    https://stackoverflow.com/questions/49624061/how-to-run-gem5-simulator-in-fs-mode-without-cache/49634544#49634544

    @@ -19603,12 +19928,12 @@ instructions 91738770
    -

    19.3.3. gem5 DRAM model

    +

    23.3.3. gem5 DRAM model

    Some info at: TimingSimpleCPU analysis #1 but highly TODO :-)

    -
    19.3.3.1. gem5 memory latency
    +
    23.3.3.1. gem5 memory latency

    TODO These look promising:

    @@ -19649,7 +19974,7 @@ instructions 91738770

    we have no caches, each instruction is fetched from memory

  • -

    each loop contains 11 instructions as shown at Section 32.2, “C busy loop”

    +

    each loop contains 11 instructions as shown at Section 35.2, “C busy loop”

  • and supposing that the loop dominated executable pre/post main, which we know is true since as shown in Benchmark emulators on userland executables an empty dynamically linked C program only as about 100k instructions, while our loop runs 1000000 * 11 = 12M.

    @@ -19677,7 +20002,7 @@ instructions 91738770
    • -
    19.3.3.2. Memory size
    +
    23.3.3.2. Memory size

    Can be set across emulators with:

    @@ -19781,7 +20106,7 @@ get_avphys_pages() * sysconf(_SC_PAGESIZE) = 0x1D178000
    -
    19.3.3.3. gem5 DRAM setup
    +
    23.3.3.3. gem5 DRAM setup

    This can be explored pretty well from gem5 config.ini.

    @@ -19839,7 +20164,7 @@ get_avphys_pages() * sysconf(_SC_PAGESIZE) = 0x1D178000
    -

    19.3.4. gem5 disk and network latency

    +

    23.3.4. gem5 disk and network latency

    TODO These look promising:

    @@ -19854,7 +20179,7 @@ get_avphys_pages() * sysconf(_SC_PAGESIZE) = 0x1D178000
    -

    19.3.5. gem5 clock frequency

    +

    23.3.5. gem5 clock frequency

    As of gem5 872cb227fdc0b4d60acc7840889d567a6936b6e1 defaults to 2GHz for fs.py:

    @@ -19939,7 +20264,7 @@ hello
    -

    19.4. gem5 kernel command line parameters

    +

    23.4. gem5 kernel command line parameters

    Analogous to QEMU:

    @@ -19972,9 +20297,9 @@ hello
    -

    19.5. gem5 GDB step debug

    +

    23.5. gem5 GDB step debug

    -

    19.5.1. gem5 GDB step debug kernel

    +

    23.5.1. gem5 GDB step debug kernel

    Analogous to QEMU, on the first shell:

    @@ -20007,7 +20332,7 @@ hello
    -

    19.5.2. gem5 GDB step debug userland process

    +

    23.5.2. gem5 GDB step debug userland process

    We are unable to use gdbserver because of networking as mentioned at: Section 14.3.1.3, “gem5 host to guest networking”

    @@ -20042,7 +20367,7 @@ hello
    -

    19.5.3. gem5 GDB step debug secondary cores

    +

    23.5.3. gem5 GDB step debug secondary cores

    gem5’s secondary core GDB setup is a hack and spawns one gdbserver for each core in separate ports, e.g. 7000, 7001, etc.

    @@ -20063,7 +20388,7 @@ hello
    -

    19.6. gem5 checkpoint

    +

    23.6. gem5 checkpoint

    Analogous to QEMU’s Snapshot, but better since it can be started from inside the guest, so we can easily checkpoint after a specific guest event, e.g. just before init is done.

    @@ -20151,7 +20476,7 @@ m5 checkpoint

    since boot has already happened, and the parameters are already in the RAM of the snapshot.

    -

    19.6.1. gem5 checkpoint userland minimal example

    +

    23.6.1. gem5 checkpoint userland minimal example

    In order to debug checkpoint restore bugs, this minimal setup using userland/freestanding/gem5_checkpoint.S can be handy:

    @@ -20205,7 +20530,7 @@ Exiting @ tick 84500 because m5_exit instruction encountered
    -

    19.6.2. gem5 checkpoint internals

    +

    23.6.2. gem5 checkpoint internals

    A quick way to get a gem5 syscall emulation mode or full system checkpoint to observe is:

    @@ -20256,7 +20581,7 @@ prvEvalTick=0
    -

    19.6.3. gem5 checkpoint restore and run a different script

    +

    23.6.3. gem5 checkpoint restore and run a different script

    You want to automate running several tests from a single pristine post-boot state.

    @@ -20404,7 +20729,7 @@ expect eof
    -

    19.6.4. gem5 restore checkpoint with a different CPU

    +

    23.6.4. gem5 restore checkpoint with a different CPU

    gem5 can switch to a different CPU model when restoring a checkpoint.

    @@ -20519,7 +20844,7 @@ cat "$(./getvar --arch aarch64 --emulator gem5 trace_txt_file)"
    -
    19.6.4.1. gem5 fast forward
    +
    23.6.4.1. gem5 fast forward

    Besides switching CPUs after a checkpoint restore, fs.py also has the --fast-forward option to automatically run the script from the start on a less detailed CPU, and switch to a more detailed CPU at a given tick.

    @@ -20645,7 +20970,7 @@ FullO3CPU: Ticking main, FullO3CPU.
    -

    19.6.5. gem5 checkpoint upgrader

    +

    23.6.5. gem5 checkpoint upgrader

    The in-tree util/cpt_upgrader.py is a tool to upgrade checkpoints taken from an older version of gem5 to be compatible with the newest version, so you can update gem5 without having to re-run the simulation that generated the checkpoints.

    @@ -20681,7 +21006,7 @@ version_tags=arm-ccregs arm-contextidr-el2 arm-gem5-gic-ext ...
    -

    19.7. Pass extra options to gem5

    +

    23.7. Pass extra options to gem5

    Remember that in the gem5 command line, we can either pass options to the script being run as in:

    @@ -20738,7 +21063,7 @@ version_tags=arm-ccregs arm-contextidr-el2 arm-gem5-gic-ext ...
    -

    19.8. m5ops

    +

    23.8. m5ops

    m5ops are magic instructions which lead gem5 to do magic things, like quitting or dumping stats.

    @@ -20778,7 +21103,7 @@ version_tags=arm-ccregs arm-contextidr-el2 arm-gem5-gic-ext ...
    -

    19.8.1. gem5 m5 executable

    +

    23.8.1. gem5 m5 executable

    m5 is a guest command line utility that is installed and run on the guest, that serves as a CLI front-end for the m5ops

    @@ -20808,7 +21133,7 @@ version_tags=arm-ccregs arm-contextidr-el2 arm-gem5-gic-ext ...

    This can be a good test m5ops since it executes very quickly.

    -
    19.8.1.1. m5 exit
    +
    23.8.1.1. m5 exit

    End the simulation.

    @@ -20817,13 +21142,13 @@ version_tags=arm-ccregs arm-contextidr-el2 arm-gem5-gic-ext ...
    -
    19.8.1.2. m5 dumpstats
    +
    23.8.1.2. m5 dumpstats

    Makes gem5 dump one more statistics entry to the gem5 m5out/stats.txt file.

    -
    19.8.1.3. m5 fail
    +
    23.8.1.3. m5 fail

    End the simulation with a failure exit event:

    @@ -20862,7 +21187,7 @@ version_tags=arm-ccregs arm-contextidr-el2 arm-gem5-gic-ext ...
    -
    19.8.1.4. m5 writefile
    +
    23.8.1.4. m5 writefile

    Send a guest file to the host. 9P is a more advanced alternative.

    @@ -20893,7 +21218,7 @@ m5 writefile myfileguest myfilehost
    -
    19.8.1.5. m5 readfile
    +
    23.8.1.5. m5 readfile

    Read a host file pointed to by the fs.py --script option to stdout.

    @@ -20921,7 +21246,7 @@ m5 writefile myfileguest myfilehost
    -
    19.8.1.6. m5 initparam
    +
    23.8.1.6. m5 initparam

    Ermm, just another m5 readfile that only takes integers and only from CLI options? Is this software so redundant?

    @@ -20947,7 +21272,7 @@ m5 writefile myfileguest myfilehost
    -
    19.8.1.7. m5 execfile
    +
    23.8.1.7. m5 execfile

    Trivial combination of m5 readfile + execute the script.

    @@ -20982,7 +21307,7 @@ m5 execfile
    -

    19.8.2. m5ops instructions

    +

    23.8.2. m5ops instructions

    gem5 allocates some magic instructions on unused instruction encodings for convenient guest instrumentation.

    @@ -21075,7 +21400,7 @@ m5 execfile
    -
    19.8.2.1. m5ops instructions interface
    +
    23.8.2.1. m5ops instructions interface

    Let’s study how the gem5 m5 executable uses them:

    @@ -21189,7 +21514,7 @@ m5_fail(ints[1], ints[0]);
    -
    19.8.2.2. m5op annotations
    +
    23.8.2.2. m5op annotations

    include/gem5/asm/generic/m5ops.h also describes some annotation instructions.

    @@ -21200,7 +21525,7 @@ m5_fail(ints[1], ints[0]);
    -

    19.9. gem5 arm Linux kernel patches

    +

    23.9. gem5 arm Linux kernel patches

    https://gem5.googlesource.com/arm/linux/ contains an ARM Linux kernel forks with a few gem5 specific Linux kernel patches on top of mainline created by ARM Holdings on top of a few upstream kernel releases.

    @@ -21280,7 +21605,7 @@ git -C "$(./getvar linux_source_dir)" checkout -

    drm: Add component-aware simple encoder allows you to see images through VNC, see: Section 13.3, “gem5 graphic mode”

  • -

    gem5: Add support for gem5’s extended GIC mode adds support for more than 8 cores, see: Section 19.3.1.2, “gem5 ARM full system with more than 8 cores”

    +

    gem5: Add support for gem5’s extended GIC mode adds support for more than 8 cores, see: Section 23.3.1.2, “gem5 ARM full system with more than 8 cores”

    • @@ -21288,7 +21613,7 @@ git -C "$(./getvar linux_source_dir)" checkout -

    Tested on 649d06d6758cefd080d04dc47fd6a5a26a620874 + 1.

    -

    19.9.1. gem5 arm Linux kernel patches boot speedup

    +

    23.9.1. gem5 arm Linux kernel patches boot speedup

    We have observed that with the kernel patches, boot is 2x faster, falling from 1m40s to 50s.

    @@ -21306,7 +21631,7 @@ git -C "$(./getvar linux_source_dir)" checkout -
    -

    19.10. m5out directory

    +

    23.10. m5out directory

    When you run gem5, it generates an m5out directory at:

    @@ -21322,7 +21647,7 @@ git -C "$(./getvar linux_source_dir)" checkout -

    The files in that directory contains some very important information about the run, and you should become familiar with every one of them.

    -

    19.10.1. gem5 m5out/system.terminal file

    +

    23.10.1. gem5 m5out/system.terminal file

    Contains UART output, both from the Linux kernel or from the baremetal system.

    @@ -21331,7 +21656,7 @@ git -C "$(./getvar linux_source_dir)" checkout -
    -

    19.10.2. gem5 m5out/system.workload.dmesg file

    +

    23.10.2. gem5 m5out/system.workload.dmesg file

    This file used to be called just m5out/system.dmesg, but the name was changed after the workload refactorings of March 2020.

    @@ -21405,7 +21730,7 @@ index f296d89be757..3e79916322c2 100644
    -

    19.10.3. gem5 m5out/stats.txt file

    +

    23.10.3. gem5 m5out/stats.txt file

    This file contains important statistics about the run:

    @@ -21504,7 +21829,7 @@ system.cpu.dtb.inst_hits

    and after that the file size went down to 21KB.

    -
    19.10.3.1. gem5 HDF5 statistics
    +
    23.10.3.1. gem5 HDF5 statistics

    We can make gem5 dump statistics in the HDF5 format by adding the magic h5:// prefix to the file name as in:

    @@ -21554,7 +21879,7 @@ system.cpu.dtb.inst_hits
    -
    19.10.3.2. gem5 only dump selected stats
    +
    23.10.3.2. gem5 only dump selected stats

    https://stackoverflow.com/questions/52014953/how-to-dump-only-a-single-or-certain-selected-stats-in-gem5

    @@ -21566,7 +21891,7 @@ system.cpu.dtb.inst_hits
    -
    19.10.3.3. Meaning of each gem5 stat
    +
    23.10.3.3. Meaning of each gem5 stat

    Well, run minimal examples, and reverse engineer them up!

    @@ -21624,7 +21949,7 @@ sim_ops 6 # Number of ops (including micro ops) simulated
    -
    19.10.3.4. gem5 stats internals
    +
    23.10.3.4. gem5 stats internals

    This describes the internals of the gem5 m5out/stats.txt file.

    @@ -21698,7 +22023,7 @@ Text::end()
    -

    19.10.4. gem5 config.ini

    +

    23.10.4. gem5 config.ini

    The m5out/config.ini file, contains a very good high level description of the system:

    @@ -21771,7 +22096,7 @@ clock=500

    Modifying the config.ini file manually does nothing since it gets overwritten every time.

    -
    19.10.4.1. gem5 config.dot
    +
    23.10.4.1. gem5 config.dot

    The m5out/config.dot file contains a graphviz .dot file that provides a simplified graphical view of a subset of the gem5 config.ini.

    @@ -21852,7 +22177,7 @@ xdg-open "$(./getvar --arch arm --emulator gem5 m5out_dir)/config.dot.svg"
    -

    19.11. m5term

    +

    23.11. m5term

    We use the m5term in-tree executable to connect to the terminal instead of a direct telnet.

    @@ -21877,7 +22202,7 @@ xdg-open "$(./getvar --arch arm --emulator gem5 m5out_dir)/config.dot.svg"
    -

    19.12. gem5 Python scripts without rebuild

    +

    23.12. gem5 Python scripts without rebuild

    We have made a crazy setup that allows you to just cd into submodules/gem5, and edit Python scripts directly there.

    @@ -21911,7 +22236,7 @@ xdg-open "$(./getvar --arch arm --emulator gem5 m5out_dir)/config.dot.svg"
    -

    19.13. gem5 fs_bigLITTLE

    +

    23.13. gem5 fs_bigLITTLE

    By default, we use configs/example/fs.py script.

    @@ -21960,7 +22285,7 @@ xdg-open "$(./getvar --arch arm --emulator gem5 m5out_dir)/config.dot.svg"
    -

    19.14. gem5 in-tree tests

    +

    23.14. gem5 in-tree tests

    https://stackoverflow.com/questions/52279971/how-to-run-the-gem5-unit-tests

    @@ -21971,7 +22296,7 @@ xdg-open "$(./getvar --arch arm --emulator gem5 m5out_dir)/config.dot.svg"

    But can the people from the project be convinced of that?

    -

    19.14.1. gem5 unit tests

    +

    23.14.1. gem5 unit tests

    These are just very small GTest tests that test a single class in isolation, they don’t run any executables.

    @@ -22026,7 +22351,7 @@ xdg-open "$(./getvar --arch arm --emulator gem5 m5out_dir)/config.dot.svg"
    -

    19.14.2. gem5 regression tests

    +

    23.14.2. gem5 regression tests

    This section is about running the gem5 in-tree tests.

    @@ -22075,7 +22400,7 @@ xdg-open "$(./getvar --arch arm --emulator gem5 m5out_dir)/config.dot.svg"
    -

    19.15. gem5 simulate() limit reached

    +

    23.15. gem5 simulate() limit reached

    This error happens when the following instruction limits are reached:

    @@ -22211,24 +22536,24 @@ Exiting @ tick 18446744073709551615 because simulate() limit reached
    -

    19.16. gem5 build options

    +

    23.16. gem5 build options

    In order to use different build options, you might also want to use gem5 build variants to keep the build outputs separate from one another.

    -

    19.16.1. gem5 debug build

    +

    23.16.1. gem5 debug build

    -

    How to use it in LKMC: Section 18.8, “Debug the emulator”.

    +

    How to use it in LKMC: Section 22.8, “Debug the emulator”.

    If you build gem5 with scons build/ARM/gem5.debug, then that is a .debug build.

    -

    It relates to the more common .opt build just as explained at Section 18.8, “Debug the emulator”: both .opt and .debug have -g, but .opt uses -O2 while .debug uses -O0.

    +

    It relates to the more common .opt build just as explained at Section 22.8, “Debug the emulator”: both .opt and .debug have -g, but .opt uses -O2 while .debug uses -O0.

    -

    19.16.2. gem5 fast build

    +

    23.16.2. gem5 fast build

    ./build-gem5 --gem5-build-type fast
    @@ -22246,13 +22571,13 @@ Exiting @ tick 18446744073709551615 because simulate() limit reached
    -

    19.16.3. gem5 prof and perf builds

    +

    23.16.3. gem5 prof and perf builds

    Profiling builds as of 3cea7d9ce49bda49c50e756339ff1287fd55df77 both use: -g -O3 and disable asserts and logging like the gem5 fast build and:

    @@ -22280,7 +22605,7 @@ gprof "$(./getvar --arch aarch64 gem5_executable)" > tmp.gprof
    -

    19.16.4. gem5 clang build

    +

    23.16.4. gem5 clang build

    TODO test properly, benchmark vs GCC.

    @@ -22293,7 +22618,7 @@ gprof "$(./getvar --arch aarch64 gem5_executable)" > tmp.gprof
    -

    19.16.5. gem5 sanitation build

    +

    23.16.5. gem5 sanitation build

    If there gem5 appears to have a C++ undefined behaviour bug, which is often very difficult to track down, you can try to build it with the following extra SCons options:

    @@ -22367,7 +22692,7 @@ Indirect leak of 1346 byte(s) in 2 object(s) allocated from:
    -

    19.16.6. gem5 Ruby build

    +

    23.16.6. gem5 Ruby build

    gem5 has two types of memory system:

    @@ -22503,7 +22828,7 @@ cat "$(./getvar --arch aarch64 --emulator gem5 trace_txt_file)"

    Tested in gem5 d7d9bc240615625141cd6feddbadd392457e49eb.

    -
    19.16.6.1. gem5 Ruby MI_example protocol
    +
    23.16.6.1. gem5 Ruby MI_example protocol

    This is the simplest of all protocols, and therefore the first one you should study to learn how Ruby works.

    @@ -22539,7 +22864,7 @@ cat "$(./getvar --arch aarch64 --emulator gem5 trace_txt_file)"
    -
    19.16.6.2. gem5 crossbar interconnect
    +
    23.16.6.2. gem5 crossbar interconnect

    Crossbar or XBar in the code, is the default CPU interconnect that gets used by fs.py if --ruby is not given.

    @@ -22590,7 +22915,7 @@ class SystemXBar(CoherentXBar):
    -

    19.16.7. gem5 Python 3 build

    +

    23.16.7. gem5 Python 3 build

    Python 3 support was mostly added in 2019 Q3 at arounda347a1a68b8a6e370334be3a1d2d66675891e0f1 but remained buggy for some time afterwards.

    @@ -22608,7 +22933,7 @@ class SystemXBar(CoherentXBar):
    -

    19.17. gem5 CPU types

    +

    23.17. gem5 CPU types

    gem5 has a few in tree CPU models for different purposes.

    @@ -22691,9 +23016,9 @@ class SystemXBar(CoherentXBar):

    From this we see that there are basically only 4 C++ CPU models in gem5: Atomic, Timing, Minor and O3. All others are basically parametrizations of those base types.

    -

    19.17.1. List of gem5 CPU types

    +

    23.17.1. List of gem5 CPU types

    -
    19.17.1.1. gem5 BaseSimpleCPU
    +
    23.17.1.1. gem5 BaseSimpleCPU

    Simple abstract CPU without a pipeline.

    @@ -22714,7 +23039,7 @@ class SystemXBar(CoherentXBar):
    -
    19.17.1.1.1. gem5 AtomicSimpleCPU
    +
    23.17.1.1.1. gem5 AtomicSimpleCPU

    AtomicSimpleCPU: the default one. Memory accesses happen instantaneously. The fastest simulation except for KVM, but not realistic at all.

    @@ -22723,7 +23048,7 @@ class SystemXBar(CoherentXBar):
    -
    19.17.1.1.2. gem5 TimingSimpleCPU
    +
    23.17.1.1.2. gem5 TimingSimpleCPU

    TimingSimpleCPU: memory accesses are realistic, but the CPU has no pipeline. The simulation is faster than detailed models, but slower than AtomicSimpleCPU.

    @@ -22739,7 +23064,7 @@ class SystemXBar(CoherentXBar):
    -
    19.17.1.2. gem5 MinorCPU
    +
    23.17.1.2. gem5 MinorCPU

    Generic in-order superscalar core.

    @@ -22805,7 +23130,7 @@ class SystemXBar(CoherentXBar):
    -
    19.17.1.3. gem5 DerivO3CPU
    +
    23.17.1.3. gem5 DerivO3CPU

    Generic out-of-order core. "O3" Stands for "Out Of Order"!

    @@ -22865,7 +23190,7 @@ wbWidth=8
    -
    19.17.1.3.1. gem5 DerivO3CPU pipeline stages
    +
    23.17.1.3.1. gem5 DerivO3CPU pipeline stages
    -
    19.17.1.3.2. gem5 util/o3-pipeview.py O3 pipeline viewer
    +
    23.17.1.3.2. gem5 util/o3-pipeview.py O3 pipeline viewer

    Mentioned at: http://www.m5sim.org/Visualization

    @@ -22918,7 +23243,7 @@ less o3pipeview.tmp.log
    -
    19.17.1.3.3. gem5 Konata O3 pipeline viewer
    +
    23.17.1.3.3. gem5 Konata O3 pipeline viewer

    https://github.com/shioyadan/Konata

    @@ -22938,7 +23263,7 @@ less o3pipeview.tmp.log
    -

    19.17.2. gem5 ARM RSK

    +

    23.17.2. gem5 ARM RSK

    https://github.com/arm-university/arm-gem5-rsk/blob/aa3b51b175a0f3b6e75c9c856092ae0c8f2a7cdc/gem5_rsk.pdf

    @@ -22948,7 +23273,7 @@ less o3pipeview.tmp.log
    -

    19.18. gem5 ARM platforms

    +

    23.18. gem5 ARM platforms

    The gem5 platform is selectable with the --machine option, which is named after the analogous QEMU -machine option, and which sets the --machine-type.

    @@ -22976,7 +23301,7 @@ less o3pipeview.tmp.log
    -

    19.19. gem5 upstream images

    +

    23.19. gem5 upstream images

    Present at:

    @@ -23030,7 +23355,7 @@ cd ..
    -

    19.20. gem5 bootloaders

    +

    23.20. gem5 bootloaders

    Certain ISAs like ARM have bootloaders that are automatically run before the main image to setup basic system state.

    @@ -23063,12 +23388,12 @@ cd ..
    -

    19.21. gem5 memory system

    +

    23.21. gem5 memory system

    Parent section: gem5 internals.

    -

    19.21.1. gem5 port system

    +

    23.21.1. gem5 port system

    The gem5 memory system is connected in a very flexible way through the port system.

    @@ -23079,7 +23404,7 @@ cd ..

    A Packet is the basic information unit that gets sent across ports.

    -
    19.21.1.1. gem5 functional vs atomic vs timing memory requests
    +
    23.21.1.1. gem5 functional vs atomic vs timing memory requests

    gem5 memory requests can be classified in the following broad categories:

    @@ -23289,7 +23614,7 @@ TimingSimpleCPU::finishTranslation(WholeTranslationState *state)

    Tested in gem5 b1623cb2087873f64197e503ab8894b5e4d4c7b4.

    -
    19.21.1.1.1. gem5 functional requests
    +
    23.21.1.1.1. gem5 functional requests

    As seen at gem5 functional vs atomic vs timing memory requests, functional requests are not used in common simulation, since the core must always go through caches.

    @@ -23336,9 +23661,9 @@ TimingSimpleCPU::finishTranslation(WholeTranslationState *state)
    -

    19.21.2. gem5 Packet vs Request

    +

    23.21.2. gem5 Packet vs Request

    -
    19.21.2.1. gem5 Packet
    +
    23.21.2.1. gem5 Packet

    Packet is what goes through ports: a single packet is sent out to the memory system, gets modified when it hits valid data, and then returns with the reply.

    @@ -23421,7 +23746,7 @@ Addr addr;
    -
    19.21.2.1.1. gem5 MemCmd
    +
    23.21.2.1.1. gem5 MemCmd

    Each gem5 Packet contains a MemCmd

    @@ -23507,7 +23832,7 @@ MemCmd::commandInfo[] =
    -
    19.21.2.2. gem5 Request
    +
    23.21.2.2. gem5 Request

    One good way to think about Request vs Packet could be "it is what the instruction definitions see", a bit like ExecContext vs ThreadContext.

    @@ -23574,7 +23899,7 @@ Addr _vaddr = MaxAddr;
    -
    19.21.2.2.1. gem5 Request in AtomicSimpleCPU
    +
    23.21.2.2.1. gem5 Request in AtomicSimpleCPU

    In AtomicSimpleCPU, a single packet of each type is kept for the entire CPU, e.g.:

    @@ -23635,7 +23960,7 @@ TLB::translateMmuOn(ThreadContext* tc, const RequestPtr &req, Mode mode,
    -
    19.21.2.2.2. gem5 Request in TimingSimpleCPU
    +
    23.21.2.2.2. gem5 Request in TimingSimpleCPU

    In TimingSimpleCPU, the request gets created per memory read:

    @@ -23670,7 +23995,7 @@ TimingSimpleCPU::initiateMemRead(Addr addr, unsigned size,
    -

    19.21.3. gem5 MSHR

    +

    23.21.3. gem5 MSHR

    Mentioned at: http://pages.cs.wisc.edu/~swilson/gem5-docs/gem5MemorySystem.html

    @@ -23732,7 +24057,7 @@ TimingSimpleCPU::initiateMemRead(Addr addr, unsigned size,
    -

    19.21.4. gem5 CommMonitor

    +

    23.21.4. gem5 CommMonitor

    You can place this SimObject in between two ports to get extra statistics about the packets that are going through.

    @@ -23777,9 +24102,32 @@ TimingSimpleCPU::initiateMemRead(Addr addr, unsigned size,

    One neat thing about this is that it is agnostic to the memory object type, so you don’t have to recode those statistics for every new type of object that operates on memory packets.

    +
    +

    23.21.5. gem5 SimpleMemory

    +
    +

    SimpleMemory is a highly simplified memory system. It can replace a more complex DRAM model if you use it e.g. as:

    +
    +
    +
    +
    ./run --emulator gem5 -- --mem-type SimpleMemory
    +
    +
    +
    +

    and it also gets used in certain system-y memories present in ARM systems by default e.g. Flash memory:

    +
    +
    +
    +
    [system.realview.flash0]
+type=SimpleMemory
    +
    +
    +
    +

    As of gem5 3ca404da175a66e0b958165ad75eb5f54cb5e772 LKMC 059a7ef9d9c378a6d1d327ae97d90b78183680b2 it did not provide any speedup to the Linux kernel boot according to a quick test.

    +
    +
    -

    19.22. gem5 internals

    +

    23.22. gem5 internals

    Internals under other sections:

    @@ -23800,7 +24148,7 @@ TimingSimpleCPU::initiateMemRead(Addr addr, unsigned size,
    -

    19.22.1. gem5 Eclipse configuration

    +

    23.22.1. gem5 Eclipse configuration

    https://stackoverflow.com/questions/61656709/how-to-setup-eclipse-ide-for-gem5-development

    @@ -23862,7 +24210,7 @@ TimingSimpleCPU::initiateMemRead(Addr addr, unsigned size,
    -

    19.22.2. gem5 Python C++ interaction

    +

    23.22.2. gem5 Python C++ interaction

    The interaction uses the Python C extension interface https://docs.python.org/2/extending/extending.html interface through the pybind11 helper library: https://github.com/pybind/pybind11

    @@ -24047,7 +24395,7 @@ static EmbeddedPyBind embed_obj("BadDevice", module_init, "BasicPioDevice");
    -

    19.22.3. gem5 entry point

    +

    23.22.3. gem5 entry point

    The main is at: src/sim/main.cc. It calls:

    @@ -24135,7 +24483,7 @@ exec filecode in scope

    Tested at gem5 b4879ae5b0b6644e6836b0881e4da05c64a6550d.

    -
    19.22.3.1. gem5 m5.objects module
    +
    23.22.3.1. gem5 m5.objects module

    All SimObjects seem to be automatically added to the m5.objects namespace, and this is done in a very convoluted way, let’s try to understand a bit:

    @@ -24300,7 +24648,7 @@ for source in PySource.all:
    -

    19.22.4. gem5 event queue

    +

    23.22.4. gem5 event queue

    gem5 is an event based simulator, and as such the event queue is of of the crucial elements in the system.

    @@ -24406,7 +24754,7 @@ b EventFunctionWrapper::process

    Then, once we had that, the most perfect thing ever would be to make the full event graph containing which events schedule which events!

    -
    19.22.4.1. gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis
    +
    23.22.4.1. gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis

    Let’s now analyze every single event on a minimal gem5 syscall emulation mode in the simplest CPU that we have:

    @@ -24542,7 +24890,7 @@ AtomicSimpleCPU::tick() at atomic.cc:757 0x55555907834c

    Tested in gem5 12c917de54145d2d50260035ba7fa614e25317a3.

    -
    19.22.4.1.1. AtomicSimpleCPU initial events
    +
    23.22.4.1.1. AtomicSimpleCPU initial events

    Let’s have a closer look at the initial magically scheduled events of the simulation.

    @@ -24761,7 +25109,7 @@ simulate() at simulate.cc:104 0x555559476d6f
    -
    19.22.4.1.2. AtomicSimpleCPU tick reschedule timing
    +
    23.22.4.1.2. AtomicSimpleCPU tick reschedule timing

    Inside AtomicSimpleCPU::tick() we saw previously that the reschedule happens at:

    @@ -24801,7 +25149,7 @@ clock=500
    -
    19.22.4.1.3. AtomicSimpleCPU memory access
    +
    23.22.4.1.3. AtomicSimpleCPU memory access

    It will be interesting to see how AtomicSimpleCPU makes memory access on GDB and to compare that with TimingSimpleCPU.

    @@ -24855,7 +25203,7 @@ clock=500
    -
    19.22.4.1.4. gem5 se.py page translation
    +
    23.22.4.1.4. gem5 se.py page translation

    Happens on EmulationPageTable, and seems to happen atomically without making any extra memory requests.

    @@ -24926,7 +25274,7 @@ Exiting @ tick 3500 because exiting with last active thread context
    -
    19.22.4.2. gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis
    +
    23.22.4.2. gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis

    Now, let’s move on to TimingSimpleCPU, which is just like AtomicSimpleCPU internally, but now the memory requests don’t actually finish immediately: gem5 CPU types!

    @@ -25207,7 +25555,7 @@ info: Entering event queue @ 0. Starting simulation...
    -
    19.22.4.2.1. TimingSimpleCPU analysis #0
    +
    23.22.4.2.1. TimingSimpleCPU analysis #0

    Schedules TimingSimpleCPU::fetch through:

    @@ -25252,7 +25600,7 @@ ArmLinuxProcess64::initState
    -
    19.22.4.2.2. TimingSimpleCPU analysis #1
    +
    23.22.4.2.2. TimingSimpleCPU analysis #1

    Backtrace:

    @@ -25383,7 +25731,7 @@ DRAMCtrl::Rank::startup(Tick ref_tick)
    -
    19.22.4.2.3. TimingSimpleCPU analysis #2
    +
    23.22.4.2.3. TimingSimpleCPU analysis #2

    This is just the startup of the second rank, see: TimingSimpleCPU analysis #1.

    @@ -25416,13 +25764,13 @@ DRAMCtrl::Rank::startup(Tick ref_tick)
    -
    19.22.4.2.4. TimingSimpleCPU analysis #3 and #4
    +
    23.22.4.2.4. TimingSimpleCPU analysis #3 and #4

    From the timing we know what that one is: the end of time exit event, like for AtomicSimpleCPU.

    -
    19.22.4.2.5. TimingSimpleCPU analysis #5
    +
    23.22.4.2.5. TimingSimpleCPU analysis #5

    Executes TimingSimpleCPU::fetch().

    @@ -25530,7 +25878,7 @@ DRAMCtrl::Rank::startup(Tick ref_tick)
    -
    19.22.4.2.6. TimingSimpleCPU analysis #6
    +
    23.22.4.2.6. TimingSimpleCPU analysis #6

    Schedules DRAMCtrl::processNextReqEvent through:

    @@ -25667,7 +26015,7 @@ TimingSimpleCPU::fetch
    -
    19.22.4.2.7. TimingSimpleCPU analysis #7
    +
    23.22.4.2.7. TimingSimpleCPU analysis #7

    Schedules BaseXBar::Layer::releaseLayer through:

    @@ -25693,13 +26041,13 @@ TimingSimpleCPU::fetch
    -
    19.22.4.2.8. TimingSimpleCPU analysis #8
    +
    23.22.4.2.8. TimingSimpleCPU analysis #8

    Executes DRAMCtrl::processNextReqEvent.

    -
    19.22.4.2.9. TimingSimpleCPU analysis #9
    +
    23.22.4.2.9. TimingSimpleCPU analysis #9

    Schedules DRAMCtrl::Rank::processActivateEvent through:

    @@ -25713,7 +26061,7 @@ DRAMCtrl::processNextReqEvent
    -
    19.22.4.2.10. TimingSimpleCPU analysis #10
    +
    23.22.4.2.10. TimingSimpleCPU analysis #10

    Schedules DRAMCtrl::processRespondEvent through:

    @@ -25725,7 +26073,7 @@ DRAMCtrl::processNextReqEvent
    -
    19.22.4.2.11. TimingSimpleCPU analysis #11
    +
    23.22.4.2.11. TimingSimpleCPU analysis #11

    Schedules DRAMCtrl::processNextReqEvent through:

    @@ -25737,7 +26085,7 @@ DRAMCtrl::processNextReqEvent
    -
    19.22.4.2.12. TimingSimpleCPU analysis #12
    +
    23.22.4.2.12. TimingSimpleCPU analysis #12

    Executes DRAMCtrl::Rank::processActivateEvent.

    @@ -25746,7 +26094,7 @@ DRAMCtrl::processNextReqEvent
    -
    19.22.4.2.13. TimingSimpleCPU analysis #13
    +
    23.22.4.2.13. TimingSimpleCPU analysis #13

    Schedules DRAMCtrl::Rank::processPowerEvent through:

    @@ -25759,7 +26107,7 @@ DRAMCtrl::Rank::processActivateEvent
    -
    19.22.4.2.14. TimingSimpleCPU analysis #14
    +
    23.22.4.2.14. TimingSimpleCPU analysis #14

    Executes DRAMCtrl::Rank::processPowerEvent.

    @@ -25768,25 +26116,25 @@ DRAMCtrl::Rank::processActivateEvent
    -
    19.22.4.2.15. TimingSimpleCPU analysis #15
    +
    23.22.4.2.15. TimingSimpleCPU analysis #15

    Executes BaseXBar::Layer<SrcType, DstType>::releaseLayer.

    -
    19.22.4.2.16. TimingSimpleCPU analysis #16
    +
    23.22.4.2.16. TimingSimpleCPU analysis #16

    Executes DRAMCtrl::processNextReqEvent().

    -
    19.22.4.2.17. TimingSimpleCPU analysis #17
    +
    23.22.4.2.17. TimingSimpleCPU analysis #17

    Executes DRAMCtrl::processRespondEvent().

    -
    19.22.4.2.18. TimingSimpleCPU analysis #18
    +
    23.22.4.2.18. TimingSimpleCPU analysis #18

    Schedules PacketQueue::processSendEvent() through:

    @@ -25801,13 +26149,13 @@ DRAMCtrl::processRespondEvent
    -
    19.22.4.2.19. TimingSimpleCPU analysis #19
    +
    23.22.4.2.19. TimingSimpleCPU analysis #19

    Executes PacketQueue::processSendEvent().

    -
    19.22.4.2.20. TimingSimpleCPU analysis #20
    +
    23.22.4.2.20. TimingSimpleCPU analysis #20

    Schedules PacketQueue::processSendEvent through:

    @@ -25831,7 +26179,7 @@ PacketQueue::processSendEvent
    -
    19.22.4.2.21. TimingSimpleCPU analysis #21
    +
    23.22.4.2.21. TimingSimpleCPU analysis #21

    Schedules BaseXBar::Layer<SrcType, DstType>::releaseLayer through:

    @@ -25851,19 +26199,19 @@ PacketQueue::processSendEvent
    -
    19.22.4.2.22. TimingSimpleCPU analysis #22
    +
    23.22.4.2.22. TimingSimpleCPU analysis #22

    Executes BaseXBar::Layer<SrcType, DstType>::releaseLayer.

    -
    19.22.4.2.23. TimingSimpleCPU analysis #23
    +
    23.22.4.2.23. TimingSimpleCPU analysis #23

    Executes PacketQueue::processSendEvent.

    -
    19.22.4.2.24. TimingSimpleCPU analysis #24
    +
    23.22.4.2.24. TimingSimpleCPU analysis #24

    Schedules TimingSimpleCPU::IcachePort::ITickEvent::process() through:

    @@ -25881,7 +26229,7 @@ PacketQueue::processSendEvent
    -
    19.22.4.2.25. TimingSimpleCPU analysis #25
    +
    23.22.4.2.25. TimingSimpleCPU analysis #25

    Executes TimingSimpleCPU::IcachePort::ITickEvent::process().

    @@ -25901,7 +26249,7 @@ PacketQueue::processSendEvent
    -
    19.22.4.2.26. TimingSimpleCPU analysis #26
    +
    23.22.4.2.26. TimingSimpleCPU analysis #26

    Schedules DRAMCtrl::processNextReqEvent through:

    @@ -25930,7 +26278,7 @@ TimingSimpleCPU::IcachePort::ITickEvent::process
    -
    19.22.4.2.27. TimingSimpleCPU analysis #27
    +
    23.22.4.2.27. TimingSimpleCPU analysis #27

    Schedules BaseXBar::Layer<SrcType, DstType>::releaseLayer through:

    @@ -25956,19 +26304,19 @@ TimingSimpleCPU::IcachePort::ITickEvent::process
    -
    19.22.4.2.28. TimingSimpleCPU analysis #28
    +
    23.22.4.2.28. TimingSimpleCPU analysis #28

    Execute DRAMCtrl::processNextReqEvent.

    -
    19.22.4.2.29. TimingSimpleCPU analysis #29
    +
    23.22.4.2.29. TimingSimpleCPU analysis #29

    Schedule DRAMCtrl::processRespondEvent().

    -
    19.22.4.2.30. TimingSimpleCPU analysis: LDR stall
    +
    23.22.4.2.30. TimingSimpleCPU analysis: LDR stall

    One important thing we want to check now, is how the memory reads are going to make the processor stall in the middle of an instruction.

    @@ -26086,7 +26434,7 @@ TimingSimpleCPU::IcachePort::ITickEvent::process
    -
    19.22.4.3. gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches
    +
    23.22.4.3. gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches

    Let’s just add --caches to gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis to see if things go any faster, and add Cache to --trace as in:

    @@ -26402,7 +26750,7 @@ type=SetAssociative

    At 1000, the future event is executed, and so it reads the original packet from the MSHR, and uses that to create a new request [40:7f] which gets forwarded.

    -
    19.22.4.3.1. What is the coherency protocol implemented by the classic cache system in gem5?
    +
    23.22.4.3.1. What is the coherency protocol implemented by the classic cache system in gem5?

    MOESI cache coherence protocol: https://github.com/gem5/gem5/blob/9fc9c67b4242c03f165951775be5cd0812f2a705/src/mem/cache/cache_blk.hh#L352

    @@ -26410,12 +26758,12 @@ type=SetAssociative

    The actual representation is done via separate state bits: https://github.com/gem5/gem5/blob/9fc9c67b4242c03f165951775be5cd0812f2a705/src/mem/cache/cache_blk.hh#L66 and MOESI appears explicitly only on the pretty printing.

    -

    This pretty printing appears for example in the --trace Cache lines as shown at gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches and with a few more transitions visible at Section 19.22.4.4, “gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs”.

    +

    This pretty printing appears for example in the --trace Cache lines as shown at gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches and with a few more transitions visible at Section 23.22.4.4, “gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs”.

    -
    19.22.4.4. gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs
    +
    23.22.4.4. gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs

    It would be amazing to analyze a simple example with interconnect packets possibly invalidating caches of other CPUs.

    @@ -26625,7 +26973,7 @@ type=SetAssociative
    -
    19.22.4.5. gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs
    +
    23.22.4.5. gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs

    Like gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs but with gem5 TimingSimpleCPU and userland/c/atomic/aarch64_add.c:

    @@ -26950,7 +27298,7 @@ global 147
    -
    19.22.4.6. gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs and Ruby
    +
    23.22.4.6. gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs and Ruby

    Now let’s do the exact same we did for gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs, but with Ruby rather than the classic system and TimingSimpleCPU (atomic does not work with Ruby)

    @@ -26992,7 +27340,7 @@ non-atomic 19
    -
    19.22.4.7. gem5 event queue MinorCPU syscall emulation freestanding example analysis
    +
    23.22.4.7. gem5 event queue MinorCPU syscall emulation freestanding example analysis

    The events for the Atomic CPU were pretty simple: basically just ticks.

    @@ -27162,14 +27510,14 @@ non-atomic 19
    -
    19.22.4.7.1. gem5 event queue MinorCPU syscall emulation freestanding example analysis: hazard
    +
    23.22.4.7.1. gem5 event queue MinorCPU syscall emulation freestanding example analysis: hazard

    TODO like gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazard but with the hazard.

    -
    19.22.4.8. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis
    +
    23.22.4.8. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis

    Like gem5 event queue MinorCPU syscall emulation freestanding example analysis but even more complex since for the gem5 DerivO3CPU!

    @@ -27197,7 +27545,7 @@ non-atomic 19

    This section and children are tested at LKMC 144a552cf926ea630ef9eadbb22b79fe2468c456.

    -
    19.22.4.8.1. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazardless
    +
    23.22.4.8.1. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazardless

    Let’s have a look at the arguably simplest example userland/arch/aarch64/freestanding/linux/hazardless.S.

    @@ -27436,7 +27784,7 @@ non-atomic 19
    -
    19.22.4.8.2. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazard
    +
    23.22.4.8.2. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazard

    Now let’s do the same as in gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazardless but with a hazard: userland/arch/aarch64/freestanding/linux/hazard.S.

    @@ -27480,7 +27828,7 @@ non-atomic 19
    -
    19.22.4.8.3. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazard4
    +
    23.22.4.8.3. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazard4

    Like gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazard but a hazard of depth 4: userland/arch/aarch64/freestanding/linux/hazard.S.

    @@ -27521,7 +27869,7 @@ non-atomic 19
    -
    19.22.4.8.4. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: stall
    +
    23.22.4.8.4. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: stall

    Like gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazard but now with an LDR stall: userland/arch/aarch64/freestanding/linux/stall.S.

    @@ -27572,7 +27920,7 @@ non-atomic 19
    -
    19.22.4.8.5. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: stall-gain
    +
    23.22.4.8.5. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: stall-gain

    Like gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: stall but now with an LDR stall: userland/arch/aarch64/freestanding/linux/stall-gain.S.

    @@ -27659,7 +28007,7 @@ non-atomic 19
    -
    19.22.4.8.6. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: stall-hazard4
    +
    23.22.4.8.6. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: stall-hazard4

    Like gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: stall-gain but now with some dependencies after the LDR: userland/arch/aarch64/freestanding/linux/stall-hazard4.S.

    @@ -27726,7 +28074,7 @@ non-atomic 19
    -
    19.22.4.8.7. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: speculative
    +
    23.22.4.8.7. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: speculative

    Now let’s try to see some Speculative execution in action with userland/arch/aarch64/freestanding/linux/speculative.S.

    @@ -27915,7 +28263,7 @@ wbActual:0
    -

    19.22.5. gem5 instruction definitions

    +

    23.22.5. gem5 instruction definitions

    This is one of the parts of gem5 that rely on semi-useless code generation inside the .isa sublanguage.

    @@ -27958,7 +28306,7 @@ wbActual:0
    -

    We also notice that the key argument passed to those instructions is of type ExecContext, which is discussed further at: Section 19.22.6.3, “gem5 ExecContext”.

    +

    We also notice that the key argument passed to those instructions is of type ExecContext, which is discussed further at: Section 23.22.6.3, “gem5 ExecContext”.

    The file is an include so that compilation can be split up into chunks by the autogenerated includers

    @@ -28163,7 +28511,7 @@ namespace ArmISAInst {

    Tested in gem5 b1623cb2087873f64197e503ab8894b5e4d4c7b4.

    -
    19.22.5.1. gem5 execute vs initiateAcc vs completeAcc
    +
    23.22.5.1. gem5 execute vs initiateAcc vs completeAcc

    These are the key methods defined in instruction definitions, so lets see when each one gets called and what they do more or less.

    @@ -28217,7 +28565,7 @@ namespace ArmISAInst {

    This can be seen concretely in GDB from the analysis done at: TimingSimpleCPU analysis: LDR stall and for more memory details see gem5 functional vs atomic vs timing memory requests.

    -
    19.22.5.1.1. gem5 completeAcc
    +
    23.22.5.1.1. gem5 completeAcc

    completeAcc is boring on most simple store memory instructions, e.g. a simple STR:

    @@ -28270,7 +28618,7 @@ namespace ArmISAInst {
    -
    19.22.5.2. gem5 microops
    +
    23.22.5.2. gem5 microops

    Some gem5 instructions break down into multiple microops.

    @@ -28331,7 +28679,7 @@ namespace ArmISAInst {
    -

    19.22.6. gem5 ThreadContext vs ThreadState vs ExecContext vs Process

    +

    23.22.6. gem5 ThreadContext vs ThreadState vs ExecContext vs Process

    These classes get used everywhere, and they have a somewhat convoluted relation with one another, so let’s figure it out this mess.

    @@ -28342,7 +28690,7 @@ namespace ArmISAInst {

    This section and all children tested at gem5 b1623cb2087873f64197e503ab8894b5e4d4c7b4.

    -
    19.22.6.1. gem5 ThreadContext
    +
    23.22.6.1. gem5 ThreadContext

    As we delve into more details below, we will reach the following conclusion: a ThreadContext represents on thread of a CPU with multiple Hardware threads.

    @@ -28392,7 +28740,7 @@ typedef SimpleThread MinorThread;

    Essentially all methods of the base ThreadContext are pure virtual.

    -
    19.22.6.1.1. gem5 SimpleThread
    +
    23.22.6.1.1. gem5 SimpleThread

    SimpleThread storage defined on BaseSimpleCPU for simple CPUs like AtomicSimpleCPU:

    @@ -28487,7 +28835,7 @@ typedef SimpleThread MinorThread;
    -
    19.22.6.1.2. gem5 O3ThreadContext
    +
    23.22.6.1.2. gem5 O3ThreadContext

    Instantiation happens in the FullO3CPU constructor:

    @@ -28588,7 +28936,7 @@ FullO3CPU<Impl>::readArchIntReg(int reg_idx, ThreadID tid)
    -
    19.22.6.2. gem5 ThreadState
    +
    23.22.6.2. gem5 ThreadState

    Owned one per ThreadContext.

    @@ -28634,7 +28982,7 @@ class O3ThreadContext : public ThreadContext
    -
    19.22.6.3. gem5 ExecContext
    +
    23.22.6.3. gem5 ExecContext

    ExecContext gets used in gem5 instruction definitions, e.g.:

    @@ -28794,7 +29142,7 @@ class O3ThreadContext : public ThreadContext

    This makes sense, since each ThreadContext represents one CPU register set, and therefore needs a separate ExecContext which allows instruction implementations to access those registers.

    -
    19.22.6.3.1. gem5 ExecContext::readIntRegOperand register resolution
    +
    23.22.6.3.1. gem5 ExecContext::readIntRegOperand register resolution

    Let’s have a look at how ExecContext::readIntRegOperand actually matches registers to decoded registers IDs, since it is not obvious.

    @@ -28833,7 +29181,7 @@ class O3ThreadContext : public ThreadContext

    First, we guess that they must be related to the reading of x1 and x2, which are the inputs of the addition.

    -

    Next, we also guess that the 0 read must correspond to x2, since it later gets potentially shifted as mentioned at Section 25.4.4.1, “ARM shift suffixes”.

    +

    Next, we also guess that the 0 read must correspond to x2, since it later gets potentially shifted as mentioned at Section 29.4.4.1, “ARM shift suffixes”.

    Let’s also have a look at the decoder code that builds the instruction instance in build/ARM/arch/arm/generated/decoder-ns.cc.inc:

    @@ -29067,7 +29415,7 @@ flattenIntIndex(int reg) const
    -
    19.22.6.4. gem5 Process
    +
    23.22.6.4. gem5 Process

    The Process class is used only for gem5 syscall emulation mode, and it represents a process like a Linux userland process, in addition to any further gem5 specific data needed to represent the process.

    @@ -29155,12 +29503,12 @@ readFunc(SyscallDesc *desc, ThreadContext *tc,
    -

    19.22.7. gem5 functional units

    +

    23.22.7. gem5 functional units

    Each instruction is marked with a class, and each class can execute in a given functional unit.

    -
    19.22.7.1. gem5 MinorCPU default functional units
    +
    23.22.7.1. gem5 MinorCPU default functional units

    Which units are available is visible for example on the gem5 config.ini of a gem5 MinorCPU run. Functional units are not present in simple CPUs like gem5 TimingSimpleCPU.

    @@ -29319,7 +29667,7 @@ opClass=IntAlu
    -
    19.22.7.2. gem5 DerivO3CPU default functional units
    +
    23.22.7.2. gem5 DerivO3CPU default functional units

    On gem5 3ca404da175a66e0b958165ad75eb5f54cb5e772, after running:

    @@ -29417,7 +29765,7 @@ pipelined=false
    -

    19.22.8. gem5 code generation

    +

    23.22.8. gem5 code generation

    gem5 uses a ton of code generation, which makes the project horrendous:

    @@ -29462,7 +29810,7 @@ pipelined=false

    But it has been widely overused to insanity. It likely also exists partly because when the project started in 2003 C++ compilers weren’t that good, so you couldn’t rely on features like templates that much.

    -
    19.22.8.1. gem5 THE_ISA
    +
    23.22.8.1. gem5 THE_ISA

    Generated code at: build/<ISA>/config/the_isa.hh which e.g. for ARM contains:

    @@ -29508,9 +29856,9 @@ enum class Arch {
    -

    19.22.9. gem5 build system

    +

    23.22.9. gem5 build system

    -
    19.22.9.1. M5_OVERRIDE_PY_SOURCE
    +
    23.22.9.1. M5_OVERRIDE_PY_SOURCE

    https://stackoverflow.com/questions/52312070/how-to-modify-a-file-under-src-python-and-run-it-without-rebuilding-in-gem5

    @@ -29525,7 +29873,7 @@ enum class Arch {
    -
    19.22.9.2. gem5 build broken on recent compiler version
    +
    23.22.9.2. gem5 build broken on recent compiler version

    gem5 moves a bit slowly, and if your host compiler is very new, the gem5 build might be broken for it, e.g. this was the case for Ubuntu 19.10 with GCC 9 and gem5 62d75e7105fe172eb906d4f80f360ff8591d4178 from Dec 2019.

    @@ -29550,7 +29898,7 @@ enum class Arch {
    -
    19.22.9.3. gem5 polymorphic ISA includes
    +
    23.22.9.3. gem5 polymorphic ISA includes

    E.g. src/cpu/decode_cache.hh includes:

    @@ -29629,7 +29977,7 @@ build/ARM/config/the_isa.hh
    -
    19.22.9.4. Why are all C++ symlinked into the gem5 build dir?
    +
    23.22.9.4. Why are all C++ symlinked into the gem5 build dir?

    Upstream request: https://gem5.atlassian.net/browse/GEM5-469

    @@ -29670,7 +30018,7 @@ build/ARM/config/the_isa.hh
    -

    20. Gensim

    +

    24. Gensim

    https://gensim.org

    @@ -29774,10 +30122,10 @@ gensim/models/armv8/isa.ac
    -

    21. Buildroot

    +

    25. Buildroot

    -

    21.1. Introduction to Buildroot

    +

    25.1. Introduction to Buildroot

    Buildroot is a set of Make scripts that download and compile from source compatible versions of:

    @@ -29790,7 +30138,7 @@ gensim/models/armv8/isa.ac

    Linux kernel

  • -

    C standard library: Buildroot supports several implementations, see: Section 21.10, “libc choice”

    +

    C standard library: Buildroot supports several implementations, see: Section 25.10, “libc choice”

  • BusyBox: provides the shell and basic command line utilities

    @@ -29801,7 +30149,7 @@ gensim/models/armv8/isa.ac

    It therefore produces a pristine, blob-less, debuggable setup, where all moving parts are configured to work perfectly together.

    • -

    Perhaps the awesomeness of Buildroot only sinks in once you notice that all it takes is 4 commands as explained at Section 21.11, “Buildroot hello world”.

    +

    Perhaps the awesomeness of Buildroot only sinks in once you notice that all it takes is 4 commands as explained at Section 25.11, “Buildroot hello world”.

    The downsides of Buildroot are:

    @@ -29846,7 +30194,7 @@ gensim/models/armv8/isa.ac
    -

    21.2. Custom Buildroot configs

    +

    25.2. Custom Buildroot configs

    We provide the following mechanisms:

    @@ -29881,10 +30229,10 @@ gensim/models/armv8/isa.ac

    The clean is necessary because the source files didn’t change, so make would just check the timestamps and not build anything.

    -

    You will then likely want to make those more permanent as explained at: Section 34.4, “Default command line arguments”.

    +

    You will then likely want to make those more permanent as explained at: Section 37.4, “Default command line arguments”.

    -

    21.2.1. Enable Buildroot compiler optimizations

    +

    25.2.1. Enable Buildroot compiler optimizations

    If you are benchmarking compiled programs instead of hand written assembly, remember that we configure Buildroot to disable optimizations by default with:

    @@ -29916,7 +30264,7 @@ gensim/models/armv8/isa.ac
    -

    21.4. Change user

    +

    25.4. Change user

    At startup, we login automatically as the root user.

    @@ -30015,7 +30363,7 @@ make menuconfig
    -

    21.4.1. Login as a non-root user without password

    +

    25.4.1. Login as a non-root user without password

    Replace on inittab:

    @@ -30038,7 +30386,7 @@ make menuconfig
    -

    21.5. Add new files to the Buildroot image

    +

    25.5. Add new files to the Buildroot image

    These are your options:

    @@ -30102,7 +30450,7 @@ make menuconfig
    -

    21.5.1. Add new Buildroot packages

    +

    25.5.1. Add new Buildroot packages

    First, see if you can’t get away without actually adding a new package, for example:

    @@ -30112,7 +30460,7 @@ make menuconfig

    if you have a standalone C file with no dependencies besides the C standard library to be compiled with GCC, just add a new file under buildroot_packages/sample_package and you are done

  • -

    if you have a dependency on a library, first check if Buildroot doesn’t have a package for it already with ls buildroot/package. If yes, just enable that package as explained at: Section 21.2, “Custom Buildroot configs”

    +

    if you have a dependency on a library, first check if Buildroot doesn’t have a package for it already with ls buildroot/package. If yes, just enable that package as explained at: Section 25.2, “Custom Buildroot configs”

    • @@ -30120,7 +30468,7 @@ make menuconfig

    If none of those methods are flexible enough for you, you can just fork or hack up buildroot_packages/sample_package the sample package to do what you want.

    -

    For how to use that package, see: Section 34.15.2, “buildroot_packages directory”.

    +

    For how to use that package, see: Section 37.15.2, “buildroot_packages directory”.

    Then iterate trying to do what you want and reading the manual until it works: https://buildroot.org/downloads/manual/manual.html

    @@ -30128,7 +30476,7 @@ make menuconfig
    -

    21.6. Remove Buildroot packages

    +

    25.6. Remove Buildroot packages

    Once you’ve built a package in to the image, there is no easy way to remove it.

    @@ -30139,11 +30487,11 @@ make menuconfig

    Also mentioned at: https://stackoverflow.com/questions/47320800/how-to-clean-only-target-in-buildroot

    -

    See this for a sample manual workaround: Section 22.9.1.4, “PARSEC uninstall”.

    +

    See this for a sample manual workaround: Section 26.9.1.4, “PARSEC uninstall”.

    -

    21.7. BR2_TARGET_ROOTFS_EXT2_SIZE

    +

    25.7. BR2_TARGET_ROOTFS_EXT2_SIZE

    When adding new large package to the Buildroot root filesystem, it may fail with the message:

    @@ -30187,7 +30535,7 @@ TODO benchmark: would gem5 suffer a considerable disk read performance hit due t

    libguestfs: https://serverfault.com/questions/246835/convert-directory-to-qemu-kvm-virtual-disk-image/916697#916697, in particular vfs-minimum-size

  • -

    use methods described at: Section 19.6.3, “gem5 checkpoint restore and run a different script” instead of putting builds on the root filesystem

    +

    use methods described at: Section 23.6.3, “gem5 checkpoint restore and run a different script” instead of putting builds on the root filesystem

    • @@ -30195,7 +30543,7 @@ TODO benchmark: would gem5 suffer a considerable disk read performance hit due t

    Bibliography: https://stackoverflow.com/questions/49211241/is-there-a-way-to-automatically-detect-the-minimum-required-br2-target-rootfs-ex

    -

    21.7.1. SquashFS

    +

    25.7.1. SquashFS

    SquashFS creation with mksquashfs does not take fixed sizes, and I have successfully booted from it, but it is readonly, which is unacceptable.

    @@ -30208,7 +30556,7 @@ TODO benchmark: would gem5 suffer a considerable disk read performance hit due t
    -

    21.8. Buildroot rebuild is slow when the root filesystem is large

    +

    25.8. Buildroot rebuild is slow when the root filesystem is large

    Buildroot is not designed for large root filesystem images, and the rebuild becomes very slow when we add a large package to it.

    @@ -30246,7 +30594,7 @@ TODO benchmark: would gem5 suffer a considerable disk read performance hit due t
    -

    21.9. Report upstream bugs

    +

    25.9. Report upstream bugs

    When asking for help on upstream repositories outside of this repository, you will need to provide the commands that you are running in detail without referencing our scripts.

    @@ -30306,7 +30654,7 @@ git -C "$(./getvar qemu_source_dir)" checkout -

    Then, you will also want to do a Bisection to pinpoint the exact commit to blame, and CC that developer.

    -

    Finally, give the images you used save upstream developers' time as shown at: Section 34.19.2, “release-zip”.

    +

    Finally, give the images you used save upstream developers' time as shown at: Section 37.19.2, “release-zip”.

    For Buildroot problems, you should wither provide the config you have:

    @@ -30321,7 +30669,7 @@ git -C "$(./getvar qemu_source_dir)" checkout -
    -

    21.10. libc choice

    +

    25.10. libc choice

    Buildroot supports several libc implementations, including:

    @@ -30369,7 +30717,7 @@ git -C "$(./getvar qemu_source_dir)" checkout -
    -

    21.11. Buildroot hello world

    +

    25.11. Buildroot hello world

    This repo doesn’t do much more other than setting a bunch of Buildroot configurations and building it.

    @@ -30414,7 +30762,7 @@ git -C "$(./getvar qemu_source_dir)" checkout -
    -

    21.12. Update the Buildroot toolchain

    +

    25.12. Update the Buildroot toolchain

    Users of this repo will often want to update the compilation toolchain to the latest version to get fresh new features like new ISA instructions.

    @@ -30428,7 +30776,7 @@ git -C "$(./getvar qemu_source_dir)" checkout -

    In this section we cover the most common cases.

    -

    21.12.1. Update GCC: GCC supported by Buildroot

    +

    25.12.1. Update GCC: GCC supported by Buildroot

    This is of course the simplest case.

    @@ -30546,9 +30894,9 @@ cd ../..
    -

    21.12.2. Update GCC: GCC not supported by Buildroot

    +

    25.12.2. Update GCC: GCC not supported by Buildroot

    -

    Now it gets fun, but well, guess what, we will try to do the same as Section 21.12.1, “Update GCC: GCC supported by Buildroot” but:

    +

    Now it gets fun, but well, guess what, we will try to do the same as Section 25.12.1, “Update GCC: GCC supported by Buildroot” but:

    -

    21.13. Buildroot vanilla kernel

    +

    25.13. Buildroot vanilla kernel

    By default, our build system uses build-linux, and the Buildroot kernel build is disabled: https://stackoverflow.com/questions/52231793/can-buildroot-build-the-root-filesystem-without-building-the-linux-kernel

    @@ -30638,7 +30986,7 @@ cd ../..
    -

    22. Userland content

    +

    26. Userland content

    This section documents our test and educational userland content, such as C, C++ and POSIX examples, present mostly under userland/.

    @@ -30647,7 +30995,7 @@ cd ../..

    Getting started at: Section 1.8, “Userland setup”

    -

    Userland assembly content is located at: Section 23, “Userland assembly”. It was split from this section basically because we were hitting the HTML h6 limit, stupid web :-)

    +

    Userland assembly content is located at: Section 27, “Userland assembly”. It was split from this section basically because we were hitting the HTML h6 limit, stupid web :-)

    This content makes up the bulk of the userland/ directory.

    @@ -30659,7 +31007,7 @@ cd ../..

    This section was originally moved in here from: https://github.com/cirosantilli/cpp-cheat

    -

    22.1. build-userland

    +

    26.1. build-userland

    build-userland

    @@ -30716,7 +31064,7 @@ cd ../..
    -

    22.2. C

    +

    26.2. C

    Programs under userland/c/ are examples of ANSI C programming:

    @@ -30855,7 +31203,7 @@ cd ../..
    -

    22.2.1. malloc

    +

    26.2.1. malloc

    Allocate memory! Vs using the stack: https://stackoverflow.com/questions/4584089/what-is-the-function-of-the-push-pop-instructions-used-on-registers-in-x86-ass/33583134#33583134

    @@ -30869,7 +31217,7 @@ cd ../..

    malloc leads to the infinite joys of Memory leaks.

    -
    22.2.1.1. malloc implementation
    +
    26.2.1.1. malloc implementation

    TODO: the exact answer is going to be hard.

    @@ -30914,7 +31262,7 @@ printf '%x\n' 4198400
    -
    22.2.1.2. malloc maximum size
    +
    26.2.1.2. malloc maximum size

    General overview at: https://stackoverflow.com/questions/2798330/maximum-memory-which-malloc-can-allocate

    @@ -30980,7 +31328,7 @@ echo 1 > /proc/sys/vm/overcommit_memory

    If we start using the pages, the OOM killer would sooner or later step in and kill our process: Linux out-of-memory killer.

    -
    22.2.1.2.1. Linux out-of-memory killer
    +
    26.2.1.2.1. Linux out-of-memory killer

    We can observe the OOM in LKMC 1e969e832f66cb5a72d12d57c53fb09e9721d589 which defaults to 256MiB of memory with:

    @@ -31006,7 +31354,7 @@ echo 1 > /proc/sys/vm/overcommit_memory
    -

    22.2.2. C multithreading

    +

    26.2.2. C multithreading

    Added in C11!

    @@ -31024,7 +31372,7 @@ echo 1 > /proc/sys/vm/overcommit_memory
    -
    22.2.2.1. atomic.c
    +
    26.2.2.1. atomic.c
    -

    22.2.3. GCC C extensions

    +

    26.2.3. GCC C extensions

    -
    22.2.3.1. C empty struct
    +
    26.2.3.1. C empty struct

    Example: userland/gcc/empty_struct.c

    @@ -31130,7 +31478,7 @@ echo 1 > /proc/sys/vm/overcommit_memory
    -
    22.2.3.2. OpenMP
    +
    26.2.3.2. OpenMP

    GCC implements the OpenMP threading implementation: https://stackoverflow.com/questions/3949901/pthreads-vs-openmp

    @@ -31153,7 +31501,7 @@ echo 1 > /proc/sys/vm/overcommit_memory

    strace shows that OpenMP makes clone() syscalls in Linux. TODO: does it actually call pthread_ functions, or does it make syscalls directly? Or in other words, can it work on Freestanding programs? A quick grep shows many references to pthreads.

    -
    22.2.3.2.1. OpenMP validation
    +
    26.2.3.2.1. OpenMP validation

    https://github.com/uhhpctools/omp-validation

    @@ -31251,7 +31599,7 @@ mkdir -p bin/c
    -

    22.3. C++

    +

    26.3. C++

    Programs under userland/cpp/ are examples of ISO C programming.

    @@ -31361,21 +31709,27 @@ mkdir -p bin/c
    +
  • +

    Algorithms contains a benchmark comparison of different c++ containers

    +
    • -

    22.3.1. C++ initialization types

    +

    26.3.1. C++ initialization types

    OMG this is hell, understand when primitive variables are initialized or not:

    @@ -31423,7 +31777,7 @@ mkdir -p bin/c
    -

    22.3.2. C++ multithreading

    +

    26.3.2. C++ multithreading

    -
    22.3.2.1. atomic.cpp
    +
    26.3.2.1. atomic.cpp

    userland/cpp/atomic/

    @@ -31657,7 +32011,7 @@ time ./mutex.out 4 100000000
    -
    22.3.2.1.1. Detailed gem5 analysis of how data races happen
    +
    26.3.2.1.1. Detailed gem5 analysis of how data races happen

    The smallest data race we managed to come up as of LKMC 7c01b29f1ee7da878c7cc9cb4565f3f3cf516a92 and gem5 872cb227fdc0b4d60acc7840889d567a6936b6e1 was with userland/c/atomic.c (see also C multithreading):

    @@ -31762,7 +32116,7 @@ non-atomic 19
    -
    22.3.2.2. C++ std::memory_order
    +
    26.3.2.2. C++ std::memory_order

    https://stackoverflow.com/questions/12346487/what-do-each-memory-order-mean

    @@ -31774,7 +32128,7 @@ non-atomic 19
    -
    22.3.2.3. C++ parallel algorithms
    +
    26.3.2.3. C++ parallel algorithms

    https://stackoverflow.com/questions/51031060/are-c17-parallel-algorithms-implemented-already/55989883#55989883

    @@ -31784,7 +32138,7 @@ non-atomic 19
    -

    22.3.3. C++ standards

    +

    26.3.3. C++ standards

    Like for C, you have to pay for the standards…​ insane. So we just use the closest free drafts instead.

    @@ -31792,14 +32146,14 @@ non-atomic 19

    https://stackoverflow.com/questions/81656/where-do-i-find-the-current-c-or-c-standard-documents

    -
    22.3.3.1. C++17 N4659 standards draft
    +
    26.3.3.1. C++17 N4659 standards draft

    http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/n4659.pdf

    -

    22.3.4. C++ type casting

    +

    26.3.4. C++ type casting

    userland/cpp/static_dynamic_reinterpret_cast.cpp

    @@ -31809,7 +32163,7 @@ non-atomic 19
    -

    22.4. POSIX

    +

    26.4. POSIX

    Programs under userland/posix/ are examples of POSIX C programming.

    @@ -31827,13 +32181,13 @@ non-atomic 19
    -

    22.4.1. Environment variables

    +

    26.4.1. Environment variables

    POSIX C example that prints all environment variables: userland/posix/environ.c

    -

    22.4.2. unistd.h

    +

    26.4.2. unistd.h

    -

    22.4.3. fork

    +

    26.4.3. fork

    POSIX' multiprocess API. Contrast with pthreads which are for threads.

    @@ -31871,7 +32225,7 @@ fork() return = 13039

    Read the source comments and understand everything that is going on!

    -
    22.4.3.1. getpid
    +
    26.4.3.1. getpid

    The minimal interesting example is to use fork and observe different PIDs.

    @@ -31883,7 +32237,7 @@ fork() return = 13039
    -
    22.4.3.2. Fork bomb
    +
    26.4.3.2. Fork bomb

    https://en.wikipedia.org/wiki/Fork_bomb

    @@ -31918,7 +32272,7 @@ fork() return = 13039
    -

    22.4.4. pthreads

    +

    26.4.4. pthreads

    POSIX' multithreading API. Contrast with fork which is for processes.

    @@ -31942,7 +32296,7 @@ fork() return = 13039
    -
    22.4.4.1. pthread_mutex
    +
    26.4.4.1. pthread_mutex

    userland/posix/pthread_count.c exemplifies the functions:

    @@ -31979,7 +32333,7 @@ There are no non-locking atomic types or atomic primitives in POSIX: -

    22.4.5. sysconf

    +

    26.4.5. sysconf

    https://pubs.opengroup.org/onlinepubs/9699919799/functions/sysconf.html

    @@ -32025,7 +32379,7 @@ There are no non-locking atomic types or atomic primitives in POSIX: -

    22.4.6. mmap

    +

    26.4.6. mmap

    The mmap system call allows advanced memory operations.

    @@ -32036,7 +32390,7 @@ There are no non-locking atomic types or atomic primitives in POSIX: -
    22.4.6.1. mmap MAP_ANONYMOUS
    +
    26.4.6.1. mmap MAP_ANONYMOUS

    Basic mmap example, do the same as userland/c/malloc.c, but with mmap.

    @@ -32054,7 +32408,7 @@ There are no non-locking atomic types or atomic primitives in POSIX: -
    22.4.6.2. mmap file
    +
    26.4.6.2. mmap file

    Memory mapped file example: userland/posix/mmap_file.c

    @@ -32066,7 +32420,7 @@ There are no non-locking atomic types or atomic primitives in POSIX: -
    22.4.6.3. brk
    +
    26.4.6.3. brk

    Previously POSIX, but was deprecated in favor of malloc

    @@ -32082,7 +32436,7 @@ There are no non-locking atomic types or atomic primitives in POSIX: -

    22.4.7. socket

    +

    26.4.7. socket

    A bit like read and write, but from / to the Internet!

    @@ -32096,7 +32450,7 @@ There are no non-locking atomic types or atomic primitives in POSIX: -

    22.5. Userland multithreading

    +

    26.5. Userland multithreading

    The following sections are related to multithreading in userland:

    @@ -32158,12 +32512,12 @@ There are no non-locking atomic types or atomic primitives in POSIX: -

    22.6. C debugging

    +

    26.6. C debugging

    Let’s group the hard-to-debug undefined-behaviour-like stuff found in C / C+ here and how to tackle those problems.

    -

    22.6.1. Stack smashing

    +

    26.6.1. Stack smashing

    https://stackoverflow.com/questions/1345670/stack-smashing-detected/51897264#51897264

    @@ -32183,7 +32537,7 @@ There are no non-locking atomic types or atomic primitives in POSIX: -

    22.6.2. Memory leaks

    +

    26.6.2. Memory leaks

    How to debug: https://stackoverflow.com/questions/6261201/how-to-find-memory-leak-in-a-c-code-project/57877190#57877190

    @@ -32192,7 +32546,7 @@ There are no non-locking atomic types or atomic primitives in POSIX: -

    22.6.3. Profiling userland programs

    +

    26.6.3. Profiling userland programs

    https://stackoverflow.com/questions/375913/how-can-i-profile-c-code-running-on-linux/60265409#60265409

    @@ -32212,12 +32566,12 @@ There are no non-locking atomic types or atomic primitives in POSIX: -

    22.7. Interpreted languages

    +

    26.7. Interpreted languages

    Maybe some day someone will use this setup to study the performance of interpreters.

    -

    22.7.1. Python

    +

    26.7.1. Python

    Examples:

    @@ -32242,7 +32596,7 @@ There are no non-locking atomic types or atomic primitives in POSIX: -
    22.7.1.1. Build and install the interpreter
    +
    26.7.1.1. Build and install the interpreter

    Buildroot has a Python package that can be added to the guest image:

    @@ -32301,7 +32655,7 @@ There are no non-locking atomic types or atomic primitives in POSIX: -
    22.7.1.2. Python gem5 user mode simulation
    +
    26.7.1.2. Python gem5 user mode simulation

    At LKMC 50ac89b779363774325c81157ec8b9a6bdb50a2f gem5 390a74f59934b85d91489f8a563450d8321b602da:

    @@ -32349,9 +32703,19 @@ There are no non-locking atomic types or atomic primitives in POSIX:

    which corresponds to the glorious getrandom syscall: https://github.com/torvalds/linux/blob/v4.17/include/uapi/asm-generic/unistd.h#L707

    +
    +

    Bibliography:

    +
    +
    + +
    -
    22.7.1.3. Embedding Python in another application
    +
    26.7.1.3. Embedding Python in another application

    Here we will add some better examples and explanations for: https://docs.python.org/3/extending/embedding.html#very-high-level-embedding

    @@ -32402,7 +32766,7 @@ There are no non-locking atomic types or atomic primitives in POSIX: -
    22.7.1.4. pybind11
    +
    26.7.1.4. pybind11

    userland/libs/pybind11

    @@ -32425,7 +32789,7 @@ There are no non-locking atomic types or atomic primitives in POSIX: -

    22.7.2. Node.js

    +

    26.7.2. Node.js

    Host installation shown at: https://askubuntu.com/questions/594656/how-to-install-the-latest-versions-of-nodejs-and-npm/971612#971612

    @@ -32525,7 +32889,7 @@ my type is MyClassToString and a is 1 and b is 2
    -
    22.7.2.1. NPM
    +
    26.7.2.1. NPM

    https://en.wikipedia.org/wiki/Npm_(software)

    @@ -32544,7 +32908,7 @@ my type is MyClassToString and a is 1 and b is 2
    -
    22.7.2.1.1. NPM data-files
    +
    26.7.2.1.1. NPM data-files

    Illustrates how to add extra non-code data files to an NPM package, and then use those files at runtime.

    @@ -32555,7 +32919,7 @@ my type is MyClassToString and a is 1 and b is 2
    -

    22.7.3. Java

    +

    26.7.3. Java

    No OpenJDK package as of 2018.08: https://stackoverflow.com/questions/28874150/buildroot-with-jamvm-2-0-for-java-8/59290927#59290927 partly because their build system is shit like the rest of the project’s setup.

    @@ -32571,7 +32935,7 @@ my type is MyClassToString and a is 1 and b is 2
    -

    22.8. Algorithms

    +

    26.8. Algorithms

    userland/algorithm

    @@ -32731,7 +33095,7 @@ cmp tmp.o tmp.e

    These are good targets for performance analysis with gem5, and there is some overlap between this section and Benchmarks.

    -

    22.8.1. BST vs heap vs hashmap

    +

    26.8.1. BST vs heap vs hashmap

    TODO: move benchmark graph from userland/cpp/bst_vs_heap_vs_hashmap.cpp to userland/algorithm/set.

    @@ -32829,7 +33193,7 @@ xdg-open bst_vs_heap_vs_hashmap_gem5.tmp.png

    The cache sizes were chosen to match the host 2017 Lenovo ThinkPad P51 to improve the comparison. Ideally we should also use the same standard library.

    -

    Note that this will take a long time, and will produce a humongous ~40Gb stats file as explained at: Section 19.10.3.2, “gem5 only dump selected stats”

    +

    Note that this will take a long time, and will produce a humongous ~40Gb stats file as explained at: Section 23.10.3.2, “gem5 only dump selected stats”

    Sources:

    @@ -32849,7 +33213,7 @@ xdg-open bst_vs_heap_vs_hashmap_gem5.tmp.png
    -

    22.8.2. BLAS

    +

    26.8.2. BLAS

    Buildroot supports it, which makes everything just trivial:

    @@ -32901,7 +33265,7 @@ cblas_dgemm( CblasColMajor, CblasNoTrans, CblasTrans,3,3,2 ,1, A,3, B,
    -

    22.8.3. Eigen

    +

    26.8.3. Eigen

    Header only linear algebra library with a mainline Buildroot package:

    @@ -32940,7 +33304,7 @@ cblas_dgemm( CblasColMajor, CblasNoTrans, CblasTrans,3,3,2 ,1, A,3, B,
    -

    22.9. Benchmarks

    +

    26.9. Benchmarks

    These are good targets for performance analysis with gem5.

    @@ -32958,7 +33322,7 @@ cblas_dgemm( CblasColMajor, CblasNoTrans, CblasTrans,3,3,2 ,1, A,3, B,
    -

    22.9.1. PARSEC benchmark

    +

    26.9.1. PARSEC benchmark

    We have ported parts of the PARSEC benchmark for cross compilation at: https://github.com/cirosantilli/parsec-benchmark See the documentation on that repo to find out which benchmarks have been ported. Some of the benchmarks were are segfaulting, they are documented in that repo.

    @@ -32976,7 +33340,7 @@ cblas_dgemm( CblasColMajor, CblasNoTrans, CblasTrans,3,3,2 ,1, A,3, B,
    -
    22.9.1.1. PARSEC benchmark without parsecmgmt
    +
    26.9.1.1. PARSEC benchmark without parsecmgmt
    ./build --arch arm --download-dependencies gem5-buildroot parsec-benchmark
@@ -33010,7 +33374,7 @@ cblas_dgemm(      CblasColMajor, CblasNoTrans, CblasTrans,3,3,2  ,1,    A,3,  B,
    -
    22.9.1.2. PARSEC change the input size
    +
    26.9.1.2. PARSEC change the input size

    Running a benchmark of a size different than test, e.g. simsmall, requires a rebuild with:

    @@ -33074,7 +33438,7 @@ cblas_dgemm( CblasColMajor, CblasNoTrans, CblasTrans,3,3,2 ,1, A,3, B,
    -
    22.9.1.3. PARSEC benchmark with parsecmgmt
    +
    26.9.1.3. PARSEC benchmark with parsecmgmt

    Most users won’t want to use this method because:

    @@ -33137,9 +33501,9 @@ parsecmgmt -a run -p splash2x.fmm -i test
    -
    22.9.1.4. PARSEC uninstall
    +
    26.9.1.4. PARSEC uninstall
    -

    If you want to remove PARSEC later, Buildroot doesn’t provide an automated package removal mechanism as mentioned at: Section 21.6, “Remove Buildroot packages”, but the following procedure should be satisfactory:

    +

    If you want to remove PARSEC later, Buildroot doesn’t provide an automated package removal mechanism as mentioned at: Section 25.6, “Remove Buildroot packages”, but the following procedure should be satisfactory:

    @@ -33155,7 +33519,7 @@ parsecmgmt -a run -p splash2x.fmm -i test
    -
    22.9.1.5. PARSEC benchmark hacking
    +
    26.9.1.5. PARSEC benchmark hacking

    If you end up going inside submodules/parsec-benchmark to hack up the benchmark (you will!), these tips will be helpful.

    @@ -33207,7 +33571,7 @@ git clean -xdf .
    -
    22.9.1.6. Coremark
    +
    26.9.1.6. Coremark

    https://en.wikipedia.org/wiki/Coremark

    @@ -33420,7 +33784,7 @@ RUN_FLAGS =
    -

    22.9.2. Microbenchmarks

    +

    26.9.2. Microbenchmarks

    It eventually has to come to that, hasn’t it?

    @@ -33457,7 +33821,7 @@ RUN_FLAGS =
    -
    22.9.2.1. Dhrystone
    +
    26.9.2.1. Dhrystone

    https://en.wikipedia.org/wiki/Dhrystone

    @@ -33574,7 +33938,7 @@ Dhrystones per Second: 16152479.0
    -
    22.9.2.2. LMbench
    +
    26.9.2.2. LMbench

    http://www.bitmover.com/lmbench/

    @@ -33692,7 +34056,7 @@ make
    -
    22.9.2.3. STREAM benchmark
    +
    26.9.2.3. STREAM benchmark

    http://www.cs.virginia.edu/stream/ref.html

    @@ -33818,7 +34182,7 @@ Solution Validates: avg error less than 1.000000e-13 on all three arrays
    -

    22.10. userland/libs directory

    +

    26.10. userland/libs directory

    Tests under userland/libs require certain optional libraries to be installed on the target, and are not built or tested by default, you must enable them with either:

    @@ -33838,7 +34202,7 @@ Solution Validates: avg error less than 1.000000e-13 on all three arrays
    -

    22.10.1. Boost

    +

    26.10.1. Boost

    https://en.wikipedia.org/wiki/Boost_(C%2B%2B_libraries)

    @@ -33854,7 +34218,7 @@ Solution Validates: avg error less than 1.000000e-13 on all three arrays
    -

    22.10.2. HDF5

    +

    26.10.2. HDF5

    https://en.wikipedia.org/wiki/Hierarchical_Data_Format

    @@ -33877,7 +34241,7 @@ Solution Validates: avg error less than 1.000000e-13 on all three arrays
    -

    22.11. Userland content filename conventions

    +

    26.11. Userland content filename conventions

    The following basenames should always refer to programs that do the same thing, but in different languages:

    @@ -33906,7 +34270,7 @@ Solution Validates: avg error less than 1.000000e-13 on all three arrays
    -

    22.12. Userland content bibliography

    +

    26.12. Userland content bibliography

    -

    23. Userland assembly

    +

    27. Userland assembly

    Programs under userland/arch/<arch>/ are examples of userland assembly programming.

    @@ -34019,7 +34383,7 @@ Solution Validates: avg error less than 1.000000e-13 on all three arrays
  • -

    registers, see: Section 23.1, “Assembly registers”

    +

    registers, see: Section 27.1, “Assembly registers”

  • jumping:

    @@ -34162,14 +34526,14 @@ error: asm_main returned 1 at line 8
    • -

    23.1. Assembly registers

    +

    27.1. Assembly registers

    After seeing an ADD hello world, you need to learn the general registers:

    -

    23.1.1. ARMv8 aarch64 x31 register

    +

    27.1.1. ARMv8 aarch64 x31 register

    Example: userland/arch/aarch64/x31.S

    @@ -34285,7 +34649,7 @@ When instructions do not interpret this operand encoding as the zero register, u
    -

    23.2. Floating point assembly

    +

    27.2. Floating point assembly

    Keep in mind that many ISAs started floating point as an optional thing, and it later got better integrated into the main CPU, side by side with SIMD.

    @@ -34327,7 +34691,7 @@ When instructions do not interpret this operand encoding as the zero register, u
    -

    23.3. SIMD assembly

    +

    27.3. SIMD assembly

    Much like ADD for non-SIMD, start learning SIMD instructions by looking at the integer and floating point SIMD ADD instructions of each ISA:

    @@ -34417,14 +34781,14 @@ When instructions do not interpret this operand encoding as the zero register, u

    Bibliography: https://stackoverflow.com/questions/1389712/getting-started-with-intel-x86-sse-simd-instructions/56409539#56409539

    -

    23.3.1. FMA instruction

    +

    27.3.1. FMA instruction

    Fused multiply add:

    @@ -34466,7 +34830,7 @@ When instructions do not interpret this operand encoding as the zero register, u
    -

    23.4. User vs system assembly

    +

    27.4. User vs system assembly

    By "userland assembly", we mean "the parts of the ISA which can be freely used from userland".

    @@ -34477,7 +34841,7 @@ When instructions do not interpret this operand encoding as the zero register, u

    One big difference between both is that we can run userland assembly on Userland setup, which is easier to get running and debug.

    -

    In particular, most userland assembly examples link to the C standard library, see: Section 23.5, “Userland assembly C standard library”.

    +

    In particular, most userland assembly examples link to the C standard library, see: Section 27.5, “Userland assembly C standard library”.

    Userland assembly is generally simpler, and a pre-requisite for Baremetal setup.

    @@ -34487,7 +34851,7 @@ When instructions do not interpret this operand encoding as the zero register, u
    -

    23.5. Userland assembly C standard library

    +

    27.5. Userland assembly C standard library

    All examples except the Freestanding programs link to the C standard library.

    @@ -34520,7 +34884,7 @@ When instructions do not interpret this operand encoding as the zero register, u
    -

    23.5.1. Freestanding programs

    +

    27.5.1. Freestanding programs

    Unlike most our other assembly examples, which use the C standard library for portability, examples under freestanding/ directories don’t link to the C standard library:

    @@ -34575,7 +34939,7 @@ When instructions do not interpret this operand encoding as the zero register, u

    This is analogous to step debugging baremetal examples.

    -
    23.5.1.1. nostartfiles programs
    +
    27.5.1.1. nostartfiles programs

    Assembly examples under nostartfiles directories can use the standard library, but they don’t use the pre-main boilerplate and start directly at our explicitly given _start:

    @@ -34658,7 +35022,7 @@ Is it any easy to determine which functions I can use or not, in case there are
    -

    23.6. GCC inline assembly

    +

    27.6. GCC inline assembly

    Examples under arch/<arch>/c/ directories show to how use inline assembly from higher level languages such as C:

    @@ -34721,7 +35085,7 @@ Is it any easy to determine which functions I can use or not, in case there are
    -

    23.6.1. GCC inline assembly register variables

    +

    27.6.1. GCC inline assembly register variables

    Used notably in some of the Linux system calls setups:

    @@ -34745,14 +35109,14 @@ Is it any easy to determine which functions I can use or not, in case there are

    In arm, it is the only way to achieve this effect: https://stackoverflow.com/questions/10831792/how-to-use-specific-register-in-arm-inline-assembler

    -

    This feature notably useful for making system calls from C, see: Section 23.7, “Linux system calls”.

    +

    This feature notably useful for making system calls from C, see: Section 27.7, “Linux system calls”.

    Documentation: https://gcc.gnu.org/onlinedocs/gcc-4.4.2/gcc/Explicit-Reg-Vars.html

    -

    23.6.2. GCC inline assembly scratch registers

    +

    27.6.2. GCC inline assembly scratch registers

    How to use temporary registers in inline assembly:

    @@ -34778,7 +35142,7 @@ Is it any easy to determine which functions I can use or not, in case there are
    -

    23.6.3. GCC inline assembly early-clobbers

    +

    27.6.3. GCC inline assembly early-clobbers

    An example of using the & early-clobber modifier: link:userland/arch/aarch64/earlyclobber.c

    @@ -34790,7 +35154,7 @@ Is it any easy to determine which functions I can use or not, in case there are
    -

    23.6.4. GCC inline assembly floating point ARM

    +

    27.6.4. GCC inline assembly floating point ARM

    Not documented as of GCC 8.2, but possible: https://stackoverflow.com/questions/53960240/armv8-floating-point-output-inline-assembly

    @@ -34806,7 +35170,7 @@ Is it any easy to determine which functions I can use or not, in case there are
    -

    23.6.5. GCC intrinsics

    +

    27.6.5. GCC intrinsics

    Pre-existing C wrappers using inline assembly, this is what production programs should use instead of inline assembly for SIMD:

    @@ -34828,7 +35192,7 @@ Is it any easy to determine which functions I can use or not, in case there are
    -
    23.6.5.1. GCC x86 intrinsics
    +
    27.6.5.1. GCC x86 intrinsics

    Good official cheatsheet with all intrinsics and what they expand to: https://software.intel.com/sites/landingpage/IntrinsicsGuide

    @@ -34956,7 +35320,7 @@ zmmintrin.h AVX512
    -

    23.7. Linux system calls

    +

    27.7. Linux system calls

    The following Userland setup programs illustrate how to make system calls:

    @@ -35055,7 +35419,7 @@ zmmintrin.h AVX512
    -

    23.7.1. futex system call

    +

    27.7.1. futex system call

    This is how threads either:

    @@ -35117,7 +35481,7 @@ child after parent sleep
    -
    23.7.1.1. Userland mutex implementation
    +
    27.7.1.1. Userland mutex implementation

    The best article to understand spinlocks is: https://eli.thegreenplace.net/2018/basics-of-futexes/

    @@ -35127,7 +35491,7 @@ child after parent sleep
    -

    23.7.2. getcpu system call and the sched_getaffinity glibc wrapper

    +

    27.7.2. getcpu system call and the sched_getaffinity glibc wrapper

    Examples:

    @@ -35202,7 +35566,7 @@ child after parent sleep
    -

    23.8. Linux calling conventions

    +

    27.8. Linux calling conventions

    A summary of results is shown at: Table 3, “Summary of Linux calling conventions for several architectures”.

    @@ -35244,7 +35608,7 @@ child after parent sleep
    -

    23.8.1. x86_64 calling convention

    +

    27.8.1. x86_64 calling convention

    Examples:

    @@ -35273,7 +35637,7 @@ child after parent sleep
    -

    23.8.2. ARM calling convention

    +

    27.8.2. ARM calling convention

    Call C standard library functions from assembly and vice versa.

    @@ -35335,7 +35699,7 @@ child after parent sleep
    -

    23.9. GNU GAS assembler

    +

    27.9. GNU GAS assembler

    GNU GAS is the default assembler used by GDB, and therefore it completely dominates in Linux.

    @@ -35343,7 +35707,7 @@ child after parent sleep

    The Linux kernel in particular uses GNU GAS assembly extensively for the arch specific parts under arch/.

    -

    23.9.1. GNU GAS assembler comments

    +

    27.9.1. GNU GAS assembler comments

    In this tutorial, we use exclusively C Preprocessor /**/ comments because:

    @@ -35378,7 +35742,7 @@ child after parent sleep
    -

    23.9.2. GNU GAS assembler immediates

    +

    27.9.2. GNU GAS assembler immediates

    Summary:

    @@ -35410,7 +35774,7 @@ child after parent sleep
    -

    23.9.3. GNU GAS assembler data sizes

    +

    27.9.3. GNU GAS assembler data sizes

    Let’s see how many bytes go into each data type:

    @@ -35502,9 +35866,9 @@ child after parent sleep
    -
    23.9.3.1. GNU GAS assembler ARM specifics
    +
    27.9.3.1. GNU GAS assembler ARM specifics
    -
    23.9.3.1.1. GNU GAS assembler ARM unified syntax
    +
    27.9.3.1.1. GNU GAS assembler ARM unified syntax

    There are two types of ARMv7 assemblies:

    @@ -35549,14 +35913,14 @@ child after parent sleep
  • -

    cannot have implicit destination with shift, see: Section 25.4.4.1, “ARM shift suffixes”

    +

    cannot have implicit destination with shift, see: Section 29.4.4.1, “ARM shift suffixes”

    • -
    23.9.3.2. GNU GAS assembler ARM .n and .w suffixes
    +
    27.9.3.2. GNU GAS assembler ARM .n and .w suffixes

    When reading disassembly, many instructions have either a .n or .w suffix.

    @@ -35569,7 +35933,7 @@ child after parent sleep
    -

    23.9.4. GNU GAS assembler char literals

    +

    27.9.4. GNU GAS assembler char literals

    userland/arch/x86_64/char_literals.S

    @@ -35590,14 +35954,14 @@ child after parent sleep
    -

    23.10. NOP instructions

    +

    27.10. NOP instructions

    @@ -35614,13 +35978,13 @@ child after parent sleep
    -

    24. x86 userland assembly

    +

    28. x86 userland assembly

    -

    Arch agnostic infrastructure getting started at: Section 23, “Userland assembly”.

    +

    Arch agnostic infrastructure getting started at: Section 27, “Userland assembly”.

    -

    24.1. x86 registers

    +

    28.1. x86 registers

    link:userland/arch/x86_64/registers.S

    @@ -35671,7 +36035,7 @@ child after parent sleep
    -

    24.1.1. x86 FLAGS registers

    +

    28.1.1. x86 FLAGS registers

    https://en.wikipedia.org/wiki/FLAGS_register

    @@ -35681,7 +36045,7 @@ child after parent sleep
    -

    24.2. x86 addressing modes

    +

    28.2. x86 addressing modes

    Example: userland/arch/x86_64/address_modes.S

    @@ -35754,7 +36118,7 @@ child after parent sleep
    -

    24.3. x86 data transfer instructions

    +

    28.3. x86 data transfer instructions

    5.1.1 "Data Transfer Instructions"

    @@ -35785,7 +36149,7 @@ child after parent sleep
    -

    24.3.1. x86 exchange instructions

    +

    28.3.1. x86 exchange instructions

    Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 7.3.1.2 "Exchange Instructions":

    @@ -35803,7 +36167,7 @@ child after parent sleep

    TODO: concrete multi-thread GCC inline assembly examples of how all those instructions are normally used as synchronization primitives.

    -
    24.3.1.1. x86 CMPXCHG instruction
    +
    28.3.1.1. x86 CMPXCHG instruction

    userland/arch/x86_64/cmpxchg.S

    @@ -35827,7 +36191,7 @@ child after parent sleep
    -

    24.3.2. x86 PUSH and POP instructions

    +

    28.3.2. x86 PUSH and POP instructions

    userland/arch/x86_64/push.S

    @@ -35854,7 +36218,7 @@ add $8, %rsp
    -

    24.3.3. x86 CQTO and CLTQ instructions

    +

    28.3.3. x86 CQTO and CLTQ instructions

    Examples:

    @@ -35955,7 +36319,7 @@ add $8, %rsp
    -

    24.3.4. x86 CMOVcc instructions

    +

    28.3.4. x86 CMOVcc instructions

    -

    It is interesting to compare this with ARMv7 conditional execution: which is available for all instructions, as shown at: Section 25.2.5, “ARM conditional execution”.

    +

    It is interesting to compare this with ARMv7 conditional execution: which is available for all instructions, as shown at: Section 29.2.5, “ARM conditional execution”.

    -

    24.4. x86 binary arithmetic instructions

    +

    28.4. x86 binary arithmetic instructions

    Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.1.2 "Binary Arithmetic Instructions":

    @@ -36081,7 +36445,7 @@ add $8, %rsp
    -

    24.5. x86 logical instructions

    +

    28.5. x86 logical instructions

    Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.1.4 "Logical Instructions"

    @@ -36103,7 +36467,7 @@ add $8, %rsp
    -

    24.6. x86 shift and rotate instructions

    +

    28.6. x86 shift and rotate instructions

    Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.1.5 "Shift and Rotate Instructions"

    @@ -36155,7 +36519,7 @@ add $8, %rsp
    -

    24.7. x86 bit and byte instructions

    +

    28.7. x86 bit and byte instructions

    Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.1.6 "Bit and Byte Instructions"

    @@ -36214,7 +36578,7 @@ add $8, %rsp
    -

    24.8. x86 control transfer instructions

    +

    28.8. x86 control transfer instructions

    Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.1.7 "Control Transfer Instructions"

    @@ -36233,7 +36597,7 @@ add $8, %rsp
    -

    24.8.1. x86 Jcc instructions

    +

    28.8.1. x86 Jcc instructions

    userland/arch/x86_64/jcc.S

    @@ -36307,7 +36671,7 @@ add $8, %rsp
    -

    24.8.2. x86 LOOP instruction

    +

    28.8.2. x86 LOOP instruction

    userland/arch/x86_64/loop.S

    @@ -36316,7 +36680,7 @@ add $8, %rsp
    -

    24.8.3. x86 string instructions

    +

    28.8.3. x86 string instructions

    Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.1.8 "String Instructions"

    @@ -36369,7 +36733,7 @@ add $8, %rsp

    However, as computer architecture evolved, those instructions might not offer considerable speedups anymore, and modern glibc such as 2.29 just uses x86 SIMD operations instead:, see also: https://stackoverflow.com/questions/33480999/how-can-the-rep-stosb-instruction-execute-faster-than-the-equivalent-loop

    -
    24.8.3.1. x86 REP prefix
    +
    28.8.3.1. x86 REP prefix

    Example: userland/arch/x86_64/rep.S

    @@ -36408,7 +36772,7 @@ add $8, %rsp
    -

    24.8.4. x86 ENTER and LEAVE instructions

    +

    28.8.4. x86 ENTER and LEAVE instructions

    userland/arch/x86_64/enter.S

    @@ -36459,16 +36823,16 @@ pop %rbp
    -

    24.9. x86 miscellaneous instructions

    +

    28.9. x86 miscellaneous instructions

    Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.1.13 "Miscellaneous Instructions"

    -

    NOP: Section 23.10, “NOP instructions”

    +

    NOP: Section 27.10, “NOP instructions”

    -

    24.10. x86 random number generator instructions

    +

    28.10. x86 random number generator instructions

    Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.1.15 Random Number Generator Instructions

    @@ -36491,7 +36855,7 @@ pop %rbp

    RDRAND sets the carry flag when data is ready so we must loop if the carry flag isn’t set.

    -

    24.10.1. x86 CPUID instruction

    +

    28.10.1. x86 CPUID instruction

    Example: userland/arch/x86_64/cpuid.S

    @@ -36562,7 +36926,7 @@ pop %rbp
    -

    24.11. x86 x87 FPU instructions

    +

    28.11. x86 x87 FPU instructions

    Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.2 "X87 FPU INSTRUCTIONS"

    @@ -36655,7 +37019,7 @@ pop %rbp
    -

    24.11.1. x86 x87 FPU vs SIMD

    +

    28.11.1. x86 x87 FPU vs SIMD

    https://stackoverflow.com/questions/1844669/benefits-of-x87-over-sse

    @@ -36694,9 +37058,9 @@ pop %rbp
    -

    24.12. x86 SIMD

    +

    28.12. x86 SIMD

    -

    Parent section: Section 23.3, “SIMD assembly”

    +

    Parent section: Section 27.3, “SIMD assembly”

    History:

    @@ -36730,12 +37094,12 @@ pop %rbp
    -

    24.12.1. x86 SSE instructions

    +

    28.12.1. x86 SSE instructions

    Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.5 "SSE INSTRUCTIONS"

    -
    24.12.1.1. x86 SSE data transfer instructions
    +
    28.12.1.1. x86 SSE data transfer instructions

    Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.5.1.1 "SSE Data Transfer Instructions"

    @@ -36754,7 +37118,7 @@ pop %rbp
    -
    24.12.1.2. x86 SSE packed arithmetic instructions
    +
    28.12.1.2. x86 SSE packed arithmetic instructions

    Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.5.1.2 "SSE Packed Arithmetic Instructions"

    @@ -36767,14 +37131,14 @@ pop %rbp
    -
    24.12.1.3. x86 SSE conversion instructions
    +
    28.12.1.3. x86 SSE conversion instructions

    Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.5.1.6 "SSE Conversion Instructions"

    -

    24.12.2. x86 SSE2 instructions

    +

    28.12.2. x86 SSE2 instructions

    Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.6 "SSE2 INSTRUCTIONS"

    @@ -36786,7 +37150,7 @@ pop %rbp
    -
    24.12.2.1. x86 PADDQ instruction
    +
    28.12.2.1. x86 PADDQ instruction

    userland/arch/x86_64/paddq.S: PADDQ, PADDL, PADDW, PADDB

    @@ -36796,7 +37160,7 @@ pop %rbp
    -

    24.12.3. x86 fused multiply add (FMA)

    +

    28.12.3. x86 fused multiply add (FMA)

    Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.15 "FUSED-MULTIPLY-ADD (FMA)"

    @@ -36816,12 +37180,12 @@ pop %rbp
    -

    24.13. x86 system instructions

    +

    28.13. x86 system instructions

    Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1 5.20 "SYSTEM INSTRUCTIONS"

    -

    24.13.1. x86 RDTSC instruction

    +

    28.13.1. x86 RDTSC instruction

    Sources:

    @@ -36895,7 +37259,7 @@ pop %rbp
    -
    24.13.1.1. x86 RDTSCP instruction
    +
    28.13.1.1. x86 RDTSCP instruction

    RDTSCP is like RDTSP, but it also stores the CPU ID into ECX: this is convenient because the value of RDTSC depends on which core we are currently on, so you often also want the core ID when you want the RDTSC.

    @@ -36938,7 +37302,7 @@ taskset -c 1 ./userland/arch/x86_64/rdtscp.out | tail -n 1
    -
    24.13.1.2. ARM PMCCNTR register
    +
    28.13.1.2. ARM PMCCNTR register

    TODO We didn’t manage to find a working ARM analogue to x86 RDTSC instruction: kernel_modules/pmccntr.c is oopsing, and even it if weren’t, it likely won’t give the cycle count since boot since it needs to be activate before it starts counting anything:

    @@ -36959,9 +37323,9 @@ taskset -c 1 ./userland/arch/x86_64/rdtscp.out | tail -n 1
    -

    24.14. x86 thread synchronization primitives

    +

    28.14. x86 thread synchronization primitives

    -

    24.14.1. x86 LOCK prefix

    +

    28.14.1. x86 LOCK prefix

    Inline assembly example at: userland/cpp/atomic/x86_64_lock_inc.cpp, see also: atomic.cpp.

    @@ -36987,11 +37351,11 @@ taskset -c 1 ./userland/arch/x86_64/rdtscp.out | tail -n 1
    -

    24.15. x86 assembly bibliography

    +

    28.15. x86 assembly bibliography

    -

    24.15.1. x86 official bibliography

    +

    28.15.1. x86 official bibliography

    -
    24.15.1.1. Intel 64 and IA-32 Architectures Software Developer’s Manuals
    +
    28.15.1.1. Intel 64 and IA-32 Architectures Software Developer’s Manuals

    We are using the May 2019 version unless otherwise noted.

    @@ -37008,25 +37372,25 @@ taskset -c 1 ./userland/arch/x86_64/rdtscp.out | tail -n 1

    Also I can’t find older versions on the website easily, so I just web archive everything.

    -
    24.15.1.1.1. Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1
    +
    28.15.1.1.1. Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 1

    Userland basics: http://web.archive.org/web/20190606075544/https://software.intel.com/sites/default/files/managed/a4/60/253665-sdm-vol-1.pdf

    -
    24.15.1.1.2. Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 2
    +
    28.15.1.1.2. Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 2

    Instruction list: http://web.archive.org/web/20190606075330/https://software.intel.com/sites/default/files/managed/a4/60/325383-sdm-vol-2abcd.pdf

    -
    24.15.1.1.3. Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 3
    +
    28.15.1.1.3. Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 3

    Kernel land: http://web.archive.org/web/20190606075534/https://software.intel.com/sites/default/files/managed/a4/60/325384-sdm-vol-3abcd.pdf

    -
    24.15.1.1.4. Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 4
    +
    28.15.1.1.4. Intel 64 and IA-32 Architectures Software Developer’s Manuals Volume 4

    Model specific extensions: http://web.archive.org/web/20190606075325/https://software.intel.com/sites/default/files/managed/22/0d/335592-sdm-vol-4.pdf

    @@ -37037,10 +37401,10 @@ taskset -c 1 ./userland/arch/x86_64/rdtscp.out | tail -n 1
    -

    25. ARM userland assembly

    +

    29. ARM userland assembly

    -

    Arch general getting started at: Section 23, “Userland assembly”.

    +

    Arch general getting started at: Section 27, “Userland assembly”.

    Instructions here loosely grouped based on that of the ARMv7 architecture reference manual Chapter A4 "The Instruction Sets".

    @@ -37049,7 +37413,7 @@ taskset -c 1 ./userland/arch/x86_64/rdtscp.out | tail -n 1

    We cover here mostly ARMv7, and then treat aarch64 differentially, since much of the ARMv7 userland is the same in aarch32.

    -

    25.1. Introduction to the ARM architecture

    +

    29.1. Introduction to the ARM architecture

    The ARM architecture is has been used on the vast majority of mobile phones in the 2010’s, and on a large fraction of micro controllers.

    @@ -37066,7 +37430,7 @@ taskset -c 1 ./userland/arch/x86_64/rdtscp.out | tail -n 1

    ARM Holdings was bought by the Japanese giant SoftBank in 2016.

    -

    25.1.1. ARMv8 vs ARMv7 vs AArch64 vs AArch32

    +

    29.1.1. ARMv8 vs ARMv7 vs AArch64 vs AArch32

    ARMv7 is the older architecture described at: ARMv7 architecture reference manual.

    @@ -37122,7 +37486,7 @@ taskset -c 1 ./userland/arch/x86_64/rdtscp.out | tail -n 1

    They are described at: ARMv8 architecture reference manual A1.7 "ARMv8 architecture extensions".

    -
    25.1.1.1. AArch32
    +
    29.1.1.1. AArch32

    32-bit mode of operation of ARMv8.

    @@ -37154,7 +37518,7 @@ taskset -c 1 ./userland/arch/x86_64/rdtscp.out | tail -n 1
    -
    25.1.1.2. AArch32 vs AArch64
    +
    29.1.1.2. AArch32 vs AArch64

    A great summary of differences can be found at: https://en.wikipedia.org/wiki/ARM_architecture#AArch64_features

    @@ -37164,17 +37528,17 @@ taskset -c 1 ./userland/arch/x86_64/rdtscp.out | tail -n 1
    -

    25.1.2. Free ARM implementations

    +

    29.1.2. Free ARM implementations

    The ARM instruction set is itself protected by patents / copyright / whatever, and you have to pay ARM Holdings a licence to implement it, even if you are creating your own custom Verilog code.

    @@ -37213,7 +37577,7 @@ Bibliography: -

    25.1.3. ARM instruction encodings

    +

    29.1.3. ARM instruction encodings

    Understanding the basics of instruction encodings is fundamental to help you to remember what instructions do and why some things are possible or not, notably the ARM LDR pseudo-instruction and the ADRP instruction.

    @@ -37325,7 +37689,7 @@ Bibliography: -
    25.1.3.1. ARM Thumb encoding
    +
    29.1.3.1. ARM Thumb encoding

    Thumb examples are available at:

    @@ -37384,7 +37748,7 @@ Bibliography: -
    25.1.3.2. ARM big endian mode
    +
    29.1.3.2. ARM big endian mode

    ARM can switch between big and little endian mode on the fly!

    @@ -37480,9 +37844,9 @@ Bibliography: -

    25.2. ARM branch instructions

    +

    29.2. ARM branch instructions

    -

    25.2.1. ARM B instruction

    +

    29.2.1. ARM B instruction

    Unconditional branch.

    @@ -37500,7 +37864,7 @@ Bibliography: -

    25.2.2. ARM BEQ instruction

    +

    29.2.2. ARM BEQ instruction

    Branch if equal based on the status registers.

    @@ -37544,7 +37908,7 @@ Bibliography: -

    25.2.3. ARM BL instruction

    +

    29.2.3. ARM BL instruction

    Branch with link, i.e. branch and store the return address on the RL register.

    @@ -37558,13 +37922,13 @@ Bibliography: -
    25.2.3.1. ARM BX instruction
    +
    29.2.3.1. ARM BX instruction
    -

    See: Section 25.1.3.1, “ARM Thumb encoding”

    +

    See: Section 29.1.3.1, “ARM Thumb encoding”

    -
    25.2.3.2. ARMv8 aarch64 ret instruction
    +
    29.2.3.2. ARMv8 aarch64 ret instruction

    Example: userland/arch/aarch64/ret.S

    @@ -37597,7 +37961,7 @@ Bibliography: -

    25.2.4. ARM CBZ instruction

    +

    29.2.4. ARM CBZ instruction

    Compare and branch if zero.

    @@ -37612,7 +37976,7 @@ Bibliography: -

    25.2.5. ARM conditional execution

    +

    29.2.5. ARM conditional execution

    Weirdly, ARM B instruction and family are not the only instructions that can execute conditionally on the flags: the same also applies to most instructions, e.g. ADD.

    @@ -37628,7 +37992,7 @@ Bibliography: -

    25.3. ARM load and store instructions

    +

    29.3. ARM load and store instructions

    In ARM, there are only two instruction families that do memory access:

    @@ -37652,9 +38016,9 @@ Bibliography: Load/store architecture.

    -

    25.3.1. ARM LDR instruction

    +

    29.3.1. ARM LDR instruction

    -
    25.3.1.1. ARM LDR pseudo-instruction
    +
    29.3.1.1. ARM LDR pseudo-instruction

    LDR can be either a regular instruction that loads stuff into memory, or also a pseudo-instruction (assembler magic): http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0041c/Babbfdih.html

    @@ -37688,7 +38052,7 @@ Bibliography: -
    25.3.1.2. ARM addressing modes
    +
    29.3.1.2. ARM addressing modes

    Example: userland/arch/arm/address_modes.S

    @@ -37759,7 +38123,7 @@ Bibliography: ARMv8 architecture reference manual: C1.3.3 "Load/Store addressing modes"

    -
    25.3.1.2.1. ARM loop over array
    +
    29.3.1.2.1. ARM loop over array

    As an application of the post-indexed addressing mode, let’s increment an array.

    @@ -37769,7 +38133,7 @@ Bibliography: -
    25.3.1.3. ARM LDRH and LDRB instructions
    +
    29.3.1.3. ARM LDRH and LDRB instructions

    There are LDR variants that load less than full 4 bytes:

    @@ -37796,7 +38160,7 @@ Bibliography: -

    25.3.2. ARM STR instruction

    +

    29.3.2. ARM STR instruction

    Store from memory into registers.

    @@ -37807,7 +38171,7 @@ Bibliography: ARM LDR instruction also applies here so we won’t go into much detail.

    -
    25.3.2.1. ARMv8 aarch64 STR instruction
    +
    29.3.2.1. ARMv8 aarch64 STR instruction

    PC-relative STR is not possible in aarch64.

    @@ -37825,7 +38189,7 @@ Bibliography: -
    25.3.2.2. ARMv8 aarch64 LDP and STP instructions
    +
    29.3.2.2. ARMv8 aarch64 LDP and STP instructions

    Push a pair of registers to the stack.

    @@ -37833,7 +38197,7 @@ Bibliography: lkmc/aarch64.h since it is the main way to restore register state.

    -
    25.3.2.2.1. ARMV8 aarch64 stack alignment
    +
    29.3.2.2.1. ARMV8 aarch64 stack alignment

    In ARMv8, the stack can be enforced to 16-byte alignment.

    @@ -37880,7 +38244,7 @@ Bibliography: -

    25.3.3. ARM LDMIA instruction

    +

    29.3.3. ARM LDMIA instruction

    Pop values form stack into the register and optionally update the address register.

    @@ -37930,7 +38294,7 @@ ldmia sp!, reglist
    -

    25.4. ARM data processing instructions

    +

    29.4. ARM data processing instructions

    Arithmetic:

    @@ -37954,7 +38318,7 @@ ldmia sp!, reglist
    -

    25.4.1. ARM CSET instruction

    +

    29.4.1. ARM CSET instruction

    Example: userland/arch/aarch64/cset.S

    @@ -37966,7 +38330,7 @@ ldmia sp!, reglist
    -

    25.4.2. ARM bitwise instructions

    +

    29.4.2. ARM bitwise instructions

    • @@ -37984,7 +38348,7 @@ ldmia sp!, reglist
    -
    25.4.2.1. ARM BIC instruction
    +
    29.4.2.1. ARM BIC instruction

    Bitwise Bit Clear: clear some bits.

    @@ -37998,7 +38362,7 @@ ldmia sp!, reglist
    -
    25.4.2.2. ARM UBFM instruction
    +
    29.4.2.2. ARM UBFM instruction

    Unsigned Bitfield Move.

    @@ -38016,7 +38380,7 @@ ldmia sp!, reglist

    TODO: explain full behaviour. Very complicated. Has several simpler to understand aliases.

    -
    25.4.2.2.1. ARM UBFX instruction
    +
    29.4.2.2.1. ARM UBFX instruction

    Alias for:

    @@ -38050,12 +38414,12 @@ ldmia sp!, reglist
    -
    25.4.2.3. ARM BFM instruction
    +
    29.4.2.3. ARM BFM instruction

    TODO: explain. Similar to UBFM but leave untouched bits unmodified.

    -
    25.4.2.3.1. ARM BFI instruction
    +
    29.4.2.3.1. ARM BFI instruction

    Examples:

    @@ -38086,12 +38450,12 @@ ldmia sp!, reglist
    -

    25.4.3. ARM MOV instruction

    +

    29.4.3. ARM MOV instruction

    Move an immediate to a register, or a register to another register.

    -

    Cannot load from or to memory, since only the LDR and STR instruction families can do that in ARM as mentioned at: Section 25.3, “ARM load and store instructions”.

    +

    Cannot load from or to memory, since only the LDR and STR instruction families can do that in ARM as mentioned at: Section 29.3, “ARM load and store instructions”.

    Example: userland/arch/arm/mov.S

    @@ -38152,7 +38516,7 @@ ldmia sp!, reglist

    Assemblers however support magic memory allocations which may hide what is truly going on: https://stackoverflow.com/questions/14046686/why-use-ldr-over-mov-or-vice-versa-in-arm-assembly Always ask your friendly disassembly for a good confirmation.

    -
    25.4.3.1. ARM movw and movt instructions
    +
    29.4.3.1. ARM movw and movt instructions

    Set the higher or lower 16 bits of a register to an immediate in one go.

    @@ -38164,7 +38528,7 @@ ldmia sp!, reglist
    -
    25.4.3.2. ARMv8 aarch64 movk instruction
    +
    29.4.3.2. ARMv8 aarch64 movk instruction

    Fill a 64 bit register with 4 16-bit instructions one at a time.

    @@ -38179,7 +38543,7 @@ ldmia sp!, reglist
    -
    25.4.3.3. ARMv8 aarch64 movn instruction
    +
    29.4.3.3. ARMv8 aarch64 movn instruction

    Set 16-bits negated and the rest to 1.

    @@ -38189,9 +38553,9 @@ ldmia sp!, reglist
    -

    25.4.4. ARM data processing instruction suffixes

    +

    29.4.4. ARM data processing instruction suffixes

    -
    25.4.4.1. ARM shift suffixes
    +
    29.4.4.1. ARM shift suffixes

    Most data processing instructions can also optionally shift the second register operand.

    @@ -38219,7 +38583,7 @@ ldmia sp!, reglist
    -
    25.4.4.2. ARM S suffix
    +
    29.4.4.2. ARM S suffix

    Example: userland/arch/arm/s_suffix.S

    @@ -38235,7 +38599,7 @@ ldmia sp!, reglist
    -

    25.4.5. ARM ADR instruction

    +

    29.4.5. ARM ADR instruction

    Similar rationale to the ARM LDR pseudo-instruction, allowing to easily store a PC-relative reachable address into a register in one go, to overcome the 4-byte fixed instruction size.

    @@ -38259,19 +38623,19 @@ ldmia sp!, reglist

    More details: https://stackoverflow.com/questions/41906688/what-are-the-semantics-of-adrp-and-adrl-instructions-in-arm-assembly/54042899#54042899

    -
    25.4.5.1. ARM ADRL instruction
    +
    29.4.5.1. ARM ADRL instruction
    -

    See: Section 25.4.5, “ARM ADR instruction”.

    +

    See: Section 29.4.5, “ARM ADR instruction”.

    -

    25.5. ARM miscellaneous instructions

    +

    29.5. ARM miscellaneous instructions

    -

    25.5.1. ARM NOP instruction

    +

    29.5.1. ARM NOP instruction

    -

    Parent section: Section 23.10, “NOP instructions”

    +

    Parent section: Section 27.10, “NOP instructions”

    There are a few different ways to encode NOP, notably MOV a register into itself, and a dedicated miscellaneous instruction.

    @@ -38292,7 +38656,7 @@ ldmia sp!, reglist
    -

    25.5.2. ARM UDF instruction

    +

    29.5.2. ARM UDF instruction

    Guaranteed undefined! Therefore raise illegal instruction signal. Used by GCC __builtin_trap apparently: https://stackoverflow.com/questions/16081618/programmatically-cause-undefined-instruction-exception

    @@ -38311,7 +38675,7 @@ ldmia sp!, reglist
    -

    25.5.3. ARM system register instructions

    +

    29.5.3. ARM system register instructions

    Examples of using them can be found at: dump_regs

    @@ -38418,7 +38782,7 @@ dc isw
    -
    25.5.3.1. ARM system register encodings
    +
    29.5.3.1. ARM system register encodings

    Each aarch64 system register is specified in the encoding of ARM system register instructions by 5 integer numbers:

    @@ -38464,12 +38828,12 @@ LKMC_DUMP_SYSTEM_REGS_PRINTF("ID_ISAR6_EL1 0x%" PRIX32 "\n", id_isar6_el1);
    -

    25.6. ARM SIMD

    +

    29.6. ARM SIMD

    -

    Parent section: Section 23.3, “SIMD assembly”

    +

    Parent section: Section 27.3, “SIMD assembly”

    -

    25.6.1. ARM VFP

    +

    29.6.1. ARM VFP

    The name for the ARMv7 and AArch32 floating point and SIMD instructions / registers.

    @@ -38515,7 +38879,7 @@ LKMC_DUMP_SYSTEM_REGS_PRINTF("ID_ISAR6_EL1 0x%" PRIX32 "\n", id_isar6_el1);
    -
    25.6.1.1. ARM VFP registers
    +
    29.6.1.1. ARM VFP registers

    TODO example

    @@ -38551,20 +38915,20 @@ LKMC_DUMP_SYSTEM_REGS_PRINTF("ID_ISAR6_EL1 0x%" PRIX32 "\n", id_isar6_el1);
    -
    25.6.1.2. ARM VADD instruction
    +
    29.6.1.2. ARM VADD instruction
    -
    25.6.1.3. ARM VCVT instruction
    +
    29.6.1.3. ARM VCVT instruction

    Example: userland/arch/arm/vcvt.S

    @@ -38593,7 +38957,7 @@ LKMC_DUMP_SYSTEM_REGS_PRINTF("ID_ISAR6_EL1 0x%" PRIX32 "\n", id_isar6_el1);
    -
    25.6.1.3.1. ARM VCVTR instruction
    +
    29.6.1.3.1. ARM VCVTR instruction

    Example: userland/arch/arm/vcvtr.S

    @@ -38611,7 +38975,7 @@ LKMC_DUMP_SYSTEM_REGS_PRINTF("ID_ISAR6_EL1 0x%" PRIX32 "\n", id_isar6_el1);
    -
    25.6.1.3.2. ARMv8 AArch32 VCVTA instruction
    +
    29.6.1.3.2. ARMv8 AArch32 VCVTA instruction

    Example: userland/arch/arm/vcvt.S

    @@ -38631,7 +38995,7 @@ LKMC_DUMP_SYSTEM_REGS_PRINTF("ID_ISAR6_EL1 0x%" PRIX32 "\n", id_isar6_el1);
    -

    25.6.2. ARMv8 Advanced SIMD and floating-point support

    +

    29.6.2. ARMv8 Advanced SIMD and floating-point support

    The ARMv8 architecture reference manual specifies floating point and SIMD support in the main architecture at A1.5 "Advanced SIMD and floating-point support".

    @@ -38639,13 +39003,13 @@ LKMC_DUMP_SYSTEM_REGS_PRINTF("ID_ISAR6_EL1 0x%" PRIX32 "\n", id_isar6_el1);The feature is often refered to simply as "SIMD&FP" throughout the manual.

    -

    The Linux kernel shows /proc/cpuinfo compatibility as neon, which is yet another intermediate name that came up at some point, see: Section 25.6.2.2, “ARM NEON”.

    +

    The Linux kernel shows /proc/cpuinfo compatibility as neon, which is yet another intermediate name that came up at some point, see: Section 29.6.2.2, “ARM NEON”.

    Vs ARM VFP: https://stackoverflow.com/questions/4097034/arm-cortex-a8-whats-the-difference-between-vfp-and-neon

    -
    25.6.2.1. ARMv8 floating point availability
    +
    29.6.2.1. ARMv8 floating point availability

    Support is semi-mandatory. ARMv8 architecture reference manual A1.5 "Advanced SIMD and floating-point support":

    @@ -38682,7 +39046,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -
    25.6.2.2. ARM NEON
    +
    29.6.2.2. ARM NEON

    Just an informal name for the "Advanced SIMD instructions"? Very confusing.

    @@ -38709,7 +39073,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -

    25.6.3. ARMv8 AArch64 floating point registers

    +

    29.6.3. ARMv8 AArch64 floating point registers

    TODO example.

    @@ -38764,7 +39128,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -
    25.6.3.1. ARMv8 aarch64 add vector instruction
    +
    29.6.3.1. ARMv8 aarch64 add vector instruction

    userland/arch/aarch64/add_vector.S

    @@ -38773,21 +39137,21 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -
    25.6.3.2. ARMv8 aarch64 FADD instruction
    +
    29.6.3.2. ARMv8 aarch64 FADD instruction
    -
    25.6.3.2.1. ARM FADD vs VADD
    +
    29.6.3.2.1. ARM FADD vs VADD
    -

    It is very confusing, but FADDS and FADDD in Aarch32 are pre-UAL for vadd.f32 and vadd.f64 which we use in this tutorial, see: Section 25.6.1.2, “ARM VADD instruction”

    +

    It is very confusing, but FADDS and FADDD in Aarch32 are pre-UAL for vadd.f32 and vadd.f64 which we use in this tutorial, see: Section 29.6.1.2, “ARM VADD instruction”

    The same goes for most ARMv7 mnemonics: f* is old, and v* is the newer better syntax.

    @@ -38799,12 +39163,12 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    Also keep in mind that fused multiply add is FMADD.

    -

    Examples at: Section 23.3, “SIMD assembly”

    +

    Examples at: Section 27.3, “SIMD assembly”

    -
    25.6.3.3. ARMv8 aarch64 LD2 instruction
    +
    29.6.3.3. ARMv8 aarch64 LD2 instruction

    Example: userland/arch/aarch64/ld2.S

    @@ -38820,7 +39184,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -

    25.6.4. ARM SIMD bibliography

    +

    29.6.4. ARM SIMD bibliography

    -

    25.6.5. ARM SVE

    +

    29.6.5. ARM SVE

    Scalable Vector Extension.

    @@ -38898,7 +39262,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    Using SVE normally requires setting the CPACR_EL1.FPEN and ZEN bits, which as as of lkmc 29fd625f3fda79f5e0ee6cac43517ba74340d513 + 1 we also enable in our Baremetal bootloaders, see also: aarch64 baremetal NEON setup.

    -
    25.6.5.1. ARM SVE VADDL instruction
    +
    29.6.5.1. ARM SVE VADDL instruction

    Get the SVE vector length. The following programs do that and print it to stdout:

    @@ -38914,7 +39278,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -
    25.6.5.2. Change ARM SVE vector length in emulators
    +
    29.6.5.2. Change ARM SVE vector length in emulators

    gem5 covered at: https://stackoverflow.com/questions/57692765/how-to-change-the-gem5-arm-sve-vector-length

    @@ -38951,7 +39315,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -
    25.6.5.3. SVE bibliography
    +
    29.6.5.3. SVE bibliography
    -
    25.6.5.3.1. SVE spec
    +
    29.6.5.3.1. SVE spec

    ARMv8 architecture reference manual A1.7 "ARMv8 architecture extensions" says:

    @@ -38991,12 +39355,12 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -

    25.7. ARM thread synchronization primitives

    +

    29.7. ARM thread synchronization primitives

    Parent section: Userland multithreading.

    -

    25.7.1. ARM LDXR and STXR instructions

    +

    29.7.1. ARM LDXR and STXR instructions

    Parent section: atomic.cpp

    @@ -39046,7 +39410,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -

    25.7.2. ARM Large System Extensions (LSE)

    +

    29.7.2. ARM Large System Extensions (LSE)

    Set of atomic and synchronization primitives added in ARMv8.1 architecture extension.

    @@ -39073,9 +39437,9 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -

    25.8. ARMv8 architecture extensions

    +

    29.8. ARMv8 architecture extensions

    -

    25.8.1. ARMv8.1 architecture extension

    +

    29.8.1. ARMv8.1 architecture extension

    ARMv8 architecture reference manual db A1.7.3 "The ARMv8.1 architecture extension"

    @@ -39089,9 +39453,9 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -

    25.9. ARM assembly bibliography

    +

    29.9. ARM assembly bibliography

    -

    25.9.1. ARM non-official bibliography

    +

    29.9.1. ARM non-official bibliography

    Good getting started tutorials:

    @@ -39113,7 +39477,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -

    25.9.2. ARM official bibliography

    +

    29.9.2. ARM official bibliography

    The official manuals were stored in http://infocenter.arm.com but as of 2017 they started to slowly move to https://developer.arm.com.

    @@ -39127,7 +39491,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    Bibliography: https://www.quora.com/Where-can-I-find-the-official-documentation-of-ARM-instruction-set-architectures-ISAs

    -
    25.9.2.1. ARMv7 architecture reference manual
    +
    29.9.2.1. ARMv7 architecture reference manual

    https://developer.arm.com/products/architecture/a-profile/docs/ddi0406/latest/arm-architecture-reference-manual-armv7-a-and-armv7-r-edition

    @@ -39139,7 +39503,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -
    25.9.2.2. ARMv8 architecture reference manual
    +
    29.9.2.2. ARMv8 architecture reference manual

    https://static.docs.arm.com/ddi0487/ca/DDI0487C_a_armv8_arm.pdf

    @@ -39195,19 +39559,19 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -
    25.9.2.3. ARMv8 architecture reference manual db
    +
    29.9.2.3. ARMv8 architecture reference manual db

    https://static.docs.arm.com/ddi0487/db/DDI0487D_b_armv8_arm.pdf

    -
    25.9.2.4. ARMv8 architecture reference manual db
    +
    29.9.2.4. ARMv8 architecture reference manual db

    https://static.docs.arm.com/ddi0487/fa/DDI0487F_a_armv8_arm.pdf

    -
    25.9.2.5. Programmer’s Guide for ARMv8-A
    +
    29.9.2.5. Programmer’s Guide for ARMv8-A

    https://static.docs.arm.com/den0024/a/DEN0024A_v8_architecture_PG.pdf

    @@ -39222,7 +39586,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -
    25.9.2.6. Arm A64 Instruction Set Architecture: Future Architecture Technologies in the A architecture profile Documentation
    +
    29.9.2.6. Arm A64 Instruction Set Architecture: Future Architecture Technologies in the A architecture profile Documentation

    https://developer.arm.com/docs/ddi0602/b

    @@ -39231,7 +39595,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -
    25.9.2.7. ARM processor documentation
    +
    29.9.2.7. ARM processor documentation

    ARM also releases documentation specific to each given processor.

    @@ -39255,7 +39619,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -
    25.9.2.7.1. ARM Cortex-A15 MPCore Processor Technical Reference Manual r4p0
    +
    29.9.2.7.1. ARM Cortex-A15 MPCore Processor Technical Reference Manual r4p0

    http://infocenter.arm.com/help/topic/com.arm.doc.ddi0438i/DDI0438I_cortex_a15_r4p0_trm.pdf

    @@ -39265,13 +39629,13 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -
    25.9.2.8. Arm Cortex‑A77 Technical Reference Manual r1p1
    +
    29.9.2.8. Arm Cortex‑A77 Technical Reference Manual r1p1

    https://static.docs.arm.com/101111/0101/arm_cortex_a77_trm_101111_0101_04_en.pdf

    -
    25.9.2.9. Arm Cortex‑A77 Software Optimization Guide r1p1
    +
    29.9.2.9. Arm Cortex‑A77 Software Optimization Guide r1p1

    https://static.docs.arm.com/swog011050/c/Arm_Cortex-A77_Software_Optimization_Guide.pdf

    @@ -39281,7 +39645,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -

    26. ELF

    +

    30. ELF

    https://en.wikipedia.org/wiki/Executable_and_Linkable_Format

    @@ -39295,7 +39659,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -

    27. IEEE 754

    +

    31. IEEE 754

    https://en.wikipedia.org/wiki/IEEE_754

    @@ -39325,13 +39689,13 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -

    28. Baremetal

    +

    32. Baremetal

    Getting started at: Section 1.9, “Baremetal setup”

    -

    28.1. Baremetal GDB step debug

    +

    32.1. Baremetal GDB step debug

    GDB step debug works on baremetal exactly as it does on the Linux kernel, which is described at: Section 2, “GDB step debug”.

    @@ -39402,7 +39766,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    -

    28.2. Baremetal bootloaders

    +

    32.2. Baremetal bootloaders

    As can be seen from Baremetal GDB step debug, all examples under baremetal/, with the exception of baremetal/arch/<arch>/no_bootloader, start from our tiny bootloaders:

    @@ -39438,7 +39802,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    the stack pointer

  • -

    NEON: Section 28.11.2, “aarch64 baremetal NEON setup”

    +

    NEON: Section 32.11.2, “aarch64 baremetal NEON setup”

  • TODO: we don’t do this currently but maybe we should setup BSS

    @@ -39466,7 +39830,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.

    • -

    28.3. Baremetal linker script

    +

    32.3. Baremetal linker script

    For things to work in baremetal, we often have to layout memory in specific ways.

    @@ -39495,7 +39859,7 @@ lkmc_heap_top = .;
    -

    28.4. Baremetal command line arguments

    +

    32.4. Baremetal command line arguments

    QEMU and gem5 currently supports baremetal CLI arguments!

    @@ -39544,7 +39908,7 @@ cc

    It is worth noting that e.g. ARM has a Semihosting mechanism for loading CLI arguments through SYS_GET_CMDLINE, but our mechanism works in principle for any ISA.

    -

    28.4.1. gem5 baremetal arm CLI args

    +

    32.4.1. gem5 baremetal arm CLI args

    Currently not supported, so we just hardcode argc 0 on the arm baremetal bootloader.

    @@ -39554,7 +39918,7 @@ cc
    -

    28.5. Semihosting

    +

    32.5. Semihosting

    Semihosting is a publicly documented interface specified by ARM Holdings that allows us to do some magic operations very useful in development, such as writting to the terminal or reading and writing host files.

    @@ -39672,7 +40036,7 @@ svc 0x00123456
    -

    28.5.1. gem5 semihosting

    +

    32.5.1. gem5 semihosting

    For gem5, you need patches/manual/gem5-semihost.patch:

    @@ -39687,7 +40051,7 @@ svc 0x00123456
    -

    28.6. gem5 baremetal carriage return

    +

    32.6. gem5 baremetal carriage return

    TODO: our example is printing newlines without automatic carriage return \r as in:

    @@ -39710,7 +40074,7 @@ svc 0x00123456
    -

    28.7. Baremetal host packaged toolchain

    +

    32.7. Baremetal host packaged toolchain

    For arm, some baremetal examples compile fine with:

    @@ -39746,13 +40110,13 @@ collect2: error: ld returned 1 exit status
    -

    28.8. Baremetal C++

    +

    32.8. Baremetal C++

    Didn’t get it working, traking at: https://github.com/cirosantilli/linux-kernel-module-cheat/issues/119

    -

    28.9. GDB builtin CPU simulator

    +

    32.9. GDB builtin CPU simulator

    It is incredible, but GDB also has a CPU simulator inside of it as documented at: https://sourceware.org/gdb/onlinedocs/gdb/Target-Commands.html

    @@ -39812,7 +40176,7 @@ starti
    -

    28.9.1. GDB builtin CPU simulator userland

    +

    32.9.1. GDB builtin CPU simulator userland

    Since I had this compiled, I also decided to try it out on userland.

    @@ -39847,7 +40211,7 @@ starti
    -

    28.10. ARM baremetal

    +

    32.10. ARM baremetal

    In this section we will focus on learning ARM architecture concepts that can only learnt on baremetal setups.

    @@ -39855,7 +40219,7 @@ starti

    Userland information can be found at: https://github.com/cirosantilli/arm-assembly-cheat

    -

    28.10.1. ARM exception levels

    +

    32.10.1. ARM exception levels

    ARM exception levels are analogous to x86 rings.

    @@ -39984,13 +40348,13 @@ CurrentEL.EL 0x3

    According to ARMv7 architecture reference manual, access to that register is controlled by other registers NSACR.{CP11, CP10} and HCPTR so those must be turned off, but I’m lazy to investigate now, even just trying to dump those registers in userland/arch/arm/dump_regs.c also leads to exceptions…​

    -
    28.10.1.1. ARM change exception level
    +
    32.10.1.1. ARM change exception level

    TODO. Create a minimal runnable example of going into EL0 and jumping to EL1.

    -
    28.10.1.2. ARM SP0 vs SPx
    +
    32.10.1.2. ARM SP0 vs SPx

    See ARMv8 architecture reference manual db D1.6.2 "The stack pointer registers".

    @@ -40009,7 +40373,7 @@ CurrentEL.EL 0x3
    -

    28.10.2. ARM SVC instruction

    +

    32.10.2. ARM SVC instruction

    This is the most basic example of exception handling we have.

    @@ -40358,7 +40722,7 @@ IN: main
    -
    28.10.2.1. ARMv8 exception vector table format
    +
    32.10.2.1. ARMv8 exception vector table format

    The vector table format is described on ARMv8 architecture reference manual Table D1-7 "Vector offsets from vector table base address".

    @@ -40498,29 +40862,29 @@ IN: main
    -
    28.10.2.2. ARM ESR register
    +
    32.10.2.2. ARM ESR register

    Exception Syndrome Register.

    -

    See example at: Section 28.10.2, “ARM SVC instruction”

    +

    See example at: Section 32.10.2, “ARM SVC instruction”

    Documentation: ARMv8 architecture reference manual db D12.2.36 "ESR_EL1, Exception Syndrome Register (EL1)".

    -
    28.10.2.3. ARM ELR register
    +
    32.10.2.3. ARM ELR register

    Exception Link Register.

    -

    See the example at: Section 28.10.2, “ARM SVC instruction”

    +

    See the example at: Section 32.10.2, “ARM SVC instruction”

    -

    28.10.3. ARM baremetal multicore

    +

    32.10.3. ARM baremetal multicore

    Examples:

    @@ -40599,7 +40963,7 @@ IN: main

    Bibliography: https://stackoverflow.com/questions/980999/what-does-multicore-assembly-language-look-like/33651438#33651438

    -
    28.10.3.1. ARM WFE and SEV instructions
    +
    32.10.3.1. ARM WFE and SEV instructions

    The WFE and SEV instructions are just hints: a compliant implementation can treat them as NOPs.

    @@ -40752,7 +41116,7 @@ IN: main

    For how userland spinlocks and mutexes are implemented see Userland mutex implementation.

    -
    28.10.3.1.1. ARM WFE global monitor events
    +
    32.10.3.1.1. ARM WFE global monitor events

    Examples:

    @@ -40792,7 +41156,7 @@ IN: main
    -
    28.10.3.1.2. WFE from userland
    +
    32.10.3.1.2. WFE from userland

    WFE and SEV are usable from userland, and are part of an efficient spinlock implementation (which userland should arguably stay away from and rather use the futex system call which allow for non busy sleep instead), which maybe is not something that userland should ever tho and just stick to mutexes?

    @@ -40899,7 +41263,7 @@ IN: main
    -
    28.10.3.1.3. ARMv8 spinlock pattern
    +
    32.10.3.1.3. ARMv8 spinlock pattern

    http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.faqs/ka16277.html

    @@ -40918,7 +41282,7 @@ IN: main
    -
    28.10.3.1.4. gem5 ARM WFE
    +
    32.10.3.1.4. gem5 ARM WFE

    gem5 390a74f59934b85d91489f8a563450d8321b602d does not sleep on the first WFE on either syscall emulation or full system, because the code does:

    @@ -40960,14 +41324,14 @@ IN: main
    -
    28.10.3.1.5. ARM YIELD instruction
    +
    32.10.3.1.5. ARM YIELD instruction

    https://stackoverflow.com/questions/59311066/how-does-the-arm-yield-instruction-inform-other-threads-that-they-could-start-a

    -
    28.10.3.2. ARM LDAXR and STLXR instructions
    +
    32.10.3.2. ARM LDAXR and STLXR instructions

    Can be used to implement atomic variables, see also:

    @@ -40986,7 +41350,7 @@ IN: main
    -
    28.10.3.3. ARM PSCI
    +
    32.10.3.3. ARM PSCI

    In QEMU, CPU 1 starts in a halted state. This can be observed from GDB, where:

    @@ -41036,14 +41400,14 @@ IN: main
    -
    28.10.3.4. ARM DMB instruction
    +
    32.10.3.4. ARM DMB instruction

    TODO: create and study a minimal examples in gem5 where the DMB instruction leads to less cycles: https://stackoverflow.com/questions/15491751/real-life-use-cases-of-barriers-dsb-dmb-isb-in-arm

    -

    28.10.4. ARM timer

    +

    32.10.4. ARM timer

    The ARM timer is the simplest way to generate hardware interrupts periodically, and therefore serves as the simples example of ARM GIC usage.

    @@ -41196,7 +41560,7 @@ cntvct_el0 0x3CF516F
    -

    28.10.5. ARM GIC

    +

    32.10.5. ARM GIC

    Generic Interrupt Controller.

    @@ -41238,7 +41602,7 @@ cntvct_el0 0x3CF516F
    -

    28.10.6. ARM paging

    +

    32.10.6. ARM paging

    TODO create a minimal working aarch64 example analogous to the x86 one at: https://github.com/cirosantilli/x86-bare-metal-examples/blob/6dc9a73830fc05358d8d66128f740ef9906f7677/paging.S

    @@ -41268,9 +41632,9 @@ cntvct_el0 0x3CF516F
    -

    28.10.7. ARM baremetal bibliography

    +

    32.10.7. ARM baremetal bibliography

    -

    First, also consider the userland bibliography: Section 25.9, “ARM assembly bibliography”.

    +

    First, also consider the userland bibliography: Section 29.9, “ARM assembly bibliography”.

    The most useful ARM baremetal example sets we’ve seen so far are:

    @@ -41295,7 +41659,7 @@ cntvct_el0 0x3CF516F
    -
    28.10.7.1. NienfengYao/armv8-bare-metal
    +
    32.10.7.1. NienfengYao/armv8-bare-metal

    https://github.com/NienfengYao/armv8-bare-metal

    @@ -41354,7 +41718,7 @@ cntvct_el0 0x3CF516F
    -
    28.10.7.2. tukl-msd/gem5.bare-metal
    +
    32.10.7.2. tukl-msd/gem5.bare-metal

    https://github.com/tukl-msd/gem5.bare-metal

    @@ -41396,7 +41760,7 @@ make CROSS_COMPILE_DIR=/usr/bin
    -

    28.11. How we got some baremetal stuff to work

    +

    32.11. How we got some baremetal stuff to work

    It is nice when thing just work.

    @@ -41404,7 +41768,7 @@ make CROSS_COMPILE_DIR=/usr/bin

    But you can also learn a thing or two from how I actually made them work in the first place.

    -

    28.11.1. Find the UART address

    +

    32.11.1. Find the UART address

    Enter the QEMU console:

    @@ -41440,7 +41804,7 @@ make CROSS_COMPILE_DIR=/usr/bin
    -

    28.11.2. aarch64 baremetal NEON setup

    +

    32.11.2. aarch64 baremetal NEON setup

    Inside baremetal/lib/aarch64.S there is a chunk of code that enables floating point operations:

    @@ -41564,7 +41928,7 @@ ISB
    -

    28.12. Baremetal tests

    +

    32.12. Baremetal tests

    Baremetal tests work exactly like User mode tests, except that you have to add the --mode baremetal option, for example:

    @@ -41577,13 +41941,13 @@ ISB

    In baremetal, we detect if tests failed by parsing logs for the Magic failure string.

    -

    See: Section 34.16, “Test this repo” for more useful testing tips.

    +

    See: Section 37.16, “Test this repo” for more useful testing tips.

    -

    29. Android

    +

    33. Android

    Remember: Android AOSP is a huge undocumented piece of bloatware. It’s integration into this repo will likely never be super good. See also: https://cirosantilli.com#android

    @@ -41631,7 +41995,7 @@ ISB

    Tested on: 8.1.0_r60.

    -

    29.1. Android image structure

    +

    33.1. Android image structure

    https://source.android.com/devices/bootloader/partitions-images

    @@ -41715,7 +42079,7 @@ vendor-qemu

    Tested on: 8.1.0_r60.

    -

    29.1.1. Android images read-only

    +

    33.1.1. Android images read-only

    From mount, we can see that some of the mounted images are ro.

    @@ -41788,7 +42152,7 @@ date >/system/a
    -

    29.1.2. Android /data partition

    +

    33.1.2. Android /data partition

    When I install an app like F-Droid, it goes under /data according to:

    @@ -41849,7 +42213,7 @@ date >/system/a
    -

    29.2. Install Android apps

    +

    33.2. Install Android apps

    I don’t know how to download files from the web on Vanilla android, the default browser does not download anything, and there is no wget:

    @@ -41899,7 +42263,7 @@ date >/system/a
    -

    29.3. Android init

    +

    33.3. Android init

    For Linux in general, see: Section 6, “init”.

    @@ -41948,7 +42312,7 @@ import /init.${ro.zygote}.rc
    -

    30. Benchmark this repo

    +

    34. Benchmark this repo

    TODO: didn’t fully port during refactor after 3b0a343647bed577586989fb702b760bd280844a. Reimplementing should not be hard.

    @@ -41977,7 +42341,7 @@ cd -
    -

    30.1. Continuous integration

    +

    34.1. Continuous integration

    We have explored a few Continuous integration solutions.

    @@ -41985,13 +42349,13 @@ cd -

    We haven’t setup any of them yet.

    -

    30.1.1. Travis

    +

    34.1.1. Travis

    We tried to automate it on Travis with .travis.yml but it hits the current 50 minute job timeout: https://travis-ci.org/cirosantilli/linux-kernel-module-cheat/builds/296454523 And I bet it would likely hit a disk maxout either way if it went on.

    -

    30.1.2. CircleCI

    +

    34.1.2. CircleCI

    This setup successfully built gem5 on every commit: .circleci/config.yml

    @@ -42020,9 +42384,9 @@ cd -
    -

    30.2. Benchmark this repo benchmarks

    +

    34.2. Benchmark this repo benchmarks

    -

    30.2.1. Benchmark Linux kernel boot

    +

    34.2.1. Benchmark Linux kernel boot

    Run all kernel boot benchmarks for one arch:

    @@ -42123,6 +42487,27 @@ instructions 124346081

    Same but with: gem5 arm Linux kernel patches at v4.15: 73s, kernel size: 132M.

    +

    On Ubuntu 20.04, LKMC d3f8d3e99f2e554aae6c3b325b350bcf7f3f087f (Linux kernel 5.4.3), gem5 6bc2111c9674d0c8db22f6a6adcc00e49625aabd (sept 2020):

    +
    +
    +
    +
    ./run --arch aarch64 --emulator gem5 --quit-after-boot
    +
    +
    +
    +

    took 193s. With some minimal newer kernel boot patches:

    +
    +
    + +
    +

    On Ubuntu 20.04 gem5 3ca404da175a66e0b958165ad75eb5f54cb5e772 this took 22 minutes 53 seconds:

    @@ -42131,7 +42516,7 @@ instructions 124346081
    -
    30.2.1.1. gem5 arm HPI boot takes much longer than aarch64
    +
    34.2.1.1. gem5 arm HPI boot takes much longer than aarch64

    TODO 62f6870e4e0b384c4bd2d514116247e81b241251 takes 33 minutes to finish at 62f6870e4e0b384c4bd2d514116247e81b241251:

    @@ -42157,7 +42542,7 @@ instructions 124346081
    -
    30.2.1.2. gem5 x86_64 DerivO3CPU boot panics
    +
    34.2.1.2. gem5 x86_64 DerivO3CPU boot panics

    https://github.com/cirosantilli2/gem5-issues/issues/2

    @@ -42169,7 +42554,7 @@ instructions 124346081
    -

    30.2.2. Benchmark emulators on userland executables

    +

    34.2.2. Benchmark emulators on userland executables

    Let’s see how fast our simulators are running some well known or easy to understand userland benchmarks!

    @@ -42488,7 +42873,7 @@ instructions 124346081

    so ~ 110 million instructions / 100 seconds makes ~ 1 MIPS (million instructions per second).

    -

    This experiment also suggests that each loop is about 11 instructions long (110M instructions / 10M loops), which we confirm at Section 32.2, “C busy loop”, bingo!

    +

    This experiment also suggests that each loop is about 11 instructions long (110M instructions / 10M loops), which we confirm at Section 35.2, “C busy loop”, bingo!

    Then for QEMU, we experimentally turn the number of loops up to 10^10 loops (100000 100000), which contains an expected 11 * 10^10 instructions, and the runtime is 00:01:08, so we have 1.1 * 10^11 instruction / 68 seconds ~ 2 * 10^9 = 2000 MIPS!

    @@ -42497,7 +42882,7 @@ instructions 124346081

    We can then repeat the experiment for other gem5 CPUs to see how they compare.

    -
    30.2.2.1. User mode vs full system benchmark
    +
    34.2.2.1. User mode vs full system benchmark

    Let’s see if user mode runs considerably faster than full system or not, ignoring the kernel boot.

    @@ -42505,7 +42890,7 @@ instructions 124346081

    First we build Dhrystone manually statically since dynamic linking is broken in gem5 as explained at: Section 10.7, “gem5 syscall emulation mode”.

    -

    TODO: move this section to our new custom dhrystone setup: Section 22.9.2.1, “Dhrystone”.

    +

    TODO: move this section to our new custom dhrystone setup: Section 26.9.2.1, “Dhrystone”.

    gem5 user mode:

    @@ -42584,7 +42969,7 @@ time \
    -

    30.2.3. Benchmark builds

    +

    34.2.3. Benchmark builds

    The build times are calculated after doing ./configure and make source, which downloads the sources, and basically benchmarks the Internet.

    @@ -42609,7 +42994,7 @@ cat ../linux-kernel-module-cheat-regression/*/build-time.log
    -
    30.2.3.1. Find which Buildroot packages are making the build slow and big
    +
    34.2.3.1. Find which Buildroot packages are making the build slow and big
    ./build-buildroot -- graph-build graph-size graph-depends
@@ -42620,14 +43005,14 @@ xdg-open graph-size.pdf
    -
    30.2.3.1.1. Buildroot use prebuilt host toolchain
    +
    34.2.3.1.1. Buildroot use prebuilt host toolchain

    The biggest build time hog is always GCC, and it does not look like we can use a precompiled one: https://stackoverflow.com/questions/10833672/buildroot-environment-with-host-toolchain

    -
    30.2.3.2. Benchmark Buildroot build baseline
    +
    34.2.3.2. Benchmark Buildroot build baseline

    This is the minimal build we could expect to get away with.

    @@ -42695,7 +43080,7 @@ xdg-open graph-size.pdf
    -
    30.2.3.3. Benchmark gem5 build
    +
    34.2.3.3. Benchmark gem5 build

    How long it takes to build gem5 itself.

    @@ -42727,7 +43112,7 @@ tail -n+1 ../linux-kernel-module-cheat-regression/*/gem5-bench-build-*.txt

    A profiling of the build has been done at: https://gem5.atlassian.net/browse/GEM5-277 Analysis there showed that d7d9bc240615625141cd6feddbadd392457e49eb (2018-06-17) is also composed of 50% pybind11 and with no obvious time sinks.

    -
    30.2.3.3.1. pybind11 accounts for 50% of gem5 build time
    +
    34.2.3.3.1. pybind11 accounts for 50% of gem5 build time

    https://gem5.atlassian.net/browse/GEM5-366

    @@ -42739,7 +43124,7 @@ tail -n+1 ../linux-kernel-module-cheat-regression/*/gem5-bench-build-*.txt
    -
    30.2.3.3.2. Benchmark gem5 single file change rebuild time
    +
    34.2.3.3.2. Benchmark gem5 single file change rebuild time

    This is the critical development parameter, and is dominated by the link time of huge binaries.

    @@ -42775,7 +43160,20 @@ tail -n+1 ../linux-kernel-module-cheat-regression/*/gem5-bench-build-*.txt

    unmodified: 10 seconds

  • -

    hack with -fuse-ld=gold: 6 seconds. Huge improvement!

    +

    LDFLAGS_EXTRA=-fuse-ld=gold: 6 seconds. Huge improvement! Note that in general you have to do a full rebuild or else link may fail: https://sourceware.org/bugzilla/show_bug.cgi?id=23869

    +
    +

    More info on gold:

    +
    +
    + +
    • @@ -42785,10 +43183,10 @@ tail -n+1 ../linux-kernel-module-cheat-regression/*/gem5-bench-build-*.txt
    @@ -42803,12 +43201,28 @@ tail -n+1 ../linux-kernel-module-cheat-regression/*/gem5-bench-build-*.txt +
  • +

    opt LDFLAGS_EXTRA=-s: stripping the executable greatly reduces link time, but you get no symbols

    +

    • ramfs made no difference, the kernel must be caching files in memory very efficiently already.

    +

    In addition to the link time, scons startup time can also be considerable:

    +
    +
    + +
    +

    Tested at: d4b3e064adeeace3c3e7d106801f95c14637c12f + 1.

    @@ -42816,9 +43230,9 @@ tail -n+1 ../linux-kernel-module-cheat-regression/*/gem5-bench-build-*.txt
    -

    30.3. Benchmark machines

    +

    34.3. Benchmark machines

    -

    30.3.1. 2017 Lenovo ThinkPad P51

    +

    34.3.1. 2017 Lenovo ThinkPad P51

    Serial number: TYPE 20HH-CTO1WW S/N PF-0V5V5N 17/11

    @@ -42924,7 +43338,7 @@ tail -n+1 ../linux-kernel-module-cheat-regression/*/gem5-bench-build-*.txt
    -
    30.3.1.1. P51 benchmarks
    +
    34.3.1.1. P51 benchmarks

    Dhrystone on Ubuntu 20.04 results at Dhrystone.

    @@ -42932,7 +43346,7 @@ tail -n+1 ../linux-kernel-module-cheat-regression/*/gem5-bench-build-*.txt

    STREAM benchmark on Ubuntu 20.04 results at STREAM benchmark.

    -
    30.3.1.1.1. P51 CoreMark-Pro
    +
    34.3.1.1.1. P51 CoreMark-Pro

    CoreMark-Pro d5b4f2ba7ba31e37a5aa93423831e7d5eb933868 on Ubuntu 20.04 with XCMD="-c$(nproc)":

    @@ -42961,7 +43375,7 @@ CoreMark-PRO 25016.00 6079.70 4.11
    -
    30.3.1.2. P51 maintenance history
    +
    34.3.1.2. P51 maintenance history

    Bought: 2017 for approximately 2400 pounds.

    @@ -43019,7 +43433,7 @@ CoreMark-PRO 25016.00 6079.70 4.11
    -
    30.3.1.3. Intel Core i7-7820HQ CPU
    +
    34.3.1.3. Intel Core i7-7820HQ CPU

    https://ark.intel.com/products/97496/Intel-Core-i7-7820HQ-Processor-8M-Cache-up-to-3-90-GHz- (archive).

    @@ -43101,7 +43515,7 @@ LEVEL4_CACHE_LINESIZE 0
    -
    30.3.1.4. Samsung M471A2K43BB1-CRC 16GB DRAM
    +
    34.3.1.4. Samsung M471A2K43BB1-CRC 16GB DRAM

    Nominal speed: 2400 Mbps

    @@ -43116,7 +43530,7 @@ LEVEL4_CACHE_LINESIZE 0
    -
    30.3.1.5. Samsung MZVLB512HAJQ-000L7 512GB SSD
    +
    34.3.1.5. Samsung MZVLB512HAJQ-000L7 512GB SSD

    PCIe TLC OPAL2.

    @@ -43141,7 +43555,7 @@ LEVEL4_CACHE_LINESIZE 0
    -
    30.3.1.6. Seagate ST1000LM035-1RK1 1TB hard disk
    +
    34.3.1.6. Seagate ST1000LM035-1RK1 1TB hard disk

    1TB.

    @@ -43165,15 +43579,15 @@ LEVEL4_CACHE_LINESIZE 0
    -
    30.3.1.7. NVIDIA Quadro M1200 4GB GDDR5 GPU
    +
    34.3.1.7. NVIDIA Quadro M1200 4GB GDDR5 GPU
    -

    30.4. Benchmark Internets

    +

    34.4. Benchmark Internets

    -

    30.4.1. 38Mbps internet

    +

    34.4.1. 38Mbps internet

    2c12b21b304178a81c9912817b782ead0286d282:

    @@ -43193,7 +43607,7 @@ LEVEL4_CACHE_LINESIZE 0
    -

    30.5. Benchmark this repo bibliography

    +

    34.5. Benchmark this repo bibliography

    gem5:

    @@ -43221,70 +43635,13 @@ LEVEL4_CACHE_LINESIZE 0
    -

    31. RTOS

    -
    -
    -

    31.1. Zephyr

    -
    -

    https://en.wikipedia.org/wiki/Zephyr_(operating_system)

    -
    -
    -

    Zephyr is an RTOS that has POSIX support. I think it works much like our Baremetal setup which uses Newlib and generates individual ELF files that contain both our C program’s code, and the Zephyr libraries.

    -
    -
    -

    TODO get a hello world working, and then consider further integration in this repo, e.g. being able to run all C userland content on it.

    -
    -
    -

    TODO: Cortex-A CPUs are not currently supported, there are some qemu_cortex_m0 boards, but can’t find a QEMU Cortex-A. There is an x86_64 qemu board, but we don’t currently have an x86 baremetal toolchain. For this reason, we won’t touch this further for now.

    -
    -
    -

    However, unlike Newlib, Zephyr must be setting up a simple pre-main runtime to be able to handle threads.

    -
    -
    -

    Failed attempt:

    -
    -
    -
    -
    # https://askubuntu.com/questions/952429/is-there-a-good-ppa-for-cmake-backports
-wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | sudo apt-key add -
-sudo apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic-rc main'
-sudo apt-get update
-sudo apt-get install cmake
-git clone https://github.com/zephyrproject-rtos/zephyr
-pip3 install --user -U west packaging
-cd zephyr
-git checkout v1.14.1
-west init zephyrproject
-west update
-export ZEPHYR_TOOLCHAIN_VARIANT=xtools
-export XTOOLS_TOOLCHAIN_PATH="$(pwd)/out/crosstool-ng/build/default/install/aarch64/bin/"
-source zephyr-env.sh
-west build -b qemu_aarch64 samples/hello_world
    -
    -
    -
    -

    The build system of that project is a bit excessive / wonky. You need an edge CMake not present in Ubuntu 18.04, which I don’t want to install right now, and it uses the weird custom west build tool frontend.

    -
    -
    -
    -

    31.2. ARM Mbed

    -
    -

    https://en.wikipedia.org/wiki/Mbed

    -
    -
    -

    TODO minimal setup to run it on QEMU? Possible?

    -
    -
    -
    -
    -
    -

    32. Compilers

    +

    35. Compilers

    Argh, compilers are boring, let’s learn a bit about them.

    -

    32.1. Prevent statement reordering

    +

    35.1. Prevent statement reordering

    userland/gcc/prevent_reorder.cpp

    @@ -43296,7 +43653,7 @@ west build -b qemu_aarch64 samples/hello_world
    -

    32.2. C busy loop

    +

    35.2. C busy loop

    userland/gcc/busy_loop.c

    @@ -43380,10 +43737,10 @@ west build -b qemu_aarch64 samples/hello_world
    -

    33. Computer architecture

    +

    36. Computer architecture

    -

    33.1. Instruction pipelining

    +

    36.1. Instruction pipelining

    In gem5, can be seen on:

    @@ -43398,7 +43755,7 @@ west build -b qemu_aarch64 samples/hello_world
    -

    33.1.1. Classic RISC pipeline

    +

    36.1.1. Classic RISC pipeline

    https://en.wikipedia.org/wiki/Classic_RISC_pipeline

    @@ -43408,7 +43765,7 @@ west build -b qemu_aarch64 samples/hello_world
    -

    33.2. Superscalar processor

    +

    36.2. Superscalar processor

    https://en.wikipedia.org/wiki/Superscalar_processor

    @@ -43435,7 +43792,7 @@ west build -b qemu_aarch64 samples/hello_world
    -

    33.2.1. Execution unit

    +

    36.2.1. Execution unit

    https://en.wikipedia.org/wiki/Execution_unit

    @@ -43448,7 +43805,7 @@ west build -b qemu_aarch64 samples/hello_world
    -

    33.3. Out-of-order execution

    +

    36.3. Out-of-order execution

    https://en.wikipedia.org/wiki/Out-of-order_execution

    @@ -43465,7 +43822,7 @@ west build -b qemu_aarch64 samples/hello_world

    As mentioned at: https://stackoverflow.com/questions/10074831/what-is-general-difference-between-superscalar-and-ooo-execution it is in theory possible for an out-of-order CPU to not a Superscalar processor, but the combination is so natural (since you can look ahead, you might as well run it!) that it is not super common.

    -

    33.3.1. Speculative execution

    +

    36.3.1. Speculative execution

    https://en.wikipedia.org/wiki/Speculative_execution

    @@ -43483,7 +43840,7 @@ west build -b qemu_aarch64 samples/hello_world
    -
    33.3.1.1. Branch predictor
    +
    36.3.1.1. Branch predictor

    https://en.wikipedia.org/wiki/Branch_predictor

    @@ -43496,20 +43853,20 @@ west build -b qemu_aarch64 samples/hello_world
    -

    33.3.2. Re-order buffer

    +

    36.3.2. Re-order buffer

    https://en.wikipedia.org/wiki/Re-order_buffer

    -

    33.3.3. Register renaming

    +

    36.3.3. Register renaming

    https://en.wikipedia.org/wiki/Register_renaming

    -

    33.4. Instruction level parallelism

    +

    36.4. Instruction level parallelism

    https://en.wikipedia.org/wiki/Instruction-level_parallelism

    @@ -43528,7 +43885,7 @@ west build -b qemu_aarch64 samples/hello_world
    -

    33.5. Hardware threads

    +

    36.5. Hardware threads

    Intel name: "Hyperthreading"

    @@ -43578,7 +43935,7 @@ west build -b qemu_aarch64 samples/hello_world
    -

    33.6. Caches

    +

    36.6. Caches

    https://courses.cs.washington.edu/courses/cse378/09wi/lectures/lec15.pdf contains some of the first pictures you should see.

    @@ -43620,7 +43977,7 @@ west build -b qemu_aarch64 samples/hello_world

    For example, for a 2-way associative cache, we remove on bit from the index, and add it to the tag.

    -

    33.6.1. Cache coherence

    +

    36.6.1. Cache coherence

    https://en.wikipedia.org/wiki/Cache_coherence

    @@ -43662,7 +44019,7 @@ west build -b qemu_aarch64 samples/hello_world

    Even if caches are coherent, this is still not enough to avoid data race conditions, because this does not enforce atomicity of read modify write sequences. This is for example shown at: Detailed gem5 analysis of how data races happen.

    -
    33.6.1.1. Memory consistency
    +
    36.6.1.1. Memory consistency

    According to http://www.inf.ed.ac.uk/teaching/courses/pa/Notes/lecture07-sc.pdf "memory consistency" is about ordering requirements of different memory addresses.

    @@ -43670,14 +44027,14 @@ west build -b qemu_aarch64 samples/hello_world

    This is represented explicitly in C++ for example C++ std::memory_order.

    -
    33.6.1.1.1. Sequential Consistency
    +
    36.6.1.1.1. Sequential Consistency

    According to http://www.inf.ed.ac.uk/teaching/courses/pa/Notes/lecture07-sc.pdf, the strongest possible consistency, everything nicely ordered as you’d expect.

    -
    33.6.1.2. Can caches snoop data from other caches?
    +
    36.6.1.2. Can caches snoop data from other caches?

    Either they can snoop only control, or both control and data can be snooped.

    @@ -43692,7 +44049,7 @@ west build -b qemu_aarch64 samples/hello_world
    -
    33.6.1.3. VI cache coherence protocol
    +
    36.6.1.3. VI cache coherence protocol

    Mentioned at:

    @@ -43939,7 +44296,7 @@ west build -b qemu_aarch64 samples/hello_world
    -
    33.6.1.4. MSI cache coherence protocol
    +
    36.6.1.4. MSI cache coherence protocol

    https://en.wikipedia.org/wiki/MSI_protocol

    @@ -44251,7 +44608,7 @@ CACHE2 S nyy

    TODO gem5 concrete example.

    -
    33.6.1.4.1. MSI cache coherence protocol with transient states
    +
    36.6.1.4.1. MSI cache coherence protocol with transient states

    TODO understand well why those are needed.

    @@ -44271,7 +44628,7 @@ CACHE2 S nyy
    -
    33.6.1.5. MESI cache coherence protocol
    +
    36.6.1.5. MESI cache coherence protocol

    https://en.wikipedia.org/wiki/MESI_protocol

    @@ -44331,7 +44688,7 @@ CACHE2 S nyy
    -
    33.6.1.6. MOSI cache coherence protocol
    +
    36.6.1.6. MOSI cache coherence protocol

    https://en.wikipedia.org/wiki/MOSI_protocol The critical MSI vs MOSI section was a bit bogus though: https://en.wikipedia.org/w/index.php?title=MOSI_protocol&oldid=895443023 but I edited it :-)

    @@ -44391,7 +44748,7 @@ CACHE2 S nyy
    -
    33.6.1.7. MOESI cache coherence protocol
    +
    36.6.1.7. MOESI cache coherence protocol

    https://en.wikipedia.org/wiki/MOESI_protocol

    @@ -44399,10 +44756,10 @@ CACHE2 S nyy

    MESI cache coherence protocol + MOSI cache coherence protocol, not much else to it!

    -

    In gem5 9fc9c67b4242c03f165951775be5cd0812f2a705, MOESI is the default cache coherency protocol of the classic memory system as shown at Section 19.22.4.3.1, “What is the coherency protocol implemented by the classic cache system in gem5?”.

    +

    In gem5 9fc9c67b4242c03f165951775be5cd0812f2a705, MOESI is the default cache coherency protocol of the classic memory system as shown at Section 23.22.4.3.1, “What is the coherency protocol implemented by the classic cache system in gem5?”.

    -

    A good an simple example showing several MOESI transitions in the classic memory model can be seen at: Section 19.22.4.4, “gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs”.

    +

    A good an simple example showing several MOESI transitions in the classic memory model can be seen at: Section 23.22.4.4, “gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs”.

    gem5 12c917de54145d2d50260035ba7fa614e25317a3 has several Ruby MOESI models implemented: MOESI_AMD_Base, MOESI_CMP_directory, MOESI_CMP_token and MOESI_hammer.

    @@ -44413,10 +44770,10 @@ CACHE2 S nyy
    -

    34. About this repo

    +

    37. About this repo

    -

    34.1. Supported hosts

    +

    37.1. Supported hosts

    The host requirements depend a lot on which examples you want to run.

    @@ -44465,9 +44822,9 @@ CACHE2 S nyy
    -

    34.2. Common build issues

    +

    37.2. Common build issues

    -

    34.2.1. You must put some 'source' URIs in your sources.list

    +

    37.2.1. You must put some 'source' URIs in your sources.list

    If ./build --download-dependencies fails with:

    @@ -44481,7 +44838,7 @@ CACHE2 S nyy
    -

    34.2.2. Build from downloaded source zip files

    +

    37.2.2. Build from downloaded source zip files

    It does not work if you just download the .zip with the sources for this repository from GitHub because we use Git submodules, you must clone this repo.

    @@ -44491,7 +44848,7 @@ CACHE2 S nyy
    -

    34.3. Run command after boot

    +

    37.3. Run command after boot

    If you just want to run a command after boot ends without thinking much about it, just use the --eval-after option, e.g.:

    @@ -44508,7 +44865,7 @@ CACHE2 S nyy
    -

    34.4. Default command line arguments

    +

    37.4. Default command line arguments

    It gets annoying to retype --arch aarch64 for every single command, or to remember --config setups.

    @@ -44553,12 +44910,12 @@ CACHE2 S nyy
    -

    34.5. Documentation

    +

    37.5. Documentation

    To learn how to build the documentation see: Section 1.10, “Build the documentation”.

    -

    34.5.1. Documentation verification

    +

    37.5.1. Documentation verification

    When running build-doc, we do the following checks:

    @@ -44579,7 +44936,7 @@ CACHE2 S nyy

    The scripts prints what you have to fix and exits with an error status if there are any errors.

    - +

    Documentation for asciidoctor/extract-link-targets

    @@ -44602,7 +44959,7 @@ CACHE2 S nyy
    -
    34.5.1.2. asciidoctor/extract-header-ids
    +
    37.5.1.2. asciidoctor/extract-header-ids

    Documentation for asciidoctor/extract-header-ids

    @@ -44647,7 +45004,7 @@ explicitly-given
    - +

    The Asciidoctor extension scripts:

    @@ -44675,7 +45032,7 @@ explicitly-given
    -

    34.6.1. GitHub pages

    +

    37.6.1. GitHub pages

    As mentioned before the TOC, we have to push this README to GitHub pages due to: https://github.com/isaacs/github/issues/1610

    @@ -44725,7 +45082,7 @@ explicitly-given
    -

    34.7. Clean the build

    +

    37.7. Clean the build

    You did something crazy, and nothing seems to work anymore?

    @@ -44789,7 +45146,7 @@ ls "$(./getvar buildroot_build_dir)"
    -

    34.8. Custom build directory

    +

    37.8. Custom build directory

    For now there is no way to change the build directory from out/ (resp. out.docker for <<docker>.) to something else.

    @@ -44804,7 +45161,7 @@ ln -s out /mnt/hd/linux-kernel-module-cheat-out
    -

    34.9. ccache

    +

    37.9. ccache

    ccache might save you a lot of re-build when you decide to Clean the build or create a new build variant.

    @@ -44884,7 +45241,7 @@ export CCACHE_MAXSIZE="20G"
    -

    34.10. getvar

    +

    37.10. getvar

    The getvar helper script can print the values of internal LKMC variables.

    @@ -44922,7 +45279,7 @@ export CCACHE_MAXSIZE="20G"

    For this reason, we use it in particular often in this README to reduce the need for refactoring.

    -

    34.10.1. run-toolchain

    +

    37.10.1. run-toolchain

    While you could just manually find/learn the path to toolchain tools, e.g. in LKMC b15a0e455d691afa49f3b813ad9b09394dfb02b7 they are:

    @@ -44969,7 +45326,7 @@ export CCACHE_MAXSIZE="20G"
    -
    34.10.1.1. disas
    +
    37.10.1.1. disas

    Since disassembly of a single function of a LKMC executable with GDB is such a common use case for run-toolchain via https://stackoverflow.com/questions/22769246/how-to-disassemble-one-single-function-using-objdump, we have this shortcut for it.

    @@ -45001,7 +45358,7 @@ export CCACHE_MAXSIZE="20G"
    -

    34.11. Rebuild Buildroot while running

    +

    37.11. Rebuild Buildroot while running

    It is not possible to rebuild the root filesystem while running QEMU because QEMU holds the file qcow2 file:

    @@ -45012,7 +45369,7 @@ export CCACHE_MAXSIZE="20G"
    -

    34.12. Simultaneous runs

    +

    37.12. Simultaneous runs

    When doing long simulations sweeping across multiple system parameters, it becomes fundamental to do multiple simulations in parallel.

    @@ -45108,7 +45465,7 @@ less "$(./getvar --arch aarch64 --emulator gem5 --run-id 1 termout_file)"
    -

    To run multiple gem5 checkouts, see: Section 34.13.3.1, “gem5 worktree”.

    +

    To run multiple gem5 checkouts, see: Section 37.13.3.1, “gem5 worktree”.

    Implementation note: we create multiple namespaces for two things:

    @@ -45147,7 +45504,7 @@ less "$(./getvar --arch aarch64 --emulator gem5 --run-id 1 termout_file)"
    -

    34.13. Build variants

    +

    37.13. Build variants

    It often happens that you are comparing two versions of the build, a good and a bad one, and trying to figure out why the bad one is bad.

    @@ -45155,7 +45512,7 @@ less "$(./getvar --arch aarch64 --emulator gem5 --run-id 1 termout_file)"

    Our build variants system allows you to keep multiple built versions of all major components, so that you can easily switching between running one or the other.

    -

    34.13.1. Linux kernel build variants

    +

    37.13.1. Linux kernel build variants

    If you want to keep two builds around, one for the latest Linux version, and the other for Linux v4.16:

    @@ -45191,11 +45548,11 @@ git -C "$(./getvar linux_source_dir)" checkout -
    -

    To run both kernels simultaneously, one on each QEMU instance, see: Section 34.12, “Simultaneous runs”.

    +

    To run both kernels simultaneously, one on each QEMU instance, see: Section 37.12, “Simultaneous runs”.

    -

    34.13.2. QEMU build variants

    +

    37.13.2. QEMU build variants

    Analogous to the Linux kernel build variants but with the --qemu-build-id option instead:

    @@ -45211,7 +45568,7 @@ git -C "$(./getvar qemu_source_dir)" checkout -
    -

    34.13.3. gem5 build variants

    +

    37.13.3. gem5 build variants

    Analogous to the Linux kernel build variants but with the --gem5-build-id option instead:

    @@ -45242,7 +45599,7 @@ git -C "$(./getvar gem5_source_dir)" checkout some-branch

    Therefore, you can’t forget to checkout to the sources to that of the corresponding build before running, unless you explicitly tell gem5 to use a non-default source tree with gem5 worktree. This becomes inevitable when you want to launch multiple simultaneous runs at different checkouts.

    -
    34.13.3.1. gem5 worktree
    +
    37.13.3.1. gem5 worktree

    --gem5-build-id goes a long way, but if you want to seamlessly switch between two gem5 tress without checking out multiple times, then --gem5-worktree is for you.

    @@ -45295,7 +45652,7 @@ cd -
    -
    34.13.3.2. gem5 private source trees
    +
    37.13.3.2. gem5 private source trees

    Suppose that you are working on a private fork of gem5, but you want to use this repository to develop it as well.

    @@ -45339,7 +45696,7 @@ gem5_internal="$(pwd)/gem5-internal"
    -

    34.13.4. Buildroot build variants

    +

    37.13.4. Buildroot build variants

    Allows you to have multiple versions of the GCC toolchain or root filesystem.

    @@ -45359,7 +45716,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -
    -

    34.14. Optimization level of a build

    +

    37.14. Optimization level of a build

    The --optimization-level option is available on all build scripts and sets the given GCC `-`O optimization level where it has been implemented for guest binaries.

    @@ -45386,9 +45743,9 @@ git -C "$(./getvar buildroot_source_dir)" checkout -
    -

    34.15. Directory structure

    +

    37.15. Directory structure

    -

    34.15.1. lkmc directory

    +

    37.15.1. lkmc directory

    lkmc/ contains sources and headers that are shared across kernel modules, userland and baremetal examples.

    @@ -45399,7 +45756,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -

    Another option would have been to name it as includes/lkmc, but that would make paths longer, and we might want to store source code in that directory as well in the future.

    -
    34.15.1.1. Userland objects vs header-only
    +
    37.15.1.1. Userland objects vs header-only

    When factoring out functionality across userland examples, there are two main options:

    @@ -45458,7 +45815,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -
    -

    34.15.2. buildroot_packages directory

    +

    37.15.2. buildroot_packages directory

    Source: buildroot_packages/.

    @@ -45507,7 +45864,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -

    A custom build script can give you more flexibility: e.g. the package can be made work with other root filesystems more easily, have better 9P support, and rebuild faster as it evades some Buildroot boilerplate.

    -
    34.15.2.1. kernel_modules buildroot package
    +
    37.15.2.1. kernel_modules buildroot package

    Source: buildroot_packages/kernel_modules/

    @@ -45554,9 +45911,9 @@ git -C "$(./getvar buildroot_source_dir)" checkout -
    -

    34.15.3. patches directory

    +

    37.15.3. patches directory

    -
    34.15.3.1. patches/global directory
    +
    37.15.3.1. patches/global directory

    Has the following structure:

    @@ -45573,7 +45930,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -
    -
    34.15.3.2. patches/manual directory
    +
    37.15.3.2. patches/manual directory

    Patches in this directory are never applied automatically: it is up to users to manually apply them before usage following the instructions in this documentation.

    @@ -45583,7 +45940,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -
    -

    34.15.4. rootfs_overlay

    +

    37.15.4. rootfs_overlay

    Source: rootfs_overlay.

    @@ -45630,7 +45987,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -

    This way you can just hack away the scripts and try them out immediately without any further operations.

    -
    34.15.4.1. out_rootfs_overlay_dir
    +
    37.15.4.1. out_rootfs_overlay_dir

    This path can be found with:

    @@ -45664,7 +46021,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -
    -

    34.15.5. lkmc.c

    +

    37.15.5. lkmc.c

    The files:

    @@ -45694,7 +46051,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -
    -

    34.15.6. lkmc_home

    +

    37.15.6. lkmc_home

    lkmc_home refers to the target base directory in which we put all our custom built stuff, such as userland executables and kernel modules.

    @@ -45727,7 +46084,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -
    -

    34.15.7. path_properties.py

    +

    37.15.7. path_properties.py

    In order to build and run each userland and baremetal example properly, we need per-file metadata such as compiler flags and required number of cores.

    @@ -45790,7 +46147,7 @@ baremetal=True
    -

    34.15.8. rand_check.out

    +

    37.15.8. rand_check.out

    Print out several parameters that normally change randomly from boot to boot:

    @@ -45818,9 +46175,9 @@ baremetal=True
    -

    34.16. Test this repo

    +

    37.16. Test this repo

    -

    34.16.1. Automated tests

    +

    37.16.1. Automated tests

    Run almost all tests:

    @@ -45876,7 +46233,7 @@ echo $?

    test does not all possible tests, because there are too many possible variations and that would take forever. The rationale is the same as for ./build all and is explained in ./build --help.

    -
    34.16.1.1. Test arch and emulator selection
    +
    37.16.1.1. Test arch and emulator selection

    You can select multiple archs and emulators of interest, as for an other command, with:

    @@ -45909,7 +46266,7 @@ echo $?
    -
    34.16.1.2. Quit on fail
    +
    37.16.1.2. Quit on fail

    By default, continue running even after the first failure happens, and they show a summary at the end.

    @@ -45923,7 +46280,7 @@ echo $?
    -
    34.16.1.3. Test userland in full system
    +
    37.16.1.3. Test userland in full system

    TODO: we really need a mechanism to automatically generate the test list automatically e.g. based on path_properties.py, currently there are many tests missing, and we have to add everything manually which is very annoying.

    @@ -45952,7 +46309,7 @@ echo $?
    -
    34.16.1.4. GDB tests
    +
    37.16.1.4. GDB tests

    We have some pexpect automated tests for GDB for both userland and baremetal programs!

    @@ -46025,7 +46382,7 @@ echo $?
    -
    34.16.1.5. Magic failure string
    +
    37.16.1.5. Magic failure string

    We do not know of any way to set the emulator exit status in QEMU arm full system.

    @@ -46128,9 +46485,9 @@ echo $?
    -

    34.16.2. Non-automated tests

    +

    37.16.2. Non-automated tests

    -
    34.16.2.1. Test GDB Linux kernel
    +
    37.16.2.1. Test GDB Linux kernel

    For the Linux kernel, do the following manual tests for now.

    @@ -46168,7 +46525,7 @@ echo $?
    -
    34.16.2.2. Test the Internet
    +
    37.16.2.2. Test the Internet

    You should also test that the Internet works:

    @@ -46179,7 +46536,7 @@ echo $?
    -
    34.16.2.3. CLI script tests
    +
    37.16.2.3. CLI script tests

    build-userland and test-executables have a wide variety of target selection modes, and it was hard to keep them all working without some tests:

    @@ -46197,12 +46554,12 @@ echo $?
    -

    34.17. Bisection

    +

    37.17. Bisection

    When updating the Linux kernel, QEMU and gem5, things sometimes break.

    -

    However, for many types of crashes, it is trivial to bisect down to the offending commit, in particular because we can make QEMU and gem5 exit with status 1 on kernel panic as mentioned at: Section 15.6.1.3, “Exit emulator on panic”.

    +

    However, for many types of crashes, it is trivial to bisect down to the offending commit, in particular because we can make QEMU and gem5 exit with status 1 on kernel panic as mentioned at: Section 16.6.1.3, “Exit emulator on panic”.

    For example, when updating from QEMU v2.12.0 to v3.0.0-rc3, the Linux kernel boot started to panic for arm.

    @@ -46253,7 +46610,7 @@ git submodule update
    -

    34.18. Update a forked submodule

    +

    37.18. Update a forked submodule

    This is a template update procedure for submodules for which we have some patches on on top of mainline.

    @@ -46282,9 +46639,9 @@ git commit -m "linux: update to ${next_mainline_revision}"
    -

    34.19. Release

    +

    37.19. Release

    -

    34.19.1. Release procedure

    +

    37.19.1. Release procedure

    Ensure that the Automated tests are passing on a clean build:

    @@ -46295,7 +46652,7 @@ git commit -m "linux: update to ${next_mainline_revision}"
    -

    The ./build-test command builds a superset of what will be downloaded which also tests other things we would like to be working on the release. For the minimal build to generate the files to be uploaded, see: Section 34.19.2, “release-zip”

    +

    The ./build-test command builds a superset of what will be downloaded which also tests other things we would like to be working on the release. For the minimal build to generate the files to be uploaded, see: Section 37.19.2, “release-zip”

    The clean build is necessary as it generates clean images since it is not possible to remove Buildroot packages

    @@ -46365,7 +46722,7 @@ git push --follow-tags
    -

    34.19.2. release-zip

    +

    37.19.2. release-zip

    Create a zip containing all files required for Prebuilt setup:

    @@ -46390,7 +46747,7 @@ git push --follow-tags
    -

    34.19.3. release-upload

    +

    37.19.3. release-upload

    After:

    @@ -46438,9 +46795,9 @@ git push --follow-tags
    -

    34.20. Design rationale

    +

    37.20. Design rationale

    -

    34.20.1. Design goals

    +

    37.20.1. Design goals

    This project was created to help me understand, modify and test low level system components by using system simulators.

    @@ -46516,7 +46873,7 @@ git push --follow-tags
    -

    34.20.2. Setup trade-offs

    +

    37.20.2. Setup trade-offs

    The trade-offs between the different setups are basically a balance between:

    @@ -46541,13 +46898,13 @@ git push --follow-tags

    compatibility: how likely is is that all the components will work well together: emulator, compiler, kernel, standard library, …​

  • -

    guest software availability: how wide is your choice of easily installed guest software packages? See also: Section 34.20.4, “Linux distro choice”

    +

    guest software availability: how wide is your choice of easily installed guest software packages? See also: Section 37.20.4, “Linux distro choice”

    • -

    34.20.3. Resource tradeoff guidelines

    +

    37.20.3. Resource tradeoff guidelines

    Choosing which features go into our default builds means making tradeoffs, here are our guidelines:

    @@ -46588,11 +46945,11 @@ git push --follow-tags
    -

    In order to learn how to measure some of those aspects, see: Section 30, “Benchmark this repo”.

    +

    In order to learn how to measure some of those aspects, see: Section 34, “Benchmark this repo”.

    -

    34.20.4. Linux distro choice

    +

    37.20.4. Linux distro choice

    We haven’t found the ultimate distro yet, here is a summary table of trade-offs that we care about: Table 8, “Comparison of Linux distros for usage in this repository”.

    @@ -46695,9 +47052,9 @@ git push --follow-tags
    -

    34.21. Soft topics

    +

    37.21. Soft topics

    -

    34.21.1. Fairy tale

    +

    37.21.1. Fairy tale

    @@ -46735,7 +47092,7 @@ git push --follow-tags
    -

    34.22. Bibliography

    +

    37.22. Bibliography

    Runnable stuff: