From dc4783a168f559876aaf90105c4f1595d44f5d08 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ciro=20Santilli=20=E5=85=AD=E5=9B=9B=E4=BA=8B=E4=BB=B6=20?=
 =?UTF-8?q?=E6=B3=95=E8=BD=AE=E5=8A=9F?= <ciro.santilli@gmail.com>
Date: Sun, 17 Nov 2019 00:00:00 +0000
Subject: [PATCH] d43ca360eb0a7a9edd28bc8da0a68e68c500cec6

---
 index.html | 343 ++++++++++++++++++++++++++++++++++-------------------
 1 file changed, 221 insertions(+), 122 deletions(-)

diff --git a/index.html b/index.html
index 731ade9..a7be977 100644
--- a/index.html
+++ b/index.html
@@ -5,7 +5,7 @@
 <meta http-equiv="X-UA-Compatible" content="IE=edge">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
 <meta name="generator" content="Asciidoctor 2.0.10">
-<meta name="description" content="The perfect emulation setup to study and develop the &lt;&lt;linux-kernel&gt;&gt; v5.2.1, kernel modules, &lt;&lt;qemu-buildroot-setup,QEMU&gt;&gt;, &lt;&lt;gem5-buildroot-setup,gem5&gt;&gt; and x86_64, ARMv7 and ARMv8 &lt;&lt;userland-assembly,userland&gt;&gt; and &lt;&lt;baremetal-setup,baremetal&gt;&gt; assembly, &lt;&lt;c,ANSI C&gt;&gt;, &lt;&lt;cpp,C++&gt;&gt; and &lt;&lt;posix,POSIX&gt;&gt;. &lt;&lt;gdb&gt;&gt; and &lt;&lt;kgdb&gt;&gt; just work. Powered by &lt;&lt;about-the-qemu-buildroot-setup,Buildroot&gt;&gt; and &lt;&lt;about-the-baremetal-setup,crosstool-NG&gt;&gt;.  Highly automated. Thoroughly documented. Automated &lt;&lt;test-this-repo,tests&gt;&gt;. "Tested" in an Ubuntu 18.04 host.">
+<meta name="description" content="The perfect emulation setup to study and develop the &lt;&lt;linux-kernel&gt;&gt; v5.2.1, kernel modules, &lt;&lt;qemu-buildroot-setup,QEMU&gt;&gt;, &lt;&lt;gem5-buildroot-setup,gem5&gt;&gt; and x86_64, ARMv7 and ARMv8 &lt;&lt;userland-assembly,userland&gt;&gt; and &lt;&lt;baremetal-setup,baremetal&gt;&gt; assembly, &lt;&lt;c,ANSI C&gt;&gt;, &lt;&lt;cpp,C++&gt;&gt; and &lt;&lt;posix,POSIX&gt;&gt;. &lt;&lt;gdb&gt;&gt; and &lt;&lt;kgdb&gt;&gt; just work. Powered by &lt;&lt;about-the-qemu-buildroot-setup,Buildroot&gt;&gt; and &lt;&lt;about-the-baremetal-setup,crosstool-NG&gt;&gt;. Highly automated. Thoroughly documented. Automated &lt;&lt;test-this-repo,tests&gt;&gt;. "Tested" in an Ubuntu 18.04 host.">
 <title>Linux Kernel Module Cheat</title>
 <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700">
 <style>
@@ -448,7 +448,7 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 <p><a href="https://zenodo.org/badge/latestdoi/64534859"><span class="image"><img src="https://zenodo.org/badge/64534859.svg" alt="64534859"></span></a></p>
 </div>
 <div class="paragraph">
-<p>The perfect emulation setup to study and develop the <a href="#linux-kernel">Linux kernel</a> v5.2.1, kernel modules, <a href="#qemu-buildroot-setup">QEMU</a>, <a href="#gem5-buildroot-setup">gem5</a> and x86_64, ARMv7 and ARMv8 <a href="#userland-assembly">userland</a> and <a href="#baremetal-setup">baremetal</a> assembly, <a href="#c">ANSI C</a>, <a href="#cpp">C++</a> and <a href="#posix">POSIX</a>. <a href="#gdb">GDB step debug</a> and <a href="#kgdb">KGDB</a> just work. Powered by <a href="#about-the-qemu-buildroot-setup">Buildroot</a> and <a href="#about-the-baremetal-setup">crosstool-NG</a>.  Highly automated. Thoroughly documented. Automated <a href="#test-this-repo">tests</a>. "Tested" in an Ubuntu 18.04 host.</p>
+<p>The perfect emulation setup to study and develop the <a href="#linux-kernel">Linux kernel</a> v5.2.1, kernel modules, <a href="#qemu-buildroot-setup">QEMU</a>, <a href="#gem5-buildroot-setup">gem5</a> and x86_64, ARMv7 and ARMv8 <a href="#userland-assembly">userland</a> and <a href="#baremetal-setup">baremetal</a> assembly, <a href="#c">ANSI C</a>, <a href="#cpp">C++</a> and <a href="#posix">POSIX</a>. <a href="#gdb">GDB step debug</a> and <a href="#kgdb">KGDB</a> just work. Powered by <a href="#about-the-qemu-buildroot-setup">Buildroot</a> and <a href="#about-the-baremetal-setup">crosstool-NG</a>. Highly automated. Thoroughly documented. Automated <a href="#test-this-repo">tests</a>. "Tested" in an Ubuntu 18.04 host.</p>
 </div>
 <div class="paragraph">
 <p>TL;DR: <a href="#qemu-buildroot-setup-getting-started">Section 1.1.1, &#8220;QEMU Buildroot setup getting started&#8221;</a></p>
@@ -534,12 +534,12 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 <li><a href="#gdb-step-debug-kernel-module">2.4. GDB step debug kernel module</a>
 <ul class="sectlevel3">
 <li><a href="#gdb-step-debug-kernel-module-arm">2.4.1. GDB step debug kernel module insmodded by init on ARM</a></li>
-<li><a href="#gdb-module_init">2.4.2. GDB module_init</a>
+<li><a href="#gdb-module-init">2.4.2. GDB module_init</a>
 <ul class="sectlevel4">
-<li><a href="#gdb-module_init-step-into-it">2.4.2.1. GDB module_init step into it</a></li>
-<li><a href="#gdb-module_init-calculate-entry-address">2.4.2.2. GDB module_init calculate entry address</a></li>
-<li><a href="#gdb-module_init-break-at-the-end-of-sys_init_module">2.4.2.3. GDB module_init break at the end of sys_init_module</a></li>
-<li><a href="#gdb-module_init-add-trap-instruction">2.4.2.4. GDB module_init add trap instruction</a></li>
+<li><a href="#gdb-module-init-step-into-it">2.4.2.1. GDB module_init step into it</a></li>
+<li><a href="#gdb-module-init-calculate-entry-address">2.4.2.2. GDB module_init calculate entry address</a></li>
+<li><a href="#gdb-module-init-break-at-the-end-of-sys-init-module">2.4.2.3. GDB module_init break at the end of sys_init_module</a></li>
+<li><a href="#gdb-module-init-add-trap-instruction">2.4.2.4. GDB module_init add trap instruction</a></li>
 </ul>
 </li>
 <li><a href="#bypass-lx-symbols">2.4.3. Bypass lx-symbols</a></li>
@@ -597,7 +597,7 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 </li>
 <li><a href="#cpu-architecture">5. CPU architecture</a>
 <ul class="sectlevel2">
-<li><a href="#x86_64">5.1. x86_64</a>
+<li><a href="#x86-64">5.1. x86_64</a>
 <ul class="sectlevel3">
 <li><a href="#ring0">5.1.1. ring0</a></li>
 </ul>
@@ -616,8 +616,8 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 <li><a href="#replace-init">6.1. Replace init</a>
 <ul class="sectlevel3">
 <li><a href="#poweroff-out">6.1.1. poweroff.out</a></li>
-<li><a href="#sleep_forever-out">6.1.2. sleep_forever.out</a></li>
-<li><a href="#time_boot-out">6.1.3. time_boot.out</a></li>
+<li><a href="#sleep-forever-out">6.1.2. sleep_forever.out</a></li>
+<li><a href="#time-boot-out">6.1.3. time_boot.out</a></li>
 </ul>
 </li>
 <li><a href="#init-busybox">6.2. Run command at the end of BusyBox init</a></li>
@@ -679,7 +679,11 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 </li>
 <li><a href="#user-mode-static-executables">10.5. User mode static executables</a>
 <ul class="sectlevel3">
-<li><a href="#user-mode-static-executables-with-dynamic-libraries">10.5.1. User mode static executables with dynamic libraries</a></li>
+<li><a href="#user-mode-static-executables-with-dynamic-libraries">10.5.1. User mode static executables with dynamic libraries</a>
+<ul class="sectlevel4">
+<li><a href="#cpp-static-and-pthreads">10.5.1.1. C++ static and pthreads</a></li>
+</ul>
+</li>
 </ul>
 </li>
 <li><a href="#gem5-syscall-emulation-mode">10.6. gem5 syscall emulation mode</a>
@@ -820,10 +824,10 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 <li><a href="#printk">15.4. printk</a>
 <ul class="sectlevel3">
 <li><a href="#procsyskernelprintk">15.4.1. /proc/sys/kernel/printk</a></li>
-<li><a href="#ignore_loglevel">15.4.2. ignore_loglevel</a></li>
-<li><a href="#pr_debug">15.4.3. pr_debug</a>
+<li><a href="#ignore-loglevel">15.4.2. ignore_loglevel</a></li>
+<li><a href="#pr-debug">15.4.3. pr_debug</a>
 <ul class="sectlevel4">
-<li><a href="#pr_debug-printkkern_debug">15.4.3.1. pr_debug != printk(KERN_DEBUG</a></li>
+<li><a href="#pr-debug-is-different-from-printk-kern-debug">15.4.3.1. pr_debug != printk(KERN_DEBUG</a></li>
 </ul>
 </li>
 </ul>
@@ -841,9 +845,9 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 <li><a href="#kernel-module-dependencies-with-modprobe">15.6.2.1. Kernel module dependencies with modprobe</a></li>
 </ul>
 </li>
-<li><a href="#module_info">15.6.3. MODULE_INFO</a></li>
+<li><a href="#module-info">15.6.3. MODULE_INFO</a></li>
 <li><a href="#vermagic">15.6.4. vermagic</a></li>
-<li><a href="#init_module">15.6.5. init_module</a></li>
+<li><a href="#init-module">15.6.5. init_module</a></li>
 <li><a href="#floating-point-in-kernel-modules">15.6.6. Floating point in kernel modules</a></li>
 </ul>
 </li>
@@ -852,7 +856,7 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 <li><a href="#kernel-panic">15.7.1. Kernel panic</a>
 <ul class="sectlevel4">
 <li><a href="#kernel-module-stack-trace-to-source-line">15.7.1.1. Kernel module stack trace to source line</a></li>
-<li><a href="#bug_on">15.7.1.2. BUG_ON</a></li>
+<li><a href="#bug-on">15.7.1.2. BUG_ON</a></li>
 <li><a href="#exit-emulator-on-panic">15.7.1.3. Exit emulator on panic</a>
 <ul class="sectlevel5">
 <li><a href="#exit-qemu-on-panic">15.7.1.3.1. Exit QEMU on panic</a></li>
@@ -864,8 +868,8 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 </ul>
 </li>
 <li><a href="#oops">15.7.2. Kernel oops</a></li>
-<li><a href="#dump_stack">15.7.3. dump_stack</a></li>
-<li><a href="#warn_on">15.7.4. WARN_ON</a></li>
+<li><a href="#dump-stack">15.7.3. dump_stack</a></li>
+<li><a href="#warn-on">15.7.4. WARN_ON</a></li>
 </ul>
 </li>
 <li><a href="#pseudo-filesystems">15.8. Pseudo filesystems</a>
@@ -887,9 +891,9 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 <li><a href="#pseudo-files">15.9. Pseudo files</a>
 <ul class="sectlevel3">
 <li><a href="#file-operations">15.9.1. File operations</a></li>
-<li><a href="#seq_file">15.9.2. seq_file</a>
+<li><a href="#seq-file">15.9.2. seq_file</a>
 <ul class="sectlevel4">
-<li><a href="#seq_file-single_open">15.9.2.1. seq_file single_open</a></li>
+<li><a href="#seq-file-single-open">15.9.2.1. seq_file single_open</a></li>
 </ul>
 </li>
 <li><a href="#poll">15.9.3. poll</a></li>
@@ -923,13 +927,13 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 <li><a href="#kernel-utility-functions">15.13. Kernel utility functions</a>
 <ul class="sectlevel3">
 <li><a href="#kstrto">15.13.1. kstrto</a></li>
-<li><a href="#virt_to_phys">15.13.2. virt_to_phys</a>
+<li><a href="#virt-to-phys">15.13.2. virt_to_phys</a>
 <ul class="sectlevel4">
 <li><a href="#userland-physical-address-experiments">15.13.2.1. Userland physical address experiments</a>
 <ul class="sectlevel5">
 <li><a href="#qemu-xp">15.13.2.1.1. QEMU xp</a></li>
 <li><a href="#dev-mem">15.13.2.1.2. /dev/mem</a></li>
-<li><a href="#pagemap_dump-out">15.13.2.1.3. pagemap_dump.out</a></li>
+<li><a href="#pagemap-dump-out">15.13.2.1.3. pagemap_dump.out</a></li>
 </ul>
 </li>
 </ul>
@@ -938,9 +942,9 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 </li>
 <li><a href="#linux-kernel-tracing">15.14. Linux kernel tracing</a>
 <ul class="sectlevel3">
-<li><a href="#config_proc_events">15.14.1. CONFIG_PROC_EVENTS</a>
+<li><a href="#config-proc-events">15.14.1. CONFIG_PROC_EVENTS</a>
 <ul class="sectlevel4">
-<li><a href="#config_proc_events-aarch64">15.14.1.1. CONFIG_PROC_EVENTS aarch64</a></li>
+<li><a href="#config-proc-events-aarch64">15.14.1.1. CONFIG_PROC_EVENTS aarch64</a></li>
 </ul>
 </li>
 <li><a href="#ftrace">15.14.2. ftrace</a>
@@ -955,7 +959,7 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 </li>
 <li><a href="#linux-kernel-hardening">15.15. Linux kernel hardening</a>
 <ul class="sectlevel3">
-<li><a href="#config_fortify_source">15.15.1. CONFIG_FORTIFY_SOURCE</a></li>
+<li><a href="#config-fortify-source">15.15.1. CONFIG_FORTIFY_SOURCE</a></li>
 <li><a href="#linux-security-modules">15.15.2. Linux security modules</a>
 <ul class="sectlevel4">
 <li><a href="#selinux">15.15.2.1. SELinux</a></li>
@@ -980,7 +984,7 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 <li><a href="#console-kernel-boot-parameter">15.18.3.2. console kernel boot parameter</a></li>
 </ul>
 </li>
-<li><a href="#config_logo">15.18.4. CONFIG_LOGO</a></li>
+<li><a href="#config-logo">15.18.4. CONFIG_LOGO</a></li>
 </ul>
 </li>
 <li><a href="#drm">15.19. DRM</a>
@@ -1023,7 +1027,7 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 <ul class="sectlevel3">
 <li><a href="#pci">18.5.1. PCI</a>
 <ul class="sectlevel4">
-<li><a href="#pci_min">18.5.1.1. pci_min</a></li>
+<li><a href="#pci-min">18.5.1.1. pci_min</a></li>
 <li><a href="#qemu-edu">18.5.1.2. QEMU edu PCI device</a></li>
 <li><a href="#manipulate-pci-registers-directly">18.5.1.3. Manipulate PCI registers directly</a></li>
 <li><a href="#pciutils">18.5.1.4. pciutils</a></li>
@@ -1034,7 +1038,7 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 </li>
 <li><a href="#gpio">18.5.2. GPIO</a></li>
 <li><a href="#leds">18.5.3. LEDs</a></li>
-<li><a href="#platform_device">18.5.4. platform_device</a></li>
+<li><a href="#platform-device">18.5.4. platform_device</a></li>
 <li><a href="#gem5-educational-hardware-models">18.5.5. gem5 educational hardware models</a></li>
 </ul>
 </li>
@@ -1172,7 +1176,7 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 </li>
 <li><a href="#m5term">19.10. m5term</a></li>
 <li><a href="#gem5-python-scripts-without-rebuild">19.11. gem5 Python scripts without rebuild</a></li>
-<li><a href="#gem5-fs_biglittle">19.12. gem5 fs_bigLITTLE</a></li>
+<li><a href="#gem5-fs-biglittle">19.12. gem5 fs_bigLITTLE</a></li>
 <li><a href="#gem5-unit-tests">19.13. gem5 unit tests</a></li>
 <li><a href="#gem5-regression-tests">19.14. gem5 regression tests</a></li>
 <li><a href="#gem5-simulate-limit-reached">19.15. gem5 simulate() limit reached</a></li>
@@ -1208,7 +1212,7 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 <li><a href="#gem5-stats-internals">19.19.5. gem5 stats internals</a></li>
 <li><a href="#gem5-code-generation">19.19.6. gem5 code generation</a>
 <ul class="sectlevel4">
-<li><a href="#gem5-the_isa">19.19.6.1. gem5 THE_ISA</a></li>
+<li><a href="#gem5-the-isa">19.19.6.1. gem5 THE_ISA</a></li>
 </ul>
 </li>
 <li><a href="#gem5-build-system">19.19.7. gem5 build system</a>
@@ -1237,7 +1241,7 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 </li>
 <li><a href="#add-new-buildroot-packages">20.5. Add new Buildroot packages</a></li>
 <li><a href="#remove-buildroot-packages">20.6. Remove Buildroot packages</a></li>
-<li><a href="#br2_target_rootfs_ext2_size">20.7. BR2_TARGET_ROOTFS_EXT2_SIZE</a>
+<li><a href="#br2-target-rootfs-ext2-size">20.7. BR2_TARGET_ROOTFS_EXT2_SIZE</a>
 <ul class="sectlevel3">
 <li><a href="#squashfs">20.7.1. SquashFS</a></li>
 </ul>
@@ -1279,7 +1283,11 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 </li>
 <li><a href="#cpp">21.2. C++</a>
 <ul class="sectlevel3">
-<li><a href="#cpp-multithreading">21.2.1. C++ multithreading</a></li>
+<li><a href="#cpp-multithreading">21.2.1. C++ multithreading</a>
+<ul class="sectlevel4">
+<li><a href="#atomic-cpp">21.2.1.1. atomic.cpp</a></li>
+</ul>
+</li>
 <li><a href="#cpp-standards">21.2.2. C++ standards</a>
 <ul class="sectlevel4">
 <li><a href="#cpp17">21.2.2.1. C++17 N4659 standards draft</a></li>
@@ -3686,7 +3694,7 @@ cd userland
 <p>And the terminal prints the values of certain system registers. This example prints registers that are only accessible from <a href="#arm-exception-levels">EL1</a> or higher, and thus could not be run in userland.</p>
 </div>
 <div class="paragraph">
-<p>In addition to the examples under <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/">baremetal/</a>,  several of the <a href="#userland-content">userland examples</a> can also be run in baremetal! This is largely due to the <a href="#about-the-baremetal-setup">awesomeness of Newlib</a>.</p>
+<p>In addition to the examples under <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/">baremetal/</a>, several of the <a href="#userland-content">userland examples</a> can also be run in baremetal! This is largely due to the <a href="#about-the-baremetal-setup">awesomeness of Newlib</a>.</p>
 </div>
 <div class="paragraph">
 <p>The examples that work include most <a href="#c">C examples</a> that don&#8217;t rely on complicated syscalls such as threads, and almost all the <a href="#userland-assembly">Userland assembly</a> examples.</p>
@@ -4382,7 +4390,7 @@ Error occurred in Python command: Cannot access memory at address 0xbf0000cc</pr
 </div>
 </div>
 <div class="sect3">
-<h4 id="gdb-module_init"><a class="anchor" href="#gdb-module_init"></a><a class="link" href="#gdb-module_init">2.4.2. GDB module_init</a></h4>
+<h4 id="gdb-module-init"><a class="anchor" href="#gdb-module-init"></a><a class="link" href="#gdb-module-init">2.4.2. GDB module_init</a></h4>
 <div class="paragraph">
 <p>TODO find a more convenient method. We have working methods, but they are not ideal.</p>
 </div>
@@ -4403,7 +4411,7 @@ Error occurred in Python command: Cannot access memory at address 0xbf0000cc</pr
 </ul>
 </div>
 <div class="sect4">
-<h5 id="gdb-module_init-step-into-it"><a class="anchor" href="#gdb-module_init-step-into-it"></a><a class="link" href="#gdb-module_init-step-into-it">2.4.2.1. GDB module_init step into it</a></h5>
+<h5 id="gdb-module-init-step-into-it"><a class="anchor" href="#gdb-module-init-step-into-it"></a><a class="link" href="#gdb-module-init-step-into-it">2.4.2.1. GDB module_init step into it</a></h5>
 <div class="paragraph">
 <p>This is the best method we&#8217;ve found so far.</p>
 </div>
@@ -4446,11 +4454,11 @@ Error occurred in Python command: Cannot access memory at address 0xbf0000cc</pr
 </div>
 </div>
 <div class="paragraph">
-<p>How we found this out: first we got <a href="#gdb-module_init-calculate-entry-address">GDB module_init calculate entry address</a> working, and then we did a <code>bt</code>. AKA cheating :-)</p>
+<p>How we found this out: first we got <a href="#gdb-module-init-calculate-entry-address">GDB module_init calculate entry address</a> working, and then we did a <code>bt</code>. AKA cheating :-)</p>
 </div>
 </div>
 <div class="sect4">
-<h5 id="gdb-module_init-calculate-entry-address"><a class="anchor" href="#gdb-module_init-calculate-entry-address"></a><a class="link" href="#gdb-module_init-calculate-entry-address">2.4.2.2. GDB module_init calculate entry address</a></h5>
+<h5 id="gdb-module-init-calculate-entry-address"><a class="anchor" href="#gdb-module-init-calculate-entry-address"></a><a class="link" href="#gdb-module-init-calculate-entry-address">2.4.2.2. GDB module_init calculate entry address</a></h5>
 <div class="paragraph">
 <p>This works, but is a bit annoying.</p>
 </div>
@@ -4526,7 +4534,7 @@ Error occurred in Python command: Cannot access memory at address 0xbf0000cc</pr
 </div>
 </div>
 <div class="sect4">
-<h5 id="gdb-module_init-break-at-the-end-of-sys_init_module"><a class="anchor" href="#gdb-module_init-break-at-the-end-of-sys_init_module"></a><a class="link" href="#gdb-module_init-break-at-the-end-of-sys_init_module">2.4.2.3. GDB module_init break at the end of sys_init_module</a></h5>
+<h5 id="gdb-module-init-break-at-the-end-of-sys-init-module"><a class="anchor" href="#gdb-module-init-break-at-the-end-of-sys-init-module"></a><a class="link" href="#gdb-module-init-break-at-the-end-of-sys-init-module">2.4.2.3. GDB module_init break at the end of sys_init_module</a></h5>
 <div class="paragraph">
 <p>TODO not working. This could be potentially very convenient.</p>
 </div>
@@ -4578,14 +4586,14 @@ Error occurred in Python command: Cannot access memory at address 0xbf0000cc</pr
 <p>also fails to break!</p>
 </div>
 <div class="paragraph">
-<p>Finally, in despair we notice that <a href="#pr_debug">pr_debug</a> prints the kernel load address as explained at <a href="#bypass-lx-symbols">Bypass lx-symbols</a>.</p>
+<p>Finally, in despair we notice that <a href="#pr-debug">pr_debug</a> prints the kernel load address as explained at <a href="#bypass-lx-symbols">Bypass lx-symbols</a>.</p>
 </div>
 <div class="paragraph">
 <p>So, if we set a breakpoint just after that message is printed by searching where that happens on the Linux source code, we must be able to get the correct load address before <code>init_module</code> happens.</p>
 </div>
 </div>
 <div class="sect4">
-<h5 id="gdb-module_init-add-trap-instruction"><a class="anchor" href="#gdb-module_init-add-trap-instruction"></a><a class="link" href="#gdb-module_init-add-trap-instruction">2.4.2.4. GDB module_init add trap instruction</a></h5>
+<h5 id="gdb-module-init-add-trap-instruction"><a class="anchor" href="#gdb-module-init-add-trap-instruction"></a><a class="link" href="#gdb-module-init-add-trap-instruction">2.4.2.4. GDB module_init add trap instruction</a></h5>
 <div class="paragraph">
 <p>This is another possibility: we could modify the module source by adding a trap instruction of some kind.</p>
 </div>
@@ -4659,7 +4667,7 @@ add-symbol-file ../../../rootfs_overlay/x86_64/timer.ko 0xffffffffc0000000
 </div>
 </div>
 <div class="paragraph">
-<p>Alternatively, if the module panics before you can read <code>/proc/modules</code>, there is a <a href="#pr_debug">pr_debug</a> which shows the load address:</p>
+<p>Alternatively, if the module panics before you can read <code>/proc/modules</code>, there is a <a href="#pr-debug">pr_debug</a> which shows the load address:</p>
 </div>
 <div class="literalblock">
 <div class="content">
@@ -5915,7 +5923,7 @@ continue</pre>
 <p>Known quirks of the supported architectures are documented in this section.</p>
 </div>
 <div class="sect2">
-<h3 id="x86_64"><a class="anchor" href="#x86_64"></a><a class="link" href="#x86_64">5.1. x86_64</a></h3>
+<h3 id="x86-64"><a class="anchor" href="#x86-64"></a><a class="link" href="#x86-64">5.1. x86_64</a></h3>
 <div class="sect3">
 <h4 id="ring0"><a class="anchor" href="#ring0"></a><a class="link" href="#ring0">5.1.1. ring0</a></h4>
 <div class="paragraph">
@@ -6215,7 +6223,7 @@ chmod +x rootfs_overlay/lkmc/gitignore.sh
 </div>
 </div>
 <div class="sect3">
-<h4 id="sleep_forever-out"><a class="anchor" href="#sleep_forever-out"></a><a class="link" href="#sleep_forever-out">6.1.2. sleep_forever.out</a></h4>
+<h4 id="sleep-forever-out"><a class="anchor" href="#sleep-forever-out"></a><a class="link" href="#sleep-forever-out">6.1.2. sleep_forever.out</a></h4>
 <div class="paragraph">
 <p>I dare you to guess what this does:</p>
 </div>
@@ -6232,7 +6240,7 @@ chmod +x rootfs_overlay/lkmc/gitignore.sh
 </div>
 </div>
 <div class="sect3">
-<h4 id="time_boot-out"><a class="anchor" href="#time_boot-out"></a><a class="link" href="#time_boot-out">6.1.3. time_boot.out</a></h4>
+<h4 id="time-boot-out"><a class="anchor" href="#time-boot-out"></a><a class="link" href="#time-boot-out">6.1.3. time_boot.out</a></h4>
 <div class="paragraph">
 <p>Get a reasonable answer to "how long does boot take in guest time?":</p>
 </div>
@@ -6767,7 +6775,7 @@ cat f
 <p>The device tree is a Linux kernel defined data structure that serves to inform the kernel how the hardware is setup.</p>
 </div>
 <div class="paragraph">
-<p><a href="#platform_device">platform_device</a> contains a minimal runnable example of device tree manipulation.</p>
+<p><a href="#platform-device">platform_device</a> contains a minimal runnable example of device tree manipulation.</p>
 </div>
 <div class="paragraph">
 <p>Device trees serve to reduce the need for hardware vendors to patch the kernel: they just provide a device tree file instead, which is much simpler.</p>
@@ -6998,7 +7006,7 @@ cat f
 <p>as mentioned at: <a href="https://lists.gnu.org/archive/html/qemu-discuss/2017-02/msg00051.html" class="bare">https://lists.gnu.org/archive/html/qemu-discuss/2017-02/msg00051.html</a></p>
 </div>
 <div class="paragraph">
-<p><a href="#gem5-fs_biglittle">gem5 fs_bigLITTLE</a> 2a9573f5942b5416fb0570cf5cb6cdecba733392 can also generate its own DTB.</p>
+<p><a href="#gem5-fs-biglittle">gem5 fs_bigLITTLE</a> 2a9573f5942b5416fb0570cf5cb6cdecba733392 can also generate its own DTB.</p>
 </div>
 <div class="paragraph">
 <p>gem5 can generate DTBs on ARM with <code>--generate-dtb</code>. The generated DTB is placed in the <a href="#m5out-directory">m5out directory</a> named as <code>system.dtb</code>.</p>
@@ -7349,7 +7357,7 @@ qw er</pre>
 <div class="content">
 <pre>./run \
   --arch aarch64 \
-  --userland "$(./getvar --arch aarch64 buildroot_target_dir)/bin/sh"  \
+  --userland "$(./getvar --arch aarch64 buildroot_target_dir)/bin/sh" \
   --userland-args='-c "uname -a &amp;&amp; pwd"' \
 ;</pre>
 </div>
@@ -7550,19 +7558,31 @@ qemu: uncaught target signal 6 (Aborted) - core dumped</pre>
 <pre>ld: cannot find -lopenblas</pre>
 </div>
 </div>
+<div class="sect4">
+<h5 id="cpp-static-and-pthreads"><a class="anchor" href="#cpp-static-and-pthreads"></a><a class="link" href="#cpp-static-and-pthreads">10.5.1.1. C++ static and pthreads</a></h5>
 <div class="paragraph">
-<p><code>g++</code> and pthreads also causes issues: <a href="https://stackoverflow.com/questions/35116327/when-g-static-link-pthread-cause-segmentation-fault-why" class="bare">https://stackoverflow.com/questions/35116327/when-g-static-link-pthread-cause-segmentation-fault-why</a></p>
+<p><code>g++</code> and pthreads also causes issues:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p><a href="https://stackoverflow.com/questions/35116327/when-g-static-link-pthread-cause-segmentation-fault-why" class="bare">https://stackoverflow.com/questions/35116327/when-g-static-link-pthread-cause-segmentation-fault-why</a></p>
+</li>
+<li>
+<p><a href="https://stackoverflow.com/questions/58848694/gcc-whole-archive-recipe-for-static-linking-to-pthread-stopped-working-in-rec" class="bare">https://stackoverflow.com/questions/58848694/gcc-whole-archive-recipe-for-static-linking-to-pthread-stopped-working-in-rec</a></p>
+</li>
+</ul>
 </div>
 <div class="paragraph">
-<p>As a consequence, the following fails:</p>
+<p>As a consequence, the following just hangs as of LKMC ca0403849e03844a328029d70c08556155dc1cd0 + 1 the example <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/std_atomic.cpp">userland/cpp/atomic/std_atomic.cpp</a>:</p>
 </div>
 <div class="literalblock">
 <div class="content">
-<pre>./run --userland userland/cpp/atomic.cpp --static</pre>
+<pre>./run --userland userland/cpp/atomic/std_atomic.cpp --static</pre>
 </div>
 </div>
 <div class="paragraph">
-<p>with error:</p>
+<p>And before that, it used to fail with other randomly different errors, e.g.:</p>
 </div>
 <div class="literalblock">
 <div class="content">
@@ -7571,10 +7591,10 @@ qemu-x86_64: /path/to/linux-kernel-module-cheat/submodules/qemu/accel/tcg/cpu-ex
 </div>
 </div>
 <div class="paragraph">
-<p>and if we manually build and run natively on host it segfaults.</p>
+<p>And a native Ubuntu 18.04 AMD64 run with static compilation segfaults.</p>
 </div>
 <div class="paragraph">
-<p>If we hack the compilation command to do instead:</p>
+<p>The workaround:</p>
 </div>
 <div class="literalblock">
 <div class="content">
@@ -7582,7 +7602,8 @@ qemu-x86_64: /path/to/linux-kernel-module-cheat/submodules/qemu/accel/tcg/cpu-ex
 </div>
 </div>
 <div class="paragraph">
-<p>then it works. We should automate that at some point.</p>
+<p>fixes some of the problems, but not all, so we are just skipping those tests for now.</p>
+</div>
 </div>
 </div>
 </div>
@@ -7739,7 +7760,7 @@ qemu-x86_64: /path/to/linux-kernel-module-cheat/submodules/qemu/accel/tcg/cpu-ex
 <pre>./run \
   --emulator gem5 \
   --static userland/arch/x86_64/freestanding/linux/hello.S \
-  --userland  \
+  --userland \
   --trace-stdout \
   --trace ExecAll,SyscallBase,SyscallVerbose \
 ;</pre>
@@ -8072,7 +8093,7 @@ hello
 <p>less host disk usage, no need to copy the entire <code>./getvar out_rootfs_overlay_dir</code> to the image again</p>
 </li>
 <li>
-<p>no need to worry about <a href="#br2_target_rootfs_ext2_size">BR2_TARGET_ROOTFS_EXT2_SIZE</a></p>
+<p>no need to worry about <a href="#br2-target-rootfs-ext2-size">BR2_TARGET_ROOTFS_EXT2_SIZE</a></p>
 </li>
 </ul>
 </div>
@@ -8234,7 +8255,7 @@ a crash or deadlock.</pre>
 </div>
 </div>
 <div class="paragraph">
-<p>but there is not terminal on the VNC window, just the <a href="#config_logo">CONFIG_LOGO</a> penguin.</p>
+<p>but there is not terminal on the VNC window, just the <a href="#config-logo">CONFIG_LOGO</a> penguin.</p>
 </div>
 <div class="sect3">
 <h4 id="quit-qemu-from-text-mode"><a class="anchor" href="#quit-qemu-from-text-mode"></a><a class="link" href="#quit-qemu-from-text-mode">13.1.1. Quit QEMU from text mode</a></h4>
@@ -8278,7 +8299,7 @@ a crash or deadlock.</pre>
 </div>
 </div>
 <div class="paragraph">
-<p>Outcome: you see a penguin due to <a href="#config_logo">CONFIG_LOGO</a>.</p>
+<p>Outcome: you see a penguin due to <a href="#config-logo">CONFIG_LOGO</a>.</p>
 </div>
 <div class="paragraph">
 <p>For a more exciting GUI experience, see: <a href="#x11">Section 13.4, &#8220;X11 Buildroot&#8221;</a></p>
@@ -8484,7 +8505,7 @@ CONFIG_DRM_VIRTIO_GPU=y</pre>
 </div>
 </div>
 <div class="paragraph">
-<p>The <a href="#config_logo">CONFIG_LOGO</a> penguin only appears after several seconds, together with kernel messages of type:</p>
+<p>The <a href="#config-logo">CONFIG_LOGO</a> penguin only appears after several seconds, together with kernel messages of type:</p>
 </div>
 <div class="literalblock">
 <div class="content">
@@ -8718,7 +8739,7 @@ xeyes</pre>
 <p>the easiest way to find them out is to just list <code>"$(./getvar buildroot_build_build_dir)/x*</code>.</p>
 </div>
 <div class="paragraph">
-<p>TODO as of: c2696c978d6ca88e8b8599c92b1beeda80eb62b2 I noticed that <code>startx</code> leads to a <a href="#bug_on">BUG_ON</a>:</p>
+<p>TODO as of: c2696c978d6ca88e8b8599c92b1beeda80eb62b2 I noticed that <code>startx</code> leads to a <a href="#bug-on">BUG_ON</a>:</p>
 </div>
 <div class="literalblock">
 <div class="content">
@@ -10092,7 +10113,7 @@ mount</pre>
 </ul>
 </div>
 <div class="paragraph">
-<p>The debug highest level is a bit more magic, see: <a href="#pr_debug">Section 15.4.3, &#8220;pr_debug&#8221;</a> for more info.</p>
+<p>The debug highest level is a bit more magic, see: <a href="#pr-debug">Section 15.4.3, &#8220;pr_debug&#8221;</a> for more info.</p>
 </div>
 <div class="sect3">
 <h4 id="procsyskernelprintk"><a class="anchor" href="#procsyskernelprintk"></a><a class="link" href="#procsyskernelprintk">15.4.1. /proc/sys/kernel/printk</a></h4>
@@ -10282,7 +10303,7 @@ early_param("quiet", quiet_kernel);</pre>
 </div>
 </div>
 <div class="sect3">
-<h4 id="ignore_loglevel"><a class="anchor" href="#ignore_loglevel"></a><a class="link" href="#ignore_loglevel">15.4.2. ignore_loglevel</a></h4>
+<h4 id="ignore-loglevel"><a class="anchor" href="#ignore-loglevel"></a><a class="link" href="#ignore-loglevel">15.4.2. ignore_loglevel</a></h4>
 <div class="literalblock">
 <div class="content">
 <pre>./run --kernel-cli 'ignore_loglevel'</pre>
@@ -10301,7 +10322,7 @@ early_param("quiet", quiet_kernel);</pre>
 </div>
 </div>
 <div class="sect3">
-<h4 id="pr_debug"><a class="anchor" href="#pr_debug"></a><a class="link" href="#pr_debug">15.4.3. pr_debug</a></h4>
+<h4 id="pr-debug"><a class="anchor" href="#pr-debug"></a><a class="link" href="#pr-debug">15.4.3. pr_debug</a></h4>
 <div class="paragraph">
 <p><a href="https://stackoverflow.com/questions/28936199/why-is-pr-debug-of-the-linux-kernel-not-giving-any-output/49835405#49835405" class="bare">https://stackoverflow.com/questions/28936199/why-is-pr-debug-of-the-linux-kernel-not-giving-any-output/49835405#49835405</a></p>
 </div>
@@ -10399,7 +10420,7 @@ insmod myprintk.ko</pre>
 <p>Get ready for the noisiest boot ever, I think it overflows the <code>printk</code> buffer and funny things happen.</p>
 </div>
 <div class="sect4">
-<h5 id="pr_debug-printkkern_debug"><a class="anchor" href="#pr_debug-printkkern_debug"></a><a class="link" href="#pr_debug-printkkern_debug">15.4.3.1. pr_debug != printk(KERN_DEBUG</a></h5>
+<h5 id="pr-debug-is-different-from-printk-kern-debug"><a class="anchor" href="#pr-debug-is-different-from-printk-kern-debug"></a><a class="link" href="#pr-debug-is-different-from-printk-kern-debug">15.4.3.1. pr_debug != printk(KERN_DEBUG</a></h5>
 <div class="paragraph">
 <p>When <code>CONFIG_DYNAMIC_DEBUG</code> is set,  <code>printk(KERN_DEBUG</code> is not the exact same as <code>pr_debug(</code> since <code>printk(KERN_DEBUG</code> messages are visible with:</p>
 </div>
@@ -10616,7 +10637,7 @@ ffffffffc0002300 B lkmc_dep     [dep]</pre>
 <p>This requires <code>CONFIG_KALLSYMS_ALL=y</code>.</p>
 </div>
 <div class="paragraph">
-<p>Dependency information is stored by the kernel module build system in the <code>.ko</code> files' <a href="#module_info">MODULE_INFO</a>, e.g.:</p>
+<p>Dependency information is stored by the kernel module build system in the <code>.ko</code> files' <a href="#module-info">MODULE_INFO</a>, e.g.:</p>
 </div>
 <div class="literalblock">
 <div class="content">
@@ -10636,7 +10657,7 @@ ffffffffc0002300 B lkmc_dep     [dep]</pre>
 </div>
 <div class="literalblock">
 <div class="content">
-<pre>strings 3 dep2.ko  | grep -E 'depends'</pre>
+<pre>strings 3 dep2.ko | grep -E 'depends'</pre>
 </div>
 </div>
 <div class="paragraph">
@@ -10755,7 +10776,7 @@ buildroot_dep          16384  1 buildroot_dep2</pre>
 </div>
 </div>
 <div class="sect3">
-<h4 id="module_info"><a class="anchor" href="#module_info"></a><a class="link" href="#module_info">15.6.3. MODULE_INFO</a></h4>
+<h4 id="module-info"><a class="anchor" href="#module-info"></a><a class="link" href="#module-info">15.6.3. MODULE_INFO</a></h4>
 <div class="paragraph">
 <p>Module metadata is stored on module files at compile time. Some of the fields can be retrieved through the <code>THIS_MODULE</code> <code>struct module</code>:</p>
 </div>
@@ -10877,7 +10898,7 @@ vermagic:       4.17.0 SMP mod_unload modversions</pre>
 <div class="sect3">
 <h4 id="vermagic"><a class="anchor" href="#vermagic"></a><a class="link" href="#vermagic">15.6.4. vermagic</a></h4>
 <div class="paragraph">
-<p>Vermagic is a magic string present in the kernel and on <a href="#module_info">MODULE_INFO</a> of kernel modules. It is used to verify that the kernel module was compiled against a compatible kernel version and relevant configuration:</p>
+<p>Vermagic is a magic string present in the kernel and on <a href="#module-info">MODULE_INFO</a> of kernel modules. It is used to verify that the kernel module was compiled against a compatible kernel version and relevant configuration:</p>
 </div>
 <div class="literalblock">
 <div class="content">
@@ -10954,7 +10975,7 @@ vermagic:       4.17.0 SMP mod_unload modversions</pre>
 </div>
 </div>
 <div class="sect3">
-<h4 id="init_module"><a class="anchor" href="#init_module"></a><a class="link" href="#init_module">15.6.5. init_module</a></h4>
+<h4 id="init-module"><a class="anchor" href="#init-module"></a><a class="link" href="#init-module">15.6.5. init_module</a></h4>
 <div class="paragraph">
 <p><code>init_module</code> and <code>cleanup_module</code> are an older alternative to the <code>module_init</code> and <code>module_exit</code> macros:</p>
 </div>
@@ -11232,7 +11253,7 @@ Kernel Offset: disabled
 </div>
 </div>
 <div class="sect4">
-<h5 id="bug_on"><a class="anchor" href="#bug_on"></a><a class="link" href="#bug_on">15.7.1.2. BUG_ON</a></h5>
+<h5 id="bug-on"><a class="anchor" href="#bug-on"></a><a class="link" href="#bug-on">15.7.1.2. BUG_ON</a></h5>
 <div class="paragraph">
 <p>Basically just calls <code>panic("BUG!")</code> for most archs.</p>
 </div>
@@ -11506,7 +11527,7 @@ CR2: 0000000000000000
 <div class="ulist">
 <ul>
 <li>
-<p><a href="#gdb-module_init">GDB module_init</a></p>
+<p><a href="#gdb-module-init">GDB module_init</a></p>
 </li>
 <li>
 <p><a href="#kernel-module-stack-trace-to-source-line">Kernel module stack trace to source line</a> post-mortem method</p>
@@ -11515,7 +11536,7 @@ CR2: 0000000000000000
 </div>
 </div>
 <div class="sect3">
-<h4 id="dump_stack"><a class="anchor" href="#dump_stack"></a><a class="link" href="#dump_stack">15.7.3. dump_stack</a></h4>
+<h4 id="dump-stack"><a class="anchor" href="#dump-stack"></a><a class="link" href="#dump-stack">15.7.3. dump_stack</a></h4>
 <div class="paragraph">
 <p>The <code>dump_stack</code> function produces a stack trace much like panic and oops, but causes no problems and we return to the normal control flow, and can cleanly remove the module afterwards:</p>
 </div>
@@ -11529,9 +11550,9 @@ CR2: 0000000000000000
 </div>
 </div>
 <div class="sect3">
-<h4 id="warn_on"><a class="anchor" href="#warn_on"></a><a class="link" href="#warn_on">15.7.4. WARN_ON</a></h4>
+<h4 id="warn-on"><a class="anchor" href="#warn-on"></a><a class="link" href="#warn-on">15.7.4. WARN_ON</a></h4>
 <div class="paragraph">
-<p>The <code>WARN_ON</code> macro basically just calls <a href="#dump_stack">dump_stack</a>.</p>
+<p>The <code>WARN_ON</code> macro basically just calls <a href="#dump-stack">dump_stack</a>.</p>
 </div>
 <div class="paragraph">
 <p>One extra side effect is that we can make it also panic with:</p>
@@ -11713,7 +11734,7 @@ echo $?</pre>
 <p>user and date to dummy values with <code>KBUILD_BUILD_USER</code> and <code>KBUILD_BUILD_TIMESTAMP</code></p>
 </li>
 <li>
-<p>hostname to the kernel git commit with <code>KBUILD_BUILD_HOST</code> and  <code>KBUILD_BUILD_VERSION</code></p>
+<p>hostname to the kernel git commit with <code>KBUILD_BUILD_HOST</code> and <code>KBUILD_BUILD_VERSION</code></p>
 </li>
 </ul>
 </div>
@@ -11779,7 +11800,7 @@ echo $?</pre>
 <p>You basically can only do <code>open</code>, <code>close</code>, <code>read</code>, <code>write</code>, and <code>lseek</code> on sysfs files.</p>
 </div>
 <div class="paragraph">
-<p>It is similar to a <a href="#seq_file">seq_file</a> file operation, except that write is also implemented.</p>
+<p>It is similar to a <a href="#seq-file">seq_file</a> file operation, except that write is also implemented.</p>
 </div>
 <div class="paragraph">
 <p>TODO: what are those <code>kobject</code> structs? Make a more complex example that shows what they can do.</p>
@@ -11992,7 +12013,7 @@ echo $?</pre>
 </div>
 </div>
 <div class="sect3">
-<h4 id="seq_file"><a class="anchor" href="#seq_file"></a><a class="link" href="#seq_file">15.9.2. seq_file</a></h4>
+<h4 id="seq-file"><a class="anchor" href="#seq-file"></a><a class="link" href="#seq-file">15.9.2. seq_file</a></h4>
 <div class="paragraph">
 <p>Writing trivial read <a href="#file-operations">File operations</a> is repetitive and error prone. The <code>seq_file</code> API makes the process much easier for those trivial cases:</p>
 </div>
@@ -12053,9 +12074,9 @@ echo $?</pre>
 </ul>
 </div>
 <div class="sect4">
-<h5 id="seq_file-single_open"><a class="anchor" href="#seq_file-single_open"></a><a class="link" href="#seq_file-single_open">15.9.2.1. seq_file single_open</a></h5>
+<h5 id="seq-file-single-open"><a class="anchor" href="#seq-file-single-open"></a><a class="link" href="#seq-file-single-open">15.9.2.1. seq_file single_open</a></h5>
 <div class="paragraph">
-<p>If you have the entire read output upfront, <code>single_open</code> is an even more convenient version of <a href="#seq_file">seq_file</a>:</p>
+<p>If you have the entire read output upfront, <code>single_open</code> is an even more convenient version of <a href="#seq-file">seq_file</a>:</p>
 </div>
 <div class="literalblock">
 <div class="content">
@@ -12915,7 +12936,7 @@ request_irq irq = 1 ret = 0</pre>
 <p>and furthermore interrupt <code>1</code> and <code>12</code> happen immediately TODO why, were they somehow pending?</p>
 </div>
 <div class="paragraph">
-<p>So so see something interesting, you need to monitor an interrupt that is more rare than the keyboard, e.g. <a href="#platform_device">platform_device</a>.</p>
+<p>So so see something interesting, you need to monitor an interrupt that is more rare than the keyboard, e.g. <a href="#platform-device">platform_device</a>.</p>
 </div>
 </div>
 <div class="sect3">
@@ -12994,7 +13015,7 @@ echo $?</pre>
 </div>
 </div>
 <div class="sect3">
-<h4 id="virt_to_phys"><a class="anchor" href="#virt_to_phys"></a><a class="link" href="#virt_to_phys">15.13.2. virt_to_phys</a></h4>
+<h4 id="virt-to-phys"><a class="anchor" href="#virt-to-phys"></a><a class="link" href="#virt-to-phys">15.13.2. virt_to_phys</a></h4>
 <div class="paragraph">
 <p>Convert a virtual address to physical:</p>
 </div>
@@ -13276,7 +13297,7 @@ Value at address 0X7C7B800 (0x7ff7dbe01800): 0x12345678</pre>
 </div>
 </div>
 <div class="sect5">
-<h6 id="pagemap_dump-out"><a class="anchor" href="#pagemap_dump-out"></a><a class="link" href="#pagemap_dump-out">15.13.2.1.3. pagemap_dump.out</a></h6>
+<h6 id="pagemap-dump-out"><a class="anchor" href="#pagemap-dump-out"></a><a class="link" href="#pagemap-dump-out">15.13.2.1.3. pagemap_dump.out</a></h6>
 <div class="paragraph">
 <p>Dump the physical address of all pages mapped to a given process using <code>/proc/&lt;pid&gt;/maps</code> and <code>/proc/&lt;pid&gt;/pagemap</code>.</p>
 </div>
@@ -13455,7 +13476,7 @@ pid 63
 <p>I hope to have examples of all methods some day, since I&#8217;m obsessed with visibility.</p>
 </div>
 <div class="sect3">
-<h4 id="config_proc_events"><a class="anchor" href="#config_proc_events"></a><a class="link" href="#config_proc_events">15.14.1. CONFIG_PROC_EVENTS</a></h4>
+<h4 id="config-proc-events"><a class="anchor" href="#config-proc-events"></a><a class="link" href="#config-proc-events">15.14.1. CONFIG_PROC_EVENTS</a></h4>
 <div class="paragraph">
 <p>Logs proc events such as process creation to a <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/netlink.c">netlink socket</a>.</p>
 </div>
@@ -13522,7 +13543,7 @@ a
 </ul>
 </div>
 <div class="sect4">
-<h5 id="config_proc_events-aarch64"><a class="anchor" href="#config_proc_events-aarch64"></a><a class="link" href="#config_proc_events-aarch64">15.14.1.1. CONFIG_PROC_EVENTS aarch64</a></h5>
+<h5 id="config-proc-events-aarch64"><a class="anchor" href="#config-proc-events-aarch64"></a><a class="link" href="#config-proc-events-aarch64">15.14.1.1. CONFIG_PROC_EVENTS aarch64</a></h5>
 <div class="paragraph">
 <p>0111ca406bdfa6fd65a2605d353583b4c4051781 was failing with:</p>
 </div>
@@ -13980,7 +14001,7 @@ instructions_firmware 20708</pre>
 <p>Make it harder to get hacked and easier to notice that you were, at the cost of some (small?) runtime overhead.</p>
 </div>
 <div class="sect3">
-<h4 id="config_fortify_source"><a class="anchor" href="#config_fortify_source"></a><a class="link" href="#config_fortify_source">15.15.1. CONFIG_FORTIFY_SOURCE</a></h4>
+<h4 id="config-fortify-source"><a class="anchor" href="#config-fortify-source"></a><a class="link" href="#config-fortify-source">15.15.1. CONFIG_FORTIFY_SOURCE</a></h4>
 <div class="paragraph">
 <p>Detects buffer overflows for us:</p>
 </div>
@@ -14951,7 +14972,7 @@ tty63::respawn:-/bin/sh
 </div>
 </div>
 <div class="sect3">
-<h4 id="config_logo"><a class="anchor" href="#config_logo"></a><a class="link" href="#config_logo">15.18.4. CONFIG_LOGO</a></h4>
+<h4 id="config-logo"><a class="anchor" href="#config-logo"></a><a class="link" href="#config-logo">15.18.4. CONFIG_LOGO</a></h4>
 <div class="paragraph">
 <p>If you run in <a href="#graphics">Graphics</a>, then you get a Penguin image for <a href="#number-of-cores">every core</a> above the console! <a href="https://askubuntu.com/questions/80938/is-it-possible-to-get-the-tux-logo-on-the-text-based-boot" class="bare">https://askubuntu.com/questions/80938/is-it-possible-to-get-the-tux-logo-on-the-text-based-boot</a></p>
 </div>
@@ -15713,10 +15734,10 @@ Format specific information:
 <div class="ulist">
 <ul>
 <li>
-<p><a href="#pci_min">pci_min</a></p>
+<p><a href="#pci-min">pci_min</a></p>
 </li>
 <li>
-<p><a href="#platform_device">platform_device</a></p>
+<p><a href="#platform-device">platform_device</a></p>
 </li>
 </ul>
 </div>
@@ -15729,7 +15750,7 @@ Format specific information:
 <p>Only tested in x86.</p>
 </div>
 <div class="sect4">
-<h5 id="pci_min"><a class="anchor" href="#pci_min"></a><a class="link" href="#pci_min">18.5.1.1. pci_min</a></h5>
+<h5 id="pci-min"><a class="anchor" href="#pci-min"></a><a class="link" href="#pci-min">18.5.1.1. pci_min</a></h5>
 <div class="paragraph">
 <p>PCI driver for our minimal <code>pci_min.c</code> QEMU fork device:</p>
 </div>
@@ -15978,7 +15999,7 @@ setpci -d 1234:11e9 BASE_ADDRESS_0</pre>
 </div>
 </div>
 <div class="paragraph">
-<p>which writes to the first register of our <a href="#pci_min">pci_min</a> device.</p>
+<p>which writes to the first register of our <a href="#pci-min">pci_min</a> device.</p>
 </div>
 <div class="paragraph">
 <p>The device then fires an interrupt at irq 11, which is unhandled, which leads the kernel to say you are a bad boy:</p>
@@ -16003,7 +16024,7 @@ devmem 0xfeb54000 w 0x12345678</pre>
 </div>
 </div>
 <div class="paragraph">
-<p>Our kernel module handles the interrupt, but does not acknowledge it like our proper <a href="#pci_min">pci_min</a> kernel module, and so it keeps firing, which leads to infinitely many messages being printed:</p>
+<p>Our kernel module handles the interrupt, but does not acknowledge it like our proper <a href="#pci-min">pci_min</a> kernel module, and so it keeps firing, which leads to infinitely many messages being printed:</p>
 </div>
 <div class="literalblock">
 <div class="content">
@@ -16229,7 +16250,7 @@ pci_register_bar(pdev, 0, PCI_BASE_ADDRESS_SPACE_MEMORY, &amp;edu-&gt;mmio);</pr
 <div class="sect3">
 <h4 id="leds"><a class="anchor" href="#leds"></a><a class="link" href="#leds">18.5.3. LEDs</a></h4>
 <div class="paragraph">
-<p>TODO: broken when <code>arm</code>  moved to <code>-M virt</code>, same as <a href="#gpio">GPIO</a>.</p>
+<p>TODO: broken when <code>arm</code> moved to <code>-M virt</code>, same as <a href="#gpio">GPIO</a>.</p>
 </div>
 <div class="paragraph">
 <p>Hack QEMU&#8217;s <code>hw/misc/arm_sysctl.c</code> with a printf:</p>
@@ -16299,7 +16320,7 @@ echo 255 &gt;brightness</pre>
 </div>
 </div>
 <div class="sect3">
-<h4 id="platform_device"><a class="anchor" href="#platform_device"></a><a class="link" href="#platform_device">18.5.4. platform_device</a></h4>
+<h4 id="platform-device"><a class="anchor" href="#platform-device"></a><a class="link" href="#platform-device">18.5.4. platform_device</a></h4>
 <div class="paragraph">
 <p>Minimal platform device example coded into the <code>-M versatilepb</code> SoC of our QEMU fork.</p>
 </div>
@@ -17824,15 +17845,30 @@ cat out/gem5-bench-dhrystone.txt</pre>
 getconf _NPROCESSORS_CONF</pre>
 </div>
 </div>
+<div class="paragraph">
+<p>Or from <a href="#user-mode-simulation">User mode simulation</a>, we can use <a href="#sysconf">sysconf</a> with <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/sysconf.c">userland/linux/sysconf.c</a> or <a href="#cpp-multithreading">C++ multithreading</a>'s <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/thread_hardware_concurrency.cpp">userland/cpp/thread_hardware_concurrency.cpp</a>:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>./run --cpus 2 --emulator gem5 --userland userland/linux/sysconf.c | grep _SC_NPROCESSORS_ONLN
+./run --cpus 2 --emulator gem5 --userland userland/cpp/thread_hardware_concurrency.cpp</pre>
+</div>
+</div>
 <div class="sect5">
 <h6 id="qemu-user-mode-multithreading"><a class="anchor" href="#qemu-user-mode-multithreading"></a><a class="link" href="#qemu-user-mode-multithreading">19.2.2.1.1. QEMU user mode multithreading</a></h6>
 <div class="paragraph">
-<p>TODO why in <a href="#user-mode-simulation">User mode simulation</a> QEMU always shows the number of cores of the host. E.g., both of the following output the same as <code>nproc</code> on the host:</p>
+<p><a href="#user-mode-simulation">User mode simulation</a> QEMU v4.0.0 always shows the number of cores of the host, presumably because the thread switching uses host threads directly which would make that harder to implement.</p>
+</div>
+<div class="paragraph">
+<p>It does not seem possible to make the guest see a different number of cores than what the host has. Full system does have the <code>-smp</code> options, which controls this.</p>
+</div>
+<div class="paragraph">
+<p>E.g., all of of the following output the same as <code>nproc</code> on the host:</p>
 </div>
 <div class="literalblock">
 <div class="content">
 <pre>nproc
-./run --userland userland/cpp/thread_hardware_concurrency.cpp
+./run --cpus 1 --userland userland/cpp/thread_hardware_concurrency.cpp
 ./run --cpus 2 --userland userland/cpp/thread_hardware_concurrency.cpp</pre>
 </div>
 </div>
@@ -17852,7 +17888,7 @@ ps Haux | grep qemu | wc</pre>
 <p>Remember <a href="#qemu-user-mode-does-not-show-stdout-immediately">QEMU user mode does not show stdout immediately</a> though.</p>
 </div>
 <div class="paragraph">
-<p>At 369a47fc6e5c2f4a7f911c1c058b6088f8824463 + 1 QEMU appears to spawn 3 host threads plus one for every new guest thread created.  Remember that <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/pthread_count.c">userland/posix/pthread_count.c</a> spawns N + 1 total threads if you count the <code>main</code> thread.</p>
+<p>At 369a47fc6e5c2f4a7f911c1c058b6088f8824463 + 1 QEMU appears to spawn 3 host threads plus one for every new guest thread created. Remember that <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/pthread_count.c">userland/posix/pthread_count.c</a> spawns N + 1 total threads if you count the <code>main</code> thread.</p>
 </div>
 </div>
 <div class="sect5">
@@ -18401,6 +18437,9 @@ m5 dumpstats</pre>
 <div class="sect4">
 <h5 id="bst-vs-heap-vs-hashmap"><a class="anchor" href="#bst-vs-heap-vs-hashmap"></a><a class="link" href="#bst-vs-heap-vs-hashmap">19.2.3.2. BST vs heap vs hashmap</a></h5>
 <div class="paragraph">
+<p>TODO: move benchmark graph from <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/bst_vs_heap_vs_hashmap.cpp">userland/cpp/bst_vs_heap_vs_hashmap.cpp</a> to <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/algorithm/set">userland/algorithm/set</a>.</p>
+</div>
+<div class="paragraph">
 <p>The following benchmark setup works both:</p>
 </div>
 <div class="ulist">
@@ -18434,8 +18473,12 @@ m5 dumpstats</pre>
 </div>
 <div class="literalblock">
 <div class="content">
-<pre>./build-userland-in-tree --force-rebuild --optimization-level 3 ./userland/cpp/bst_vs_heap_vs_hashmap.cpp
-./userland/cpp/bst_vs_heap_vs_hashmap.out 10000000 10000 | tee bst_vs_heap_vs_hashmap.dat
+<pre>./build-userland-in-tree \
+  --force-rebuild \
+  --optimization-level 3 \
+  ./userland/cpp/bst_vs_heap_vs_hashmap.cpp \
+;
+./userland/cpp/bst_vs_heap_vs_hashmap.out 10000000 10000 0 | tee bst_vs_heap_vs_hashmap.dat
 gnuplot \
   -e 'input_noext="bst_vs_heap_vs_hashmap"' \
   -e 'heap_zoom_max=50' \
@@ -18446,10 +18489,10 @@ xdg-open bst_vs_heap_vs_hashmap.tmp.png</pre>
 </div>
 </div>
 <div class="paragraph">
-<p>The parameters <code>heap_zoom_max</code> and <code>hashmap_zoom_max</code> are chosen manually interactively to best showcase the regions of interest in  those plots.</p>
+<p>The parameters <code>heap_zoom_max</code> and <code>hashmap_zoom_max</code> are chosen manually interactively to best showcase the regions of interest in those plots.</p>
 </div>
 <div class="paragraph">
-<p>First we build the benchmark with <a href="#m5ops-instructions">m5ops instructions</a> enabled, and then we run it and extract the stats:</p>
+<p>To benchmark on gem5, we first build the benchmark with <a href="#m5ops-instructions">m5ops instructions</a> enabled, and then we run it and extract the stats:</p>
 </div>
 <div class="literalblock">
 <div class="content">
@@ -18465,7 +18508,7 @@ xdg-open bst_vs_heap_vs_hashmap.tmp.png</pre>
   --emulator gem5 \
   --static \
   --userland userland/cpp/bst_vs_heap_vs_hashmap.cpp \
-  --userland-args='100000' \
+  --userland-args='100000 1 0' \
   -- \
   --cpu-type=DerivO3CPU \
   --caches \
@@ -18674,7 +18717,7 @@ cblas_dgemm(      CblasColMajor, CblasNoTrans, CblasTrans,3,3,2  ,1,    A,3,  B,
 <div class="ulist">
 <ul>
 <li>
-<p><a href="#br2_target_rootfs_ext2_size">BR2_TARGET_ROOTFS_EXT2_SIZE</a> if the unpacked inputs are large</p>
+<p><a href="#br2-target-rootfs-ext2-size">BR2_TARGET_ROOTFS_EXT2_SIZE</a> if the unpacked inputs are large</p>
 </li>
 <li>
 <p><a href="#memory-size">Memory size</a>, unless you want to meet the OOM killer, which is admittedly kind of fun</p>
@@ -20050,7 +20093,7 @@ clock=500</pre>
 </div>
 </div>
 <div class="sect2">
-<h3 id="gem5-fs_biglittle"><a class="anchor" href="#gem5-fs_biglittle"></a><a class="link" href="#gem5-fs_biglittle">19.12. gem5 fs_bigLITTLE</a></h3>
+<h3 id="gem5-fs-biglittle"><a class="anchor" href="#gem5-fs-biglittle"></a><a class="link" href="#gem5-fs-biglittle">19.12. gem5 fs_bigLITTLE</a></h3>
 <div class="paragraph">
 <p>By default, we use <code>configs/example/fs.py</code> script.</p>
 </div>
@@ -20068,7 +20111,7 @@ clock=500</pre>
 <div class="ulist">
 <ul>
 <li>
-<p>more representative of mobile ARM SoCs, which almost always have  big little cluster</p>
+<p>more representative of mobile ARM SoCs, which almost always have big little cluster</p>
 </li>
 <li>
 <p>simpler than <code>fs.py</code>, and therefore easier to understand and modify</p>
@@ -20321,8 +20364,8 @@ Exiting @ tick 18446744073709551615 because simulate() limit reached</pre>
 <div class="literalblock">
 <div class="content">
 <pre>sudo apt-get install clang
-./build-gem5 --clang
-./run --clang --emulator gem5</pre>
+./build-gem5 --gem5-clang
+./run --emulator gem5 --gem5-clang</pre>
 </div>
 </div>
 </div>
@@ -21331,7 +21374,7 @@ Text::end()
 <p>But it has been widely overused to insanity. It likely also exists partly because when the project started in 2003 C++ compilers weren&#8217;t that good, so you couldn&#8217;t rely on features like templates that much.</p>
 </div>
 <div class="sect4">
-<h5 id="gem5-the_isa"><a class="anchor" href="#gem5-the_isa"></a><a class="link" href="#gem5-the_isa">19.19.6.1. gem5 THE_ISA</a></h5>
+<h5 id="gem5-the-isa"><a class="anchor" href="#gem5-the-isa"></a><a class="link" href="#gem5-the-isa">19.19.6.1. gem5 THE_ISA</a></h5>
 <div class="paragraph">
 <p>Generated code at: <code>build/&lt;ISA&gt;/config/the_isa.hh</code> which contains amongst other lines:</p>
 </div>
@@ -21756,7 +21799,7 @@ make menuconfig</pre>
 </div>
 </div>
 <div class="sect2">
-<h3 id="br2_target_rootfs_ext2_size"><a class="anchor" href="#br2_target_rootfs_ext2_size"></a><a class="link" href="#br2_target_rootfs_ext2_size">20.7. BR2_TARGET_ROOTFS_EXT2_SIZE</a></h3>
+<h3 id="br2-target-rootfs-ext2-size"><a class="anchor" href="#br2-target-rootfs-ext2-size"></a><a class="link" href="#br2-target-rootfs-ext2-size">20.7. BR2_TARGET_ROOTFS_EXT2_SIZE</a></h3>
 <div class="paragraph">
 <p>When adding new large package to the Buildroot root filesystem, it may fail with the message:</p>
 </div>
@@ -22660,16 +22703,75 @@ echo 1 &gt; /proc/sys/vm/overcommit_memory
 </li>
 <li>
 <p><a href="https://en.cppreference.com/w/cpp/header/atomic"><code>&lt;atomic&gt;</code></a>: <a href="#cpp17">C++17 N4659 standards draft</a> 32 "Atomic operations library"</p>
-<div class="ulist">
-<ul>
-<li>
-<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic.cpp">userland/cpp/atomic.cpp</a></p>
 </li>
 </ul>
 </div>
+<div class="sect4">
+<h5 id="atomic-cpp"><a class="anchor" href="#atomic-cpp"></a><a class="link" href="#atomic-cpp">21.2.1.1. atomic.cpp</a></h5>
+<div class="paragraph">
+<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/">userland/cpp/atomic/</a></p>
+</div>
+<div class="paragraph">
+<p>In this set of examples, we exemplify various synchronization mechanisms, including assembly specific ones, by using the convenience of C++ multithreading:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/main.hpp">userland/cpp/atomic/main.hpp</a>: contains all the code which is then specialized in spearated cpp files with macros</p>
+</li>
+<li>
+<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/aarch64_add.cpp">userland/cpp/atomic/aarch64_add.cpp</a>: non synchronized aarch64 inline assembly</p>
+</li>
+<li>
+<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/aarch64_ldadd.cpp">userland/cpp/atomic/aarch64_ldadd.cpp</a>: synchronized aarch64 inline assembly</p>
+</li>
+<li>
+<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/fail.cpp">userland/cpp/atomic/fail.cpp</a>: non synchronized C operator ``</p>
+</li>
+<li>
+<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/mutex.cpp">userland/cpp/atomic/mutex.cpp</a>: synchronized <code>std::mutex</code></p>
+</li>
+<li>
+<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/std_atomic.cpp">userland/cpp/atomic/std_atomic.cpp</a>: synchronized <code>std::atomic_ulong</code></p>
+</li>
+<li>
+<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/x86_64_inc.cpp">userland/cpp/atomic/x86_64_inc.cpp</a>: non synchronized x86_64 inline assembly</p>
+</li>
+<li>
+<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/x86_64_lock_inc.cpp">userland/cpp/atomic/x86_64_lock_inc.cpp</a>: synchronized x86_64 inline assembly</p>
 </li>
 </ul>
 </div>
+<div class="paragraph">
+<p>All examples do exactly the same thing: span N threads and loop M times in each thread incrementing a global integer.</p>
+</div>
+<div class="paragraph">
+<p>For inputs large enough, the non-synchronized examples are extremely likely to produce "wrong" results, for example on <a href="#p51">P51</a> Ubuntu 18.04 native with 2 threads and 10000 loops:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>./fail.out 2 10000</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>we could get an output such as:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>expect 20000
+global 12676</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The actual value is much smaller, because the threads have often overwritten one another with older values.</p>
+</div>
+<div class="paragraph">
+<p>Interestingly, with <code>--optimization-level 3</code>, the results almost always match "by chance", because GCC optimizes our for loop to a single addition! Not sure how to force things nicely here without having arch specific assembly, the following technique comes somewhat close: <a href="https://stackoverflow.com/questions/37786547/enforcing-statement-order-in-c/56865717#56865717" class="bare">https://stackoverflow.com/questions/37786547/enforcing-statement-order-in-c/56865717#56865717</a> but I don&#8217;t want to put our addition in a <code>noinline</code> function to avoid the extra function call!</p>
+</div>
+<div class="paragraph">
+<p>This setup can also be used to benchmark different synchronization mechanisms. <code>std::mutex</code> was about 2x slower with two cores than <code>std::atomic</code>, presumably because it relies on the <code>futex</code> system call as can be seen from <code>sudo strace -f -s999 -v</code> logs, while <code>std::atomic</code> uses just userland instructions: <a href="https://www.quora.com/How-does-std-atomic-work-in-C++11/answer/Ciro-Santilli" class="bare">https://www.quora.com/How-does-std-atomic-work-in-C++11/answer/Ciro-Santilli</a></p>
+</div>
+</div>
 </div>
 <div class="sect3">
 <h4 id="cpp-standards"><a class="anchor" href="#cpp-standards"></a><a class="link" href="#cpp-standards">21.2.2. C++ standards</a></h4>
@@ -25904,7 +26006,7 @@ taskset -c 1 ./userland/arch/x86_64/rdtscp.out | tail -n 1</pre>
 <div class="sect3">
 <h4 id="x86-lock-prefix"><a class="anchor" href="#x86-lock-prefix"></a><a class="link" href="#x86-lock-prefix">23.14.1. x86 LOCK prefix</a></h4>
 <div class="paragraph">
-<p>Inline assembly example at: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic.cpp">userland/cpp/atomic.cpp</a></p>
+<p>Inline assembly example at: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/x86_64_lock_inc.cpp">userland/cpp/atomic/x86_64_lock_inc.cpp</a>, see also: <a href="#atomic-cpp">atomic.cpp</a>.</p>
 </div>
 <div class="paragraph">
 <p>Ensures that memory modifications are visible across all CPUs, which is fundamental for thread synchronization.</p>
@@ -26001,7 +26103,7 @@ taskset -c 1 ./userland/arch/x86_64/rdtscp.out | tail -n 1</pre>
 <p>ARM is generally considered a RISC instruction set, although there are some more complex instructions which would not generally be classified as purely RISC.</p>
 </div>
 <div class="paragraph">
-<p>ARM is developed by the British funded company ARM Holdings: <a href="https://en.wikipedia.org/wiki/Arm_Holdings" class="bare">https://en.wikipedia.org/wiki/Arm_Holdings</a> which originated as a joint venture between Acorn Computers, Apple  and VLSI Technology in 1990.</p>
+<p>ARM is developed by the British funded company ARM Holdings: <a href="https://en.wikipedia.org/wiki/Arm_Holdings" class="bare">https://en.wikipedia.org/wiki/Arm_Holdings</a> which originated as a joint venture between Acorn Computers, Apple and VLSI Technology in 1990.</p>
 </div>
 <div class="paragraph">
 <p>ARM Holdings was bought by the Japanese giant SoftBank in 2016.</p>
@@ -27794,7 +27896,7 @@ AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.</p>
 <div class="ulist">
 <ul>
 <li>
-<p>LDADD: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic.cpp">userland/cpp/atomic.cpp</a></p>
+<p>LDADD: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/aarch64_ldadd.cpp">userland/cpp/atomic/aarch64_ldadd.cpp</a>, see also: <a href="#atomic-cpp">atomic.cpp</a>.</p>
 </li>
 </ul>
 </div>
@@ -28966,9 +29068,6 @@ IN: main
 <p>which does an <code>eret</code> and jumps back to 0x4000209c, which is 4 bytes and therefore one instruction after where SVC was taken at 0x40002098.</p>
 </div>
 <div class="paragraph">
-<p>In QEMU,  and then we just continue running from the exception handler address.</p>
-</div>
-<div class="paragraph">
 <p>On the terminal output, we observe the initial values of:</p>
 </div>
 <div class="ulist">
@@ -29390,7 +29489,7 @@ IN: main
 <div class="sect3">
 <h4 id="arm-timer"><a class="anchor" href="#arm-timer"></a><a class="link" href="#arm-timer">27.8.4. ARM timer</a></h4>
 <div class="paragraph">
-<p>The ARM timer is the simplest way to generate hardware interrupts periodically, and therefore serves as the simples example of <a href="#arm-gic">ARM GIC</a>  usage.</p>
+<p>The ARM timer is the simplest way to generate hardware interrupts periodically, and therefore serves as the simples example of <a href="#arm-gic">ARM GIC</a> usage.</p>
 </div>
 <div class="paragraph">
 <p>Working on QEMU: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/aarch64/timer.c">baremetal/arch/aarch64/timer.c</a></p>
@@ -32226,7 +32325,7 @@ gem5_internal="$(pwd)/gem5-internal"</pre>
 </div>
 </div>
 <div class="paragraph">
-<p>Next, when you want to build with the private repository, use the <code>--gem5-build-dir</code> and  <code>--gem5-source-dir</code> argument to override our default gem5 source and build locations:</p>
+<p>Next, when you want to build with the private repository, use the <code>--gem5-build-dir</code> and <code>--gem5-source-dir</code> argument to override our default gem5 source and build locations:</p>
 </div>
 <div class="literalblock">
 <div class="content">
@@ -33216,7 +33315,7 @@ cd linux-kernel-module-cheat-release</pre>
 </div>
 <div class="literalblock">
 <div class="content">
-<pre>git tag -a  v3.0
+<pre>git tag -a v3.0
 # Describe the release int the tag message.
 git push --follow-tags
 ./release-zip --all-archs