831d9d8372

2026-01-24 18:51:36 +01:00 · 2020-03-20 00:00:00 +00:00
parent 45877a196f
commit 3b4929ec44
1 changed files with 412 additions and 134 deletions
--- a/index.html
+++ b/index.html
@@ -477,7 +477,7 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 <li><a href="#about-the-qemu-buildroot-setup">1.1.3. About the QEMU Buildroot setup</a></li>
 </ul>
 </li>
-<li><a href="#dry-run-to-get-commands-for-your-project">1.2. Dry run to get commands for your project</a></li>
+<li><a href="#dry-run">1.2. Dry run to get commands for your project</a></li>
 <li><a href="#gem5-buildroot-setup">1.3. gem5 Buildroot setup</a>
 <ul class="sectlevel3">
 <li><a href="#about-the-gem5-buildroot-setup">1.3.1. About the gem5 Buildroot setup</a></li>
@@ -1135,14 +1135,15 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 <li><a href="#pass-extra-options-to-gem5">19.6. Pass extra options to gem5</a></li>
 <li><a href="#m5ops">19.7. m5ops</a>
 <ul class="sectlevel3">
-<li><a href="#m5">19.7.1. m5</a>
+<li><a href="#gem5-m5-executable">19.7.1. gem5 m5 executable</a>
 <ul class="sectlevel4">
 <li><a href="#m5-exit">19.7.1.1. m5 exit</a></li>
-<li><a href="#m5-fail">19.7.1.2. m5 fail</a></li>
-<li><a href="#m5-writefile">19.7.1.3. m5 writefile</a></li>
-<li><a href="#m5-readfile">19.7.1.4. m5 readfile</a></li>
-<li><a href="#m5-initparam">19.7.1.5. m5 initparam</a></li>
-<li><a href="#m5-execfile">19.7.1.6. m5 execfile</a></li>
+<li><a href="#m5-dumpstats">19.7.1.2. m5 dumpstats</a></li>
+<li><a href="#m5-fail">19.7.1.3. m5 fail</a></li>
+<li><a href="#m5-writefile">19.7.1.4. m5 writefile</a></li>
+<li><a href="#m5-readfile">19.7.1.5. m5 readfile</a></li>
+<li><a href="#m5-initparam">19.7.1.6. m5 initparam</a></li>
+<li><a href="#m5-execfile">19.7.1.7. m5 execfile</a></li>
 </ul>
 </li>
 <li><a href="#m5ops-instructions">19.7.2. m5ops instructions</a>
@@ -1161,10 +1162,12 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 <li><a href="#m5out-directory">19.9. m5out directory</a>
 <ul class="sectlevel3">
 <li><a href="#gem5-m5out-system-terminal-file">19.9.1. gem5 m5out/system.terminal file</a></li>
-<li><a href="#gem5-m5out-system-dmesg-file">19.9.2. gem5 m5out/system.dmesg file</a></li>
+<li><a href="#gem5-m5out-system-dmesg-file">19.9.2. gem5 <code>m5out/system.workload.dmesg</code> file</a></li>
 <li><a href="#gem5-m5out-stats-txt-file">19.9.3. gem5 m5out/stats.txt file</a>
 <ul class="sectlevel4">
-<li><a href="#gem5-only-dump-selected-stats">19.9.3.1. gem5 only dump selected stats</a></li>
+<li><a href="#gem5-hdf5-statistics">19.9.3.1. gem5 HDF5 statistics</a></li>
+<li><a href="#gem5-only-dump-selected-stats">19.9.3.2. gem5 only dump selected stats</a></li>
+<li><a href="#gem5-stats-internals">19.9.3.3. gem5 stats internals</a></li>
 </ul>
 </li>
 <li><a href="#gem5-config-ini">19.9.4. gem5 config.ini</a></li>
@@ -1264,17 +1267,16 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 <li><a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis">19.19.4.6. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis</a></li>
 </ul>
 </li>
-<li><a href="#gem5-stats-internals">19.19.5. gem5 stats internals</a></li>
-<li><a href="#gem5-code-generation">19.19.6. gem5 code generation</a>
+<li><a href="#gem5-code-generation">19.19.5. gem5 code generation</a>
 <ul class="sectlevel4">
-<li><a href="#gem5-the-isa">19.19.6.1. gem5 THE_ISA</a></li>
+<li><a href="#gem5-the-isa">19.19.5.1. gem5 THE_ISA</a></li>
 </ul>
 </li>
-<li><a href="#gem5-build-system">19.19.7. gem5 build system</a>
+<li><a href="#gem5-build-system">19.19.6. gem5 build system</a>
 <ul class="sectlevel4">
-<li><a href="#gem5-build-broken-on-recent-compiler-version">19.19.7.1. gem5 build broken on recent compiler version</a></li>
-<li><a href="#gem5-polymorphic-isa-includes">19.19.7.2. gem5 polymorphic ISA includes</a></li>
-<li><a href="#why-are-all-c-symlinked-into-the-gem5-build-dir">19.19.7.3. Why are all C++ symlinked into the gem5 build dir?</a></li>
+<li><a href="#gem5-build-broken-on-recent-compiler-version">19.19.6.1. gem5 build broken on recent compiler version</a></li>
+<li><a href="#gem5-polymorphic-isa-includes">19.19.6.2. gem5 polymorphic ISA includes</a></li>
+<li><a href="#why-are-all-c-symlinked-into-the-gem5-build-dir">19.19.6.3. Why are all C++ symlinked into the gem5 build dir?</a></li>
 </ul>
 </li>
 </ul>
@@ -1345,19 +1347,20 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
 </li>
 <li><a href="#cpp">21.2. C++</a>
 <ul class="sectlevel3">
-<li><a href="#cpp-multithreading">21.2.1. C++ multithreading</a>
+<li><a href="#cpp-initialization-types">21.2.1. C++ initialization types</a></li>
+<li><a href="#cpp-multithreading">21.2.2. C++ multithreading</a>
 <ul class="sectlevel4">
-<li><a href="#atomic-cpp">21.2.1.1. atomic.cpp</a></li>
-<li><a href="#cpp-memory-order">21.2.1.2. C++ std::memory_order</a></li>
-<li><a href="#cpp-parallel-algorithms">21.2.1.3. C++ parallel algorithms</a></li>
+<li><a href="#atomic-cpp">21.2.2.1. atomic.cpp</a></li>
+<li><a href="#cpp-memory-order">21.2.2.2. C++ std::memory_order</a></li>
+<li><a href="#cpp-parallel-algorithms">21.2.2.3. C++ parallel algorithms</a></li>
 </ul>
 </li>
-<li><a href="#cpp-standards">21.2.2. C++ standards</a>
+<li><a href="#cpp-standards">21.2.3. C++ standards</a>
 <ul class="sectlevel4">
-<li><a href="#cpp17">21.2.2.1. C++17 N4659 standards draft</a></li>
+<li><a href="#cpp17">21.2.3.1. C++17 N4659 standards draft</a></li>
 </ul>
 </li>
-<li><a href="#cpp-type-casting">21.2.3. C++ type casting</a></li>
+<li><a href="#cpp-type-casting">21.2.4. C++ type casting</a></li>
 </ul>
 </li>
 <li><a href="#posix">21.3. POSIX</a>
@@ -2353,6 +2356,9 @@ hello /root/.profile
 </div>
 </div>
 <div class="paragraph">
+<p>see also: <a href="#dry-run">Dry run to get commands for your project</a>.</p>
+</div>
+<div class="paragraph">
 <p>When you reach difficulties, QEMU makes it possible to easily GDB step debug the Linux kernel source code, see: <a href="#gdb">Section 2, &#8220;GDB step debug&#8221;</a>.</p>
 </div>
 </div>
@@ -2424,6 +2430,9 @@ hello /root/.profile
 </div>
 </div>
 <div class="paragraph">
+<p>See also: <a href="#dry-run">Dry run to get commands for your project</a>.</p>
+</div>
+<div class="paragraph">
 <p><code>--eval-after</code> is optional: you could just type <code>insmod hello.ko</code> in the terminal, but this makes it run automatically at the end of boot, and then drops you into a shell.</p>
 </div>
 <div class="paragraph">
@@ -2815,7 +2824,7 @@ j = 0</pre>
 </div>
 </div>
 <div class="sect2">
-<h3 id="dry-run-to-get-commands-for-your-project"><a class="anchor" href="#dry-run-to-get-commands-for-your-project"></a><a class="link" href="#dry-run-to-get-commands-for-your-project">1.2. Dry run to get commands for your project</a></h3>
+<h3 id="dry-run"><a class="anchor" href="#dry-run"></a><a class="link" href="#dry-run">1.2. Dry run to get commands for your project</a></h3>
 <div class="paragraph">
 <p>One of the major features of this repository is that we try to support the <code>--dry-run</code> option really well for all scripts.</p>
 </div>
@@ -2873,6 +2882,17 @@ j = 0</pre>
 </div>
 </div>
 <div class="paragraph">
+<p>Since we need this so often, the last run command is also stored for convenience at:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>cat out/run.sh</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>although this won&#8217;t of course work well for <a href="#simultaneous-runs">Simultaneous runs</a>.</p>
+</div>
+<div class="paragraph">
 <p>Furthermore, <code>--dry-run</code> also automatically specifies, in valid Bash shell syntax:</p>
 </div>
 <div class="ulist">
@@ -4979,7 +4999,7 @@ echo 'file kernel/module.c +p' &gt; /sys/kernel/debug/dynamic_debug/control
 <p>If you get a failure before that, it will be hard to see the print messages.</p>
 </div>
 <div class="paragraph">
-<p>One possible solution is to parse the dmesg buffer, gem5 actually implements that: <a href="#gem5-m5out-system-dmesg-file">gem5 m5out/system.dmesg file</a>.</p>
+<p>One possible solution is to parse the dmesg buffer, gem5 actually implements that: <a href="#gem5-m5out-system-dmesg-file">gem5 <code>m5out/system.workload.dmesg</code> file</a>.</p>
 </div>
 </div>
 </div>
@@ -7592,7 +7612,7 @@ qw er</pre>
 <p>QEMU by default copies the host <code>uname</code> value, but we always override it in our scripts.</p>
 </div>
 <div class="paragraph">
-<p>Determining the right number to use for the kernel version is of course highly non-trivial and would require an extensive userland test suite, which most emulator don&#8217;t have.</p>
+<p>Determining the right number to use for the kernel version is of course highly non-trivial and would require an extensive userland test suite, which most emulators don&#8217;t have.</p>
 </div>
 <div class="literalblock">
 <div class="content">
@@ -18848,7 +18868,7 @@ m5 dumpstats</pre>
 </div>
 </div>
 <div class="paragraph">
-<p>where <a href="#m5">m5</a> is a guest utility present inside the gem5 tree which we cross-compiled and installed into the guest.</p>
+<p>where <a href="#gem5-m5-executable">gem5 m5 executable</a> is a guest utility present inside the gem5 tree which we cross-compiled and installed into the guest.</p>
 </div>
 <div class="paragraph">
 <p>To restore the checkpoint, kill the VM and run:</p>
@@ -19449,7 +19469,7 @@ FullO3CPU: Ticking main, FullO3CPU.
 <div class="ulist">
 <ul>
 <li>
-<p><a href="#m5">m5</a></p>
+<p><a href="#gem5-m5-executable">gem5 m5 executable</a></p>
 </li>
 <li>
 <p><a href="#m5ops-instructions">m5ops instructions</a></p>
@@ -19476,7 +19496,7 @@ FullO3CPU: Ticking main, FullO3CPU.
 </ul>
 </div>
 <div class="sect3">
-<h4 id="m5"><a class="anchor" href="#m5"></a><a class="link" href="#m5">19.7.1. m5</a></h4>
+<h4 id="gem5-m5-executable"><a class="anchor" href="#gem5-m5-executable"></a><a class="link" href="#gem5-m5-executable">19.7.1. gem5 m5 executable</a></h4>
 <div class="paragraph">
 <p><code>m5</code> is a guest command line utility that is installed and run on the guest, that serves as a CLI front-end for the <a href="#m5ops">m5ops</a></p>
 </div>
@@ -19486,6 +19506,25 @@ FullO3CPU: Ticking main, FullO3CPU.
 <div class="paragraph">
 <p>It is possible to guess what most tools do from the corresponding <a href="#m5ops">m5ops</a>, but let&#8217;s at least document the less obvious ones here.</p>
 </div>
+<div class="paragraph">
+<p>In LKMC we build <code>m5</code> with:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>./build-m5 --arch aarch64</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The <code>m5</code> executable can be run on <a href="#user-mode-simulation">User mode simulation</a> as normal with:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>./run --arch aarch64 --emulator gem5 --userland "$(./getvar --arch aarch64 out_rootfs_overlay_bin_dir)/m5" --userland-args dumpstats</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>This can be a good test <a href="#m5ops">m5ops</a> since it executes very quickly.</p>
+</div>
 <div class="sect4">
 <h5 id="m5-exit"><a class="anchor" href="#m5-exit"></a><a class="link" href="#m5-exit">19.7.1.1. m5 exit</a></h5>
 <div class="paragraph">
@@ -19496,7 +19535,13 @@ FullO3CPU: Ticking main, FullO3CPU.
 </div>
 </div>
 <div class="sect4">
-<h5 id="m5-fail"><a class="anchor" href="#m5-fail"></a><a class="link" href="#m5-fail">19.7.1.2. m5 fail</a></h5>
+<h5 id="m5-dumpstats"><a class="anchor" href="#m5-dumpstats"></a><a class="link" href="#m5-dumpstats">19.7.1.2. m5 dumpstats</a></h5>
+<div class="paragraph">
+<p>Makes gem5 dump one more statistics entry to the <a href="#gem5-m5out-stats-txt-file">gem5 m5out/stats.txt file</a>.</p>
+</div>
+</div>
+<div class="sect4">
+<h5 id="m5-fail"><a class="anchor" href="#m5-fail"></a><a class="link" href="#m5-fail">19.7.1.3. m5 fail</a></h5>
 <div class="paragraph">
 <p>End the simulation with a failure exit event:</p>
 </div>
@@ -19535,7 +19580,7 @@ FullO3CPU: Ticking main, FullO3CPU.
 </div>
 </div>
 <div class="sect4">
-<h5 id="m5-writefile"><a class="anchor" href="#m5-writefile"></a><a class="link" href="#m5-writefile">19.7.1.3. m5 writefile</a></h5>
+<h5 id="m5-writefile"><a class="anchor" href="#m5-writefile"></a><a class="link" href="#m5-writefile">19.7.1.4. m5 writefile</a></h5>
 <div class="paragraph">
 <p>Send a guest file to the host. <a href="#9p">9P</a> is a more advanced alternative.</p>
 </div>
@@ -19566,7 +19611,7 @@ m5 writefile myfileguest myfilehost</pre>
 </div>
 </div>
 <div class="sect4">
-<h5 id="m5-readfile"><a class="anchor" href="#m5-readfile"></a><a class="link" href="#m5-readfile">19.7.1.4. m5 readfile</a></h5>
+<h5 id="m5-readfile"><a class="anchor" href="#m5-readfile"></a><a class="link" href="#m5-readfile">19.7.1.5. m5 readfile</a></h5>
 <div class="paragraph">
 <p>Read a host file pointed to by the <code>fs.py --script</code> option to stdout.</p>
 </div>
@@ -19594,7 +19639,7 @@ m5 writefile myfileguest myfilehost</pre>
 </div>
 </div>
 <div class="sect4">
-<h5 id="m5-initparam"><a class="anchor" href="#m5-initparam"></a><a class="link" href="#m5-initparam">19.7.1.5. m5 initparam</a></h5>
+<h5 id="m5-initparam"><a class="anchor" href="#m5-initparam"></a><a class="link" href="#m5-initparam">19.7.1.6. m5 initparam</a></h5>
 <div class="paragraph">
 <p>Ermm, just another <a href="#m5-readfile">m5 readfile</a> that only takes integers and only from CLI options? Is this software so redundant?</p>
 </div>
@@ -19620,7 +19665,7 @@ m5 writefile myfileguest myfilehost</pre>
 </div>
 </div>
 <div class="sect4">
-<h5 id="m5-execfile"><a class="anchor" href="#m5-execfile"></a><a class="link" href="#m5-execfile">19.7.1.6. m5 execfile</a></h5>
+<h5 id="m5-execfile"><a class="anchor" href="#m5-execfile"></a><a class="link" href="#m5-execfile">19.7.1.7. m5 execfile</a></h5>
 <div class="paragraph">
 <p>Trivial combination of <code>m5 readfile</code> + execute the script.</p>
 </div>
@@ -19660,7 +19705,7 @@ m5 execfile</pre>
 <p>gem5 allocates some magic instructions on unused instruction encodings for convenient guest instrumentation.</p>
 </div>
 <div class="paragraph">
-<p>Those instructions are exposed through the <a href="#m5">m5</a> in tree executable.</p>
+<p>Those instructions are exposed through the <a href="#gem5-m5-executable">gem5 m5 executable</a> in tree executable.</p>
 </div>
 <div class="paragraph">
 <p>To make things simpler to understand, you can play around with our own minimized educational <code>m5</code> subset:</p>
@@ -19685,7 +19730,6 @@ m5 execfile</pre>
 <div class="content">
 <pre>./build-userland \
  --arch aarch64 \
-  --ccflags='-DLKMC_M5OPS_ENABLE=1' \
  --force-rebuild \
  userland/c/m5ops.c \
 ;
@@ -19745,7 +19789,7 @@ m5 execfile</pre>
 <div class="sect4">
 <h5 id="m5ops-instructions-interface"><a class="anchor" href="#m5ops-instructions-interface"></a><a class="link" href="#m5ops-instructions-interface">19.7.2.1. m5ops instructions interface</a></h5>
 <div class="paragraph">
-<p>Let&#8217;s study how <a href="#m5">m5</a> uses them:</p>
+<p>Let&#8217;s study how the <a href="#gem5-m5-executable">gem5 m5 executable</a> uses them:</p>
 </div>
 <div class="ulist">
 <ul>
@@ -19999,18 +20043,77 @@ git -C "$(./getvar linux_source_dir)" checkout -
 </div>
 </div>
 <div class="sect3">
-<h4 id="gem5-m5out-system-dmesg-file"><a class="anchor" href="#gem5-m5out-system-dmesg-file"></a><a class="link" href="#gem5-m5out-system-dmesg-file">19.9.2. gem5 m5out/system.dmesg file</a></h4>
+<h4 id="gem5-m5out-system-dmesg-file"><a class="anchor" href="#gem5-m5out-system-dmesg-file"></a><a class="link" href="#gem5-m5out-system-dmesg-file">19.9.2. gem5 <code>m5out/system.workload.dmesg</code> file</a></h4>
 <div class="paragraph">
-<p>TODO confirm and create minimal example.</p>
+<p>This file used to be called just <code>m5out/system.dmesg</code>, but the name was changed after the workload refactorings of March 2020.</p>
 </div>
 <div class="paragraph">
-<p>I think this file is capable of showing terminal messages before they reach the terminal by parsing the dmesg buffer from memory.</p>
+<p>This file is capable of showing terminal messages that are <code>printk</code> before the serial is enabled as described at: <a href="#linux-kernel-early-boot-messages">Linux kernel early boot messages</a>.</p>
 </div>
 <div class="paragraph">
-<p>This could be used to debug the Linux kernel boot if problems happen before the serial is enabled: <a href="#linux-kernel-early-boot-messages">Linux kernel early boot messages</a>.</p>
+<p>The file is dumped only on kernel panics which gem5 can detect by the PC address: <a href="#exit-gem5-on-panic">Exit gem5 on panic</a>.</p>
 </div>
 <div class="paragraph">
-<p>The file appears to get dumped only on kernel panic which gem5 can detect by the PC address: <a href="#exit-gem5-on-panic">Exit gem5 on panic</a>.</p>
+<p>This mechanism can be very useful to debug the Linux kernel boot if problems happen before the serial is enabled.</p>
+</div>
+<div class="paragraph">
+<p>This magic mechanism works by activating an event when the PC reaches the <code>printk</code> address, much like gem5 <a href="#exit-gem5-on-panic">can detect <code>panic</code> by PC</a> and then parsing printk function arguments and buffers!</p>
+</div>
+<div class="paragraph">
+<p>The relevant source is at <a href="https://github.com/gem5/gem5/blob/cd69bb50414450c3bb5ef41dce676b75fd42c0ee/src/kern/linux/printk.cc"><code>src/kern/linux/printk.c</code></a>.</p>
+</div>
+<div class="paragraph">
+<p>We can test this mechanism in a controlled way by hacking a <code>panic()</code> into the kernel next to a <code>printk</code> that shows up before the serial is enabled, e.g. on Linux v5.4.3 we could do:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
+index f296d89be757..3e79916322c2 100644
+--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
+@@ -6207,6 +6207,7 @@ void __init ftrace_init(void)
+
+    pr_info("ftrace: allocating %ld entries in %ld pages\n",
+        count, count / ENTRIES_PER_PAGE + 1);
+   panic("foobar");
+
+    last_ftrace_enabled = ftrace_enabled = 1;</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>With this, after the panic, <code>system.workload.dmesg</code> contains on LKMC d09a0d97b81582cc88381c4112db631da61a048d aarch64:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>[0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd070]
+[0.000000] Linux version 5.4.3-dirty (lkmc@f7688b48ac46e9a669e279f1bc167722d5141eda) (gcc version 8.3.0 (Buildroot 2019.11-00002-g157ac499cf)) #1 SMP Thu Jan 1 00:00:00 UTC 1970
+[0.000000] Machine model: V2P-CA15
+[0.000000] Memory limited to 256MB
+[0.000000] efi: Getting EFI parameters from FDT:
+[0.000000] efi: UEFI not found.
+[0.000000] On node 0 totalpages: 65536
+[0.000000]   DMA32 zone: 1024 pages used for memmap
+[0.000000]   DMA32 zone: 0 pages reserved
+[0.000000]   DMA32 zone: 65536 pages, LIFO batch:15
+[0.000000] percpu: Embedded 29 pages/cpu s79960 r8192 d30632 u118784
+[0.000000] pcpu-alloc: s79960 r8192 d30632 u118784 alloc=29*4096
+[0.000000] pcpu-alloc: [0] 0
+[0.000000] Detected PIPT I-cache on CPU0
+[0.000000] CPU features: detected: ARM erratum 832075
+[0.000000] CPU features: detected: EL2 vector hardening
+[0.000000] ARM_SMCCC_ARCH_WORKAROUND_1 missing from firmware
+[0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 64512
+[0.000000] Kernel command line: earlyprintk=pl011,0x1c090000 lpj=19988480 rw loglevel=8 mem=256MB root=/dev/sda console_msg_format=syslog nokaslr norandmaps panic=-1 printk.devkmsg=on printk.time=y rw console=ttyAMA0 - lkmc_home=/lkmc
+[0.000000] Dentry cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
+[0.000000] Inode-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
+[0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
+[0.000000] Memory: 233432K/262144K available (6652K kernel code, 792K rwdata, 2176K rodata, 896K init, 659K bss, 28712K reserved, 0K cma-reserved)
+[0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
+[0.000000] ftrace: allocating 22067 entries in 87 pages</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>So we see that messages up to the <code>ftrace</code> do show up!</p>
 </div>
 </div>
 <div class="sect3">
@@ -20024,7 +20127,7 @@ git -C "$(./getvar linux_source_dir)" checkout -
 </div>
 </div>
 <div class="paragraph">
-<p>Whenever we run <code>m5 dumpstats</code> or <code>m5 exit</code>, a section with the following format is added to that file:</p>
+<p>Whenever we run <code>m5 dumpstats</code> or when fs.py and se.py are exiting (TODO other scripts?), a section with the following format is added to that file:</p>
 </div>
 <div class="literalblock">
 <div class="content">
@@ -20046,8 +20149,113 @@ system.cpu.dtb.inst_hits</pre>
 <div class="paragraph">
 <p>For x86, it is interesting to try and correlate <code>numCycles</code> with:</p>
 </div>
+<div class="paragraph">
+<p>In LKMC f42c525d7973d70f4c836d2169cc2bd2893b4197 gem5 5af26353b532d7b5988cf0f6f3d0fbc5087dd1df, the stat file for a <a href="#c">C</a> hello world:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>./run --arch aarch64 --emulator gem5 --userland userland/c/hello.c</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>which has a single dump done at the exit, has size 59KB and stat lines of form:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>final_tick                                   91432000                       # Number of ticks from beginning of simulation (restored from checkpoints and never reset)</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>We can reduce the file size by adding the <code>?desc=False</code> magic suffix to the stat flie name:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>--stats-file stats.txt?desc=false</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>as explained in:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>gem5.opt --stats-help</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>and this reduces the file size to 39KB by removing those excessive comments:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>final_tick                                   91432000</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>although trailing spaces are still prse</p>
+</div>
+<div class="paragraph">
+<p>We can further reduce this size by removing spaces from the dumps with this hack:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>         ccprintf(stream, " |%12s %10s %10s",
+                  ValueToString(value, precision), pdfstr.str(), cdfstr.str());
+     } else {
+-        ccprintf(stream, "%-40s %12s %10s %10s", name,
+-                 ValueToString(value, precision), pdfstr.str(), cdfstr.str());
+        ccprintf(stream, "%s %s", name, ValueToString(value, precision));
+        if (pdfstr.rdbuf()-&gt;in_avail())
+            stream &lt;&lt; " " &lt;&lt; pdfstr.str();
+        if (cdfstr.rdbuf()-&gt;in_avail())
+            stream &lt;&lt; " " &lt;&lt; cdfstr.str();
+
+         if (descriptions) {
+             if (!desc.empty())</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>and after that the file size went down to 21KB.</p>
+</div>
 <div class="sect4">
-<h5 id="gem5-only-dump-selected-stats"><a class="anchor" href="#gem5-only-dump-selected-stats"></a><a class="link" href="#gem5-only-dump-selected-stats">19.9.3.1. gem5 only dump selected stats</a></h5>
+<h5 id="gem5-hdf5-statistics"><a class="anchor" href="#gem5-hdf5-statistics"></a><a class="link" href="#gem5-hdf5-statistics">19.9.3.1. gem5 HDF5 statistics</a></h5>
+<div class="paragraph">
+<p>We can make gem5 dump statistics in the <a href="#hdf5">HDF5</a> format by adding the magic <code>h5://</code> prefix to the file name as in:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>gem5.opt --stats-file h5://stats.h5</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>as explained in:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>gem5.opt --stats-help</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>This is not exposed in LKMC f42c525d7973d70f4c836d2169cc2bd2893b4197 however, you just have to <a href="#dry-run">hack the gem5 CLI for now</a>.</p>
+</div>
+<div class="paragraph">
+<p>TODO what is the advantage? The generated file for <code>--stats-file h5://stats.h5?desc=False</code> in LKMC f42c525d7973d70f4c836d2169cc2bd2893b4197 gem5 5af26353b532d7b5988cf0f6f3d0fbc5087dd1df for a single dump was 946K, so much larger than the text version seen at <a href="#gem5-m5out-stats-txt-file">gem5 m5out/stats.txt file</a> which was only 59KB max!</p>
+</div>
+<div class="paragraph">
+<p>We then try to see if it is any better when you have a bunch of dump events:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>./run --arch aarch64 --emulator gem5 --userland userland/c/m5ops.c --userland-args 'd 1000'</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>and there yes, we see that the file size fell from 39MB on <code>stats.txt</code> to 3.2MB on <code>stats.m5</code>, so the increase observed previously was just due to some initial size overhead (considering the patched gem5 with no spaces in the text file).</p>
+</div>
+<div class="paragraph">
+<p>We also note however that the stat dump made the such a simulation that just loops and dumps considerably slower, from 3s to 15s on <a href="#p51">P51</a>. Fascinating, we are definitely not disk bound there.</p>
+</div>
+</div>
+<div class="sect4">
+<h5 id="gem5-only-dump-selected-stats"><a class="anchor" href="#gem5-only-dump-selected-stats"></a><a class="link" href="#gem5-only-dump-selected-stats">19.9.3.2. gem5 only dump selected stats</a></h5>
 <div class="paragraph">
 <p>TODO</p>
 </div>
@@ -20058,6 +20266,79 @@ system.cpu.dtb.inst_hits</pre>
 <p>To prevent the stats file from becoming humongous.</p>
 </div>
 </div>
+<div class="sect4">
+<h5 id="gem5-stats-internals"><a class="anchor" href="#gem5-stats-internals"></a><a class="link" href="#gem5-stats-internals">19.9.3.3. gem5 stats internals</a></h5>
+<div class="paragraph">
+<p>This describes the internals of the <a href="#gem5-m5out-stats-txt-file">gem5 m5out/stats.txt file</a>.</p>
+</div>
+<div class="paragraph">
+<p>GDB call stack to <code>dumpstats</code>:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>Stats::pythonDump () at build/ARM/python/pybind11/stats.cc:58
+Stats::StatEvent::process() ()
+GlobalEvent::BarrierEvent::process (this=0x555559fa6a80) at build/ARM/sim/global_event.cc:131
+EventQueue::serviceOne (this=this@entry=0x555558c36080) at build/ARM/sim/eventq.cc:228
+doSimLoop (eventq=0x555558c36080) at build/ARM/sim/simulate.cc:219
+simulate (num_cycles=&lt;optimized out&gt;) at build/ARM/sim/simulate.cc:132</pre>
+</div>
+</div>
+<div class="paragraph">
+<p><code>Stats::pythonDump</code> does:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>void
+pythonDump()
+{
+    py::module m = py::module::import("m5.stats");
+    m.attr("dump")();
+}</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>This calls <code>src/python/m5/stats/<em>init</em>.py</code> in <code>def dump</code> does the main dumping</p>
+</div>
+<div class="paragraph">
+<p>That function does notably:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>    for output in outputList:
+        if output.valid():
+            output.begin()
+            for stat in stats_list:
+                stat.visit(output)
+            output.end()</pre>
+</div>
+</div>
+<div class="paragraph">
+<p><code>begin</code> and <code>end</code> are defined in C++ and output the header and tail respectively</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>void
+Text::begin()
+{
+    ccprintf(*stream, "\n---------- Begin Simulation Statistics ----------\n");
+}
+
+void
+Text::end()
+{
+    ccprintf(*stream, "\n---------- End Simulation Statistics   ----------\n");
+    stream-&gt;flush();
+}</pre>
+</div>
+</div>
+<div class="paragraph">
+<p><code>stats_list</code> contains the stats, and <code>stat.visit</code> prints them, <code>outputList</code> contains by default just the text output. I don&#8217;t see any other types of output in gem5, but likely JSON / binary formats could be envisioned.</p>
+</div>
+<div class="paragraph">
+<p>Tested in gem5 b4879ae5b0b6644e6836b0881e4da05c64a6550d.</p>
+</div>
+</div>
 </div>
 <div class="sect3">
 <h4 id="gem5-config-ini"><a class="anchor" href="#gem5-config-ini"></a><a class="link" href="#gem5-config-ini">19.9.4. gem5 config.ini</a></h4>
@@ -20546,7 +20827,16 @@ Exiting @ tick 18446744073709551615 because simulate() limit reached</pre>
 </ul>
 </div>
 <div class="paragraph">
-<p>See also: <a href="#profiling-userland-programs">Profiling userland programs</a>.</p>
+<p>Profiling techniques are discussed in more detail at: <a href="#profiling-userland-programs">Profiling userland programs</a>.</p>
+</div>
+<div class="paragraph">
+<p>For the <code>prof</code> build, you can get the <code>gmon.out</code> file with:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>./run --arch aarch64 --emulator gem5 --userland userland/c/hello.c --gem5-build-type prof
+gprof "$(./getvar --arch aarch64 gem5_executable)" &gt; tmp.gprof</pre>
+</div>
 </div>
 </div>
 <div class="sect3">
@@ -23264,80 +23554,7 @@ info: Entering event queue @ 0.  Starting simulation...
 </div>
 </div>
 <div class="sect3">
-<h4 id="gem5-stats-internals"><a class="anchor" href="#gem5-stats-internals"></a><a class="link" href="#gem5-stats-internals">19.19.5. gem5 stats internals</a></h4>
-<div class="paragraph">
-<p>This describes the internals of the <a href="#gem5-m5out-stats-txt-file">gem5 m5out/stats.txt file</a>.</p>
-</div>
-<div class="paragraph">
-<p>GDB call stack to <code>dumpstats</code>:</p>
-</div>
-<div class="literalblock">
-<div class="content">
-<pre>Stats::pythonDump () at build/ARM/python/pybind11/stats.cc:58
-Stats::StatEvent::process() ()
-GlobalEvent::BarrierEvent::process (this=0x555559fa6a80) at build/ARM/sim/global_event.cc:131
-EventQueue::serviceOne (this=this@entry=0x555558c36080) at build/ARM/sim/eventq.cc:228
-doSimLoop (eventq=0x555558c36080) at build/ARM/sim/simulate.cc:219
-simulate (num_cycles=&lt;optimized out&gt;) at build/ARM/sim/simulate.cc:132</pre>
-</div>
-</div>
-<div class="paragraph">
-<p><code>Stats::pythonDump</code> does:</p>
-</div>
-<div class="literalblock">
-<div class="content">
-<pre>void
-pythonDump()
-{
-    py::module m = py::module::import("m5.stats");
-    m.attr("dump")();
-}</pre>
-</div>
-</div>
-<div class="paragraph">
-<p>This calls <code>src/python/m5/stats/<em>init</em>.py</code> in <code>def dump</code> does the main dumping</p>
-</div>
-<div class="paragraph">
-<p>That function does notably:</p>
-</div>
-<div class="literalblock">
-<div class="content">
-<pre>    for output in outputList:
-        if output.valid():
-            output.begin()
-            for stat in stats_list:
-                stat.visit(output)
-            output.end()</pre>
-</div>
-</div>
-<div class="paragraph">
-<p><code>begin</code> and <code>end</code> are defined in C++ and output the header and tail respectively</p>
-</div>
-<div class="literalblock">
-<div class="content">
-<pre>void
-Text::begin()
-{
-    ccprintf(*stream, "\n---------- Begin Simulation Statistics ----------\n");
-}
-
-void
-Text::end()
-{
-    ccprintf(*stream, "\n---------- End Simulation Statistics   ----------\n");
-    stream-&gt;flush();
-}</pre>
-</div>
-</div>
-<div class="paragraph">
-<p><code>stats_list</code> contains the stats, and <code>stat.visit</code> prints them, <code>outputList</code> contains by default just the text output. I don&#8217;t see any other types of output in gem5, but likely JSON / binary formats could be envisioned.</p>
-</div>
-<div class="paragraph">
-<p>Tested in gem5 b4879ae5b0b6644e6836b0881e4da05c64a6550d.</p>
-</div>
-</div>
-<div class="sect3">
-<h4 id="gem5-code-generation"><a class="anchor" href="#gem5-code-generation"></a><a class="link" href="#gem5-code-generation">19.19.6. gem5 code generation</a></h4>
+<h4 id="gem5-code-generation"><a class="anchor" href="#gem5-code-generation"></a><a class="link" href="#gem5-code-generation">19.19.5. gem5 code generation</a></h4>
 <div class="paragraph">
 <p>gem5 uses a ton of code generation, which makes the project horrendous:</p>
 </div>
@@ -23382,7 +23599,7 @@ Text::end()
 <p>But it has been widely overused to insanity. It likely also exists partly because when the project started in 2003 C++ compilers weren&#8217;t that good, so you couldn&#8217;t rely on features like templates that much.</p>
 </div>
 <div class="sect4">
-<h5 id="gem5-the-isa"><a class="anchor" href="#gem5-the-isa"></a><a class="link" href="#gem5-the-isa">19.19.6.1. gem5 THE_ISA</a></h5>
+<h5 id="gem5-the-isa"><a class="anchor" href="#gem5-the-isa"></a><a class="link" href="#gem5-the-isa">19.19.5.1. gem5 THE_ISA</a></h5>
 <div class="paragraph">
 <p>Generated code at: <code>build/&lt;ISA&gt;/config/the_isa.hh</code> which contains amongst other lines:</p>
 </div>
@@ -23409,9 +23626,9 @@ enum class Arch {
 </div>
 </div>
 <div class="sect3">
-<h4 id="gem5-build-system"><a class="anchor" href="#gem5-build-system"></a><a class="link" href="#gem5-build-system">19.19.7. gem5 build system</a></h4>
+<h4 id="gem5-build-system"><a class="anchor" href="#gem5-build-system"></a><a class="link" href="#gem5-build-system">19.19.6. gem5 build system</a></h4>
 <div class="sect4">
-<h5 id="gem5-build-broken-on-recent-compiler-version"><a class="anchor" href="#gem5-build-broken-on-recent-compiler-version"></a><a class="link" href="#gem5-build-broken-on-recent-compiler-version">19.19.7.1. gem5 build broken on recent compiler version</a></h5>
+<h5 id="gem5-build-broken-on-recent-compiler-version"><a class="anchor" href="#gem5-build-broken-on-recent-compiler-version"></a><a class="link" href="#gem5-build-broken-on-recent-compiler-version">19.19.6.1. gem5 build broken on recent compiler version</a></h5>
 <div class="paragraph">
 <p>gem5 moves a bit slowly, and if your host compiler is very new, the gem5 build might be broken for it, e.g. this was the case for Ubuntu 19.10 with GCC 9 and gem5 62d75e7105fe172eb906d4f80f360ff8591d4178 from Dec 2019.</p>
 </div>
@@ -23436,7 +23653,7 @@ enum class Arch {
 </div>
 </div>
 <div class="sect4">
-<h5 id="gem5-polymorphic-isa-includes"><a class="anchor" href="#gem5-polymorphic-isa-includes"></a><a class="link" href="#gem5-polymorphic-isa-includes">19.19.7.2. gem5 polymorphic ISA includes</a></h5>
+<h5 id="gem5-polymorphic-isa-includes"><a class="anchor" href="#gem5-polymorphic-isa-includes"></a><a class="link" href="#gem5-polymorphic-isa-includes">19.19.6.2. gem5 polymorphic ISA includes</a></h5>
 <div class="paragraph">
 <p>E.g. <code>src/cpu/decode_cache.hh</code> includes:</p>
 </div>
@@ -23515,7 +23732,7 @@ build/ARM/config/the_isa.hh
 </div>
 </div>
 <div class="sect4">
-<h5 id="why-are-all-c-symlinked-into-the-gem5-build-dir"><a class="anchor" href="#why-are-all-c-symlinked-into-the-gem5-build-dir"></a><a class="link" href="#why-are-all-c-symlinked-into-the-gem5-build-dir">19.19.7.3. Why are all C++ symlinked into the gem5 build dir?</a></h5>
+<h5 id="why-are-all-c-symlinked-into-the-gem5-build-dir"><a class="anchor" href="#why-are-all-c-symlinked-into-the-gem5-build-dir"></a><a class="link" href="#why-are-all-c-symlinked-into-the-gem5-build-dir">19.19.6.3. Why are all C++ symlinked into the gem5 build dir?</a></h5>
 <div class="paragraph">
 <p>Some scons madness.</p>
 </div>
@@ -24834,7 +25051,7 @@ echo 1 &gt; /proc/sys/vm/overcommit_memory
 <p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/template_class_with_static_member.cpp">userland/cpp/template_class_with_static_member.cpp</a>: <a href="https://stackoverflow.com/questions/3229883/static-member-initialization-in-a-class-template" class="bare">https://stackoverflow.com/questions/3229883/static-member-initialization-in-a-class-template</a></p>
 </li>
 <li>
-<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/if_constexpr.cpp">userland/cpp/if_constexpr.cpp</a>: C++17 <code>if constexpr</code></p>
+<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/if_constexpr.cpp">userland/cpp/if_constexpr.cpp</a>: C++17 <code>if constexpr</code>: <a href="https://stackoverflow.com/questions/12160765/if-else-at-compile-time-in-c/54647315#54647315" class="bare">https://stackoverflow.com/questions/12160765/if-else-at-compile-time-in-c/54647315#54647315</a></p>
 </li>
 </ul>
 </div>
@@ -24895,7 +25112,55 @@ echo 1 &gt; /proc/sys/vm/overcommit_memory
 </ul>
 </div>
 <div class="sect3">
-<h4 id="cpp-multithreading"><a class="anchor" href="#cpp-multithreading"></a><a class="link" href="#cpp-multithreading">21.2.1. C++ multithreading</a></h4>
+<h4 id="cpp-initialization-types"><a class="anchor" href="#cpp-initialization-types"></a><a class="link" href="#cpp-initialization-types">21.2.1. C++ initialization types</a></h4>
+<div class="paragraph">
+<p>OMG this is hell, understand when primitive variables are initialized or not:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p><a href="https://stackoverflow.com/questions/3127454/how-do-c-class-members-get-initialized-if-i-dont-do-it-explicitly" class="bare">https://stackoverflow.com/questions/3127454/how-do-c-class-members-get-initialized-if-i-dont-do-it-explicitly</a></p>
+</li>
+<li>
+<p><a href="https://blog.tartanllama.xyz/initialization-is-bonkers/" class="bare">https://blog.tartanllama.xyz/initialization-is-bonkers/</a></p>
+</li>
+</ul>
+</div>
+<div class="paragraph">
+<p>Intuition:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p>direct initialization: a constructor called explicitly with at least one argument: <a href="https://en.cppreference.com/w/cpp/language/direct_initialization" class="bare">https://en.cppreference.com/w/cpp/language/direct_initialization</a></p>
+</li>
+<li>
+<p>default initialization: does not initialize primitive types: <a href="https://en.cppreference.com/w/cpp/language/default_initialization" class="bare">https://en.cppreference.com/w/cpp/language/default_initialization</a></p>
+</li>
+<li>
+<p>value initialization: maybe initializes primitive types: <a href="https://en.cppreference.com/w/cpp/language/value_initialization" class="bare">https://en.cppreference.com/w/cpp/language/value_initialization</a></p>
+</li>
+<li>
+<p>zero initialization: initializes primitive types</p>
+</li>
+</ul>
+</div>
+<div class="paragraph">
+<p>Good rule:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p>initialize every single variable explicitly to prevent the risk of having uninitialized variables due to programmer error (which is easy to get wrong due to insane rules)</p>
+</li>
+<li>
+<p>if you don&#8217;t define your own default constructor, always <code>= delete</code> it instead. This prevents the possibility that variables will be assigned twice due to zero initialization</p>
+</li>
+</ul>
+</div>
+</div>
+<div class="sect3">
+<h4 id="cpp-multithreading"><a class="anchor" href="#cpp-multithreading"></a><a class="link" href="#cpp-multithreading">21.2.2. C++ multithreading</a></h4>
 <div class="ulist">
 <ul>
 <li>
@@ -24923,7 +25188,7 @@ echo 1 &gt; /proc/sys/vm/overcommit_memory
 </ul>
 </div>
 <div class="sect4">
-<h5 id="atomic-cpp"><a class="anchor" href="#atomic-cpp"></a><a class="link" href="#atomic-cpp">21.2.1.1. atomic.cpp</a></h5>
+<h5 id="atomic-cpp"><a class="anchor" href="#atomic-cpp"></a><a class="link" href="#atomic-cpp">21.2.2.1. atomic.cpp</a></h5>
 <div class="paragraph">
 <p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/">userland/cpp/atomic/</a></p>
 </div>
@@ -25127,7 +25392,7 @@ time ./mutex.out 4 100000000</pre>
 </div>
 </div>
 <div class="sect4">
-<h5 id="cpp-memory-order"><a class="anchor" href="#cpp-memory-order"></a><a class="link" href="#cpp-memory-order">21.2.1.2. C++ std::memory_order</a></h5>
+<h5 id="cpp-memory-order"><a class="anchor" href="#cpp-memory-order"></a><a class="link" href="#cpp-memory-order">21.2.2.2. C++ std::memory_order</a></h5>
 <div class="paragraph">
 <p><a href="https://stackoverflow.com/questions/12346487/what-do-each-memory-order-mean" class="bare">https://stackoverflow.com/questions/12346487/what-do-each-memory-order-mean</a></p>
 </div>
@@ -25136,7 +25401,7 @@ time ./mutex.out 4 100000000</pre>
 </div>
 </div>
 <div class="sect4">
-<h5 id="cpp-parallel-algorithms"><a class="anchor" href="#cpp-parallel-algorithms"></a><a class="link" href="#cpp-parallel-algorithms">21.2.1.3. C++ parallel algorithms</a></h5>
+<h5 id="cpp-parallel-algorithms"><a class="anchor" href="#cpp-parallel-algorithms"></a><a class="link" href="#cpp-parallel-algorithms">21.2.2.3. C++ parallel algorithms</a></h5>
 <div class="paragraph">
 <p><a href="https://stackoverflow.com/questions/51031060/are-c17-parallel-algorithms-implemented-already/55989883#55989883" class="bare">https://stackoverflow.com/questions/51031060/are-c17-parallel-algorithms-implemented-already/55989883#55989883</a></p>
 </div>
@@ -25146,7 +25411,7 @@ time ./mutex.out 4 100000000</pre>
 </div>
 </div>
 <div class="sect3">
-<h4 id="cpp-standards"><a class="anchor" href="#cpp-standards"></a><a class="link" href="#cpp-standards">21.2.2. C++ standards</a></h4>
+<h4 id="cpp-standards"><a class="anchor" href="#cpp-standards"></a><a class="link" href="#cpp-standards">21.2.3. C++ standards</a></h4>
 <div class="paragraph">
 <p>Like for C, you have to pay for the standards&#8230;&#8203; insane. So we just use the closest free drafts instead.</p>
 </div>
@@ -25154,14 +25419,14 @@ time ./mutex.out 4 100000000</pre>
 <p><a href="https://stackoverflow.com/questions/81656/where-do-i-find-the-current-c-or-c-standard-documents" class="bare">https://stackoverflow.com/questions/81656/where-do-i-find-the-current-c-or-c-standard-documents</a></p>
 </div>
 <div class="sect4">
-<h5 id="cpp17"><a class="anchor" href="#cpp17"></a><a class="link" href="#cpp17">21.2.2.1. C++17 N4659 standards draft</a></h5>
+<h5 id="cpp17"><a class="anchor" href="#cpp17"></a><a class="link" href="#cpp17">21.2.3.1. C++17 N4659 standards draft</a></h5>
 <div class="paragraph">
 <p><a href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/n4659.pdf" class="bare">http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/n4659.pdf</a></p>
 </div>
 </div>
 </div>
 <div class="sect3">
-<h4 id="cpp-type-casting"><a class="anchor" href="#cpp-type-casting"></a><a class="link" href="#cpp-type-casting">21.2.3. C++ type casting</a></h4>
+<h4 id="cpp-type-casting"><a class="anchor" href="#cpp-type-casting"></a><a class="link" href="#cpp-type-casting">21.2.4. C++ type casting</a></h4>
 <div class="paragraph">
 <p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/static_dynamic_reinterpret_cast.cpp">userland/cpp/static_dynamic_reinterpret_cast.cpp</a></p>
 </div>
@@ -26111,7 +26376,7 @@ xdg-open bst_vs_heap_vs_hashmap_gem5.tmp.png</pre>
 <p>The cache sizes were chosen to match the host <a href="#p51">P51</a> to improve the comparison. Ideally we should also use the same standard library.</p>
 </div>
 <div class="paragraph">
-<p>Note that this will take a long time, and will produce a humongous ~40Gb stats file as explained at: <a href="#gem5-only-dump-selected-stats">Section 19.9.3.1, &#8220;gem5 only dump selected stats&#8221;</a></p>
+<p>Note that this will take a long time, and will produce a humongous ~40Gb stats file as explained at: <a href="#gem5-only-dump-selected-stats">Section 19.9.3.2, &#8220;gem5 only dump selected stats&#8221;</a></p>
 </div>
 <div class="paragraph">
 <p>Sources:</p>
@@ -26713,7 +26978,17 @@ git clean -xdf .</pre>
 <p>Binary format to store data. TODO vs databases, notably SQLite: <a href="https://datascience.stackexchange.com/questions/262/hierarchical-data-format-what-are-the-advantages-compared-to-alternative-format" class="bare">https://datascience.stackexchange.com/questions/262/hierarchical-data-format-what-are-the-advantages-compared-to-alternative-format</a></p>
 </div>
 <div class="paragraph">
-<p>Examples: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/libs/hdf5">userland/libs/hdf5</a></p>
+<p>Examples:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/libs/hdf5">userland/libs/hdf5</a></p>
+</li>
+<li>
+<p>gem5 can dump statistics as HDF5: <a href="#gem5-hdf5-statistics">gem5 HDF5 statistics</a></p>
+</li>
+</ul>
 </div>
 </div>
 </div>
@@ -34649,6 +34924,9 @@ instructions 124346081</pre>
 <p>Same but with <a href="#buildroot-vanilla-kernel">Buildroot vanilla kernel</a> (kernel v4.19): 44s to blow up at "Please append a correct "root=" boot option; here are the available partitions" because missing some filesystem mount option. But likely wouldn&#8217;t be much more until after boot since we are almost already done by then! Therefore this vanilla kernel is much much faster! TODO find which config or kernel commit added so much time! Also that kernel is tiny at 8.5MB.</p>
 </div>
 <div class="paragraph">
+<p>Same but hacking <code>BR2_LINUX_KERNEL_LATEST_VERSION=y</code> and <code>BR2_PACKAGE_HOST_LINUX_HEADERS_CUSTOM_5_3=y</code> which reaches kernel 5.3.14 which closer to the LKMC one 5.4.3: 40s, which is very similar for the older kernel. Therefore it does not loook like it is a problem of kernel code changes, but rather of configs.</p>
+</div>
+<div class="paragraph">
 <p>Same but with: <a href="#gem5-arm-linux-kernel-patches">gem5 arm Linux kernel patches</a> at v4.15: 73s, kernel size: 132M.</p>
 </div>
 <div class="sect4">