baremetal: all examples working, all failures accounted for!

SIMD&FP is now enabled in arm from bootloader.

README.adoc

@@ -9795,7 +9795,7 @@ but the approximation is reasonable.
 +
 It is used mostly for microarchitecture research purposes: when you are making a new chip technology, you don't really need to specialize enormously to an existing microarchitecture, but rather develop something that will work with a wide range of future architectures.
 ** runs are deterministic by default, unlike QEMU which has a special <<qemu-record-and-replay>> mode, that requires first playing the content once and then replaying
-** gem5 ARM at least appears to implement more low level CPU functionality than QEMU, e.g. QEMU only added EL2 in 2018: https://stackoverflow.com/questions/42824706/qemu-system-aarch64-entering-el1-when-emulating-a53-power-up See also: <<arm-exception-level>>
+** gem5 ARM at least appears to implement more low level CPU functionality than QEMU, e.g. QEMU only added EL2 in 2018: https://stackoverflow.com/questions/42824706/qemu-system-aarch64-entering-el1-when-emulating-a53-power-up See also: <<arm-exception-levels>>
 * disadvantage of gem5: slower than QEMU, see: <<benchmark-linux-kernel-boot>>
 +
 This implies that the user base is much smaller, since no Android devs.
@@ -13034,6 +13034,19 @@ Does not have as many assembly code examples as you'd hope however...
 
 Latest version at: https://developer.arm.com/docs/den0024/latest/preface
 
+===== ARM processor documentation
+
+ARM also releases documentation specific to each given processor.
+
+This adds extra details to the more portable <<armarm8>> ISA documentation.
+
+[[arm-cortex15-trm]]
+===== ARM Cortex-A15 MPCore Processor Technical Reference Manual r4p0
+
+http://infocenter.arm.com/help/topic/com.arm.doc.ddi0438i/DDI0438I_cortex_a15_r4p0_trm.pdf
+
+2013.
+
 == Baremetal
 
 Getting started at: <<baremetal-setup>>
@@ -13226,9 +13239,51 @@ collect2: error: ld returned 1 exit status
 with the prebuilt toolchain, and I'm lazy to debug.
 * there seems to to be no analogous `aarch64` Ubuntu package to `gcc-arm-none-eabi`: https://askubuntu.com/questions/1049249/is-there-a-package-with-the-aarch64-version-of-gcc-arm-none-eabi-for-bare-metal
 
-=== C++ baremetal
+[[baremetal-cpp]]
+=== Baremetal C++
 
-TODO I tried by there was an error. Not yet properly reported. Should not be hard in theory since `libstdc++` is just part of GCC, as shown at: https://stackoverflow.com/questions/21872229/how-to-edit-and-re-build-the-gcc-libstdc-c-standard-library-source/51946224#51946224
+TODO not working as of 8825222579767f2ee7e46ffd8204b9e509440759 + 1. Not yet properly researched / reported upstream yet.
+
+Should not be hard in theory since `libstdc++` is just part of GCC, as shown at: https://stackoverflow.com/questions/21872229/how-to-edit-and-re-build-the-gcc-libstdc-c-standard-library-source/51946224#51946224
+
+To test it out, I first hack link:common.py[] to enable `C++`:
+
+....
+consts['baremetal_build_in_exts'] = consts['build_in_exts']
+....
+
+and then I hack link:userland/arch/aarch64/c/multiline.cpp[] to consist only of an empty main:
+
+....
+int main() {}
+....
+
+then for example:
+
+....
+./build-baremetal --arch aarch64
+./run --arch aarch64 --baremetal userland/arch/aarch64/c/multiline.cpp
+....
+
+fails with:
+
+....
+rom: requested regions overlap (rom dtb. free=0x00000000000000a0, addr=0x0000000000000000)
+qemu-system-aarch64: rom check and register reset failed
+....
+
+and the gem5 build fails completely:
+
+....
+./build-baremetal --arch aarch64 --emulator gem5 userland/arch/aarch64/c/multiline.cpp
+....
+
+fails with:
+
+....
+/tmp/ccFd2YIB.o:(.eh_frame+0x1c): relocation truncated to fit: R_AARCH64_PREL32 against `.text'
+collect2: error: ld returned 1 exit status
+....
 
 === GDB builtin CPU simulator
 
@@ -13300,7 +13355,7 @@ In this section we will focus on learning ARM architecture concepts that can onl
 
 Userland information can be found at: https://github.com/cirosantilli/arm-assembly-cheat
 
-==== ARM exception level
+==== ARM exception levels
 
 ARM exception levels are analogous to x86 <<ring0,rings>>.
 
@@ -13512,7 +13567,7 @@ The first part of the table contains:
 
 and the following other parts are analogous, but referring to `SPx` and lower ELs.
 
-We are going to do everything in <<arm-exception-level,EL1>> for now.
+We are going to do everything in <<arm-exception-levels,EL1>> for now.
 
 On the terminal output, we observe the initial values of:
 
@@ -13729,7 +13784,21 @@ class RealViewPBX(RealView):
 
 ==== aarch64 baremetal NEON setup
 
-Inside link:baremetal/lib/aarch64.S[] there is a chunk of code called "NEON setup".
+Inside link:baremetal/lib/aarch64.S[] there is a chunk of code that enables floating point operations:
+
+....
+mov x1, 0x3 << 20
+msr cpacr_el1, x1
+isb
+....
+
+`cpacr_el1` is documented at <<armarm8>> D10.2.29 "CPACR_EL1, Architectural Feature Access Control Register".
+
+Here we touch the FPEN bits to 3, which enable floating point operations:
+
+____
+11 This control does not cause any instructions to be trapped.
+____
 
 Without that, the `printf`:
 
@@ -13787,7 +13856,7 @@ ISB
 
 but it entered an exception loop at `MSR CPTR_EL3, XZR`.
 
-We then found out that QEMU starts in EL1, and so we kept just the EL1 part, and it worked. Related:
+We then found out that QEMU <<arm-exception-levels,<starts in EL1>>, and so we kept just the EL1 part, and it worked. Related:
 
 * https://stackoverflow.com/questions/42824706/qemu-system-aarch64-entering-el1-when-emulating-a53-power-up
 * https://stackoverflow.com/questions/37299524/neon-support-in-armv8-system-mode-qemu
@@ -15248,13 +15317,13 @@ So setup this `on_exit` automatically from all our <<baremetal-bootloaders>>, so
 +
 The following examples end up testing that our setup is working:
 +
-* link:baremetal/c/assert_fail.c[]
+* link:userland/c/assert_fail.c[]
-* link:baremetal/return1.c[]
+* link:userland/c/return0.c[]
-* link:baremetal/return2.c[]
+* link:userland/c/return1.c[]
-* link:baremetal/exit0.c[]
+* link:userland/c/return2.c[]
-* link:baremetal/exit1.c[]
+* link:userland/c/exit0.c[]
-* link:baremetal/arch/arm/return1.S[]
+* link:userland/c/exit1.c[]
-* link:baremetal/arch/aarch64/return1.S[]
+* link:userland/c/exit2.c[]
 
 Beware that on Linux kernel simulations, you cannot even echo that string from userland, since userland stdout shows up on the serial.
 

baremetal/arch/aarch64/el.c

@@ -1,4 +1,4 @@
-/* https://github.com/cirosantilli/linux-kernel-module-cheat#arm-exception-level */
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#arm-exception-levels */
 
 #include <stdio.h>
 #include <inttypes.h>

baremetal/arch/aarch64/multicore.S

@@ -1,7 +1,8 @@
 /* https://github.com/cirosantilli/linux-kernel-module-cheat#arm-multicore */
 
-.global main
+#include <lkmc.h>
-main:
+
+LKMC_PROLOGUE
     /* Reset spinlock. */
     mov x0, 0
     ldr x1, =spinlock
@@ -65,9 +66,6 @@ spinlock_start:
     /* Hint CPU 0 to enter low power mode. */
     wfe
     cbz x0, spinlock_start
-
+LKMC_EPILOGUE
-    mov x0, 0
-    ret
-
 spinlock:
     .skip 8

baremetal/arch/arm/el.c

@@ -1,4 +1,4 @@
-/* https://github.com/cirosantilli/linux-kernel-module-cheat#arm-exception-level */
+/* https://github.com/cirosantilli/linux-kernel-module-cheat#arm-exception-levels */
 
 #include <stdio.h>
 #include <inttypes.h>

baremetal/arch/arm/gem5_assert.S

@@ -1,19 +0,0 @@
-/* assert 0x12345678 + 1 == 0x12345679 */
-
-#include <lkmc/m5ops.h>
-
-.global main
-main:
-    adr r0, myvar
-    ldr r1, [r0]
-    add r1, r1, #1
-    str r1, [r0]
-    movw r2, #0x5679
-    movt r2, #0x1234
-    cmp r1, r2
-    beq ok
-    LKMC_M5OPS_FAIL_1_ASM
-ok:
-    LKMC_M5OPS_EXIT_ASM
-myvar:
-    .word 0x12345678

baremetal/arch/arm/multicore.S

@@ -1,16 +1,17 @@
 /* https://github.com/cirosantilli/linux-kernel-module-cheat#arm-multicore */
 
-.global main
+#include <lkmc.h>
-main:
+
-    mov r0, #0
+LKMC_PROLOGUE
+    mov r0, 0
     ldr r1, =spinlock
     str r0, [r1]
     /* Get CPU ID. */
     mrc p15, 0, r1, c0, c0, 5
-    ands r1, r1, #3
+    ands r1, r1, 3
     beq cpu0_only
 cpu1_only:
-    mov r0, #1
+    mov r0, 1
     ldr r1, =spinlock
     str r0, [r1]
     dmb sy
@@ -22,17 +23,16 @@ cpu0_only:
 #if !LKMC_GEM5
     /* PSCI CPU_ON. */
     ldr r0, =0x84000003
-    mov r1, #1
+    mov r1, 1
     ldr r2, =cpu1_only
-    mov r3, #0
+    mov r3, 0
     hvc 0
 #endif
 spinlock_start:
     ldr r0, spinlock
     wfe
-    cmp r0, #0
+    cmp r0, 0
     beq spinlock_start
-    mov r0, #0
+LKMC_EPILOGUE
-    bx lr
 spinlock:
     .skip 4

baremetal/arch/arm/regs.S

@@ -1,9 +1,11 @@
 /* See the aarch64 version. */
-.global main
+
-main:
+#include <lkmc.h>
+
+LKMC_PROLOGUE
     mov r0, #1
     /* test-gdb-r0 */
     mov r1, #2
     /* test-gdb-r1 */
     mov r0, #0
-    bx lr
+LKMC_EPILOGUE

baremetal/arch/arm/return1.S

@@ -1,5 +0,0 @@
-/* https://github.com/cirosantilli/linux-kernel-module-cheat#magic-failure-string */
-.global main
-main:
-    mov r0, #1
-    bx lr

baremetal/lib/aarch64.S

@@ -2,15 +2,15 @@
 
 .global lkmc_start
 lkmc_start:
-    /* = NEON setup */
-    mov x1, #(0x3 << 20)
-    msr cpacr_el1, x1
-    isb
-
     /* Load the vector table. */
     ldr x0, =lkmc_vector_table
     msr vbar_el1, x0
 
+    /* https://github.com/cirosantilli/linux-kernel-module-cheat#aarch64-baremetal-neon-setup */
+    mov x1, 0x3 << 20
+    msr cpacr_el1, x1
+    isb
+
     /* Prepare the stack for main, mandatory for C code. */
     ldr x0, =stack_top
     mov sp, x0
@@ -30,8 +30,8 @@ LKMC_VECTOR_TABLE
 
 /* Default trap handler. */
 LKMC_WEAK(lkmc_vector_trap_handler)
-    ldr x0, =lkmc_vector_trap_handler_error
+    ldr x0, =lkmc_vector_trap_handler_error_message
     bl puts
     bl abort
-lkmc_vector_trap_handler_error:
+lkmc_vector_trap_handler_error_message:
     .asciz "error: unexpected interrupt"

baremetal/lib/arm.S

@@ -5,6 +5,19 @@ lkmc_start:
     /* Prepare the stack for main, mandatory for C code. */
     ldr sp, =stack_top
 
+    /* Enable floating point.
+     * Code copied from: http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0409h/CHDEGGFF.html
+     * Without this, SIMD operations such as vmov raise an exception.
+     */
+    mrc p15, 0, r0, c1, c0, 2 /* Read CPACR into r0 */
+    orr r0, r0, 3 << 20       /* OR in User and Privileged access for CP10 */
+    orr r0, r0, 3 << 22       /* OR in User and Privileged access for CP11 */
+    bic r0, r0, 3 << 30       /* Clear ASEDIS/D32DIS if set */
+    mcr p15, 0, r0, c1, c0, 2 /* Store new access permissions into CPACR */
+    isb                       /* Ensure side-effect of CPACR is visible */
+    mov r0, 1 << 30           /* Create value with FPEXC (bit 30) set in r0 */
+    vmsr fpexc, r0            /* Enable VFP and SIMD extensions */
+
     /* https://github.com/cirosantilli/linux-kernel-module-cheat#magic-failure-string */
     ldr r0, =lkmc_baremetal_on_exit_callback
     bl on_exit

build-baremetal

@@ -116,7 +116,7 @@ Build the baremetal examples with crosstool-NG.
                 for path, in_dirnames, in_filenames in self.sh.walk(target):
                     for in_filename in in_filenames:
                         in_ext = os.path.splitext(in_filename)[1]
-                        if not in_ext in self.env['build_in_exts']:
+                        if not in_ext in self.env['baremetal_build_in_exts']:
                             continue
                         in_path = os.path.join(path, in_filename)
                         my_thread_pool.submit({

common.py

@@ -107,10 +107,13 @@ consts['cxx_ext'] = '.cpp'
 consts['header_ext'] = '.h'
 consts['kernel_module_ext'] = '.ko'
 consts['obj_ext'] = '.o'
-consts['build_in_exts'] = [
+# https://github.com/cirosantilli/linux-kernel-module-cheat#baremetal-cpp
+consts['baremetal_build_in_exts'] = [
     consts['asm_ext'],
     consts['c_ext'],
-    consts['cxx_ext'],
+]
+consts['build_in_exts'] = consts['baremetal_build_in_exts'] + [
+    consts['cxx_ext']
 ]
 consts['userland_out_exts'] = [
     consts['userland_executable_ext'],
@@ -939,7 +942,7 @@ Incompatible archs are skipped.
             env['disk_image'] = env['gem5_fake_iso']
             path = self.resolve_baremetal_executable(env['baremetal'])
             source_path_noext = os.path.splitext(join(
-                env['baremetal_source_dir'],
+                env['root_dir'],
                 os.path.relpath(path, env['baremetal_build_dir'])
             ))[0]
             for ext in [env['c_ext'], env['asm_ext']]:

path_properties.py

@@ -11,6 +11,10 @@ class PathProperties:
     # All new properties must be listed here or else you get an error.
     default_properties = {
         'allowed_archs': None,
+        # The example uses aarch32 instructions which are not present in ARMv7.
+        # Therefore, it cannot be run in baremetal ARMv7 CPUs.
+        # User mode simulation however seems to enable aarch32 so these run fine.
+        'arm_aarch32': False,
         # Examples that can be built in baremetal.
         'baremetal': False,
         'c_std': default_c_std,
@@ -52,7 +56,6 @@ class PathProperties:
         # it only generates intermediate object files. Therefore it
         # should not be run while testing.
         'no_executable': False,
-        'receives_signal': None,
         # The script requires a non-trivial argument to be passed to run properly.
         'requires_argument': False,
         'requires_dynamic_library': False,
@@ -66,6 +69,13 @@ class PathProperties:
         # deeply to the system it runs on, which would preventing further interactive
         # or test usage of the system, for example poweroff or messing up the GUI.
         'requires_sudo': False,
+        # The signal received is generated by the OS implicitly rather than explicitly
+        # done sith signal(), e.g. Illegal instruction or sigsegv.
+        # Therefore, it won't behave in the same way in baremetal. aarch64 already
+        # has an exception handler which we could use but arm doesn't and the IP
+        # goes astray, so let's just skip this for now.
+        'signal_generated_by_os': False,
+        'signal_received': None,
         # We were lazy to properly classify why we are skipping these tests.
         # TODO get it done.
         'skip_run_unclassified': False,
@@ -133,6 +143,12 @@ class PathProperties:
                 is_baremetal=is_baremetal,
                 is_userland=is_userland
             ) and
+            not (
+                is_baremetal and (
+                    self['arm_aarch32'] or
+                    self['signal_generated_by_os']
+                )
+            ) and
             not self['interactive'] and
             not self['more_than_1s'] and
             not self['no_executable'] and
@@ -246,7 +262,6 @@ path_properties_tuples = (
                         'arm': (
                             {'allowed_archs': {'arm'}},
                             {
-                                'gem5_assert.S': {'requires_m5ops': True},
                                 'multicore.S': {'test_run_args': {'cpus': 2}},
                                 'no_bootloader': (
                                     {'extra_objs_disable_baremetal_bootloader': True},
@@ -326,11 +341,13 @@ path_properties_tuples = (
                                     },
                                 ),
                                 'freestanding': freestanding_properties,
-                                'lkmc_assert_eq_fail.S': {'receives_signal': signal.Signals.SIGABRT},
+                                'lkmc_assert_eq_fail.S': {'signal_received': signal.Signals.SIGABRT},
-                                'lkmc_assert_memcmp_fail.S': {'receives_signal': signal.Signals.SIGABRT},
+                                'lkmc_assert_memcmp_fail.S': {'signal_received': signal.Signals.SIGABRT},
                                 'udf.S': {
-                                    'receives_signal': signal.Signals.SIGILL
+                                    'signal_generated_by_os': True,
+                                    'signal_received': signal.Signals.SIGILL,
                                 },
+                                'vcvta.S': {'arm_aarch32': True},
                             }
                         ),
                         'aarch64': (
@@ -344,15 +361,16 @@ path_properties_tuples = (
                                     },
                                 ),
                                 'freestanding': freestanding_properties,
-                                'lkmc_assert_eq_fail.S': {'receives_signal': signal.Signals.SIGABRT},
+                                'lkmc_assert_eq_fail.S': {'signal_received': signal.Signals.SIGABRT},
-                                'lkmc_assert_memcmp_fail.S': {'receives_signal': signal.Signals.SIGABRT},
+                                'lkmc_assert_memcmp_fail.S': {'signal_received': signal.Signals.SIGABRT},
                                 'udf.S': {
-                                    'receives_signal': signal.Signals.SIGILL
+                                    'signal_generated_by_os': True,
+                                    'signal_received': signal.Signals.SIGILL,
                                 },
                             }
                         ),
                         'lkmc_assert_fail.S': {
-                            'receives_signal': signal.Signals.SIGABRT,
+                            'signal_received': signal.Signals.SIGABRT,
                         },
                         'x86_64': (
                             {'allowed_archs': {'x86_64'}},
@@ -363,13 +381,13 @@ path_properties_tuples = (
                                     {
                                         'freestanding': freestanding_properties,
                                         'ring0.c': {
-                                            'receives_signal': signal.Signals.SIGSEGV
+                                            'signal_received': signal.Signals.SIGSEGV
                                         }
                                     }
                                 ),
                                 'freestanding': freestanding_properties,
-                                'lkmc_assert_eq_fail.S': {'receives_signal': signal.Signals.SIGABRT},
+                                'lkmc_assert_eq_fail.S': {'signal_received': signal.Signals.SIGABRT},
-                                'lkmc_assert_memcmp_fail.S': {'receives_signal': signal.Signals.SIGABRT},
+                                'lkmc_assert_memcmp_fail.S': {'signal_received': signal.Signals.SIGABRT},
                             }
                         ),
                     }
@@ -380,10 +398,10 @@ path_properties_tuples = (
                     },
                     {
                         'abort.c': {
-                            'receives_signal': signal.Signals.SIGABRT,
+                            'signal_received': signal.Signals.SIGABRT,
                         },
                         'assert_fail.c': {
-                            'receives_signal': signal.Signals.SIGABRT,
+                            'signal_received': signal.Signals.SIGABRT,
                         },
                         'exit1.c': {'exit_status': 1},
                         'exit2.c': {'exit_status': 2},

test-baremetal

@@ -46,7 +46,7 @@ If given, run only the given tests. Otherwise, run all tests.
                     path_abs = os.path.abspath(path)
                     dirpath_relative_root = path_abs[rootdir_abs_len + 1:]
                     for in_filename in in_filenames:
-                        if os.path.splitext(in_filename)[1] in (self.env['c_ext'], self.env['asm_ext']):
+                        if os.path.splitext(in_filename)[1] in self.env['baremetal_build_in_exts']:
                             path_relative_root = os.path.join(dirpath_relative_root, in_filename)
                             my_path_properties = path_properties.get(path_relative_root)
                             if my_path_properties.should_be_tested(self.env, is_baremetal=True):
@@ -55,12 +55,16 @@ If given, run only the given tests. Otherwise, run all tests.
                                     'baremetal': os.path.relpath(os.path.join(path_abs, in_filename), os.getcwd()),
                                 })
                                 cur_run_args.update(my_path_properties['test_run_args'])
-                                my_thread_pool.submit({
+                                run_test_args = {
                                     'expected_exit_status': my_path_properties['exit_status'],
                                     'run_args': cur_run_args,
                                     'run_obj': lkmc.import_path.import_path_main('run'),
                                     'test_id': path_relative_root,
-                                })
+                                }
+                                signal = my_path_properties['signal_received']
+                                if signal is not None:
+                                    run_test_args['expected_exit_status'] = 128 + signal.value
+                                my_thread_pool.submit(run_test_args)
         return self._handle_thread_pool_errors(my_thread_pool)
 
 if __name__ == '__main__':

test-user-mode

@@ -65,8 +65,9 @@ If given, run only the given tests. Otherwise, run all tests.
                                     'run_obj': lkmc.import_path.import_path_main('run'),
                                     'test_id': path_relative_root,
                                 }
-                                signal = my_path_properties['receives_signal']
+                                signal = my_path_properties['signal_received']
                                 if signal is not None:
+                                    # Python subprocess reports signals differently from Bash's 128 + signal rule.
                                     run_test_args['expected_exit_status'] = -signal.value
                                 my_thread_pool.submit(run_test_args)
         return self._handle_thread_pool_errors(my_thread_pool)

userland/arch/aarch64/c/multiline.cpp

@@ -3,7 +3,7 @@
 #include <assert.h>
 #include <inttypes.h>
 
-int main(void) {
+int main() {
     uint64_t io = 0;
     __asm__ (
         R"(

userland/arch/arm/ldmia.S

@@ -6,6 +6,7 @@
 #define ELEM_SIZE 4
 
 .data;
+.align 4
 my_array_0:
     .word 0x11111111, 0x22222222, 0x33333333, 0x44444444
 my_array_1:

userland/arch/arm/vadd_scalar.S

@@ -22,6 +22,7 @@ LKMC_PROLOGUE
 
     /* Now the same from memory with vldr and vstr. */
 .data
+.align 4
 my_float_0:
     .float 1.5
 my_float_1:

userland/c/empty.c

@@ -0,0 +1 @@
+int main(void) {}

userland/c/false.c

@@ -9,7 +9,7 @@
 
 int main(int argc, char **argv) {
     int ret;
-    if (argc == 1) {
+    if (argc <= 1) {
         ret = 1;
     } else {
         ret = strtoull(argv[1], NULL, 0);

userland/cpp/empty.cpp

@@ -0,0 +1,4 @@
+// Sanity checking low level stuff, initially inspired by baremetal.
+// https://github.com/cirosantilli/linux-kernel-module-cheat#baremetal-cpp
+
+int main() {}