diff --git a/README.adoc b/README.adoc index 07d1513..15c96cb 100644 --- a/README.adoc +++ b/README.adoc @@ -2243,7 +2243,7 @@ The implementation is described at: https://stackoverflow.com/questions/46415059 === GDB step debug multicore userland -For a more minimal baremetal multicore setup, see: xref:arm-multicore[xrefstyle=full]. +For a more minimal baremetal multicore setup, see: xref:arm-baremetal-multicore[xrefstyle=full]. We can set and get which cores the Linux kernel allows a program to run on with `sched_getaffinity` and `sched_setaffinity`: @@ -10391,10 +10391,10 @@ There are two types of lines: Breakdown: * `25007500`: time count in some unit. Note how the microops execute at further timestamps. -* `system.cpu`: distinguishes between CPUs when there are more than one. For example, running xref:arm-multicore[xrefstyle=full] with two cores produces `system.cpu0` and `system.cpu1` +* `system.cpu`: distinguishes between CPUs when there are more than one. For example, running xref:arm-baremetal-multicore[xrefstyle=full] with two cores produces `system.cpu0` and `system.cpu1` * `T0`: thread number. TODO: https://superuser.com/questions/133082/hyper-threading-and-dual-core-whats-the-difference/995858#995858[hyperthread]? How to play with it? + -`config`.ini has `--param 'system.multi_thread = True' --param 'system.cpu[0].numThreads = 2'`, but in <> the first one alone does not produce `T1`, and with the second one simulation blows up with: +`config`.ini has `--param 'system.multi_thread = True' --param 'system.cpu[0].numThreads = 2'`, but in <> the first one alone does not produce `T1`, and with the second one simulation blows up with: + .... fatal: fatal condition interrupts.size() != numThreads occurred: CPU system.cpu has 1 interrupt controllers, but is expecting one per thread (2) @@ -12052,7 +12052,7 @@ Exiting @ tick 18446744073709551615 because simulate() limit reached Other examples of the message: -* <> with a single CPU stays stopped at an WFE sleep instruction +* <> with a single CPU stays stopped at an WFE sleep instruction * this sample bug on se.py multithreading: https://github.com/cirosantilli/linux-kernel-module-cheat/issues/81 === gem5 build options @@ -15114,7 +15114,7 @@ Then it is just a huge copy paste of infinite boring details: * <> * <> -To debug these instructoins, you can see the register values in GDB with: +To debug these instructions, you can see the register values in GDB with: .... info registers float @@ -15223,10 +15223,58 @@ This is analogous to </c/` directories show to how use inline assembly from higher level languages such as C: @@ -17345,7 +17393,7 @@ That document then describes the SVE instructions and registers. [[arm-lse]] ===== ARM Large System Extensions (LSE) -Parent section: <>. +Parent section: <>. <> "ARMv8.1-LSE, ARMv8.1 Large System Extensions" @@ -18265,7 +18313,7 @@ Exception Link Register. See the example at: xref:arm-svc-instruction[xrefstyle=full] -==== ARM multicore +==== ARM baremetal multicore Examples: @@ -18322,6 +18370,11 @@ Bibliography: https://stackoverflow.com/questions/980999/what-does-multicore-ass The WFE and SEV instructions are just hints: a compliant implementation can treat them as NOPs. +Concrete examples of the instruction can be seen at: + +* link:userland/arch/aarch64/nostartfiles/wfe.S[] +* <> + However, likely no implementation likely does (TODO confirm), since: * WFE is intended to put the core in a low power mode @@ -18331,15 +18384,37 @@ and power consumption is key in ARM applications. SEV is not the only thing that can wake up a WFE, it is only an explicit software way to do it. Notably, global monitor operations on memory accesses of regions marked by LDAXR and STLXR instructions can also wake up a WFE sleeping core. This is done to allow spinlocks opens to automatically wake up WFE sleeping cores at free time without the need for a explicit SEV. -WFE and SEV are usable from userland, and are part of a efficient spinlock implementation. +WFE and SEV are usable from userland, and are part of an efficient spinlock implementation, which maybe is not something that userland should ever tho and just stick to mutexes? + +There is a control bit `SCTLR_EL1.nTWE` that determines if WFE is trapped or not, i.e.: is that bit is set, then it is trapped and EL0 execution raises an exception in EL1. Linux v5.2.1 does not seem to trap however, tested with `--trace ExecAll` in a full system simulation. But then, how does the kernel prevent CPUs from going to sleep randomly and instead reschedules other tasks? Does the kernel check if CPUs are in WFE when it wakes up on the timer, and only then reschedules? This would allow for userland to implement fast spinlocks if the spinlock returns faster than the timer. The kernel seems to setup NTWE at: + +include/asm/sysreg.h + +.... +#define SCTLR_EL1_SET (SCTLR_ELx_M | SCTLR_ELx_C | SCTLR_ELx_SA |\ + ... + SCTLR_EL1_NTWE | SCTLR_ELx_IESB | SCTLR_EL1_SPAN |\ +.... + +and: + +mm/proc.S + +.... + /* + * Prepare SCTLR + */ + mov_q x0, SCTLR_EL1_SET +.... Quotes for the above <> G1.18.1 "Wait For Event and Send Event": + ____ The following events are WFE wake-up events: \[...] -- An event caused by the clearing of the global monitor associated with the PE +* An event caused by the clearing of the global monitor associated with the PE ____ and <> E2.9.6 "Use of WFE and SEV instructions by spin-locks": diff --git a/baremetal/arch/aarch64/multicore.c b/baremetal/arch/aarch64/multicore.c index 151ed35..f371a8b 100644 --- a/baremetal/arch/aarch64/multicore.c +++ b/baremetal/arch/aarch64/multicore.c @@ -1,4 +1,4 @@ -/* https://cirosantilli.com/linux-kernel-module-cheat#arm-multicore +/* https://cirosantilli.com/linux-kernel-module-cheat#arm-baremetal-multicore * * Beware: things will blow up if the stack for CPU0 grow too much and * reaches that of CPU1. This is why it is so hard to do multithreading diff --git a/baremetal/arch/aarch64/no_bootloader/multicore_asm.S b/baremetal/arch/aarch64/no_bootloader/multicore_asm.S index d0a270f..ac240dc 100644 --- a/baremetal/arch/aarch64/no_bootloader/multicore_asm.S +++ b/baremetal/arch/aarch64/no_bootloader/multicore_asm.S @@ -1,4 +1,4 @@ -/* https://cirosantilli.com/linux-kernel-module-cheat#arm-multicore +/* https://cirosantilli.com/linux-kernel-module-cheat#arm-baremetal-multicore * * This has to be in no_bootloader */ diff --git a/baremetal/arch/arm/multicore.c b/baremetal/arch/arm/multicore.c index c29ea76..9c7599f 100644 --- a/baremetal/arch/arm/multicore.c +++ b/baremetal/arch/arm/multicore.c @@ -1,4 +1,4 @@ -/* https://cirosantilli.com/linux-kernel-module-cheat#arm-multicore */ +/* https://cirosantilli.com/linux-kernel-module-cheat#arm-baremetal-multicore */ #include diff --git a/baremetal/arch/arm/no_bootloader/multicore_asm.S b/baremetal/arch/arm/no_bootloader/multicore_asm.S index ba87d4e..9bd944d 100644 --- a/baremetal/arch/arm/no_bootloader/multicore_asm.S +++ b/baremetal/arch/arm/no_bootloader/multicore_asm.S @@ -1,4 +1,4 @@ -/* https://cirosantilli.com/linux-kernel-module-cheat#arm-multicore */ +/* https://cirosantilli.com/linux-kernel-module-cheat#arm-baremetal-multicore */ #include diff --git a/path_properties.py b/path_properties.py index 37e1d53..890c0f3 100644 --- a/path_properties.py +++ b/path_properties.py @@ -535,7 +535,7 @@ path_properties_tuples = ( nostartfiles_properties, { # https://github.com/cirosantilli/linux-kernel-module-cheat/issues/107 - 'exit.s': {'skip_run_unclassified': True}, + 'exit.S': {'skip_run_unclassified': True}, } ), 'udf.S': { @@ -573,6 +573,13 @@ path_properties_tuples = ( 'rdtscp.c': {'uses_instructions': {'x86_64': {'rdtscp'}}}, } ), + 'nostartfiles': ( + nostartfiles_properties, + { + # https://github.com/cirosantilli/linux-kernel-module-cheat/issues/107 + 'exit.S': {'skip_run_unclassified': True}, + } + ), 'div_overflow.S': {'signal_received': signal.Signals.SIGFPE}, 'div_zero.S': {'signal_received': signal.Signals.SIGFPE}, 'fabs.S': {'uses_instructions': {'x86_64': {'fcomip'}}}, diff --git a/userland/arch/aarch64/nostartfiles/wfe.S b/userland/arch/aarch64/nostartfiles/wfe.S new file mode 100644 index 0000000..ae1584e --- /dev/null +++ b/userland/arch/aarch64/nostartfiles/wfe.S @@ -0,0 +1,6 @@ +/* https://cirosantilli.com/linux-kernel-module-cheat#arm-wfe-and-sev-instructions */ +.global _start +_start: + wfe + mov x0, 0 + bl exit diff --git a/userland/arch/x86_64/nostartfiles/README.adoc b/userland/arch/x86_64/nostartfiles/README.adoc new file mode 100644 index 0000000..488ce67 --- /dev/null +++ b/userland/arch/x86_64/nostartfiles/README.adoc @@ -0,0 +1 @@ +https://cirosantilli.com/linux-kernel-module-cheat#nostartfiles-programs diff --git a/userland/arch/x86_64/nostartfiles/build b/userland/arch/x86_64/nostartfiles/build new file mode 120000 index 0000000..ab18017 --- /dev/null +++ b/userland/arch/x86_64/nostartfiles/build @@ -0,0 +1 @@ +../build \ No newline at end of file diff --git a/userland/arch/x86_64/nostartfiles/exit.S b/userland/arch/x86_64/nostartfiles/exit.S new file mode 100644 index 0000000..581d017 --- /dev/null +++ b/userland/arch/x86_64/nostartfiles/exit.S @@ -0,0 +1,4 @@ +.global _start +_start: + mov $0, %rdi + call exit diff --git a/userland/arch/x86_64/nostartfiles/test b/userland/arch/x86_64/nostartfiles/test new file mode 120000 index 0000000..419df4f --- /dev/null +++ b/userland/arch/x86_64/nostartfiles/test @@ -0,0 +1 @@ +../test \ No newline at end of file