futex: fix example, could go wrong in theory

Start std::memory_order stub...

README.adoc

@@ -14114,6 +14114,13 @@ Bibliography:
 
 * https://stackoverflow.com/questions/31978324/what-exactly-is-stdatomic/58904448#58904448 "What exactly is std::atomic?"
 
+[[cpp-memory-order]]
+===== C++ std::memory_order
+
+https://stackoverflow.com/questions/12346487/what-do-each-memory-order-mean
+
+TODO let's understand that fully one day.
+
 [[cpp-parallel-algorithms]]
 ===== C++ parallel algorithms
 
@@ -15492,17 +15499,17 @@ This is how threads either:
 
 This syscall is rarely used on its own, and there isn't even a glibc wrapper for it: you almost always just want to use the <<pthreads>> or <<cpp-multithreading>> wrappers which use it for you to <<userland-mutex-implementation,implement higher level constructs like mutexes>>.
 
-Futexes are bit complicated, because in order to achieve their efficiency, basically nothing is guaranteed: the wait might not wait, and the wakes might not wake. So you are just basically forced to use atomic operations on the futex memory address in order to be sure of anything.
+Futexes are bit complicated, because in order to achieve their efficiency, basically nothing is guaranteed: the wait might not wait, and the wakes might not wake.
+
+So you are just basically forced to use atomic operations on the futex memory address in order to be sure of anything (we encourage you to try without :-)).
 
 Minimal examples:
 
 * link:lkmc/futex.h[]: our futex wrapper
-* link:userland/linux/futex.c[]: minimal example, the main thread:
-** spawns a child
-** the child waits on a futex
-** the main thread sleeps for one second
-** the main thread wakes up the child
-** the child returns
+* link:userland/linux/futex.c[]: minimal example. It:
+** first spawns a child
+** then sleeps for 1 second and wakes up the futex if anyone is sleeping on it
+** the child sleeps on the futex if it reaches that futex before the end of the parent's sleep (likely). If it did reach that `FUTEX_WAIT` there, it gets awoken by the parent.
 +
 So what you see is:
 +
@@ -15510,8 +15517,8 @@ So what you see is:
 main start
 child start
 [wait 1s]
-main after sleep
-child end
+parent after sleep
+child after parent sleep
 ....
 
 ===== Userland mutex implementation

userland/linux/futex.c

@@ -8,14 +8,25 @@
 
 #include <lkmc/futex.h>
 
-static int futex1, futex2;
+static int sleep_done;
 
 void* main_thread(void *arg) {
     (void)arg;
     puts("child start");
-    lkmc_futex(&futex1, FUTEX_WAKE, 1, NULL, NULL, 0);
-    lkmc_futex(&futex2, FUTEX_WAIT, 0, NULL, NULL, 0);
-    puts("child end");
+    /* We only sleep if sleep_done is 0. In that case, we are certain
+     * that a FUTEX_WAKE is coming later on, so we won't deadlock.
+     *
+     * The while is needed due to the possibility of spurious wakeups.
+     *
+     * The __atomic_load_n is not enough to ensure that we only sleep
+     * if sleep_done is 0, because a sleep_done could happen after the check:
+     * this is guaranteed by the system call itself which atomically checks that
+     * for us as well.
+     */
+    while (!__atomic_load_n(&sleep_done, __ATOMIC_ACQUIRE)) {
+        lkmc_futex(&sleep_done, FUTEX_WAIT, 0, NULL, NULL, 0);
+    }
+    puts("child after parent sleep");
     return NULL;
 }
 
@@ -29,9 +40,19 @@ int main(void) {
         main_thread,
         NULL
     ));
-    lkmc_futex(&futex1, FUTEX_WAIT, 0, NULL, NULL, 0);
     sleep(1);
-    puts("main after sleep");
-    lkmc_futex(&futex2, FUTEX_WAKE, 1, NULL, NULL, 0);
+    puts("parent after sleep");
+    /* Mark the sleep as done. If the child still didn't reach the FUTEX_WAIT for some miracle,
+     * (extremely unlikely because we've just slept for one entire second)
+     * we don't want it to sleep, otherwise the FUTEX_WAKE that we will be done next might happen
+     * before the FUTEX_WAIT, which would have no effect, and so FUTEX_WAIT would sleep forever.
+     *
+     * __ATOMIC_ACQUIRE and __ATOMIC_RELEASE are enough here: together they ensure that once this
+     * store is done, then the load MUST see it, which is exactly what we need to avoid deadlocks.
+     * TODO is RELAXED enough? Why yes/not?
+     * See also: https://cirosantilli.com/linux-kernel-module-cheat#cpp-memory-order
+     */
+    __atomic_store_n(&sleep_done, 1, __ATOMIC_RELEASE);
+    lkmc_futex(&sleep_done, FUTEX_WAKE, 1, NULL, NULL, 0);
     assert(!pthread_join(thread, NULL));
 }