user mode: use the nonexistent symlink workaround

I was considering setting --static by default to match gem5, but then
that breaks shared libraries like openblas... so let's just use the
ugly workaround for now as it seems to work...

README.adoc

@@ -1090,10 +1090,10 @@ The following subjects are particularly important:
 
 Much like <<baremetal-setup>>, this is another fun setup that does not require Buildroot or the Linux kernel.
 
-Introduction at: <<user-mode-simulation>>.
-
 Getting started at: <<qemu-user-mode-getting-started>>.
 
+Introduction at: <<user-mode-simulation>>.
+
 [[gdb]]
 == GDB step debug
 
@@ -3269,20 +3269,11 @@ See also: <<user-mode-simulation-with-glibc>>
 
 === QEMU user mode getting started
 
-TODO: at 125d14805f769104f93c510bedaa685a52ec025d we <<libc-choice,moved Buildroot from uClibc to glibc>>, and this broke running user mode executables built with Buildroot in many setups, which would be ideal setup. Fixing them requires hacking up the emulators / glibc, but I'm lazy now. More details at: <<user-mode-simulation-with-glibc>>.
+Let's run link:userland/print_argv.c[] built with the Buildroot toolchain on QEMU user mode:
-
-Running with Ubuntu 18.04 host packaged toolchains works however, as emulator maintainers have had more time to iron out the issues there, so let's go that route for now:
 
 ....
-sudo apt-get install gcc-aarch64-linux-gnu
+./build user-mode-qemu
-./build-userland \
-  --arch aarch64 \
-  --gcc-which host \
-  --userland-build-id host \
-;
 ./run \
-  --arch aarch64 \
-  --userland-build-id host \
   --userland print_argv \
   --userland-args 'asdf "qw er"' \
 ;
@@ -3295,37 +3286,52 @@ asdf
 qw er
 ....
 
-This runs link:userland/print_argv.c[].
+`./run --userland` path resolution is analogous to <<baremetal-setup-getting-started,that of `./run --baremetal`>>.
 
-All further sections assume that you are using the host toolchain unless otherwise noted:
+`./build user-mode-qemu` first builds Buildroot, and then runs `./build-userland`, which is further documented at: <<userland-directory>>. It also builds QEMU. If you ahve already done a <<qemu-buildroot-setup>> previously, this will be very fast.
+
+If you modify the userland programs, rebuild simply with:
+
+....
+./build-userland
+....
+
+==== User mode with host toolchain and QEMU
+
+If you are lazy to built the Buildroot toolchain and QEMU, you can get away on Ubuntu 18.04 with just:
+
+....
+sudo apt-get install gcc-aarch64-linux-gnu qemu-system-aarch64
+./build-userland \
+  --arch aarch64 \
+  --gcc-which host \
+  --userland-build-id host \
+;
+./run \
+  --arch aarch64 \
+  --qemu-which host
+  --userland-build-id host \
+  --userland print_argv \
+  --userland-args 'asdf "qw er"' \
+;
+....
+
+where:
 
 * `--gcc-which host`: use the host toolchain.
 +
 We must pass this to `./run` as well because QEMU must know which dynamic libraries to use. See also: <<user-mode-static-executables>>.
 * `--userland-build-id host`: put the host built into a <<build-variants>>
 
-In order to build and run with the Buildroot toolchain to try and fix the emulators, do:
+This present the usual trade-offs of using prebuilts as mentioned at: <<prebuilt>>.
-
-....
-./build --arch aarch64 user-mode-qemu
-./run \
-  --arch aarch64 \
-  --userland print_argv \
-  --userland-args 'asdf "qw er"' \
-;
-....
-
-where `build` builds the whole toolchain for us.
-
-`./run --userland` path resolution is analogous to <<baremetal-setup-getting-started,that of `./run --baremetal`>>.
-
-`./build user-mode-qemu` first builds Buildroot, and then runs `./build-userland`, which is further documented at: <<userland-directory>>. It also builds QEMU.
 
 ==== User mode simulation with glibc
 
+At 125d14805f769104f93c510bedaa685a52ec025d we <<libc-choice,moved Buildroot from uClibc to glibc>>, and caused some user mode pain, which we document here.
+
 ===== FATAL: kernel too old
 
-Happens on all gem5 setups, but not on QEMU.
+Happens on all gem5 setups, but not on QEMU on Ubuntu 18.04 host.
 
 glibc has a check for kernel version, likely obtained from the `uname` syscall, and if the kernel is not new enough, it quits.
 
@@ -3351,11 +3357,16 @@ The ID is just hardcoded on the source:
 
 ===== stack smashing detected
 
-Bug report and workaround: https://bugs.launchpad.net/qemu/+bug/1701798/comments/16
+For some reason QEMU / glibc x86_64 picks up the host libc, which breaks things.
+
+Other archs work as they different host libc is skipped. <<user-mode-static-executables>> also work.
+
+We have worked around this with with https://bugs.launchpad.net/qemu/+bug/1701798/comments/12 from the thread: https://bugs.launchpad.net/qemu/+bug/1701798 by creating the file: link:rootfs_overlay/etc/ld.so.cache[] which is a symlink to a file that cannot exist: `/dev/null/nonexistent`.
 
 Reproduction:
 
 ....
+rm -f "$(./getvar buildroot_target_dir)/etc/ld.so.cache"
 ./run --userland hello
 ./run --userland hello --qemu-which host
 ....
@@ -3367,34 +3378,25 @@ Outcome:
 qemu: uncaught target signal 6 (Aborted) - core dumped
 ....
 
-The following all work however:
+To get things working again, restore `ld.so.cache` with:
 
 ....
-./run --arch aarch64 --userland hello
+./build-buildroot
-./run --static --userland hello
 ....
 
-A non-QEMU example of stack smashing is shown at: https://stackoverflow.com/questions/1345670/stack-smashing-detected/51897264#51897264
-
 I've also tested on an Ubuntu 16.04 guest and the failure is different one:
 
 ....
 qemu: uncaught target signal 4 (Illegal instruction) - core dumped
 ....
 
-So my theory is that it must be picking up something from the host as described at: https://bugs.launchpad.net/qemu/+bug/1701798 since there are different errors in different hosts.
+A non-QEMU-specific example of stack smashing is shown at: https://stackoverflow.com/questions/1345670/stack-smashing-detected/51897264#51897264
-
-This is also consistent with the fact that aarch64 worked: those binaries are not present for QEMU to get confused in that case.
 
 Tested at: 2e32389ebf1bedd89c682aa7b8fe42c3c0cf96e5 + 1.
 
 ==== User mode static executables
 
-Running dynamically linked executables in QEMU requires pointing it to the root filesystem with the `-L` option so that it can find the dynamic linker and shared libraries.
+Example:
-
-We pass `-L` by default, so everything just works.
-
-However, in case something goes wrong, you can also try statically linked executables with:
 
 ....
 ./build-userland \
@@ -3409,7 +3411,32 @@ However, in case something goes wrong, you can also try statically linked execut
 ;
 ....
 
-gem5 user mode currently only supports static executables: <<gem5-syscall-emulation-mode>>.
+Running dynamically linked executables in QEMU requires pointing it to the root filesystem with the `-L` option so that it can find the dynamic linker and shared libraries.
+
+We pass `-L` by default, so everything just works.
+
+However, in case something goes wrong, you can also try statically linked executables, since this mechanism tends to be a bit more stable, for example:
+
+* gem5 user mode currently only supports static executables: <<gem5-syscall-emulation-mode>>
+* QEMU x86_64 guest on x86_64 host was failing with <<stack-smashing-detected>>, but we found a workaround
+
+===== User mode static executables with dynamic libraries
+
+One limitation of static executables is that Buildroot mostly only builds dynamic versions of libraries (the libc is an exception).
+
+So programs that rely on those libraries might not compile as GCC can't find the `.a` version of the library.
+
+For example, if we try to build <<blas>> statically:
+
+....
+./build-userland --has-package openblas --static -- openblas_hello
+....
+
+it fails with:
+
+....
+ld: cannot find -lopenblas
+....
 
 ==== User mode GDB
 

common.py

@@ -569,7 +569,7 @@ Valid emulators: {}
         env['buildroot_cpio'] = join(env['buildroot_images_dir'], 'rootfs.cpio')
         env['staging_dir'] = join(env['out_dir'], 'staging', env['arch'])
         env['buildroot_staging_dir'] = join(env['buildroot_build_dir'], 'staging')
-        env['target_dir'] = join(env['buildroot_build_dir'], 'target')
+        env['buildroot_target_dir'] = join(env['buildroot_build_dir'], 'target')
         if not env['_args_given']['linux_source_dir']:
             env['linux_source_dir'] = os.path.join(consts['submodules_dir'], 'linux')
         common.extract_vmlinux = os.path.join(env['linux_source_dir'], 'scripts', 'extract-vmlinux')
@@ -805,7 +805,7 @@ lunch aosp_{}-eng
                 env['buildroot_host_bin_dir'],
                 env['buildroot_toolchain_prefix']
             )
-            env['userland_library_dir'] = env['target_dir']
+            env['userland_library_dir'] = env['buildroot_target_dir']
         elif env['gcc_which'] == 'crosstool-ng':
             env['toolchain_prefix'] = os.path.join(
                 env['crosstool_ng_bin_dir'],

copy-overlay

@@ -19,6 +19,7 @@ https://github.com/cirosantilli/linux-kernel-module-cheat#rootfs_overlay
         distutils.dir_util.copy_tree(
             self.env['rootfs_overlay_dir'],
             self.env['out_rootfs_overlay_dir'],
+            preserve_symlinks=True,
             update=1,
         )
 

rootfs_overlay/etc/ld.so.cache

@@ -0,0 +1 @@
+/dev/null/nonexistent

test-user-mode

@@ -11,7 +11,6 @@ class Main(common.TestCliFunction):
             description='''\
 https://github.com/cirosantilli/linux-kernel-module-cheat#user-mode-tests
 '''
-            ,
         )
         self.add_argument(
             'tests',