lkmc_many_files: create package to test a rootfs with many files

Automatically pick up packages from under packages/ into BR2_EXTERNAL.

Move many subdirectory READMEs into the toplevel and link to toplevel from those subreadmes instead.

README.adoc

@@ -6578,7 +6578,7 @@ First, see if you can't get away without actually adding a new package, for exam
 
 If none of those methods are flexible enough for you, create a new package as follows:
 
-* use link:sample_package[] as a starting point
+* use link:packages/sample_package[] as a starting point
 * fork this repository, and modify that package to do what you want
 * read the comments on that package to get an idea of how to start
 * check the main manual for more complicated things: https://buildroot.org/downloads/manual/manual.html
@@ -6586,6 +6586,7 @@ If none of those methods are flexible enough for you, create a new package as fo
 +
 ....
 ./build -- sample_package-reconfigure
+./run -F '/sample_package.out'
 ....
 +
 if you make any changes to that package after the initial build: <<rebuild>>
@@ -6744,6 +6745,15 @@ We have applied link:https://github.com/cirosantilli/buildroot/commit/983fe7910a
 
 which contributed to a large part of the slowness.
 
+Test how Buildroot deals with many files with:
+
+....
+./build -B BR2_PACKAGE_LKMC_MANY_FILES=y -- lkmc_many_files-reconfigure |& ts -i '%.s'
+./build |& ts -i '%.s'
+....
+
+and notice how the second build, which does not rebuilt the package at all, still gets stuck in the `RPATH` check forever without our Buildroot patch.
+
 === Report upstream bugs
 
 When asking for help on upstream repositories outside of this repository, you will need to provide the commands that you are running in detail without referencing our scripts.
@@ -7206,20 +7216,55 @@ Every `.patch` file in this directory gets applied to Buildroot before anything
 
 This directory has been made kind of useless when we decided to use our own Buildroot fork, but we've kept the functionality just in case we someday go back to upstream Buildroot.
 
-:leveloffset: +3
+==== gem5 directory
 
-include::global_patch_dir/README.adoc[]
+We Build the gem5 emulator through Buildroot basically just to reuse its timestamping system to avoid rebuilds.
+
+There is also the `m5` tool that we must build through Buildroot ans install on the root filesystem, but we could just make two separate builds.
+
+This directory has the following structure:
+
+==== global_patch_dir
+
+Has the following structure:
+
+....
+package-name/00001-do-something.patch
+....
+
+The patches are then applied to the corresponding packages before build.
+
+Uses `BR2_GLOBAL_PATCH_DIR`.
+
+==== packages directory
+
+Any directory in that subdirectory is added to `BR2_EXTERNAL` and become available to the build.
+
+==== rootfs_overlay
+
+Copied into the target filesystem.
+
+We use it for:
+
+* customized configuration files
+* userland module test scripts that don't need to be compiled.
++
+C files for example need compilation, and must go through the regular package system, e.g. through link:kernel_module/user[].
+
+:leveloffset: +3
 
 include::kernel_module/README.adoc[]
 
 include::kernel_module/user/README.adoc[]
 
-include::rootfs_overlay/README.adoc[]
-
 :leveloffset: -3
 
 === Script man pages
 
+These appear when you do `./some-script -h`.
+
+We have to keep them as separate files from the README for that to be possible.
+
 :leveloffset: +3
 
 include::build-usage.adoc[]

build

@@ -137,7 +137,12 @@ if "$configure"; then
   for p in $(find "${root_dir}/buildroot_patches/" -maxdepth 1 -name '*.patch' -print); do
     patch -N -r - -p 1 < "$p" || :
   done
-  make O="$buildroot_out_dir" BR2_EXTERNAL="../kernel_module:../gem5:../parsec-benchmark:../sample_package" "$defconfig"
+  br2_external='../kernel_module:../gem5:../parsec-benchmark'
+  packages_dir="${root_dir}/packages"
+  for package_dir in "${packages_dir}"/*/; do
+    br2_external="${br2_external}:../packages/$(basename "${package_dir}")"
+  done
+  make O="$buildroot_out_dir" BR2_EXTERNAL="$br2_external" "$defconfig"
   # TODO Can't get rid of these for now.
   # http://stackoverflow.com/questions/44078245/is-it-possible-to-use-config-fragments-with-buildroots-config
   for config_fragment in $config_fragments; do

gem5/README.adoc

@@ -1,10 +1 @@
-= gem5
-
-We Build gem5 as a Buildroot package for the following reasons:
-
-* if we build in tree without Buildroot, we still want to build the `m5` tool through Buildroot. But then Buildroot rsyncs the build for all archs into `output/build`, which takes quite some disk space. There seems to be no way to exclude some packages from the sync.
-* it allows us to reuse Buildroot's:
-** configuration system, which reduces duplication: set the gem5 package option, everything just gets built
-** timestamping system, which could save a few seconds on rebuilds, since we then don't ever run `scons` when gem5 is not modified
-
-What is preventing upstreaming to Buildroot: a minimal kernel config like the ones QEMU has.
+https://github.com/cirosantilli/linux-kernel-module-cheat#gem5-directory

global_patch_dir/README.adoc

@@ -1,7 +1 @@
-= global_patch_dir
-
-This directory has the following structure:
-
-....
-package-name/00001-do-something.patch
-....
+https://github.com/cirosantilli/linux-kernel-module-cheat#global_patch_dir

packages/README.adoc

@@ -0,0 +1 @@
+https://github.com/cirosantilli/linux-kernel-module-cheat#packages_directory

packages/lkmc_many_files/Config.in

@@ -0,0 +1,4 @@
+config BR2_PACKAGE_LKMC_MANY_FILES
+	bool "lkmc_many_files"
+	help
+		Create a bunch of files in target/ to benchmark Buildroot.

packages/lkmc_many_files/external.desc

@@ -0,0 +1 @@
+name: LKMC_MANY_FILES

packages/lkmc_many_files/external.mk

@@ -0,0 +1,30 @@
+################################################################################
+#
+# lkmc_many_files
+#
+################################################################################
+
+LKMC_MANY_FILES_VERSION = 1.0
+LKMC_MANY_FILES_SITE = $(BR2_EXTERNAL_LKMC_MANY_FILES_PATH)
+LKMC_MANY_FILES_SITE_METHOD = local
+
+define LKMC_MANY_FILES_BUILD_CMDS
+	# D contains the source code of this package.
+	# TARGET_CONFIGURE_OPTS contains several common options such as CFLAGS and LDFLAGS.
+	$(MAKE) -C '$(@D)' $(TARGET_CONFIGURE_OPTS)
+endef
+
+define LKMC_MANY_FILES_INSTALL_TARGET_CMDS
+  # Anything put inside TARGET_DIR will end up on the guest relative to the root directory.
+  i=0; \
+  while [ $$i -le 128 ]; do \
+    j=0; \
+    while [ $$j -le 128 ]; do \
+      $(INSTALL) -D -m 0755 $(@D)/lkmc_many_files.out $(TARGET_DIR)/lkmc_many_files/$${i}_$${j}.out; \
+      j=$$(($$j+1)); \
+    done; \
+    i=$$(($$i+1)); \
+  done
+endef
+
+$(eval $(generic-package))

packages/lkmc_many_files/lkmc_many_files.c

@@ -0,0 +1,7 @@
+#include <stdio.h>
+#include <stdlib.h>
+
+int main(void) {
+	puts("hello lkmc_many_files");
+	return EXIT_SUCCESS;
+}

packages/sample_package/Makefile

@@ -0,0 +1,16 @@
+.PHONY: all clean
+
+CFLAGS_EXTRA ?= -fopenmp -std=c99 -Wall -Werror -Wextra
+IN_EXT ?= .c
+LIBS :=
+OUT_EXT ?= .out
+
+OUTS := $(addsuffix $(OUT_EXT), $(basename $(wildcard *$(IN_EXT))))
+
+all: $(OUTS)
+
+%$(OUT_EXT): %$(IN_EXT)
+	$(CC) $(CFLAGS) $(CFLAGS_EXTRA) -o '$@' '$<' $(LIBS)
+
+clean:
+	rm -f *'$(OUT_EXT)'

rootfs_overlay/README.adoc

@@ -1,8 +1 @@
-= rootfs_overlay
-
-This directory copied into the target filesystem.
-
-We use it to for things like:
-
-* customized configuration files
-* userland module test scripts
+https://github.com/cirosantilli/linux-kernel-module-cheat#rootfs_overlay