From 824f8c38f470491f89b787107ca35e50686c090d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ciro=20Santilli=20=E5=85=AD=E5=9B=9B=E4=BA=8B=E4=BB=B6=20?= =?UTF-8?q?=E6=B3=95=E8=BD=AE=E5=8A=9F?= Date: Mon, 30 Dec 2019 00:00:01 +0000 Subject: [PATCH] 3a8f2fcff5e828dfb4eb93932a25375dae8a1d4c --- index.html | 254 +++++++++++++++++++++++++++++------------------------ 1 file changed, 139 insertions(+), 115 deletions(-) diff --git a/index.html b/index.html index bbad346..e882eed 100644 --- a/index.html +++ b/index.html @@ -1888,95 +1888,96 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b
  • 33.7. Clean the build
    • -
  • 33.8. ccache
    • -
  • 33.9. Rebuild Buildroot while running
    • -
  • 33.10. Simultaneous runs
    • -
  • 33.11. Build variants +
  • 33.8. Custom build directory
    • +
  • 33.9. ccache
    • +
  • 33.10. Rebuild Buildroot while running
    • +
  • 33.11. Simultaneous runs
    • +
  • 33.12. Build variants
    • -
  • 33.12. Directory structure +
  • 33.13. Directory structure
    • -
  • 33.13. Test this repo +
  • 33.14. Test this repo
    • -
  • 33.14. Bisection
    • -
  • 33.15. path_properties
    • -
  • 33.16. Update a forked submodule
    • -
  • 33.17. Release +
  • 33.15. Bisection
    • +
  • 33.16. path_properties
    • +
  • 33.17. Update a forked submodule
    • +
  • 33.18. Release
    • -
  • 33.18. Design rationale +
  • 33.19. Design rationale
    • -
  • 33.19. Soft topics +
  • 33.20. Soft topics
    • -
  • 33.20. Bibliography
    • +
  • 33.21. Bibliography
    • @@ -1993,7 +1994,7 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b

    If you don’t know which one to go for, start with QEMU Buildroot setup getting started.

    -

    Design goals of this project are documented at: Section 33.18.1, “Design goals”.

    +

    Design goals of this project are documented at: Section 33.19.1, “Design goals”.

    1.1. QEMU Buildroot setup

    @@ -2404,7 +2405,7 @@ hello /root/.profile

    If you really want to develop semiconductors, your only choice is to join an university or a semiconductor company that has the EDA licenses.

    -

    See also: Section 33.19.2, “Should you waste your life with systems programming?”.

    +

    See also: Section 33.20.2, “Should you waste your life with systems programming?”.

    While hacking QEMU, you will likely want to GDB step its source. That is trivial since QEMU is just another userland program like any other, but our setup has a shortcut to make it even more convenient, see: Section 18.7, “Debug the emulator”.

    @@ -2842,6 +2843,14 @@ j = 0

    Remember that the gem5 boot is considerably slower than QEMU since the simulation is more detailed.

    +

    gem5 moves a bit slowly, and if your host compiler is very new, the build might be broken for it, e.g. this was the case for Ubuntu 19.10 with GCC 9 and gem5 62d75e7105fe172eb906d4f80f360ff8591d4178 from Dec 2019. You can work around that by installing an older compiler and using it with something like:

    +
    +
    +
    +
    ./build-gem5 -- CC=gcc-8 CXX=g++-8
    +
    +
    +

    To get a terminal, either open a new shell and run:

    @@ -7380,7 +7389,7 @@ qw er

    The gem5 tests require building statically with build id static, see also: Section 10.7, “gem5 syscall emulation mode”. TODO automate this better.

    -

    See: Section 33.13, “Test this repo” for more useful testing tips.

    +

    See: Section 33.14, “Test this repo” for more useful testing tips.

    @@ -8018,7 +8027,7 @@ hello
    -

    33.8. ccache

    +

    33.8. Custom build directory

    +
    +

    For now there is no way to change the build directory from out/ (resp. out.docker for <<docker>.) to something else.

    +
    +
    +

    However, if you just want to place the build storage in your hard drive and the source in your SSD, which is a good configuration if you are doing lots of builds, just create a symlink as:

    +
    +
    +
    +
    mkdir -p /mnt/hd/linux-kernel-module-cheat-out
+ln -s out /mnt/hd/linux-kernel-module-cheat-out
    +
    +
    +
    +
    +

    33.9. ccache

    ccache might save you a lot of re-build when you decide to Clean the build or create a new build variant.

    @@ -33835,7 +33859,7 @@ export CCACHE_MAXSIZE="20G"
    -

    33.9. Rebuild Buildroot while running

    +

    33.10. Rebuild Buildroot while running

    It is not possible to rebuild the root filesystem while running QEMU because QEMU holds the file qcow2 file:

    @@ -33846,7 +33870,7 @@ export CCACHE_MAXSIZE="20G"
    -

    33.10. Simultaneous runs

    +

    33.11. Simultaneous runs

    When doing long simulations sweeping across multiple system parameters, it becomes fundamental to do multiple simulations in parallel.

    @@ -33942,7 +33966,7 @@ less "$(./getvar --arch aarch64 --emulator gem5 --run-id 1 termout_file)"
    -

    To run multiple gem5 checkouts, see: Section 33.11.3.1, “gem5 worktree”.

    +

    To run multiple gem5 checkouts, see: Section 33.12.3.1, “gem5 worktree”.

    Implementation note: we create multiple namespaces for two things:

    @@ -33981,7 +34005,7 @@ less "$(./getvar --arch aarch64 --emulator gem5 --run-id 1 termout_file)"
    -

    33.11. Build variants

    +

    33.12. Build variants

    It often happens that you are comparing two versions of the build, a good and a bad one, and trying to figure out why the bad one is bad.

    @@ -33989,7 +34013,7 @@ less "$(./getvar --arch aarch64 --emulator gem5 --run-id 1 termout_file)"

    Our build variants system allows you to keep multiple built versions of all major components, so that you can easily switching between running one or the other.

    -

    33.11.1. Linux kernel build variants

    +

    33.12.1. Linux kernel build variants

    If you want to keep two builds around, one for the latest Linux version, and the other for Linux v4.16:

    @@ -34025,11 +34049,11 @@ git -C "$(./getvar linux_source_dir)" checkout -
    -

    To run both kernels simultaneously, one on each QEMU instance, see: Section 33.10, “Simultaneous runs”.

    +

    To run both kernels simultaneously, one on each QEMU instance, see: Section 33.11, “Simultaneous runs”.

    -

    33.11.2. QEMU build variants

    +

    33.12.2. QEMU build variants

    Analogous to the Linux kernel build variants but with the --qemu-build-id option instead:

    @@ -34045,7 +34069,7 @@ git -C "$(./getvar qemu_source_dir)" checkout -
    -

    33.11.3. gem5 build variants

    +

    33.12.3. gem5 build variants

    Analogous to the Linux kernel build variants but with the --gem5-build-id option instead:

    @@ -34076,7 +34100,7 @@ git -C "$(./getvar gem5_source_dir)" checkout some-branch

    Therefore, you can’t forget to checkout to the sources to that of the corresponding build before running, unless you explicitly tell gem5 to use a non-default source tree with gem5 worktree. This becomes inevitable when you want to launch multiple simultaneous runs at different checkouts.

    -
    33.11.3.1. gem5 worktree
    +
    33.12.3.1. gem5 worktree

    --gem5-build-id goes a long way, but if you want to seamlessly switch between two gem5 tress without checking out multiple times, then --gem5-worktree is for you.

    @@ -34129,7 +34153,7 @@ cd -
    -
    33.11.3.2. gem5 private source trees
    +
    33.12.3.2. gem5 private source trees

    Suppose that you are working on a private fork of gem5, but you want to use this repository to develop it as well.

    @@ -34173,7 +34197,7 @@ gem5_internal="$(pwd)/gem5-internal"
    -

    33.11.4. Buildroot build variants

    +

    33.12.4. Buildroot build variants

    Allows you to have multiple versions of the GCC toolchain or root filesystem.

    @@ -34193,9 +34217,9 @@ git -C "$(./getvar buildroot_source_dir)" checkout -
    -

    33.12. Directory structure

    +

    33.13. Directory structure

    -

    33.12.1. lkmc directory

    +

    33.13.1. lkmc directory

    lkmc/ contains sources and headers that are shared across kernel modules, userland and baremetal examples.

    @@ -34206,7 +34230,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -

    Another option would have been to name it as includes/lkmc, but that would make paths longer, and we might want to store source code in that directory as well in the future.

    -
    33.12.1.1. Userland objects vs header-only
    +
    33.13.1.1. Userland objects vs header-only

    When factoring out functionality across userland examples, there are two main options:

    @@ -34265,7 +34289,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -
    -

    33.12.2. buildroot_packages directory

    +

    33.13.2. buildroot_packages directory

    Source: buildroot_packages/.

    @@ -34314,7 +34338,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -

    A custom build script can give you more flexibility: e.g. the package can be made work with other root filesystems more easily, have better 9P support, and rebuild faster as it evades some Buildroot boilerplate.

    -
    33.12.2.1. kernel_modules buildroot package
    +
    33.13.2.1. kernel_modules buildroot package

    Source: buildroot_packages/kernel_modules/

    @@ -34361,9 +34385,9 @@ git -C "$(./getvar buildroot_source_dir)" checkout -
    -

    33.12.3. patches directory

    +

    33.13.3. patches directory

    -
    33.12.3.1. patches/global directory
    +
    33.13.3.1. patches/global directory

    Has the following structure:

    @@ -34380,7 +34404,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -
    -
    33.12.3.2. patches/manual directory
    +
    33.13.3.2. patches/manual directory

    Patches in this directory are never applied automatically: it is up to users to manually apply them before usage following the instructions in this documentation.

    @@ -34390,7 +34414,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -
    -

    33.12.4. rootfs_overlay

    +

    33.13.4. rootfs_overlay

    Source: rootfs_overlay.

    @@ -34437,7 +34461,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -

    This way you can just hack away the scripts and try them out immediately without any further operations.

    -
    33.12.4.1. out_rootfs_overlay_dir
    +
    33.13.4.1. out_rootfs_overlay_dir

    This path can be found with:

    @@ -34471,7 +34495,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -
    -

    33.12.5. lkmc.c

    +

    33.13.5. lkmc.c

    The files:

    @@ -34501,7 +34525,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -
    -

    33.12.6. rand_check.out

    +

    33.13.6. rand_check.out

    Print out several parameters that normally change randomly from boot to boot:

    @@ -34528,7 +34552,7 @@ git -C "$(./getvar buildroot_source_dir)" checkout -
    -

    33.12.7. lkmc_home

    +

    33.13.7. lkmc_home

    lkmc_home refers to the target base directory in which we put all our custom built stuff, such as userland executables and kernel modules.

    @@ -34562,9 +34586,9 @@ git -C "$(./getvar buildroot_source_dir)" checkout -
    -

    33.13. Test this repo

    +

    33.14. Test this repo

    -

    33.13.1. Automated tests

    +

    33.14.1. Automated tests

    Run almost all tests:

    @@ -34620,7 +34644,7 @@ echo $?

    test does not all possible tests, because there are too many possible variations and that would take forever. The rationale is the same as for ./build all and is explained in ./build --help.

    -
    33.13.1.1. Test arch and emulator selection
    +
    33.14.1.1. Test arch and emulator selection

    You can select multiple archs and emulators of interest, as for an other command, with:

    @@ -34653,7 +34677,7 @@ echo $?
    -
    33.13.1.2. Quit on fail
    +
    33.14.1.2. Quit on fail

    By default, continue running even after the first failure happens, and they show a summary at the end.

    @@ -34667,7 +34691,7 @@ echo $?
    -
    33.13.1.3. Test userland in full system
    +
    33.14.1.3. Test userland in full system

    TODO: we really need a mechanism to automatically generate the test list automatically e.g. based on path_properties, currently there are many tests missing, and we have to add everything manually which is very annoying.

    @@ -34696,7 +34720,7 @@ echo $?
    -
    33.13.1.4. GDB tests
    +
    33.14.1.4. GDB tests

    We have some pexpect automated tests for GDB for both userland and baremetal programs!

    @@ -34769,7 +34793,7 @@ echo $?
    -
    33.13.1.5. Magic failure string
    +
    33.14.1.5. Magic failure string

    We do not know of any way to set the emulator exit status in QEMU arm full system.

    @@ -34872,9 +34896,9 @@ echo $?
    -

    33.13.2. Non-automated tests

    +

    33.14.2. Non-automated tests

    -
    33.13.2.1. Test GDB Linux kernel
    +
    33.14.2.1. Test GDB Linux kernel

    For the Linux kernel, do the following manual tests for now.

    @@ -34912,7 +34936,7 @@ echo $?
    -
    33.13.2.2. Test the Internet
    +
    33.14.2.2. Test the Internet

    You should also test that the Internet works:

    @@ -34923,7 +34947,7 @@ echo $?
    -
    33.13.2.3. CLI script tests
    +
    33.14.2.3. CLI script tests

    build-userland and test-executables have a wide variety of target selection modes, and it was hard to keep them all working without some tests:

    @@ -34941,7 +34965,7 @@ echo $?
    -

    33.14. Bisection

    +

    33.15. Bisection

    When updating the Linux kernel, QEMU and gem5, things sometimes break.

    @@ -34997,7 +35021,7 @@ git submodule update
    -

    33.15. path_properties

    +

    33.16. path_properties

    In order to build and run each userland and baremetal example properly, we need per-file metadata such as compiler flags and required number of cores.

    @@ -35060,7 +35084,7 @@ baremetal=True
    -

    33.16. Update a forked submodule

    +

    33.17. Update a forked submodule

    This is a template update procedure for submodules for which we have some patches on on top of mainline.

    @@ -35089,9 +35113,9 @@ git commit -m "linux: update to ${next_mainline_revision}"
    -

    33.17. Release

    +

    33.18. Release

    -

    33.17.1. Release procedure

    +

    33.18.1. Release procedure

    Ensure that the Automated tests are passing on a clean build:

    @@ -35102,7 +35126,7 @@ git commit -m "linux: update to ${next_mainline_revision}"
    -

    The ./build-test command builds a superset of what will be downloaded which also tests other things we would like to be working on the release. For the minimal build to generate the files to be uploaded, see: Section 33.17.2, “release-zip”

    +

    The ./build-test command builds a superset of what will be downloaded which also tests other things we would like to be working on the release. For the minimal build to generate the files to be uploaded, see: Section 33.18.2, “release-zip”

    The clean build is necessary as it generates clean images since it is not possible to remove Buildroot packages

    @@ -35172,7 +35196,7 @@ git push --follow-tags
    -

    33.17.2. release-zip

    +

    33.18.2. release-zip

    Create a zip containing all files required for Prebuilt setup:

    @@ -35197,7 +35221,7 @@ git push --follow-tags
    -

    33.17.3. release-upload

    +

    33.18.3. release-upload

    After:

    @@ -35245,9 +35269,9 @@ git push --follow-tags
    -

    33.18. Design rationale

    +

    33.19. Design rationale

    -

    33.18.1. Design goals

    +

    33.19.1. Design goals

    This project was created to help me understand, modify and test low level system components by using system simulators.

    @@ -35323,7 +35347,7 @@ git push --follow-tags
    -

    33.18.2. Setup trade-offs

    +

    33.19.2. Setup trade-offs

    The trade-offs between the different setups are basically a balance between:

    @@ -35348,13 +35372,13 @@ git push --follow-tags

    compatibility: how likely is is that all the components will work well together: emulator, compiler, kernel, standard library, …​

  • -

    guest software availability: how wide is your choice of easily installed guest software packages? See also: Section 33.18.4, “Linux distro choice”

    +

    guest software availability: how wide is your choice of easily installed guest software packages? See also: Section 33.19.4, “Linux distro choice”

    • -

    33.18.3. Resource tradeoff guidelines

    +

    33.19.3. Resource tradeoff guidelines

    Choosing which features go into our default builds means making tradeoffs, here are our guidelines:

    @@ -35399,7 +35423,7 @@ git push --follow-tags
    -

    33.18.4. Linux distro choice

    +

    33.19.4. Linux distro choice

    We haven’t found the ultimate distro yet, here is a summary table of trade-offs that we care about: Table 8, “Comparison of Linux distros for usage in this repository”.

    @@ -35502,9 +35526,9 @@ git push --follow-tags
    -

    33.19. Soft topics

    +

    33.20. Soft topics

    -

    33.19.1. Fairy tale

    +

    33.20.1. Fairy tale

    @@ -35541,7 +35565,7 @@ git push --follow-tags
    -

    33.19.2. Should you waste your life with systems programming?

    +

    33.20.2. Should you waste your life with systems programming?

    Being the hardcore person who fully understands an important complex system such as a computer, it does have a nice ring to it doesn’t it?

    @@ -35626,7 +35650,7 @@ git push --follow-tags
    -

    33.20. Bibliography

    +

    33.21. Bibliography

    Runnable stuff: