mirror of
https://github.com/cirosantilli/linux-kernel-module-cheat.git
synced 2026-01-23 02:05:57 +01:00
move user mode to toplevel section, because too good / large :-)
This commit is contained in:
Ciro Santilli 六四事件 法轮功
451
README.adoc
451
README.adoc
@@ -98,7 +98,6 @@ See also: <<cpu-architecture,CPU architectures>>.
|
||||
I now urge you to read the following sections which contain widely applicable information:
|
||||
|
||||
* <<run-command-after-boot>>
|
||||
* <<rebuild-buildroot-packages>>
|
||||
* <<clean-the-build>>
|
||||
* <<build-the-documentation>>
|
||||
* Linux kernel
|
||||
@@ -1079,6 +1078,7 @@ Just don't forget to remove your breakpoints after `rmmod`, or they will point t
|
||||
|
||||
TODO: why does `break work_func` for `insmod kthread.ko` not very well? Sometimes it breaks but not others.
|
||||
|
||||
[[gdb-step-debug-kernel-module-arm]]
|
||||
==== GDB step debug kernel module insmodded by init on ARM
|
||||
|
||||
TODO on `arm` 51e31cdc2933a774c2a0dc62664ad8acec1d2dbe it does not always work, and `lx-symbols` fails with the message:
|
||||
@@ -1969,7 +1969,7 @@ instead of `kgdboc=ttyS0,115200`, you enter a different debugging mode called KD
|
||||
|
||||
TODO is there any advantage of using KDB over GDB? Except for the fact that you need potentially less setup?
|
||||
|
||||
TODO: only works in <<graphic-mode>>. On the serial, prompt hangs, and the characters I type don't show up at all.
|
||||
TODO: only works in <<graphics,graphic mode>>. On the serial, prompt hangs, and the characters I type don't show up at all.
|
||||
|
||||
In QEMU:
|
||||
|
||||
@@ -2833,6 +2833,242 @@ The main use case for `-enable-kvm` in this repository is to test if something t
|
||||
|
||||
For example, when porting a benchmark to Buildroot, you can first use QEMU's KVM to test that benchmarks is producing the correct results, before analysing them more deeply in gem5, which runs much slower.
|
||||
|
||||
== User mode simulation
|
||||
|
||||
Both QEMU and gem5 have an user mode simulation mode in addition to full system simulation that we consider elsewhere in this project.
|
||||
|
||||
In QEMU, it is called just <<qemu-user-mode,"user mode">>, and in gem5 it is called <<gem5-syscall-emulation-mode,syscall emulation mode>>.
|
||||
|
||||
In both, the basic idea is the same.
|
||||
|
||||
User mode simulation takes regular userland executables of any arch as input and executes them directly, without booting a kernel.
|
||||
|
||||
Instead of simulating the full system, it translates normal instructions like in full system mode, but magically forwards system calls to the host OS.
|
||||
|
||||
Advantages over full system simulation:
|
||||
|
||||
* the simulation may <<user-mode-vs-full-system-benchmark,run faster>> since you don't have to simulate the Linux kernel and several device models
|
||||
* you don't need to build your own kernel or root filesystem, which saves time. You still need a toolchain however, but the pre-packaged ones may work fine.
|
||||
|
||||
Disadvantages:
|
||||
|
||||
* lower guest to host portability:
|
||||
** TODO confirm: host OS == guest OS?
|
||||
** TODO confirm: the host Linux kernel should be newer than the kernel the executable was built for.
|
||||
+
|
||||
It may still work even if that is not the case, but could fail is a missing system call is reached.
|
||||
+
|
||||
The target Linux kernel of the executable is a GCC toolchain build-time configuration.
|
||||
* cannot be used to test the Linux kernel, and results are less representative of a real system since we are faking more
|
||||
|
||||
=== QEMU user mode
|
||||
|
||||
First let's run a dynamically linked executable built with the Buildroot toolchain:
|
||||
|
||||
....
|
||||
./build-qemu --arch arm --userland
|
||||
./build-userland --arch arm
|
||||
./build-buildroot --arch arm
|
||||
./run \
|
||||
--arch arm \
|
||||
--userland print_argv \
|
||||
-- \
|
||||
asdf qwer \
|
||||
;
|
||||
....
|
||||
|
||||
This runs link:userland/print_argv.c[]. `--userland` path resolution is analogous to <<baremetal-setup-getting-started,that of `--baremetal`>>.
|
||||
|
||||
`./build-userland` is further documented at: <<userland-directory>>.
|
||||
|
||||
Running dynamically linked executables in QEMU requires pointing it to the root filesystem with the `-L` option so that it can find the dynamic linker and shared libraries.
|
||||
|
||||
We pass `-L` by default, so everything just works:
|
||||
|
||||
You can also try statically linked executables with:
|
||||
|
||||
....
|
||||
./build-userland \
|
||||
--arch arm \
|
||||
--make-args='CCFLAGS_EXTRA=-static' \
|
||||
--userland-build-id static \
|
||||
;
|
||||
./run \
|
||||
--arch arm \
|
||||
--userland-build-id static \
|
||||
--userland print_argv \
|
||||
-- \
|
||||
asdf qwer \
|
||||
;
|
||||
....
|
||||
|
||||
Or you can run statically linked built by the host packaged toolchain with:
|
||||
|
||||
....
|
||||
./build-userland \
|
||||
--arch arm \
|
||||
--host \
|
||||
--make-args='-B CFLAGS_EXTRA=-static' \
|
||||
--userland-build-id host-static \
|
||||
;
|
||||
./run \
|
||||
--arch arm \
|
||||
--userland-build-id host-static \
|
||||
--userland print_argv \
|
||||
-- \
|
||||
asdf qwer \
|
||||
;
|
||||
....
|
||||
|
||||
TODO expose dynamically linked executables built by the host toolchain. It also works, we just have to use e.g. `-L /usr/aarch64-linux-gnu`, so it's not really hard, I'm just lazy.
|
||||
|
||||
==== QEMU user mode GDB
|
||||
|
||||
It's nice when <<gdb,the obvious>> just works, right?
|
||||
|
||||
....
|
||||
./run \
|
||||
--arch arm \
|
||||
--debug-guest \
|
||||
--userland print_argv \
|
||||
-- \
|
||||
asdf qwer \
|
||||
;
|
||||
....
|
||||
|
||||
and on another shell:
|
||||
|
||||
....
|
||||
./run-gdb \
|
||||
--arch arm \
|
||||
--userland print_argv \
|
||||
main \
|
||||
;
|
||||
....
|
||||
|
||||
or to stop at the very first instruction of a freestanding program, just use `--no-continue` TODO example.
|
||||
|
||||
=== gem5 syscall emulation mode
|
||||
|
||||
Less robust than QEMU's, but still usable:
|
||||
|
||||
* https://stackoverflow.com/questions/48986597/when-should-you-use-full-system-fs-vs-syscall-emulation-se-with-userland-program
|
||||
* https://stackoverflow.com/questions/48959349/how-to-solve-fatal-kernel-too-old-when-running-gem5-in-syscall-emulation-se-m
|
||||
|
||||
There are much more unimplemented syscalls in gem5 than in QEMU. Many of those are trivial to implement however.
|
||||
|
||||
As of 185c2730cc78d5adda683d76c0e3b35e7cb534f0, dynamically linked executables only work on x86, and they can only use the host libraries, which is ugly:
|
||||
|
||||
* https://stackoverflow.com/questions/50542222/how-to-run-a-dynamically-linked-executable-syscall-emulation-mode-se-py-in-gem5
|
||||
* https://www.mail-archive.com/gem5-users@gem5.org/msg15585.html
|
||||
|
||||
If you try dynamically linked executables on ARM, they fail with:
|
||||
|
||||
....
|
||||
fatal: Unable to open dynamic executable's interpreter.
|
||||
....
|
||||
|
||||
So let's just play with some static ones:
|
||||
|
||||
....
|
||||
./build-userland \
|
||||
--arch aarch64 \
|
||||
--userland-build-id static \
|
||||
--make-args='CCFLAGS_EXTRA=-static' \
|
||||
;
|
||||
./run \
|
||||
--arch aarch64 \
|
||||
--gem5 \
|
||||
--userland print_argv \
|
||||
--userland-build-id static \
|
||||
-- \
|
||||
--options 'asdf "qw er"' \
|
||||
;
|
||||
....
|
||||
|
||||
TODO: how to escape spaces?
|
||||
|
||||
Step debug also works:
|
||||
|
||||
....
|
||||
./run \
|
||||
--arch arm \
|
||||
--debug-guest \
|
||||
--gem5 \
|
||||
--userland print_argv \
|
||||
--userland-build-id static \
|
||||
-- \
|
||||
--options 'asdf "qw er"' \
|
||||
;
|
||||
./run-gdb \
|
||||
--arch arm \
|
||||
--gem5 \
|
||||
--userland print_argv \
|
||||
--userland-build-id static \
|
||||
main \
|
||||
;
|
||||
....
|
||||
|
||||
==== User mode vs full system benchmark
|
||||
|
||||
Let's see if user mode runs considerably faster than full system or not.
|
||||
|
||||
gem5 user mode:
|
||||
|
||||
....
|
||||
./build-buildroot --config 'BR2_PACKAGE_DHRYSTONE=y' --arch arm
|
||||
make \
|
||||
-B \
|
||||
-C "$(./getvar --arch arm build_dir)/dhrystone-2" \
|
||||
CC="$(./run-toolchain --arch arm --dry gcc)" \
|
||||
CFLAGS=-static \
|
||||
;
|
||||
time \
|
||||
./run \
|
||||
--arch arm \
|
||||
--gem5 \
|
||||
--userland \
|
||||
"$(./getvar --arch arm build_dir)/dhrystone-2/dhrystone" \
|
||||
-- \
|
||||
--options 100000 \
|
||||
;
|
||||
....
|
||||
|
||||
gem5 full system:
|
||||
|
||||
....
|
||||
time \
|
||||
./run \
|
||||
--arch arm \
|
||||
--eval-busybox '/gem5.sh' \
|
||||
--gem5
|
||||
--gem5-readfile 'dhrystone 100000' \
|
||||
;
|
||||
....
|
||||
|
||||
QEMU user mode:
|
||||
|
||||
....
|
||||
time qemu-arm "$(./getvar --arch arm build_dir)/dhrystone-2/dhrystone" 100000000
|
||||
....
|
||||
|
||||
QEMU full system:
|
||||
|
||||
....
|
||||
time \
|
||||
./run \
|
||||
--arch arm \
|
||||
--eval-busybox 'time dhrystone 100000000;/poweroff.out' \
|
||||
;
|
||||
....
|
||||
|
||||
Result on <<p51>> at bad30f513c46c1b0995d3a10c0d9bc2a33dc4fa0:
|
||||
|
||||
* gem5 user: 33 seconds
|
||||
* gem5 full system: 51 seconds
|
||||
* QEMU user: 45 seconds
|
||||
* QEMU full system: 223 seconds
|
||||
|
||||
== Kernel module utilities
|
||||
|
||||
=== insmod
|
||||
@@ -7669,211 +7905,6 @@ TODO get some working!
|
||||
|
||||
http://gedare-csphd.blogspot.co.uk/2013/02/adding-simple-io-device-to-gem5.html
|
||||
|
||||
=== QEMU user mode
|
||||
|
||||
==== QEMU user mode introduction
|
||||
|
||||
QEMU user mode is a QEMU mode that can execute userland executables of another arch directly.
|
||||
|
||||
Instead of simulating the full system, it translates normal instructions like in full system mode, but magically forwards system calls to the host OS.
|
||||
|
||||
This has the following portability implications:
|
||||
|
||||
* TODO confirm: host OS == guest OS?
|
||||
* TODO confirm: the host Linux kernel should be newer than the kernel the executable was built for.
|
||||
+
|
||||
It may still work even if that is not the case, but could fail is a missing system call is reached.
|
||||
+
|
||||
The target Linux kernel of the executable is a GCC toolchain build-time configuration.
|
||||
|
||||
QEMU user mode completely bypasses the kernel that we've built: all it takes is the userland executable. Therefore it cannot be easily used to develop the Linux kernel.
|
||||
|
||||
=== QEMU user mode getting started
|
||||
|
||||
....
|
||||
./build-userland --arch arm
|
||||
./build-buildroot --arch arm
|
||||
./run \
|
||||
--arch arm \
|
||||
--userland print_argv \
|
||||
-- \
|
||||
asdf qwer \
|
||||
;
|
||||
....
|
||||
|
||||
This runs link:userland/print_argv.c[]. `--userland` path resolution is analogous to <<baremetal-setup-getting-started,that of `--baremetal`>>.
|
||||
|
||||
`./build-userland` is further documented at: <<userland-directory>>.
|
||||
|
||||
As we've just seen, QEMU user mode supports dynamically linked executables.
|
||||
|
||||
This requires point it to the root filesystem with the `-L` option so that it can find the dynamic linker and shared libraries.
|
||||
|
||||
We pass `-L` by default, so everything just works:
|
||||
|
||||
You can also try statically linked executables with:
|
||||
|
||||
....
|
||||
./build-qemu --arch arm --userland
|
||||
./build-userland \
|
||||
--arch arm \
|
||||
--make-args='CCFLAGS_EXTRA=-static' \
|
||||
--userland-build-id static \
|
||||
;
|
||||
./run \
|
||||
--arch arm \
|
||||
--userland-build-id static \
|
||||
--userland print_argv \
|
||||
-- \
|
||||
asdf qwer \
|
||||
;
|
||||
....
|
||||
|
||||
==== QEMU user mode GDB
|
||||
|
||||
It's nice when <<gdb,the obvious>> just works, right?
|
||||
|
||||
....
|
||||
./run \
|
||||
--arch arm \
|
||||
--debug-guest \
|
||||
--userland print_argv \
|
||||
-- \
|
||||
asdf qwer \
|
||||
;
|
||||
....
|
||||
|
||||
and on another shell:
|
||||
|
||||
....
|
||||
./run-gdb \
|
||||
--arch arm \
|
||||
--userland print_argv \
|
||||
main \
|
||||
;
|
||||
....
|
||||
|
||||
or to stop at the very first instruction of a freestanding program, just use `--no-continue` TODO example.
|
||||
|
||||
==== gem5 syscall emulation mode
|
||||
|
||||
Analogous to <<qemu-user-mode>>, but less robust:
|
||||
|
||||
* https://stackoverflow.com/questions/48986597/when-should-you-use-full-system-fs-vs-syscall-emulation-se-with-userland-program
|
||||
* https://stackoverflow.com/questions/48959349/how-to-solve-fatal-kernel-too-old-when-running-gem5-in-syscall-emulation-se-m
|
||||
|
||||
As of 185c2730cc78d5adda683d76c0e3b35e7cb534f0, dynamically linked executables only work on x86, and they can only use the host libraries, which is ugly:
|
||||
|
||||
* https://stackoverflow.com/questions/50542222/how-to-run-a-dynamically-linked-executable-syscall-emulation-mode-se-py-in-gem5
|
||||
* https://www.mail-archive.com/gem5-users@gem5.org/msg15585.html
|
||||
|
||||
If you try dynamically linked executables on ARM, they fail with:
|
||||
|
||||
....
|
||||
fatal: Unable to open dynamic executable's interpreter.
|
||||
....
|
||||
|
||||
So let's just play with some static ones:
|
||||
|
||||
....
|
||||
./build-userland \
|
||||
--arch aarch64 \
|
||||
--userland-build-id static \
|
||||
--make-args='CCFLAGS_EXTRA=-static' \
|
||||
;
|
||||
./run \
|
||||
--arch aarch64 \
|
||||
--gem5 \
|
||||
--userland print_argv \
|
||||
--userland-build-id static \
|
||||
-- \
|
||||
--options 'asdf "qw er"' \
|
||||
;
|
||||
....
|
||||
|
||||
TODO: how to escape spaces?
|
||||
|
||||
Step debug also works:
|
||||
|
||||
....
|
||||
./run \
|
||||
--arch arm \
|
||||
--debug-guest \
|
||||
--gem5 \
|
||||
--userland print_argv \
|
||||
--userland-build-id static \
|
||||
-- \
|
||||
--options 'asdf "qw er"' \
|
||||
;
|
||||
./run-gdb \
|
||||
--arch arm \
|
||||
--gem5 \
|
||||
--userland print_argv \
|
||||
--userland-build-id static \
|
||||
main \
|
||||
;
|
||||
....
|
||||
|
||||
==== User mode vs full system benchmark
|
||||
|
||||
Let's see if user mode runs considerably faster than full system or not.
|
||||
|
||||
gem5 user mode:
|
||||
|
||||
....
|
||||
./build-buildroot --config 'BR2_PACKAGE_DHRYSTONE=y' --arch arm
|
||||
make \
|
||||
-B \
|
||||
-C "$(./getvar --arch arm build_dir)/dhrystone-2" \
|
||||
CC="$(./run-toolchain --arch arm --dry gcc)" \
|
||||
CFLAGS=-static \
|
||||
;
|
||||
time \
|
||||
./run \
|
||||
--arch arm \
|
||||
--gem5 \
|
||||
--userland \
|
||||
"$(./getvar --arch arm build_dir)/dhrystone-2/dhrystone" \
|
||||
-- \
|
||||
--options 100000 \
|
||||
;
|
||||
....
|
||||
|
||||
gem5 full system:
|
||||
|
||||
....
|
||||
time \
|
||||
./run \
|
||||
--arch arm \
|
||||
--eval-busybox '/gem5.sh' \
|
||||
--gem5
|
||||
--gem5-readfile 'dhrystone 100000' \
|
||||
;
|
||||
....
|
||||
|
||||
QEMU user mode:
|
||||
|
||||
....
|
||||
time qemu-arm "$(./getvar --arch arm build_dir)/dhrystone-2/dhrystone" 100000000
|
||||
....
|
||||
|
||||
QEMU full system:
|
||||
|
||||
....
|
||||
time \
|
||||
./run \
|
||||
--arch arm \
|
||||
--eval-busybox 'time dhrystone 100000000;/poweroff.out' \
|
||||
;
|
||||
....
|
||||
|
||||
Result on <<p51>> at bad30f513c46c1b0995d3a10c0d9bc2a33dc4fa0:
|
||||
|
||||
* gem5 user: 33 seconds
|
||||
* gem5 full system: 51 seconds
|
||||
* QEMU user: 45 seconds
|
||||
* QEMU full system: 223 seconds
|
||||
|
||||
=== QEMU monitor
|
||||
|
||||
The QEMU monitor is a terminal that allows you to send text commands to the QEMU VM: https://en.wikibooks.org/wiki/QEMU/Monitor
|
||||
@@ -11065,14 +11096,14 @@ or more cleanly out of tree:
|
||||
|
||||
....
|
||||
./build-userland --host --userland-build-id host
|
||||
$(./getvar --userland-build-id host userland_build_dir)/hello.out
|
||||
"$(./getvar --userland-build-id host userland_build_dir)/hello.out"
|
||||
....
|
||||
|
||||
Extra make flags may be passed as:
|
||||
|
||||
....
|
||||
./build-userland --host --userland-build-id host-static --make-args="-B CFLAGS_EXTRA=-static"
|
||||
$(./getvar --userland-build-id host-static userland_build_dir)/hello.out
|
||||
./build-userland --host --userland-build-id host-static --make-args='-B CFLAGS_EXTRA=-static'
|
||||
"$(./getvar --userland-build-id host-static userland_build_dir)/hello.out"
|
||||
....
|
||||
|
||||
This for example would both force a rebuild due to `-B` and link statically due to `CFLAGS_EXTRA=-static`.
|
||||
|
||||
Reference in New Issue
Block a user