This commit is a large squash, the full development branch is available at:
https://github.com/cirosantilli/linux-kernel-module-cheat/tree/asm
This notably means a refactor of the userland build and testing, to support:
- improved assembly infrastructure unified across arm and x86
- native in-tree build and test helpers
- parallel building and testing, which implies thread_pool.py
- selection of what to build and test from the CLI
- path_properties.py to indicate how to build and run different examples
- in full system, move all userland stuff into /lkmc
- prefix everything that we defined across files with LKMC
- --gdb uber convenient helper
- remove import imp which was deprecated
Full commit messages from the branch follow:
1:
userland: add assembly support
Move arm assembly cheat here, and start some work on x86 cheat as well.
2:
document userland asm syscall interfaces
3:
userland assembly: structure readme
4:
x86 fail works
5:
asm: more links
6:
userland: add ported to all archs
7:
move all our stuff into /lkmc in guest
Motivation: userland is getting several new subdirectories, it would be
too insane to just dump all of that in the guest root filesystem.
To alleviate the cd pain, .profile puts user inside /lkmc by default.
8:
start the big userland migration
9:
migrate all
10:
bak
11:
build-userland-in-tree is now a Python command
./build calls it, we did this to allow --download-dependencies to work
perfectly.
12:
rename include to lkmc
13:
mtops.h is perfect now
14:
userland: make build perfect
15:
preparing test_user_mode, need to generalize stuff as usual
16:
asm: prefix every linux specific with linux/
17:
userland: maybe it really works
18:
userland: fix kernel version to work on older ubuntu
Expose --kernel-version to allow customization.
Update LTP info.
19:
userland: build really truly working now
userland test: start work, in a working state, but no features
20:
test-user-mode: make perfect like build-userland
Multithreading and target selection.
21:
userland: get a bit closer to perfection
22:
thread_pool: support passing thread IDs
Then use that to fix gem5 error log read race.
23:
userland: native testing
24:
userland: path properties getting nice!
25:
userland: move posix/environ from cpp-cheat
26:
gem5: --debug-flags without =, looks nicer whenever it can be done
27:
run: rename --wait-gdb in --gdb-wait, --gdb prefix might become a thing
28:
run: create --tmux-program gdb to open gem5 GDB
29:
run: create the uber convenient --gdb option
30:
userland: move getchar from cpp-cheat
31:
prebuilt: kernel boot aarch64 does not work on Ubuntu 16.04
32:
userland: x86_64 linux hello world make PIE
33:
userland: try to make userland executable selection saner
Only allow existing files to be built, stop extension expansion madness.
cli_function: get_cli print booleans properly, was printing without --no-
for negations.
34:
userland: only link to lkmc.o if needed
35:
path_properties: make data very compact with only tuples and dicts
Spend 2 hours of my life thinking about low value tree walks ;-)
36:
userland: move more userland/arch/ logic into property tree
37:
userland: make libs work
Working for build, but now test-user-mode-in-tree is not using --in-tree,
TODO fix later on.
38:
userland: make libs really work
39:
userland: document path_properties
40:
userland: classify linux
41:
waste your life
42:
common: fix absolute path runs
--gdb: allow running from arbitrary directory
43:
baremetal: arm allow using floating point instructions
44:
baremetal: stat preparing to make perfect like userland/
45:
run: fix image check logic accounting for userland
Was failing if I try to run userland (with abspath) when out/
directory is not present.
46:
cli-function: raise if the config file is given and does not exist
47:
common: define missing 'ld' variable, this broke m5 build
48:
rum: --qemu-which host now works for user mode as well as system
Don't fall back on host QEMU automatically, too much insanity.
49:
userland: refix silly mistakes
50:
userland: use path_properties flags for all builds, including lkmc. and userland/arch/main.c
Without this in particular, --gdb fails on assembly because main.c
was not being built with -ggdb3.
51:
userland: start refactor to show failing values on failure!
aarch64 basically done, but missing:
- other archs
- maybe convert main.c into C++ to use templates?
- full review of ASSERT_EQ calling convention issues not seen by tests
by chance
- documentation
52:
readme: releases are more stable...
53:
submodules: sort gitmodules
54:
test-baremetal: same interface as test-user-mode
In particular, runs tests in parallel, and allows selecting given tests
55:
baremetal: allow arbitrary exit status with the magic string
test-baremetal: fix missing setting x0 return value
Examples were just returning on ret without setting x0, which led to
failures... those were not noticed because of how broken the testing system
was ;-)
56:
baremetal: ah, actually nope, it didn't work :-(
Workaround for now. Works on asserts, but not on exit 1.
Some other day, maybe.
https://github.com/cirosantilli/linux-kernel-module-cheat/issues/59
57:
panic on panic: improve behaviour description
58:
baremetal: get exit status working with on_exit :-)
59:
baremetal: implement C assert
60:
test-baremetal: remove commented out exit status workaround
61:
test-user-mode: handle exit status for signals. Fix#61.
62:
aarch64: fix ASSERT_EQ_REG tests on gem5
Was doing an 8-byte aligned store, which gem5 dislikes.
But the ARMARM says bad things may happen there, notably a signal:
"D1.8.2 SP alignment checking" so gem5 is not really too wrong,
QEMU just happens to work by chance.
63:
userland assembly: build empty.S and fail.S to toplevel and run fail.S with path_properties exit_status
They were just duplicating stuff needlessly while we don't support non-native in-tree builds,
which leads to executable conflicts for C file anyways.
64:
gem5: use a single build tree for all build types
gem5 already has different object names for each build type it seems, so
let's just make sure that works and save some disk space.
65:
userland x86_64: ASSERT_EQ show actual and expected values
66:
assert_fail.c: add to readme index
67:
userland x86_64: implement ASSERT_MEMCMP
68:
userland x86_64: allow ASSERT_EQ to take just about anything
69:
gas data sizes
70:
gas_data_sizes.S: make PIE for all ISAs
71:
x86: paddq
72:
x86 paddq: test entire family
73:
Get rid of imp, started giving deprecation warning every time in Python 3.7 in Ubuntu 19.04.
Please python stop torturing me with refactors.
Make ./run -u blow up if executable not found, otherwise I go crazy.
Get ./test-gdb back to life after the ./run relative path refactor, forgot to test this.
74:
fix run-toolchain, qemu-monitor, trace-boot, trace2line, bisect-linux-boot-gem5. Fixes part of #63
I'm sad no one reported qemu-monitor break, that one is kind of important.
count.out arguments broke it as an init program, since the kernel adds trash
parameters to every init.
Is anyone using this repo, I wonder? Keep pushing, keep pushing.
One day it gets good enough, and the whole world will see.
75:
x86 assembly: addpd
76:
Fix import_path circular dependency by splitting it out.
Use import thread_pool instead from, from is evil.
Fix poweroff.out path for ./trace-boot.
77:
run: rename cryptic tmu to tmux-split, ./run is good now so I never use it anymore explicitly
78:
assembly SIMD add: make uniform for all ISAs, mark as entry point to learning SIMD
79:
start moving arm-assembly-cheat readme in here
80:
arm assembly: move some more in
81:
move more arm in
82:
userland: attempt to fix all assembly example links to README
83:
assembly: improve organization of simd add
84:
ld2 move in
85:
Make userland / assembly getting started more uniform / visible
Forward --gcc-which to ./run --tmux.
Use gdb-multiarch for --gcc-which host.
86:
userland: disable PIE explicitly on command line for all executables
87:
userland: make userland content a better landing page
88:
build: check git version from --version and degrade gracefully
89:
build: make --dry-run work again on all
90:
import_path: importlib explicit for Ubuntu 16.04
91:
make all submodules point to my forks
git servers are insane, submodule implementation is crap, what can you do
92:
build: log warning on git too old for --update
93:
build-linux: do olddefconfig even if no fragments
In particular, gem5 kernel 4.15 needs it
94:
userland content: improve a bit landing page for cpp-cheat
Unsquashed version at v2-rc-unsquashed, but that cannot be merged as it
breaks bisects at several points. All bugs will not bisect to this
humongous change.
It all started with a conversion of the Bash scripts to Python, mainly
because I couldn't stand not being able to properly use --options for
run which has a million options.
Then since that required a full testing, I decided to do all the
refactorings that I had in mind at once, and so I did and it became
v2-rc.
This is the largest patch I have ever done! OMG a few weeks of extra time.
I'm never writing a Bash script for anything that starts getting big again.
Some of the features are:
* separate build-qemu and build-gem5 commands
* common: convert scripts to python. Add --option for everything
* rename build to build-buildroot now that we are splitting all the build
commands, Linux kernel to follow
* move all git submodules to submodules/ and all buildroot packages to
packages/
* refactor the out/ structure. Keep projects on toplevel, because guest
projects separate archs and host ones don't, making a toplevel arch wrong
* do-release: rename to just release
https://stackoverflow.com/questions/16174992/cant-get-argparse-to-read-quoted-string-with-dashes-in-it
* run: add --terminal and explain gem5 pdb
* just track the lvimrc
* store CLI kernel config fragment inside buildlroot to avoid conflicts
* gem5: document m5 initparam
* readme: make a bunch of things awesomer
* readme: fix broken refs
* parsec-benchmark: update to 75d55ac446a43c47efb1044844a108c6c330184c
Could not fetch otherwise.
* gem5: M5_OVERRIDE_PY_SOURCE
Fix gem5/build wrong path for dd swapfile. Put it under out/ while we are at it
now that we have out of tree builds.
Allow seq_id larger than 7: broke previously because 08 is not valid octal.
Use timeout --foreground to allow killing scons
Don't upload results by default, too insane.
This is cleaner as it removes yet another .gitignore rule and allows
for one command per arch.
This was prompted by the recent Docker patch creating files as root,
which then breaks future host runs.
Also separate all arguments by newlines. This:
- makes it much easier to minimize when reporting QEMU bugs
- shows the full command on the screen despite line wrapping being turned off
by one of the annoying emulators, to be investigated.
The commands are not yet pristine, because:
- there are some leading spaces
- options not sorted
but it is already a huge improvement, and those operations are now be easy from Vim.
The general strategy is:
- end every variable containing a command with a newline
- when expanding such variables, don't add an extra newline
Most of it was present inside buildroot/output.* and the rest scattered
on top level.
This came about for the n-th time when we were reviewing QEMU trace file
locations.
On one hand, it would be cool to have per arch traces.
This made buildroot/output.${arch}~/ a natural choice.
But on the other, those traces have nothing to do with Buildroot,
and could potentially interfere with Buildroot build files.
It also feels nicer to have buildroot/ pristine source code only,
and keep all output under a single directory out/
Otherwise it was getting messy to distinguish between trace-events
and trace-<pid> in the .gitignore.
Now only the latest trace will be kept, with a fixed name.
This lead to a re-factoring of count boot instructions, since trace2line
relies on the same tracing mechanism.
Also, now that we have common factored, I've decided to put the traces on
the out dir, to concentrate all outputs in a single place, and allow
easy switching between archs.
Don't pass -e on checkpoint restore.
Add benchmarks to how much GEM5 is slower than QEMU.
Rename Kernel boot command line arguments to match kernel docs name.
Document how to pass extra options to GEM5.
Start listing interesting benchmarks to run on GEM5.
Add an openmp hello world.
Semi automated conversion, in part because Pandoc is kind of buggy for adoc:
Some of the commands were along the lines:
for f in *.md; do pandoc --atx-headers --base-header-level=3 -o ${f%.md}.adoc --wrap=none $f; done
sed -Ei '/\[\[.*\]\]/d' *.adoc
while read -r f; do cat $f; echo; done <f >g
