# Linux Kernel Module Cheat

### 3.9. GDB step debug multicore userland

For a more minimal baremetal multicore setup, see: Section 33.10.3, “ARM baremetal multicore”.

We can set and get which cores the Linux kernel allows a program to run on with `sched_getaffinity` and `sched_setaffinity`:

    ./run --cpus 2 --eval-after './linux/sched_getaffinity.out'

Source: userland/linux/sched_getaffinity.c

Sample output:

    sched_getaffinity = 1 1
    sched_getcpu = 1
    sched_getaffinity = 1 0
    sched_getcpu = 0

Which shows us that:

- initially:

  - all 2 cores were enabled as shown by `sched_getaffinity = 1 1`

  - the process was randomly assigned to run on core 1 (the second one) as shown by `sched_getcpu = 1`. If we run this several times, it will also run on core 0 sometimes.

- then we restrict the affinity to just core 0, and we see that the program was actually moved to core 0

The number of cores is modified as explained at: Section 24.3.1, “Number of cores”

`taskset` from the util-linux package sets the initial core affinity of a program:

    ./build-buildroot \
      --config 'BR2_PACKAGE_UTIL_LINUX=y' \
      --config 'BR2_PACKAGE_UTIL_LINUX_SCHEDUTILS=y' \
    ;
    ./run --eval-after 'taskset -c 1,1 ./linux/sched_getaffinity.out'

output:

    sched_getaffinity = 0 1
    sched_getcpu = 1
    sched_getaffinity = 1 0
    sched_getcpu = 0

so we see that the affinity was restricted to the second core from the start.

Let’s do a QEMU observation to justify this example being in the repository with userland breakpoints.

We will run our `./linux/sched_getaffinity.out` infinitely many times, on core 0 and core 1 alternatively:

    ./run \
      --cpus 2 \
      --eval-after 'i=0; while true; do taskset -c $i,$i ./linux/sched_getaffinity.out; i=$((! $i)); done' \
      --gdb-wait \
    ;

on another shell:

    ./run-gdb --userland "$(./getvar userland_build_dir)/linux/sched_getaffinity.out" main

Then, inside GDB:

    (gdb) info threads
      Id   Target Id         Frame
    * 1    Thread 1 (CPU#0 [running]) main () at sched_getaffinity.c:30
      2    Thread 2 (CPU#1 [halted ]) native_safe_halt () at ./arch/x86/include/asm/irqflags.h:55
    (gdb) c
    (gdb) info threads
      Id   Target Id         Frame
      1    Thread 1 (CPU#0 [halted ]) native_safe_halt () at ./arch/x86/include/asm/irqflags.h:55
    * 2    Thread 2 (CPU#1 [running]) main () at sched_getaffinity.c:30
    (gdb) c

and we observe that `info threads` shows the actual correct core on which the process was restricted to run by `taskset`!

We should also try it out with kernel modules: https://stackoverflow.com/questions/28347876/set-cpu-affinity-on-a-loadable-linux-kernel-module

TODO we then tried:

    ./run --cpus 2 --eval-after './linux/sched_getaffinity_threads.out'

and:

    ./run-gdb --userland "$(./getvar userland_build_dir)/linux/sched_getaffinity_threads.out"

to switch between two simultaneous live threads with different affinities, it just didn’t break on our threads:

    b main_thread_0

Note that secondary cores in gem5 are kind of broken however: gem5 GDB step debug secondary cores.

Bibliography:

- https://stackoverflow.com/questions/10490756/how-to-use-sched-getaffinity-and-sched-setaffinity-in-linux-from-c/50117787#50117787

  - https://stackoverflow.com/questions/663958/how-to-control-which-core-a-process-runs-on/50210009#50210009

  - https://stackoverflow.com/questions/280909/cpu-affinity/54478296#54478296

  - https://unix.stackexchange.com/questions/73/how-can-i-set-the-processor-affinity-of-a-process-on-linux/441098#441098 (summary only)

- https://stackoverflow.com/questions/42800801/how-to-use-gdb-to-debug-qemu-with-smp-symmetric-multiple-processors