pci: review setpci and devmem manipulations

Make if you do this then this happens relations clearer.

README.adoc

@@ -2760,7 +2760,7 @@ Now try the following:
 * press a mouse key, and release it after a few seconds
 * move the mouse around
 
-Outcome: dmesg shows which IRQ was fired for each action throug messages of type:
+Outcome: dmesg shows which IRQ was fired for each action through messages of type:
 
 ....
 handler irq = 1 dev = 250
@@ -2769,7 +2769,7 @@ handler irq = 1 dev = 250
 `dev` is the character device for the module and never changes, as can be confirmed by:
 
 ....
-cat /proc/devices | grep lkmc_irq
+grep lkmc_irq /proc/devices
 ....
 
 The IRQs that we observe are:
@@ -3776,6 +3776,25 @@ PCI driver for our minimal `pci_min.c` QEMU fork device:
 insmod /pci_min.ko
 ....
 
+Outcome:
+
+....
+<4>[   10.608241] pci_min: loading out-of-tree module taints kernel.
+<6>[   10.609935] probe
+<6>[   10.651881] dev->irq = 11
+lkmc_pci_min mmio_write addr = 0 val = 12345678 size = 4
+<6>[   10.668515] irq_handler irq = 11 dev = 251
+lkmc_pci_min mmio_write addr = 4 val = 0 size = 4
+....
+
+What happened:
+
+* right at probe time, we write to a register
+* our hardware model is coded such that it generates an interrupt when written to
+* the Linux kernel interrupt handler write to another register, which tells the hardware to stop sending interrupts
+
+Kernel messages and printks from inside QEMU are shown all together, to see that more clearly, run in <<graphic-mode>> instead.
+
 Source:
 
 * Kernel module: link:kernel_module/pci_min.c[].
@@ -3798,12 +3817,12 @@ Small upstream educational PCI device:
 /pci.sh
 ....
 
-Source:
+This tests a lot of features of the edu device, to understand the results, compare the inputs with the documentation of the hardware: https://github.com/qemu/qemu/blob/v2.12.0/docs/specs/edu.txt
+
+Sources:
 
 * kernel module: link:kernel_module/pci.c[]
-* QEMU device:
-** source: https://github.com/qemu/qemu/blob/v2.12.0/hw/misc/edu.c
-** documentation: https://github.com/qemu/qemu/blob/v2.12.0/docs/specs/edu.txt
+* QEMU device: https://github.com/qemu/qemu/blob/v2.12.0/hw/misc/edu.c
 * test script: link:rootfs_overlay/pci.sh[]
 
 Works because we add to our default QEMU CLI:
@@ -3824,31 +3843,41 @@ TODO exercise DMA on the kernel module. The `edu` hardware model has that featur
 * https://stackoverflow.com/questions/32592734/are-there-any-dma-driver-example-pcie-and-fpga/44716747#44716747
 * https://stackoverflow.com/questions/17913679/how-to-instantiate-and-use-a-dma-driver-linux-module
 
-===== setpci
+===== Manipulate PCI registers directly
 
-There are two versions of `setpci` and `lspci`:
+In this section we will try to interact with PCI devices directly from userland without kernel modules.
 
-* a simple one from BusyBox
-* a more complete one from link:https://github.com/pciutils/pciutils[pciutils] which Buildroot has a package for. This is the one we enable by default.
-
-`setpci` can read and write to PCI configuration registers.
-
-Read is possible from Linux with:
+First identify the PCI device with:
 
 ....
-hexdump /sys/bus/pci/devices/0000:00:05.0/config
+lspci
 ....
 
-and `/dev/mem` can of course do both reads and writes, but `setpci` provides nice human readable register names, e.g.:
+In our case for example, we see:
+
+....
+00:06.0 Unclassified device [00ff]: Device 1234:11e8 (rev 10)
+00:07.0 Unclassified device [00ff]: Device 1234:11e9
+....
+
+which we identify as being `edu` and `pci_min` respectively by the magic numbers: `1234:11e?`
+
+Read the configuration registers as binary:
+
+....
+hexdump /sys/bus/pci/devices/0000:00:06.0/config
+....
+
+Get nice human readable names and offsets of the registers and some enums:
 
 ....
 setpci --dumpregs
 ....
 
-then and then get the values with either bus or device id:
+Get the values of a given config register from its human readable name, either with either bus or device id:
 
 ....
-setpci -s 0000:00:05.0 BASE_ADDRESS_0
+setpci -s 0000:00:06.0 BASE_ADDRESS_0
 setpci -d 1234:11e9 BASE_ADDRESS_0
 ....
 
@@ -3858,13 +3887,49 @@ Note however that `BASE_ADDRESS_0` also appears when you do:
 lspci -v
 ....
 
+as:
+
+....
+Memory at feb54000
+....
+
 Then you can try messing with that address with:
 
 ....
-devmem 0xfeb52000 w 0x12345678
+devmem 0xfeb54000 w 0x12345678
 ....
 
-which for our <<pci_min>> device fires interrupts.
+which writes to the first register of our <<pci_min>> device.
+
+The device then fires an interrupt at irq 11, which is unhandled, which leads the kernel to say you are a bad boy:
+
+....
+lkmc_pci_min mmio_write addr = 0 val = 12345678 size = 4
+<5>[ 1064.042435] random: crng init done
+<3>[ 1065.567742] irq 11: nobody cared (try booting with the "irqpoll" option)
+....
+
+followed by a trace.
+
+Next, also try using our <<irq-ko>> IRQ monitoring module before triggering the interrupt:
+
+....
+insmod /irq.ko
+devmem 0xfeb54000 w 0x12345678
+....
+
+Our kernel module handles the interrupt, but does not acknowledge it like our proper <<pci_min>> kernel module, and so it keeps firing, which leads to infinitely many messages being printed:
+
+....
+handler irq = 11 dev = 251
+....
+
+===== pciutils
+
+There are two versions of `setpci` and `lspci`:
+
+* a simple one from BusyBox
+* a more complete one from link:https://github.com/pciutils/pciutils[pciutils] which Buildroot has a package for, and is the default on Ubuntu 18.04 host. This is the one we enable by default.
 
 ===== Introduction to PCI