From 922795cbc4ead88c59cc94b3e5dc3d454a6002c4 Mon Sep 17 00:00:00 2001 From: Ciro Santilli Date: Tue, 22 May 2018 17:29:50 +0100 Subject: [PATCH] pci: review setpci and devmem manipulations Make if you do this then this happens relations clearer. --- README.adoc | 105 ++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 85 insertions(+), 20 deletions(-) diff --git a/README.adoc b/README.adoc index d741d53..d5f6369 100644 --- a/README.adoc +++ b/README.adoc @@ -2760,7 +2760,7 @@ Now try the following: * press a mouse key, and release it after a few seconds * move the mouse around -Outcome: dmesg shows which IRQ was fired for each action throug messages of type: +Outcome: dmesg shows which IRQ was fired for each action through messages of type: .... handler irq = 1 dev = 250 @@ -2769,7 +2769,7 @@ handler irq = 1 dev = 250 `dev` is the character device for the module and never changes, as can be confirmed by: .... -cat /proc/devices | grep lkmc_irq +grep lkmc_irq /proc/devices .... The IRQs that we observe are: @@ -3776,6 +3776,25 @@ PCI driver for our minimal `pci_min.c` QEMU fork device: insmod /pci_min.ko .... +Outcome: + +.... +<4>[ 10.608241] pci_min: loading out-of-tree module taints kernel. +<6>[ 10.609935] probe +<6>[ 10.651881] dev->irq = 11 +lkmc_pci_min mmio_write addr = 0 val = 12345678 size = 4 +<6>[ 10.668515] irq_handler irq = 11 dev = 251 +lkmc_pci_min mmio_write addr = 4 val = 0 size = 4 +.... + +What happened: + +* right at probe time, we write to a register +* our hardware model is coded such that it generates an interrupt when written to +* the Linux kernel interrupt handler write to another register, which tells the hardware to stop sending interrupts + +Kernel messages and printks from inside QEMU are shown all together, to see that more clearly, run in <> instead. + Source: * Kernel module: link:kernel_module/pci_min.c[]. @@ -3798,12 +3817,12 @@ Small upstream educational PCI device: /pci.sh .... -Source: +This tests a lot of features of the edu device, to understand the results, compare the inputs with the documentation of the hardware: https://github.com/qemu/qemu/blob/v2.12.0/docs/specs/edu.txt + +Sources: * kernel module: link:kernel_module/pci.c[] -* QEMU device: -** source: https://github.com/qemu/qemu/blob/v2.12.0/hw/misc/edu.c -** documentation: https://github.com/qemu/qemu/blob/v2.12.0/docs/specs/edu.txt +* QEMU device: https://github.com/qemu/qemu/blob/v2.12.0/hw/misc/edu.c * test script: link:rootfs_overlay/pci.sh[] Works because we add to our default QEMU CLI: @@ -3824,31 +3843,41 @@ TODO exercise DMA on the kernel module. The `edu` hardware model has that featur * https://stackoverflow.com/questions/32592734/are-there-any-dma-driver-example-pcie-and-fpga/44716747#44716747 * https://stackoverflow.com/questions/17913679/how-to-instantiate-and-use-a-dma-driver-linux-module -===== setpci +===== Manipulate PCI registers directly -There are two versions of `setpci` and `lspci`: +In this section we will try to interact with PCI devices directly from userland without kernel modules. -* a simple one from BusyBox -* a more complete one from link:https://github.com/pciutils/pciutils[pciutils] which Buildroot has a package for. This is the one we enable by default. - -`setpci` can read and write to PCI configuration registers. - -Read is possible from Linux with: +First identify the PCI device with: .... -hexdump /sys/bus/pci/devices/0000:00:05.0/config +lspci .... -and `/dev/mem` can of course do both reads and writes, but `setpci` provides nice human readable register names, e.g.: +In our case for example, we see: + +.... +00:06.0 Unclassified device [00ff]: Device 1234:11e8 (rev 10) +00:07.0 Unclassified device [00ff]: Device 1234:11e9 +.... + +which we identify as being `edu` and `pci_min` respectively by the magic numbers: `1234:11e?` + +Read the configuration registers as binary: + +.... +hexdump /sys/bus/pci/devices/0000:00:06.0/config +.... + +Get nice human readable names and offsets of the registers and some enums: .... setpci --dumpregs .... -then and then get the values with either bus or device id: +Get the values of a given config register from its human readable name, either with either bus or device id: .... -setpci -s 0000:00:05.0 BASE_ADDRESS_0 +setpci -s 0000:00:06.0 BASE_ADDRESS_0 setpci -d 1234:11e9 BASE_ADDRESS_0 .... @@ -3858,13 +3887,49 @@ Note however that `BASE_ADDRESS_0` also appears when you do: lspci -v .... +as: + +.... +Memory at feb54000 +.... + Then you can try messing with that address with: .... -devmem 0xfeb52000 w 0x12345678 +devmem 0xfeb54000 w 0x12345678 .... -which for our <> device fires interrupts. +which writes to the first register of our <> device. + +The device then fires an interrupt at irq 11, which is unhandled, which leads the kernel to say you are a bad boy: + +.... +lkmc_pci_min mmio_write addr = 0 val = 12345678 size = 4 +<5>[ 1064.042435] random: crng init done +<3>[ 1065.567742] irq 11: nobody cared (try booting with the "irqpoll" option) +.... + +followed by a trace. + +Next, also try using our <> IRQ monitoring module before triggering the interrupt: + +.... +insmod /irq.ko +devmem 0xfeb54000 w 0x12345678 +.... + +Our kernel module handles the interrupt, but does not acknowledge it like our proper <> kernel module, and so it keeps firing, which leads to infinitely many messages being printed: + +.... +handler irq = 11 dev = 251 +.... + +===== pciutils + +There are two versions of `setpci` and `lspci`: + +* a simple one from BusyBox +* a more complete one from link:https://github.com/pciutils/pciutils[pciutils] which Buildroot has a package for, and is the default on Ubuntu 18.04 host. This is the one we enable by default. ===== Introduction to PCI