# Linux Kernel Module Cheat

#### 24.16.6. gem5 Ruby build

gem5 has two types of memory system:

- the classic memory system, which is used by default, its caches are covered at: gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches

- the Ruby memory system

The Ruby memory system includes the SLICC domain specific language to describe memory systems: http://gem5.org/Ruby SLICC transpiles to C++ auto-generated files under `build/<isa>/mem/ruby/protocol/`.

Ruby seems to have usage outside of gem5, but the naming overload with the Ruby programming language, which also has domain specific languages as a concept, makes it impossible to google anything about it!

Since it is not the default, Ruby is generally less stable that the classic memory model. However, because it allows describing a wide variety of important cache coherence protocols, while the classic system only describes a single protocol, Ruby is very importanonly describes a single protocol, Ruby is a very important feature of gem5.

Ruby support must be enabled at compile time with the `scons PROTOCOL=` flag, which compiles support for the desired memory system type.

Note however that most ISAs already implicitly set `PROTOCOL` via the `build_opts/` directory, e.g. `build_opts/ARM` contains:

    PROTOCOL = 'MOESI_CMP_directory'

and therefore ARM already compiles `MOESI_CMP_directory` by default.

Then, with `fs.py` and `se.py`, you can choose to use either the classic or the ruby system type selected at build time with `PROTOCOL=` at runtime by passing the `--ruby` option:

- if `--ruby` is given, use the ruby memory system that was compiled into gem5. Caches are always present when Ruby is used, since the main goal of Ruby is to specify the cache coherence protocol, and it therefore hardcodes cache hierarchies.

- otherwise, use the classic memory system. Caches may be optional for certain CPU types and are enabled with `--caches`.

Note that the `--ruby` option has some crazy side effects besides enabling Ruby, e.g. it sets the default `--cpu-type` to `TimingSimpleCPU` instead of the otherwise default `AtomicSimpleCPU`. TODO: I have been told that this is because sends the packet atomically,atomic requests do not work with Ruby, only timing.

It is not possible to build more than one Ruby system into a single build, and this is a major pain point for testing Ruby: https://gem5.atlassian.net/browse/GEM5-467

For example, to use a two level MESI cache coherence protocol we can do:

    ./build-gem5 --arch aarch64 --gem5-build-id ruby -- PROTOCOL=MESI_Two_Level
    ./run --arch aarch64 --emulator -gem5 --gem5-build-id ruby -- --ruby

and during build we see a humongous line of type:

    [   SLICC] src/mem/protocol/MESI_Two_Level.slicc -> ARM/mem/protocol/AccessPermission.cc, ARM/mem/protocol/AccessPermission.hh, ...

which shows that dozens of C++ files are being generated from Ruby SLICC.

The relevant Ruby source files live in the source tree under:

    src/mem/protocol/MESI_Two_Level*

We already pass the `SLICC_HTML` flag by default to the build, which generates an HTML summary of each memory protocol under (TODO broken: https://gem5.atlassian.net/browse/GEM5-357):

    xdg-open "$(./getvar --arch aarch64 --gem5-build-id ruby gem5_build_build_dir)/ARM/mem/protocol/html/index.html"

A minimized ruby config which was not merged upstream can be found for study at: https://gem5-review.googlesource.com/c/public/gem5/+/13599/1

One easy way to see that Ruby is being used without understanding it in detail is to enable some logging:

    ./run \
      --arch aarch64 \
      --emulator gem5 \
      --gem5-worktree master \
      --userland userland/arch/aarch64/freestanding/linux/hello.S \
      --static \
      --trace ExecAll,FmtFlag,Ruby,XBar \
      -- \
      --ruby \
    ;
    cat "$(./getvar --arch aarch64 --emulator gem5 trace_txt_file)"

Then:

- when the `--ruby` flag is given, we see a gazillion Ruby related messages prefixed e.g. by `RubyPort:`.

  We also observe from `ExecEnable` lines that instruction timing is not simple anymore, so the memory system must have latencies

- without `--ruby`, we instead see `XBar` (Coherent Crossbar) related messages such as `CoherentXBar:`, which I believe is the more precise name for the memory model that the classic memory system uses: gem5 crossbar interconnect.

Certain features may not work in Ruby. For example, gem5 checkpoint creation is only possible in Ruby protocols that support flush, which is the case for `PROTOCOL=MOESI_hammer` but not `PROTOCOL=MESI_Three_Level`: https://www.mail-archive.com/gem5-users@gem5.org/msg17418.html

Tested in gem5 d7d9bc240615625141cd6feddbadd392457e49eb.

- gem5 Ruby MI_example protocol

- gem5 crossbar interconnect