gem5 m5.objects module

README.adoc

@@ -13418,6 +13418,147 @@ and that is where the main event loop, `doSimLoop`, gets called and starts kicki
 
 Tested at gem5 b4879ae5b0b6644e6836b0881e4da05c64a6550d.
 
+===== gem5 `m5.objects` module
+
+All `SimObjects` seem to be automatically added to the `m5.objects` namespace, and this is done in a very convoluted way, let's try to understand a bit:
+
+....
+src/python/m5/objects/__init__.py
+....
+
+contains:
+
+....
+modules = __loader__.modules
+
+for module in modules.keys():
+    if module.startswith('m5.objects.'):
+        exec("from %s import *" % module)
+....
+
+And from <<debug-gem5-python-scripts,IPDB>> we see that this appears to loop over every object string of type `m5.objects.modulename`.
+
+This `__init__` gets called from `src/python/importer.py` at the `exec`:
+
+....
+class CodeImporter(object):
+    def load_module(self, fullname):
+            override = os.environ.get('M5_OVERRIDE_PY_SOURCE', 'false').lower()
+            if override in ('true', 'yes') and  os.path.exists(abspath):
+                src = open(abspath, 'r').read()
+                code = compile(src, abspath, 'exec')
+
+            if os.path.basename(srcfile) == '__init__.py':
+                mod.__path__ = fullname.split('.')
+                mod.__package__ = fullname
+            else:
+                mod.__package__ = fullname.rpartition('.')[0]
+            mod.__file__ = srcfile
+
+            exec(code, mod.__dict__)
+
+import sys
+importer = CodeImporter()
+add_module = importer.add_module
+sys.meta_path.append(importer)
+....
+
+Here as a bonus here we also see how <<m5-override-py-source,`M5_OVERRIDE_PY_SOURCE`>> works.
+
+In `src/SConscript` we see that `SimObject` is just a `PySource` with module equals to `m5.objects`:
+
+....
+class SimObject(PySource):
+    def __init__(self, source, tags=None, add_tags=None):
+        '''Specify the source file and any tags (automatically in
+        the m5.objects package)'''
+        super(SimObject, self).__init__('m5.objects', source, tags, add_tags)
+....
+
+The `add_module` method seems to be doing the magic and is called from `src/sim/init.cc`:
+
+....
+bool
+EmbeddedPython::addModule() const
+{
+    PyObject *code = getCode();
+    PyObject *result = PyObject_CallMethod(importerModule, PyCC("add_module"),
+....
+
+which is called from:
+
+....
+int
+EmbeddedPython::initAll()
+{
+    // Load the importer module
+    PyObject *code = importer->getCode();
+    importerModule = PyImport_ExecCodeModule(PyCC("importer"), code);
+    if (!importerModule) {
+        PyErr_Print();
+        return 1;
+    }
+
+    // Load the rest of the embedded python files into the embedded
+    // python importer
+    list<EmbeddedPython *>::iterator i = getList().begin();
+    list<EmbeddedPython *>::iterator end = getList().end();
+    for (; i != end; ++i)
+        if (!(*i)->addModule())
+....
+
+and `getList` comes from:
+
+....
+EmbeddedPython::EmbeddedPython(const char *filename, const char *abspath,
+    const char *modpath, const unsigned char *code, int zlen, int len)
+    : filename(filename), abspath(abspath), modpath(modpath), code(code),
+      zlen(zlen), len(len)
+{
+    // if we've added the importer keep track of it because we need it
+    // to bootstrap.
+    if (string(modpath) == string("importer"))
+        importer = this;
+    else
+        getList().push_back(this);
+}
+
+list<EmbeddedPython *> &
+EmbeddedPython::getList()
+{
+    static list<EmbeddedPython *> the_list;
+    return the_list;
+}
+....
+
+and the constructor in turn gets called from per `SimObject` autogenerated files such as e.g. `dev/storage/Ide.py.cc` for `src/dev/storage/Ide.py`:
+
+....
+EmbeddedPython embedded_m5_objects_Ide(
+    "m5/objects/Ide.py",
+    "/home/ciro/bak/git/linux-kernel-module-cheat/data/gem5/master4/src/dev/storage/Ide.py",
+    "m5.objects.Ide",
+    data_m5_objects_Ide,
+    947,
+    2099);
+
+} // anonymous namespace
+....
+
+which get autogenerated at `src/SConscript`:
+
+....
+def embedPyFile(target, source, env):
+
+for source in PySource.all:
+    base_py_env.Command(source.cpp, [ py_marshal, source.tnode ],
+                        MakeAction(embedPyFile, Transform("EMBED PY")))
+....
+
+where the `PySource.all` thing as you might expect is a static list of all `PySource` source files as they get updated in the constructor.
+
+Tested in gem5 d9cb548d83fa81858599807f54b52e5be35a6b03.
+
 ==== gem5 event queue
 
 gem5 is an event based simulator, and as such the event queue is of of the crucial elements in the system.
@@ -16248,6 +16389,17 @@ Bibliography: https://www.mail-archive.com/gem5-users@gem5.org/msg16989.html
 
 ==== gem5 build system
 
+[[m5-override-py-source]]
+===== M5_OVERRIDE_PY_SOURCE
+
+https://stackoverflow.com/questions/52312070/how-to-modify-a-file-under-src-python-and-run-it-without-rebuilding-in-gem5
+
+Running gem5 with the `M5_OVERRIDE_PY_SOURCE=true` environment variable allows you to modify a file under src/python and run it without rebuilding in gem5?
+
+We set this environment variable by default in our link:run[] script.
+
+How `M5_OVERRID_PY_SOURCE` works is shown at: <<gem5-m5-objects-module>>.
+
 ===== gem5 build broken on recent compiler version
 
 gem5 moves a bit slowly, and if your host compiler is very new, the gem5 build might be broken for it, e.g. this was the case for Ubuntu 19.10 with GCC 9 and gem5 62d75e7105fe172eb906d4f80f360ff8591d4178 from Dec 2019.

run

@@ -518,7 +518,7 @@ Extra options to append at the end of the emulator command line.
             gem5_exe_args = self.sh.shlex_split(self.env['gem5_exe_args'])
             if do_trace:
                 gem5_exe_args.extend(['--debug-flags', trace_type, LF])
-            # https://stackoverflow.com/questions/52312070/how-to-modify-a-file-under-src-python-and-run-it-without-rebuilding-in-gem5/52312071#52312071
+            # https://cirosantilli.com/linux-kernel-module-cheat#m5-override-py-source
             extra_env['M5_OVERRIDE_PY_SOURCE'] = 'true'
             if self.env['trace_stdout']:
                 debug_file = 'cout'