GNOME Bugzilla – Bug 660647
Use a search path for overrides
Last modified: 2012-09-29 10:00:25 UTC
Overrides for GI modules must be put into gi._overridesdir directory to be found. Why it is a problem: For system installations of pygobject (i.e. to prefix such as /usr) this means packages compiled by a user cannot install overrides. At least unless he also installs pygobject from source together with anything else that has GI overrides (in order to get everything to one directory). Even in cases when the pygobject override directory is writeable it may be preferred to install overrides of an additional package elsewhere, e.g. to avoid installing anything outside its prefix. How it happens: The overrides for a GI module are loaded with overrides_modules = __import__('gi.overrides', fromlist=[self._namespace]) in gi/module.py. This employs the standard Python package lookup mechanism which means if it finds package gi.overrides somewhere it concludes this is the directory where gi.overrides resides and hence it only looks for the subpackages in this directory. It should search sys.path (possibly some other path set but using the python search path seems reasonable) for each subpackage. This might be fixable by adding another module finder/loader, this time for the overrides.
A simple fix would be something like import gi.overrides, sys, os.path for pythondir in sys.path: odir = os.path.join(pythondir, 'gi', 'overrides') if odir not in gi.overrides.__path__: gi.overrides.__path__.append(os.path.join(pythondir, 'gi', 'overrides')) in gi/__init__.py. Of course, this bases the search path on sys.path value at the time the module is imported so it is probably not good enough.
gobject-introspection uses XDG_DATA_DIRS to locate it's gir's: http://git.gnome.org/browse/gobject-introspection/tree/girepository/girparser.c#n26 Note the g_get_system_data_dirs call. So maybe it would make sense if PyGObject does the same. If carefully designed, it would allow a module to install both it's gir/typelibs and it's Python overrides into some prefix and we'd be able to get things working with a single "user interface": the XDG_DATA_DIRS environment variable. I'm having for example jhbuild in mind here, but haven't thought it through completely. Just an idea...
I must say I fail to see what relevance XDG_DATA_DIRS has to location of Python modules, as opposed to PYTHONPATH. First, XDG_DATA_DIR it is completely alien to Python. Second, the override modules can be shared libraries (at present I am struggling to get overrides written in C working but in principle it should work). This means the location is libdir-like, not datadir-like. An analogy could perhaps be found with GI_TYPELIB_PATH which is again a libdir-like path (and separated from the XDG stuff). Since sys.path (influenced by PYTHONPATH) is what says where Python looks for modules, if you have an override module foobar in gi/overrides subdirectory somewhere in sys.path in my opinion it should be simply found as gi.overrides.foobar. This permits installation consistent with other Python modules the package may install (and essentially eliminates the need for an overridesdir variable which might be considered a bonus). While additional search paths can be imagined in my opinion this one would be sufficient. That said, we do not have the actual mechanism yet so all this is a bit premature...
Kind of agree with you both, while gi uses XDG for girs, it uses GI_TYPELIB_PATH for typelibs, the former are more data-like, while the latter are more lib-like.
Created attachment 198256 [details] [review] hackish patch A hackish patch that simply reconstructs gi.overrides.__path__ based on sys.path every time before an import from gi.overrides is attempted. This works nicely for me but the approach is somewhat aggressive.
Created attachment 198257 [details] [review] hackish patch Wrong patch...
I am having problems with something tightly related to this as well. I want to add unit tests for my Python bindings, but I can't figure out how to get my test suite to pick up the overrides .py file from the source tree. Hacking gi._overridesdir or gi.overrides.__path__ does not seem to be enough (or something I can rely on at all).
This ended up being fixed in another ticket. You can now use standard python path mechanisms to pickup additional override dirs, as in setting PYTHONPATH or making sure the parent dir to your custom gi/overrides dir is in sys.path. However, you must add pkgutil magic to your __init__.py files: from pkgutil import extend_path __path__ = extend_path(__path__, __name__) ..../custom_folder/ gi/ __ini__.py <-- pkgutil stuff in here overrides/ __init__.py <-- pkgutil stuff in here MyOverride.py *** This bug has been marked as a duplicate of bug 680913 ***