GNOME Bugzilla – Bug 786785
Commit #fe2a9887a8 breaks gdbus-codegen, cannot find its module sources due to incorrect sys.path
Last modified: 2017-10-31 16:54:23 UTC
Commit #fe2a9887a8 introduced this change at line 30 in 'gdbus/codegen/gdbus-codegen.in' (20/12/2016):- This line:- elif os.name == 'nt': got changed to this:- elif os.path.basename(filedir) == 'bin': According to the description, the intention was to improve support for building with MSVC and MinGW - but in fact, this has broken the build on my system (MSVC). A gdbus-codegen section which was previously being executed during my build process is now getting ignored (which breaks the build at a later stage). If I'm the only person affected by this I'll happily make a local change at my end. But it'd be helpul to know why that line got changed. The previous line and the new line are clearly not equivalent.
Hi John, It seems that running '$(PYTHON) gdbus-codegen' directly in where gdbus-codegen is located is causing the issue. Can you try to run something like '$(PYTHON) <full_path_to_gdbus-codegen>' and see how things go? Seems like that patch introduced a regression on its PYTHONPATH (i.e. sys.path) initialization. With blesings, thank you!
Hi Chun-wei, Previously I was running from my "gio" subfolder so I've now changed it to run from my main (top-level) project folder. As well as providing a full path like you suggested, I also needed to provide the full path to 'dbus-daemon.xml' - in other words:- python "F:/GTK-SOURCES/gnu-windows/src/MB3Glib/gio/gdbus-2.0/codegen/gdbus-codegen" --interface-prefix org. --generate-c-code gdbus-daemon-generated --c-namespace _G F:/GTK-SOURCES/gnu-windows/src/MB3Glib/gio/dbus-daemon.xml However... python gives me the same error as previously - namely:- line 196, in codegen_main gen = codegen.CodeGenerator(all_ifaces, AttributeError: 'module' object has no attribute 'CodeGenerator' I mentioned in my emails that the problem seems to be connected to this line (in 'gdbus-codegen.in'):- sys.path.insert(0, os.path.abspath(path)) After Nirbheek's change (2oth Dec 2016) the above line now places the following folder into my path:- F:\\GTK-SOURCES\\gnu-windows\\src\\MB3Glib\\gio\\gdbus-2.0 and it seems to be the presence of that path folder which causes the problem.
If no-one else can reproduce this problem, I'll find some kludge to fix it locally.
Hi John, Can you try to run it again with the following file layout for the gdbus-codegen scripts (in the way I mentioned in comment 1): $(prefix) |-bin | |-gdbus-codegen | |-share |-glib-2.0 |-codegen |-__init__.py |-codegen.py |-codegen_docbook.py |-codegen_main.py |-config.py |-dbustypes.py |-parser.py |-utils.py This is the layout of how gdbus-codegen is supposed to be when it is properly installed. With blessings, and cheers!
Hi Chub-wei - I needed to make a slight change to your folder structure:- $(prefix) |-bin | |-gdbus-codegen | |-share |-glib-2.0 |-__init__.py |-codegen.py |-codegen_docbook.py |-codegen_main.py |-config.py |-dbustypes.py |-parser.py |-utils.py and I still need to provide the full path to dbus-daemon.xml (which on my system is this):- F:/GTK-SOURCES/gnu-windows/src/MB3Glib/gio/dbus-daemon.xml but python still gives me the same error message. Do I maybe need to move 'dbus-daemon.xml' somewhere else? In case it's helpful, here's what python prints out as being my path:- ['F:\\GTK-SOURCES\\gnu-windows\\share\\glib-2.0', 'F:\\GTK-SOURCES\\gnu-window s\\bin', 'C:\\Windows\\SYSTEM32\\python33.zip', 'C:\\Python33\\DLLs', 'C:\\Pytho n33\\lib', 'C:\\Python33', 'C:\\Python33\\lib\\site-packages']
Hello John, Ah, I see, it seems that you are trying to generate the gdbus source files for building GIO... You need to define these envvars: UNINSTALLED_GLIB_SRCDIR and UNINSTALLED_GLIB_BUILDDIR before you run the scripts. So, for your case, you need to do (and forget what I mentioned about the file layout) set UNINSTALLED_GLIB_SRCDIR=F:\\GTK-SOURCES\\gnu-windows\\src\\MB3Glib set UNINSTALLED_GLIB_BUILDDIR=F:\\GTK-SOURCES\\gnu-windows\\src\\MB3Glib (presuming you are building in your source tree, substitute UNINSTALLED_GLIB_BUILDDIR otherwise) And then run: python "F:/GTK-SOURCES/gnu-windows/src/MB3Glib/gio/gdbus-2.0/codegen/gdbus-codegen" --interface-prefix org. --generate-c-code gdbus-daemon-generated --c-namespace _G F:/GTK-SOURCES/gnu-windows/src/MB3Glib/gio/dbus-daemon.xml Let me know if this went ok. With blessings, and cheers!
This is just weird... setting those 2 environment variables causes my build to fail in exactly the same way - BUT - setting them to anything else causes it to succeed?!? It's almost like I mentioned in comment 2 (where the presence of a specific path folder was causing my build to fail...) Which brings me back to Nirbheek's change from 20th Dec 2016. In commit fe2a9887a8 he changed line 30 in 'gdbus/codegen/gdbus-codegen.in' to look like this:- elif os.path.basename(filedir) == 'bin': when it originally said this:- elif os.name == 'nt': So (in my Windows build) a section that previously got executed now gets ignored. Can't we just combine those two statements somehow?
John Emmas wrote: "It's almost like I mentioned in comment 2 (where the presence of a specific path folder was causing my build to fail...)" Yes - after some further investigation, that's exactly the problem. Near the top of 'gdbus-codegen' we have these lines:- if srcdir is not None: path = os.path.join(srcdir, 'gio', 'gdbus-2.0') elif os.path.basename(filedir) == 'bin': If I set those two environment variables (see comment 6) that section now gets executed. Ultimately, this results in sys.path containing the following folder (on my system):- F:\\GTK-SOURCES\\gnu-windows\\src\\MB3Glib\\gio\\gdbus-2.0 If sys.path contains that specific folder my build invariably fails. If it doesn't contain it, my build succeeds. I can't claim to understand what's happening but that's definitely the problem!
Fan, Chun-wei wrote: "You need to define these envvars: UNINSTALLED_GLIB_SRCDIR and UNINSTALLED_GLIB_BUILDDIR before you run the scripts. So, for your case, you need to do (and forget what I mentioned about the file layout) set UNINSTALLED_GLIB_SRCDIR=F:\\GTK-SOURCES\\gnu-windows\\src\\MB3Glib set UNINSTALLED_GLIB_BUILDDIR=F:\\GTK-SOURCES\\gnu-windows\\src\\MB3Glib" I've realised now that this has probably always been faulty. Like I said in comment 2, the problem happens if a specific folder path is present in my (python) path. And the presence of this statement (in gdbus-codegen):- elif os.name == 'nt': wasn't doing anything magical - it was simply ensuring that that particular path was NOT present! So I've now gone back to Chun-wei's suggestion from comment 6. But instead of using his suggested path, I've simply used some arbitrary path (which again ensures that the troublesome path will NOT be present). Voila!! - Build now works again !!
(In reply to Fan, Chun-wei from comment #6) > Hello John, > > Ah, I see, it seems that you are trying to generate the gdbus source files > for building GIO... > > You need to define these envvars: UNINSTALLED_GLIB_SRCDIR and > UNINSTALLED_GLIB_BUILDDIR before you run the scripts. So, for your case, > you need to do (and forget what I mentioned about the file layout) > > set UNINSTALLED_GLIB_SRCDIR=F:\\GTK-SOURCES\\gnu-windows\\src\\MB3Glib > > set UNINSTALLED_GLIB_BUILDDIR=F:\\GTK-SOURCES\\gnu-windows\\src\\MB3Glib > Setting these is one way (which is anyway broken), but we shouldn't need to do that for in-source running. The purpose of that code is to set the PYTHONPATH such that gdbus-codegen can find the module source without user intervention.
*** Bug 787032 has been marked as a duplicate of this bug. ***
Created attachment 359606 [details] [review] gdbus-codegen: Don't assume bindir and datadir share prefix This assumption breaks when, for instance: * Called as /bin/gdbus-codegen * Installed on Windows in a directory that is not `bin/` For such cases, we cannot make any assumptions about the directory structure, and must hard-code the datadir.
Created attachment 359607 [details] [review] gdbus-codegen: Don't assume bindir and datadir share prefix This assumption breaks when, for instance: * Called as /bin/gdbus-codegen * Installed on Windows in a directory that is not `bin/` For such cases, we cannot make any assumptions about the directory structure, and must hard-code the datadir.
Thanks for all your hard work Nirbheek - but I'm afraid (on my system) it hasn't fixed the problem... :-( In other words... if (after applying your patch) I set those 2 environment variables to the paths suggested by Chun-wei (in comment #6) I'll see the error that I reported in comment #2. OTOH if I set them to any other value, the build works fine, without any error. gdbus-codegen ends up creating something from 'sys.path' (a Python path, I think). The basic problem here is that if that path contains the path to my top-level Glib folder, the build fails. If it doesn't, the build succeeds. On my system, that path is:- F:/GTK-SOURCES/gnu-windows/src/MB3Glib
Did you change the directory structure inside `F:/GTK-SOURCES/gnu-windows/src/MB3Glib` or is it the same as what is shipped with glib itself? Also, if you set `UNINSTALLED_GLIB_SRCDIR` in the environment, `sys.path` should contain `F:/GTK-SOURCES/gnu-windows/src/MB3Glib/gio/gdbus-2.0` (note the last two path components). Could you place some `print(path)` statements at various places in `gdbus-codegen` so we can figure out what's happening?
Nirbheek Chauhan wrote: " Did you change the directory structure inside `F:/GTK-SOURCES/gnu-windows/src/MB3Glib` or is it the same as what is shipped with glib itself? " Ah.. you might have spotted something. Inside the folder 'gio/gdbus-2.0/codegen' there's a file called 'parser.py'. For some reason, I've always needed to run that from a different folder:- (gio/gdbus-2.0/codegen/parse). Maybe that's what's causing the problem!? Nirbheek Chauhan wrote: " `sys.path` should contain `F:/GTK-SOURCES/gnu-windows/src/MB3Glib/gio/gdbus-2.0` (note the last two path components) " Yes, in my case that's all that sys.path contains now. The forward slashes are back slashes in my particular case (but the folder path itself is correct):- F:\GTK-SOURCES\gnu-windows\src\MB3Glib\gio\gdbus-2.0 As long as I'm the only person encountering this, let's just forget about it. I can work around it simply by modifying 'UNINSTALLED_GLIB_SRCDIR' slightly. It's really not a problem.
(In reply to John Emmas from comment #16) > Nirbheek Chauhan wrote: > " `sys.path` should contain > `F:/GTK-SOURCES/gnu-windows/src/MB3Glib/gio/gdbus-2.0` (note the last two > path components) " > > Yes, in my case that's all that sys.path contains now. The forward slashes > are back slashes in my particular case (but the folder path itself is > correct):- F:\GTK-SOURCES\gnu-windows\src\MB3Glib\gio\gdbus-2.0 > > As long as I'm the only person encountering this, let's just forget about > it. I can work around it simply by modifying 'UNINSTALLED_GLIB_SRCDIR' > slightly. It's really not a problem. I think there's something strange about your setup because the sys.path value is right from what I can tell. The patch itself seems correct. :) Just need a review and we can get this in.
Review of attachment 359607 [details] [review]: Looks reasonable to me
Attachment 359607 [details] pushed as b9f2ea4 - gdbus-codegen: Don't assume bindir and datadir share prefix
Followup in https://bugzilla.gnome.org/show_bug.cgi?id=789723