GNOME Bugzilla – Bug 562790
libpyglib should have libpython linked in explicitly to avoid undefined symbols
Last modified: 2010-11-28 00:02:03 UTC
only an issue if you're building with -no-undefined unavoidable in certain situations (cygwin)
Looks like there are several libs that have this problem, not just libpyglib. Easy enough to propagate PYTHON_LIBS with the appropriate -L and -l flags from ./configure into the Makefiles, but my python-fu is too weak to know the most portable way to determine those flags:(
Created attachment 125841 [details] [review] link against libpython to get all symbols defined Works For Me(tm) but note that the PYTHON_LIBS is hardcoded to where my machine stashes libpython rather than interrogating python itself to find out how it was configured. With this patch, the shared lib and all python mods have all symbols defined.
unfortunately, if one has a statically built libpython, then all the python symbols become exposed through libpyglib (as well as ballooning up the size signifigantly), so that'd need something like libpyglib_2_0_la_LDFLAGS = -export-symbols-regex '^pyglib_.*|^_pyglib_.*'
I'm working on a patch, I will attach it in a minute (or so).
I hope this is optional so everyone isn't forced to live with the win32 behavior
Created attachment 125873 [details] [review] patch
(In reply to comment #5) > I hope this is optional so everyone isn't forced to live with the win32 > behavior > Yes John, it's optional
Comment on attachment 125873 [details] [review] patch >+ AC_MSG_CHECKING([for Python library path]) Please rewrite so that the check is performed only if the result is going to be used, i.e. only on win32. >+ AC_MSG_ERROR([cannot find Python library path]) Particularly because of this. Although it didn't fail here, it would be stupid to err just because of some test that is not even needed. >+ AC_SUBST([PYTHON_LDFLAGS],["-L$python_path -l$python_lib"]) Please rename to WIN32_PYTHON_LDFLAGS or something, because this name is misleading. Do we also need similar patch for PyGTK?
(In reply to comment #6) > Created an attachment (id=125873) [edit] > patch > works, aside from when it dies out on /usr/bin/libtool --tag=CC --mode=link gcc -g -pipe -Wall -fno-strict-aliasing -std=c9x -o testhelper.la -rpath /usr/lib/python2.5/site-packages/pygobject -module -avoid-version testhelpermodule.lo test-thread.lo test-unknown.lo -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lintl -liconv rm -fr .libs/testhelper.a .libs/testhelper.dll .libs/testhelper.dll.a .libs/testhelper.la .libs/testhelper.lai gcc -shared .libs/testhelpermodule.o .libs/test-thread.o .libs/test-unknown.o /usr/lib/libgobject-2.0.dll.a -L/usr/lib /usr/lib/libgthread-2.0.dll.a /usr/lib/libglib-2.0.dll.a /usr/lib/libintl.dll.a /usr/lib/libiconv.dll.a -o .libs/testhelper.dll -Wl,--enable-auto-image-base -Xlinker --out-implib -Xlinker .libs/testhelper.dll.a .libs/testhelpermodule.o: In function `_wrap_test_g_object_new': /src/pygobject/tests/testhelpermodule.c:85: undefined reference to `_PyInt_FromLong' .libs/testhelpermodule.o: In function `_wrap_test_interface_iface_method': /src/pygobject/tests/testhelpermodule.c:96: undefined reference to `_PyArg_ParseTupleAndKeywords' /src/pygobject/tests/testhelpermodule.c:101: undefined reference to `__imp___Py_NoneStruct' /src/pygobject/tests/testhelpermodule.c:102: undefined reference to `__imp___Py_NoneStruct' .libs/testhelpermodule.o: In function `_wrap_TestInterface__do_iface_method': /src/pygobject/tests/testhelpermodule.c:169: undefined reference to `_PyArg_ParseTupleAndKeywords' /src/pygobject/tests/testhelpermodule.c:177: undefined reference to `__imp__PyExc_NotImplementedError' /src/pygobject/tests/testhelpermodule.c:177: undefined reference to `_PyErr_SetString' /src/pygobject/tests/testhelpermodule.c:181: undefined reference to `__imp___Py_NoneStruct' /src/pygobject/tests/testhelpermodule.c:182: undefined reference to `__imp___Py_NoneStruct' .libs/testhelpermodule.o: In function `_wrap_TestInterface__proxy_do_iface_method': /src/pygobject/tests/testhelpermodule.c:243: undefined reference to `_PyErr_Occurred' /src/pygobject/tests/testhelpermodule.c:244: undefined reference to `_PyErr_Print' /src/pygobject/tests/testhelpermodule.c:248: undefined reference to `_PyTuple_New' /src/pygobject/tests/testhelpermodule.c:249: undefined reference to `_PyObject_GetAttrString' /src/pygobject/tests/testhelpermodule.c:251: undefined reference to `_PyErr_Occurred' /src/pygobject/tests/testhelpermodule.c:252: undefined reference to `_PyErr_Print' /src/pygobject/tests/testhelpermodule.c:258: undefined reference to `_PyObject_CallObject' /src/pygobject/tests/testhelpermodule.c:260: undefined reference to `_PyErr_Occurred' /src/pygobject/tests/testhelpermodule.c:261: undefined reference to `_PyErr_Print' /src/pygobject/tests/testhelpermodule.c:268: undefined reference to `__imp___Py_NoneStruct' /src/pygobject/tests/testhelpermodule.c:269: undefined reference to `_PyErr_Occurred' /src/pygobject/tests/testhelpermodule.c:270: undefined reference to `_PyErr_Print' /src/pygobject/tests/testhelpermodule.c:271: undefined reference to `__imp__PyExc_TypeError' /src/pygobject/tests/testhelpermodule.c:271: undefined reference to `_PyErr_SetString' .libs/testhelpermodule.o: In function `__TestInterface__interface_init': /src/pygobject/tests/testhelpermodule.c:294: undefined reference to `_PyObject_GetAttrString' /src/pygobject/tests/testhelpermodule.c:297: undefined reference to `__imp__PyCFunction_Type' /src/pygobject/tests/testhelpermodule.c:297: undefined reference to `__imp__PyCFunction_Type' /src/pygobject/tests/testhelpermodule.c:297: undefined reference to `_PyType_IsSubtype' /src/pygobject/tests/testhelpermodule.c:300: undefined reference to `_PyErr_Clear' .libs/testhelpermodule.o: In function `_wrap_connectcallbacks': /src/pygobject/tests/testhelpermodule.c:444: undefined reference to `_PyArg_ParseTuple' /src/pygobject/tests/testhelpermodule.c:449: undefined reference to `__imp___Py_NoneStruct' /src/pygobject/tests/testhelpermodule.c:450: undefined reference to `__imp___Py_NoneStruct' .libs/testhelpermodule.o: In function `_wrap_test_value': /src/pygobject/tests/testhelpermodule.c:459: undefined reference to `_PyArg_ParseTuple' /src/pygobject/tests/testhelpermodule.c:464: undefined reference to `__imp__PyExc_TypeError' /src/pygobject/tests/testhelpermodule.c:464: undefined reference to `_PyErr_SetString' .libs/testhelpermodule.o: In function `_wrap_test_value_array': /src/pygobject/tests/testhelpermodule.c:477: undefined reference to `_PyArg_ParseTuple' /src/pygobject/tests/testhelpermodule.c:482: undefined reference to `__imp__PyExc_TypeError' /src/pygobject/tests/testhelpermodule.c:482: undefined reference to `_PyErr_SetString' .libs/testhelpermodule.o: In function `_wrap_test_gerror_exception': /src/pygobject/tests/testhelpermodule.c:497: undefined reference to `_PyArg_ParseTuple' /src/pygobject/tests/testhelpermodule.c:500: undefined reference to `_PyTuple_New' /src/pygobject/tests/testhelpermodule.c:501: undefined reference to `_PyObject_CallObject' /src/pygobject/tests/testhelpermodule.c:511: undefined reference to `__imp___Py_NoneStruct' /src/pygobject/tests/testhelpermodule.c:512: undefined reference to `__imp___Py_NoneStruct' .libs/testhelpermodule.o: In function `inittesthelper': /src/pygobject/tests/testhelpermodule.c:535: undefined reference to `_Py_InitModule4' /src/pygobject/tests/testhelpermodule.c:537: undefined reference to `_PyModule_GetDict' /src/pygobject/tests/testhelpermodule.c:539: undefined reference to `_PyImport_ImportModule' /src/pygobject/tests/testhelpermodule.c:540: undefined reference to `_PyModule_GetDict' /src/pygobject/tests/testhelpermodule.c:542: undefined reference to `_PyDict_GetItemString' /src/pygobject/tests/testhelpermodule.c:544: undefined reference to `__imp__PyExc_ImportError' /src/pygobject/tests/testhelpermodule.c:544: undefined reference to `_PyErr_SetString' /src/pygobject/tests/testhelpermodule.c:549: undefined reference to `__imp__PyExc_ImportError' /src/pygobject/tests/testhelpermodule.c:549: undefined reference to `_PyErr_SetString' /src/pygobject/tests/testhelpermodule.c:561: undefined reference to `_Py_BuildValue' .libs/testhelpermodule.o: In function `pygobject_init': /src/pygobject/tests/../gobject/pygobject.h:309: undefined reference to `_PyImport_ImportModule' /src/pygobject/tests/../gobject/pygobject.h:311: undefined reference to `_PyErr_Occurred' /src/pygobject/tests/../gobject/pygobject.h:315: undefined reference to `_PyErr_Fetch' /src/pygobject/tests/../gobject/pygobject.h:316: undefined reference to `_PyObject_Repr' /src/pygobject/tests/../gobject/pygobject.h:320: undefined reference to `_PyString_AsString' /src/pygobject/tests/../gobject/pygobject.h:320: undefined reference to `__imp__PyExc_ImportError' /src/pygobject/tests/../gobject/pygobject.h:320: undefined reference to `_PyErr_Format' /src/pygobject/tests/../gobject/pygobject.h:325: undefined reference to `__imp__PyExc_ImportError' /src/pygobject/tests/../gobject/pygobject.h:325: undefined reference to `_PyErr_SetString' /src/pygobject/tests/../gobject/pygobject.h:331: undefined reference to `_PyObject_GetAttrString' /src/pygobject/tests/../gobject/pygobject.h:332: undefined reference to `__imp__PyCObject_Type' /src/pygobject/tests/../gobject/pygobject.h:333: undefined reference to `_PyCObject_AsVoidPtr' /src/pygobject/tests/../gobject/pygobject.h:335: undefined reference to `__imp__PyExc_ImportError' /src/pygobject/tests/../gobject/pygobject.h:335: undefined reference to `_PyErr_SetString' /src/pygobject/tests/../gobject/pygobject.h:346: undefined reference to `_PyObject_GetAttrString' /src/pygobject/tests/../gobject/pygobject.h:348: undefined reference to `_PyObject_GetAttrString' /src/pygobject/tests/../gobject/pygobject.h:350: undefined reference to `__imp__PyExc_ImportError' /src/pygobject/tests/../gobject/pygobject.h:350: undefined reference to `_PyErr_SetString' /src/pygobject/tests/../gobject/pygobject.h:355: undefined reference to `_PyArg_ParseTuple' /src/pygobject/tests/../gobject/pygobject.h:357: undefined reference to `__imp__PyExc_ImportError' /src/pygobject/tests/../gobject/pygobject.h:357: undefined reference to `_PyErr_SetString' /src/pygobject/tests/../gobject/pygobject.h:367: undefined reference to `__imp__PyExc_ImportError' /src/pygobject/tests/../gobject/pygobject.h:367: undefined reference to `_PyErr_Format' Creating library file: .libs/testhelper.dll.a collect2: ld returned 1 exit status make[2]: *** [testhelper.la] Error 1
(In reply to comment #9) > works, aside from when it dies out on well it looks like it's not... > /usr/bin/libtool --tag=CC --mode=link gcc -g -pipe -Wall > -fno-strict-aliasing -std=c9x -o testhelper.la -rpath > /usr/lib/python2.5/site-packages/pygobject -module -avoid-version > testhelpermodule.lo test-thread.lo test-unknown.lo -lgobject-2.0 -lgthread-2.0 > -lglib-2.0 -lintl -liconv in the tests/Makefile.am add $(PYTHON_LDFLAGS) as well, and it looks like that -no-undefined is missing, I forgot that.
sysconfig.get_config_var("LIBPL") gives the dir containing the static libpython on both linux and darwin (fink's lib build, not apple's framework...the latter doesn't have a static libpython at all). Do those work on win? So instead of for looping on "various possible places", could just look where python itself says the file will be (if it exists). Something like: >python_path=`$PYTHON -c 'from distutils import sysconfig; print sysconfig.get_config_var("LIBPL")'` (NB: confusing to use $python_path two different ways attachment 125873 [details] [review] !)
Created attachment 128426 [details] [review] distutils patch to build on win32 Hi guys... Gian Mario suggested to attach my patch to this bug because the problem is somehow related. I'll try to be verbose about the patch also for my own good since otherwise I'll forget most of the details soon... bear with me :) So... since gedit now works on win32 and gedit plugins require a newish pygobject I really really wanted a working pygobject on win32. After seeing Gian Mario struggle with auto* I decided to try to resuscitate the unsupported distutils based build system, since it is also the one that is used to create the old pygobject installers. With this patch (plus some manual tweaking I'll explain below) I managed to get an installer build and more or less working (ok... I just tried "import glib" and it works, more testing is needed) Without further ado, let's get to what the patch does and what it is still missing: 1) the patch modifies dsextras.py for two reasons 1 - a) Allow to pass --export-all-symbols when creating the pyglib utility dll without letting distutils create a .def file with the bogus "initpyglib" exported symbol (more on this below) 1 - b) Allow to propagate include_dirs etc specified when instanciating a PkgConfigExtension instead of overwriting them 2) The patch updates setup.py to build all the new stuff, in particular: 2 - a) the pyglib utility dll 2 - b) the new glib module 2 - c) the new gio module Of these the most problematoc is the pyglib dll (2-a) since I did not find an official way to build shared libs which are not modules. What I did is simply use the same PkgConfigExtension class used to build modules and force it to export all the symbols (see the modification to dsextras). In the other modules I then tell the compiler to include and link this utility library. 3) Last thing the patch does is fix compilation due to a missing header... there are also many scary warnings that should be fixed on windows What is still misssing (and where I would really appreciate help and suggestions): - my trick to create the pyglib dll creates a .pyd file... I manually rename it to dll so that the modules manage to link to it - I put the pyglib dll in the glib subdir... seems to work but I doubt it is the right place. On unix it ends up in /usr/lib... I have no idea where it should go on win32... in the bin dir of the python install? If yes, how do we move it there? - I have hardcoded the path to the pyglib dll passed in the libraries_dirs paramater since I had no idea how to programmatically get the build dir and also given the fact that as said above I do not think it is the right place for the lib That's all for tonight :-)
yeah, just wondering, does libpyglib actually need to be a shared library? this could be much easier handled if it were a static archive, and google says nothing outside of pygobject links to it (yet), so there'd be minimal impact
Hello, I think I've made some progress getting pygobject 2.18.0 to build on MS Windows. I'm attaching a patch of the changes I needed to make to get it to build. I used MingW and MSYS to do the build. I needed to modify the --export-symbols-regex argument for glib and gobject. The vertical bar was causing shell problems when the regex was only quoted with double quotes so I changed them to single quotes. I also needed to add the include line to get glib/pygiochannel.c to build because PyGPollFD_Type was undeclared. I had to change the *_LIBADD lines to link against the glib .lo file instead of the .la file. Otherwise, a dll wouldn't get built (libtool complained about linking to a static lib). Finally I had to add -lgthread-2.0 to the link line for the gio/ module. That's not in the patch, I modified the Makefile directly because I couldn't figure out where to patch. I built it on Windows XP using the version of glib that you get with the GTK+ stack bundle available here: http://www.gtk.org/download-windows.html#StableRelease Which seems to be glib 2.20.3. After building and installing and renaming the .dll files to .pyd, I was able to import glib and gobject successfully. Importing gio resulted in a crash in malloc.c. Hope this helps, dave
Created attachment 136795 [details] [review] patch for building on windows with mingw (./configure)
Created attachment 138720 [details] Updated setup.py build Just a note to say that I have continued with the work paolo borelli started (#c12) The attached patch builds pygobject 2.19 from master correctly. I chose to build glib/pyglib.c into all the extensions, rather than creating a shared library, or libtool archive against which they link. This seems to work fine. I still need to work out how to do the generate-constants.c -> constants.py magic.
Ok, I think I have this beat. Complete. Victory. I was able to generate correct pygobject installers and have been testing them for the last day or so. The branch (master-windows) with all the fixes can be found at http://github.com/nzjrs/pygobject/tree The most invasive change was to remove generate-constants.c (because it cannot be run during a distutils build), and add the constants in gobjectmodule.c directly. I would like to know if this change is acceptable to be upstreamed (i.e. for the linux builds as well). If so then I will rebase the changes into a more coherent patch series and push to git.gnome in a new windows branch. Thoughts?
Build instructions: * Install gtk+ bundle from gtk.org * Install Python2.6 * Install MinGW-5.14.exe and MSYS-1.0.11-rc-1.exe * Add C:\GTK\bin to path (from windows) * Add the following to your msys environment variables (open and append to C:\msys\1.0\etc\profile) export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/c/devel/lib/pkgconfig:/c/Python26/Lib/pkgconfig export PATH=$PATH:/c/Python26:/c/Python26/bin export CPPFLAGS="-I/c/Python26/include" export LDFLAGS="-L/c/Python26/lib" * In msys you can build with python setup.py build --compiler=mingw32 --enable-threading --yes-i-know-its- not-supported bdist_wininst
wow... not sure why I was not on CC. Great to see progress finally made!
(In reply to comment #17) > Ok, I think I have this beat. Complete. Victory. Thanks John > The most invasive change was to remove generate-constants.c (because it cannot > be run during a distutils build), and add the constants in gobjectmodule.c > directly. I would like to know if this change is acceptable to be upstreamed > (i.e. for the linux builds as well). I don't know that part of the code, I guess if it works the same (also checking the python version) it could be acceptable also building on GNU/Linux.
Just a friendly comment to find out the status on this bug. From the above comments it seems to me like this issue is solved. Does it simply need more testing before it can be fixed/closed? I'm patiently waiting on another bug that this one is currently blocking, so if there is anything I can do to help I'd be willing to give it a shot.
ping? It's been 5 months and several release-versions. If the fix fixes it according to those who have the problem and the fix is conditionalized to apply only to those platforms rather than affecting anyone else, is there some reason it's not committed yet?
Agree, the patch to fix this and Gtk+ lives on bug 589671 John (patiently waiting for a review)
Is this still relevant?
Paul, Gian, Paolo: Is this still relevant?
I fixed this ages ago in the windows pygobject/pygtk builds.