GNOME Bugzilla – Bug 753428
Use Distutils To Build the Dumper Binaries
Last modified: 2015-09-10 12:44:36 UTC
Hi, Since bug 728313 has already fulfilled its original intention of allowing the .gir files to be generated with Visual C++ without needing some Windows GCC, this bug is opened to track the process of making the build of the dumper program done with Python's distutils, so that it makes the .gir generation process more portable. Please note that the preprocessing of the headers and sources for generating the dumper program source is already done with distutils at this time, from bug 728313. I will be adding the people on the CC list of 728313 to track the progress here, as I think it would be a good idea to get this in time for g-i 1.46, ASAP. With blessings, thank you!
Hi, Since this needs people's help to ensure that everything runs on the platforms that g-i currently supports, I am adding people on the CC list from bug 728313 here for this purpose. Any help, comments and observations are really appreciated, as I am definitely not an expert in the distutils situation on UnixCCompiler. With blessings, thank you!
Created attachment 309596 [details] [review] giscanner: Use Distutils to Build And Link Dumper Binary, with libtool library handled Hi Ting-wei and Emmanuele, Can you do me a favor? Can you please test this patch and see whether the introspection files build correctly for you with this patch, as I heard from you both in bug 728313? It seems that we need to try to parse the .la libraries for both internal and external linking, which was the cause of the problems, AFAIK. I tried to build gnome-shell with this patch, which does succeed by running make -j8 (after autogen'ing/configure'ing with the correct LD_LIBRARY_PATH and PKG_CONFIG_PATH). This also passed g-i's 'make distcheck'. --- Hi Ting-wei, I see from the warning you were talking about in bug 728313 (note: not the -lintl problem)-does that break the build for you? If that warning does not break the build for you, and if this patch works for you with or without that warning, I hope that we can get this patch (or so) in, and try to deal with that warning with a separate patch, if that is alright with all, since I don't have access to FreeBSD. --- With blessings, thank you very much!
Sorry, Please substitute -lintl with -ldbind from the above comment....
(In reply to Fan, Chun-wei from comment #2) > I see from the warning you were talking about in bug 728313 (note: not the > -lintl problem)-does that break the build for you? If that warning does not > break the build for you, and if this patch works for you with or without > that warning, I hope that we can get this patch (or so) in, and try to deal > with that warning with a separate patch, if that is alright with all, since > I don't have access to FreeBSD. -export-dynamic is a libtool option. If we don't use libtool, we should not use it. GCC accepts -export-dynamic (found in gcc/common.opt, https://gcc.gnu.org/bugzilla/show_bug.cgi?id=47390), but this behavior is undocumented. Clang doesn't recognized this option, so it passes this unknown option to ld. However, ld needs --export-dynamic, not -export-dynamic. This causes ld to interpret it as -e xport-dynamic. -e option is used to set the entry point of the program. There is no xport-dynamic symbol, so a warning is showed. This usually doesn't cause problems, but I remember that I saw crashes caused by -export-dynamic. I don't think we should ignore the warning. The wrong option asks ld to do the wrong thing. Attachment 309596 [details] causes build failure in gtk+: GISCAN Gdk-3.0.gir tmp-introspect0c4jFm/home/lantw44/gnome/source/gtk+/gdk/tmp-introspect0c4jFm/Gdk-3.0.o:(.data.rel+0xb0): undefined reference to `gdk_touchpad_gesture_phase_get_type' cc: error: linker command failed with exit code 1 (use -v to see invocation) This means ld finds wrong version of libgdk-3.so. Adding -v -Wl,--verbose shows: attempt to open /home/lantw44/gnome/devinstall/lib/libgdk-3.so failed attempt to open /home/lantw44/gnome/devinstall/lib/libgdk-3.a failed attempt to open /usr/local/lib/libgdk-3.so succeeded -lgdk-3 (/usr/local/lib/libgdk-3.so) It is caused by the wrong order of -L arguments, which causes ld to use the version already installed on the system instead of the newly built version under .libs. The command used when linking: /usr/bin/cc -L/home/lantw44/gnome/devinstall/lib -L/usr/local/lib tmp-introspectPegIOh/home/lantw44/gnome/source/gtk+/gdk/tmp-introspectPegIOh/Gdk-3.0.o -L. -L.libs -L/home/lantw44/gnome/devinstall/lib -L/usr/local/lib -Wl,-rpath=.libs -Wl,-rpath=/home/lantw44/gnome/devinstall/lib.libs -Wl,-rpath=/usr/local/lib.libs -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -lgdk_pixbuf-2.0 -lcairo-gobject -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lintl -lfontconfig -lfreetype -lXinerama -lXi -lXrandr -lXcursor -lXcomposite -lXdamage -lXfixes -lX11 -lXext -lcairo -lepoxy -lm -lrt -o /home/lantw44/gnome/source/gtk+/gdk/tmp-introspectPegIOh/Gdk-3.0 -Wl,--export-dynamic -L/home/lantw44/gnome/devinstall/lib -lgio-2.0 -lgobject-2.0 -Wl,--export-dynamic -lgmodule-2.0 -pthread -lglib-2.0 -lintl -L. -L.libs should come before -L/home/lantw44/gnome/devinstall/lib -L/usr/local/lib. My LDFLAGS is -L/home/lantw44/gnome/devinstall/lib -L/usr/local/lib (set by JHBuild).
Created attachment 309853 [details] [review] giscanner: Use Distutils to Build and Link Dumper Binary, with libtool libraries handled (take ii) Hi Ting-wei, (In reply to Ting-Wei Lan from comment #4) > However, ld needs --export-dynamic... This should be addressed in this patch now, thanks for letting me know. > > Attachment 309596 [details] causes build failure in gtk+: > GISCAN Gdk-3.0.gir > tmp-introspect0c4jFm/home/lantw44/gnome/source/gtk+/gdk/tmp-introspect0c4jFm/ > Gdk-3.0.o:(.data.rel+0xb0): undefined reference to > `gdk_touchpad_gesture_phase_get_type' Thanks, I was able to get this behavior under my Fedora 22 system... OK, so it seems that when one goes through the .la libraries, in terms of the libpaths, when LDFLAGS is set, one would: -Search the libpaths from its dependent .la's, if they are listed before any explicitly-specified libpaths, sans the libpaths specified by LDFLAGS. Otherwise first search the libpaths specified by LDFLAGS. -Then search the libpaths specified by LDFLAGS, if not already searched. -Then search the libpaths specified by the .la files. Can you try the patch here and see whether everything goes ok for you? I tried this with and without LDFLAGS set up to gnome-shell, and everything in 'make distcheck' for gobject-introspection went well. Thanks, with blessings! --- 好險這次颱風應該不會來... ---
Attachment 309853 [details] does not apply on master branch ... error: patch failed: giscanner/dumper.py:149 error: giscanner/dumper.py: patch does not apply
Review of attachment 309853 [details] [review]: If I revert commit ebe6585 to test the patch, I get this message: GISCAN GLib-2.0.gir /usr/bin/ld: cannot find -lintl cc: error: linker command failed with exit code 1 (use -v to see invocation) The command it runs is: /usr/bin/cc -v tmp-introspectKHg5Rx/home/lantw44/gnome/source/gobject-introspection/tmp-introspectKHg5Rx/GLib-2.0.o -lglib-2.0 -lgobject-2.0 -o /home/lantw44/gnome/source/gobject-introspection/tmp-introspectKHg5Rx/GLib-2.0 -Wl,--export-dynamic -L/home/lantw44/gnome/devinstall/lib -lgio-2.0 -lgobject-2.0 -Wl,--export-dynamic -lgmodule-2.0 -pthread -lglib-2.0 -lintl -L/usr/local/lib from LDFLAGS is missing ... ::: giscanner/ccompiler.py @@ -123,3 +144,3 @@ - for library in libraries: - if self.check_is_msvc(): - args.append(library + '.lib') + # Ensure libraries are always linked as we are going to use ldd to work + # out their names later + internal_link_args.append('-Wl,--no-as-neebded') I think you mean -Wl,--no-as-needed here
Created attachment 309922 [details] [review] giscanner: Use Distutils to Build and Link Dumper Binary, with libtool libraries handled (take ii) Hi Ting-wei, Can you try this patch again, which was rebased against the latest master checkout, and should at least attempt address the jhbuild issues? Thanks for your help and patience. Much fun dealing with .la libraries with distutils :| sigh... Note: At Gjs it may need the additional PangoFT2 package along with the gtk+-3.0 in gjs_gtk_packages= in configure.ac... With blessings.
Attachment 309922 [details] does not apply because of commit 82b8678. I modified your patch in order to test it. It still does not build: GISCAN Offsets-1.0.gir /usr/bin/ld: cannot find -lintl cc: error: linker command failed with exit code 1 (use -v to see invocation) The command it runs is: /usr/bin/cc tmp-introspectlDAHs4/home/lantw44/gnome/source/gobject-introspection/tests/offsets/tmp-introspectlDAHs4/Offsets-1.0.o -L. -L.libs -L/home/lantw44/gnome/devinstall/lib -Wl,-rpath=.libs -loffsets -lgio-2.0 -lgmodule-2.0 -lgobject-2.0 -lglib-2.0 -lintl -o /home/lantw44/gnome/source/gobject-introspection/tests/offsets/tmp-introspectlDAHs4/Offsets-1.0 -Wl,--export-dynamic -Wl,--export-dynamic -pthread -L/usr/local/lib is in my LDFLAGS, but it doesn't use it. --- 還好最近只是下雨而已，沒發生什麼事情 ---
Created attachment 309949 [details] [review] giscanner: Use Distutils to Build and Link Dumper Binary, with libtool libraries handled (take iii) Hi Ting-wei, This is the rebased new patch, which is also cleaned up a little bit... (In reply to Ting-Wei Lan from comment #9) > -L/usr/local/lib is in my LDFLAGS, but it doesn't use it. Hmm, I checked my builds and the LDFLAGS are indeed picked up, by printing out self.compiler.library_dirs and comparing with the LDFLAGS that are acquired from the envvars... Maybe you want to clean up the builds and retry, I couldn't reproduce your situation, even when trying to use JHBuild, which built all the way up to gnome-shell... > --- > 還好最近只是下雨而已，沒發生什麼事情 > --- --- 真的好險天鵝沒來，不然這次大概不會很好 --- With Blessings, thank you!
It still fails with the same error message: GISCAN Offsets-1.0.gir /usr/bin/ld: cannot find -lintl cc: error: linker command failed with exit code 1 (use -v to see invocation) In giscanner.dumper.DumpCompiler._link, self._options.external_library is False, so giscanner.dumper.DumpCompiler._add_link_external_args is not called. This means giscanner.ccompiler.CCompiler.get_external_link_flags and giscanner.ccompiler.CCompiler._handle_ldflag_libpaths are not called, so LDFLAGS are not used. This problem doesn't happen on GNU/Linux because we don't need -L/usr/local/lib there.
Created attachment 309989 [details] [review] giscanner/ccompiler.py: Add printout for ldflags handling (*not* for commit) Hi, This is the patch that is applied on top of attachment 309949 [details] [review] to show what happens when LDFLAGS are passed in during the autogen/configure stage. This is not for going into master, btw. With blessings, thank you!
Created attachment 309990 [details] Build log showing build output. Hi Ting-wei, I couldn't reproduce your issue... the LDFLAGS are indeed added in my case, and I think I am out of ideas here. This is my configure line: LD_LIBRARY_PATH=/home/fanc999/gnome.unstable/lib PKG_CONFIG_PATH=/home/fanc999/gnome.unstable/lib/pkgconfig LDFLAGS='-L/home/fanc999/gnome.unstable/lib -L/usr/local/lib' ../gi.working/configure --prefix=/home/fanc999/gnome.unstable I am attaching the output of my build here, for references, which is obtained by applying attachment 309989 [details] [review] on top of attachment 309949 [details] [review], which should show that the LDFLAGS are indeed handled during the build of Offsets-1.0.gir. Plus, if the LDFLAGS aren't handled, we should have bailed out during GLib-2.0.gir and/or GIRepository-2.0.gir, as both of them would look for -lintl, AFAICT. With blessings, thank you!
Created attachment 309994 [details] gobject-introspection build log with libtool 2.4.6 on FreeBSD
Hi Ting-wei, Can you change this line (around line 433 in giscanner/ccompiler.py, where both patches 209949 and 309989 are applied): if self.ldflag_libpath_added_state is 1: to: if self.ldflag_libpath_added_state == 1: In my case the giscanner.ccompiler.CCompiler._handle_ldflag_libpaths was called (since print('_handle_ldflag_libpaths! LDFLAGS=%s, libpaths after ldflags handling=%s' % (self.ldflags, self.compiler.library_dirs)) is done whenever giscanner.ccompiler.CCompiler._handle_ldflag_libpaths is called on a UnixCCompiler instance), but it wasn't at all in your case. There could be a there are some subtleties between the Linux and FreeBSD implementation of Python... With blessings, thank you!
_handle_ldflag_libpaths is not called because deplibs[startidx:] is empty. deplibs = ['-L/home/lantw44/gnome/devinstall/lib', '-L/usr/local/lib'] startidx = 2
Created attachment 310089 [details] [review] giscanner: Use Distutils to Build and Link Dumper Binary, with libtool libraries handled (take iv) Hi, Hopefully this addresses the issues for you... managed to build gnome-shell under Fedora in jhbuild. With blessings, thank you!
GTK+ doesn't build: GEN gdkconfig.h GISCAN Gdk-3.0.gir Invalid GType function: 'gdk_touchpad_gesture_phase_get_type' Failed to find symbol 'gdk_touchpad_gesture_phase_get_type' Command '['/home/lantw44/gnome/source/gtk+/gdk/tmp-introspectm5b5Dc/Gdk-3.0', '--introspect-dump=/home/lantw44/gnome/source/gtk+/gdk/tmp-introspectm5b5Dc/functions.txt,/home/lantw44/gnome/source/gtk+/gdk/tmp-introspectm5b5Dc/dump.xml']' returned non-zero exit status 1 /home/lantw44/gnome/devinstall/share/gobject-introspection-1.0/Makefile.introspection:155: recipe for target 'Gdk-3.0.gir' failed gmake[4]: *** [Gdk-3.0.gir] Error 1
Hi Ting-wei, Hmm, it seems that on your system the GDK .so that is built during your build is in gdk/, not gdk/.libs (unlike the case on Linux--since the dumper executable linked for you-so it seems that when the dumper executable is running, it was not looking for the GDK .so from the right place as a result), so another difference in the situation on Linux and FreeBSD... Can you try to add the following lines to giscanner/ccompiler.py: self.compiler.add_library_dir('.') # [around line 129 in giscanner/ccompiler.py)]: if os.name != 'nt': self.compiler.add_runtime_library_dir('.') And let me know whether things went well for you? With blessings, thank you! --- 真的多謝你的耐心以及協助... ---
(In reply to Fan, Chun-wei from comment #19) > Hi Ting-wei, > > Hmm, it seems that on your system the GDK .so that is built during your > build is in gdk/, not gdk/.libs (unlike the case on Linux--since the dumper > executable linked for you-so it seems that when the dumper executable is > running, it was not looking for the GDK .so from the right place as a > result), so another difference in the situation on Linux and FreeBSD... libgdk-3.so is in gdk/.libs. The name of the hidden directory used by Linux and FreeBSD is the same. > > Can you try to add the following lines to giscanner/ccompiler.py: > > self.compiler.add_library_dir('.') > # [around line 129 in giscanner/ccompiler.py)]: > > if os.name != 'nt': > self.compiler.add_runtime_library_dir('.') > > And let me know whether things went well for you? This still doesn't work. ldd shows: /home/lantw44/gnome/source/gtk+/gdk/tmp-introspectgCxM9u/Gdk-3.0: libgdk-3.so.0 => /usr/local/lib/libgdk-3.so.0 I think we may want to set LD_LIBRARY_PATH here. It is usually done by the wrapper script generated by libtool. If we use neither LD_LIBRARY_PATH environment variable nor DT_RPATH dynamic section, it is really possible for the runtime linker to find the wrong library.
(In reply to Ting-Wei Lan from comment #20) > I think we may want to set LD_LIBRARY_PATH here. It is usually done by the > wrapper script generated by libtool. I'm growing a bit skeptical of using distutils on all platforms if the result is that we start to reinvent parts of libtool. But if it's just LD_LIBRARY_PATH on Unix, it's not the end of the world I guess...but still.
For hat it's worth, here it fails with Caught exception: <class 'distutils.errors.LinkError'> LinkError(LinkError(DistutilsExecError("command 'x86_64-pc-linux-gnu-cc' failed with exit status 1",),),) > /var/tmp/paludis/build/gnome-desktop-gobject-introspection-1.45.4-r1/work/gobject-introspection-1.45.4/giscanner/dumper.py(282)_link() -> raise LinkError(e) x86_64-pc-linux-gnu-cc: error: unrecognized command line option '-R' because distutils acts stupid when determining the flags to pass to the linker "if the linker is gcc" it passes -Wl,-R otherwise it passes -R but as lots of compilation infrastructures ours have CC being "cc" and not "gcc" and is thus mis-detected by distutils which only does a 'return "gcc" in compiler_name or "g++" in compiler_name'
Created attachment 310325 [details] [review] giscanner: Use Distutils to Build and Link Dumper Binary (take v) Hi Colin, (In reply to Colin Walters from comment #21) > I'm growing a bit skeptical of using distutils on all platforms if the > result is that we start to reinvent parts of libtool. I agree with you, and I am getting a bit wearied trying to get the libtool stuff right on systems I don't have access to, so this new patch does the following instead: -Compile the dumper program source using distutils, for all builds, as we were able to do without much hassle so far. -For linking: --If we are not using libtool, such as on MSVC builds, use distutils to do the linking, as we were trying to accomplish in this bug. --Otherwise, do what we used to do: construct the linker command line, except acquire the compiler command using what we have from distutils. --- Hi Marc-Antoine, Is your build using libtool, or is that -R flag passed in using LDFLAGS or so? With blessings, thank you!
Created attachment 310328 [details] [review] iscanner: Use Distutils to Build and Link Dumper Binary (take v.5) Sorry, I felt that this patch needed some cleaning up...
(In reply to Fan, Chun-wei from comment #23) > Hi Marc-Antoine, > > Is your build using libtool, or is that -R flag passed in using LDFLAGS or > so? > > With blessings, thank you! It works fine with your last patch, thanks
Review of attachment 310328 [details] [review]: This patch looks reasonable at a high level. The complexity here is quite high (not your fault) - but I'm willing to try landing this again.
Hi Colin, Thanks for your review and advice along the way! I have committed the patch as d1f6206. I will keep this open for a bit, in case problems should arise from this. With blessings, thank you!
I am doing build test and run test for attachment 310328 [details] [review]. This can take much time because jhbuild is usually interrupted by unrelated build issues.
GEGL doesn't build: GISCAN Gegl-0.3.gir /home/lantw44/gnome/source/gegl/gegl/tmp-introspectTqYCem/Gegl-0.3.c:61:7: error: use of undeclared identifier 'GEGL' g_critical ("failed to write to iochannel: %s", error->message); ^ /home/lantw44/gnome/devinstall/include/glib-2.0/glib/gmessages.h:165:32: note: expanded from macro 'g_critical' #define g_critical(...) g_log (G_LOG_DOMAIN, \ ^ <command line>:2:22: note: expanded from here #define G_LOG_DOMAIN GEGL-__FILE__ ^ /home/lantw44/gnome/source/gegl/gegl/tmp-introspectTqYCem/Gegl-0.3.c:76:7: error: use of undeclared identifier 'GEGL' g_critical ("failed to write to iochannel: %s", error->message); ^ /home/lantw44/gnome/devinstall/include/glib-2.0/glib/gmessages.h:165:32: note: expanded from macro 'g_critical' #define g_critical(...) g_log (G_LOG_DOMAIN, \ ^ <command line>:2:22: note: expanded from here #define G_LOG_DOMAIN GEGL-__FILE__ ^ 2 errors generated. Caught exception: <class 'distutils.errors.CompileError'> CompileError(DistutilsExecError("command 'clang' failed with exit status 1",),) > /usr/local/lib/python2.7/distutils/unixccompiler.py(122)_compile() -> raise CompileError, msg (Pdb) /home/lantw44/gnome/devinstall/share/gobject-introspection-1.0/Makefile.introspection:155: recipe for target 'Gegl-0.3.gir' failed -DG_LOG_DOMAIN="GEGL-"__FILE__ It should use -DG_LOG_DOMAIN=GEGL-__FILE__ -DG_LOG_DOMAIN="GEGL-"__FILE__, but it uses -DG_LOG_DOMAIN=GEGL-__FILE__ now.
Hi, Hmm, it seems that distutils is trying to be clever about dealing with CFLAGS. The thing is that those -D flags need to be placed in the macros attribute to be fed into compile(), and things would work. I will try to come up with a patch later. With blessings.
Created attachment 310841 [details] [review] giscanner/ccompiler.py: Improve CFLAGS handling Hi, This deals with the less seen cases where -Dxxx=yyy is set via CFLAGS, as distutils don't handle these well, especially when quotes are involved in these. Add such flags to the macros that are passed into distutil's compile() on instances of UnixCCompiler. With blessings, thank you!
Review of attachment 310841 [details] [review]: One minor comment. ::: giscanner/ccompiler.py @@ +216,3 @@ + cflags = os.environ.get('CFLAGS') + if cflags: + for i, cflag in enumerate(cflags.split()): Do we need to `GLib.shell_parse_argv` here? Or I guess shlex.split. Eh...we don't do that elsewhere, we can ignore this for now. @@ +219,3 @@ + if cflag.startswith('-D'): + cflag_idx.append(i) + if cflag.find('=') > -1: idx = cflag.find('=') if idx > -1: macroset = (cflag[2:idx, cflag[idx + 1:]) etc?
Hi Colin, Thanks for the review, the patch (updated to your suggestions and cleaned up a bit) was pushed as 3c42163. (In reply to Colin Walters from comment #32) > Do we need to `GLib.shell_parse_argv` here? Or I guess shlex.split. > Eh...we don't do that elsewhere, we can ignore this for now. This seems like what distutils was trying to do, but doing shlex.split() actually complicates things, as it will remove the quotes in the process. I think shlex needs to be used in other ways to be more effective in this case. > idx = cflag.find('=') > if idx > -1: > macroset = (cflag[2:idx, cflag[idx + 1:]) > I updated the patch as suggested, albeit with a slightly different variable name. With blessings, thank you!
Thanks, all modules successfully build and run now.
Chun-Wei, Lan: Thanks for your work on this!
*** Bug 754828 has been marked as a duplicate of this bug. ***