GNOME Bugzilla – Bug 734954
(lt-Gst-1.0:19423): GLib-ERROR **: ../../glib/gmem.c:353: overflow allocating 1701079383*4 bytes
Last modified: 2017-05-03 14:11:13 UTC
I've been getting this error with astounding regularity for the past day or two. It has become impossible to compile lib32-gstreamer-git (an important dependency of several packages). I cannot compile lib32-gstreamer-plugins-base either because of the same error. The problem is not happening compiling for my native architecture, x86_64. My first thought was that perhaps a recent commit had broken something, so I tested all the commits for the last five days but there was no change. I also tested the last several versions of (lib32-)glib2-git but there was no change. This compile worked only a few days ago, but now I cannot find any way to get past this error (tried MALLOC_CHECK_=0, but it is ignored; --{en,dis}able-glib2 also both ignored). I need someone else to test if compiling 32bit gstreamer works on another 64bit machine. If so, then I can begin to debug the problem by comparison. There appears to be nothing out of the order, nothing has changed since the last time this compile succeeded except the date.
I compiled lib32-glib2-git with the condition in g_malloc0_n() disabled to see if compile would go through, and this is the next error message: (lt-Gst-1.0:24946): GLib-ERROR **: ../../glib/gmem.c:133: failed to allocate 2980420032 bytes Looks like it cuts off about 3GB from lt-Gst-1.0's request. Why does lib32-gstreamer-git want over 6GB (1701079383*4 bytes) of memory during compile?
what platform/distro is this ? Can you provide full logs of the compile step ? Also add logs of where/what fails. If you can, provide backtraces of the allocation failure.
>Can you provide full logs of the compile step ? I have uploaded their Archlinix PKGBUILDs to a git repository*: glib2-git-multilib https://github.com/quequotion/mmug-qq/tree/master/glib2-git-multilib gstreamer-git-multilib https://github.com/quequotion/mmug-qq/tree/master/gstreamer-git-multilib You could use them to create bash scripts and to test the procedure. *The repository contains other packages of interest, such as lib32-python2 with it's architecture-specific includes in tact for building dependent packages and gobject-introspection-git-multilib which is also a dependency of gstreamer-git-multilib. I'll be right back with some build logs...
Created attachment 283723 [details] buildlog of lib32-gstreamer-git makepkg -p PKGBUILD.lib32 32bit compile on a 64bit machine is failing due to a memory allocation overflow. The error does not happen while compiling in native 64bit.
Created attachment 283724 [details] buildlog of lib32-gstreamer-git with g_malloc0_n()'s fail condition disabled I created a separate git branch for malloc testing: https://github.com/quequotion/mmug-qq/tree/malloc_testing/glib2-git-multilib makepkg -p PKGBUILD.lib32 32bit compile on a 64bit machine is failing due to a memory allocation overflow. The error does not happen while compiling in native 64bit.
Gotcha: When generating these logs I left -Ofast enabled in makepkg.conf but I have tested the build with both -O0 and -O1, which resulted in exactly the same memory allocation error.
It seems to me this should be reported against arch linux or gobject-introspection instead?
>Tim Please test before shifting the blame off to another dev team who will not be able to do anything about it. It's not a problem with archlinux; the host distribution is irrelevant. The problem is somewhere between gstreamer, glib, and possibly gobject-introspection. I have taken great care to make sure I have all of the needed 32bit libraries installed to compile lib32-gstreamer and I have tested my packaging of glib2 and gobject-introspection against several other lib32- packages; lib32-gstreamer is the only one suffering a memory allocation error so far.
What I really need to know is: Can anyone else build lib32-gstreamer-git on x86_64? (verified working with 32bit dependants) How do you have lib32-glib2 and (lib32-)gobject-introspection installed? (what components installed where). "lib32-gobject-introspection" may be unnecessary; the build stopped working before I managed to compile this package (it works neither with nor without it). Is lib32-python2 relevant here?
It's not about shifting blame, but if I understand you correctly what happens is that the Gst-1.0 binary generated by gobject-introspection aborts with an error that indicates a programming error or bug in that program. It is not clear to me how this would be GStreamer's fault somehow, I would say that whatever annotions are in the GStreamer code, the g-i binary should never crash or abort like that. So that leaves a bug in our Makefile.am, that somehow something gets linked wrongly, but I would expect things to bomb out much earlier than that then.
Created attachment 283733 [details] installed files of lib32-glib2 For example, here are the files my lib32-glib2-git package installs. Some things have been removed to avoid collision with the native 64bit package, such as executables. Am I missing something needed while compiling lib32-gstreamer?
Created attachment 283734 [details] installed files of lib32-gobject-introspection Here are the files my installed by my lib32-gobject-introspection-git package. Again, some things have been removed to avoid collision with the native 64bit package.
>the g-i binary should never crash or abort like that. Perhaps this is the issue. Do I need the 32bit g-i executables in order to compile lib32-gstreamer? One way or another (either through proper configuration or simple renaming) I think i can get them into the package, but then how could I tell gstreamer's autoconf to use alternate g-i binaries?
Repackaged and branched: gobject-introspection-multilib https://github.com/quequotion/mmug-qq/tree/gstreamer-testing/gobject-introspection-git-multilib lib32-gobject-introspection now keeps it's includes and the binaries are suffixed with -32. gstreamer-git-multilib (32bit pkgbuild updated) https://github.com/quequotion/mmug-qq/tree/gstreamer-testing/gstreamer-git-multilib In order to force gstreamer to use /usr/bin/g-ir-{scanner,compiler,generate}-32, I patched the "common" submodule before running autogen.sh. There doesn't seem to be any other way to specify alternate g-i binaries for multilib installations. The memory allocation overflow still happens. Has anyone had time to test building lib32-gstreamer on x86_64?
Created attachment 283775 [details] installed files of lib32-gobject-introspection Now with binaries and includes; properly installed by giving the parameters to configure and still avoiding collision with the native x86_64 package.
I was unaware that introspection could be disabled. The only way to build lib32-gstreamer-git is to disable introspection. Please, somebody prove me wrong. Despite going to great effort to package lib32-gobject-introspection for use in this build, I think lib32-gstreamer is still finding x86_64 gobject-introspection and getting the wrong datatype information.
This still sounds more like a gobject-introspection issue to me. Not sure what to do about this without more concrete information on what goes wrong where in GStreamer, and I don't think anyone is likely to investigate this seeing that nothing has happened in the last 1-2 years, sorry.
>the last 1-2 years In which I have not attempted to do this. At the time, the gstreamer development tip was a build dependency for the 32bit-only pcsx2. I haven't really used pcsx2 much in the last two years either, and only compiled against precompiled dependencies. If I get around to this again, and it doesn't work, I should probably open a new bug.
Actually we've had a patch for this for a while in OpenIndiana: https://github.com/OpenIndiana/oi-userland/blob/oi/hipster/components/multimedia/gstreamer/patches/gst-02-amd64.patch Your problem is using the same name for the registry in 32-bit and 64-bit. During the build the registry in the user's home directory is loaded: if there is a previously generated 64-bit registry in the home directory and the build is 32-bit, kaboom.
Looks like this was fixed downstream by adding --build=i686-pc-linux-gnu to configure/autogen arguments. https://github.com/quequotion/mmug-qq/commit/2e7ca68d54bfd995b92d7b43917d5ba35537a72a#diff-91fc4b079409add934ca634aab8c4bf9 You can't just pass -march=m32 or such in CFLAGS.
>Looks like this was fixed downstream I am not so sure of that. You're referencing my own git repository, which is almost certainly out of date (mmug in particular hasn't been worked on much in a few years). I'm out of town and cannot test my PKGBUILD at the moment to see if it actually works. I probably enabled overkill 32bit flags everywhere because it was having some problem related to 64bit; I think Aurelien's comment is worth looking into--this sounds plausible for the problem I was having. Does --build=i686-pc-linux-gnu somehow avoid using the wrong registry?
No, but you must pass --build=i686-pc-linux-gnu for autotools to use/detect the right HOST_CPU. If it is omitted, and just -m32 or such is passed, then what Aurelien describes may happen. The patch Aurelien references is not right and should not be required if cross-compile is done correctly.
(In reply to Tim-Philipp Müller from comment #22) > No, but you must pass --build=i686-pc-linux-gnu for autotools to use/detect > the right HOST_CPU. If it is omitted, and just -m32 or such is passed, then > what Aurelien describes may happen. The patch Aurelien references is not > right and should not be required if cross-compile is done correctly. This is maybe true for Linux but Solaris/illumos has a different handling of architectures (similar to IRIX) which actually predates Linux's 64-bit support. On Solaris/illumos the host architecture may support different ISAs (i386 and amd64) and each ISA can have extensions (remember also that on IRIX up to 3 ISAs would co-exist). Tools default to the least-common denominator for each architecture: that's why for instance gcc defaults to 32-bit on 64-bit capable x86. From the beginning amd64 has been seen as an extension of i386 on 64-bit capable machines, and not as a totally different architecture. The reason is the care for backward-compatibility: providing 32- and 64-bit libs and userland is a natural continuity if the 64-bit ISA is seen as an extension, and not retrofitted with latter addition of "*-lib32" and cross-compilation. In this case, relying on HOST_CPU is incorrect because the host cpu being assumed to be possibly both 32- and 64-bit capable, the switch happens with compilation flags. I guess it is a matter of definition.