GNOME Bugzilla – Bug 412678
random segfaults or memory corruptions with multiple textoverlays (pango not reentrant)
Last modified: 2011-03-20 21:15:15 UTC
A pipeline of mine has 5 v4l2src sources and an ximagesink. A textoverlay is added onto each source, then all are mixed through a videomixer and the result is displayed in an X window. When I use textoverlays, there are random crashes (segfaults, memory corruption, etc...). Without textoverlay elements, I can't reproduce the problem. I'll attach gdb's log as well as my own program's log, which includes the pipeline's definition.
Created attachment 83468 [details] result of launching controlled from gdb, then 'thread apply all bt'
Created attachment 83469 [details] My own program's log including the pipeline's defintion
- what versions of GStreamer core, -base and -good? $ gst-inspect-0.10 filesrc | grep Version $ gst-inspect-0.10 textoverlay | grep Version $ gst-inspect-0.10 videobox | grep Version - what version of v4l2src? - what distro? - could you make sure you've got full debugging symbols for GStreamer, gst-plugins-* and GLib, by installing the appropriate debug packages please? - can you reproduce the problem by using videotestsrc (maybe followed by a capsfilter forcing the same dimensions as your v4l2src give you) in place of the v4l2src? - can you reproduce the problem if you're running your pipeline in gst-launch-0.10? If yes, could you run it through valgrind please? $ G_SLICE=always-malloc valgrind --trace-children=yes gst-launch-0.10 ...yourpipelinehere... 2>valgrind.log (this will only really be useful if you have full debug symbols).
filesrc version 0.10.10 textoverlay version 0.10.10 videobox version 0.10.4 v4l2src version 0.10.2 videotestsrc 0.10.10 Standard gstreamer* Debian Sid packages as of today March 1st 2007. The whole machine is up-to-date as far as Debian packages are concerned. I've reproduced this random problem with videotestsrc on a different machine with identical versions, and the attached pipeline script. It seems glibc detected a double free, but the crashing message differs often. You may need to launch the pipeline a few times before it crashes.
Created attachment 83693 [details] Test pipeline which reproduces the problem using videotestsrc
> - could you make sure you've got full debugging symbols for > GStreamer, gst-plugins-* and GLib, by installing the appropriate > debug packages please? Now done. > - can you reproduce the problem by using videotestsrc (maybe followed > by a capsfilter forcing the same dimensions as your v4l2src give you) > in place of the v4l2src? Yes, see comments #4 and #5 > - can you reproduce the problem if you're running your pipeline in > gst-launch-0.10? If yes, could you run it through valgrind please? > > $ G_SLICE=always-malloc valgrind --trace-children=yes gst-launch-0.10 > ...yourpipelinehere... 2>valgrind.log OK, seems to confirm the problem in textoverlay.c around line 1290, I'll attach the valgrind report ASAP.
Created attachment 83695 [details] Valgrind report
This seems to have a lot to do with pango not being reentrant during all its initialization. As a really stupid workaround, you might try starting up a pipeline with only one textoverlay, run a few frames, then shut it down, and then try your pipeline. (Of course, you can't do this with gst-launch.) I'm not sure there's much we can do about pango not being reentrant. You could also look into using cairotextoverlay. It doesn't seem to support the same parameters; please ask for features you need.
Seems to work without problem with cairotextoverlay and videotestsrc. I'll test with v4l2src ASAP. Thanks for your help !
Jerome, can you confirm it fixes your issues ?
(In reply to comment #10) > Jerome, can you confirm it fixes your issues ? Unfortunately I can't. I forgot to do it at that time, and now I don't have access to the original hardware anymore to check with multiple v4l2src sources, since I've changed job. Sorry to not be more helpful with this.
*** Bug 642960 has been marked as a duplicate of this bug. ***
and it's not been solved since 2007? shouldn't have to remove clockoverlay from base?
if we can't use timeoverlay could you add date overlay to cairotextoverlay (since currently it's doesn't support it).
Created attachment 182087 [details] [review] do a early init of pango internals This is quite a hack, but works fine for me.
Asked behdad if there is a better solution for pango: """ Your hack should work, but now that you are getting hit by this, why not make pango thread safe? It's not AS hard as one would imagine. I'd love to hack on it... We need two pieces: - GOnce'ify initializations. g_once_init/leave() make that less painful. One can start by looking up all static variables in the library and attack them. - Lock'ify shared objects (fontmap, fonts, font familys, faces, ...). Mutex's are painfully bulky and slow. gbitlock to the rescue. Lets start by finishing my patch for g_object_lock(): https://bugzilla.gnome.org/show_bug.cgi?id=608695 """ Fixing pango is a bit outside of what I can do now and it might take a bit of time to become effective to users anyway. Shall we put that hack in? I can add a bit FIXME comment to explain.
Could you put the hack in a GOnce so that it is serialized? Or put it in the class init function, which I suppose would require creation/deletion of a layout object. Otherwise, it seems like a good solution until pango is fixed (and gets out distros, etc.).
Having it in class init was not sufficient, as then I would free the pango layout and pnago would release the type-plugins. I keep it now in _init() but wrapped it with gonce (thanks for suggestion). commit fee3266056b522cdd34e606b5682553d35eec5a1 Author: Stefan Kost <ensonic@users.sf.net> Date: Fri Mar 4 14:52:28 2011 +0200 textoverlay: add a hack to init the pango engine Layout a single char to pre-create all resources. I got no more crashers, but once these warnings: (gst-launch-0.10:18177): Pango-WARNING **: failed to create cairo scaled font, expect ugly output. the offending font is 'DejaVu Sans Mono 18' (gst-launch-0.10:18177): Pango-WARNING **: font_face status is: no error has occurred (gst-launch-0.10:18177): Pango-WARNING **: scaled_font status is: invalid matrix (not invertible) (gst-launch-0.10:18177): Pango-WARNING **: shaping failure, expect ugly output. shape-engine='BasicEngineFc', font='DejaVu Sans Mono 18', text='10:40:54' That could still be a race or a different bug.
Comment on attachment 182087 [details] [review] do a early init of pango internals with small modification.
Created attachment 183320 [details] backtrace Still got one crash :/
Got more crashes with various backtraces. I reverted the previous commit and went for a class wide lock. This seem to work reliable for me. commit 9ac74c59db5b2cb7090b9e18fda1894c8511e881 Author: Stefan Kost <ensonic@users.sf.net> Date: Mon Mar 14 11:14:04 2011 +0200 textoverlay: use a class wide mutex to work around pango reentrance issues Pango is not reentrant. Use a class wide mutex to protect pange use in gst_text_overlay_render_pangocairo(). This works reliable in contrast to the hack in my previous commit. Fixes Bug #412678
*** Bug 645333 has been marked as a duplicate of this bug. ***