GNOME Bugzilla – Bug 752376
MPEG-DASH stream causes omxh264dec deadlock
Last modified: 2016-07-06 09:14:31 UTC
Created attachment 307412 [details] Tarball of debug output showing omx deadlock. Hi, I've hit a problem in gst-omx that causes issues when playing back adaptive bitrate MPEG-DASH streams on a Raspberry Pi (2). When GStreamer attempts to do a representation change, the omx decoder attempts to flush buffers in order to reconfigure the decoder for the resolution change. However, it always times out waiting for that to happen, and so the pipeline fails. So far I've only tested this on a Raspberry Pi, so I don't know if this is a problem specific to the Pi as it's the only device I use that requires gst-omx. You should be able to reproduce this behaviour with the following: gst-launch-1.0 playbin uri=http://rdmedia.bbc.co.uk/dash/ondemand/bbb/2/client_manifest-common_init.mpd At present, I'm running GStreamer Core/Plugins-Base/Good/Bad at version 1.4.5, and gst-omx is git latest. The OpenMAX support is provided by rpi-userland, which is at git version b864074d. I'm building GStreamer via buildroot (2015.08-git-00654-g5284dcf-dirty). A quick canter through the bugzilla list indicates that there have been similar problems with this in the past, but as I'm running git latest I'd have assumed that they'd all be fixed by now. Indeed, all the fixes seem to be present in git. (e.g. #710948, #726107, I will attach a log I've produced with omx:9 and omxvideodec:9 debugging, which seems to show the decoder deciding it needs to change, so sends the message "0:00:09.714889582 [...] <omxh264dec-omxh264dec0> Waiting for video_decode port 131 to release all buffers" But then when the decoder attempts to release a buffer, it says that the port is flushing or disabled, and so doesn't release the buffer. "0:00:09.750420143 [...] <omxh264dec-omxh264dec0> Releasing buffer 0xe92218 (0xf79750) to video_decode port 131 0:00:09.750522434 [...] <omxh264dec-omxh264dec0> video_decode port 131 is flushing or disabled, not releasing buffer" This seems like a bit of a deadlock to me. Looking at the debug dot file, the omx decoder module isn't showing as flushing (no upper case F), but is showing as the task being paused. I'll attach the files for these too.
I can't reproduce the deadlock here but video rendering stops nonetheless :) For some reason gst_omx_video_dec_set_format() returns early because it doesn't detect a format change. If I force is_format_change to TRUE the rendering is unblocked and playback goes on. I'll try to write a patch for this issue once I figured out what's happening.
Created attachment 309355 [details] [review] omxh264dec: implement is_format_change Trigger a decoder reconfiguration if the caps changed. The omxvideodecoder class only checks some of the caps parameters but if other fields change such as h264 profile and/or level it wouldn't trigger a reconfiguration.
Created attachment 309356 [details] [review] omxh264dec: implement is_format_change The omxvideodecoder class only checks some of the caps parameters but if other fields change such as h264 profile and/or level it wouldn't trigger a reconfiguration.
Comment on attachment 309356 [details] [review] omxh264dec: implement is_format_change The profiles/levels might be NULL, I think g_str_equal() doesn't like that much
Created attachment 309361 [details] [review] omxh264dec: implement is_format_change The omxvideodecoder class only checks some of the caps parameters but if other fields change such as h264 profile and/or level it wouldn't trigger a reconfiguration.
Review of attachment 309361 [details] [review]: ::: omx/gstomxh264dec.c @@ +105,3 @@ + + if (g_strcmp0 (old_profile, new_profile) != 0 + || !g_strcmp0 (old_level, new_level) != 0) { Boolean confusion :)
Created attachment 309362 [details] [review] omxh264dec: implement is_format_change The omxvideodecoder class only checks some of the caps parameters but if other fields change such as h264 profile and/or level it wouldn't trigger a reconfiguration.
commit e0327521f6d8340f9c901697e69ba942916b32da Author: Philippe Normand <philn@igalia.com> Date: Sun Aug 16 14:53:42 2015 +0200 omxh264dec: implement is_format_change The omxvideodecoder class only checks some of the caps parameters but if other fields change such as h264 profile and/or level it wouldn't trigger a reconfiguration. https://bugzilla.gnome.org/show_bug.cgi?id=752376
Thanks for your work on this, the original error I was seeing has gone. Unfortunately, it hasn't solved the problem entirely. Instead, I'm now getting a "decoder not initialized" error. I'll upload a second log which shows the issue in more detail. I don't think that is is necessarily just a Profile and Level change issue, as I can reproduce the new crash even with an MPD not containing any profile or level changes: http://rdmedia.bbc.co.uk/dash/ondemand/bbb/2/client_manifest-high_profile-level_40-common_init.mpd Running any MPD with only a single H.264 video representation works without any problems. If you're capable of running this on a Raspberry Pi without any crashes or glitches in your playback, could you please share exactly which versions of gstreamer/plugins-base/-good/-bad you're using so I can verify here? I'm trying to use stable releases but I understand 1.4.5 is getting a little long in the tooth by now.
Created attachment 309489 [details] Compressed Debug Log
Please try with recent git snapshots, at least of -bad because adaptivedemux and dashdemux received a fair amount of changes since the 1.4.5 release... I can test that stream on my rpi tomorrow anyway.
Here I don't see any error but the video rendering hangs after a few seconds indeed. Since the original issue was fixed I suggest we close this bug (again) and clone it to a new bug, maybe?
Yes, please create a new bug report and put some debug logs in there
I've got it up and running with the 1.5.90 release, and can confirm that the original bug is fixed. However, I also don't quite see Philippe's issue where the video rendering hangs, although I am getting very poor performance (playback at about 1fps). This isn't just happening on DASH streams, so I've opened a new ticket at https://bugzilla.gnome.org/show_bug.cgi?id=753886 to track this new bug.