After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 752376 - MPEG-DASH stream causes omxh264dec deadlock
MPEG-DASH stream causes omxh264dec deadlock
Status: RESOLVED FIXED
Product: GStreamer
Classification: Platform
Component: gst-omx
git master
Other Linux
: Normal major
: 1.9.1
Assigned To: GStreamer Maintainers
GStreamer Maintainers
Depends on:
Blocks:
 
 
Reported: 2015-07-14 14:34 UTC by Sam Hurst
Modified: 2016-07-06 09:14 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Tarball of debug output showing omx deadlock. (165.96 KB, application/x-gzip)
2015-07-14 14:34 UTC, Sam Hurst
  Details
omxh264dec: implement is_format_change (1.12 KB, patch)
2015-08-16 12:59 UTC, Philippe Normand
none Details | Review
omxh264dec: implement is_format_change (1.60 KB, patch)
2015-08-16 13:22 UTC, Philippe Normand
none Details | Review
omxh264dec: implement is_format_change (1.61 KB, patch)
2015-08-16 13:40 UTC, Philippe Normand
none Details | Review
omxh264dec: implement is_format_change (1.61 KB, patch)
2015-08-16 13:43 UTC, Philippe Normand
committed Details | Review
Compressed Debug Log (143.29 KB, application/x-gzip)
2015-08-18 16:42 UTC, Sam Hurst
  Details

Description Sam Hurst 2015-07-14 14:34:42 UTC
Created attachment 307412 [details]
Tarball of debug output showing omx deadlock.

Hi,

I've hit a problem in gst-omx that causes issues when playing back adaptive bitrate MPEG-DASH streams on a Raspberry Pi (2). When GStreamer attempts to do a representation change, the omx decoder attempts to flush buffers in order to reconfigure the decoder for the resolution change.

However, it always times out waiting for that to happen, and so the pipeline fails. So far I've only tested this on a Raspberry Pi, so I don't know if this is a problem specific to the Pi as it's the only device I use that requires gst-omx.

You should be able to reproduce this behaviour with the following:

gst-launch-1.0 playbin uri=http://rdmedia.bbc.co.uk/dash/ondemand/bbb/2/client_manifest-common_init.mpd

At present, I'm running GStreamer Core/Plugins-Base/Good/Bad at version 1.4.5, and gst-omx is git latest. The OpenMAX support is provided by rpi-userland, which is at git version b864074d. I'm building GStreamer via buildroot (2015.08-git-00654-g5284dcf-dirty).

A quick canter through the bugzilla list indicates that there have been similar problems with this in the past, but as I'm running git latest I'd have assumed that they'd all be fixed by now. Indeed, all the fixes seem to be present in git. (e.g. #710948, #726107, 

I will attach a log I've produced with omx:9 and omxvideodec:9 debugging, which seems to show the decoder deciding it needs to change, so sends the message

"0:00:09.714889582 [...] <omxh264dec-omxh264dec0> Waiting for video_decode port 131 to release all buffers"

But then when the decoder attempts to release a buffer, it says that the port is flushing or disabled, and so doesn't release the buffer.

"0:00:09.750420143 [...] <omxh264dec-omxh264dec0> Releasing buffer 0xe92218 (0xf79750) to video_decode port 131
0:00:09.750522434 [...] <omxh264dec-omxh264dec0> video_decode port 131 is flushing or disabled, not releasing buffer"

This seems like a bit of a deadlock to me. Looking at the debug dot file, the omx decoder module isn't showing as flushing (no upper case F), but is showing as the task being paused. I'll attach the files for these too.
Comment 1 Philippe Normand 2015-08-16 11:18:03 UTC
I can't reproduce the deadlock here but video rendering stops nonetheless :)
For some reason gst_omx_video_dec_set_format() returns early because it doesn't detect a format change. If I force is_format_change to TRUE the rendering is unblocked and playback goes on. I'll try to write a patch for this issue once I figured out what's happening.
Comment 2 Philippe Normand 2015-08-16 12:59:44 UTC
Created attachment 309355 [details] [review]
omxh264dec: implement is_format_change

Trigger a decoder reconfiguration if the caps changed. The
omxvideodecoder class only checks some of the caps parameters but if
other fields change such as h264 profile and/or level it wouldn't trigger a
reconfiguration.
Comment 3 Philippe Normand 2015-08-16 13:22:44 UTC
Created attachment 309356 [details] [review]
omxh264dec: implement is_format_change

The omxvideodecoder class only checks some of the caps parameters but if
other fields change such as h264 profile and/or level it wouldn't trigger a
reconfiguration.
Comment 4 Sebastian Dröge (slomo) 2015-08-16 13:30:28 UTC
Comment on attachment 309356 [details] [review]
omxh264dec: implement is_format_change

The profiles/levels might be NULL, I think g_str_equal() doesn't like that much
Comment 5 Philippe Normand 2015-08-16 13:40:19 UTC
Created attachment 309361 [details] [review]
omxh264dec: implement is_format_change

The omxvideodecoder class only checks some of the caps parameters but if
other fields change such as h264 profile and/or level it wouldn't trigger a
reconfiguration.
Comment 6 Sebastian Dröge (slomo) 2015-08-16 13:42:11 UTC
Review of attachment 309361 [details] [review]:

::: omx/gstomxh264dec.c
@@ +105,3 @@
+
+  if (g_strcmp0 (old_profile, new_profile) != 0
+      || !g_strcmp0 (old_level, new_level) != 0) {

Boolean confusion :)
Comment 7 Philippe Normand 2015-08-16 13:43:05 UTC
Created attachment 309362 [details] [review]
omxh264dec: implement is_format_change

The omxvideodecoder class only checks some of the caps parameters but if
other fields change such as h264 profile and/or level it wouldn't trigger a
reconfiguration.
Comment 8 Sebastian Dröge (slomo) 2015-08-16 13:48:44 UTC
commit e0327521f6d8340f9c901697e69ba942916b32da
Author: Philippe Normand <philn@igalia.com>
Date:   Sun Aug 16 14:53:42 2015 +0200

    omxh264dec: implement is_format_change
    
    The omxvideodecoder class only checks some of the caps parameters but if
    other fields change such as h264 profile and/or level it wouldn't trigger a
    reconfiguration.
    
    https://bugzilla.gnome.org/show_bug.cgi?id=752376
Comment 9 Sam Hurst 2015-08-18 16:41:49 UTC
Thanks for your work on this, the original error I was seeing has gone. Unfortunately, it hasn't solved the problem entirely. Instead, I'm now getting a "decoder not initialized" error. I'll upload a second log which shows the issue in more detail.

I don't think that is is necessarily just a Profile and Level change issue, as I can reproduce the new crash even with an MPD not containing any profile or level changes:

http://rdmedia.bbc.co.uk/dash/ondemand/bbb/2/client_manifest-high_profile-level_40-common_init.mpd

Running any MPD with only a single H.264 video representation works without any problems. 

If you're capable of running this on a Raspberry Pi without any crashes or glitches in your playback, could you please share exactly which versions of gstreamer/plugins-base/-good/-bad you're using so I can verify here? I'm trying to use stable releases but I understand 1.4.5 is getting a little long in the tooth by now.
Comment 10 Sam Hurst 2015-08-18 16:42:30 UTC
Created attachment 309489 [details]
Compressed Debug Log
Comment 11 Philippe Normand 2015-08-18 16:49:00 UTC
Please try with recent git snapshots, at least of -bad because adaptivedemux and dashdemux received a fair amount of changes since the 1.4.5 release...

I can test that stream on my rpi tomorrow anyway.
Comment 12 Philippe Normand 2015-08-19 06:48:32 UTC
Here I don't see any error but the video rendering hangs after a few seconds indeed.

Since the original issue was fixed I suggest we close this bug (again) and clone it to a new bug, maybe?
Comment 13 Sebastian Dröge (slomo) 2015-08-19 07:15:07 UTC
Yes, please create a new bug report and put some debug logs in there
Comment 14 Sam Hurst 2015-08-20 16:48:07 UTC
I've got it up and running with the 1.5.90 release, and can confirm that the original bug is fixed. However, I also don't quite see Philippe's issue where the video rendering hangs, although I am getting very poor performance (playback at about 1fps). This isn't just happening on DASH streams, so I've opened a new ticket at https://bugzilla.gnome.org/show_bug.cgi?id=753886 to track this new bug.