GNOME Bugzilla – Bug 734499
videoaggregator: Does not update segment position from upstream segments
Last modified: 2014-10-06 08:33:29 UTC
Created attachment 282943 [details] GLVideomixer bug example After much debugging and after adding yet another head-shaped dent in my desk, I figured out that the glvideomixer element prevents a pipeline from seeking. I've attached an example, set the URI variable at the top to a video file to run. Switch out the glvideomixer with a regular videomixer and it works perfectly.
Thanks for your report. I can not use you script here, I am getting: Traceback (most recent call last):
+ Trace 233945
mediatype = pad.get_current_caps().to_string().split(',')[0]
I have tested seeking with glvideomixer doing: gst-validate-1.0 uridecodebin uri=file:///home/thiblahute/Videos/tron.webm ! queue ! glvideomixer ! videorate ! xvimagesink --set-scenario scrub_forward_seeking and it does work here.
Sorry for the delay in responding, I just started a new job and it's kept me pretty busy. The example works fine for me using GStreamer git master and an avi-encapsulated h264 stream with audio. Maybe try a different video file?
Do you have any files where this fails? Would it be possible for you to share them somewhere?
Here's the link to the test video I use (the last few seconds of a countdown video I made): https://owncloud.isrv.pw/public.php?service=files&t=cedd3917725819b9fe1f92fc88a24baf However, on my machine the issue occurs with every video file I've tried.
That file works fine for me with the following pipeline gst-launch-1.0 -v uridecodebin uri=file:///path/to/test.avi ! glvideomixer ! navseek ! ximagesink Could you post the output of the above command on your machine?
That pipeline does work correctly, pressing the left arrow seeks to the beginning of the video. Output: Setting pipeline to PAUSED ... /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0: source = "\(GstFileSrc\)\ source" Pipeline is PREROLLING ... Got context from element 'glimagesink0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0"; /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:src: caps = video/x-msvideo /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstTypeFindElement:typefind.GstPad:src: caps = "NULL" /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstMultiQueue:multiqueue0.GstPad:sink_0: caps = "video/x-h264\,\ variant\=\(string\)itu\,\ framerate\=\(fraction\)24/1\,\ width\=\(int\)1280\,\ height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1" /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstMultiQueue:multiqueue0.GstPad:sink_0: caps = "video/x-h264\,\ variant\=\(string\)itu\,\ framerate\=\(fraction\)24/1\,\ width\=\(int\)1280\,\ height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1" /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstMultiQueue:multiqueue0.GstPad:sink_1: caps = "audio/mpeg\,\ mpegversion\=\(int\)1\,\ layer\=\(int\)2\,\ rate\=\(int\)44100\,\ channels\=\(int\)2\,\ codec_data\=\(buffer\)020000ee020001000000010010000000000000000000" /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstMultiQueue:multiqueue0.GstPad:sink_1: caps = "audio/mpeg\,\ mpegversion\=\(int\)1\,\ layer\=\(int\)2\,\ rate\=\(int\)44100\,\ channels\=\(int\)2\,\ codec_data\=\(buffer\)020000ee020001000000010010000000000000000000" /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstMpegAudioParse:mpegaudioparse0.GstPad:sink: caps = "audio/mpeg\,\ mpegversion\=\(int\)1\,\ layer\=\(int\)2\,\ rate\=\(int\)44100\,\ channels\=\(int\)2\,\ codec_data\=\(buffer\)020000ee020001000000010010000000000000000000" /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstMpegAudioParse:mpegaudioparse0.GstPad:sink: caps = "audio/mpeg\,\ mpegversion\=\(int\)1\,\ layer\=\(int\)2\,\ rate\=\(int\)44100\,\ channels\=\(int\)2\,\ codec_data\=\(buffer\)020000ee020001000000010010000000000000000000" /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstH264Parse:h264parse0.GstPad:src: caps = "video/x-h264\,\ variant\=\(string\)itu\,\ framerate\=\(fraction\)24/1\,\ width\=\(int\)1280\,\ height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ parsed\=\(boolean\)true\,\ stream-format\=\(string\)byte-stream\,\ alignment\=\(string\)au" /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstMad:mad0.GstPad:sink: caps = "audio/mpeg\,\ mpegversion\=\(int\)1\,\ mpegaudioversion\=\(int\)1\,\ layer\=\(int\)2\,\ rate\=\(int\)44100\,\ channels\=\(int\)2\,\ parsed\=\(boolean\)true" /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstMpegAudioParse:mpegaudioparse0.GstPad:src: caps = "audio/mpeg\,\ mpegversion\=\(int\)1\,\ mpegaudioversion\=\(int\)1\,\ layer\=\(int\)2\,\ rate\=\(int\)44100\,\ channels\=\(int\)2\,\ parsed\=\(boolean\)true" /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstMad:mad0.GstPad:src: caps = "audio/x-raw\,\ format\=\(string\)S32LE\,\ layout\=\(string\)interleaved\,\ rate\=\(int\)44100\,\ channels\=\(int\)2\,\ channel-mask\=\(bitmask\)0x0000000000000003" Redistribute latency... /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/avdec_h264:avdec_h264-0.GstPad:sink: caps = "video/x-h264\,\ variant\=\(string\)itu\,\ framerate\=\(fraction\)24/1\,\ width\=\(int\)1280\,\ height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ parsed\=\(boolean\)true\,\ stream-format\=\(string\)byte-stream\,\ alignment\=\(string\)au" /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0.GstPad:src: caps = "video/x-h264\,\ variant\=\(string\)itu\,\ framerate\=\(fraction\)24/1\,\ width\=\(int\)1280\,\ height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ parsed\=\(boolean\)true\,\ stream-format\=\(string\)byte-stream\,\ alignment\=\(string\)au" /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0.GstPad:sink: caps = "video/x-h264\,\ variant\=\(string\)itu\,\ framerate\=\(fraction\)24/1\,\ width\=\(int\)1280\,\ height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ parsed\=\(boolean\)true\,\ stream-format\=\(string\)byte-stream\,\ alignment\=\(string\)au" /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/avdec_h264:avdec_h264-0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)1280\,\ height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ chroma-site\=\(string\)mpeg2\,\ colorimetry\=\(string\)bt709\,\ framerate\=\(fraction\)24/1" /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0.GstGhostPad:src_1.GstProxyPad:proxypad5: caps = "audio/x-raw\,\ format\=\(string\)S32LE\,\ layout\=\(string\)interleaved\,\ rate\=\(int\)44100\,\ channels\=\(int\)2\,\ channel-mask\=\(bitmask\)0x0000000000000003" /GstPipeline:pipeline0/GstGLVideoMixer:glvideomixer0.GstGLVideoMixerPad:sink_0: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)1280\,\ height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ chroma-site\=\(string\)mpeg2\,\ colorimetry\=\(string\)bt709\,\ framerate\=\(fraction\)24/1" /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0.GstGhostPad:src_0.GstProxyPad:proxypad4: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)1280\,\ height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ chroma-site\=\(string\)mpeg2\,\ colorimetry\=\(string\)bt709\,\ framerate\=\(fraction\)24/1" /GstPipeline:pipeline0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0.GstDecodePad:src_0.GstProxyPad:proxypad2: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)1280\,\ height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ chroma-site\=\(string\)mpeg2\,\ colorimetry\=\(string\)bt709\,\ framerate\=\(fraction\)24/1" /GstPipeline:pipeline0/GstGLVideoMixer:glvideomixer0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)RGBA\,\ width\=\(int\)1280\,\ height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)1:1:0:0\,\ framerate\=\(fraction\)24/1" /GstPipeline:pipeline0/GstNavSeek:navseek0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)RGBA\,\ width\=\(int\)1280\,\ height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)1:1:0:0\,\ framerate\=\(fraction\)24/1" /GstPipeline:pipeline0/GstGLImageSink:glimagesink0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)RGBA\,\ width\=\(int\)1280\,\ height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)1:1:0:0\,\ framerate\=\(fraction\)24/1" /GstPipeline:pipeline0/GstNavSeek:navseek0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)RGBA\,\ width\=\(int\)1280\,\ height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)1:1:0:0\,\ framerate\=\(fraction\)24/1" Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock Got EOS from element "pipeline0". Execution ended after 0:00:03.119550442 Setting pipeline to PAUSED ... Setting pipeline to READY ... Setting pipeline to NULL ... Freeing pipeline ...
Can anyone shed any light on this? The bug's a complete blocker for my application. I know it's in plugins-bad, but I'd appreciate a fix.
If you seek on the pipeline it works. Not if you seek on uridecodebin. This seems to be due to the segment handling in aggregator. As a result of the seek on uridecodebin, a new segment is sent to the mixer which in videomixer's case results in a reset of the segment position. For compositor/glvideomixer, the position is not reset and ends up outside the segment bounds causing videoaggregator to drop the buffer. Another interesting point is that aggregator does not seem to get the seek event when one seeks on uridecodebin instead of the pipeline. I have no idea how this is supposed to work :)
Created attachment 286767 [details] glvideomixer-bug-example.py Updated example that does not throw a TypeError and seeks on the pipeline instead of uridecodebin
Created attachment 286773 [details] glvideomixer dynamically adding pads This example (from bug #735790) dynamically adds pads to videoaggregator. When a new source is added after videoaggregator has been running for a while, the source has to send enough buffers to match with videoaggregator's current time before any of the buffers are displayed. All of the previous buffers are dropped as being "too late". As a result there is a delay proportional to the running time between adding the new source and when the next videoaggregator output buffer is pushed out.
Created attachment 286775 [details] [review] aggregator: Update sinkpads segment position when receiving a new segment
Created attachment 286777 [details] [review] aggregator: Update sinkpads segment position when receiving a new segment https://bugzilla.gnome.org/show_bug.cgi?id=734499 Conflicts: gst-libs/gst/base/gstaggregator.c
Here is the difference in the segment info between videomixer and videoaggregator around the point of the seek. videomixer: 0:00:04.062967418 5907 0x7f39f40069e0 ERROR videomixer videomixer2.c:1266:gst_videomixer2_collected:<videomixer> timing out start 0:00:09.791666667 out end 0:00:09.833333333 segment time segment start=0:00:00.000000000, offset=0:00:00.000000000, stop=99:99:99.999999999, rate=1.000000, applied_rate=1.000000, flags=0x00, time=0:00:00.000000000, base=0:00:00.000000000, position 0:00:09.791666667, duration 99:99:99.999999999 0:00:04.079718523 5907 0x7f39f40069e0 ERROR videomixer videomixer2.c:1266:gst_videomixer2_collected:<videomixer> timing out start 0:00:09.833333333 out end 0:00:09.875000000 segment time segment start=0:00:00.000000000, offset=0:00:00.000000000, stop=99:99:99.999999999, rate=1.000000, applied_rate=1.000000, flags=0x00, time=0:00:00.000000000, base=0:00:00.000000000, position 0:00:09.833333333, duration 99:99:99.999999999 seeking 0:00:04.165913261 5907 0x7f39f40069e0 ERROR videomixer videomixer2.c:1266:gst_videomixer2_collected:<videomixer> timing out start 0:00:00.000000000 out end 0:00:00.041666667 segment time segment start=0:00:00.000000000, offset=0:00:00.000000000, stop=99:99:99.999999999, rate=1.000000, applied_rate=1.000000, flags=0x00, time=0:00:00.000000000, base=0:00:00.000000000, position 99:99:99.999999999, duration 99:99:99.999999999 0:00:04.180805716 5907 0x7f39f40069e0 ERROR videomixer videomixer2.c:1266:gst_videomixer2_collected:<videomixer> timing out start 0:00:00.041666667 out end 0:00:00.083333333 segment time segment start=0:00:00.000000000, offset=0:00:00.000000000, stop=99:99:99.999999999, rate=1.000000, applied_rate=1.000000, flags=0x00, time=0:00:00.000000000, base=0:00:00.000000000, position 0:00:00.041666667, duration 99:99:99.999999999 videoaggregator: 0:00:04.567522244 5875 0x2266cf0 ERROR videoaggregator gstvideoaggregator.c:1205:gst_videoaggregator_aggregate:<glvideomixer> timing out start 0:00:09.750000000 out end 0:00:09.791666667 segment time segment start=0:00:00.000000000, offset=0:00:00.000000000, stop=99:99:99.999999999, rate=1.000000, applied_rate=1.000000, flags=0x00, time=0:00:00.000000000, base=0:00:00.000000000, position 0:00:09.750000000, duration 99:99:99.999999999 0:00:04.584520276 5875 0x2266cf0 ERROR videoaggregator gstvideoaggregator.c:1205:gst_videoaggregator_aggregate:<glvideomixer> timing out start 0:00:09.791666667 out end 0:00:09.833333333 segment time segment start=0:00:00.000000000, offset=0:00:00.000000000, stop=99:99:99.999999999, rate=1.000000, applied_rate=1.000000, flags=0x00, time=0:00:00.000000000, base=0:00:00.000000000, position 0:00:09.791666667, duration 99:99:99.999999999 seeking 0:00:04.664475698 5875 0x2266cf0 ERROR videoaggregator gstvideoaggregator.c:1205:gst_videoaggregator_aggregate:<glvideomixer> timing out start 0:00:09.833333333 out end 0:00:09.875000000 segment time segment start=0:00:00.000000000, offset=0:00:00.000000000, stop=99:99:99.999999999, rate=1.000000, applied_rate=1.000000, flags=0x00, time=0:00:00.000000000, base=0:00:00.000000000, position 0:00:09.833333333, duration 99:99:99.999999999 0:00:04.681407947 5875 0x2266cf0 ERROR videoaggregator gstvideoaggregator.c:1205:gst_videoaggregator_aggregate:<glvideomixer> timing out start 0:00:09.833333333 out end 0:00:09.875000000 segment time segment start=0:00:00.000000000, offset=0:00:00.000000000, stop=99:99:99.999999999, rate=1.000000, applied_rate=1.000000, flags=0x00, time=0:00:00.000000000, base=0:00:00.000000000, position 0:00:09.833333333, duration 99:99:99.999999999
Thanks for responding so fast! The attached patch didn't seem to resolve the issue, though, do I need to change my code? My application needs to seek uridecodebins individually, so seeking the pipeline won't work.
Why do you need to seek the decodebin individually? You should also seek the mixer so that it knows what segment it is working with.
Comment on attachment 286777 [details] [review] aggregator: Update sinkpads segment position when receiving a new segment Why not just set it to GST_CLOCK_TIME_NONE if unknown?
(In reply to comment #16) > (From update of attachment 286777 [details] [review]) > Why not just set it to GST_CLOCK_TIME_NONE if unknown? Not sure, I just took the code from collectpads. I think that the position setting is not even the issue here.
What *is* the issue then? :)
(In reply to comment #18) > What *is* the issue then? :) The issue is: https://bugzilla.gnome.org/show_bug.cgi?id=734499#c8 but the reason of the issue is unknown afaict
Right, so the difference is that the segment position is not set to -1 on FLUSH_STOP like in videomixer2.c:1951. The aggregator equivalent, gst_videoaggregator_flush is never called, presumambly because priv->flush_seeking is FALSE due to there being no seek on the pad.
Created attachment 287809 [details] [review] (In reply to comment #20) > Right, so the difference is that the segment position is not set to -1 on > FLUSH_STOP like in videomixer2.c:1951. The aggregator equivalent, > gst_videoaggregator_flush is never called, presumambly because > priv->flush_seeking is FALSE due to there being no seek on the pad. So in videomixer we concider that we are flushed, whenever a sinkpad is flushed, that sounds conceptually weird to me and I wonder if it is what we actually want. I attach a patch here that you can check to see if it actually fixes your issue.
There's two flush situations to consider here. If just a sinkpad decides to flush you would immediately unblock its streaming thread and throw away all pending data and timing information, and accept data again after flush-stop. This would not flush the complete aggregator. If you receive flush events on the srcpad you would flush all sinkpads and would consider the aggregator flushed after all flush-start events are sent and then all flush-stops are sent. If you're seeking you would flush all sinkpads and downstream, and would consider the complete mixer flushed when all flush-start events are sent and then all flush-stops afterwards. Does this make sense and helps here?
(In reply to comment #22) > There's two flush situations to consider here. > > If just a sinkpad decides to flush you would immediately unblock its streaming > thread and throw away all pending data and timing information, and accept data > again after flush-stop. This would not flush the complete aggregator. > > If you receive flush events on the srcpad you would flush all sinkpads and > would consider the aggregator flushed after all flush-start events are sent and > then all flush-stops are sent. > > If you're seeking you would flush all sinkpads and downstream, and would > consider the complete mixer flushed when all flush-start events are sent and > then all flush-stops afterwards. > > > Does this make sense and helps here? What you describe here is exactly what we do. The issue is when we seek only the source (ie. the aggregator never sees the seek event), and thus does not recalculate it position, etc... thus the bug we are talking about here. I tend to think seeking only upstream elements is not really correct, what do you think?
Only seeking upstream elements is ok and you should not do anything in aggregator then. If the upstream seek is non-flushing you won't even notice that something has happened, and whoever did the seek needs to make sure that running times after the seek are how they're expected to be. If the seek was flushing you will have to flush your sinkpad and reset all timing information, but that's it... again whoever did the seek needs to make sure that running times after the seek are as expected.
@Isaac Smith, ok so in your case you should make sure that running times are coherent so that videoaggregator can properly operate on the buffers that gets in. The gst_pad_set_offset API[0] has been implemented exactly for that kind of use case. [0]the http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstPad.html#gst-pad-set-offset