GNOME Bugzilla – Bug 759470
input-selector element generates wrong DTS timestamps with sync-mode=0 and multiple rtspsrc (same video format/framerates)
Last modified: 2018-11-03 12:31:19 UTC
input-selector: it switches to active live video stream (rtspsrc) without checking for keyframes. This causes input-selector to miss keyframe from rtspsrc and this causes additional delay (video is paused for 1second) during camera switching using input-selector. Test case: I have 5 ip cameras (all are rtspsrc element) and one audio src (rtspsrc) All 5 ip cameras are connected with input-selector element and it switched from one camera to another camera. My code works perfectly fine [1] : http://hastebin.com/raw/adatujewes But there is one flaw in streaming. I am seeing a pause for 1 second whenever input-selector is switching between camera. Look at timestamp 1min 12sec on my streaming video url which is using my binary generated from code [1]: [2] https://www.youtube.com/watch?v=NPlN_3YhmrM You will see cameras are switched every 5 seconds after 1:12 on that url [2] However, there is a pause of 1 second when camera switching happens, I tried with changing delay of queue/ replacing queue with queue2/ making rtspsec latency=0 . But nothing helps me . Proposed Solution: Fix input-selector element so that it can wait for keyframes before switching to live stream. This will give gapless video switching from input-selector element.
Created attachment 317394 [details] test program to show delay of 1 second using input-selector element This the test program which i wrote for camera switching using input-selector. I also tried with queue2/cache-buffer/sync-mode but nothing helps me to achieve zero delay during cam switch. input-selector is unusable for rtspsrc as it cannot do gapless video playback anymore.
i am new to gstreamer . i was looking into code : [1] http://cgit.freedesktop.org/gstreamer/gstreamer/tree/plugins/elements/gstinputselector.c#n1065 . this seems to be chain function . I am thinking adding PAD PROBE BLOCK in [1] and wait till i see keyframe from active pad. is that sounds good to any expert here ?
The easiest way to implement this would be to just stay at the currently selected pad and continue as normal (that is, dropping buffers on other pads) until the to-be-selected pad gets its first buffer without DELTA_UNIT flag. And then you would do the switch from the currently active pad to the new pad.
Switching between this and the current behaviour should be done with a property btw, and the current behaviour has to stay the default.
(In reply to Sebastian Dröge (slomo) from comment #3) > The easiest way to implement this would be to just stay at the currently > selected pad and continue as normal (that is, dropping buffers on other > pads) until the to-be-selected pad gets its first buffer without DELTA_UNIT > flag. And then you would do the switch from the currently active pad to the > new pad. Could you please tell me which function is checking buffers for selected /non-selected pad in http://cgit.freedesktop.org/gstreamer/gstreamer/tree/plugins/elements/gstinputselector.c ? My guess is the chain function i.e. [1] but not sure. [1] http://cgit.freedesktop.org/gstreamer/gstreamer/tree/plugins/elements/gstinputselector.c#n1065
The chain function is where all buffers are arriving inside the element, yes.
(In reply to Sebastian Dröge (slomo) from comment #6) > The chain function is where all buffers are arriving inside the element, yes. I am thinking to check for DELTA_UNIT inside this if code block [1] if (active_sinkpad == pad) { if(// my code for checking delta unit flag) goto ignore; selpad->pushed = TRUE; } [1] http://cgit.freedesktop.org/gstreamer/gstreamer/tree/plugins/elements/gstinputselector.c#n1118 Please suggest if this is the correct place to do it or not.
I think the check/switching should happen around http://cgit.freedesktop.org/gstreamer/gstreamer/tree/plugins/elements/gstinputselector.c#n988
(In reply to Sebastian Dröge (slomo) from comment #8) > I think the check/switching should happen around > http://cgit.freedesktop.org/gstreamer/gstreamer/tree/plugins/elements/ > gstinputselector.c#n988 Thanks. here is the code which i am going to put soon just below line #988: active_sinkpad = gst_input_selector_get_active_sinkpad (sel); // line 988 if (active_sinkpad == pad) { //line 989 // this pad is active sink pad so check for DELTA UNIT and discard. // buf is passed as pointer of GstBuf in arguments to this chain func. // so check for DELTA UNIT flag in buf. if (// buf has DELTA UNIT) goto ignore; } Can I re-use ignore code block in that chain function to ignore this buffer ??? Please take a look into my above code and suggest me if my concept is wrong.
(In reply to Tapas Kumar Kundu from comment #9) > (In reply to Sebastian Dröge (slomo) from comment #8) > > I think the check/switching should happen around > > http://cgit.freedesktop.org/gstreamer/gstreamer/tree/plugins/elements/ > > gstinputselector.c#n988 > > Thanks. here is the code which i am going to put soon just below line #988: > > active_sinkpad = gst_input_selector_get_active_sinkpad (sel); // line 988 > if (active_sinkpad == pad) { //line 989 > // this pad is active sink pad so check for DELTA UNIT and discard. > // buf is passed as pointer of GstBuf in arguments to this chain func. > // so check for DELTA UNIT flag in buf. > if (// buf has DELTA UNIT) > goto ignore; > } This will make it stall anyway, you need to keep pushing buffers from the previous active pad. This code will drop buffers until the next keyframe which cause a stall just like it does currently. Here are few steps 1) Add a new enum property with 2 modes: immediate and next-keyframe to select the switching behavior. There should be plenty of examples in element's code adding enum properties, this is done on class_init() 2) Check the set_active_pad() that is the function that switches the active_pad value. In your case you don't want that to happen until the to-be-activated pad has a non-delta-unit buffer. 3) Keep in mind that the _chain() function is called for every buffer arriving on all of the pads (active or inactive). You need to check if it is the pad to be activated and check if the buffer is a non-delta, and then activate it, if it is in your new operation mode.
(In reply to Thiago Sousa Santos from comment #10) > (In reply to Tapas Kumar Kundu from comment #9) > > (In reply to Sebastian Dröge (slomo) from comment #8) > > > I think the check/switching should happen around > > > http://cgit.freedesktop.org/gstreamer/gstreamer/tree/plugins/elements/ > > > gstinputselector.c#n988 > > > > Thanks. here is the code which i am going to put soon just below line #988: > > > > active_sinkpad = gst_input_selector_get_active_sinkpad (sel); // line 988 > > if (active_sinkpad == pad) { //line 989 > > // this pad is active sink pad so check for DELTA UNIT and discard. > > // buf is passed as pointer of GstBuf in arguments to this chain func. > > // so check for DELTA UNIT flag in buf. > > if (// buf has DELTA UNIT) > > goto ignore; > > } > > This will make it stall anyway, you need to keep pushing buffers from the > previous active pad. This code will drop buffers until the next keyframe > which cause a stall just like it does currently. No it should not . Chain function is invoked for both active and inactive pads and i am checking whether this is invoked for active pad or not. If it is invoked on active pad and then only I am dropping for 'delta unit' > > 2) Check the set_active_pad() that is the function that switches the > active_pad value. In your case you don't want that to happen until the > to-be-activated pad has a non-delta-unit buffer. > If we do it in this way then the code which will use gst-input-selector, needs to activate same pad again and again untill it sees for activation is success. Please correct me if I am wrong. Thanks for your help
(In reply to Tapas Kumar Kundu from comment #11) > (In reply to Thiago Sousa Santos from comment #10) > > (In reply to Tapas Kumar Kundu from comment #9) > > > (In reply to Sebastian Dröge (slomo) from comment #8) > > > > I think the check/switching should happen around > > > > http://cgit.freedesktop.org/gstreamer/gstreamer/tree/plugins/elements/ > > > > gstinputselector.c#n988 > > > > > > Thanks. here is the code which i am going to put soon just below line #988: > > > > > > active_sinkpad = gst_input_selector_get_active_sinkpad (sel); // line 988 > > > if (active_sinkpad == pad) { //line 989 > > > // this pad is active sink pad so check for DELTA UNIT and discard. > > > // buf is passed as pointer of GstBuf in arguments to this chain func. > > > // so check for DELTA UNIT flag in buf. > > > if (// buf has DELTA UNIT) > > > goto ignore; > > > } > > > > This will make it stall anyway, you need to keep pushing buffers from the > > previous active pad. This code will drop buffers until the next keyframe > > which cause a stall just like it does currently. > > No it should not . Chain function is invoked for both active and inactive > pads and i am checking whether this is invoked for active pad or not. If it > is invoked on active pad and then only I am dropping for 'delta unit' If you switch the pad immediately you will stop pushing buffers on the old pad and downstream will receive no buffers until you get a keyframe, so yes. You get a stall just like it happens today. You need to current active pad to keep pushing buffers until the one you want to activate gets a keyframe. > > > > 2) Check the set_active_pad() that is the function that switches the > > active_pad value. In your case you don't want that to happen until the > > to-be-activated pad has a non-delta-unit buffer. > > > > If we do it in this way then the code which will use gst-input-selector, > needs to activate same pad again and again untill it sees for activation is > success. No, you will store some state that will mean that you are switching on the next keyframe and that will happen in the _chain() function once a non delta buffer arrives. > > Please correct me if I am wrong. Thanks for your help
(In reply to Thiago Sousa Santos from comment #12) > > If you switch the pad immediately you will stop pushing buffers on the old > pad and downstream will receive no buffers until you get a keyframe, so yes. > You get a stall just like it happens today. You need to current active pad > to keep pushing buffers until the one you want to activate gets a keyframe. > > > No, you will store some state that will mean that you are switching on the > next keyframe and that will happen in the _chain() function once a non delta > buffer arrives. > I really appreicate your help on this. So I will set a flag whenever set_active_pad is called. chain function will check for this flag and if the flag is set then it will try to detect keyframe on to-be-activeted-pad . When keyframe comes for to-be-activated-pad, chain function will signal set_active_pad() function to actually set active pad. One question: Are chain function and set_active_pad run by two different threads ?
(In reply to Tapas Kumar Kundu from comment #13) > (In reply to Thiago Sousa Santos from comment #12) > > > > If you switch the pad immediately you will stop pushing buffers on the old > > pad and downstream will receive no buffers until you get a keyframe, so yes. > > You get a stall just like it happens today. You need to current active pad > > to keep pushing buffers until the one you want to activate gets a keyframe. > > > > > > No, you will store some state that will mean that you are switching on the > > next keyframe and that will happen in the _chain() function once a non delta > > buffer arrives. > > > > I really appreicate your help on this. So I will set a flag whenever > set_active_pad is called. chain function will check for this flag and if the > flag is set then it will try to detect keyframe on to-be-activeted-pad . > When keyframe comes for to-be-activated-pad, chain function will signal > set_active_pad() function to actually set active pad. > > One question: > Are chain function and set_active_pad run by two different threads ? The set_active_pad() is called by the application thread because it is called from set_property(), the _chain() function is called from the streaming thread (a thread internal to the pipeline and created by gstreamer). For your change you will likely have to call the set_active_pad() from the chain function, there is no problem with that as long as you pay attention to proper locking/mutex.
Created attachment 317685 [details] [review] proposed patch which doesn't contain keyframe enum I will follow suggestion from comment #10 to add separate enum for "immediate switch" later on. This patch has all other logics which i implement from all suggestions given above. But unfortunately, I am still seeing <1 sec delay between camera switch. Delay seems to be improved but it is still there. I tried with input-selector CACHE_BUFFERS = false/true (both) and sync_mode=1. Any idea what can cause this delay even after having this patch ?
<thiagoss> tapas, do you understand why it still takes 1s to switch? Your patch makes it wait for a keyframe, so it will only switch when it gets a keyframe. Which likely happens 1s after you request the switch. That said, it doesn't mean that your patch isn't useful, it is. It just doesn't seem to do what you'd expect because it still needs to wait before switching. To switch immediately you'd need to keep the last keyframe and all the buffers after it for every pad so you could switch immediately when requested. This could be a 3rd operation mode (requires more memory).
(In reply to Thiago Sousa Santos from comment #16) > <thiagoss> tapas, do you understand why it still takes 1s to switch? Your > patch makes it wait for a keyframe, so it will only switch when it gets a > keyframe. Which likely happens 1s after you request the switch. > > That said, it doesn't mean that your patch isn't useful, it is. It just > doesn't seem to do what you'd expect because it still needs to wait before > switching. To switch immediately you'd need to keep the last keyframe and > all the buffers after it for every pad so you could switch immediately when > requested. This could be a 3rd operation mode (requires more memory). Here the debug log from input-selector element: https://drive.google.com/file/d/0B1cSMS8_GuAEeVM5MV9FOVlSZzA/view?usp=sharing You can search for "startcam" to see exact line where camera switch happens from application side. This is generated with proposed patch on git master. I really need some help to understand where the delay is occurring inside input-selector element during cam switch. Could you please take a look at this log ?
(In reply to Tapas Kumar Kundu from comment #17) > Here the debug log from input-selector element: > https://drive.google.com/file/d/0B1cSMS8_GuAEeVM5MV9FOVlSZzA/view?usp=sharing > > You can search for "startcam" to see exact line where camera switch happens > from application side. This is generated with proposed patch on git master. > > I really need some help to understand where the delay is occurring inside > input-selector element during cam switch. > > Could you please take a look at this log ? It seems that something is causing delay because of sticky events . This is just a guess by looking into time stamps of log during cam switch. Can anyone explain it to me if this can cause delay or not ? 0:00:23.073715429 3480 0x7f3db427f630 LOG input-selector gstinputselector.c:986:gst_selector_pad_chain:<video_switch:sink_0> keyframe found on to-be-activated-pad 0:00:23.073739489 3480 0x7f3db427f630 DEBUG input-selector gstinputselector.c:1366:gst_input_selector_set_active_pad:<video_switch> setting active pad to video_switch:sink_0 0:00:23.073891880 3480 0x7f3db427f630 DEBUG input-selector gstinputselector.c:1386:gst_input_selector_set_active_pad:<video_switch> New active pad is <video_switch:sink_0> 0:00:23.073925025 3480 0x7f3db427f630 LOG input-selector gstinputselector.c:997:gst_selector_pad_chain:<video_switch:sink_0> getting active pad 0:00:23.073949517 3480 0x7f3db427f630 LOG input-selector gstinputselector.c:1066:gst_selector_pad_chain:<video_switch:sink_0> received start time 0:00:22.940258943 0:00:23.074018915 3480 0x7f3db427f630 DEBUG input-selector gstinputselector.c:433:forward_sticky_events:<video_switch:sink_0> forward sticky event stream-start event: 0x7f3d4c004de0, time 99:99:99.999999999, seq-num 681, GstEventStreamStart, stream-id=(string)f980ae85b03cfb27e99e1103b54cdb1df48678ec51f0ca4e83e6082a53bf7be2, flags=(GstStreamFlags)GST_STREAM_FLAG_NONE, group-id=(uint)14; 0:00:23.073955002 3480 0x7f3db420ea30 DEBUG input-selector gstinputselector.c:978:gst_selector_pad_chain:<video_switch:sink_3> entering chain for buf 0x7f3d5800b960 with timestamp 0:00:22.990433666 =========================================================================== =========================================================================== 0:00:23.076769731 3480 0x7f3db427f680 DEBUG input-selector gstinputselector.c:804:gst_input_selector_wait_running_time:<video_switch:sink_2> Waiting for active streams to advance. 0:00:22.951005667 >= 0:00:22.940622043 0:00:23.116405692 3480 0x7f3db427f630 DEBUG input-selector gstinputselector.c:433:forward_sticky_events:<video_switch:sink_0> forward sticky event caps event: 0x7f3d8401a670, time 99:99:99.999999999, seq-num 876, GstEventCaps, caps=(GstCaps)"video/x-h264\,\ stream-format\=\(string\)avc\,\ alignment\=\(string\)au\,\ codec_data\=\(buffer\)0164001effe100196764001eac34cc0b03dff016a0c0c0c800001f480004e2042001000468ee3c30\,\ level\=\(string\)3\,\ profile\=\(string\)high"; 0:00:23.116679346 3480 0x7f3db427f630 DEBUG input-selector gstinputselector.c:433:forward_sticky_events:<video_switch:sink_0> forward sticky event segment event: 0x7f3d8401a6e0, time 99:99:99.999999999, seq-num 877, GstEventSegment, segment=(GstSegment)"GstSegment, flags=(GstSegmentFlags)GST_SEGMENT_FLAG_NONE, rate=(double)1, applied-rate=(double)1, format=(GstFormat)GST_FORMAT_TIME, base=(guint64)548821523, offset=(guint64)0, start=(guint64)548821523, stop=(guint64)18446744073709551615, time=(guint64)0, position=(guint64)548821523, duration=(guint64)18446744073709551615;"; 0:00:23.116889218 3480 0x7f3db427f630 DEBUG input-selector gstinputselector.c:1109:gst_selector_pad_chain:<video_switch:sink_0> Marking discont buffer 0x7f3d380bd7f0 0:00:23.116962912 3480 0x7f3db427f630 LOG input-selector gstinputselector.c:1116:gst_selector_pad_chain:<video_switch:sink_0> Forwarding buffer 0x7f3d380bd7f0 with timestamp 0:00:22.940258943 0:00:23.117031265 3480 0x7f3db427f540 LOG input-selector gstinputselector.c:1122:gst_selector_pad_chain:<video_switch:sink_1> Buffer 0xd8edd0 forwarded result=0
(In reply to Tapas Kumar Kundu from comment #18) > (In reply to Tapas Kumar Kundu from comment #17) > > Here the debug log from input-selector element: > > https://drive.google.com/file/d/0B1cSMS8_GuAEeVM5MV9FOVlSZzA/view?usp=sharing > > > > You can search for "startcam" to see exact line where camera switch happens > > from application side. This is generated with proposed patch on git master. > > > > I really need some help to understand where the delay is occurring inside > > input-selector element during cam switch. > > > > Could you please take a look at this log ? > > It seems that something is causing delay because of sticky events . This is > just a guess by looking into time stamps of log during cam switch. > > Can anyone explain it to me if this can cause delay or not ? > > I modified gstinputselector.c not to pass any sticky events when camera switch happens but i still same delay during camera switch. However, if I use proposed patch from comment #15 with sync-mode=0 then i can see zero delay camera switch but my pipeline every 10-15 times camera switch. Any ideas on this ? Why sync-mode=0 is not working with input-selector element with my patch ? Any help is appreciated.
Created attachment 317884 [details] dummy program with gives gapless cam switch with proposed patch and sync-mode=0 if I use proposed patch from comment #15 with this dummy test program WITH *sync-mode=0* then i can see zero delay camera switch but my pipeline gives "rtmp send error" after every 10-15 times camera switch. Any ideas on this ? Why sync-mode=0 is not working with input-selector element with my patch ? I tried by changing queue length to infinite and adding more queues but nothing helped. I am stuck here. Any help is appreciated.
Created attachment 317976 [details] Input selector element generates wrong DTS sequence with sync_mode=0 for rtspsrc I generated flv video using proposed patch and saved it to disk using filesink instead of rtmpsink. Then I tried to play saved flv file with ffmpeg. I found that ffmpeg is detecting wrong DTS timestamps every ~5sec . Interestingly, input selector element also changes rtspsrc ip cameras every 5 second. This means that input-selector element is generating wrong DTS timestamp for every cam switch which is causing rtmpsink element to behave abnormally. Please note that test patch which i uploaded in this bug id has nothing to do with inocrrect timestmaps as it is just waiting for correct keyframe before triggering cam switch inside input-selector. (see attachment #317685 [details]) . So this must be a bug for input-selector element which makes it unusable with sync-mode=0 with multiple rtspsrc elements with same video format/framerate . And sync-mode=0 is the only way to get gapless video for rtspsrc from input-selector. here is the generated flv video from input-selector element which i used to detect wrong DTS using ffmpeg. https://drive.google.com/file/d/0B1cSMS8_GuAEbWtMdFFocjZVOGM/view?usp=sharing
Bug id: 759895 and 759862 may be possible duplicate of this bug because they are also using input-selector with sync-mode=0 . But this is not confirmed yet. It is a guess.
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gstreamer/issues/144.