GNOME Bugzilla – Bug 776491
multiqueue: Update buffering level based on the lowest value
Last modified: 2018-05-06 14:46:21 UTC
Depending on use cases, application might want to stop buffering only if all queues hit the high threshold
Created attachment 342471 [details] [review] multiqueue: Calculate buffering of each queue only when it is updated Store each buffering level of singlequeue in itself
Created attachment 342472 [details] [review] multiqueue: Update buffering level based on the lowest value
Created attachment 342473 [details] [review] multiqueue: Update buffering level based on the lowest value
Created attachment 344623 [details] [review] multiqueue: Adjust buffering level to MAX if SQ's srcpad is not linked Dear All. When I changed track so quickly and repeatedly, buffering message is not posted as well. After track is changed, new activated slot still on GST_FLOW_NOT_LINKED status. Because, old pushed buffer is returned lately to the activated slot's output and non activated slot still be accepted as activated slot. So, it is better to use gst_pad_is_linked () instead of GST_FLOW_NOT_LINKED. Thanks.
Created attachment 344624 [details] [review] multiqueue: Update buffering level based on the lowest value Update more not-linked handling
(In reply to Seungha Yang from comment #1) > Created attachment 342471 [details] [review] [review] > multiqueue: Calculate buffering of each queue only when it is updated > > Store each buffering level of singlequeue in itself Looks good (In reply to HoonHee Lee from comment #4) > Created attachment 344623 [details] [review] [review] > multiqueue: Adjust buffering level to MAX if SQ's srcpad is not linked > > When I changed track so quickly and repeatedly, buffering message is not > posted as well. > > After track is changed, new activated slot still on GST_FLOW_NOT_LINKED > status. This shouldn't happen. When a pad is relinked it should get a GST_EVENT_RECONFIGURE, which switches srcresult to GST_FLOW_OK. > Because, old pushed buffer is returned lately to the activated slot's output > and non activated slot still be accepted as activated slot. > > So, it is better to use gst_pad_is_linked () instead of GST_FLOW_NOT_LINKED. > gst_pad_is_linked() should be avoided because it's expensive and it doesn't give you a full downstream visibility (there might be a change in an element/pad further downstream).
Created attachment 344674 [details] Add dot graph.
Created attachment 344675 [details] Add log for GST Hello Edward Hervey. Thanks for your comments. You are right. srcresult can be switched to GST_FLOW_OK by GST_EVENT_RECONFIGURE. ===================================================================== gst_multi_queue_src_event (GstPad * pad, GstObject * parent, GstEvent * event) { switch (GST_EVENT_TYPE (event)) { case GST_EVENT_RECONFIGURE: GST_MULTI_QUEUE_MUTEX_LOCK (mq); if (sq->srcresult == GST_FLOW_NOT_LINKED) { sq->srcresult = GST_FLOW_OK; g_cond_signal (&sq->turn); } ... } ===================================================================== For your better understanding, I want to show some logs for MQ and decodebin3. Actually, I added some logs for debug. I switched audio track from 2nd to 1st. (multiqueue0:src_1 ==> multiqueue0:src_0) After track is changed, we expect that srcresult of multiqueue0:src_1 is not-linked but, it still ok. Thus, we missed the opportunity to post buffering message even if singlequeue0 and singlequeue2 are full. ========================================================================== 0:00:40.856890106 ^[[334m 1641^[[00m 0x24e90a0 ^[[33;01mWARN ^[[00m ^[[00m decodebin3 gstdecodebin3.c:2665:handle_stream_switch:<decodebin3-0>^[[00m ##### Add an idle probe slot(multiqueue0:src_1) 0:00:40.856956565 ^[[334m 1641^[[00m 0x24e90a0 ^[[33;01mWARN ^[[00m ^[[00m decodebin3 gstdecodebin3.c:2374:reassign_slot:<multiqueue0:src_1>^[[00m ##### Unlinking from decoder 0x7fcc18042930 0:00:40.857018106 ^[[334m 1641^[[00m 0x24e90a0 ^[[33;01mWARN ^[[00m ^[[00m decodebin3 gstdecodebin3.c:2408:reassign_slot:<multiqueue0:src_1>^[[00m ##### Assigning output to slot 0x7fcc1403cce0 'de164b257930960e29a470f6f526103a3e4bd64d14f30d2f226470756af34104/src_2:1/00000100' 0:00:40.857067472 ^[[334m 1641^[[00m 0x24e90a0 ^[[33;01mWARN ^[[00m ^[[00m decodebin3 gstdecodebin3.c:2289:idle_reconfigure:<multiqueue0:src_0>^[[00m ##### output : 0x7fcc0c001e50, slot (multiqueue0:src_0) 0:00:40.857128665 ^[[334m 1641^[[00m 0x24e90a0 ^[[33;01mWARN ^[[00m ^[[00m decodebin3 gstdecodebin3.c:2166:reconfigure_output_stream:<decodebin3-0>^[[00m ##### Reusing existing decoder. Re-linking pad (multiqueue0:src_0) to decoder (avdec_aac0:sink) 0:00:40.857180932 ^[[334m 1641^[[00m 0x24e90a0 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:2562:gst_multi_queue_src_event:<multiqueue0>^[[00m ##### Got reconfigure event on SingleQueue 0, result: ok 0:00:40.857271950 ^[[334m 1641^[[00m 0x7fcc18010ad0 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1185:update_buffering_by_minimum:<multiqueue0>^[[00m ##### sq: 0, oq->buffering_level: 1001244, mq->high_watermark: 100000, srcresult: ok 0:00:40.857308341 ^[[334m 1641^[[00m 0x7fcc18010ad0 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1185:update_buffering_by_minimum:<multiqueue0>^[[00m ##### sq: 1, oq->buffering_level: 0, mq->high_watermark: 100000, srcresult: ok 0:00:40.857327418 ^[[334m 1641^[[00m 0x7fcc18010ad0 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1185:update_buffering_by_minimum:<multiqueue0>^[[00m ##### sq: 2, oq->buffering_level: 906666, mq->high_watermark: 100000, srcresult: ok 0:00:40.857343096 ^[[334m 1641^[[00m 0x7fcc18010ad0 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1194:update_buffering_by_minimum:<multiqueue0>^[[00m ##### min_level: 0, percent: 0 ... 0:00:40.977973727 ^[[334m 1641^[[00m 0x7fcc28003000 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1185:update_buffering_by_minimum:<multiqueue0>^[[00m ##### sq: 0, oq->buffering_level: 1000533, mq->high_watermark: 100000, srcresult: ok 0:00:40.978006387 ^[[334m 1641^[[00m 0x7fcc28003000 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1185:update_buffering_by_minimum:<multiqueue0>^[[00m ##### sq: 1, oq->buffering_level: 0, mq->high_watermark: 100000, srcresult: ok 0:00:40.978020714 ^[[334m 1641^[[00m 0x7fcc28003000 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1185:update_buffering_by_minimum:<multiqueue0>^[[00m ##### sq: 2, oq->buffering_level: 918666, mq->high_watermark: 100000, srcresult: ok 0:00:40.978043957 ^[[334m 1641^[[00m 0x7fcc28003000 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1194:update_buffering_by_minimum:<multiqueue0>^[[00m ##### min_level: 0, percent: 0 ========================================================================== I switched audio track from 1st to 2nd. (multiqueue0:src_0 ==> multiqueue0:src_1) After track is changed, we expect that srcresult of multiqueue0:src_0 is not-linked but, it is ok. Also, we could miss the chance the post buffering message until srcresult of multiqueue0:src_0 switches to not-linked. =========================================================================== 0:00:41.007352954 ^[[334m 1641^[[00m 0x24e90a0 ^[[33;01mWARN ^[[00m ^[[00m decodebin3 gstdecodebin3.c:2665:handle_stream_switch:<decodebin3-0>^[[00m ##### Add an idle probe slot(multiqueue0:src_0) 0:00:41.468253076 ^[[334m 1641^[[00m 0x7fcc18010ad0 ^[[33;01mWARN ^[[00m ^[[00m decodebin3 gstdecodebin3.c:2374:reassign_slot:<multiqueue0:src_0>^[[00m ##### Unlinking from decoder 0x7fcc18042930 0:00:41.468326478 ^[[334m 1641^[[00m 0x7fcc18010ad0 ^[[33;01mWARN ^[[00m ^[[00m decodebin3 gstdecodebin3.c:2408:reassign_slot:<multiqueue0:src_0>^[[00m ##### Assigning output to slot 0x7fcc18040af0 'de164b257930960e29a470f6f526103a3e4bd64d14f30d2f226470756af34104/src_1:1/00000100' 0:00:41.468370636 ^[[334m 1641^[[00m 0x7fcc18010ad0 ^[[33;01mWARN ^[[00m ^[[00m decodebin3 gstdecodebin3.c:2289:idle_reconfigure:<multiqueue0:src_1>^[[00m ##### output : 0x7fcc0c001e50, slot (multiqueue0:src_1) 0:00:41.468403311 ^[[334m 1641^[[00m 0x7fcc18010ad0 ^[[33;01mWARN ^[[00m ^[[00m decodebin3 gstdecodebin3.c:2166:reconfigure_output_stream:<decodebin3-0>^[[00m ##### Reusing existing decoder. Re-linking pad (multiqueue0:src_1) to decoder (avdec_aac0:sink) 0:00:41.468429217 ^[[334m 1641^[[00m 0x7fcc18010ad0 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:2562:gst_multi_queue_src_event:<multiqueue0>^[[00m ##### Got reconfigure event on SingleQueue 1, result: ok 0:00:41.468531981 ^[[334m 1641^[[00m 0x7fcc1403cb70 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1185:update_buffering_by_minimum:<multiqueue0>^[[00m ##### sq: 0, oq->buffering_level: 1000533, mq->high_watermark: 100000, srcresult: ok 0:00:41.468559055 ^[[334m 1641^[[00m 0x7fcc1403cb70 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1185:update_buffering_by_minimum:<multiqueue0>^[[00m ##### sq: 1, oq->buffering_level: 0, mq->high_watermark: 100000, srcresult: ok 0:00:41.468577293 ^[[334m 1641^[[00m 0x7fcc1403cb70 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1185:update_buffering_by_minimum:<multiqueue0>^[[00m ##### sq: 2, oq->buffering_level: 934666, mq->high_watermark: 100000, srcresult: ok 0:00:41.468615526 ^[[334m 1641^[[00m 0x7fcc1403cb70 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1194:update_buffering_by_minimum:<multiqueue0>^[[00m ##### min_level: 0, percent: 0 0:00:41.468688566 ^[[334m 1641^[[00m 0x7fcc2004ba30 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1185:update_buffering_by_minimum:<multiqueue0>^[[00m ##### sq: 0, oq->buffering_level: 1000533, mq->high_watermark: 100000, srcresult: ok 0:00:41.468716409 ^[[334m 1641^[[00m 0x7fcc2004ba30 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1185:update_buffering_by_minimum:<multiqueue0>^[[00m ##### sq: 1, oq->buffering_level: 0, mq->high_watermark: 100000, srcresult: ok 0:00:41.468736625 ^[[334m 1641^[[00m 0x7fcc2004ba30 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1185:update_buffering_by_minimum:<multiqueue0>^[[00m ##### sq: 2, oq->buffering_level: 933333, mq->high_watermark: 100000, srcresult: ok ... 0:00:41.469435035 ^[[334m 1641^[[00m 0x2454400 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1185:update_buffering_by_minimum:<multiqueue0>^[[00m ##### sq: 0, oq->buffering_level: 1000533, mq->high_watermark: 100000, srcresult: ok 0:00:41.469470879 ^[[334m 1641^[[00m 0x2454400 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1185:update_buffering_by_minimum:<multiqueue0>^[[00m ##### sq: 1, oq->buffering_level: 1422, mq->high_watermark: 100000, srcresult: ok 0:00:41.469490929 ^[[334m 1641^[[00m 0x2454400 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1185:update_buffering_by_minimum:<multiqueue0>^[[00m ##### sq: 2, oq->buffering_level: 933333, mq->high_watermark: 100000, srcresult: ok 0:00:41.469509451 ^[[334m 1641^[[00m 0x2454400 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1194:update_buffering_by_minimum:<multiqueue0>^[[00m ##### min_level: 1422, percent: 1 0:00:41.469543301 ^[[334m 1641^[[00m 0x7fcc18010ad0 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:2008:gst_multi_queue_loop:<multiqueue0>^[[00m ##### SingleQueue 0 : Changed from active to non-active 0:00:41.469600167 ^[[334m 1641^[[00m 0x7fcc18010ad0 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1185:update_buffering_by_minimum:<multiqueue0>^[[00m ##### sq: 0, oq->buffering_level: 1000000, mq->high_watermark: 100000, srcresult: not-linked 0:00:41.469625011 ^[[334m 1641^[[00m 0x7fcc18010ad0 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1185:update_buffering_by_minimum:<multiqueue0>^[[00m ##### sq: 1, oq->buffering_level: 1422, mq->high_watermark: 100000, srcresult: ok 0:00:41.469664259 ^[[334m 1641^[[00m 0x7fcc18010ad0 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1185:update_buffering_by_minimum:<multiqueue0>^[[00m ##### sq: 2, oq->buffering_level: 933333, mq->high_watermark: 100000, srcresult: ok 0:00:41.469684401 ^[[334m 1641^[[00m 0x7fcc18010ad0 ^[[33;01mWARN ^[[00m ^[[00m multiqueue gstmultiqueue.c:1194:update_buffering_by_minimum:<multiqueue0>^[[00m ##### min_level: 1422, percent: 1 0:00:41.470932369 ^[[334m 1641^[[00m 0x7fcc2004ba30 ^[[33;01mWARN ^[[00m ^[[00m =========================================================================== I attached the log and dot graph. Please consider my comment and review. Thanks.
Hello Edward Hervey How about using gst_pad_get_peer () instead of gst_pad_is_linked()?
So the problem doesn't exist/matter for the regular buffering use-case (i.e. by maximum). There's a fundamental issue with trying to calculate buffering levels based on lowest value. Because you can't really know whether a single queue is empty because it drained further down or because there's nothing that arrived. Can you explain the higher-level reason for wanting this information ? Multiqueue shouldn't be used for "buffering" purposes. (In reply to HoonHee Lee from comment #9) > Hello Edward Hervey > > How about using gst_pad_get_peer () instead of gst_pad_is_linked()? No, it's the same thing (is_linked() just checks if the peer is NULL). If it's unlinked at the next element (say after the decoder) you won't know about it (multiqueue source pad has a peer but there's nothing one element further down).
(In reply to Edward Hervey from comment #10) > So the problem doesn't exist/matter for the regular buffering use-case > (i.e. by maximum). > Can you explain the higher-level reason for wanting this information ? > Multiqueue shouldn't be used for "buffering" purposes. Hello Edward Hervey About multiqueue buffering, I remember your comment on bug #768897. But there was some critical issues on queue2 (in urisourcebin) buffering. That is, as we know, buffered level on queue2 is estimated value, not sophisticated one. So it may be incorrect. When general http streaming case, it's more critical. If a stream is very badly interleaved, queue2 never know whether we can play smoothly or not with/without buffering. So, I'm now trying to use multiqueue buffering with playbin3. AdaptiveDemux use-case seems to less critical because usually there are believable bitrate tags. But it also does not safer because queue2's buffering must be very sensitive to segment event and buffer timestamp. But controlling them does not easy from my experience. So, general for purpose (only my environment point of view), I'd like to use multiqueue's buffering.
Marking this as a duplicate of the core issue bug. *** This bug has been marked as a duplicate of bug 667796 ***