Created attachment 343424 [details] [review]
adaptivedemux: Do not expose/remove static streams

Depending on streaming protocol (i.e., hls) there might be streams
which don't need to be re-exposed/removed during playback

Created attachment 343425 [details] [review]
hlsdemux: Re-expose only variant stream with bitrate switching

remove/expose streams on EXT-X-MEDIA tag seems to unnecessary.
Do I missing something?

Created attachment 355514 [details] [review]
hlsdemux: Re-expose only variant stream with bitrate switching

Re-exposing all streams per variant change is very inefficient
operation because:
* EXT-X-MEDIA streams will not be changed during playback.
* baseclass should wait the preparation of task for new streams
  until all download tasks for old streams are finished.
  This might be able to break seamless bitrate switching.

When streams-aware flag is set on parent bin (i.e., playbin3),
since it's allowed that dynamic pad exposing, only the variant streams
will be re-exposed and all the other rendition streams will be reused.

Created attachment 355515 [details] [review]
urisourcebin: Set streams-aware flag

I missed one thing on commit message. In case playbin 3, I allowed hlsdemux to be able to expose subtitle streams.

Hi, how about subtitle streams, now my m3u8 file has eight subtitles, it can be displayed by playbin, but when i switch to another, the subtitle disappeared, why?

what's the meaning of GST_BIN_FLAG_STREAMS_AWARE? 
I added GST_BIN_FLAG_STREAMS_AWARE to the code refer according to your method, but the value of demux->streams_aware is FALSE, when the code is running

(In reply to tianyahai from comment #7)
> Hi, how about subtitle streams, now my m3u8 file has eight subtitles, it can
> be displayed by playbin, but when i switch to another, the subtitle
> disappeared, why?

AFAIK, subtitle is disabled in current hlsdemux. Need more context such as your m3u8 and your playbin usecase (playbin or playbin3).


(In reply to tianyahai from comment #8)
> what's the meaning of GST_BIN_FLAG_STREAMS_AWARE? 
> I added GST_BIN_FLAG_STREAMS_AWARE to the code refer according to your
> method, but the value of demux->streams_aware is FALSE, when the code is
> running

You can refer to its meaning at https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstBin.html
My understanding about it is that, it's used in playbin3 (not playbin) to indicate that corresponding element/bin can handle GstStreams API and also can handle stream add/removal during playback without any interruption.

I guess you are now use playbin (not playbin3). The playbin doesn't use urisourcebin but do uridecodebin. Please not that my patch aims to set the flag at urisourcebin and read it by adaptivedemux (this is urisourcebin's child bin)

(In reply to Seungha Yang from comment #9)
> (In reply to tianyahai from comment #7)
> > Hi, how about subtitle streams, now my m3u8 file has eight subtitles, it can
> > be displayed by playbin, but when i switch to another, the subtitle
> > disappeared, why?
> 
> AFAIK, subtitle is disabled in current hlsdemux. Need more context such as
> your m3u8 and your playbin usecase (playbin or playbin3).
> 
> 
Thank you for your reply, in deeded, the subtitles are closed by default, but in the code of gsthlsdemux.c file i enabled subtitle like this：
  select = demux->streams_aware ? TRUE : (media->mtype == GST_HLS_MEDIA_TYPE_VIDEO ||
          media->mtype == GST_HLS_MEDIA_TYPE_AUDIO || media->mtype == GST_HLS_MEDIA_TYPE_SUBTITLES);

In my m3u8 there are eight subtitles, the subtitles can be displayed when the code is running, but if i switch subtitle to another it will be disappear, i used playbin or playbin2, not playbin3

> (In reply to tianyahai from comment #8)
> > what's the meaning of GST_BIN_FLAG_STREAMS_AWARE? 
> > I added GST_BIN_FLAG_STREAMS_AWARE to the code refer according to your
> > method, but the value of demux->streams_aware is FALSE, when the code is
> > running
> 
> You can refer to its meaning at
> https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/
> GstBin.html
> My understanding about it is that, it's used in playbin3 (not playbin) to
> indicate that corresponding element/bin can handle GstStreams API and also
> can handle stream add/removal during playback without any interruption.
> 
> I guess you are now use playbin (not playbin3). The playbin doesn't use
> urisourcebin but do uridecodebin. Please not that my patch aims to set the
> flag at urisourcebin and read it by adaptivedemux (this is urisourcebin's
> child bin)

Yes,as you say,i used playbin,can i add "GST_OBJECT_FLAG_SET (dec, GST_ELEMENT_FLAG_SOURCE | GST_BIN_FLAG_STREAMS_AWARE)" to gsturidecodebin.c file according to your urisourcebin file? 
How can hlsdemux achieve switch subtitles...

I submitted it in this: https://bugzilla.gnome.org/show_bug.cgi?id=788869

I try playbin3 to play my m3u8, if it only contain one stream as MediaPlaylist, it's ok, if contains variant streams, it failed like below:
 ...
PREROLLING ...
... 100%  
(gst-launch-1.0:22408): GStreamer-CRITICAL **: gst_stream_collection_get_stream: assertion 'GST_IS_STREAM_COLLECTION (collection)' failed
Caught SIGSEGV

+ Trace 238065

Why it's not ok, does playbin3 not support multi-rate m3u8 file?

(In reply to tianyahai from comment #10)
> Thank you for your reply, in deeded, the subtitles are closed by default,
> but in the code of gsthlsdemux.c file i enabled subtitle like this：
>   select = demux->streams_aware ? TRUE : (media->mtype ==
> GST_HLS_MEDIA_TYPE_VIDEO ||
>           media->mtype == GST_HLS_MEDIA_TYPE_AUDIO || media->mtype ==
> GST_HLS_MEDIA_TYPE_SUBTITLES);
> 
> In my m3u8 there are eight subtitles, the subtitles can be displayed when
> the code is running, but if i switch subtitle to another it will be
> disappear, i used playbin or playbin2, not playbin3
This comment might be helpful for you :)
https://bugzilla.gnome.org/show_bug.cgi?id=779011#c7

> Yes,as you say,i used playbin,can i add "GST_OBJECT_FLAG_SET (dec,
> GST_ELEMENT_FLAG_SOURCE | GST_BIN_FLAG_STREAMS_AWARE)" to gsturidecodebin.c
> file according to your urisourcebin file? 
You can try it but it doesn't make sense because uridecodebin doesn't "STREAMS_AWARE".

> How can hlsdemux achieve switch subtitles...
It's totally dependent on adaptivdemux's track switch method. Firstly, adaptivdemux(hlsdemux) will expose all available tracks during preroll state. Then, stop streaming for non-activated track depending on flow return "not-linked". It's well described on comment of gstadaptivedemux.c

Then, if you try switch track, it will trigger "reconfigure" event which also trigger re-streaming for newly activated track in adaptivedemux. Basically, this is how adaptivedemux switch tracks. But when I saw comment above, there seems weak point (maybe bug?) in playbin's subtitle track change behavior.

(In reply to Seungha Yang from comment #13)
> (In reply to tianyahai from comment #10)
> > Thank you for your reply, in deeded, the subtitles are closed by default,
> > but in the code of gsthlsdemux.c file i enabled subtitle like this：
> >   select = demux->streams_aware ? TRUE : (media->mtype ==
> > GST_HLS_MEDIA_TYPE_VIDEO ||
> >           media->mtype == GST_HLS_MEDIA_TYPE_AUDIO || media->mtype ==
> > GST_HLS_MEDIA_TYPE_SUBTITLES);
> > 
> > In my m3u8 there are eight subtitles, the subtitles can be displayed when
> > the code is running, but if i switch subtitle to another it will be
> > disappear, i used playbin or playbin2, not playbin3
> This comment might be helpful for you :)
> https://bugzilla.gnome.org/show_bug.cgi?id=779011#c7

Thank you,i will try it follow this comment
> 
> > Yes,as you say,i used playbin,can i add "GST_OBJECT_FLAG_SET (dec,
> > GST_ELEMENT_FLAG_SOURCE | GST_BIN_FLAG_STREAMS_AWARE)" to gsturidecodebin.c
> > file according to your urisourcebin file? 
> You can try it but it doesn't make sense because uridecodebin doesn't
> "STREAMS_AWARE".
> 
...indeed it doesn't work...
> > How can hlsdemux achieve switch subtitles...
> It's totally dependent on adaptivdemux's track switch method. Firstly,
> adaptivdemux(hlsdemux) will expose all available tracks during preroll
> state. Then, stop streaming for non-activated track depending on flow return
> "not-linked". It's well described on comment of gstadaptivedemux.c
> 
> Then, if you try switch track, it will trigger "reconfigure" event which
> also trigger re-streaming for newly activated track in adaptivedemux.
> Basically, this is how adaptivedemux switch tracks. But when I saw comment
> above, there seems weak point (maybe bug?) in playbin's subtitle track
> change behavior.
Have you tested m3u8 file that contains mulit bitrate streams and subtitles, and can you switch subtile seccessfully? thank you again, have a nice day^_^

(In reply to tianyahai from comment #14)
> Have you tested m3u8 file that contains mulit bitrate streams and subtitles,
> and can you switch subtile seccessfully? thank you again, have a nice day^_^

I think current gstreamer is not good enough for hls subtitle track change.... DASH + multi TTML case might have similar issue I guess...

In my opinion, current adaptivedemux structure has problem from subtitle track change point of view.
First, at adaptivedemux side, streaming progress of subtitle is not fully synchornized with A/V. (since subtitle data is too small, downloading the subtitle is too faster than A/V).
Adaptivedemux's track change method is that, it tries to download new stream from 1) last matching streams time position or 2) depending on query position.
This can cause mismatch between old/new subtitle stream's time position.
Second, the fact that multiqueue element doesn't fully consider synchronization of sparse stream (subtitle stream).

Since I'm not expert... above comment might be incorrect. It's just my opinion.

(In reply to Seungha Yang from comment #15)
> (In reply to tianyahai from comment #14)
> > Have you tested m3u8 file that contains mulit bitrate streams and subtitles,
> > and can you switch subtile seccessfully? thank you again, have a nice day^_^
> 
> I think current gstreamer is not good enough for hls subtitle track
> change.... DASH + multi TTML case might have similar issue I guess...
> 
> In my opinion, current adaptivedemux structure has problem from subtitle
> track change point of view.
> First, at adaptivedemux side, streaming progress of subtitle is not fully
> synchornized with A/V. (since subtitle data is too small, downloading the
> subtitle is too faster than A/V).
> Adaptivedemux's track change method is that, it tries to download new stream
> from 1) last matching streams time position or 2) depending on query
> position.
> This can cause mismatch between old/new subtitle stream's time position.
> Second, the fact that multiqueue element doesn't fully consider
> synchronization of sparse stream (subtitle stream).
> 
> Since I'm not expert... above comment might be incorrect. It's just my
> opinion.

Ok,thank you for your help,next i will try playbin3 play m3u8...
but as you said "subtitle is not fully synchornized with A/V.."
even if it switch subtitles successful by playbin3,subtitle display maybe have some problem..

Comment on attachment 355514 [details] [review]
hlsdemux: Re-expose only variant stream with bitrate switching

Will be updated again

*** Bug 796000 has been marked as a duplicate of this bug. ***

-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/508.