GNOME Bugzilla – Bug 675625
dashdemux: a streaming client supporting the new MPEG DASH standard
Last modified: 2015-01-06 12:04:38 UTC
Created attachment 213609 [details] [review] [dashbin] a streaming client supporting the new MPEG DASH standard In attachment: DASHBIN-initial-release.patch DASH is the new Dynamic Adaptive Streaming over HTTP standard from MPEG. This patch adds the support for DASH in GStreamer, through the new bin element DASHBIN. More in detail, the patch adds 3 new elements to the gst-plugins-bad-0.10.23 plugin package: 1) mpdcommon: an helper library to browse and manage the DASH MPD manifest files; 2) mpdparse: the mpdparse plugin, that detects and parses the MPD manifest files; 3) dashbin: the DASH client, implemented as a src bin; A simple usage example is: gst-launch \ dashbin location=http://example.com/dash_manifest.mpd \ ! qtdemux ! queue ! ffdec_h264 ! xvimagesink In the initial state, DASHBIN creates a simple 2 elements pipeline: souphttpsrc ! mpdparse --> src pad whenever the mpdparse element detects a valid DASH MPD manifest file (or if a local MPD file is passed to mpdparse through the "input-mpd-file" property), it parses it and then it fires the "on-new-mpd" signal. DASHBIN catches the signal and creates from 1 to 3 new download pipelines, depending on the content of the MPD file. The maximum configuration of the bin is the following: # MPD manifest file download pipeline souphttpsrc ! mpdparse ! fakesink # video download pipeline souphttpsrc ! queue --> src pad # audio download pipeline souphttpsrc ! queue --> src_1 pad # subtitles download pipeline souphttpsrc ! queue --> src_2 pad Often, only the video pipeline is present (and the stream will contain several video/audio/subtitles tracks), but the DASH standard supports also separate delivery of video/audio/subtitles tracks. Each of the pipelines has a 3-states buffer control mechanism based on thresholds. The 3 states are PREFETCHING (during preroll, or when we are re-buffering after a buffer under-run), NORMAL (downloading new data and playing buffered data at the same time) and WAITING (after a buffer overrun, we delay the download of new segments while we consume some buffered data). The buffer thresholds can be specified by the user (in bytes or seconds) through several properties. By default, DASHBIN uses the initial buffering time from the MPD file. Automatic adaptation to the available bandwidth is not included in this release. It will be provided in a future update, probably in the form of an external library. The max-video-bw and max-audio-bw properties allow the user (or the controlling application) to select the appropriate bandwidth of the video and audio representations. The change is effective starting from the next data segment. DASHBIN wraps all the properties of the souphttpsrc element and also it exposes an URIHandler interface. Setting a RANK above GST_RANK_PRIMARY, it will overtake the souphttpsrc element as the default handler for HTTP links, enabling DASH support on all playbin2 based applications. If no MPD manifest file is detected, the mpdparse plugin will work in pass-through mode (like a typefind element) keeping full compatibility with the plain souphttpsrc element. DASHBIN partially supports the MPD syntax from the first published version of the standard (ISO/IEC 23009-1 2012-04-01), together with some syntax elements from older versions of the standard, in order to keep compatibility with all the currently available test material. In a future update, legacy support will be dropped and the support for the published version of the standard will be completed. DASHBIN currently supports the "ISO media file format On Demand" profile and the "MPEG-2 TS simple" profile. Support for more advanced profiles (starting from the Live profile) will be implemented in the future, as DASHBIN is still under heavy development. Here you can find a few test sequences (provided by Klagenfurt University): # Big Buck Bunny http://www-itec.uni-klu.ac.at/ftp/datasets/mmsys12/BigBuckBunny/MPDs/BigBuckBunnyNonSeg_1s_isoffmain_DIS_23009_1_v_2_1c2_2011_08_30.mpd http://www-itec.uni-klu.ac.at/ftp/datasets/mmsys12/BigBuckBunny/MPDs/BigBuckBunny_1s_isoffmain_DIS_23009_1_v_2_1c2_2011_08_30.mpd http://www-itec.uni-klu.ac.at/ftp/datasets/mmsys12/BigBuckBunny/MPDs/test-dashjs.mpd # Elephants Dream http://www-itec.uni-klu.ac.at/ftp/datasets/mmsys12/ElephantsDream/MPDs/ElephantsDreamNonSeg_1s_isoffmain_DIS_23009_1_v_2_1c2_2011_08_30.mpd http://www-itec.uni-klu.ac.at/ftp/datasets/mmsys12/ElephantsDream/MPDs/ElephantsDream_1s_isoffmain_DIS_23009_1_v_2_1c2_2011_08_30.mpd # Of Forest And Men http://www-itec.uni-klu.ac.at/ftp/datasets/mmsys12/OfForestAndMen/forest_15s/OfForestAndMen_15s_isoffmain_DIS_23009_1_v_2_1c2_2011_08_30.mpd # Red Bull Play Streets http://www-itec.uni-klu.ac.at/ftp/datasets/mmsys12/RedBullPlayStreets/redbull_4s/RedBullPlayStreets_4s_isoffmain_DIS_23009_1_v_2_1c2_2011_08_30.mpd # The Swiss Account http://www-itec.uni-klu.ac.at/ftp/datasets/mmsys12/TheSwissAccount/MPDs/TheSwissAccount_4s_isoffmain_DIS_23009_1_v_2_1c2_2011_08_30.mpd # Valkaama http://www-itec.uni-klu.ac.at/ftp/datasets/mmsys12/Valkaama/MPDs/Valkaama_1s_act_isoffmain_DIS_23009_1_v_2_1c2_2011_08_30.mpd # Old Syntax: http://www-itec.uni-klu.ac.at/ftp/datasets/mmsys12/Selected_VLC_Plugin_MPDs/BigBuckBunny_15s_DASH_VLC_Compatible_4-Representations_720p.mpd http://www-itec.uni-klu.ac.at/ftp/datasets/mmsys12/Selected_VLC_Plugin_MPDs/RedBullPlayStreets_4s_DASH_VLC_Compatible_3-Representatins_720p.mpd Best regards, Gianluca Gennari
I don't think it's a good idea to have just another plugin in -bad that basically has the same error-prone code at its base as hlsdemux. hlsdemux was recently refactored to allow it to be used as base class for other, similar HTTP live streaming/playlist protocols and it would really make sense to make use of this (and extend/change where necessary) instead of implementing it again.
While hlsdemux and dashbin implement similar functionalities, they are completely different from an architectural point of view. hlsdemux serializes the downloads, while dashbin implements several parallel download pipelines. This has some advantages: 1) Less latency: suppose we have to download several adaptation sets (let's say, 2 views of an H.264/MVC stream): serializing them, we have to buffer an entire segment of the base view, and then we can start playing the sequence when we finally start downloading the dependent view. With parallel downloads, both views are downloaded at the same time, with no further latency; if we consider an H.264/SVC video sequence with 3 layers, things get even worse. Also, segments in DASH are usually longer than in HLS (10-15 seconds). 2) Suppose we have multiple alternative servers available in the manifest file: with parallel downloads, the client can choose a different server for each adaptation set; or we may have to download a particular audio track from a secondary server: with parallel downloads, we avoid switching back and forth between the servers, keeping the HTTP connections always open. 3) multiple concurrent downloads often give an higher throughput than a single one. Of course, nothing prevents to implement DASH support in hlsdemux: the mpdcommon library should provide the needed parsing functions.
(Optionally) adding the multiple-download support to hlsdemux shouldn't be that much work and would keep the code duplication much smaller here :) Also this should really be optional, you don't always want that and in some cases you might want to configure how much should be downloaded in parallel, etc. Also, as DASH doesn't have its own URI protocol (and just uses HTTP) it can't be implemented as a source element but has to be done as a demuxer like hlsdemux to be able to autoplug it in playbin2 pipelines.
(In reply to comment #3) > Also, as DASH doesn't have its own URI protocol (and just uses HTTP) it can't > be implemented as a source element but has to be done as a demuxer like > hlsdemux to be able to autoplug it in playbin2 pipelines. As I wrote in the description, dashbin can be autoplugged in Playbin2 pipelines by just changing the rank to something higher than PRIMARY so that it overtakes the plain souphttpsrc element as the default handler for HTTP URIs. Then it acts as a plain souphttpsrc element, unless a DASH manifest file is downloaded and detected; in this case, the bin creates all the necessary elements (depending on the manifest content and on the user settings) and then it starts streaming the DASH content. After parsing, the manifest file is dropped. I'm not familiar with the internal structure of hlsdemux, so I don't know how it solves the same problem and it's not clear to me how a demuxer element can help. I will read the source code to better understand what you are talking about.
Cool stuff! Great to see you have finally decided to upstream this. > dashbin can be autoplugged in Playbin2 pipelines > by just changing the rank to something higher than PRIMARY so that it > overtakes the plain souphttpsrc element as the default handler for > HTTP URIs. Then it acts as a plain souphttpsrc element, unless a > DASH manifest file is downloaded and detected This doesn't sound very appealing to me. I would strongly prefer a setup where the DASH plugin comes into play only when a DASH manifest file is detected. This means there either needs to be a special URI scheme (not a great approach), or a manifest file "demuxer" approach a la hlsdemux needs to be taken. "demuxer" is a bit misleading here, since it's not really doing any demuxing at all, it's just how it appears to playbin2 really and where it sits in the autoplugging order. Once it gets the initial playlist/file, it takes over. I haven't looked at the DASH code yet and my memory of DASH is a bit hazy, but from what I remember I would think that it should be possible to adapt the existing hls-targetted base class to accommodate DASH as well.
(In reply to comment #5) > I would strongly prefer a setup where the DASH plugin comes into play only when > a DASH manifest file is detected. This means there either needs to be a special > URI scheme (not a great approach), or a manifest file "demuxer" approach a la > hlsdemux needs to be taken. "demuxer" is a bit misleading here, since it's not > really doing any demuxing at all, it's just how it appears to playbin2 really > and where it sits in the autoplugging order. Once it gets the initial > playlist/file, it takes over. Thanks for the explanation, now it makes a lot of sense. Actually, this approach will simplify quite a few things and it is easy to implement, so I'm definitely going to adopt it. Since we started talking about the interaction with Playbin2, let me ask a few question. We may have several languages for audio tracks or subtitles, available as separate streams. The list of all available options is known after we complete the parsing of the manifest file, then we need to notify Playbin2 of this list, and finally we need to get back the selected language from Playbin2, in order to set-up the download of the correct tracks. Is there a standard way to do this with Playbin2? Also, when we have several video views/layers (for H.264/MVC or H.264/SVC streams) we need to reassemble a compliant H.264 raw bitstream using a sort of "h264mux" element, before sending it to the decoder. Is Playbin2 able to auto-plug muxer elements (eventually, using some "magic" trick)? Otherwise, I will have to construct the reassembling pipeline into the dashbin. > I haven't looked at the DASH code yet and my memory of DASH is a bit hazy, but > from what I remember I would think that it should be possible to adapt the > existing hls-targetted base class to accommodate DASH as well. We are carefully looking into this, as changing the core structure at this point it's definitely not trivial.
(In reply to comment #6) > (In reply to comment #5) > > I would strongly prefer a setup where the DASH plugin comes into play only when > > a DASH manifest file is detected. This means there either needs to be a special > > URI scheme (not a great approach), or a manifest file "demuxer" approach a la > > hlsdemux needs to be taken. "demuxer" is a bit misleading here, since it's not > > really doing any demuxing at all, it's just how it appears to playbin2 really > > and where it sits in the autoplugging order. Once it gets the initial > > playlist/file, it takes over. > > Thanks for the explanation, now it makes a lot of sense. Actually, this > approach will simplify quite a few things and it is easy to implement, so I'm > definitely going to adopt it. Any news on this? :) > Since we started talking about the interaction with Playbin2, let me ask a few > question. > > We may have several languages for audio tracks or subtitles, available as > separate streams. The list of all available options is known after we complete > the parsing of the manifest file, then we need to notify Playbin2 of this list, > and finally we need to get back the selected language from Playbin2, in order > to set-up the download of the correct tracks. Is there a standard way to do > this with Playbin2? Language tags for the different streams, yes. That's how it is done with other container formats too. > Also, when we have several video views/layers (for H.264/MVC or H.264/SVC > streams) we need to reassemble a compliant H.264 raw bitstream using a sort of > "h264mux" element, before sending it to the decoder. Is Playbin2 able to > auto-plug muxer elements (eventually, using some "magic" trick)? Otherwise, I > will have to construct the reassembling pipeline into the dashbin. h264parse could do this and is autoplugged already in front of the decoders/sinks
(In reply to comment #7) > (In reply to comment #6) > > (In reply to comment #5) > > > I would strongly prefer a setup where the DASH plugin comes into play only when > > > a DASH manifest file is detected. This means there either needs to be a special > > > URI scheme (not a great approach), or a manifest file "demuxer" approach a la > > > hlsdemux needs to be taken. "demuxer" is a bit misleading here, since it's not > > > really doing any demuxing at all, it's just how it appears to playbin2 really > > > and where it sits in the autoplugging order. Once it gets the initial > > > playlist/file, it takes over. > > > > Thanks for the explanation, now it makes a lot of sense. Actually, this > > approach will simplify quite a few things and it is easy to implement, so I'm > > definitely going to adopt it. > > Any news on this? :) Hi Sebastian, there are several news indeed. We are about to start a collaboration with Orange (they have been very active on the GStreamer list about DASH developments) in order to jointly develop a new DASH client for GStreamer, based on HLSdemux, with an enhanced feature set and an up-to-date support for the DASH manifest file syntax. This will take several weeks to be released, so for the time being please consider the DASH client released in this topic as a proof-of-concept version. > > > Since we started talking about the interaction with Playbin2, let me ask a few > > question. > > > > We may have several languages for audio tracks or subtitles, available as > > separate streams. The list of all available options is known after we complete > > the parsing of the manifest file, then we need to notify Playbin2 of this list, > > and finally we need to get back the selected language from Playbin2, in order > > to set-up the download of the correct tracks. Is there a standard way to do > > this with Playbin2? > > Language tags for the different streams, yes. That's how it is done with other > container formats too. Ok thanks, I will look into this. My only doubt was about the possibility to have a notification from playbin2 about the audio track effectively selected by the user (or at least a default selection) before we start downloading the segments. > > Also, when we have several video views/layers (for H.264/MVC or H.264/SVC > > streams) we need to reassemble a compliant H.264 raw bitstream using a sort of > > "h264mux" element, before sending it to the decoder. Is Playbin2 able to > > auto-plug muxer elements (eventually, using some "magic" trick)? Otherwise, I > > will have to construct the reassembling pipeline into the dashbin. > > h264parse could do this and is autoplugged already in front of the > decoders/sinks I know h264parse quite well, as I already modified it to implement the opposite feature ("demuxing" the H.264 stream to feed the mpegtsmux element in order to produce a "layered" MPEG-2 transport stream). But I don't like its structure very much, so I will probably end up developing a new plugin dedicated to just reassembling the H.264 layers/views (as there is not much parsing involved into this process, the code duplication will be minimal, if any).
(In reply to comment #8) > (In reply to comment #7) > > (In reply to comment #6) > > > (In reply to comment #5) > > > > I would strongly prefer a setup where the DASH plugin comes into play only when > > > > a DASH manifest file is detected. This means there either needs to be a special > > > > URI scheme (not a great approach), or a manifest file "demuxer" approach a la > > > > hlsdemux needs to be taken. "demuxer" is a bit misleading here, since it's not > > > > really doing any demuxing at all, it's just how it appears to playbin2 really > > > > and where it sits in the autoplugging order. Once it gets the initial > > > > playlist/file, it takes over. > > > > > > Thanks for the explanation, now it makes a lot of sense. Actually, this > > > approach will simplify quite a few things and it is easy to implement, so I'm > > > definitely going to adopt it. > > > > Any news on this? :) > > Hi Sebastian, > there are several news indeed. We are about to start a collaboration with > Orange (they have been very active on the GStreamer list about DASH > developments) in order to jointly develop a new DASH client for GStreamer, > based on HLSdemux, with an enhanced feature set and an up-to-date support for > the DASH manifest file syntax. This will take several weeks to be released, so > for the time being please consider the DASH client released in this topic as a > proof-of-concept version. That's great news :) > > > Also, when we have several video views/layers (for H.264/MVC or H.264/SVC > > > streams) we need to reassemble a compliant H.264 raw bitstream using a sort of > > > "h264mux" element, before sending it to the decoder. Is Playbin2 able to > > > auto-plug muxer elements (eventually, using some "magic" trick)? Otherwise, I > > > will have to construct the reassembling pipeline into the dashbin. > > > > h264parse could do this and is autoplugged already in front of the > > decoders/sinks > > I know h264parse quite well, as I already modified it to implement the opposite > feature ("demuxing" the H.264 stream to feed the mpegtsmux element in order to > produce a "layered" MPEG-2 transport stream). But I don't like its structure > very much, so I will probably end up developing a new plugin dedicated to just > reassembling the H.264 layers/views (as there is not much parsing involved into > this process, the code duplication will be minimal, if any). The main problem with that is that it would be h264parse job still. And h264parse would be autoplugged anyway. It's were conversions like this should happen
Any update on this ? Is there a git repository somewhere with ongoing progress ?
(In reply to comment #10) > Any update on this ? Is there a git repository somewhere with ongoing progress > ? Hi Edward, the development of the new client is ongoing and we (Orange and STM) are almost ready to open the git repository to the public. As I said before, the new client version is based on HLSdemux and is totally different from the one released here (which is not developed anymore). The new client is still based on GStreamer 0.10.X, so we are also discussing whether we should release it as it is or if we should port it to 0.11.X/1.0.
That's great news. From our point of view it would be best if you ported it to 0.11/1.0, since the 0.10 branch is feature-frozen now (1.0 is scheduled to be released within the next 2 weeks, for what it's worth).
It would be really useful to me if you released your 0.10 version before porting it to 1.0, as many people are still stuck in the past.
Hi all, the new 'dashdemux' plugin is finally ready for testing. The git repository is on Github: https://github.com/Orange-OpenSource/gstdashdemux David Corvoysier announced the new release on the GStreamer mailing list: http://gstreamer-devel.966125.n4.nabble.com/Request-for-feedback-New-DASH-plugin-implementation-dashdemux-td4656682.html This new plugin is the result of a joint effort from ST MicroElectronics and Orange and is based on the HLSdemux structure. It is still based on GStreamer 0.10.36. For further details, please refer to David's post on the mailing list or to the README file on Github. I think this bug can be closed now, as the new plugin can be discussed on the mailing list and the old 'dashbin' is not developed anymore. Best regards, Gianluca Gennari
(In reply to comment #14) > I think this bug can be closed now, as the new plugin can be discussed on the > mailing list and the old 'dashbin' is not developed anymore. No, the discussion should happen in this bug and not on the mailing list ;) Some comments by Edward about this from the mailinglist: I'll let someone more familiar with HLS and DASH comment about the > implementation itself, but a few first comments: > * All the bugs/patches you mentioned in the README are pushed upstream > in both 0.10 and master. The problem is that ... there's no guarantee > there will be a future 0.10 release. > * The various source files you are re-using are already ported to 1.0 > (along with hlsdemux itself). > * Most of the development effort is concentrated on 1.0 these days, and > in combination with the above-mentioned release issue, I fear you might > not get much help by sticking to 0.10 :( > * For the plugin to make it into -bad (and not -good as you mentionned), > it would be a good move to make your repository based on -bad (and not a > standalone repository as is the case right now). > * Quite a bit of your code isn't related to gstreamer API but more to > protocol handling (which is agnostic to the 0.10/1.0 changes). > > Considering all the above, I'd therefore recommend you do the following: > * Re-use the current master (1.0) code of gstfragmented and related > source files > * For that, clone gst-plugins-bad and adapt your commits/code on top. > Since hlsdemux is already ported, and you inspired yourself from that > code, you can see how it is now handling the gstreamer parts in 1.0. > * Open a bug report on the gstreamer bug tracker pointing to that > repository so we can track/comment/assist on the progress
Can we close this one?
Yes, why would you keep it open? :)
Hi, Where can I find DASHBIN-initial-release.patch ?
Hi Michal, the original patch is obsolete, as a new plugin called "dashdemux" has been developed in place of the old "dashbin" and is already included in GStreamer 1.2.x. See post #14 for more details. If you are really interested on the old patch for some reason, you can still download it by clicking on the "Show Obsolete" link below.