After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 668094 - dashsink: New sink for DASH
dashsink: New sink for DASH
Status: RESOLVED OBSOLETE
Product: GStreamer
Classification: Platform
Component: gst-plugins-bad
git master
Other Linux
: Normal enhancement
: git master
Assigned To: GStreamer Maintainers
GStreamer Maintainers
Depends on: 660260 668091 668092 668093 668095
Blocks:
 
 
Reported: 2012-01-17 14:09 UTC by Andoni Morales
Modified: 2018-11-03 13:10 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Dashsink patch adding partial, untested support for the codecs property (12.79 KB, patch)
2017-05-10 18:28 UTC, Isaac Smith
needs-work Details | Review

Description Andoni Morales 2012-01-17 14:09:59 UTC
New sink for DASH that uses the GstBaseAdaptiveSink base class. The element support for now the DASH basic-on-demand profile with fragmented MP4.

Example usage:

gst-launch -e videotestsrc num-buffers=3000 is-live=true  ! "video/x-raw-yuv, framerate=30/1" ! keyunits-scheduler interval=10000000000 ! tee name=t ! queue ! textoverlay text=200kbps font-desc="sans bold 40" ! x264enc bitrate=200 ! mp4dashmux fragment-method=2 ! queue ! dashsink name=sink base-url=http://localhost/dash output-directory=/home/andoni/public_html/dash t. !  queue ! textoverlay text=100kbps font-desc="sans bold 40" ! x264enc bitrate=100 ! mp4dashmux fragment-method=2 ! queue ! sink. t. !  queue ! textoverlay text=500kbps font-desc="sans bold 40" ! x264enc bitrate=500 ! mp4dashmux fragment-method=2 ! queue ! sink. audiotestsrc is-live=true ! keyunits-scheduler interval=10000000000 ! faac ! mp4dashmux fragment-method=2 ! queue ! sink.
Comment 1 Andoni Morales 2012-01-17 14:15:19 UTC
And the missing branch where the work is beeing done

https://github.com/ylatuya/gst-plugins-bad/tree/base_fragmented
Comment 2 Sebastian Dröge (slomo) 2013-11-11 13:59:25 UTC
Can you attach patch(es) for this to the bug for easier reviewing, once the other patches have landed and this can be merged? :)
Comment 3 Thiago Sousa Santos 2015-02-18 14:26:52 UTC
Gathered the required patches in bugs and public repositories and I have dashsink working (for the basic tests I did) with those branches:

http://cgit.freedesktop.org/~thiagoss/gst-plugins-good/log/?h=dashsink
http://cgit.freedesktop.org/~thiagoss/gst-plugins-bad/log/?h=dashsink

The bugs involved are:
good:
https://bugzilla.gnome.org/show_bug.cgi?id=706509
https://bugzilla.gnome.org/show_bug.cgi?id=708221
https://bugzilla.gnome.org/show_bug.cgi?id=660260
https://bugzilla.gnome.org/show_bug.cgi?id=668091
bad:
This one

I'd recommend reviewing in those branches as minor updates have been made to the patches. If you prefer I can go over on the bugs and upload new versions for the patches if needed.
Comment 4 Nicolas Dufresne (ndufresne) 2015-03-15 12:13:47 UTC
@@ -2144,6 +2181,17 @@ flush:
GstBuffer *buffer;
guint i, total_size;
+ if (GST_BUFFER_FLAG_IS_SET (buf, GST_BUFFER_FLAG_DELTA_UNIT)) {
+ GST_WARNING ("Next fragment will not start with a keyframe");
+ }

Maybe we would post a warning in this case (I mean GST_ELEMENT_WARNING), since it means multiple issues could arrived around this area of the file.

+++ b/gst/isomp4/gstqtmux.c
@@ -104,6 +104,7 @@
#include <gst/audio/audio.h>
#include <gst/video/video.h>
#include <gst/tag/tag.h>
+#include <gst/video/video.h>

you are including video.h twice.

+ {FRAGMENT_METHOD_EVENT, "GstForceKeyUnit events", "event"},

Arguably, the GstForceKeyUnit was really designed to ask a decoder to produce a keyframe. Overriding it for this task have an interesting side effect. Let's say you have a decoder set to produce keyframe only on request. Then there would be no way to have more then 1 keyframe per fragment. By adding a specific event to force a Fragment, we could send GstForceKeyUnit to insert keyframes in the current fragment and later cut the fragment. This may or may not be very useful, but I think it's good to understand the limitation.

One thing I notice is that with time method, we use a simplistic approach which makes the fragment always slightly bigger then then target duration. I don't know if that is a problem or not.

p.s. I'll be posting multiple comment while going through
Comment 5 Nicolas Dufresne (ndufresne) 2015-03-15 15:12:39 UTC
+Android.mk: Makefile.am

Androgenizer stuff isn't maintain anymore, I would just remove it.

+VOID:STRING,STRING

We use the generic marshaller now in newly introduce code.
Comment 6 Nicolas Dufresne (ndufresne) 2015-03-15 16:44:09 UTC
Just a suspiciously C++ commented unref in gst-libs/gst/baseadaptive/gstbaseadaptivesink.c:

+ if (pad_data->pad != NULL) {
+ //gst_object_unref (pad_data->pad);
+ pad_data->pad = NULL;
+ }
Comment 7 Nicolas Dufresne (ndufresne) 2015-03-15 16:48:44 UTC
+ case GST_STATE_CHANGE_PAUSED_TO_READY:
+ gst_base_adaptive_sink_unlock (sink);
+ gst_base_adaptive_sink_stop (sink);

Hard for me to tell, but we you chain and then unlock, does it mean you may deadlock here ?
Comment 8 Alex 2015-03-24 01:08:39 UTC
(In reply to Thiago Sousa Santos from comment #3)
> Gathered the required patches in bugs and public repositories and I have
> dashsink working (for the basic tests I did) with those branches:
> 
> http://cgit.freedesktop.org/~thiagoss/gst-plugins-good/log/?h=dashsink
> http://cgit.freedesktop.org/~thiagoss/gst-plugins-bad/log/?h=dashsink
> 
> The bugs involved are:
> good:
> https://bugzilla.gnome.org/show_bug.cgi?id=706509
> https://bugzilla.gnome.org/show_bug.cgi?id=708221
> https://bugzilla.gnome.org/show_bug.cgi?id=660260
> https://bugzilla.gnome.org/show_bug.cgi?id=668091
> bad:
> This one
> 
> I'd recommend reviewing in those branches as minor updates have been made to
> the patches. If you prefer I can go over on the bugs and upload new versions
> for the patches if needed.

I am tring to reproduce your test buf I failed to use chrome to play mpd file. Any thing I missed? 

My steps are:
==================== Server side ==============================
1. Build and install these 2 packages
http://cgit.freedesktop.org/~thiagoss/gst-plugins-good/log/?h=dashsink
http://cgit.freedesktop.org/~thiagoss/gst-plugins-bad/log/?h=dashsink

2. gst-launch-1.0 videotestsrc is-live=true ! video/x-raw, format=NV12, framerate=30/1 ! queue ! x264enc bitrate=200 ! mp4dashmux fragment-method=2 ! queue ! dashsink name=sink base-url=http://localhost/dash output-directory=/home/wzw/public_html/dash title=sample

3. 
cd /home/wzw/public_html/dash
python -m SimpleHTTPServer 8111 

========= Client(chrome browser + dash.js) side ===============
4. 
open URL: http://127.0.0.1:8111/samples/dash-if-reference-player/
In the "Stream" entry, key in "http://127.0.0.1:8111/sample.mpd"
Click "Load"

5. Nothing displayed, but got these logs from dash.js:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Parsing complete: ( xml2json: 3ms, objectiron: 2ms, total: 0.005s) 
Debug.js:115 Manifest has loaded. 
Debug.js:115 Manifest has been refreshed. 
Debug.js:115 MediaSource is open! 
Debug.js:115 [object Event] 
Debug.js:115 Added 0 inline events 
Debug.js:115 video codec: video/mp4;codecs="undefined" 
Debug.js:115 videoCodec (video/mp4;codecs="undefined") is not supported. 
Debug.js:115 No buffer was created, skipping video data. 
Debug.js:115 No audio data. 
Debug.js:115 No text data. 
Debug.js:115 No fragmentedText data. 
Debug.js:115 No streams to play.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

6. The gstreamer pipline log is so large, so I put it at:
https://drive.google.com/file/d/0BzhQfLFttwrNVHhDWUZpcHNvRVU/view?usp=sharing
Comment 9 Jan Schmidt 2016-04-01 14:38:40 UTC
12 month ping: We should probably commit this and the patches in the other days one of these days.
Comment 10 Thiago Sousa Santos 2016-04-11 13:11:40 UTC
(In reply to Jan Schmidt from comment #9)
> 12 month ping: We should probably commit this and the patches in the other
> days one of these days.

IIRC, one remaining issue here is how to properly inform dashsink of bitrates and alternative content.

Should we consider all video/audio inputs as being bitrate alternatives to each other or should we design some mechanism to properly signal that?
Comment 11 Thiago Sousa Santos 2016-04-14 20:31:21 UTC
Rebased everything again:

http://cgit.freedesktop.org/~thiagoss/gst-plugins-good/log/?h=dashsink
http://cgit.freedesktop.org/~thiagoss/gst-plugins-bad/log/?h=dashsink

Tried to just make things work. Will start doing a proper review of how stuff was implemented. Hopefully we can start merging this soon unless someone has further comments or concerns.
Comment 12 Thiago Sousa Santos 2016-04-18 14:42:35 UTC
One issue is that dashsink uses discoverer to inspect the fragments it gets from mp4dashmux. It would be easier to just forward the container information to it. Can the new streams API help here? dashsink somehow needs to know the formats inside the container to properly create the mpd file.
Comment 13 Michael Savin 2016-07-12 10:30:43 UTC
Please, merge this plugin to the gst-plugins-bad. It's very helpful to support fragmented MP4 for HLS.
As described in https://developer.apple.com/services-account/download?path=/Documentation/HTTP_Live_Streaming_Internet__Draft_Preview/HLSdraftpantospreview.pdf HLS in macOS v10.12 or later, iOS 10 or later, and tvOS 10 or later supports fMP4 segments, as they are used by MPEG-DASH as well.
Comment 14 Isaac Smith 2017-04-13 16:22:04 UTC
Dashsink looks awesome! I'm running into problems using it though, it generates an MPD and segments but the MPD doesn't play in the DashIF reference player (version 2.4.1), but gives the error: 'Cannot play media. No decoders for requested formats: video/mp4;codecs="undefined"'

Here's my Python streaming script: https://gist.github.com/hunternet93/d3dc2ed304c559f51836e06c77620f3f It creates three pipelines like "videotestsrc ! x264enc ! h264parse ! mp4dashmux" that are each linked to a single dashsink. Here's the generated MPD: https://video.crosscommunity.com/dashtest/GStreamer%20Adaptive%20Streaming.mpd

It'd be awesome to see this merged into GStreamer, DASH is a cool streaming tech but there's not much FOSS that can generate it.
Comment 15 Isaac Smith 2017-04-18 17:17:49 UTC
I have placed a $200 USD bounty on this issue: https://www.bountysource.com/issues/5503093-dashsink-new-sink-for-dash
Comment 16 Isaac Smith 2017-04-20 16:56:53 UTC
I think I've found the source of the issue, dashsink doesn't set the required codecs property of the <Representation> tag in the MPD. Here's the RFC on the codec property: https://tools.ietf.org/html/rfc6381 And a more human-readable description: https://stackoverflow.com/questions/16363167/html5-video-tag-codecs-attribute

Generating the correct codec string for incoming media seems to be a fairly simple job for someone familiar with the inner workings of GStreamer, if someone could code it up I'd be very grateful.
Comment 17 Isaac Smith 2017-05-10 18:28:24 UTC
Created attachment 351575 [details] [review]
Dashsink patch adding partial, untested support for the codecs property

This needs a lot of work, I am not a C programmer so I probably missed a lot. Also, still needs proper audio codec detection.
Comment 18 Isaac Smith 2017-05-10 18:38:24 UTC
After using the above patch, the dash.js reference player attempts to play the stream but dies with a media player error. It appears that the MP4 segments are getting written incorrectly, they don't play in any player I've tested.

Here's a sample MPD: https://video.crosscommunity.com/dashtest/GStreamer%20Adaptive%20Streaming.mpd 

Produced with the pipeline:

    videotestsrc ! x264enc ! h264parse ! mp4dashmux fragment-method=1 fragment-duration=2 ! dashsink fragment-duration=2

When played in dash.js on Firefox, resulting in the error "Video Element Error: MEDIA_ERR_DECODE". When played using playbin, it fails to play after the following warnings:

    0:00:00.698391401 15039 0x7f6474005540 WARN                 qtdemux qtdemux.c:6512:gst_qtdemux_process_adapter:<qtdemux0> Unknown fourcc while parsing header : styp
    0:00:00.828871236 15039 0x7f6474005540 WARN                 qtdemux qtdemux.c:6315:gst_qtdemux_process_adapter:<qtdemux0> error: This file is invalid and cannot be played.
    0:00:00.828909916 15039 0x7f6474005540 WARN                 qtdemux qtdemux.c:6315:gst_qtdemux_process_adapter:<qtdemux0> error: atom ...Q has bogus size 904110311

My best guess is that mp4dashmux isn't working correctly, but I'm way out of my depth here so I have no idea what's wrong.
Comment 19 Alex 2017-05-18 21:11:19 UTC
(In reply to Isaac Smith from comment #18)
> After using the above patch, the dash.js reference player attempts to play
> the stream but dies with a media player error. It appears that the MP4
> segments are getting written incorrectly, they don't play in any player I've
> tested.
> 
> Here's a sample MPD:
> https://video.crosscommunity.com/dashtest/GStreamer%20Adaptive%20Streaming.
> mpd 
> 
> Produced with the pipeline:
> 
>     videotestsrc ! x264enc ! h264parse ! mp4dashmux fragment-method=1
> fragment-duration=2 ! dashsink fragment-duration=2
> 
> When played in dash.js on Firefox, resulting in the error "Video Element
> Error: MEDIA_ERR_DECODE". When played using playbin, it fails to play after
> the following warnings:
> 
>     0:00:00.698391401 15039 0x7f6474005540 WARN                 qtdemux
> qtdemux.c:6512:gst_qtdemux_process_adapter:<qtdemux0> Unknown fourcc while
> parsing header : styp
>     0:00:00.828871236 15039 0x7f6474005540 WARN                 qtdemux
> qtdemux.c:6315:gst_qtdemux_process_adapter:<qtdemux0> error: This file is
> invalid and cannot be played.
>     0:00:00.828909916 15039 0x7f6474005540 WARN                 qtdemux
> qtdemux.c:6315:gst_qtdemux_process_adapter:<qtdemux0> error: atom ...Q has
> bogus size 904110311
> 
> My best guess is that mp4dashmux isn't working correctly, but I'm way out of
> my depth here so I have no idea what's wrong.

I also have a problem corresponding to the absence of 'codecs' property. I have tested similar pipeline (but without 'h264parse' before mp4dashdemux) in Chrome, Firefox and Epiphany, and only Epiphany was able to play my stream (with dash.js library) without 'codecs' property in manifest file.

But I don't have a problem (MEDIA_ERR_DECODE), described in your last message. My guess is that 'h264parse' is not necessary in your pipeline, but I am not sure.
Comment 20 Isaac Smith 2017-05-18 21:30:18 UTC
(In reply to Alex from comment #19)
> (In reply to Isaac Smith from comment #18)
> > After using the above patch, the dash.js reference player attempts to play
> > the stream but dies with a media player error. It appears that the MP4
> > segments are getting written incorrectly, they don't play in any player I've
> > tested.
> > 
> > Here's a sample MPD:
> > https://video.crosscommunity.com/dashtest/GStreamer%20Adaptive%20Streaming.
> > mpd 
> > 
> > Produced with the pipeline:
> > 
> >     videotestsrc ! x264enc ! h264parse ! mp4dashmux fragment-method=1
> > fragment-duration=2 ! dashsink fragment-duration=2
> > 
> > When played in dash.js on Firefox, resulting in the error "Video Element
> > Error: MEDIA_ERR_DECODE". When played using playbin, it fails to play after
> > the following warnings:
> > 
> >     0:00:00.698391401 15039 0x7f6474005540 WARN                 qtdemux
> > qtdemux.c:6512:gst_qtdemux_process_adapter:<qtdemux0> Unknown fourcc while
> > parsing header : styp
> >     0:00:00.828871236 15039 0x7f6474005540 WARN                 qtdemux
> > qtdemux.c:6315:gst_qtdemux_process_adapter:<qtdemux0> error: This file is
> > invalid and cannot be played.
> >     0:00:00.828909916 15039 0x7f6474005540 WARN                 qtdemux
> > qtdemux.c:6315:gst_qtdemux_process_adapter:<qtdemux0> error: atom ...Q has
> > bogus size 904110311
> > 
> > My best guess is that mp4dashmux isn't working correctly, but I'm way out of
> > my depth here so I have no idea what's wrong.
> 
> I also have a problem corresponding to the absence of 'codecs' property. I
> have tested similar pipeline (but without 'h264parse' before mp4dashdemux)
> in Chrome, Firefox and Epiphany, and only Epiphany was able to play my
> stream (with dash.js library) without 'codecs' property in manifest file.
> 
> But I don't have a problem (MEDIA_ERR_DECODE), described in your last
> message. My guess is that 'h264parse' is not necessary in your pipeline, but
> I am not sure.

I only get the MEDIA_ERR_DECODE message after fixing the codecs property issue using the patch I posted earlier.
Comment 21 Thiago Sousa Santos 2017-09-22 18:59:26 UTC
Anyone rebased this on top of master recently?
Comment 22 Sebastian Dröge (slomo) 2018-05-04 09:48:55 UTC
What's the status here? Is anybody going to take it forward or should we close it?
Comment 23 GStreamer system administrator 2018-11-03 13:10:37 UTC
-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/59.