GNOME Bugzilla – Bug 731352
flv: Container timestamp is DTS not PTS
Last modified: 2015-06-03 09:31:52 UTC
The flv container contains DTS, not PTS. The PTS is obtain through the composition time (CTS): PTS = DTS + CTS Our implementation do the opposite, hence there is bugs when DTS != PTS. For the demuxing, it result in a lot of warning, and for muxing it produce streams that are very choppy on adobe flash and vlc. I have working patches to fix the video tag, will need to finish before publishing.
Created attachment 278077 [details] [review] [PATCH] flv: Tag timestamp are DTS not PTS The tags in FLV are DTS. In audio cases, and for many video format this makes no difference, but for AVC with B-Frames, PTS need to be computed from composition timestamp CTS, with PTS = DTS + CTS. https://bugzilla.gnome.org/show_bug.cgi?id=731352 --- gst/flv/gstflvdemux.c | 76 +++++++++++++++++++++++++-------------------------- gst/flv/gstflvdemux.h | 2 +- gst/flv/gstflvmux.c | 59 +++++++++++++++++++++++---------------- 3 files changed, 75 insertions(+), 62 deletions(-)
For the curious, try these with VLC, you'll see the difference: Before: http://people.collabora.co.uk/~nicolas/test.flv And after: http://people.collabora.co.uk/~nicolas/test-after.flv
Ok, remains one thing the need to be implemented, the spec says: Timestamp: Time in milliseconds at which the data in this tag applies. This value is relative to the first tag in the FLV file, which always has a timestamp of 0. We don't offset to zero the timestamp. I think this is important, as it will bring all streams to the same base (av sync), and will reduce the risk of overflow.
Created attachment 281894 [details] [review] [PATCH] flvmux: Ensure Timestamp starts at 0 FLV documentation stipulates that timestamp must start at zero. In order to respect this rule, keep the first timestamp around and offset the timestamp from this value. This allow for longer recording time in presence of timestamp that does not start at 0 already. https://bugzilla.gnome.org/show_bug.cgi?id=73135 --- gst/flv/gstflvmux.c | 7 +++++++ gst/flv/gstflvmux.h | 1 + 2 files changed, 8 insertions(+)
The code does not completely fix the issue with GStreamer not able to send streams with B-frames to CDNs such as YouTube Live. In fact, if released implemented like that, GStreamer can no longer be used to feed a stream to YouTube Live. More information here http://lists.freedesktop.org/archives/gstreamer-devel/2014-August/049191.html Regards Peter Maersk-Moller
I've tested these changes on top of 1.5 with live Youtube, and it works for me. It is likely that the recent timestamp changes to x264 enc were needed. gst-launch-1.0 \ videotestsrc is-live=1 \ ! video/x-raw,width=1280,height=720,framerate=30/1 \ ! timeoverlay \ ! x264enc bitrate=2000 ! video/x-h264,profile=main \ ! queue ! mux. \ audiotestsrc is-live=1 wave=12 ! faac ! queue ! mux. \ flvmux streamable=1 name=mux \ ! rtmpsink location="rtmp://a.rtmp.youtube.com/live2/x/<name> app=live2" To disable b-frames, simply select profile=constrained-baseline or bframes=0. The high profile (default) results in slightly choppy experience, though it's not rejected, so maybe it's due to my computer CPU being too slow for the task. It seems that adding flashVer and the datarate parameters (as described in Snowmix test documentation) may cause issues.
Comment on attachment 278077 [details] [review] [PATCH] flv: Tag timestamp are DTS not PTS commit ff2bce7b26a175621d41eea6071df3b268e5a18e Author: Nicolas Dufresne <nicolas.dufresne@collabora.co.uk> Date: Fri Jun 6 23:17:52 2014 -0400 flv: Tag timestamp are DTS not PTS The tags in FLV are DTS. In audio cases, and for many video format this makes no difference, but for AVC with B-Frames, PTS need to be computed from composition timestamp CTS, with PTS = DTS + CTS. https://bugzilla.gnome.org/show_bug.cgi?id=731352
Comment on attachment 281894 [details] [review] [PATCH] flvmux: Ensure Timestamp starts at 0 commit c1e7bec616d29a975a43e9fa69a5aa0250a044c4 Author: Nicolas Dufresne <nicolas.dufresne@collabora.co.uk> Date: Mon Jul 28 20:58:59 2014 -0400 flvmux: Ensure Timestamp starts at 0 FLV documentation stipulates that timestamp must start at zero. In order to respect this rule, keep the first timestamp around and offset the timestamp from this value. This allow for longer recording time in presence of timestamp that does not start at 0 already. https://bugzilla.gnome.org/show_bug.cgi?id=731352
Looks like the second will break synchronization if all stream don't start at the same time.
I just notice that the first buffer on each stream has an invalid timestamp, I'm not sure why yet, but this leads to first_timestamp to always be set to zero, hence the "offending" patch having no effect at all.
As Sebastien explained, keep the first timestamp rather then timestamp per stream. Also fixed a regression with tag timestamp. commit a7a3cb343a2da8e035539d8aa511391a4991f97d Author: Nicolas Dufresne <nicolas.dufresne@collabora.com> Date: Wed Aug 27 21:07:26 2014 -0400 flvmux: Correctly offset timestamp The previous method would break AV sync in the case audio or video didn't start at the same point in running time. https://bugzilla.gnome.org/show_bug.cgi?id=731352 commit aa5bd991276dd3ff5403890972e2aba832f07eba Author: Nicolas Dufresne <nicolas.dufresne@collabora.com> Date: Wed Aug 27 20:56:12 2014 -0400 flvmux: Save dts from buffer We no longer set dts in muxed buffer. This would lead to encoding tags with timestamp 0 instead of the timestamp of previous buffer. https://bugzilla.gnome.org/show_bug.cgi?id=731352
Hi Nicolas (In reply to comment #6) > I've tested these changes on top of 1.5 with live Youtube, and it works for me. > It is likely that the recent timestamp changes to x264 enc were needed. > > gst-launch-1.0 \ > videotestsrc is-live=1 \ > ! video/x-raw,width=1280,height=720,framerate=30/1 \ > ! timeoverlay \ > ! x264enc bitrate=2000 ! video/x-h264,profile=main \ > ! queue ! mux. \ > audiotestsrc is-live=1 wave=12 ! faac ! queue ! mux. \ > flvmux streamable=1 name=mux \ > ! rtmpsink location="rtmp://a.rtmp.youtube.com/live2/x/<name> app=live2" > > To disable b-frames, simply select profile=constrained-baseline or bframes=0. > The high profile (default) results in slightly choppy experience, though it's > not rejected, so maybe it's due to my computer CPU being too slow for the task. > It seems that adding flashVer and the datarate parameters (as described in > Snowmix test documentation) may cause issues. I have tested it with 1.4.0 version you send me the Git references for. It is correct your solution will stream with YouTube Live with the 1.4.0 modified version. Your solution will however also stream with the parameters set by the example on the Snowmix page https://sourceforge.net/p/snowmix/wiki/Snowmix%20and%20CDNs/ so I guess the extra parameters may no longer be required. Either an update in the librtmp or perhaps on the YouTube/Google side of the things. Anyway I suspect why your solution with or without the extra parameters works is that both streams start at t=0. Now if you use the following two pipelines, then it does NOT work with the modified flvmux. The stream is accepted at YouTube as 'Good', but you never can get to see the Preview nor actual start a Live session. This both applies if bframes are present and bframes not present. I suspect this is a issue with starting values of DTS and PTS. The pipelines are: The producer: gst-launch-1.0 videotestsrc is-live=1 !\ video/x-raw,width=1280,height=720,framerate=24/1 !\ timeoverlay !\ x264enc bitrate=4000 !\ video/x-h264,profile=main !\ queue !\ mux. audiotestsrc is-live=1 wave=0 !\ audio/x-raw,channels=2 !\ faac !\ queue !\ mux. matroskamux name=mux streamable=true !\ queue !\ tcpserversink host=0.0.0.0 port=5010 sync-method=2 recover-policy=keyframe The sender: gst-launch-1.0 -v tcpclientsrc host=127.0.0.1 port=5010 do-timestamp=true !\ queue !\ matroskademux name=demuxer !\ queue !\ aacparse !\ muxer. demuxer. !\ queue !\ h264parse !\ flvmux name=muxer streamable=true !\ queue !\ rtmpsink location="rtmp://a.rtmp.youtube.com/live2/x/<YOUR_AUTH> app=live2" It should be noted that IT DOES WORK with GStreamer 1.4.0 WITHOUT the modified flvmux (ie. the old flvmux) as long as no bframes are present. So I would say that the status = RESOLVED/FIXED is a little bit overstated. If you have any questions, feel free to contact me directly or through here. Is the modified flvmux I use was downloaded, compiled and installed using your instructions being: git clone git://git.collabora.co.uk/git/user/nicolas/gst-plugins-good.git git checkout flv-ts-fix-1.4 Is this still the version I should be testing? If I get Gitmaster, which I guess is 1.5.0, is it then the new not working flvmux code or is it the old that works with non bframes streams? Best regards Peter Maersk-Moller
(In reply to comment #12) > Hi Nicolas > > (In reply to comment #6) > > I've tested these changes on top of 1.5 with live Youtube, and it works for me. > > It is likely that the recent timestamp changes to x264 enc were needed. > > > > gst-launch-1.0 \ > > videotestsrc is-live=1 \ > > ! video/x-raw,width=1280,height=720,framerate=30/1 \ > > ! timeoverlay \ > > ! x264enc bitrate=2000 ! video/x-h264,profile=main \ > > ! queue ! mux. \ > > audiotestsrc is-live=1 wave=12 ! faac ! queue ! mux. \ > > flvmux streamable=1 name=mux \ > > ! rtmpsink location="rtmp://a.rtmp.youtube.com/live2/x/<name> app=live2" > > > > To disable b-frames, simply select profile=constrained-baseline or bframes=0. > > The high profile (default) results in slightly choppy experience, though it's > > not rejected, so maybe it's due to my computer CPU being too slow for the task. > > It seems that adding flashVer and the datarate parameters (as described in > > Snowmix test documentation) may cause issues. > > I have tested it with 1.4.0 version you send me the Git references for. It can't really work on 1.4 without also patching gst-plugins-ugly. flv specific timestamp as dts and positive offset toward pts. x264enc was shifting only the dts to make it positive, which would break the value for the offset. As said, if you only have the two patch I suggested on top of 1.4 it will likely not work. See patch 698714fc9781c614ea5d65744508d5db73f16066 from gst-plugins-ugly. Along with the 4 patches in the bug report. To be on the same page, I would recommend you to test on git master, these changes won't be backported to 1.4 anyway. > > It is correct your solution will stream with YouTube Live with the 1.4.0 > modified version. Your solution will however also stream with the parameters > set by the example on the Snowmix page > https://sourceforge.net/p/snowmix/wiki/Snowmix%20and%20CDNs/ so I guess the > extra parameters may no longer be required. Either an update in the librtmp or > perhaps on the YouTube/Google side of the things. I must admit I didn't test with re-adding these parameters (which are flash specific, so if they are not needed, I just prefer not setting them), so indeed it's likely working both ways now due to update on Google server side. > > Anyway I suspect why your solution with or without the extra parameters works > is that both streams start at t=0. No, that was my mistake. It's fixed now. Timestamp starts at 0 (first package is header and has pts = dts = cts = 0). Then first stream packet is offset to zero (in running time), and all following packet are offset from the first packet timestamp. All timestamp where converted to stream-time before end, so they correlate together and can be offset this way. This is to respect the spec, unless the spec is proven wrong. If the spec was wrong, there is more changes needed to get initial header timestamp right. > > Now if you use the following two pipelines, then it does NOT work with the > modified flvmux. The stream is accepted at YouTube as 'Good', but you never can > get to see the Preview nor actual start a Live session. This both applies if > bframes are present and bframes not present. I suspect this is a issue with > starting values of DTS and PTS. > > The pipelines are: > > The producer: > > gst-launch-1.0 videotestsrc is-live=1 !\ > video/x-raw,width=1280,height=720,framerate=24/1 !\ > timeoverlay !\ > x264enc bitrate=4000 !\ > video/x-h264,profile=main !\ > queue !\ > mux. audiotestsrc is-live=1 wave=0 !\ > audio/x-raw,channels=2 !\ > faac !\ > queue !\ > mux. matroskamux name=mux streamable=true !\ > queue !\ > tcpserversink host=0.0.0.0 port=5010 sync-method=2 recover-policy=keyframe > > The sender: > > gst-launch-1.0 -v tcpclientsrc host=127.0.0.1 port=5010 do-timestamp=true !\ > queue !\ > matroskademux name=demuxer !\ > queue !\ > aacparse !\ > muxer. demuxer. !\ > queue !\ > h264parse !\ > flvmux name=muxer streamable=true !\ > queue !\ > rtmpsink location="rtmp://a.rtmp.youtube.com/live2/x/<YOUR_AUTH> app=live2" > > It should be noted that IT DOES WORK with GStreamer 1.4.0 WITHOUT the modified > flvmux (ie. the old flvmux) as long as no bframes are present. I confirm this does not work for me either. I'll file a separate bug to address this. It would seem that the dts is lost along the way in the matroskamux ! matroskademux. I'll re-add support for missing DTS of course (will use PTS as DTS), but that will leave b-frames broken. Something is wrong in matroska. > > So I would say that the status = RESOLVED/FIXED is a little bit overstated. Well the bug report is about generating invalid flv stream in presence of b-frames. There is good reason to achieve this work in development branch, and good reason I'm not going to backport to 1.4. Thanks for reporting, I'll open corresponding issues for matroska stipping off dts values.
For the cross-reference, matroska issue with dts: https://bugzilla.gnome.org/show_bug.cgi?id=735628 And this brings back to working state (without b-frames) the transmux method: commit 356defdfea85c40d50283849a2ed8f28a9b64b2c Author: Nicolas Dufresne <nicolas.dufresne@collabora.co.uk> Date: Thu Aug 28 15:03:50 2014 -0400 flvmux: Fallback to PTS if DTS is missing Fixing a regression introduce when fixing: https://bugzilla.gnome.org/show_bug.cgi?id=731352
Hi Nicolas. I confirm that with the latest Git Master 1.5.0, I can now stream over YouTube Live when not using b-frames. It also works when I first matroskamux the stream over a tcpserversink as described. As you write, it does not work if I include b-frames when first matroskamuxing the stream making available over a tcpserversink. The reason I prefer matroska muxing the stream and making it available over a tcpserversink before demuxing it, flvmuxing it and then send it to YouTube is that I need to decouple the process creating/encoding the stream and the process sending the stream to YouTube. The latter process may sometime need a restart if something goes wrong in YouTube land without having to interrupt the encoding process as other processes using the stream (file recording and streaming to other CDNs like UStream and LiveStream) should be able to continue without interruption. Thanks for the effort. It is much appreciated.
Thanks for taking the time to reproduce and giving me instructions to test. Hopefully this use case will also be supported as soon as h264parse learn to generate missing timestamp (in the matroska case, the DTS is missing). See https://bugzilla.gnome.org/show_bug.cgi?id=659489 if you want to keep track of this effort.