GNOME Bugzilla – Bug 679466
[2.0] support negative DTS
Last modified: 2018-11-03 12:15:32 UTC
[I'm not hoping for this to get fixed for 1.0; filing the bug to raise awareness anyway] x264 generates one or two frames with negative DTS (one with b-frames enabled, two with b-pyramid). This is necessary for sequence such as IPBPP (PTS=0,2,1,3,4 DTS=-1,0,1,2,3) to be decoded properly - B frame in the middle has PTS 1, but needs two frames (I, B) decoded before it can be displayed. In GStreamer there is currently no way to represent a negative DTS. There are few hacks around this (encoder could compress the DTS - i.e. 0, 0.5, 1, 2, 3) or the DTS can be shifted in the encoder and shifted back in muxer, but these are either ugly or not entirely legal (compression). If anyone has a nicer/or simpler solution I'm all ears.
Some background: <d>: If your PTS starts with 0, yes, your first (or more) DTS will be negative <e>: d, it doesn't make anay sense whatsoever. <e>: it'll only be negative if you already suppress the decoding latency <d>: do what now? <d>: you mean, what you get from x264? <e>: x264 returning negative DTS <d>: what do you mean by "decoding latency"? <e>: because of reordered frames <e>: IPBPP => 1 frame delay <d>: yes, and that is why DTS is (PTS-1 frame) for non-B pictures <d>: er,well, PTS - N frames <e>: and for keyframes ? <e>: it's never PTS == DTS ? <d>: no. it's DTS = (PTS - 1) <e>: not according to 99.9999% of the streams I've seen <d>: are they reordering pictures? <e>: let's put it another way. If they weren't reordering pictures, you'd always have DTS == PTS. And if they were reordering pictures, you'd almost never have PTS == DTS <d>: if you reorder, PTS == DTS for B frames. * e scratches head <d>: say you have a standard coding sequence (in presentation order) IBBBP <d>: reordered, it is IPBBB <e>: right <d>: in order for the BBB frames to go out on time, you have to start decoding one frame early: <d>: (I)PBBB <d>: so I has a negative DTS, DTS(P) is the same as PTS(I) <d>: and the B frames have DTS == PTS
#gstreamer, 23-04-2013 20:50:05-23:50:49 UK time: matej_k: this non negative timestamps only thing really sucks matej_k: big time matej_k: also getting DTS set to -1 just because it's before segment start is pure evil matej_k: I'm really hoping gstreamer 2 will allow negative running/stream time for DTS matej_k: ds, ping ds: matej_k: yo matej_k: there's still a problem with qtmux timestamps matej_k: well, actually, it's not really qtmux matej_k: it's that first couple of frames have dts = -1 matej_k: because gstreamer can't have negative running time for DTS ds: yeah, that's a known problem. ds: it depends if we want to use the MP4 concept of DTS, or MPEG-TS concept of DTS ds: in either case, they're incompatible with each other, which will make remuxing... interesting matej_k: what do you mean incompatible? matej_k: well, actually, what do you mean by different concept? ds: MP4 uses DTS==PTS for keyframes. MPEG-TS uses DTS=PTS-1 or 2 matej_k: uh? matej_k: that's really depending on the encoder, isn't it? ds: no matej_k: how is it not? matej_k: you can't have PTS = DTS for H264 with b-pyramid matej_k: ever ds: IF you are assuming the MPEG-TS idea of DTS matej_k: but you can have same thing in MP4 matej_k: you have monotonous DTS and offset to PTS matej_k: where do you get the PTS == DTS from? matej_k: for keyframes ds: because that's how it works. It's a different definition of DTS ds: offset by one or two frames from MPEG-TS DTS values matej_k: is it in MP4 specification? ds: it's in the AVC mapping spec matej_k: that's really weird and doesn't make much sense to me. if it really is like this than maybe this should be handled at muxer / demuxed level ds: you're right, it makes no sense. matej_k: and the rest of gstreamer should use mpeg ts DTS ds: I generally agree, aside from the current failings of MP4/qtmux/qtdemux ds: Since MP4's idea of DTS is fundamentally useless matej_k: well, you still need to have the PTS offset right matej_k: anyway, mp4 doesn't bother me as much as the fact that you can't have negative running / stream time for DTS ds: indeed matej_k: ds, i added a patch https://bugzilla.gnome.org/show_bug.cgi?id=696333 ds: we probably need to do something hackish like defining -GST_SECOND to -1 as valid DTS values and go with it matej_k: ds, what's wrong with GstClockTime being guint64? ds: matej_k: GstClockTime is guint64 matej_k: ds, I meant gint matej_k: :) ds: matej_k: among other things, it would be an ABI break matej_k: the way I understand it, this won't be addressed until gst 2.0 ocrete: problem is that we're using -1 as GST_CLOCK_TIME_NONE matej_k: I'd have to be redefined ocrete: : not likely to happen.. can't you just add an offset the DTS in the demuxer ? ds: ocrete: the problem is that negative DTS values need to be carried around generally in gstreamer matej_k: ocrete, it's not just a demuxed thing matej_k: x264enc generates negative dts for example ocrete: : ds: why? ocrete: : who is -4..4 different from 0..8 ? ds: because DTS is usually negative for the first 2 frames ocrete: : matej_k: add an offset there too ? matej_k: ocrete https://bugzilla.gnome.org/show_bug.cgi?id=679466 ds: and those negative values represent a real negative value ds: er, that was circular, but you know what I mean ds: brb ocrete: : I understand that, but I expect that 2.0 is pretty far away, so it may make sense to just shift the timestamps for the forseable future ocrete: : (and manipulate the segments, etc to match) matej_k: ocrete, it still won't help you with gst_segment_to_running/stream_time ds: exactly. ds: because you still have the out-of-segment problem ocrete: : but having encoded frames out of segment are fine I think ocrete: : no? ocrete: : they'll be clipped after decoding ds: again, this is DTS, not PTS ocrete: : sure, but you can have dts timestamps outside of the segment, no ? ds: not if you expect calculations to work correctly. like converting to running time matej_k: ocrete, muxers get running time ocrete: : hmm matej_k: so the PTS that muxer gets start from 0; thus the DTS have to start from -something ds: __tim: ^ __tim: ds, matej_k: I think we de facto standardised on the mp4 variant btw matej_k: __tim, why? ds: __tim: not that I know of matej_k: why would you do that? __tim: there's quite a bit of code that assumes dts == pts for keyframes matej_k: is there? matej_k: because it's not true for most of my videos and things work __tim: that's esp. how it worked in 0.10 when you couldn't even signal dts/pts separately matej_k: also, __tim, if you say that dts == pts for keyframes, does it mean there are frames where dts > pts? matej_k: because it is bound to happen __tim: I somehow doubt that code has just gone away __tim: I don't know what it means, it's just my impression from discussions I have seen over the years matej_k: and I'm pretty sure that there is code in gstreamer that assumes that dts <= pts always ds: __tim: the code in qtmux has gone away, at least matej_k: __tim, I know base parse used to assume that dts = pts, but that's gone now __tim: ah, ok __tim: I guess that only leaves encoders+decoders then __tim: (to check) matej_k: x264enc outputs negative timestamps (mpeg ts style) matej_k: don't know about other encodes matej_k: encoders matej_k: haven't tried any, but they have same base class __tim: really? didn't it try to fix those up? matej_k: sorry, maybe you have __tim: the x264enc element matej_k: last time I looked it was few months ago i think matej_k: how did you fix it? matej_k: did you compress the timestamps ? matej_k: because it wasn't really broken, negative timestamps are a feature :) matej_k: __tim, so you added the dts_offset to x264enc? matej_k: I see now what you mean matej_k: so x264 really outputs mp4 like DTS __tim: I did not add that matej_k: well, someone must have :) matej_k: but this is not very good approach matej_k: you need to know how much to shift back when mixing mpeg ts matej_k: muxing matej_k: http://cgit.freedesktop.org/gstreamer/gst-plugins-ugly/commit/ext/x264/gstx264enc.c?id=839e75d5d145d486a5c114f8f9f7ea362d12866f matej_k: this is not good, now I have parts of pipeline that produce mp4 dts and parts that expect mpeg ts dts matej_k: __tim, would it make sense to have a sticky dts_shift event? __tim: something like that matej_k: well, I'd be nice to have something when muxing mpeg ts, 2.0 probably won't happen for a while :) MikeS: It took about 15 years for 1.0 to happen! ds: I am strongly against any method that does not put the correct DTS value in the DTS field. matej_k: ds, that's happening now matej_k: well, matej_k: depends on how you define correct DTS value I guess :) ds: The MP4 method is not correct. __tim: ds, so a) how can you do that for negative dts if you can't express negative dts directly? (other than via an additional dts offset) __tim: and I forgot what b) was ;) matej_k: well, DTS_OFFSET event would probably solve things for me right now, but it's not a pretty solution __tim: ds, I don't really care, someone just go and decide for one or the other and write a design document how it should work __tim: ds, ah yes and b) what's the "correct" dts? ds: The correct dts is the time when the decoder should start decoding a frame. Decoding is assumed to be instantaneous, so you can have dts==pts for some frames ds: but never dts > pts. __tim: right ds: it has to be exactly right to mux correct MPEG-TS streams, otherwise hardware decoders will fail matej_k: well, yeah, but maybe you can say that you can never have dts + dts_offset > pts matej_k: where dts_offset would be a sticky event ds: or you could just do it correctly from the start matej_k: ds, it's kinda late for that, isn't it ds: no matej_k: given that Gstclocktime is unsigned ds: who cares ds: it's just 64 bits. ds: we get to use them as we wish __tim: we don't really get to change the special NONE value matej_k: yeah __tim: though we can of course add a CORRECT_DTS field __tim: ds, can you always create correct-for MPEG-TS from mp4? ds: __tim: no ds: because mp4 is fundamentally broken __tim: if so, are we just talking about which is the better way of representating the same thing? matej_k: ds, isn't there way to save the offset in edit list atom? ds: matej_k: not that I know of, but I haven't read all the updates to the AVC file format spec matej_k: ds, it's not really relevant anyway. my problem now is that x264enc generates DTS and there is no way to get the exact same DTS to mpeg ts muxer __tim: matej_k, just posting "I don't think everyone agrees on this" isn't terribly helpful imho, esp. not without the background of the discussion here :) __tim: matej_k, someone needs to propose a scheme that makes sense and document it ;) matej_k: __tim, ok, I'll quote ds then :) GregDR: Does it make sense that the first buffers to hit mpegtsmux sink pads always have DTS and PTS == 0? I'm seeing that consistently with my pipeline and it is causing a bug in mpegtsmux and I just want to know if it is specific to my stream GregDR: the first video buffer has DTS == 0. After that, all video buffers only have valid PTS. This mpegtsmux prefers the last valid DTS over PTS to stamp the outgoing buffer. Therefore, my outgoing timestamps are always 0. GregDR: FYI -- this only happens on master. Not on 1.0 Also: "DTS being broken in GStreamer" post on mailing list: http://lists.freedesktop.org/archives/gstreamer-devel/2013-April/040603.html
#gstreamer, 23-04-2013 20:50:05-23:50:49 UK time: matej_k: this non negative timestamps only thing really sucks matej_k: big time matej_k: also getting DTS set to -1 just because it's before segment start is pure evil matej_k: I'm really hoping gstreamer 2 will allow negative running/stream time for DTS matej_k: ds, ping ds: matej_k: yo matej_k: there's still a problem with qtmux timestamps matej_k: well, actually, it's not really qtmux matej_k: it's that first couple of frames have dts = -1 matej_k: because gstreamer can't have negative running time for DTS ds: yeah, that's a known problem. ds: it depends if we want to use the MP4 concept of DTS, or MPEG-TS concept of DTS ds: in either case, they're incompatible with each other, which will make remuxing... interesting matej_k: what do you mean incompatible? matej_k: well, actually, what do you mean by different concept? ds: MP4 uses DTS==PTS for keyframes. MPEG-TS uses DTS=PTS-1 or 2 matej_k: uh? matej_k: that's really depending on the encoder, isn't it? ds: no matej_k: how is it not? matej_k: you can't have PTS = DTS for H264 with b-pyramid matej_k: ever ds: IF you are assuming the MPEG-TS idea of DTS matej_k: but you can have same thing in MP4 matej_k: you have monotonous DTS and offset to PTS matej_k: where do you get the PTS == DTS from? matej_k: for keyframes ds: because that's how it works. It's a different definition of DTS ds: offset by one or two frames from MPEG-TS DTS values matej_k: is it in MP4 specification? ds: it's in the AVC mapping spec matej_k: that's really weird and doesn't make much sense to me. if it really is like this than maybe this should be handled at muxer / demuxed level ds: you're right, it makes no sense. matej_k: and the rest of gstreamer should use mpeg ts DTS ds: I generally agree, aside from the current failings of MP4/qtmux/qtdemux ds: Since MP4's idea of DTS is fundamentally useless matej_k: well, you still need to have the PTS offset right matej_k: anyway, mp4 doesn't bother me as much as the fact that you can't have negative running / stream time for DTS ds: indeed matej_k: ds, i added a patch https://bugzilla.gnome.org/show_bug.cgi?id=696333 ds: we probably need to do something hackish like defining -GST_SECOND to -1 as valid DTS values and go with it matej_k: ds, what's wrong with GstClockTime being guint64? ds: matej_k: GstClockTime is guint64 matej_k: ds, I meant gint matej_k: :) ds: matej_k: among other things, it would be an ABI break matej_k: the way I understand it, this won't be addressed until gst 2.0 ocrete: problem is that we're using -1 as GST_CLOCK_TIME_NONE matej_k: I'd have to be redefined ocrete: : not likely to happen.. can't you just add an offset the DTS in the demuxer ? ds: ocrete: the problem is that negative DTS values need to be carried around generally in gstreamer matej_k: ocrete, it's not just a demuxed thing matej_k: x264enc generates negative dts for example ocrete: : ds: why? ocrete: : who is -4..4 different from 0..8 ? ds: because DTS is usually negative for the first 2 frames ocrete: : matej_k: add an offset there too ? matej_k: ocrete https://bugzilla.gnome.org/show_bug.cgi?id=679466 ds: and those negative values represent a real negative value ds: er, that was circular, but you know what I mean ds: brb ocrete: : I understand that, but I expect that 2.0 is pretty far away, so it may make sense to just shift the timestamps for the forseable future ocrete: : (and manipulate the segments, etc to match) matej_k: ocrete, it still won't help you with gst_segment_to_running/stream_time ds: exactly. ds: because you still have the out-of-segment problem ocrete: : but having encoded frames out of segment are fine I think ocrete: : no? ocrete: : they'll be clipped after decoding ds: again, this is DTS, not PTS ocrete: : sure, but you can have dts timestamps outside of the segment, no ? ds: not if you expect calculations to work correctly. like converting to running time matej_k: ocrete, muxers get running time ocrete: : hmm matej_k: so the PTS that muxer gets start from 0; thus the DTS have to start from -something ds: __tim: ^ __tim: ds, matej_k: I think we de facto standardised on the mp4 variant btw matej_k: __tim, why? ds: __tim: not that I know of matej_k: why would you do that? __tim: there's quite a bit of code that assumes dts == pts for keyframes matej_k: is there? matej_k: because it's not true for most of my videos and things work __tim: that's esp. how it worked in 0.10 when you couldn't even signal dts/pts separately matej_k: also, __tim, if you say that dts == pts for keyframes, does it mean there are frames where dts > pts? matej_k: because it is bound to happen __tim: I somehow doubt that code has just gone away __tim: I don't know what it means, it's just my impression from discussions I have seen over the years matej_k: and I'm pretty sure that there is code in gstreamer that assumes that dts <= pts always ds: __tim: the code in qtmux has gone away, at least matej_k: __tim, I know base parse used to assume that dts = pts, but that's gone now __tim: ah, ok __tim: I guess that only leaves encoders+decoders then __tim: (to check) matej_k: x264enc outputs negative timestamps (mpeg ts style) matej_k: don't know about other encodes matej_k: encoders matej_k: haven't tried any, but they have same base class __tim: really? didn't it try to fix those up? matej_k: sorry, maybe you have __tim: the x264enc element matej_k: last time I looked it was few months ago i think matej_k: how did you fix it? matej_k: did you compress the timestamps ? matej_k: because it wasn't really broken, negative timestamps are a feature :) matej_k: __tim, so you added the dts_offset to x264enc? matej_k: I see now what you mean matej_k: so x264 really outputs mp4 like DTS __tim: I did not add that matej_k: well, someone must have :) matej_k: but this is not very good approach matej_k: you need to know how much to shift back when mixing mpeg ts matej_k: muxing matej_k: http://cgit.freedesktop.org/gstreamer/gst-plugins-ugly/commit/ext/x264/gstx264enc.c?id=839e75d5d145d486a5c114f8f9f7ea362d12866f matej_k: this is not good, now I have parts of pipeline that produce mp4 dts and parts that expect mpeg ts dts matej_k: __tim, would it make sense to have a sticky dts_shift event? __tim: something like that matej_k: well, I'd be nice to have something when muxing mpeg ts, 2.0 probably won't happen for a while :) MikeS: It took about 15 years for 1.0 to happen! ds: I am strongly against any method that does not put the correct DTS value in the DTS field. matej_k: ds, that's happening now matej_k: well, matej_k: depends on how you define correct DTS value I guess :) ds: The MP4 method is not correct. __tim: ds, so a) how can you do that for negative dts if you can't express negative dts directly? (other than via an additional dts offset) __tim: and I forgot what b) was ;) matej_k: well, DTS_OFFSET event would probably solve things for me right now, but it's not a pretty solution __tim: ds, I don't really care, someone just go and decide for one or the other and write a design document how it should work __tim: ds, ah yes and b) what's the "correct" dts? ds: The correct dts is the time when the decoder should start decoding a frame. Decoding is assumed to be instantaneous, so you can have dts==pts for some frames ds: but never dts > pts. __tim: right ds: it has to be exactly right to mux correct MPEG-TS streams, otherwise hardware decoders will fail matej_k: well, yeah, but maybe you can say that you can never have dts + dts_offset > pts matej_k: where dts_offset would be a sticky event ds: or you could just do it correctly from the start matej_k: ds, it's kinda late for that, isn't it ds: no matej_k: given that Gstclocktime is unsigned ds: who cares ds: it's just 64 bits. ds: we get to use them as we wish __tim: we don't really get to change the special NONE value matej_k: yeah __tim: though we can of course add a CORRECT_DTS field __tim: ds, can you always create correct-for MPEG-TS from mp4? ds: __tim: no ds: because mp4 is fundamentally broken __tim: if so, are we just talking about which is the better way of representating the same thing? matej_k: ds, isn't there way to save the offset in edit list atom? ds: matej_k: not that I know of, but I haven't read all the updates to the AVC file format spec matej_k: ds, it's not really relevant anyway. my problem now is that x264enc generates DTS and there is no way to get the exact same DTS to mpeg ts muxer __tim: matej_k, just posting "I don't think everyone agrees on this" isn't terribly helpful imho, esp. not without the background of the discussion here :) __tim: matej_k, someone needs to propose a scheme that makes sense and document it ;) matej_k: __tim, ok, I'll quote ds then :) GregDR: Does it make sense that the first buffers to hit mpegtsmux sink pads always have DTS and PTS == 0? I'm seeing that consistently with my pipeline and it is causing a bug in mpegtsmux and I just want to know if it is specific to my stream GregDR: the first video buffer has DTS == 0. After that, all video buffers only have valid PTS. This mpegtsmux prefers the last valid DTS over PTS to stamp the outgoing buffer. Therefore, my outgoing timestamps are always 0. GregDR: FYI -- this only happens on master. Not on 1.0
This is fixed for most H264 cases now. Shall we still keep this note for 2.0 ?
Yes, it would be good to fix it properly in 2.0
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gstreamer/issues/25.