GNOME Bugzilla – Bug 758949
qtmux can't handle missing DTS
Last modified: 2018-11-03 15:06:14 UTC
So using an rtsp connection over TCP I end up with no DTS in my stream. When I try to save this to an mp4 file, I get an corrupted mp4. Switching to UDP works fine. My setup is described in this bug: https://bugzilla.gnome.org/show_bug.cgi?id=758932 To summarise, gst-rtsp-server-1.0/examples/test-launch "( videotestsrc is_live=true ! x264enc ! video/x-h264,width=1296,height=972,framerate=25/1 ! h264parse ! rtph264pay pt=96 name=pay0 )" ./gst-launch-1.0 -e rtspsrc location=rtspt://127.0.0.1:8554/test ! rtph264depay ! h264parse ! mp4mux ! filesink location=~/Downloads/localtest1.mp4 The file will not play and shows only the first frame. I believe the error is in 'gst_qt_pad_adjust_buffer_dts'.
Created attachment 316660 [details] [review] Patch to use PTS as DTS I hope the conditions for using PTS are appropriate. I did test this with my setup and get a file out that I can play :)
I'm slightly concerned that something like this may previously been reverted... https://bugzilla.gnome.org/show_bug.cgi?id=707340
Indeed, this is exactly the opposite of what you are suggesting. Can you talk with Matej. I believe what we find in bug 707340 is wrong for JPEG or PNG streams (assuming this can be muxed of course). We don't require IDR type of stream to have a DTS set. I don't know why a stream where DTS set sporadically shall be considered a valid stream. It use to be the case because of the negative DTS, but I cannot find a valid use case now. Imho, the guessing of DTS is a parser or depayloader task, while muxer shall be able to assume that there is no deltal between DTS and PTS if DTS isn't set.
Well, looking again, it seems that this block: if (!GST_CLOCK_TIME_IS_VALID (pad->dts_adjustment)) { Is only hit the first time through after a pad reset. If a DTS is present at that time, I don't think my block will ever be hit. So the remaining case is that the first buffer in the JPEG/PNG stream has missing DTS and will then contain sporadic DTS from then on. Which is possible if you switch it in part way through I suppose? Maybe I could extend the logic slightly to detect sporadic DTS and disable this mode if so? The first few buffers before the first DTS might be messed up, but isn't that the case already?
You can't mux H.264 in MP4 without proper DTS and expect it to work. Your problem is that you don't have the DTS, not what muxer does if you don't provide the DTS. For video streams with B-frames you really need to have both timestamps. The muxer is not meant to figure it out. (the parser could maybe, but that's another topic). In any case, you need to fix your connection not to drop the DTS. That's your problem, not the muxer.
Matej, seems like a bug in the TCP mode of rtspsrc (though, DTS is normally deduced through arrival time, which is inaccurate in TCP mode).
Why should it be a bug in rtspsrc's TCP mode? I'd say that works all as expected, the problem is that h264parse does not calculate the DTS if missing.
You seem to assume the DTS can be calulated without a reference time. You can "probably" do that, to be proven, but as Sree mention, probably not without increasing the latecny (latency that you will only know when you have accumulated enough). I wonder if instead we can have DTS set the same way as with UDP, and let the jitterbuffer (even if not essential) smooth it up a bit.
I don't know anything about gstreamer rtsp implementation, so I have no idea whether missing DTS is bug or intentional. IIRC h264parse does try to compute DTS if missing, but it needs bufferign period SEI for that, which is often absent.
(In reply to Nicolas Dufresne (stormer) from comment #8) > I wonder if instead we can have DTS set the same way as with UDP, and let > the jitterbuffer (even if not essential) smooth it up a bit. The problem is that TCP is a quite bursty protocol because of all the buffering that happens, the packet reordering and resending inside the protocol, etc. With UDP you don't have all that and get packets as they arrive. So with UDP you get some relatively good estimate of DTS by looking at the capture times, with TCP it can be completely off. That's why we only ever take the first timestamp as a base with TCP and let everything else be interpolated.
I agree it won't be great if we'd do that. Maybe what we can do (until we figure-out something in h264parse) accept to use the PTS when DTS is absent ? This will fix all stream without b-frames. We already behave this way in flvmux for similar reasons I believe.
I'm currently using my own fork of qtmux. I've looked at the code and the entire DTS/PTS handling is different so I can't tell if the proposed patch would break anything. But it's not a very good solution. As soon as you tried muxing something with b-frames where you have non-monotonous PTS you'll get corrupted output. The way we address missing DTS is to take PTS, sort them and shift them back just enough so that DTS <= PTS. But this of course introduces another latency, and it's not always easy to know how much the timestamps need to be shifted.
(In reply to Matej Knopp from comment #12) > The way we address missing DTS is to take PTS, sort them and shift them back > just enough so that DTS <= PTS. But this of course introduces another > latency, and it's not always easy to know how much the timestamps need to be > shifted. Ok, but that should go in the parser not the muxer. How did you figure-out when you have buffered enough ?
I think to know how much to shift back you need to determine b-pyramid level somehow. Someone with better understanding of the specs might give better insight here. Our solution is not generic enough, since we don't care about latency. And even if we find out mid-stream that our generated DTS would be higher than next PTS, we just interpolate and increase the shift by one. Given that we only target specific devices we can get away with this, but it is hardly something I'd advise for h264parse to do :)
There are some calculations for this in vtdec or vtenc for guessing the latency, IIRC there's also a formula on wikipedia somewhere. That part should be the least problem
There is calculation for max_dec_frame_buffering in vtdec, but that gives really high number for videos I've tested it with (i.e. 15 or 16), whereas the DTS only need to be shifted once for streams without b-pyramid, and 1 + b-pyramid level for streams with b-pyramid. Now if someone would tell me how to detect b-pyramid level from stream I'd be very happy :)
So is there any way to insert this change as a temporary fix-up til h264parse is given a makeover?
Why do you need change in muxer for this? You can just write a simple element that sets DTS from PTS and put it before your muxer.
That would be sensible I supposed. Main reason to have it here is for equivalent behaviour to matroskamux and other muxers.
Not sure about other muxers. Matroska is a bit special since it doesn't store DTS at all. Which makes remuxing matroska to other containers interesting.
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/issues/240.