GNOME Bugzilla – Bug 654850
rtph264depay adds one frame of latency in access-unit aggregation mode
Last modified: 2013-02-15 11:33:19 UTC
Since the aggregation code waits until the next frame to close the previous access-unit, it always adds at least one frame of latency. Two ways to mitigate this: 1. Use the marker bit, it indicates the end of an AU 2. Even after that, I think the depayloader should still report some latency in case the marker bit is missing (the RFC says to not rely on it). Since we have no way to know the framerate, I guess it should be a property. Unless someone has a smarter idea ?
Created attachment 192194 [details] [review] rtph264depay: report incurred latency when providing AU output Patch to (at least) report latency (based on estimated framerate).
Maybe there should be a way for the jitterbuffer to report that the next packet has a different RTP timestamp and is therefore in a different AU.
Created attachment 192220 [details] [review] rtph264pay: Complete merged AU on marker bit This patch (to use the RTP marker bit) mostly fixes my test case.. But corrupts the following pipeline for a reason I don't understand: v4l2src ! video/x-raw-yuv, width=320, height=240 ! x264enc tune=zerolatency profile=baseline ! rtph264pay ! rtph264depay ! video/x-h264, alignment=au, stream-format=avc ! ffdec_h264 ! xvimagesink
nevermind, it seems to work now.. Btw, ffdec_h264 fails with alignment=nal, stream-format=avc.. This should probably be represented in the caps.
Actually, your patch also solves my test case, but I'm concerned that it really only evaluates the latency from the first few frames. Also, you probably only want to increase the latency is "rtph264depay->merge" is set. And my patch does break ffdec_h264 but seems to work fine with the TI dsp decoder.
Created attachment 192223 [details] [review] rtph264pay: Only set the marker bit on the last NALU of a multi-NALU access unit Turns out the previous pipeline failed because rtph264pay fails sets the marker bit at the end of every NAL unit and not just at the end of the last one of an access unit, attached patch also fixes that. This is only triggered if you have more tghan one slice per image (that is more than one NAL per access unit).
Also, found a new pipeline which just doesn't work: v4l2src ! video/x-raw-yuv, width=640, height=480 ! x264enc tune=zerolatency ! video/x-h264, stream-format=byte-stream, profile=constrained-baseline ! rtph264pay scan-mode=1 ! udpsink
Created attachment 192224 [details] [review] rtph264pay: Only set the marker bit on the last NALU of a multi-NALU access unit Updated to only set the marker bit if alignment=au, otherwise there is no way we can know where the AU ends.
Thanks, committed the following, not sure whether that makes it resolved all the way, due to possibly remaining latency thingy ... commit 2591a882aed16fb7e8ed175af38e1ecc4e6056c8 Author: Olivier Crête <olivier.crete@collabora.com> Date: Mon Jul 18 16:46:27 2011 -0400 rtph264depay: Complete merged AU on marker bit The marker bit on a RTP packet means the AU has been completed, so push it out immediately to reduce the latency. https://bugzilla.gnome.org/show_bug.cgi?id=654850 commit 118a7cc36a0fb6b289927f199c78441a165fbae3 Author: Olivier Crête <olivier.crete@collabora.com> Date: Mon Jul 18 20:27:38 2011 -0400 rtph264pay: Only set the marker bit on the last NALU of a multi-NALU access unit An access unit could contain multiple NAL units, in that case, only the last RTP packet of the last NALU should have its marker bit set. https://bugzilla.gnome.org/show_bug.cgi?id=654850
Created attachment 192627 [details] [review] rtph264depay: Make output in AVC stream format work even without complete sprop-parameter-set This allows outputting streams in AVC format even if the SPS/PPS are sent inside the RTP stream. With the current patches, it works great with gst-dsp, because it doesn't try to do AU aggregation, but gst-ffmpeg will re-introduce the frame of latency, because it will wait for the start of the next AU before starting to decode the current one. So, sending it in AVC format fixes that, except that AVC only works if the SPS/PPS are in the caps, so we have to parse them out of the RTP stream and delay setting the SPS/PPS have arrived. There is one problem with that, the GstBaseRTPDepayload base class doesn't allow us to return the not-negotiated error back if the setcap fails, so we just ignore it.. But I guess it will get an error on the push, so it shouldn't be too bad, I guess.
Created attachment 192629 [details] [review] ffmpeg: Can only decode H.264 as whole access units, so put such in the caps In byte-stream mode, it does parsing and can aggregate access units, but that fails in avc mode.
(In reply to comment #10) > With the current patches, it works great with gst-dsp, because it doesn't try > to do AU aggregation, but gst-ffmpeg will re-introduce the frame of latency, > because it will wait for the start of the next AU before starting to decode the > current one. So, sending it in AVC format fixes that, except that AVC only > works if the SPS/PPS are in the caps, so we have to parse them out of the RTP > stream and delay setting the SPS/PPS have arrived. Should work the same with gst-av ;)
Created attachment 208652 [details] [review] rtph264depay: Make output in AVC stream format work even without complete sprop-parameter-set This allows outputting streams in AVC format even if the SPS/PPS are sent inside the RTP stream.
Comment on attachment 208652 [details] [review] rtph264depay: Make output in AVC stream format work even without complete sprop-parameter-set Alright, because of bug #654850, I pushed this patch from here. commit 053f33adc8bd1325c01ef373469dc116ec624346 Author: Olivier Crête <olivier.crete@collabora.com> Date: Thu Jul 14 16:23:49 2011 -0400 rtph264depay: Make output in AVC stream format work even without complete sp This allows outputting streams in AVC format even if the SPS/PPS are sent in the RTP stream. https://bugzilla.gnome.org/show_bug.cgi?id=654850
Please port to 0.11.
Comment on attachment 192629 [details] [review] ffmpeg: Can only decode H.264 as whole access units, so put such in the caps Seems like this is already there in 1.0
Sounds like all of these patches are in now except for the one that tries to guess the latency. Completing on markers makes the latency 0 if there is no lost packet.
This seems to have exposed some issue in GstVideoDecoder/avdec_h264 which can lead to crashes it seems, see bug #693581 .