GNOME Bugzilla – Bug 630256
rtph264-pay/depay: doesn't respect timestamps from incomming buffers
Last modified: 2010-11-30 23:26:18 UTC
I expect that a h264-stream that is passed through a rtppayloader and rtpdepayloader is still the same stream at the end. But it looks like the timestamps are not 100% the same. See bellow for an example: $gst-launch -v filesrc location=../elephants_dream/elephantsdream-480-h264.mkv ! matroskademux ! decodebin ! fakesink | grep chain | head /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (194400 bytes, timestamp: 0:00:00.000000000, duration: 0:00:00.458333333, offset: -1, offset_end: -1, flags: 32) 0x2005700" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (194400 bytes, timestamp: 0:00:00.458000000, duration: 0:00:00.041000000, offset: -1, offset_end: -1, flags: 256) 0x2005600" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (194400 bytes, timestamp: 0:00:00.500000000, duration: 0:00:00.041000000, offset: -1, offset_end: -1, flags: 256) 0x2213b90" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (194400 bytes, timestamp: 0:00:00.542000000, duration: 0:00:00.041000000, offset: -1, offset_end: -1, flags: 256) 0x2213c10" $gst-launch -v filesrc location=../elephants_dream/elephantsdream-480-h264.mkv ! matroskademux ! rtph264pay ! rtph264depay ! decodebin ! fakesink | grep chain | head /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (194400 bytes, timestamp: 0:00:00.500000000, duration: 0:00:00.040000000, offset: -1, offset_end: -1, flags: 0) 0x2618520" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (194400 bytes, timestamp: 0:00:00.458000000, duration: 0:00:00.040000000, offset: -1, offset_end: -1, flags: 256) 0x2618020" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (194400 bytes, timestamp: 0:00:00.500000000, duration: 0:00:00.040000000, offset: -1, offset_end: -1, flags: 256) 0x2618420" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (194400 bytes, timestamp: 0:00:00.542000000, duration: 0:00:00.040000000, offset: -1, offset_end: -1, flags: 256) 0x26184a0"
It looks like the depayloader does some crazy things with the timestamps.
Created attachment 170820 [details] [review] rtph264depay: also timestamp the buffers with codec-data
This fixes the wrong timestamps in the example above.
You pipeline has decodebin in it, which reorders timestamps. You probably wanted: gst-launch -v filesrc location=../elephants_dream/elephantsdream-480-h264.mkv ! matroskademux ! fakesink | grep chain | head
I rewrote that dodgy code in this commit, please try again: commit f5c65a919fc4f8bf90c0c7720c92a7471a454e6c Author: Wim Taymans <wim.taymans@collabora.co.uk> Date: Wed Sep 22 12:37:33 2010 +0200 rtph264depay: refactor and simplify AU merging Move the processing of the NALU to a separate method. Simplify the merging of NALU into AU and use common code when possible.
The issue above is fixed, however I have another issue that I think the depayloader is the cause of (and is related to this). If I do this test over network (with full rtp/rtcp) I record the data after rtph264depay (video_h264.gdp). But the decoder is producing strange timestamps for this stream. gst-launch -v filesrc location=video_h264.gdp ! gdpdepay ! ffdec_h264 ! fakesink | grep chain | head -n 15 /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:00.739234437, duration: none, offset: -1, offset_end: -1, flags: 32) 0xa0c810" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:00.739234437, duration: none, offset: -1, offset_end: -1, flags: 256) 0xa0cb90" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:00.739234436, duration: none, offset: -1, offset_end: -1, flags: 256) 0xa0cc10" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:00.739234438, duration: none, offset: -1, offset_end: -1, flags: 256) 0xa0c910" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:00.580115036, duration: none, offset: -1, offset_end: -1, flags: 256) 0x7ce600" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:00.875055806, duration: none, offset: -1, offset_end: -1, flags: 256) 0xa0cb10" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:00.875055805, duration: none, offset: -1, offset_end: -1, flags: 256) 0xa0c810" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:00.875055807, duration: none, offset: -1, offset_end: -1, flags: 256) 0x7ce700" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:00.739234439, duration: none, offset: -1, offset_end: -1, flags: 256) 0xa0ca10" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:00.933831367, duration: none, offset: -1, offset_end: -1, flags: 256) 0xa0ca90" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:00.933831366, duration: none, offset: -1, offset_end: -1, flags: 256) 0x7ce680" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:00.933831368, duration: none, offset: -1, offset_end: -1, flags: 256) 0xa0cb10" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:00.875055808, duration: none, offset: -1, offset_end: -1, flags: 256) 0xa0ca10" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:01.062411317, duration: none, offset: -1, offset_end: -1, flags: 256) 0xa0c810" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:01.062411316, duration: none, offset: -1, offset_end: -1, flags: 256) 0xa0c890"
Created attachment 170843 [details] dump of h264 stream
The code looks like it should do the right thing now. Can you provide a dump of what goes into h264depay? Are you using a jitterbuffer in your pipeline?
Do not know if it is related, but if operating in AU merging, current code might have a problem in the STAP-A case, where _handle_nal is actually handled a bunch of NALUs (whereas it expects and inspects only 1). This could confuse the parsing heuristic, and lead to wrong AU with maybe twisted timestamps. AFAICS, calling _handle_nal for each of the STAP-A fragments should handle that.
(In reply to comment #8) > The code looks like it should do the right thing now. Can you provide a dump of > what goes into h264depay? Are you using a jitterbuffer in your pipeline? Yes, I'm using full rtp/rtcp stack. This stream is fully generated with GStreamer (streamer/receiver) with GStreamer upstream elements. So should be possible to experience the same behaviour. I assume it has something todo with the reordering of the incomming buffers.
Created attachment 171191 [details] dump of rtp stream after gstrtpbin
gst-launch -v filesrc location=dump_h264.gdp ! gdpdepay ! rtph264depay ! decodebin ! fakesink | grep chain | head -n 6 /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:01.206507848, duration: none, offset: -1, offset_end: -1, flags: 32) 0x169b320" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:01.238020168, duration: none, offset: -1, offset_end: -1, flags: 256) 0x169b120" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:01.280392748, duration: none, offset: -1, offset_end: -1, flags: 256) 0x1479600" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:01.314194684, duration: none, offset: -1, offset_end: -1, flags: 256) 0x169b3a0" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:01.365749203, duration: none, offset: -1, offset_end: -1, flags: 256) 0x169b320" /GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = "chain ******* < (460800 bytes, timestamp: 0:00:01.360274697, duration: none, offset: -1, offset_end: -1, flags: 256) 0x169b220"
Created attachment 171249 [details] server-H264.sh This scripts demonstrates this bug, it's just the example script for H264/PCMA with the audio part removed. As client the example script can be used untouched, if you change the sink to fakesink you can see the problems with the timestamps. (as well as visual with xvimagesink) http://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/tests/examples/rtp/client-H264.sh
This was fixed with some commits for ffdec_* timestamp calculation.