GNOME Bugzilla – Bug 691400
rtpbin rtpsession->RTCP thread is starting to early for ntp-sync
Last modified: 2013-07-11 14:22:13 UTC
Created attachment 233057 [details] [review] RTCP startup delay patch I want to use the ntp-sync property on the rtpbin element in order to get sync audio playback on ntp synced devices. I had a problem (see here: http://lists.freedesktop.org/archives/gstreamer-devel/2012-December/038542.html and http://lists.freedesktop.org/archives/gstreamer-devel/2013-January/038673.html) that the sending rtpbin is sending at first an RTCP SR or an RTCP RR randomly. I looked into the code - the problem is that the RTCP and RTP thread are starting in parallel. If the RTP thread has received RTP data from the RTP payloader before RTCP thread is started it will send an SR. If the RTP thread has received no data from the RTP payloader it will send an RR. According to the rfc3550 this behavior is correct. But if you want to use the ntp-sync mechanism on the receiver side it's important that you have a SR with the RTP to NTP time correlation before you start the playout - otherwise it will wait for the next RTCP packet (first SR-report), which is sended with a min delay of 2s to get that correlation. And if your playout has started already this time-jump will be audible... I made a simple patch to delay the startup of the RTCP thread with a startup delay timer (default 20ms) which will be stopped if either RTP data is ready to send or received before the timer ends. I would like to have at least some feedback on my ideas (am I right?) or have some property to enable my patch in a future version of GStreamer. cheers Thomas
I don't like the idea of adding a delay like that. It should probably just wait for the first packet to be sent before doing the RTCP.
me neither, but rtpbin is acting as a receiver as well and if there's an error and no rtp data is received it should (RFC says it) start sending receiver reports. As I said according to the rfc3550 this behavior is correct. But in order to use the ntp-sync mechanism this behavior is bad to get low latency inter device synchronous audio playback, because you don't want to buffer a few seconds, you want to start playback as fast as possible. The delay is an maximum value to wait for starting the RTCP thread if nothing is received or sended. May you have an better idea?
I think it is reasonable to wait with RTCP until you get the first RTP packet when you have requested a send_rtp_sink pad from the session.
ok - then there's another problem on the receiver (rtpbin) side. If the rtpbin - when it's receiving an SR before it created the rtpsession it will throw away the SR because it doesn't know the SSRC yet. So in my experiments (wireshark) there must be 3-4 RTP packages sended out before the SR can be correcly handled by the receiver rtpbin. But all this is tricky, because it depends on the debug log output you have... ;) ->Do you have an solution for the receiver side? ->Do you plan to modify the rtpbin that the RTCP thread is started after the request for an send_rtp_sink pad? thank you very much for your reply!
(In reply to comment #4) > ->Do you have an solution for the receiver side? It sounds like you want to quickly synchronize streams. You can't do this reliably with RTCP, you need RFC 6051. > ->Do you plan to modify the rtpbin that the RTCP thread is started after the > request for an send_rtp_sink pad? Actually, when gst_rtp_session_send_rtp() is called I want to let the RTCP thread run. The reason is that it can't do anything meaningful before it has completely processed the packet anyway.
>It sounds like you want to quickly synchronize streams. You can't do this >reliably with RTCP, you need RFC 6051. yes, I want to quickly synchronize streams. Audio recorded from A-Device synchronized played out with low latency on B-Device and C-Device (inter device sync). A,B,C-Device are ntp synced. How long will it take till RFC 6051 is available in gstreamer? (acc. to http://gstreamer.freedesktop.org/wiki/ReleasePlanning/RoadMap it's this month?;) Despite it's not reliable - might a short term solution is that the receiver bin is kinda buffering rtcp a SR when there's already a rtp package received and has not already completely processed the rtp package to setup the rtpsession!? And might this feature is only enabled if the ntp-sync property is set.
commit 2d5319c1fa4487c9e050931ce8a26e427626b763 Author: Wim Taymans <wim.taymans@collabora.co.uk> Date: Fri Feb 8 17:05:27 2013 +0100 rtpsession: delay RTCP until first RTP packet Delay sending the first RTCP packet until we have sent the first RTP packet. Otherwise we will send out a Receiver Report instead of a sender report. See https://bugzilla.gnome.org/show_bug.cgi?id=691400
(In reply to comment #6) > >It sounds like you want to quickly synchronize streams. You can't do this > >reliably with RTCP, you need RFC 6051. > yes, I want to quickly synchronize streams. Audio recorded from A-Device > synchronized played out with low latency on B-Device and C-Device (inter device > sync). A,B,C-Device are ntp synced. > How long will it take till RFC 6051 is available in gstreamer? (acc. to > http://gstreamer.freedesktop.org/wiki/ReleasePlanning/RoadMap it's this > month?;) There is no schedule for this, it will happen when it happens. > Despite it's not reliable - might a short term solution is that the receiver > bin is kinda buffering rtcp a SR when there's already a rtp package received > and has not already completely processed the rtp package to setup the > rtpsession!? And might this feature is only enabled if the ntp-sync property is > set. I don't quite understand what you propose but I think you propose to queue rtp packets until you get an RTCP packet, then output the queued rtp packets with proper timestamps as calculated from RTCP. I think this option is not so interesting because there might be a lot of time between receiving RTP and the RTCP packet, you would likely exceed the latency configured in the pipeline. What sounds like a better idea is to simply drop RTP packets until you get an RTCP packet. That doesn't sound like something complicated to implement and would accomplish basically the same thing (you might loose a couple of packets before the RTCP that you could have played maybe)
No, I propose it the other way round. I like to kinda buffering (not drop) rtCp packets on the receiver side if there's already a RTP packet received, but the rtpbin has not yet a session opened for it. At the moment (with the patch) the receiver rtpbin is dropping RTCP packets if they receive shortly after the first RTP packets (up to 4 RTP packets) - depending on the debug level. So either speed up the rtp-session setup or queue RTCP packets while processing RTP (setup the rtpsession). The rtp packets are buffered in the jitterbuffer, so while they are in the jitterbuffer their timestamps can be adjusted with the RTCP timestamp!?
(In reply to comment #9) > No, I propose it the other way round. > I like to kinda buffering (not drop) rtCp packets on the receiver side if > there's already a RTP packet received, but the rtpbin has not yet a session > opened for it. what do you mean with 'session open for it'? > > At the moment (with the patch) the receiver rtpbin is dropping RTCP packets if > they receive shortly after the first RTP packets (up to 4 RTP packets) - no RTCP is ever dropped AFAIK, where do you see this being dropped? > depending on the debug level. So either speed up the rtp-session setup or queue > RTCP packets while processing RTP (setup the rtpsession). I don't understand what you mean.. > > The rtp packets are buffered in the jitterbuffer, so while they are in the > jitterbuffer their timestamps can be adjusted with the RTCP timestamp!? Yes, it can be done and that's what happens. I don't undestand what this is about
(In reply to comment #9) > > At the moment (with the patch) the receiver rtpbin is dropping RTCP packets if > they receive shortly after the first RTP packets (up to 4 RTP packets) - > depending on the debug level. So either speed up the rtp-session setup or queue > RTCP packets while processing RTP (setup the rtpsession). I see what you mean with the dropping of the RTCP packets. This actually happens in the jitterbuffer when it has no RTP packets yet.
This should improve things quite a bit regarding RTCP arriving before the first RTP packet. commit 519305d14db6049721c5ec426fe0ea6edc69cf41 Author: Wim Taymans <wim.taymans@collabora.co.uk> Date: Thu Jun 27 16:23:20 2013 +0200 jitterbuffer: improve sync on first packets Don't throw away the first RTCP packet if it arrives before the first RTP packet but remember and use it to signal sync once we get the RTP packet. See https://bugzilla.gnome.org/show_bug.cgi?id=691400
thank you a lot - I couldn't reproduce the bug anymore. I'm very happy now - like to buy you a beer in Edinburgh cheers, Thomas