GNOME Bugzilla – Bug 565319
rtpjitterbuffer: bursty reception can cause time to be reversed
Last modified: 2008-12-31 08:51:47 UTC
Please describe the problem: In rtpjitterbuffer's calculate_skew, buffer timestamps are shifted with a skew value that changes over time. However, it is possible that the skew changes more than the time between buffers, which causes buffer timestamps to go back in time. While running, this is very unlikely to occur due to the large weighting factor of the running average. It can occur, for example, when network latency (temporarily) increases, and the skew grows to e.g. 6s. Then, suddenly, all these packets arrive in a burst, which causes delta (the difference between sending and receiving clock) to shrink suddenly to e.g. 6ms. The new skew will then be 6000*124/125 + 6 = 5958ms, and the buffer timestamp is set at 2ms less than the previous buffer timestamp. On the other hand, while the window is filling, this can occur very easily. Indeed, at that point the weighting factor decreases parabolically, so that the weighting factor approaches 0 when the window fills up. Therefore, when rtp packets come in in a burst while the window is filling, the skew shrinks quickly. Steps to reproduce: Actual results: Expected results: Does this happen every time? Other information:
Created attachment 125128 [details] Log of rtpjitterbuffer creating timestamps that go back in time This is the log from the following pipeline: gst-launch -v --gst-debug=rtpjitterbuffer:5,gstrtpjitterbuffer:5 rtspsrc location='...' ! fakesink The rtsp server is a camera that starts with sending a 2-second backlog of video it still has in its buffer. Starting at the 21st packet (line 120), the timestamps of buffers coming out of calculate_skew become smaller and smaller. The minimum timestamp is reached when the window fills up (with 31 packets). Note that the skew is at that point still converging toward the -2.3s it should be.
To fix this, I'll do two things: * keep track of the last outgoing timestamp, and make sure it never decreases; * limit the parabolic weighting factor while filling to 10 to make the evolution of skew smoother when the window is almost full. I'll post them as separate patches so you can choose which one(s) to accept.
Created attachment 125129 [details] [review] Patch to make sure time never goes backwards This patch keeps track of the last outgoing timestamp, and makes sure a new timestamp is never earlier than the previous one.
Created attachment 125130 [details] [review] Alternate patch: timestamps can go back in time if they do at the sender side This patch keeps track of the last outgoing timestamp and of the last sender-side time. Timestamps can only go forward if they do at the sender side, can only go back if they do at the sender side, and remain the same if they remain the same at the sender side.
It turns out that my second idea (limiting the parabolic weighting factor) just makes things worse for the situation I'm facing, so I'm not posting that patch.
There is also the FIXME that states that we should detect spikes and disable the skew algorithm while the spike happens.
Patch by: Arnout Vandecappelle <arnout at mind dot be> * gst/rtpmanager/rtpjitterbuffer.c: (rtp_jitter_buffer_reset_skew), (calculate_skew): * gst/rtpmanager/rtpjitterbuffer.h: Keep track of the last outgoing timestamp and of the last sender-side time. Timestamps can only go forward if they do at the sender side, can only go back if they do at the sender side, and remain the same if they remain the same at the sender side. Fixes #565319.
Regarding the detection of spikes: you'd need a real mathematician to look at that to solve it correctly. However, the problem could be alleviated by making sure the increment in out_time is between 90 and 110 percent of the increment in send_diff. The idea is that the skew should be a lot less than 10% per sample. If you think this is a good idea, clone this bug, put me in CC and I'll patch it.