GNOME Bugzilla – Bug 765797
avauddec: a/v sync issues with broken streams
Last modified: 2016-05-02 07:04:51 UTC
Every time frame decoding fails, the audio data gets skipped, but audio frame is kept in gstaudiodecoder queue, which results in timestamps being shifted backwards. I think this condition inside gst_ffmpegauddec_handle_frame is not right if (ffmpegdec->outbuf) ret = gst_audio_decoder_finish_frame (GST_AUDIO_DECODER (ffmpegdec), ffmpegdec->outbuf, 1); It only finishes frame if decoding succeeds. I think it should finish the frame regardless of whether decoding failed or not; Because otherwise when next frame comes (and decodes correctly), it's timestamps are taken from previous (failed frame), which results in A/V sync issues. Sample video http://s3.amazonaws.com/MatejK/Samples/HD-OP01-av-sync-with-failed-decoding-of-initial-frames.mkv
The problem is that we might get no frame first, and then on the next call get the previous decoded frame... in which case we will also break sync again
Maybe we should do something around the pts field in the AVFrame here, or some other way of tracking input/output.
The problem here is, if you get 10 broken frames in a row, you'll get very noticeable a/v sync problem. Whereas if first frame doesn't produce anything, you get at most one frame delay. Also it is questionable whether the resulting frame should actually have PTS of first or second frame; From what I've seen, most decoders are not really bothered by this latency, and just slap current input PTS on whatever the decoder outputs; I don't think there is reliable way to determine whether the missing frame is because of decoder latency or stream corruption; and if there is stream corruption, you just have to keep going on, finishing empty frames, otherwise the A/V sync gets lost. I'm doing this with all my audio decoders and it makes huge difference for live recordings;
Can you check with the ffmpeg people if there is a way to know about this?
We need to be able to distinguish between not getting a buffer because there's not necessarily a 1:1 input/output relationship, and not getting a buffer because of a decoding error, is that right?
Yes
I really don't think it is feasible to expect to able to distinguish between decoder latency and corrupted stream. In my experience, for practical purposes, as long as you feed the decoder entire packet (which GstAudioDecoder is supposed to do), there actually is a 1:1 input/output relationship (if you consider empty output buffer an output)
commit 00ebf95e08ab9bd241aaedda279abe73d48d9b85 Author: Sebastian Dröge <sebastian@centricular.com> Date: Fri Apr 29 12:55:19 2016 +0300 avauddec: If decoding a frame failed, skip it Otherwise the next successfully decoded frame will get its timestamp and we will slowly let a/v sync drift apart. https://bugzilla.gnome.org/show_bug.cgi?id=765797
commit 3fb49b60537c64999383e058b1bbf6ff3bec87de Author: Sebastian Dröge <sebastian@centricular.com> Date: Fri Apr 29 13:06:07 2016 +0300 avauddec: Finish frames if they are header buffers only and don't produce any output Otherwise we will consider them as one frame of raw audio that is still pending, and shift all timestamps by the amount of time spent with header buffers. https://bugzilla.gnome.org/show_bug.cgi?id=765797