GNOME Bugzilla – Bug 610906
[assrender] subtitles can show up late or not at all
Last modified: 2015-07-27 22:46:01 UTC
This occurs most visibly with complex karaoke but I have seen it occur randomly with simple dialogue (extremely rare). Sample: http://www.megaupload.com/?d=TVFH66Q3 I use this sample video to stress test assrender's advanced capabilities. At any rate, in this video, both lateness and an occasion absence of subtitles can be observed. A 14 seconds (Apple! Picking! Beam!) and 19 seconds ("Your life changes everything!), text that should've been rendered already or rendered in sequence does not appear when expected. The first case does not render on time unless one seeks to the keyframe that starts the sequence. The latter case won't under any circumstance. Perhaps related, video stops briefly at 26 seconds. This occurs without fail using assrender. Subs will reliably, but not always cut out at around 40 seconds and not return unless one seeks towards the beginning of the video.
Please get a debug log with GST_DEBUG=ass*:5,subtitleoverlay:5 GST_DEBUG_NO_COLOR=1 and attach it here.
Created attachment 154591 [details] Debug of late and discarded subtitles
Seems to be caused by something sending wrong filler segments.
To give some more details, the problem here seems to be, that something sends a filler/update newsegment event where the start position is bigger than the subtitle buffers timestamp. This causes gst_segment_to_running_time() to return -1, which then causes the subtitle buffer to not be displayed. Fixing that might already fix this bug... another source of problem here could be, that subtitles are not rendered if they are pushed not synchronized to the video by downstream. They're then simply dropped if they're after the current video frame. The video frames never wait for subtitle buffers, only the subtitle buffers wait for video frames.
Created attachment 154771 [details] [review] assrender: Drop text buffers before the segment start and clip running time end to the segment stop
Created attachment 154774 [details] [review] matroskademux: Advance sparse streams only as much as required to keep the gap smaller than 500ms Changing it to the newest timestamp that was ever pushed will increase the segment start in 500ms jumps, which could be just after the next sparse stream buffer. E.g. Video at 1.0s, sparse stream at 0.5s would jump the sparse stream to 1.0s. Now a new sparse stream buffer could appear that has a timestamp of 0.9s and this would be dropped for no good reason because of bad luck.
These two patches fix it in this case but fixing #611194 and adjusting matroskademux properly would be better. The matroskademux patch will cause *many* newsegment updates to be sent.
I think bug 610904 may also benefit from these fixes. I'll test tonight.
The good and bad patches together seem to make the situation a bit worse. While the subs no longer disappear, the lag has increased to the point where they bleed into the next scene. Moving and positioned subtitles can no longer keep up with their targets. Unpatched good and patched bad prevents the loss of subs, but they may show up late. No continuous lag like the above. Patched good and unpatched bad has accurate timing but loses subs quite easily through seeking.
Ok, that's bad :) Thanks for testing though... Can you create another debug log with both patches and tell me which subtitles are wrong and where things start to get wrong?
Created attachment 154816 [details] [review] assrender: Drop text buffers before the segment start and clip running time end to the segment stop
Created attachment 154817 [details] [review] assrender: Consume text buffers if they're to be displayed in the next 500ms Instead of only consuming them if they're to be display now. Consuming them then might make following buffers with the same timestamp to be too late.
commit 852be755dde44bbb6a9d276a16915d521ee8e2df Author: Sebastian Dröge <sebastian.droege@collabora.co.uk> Date: Fri Feb 26 18:06:24 2010 +0100 assrender: Drop text buffers before the segment start and clip running time commit 659776f1b16715b413b672ab03381d0ddf786790 Author: Sebastian Dröge <sebastian.droege@collabora.co.uk> Date: Sat Feb 27 11:23:14 2010 +0100 assrender: Consume text buffers if they're to be displayed in the next 500ms Instead of only consuming them if they're to be display now. Consuming them then might make following buffers with the same timestamp to be too late.
The assrender patches are now pushed as they definitely improve the situation without making anything worse. Could you check again with the latest patches and unpatched or patched gst-plugins-good?
Comment on attachment 154774 [details] [review] matroskademux: Advance sparse streams only as much as required to keep the gap smaller than 500ms commit ad71d43f5221fedd4a8b2733a5e572ced93b3016 Author: Sebastian Dröge <sebastian.droege@collabora.co.uk> Date: Fri Feb 26 18:10:32 2010 +0100 matroskademux: Advance sparse streams only as much as required to keep the g Changing it to the newest timestamp that was ever pushed will increase the segment start in 500ms jumps, which could be just after the next sparse stream buffer. E.g. Video at 1.0s, sparse stream at 0.5s would jump the sparse stream to 1.0s. Now a new sparse stream buffer could appear that has a timestamp of 0.9s and this would be dropped for no good reason because of bad luck.
Created attachment 154865 [details] Debug log of newer good and bad patches combined. Subtitle timing is perfect until after seeking.
So these patches fix everything unless you seek in the movie, in which case the subtitles have wrong timing?
With respect to the patches, I will cautiously say "yes" for the time being. Seeking definitely disrupts timing though.
I'd like to amend my statement slightly. Seeking doesn't always disrupt timing. If the script is simple, there will be no observable mistiming when seeking (this doesn't mean there is none). If the script has complex elements--that is, it has moving text or crazy effects--lag will be introduced through seeking to those scenes. Seeking away from them can correct the lag.
That's a bit scary as it means that something down the path creates wrong segment or timestamp information. Could you please create a log again where you do a single seek and where subtitles are then not synchronized to a/v anymore?
Created attachment 154998 [details] Debug of single-seek distortion of subtitle timing
Reopening as the requested information has been provided.
Does this still happen with git master as of now after commit 8d837cbac41b121760207e784ba7b54f1ef84f32 Author: Arnaud Vrac <avrac@freebox.fr> Date: Thu Jan 17 19:02:48 2013 +0100 assrender: reenable unit tests and up rank to primary again ? I don't have the sample file to test myself unfortunately.
Closing this bug report as no further information has been provided. Please feel free to reopen this bug report if you can provide the information that was asked for in a previous comment. Thanks!