After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 610906 - [assrender] subtitles can show up late or not at all
[assrender] subtitles can show up late or not at all
Status: RESOLVED INCOMPLETE
Product: GStreamer
Classification: Platform
Component: gst-plugins-bad
git master
Other Linux
: Normal normal
: git master
Assigned To: GStreamer Maintainers
GStreamer Maintainers
Depends on: 610904 611194
Blocks: 604067
 
 
Reported: 2010-02-24 04:22 UTC by Eric Appleman
Modified: 2015-07-27 22:46 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Debug of late and discarded subtitles (111.78 KB, application/x-gzip)
2010-02-24 14:26 UTC, Eric Appleman
  Details
assrender: Drop text buffers before the segment start and clip running time end to the segment stop (2.70 KB, patch)
2010-02-26 17:10 UTC, Sebastian Dröge (slomo)
none Details | Review
matroskademux: Advance sparse streams only as much as required to keep the gap smaller than 500ms (2.09 KB, patch)
2010-02-26 17:13 UTC, Sebastian Dröge (slomo)
committed Details | Review
assrender: Drop text buffers before the segment start and clip running time end to the segment stop (1.67 KB, patch)
2010-02-27 10:24 UTC, Sebastian Dröge (slomo)
committed Details | Review
assrender: Consume text buffers if they're to be displayed in the next 500ms (2.46 KB, patch)
2010-02-27 10:24 UTC, Sebastian Dröge (slomo)
committed Details | Review
Debug log of newer good and bad patches combined. (616.18 KB, application/x-gzip)
2010-02-27 22:21 UTC, Eric Appleman
  Details
Debug of single-seek distortion of subtitle timing (204.88 KB, application/gzip)
2010-03-02 02:22 UTC, Eric Appleman
  Details

Description Eric Appleman 2010-02-24 04:22:32 UTC
This occurs most visibly with complex karaoke but I have seen it occur randomly with simple dialogue (extremely rare).

Sample: http://www.megaupload.com/?d=TVFH66Q3

I use this sample video to stress test assrender's advanced capabilities.

At any rate, in this video, both lateness and an occasion absence of subtitles can be observed.

A 14 seconds (Apple! Picking! Beam!) and 19 seconds ("Your life changes everything!), text that should've been rendered already or rendered in sequence does not appear when expected. The first case does not render on time unless one seeks to the keyframe that starts the sequence. The latter case won't under any circumstance.

Perhaps related, video stops briefly at 26 seconds. This occurs without fail using assrender.

Subs will reliably, but not always cut out at around 40 seconds and not return unless one seeks towards the beginning of the video.
Comment 1 Sebastian Dröge (slomo) 2010-02-24 14:01:31 UTC
Please get a debug log with GST_DEBUG=ass*:5,subtitleoverlay:5 GST_DEBUG_NO_COLOR=1 and attach it here.
Comment 2 Eric Appleman 2010-02-24 14:26:12 UTC
Created attachment 154591 [details]
Debug of late and discarded subtitles
Comment 3 Sebastian Dröge (slomo) 2010-02-25 06:28:08 UTC
Seems to be caused by something sending wrong filler segments.
Comment 4 Sebastian Dröge (slomo) 2010-02-25 10:15:09 UTC
To give some more details, the problem here seems to be, that something sends a filler/update newsegment event where the start position is bigger than the subtitle buffers timestamp. This causes gst_segment_to_running_time() to return -1, which then causes the subtitle buffer to not be displayed.

Fixing that might already fix this bug... another source of problem here could be, that subtitles are not rendered if they are pushed not synchronized to the video by downstream. They're then simply dropped if they're after the current video frame. The video frames never wait for subtitle buffers, only the subtitle buffers wait for video frames.
Comment 5 Sebastian Dröge (slomo) 2010-02-26 17:10:23 UTC
Created attachment 154771 [details] [review]
assrender: Drop text buffers before the segment start and clip running time end to the segment stop
Comment 6 Sebastian Dröge (slomo) 2010-02-26 17:13:25 UTC
Created attachment 154774 [details] [review]
matroskademux: Advance sparse streams only as much as required to keep the gap smaller than 500ms

Changing it to the newest timestamp that was ever pushed will
increase the segment start in 500ms jumps, which could be just
after the next sparse stream buffer. E.g.

Video at 1.0s, sparse stream at 0.5s would jump the
sparse stream to 1.0s. Now a new sparse stream buffer could
appear that has a timestamp of 0.9s and this would be
dropped for no good reason because of bad luck.
Comment 7 Sebastian Dröge (slomo) 2010-02-26 17:14:51 UTC
These two patches fix it in this case but fixing #611194 and adjusting matroskademux properly would be better. The matroskademux patch will cause *many* newsegment updates to be sent.
Comment 8 Eric Appleman 2010-02-26 20:20:35 UTC
I think bug 610904 may also benefit from these fixes.

I'll test tonight.
Comment 9 Eric Appleman 2010-02-27 04:08:42 UTC
The good and bad patches together seem to make the situation a bit worse. While the subs no longer disappear, the lag has increased to the point where they bleed into the next scene. Moving and positioned subtitles can no longer keep up with their targets.

Unpatched good and patched bad prevents the loss of subs, but they may show up late. No continuous lag like the above.

Patched good and unpatched bad has accurate timing but loses subs quite easily through seeking.
Comment 10 Sebastian Dröge (slomo) 2010-02-27 07:42:25 UTC
Ok, that's bad :) Thanks for testing though...

Can you create another debug log with both patches and tell me which subtitles are wrong and where things start to get wrong?
Comment 11 Sebastian Dröge (slomo) 2010-02-27 10:24:36 UTC
Created attachment 154816 [details] [review]
assrender: Drop text buffers before the segment start and clip running time end to the segment stop
Comment 12 Sebastian Dröge (slomo) 2010-02-27 10:24:44 UTC
Created attachment 154817 [details] [review]
assrender: Consume text buffers if they're to be displayed in the next 500ms

Instead of only consuming them if they're to be display now. Consuming
them then might make following buffers with the same timestamp to
be too late.
Comment 13 Sebastian Dröge (slomo) 2010-02-27 10:44:07 UTC
commit 852be755dde44bbb6a9d276a16915d521ee8e2df
Author: Sebastian Dröge <sebastian.droege@collabora.co.uk>
Date:   Fri Feb 26 18:06:24 2010 +0100

    assrender: Drop text buffers before the segment start and clip running time 

commit 659776f1b16715b413b672ab03381d0ddf786790
Author: Sebastian Dröge <sebastian.droege@collabora.co.uk>
Date:   Sat Feb 27 11:23:14 2010 +0100

    assrender: Consume text buffers if they're to be displayed in the next 500ms
    
    Instead of only consuming them if they're to be display now. Consuming
    them then might make following buffers with the same timestamp to
    be too late.
Comment 14 Sebastian Dröge (slomo) 2010-02-27 10:47:27 UTC
The assrender patches are now pushed as they definitely improve the situation without making anything worse. Could you check again with the latest patches and unpatched or patched gst-plugins-good?
Comment 15 Sebastian Dröge (slomo) 2010-02-27 11:20:27 UTC
Comment on attachment 154774 [details] [review]
matroskademux: Advance sparse streams only as much as required to keep the gap smaller than 500ms

commit ad71d43f5221fedd4a8b2733a5e572ced93b3016
Author: Sebastian Dröge <sebastian.droege@collabora.co.uk>
Date:   Fri Feb 26 18:10:32 2010 +0100

    matroskademux: Advance sparse streams only as much as required to keep the g
    
    Changing it to the newest timestamp that was ever pushed will
    increase the segment start in 500ms jumps, which could be just
    after the next sparse stream buffer. E.g.
    
    Video at 1.0s, sparse stream at 0.5s would jump the
    sparse stream to 1.0s. Now a new sparse stream buffer could
    appear that has a timestamp of 0.9s and this would be
    dropped for no good reason because of bad luck.
Comment 16 Eric Appleman 2010-02-27 22:21:48 UTC
Created attachment 154865 [details]
Debug log of newer good and bad patches combined.

Subtitle timing is perfect until after seeking.
Comment 17 Sebastian Dröge (slomo) 2010-03-01 05:17:17 UTC
So these patches fix everything unless you seek in the movie, in which case the subtitles have wrong timing?
Comment 18 Eric Appleman 2010-03-01 05:29:34 UTC
With respect to the patches, I will cautiously say "yes" for the time being.

Seeking definitely disrupts timing though.
Comment 19 Eric Appleman 2010-03-01 07:22:38 UTC
I'd like to amend my statement slightly. Seeking doesn't always disrupt timing. If the script is simple, there will be no observable mistiming when seeking (this doesn't mean there is none). If the script has complex elements--that is, it has moving text or crazy effects--lag will be introduced through seeking to those scenes. Seeking away from them can correct the lag.
Comment 20 Sebastian Dröge (slomo) 2010-03-01 16:14:11 UTC
That's a bit scary as it means that something down the path creates wrong segment or timestamp information.

Could you please create a log again where you do a single seek and where subtitles are then not synchronized to a/v anymore?
Comment 21 Eric Appleman 2010-03-02 02:22:33 UTC
Created attachment 154998 [details]
Debug of single-seek distortion of subtitle timing
Comment 22 Tobias Mueller 2010-10-16 14:50:28 UTC
Reopening as the requested information has been provided.
Comment 23 Tim-Philipp Müller 2013-01-22 00:40:43 UTC
Does this still happen with git master as of now after

 commit 8d837cbac41b121760207e784ba7b54f1ef84f32
 Author: Arnaud Vrac <avrac@freebox.fr>
 Date:   Thu Jan 17 19:02:48 2013 +0100

    assrender: reenable unit tests and up rank to primary again

? 

I don't have the sample file to test myself unfortunately.
Comment 24 Alexandre Franke 2015-07-27 22:46:01 UTC
Closing this bug report as no further information has been provided. Please feel free to reopen this bug report if you can provide the information that was asked for in a previous comment.
Thanks!