GNOME Bugzilla – Bug 678146
Closed Caption support
Last modified: 2018-11-03 13:12:13 UTC
Right now we don't suport any kind of Closed Caption (not to be confused with subtitles). https://en.wikipedia.org/wiki/Closed_captioning There are a few problems that need to be solved: 1) CC format support There are a few standards out there for CC, the main ones are: * CEA 608 and 708 * teletext teletext is supported by teletextdec (needs to be checked). An alternative would be the vbidec element (which wasn't ported to 0.10). the zvbi library *seems* to have support for CEA 608 and maybe 708, this would need to be investigated. Proper caps need to be determined also. CEA 608/708 require a new element 2) Extracting CC stream For teletext it is always contained as a separate stream in the container format (mostly mpeg-ts). This is where it gets really painful. For CEA 608/708 it is almost always contained in either the mpeg video stream (as user data) or in the h264 RBSP stream (as SEI data). So the question is: * where to parse it. The logical place would be in the mpegvideo/h264 parsers. * How to expose it. Do we create a new pad ? Do we emit them as messages ? There is also CEA 608/708 in quicktime/iso file formats, see bug #606643
teletextdec will need some loving to be properly auto-plugged, used, and default to subtitle page (888) instead of the 100 page.
For extract the CEA 608/708 from the h264 anciliary streams the idea would be to do that in h264parse, as there we already have the information to be able to extract it. This would then need to be either exposed as a separate stream via a pad, or as GstMeta on the h264 buffers. For subtitles, exposing as a separate stream has the big problem that we can't now beforehand if there ever will be such a subtitle stream or not. So we could add it far too late and decodebin would ignore it. In decodebin's current design. So I see the following options here: 1) Expose as a separate stream and have CC stream decoder elements, fix decodebin to accept streams after it already exposed streams. This would have to be done very carefully to not cause any problems with the no-more-pads logic in decodebin and detecting if something is "just" a new stream, or something is the start of a complete new stream group (think: HLS, chained oggs, etc). 2) Parse the CC subtitles into a GstMeta, put them on the h264 buffers in the parsers. Then inside playbin redirect these subtitle metas as a new subtitle stream from the video stream to the subtitle input selector. For this to be meaningful we would need a generic way of specifying if something is a subtitle meta and also identify different subtitle meta streams (to allow playbin to expose multiple of them...).
*** Bug 140220 has been marked as a duplicate of this bug. ***
I created a branch of get-plugins-bad and added elements mpeguserdatademux and ccfilter. I modified mpegvparse to add a boolean user-data to its src caps (much like system-stream etc). Autoplug finds mpeguserdatademux based on this new cap and plugs after mpegvparse. mpeguserdatademux has an always src pad for video data passthrough which ends up connecting to the decoder via multiqueue. It also has a sometimes src pad for MPEG user data (current caps: video/x-user-data). The new ccfilter element autoplugs based on caps video/x-user-data and has a sometimes src pad with caps text/vtt. Updated versions of uridecodebin connect to these pads with the assistance of WebKit courtesy of Brendan Long. I have been trying to complete this work on closed captions support and am having some difficulties getting the text cue timing correct. In my original test implementation I had been running in parallel with tsdemux and the element received PES headers where I extracted the 90KHz PTS clock information and used that for the CC text cue timing. In my refactored implementation the mpeguserdatademux element is plugged after mpegvparse and I no longer receive PES headers. I tried using GST_BUFFER_PTS(buf) and the resulting CC cue times appeared not to be linear with the time stamps in the logs and cues rendered at the wrong time. I have tried using the element clock which is valid after the NewClock event but that time requires an offset based on user interaction with WebKit (or not when running with gst-launch). Any comments, thoughts, or insights would be appreciated. I eventually hope to submit this branch (text_track_refactor) for your approval to begin to address this issue.
Steve, it would be easier to review/comment if you pointed us to where your branch is :) Put it on github for example
Agreed - the branch is currently in bitbucket and CableLabs will be open-sourcing the project soon... In the meantime, would you expect PTS times from the stream buffers and their respective log times to increment with near lock-step difference? e.g. If I just simply log the PTS from all the buffers received in my chain and the first log showed a timestamp of 0.00.05.080 and the PTS showed 0.00.00.780. Would you expect that after one minute of being in the playing state that the timestamp difference would still be close to 0.00.05.700? (+/- some nominal tolerance)
the user-data (whatever it is) should have the same GST_BUFFER_PTS and GST_BUFFER_DTS as the mpeg video buffer it was extracted from. You should not have to deal with PES (that is done by upstream elements such as a mpeg-ts demuxer).
For this to work properly we also need to improve decodebin (and probably playbin/uridecodebin too) to allow pads to be added at any later time.
(In reply to comment #4) > I created a branch of get-plugins-bad and added elements mpeguserdatademux and > ccfilter. I modified mpegvparse to add a boolean user-data to its src caps > (much like system-stream etc). Autoplug finds mpeguserdatademux based on this > new cap and plugs after mpegvparse. mpeguserdatademux has an always src pad > for video data passthrough which ends up connecting to the decoder via > multiqueue. It also has a sometimes src pad for MPEG user data (current caps: > video/x-user-data). The new ccfilter element autoplugs based on caps > video/x-user-data and has a sometimes src pad with caps text/vtt. Updated > versions of uridecodebin connect to these pads with the assistance of WebKit > courtesy of Brendan Long. After more thinking and discussions. This seems like a wrong approach. Just having the parser expose any user data without any identification/parsing might cause several problems. It's as if we had mpeg-ts demuxers only expose data/x-pes and losing any context downstream. For each parser that sees private data they should: * identify the type of private data * Is the user-data metadata (example: AVCHD original recording time) or actual data (like CC) ? Depending on the type some user-data should be sent as tag events, and others should have a new pad created. * Use identifiable caps (subtitle/x-cea-608 for example ?) * If the type is not identifyable, notify it with GST_FIXME() * Note: some user_data might also be needed by the parser itself also
bug #704881 describes an attempt to begin to address a portion of this...
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/72.