GNOME Bugzilla – Bug 665080
API: subtitle overlays for raw and non-raw video buffers
Last modified: 2011-12-05 15:38:26 UTC
I would like to propose a simple abstraction for representing one or more subtitle overlay rectangles that should be blended on top of a video. These can then be blended on top of raw video using the new gst_video_overlay_composition_blend() convenience API, or they can be attached to buffers that represent vaapi/vdpau/whatever surfaces for later rendering in the video sink using the gst_buffer_set_qdata() API. This follows the design document/draft at [0]. Primary goals: - keep it simple and non-intrusive - add support for subtitles when using vaapi/vdpau decoders - provide utility API for overlay elements so they don't have to re-implement the same blending code everywhere, and then usually only for a limited number of video formats. - make sure we can easily add more features later The proposed new API can be found here: http://cgit.freedesktop.org/~tpm/gst-plugins-base/tree/gst-libs/gst/video/video-overlay-composition.h?h=video-overlay-composition-textoverlay It adds two new mini-object classes: - GstVideoOverlayComposition: contains one or more rectangles, may be attached to buffers or blended directly onto raw video buffers - GstVideoOverlayRectangle: represents an overlay rectangle, including ARGB pixel data, overlay position/size/scaling etc. In the most basic case, an overlay element would simply modify its code to do something along the lines of: comp = get_current_composition (overlay); if (overlay->input_is_raw_video) { composition_blend (comp, video_buffer); } else { video_buffer_set_overlay_composition (video_buffer, comp); } The composition is metadata as far as GstBuffer is concerned, and the API itself follows the same logic, including writability-based-on-refcount. Even though quite simple, we don't expose the structures directly and require access via the function API. This is to enforce/ensure writability and proper locking (multiple threads may be accessing these at the same time), and to allow for easy extension in future. There's also some "magic" going on under the hood (like caching of scaled rectangles). Sequence numbers on both objects will allow for efficient caching/update mechanisms. Where compositions are attached to buffers they will be attached to all buffers, not only when they change. This ensures proper state even when buffers get dropped for some reason (like QoS), and is also more in line with how we prefer to do it with e.g. caps. The sink/mixer can easily find out whether anything changed using the sequence numbers. Future features may include attaching the original text / pango markup in case the sink (or application) can render that directly; more formats for the overlay; letting the sinks do the overlaying even for raw video (requires a new query); support for pre-multiplied alpha; etc. This will eventually also allow us to have e.g. text rendered at the resolution of the output surface/display even if the actual video resolution is quite small and the video gets upscaled for display (currently we'll just blend the text on top of the video in low resolution and then the text gets upscaled as well, which results in ugly pixelated subtitles). Of course there are many alternative ways how one could solve all this, but the goal was to pick something that's simple, not very intrusive, and doesn't require major modifications in e.g. playbin2. [0] http://cgit.freedesktop.org/gstreamer/gst-plugins-base/tree/docs/design/draft-subtitle-overlays.txt
Looks good to me
commit a741c0cf11e1747cb58023c2ad3f979fc5b23b79 Author: Thibault Saunier <thibault.saunier@collabora.com> Date: Fri Nov 18 11:04:47 2011 -0300 textoverlay: Sync the caps with the new supported formats Thanks to the use of the new video composition library, we gain support to more colospaces and formats, let's state it. commit a018c187a0b6ad1010d8b9142e4606b069f0c55c Author: Thibault Saunier <thibault.saunier@collabora.com> Date: Wed Nov 16 17:54:43 2011 -0300 textoverlay: Make use of the new video blending utility commit 00d6ffd9f56e6ce636f7d61bc1ab2e9ff1227889 Author: Tim-Philipp Müller <tim.muller@collabora.co.uk> Date: Fri Nov 25 16:46:09 2011 +0000 tests: add basic unit test for video overlay composition and rectangles commit 6630236af4e7ab3dfbce2265c7ad7ff9b59c302c Author: Tim-Philipp Müller <tim.muller@collabora.co.uk> Date: Sat Nov 12 14:59:35 2011 +0000 video: add video overlay composition API for subtitles Basic API to attach overlay rectangles to buffers, or blend them directly onto raw video buffers. To be used primarily for things like subtitles or logo overlays, not meant to replace videomixer. Allows us to associate subtitle overlays with non-raw video surface buffers, so that subtitles are not lost and can instead be rendered later when those surfaces are displayed or converted, whilst re-using all the existing overlay plugins and not having to teach them about our special video surfaces. Could also have been made part of the surface buffer abstraction of course, but a secondary goal was to consolidate the blending code for raw video into libgstvideo, and this kind of API allows us to do both in a way that's minimally invasive to existing elements, and at the same time is fairly intuitive. More features and extensions like the ability to pass the source data or text/markup directly will be added later. https://bugzilla.gnome.org/show_bug.cgi?id=665080 API: gst_video_buffer_get_overlay_composition() API: gst_video_buffer_set_overlay_composition() API: gst_video_overlay_composition_new() API: gst_video_overlay_composition_add_rectangle() API: gst_video_overlay_composition_n_rectangles() API: gst_video_overlay_composition_get_rectangle() API: gst_video_overlay_composition_make_writable() API: gst_video_overlay_composition_copy() API: gst_video_overlay_composition_ref() API: gst_video_overlay_composition_unref() API: gst_video_overlay_composition_blend() API: gst_video_overlay_rectangle_new_argb() API: gst_video_overlay_rectangle_get_pixels_argb() API: gst_video_overlay_rectangle_get_pixels_unscaled_argb() API: gst_video_overlay_rectangle_get_render_rectangle() API: gst_video_overlay_rectangle_set_render_rectangle() API: gst_video_overlay_rectangle_copy() API: gst_video_overlay_rectangle_ref() API: gst_video_overlay_rectangle_unref() commit a7a3f969b306c1747972f5e662fd4b186cd6ecaa Author: Tim-Philipp Müller <tim.muller@collabora.co.uk> Date: Wed Nov 23 00:31:18 2011 +0000 video: hide private video-blend.[ch] from gobject-introspection And remove unused fields from helper structure. commit b0ff1d22e91142ec7335aec9e6dceefb34ae2a68 Author: Tim-Philipp Müller <tim.muller@collabora.co.uk> Date: Tue Nov 15 18:00:00 2011 +0000 video: add fallbacks for compilation without orc commit 80054a3b1e47a4257044b0a415743c717c73fbe8 Author: Thibault Saunier <thibault.saunier@collabora.com> Date: Mon Oct 17 17:25:11 2011 +0200 video: add some internal helper functions for image blending This could be improved if we decide we don't need it to be this generic/flexible.