GNOME Bugzilla – Bug 663204
Video Sink should support color balance
Last modified: 2014-05-28 14:39:47 UTC
While digging Totem code I found that they where using videobalance element to handle saturation, hue, brightness and contrast. I think we could improve performance by implementing GstColorBalance interface using shaders. Some upstream clutter work to add such effects seems inline. Also, this is one of the current blocker for enabling HW accelerated buffers being rendered in Totem. Add lightness, brightness, and contrast effects https://bugzilla.gnome.org/show_bug.cgi?id=656156
I'm currently looking at adding that feature into cogl-gst : https://github.com/djdeath/cogl/commit/8f1091f26b6274722f044b09bc0fc7623d89b90b Should be available in ClutterGst 3.0.
Created attachment 269138 [details] [review] video-sink: implement colorbalance interface The following patch implement the colorbalance interface for the clutter-gst-2.0 video-sink. The current limitation on this implementation is relative to the clutter desaturate effect which only supports values from 0.0 (fully saturated, ie: original image) to 1.0 (fully desaturated).
Thanks a lot for looking into that for ClutterGst 2.0, but this implementation is unlikely to perform well. Here you're using 3 clutter effects derived from ClutterOffscreenEffect. That means that when you're using the 3 of them at the same time, you're copying the video texture 3 times before it finally ends on the stage. It's also potentially consuming a fair amount of memory. I would recommend that you look at the stuff I'm trying to get into Cogl 1.18 : https://github.com/djdeath/cogl/commit/5904aac57b4e8733cefd84a7f78e1ec0d2b82c31 It pretty similar to what the videobalance gst element does, except it's done on the gpu : we compute 3 lookup tables for y, u and v, which are then uploaded on the gpu memory, what's left to do by the gpu, is the actual lookup for each pixel.
(In reply to comment #3) > Thanks a lot for looking into that for ClutterGst 2.0, but this implementation > is unlikely to perform well. > Here you're using 3 clutter effects derived from ClutterOffscreenEffect. That > means that when you're using the 3 of them at the same time, you're copying the > video texture 3 times before it finally ends on the stage. It's also > potentially consuming a fair amount of memory. Thanks for the review. I agree that it is not the most efficient way to implement it and that it can be done within a single pass. Do you know some kind of tool that can be used to mesure the GPU performance and mesure the impact of this code ? > > I would recommend that you look at the stuff I'm trying to get into Cogl 1.18 : > > https://github.com/djdeath/cogl/commit/5904aac57b4e8733cefd84a7f78e1ec0d2b82c31 > > It pretty similar to what the videobalance gst element does, except it's done > on the gpu : we compute 3 lookup tables for y, u and v, which are then uploaded > on the gpu memory, what's left to do by the gpu, is the actual lookup for each > pixel. Thanks, i will try to implement a shader effect using this method.
Created attachment 269606 [details] [review] video-sink: implement colorbalance interface Patch updated. It now uses a single shader to perform the colorbalance operation.
Created attachment 269607 [details] [review] video-sink: implement colorbalance interface Right patch attached, sorry for the noise :(
Created attachment 269985 [details] [review] video-sink: implement colorbalance interface Fixes comment about Y/U/V ranges for the ycbcr2rgb shader function and fixes CLUTTER_GST_*_DEFAULT declarations (BRIGHTNESS and CONTRAST was inverted).
Here is some benchmark. intel_gpu_top is used to mesure how busy the gpu is. The following pipeline has been used: LIBGL_SHOW_FPS=1 gst-launch-1.0 videotestsrc ! video/x-raw,format=RGBA,width=1920,height=1080,framerate=60/1 ! cluttersink CPU: Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz Graphics: Intell HD4000. cluttersink window (on screen) is of size 2300x1276. * 1920x1080p @60fps, colorbalance disabled average FPS = 60fps average render busy: 29% * 1920x1080p @60fps, colorbalance enabled average FPS = 60fps average render busy: 45% * 3800x1080 @60fps, colorbalance disabled average FPS = 60fps average render busy: 29% * 3800x1080 @60fps, colorbalance enabled average FPS = 60fps average render busy: not stable ~83% jumps down to 50% for short period of times.
In the last patch you attached, there is still an additional step of going through a ClutterOffscreenEffect. This basically means the application keeps a buffer of the size of the video and goes through video texture -> offscreen buffer -> application buffer (+ composition of the window manager) The patch I've mentioned on Cogl doesn't use an intermediate buffer to do this. It obviously requires more invasive changes to ClutterGst which is why I'm dragging my feet on this. This is basically rewriting something already done somewhere else. Though, since the CoglGst transition didn't happen for 3.12, I'll try to find some time to do it.
I reimported the video sink into ClutterGst. It uses the color correction implementation I pointed to. It's on master.
Great ! thanks.
Thanks :)