GNOME Bugzilla – Bug 625722
[geometrictransform] Some new effect elements for cheese
Last modified: 2010-08-03 07:45:11 UTC
Hi, I'd like to propose for inclusion a new set of video filters meant for use with camera applications like Cheese and Empathy. It's an almost straightforward port of gleffects (from gst-plugins-gl) distortion (and maybe some of the lut mapping ones, e.g. heat and sepia, soon) filters to GstVideoFilter. The rationale behind this port is that gst-plugins-gl support in Cheese is still far away to be done, even farther now that we're moving to a Clutter based UI because sharing texture data between different threads is still tricky. This will eventually become easier when GPU data will be a first class citizen in the gstreamer pipeline (when Benjamin's plan of taking over the world will succeed). Meanwhile, this filter set works around the issue porting some of the effects from gst-gl to work on the cpu so that they are immediately (well, if and when you will accept them :-P) usable from Cheese. It also enables a wider range of setups to use the filters (no need for modern graphic hardware). Implementation wise, all filters share almost the same code, they fill a distortion map in set_caps with a pair of transformed coordinates for each source pixel and then fill the dest buffer with source pixels sampled at coords from the distortion map. Given that the transformed pixel could fall at fractional coordinates all the filters but "cheesemirror" (which doesn't need it because it gives exact results with integer math) provide two interpolation methods: - nearest-neighbour: the dest pixel is taken to be the nearest one in src buffer to the fractional transformed coords - bilinear: the dest pixel is calculated from a weighted average of the 4 pixels in the neighbourhood of the fractional transformed coords Transformed coords are stored in fixed point into 32bit integers with 8 bits for the fractional part. This turned out to be faster than floating point for bilinear interpolation. Bilinear interpolation is still slow enough to make the filters drop frames in my netbook (with 640x480@30fps webcam video) so nearest-neighbor is enabled by default. It could probably use some optimization love, it seems the kind of thing orc could improve a lot (if anyone wants to step in any help is welcome). Plugin's name and cheese namespace seemed the most obvious ones, if you have better ideas please suggest.
Created attachment 166883 [details] [review] cheeseeffects: adds new plugin 'cheeseeffects' Rewrite of gstgl distortion effects as GstVideoFilter subclasses. Currently only contains a single element: cheesesqueeze.
Created attachment 166884 [details] [review] cheeseeffects: adds stretch element
Created attachment 166885 [details] [review] cheeseeffects: adds fake fisheye element
Created attachment 166886 [details] [review] cheeseeffects: adds bulge element
Created attachment 166887 [details] [review] cheeseeffects: adds square element
Created attachment 166888 [details] [review] cheeseeffects: adds twirl element
Created attachment 166889 [details] [review] cheeseeffects: adds tunnel element
Created attachment 166890 [details] [review] cheeseeffects: adds mirror element
I think for many of these effects we have something similar in the effectv, gaudieffects and videofilter plugins already
> I think for many of these effects we have something similar in the effectv, > gaudieffects and videofilter plugins already Are you sure? gaudieffects seem to be color transform filters, videofilter don't know, if you mean videoflip it doesn't do the mirror thing, effectv has a distortion filter (warptv) but it's time dependent, these ones are static, and IMHO they fit a lot better the Cheese use case since you don't want the effect to change from the time you press "Take a photo" to when the countdown ends.
You're probably right but all the effect names sound so familiar :) You could also take a look at the geometrictransform element from gst-plugins-bad btw, it has a lot of effects that might be similar too. And it has a nice base class for geometric transformations, which you could probably use too (i.e. it does the original coordinate -> new coordinate mapping).
(In reply to comment #11) > You're probably right but all the effect names sound so familiar :) > > You could also take a look at the geometrictransform element from > gst-plugins-bad btw, it has a lot of effects that might be similar too. And it > has a nice base class for geometric transformations, which you could probably > use too (i.e. it does the original coordinate -> new coordinate mapping). Uh, that's one I didn't know about :) It has a nice base class you're right and has at least two filters, twirl and pinch, which overlap with cheesetwirl and cheesesqueeze. It doesn't do the interpolating thing though, it just maps the new pixel to nearest neighbour pixel (well it casts doubles to (int), not sure if that actually rounds to nearest int) in the source at transformed coords (often resulting in jagged output). It is not straightly portable to geometrictransform because it currently works with 32 bit pixels while that supports also RGB, GRAY8 and GRAY16 caps. Also I'd like to add at least two more filters, lookup table based color transforms, "heat" and "sepia". It would be great to have them in Cheese but they don't surely fit the geometrictransform plugin. I'm open to any suggestion, I'd really like to have these filters in Cheese soon.
(In reply to comment #12) > (In reply to comment #11) > > You're probably right but all the effect names sound so familiar :) > > > > You could also take a look at the geometrictransform element from > > gst-plugins-bad btw, it has a lot of effects that might be similar too. And it > > has a nice base class for geometric transformations, which you could probably > > use too (i.e. it does the original coordinate -> new coordinate mapping). > > Uh, that's one I didn't know about :) > It has a nice base class you're right and has at least two filters, twirl and > pinch, which overlap with cheesetwirl and cheesesqueeze. > > It doesn't do the interpolating thing though, it just maps the new pixel to > nearest neighbour pixel (well it casts doubles to (int), not sure if that > actually rounds to nearest int) in the source at transformed coords (often > resulting in jagged output). > It is not straightly portable to geometrictransform because it currently works > with 32 bit pixels while that supports also RGB, GRAY8 and GRAY16 caps. Well, the interpolation thing could be added to the geometrictransform baseclass I guess. And then you could port most of these effects to that and automagically get RGB, etc support :) What do you think? > Also I'd like to add at least two more filters, lookup table based color > transforms, "heat" and "sepia". It would be great to have them in Cheese but > they don't surely fit the geometrictransform plugin. Yes, that's something different then. How exactly do they work? Is it just a function from source color to destination color, e.g. mapping of green to blue? Or does it also take the position into account? Anyway, I think it might make sense to have a new plugin with a base class for this and then add the heat/sepia elements as subtypes there. I guess something like current cheese's "hulk" effect could be done with this too, IIRC it was just adjusting the hue a bit.
(In reply to comment #13) > > It is not straightly portable to geometrictransform because it currently works > > with 32 bit pixels while that supports also RGB, GRAY8 and GRAY16 caps. > > Well, the interpolation thing could be added to the geometrictransform > baseclass I guess. And then you could port most of these effects to that and > automagically get RGB, etc support :) > What do you think? Sorry for my bad english, I meant that my interpolation function is not directly portable as is because it doesn't take video format into account right now and just assumes 8888 pixels, so it may take some work to do it properly. Anyway, I'll drop interpolation for now and port the filters to geometric transform. Then I'll try to find some time to port the interpolation thing too. > > Also I'd like to add at least two more filters, lookup table based color > > transforms, "heat" and "sepia". It would be great to have them in Cheese but > > they don't surely fit the geometrictransform plugin. > > Yes, that's something different then. How exactly do they work? Is it just a > function from source color to destination color, e.g. mapping of green to blue? > Or does it also take the position into account? They just map source color to output color using a lookup table. Heat and sepia calculate current color luminance and sample the new color from a 256 pixels array. No position involved. > Anyway, I think it might make sense to have a new plugin with a base class for > this and then add the heat/sepia elements as subtypes there. I guess something > like current cheese's "hulk" effect could be done with this too, IIRC it was > just adjusting the hue a bit. Yep it just uses videobalance for that but it could very well be done this way too. Again, will see if time allows me to do this properly. Not sure there is much material for a base class, it wouldn't be that different from plain basetransform/videofilter.
(In reply to comment #14) > (In reply to comment #13) > > > It is not straightly portable to geometrictransform because it currently works > > > with 32 bit pixels while that supports also RGB, GRAY8 and GRAY16 caps. > > > > Well, the interpolation thing could be added to the geometrictransform > > baseclass I guess. And then you could port most of these effects to that and > > automagically get RGB, etc support :) > > What do you think? > > Sorry for my bad english, I meant that my interpolation function is not > directly portable as is because it doesn't take video format into account right > now and just assumes 8888 pixels, so it may take some work to do it properly. > Anyway, I'll drop interpolation for now and port the filters to geometric > transform. Then I'll try to find some time to port the interpolation thing too. Doing the interpolation is essentially the same for other pixel formats... you simply do it by component. > > > Also I'd like to add at least two more filters, lookup table based color > > > transforms, "heat" and "sepia". It would be great to have them in Cheese but > > > they don't surely fit the geometrictransform plugin. > > > > Yes, that's something different then. How exactly do they work? Is it just a > > function from source color to destination color, e.g. mapping of green to blue? > > Or does it also take the position into account? > > They just map source color to output color using a lookup table. Heat and sepia > calculate current color luminance and sample the new color from a 256 pixels > array. > No position involved. How do you build the lookup table btw? It would be *really* large in the general case. > > Anyway, I think it might make sense to have a new plugin with a base class for > > this and then add the heat/sepia elements as subtypes there. I guess something > > like current cheese's "hulk" effect could be done with this too, IIRC it was > > just adjusting the hue a bit. > > Yep it just uses videobalance for that but it could very well be done this way > too. Again, will see if time allows me to do this properly. > Not sure there is much material for a base class, it wouldn't be that different > from plain basetransform/videofilter. I don't know, maybe the base class could handle the lookup table and apply it to different color formats. And the subclasses only provide the different lookup tables for the different effects then. But if all these effects could be implemented with videobalance a bin around videobalance would definitely be preffered.
> How do you build the lookup table btw? It would be *really* large in the > general case. > Each lookup table is an array of 256 pixels, 768 bytes. I build them with the gimp. See below. > I don't know, maybe the base class could handle the lookup table and apply it > to different color formats. And the subclasses only provide the different > lookup tables for the different effects then. > > But if all these effects could be implemented with videobalance a bin around > videobalance would definitely be preffered. Nope the same result is not achievable with videobalance. It's not a hue/contrast/brightness change, each color is mapped to a new one with no simple relation with the previous. If you give me a couple of hours (maximum tomorrow morning) I'll post a simple patch with a single plugin + several presets so that it's clear what we're talking about, then we'll see if a baseclass it's really worth.
Created attachment 166941 [details] [review] colorlut: new plugin for lookup table color mapping Implements a color lookup table filter with 4 presets: - heat: fake heat camera effect - sepia: sepia toning - xray: invert + shade to blue - xpro: cross process
(In reply to comment #17) > Created an attachment (id=166941) [details] [review] > colorlut: new plugin for lookup table color mapping Note that I wrote it quickly as a proof of concept so I could have missed something, but the idea is more or less implemented. See the comments in the code for a complete description about how the lookup tables were generated and how the filter works.
Ah, so it's just a luma->(r,g,b) function that is implemented by the LUT. Right, the plugin as you've attached it here looks good. Clean it up a bit and file a new bug for it and then we can get it included :) Btw, it would be nice if you would add support for xRGB/ARGB and some YUV formats (at least AYUV) too Let's track the geometrictransform changes in this bug here
Created attachment 166952 [details] [review] geometrictransform: new filter "stretch" Ports gleffects "stretch" filter to geometrictransform. Shrinks the image around the center and gradually return to normal zoom creating funny caricatures.
Created attachment 166954 [details] [review] geometrictransform: new filter "bulge" Ports gleffects "bulge" filter to geometrictransform. Adds a protuberance around the center point.
Created attachment 166955 [details] [review] geometrictransform: new filter "tunnel" Ports gleffects "tunnel" filter to geometrictransform. Do nothing in a circle around the center and zoom outside.
Created attachment 166956 [details] [review] geometrictransform: new filter "square" Ports gleffects "square" filter to geometrictransform. Maps a region around the center into a zoomed square and smoothly get back to normal zoom. With faces it makes a funny "cube-face" effect.
Created attachment 166957 [details] [review] geometrictransform: new filter "mirror" Ports gleffects "mirror" filter to geometrictransform. Simple yet effective mirror effect, splits the image into halves and reflect the first into the second.
Created attachment 166958 [details] [review] geometrictransform: new filter "fisheye" Ports gleffects "fisheye" filter to geometrictransform. Fake fisheye lens filter. Somewhat empiric implementation because I didn't find any good algorithm that does it with nice results.
I really like having these effects, I'm just not sure the name cheeseeffects is good, because really they're not cheese specific. Something more descriptive may be better.
(In reply to comment #26) > I really like having these effects, I'm just not sure the name cheeseeffects is > good, because really they're not cheese specific. Something more descriptive > may be better. They're not called cheeseeffects anymore but will be new elements in the geometrictransform element. Filippo, I'll review your patches later but from a short look they seem to be good :)
commit 0c0adffce756686ddfcf99b0208b295c8cd994f4 Author: Filippo Argiolas <filippo.argiolas@gmail.com> Date: Mon Aug 2 11:30:50 2010 +0200 geometrictransform: new filter "fisheye" Ports gleffects "fisheye" filter to geometrictransform. Fake fisheye lens filter. Somewhat empiric implementation because I didn't find any good algorithm that does it with nice results. https://bugzilla.gnome.org/show_bug.cgi?id=625722 commit 9ee12f94deee3c9bf7abf4e1b3afb1434c6147e2 Author: Filippo Argiolas <filippo.argiolas@gmail.com> Date: Mon Aug 2 11:12:42 2010 +0200 geometrictransform: new filter "mirror" Ports gleffects "mirror" filter to geometrictransform. Simple yet effective mirror effect, splits the image into halves and reflect the first into the second. https://bugzilla.gnome.org/show_bug.cgi?id=625722 commit 5c4915d917ab969ed2417b5f64dbea907b8602ff Author: Filippo Argiolas <filippo.argiolas@gmail.com> Date: Mon Aug 2 11:01:31 2010 +0200 geometrictransform: new filter "square" Ports gleffects "square" filter to geometrictransform. Maps a region around the center into a zoomed square and smoothly get back to normal zoom. With faces it makes a funny "cube-face" effect. https://bugzilla.gnome.org/show_bug.cgi?id=625722 commit a7fb7ae06c53a19224cf83dde75812969588d3f5 Author: Filippo Argiolas <filippo.argiolas@gmail.com> Date: Mon Aug 2 10:46:44 2010 +0200 geometrictransform: new filter "tunnel" Ports gleffects "tunnel" filter to geometrictransform. Do nothing in a circle around the center and zoom outside. https://bugzilla.gnome.org/show_bug.cgi?id=625722 commit 3ba3310b013c8c6a4301d0ec630501dd530b5eb5 Author: Filippo Argiolas <filippo.argiolas@gmail.com> Date: Mon Aug 2 09:39:51 2010 +0200 geometrictransform: new filter "bulge" Ports gleffects "bulge" filter to geometrictransform. Adds a protuberance around the center point. https://bugzilla.gnome.org/show_bug.cgi?id=625722 commit 2edd185a9a3f03280d4184ddd5a81c20f52845f7 Author: Filippo Argiolas <filippo.argiolas@gmail.com> Date: Mon Aug 2 09:17:03 2010 +0200 geometrictransform: new filter "stretch" Ports gleffects "stretch" filter to geometrictransform. Shrinks the image around the center and gradually return to normal zoom creating funny caricatures. https://bugzilla.gnome.org/show_bug.cgi?id=625722
It'd be great if the center of the effects and the "strength" could be configurable later :)
Hey Sebastian, thanks, really, for the super quick review and for committing my filters :) Center of the circle based effects (bulge, tunnel and stretch) is already configurable, radius is currently hard coded because I don't like the current way it is handled (absolute radius instead of a video size independent parameter). I talked with Thiago about this and he almost agrees but couldn't find a good way of defining a relative radius, will try to come up with a patch for that soon, and make the radius configurable then. Then I'll add a strength/intensity parameter where it makes sense (at least bulge and stretch, maybe fisheye if I find a good way of defining its "strength").