GNOME Bugzilla – Bug 678485
templatematch: element improvements
Last modified: 2014-07-28 19:10:51 UTC
Created attachment 216843 [details] [review] General tidy-up patch We have made some changes to the templatematch element so it works better for our application: # Set caps to BGR rather than RGB. OpenCV expects BGR whereas templatematch has until now used GST_VIDEO_CAPS_RGB. Previously templatematch had difficulty matching anything colorful. # Allow changing the template to match on-the-fly, even when the pipeline is PLAYING # Adjust the color of the match rectangle according to the match certainty # We pass the video through unchanged if no template has been specified rather than erroring out # General tidyups
Created attachment 216844 [details] [review] Pass video through when nothing to match against opencv templatematch: Pass video through when nothing to match against The early return was bypassing the call to gst_pad_push. With no filter->template (and thus no filter->cvTemplateImage) the rest of this function is essentially a no-op (except for the call to gst_pad_push). This (plus the previous commit) allows templatematch to be enabled/disabled without removing it entirely from the pipeline, by setting/unsetting the template property.
Created attachment 216845 [details] [review] Allow changing template property on the fly Previously changing the template property resulted in an exception thrown from cvMatchTemplate, because "dist_image" (the intermediate match-certainty-distribution) was the wrong size (because the template image size had changed). Locking has also been added to allow changing the properties (e.g. the pattern to match) while the pipeline is playing. * gst_element_post_message is moved outside of the lock, because it will call into arbitrary user code (otherwise, if that user code calls into gst_templatematch_set_property on this same thread it would deadlock). * gst_template_match_load_template: If we fail to load the new template we still unload the previous template, so this element becomes a no-op in the pipeline. The alternative would be to keep the previous template; I believe unloading the previous template is a better choice, because it is consistent with the state this element would be in if it fails to load the very first template at start-up.
Created attachment 216846 [details] [review] Set caps to BGR order templatematch operates on BGR data. In fact, OpenCV's IplImage always stores color image data in BGR order -- this isn't documented at all in the OpenCV source code, but there are hints around the web (see for example http://www.cs.iit.edu/~agam/cs512/lect-notes/opencv-intro/opencv-intro.html#SECTION00041000000000000000 and http://www.comp.leeds.ac.uk/vision/opencv/iplimage.html ). gst_templatematch_load_template loads the template (the image to find) from disk using OpenCV's cvLoadImage, so it is stored in an IplImage in BGR order. But in gst_templatematch_chain, no OpenCV conversion functions are used: the imageData pointer of the IplImage for the video frame (the image to search in) is just set to point to the raw buffer data. Without this fix, that raw data is in RGB order, so the call to cvMatchTemplate ends up comparing the template's Blue channel against the frame's Red channel, producing very poor results. The BGR order probably needs to be applied to the other opencv elements too, but we haven't had time to confirm that. This patch includes a test that a template will match against itself which will be run with `make check`
Created attachment 216847 [details] [review] Match rectangle grows redder with increased match certainty This is useful for debugging your matches as it indicates how certain the match was in addition to its position.
Created attachment 217239 [details] [review] Produce a warning message if loading template image fails
Never too late, right? :) Updated the patches for 1.0 API and implemented a better test than the one that used the script. Thanks for the patches and sorry it took so long. commit 06243b7eeee9ab09515e7367c189ef99bf5b652f Author: Thiago Santos <ts.santos@osg.sisa.samsung.com> Date: Mon Jul 28 15:45:09 2014 -0300 tests: templatematch: add test to check that we use the correct rgb format templatematch should use the same RGB format that opencv uses (BGR), make sure we keep it that way with this test. https://bugzilla.gnome.org/show_bug.cgi?id=678485 commit f5992a6cdd13c5460a9dd2bef17c7502a0816237 Author: David Rothlisberger <david.rothlisberger@youview.com> Date: Fri Jun 15 13:19:06 2012 +0100 opencv templatematch: Set caps to BGR order templatematch operates on BGR data. In fact, OpenCV's IplImage always stores color image data in BGR order -- this isn't documented at all in the OpenCV source code, but there are hints around the web (see for example http://www.cs.iit.edu/~agam/cs512/lect-notes/opencv-intro/opencv-intro.html#SECTION00041000000000000000 and http://www.comp.leeds.ac.uk/vision/opencv/iplimage.html ). gst_templatematch_load_template loads the template (the image to find) from disk using OpenCV's cvLoadImage, so it is stored in an IplImage in BGR order. But in gst_templatematch_chain, no OpenCV conversion functions are used: the imageData pointer of the IplImage for the video frame (the image to search in) is just set to point to the raw buffer data. Without this fix, that raw data is in RGB order, so the call to cvMatchTemplate ends up comparing the template's Blue channel against the frame's Red channel, producing very poor results. commit aa5e25820b0a721580b5d4054477345f66352bb8 Author: Thiago Santos <ts.santos@osg.sisa.samsung.com> Date: Mon Jul 28 13:10:35 2014 -0300 templatematch: mark pads as proxy caps Allows negotiation to happen properly commit 1b5483eef98928d000acea579d4e68a5374557f1 Author: William Manley <william.manley@youview.com> Date: Mon Jun 25 20:37:01 2012 +0100 templatematch: Produce a warning message if we can't load a template image commit 88cb33a1dce12ab6ae847dd417effc00d6954688 Author: William Manley <william.manley@youview.com> Date: Wed Jun 20 15:22:52 2012 +0100 templatematch: Match rectangle grows redder with increased match certainty This is useful for debugging your matches as it indicates how certain the match was in addition to its position. commit b608767288b738889b4afaf6141fa92c72bb5645 Author: William Manley <william.manley@youview.com> Date: Wed Jun 20 15:05:40 2012 +0100 templatematch: Allow changing template property on the fly Previously changing the template property resulted in an exception thrown from cvMatchTemplate, because "dist_image" (the intermediate match-certainty-distribution) was the wrong size (because the template image size had changed). Locking has also been added to allow changing the properties (e.g. the pattern to match) while the pipeline is playing. * gst_element_post_message is moved outside of the lock, because it will call into arbitrary user code (otherwise, if that user code calls into gst_templatematch_set_property on this same thread it would deadlock). * gst_template_match_load_template: If we fail to load the new template we still unload the previous template, so this element becomes a no-op in the pipeline. The alternative would be to keep the previous template; I believe unloading the previous template is a better choice, because it is consistent with the state this element would be in if it fails to load the very first template at start-up. Thanks to Will Manley for the bulk of this work; any errors are probably mine. commit c77808055de487260363adf4969a4b62cc9048a8 Author: David Rothlisberger <david.rothlisberger@youview.com> Date: Fri Jun 1 16:07:34 2012 +0100 templatematch: Pass video through when nothing to match against The early return was bypassing the call to gst_pad_push. With no filter->template (and thus no filter->cvTemplateImage) the rest of this function is essentially a no-op (except for the call to gst_pad_push). This (plus the previous commit) allows templatematch to be enabled/disabled without removing it entirely from the pipeline, by setting/unsetting the template property. commit 056d652b922c6bf926bbfdcd3e9edbf31944e605 Author: William Manley <william.manley@youview.com> Date: Wed Jun 20 15:05:06 2012 +0100 templatematch: Remove no-op call to gst_templatematch_load_template We have just set filter->template to NULL, so gst_templatematch_load_template did nothing.
Pushed the 2 bugfixes from the patches above to 1.4 branch: commit 1f30334b5d16a62bec2af526bdb365a986bfd774 Author: David Rothlisberger <david.rothlisberger@youview.com> Date: Fri Jun 15 13:19:06 2012 +0100 opencv templatematch: Set caps to BGR order templatematch operates on BGR data. In fact, OpenCV's IplImage always stores color image data in BGR order -- this isn't documented at all in the OpenCV source code, but there are hints around the web (see for example http://www.cs.iit.edu/~agam/cs512/lect-notes/opencv-intro/opencv-intro.html#SECTION00041000000000000000 and http://www.comp.leeds.ac.uk/vision/opencv/iplimage.html ). gst_templatematch_load_template loads the template (the image to find) from disk using OpenCV's cvLoadImage, so it is stored in an IplImage in BGR order. But in gst_templatematch_chain, no OpenCV conversion functions are used: the imageData pointer of the IplImage for the video frame (the image to search in) is just set to point to the raw buffer data. Without this fix, that raw data is in RGB order, so the call to cvMatchTemplate ends up comparing the template's Blue channel against the frame's Red channel, producing very poor results. commit 9bdb536eec8d0685979a8d9509978cf709c4f01f Author: Thiago Santos <ts.santos@osg.sisa.samsung.com> Date: Mon Jul 28 13:10:35 2014 -0300 templatematch: mark pads as proxy caps Allows negotiation to happen properly