After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 678485 - templatematch: element improvements
templatematch: element improvements
Status: RESOLVED FIXED
Product: GStreamer
Classification: Platform
Component: gst-plugins-bad
1.x
Other All
: Normal normal
: 1.4.1
Assigned To: GStreamer Maintainers
GStreamer Maintainers
Depends on:
Blocks:
 
 
Reported: 2012-06-20 15:36 UTC by Will Manley
Modified: 2014-07-28 19:10 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
General tidy-up patch (845 bytes, patch)
2012-06-20 15:36 UTC, Will Manley
committed Details | Review
Pass video through when nothing to match against (1.68 KB, patch)
2012-06-20 15:37 UTC, Will Manley
committed Details | Review
Allow changing template property on the fly (6.15 KB, patch)
2012-06-20 15:38 UTC, Will Manley
committed Details | Review
Set caps to BGR order (4.86 KB, patch)
2012-06-20 15:39 UTC, Will Manley
committed Details | Review
Match rectangle grows redder with increased match certainty (1.64 KB, patch)
2012-06-20 15:40 UTC, Will Manley
committed Details | Review
Produce a warning message if loading template image fails (2.80 KB, patch)
2012-06-25 19:56 UTC, Will Manley
committed Details | Review

Description Will Manley 2012-06-20 15:36:23 UTC
Created attachment 216843 [details] [review]
General tidy-up patch

We have made some changes to the templatematch element so it works better for our application:
# Set caps to BGR rather than RGB.  OpenCV expects BGR whereas templatematch has until now used GST_VIDEO_CAPS_RGB.  Previously templatematch had difficulty matching anything colorful.
# Allow changing the template to match on-the-fly, even when the pipeline is PLAYING
# Adjust the color of the match rectangle according to the match certainty
# We pass the video through unchanged if no template has been specified rather than erroring out
# General tidyups
Comment 1 Will Manley 2012-06-20 15:37:32 UTC
Created attachment 216844 [details] [review]
Pass video through when nothing to match against

opencv templatematch: Pass video through when nothing to match against

The early return was bypassing the call to gst_pad_push. With no filter->template (and thus no filter->cvTemplateImage) the rest of this function is essentially a no-op (except for the call to gst_pad_push).

This (plus the previous commit) allows templatematch to be enabled/disabled without removing it entirely from the pipeline, by setting/unsetting the template property.
Comment 2 Will Manley 2012-06-20 15:38:53 UTC
Created attachment 216845 [details] [review]
Allow changing template property on the fly

Previously changing the template property resulted in an exception
thrown from cvMatchTemplate, because "dist_image" (the intermediate
match-certainty-distribution) was the wrong size (because the
template image size had changed).

Locking has also been added to allow changing the properties (e.g. the
pattern to match) while the pipeline is playing.

 * gst_element_post_message is moved outside of the lock, because it will
   call into arbitrary user code (otherwise, if that user code calls into
   gst_templatematch_set_property on this same thread it would deadlock).

 * gst_template_match_load_template: If we fail to load the new template
   we still unload the previous template, so this element becomes a no-op
   in the pipeline. The alternative would be to keep the previous template;
   I believe unloading the previous template is a better choice, because it
   is consistent with the state this element would be in if it fails to
   load the very first template at start-up.
Comment 3 Will Manley 2012-06-20 15:39:48 UTC
Created attachment 216846 [details] [review]
Set caps to BGR order

templatematch operates on BGR data. In fact, OpenCV's IplImage always
stores color image data in BGR order -- this isn't documented at all in
the OpenCV source code, but there are hints around the web (see for
example
http://www.cs.iit.edu/~agam/cs512/lect-notes/opencv-intro/opencv-intro.html#SECTION00041000000000000000
and http://www.comp.leeds.ac.uk/vision/opencv/iplimage.html ).

gst_templatematch_load_template loads the template (the image to find)
from disk using OpenCV's cvLoadImage, so it is stored in an IplImage in
BGR order. But in gst_templatematch_chain, no OpenCV conversion
functions are used: the imageData pointer of the IplImage for the video
frame (the image to search in) is just set to point to the raw buffer
data. Without this fix, that raw data is in RGB order, so the call to
cvMatchTemplate ends up comparing the template's Blue channel against
the frame's Red channel, producing very poor results.

The BGR order probably needs to be applied to the other opencv elements
too, but we haven't had time to confirm that.

This patch includes a test that a template will match against itself
which will be run with `make check`
Comment 4 Will Manley 2012-06-20 15:40:22 UTC
Created attachment 216847 [details] [review]
Match rectangle grows redder with increased match certainty

This is useful for debugging your matches as it indicates how certain the
match was in addition to its position.
Comment 5 Will Manley 2012-06-25 19:56:26 UTC
Created attachment 217239 [details] [review]
Produce a warning message if loading template image fails
Comment 6 Thiago Sousa Santos 2014-07-28 18:59:53 UTC
Never too late, right? :)
Updated the patches for 1.0 API and implemented a better test than the one that used the script. Thanks for the patches and sorry it took so long.

commit 06243b7eeee9ab09515e7367c189ef99bf5b652f
Author: Thiago Santos <ts.santos@osg.sisa.samsung.com>
Date:   Mon Jul 28 15:45:09 2014 -0300

    tests: templatematch: add test to check that we use the correct rgb format
    
    templatematch should use the same RGB format that opencv uses (BGR), make sure
    we keep it that way with this test.
    
    https://bugzilla.gnome.org/show_bug.cgi?id=678485

commit f5992a6cdd13c5460a9dd2bef17c7502a0816237
Author: David Rothlisberger <david.rothlisberger@youview.com>
Date:   Fri Jun 15 13:19:06 2012 +0100

    opencv templatematch: Set caps to BGR order
    
    templatematch operates on BGR data. In fact, OpenCV's IplImage always
    stores color image data in BGR order -- this isn't documented at all in
    the OpenCV source code, but there are hints around the web (see for
    example
    http://www.cs.iit.edu/~agam/cs512/lect-notes/opencv-intro/opencv-intro.html#SECTION00041000000000000000
    and http://www.comp.leeds.ac.uk/vision/opencv/iplimage.html ).
    
    gst_templatematch_load_template loads the template (the image to find)
    from disk using OpenCV's cvLoadImage, so it is stored in an IplImage in
    BGR order. But in gst_templatematch_chain, no OpenCV conversion
    functions are used: the imageData pointer of the IplImage for the video
    frame (the image to search in) is just set to point to the raw buffer
    data. Without this fix, that raw data is in RGB order, so the call to
    cvMatchTemplate ends up comparing the template's Blue channel against
    the frame's Red channel, producing very poor results.

commit aa5e25820b0a721580b5d4054477345f66352bb8
Author: Thiago Santos <ts.santos@osg.sisa.samsung.com>
Date:   Mon Jul 28 13:10:35 2014 -0300

    templatematch: mark pads as proxy caps
    
    Allows negotiation to happen properly

commit 1b5483eef98928d000acea579d4e68a5374557f1
Author: William Manley <william.manley@youview.com>
Date:   Mon Jun 25 20:37:01 2012 +0100

    templatematch: Produce a warning message if we can't load a template image

commit 88cb33a1dce12ab6ae847dd417effc00d6954688
Author: William Manley <william.manley@youview.com>
Date:   Wed Jun 20 15:22:52 2012 +0100

    templatematch: Match rectangle grows redder with increased match certainty
    
    This is useful for debugging your matches as it indicates how certain the
    match was in addition to its position.

commit b608767288b738889b4afaf6141fa92c72bb5645
Author: William Manley <william.manley@youview.com>
Date:   Wed Jun 20 15:05:40 2012 +0100

    templatematch: Allow changing template property on the fly
    
    Previously changing the template property resulted in an exception
    thrown from cvMatchTemplate, because "dist_image" (the intermediate
    match-certainty-distribution) was the wrong size (because the
    template image size had changed).
    
    Locking has also been added to allow changing the properties (e.g. the
    pattern to match) while the pipeline is playing.
    
     * gst_element_post_message is moved outside of the lock, because it will
       call into arbitrary user code (otherwise, if that user code calls into
       gst_templatematch_set_property on this same thread it would deadlock).
    
     * gst_template_match_load_template: If we fail to load the new template
       we still unload the previous template, so this element becomes a no-op
       in the pipeline. The alternative would be to keep the previous template;
       I believe unloading the previous template is a better choice, because it
       is consistent with the state this element would be in if it fails to
       load the very first template at start-up.
    
    Thanks to Will Manley for the bulk of this work; any errors are probably
    mine.

commit c77808055de487260363adf4969a4b62cc9048a8
Author: David Rothlisberger <david.rothlisberger@youview.com>
Date:   Fri Jun 1 16:07:34 2012 +0100

    templatematch: Pass video through when nothing to match against
    
    The early return was bypassing the call to gst_pad_push. With no
    filter->template (and thus no filter->cvTemplateImage) the rest of this
    function is essentially a no-op (except for the call to gst_pad_push).
    
    This (plus the previous commit) allows templatematch to be
    enabled/disabled without removing it entirely from the pipeline, by
    setting/unsetting the template property.

commit 056d652b922c6bf926bbfdcd3e9edbf31944e605
Author: William Manley <william.manley@youview.com>
Date:   Wed Jun 20 15:05:06 2012 +0100

    templatematch: Remove no-op call to gst_templatematch_load_template
    
    We have just set filter->template to NULL, so
    gst_templatematch_load_template did nothing.
Comment 7 Thiago Sousa Santos 2014-07-28 19:10:51 UTC
Pushed the 2 bugfixes from the patches above to 1.4 branch:
commit 1f30334b5d16a62bec2af526bdb365a986bfd774
Author: David Rothlisberger <david.rothlisberger@youview.com>
Date:   Fri Jun 15 13:19:06 2012 +0100

    opencv templatematch: Set caps to BGR order
    
    templatematch operates on BGR data. In fact, OpenCV's IplImage always
    stores color image data in BGR order -- this isn't documented at all in
    the OpenCV source code, but there are hints around the web (see for
    example
    http://www.cs.iit.edu/~agam/cs512/lect-notes/opencv-intro/opencv-intro.html#SECTION00041000000000000000
    and http://www.comp.leeds.ac.uk/vision/opencv/iplimage.html ).
    
    gst_templatematch_load_template loads the template (the image to find)
    from disk using OpenCV's cvLoadImage, so it is stored in an IplImage in
    BGR order. But in gst_templatematch_chain, no OpenCV conversion
    functions are used: the imageData pointer of the IplImage for the video
    frame (the image to search in) is just set to point to the raw buffer
    data. Without this fix, that raw data is in RGB order, so the call to
    cvMatchTemplate ends up comparing the template's Blue channel against
    the frame's Red channel, producing very poor results.

commit 9bdb536eec8d0685979a8d9509978cf709c4f01f
Author: Thiago Santos <ts.santos@osg.sisa.samsung.com>
Date:   Mon Jul 28 13:10:35 2014 -0300

    templatematch: mark pads as proxy caps
    
    Allows negotiation to happen properly