After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 643252 - Improve opencv facedetect performance
Improve opencv facedetect performance
Status: RESOLVED FIXED
Product: GStreamer
Classification: Platform
Component: gst-plugins-bad
git master
Other All
: Normal normal
: 1.5.1
Assigned To: GStreamer Maintainers
GStreamer Maintainers
Depends on:
Blocks:
 
 
Reported: 2011-02-25 01:14 UTC by Laura Lucas Alday
Modified: 2014-07-30 19:52 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
always pass a 320x280 image to cvHaarDetectObjects () (5.17 KB, patch)
2011-02-25 01:18 UTC, Laura Lucas Alday
none Details | Review
facedetect and faceblur speed up patch (13.53 KB, patch)
2011-04-30 11:58 UTC, Robert Jobbagy
none Details | Review
facedetect and faceblur fixed speed up patch (13.94 KB, patch)
2011-04-30 12:19 UTC, Robert Jobbagy
reviewed Details | Review
facedetect and faceblur speed up patch2 (17.19 KB, patch)
2011-07-30 11:59 UTC, Robert Jobbagy
needs-work Details | Review
facedetect and faceblur speed up patch 3 (13.84 KB, patch)
2011-08-02 18:20 UTC, Robert Jobbagy
reviewed Details | Review
replace static caps (1.08 KB, patch)
2011-09-10 06:43 UTC, Robert Jobbagy
committed Details | Review
facedetect: faceblur: new properties for speed up (12.14 KB, patch)
2014-07-28 22:44 UTC, Thiago Sousa Santos
reviewed Details | Review

Description Laura Lucas Alday 2011-02-25 01:14:55 UTC
Improve opencv facedetect plugin performance.
Comment 1 Laura Lucas Alday 2011-02-25 01:18:21 UTC
Created attachment 181884 [details] [review]
always pass a 320x280 image to cvHaarDetectObjects ()

This is an idea I tested for improving facedetect performance in resolutions higher than 320x280. It always passes a 320x280 image to cvHaarDetectObjects () and the performance improvement in 640x480 or 800x600 is very noticeable.
Comment 2 Stefan Sauer (gstreamer, gtkdoc dev) 2011-02-27 20:25:58 UTC
Good idea. What about making it configurable with two gobject properties?
Comment 3 Robert Jobbagy 2011-04-29 11:59:55 UTC
I have more idea how can be faster facedetect, my modification reduce cpu usage with 50 percent. I can send my patch if you needs it.
Comment 4 Laura Lucas Alday 2011-04-30 03:23:32 UTC
Of course, please upload it so I can take a look.
Comment 5 Robert Jobbagy 2011-04-30 11:57:26 UTC
Hi Guys,

I fixed and tuning facedetect and faceblur elements.
I add a new bus msg for both of two elements if element couldnt load profile file,it send a bus msg what value is this : profile_load_failed it send this msg once.

after my modification these elements works fine with lower CPU usage with this pipeline :

gst-launch -m v4l2src ! videoscale ! video/x-raw-yuv,width=320,height=240 ! ffmpegcolorspace ! facedetect/faceblur ! ffmpegcolorspace ! xvimagesink

I attached my patches for these elements.

I'm waiting for your response.

Enjoy :)
Comment 6 Robert Jobbagy 2011-04-30 11:58:57 UTC
Created attachment 186929 [details] [review]
facedetect and faceblur speed up patch

increase scale factor
use canny prunning
change profile
increase min size
add profile load failed bus msg
Comment 7 Robert Jobbagy 2011-04-30 12:18:25 UTC
Sorry something small thing missing previous patch. 
I fixed and send again the latest patch.
Comment 8 Robert Jobbagy 2011-04-30 12:19:15 UTC
Created attachment 186932 [details] [review]
facedetect and faceblur fixed speed up patch

fixed profile load failed bus msg
Comment 9 Robert Jobbagy 2011-05-18 22:00:52 UTC
Laura any response ? 

or someone else ? 

Robert
Comment 10 Nicola 2011-07-17 18:39:14 UTC
Hi,

I tested Robert patch and it works fine for me,

Nicola
Comment 11 Stefan Sauer (gstreamer, gtkdoc dev) 2011-07-20 08:54:14 UTC
Review of attachment 186932 [details] [review]:

::: ext/opencv/gstfaceblur.c
@@ +397,3 @@
+     GstMessage *m = gst_message_new_element (GST_OBJECT (filter), s);
+	 gst_element_post_message (GST_ELEMENT (filter), m);
+	 filter->sent_profile_load_failed_msg = TRUE;

I think you could just use GST_ELEMENT_WARNING() here, something like this (please review the actual message and chage as appropriate):

GST_ELEMENT_WARNING (filter, RESOURCE, NOT_FOUND,
  (_("Profile %s is missing.",profile_name)),
  ("missing facedetect profile file %s",profile_name);

This will send a warning message and log a warning (with the 2nd text) as well.
Comment 12 Stefan Sauer (gstreamer, gtkdoc dev) 2011-07-20 09:36:49 UTC
Review of attachment 186932 [details] [review]:

::: ext/opencv/gstfaceblur.c
@@ +70,3 @@
 #define GST_CAT_DEFAULT gst_faceblur_debug
 
+#define DEFAULT_PROFILE "/usr/share/doc/opencv-doc/examples/haarcascades/haarcascades/haarcascade_frontalface_alt2.xml"

I think this should be:
/usr/share/opencv/haarcascades/haarcascade_frontalface_alt2.xml
Comment 13 Stefan Sauer (gstreamer, gtkdoc dev) 2011-07-20 16:33:34 UTC
Review of attachment 186932 [details] [review]:

::: ext/opencv/gstfaceblur.c
@@ +118,3 @@
+  };
+  *id = g_flags_register_static ("GstOpencvFaceDetectFlags", values);
+

*id = g_flags_register_static ("GstOpencvFaceBlurFlags", values);

otherwise you'll get
gst-plugin-scanner:7672): GLib-GObject-WARNING **: cannot register existing type `GstOpencvFaceDetectFlags'

please also rename the flags itself.
Comment 14 Robert Jobbagy 2011-07-20 16:49:18 UTC
Stefan,

Thanks your guide.

It's an old code.
I learned/gave lot of useful advice about gstreamer for you recently,
so I'll update this patch too.

Robert
Comment 15 Robert Jobbagy 2011-07-30 11:43:20 UTC
Test Pipeline:

gst-launch-0.10 -m v4l2src ! video/x-raw-yuv,width=320,height=240,framerate=20/1 ! ffmpegcolorspace ! facedetect profile=/usr/local/share/opencv/haarcascades/haarcascade_frontalface_alt2.xml display=true ! ffmpegcolorspace ! xvimagesink

Test notebook: Asus K42Jv

Without my patch

works all 4 thread about 70-80 % 

and display play ~ 1 fps :(

With my patch:

one thread ~ 25 % 
and other three thread ~ 15 %
Comment 16 Robert Jobbagy 2011-07-30 11:59:54 UTC
Created attachment 192905 [details] [review]
facedetect and faceblur speed up patch2
Comment 17 Stefan Sauer (gstreamer, gtkdoc dev) 2011-08-01 12:45:41 UTC
Review of attachment 192905 [details] [review]:

::: ext/opencv/gstfacedetect.c
@@ +158,1 @@
 

why those arbitrary resolutions? I don't think we want to force everyone to such a low resolution. I would just recommend to use it with videoscale and capsfilter in case of too high cpu load in a example.

@@ +323,1 @@
 }

if you just lock/unlock the mutex in the property setter you can also remove it. The setters never run from different processes at the same time (if the app is done right). We should only use mutexes if we need to ensure that the processing function is getting a coherent set of properties.
Comment 18 Stefan Sauer (gstreamer, gtkdoc dev) 2011-08-01 13:05:00 UTC
Right now I have to use:
gst-launch-0.10 <camerasrc> ! colorspace ! videoscale ! facedetect ! colorspace ! xvimagesink

if I just run
gst-launch-0.10 <camerasrc> ! colorspace ! facedetect ! colorspace ! xvimagesink

it does not show anything, but should actually fail to negotiate.
Comment 19 Robert Jobbagy 2011-08-02 18:20:25 UTC
Created attachment 193097 [details] [review]
facedetect and faceblur speed up patch 3
Comment 20 Robert Jobbagy 2011-08-02 18:21:23 UTC
Stefan,

I fixed these things what we discussed.

Pls check this patch.

Thanks.

Robert
Comment 21 Stefan Sauer (gstreamer, gtkdoc dev) 2011-08-06 16:54:30 UTC
Review of attachment 193097 [details] [review]:

Just one small question. Otherwise this is ready to go.

::: ext/opencv/gstfaceblur.c
@@ +137,2 @@
 static GstStaticPadTemplate src_factory = GST_STATIC_PAD_TEMPLATE ("src",
+    GST_PAD_SRC, GST_PAD_ALWAYS, GST_STATIC_CAPS ("video/x-raw-rgb "));

Is this change needed? (GST_STATIC_CAPS (GST_VIDEO_CAPS_RGB) -> GST_STATIC_CAPS ("video/x-raw-rgb "))
Comment 22 Robert Jobbagy 2011-08-07 11:40:17 UTC
Of course no, this indentation change was in caps update, when I removed it I dont undo this. Sorry for that.
Comment 23 Robert Jobbagy 2011-09-10 06:43:09 UTC
Hi,

I attached this little modification.

Robert
Comment 24 Robert Jobbagy 2011-09-10 06:43:43 UTC
Created attachment 196163 [details] [review]
replace static caps
Comment 25 Stefan Sauer (gstreamer, gtkdoc dev) 2011-11-28 08:57:52 UTC
Review of attachment 193097 [details] [review]:

::: ext/opencv/gstfaceblur.c
@@ +53,3 @@
  * <title>Example launch line</title>
  * |[
+ * gst-launch-0.10 <camerasrc> ! ffmpegcolorspace ! faceblur ! ffmpegcolorspace ! xvimagesink

we have "autovideosrc" for this.
Comment 26 Stefan Sauer (gstreamer, gtkdoc dev) 2011-11-28 08:59:21 UTC
Robert, if you could update the patch, I'll apply it right away. Sorry for not getting back to this any earlier.
Comment 27 Robert Jobbagy 2011-12-01 20:06:30 UTC
Hi Stefan,

No problem, I'll have time on this weekend and I can do that.
It's okay for you ? 

Robert
Comment 28 Stefan Sauer (gstreamer, gtkdoc dev) 2011-12-01 21:36:52 UTC
Yes, that will work. I hope we won't freeze before.
Comment 29 Sebastian Dröge (slomo) 2013-08-23 11:00:44 UTC
Robert, any progress on this? A new patch against latest git master of gst-plugins-bad would be good to have here
Comment 30 Thiago Sousa Santos 2014-07-28 22:44:09 UTC
Created attachment 281892 [details] [review]
facedetect: faceblur: new properties for speed up

Updated for latest master, I didn't incorporate the change in the haar
profile file as I think the one named 'default' should be the default unless
someone give me a good reason for the change.
Comment 31 Nicolas Dufresne (ndufresne) 2014-07-28 22:55:10 UTC
Review of attachment 281892 [details] [review]:

Looks good to me. I would suggest to split this though 1) Port gst 0.10 doc 2) send warning if cascade isn't found and 3) Well this one, but with a more descriptive comment.

::: ext/opencv/gstfaceblur.c
@@ +354,3 @@
+          ("missing faceblur profile file %s", filter->profile));
+      filter->sent_profile_load_failed_msg = TRUE;
+    }

This is slightly unrelated, could have been a seperate patch.
Comment 32 Thiago Sousa Santos 2014-07-30 19:52:21 UTC
Updated and pushed. Thanks for the review.
As I said I didn't update the default profile as there was no reasoning to do so. If there is one, please open a new bug about it.


commit bd3cefeecc732cb0aa376a508165bcd04e9f950f
Author: Robert Jobbagy <jobbagy.robert@gmail.com>
Date:   Wed Jul 30 16:32:01 2014 -0300

    faceblur: facedetect: add properties to faceblur
    
    Makes faceblur have the same properties as facedetect. Also improves
    the standard defaults and makes them the same in the 2 elements

commit 43a4498569f1939e409a08b88445cafe10e26f94
Author: Robert Jobbagy <jobbagy.robert@gmail.com>
Date:   Wed Jul 30 16:30:39 2014 -0300

    faceblur: post message when profile loading fails

commit 555d29681bd661cf8fd5d1681a695f2f2196874a
Author: Robert Jobbagy <jobbagy.robert@gmail.com>
Date:   Wed Jul 30 16:07:17 2014 -0300

    facedetect: faceblur: update launch line examples
    
    Reflect 1.0 and more useful examples