After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 774576 - opencv: caps to IplImage format conversion fails for sparse formats
opencv: caps to IplImage format conversion fails for sparse formats
Status: RESOLVED FIXED
Product: GStreamer
Classification: Platform
Component: gst-plugins-bad
git master
Other All
: Normal normal
: 1.11.1
Assigned To: GStreamer Maintainers
GStreamer Maintainers
Depends on:
Blocks:
 
 
Reported: 2016-11-16 17:04 UTC by Philippe Renon
Modified: 2016-11-30 02:23 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
opencv: account for sparse formats when converting caps to IPlImage format (1.15 KB, patch)
2016-11-16 17:08 UTC, Philippe Renon
none Details | Review
opencv: account for sparse/padded formats when converting caps to cv image type (6.99 KB, patch)
2016-11-26 12:37 UTC, Philippe Renon
none Details | Review
opencv: account for sparse/padded formats when converting caps to cv image type (4.29 KB, patch)
2016-11-26 16:03 UTC, Philippe Renon
committed Details | Review
opencv: add opencv image format documentation (2.77 KB, patch)
2016-11-26 16:03 UTC, Philippe Renon
committed Details | Review

Description Philippe Renon 2016-11-16 17:04:44 UTC
gstopencvutils has utility to extract the number of channel and depth from caps.
This is later used to select an appropriate IplImage format.

This utility does not handle sparse formats well. It will report 3 channels when it should report 4 channels.
Comment 1 Philippe Renon 2016-11-16 17:08:38 UTC
Created attachment 340032 [details] [review]
opencv: account for sparse formats when converting caps to IPlImage format

This demonstrates a use case of https://bugzilla.gnome.org/show_bug.cgi?id=773433
Comment 2 Philippe Renon 2016-11-23 21:17:33 UTC
Here is some background about how images are handled in opencv.

The various opencv image containers or headers carry the following information:
- number of channels (usually 1 to 4)
- depth (8, 16, 32, 64...); all channels have the same depth.
The channel layout (BGR vs RGB) is not carried...

This gives us the following list of supported image formats:
  CV_8UC1, CV_8UC2, CV_8UC3, CV_8UC4
  CV_8SC1, CV_8SC2, CV_8SC3, CV_8SC4
  CV_16UC1, CV_16UC2, CV_16UC3, CV_16UC4
  CV_16SC1, CV_16SC2, CV_16SC3, CV_16SC4
  CV_32SC1, CV_32SC2, CV_32SC3, CV_32SC4
  CV_32FC1, CV_32FC2, CV_32FC3, CV_32FC4
  CV_64FC1, CV_64FC2, CV_64FC3, CV_64FC4

Where the first part of the format name is the depth followed by a digit representing the number of channels. Note that opencv supports more that 4 channels.

The opencv algorithms don't all support all the image types.
For example findChessboardCorners() supports only 8 bits formats (gray scale and color).

And, typically, this algorithm will convert the image to gray scale before proceeding. It will do so with something like this:
   cvtColor(srcImg, destImg, CV_BGR2GRAY);

The conversion will work on any BGR format (BGR, BGRA, BGRx). The extra channel(s) will be ignored.
It will also produce a result for any RGB format. The result will be "wrong" to the human eye and might affect some algorithms (not findChessboardCorners() afaik...). This is due to how RGB gets converted to gray where each color has a different weight.

Another example is the 2D rendering API. It work with RGB but the colors will be wrong.

The bad thing is that it is not possible to change the "default" BGR format.
Safest is to not assume that RGB will work and always convert to BGR.

That said, the current opencv gstreamer elements all accept BGR and RGB caps ! Some have restrictions but if a format is supported then both BGR and RGB layouts will be supported.

The patch that I am about to push does not address the above.
Instead it makes the situation worse ;) by adding support for the x layouts (BGRx, RGBx).
Testing shows that x layouts work with expected color errors when using RGB layouts.
Comment 3 Philippe Renon 2016-11-26 12:37:06 UTC
Created attachment 340796 [details] [review]
opencv: account for sparse/padded formats when converting caps to cv image type
Comment 4 Philippe Renon 2016-11-26 16:03:01 UTC
Created attachment 340804 [details] [review]
opencv: account for sparse/padded formats when converting  caps to cv image type
Comment 5 Philippe Renon 2016-11-26 16:03:32 UTC
Created attachment 340805 [details] [review]
opencv: add opencv image format documentation
Comment 6 Luis de Bethencourt 2016-11-26 17:53:30 UTC
I will review this on Monday. If I forget, remind me via email.
Comment 7 Nicolas Dufresne (ndufresne) 2016-11-30 02:13:40 UTC
Review of attachment 340804 [details] [review]:

Looks good, I have conflicting patch here, will try to merge it.
Comment 8 Nicolas Dufresne (ndufresne) 2016-11-30 02:14:36 UTC
Review of attachment 340805 [details] [review]:

Looks good.
Comment 9 Nicolas Dufresne (ndufresne) 2016-11-30 02:22:25 UTC
Attachment 340805 [details] pushed as 4664fc0 - opencv: add opencv image format documentation