After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 724521 - v4l2object: Inconvenient handling of V4L2_FRMSIZE_STEPWISE
v4l2object: Inconvenient handling of V4L2_FRMSIZE_STEPWISE
Status: RESOLVED FIXED
Product: GStreamer
Classification: Platform
Component: gst-plugins-good
git master
Other Linux
: Normal normal
: 1.3.91
Assigned To: GStreamer Maintainers
GStreamer Maintainers
: 726521 728184 (view as bug list)
Depends on:
Blocks:
 
 
Reported: 2014-02-17 03:34 UTC by Pavel Bludov
Modified: 2014-07-04 12:57 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
$gst-launch-1.0 --gst-debug=v4l2:5 v4l2src ! video/x-raw,width=720,height=576 ! fakesink (735.08 KB, text/plain)
2014-02-18 00:43 UTC, Pavel Bludov
  Details
Cheese preferences screenshot (25.33 KB, image/png)
2014-02-18 01:49 UTC, Pavel Bludov
  Details
v4l2: fix probing and enumeration of stepwise frame sizes (untested) (3.60 KB, patch)
2014-06-30 09:38 UTC, Tim-Philipp Müller
committed Details | Review
v4l2src num-buffers=1 ! fakesink on RPi (8.33 KB, application/octet-stream)
2014-06-30 12:47 UTC, Robert Swain
  Details
v4l2src ! video/x-h264,width=1280,height=720 ! h264parse ! mp4mux ! filesink location=test.mp4 (7.43 KB, application/octet-stream)
2014-06-30 12:48 UTC, Robert Swain
  Details

Description Pavel Bludov 2014-02-17 03:34:42 UTC
Some v4l2 driver can return the list of frame sizes as follows:

 *   Discrete: A list of discrete sizes.
 *   Step-wise: The width/height is a range with fixed step.
 *   Continuous: This is a special case of the step-wise type above.
     The step_width and step_height values are set to 1.

Most often there is discrete and continuous types. They're both ok.

How implemented the third way? There
    http://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/sys/v4l2/gstv4l2object.c#n2212
it coded as

if (size.type == V4L2_FRMSIZE_TYPE_STEPWISE) {

    for (w = size.stepwise.min_width, h = size.stepwise.min_height;
        w <= size.stepwise.max_width && h <= size.stepwise.max_height;
        w += size.stepwise.step_width, h += size.stepwise.step_height) {

      tmp =
          gst_v4l2_object_probe_caps_for_format_and_size (v4l2object,
          pixelformat, w, h, template);

      if (tmp)
        results = g_list_prepend (results, tmp);
        
    }
  }

What is wrong with it? Let's see all cases:
$grep -n -A 10 -B 10 -r '--include=*.c' V4L2_FRMSIZE_TYPE_STEPWISE

drivers/media/platform/vivi.c-51	#define MAX_WIDTH 1920
drivers/media/platform/vivi.c-52	#define MAX_HEIGHT 1200	
drivers/media/platform/vivi.c-1048	static const struct v4l2_frmsize_stepwise sizes = {
drivers/media/platform/vivi.c-1049		48, MAX_WIDTH, 4,
drivers/media/platform/vivi.c-1050		32, MAX_HEIGHT, 1
drivers/media/platform/vivi.c-1051	};
drivers/media/platform/vivi.c:1061	fsize->type = V4L2_FRMSIZE_TYPE_STEPWISE;
drivers/media/platform/vivi.c-1062	fsize->stepwise = sizes;

We have 469 frame sizes from 48x32 till 1920x500

drivers/media/usb/em28xx/em28xx-video.c-1477	/* Report a continuous range */
drivers/media/usb/em28xx/em28xx-video.c:1478	fsize->type = V4L2_FRMSIZE_TYPE_STEPWISE;
drivers/media/usb/em28xx/em28xx-video.c-1479	scale_to_size(dev, EM28XX_HVSCALE_MAX, EM28XX_HVSCALE_MAX,
drivers/media/usb/em28xx/em28xx-video.c-1480		      &fsize->stepwise.min_width, &fsize->stepwise.min_height);
drivers/media/usb/em28xx/em28xx-video.c-1481	if (fsize->stepwise.min_width < 48)
drivers/media/usb/em28xx/em28xx-video.c-1482		fsize->stepwise.min_width = 48;
drivers/media/usb/em28xx/em28xx-video.c-1483	if (fsize->stepwise.min_height < 38)
drivers/media/usb/em28xx/em28xx-video.c-1484		fsize->stepwise.min_height = 38;
drivers/media/usb/em28xx/em28xx-video.c-1485	fsize->stepwise.max_width = maxw;
drivers/media/usb/em28xx/em28xx-video.c-1486	fsize->stepwise.max_height = maxh;
drivers/media/usb/em28xx/em28xx-video.c-1487	fsize->stepwise.step_width = 1;
drivers/media/usb/em28xx/em28xx-video.c-1488	fsize->stepwise.step_height = 1;

We have 539 frame sizes from 48x38 till 586x576 

drivers/media/usb/sn9c102/sn9c102_core.c:2495	frmsize.type = V4L2_FRMSIZE_TYPE_STEPWISE;
drivers/media/usb/sn9c102/sn9c102_core.c-2496	frmsize.stepwise.min_width = frmsize.stepwise.step_width = 16;
drivers/media/usb/sn9c102/sn9c102_core.c-2497	frmsize.stepwise.min_height = frmsize.stepwise.step_height = 16;
drivers/media/usb/sn9c102/sn9c102_core.c-2498	frmsize.stepwise.max_width = cam->sensor.cropcap.bounds.width;
drivers/media/usb/sn9c102/sn9c102_core.c-2499	frmsize.stepwise.max_height = cam->sensor.cropcap.bounds.height;
drivers/media/usb/sn9c102/sn9c102_core.c-2500	memset(&frmsize.reserved, 0, sizeof(frmsize.reserved));

We have 30 frame sizes from 16x16 till 480x480 

drivers/media/usb/gspca/stk1135.c:636	fsize->type = V4L2_FRMSIZE_TYPE_STEPWISE;
drivers/media/usb/gspca/stk1135.c-637	fsize->stepwise.min_width = 32;
drivers/media/usb/gspca/stk1135.c-638	fsize->stepwise.min_height = 32;
drivers/media/usb/gspca/stk1135.c-639	fsize->stepwise.max_width = 1280;
drivers/media/usb/gspca/stk1135.c-640	fsize->stepwise.max_height = 1024;
drivers/media/usb/gspca/stk1135.c-641	fsize->stepwise.step_width = 2;
drivers/media/usb/gspca/stk1135.c-642	fsize->stepwise.step_height = 2;

We have 497 frame sizes from 32x32 till 1024x1024

Do you see? All of them are limited to square sizes, except the vivi,
which is even worse.
I think the best way is to handle V4L2_FRMSIZE_TYPE_STEPWISE in the same
way as V4L2_FRMSIZE_TYPE_CONTINUOUS and ignore the step.
Comment 1 Nicolas Dufresne (ndufresne) 2014-02-17 14:41:27 UTC
It's unclear what is the bug you are reporting. Are you experiencing incorrectness, performance issues or some other issue ? Thanks for clarifying.
Comment 2 Pavel Bludov 2014-02-18 00:43:49 UTC
Created attachment 269485 [details]
$gst-launch-1.0 --gst-debug=v4l2:5 v4l2src ! video/x-raw,width=720,height=576 ! fakesink

Trying to capture with 'Pinnacle Hybrid Pro (330e)' on maximum possible frame size.
Comment 3 Pavel Bludov 2014-02-18 01:40:11 UTC
In short, it is impossible to capture at maximum frame size.
From v4l2src point of view, the tuner can capture frames with size from 144x115 to 605x576.
But the driver reports its maximum frame size as 720x576.

$gst-launch-1.0 --gst-debug=v4l2:5 v4l2src ! video/x-raw,width=720,height=576 ! fakesink

gst_v4l2_open:<v4l2src0> Trying to open device /dev/video0
gst_v4l2_get_capabilities:<v4l2src0> getting capabilities
gst_v4l2_fill_lists:<v4l2src0> getting enumerations
gst_v4l2_fill_lists:<v4l2src0>   channels
...
gst_v4l2_object_probe_caps_for_format:<v4l2src0> Enumerating frame sizes
gst_v4l2_object_probe_caps_for_format:<v4l2src0> we have stepwise frame sizes:
gst_v4l2_object_probe_caps_for_format:<v4l2src0> min width:   144
gst_v4l2_object_probe_caps_for_format:<v4l2src0> min height:  115
gst_v4l2_object_probe_caps_for_format:<v4l2src0> max width:   720
gst_v4l2_object_probe_caps_for_format:<v4l2src0> min height:  576
gst_v4l2_object_probe_caps_for_format:<v4l2src0> step width:  1
gst_v4l2_object_probe_caps_for_format:<v4l2src0> step height: 1
gst_v4l2_object_probe_caps_for_format_and_size:<v4l2src0> Unable to enumerate intervals for YUYV@144x115
gst_v4l2_object_probe_caps_for_format_and_size:<v4l2src0> Unable to enumerate intervals for YUYV@145x116
...
gst_v4l2_object_probe_caps_for_format_and_size:<v4l2src0> Unable to enumerate intervals for YUYV@604x575
gst_v4l2_object_probe_caps_for_format_and_size:<v4l2src0> Unable to enumerate intervals for YUYV@605x576
gst_v4l2_object_probe_caps_for_format:<v4l2src0> done iterating stepwise frame sizes
...
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data flow error.
Additional debug info:
gstbasesrc.c(2809): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming task paused, reason not-negotiated (-4)
Comment 4 Pavel Bludov 2014-02-18 01:49:13 UTC
Created attachment 269492 [details]
Cheese preferences screenshot

There are more then 500 rows in each combobox.
Btw, it takes 3 min to start the program.
I'm sure the delay is related to this bug.
Comment 5 Nicolas Dufresne (ndufresne) 2014-03-15 10:42:49 UTC
I guess we need some conditions before changing steps into contiguous as this may cause negotiation to fail. I've seen few drivers having these steps, as they expose their alignment requirement there, but also have crop/scale support that would completely mitigate this. Maybe we could try and merge these into contiguous if crop (or selection) is support ? Though there is no probing to then detect the crop/scale limitations.
Comment 6 Tobias Wolf 2014-03-17 12:48:01 UTC
I just filed bug #726521, which seems to be related to this. It regards the Raspberry Pi camera board v4l2 driver, which introduced stepwise as well (14.9 million possible combinations). Doing this on the Raspberry Pi takes a long time and then it fails (see my reference).


Driver Info (not using libv4l2):
	Driver name   : bm2835 mmal
	Card type     : mmal service 15.1
	Bus info      : platform:bcm2835-v4l2
	Driver version: 3.10.33
	Capabilities  : 0x85000005
		Video Capture
		Video Overlay
		Read/Write
		Streaming
		Device Capabilities
	Device Caps   : 0x05000005
		Video Capture
		Video Overlay
		Read/Write
		Streaming

	Index       : 0
	Type        : Video Capture
	Pixel Format: 'YU12'
	Name        : 4:2:0, packed YUV
		Size: Stepwise 16x16 - 2592x1944 with step 2/2

	Index       : 1
	Type        : Video Capture
	Pixel Format: 'YUYV'
	Name        : 4:2:2, packed, YUYV
		Size: Stepwise 16x16 - 2592x1944 with step 2/2

	Index       : 2
	Type        : Video Capture
	Pixel Format: 'RGB3'
	Name        : RGB24 (LE)
		Size: Stepwise 16x16 - 2592x1944 with step 2/2

	Index       : 3
	Type        : Video Capture
	Pixel Format: 'JPEG' (compressed)
	Name        : JPEG
		Size: Stepwise 16x16 - 2592x1944 with step 2/2

	Index       : 4
	Type        : Video Capture
	Pixel Format: 'H264' (compressed)
	Name        : H264
		Size: Stepwise 16x16 - 2592x1944 with step 2/2

	Index       : 5
	Type        : Video Capture
	Pixel Format: 'MJPG' (compressed)
	Name        : MJPEG
		Size: Stepwise 16x16 - 2592x1944 with step 2/2

	Index       : 6
	Type        : Video Capture
	Pixel Format: 'YVYU'
	Name        : 4:2:2, packed, YVYU
		Size: Stepwise 16x16 - 2592x1944 with step 2/2

	Index       : 7
	Type        : Video Capture
	Pixel Format: 'VYUY'
	Name        : 4:2:2, packed, VYUY
		Size: Stepwise 16x16 - 2592x1944 with step 2/2

	Index       : 8
	Type        : Video Capture
	Pixel Format: 'UYVY'
	Name        : 4:2:2, packed, UYVY
		Size: Stepwise 16x16 - 2592x1944 with step 2/2

	Index       : 9
	Type        : Video Capture
	Pixel Format: 'NV12'
	Name        : 4:2:0, packed, NV12
		Size: Stepwise 16x16 - 2592x1944 with step 2/2
Comment 7 Nicolas Dufresne (ndufresne) 2014-03-17 13:58:16 UTC
Thanks, this is good data.
Comment 8 Ricardo Ribalda 2014-04-14 14:21:29 UTC
*** Bug 728184 has been marked as a duplicate of this bug. ***
Comment 9 Ricardo Ribalda 2014-04-14 14:26:16 UTC
I have created (and now closed as duplicated) a very similar bug to this one.

The problem is not only that it takes a long time to start, it is also missing framesizes:


gst_v4l2_object_probe_caps_for_format_and_size finds out all the possible frame
sizes for a specific device.

The device enumerates the possible frame sizes with this ioctl
http://linuxtv.org/downloads/v4l-dvb-apis/vidioc-enum-framesizes.html

When the devices provides a V4L2_FRMIVAL_TYPE_STEPWISE type of enumeration, a
minimum, a maximum and a step is provided.

The current code considers that the valid frames sizes are obtained adding step
to minimum until maximum is reached when the correct interpretation of the
standard takes independently the step with and step height.

To make it more clear:


With the current code the list of frames is obtained like this (pseudocode)

size.x=minsize.x
size.y=minsize.y

while size < maxsize:
    addsize(size)
    size.x += step.x
    size.y += step.y

But it should look like:

size.x=minsize.x

while size.x < maxsize.x:
  size.y=minsize=y
  while size.y < maxsize.y:
     addsize(size)
     size.y += step.y
  size.x += step.y
Comment 10 Nicolas Dufresne (ndufresne) 2014-04-14 15:06:25 UTC
Thanks, good to know, again for small steps, I suggest to expose it as a range in GST to prevent the caps size becoming too big. For sources, we can always get it to round up, and crop. Loosing a border of 1 or 2 pixels won't make much difference (cameras are already cropping a bit), and I expect that standard resolution are supported in most cases.

Right now, a lot of HW have hidden steps, so we need a way to handle that anyway. On blocker I have, is that v4l2 device do a round neared, rather then useful round up.
Comment 11 Tim-Philipp Müller 2014-06-30 09:38:18 UTC
Created attachment 279583 [details] [review]
v4l2: fix probing and enumeration of stepwise frame sizes (untested)

Possible patch, completely untested so far.
Comment 12 Robert Swain 2014-06-30 12:28:02 UTC
(In reply to comment #11)
> Created an attachment (id=279583) [details] [review]
> v4l2: fix probing and enumeration of stepwise frame sizes (untested)
> 
> Possible patch, completely untested so far.

This patch at least allows some video to be output on the Raspberry Pi with the following command line:

gst-launch-1.0 v4l2src ! video/x-raw,width=1280,height=720 ! glimagesink

However, it does not look like it changed the resolution of the video being captured as I could see a lot of aliasing on diagonal edges that I had not seen previously.

Also, using video/x-h264,width...,height... ! h264parse ! mp4mux ! filesink does not output any data. v4l2src occasionally spits messages about incorrect pixel format.
Comment 13 Tim-Philipp Müller 2014-06-30 12:39:49 UTC
I'm mostly concerned about the caps it probes in this case. Any other issues that remain in addition to that should be handled in a separate bug IMHO.

Could you make a new GST_DEBUG=*v4l2*:6 log with the patch applied? (just v4l2src num-buffers=1 ! fakesink)
Comment 14 Robert Swain 2014-06-30 12:47:55 UTC
Created attachment 279595 [details]
v4l2src num-buffers=1 ! fakesink on RPi
Comment 15 Robert Swain 2014-06-30 12:48:50 UTC
Created attachment 279596 [details]
v4l2src ! video/x-h264,width=1280,height=720 ! h264parse ! mp4mux ! filesink location=test.mp4
Comment 16 Robert Swain 2014-06-30 12:49:39 UTC
Uploaded logs from the 1 buffer and H264 cases just in case they're interesting.
Comment 17 Nicolas Dufresne (ndufresne) 2014-06-30 15:07:21 UTC
Review of attachment 279583 [details] [review]:

This looks correct to me, do it need more testing ? Vivi test driver would help btw.
Comment 18 Nicolas Dufresne (ndufresne) 2014-07-01 17:03:24 UTC
Review of attachment 279583 [details] [review]:

Can anyone test and report ? If it does not introduce any regression, I'd like to merge this one for 1.4.
Comment 19 Tim-Philipp Müller 2014-07-01 17:25:19 UTC
I'll push it later. Rob tested it on the RPi, so I guess it generally works.
Comment 20 Tim-Philipp Müller 2014-07-01 19:09:24 UTC
*** Bug 726521 has been marked as a duplicate of this bug. ***
Comment 21 Tim-Philipp Müller 2014-07-01 19:29:08 UTC
Pushed:

commit a016f19de8ffb758c87db46be2c636d4ac4bfa44
Author: Tim-Philipp Müller <tim@centricular.com>
Date:   Mon Jun 30 10:29:54 2014 +0100

    v4l2: fix probing and enumeration of stepwise frame sizes
    
    The code enumerating STEPWISE framesizes would start from
    (min_w, min_h) and then add (step_w, step_h) to get the
    next framesize. However, it should really allow any width
    from min_w to max_w with step_w and same for heights.
    Secondly, we would add and probe each individual stepped
    frame size to the caps as separate structure, which would
    lead to hundreds if not thousands of structs ending up in
    the probed caps. Use integer ranges with steps instead.
    
    This was particularly noticable with the Raspberry Pi Cam.
    
    https://bugzilla.gnome.org/show_bug.cgi?id=724521
    https://bugzilla.gnome.org/show_bug.cgi?id=732458
    https://bugzilla.gnome.org/show_bug.cgi?id=726521