GNOME Bugzilla – Bug 724521
v4l2object: Inconvenient handling of V4L2_FRMSIZE_STEPWISE
Last modified: 2014-07-04 12:57:38 UTC
Some v4l2 driver can return the list of frame sizes as follows: * Discrete: A list of discrete sizes. * Step-wise: The width/height is a range with fixed step. * Continuous: This is a special case of the step-wise type above. The step_width and step_height values are set to 1. Most often there is discrete and continuous types. They're both ok. How implemented the third way? There http://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/sys/v4l2/gstv4l2object.c#n2212 it coded as if (size.type == V4L2_FRMSIZE_TYPE_STEPWISE) { for (w = size.stepwise.min_width, h = size.stepwise.min_height; w <= size.stepwise.max_width && h <= size.stepwise.max_height; w += size.stepwise.step_width, h += size.stepwise.step_height) { tmp = gst_v4l2_object_probe_caps_for_format_and_size (v4l2object, pixelformat, w, h, template); if (tmp) results = g_list_prepend (results, tmp); } } What is wrong with it? Let's see all cases: $grep -n -A 10 -B 10 -r '--include=*.c' V4L2_FRMSIZE_TYPE_STEPWISE drivers/media/platform/vivi.c-51 #define MAX_WIDTH 1920 drivers/media/platform/vivi.c-52 #define MAX_HEIGHT 1200 drivers/media/platform/vivi.c-1048 static const struct v4l2_frmsize_stepwise sizes = { drivers/media/platform/vivi.c-1049 48, MAX_WIDTH, 4, drivers/media/platform/vivi.c-1050 32, MAX_HEIGHT, 1 drivers/media/platform/vivi.c-1051 }; drivers/media/platform/vivi.c:1061 fsize->type = V4L2_FRMSIZE_TYPE_STEPWISE; drivers/media/platform/vivi.c-1062 fsize->stepwise = sizes; We have 469 frame sizes from 48x32 till 1920x500 drivers/media/usb/em28xx/em28xx-video.c-1477 /* Report a continuous range */ drivers/media/usb/em28xx/em28xx-video.c:1478 fsize->type = V4L2_FRMSIZE_TYPE_STEPWISE; drivers/media/usb/em28xx/em28xx-video.c-1479 scale_to_size(dev, EM28XX_HVSCALE_MAX, EM28XX_HVSCALE_MAX, drivers/media/usb/em28xx/em28xx-video.c-1480 &fsize->stepwise.min_width, &fsize->stepwise.min_height); drivers/media/usb/em28xx/em28xx-video.c-1481 if (fsize->stepwise.min_width < 48) drivers/media/usb/em28xx/em28xx-video.c-1482 fsize->stepwise.min_width = 48; drivers/media/usb/em28xx/em28xx-video.c-1483 if (fsize->stepwise.min_height < 38) drivers/media/usb/em28xx/em28xx-video.c-1484 fsize->stepwise.min_height = 38; drivers/media/usb/em28xx/em28xx-video.c-1485 fsize->stepwise.max_width = maxw; drivers/media/usb/em28xx/em28xx-video.c-1486 fsize->stepwise.max_height = maxh; drivers/media/usb/em28xx/em28xx-video.c-1487 fsize->stepwise.step_width = 1; drivers/media/usb/em28xx/em28xx-video.c-1488 fsize->stepwise.step_height = 1; We have 539 frame sizes from 48x38 till 586x576 drivers/media/usb/sn9c102/sn9c102_core.c:2495 frmsize.type = V4L2_FRMSIZE_TYPE_STEPWISE; drivers/media/usb/sn9c102/sn9c102_core.c-2496 frmsize.stepwise.min_width = frmsize.stepwise.step_width = 16; drivers/media/usb/sn9c102/sn9c102_core.c-2497 frmsize.stepwise.min_height = frmsize.stepwise.step_height = 16; drivers/media/usb/sn9c102/sn9c102_core.c-2498 frmsize.stepwise.max_width = cam->sensor.cropcap.bounds.width; drivers/media/usb/sn9c102/sn9c102_core.c-2499 frmsize.stepwise.max_height = cam->sensor.cropcap.bounds.height; drivers/media/usb/sn9c102/sn9c102_core.c-2500 memset(&frmsize.reserved, 0, sizeof(frmsize.reserved)); We have 30 frame sizes from 16x16 till 480x480 drivers/media/usb/gspca/stk1135.c:636 fsize->type = V4L2_FRMSIZE_TYPE_STEPWISE; drivers/media/usb/gspca/stk1135.c-637 fsize->stepwise.min_width = 32; drivers/media/usb/gspca/stk1135.c-638 fsize->stepwise.min_height = 32; drivers/media/usb/gspca/stk1135.c-639 fsize->stepwise.max_width = 1280; drivers/media/usb/gspca/stk1135.c-640 fsize->stepwise.max_height = 1024; drivers/media/usb/gspca/stk1135.c-641 fsize->stepwise.step_width = 2; drivers/media/usb/gspca/stk1135.c-642 fsize->stepwise.step_height = 2; We have 497 frame sizes from 32x32 till 1024x1024 Do you see? All of them are limited to square sizes, except the vivi, which is even worse. I think the best way is to handle V4L2_FRMSIZE_TYPE_STEPWISE in the same way as V4L2_FRMSIZE_TYPE_CONTINUOUS and ignore the step.
It's unclear what is the bug you are reporting. Are you experiencing incorrectness, performance issues or some other issue ? Thanks for clarifying.
Created attachment 269485 [details] $gst-launch-1.0 --gst-debug=v4l2:5 v4l2src ! video/x-raw,width=720,height=576 ! fakesink Trying to capture with 'Pinnacle Hybrid Pro (330e)' on maximum possible frame size.
In short, it is impossible to capture at maximum frame size. From v4l2src point of view, the tuner can capture frames with size from 144x115 to 605x576. But the driver reports its maximum frame size as 720x576. $gst-launch-1.0 --gst-debug=v4l2:5 v4l2src ! video/x-raw,width=720,height=576 ! fakesink gst_v4l2_open:<v4l2src0> Trying to open device /dev/video0 gst_v4l2_get_capabilities:<v4l2src0> getting capabilities gst_v4l2_fill_lists:<v4l2src0> getting enumerations gst_v4l2_fill_lists:<v4l2src0> channels ... gst_v4l2_object_probe_caps_for_format:<v4l2src0> Enumerating frame sizes gst_v4l2_object_probe_caps_for_format:<v4l2src0> we have stepwise frame sizes: gst_v4l2_object_probe_caps_for_format:<v4l2src0> min width: 144 gst_v4l2_object_probe_caps_for_format:<v4l2src0> min height: 115 gst_v4l2_object_probe_caps_for_format:<v4l2src0> max width: 720 gst_v4l2_object_probe_caps_for_format:<v4l2src0> min height: 576 gst_v4l2_object_probe_caps_for_format:<v4l2src0> step width: 1 gst_v4l2_object_probe_caps_for_format:<v4l2src0> step height: 1 gst_v4l2_object_probe_caps_for_format_and_size:<v4l2src0> Unable to enumerate intervals for YUYV@144x115 gst_v4l2_object_probe_caps_for_format_and_size:<v4l2src0> Unable to enumerate intervals for YUYV@145x116 ... gst_v4l2_object_probe_caps_for_format_and_size:<v4l2src0> Unable to enumerate intervals for YUYV@604x575 gst_v4l2_object_probe_caps_for_format_and_size:<v4l2src0> Unable to enumerate intervals for YUYV@605x576 gst_v4l2_object_probe_caps_for_format:<v4l2src0> done iterating stepwise frame sizes ... ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data flow error. Additional debug info: gstbasesrc.c(2809): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: streaming task paused, reason not-negotiated (-4)
Created attachment 269492 [details] Cheese preferences screenshot There are more then 500 rows in each combobox. Btw, it takes 3 min to start the program. I'm sure the delay is related to this bug.
I guess we need some conditions before changing steps into contiguous as this may cause negotiation to fail. I've seen few drivers having these steps, as they expose their alignment requirement there, but also have crop/scale support that would completely mitigate this. Maybe we could try and merge these into contiguous if crop (or selection) is support ? Though there is no probing to then detect the crop/scale limitations.
I just filed bug #726521, which seems to be related to this. It regards the Raspberry Pi camera board v4l2 driver, which introduced stepwise as well (14.9 million possible combinations). Doing this on the Raspberry Pi takes a long time and then it fails (see my reference). Driver Info (not using libv4l2): Driver name : bm2835 mmal Card type : mmal service 15.1 Bus info : platform:bcm2835-v4l2 Driver version: 3.10.33 Capabilities : 0x85000005 Video Capture Video Overlay Read/Write Streaming Device Capabilities Device Caps : 0x05000005 Video Capture Video Overlay Read/Write Streaming Index : 0 Type : Video Capture Pixel Format: 'YU12' Name : 4:2:0, packed YUV Size: Stepwise 16x16 - 2592x1944 with step 2/2 Index : 1 Type : Video Capture Pixel Format: 'YUYV' Name : 4:2:2, packed, YUYV Size: Stepwise 16x16 - 2592x1944 with step 2/2 Index : 2 Type : Video Capture Pixel Format: 'RGB3' Name : RGB24 (LE) Size: Stepwise 16x16 - 2592x1944 with step 2/2 Index : 3 Type : Video Capture Pixel Format: 'JPEG' (compressed) Name : JPEG Size: Stepwise 16x16 - 2592x1944 with step 2/2 Index : 4 Type : Video Capture Pixel Format: 'H264' (compressed) Name : H264 Size: Stepwise 16x16 - 2592x1944 with step 2/2 Index : 5 Type : Video Capture Pixel Format: 'MJPG' (compressed) Name : MJPEG Size: Stepwise 16x16 - 2592x1944 with step 2/2 Index : 6 Type : Video Capture Pixel Format: 'YVYU' Name : 4:2:2, packed, YVYU Size: Stepwise 16x16 - 2592x1944 with step 2/2 Index : 7 Type : Video Capture Pixel Format: 'VYUY' Name : 4:2:2, packed, VYUY Size: Stepwise 16x16 - 2592x1944 with step 2/2 Index : 8 Type : Video Capture Pixel Format: 'UYVY' Name : 4:2:2, packed, UYVY Size: Stepwise 16x16 - 2592x1944 with step 2/2 Index : 9 Type : Video Capture Pixel Format: 'NV12' Name : 4:2:0, packed, NV12 Size: Stepwise 16x16 - 2592x1944 with step 2/2
Thanks, this is good data.
*** Bug 728184 has been marked as a duplicate of this bug. ***
I have created (and now closed as duplicated) a very similar bug to this one. The problem is not only that it takes a long time to start, it is also missing framesizes: gst_v4l2_object_probe_caps_for_format_and_size finds out all the possible frame sizes for a specific device. The device enumerates the possible frame sizes with this ioctl http://linuxtv.org/downloads/v4l-dvb-apis/vidioc-enum-framesizes.html When the devices provides a V4L2_FRMIVAL_TYPE_STEPWISE type of enumeration, a minimum, a maximum and a step is provided. The current code considers that the valid frames sizes are obtained adding step to minimum until maximum is reached when the correct interpretation of the standard takes independently the step with and step height. To make it more clear: With the current code the list of frames is obtained like this (pseudocode) size.x=minsize.x size.y=minsize.y while size < maxsize: addsize(size) size.x += step.x size.y += step.y But it should look like: size.x=minsize.x while size.x < maxsize.x: size.y=minsize=y while size.y < maxsize.y: addsize(size) size.y += step.y size.x += step.y
Thanks, good to know, again for small steps, I suggest to expose it as a range in GST to prevent the caps size becoming too big. For sources, we can always get it to round up, and crop. Loosing a border of 1 or 2 pixels won't make much difference (cameras are already cropping a bit), and I expect that standard resolution are supported in most cases. Right now, a lot of HW have hidden steps, so we need a way to handle that anyway. On blocker I have, is that v4l2 device do a round neared, rather then useful round up.
Created attachment 279583 [details] [review] v4l2: fix probing and enumeration of stepwise frame sizes (untested) Possible patch, completely untested so far.
(In reply to comment #11) > Created an attachment (id=279583) [details] [review] > v4l2: fix probing and enumeration of stepwise frame sizes (untested) > > Possible patch, completely untested so far. This patch at least allows some video to be output on the Raspberry Pi with the following command line: gst-launch-1.0 v4l2src ! video/x-raw,width=1280,height=720 ! glimagesink However, it does not look like it changed the resolution of the video being captured as I could see a lot of aliasing on diagonal edges that I had not seen previously. Also, using video/x-h264,width...,height... ! h264parse ! mp4mux ! filesink does not output any data. v4l2src occasionally spits messages about incorrect pixel format.
I'm mostly concerned about the caps it probes in this case. Any other issues that remain in addition to that should be handled in a separate bug IMHO. Could you make a new GST_DEBUG=*v4l2*:6 log with the patch applied? (just v4l2src num-buffers=1 ! fakesink)
Created attachment 279595 [details] v4l2src num-buffers=1 ! fakesink on RPi
Created attachment 279596 [details] v4l2src ! video/x-h264,width=1280,height=720 ! h264parse ! mp4mux ! filesink location=test.mp4
Uploaded logs from the 1 buffer and H264 cases just in case they're interesting.
Review of attachment 279583 [details] [review]: This looks correct to me, do it need more testing ? Vivi test driver would help btw.
Review of attachment 279583 [details] [review]: Can anyone test and report ? If it does not introduce any regression, I'd like to merge this one for 1.4.
I'll push it later. Rob tested it on the RPi, so I guess it generally works.
*** Bug 726521 has been marked as a duplicate of this bug. ***
Pushed: commit a016f19de8ffb758c87db46be2c636d4ac4bfa44 Author: Tim-Philipp Müller <tim@centricular.com> Date: Mon Jun 30 10:29:54 2014 +0100 v4l2: fix probing and enumeration of stepwise frame sizes The code enumerating STEPWISE framesizes would start from (min_w, min_h) and then add (step_w, step_h) to get the next framesize. However, it should really allow any width from min_w to max_w with step_w and same for heights. Secondly, we would add and probe each individual stepped frame size to the caps as separate structure, which would lead to hundreds if not thousands of structs ending up in the probed caps. Use integer ranges with steps instead. This was particularly noticable with the Raspberry Pi Cam. https://bugzilla.gnome.org/show_bug.cgi?id=724521 https://bugzilla.gnome.org/show_bug.cgi?id=732458 https://bugzilla.gnome.org/show_bug.cgi?id=726521