GNOME Bugzilla – Bug 772766
V4L2 compositor proposal
Last modified: 2018-11-03 15:12:33 UTC
Created attachment 337446 [details] [review] V4L2 compositor code Hi folks, Here is a proposal of a new GStreamer element which performs video composition using video4linux2 M2M devices. Today we are able to use our new element with simple pipelines using gst-launch and also more complex ones using our application on our platform (imx6 board). In both cases, fbdevsink element (with sometimes others sinks) is used to display the composition's result. Used capture/output resolution is 720p or 1080p and we use various video format: YUV2, NV12 and RGB16. Ouput IO-mode is always dmabuf-import and capture IO-mode is always dmabuf. The current status of our development is work-in-progress and well there is still some bugs. It seems to work as expected with gst-launch but we experiment different errors when using it in our application after some start/stop cycles (== gstreamer pipeline reinstanciation from a same process). That is why life cycle of our buffers is greatly suspected. Typically encountered errors are after a failing start are: ERROR v4l2allocator gstv4l2allocator.c:1245:gst_v4l2_allocator_qbuf:<ipu1-capture:pool:src:allocator> failed queueing buffer 0: Invalid argument ERROR v4l2bufferpool gstv4l2bufferpool.c:1127:gst_v4l2_buffer_pool_qbuf:<ipu1-capture:pool:src> could not queue a buffer 0 ERROR v4l2allocator gstv4l2allocator.c:1245:gst_v4l2_allocator_qbuf:<ipu1-capture:pool:src:allocator> failed queueing buffer 1: Invalid argument ERROR v4l2bufferpool gstv4l2bufferpool.c:1127:gst_v4l2_buffer_pool_qbuf:<ipu1-capture:pool:src> could not queue a buffer 1 (...) *or* ERROR gst_v4l2_compositor_queue_jobs() failed which means buffers are still owned somewhere in the pipeline (downstream). *or* process freezing :( A this point I cannot find where is/are the problem(s). We also use non-standard elements in our application or plaform (gstreamer elements, drivers, ...). So at this point some help/feedback should be appreciated. Can someone try to run and compile it on another platform ? Is there an easy way to restart the pipeline from gst-launch (gstreamer pipeline reinstanciation from a same process ?) I put two patches in attachment. Both should apply on the current master branch. First one (v4l2compositor.patch) is the code of the compositor itself. Note that we use gstvideoaggregator (from gst-plugins-bad) as base class after importing and renaming it in the gst-plugins-good repository. Second one (testing_with_gst-launch.patch) is only for testing purpose only. It makes some modifications in v4l2convert that we put in front of fbdevsink in some of our gst-launch commands (see below). Testing v4l2compositor with gst-launch ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Here is a complete example with gst-launch with some logs. It composes two stream into one. Here all streams are 720p: # GST_DEBUG_NO_COLOR=1 GST_DEBUG=*:3 gst-launch-1.0 -v v4l2compositor num_jobs=12 device=/dev/v4l/by-path/ipu1-scaler output-io-mode=5 capture-io-mode=4 name=comp sink_0::xpos=8 sink_0::ypos=8 sink_0::width=624 sin k_0::height=344 sink_1::xpos=648 sink_1::ypos=8 sink_1::width=624 sink_1::height =344 ! "video/x-raw,pixel-aspect-ratio=1/1,colorimetry=bt709,height=720,framer ate=24/1,width=1280" ! v4l2convert device=/dev/v4l/by-path/ipu2-scaler output-io-mode=5 capture-io-mode=auto ! fbdevsink v4l2src device=/dev/v4l/by-path/ipu1-capture io-mode=4 ! "video/x-raw,pixel-aspect-ratio=1/1,colorimetry=bt709,height=720,framerate=24/1,width=1280,format=YUY2" ! comp. v4l2src device=/dev/v4l/by-path/ipu2-capture io-mode=4 ! "video/x-raw,pixel-aspect-ratio=1/1,colorimetry=bt709,height=720,framerate=24/1,width=1280,format=YUY2" ! comp. Setting pipeline to PAUSED ... WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:<v4l2transform0> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:<v4l2transform0> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:<v4l2transform0> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:<v4l2transform0> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:<v4l2transform0> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:<v4l2transform0> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:<v4l2transform0> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:<v4l2transform0> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:<v4l2transform0> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:<v4l2transform0> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument WARN v4l2 gstv4l2object.c:1849:gst_v4l2_object_add_aspect_ratio:<v4l2transform0> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument WARN v4l2src gstv4l2src.c:545:gst_v4l2src_query:<v4l2src0> Can't give latency since framerate isn't fixated ! Pipeline is live and does not need PREROLL ... WARN v4l2src gstv4l2src.c:545:gst_v4l2src_query:<v4l2src1> Can't give latency since framerate isn't fixated ! WARN v4l2_aggregator gstv4l2aggregator.c:1409:gst_v4l2_aggregator_query_latency_unlocked:<comp> Latency query failed Setting pipeline to PLAYING ... New clock: GstSystemClock /GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1 /GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1 /GstPipeline:pipeline0/GstV4l2Compositor:comp.GstV4l2CompositorPad:sink_0: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1 /GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1 /GstPipeline:pipeline0/GstV4l2Src:v4l2src1.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1 /GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1 /GstPipeline:pipeline0/GstV4l2Compositor:comp.GstV4l2CompositorPad:sink_1: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1 /GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, framerate=(fraction)24/1 WARN v4l2bufferpool gstv4l2bufferpool.c:748:gst_v4l2_buffer_pool_start:<v4l2src0:pool:src> Uncertain or not enough buffers, enabling copy threshold WARN v4l2bufferpool gstv4l2bufferpool.c:748:gst_v4l2_buffer_pool_start:<v4l2src1:pool:src> Uncertain or not enough buffers, enabling copy threshold FIXME basesink gstbasesink.c:3124:gst_base_sink_default_event:<fbdevsink0> stream-start event without group-id. Consider implementing group-id handling in the upstream elements WARN structure gststructure.c:1935:priv_gst_structure_append_to_gstring: No value transform to serialize field 'chroma-site' of type 'string' /GstPipeline:pipeline0/GstV4l2Compositor:comp.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive, chroma-site=(string)NULL WARN structure gststructure.c:1935:priv_gst_structure_append_to_gstring: No value transform to serialize field 'chroma-site' of type 'string' /GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive, chroma-site=(string)NULL /GstPipeline:pipeline0/GstV4l2Transform:v4l2transform0.GstPad:src: caps = video/x-raw, format=(string)RGB16, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive /GstPipeline:pipeline0/GstFBDEVSink:fbdevsink0.GstPad:sink: caps = video/x-raw, format=(string)RGB16, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive WARN v4l2bufferpool gstv4l2bufferpool.c:748:gst_v4l2_buffer_pool_start:<v4l2transform0:pool:src> Uncertain or not enough buffers, enabling copy threshold WARN structure gststructure.c:1935:priv_gst_structure_append_to_gstring: No value transform to serialize field 'chroma-site' of type 'string' /GstPipeline:pipeline0/GstV4l2Transform:v4l2transform0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive, chroma-site=(string)NULL WARN structure gststructure.c:1935:priv_gst_structure_append_to_gstring: No value transform to serialize field 'chroma-site' of type 'string' /GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive, chroma-site=(string)NULL Redistribute latency... <Ctrl+C> Execution ended after 0:00:36.615017671 Setting pipeline to PAUSED ... Setting pipeline to READY ... Setting pipeline to NULL ... Freeing pipeline ... Regards, Sebastien
Created attachment 337447 [details] [review] V4l2convert patches
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/issues/308.