GNOME Bugzilla – Bug 725835
v4l2sink: Use GstVideoCropMeta to set overlay window if available
Last modified: 2018-11-03 14:51:52 UTC
Created attachment 271119 [details] [review] v4l2sink: Use GstVideoCropMeta to set overlay window if available An upstream element could provide buffers with GstVideoCropMeta to v4l2sink. The main idea of the attached patch is to update the V4L2 overlay window according to incoming GstVideoCropMeta (if available) when rendering frames. However, current implementation of this will override any overlay rectangle set by set_property() method. Maybe it should be property first then GstVideoCropMeta. More background: Our H264 decoder output non-cropped frames with a definition multiple of 16, which is the h264 macroblock size. When input video definition is not a multiple of 16, encoding/decoding process add padding and provide crop information. Since our gst pipeline is decoder --> v4l2sink, we want to let the cropping job to the v4l2 device. So decoder send cropping info through GstVideoCropMeta and v4l2sink set overlay window accordingly. Current patch apply on master and 1.2.3, however i only tested it on 1.2.3
Review of attachment 271119 [details] [review]: I think it's indeed a good idea to handle crop meta, specially that we advertise doing so gst_query_add_allocation_meta (query, GST_VIDEO_CROP_META_API_TYPE, NULL); Though, I this solution is not complete, and does not correlate with V4L2 specification. Overlay in V4L2 is designed initially for sources (capture) that renders directly on top of the frame buffer. It is now experimentally accepted for sink (output) though the effect need to be correlated with the crop information in order to preserve aspect ratio. The spec is still very vague on how to probe this. But what is stated is that both overlay and crop settings can crop or scale. The spec also seem to say that there exist a way to check if it's being scaled and what factor, but it's still very unclear in my mind. http://linuxtv.org/downloads/v4l-dvb-apis/osd.html http://linuxtv.org/downloads/v4l-dvb-apis/crop.html Also, if overlay is not supported, fallback to using crop would be nice, and vis-versa. There exist a rather large overlap between the two API.
Btw, the correct solution should ensure that pixel aspect ratio is maintained, and that cropped portion of a video are not visible to the end user (regardless of the HW limitations).
Crop and scale part for output device is also unclear in my mind at this moment. The spec say cropping ioctls on an output device determine the area where images are inserted. So i don't think we can really crop in this case using cropping ioctls. It seems that we can determine if a device does scaling or not with cropcaps. However it is not really clear how, maybe we should check if cropcaps bounds are equal to cropcaps defrect. For now, I will try first to ensure that pixel aspect ratio remains the same Moreover, since all this behavior with output device seems to be an abuse of the V4L2 cropping/overlay API, it could be interesting to rely on the experimental selection API. Drawback is device compatibilty issue.
(In reply to comment #3) > Crop and scale part for output device is also unclear in my mind at this > moment. > The spec say cropping ioctls on an output device determine the area where > images are inserted. So i don't think we can really crop in this case using > cropping ioctls. Well, if you place your video at a negative location and make the window larger then display, what you get is a crop (assuming the HW can scale). > It seems that we can determine if a device does scaling or not with cropcaps. > However it is not really clear how, maybe we should check if cropcaps bounds > are equal to cropcaps defrect. I made the same conclusion here. > For now, I will try first to ensure that pixel aspect ratio remains the same I'd love to know, specially, how to userspace know that the driver is doing it right. > Moreover, since all this behavior with output device seems to be an abuse of > the V4L2 cropping/overlay API, it could be interesting to rely on the > experimental selection API. Drawback is device compatibilty issue. Heard about this one, obviously not yet implemented in GStreamer, but I'd like to know more at some point.
(In reply to comment #4) > Well, if you place your video at a negative location and make the window larger > then display, what you get is a crop (assuming the HW can scale). I think it could work but it depends of the drivers. For instance my v4l2 output doesn't support negative coordinates. > I'd love to know, specially, how to userspace know that the driver is doing it > right. Well in fact it doesn't know and we shouldn't assume that it does. This work should be done by the application according with the pixelaspect given by CROPCAP ioctl. I wrote a draft of a solution but it may be better to submit it in another issue, no ? > > Moreover, since all this behavior with output device seems to be an abuse of > > the V4L2 cropping/overlay API, it could be interesting to rely on the > > experimental selection API. Drawback is device compatibilty issue. > > Heard about this one, obviously not yet implemented in GStreamer, but I'd like > to know more at some point. Just for reference : http://linuxtv.org/downloads/v4l-dvb-apis/selection-api.html
For now it's has been disabled, so we don't advertise something we don't do.
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/issues/110.