GNOME Bugzilla – Bug 796723
v4l2src + jpegdec dropping every second frame (timestamping / clocking / QoS issues?)
Last modified: 2018-11-03 15:31:36 UTC
I'd been using GStreamer 1.12, which uses libv4l2, with this v4l2src usage: gst-launch-1.0 v4l2src device=/dev/v4l/buffalo ! 'video/x-raw,width=1280,height=720,framerate=30/1' ! videoconvert ! fpsdisplaysink AFAIK this works because the webcam outputs MJPEG and libv4l2 decodes it (the camera cannot output uncompressed video at 30FPS). GStreamer 1.14 disabled use of libv4l2 by default, though the above pipeline still works with GST_V4L2_USE_LIBV4L2=1. I used this with kind of basic pattern with a more complicated pipeline compositing multiple cameras and a screen capture. However, trying to use gstreamer's own jpegdec instead, as is required without libv4l2, does not work properly (using libv4l2 with jpegdec doesn't work either). This is the test pipeline: gst-launch-1.0 v4l2src device=/dev/v4l/buffalo ! 'image/jpeg,width=1280,height=720,framerate=30/1' ! jpegdec ! videoconvert ! fpsdisplaysink I get 15 FPS output, and every second frame is dropped. This only happens with one of my two webcams - the other yields a clean 30 FPS. With GST_DEBUG=5 I see a line like this every second frame: 0:00:06.137449218 12416 0x55621e964590 DEBUG videodecoder gstvideodecoder.c:3155:gst_video_decoder_clip_and_push_buf:<jpegdec0> Dropping frame due to QoS. start:0:00:04.839125944 deadline:0:00:04.839125944 earliest_time:0:00:04.854135807 Using "fpsdisplaysink sync=false" works around the issue and yields 30 FPS (for both cameras). I get the feeling there's a clocking issue. Having both the webcam produce frames at a certain rate and the display consume them at its own rate (60Hz, which is a multiple of 30Hz) seems to make jpegdec think it's too late to the party after every output frame, on the next frame. Both webcams are nominally 30 FPS, but I guess one is probably slightly under and one slightly over that. I'll attach debug logs for the second pipeline above, with both webcams (the one that causes dropped frames and the one that doesn't).
Created attachment 372892 [details] fragment of problem log (every second frame dropped)
Created attachment 372893 [details] fragment of OK log
Interesting, I can reproduce on both camera's here.
It's a problem with latency reporting apparently. v4l2src hardcodes 1 frames latency (33ms for this test case). And jpegdec is suppose to declare another frame latency (because it decodes the current frames when it find the start of the next one). But the total latency is 33ms: GST_DEBUG="bin:4" gst-launch-1.0 v4l2src device=/dev/video1 ! 'image/jpeg,width=1280,height=720,framerate=30/1' ! jpegdec ! videoconvert ! fpsdisplaysink . . . New clock: GstSystemClock 0:00:01.041722891 26674 0x7ffad0004c00 INFO bin gstbin.c:2778:gst_bin_do_latency_func:<pipeline0> configured latency of 0:00:00.033333333 If I manually fix the latency, it works fine: GST_DEBUG="bin:4" gst-launch-1.0 pipeline. \( latency=66666666 v4l2src device=/dev/video1 ! 'image/jpeg,width=1280,height=720,framerate=30/1' ! identity drop-allocation=1 ! jpegdec ! videoconvert ! fpsdisplaysink \) So there is two things to work on, a) fix the latency reporting in jpegdec (and avdec_mjpeg), and b) get v4l2src to signal jpegdec that the jpeg images are already framed, so we get back this 1 frame latency. libv4l2 already knows the jpeg are framed, hence the reason we didn't have this latency.
Yup, I can confirm that your pipeline with the hardcoded latency works fine with both webcams. Is there a way of applying this only for the jpegdec element, so I can try it on more complex pipelines without having to manually figure out what the expected whole-pipeline latency is supposed to be?
Created attachment 372894 [details] [review] jpegdec: Declare the latency introduced by the parser The parser looks for the start of a jpeg frame and will only complete the frame when the couple first bytes of the following frame is received. This has the side effect on adding one frame latency when the source is live.
That's step one, now I'll try and get rid of the extra frame latency.
Hmm, that is super odd, the parser is not called with v4l2src, jpegdec only parse if the segment is not in TIME format. I really need to find were this latency comes from then.
I'm actually on the wrong track. I've tested manually increasing latency in 720, 1080, and 2160p, and found out that it's not exactly one frame. The bigger the resolution, the higher the latency. So this is in fact processing latency. On my laptop, processing a 1080p jpeg takes about 33ms, a 720p buffer a little less, so it was really luck that it worked.
Ok, so the difference is that the QoS is implemented in the decoder does not allow any latenesss, while the videosink allow up to 1 frame late and/or max-lateness (apparently depends on the sink). This can be showed by doing this: gst-launch-1.0 v4l2src device=/dev/video1 ! jpegdec ! video/x-raw,framerate=30/1,height=1080 ! queue ! fpsdisplaysink video-sink="glimagesink qos=0" The lie about this pipeline is that all the frames are displayed late on screen (about 1 frame). As v4l2src have no reason to implement qos, it does not, and just pushes the late frames. that's why you get better results. I think one of the solution is: https://bugzilla.gnome.org/show_bug.cgi?id=640610
Meanwhile, we should fix GstVideoDecoder, it drops frames but does not post QoS messages in that specific case.
The core of this issue is fixed by 640610, but this fix cannot be backported to 1.14. Meanwhile you can workaround by increasing the latency of your pipeline by handling the do-latency signal or by forcing a latency on the pipeline.
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/issues/488.