Comment on attachment 287152 [details]
Minimal code to reproduce.

The reason for a stall would normally be that the upstream buffer pool is exhausted (otherwise a deadlock).

I don't see obvious leak in that sample code, so I'll have to reproduce it. Will do a little later, thanks for reporting, and special thanks for taking the time to provide a sample application.

Ok, I confirm I could reproduce. Tested against 1.2, it works, and git master when it hangs.

I need to think a little more about this one. Adding enable-last-sample=0 to appsink makes it work again. Though, it should still work (ineffectively) with this option set:

If I understand well, appsink as a queue of 1 buffer by default, enable-last-sample is true by default, hence another buffer will be stored. This mean appsink can effectively start v4l2src which allocates 2 buffers.

v4l2src: DQ and push buffer 1
appsink: queue buffer 1
v4l2src: DQ and push buffer 2 (blocks)
app: pull buffer 1 and unref
appsink: keeps buffers 1 as the last sample
appsink: queue buffer 2 (unblock v4l2src)
v4l2src: blocks, no more capture buffer
app: pull buffer 2 and unref
appsink: replace last-sample, releasing buffer 1
v4l2src: unblock, Q buffer 1, capture buffer 1 and push
etc.

But buffer never get released for some reason I don't understand yet. So just saying appsink should request more buffers does not seems fully right.

Looks like it only occure if using libv4l2, and using one of the internal converters. This test forces RGB, which is rarely supported by HW.

We found some additional info about the issue. The following are changes to original sample code pipeline.

1. This one still hangs, even though RGB is not requested there:
szPipelineDesc = g_strdup_printf("%s name=source device=%s ! queue ! videoconvert ! appsink name=sink", "v4l2src", "/dev/video0", format);
2. On the other hand, this pipeline works (it's the original with RGB caps filter, but with NV12 format on appsink itself):
const gchar * format = "video/x-raw,format=RGB,width=640,height=480";
szPipelineDesc = g_strdup_printf("%s name=source device=%s ! %s ! queue ! videoconvert ! appsink name=sink caps=video/x-raw,format=NV12", "v4l2src", "/dev/video0", format);

We are finding this problem too in openframeworks which uses gstreamer to access video under linux. According to a user, going back to kernel 3.14 instead of 3.16 solves the problem:

http://forum.openframeworks.cc/t/arch-linux-ofvideograbber-only-showing-first-frame-gstreamer-error-on-exit/17178

also found that asking for nv12 (i guess any other format not directly supported by the device will do) and having a videoconvert between v4l2src and appsink fixes the issue. 

setting enable-last-sample to false doesn't seem to work for me

also adding a videorate like:

v4l2src device=/dev/video0 ! video/x-raw,format=RGB,width=1280,height=720,framerate=30/1 ! videorate ! appsink name=ofappsink caps="video/x-raw, format=RGB, width=1280, height=720

solves the issue

Just to double check, on which version of GStreamer are these test being run ? The fact that changing the kernel version changes the behaviour is very suspicious. GStreamer is behaving the same, what was changed in your specific v4l2 driver ?

i'm using gstreamer 1.4.3 on ubuntu 14.10 not sure what's the difference with the kernel it was someone else who reported that changing the kernel version fixes the problem and in any case he just downgraded the kernel i don't think he knows what's the difference in the driver either. 

for me adding a videorate or any element that changes the frames (like a videoconvert that is not just a passthrough) fixes it too as explained above.

might it be that newest vl42 modules are generating some different timestamping? something like a videorate would be overwrtting those making appsink work again properly

No, this is not a possibility. The most probable cause of hangs here is that the v4l2 buffer pool get exhausted. Do you keep buffers out of appsink around ? Since as soon as you keep two, the pipeline will hang. To be able to keep more, you would have to find a way to override the allocation query and request the amount of buffers you want. Elements that copy the buffers will indeed workaround this.

I'm starting to think this behaviour is fine and v4l2src has nothing special to do with that. What would be missing is a way to tell appsink how many buffers we want.

yes i'm using this in an openGL loop so i'm keeping a frame from appsink around until the next gl frame is rendered. this has never been a problem before and other elements like playbin playsrc... are totally fine

(In reply to arturo from comment #11)
> yes i'm using this in an openGL loop so i'm keeping a frame from appsink
> around until the next gl frame is rendered. this has never been a problem
> before and other elements like playbin playsrc... are totally fine

Ok noted. Before we were over allocating in v4l2src. That would work around your problem, but not really solve it. Of course if you had a color conversion, this also hide the problem. In 0.10 one could influence the size of v4l2 queue. But this makes very little sense, since that queue might not be pushing buffers directly.

The right solution would be to add propose_allocation() implementation in appsink, and a way for the application to influence that. I guess it would be nice if this isn't limited to choosing min/max pool size, but also allow exposing support for special features like GstVideoMeta, GstVideoCrop, etc.

thanks. btw i'm testing by not keeping frames around and still have the same problem in case that's of any help.

would there be a way to solve this in current versions? i'm using a videorate in between the v4l2src and the appsink but for some reason it makes the video super laggy.

There is possibility you are hitting more then one bugs. Obviously don't forget to disable "enable-last-sample" property. Videorate will keep a buffer around, there is path in newer version of GStreamer the fixes that, not backported yet, but we could if that helps.


Ideally, re-test this pipeline with git master. In git master there is many fixes that cannot really be backported since they are new features. We detect cameras that produce broken timestamp now. Broken timestamp can lead to similar symptom. For sure, update to 1.4.5, as there is many bug fix between 1.4.3 and 1.4.5.

Sorry for the late reply but i just had a chance to test this on 1.4.5 and the problem is still there. Now not even adding a videorate solves it i need to immediately copy and release the buffers as soon as i get them instead of being able to hold them until i don't need them any more which introduces an extra copy that I didn't need before.

> I'm starting to think this behaviour is fine and v4l2src has nothing special
> to do with that. What would be missing is a way to tell appsink how many
> buffers we want.

It seems to me that v4l2src allocates too few buffers by default and this is causing problems pretty much everywhere for everything. Can't we default to something higher? (People on embedded devices can make it lower again if they want and fine-tune it to their pipelines, but that doesn't mean we should default to that IMHO).

Is there any metric that would justify allocating more (performance, or a logic that would rationalise this) ? My perspective so far is that the pipeline should report their needs and appsink *is* the problem. Other elements have been fixed (like videorate). Appsink/appsrc disable everything allocation query related (not just choosing the number of buffers).

By the way, it was already agreed that a queue-size property (in fact min-queue-size), would be re-added at least to v4l2src. Patches are welcome. Increasing that default is a matter of taste. An analogy is the encoder, one could say the default should be good for VOIP, other may pretend it should be good for films and local playback. It's the same, except that saving memory can benefit both embedded and desktop world.

I'm looking at this mostly from a user perspective. Requiring that users set properties on appsink (or elsewhere) to make simple stuff like this work means GStreamer has failed the user. If we can't make basic stuff work out of the box everywhere we are better off just copying all output frames like we used to and leaving the enabling of more performant modes of operation as an exercise to the user.

Just to be sure, the pipeline-with-x264enc-doesn't-preroll situation is also us failing the user for example, so this is not just v4l2src specific, I'm just giving my rationale here. If we require users to understand too much of how specific stuff works under the hood, we have failed them and half of GStreamer's raison d'etre disappears, IMHO ;)

Actually, no need argument, I already implemented and pushed a solution to this bug apparently. I wrote this after a discussion with Sebastien. It's very simple, if the allocation is uncertain (or driver have given less then needed), we now allocate at least 4 buffers (the default we had in the last decade) and enable the copy threshold in case the pipeline need even more.

Great :)

In case someone want to backport, commit 6afd1c5d

Oh, and by uncertain, I mean that no allocation pool has been set at all in the query (hence no min/max).

I believe this is working fine (even though not optimal) in master ?