Created attachment 205374 [details] [review]
Adds generate-frames option to v4l2src for better support of signal loss cases

wouldn't it be smarter to not send frames and have a videorate element afterwards duplicate frames. v4l2src would mark the first frame after it regained the signal with discont. 

We could make the signal loss not fatal and just post a warning/info message on the bus. Dunno if it would make sense to have a timeout and error out if the signal is not regained after some time.

I'd vote for any solution that makes the loss of signal non-fatal.  As long as there is a message on the bus, the app can substitute the v4l source with any other means of generating a fallback signal.

I'm not sure of a clean way in the current system to catch the v4l errors that are posted and intercept the EOS before the pipeline comes down (though there's likely a better way I missed).  If the v4l source can handle the logic of detecting re-availability of the device, that would be very useful.

See also #667724

Actually it would be nice to detect as early as possible that a frame is lost in v4l2src ... and notify that with a GAP event (on which videorate can react quickly).

Review of attachment 205374 [details] [review]:

This patch seems to depend on another one not included, is that right ? Also, as Edward mention, we need to generate GAP event, not frames.

(In reply to comment #5)
> Actually it would be nice to detect as early as possible that a frame is lost
> in v4l2src ... and notify that with a GAP event (on which videorate can react
> quickly).

Any idea, design in mind ? I'm myself unsure how signal lost is detected, and if this is only for antenna base sources ? Do we get an event from v4l2, or simply the streams stops (or never starts).

In the second scenario, I may propose having deadline timer the get reset on each frames, and send gaps event if frame did not arrive on expected time. It might solve some issues with cameras, as they often take takes ages to come up and don't respect the configured framerate (or maybe I'm mixing stuff here, just tell me if so).

Ping ?  Can anyone clarify what definition of signal lost we are talking about ?  It is signal-lost in the ATSC/DVB sense, or is it more like, someone has pulled out the USB plug ?

For tracking devices, we have DeviceMonitor now, though I feel swapping element and dropping erros should be done outside of the src (e.g. camerabin ?)

it's signal lost in the sense that when you have a capture card the input source is not actually generating a signal. Maybe because the cable got plugged or the device got switched off or whatever reason.

Someone pulling out the capture device from the USB plug is a different scenario, that should just generate a normal error.

I'm not sure if v4l2src should really be in the business of generating black frames, although it should notify the loss of signal and when it's re-established again.

I can't find anything in V4L2 API that would clearly let us know when signal is lost. Can anyone explain what drivers would do to let the application know ?

I think it's status field and V4L2_IN_ST_NO_SIGNAL, but not 100% sure. I am pretty sure signalling for this exists in some form even if it's not necessarily implemented by all drivers, I've seen it used somewhere in a product.

This seems to come out of VIDIOC_ENUMINPUT call, which we probe once and we ignore the status field where this could be set. I do see yet how this can be detected, Anthony are you still willing to work on that ? Are you willing to post the entire patchset ?

I can work on it. 
I can looking for in Vl2 API to know if it can return an information when the signal is lost but I can not to tell you when I will work on it.
I hope to have time next week to propose a patch with this feature.

Any news ?

(In reply to comment #14)
> Any news ?

sorry I have not started yet. I am trying to see what we can do.
V4l2.VIDIOC_ENUMINPUT can be used and then recover the status to see if there is signal.
For the GAP event can we define how long before videorate duplicates the last frame?

Indeed, the "official" way of querying the device for signal presence is currently V4l2.VIDIOC_ENUMINPUT.status (see http://www.linuxtv.org/downloads/v4l-dvb-apis/vidioc-enuminput.html -- which provides details about the cause, see http://www.linuxtv.org/downloads/v4l-dvb-apis/vidioc-enuminput.html#input-status). However, i believe this may be driver-dependant, which means we may not rely (only) on this. All devices we had our hands on did expose that, but may only return 0 when signal is present and not fully provide the exact reason. I don't know if all the drivers do implement the status properly though, any counter examples would be interesting.

However, in some cases the device may become unresponding (e.g. you plug a USB webcam out during capture), so no more querying. The only reliable thing is whether open() and dqbuf ioctl does not return -1.

Overall, the first big step is indeed to make the loss of signal non-fatal, i.e. what we have done in our current patch
* v4l2 ioctl node opening should be done with the O_NONBLOCK flag (http://linuxtv.org/downloads/v4l-dvb-apis/func-open.html) so that the whole plugin does not freeze when a call blocks (otherwise it will wait until the generic kernel 120s (!!!) timeout /proc/sys/kernel/hung_task_timeout_secs)
* v4l2src does not emit EOS in case of error, but either pushes a generated picture (scenario #1) or pushes down an event (scenario #2)
* keeps trying to restart capture by closing and re-opening the device node every N seconds (defined by the 'retry-delay' property in our implementation)
* whenever the device opening or dqbuf succeeds again, push the new buffers and stop the workaround (stop generating pictures or push down an event)

Regarding the vent (custom meta or GAP event), since you can't really predict how long the no-signal will be, is it okay if you push several contiguous GAP events with a constant duration (e.g. defined after the retry-delay property) ?

I agree with separating the frame generation in videorate or a dedicated element so that it can also be used with decklinksrc, dv1394src, ... As long as they emit GAP events.

The mitigation strategy defined in the target element should be configurable among:
* duplicate the last known frame (could be something not so nice, like an incomplete buffer, and will require a copy per buffer or keep the previous buffer allocated)
* black picture, or even color-selectable, with an optional user-defined text, or even text containing details on the failure (e.g. unsupported input, out-of-range...)
* a png picture ?
* ...

Here is a WIP list of known issues that we want to mitigate on a generic manner:
* signal unplugged during capture (dqbuf fails)
* no signal when starting capture (requires defining a default caps negotiation strategy -- will not work with devices/drivers that don't support scaling)
* v4l2 device disappears (USB unplugged example)
* v4l2 device driver crashes and stops responding (buggy hardware / overheating hardware)
* random "temporary" kernel errors, which require to restart the capture after some "quiet time"
* override device-provided fake frames by the user-defined strategy (some devices actually do it) by querying V4l2.VIDIOC_ENUMINPUT.status 

Open questions:
* what about compressed streams ?
* is this feature generic enough to integrate into videorate or is a separate element better ?

(In reply to comment #16)
> However, in some cases the device may become unresponding (e.g. you plug a USB
> webcam out during capture), so no more querying. The only reliable thing is
> whether open() and dqbuf ioctl does not return -1.

Indeed, as soon as USB device get unplugegd, ENODEV is returned and an error is posted.

> 
> Overall, the first big step is indeed to make the loss of signal non-fatal,
> i.e. what we have done in our current patch
> * v4l2 ioctl node opening should be done with the O_NONBLOCK flag
> (http://linuxtv.org/downloads/v4l-dvb-apis/func-open.html) so that the whole
> plugin does not freeze when a call blocks (otherwise it will wait until the
> generic kernel 120s (!!!) timeout /proc/sys/kernel/hung_task_timeout_secs)

Would be nice to provide some clarification, since we should never be doing an block ioctl(), as we poll before action. If an ioctl() blocks after polling succeed, then it's likely a driver bug.

Note that NONBLOCK was commented out in 2006, would be nice if someone could dig up associated bug and figure-out why before going this way:

https://bugzilla.gnome.org/show_bug.cgi?id=338818

> * v4l2src does not emit EOS in case of error, but either pushes a generated
> picture (scenario #1) or pushes down an event (scenario #2)
> * keeps trying to restart capture by closing and re-opening the device node
> every N seconds (defined by the 'retry-delay' property in our implementation)
> * whenever the device opening or dqbuf succeeds again, push the new buffers and
> stop the workaround (stop generating pictures or push down an event)

In my opinion, this is too much for an element. The element could instead provide (if not already) ways to let application handle it. I remember Youness Alaoui that propsed an API to transparently handle WebCAMs going away, flipping it with a black frame generator and letting application dynamically select a new one, or the same if it came back. This is all outdated, but if it was possible when he did it, it should still be possible now. Except now we have new discovery API (GstDeviceMonitor) which should help.


> Here is a WIP list of known issues that we want to mitigate on a generic
> manner:
> * signal unplugged during capture (dqbuf fails)

As there is no event from the device, application can grab the device FD and poll for it. If signal is lost, it would swap the source, and keep polling, when it comes back, it would swap again.

> * no signal when starting capture (requires defining a default caps negotiation
> strategy -- will not work with devices/drivers that don't support scaling)

Not sure what happen, will probably lead to no-negotiated on certain devices. Application can catch the error on sync bus, remove it. We could have a way to poll on the element while it's in READY state. This will always remain specific to this element.

> * v4l2 device disappears (USB unplugged example)

That will throw an error, application can remove this error and swap.

> * v4l2 device driver crashes and stops responding (buggy hardware / overheating
> hardware)

This case might be unrecoverable, no luck, fix your driver. Per experience, this often lead to the lost of a CPU and all attached process becomes zomby.

> * random "temporary" kernel errors, which require to restart the capture after
> some "quiet time"

Will error out, same thing.

> * override device-provided fake frames by the user-defined strategy (some
> devices actually do it) by querying V4l2.VIDIOC_ENUMINPUT.status 

Should be application side, with may be property to help.

> 
> Open questions:
> * what about compressed streams ?

There is nothing you can do, unless you are allowed to swap with something that encode.

> * is this feature generic enough to integrate into videorate or is a separate
> element better ?

I don't think it's inline with videorate role in my opinion.

I just fixed the bug title so we need what need to be

*** This bug has been marked as a duplicate of bug 745441 ***