After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 668018 - v4l2src: Adds generate-frames option to v4l2src for better support of signal loss cases
v4l2src: Adds generate-frames option to v4l2src for better support of signal ...
Status: RESOLVED DUPLICATE of bug 745441
Product: GStreamer
Classification: Platform
Component: gst-plugins-good
git master
Other Linux
: Normal normal
: git master
Assigned To: GStreamer Maintainers
GStreamer Maintainers
Depends on:
Blocks:
 
 
Reported: 2012-01-16 15:10 UTC by anthony
Modified: 2015-03-02 13:58 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Adds generate-frames option to v4l2src for better support of signal loss cases (4.10 KB, patch)
2012-01-16 15:11 UTC, anthony
rejected Details | Review

Description anthony 2012-01-16 15:10:39 UTC
If generate-frames is set to True, black frames will be generated during capture when the v4l device loses signal.

Exemple pipeline with generate-frames=true:

gst-launch v4l2src device=/dev/video0 generate-frames=true ! xvimagesink -v

If the device loses the signal, instead of issuing an error, v4l2src will continue generating frames similar (in resolution and format) until the signal is up again.

Note: this case does not appear with webcams, mostly with frame grabbers and similar.
Comment 1 anthony 2012-01-16 15:11:49 UTC
Created attachment 205374 [details] [review]
Adds generate-frames option to v4l2src for better support of signal loss cases
Comment 2 Stefan Sauer (gstreamer, gtkdoc dev) 2012-07-08 18:24:13 UTC
wouldn't it be smarter to not send frames and have a videorate element afterwards duplicate frames. v4l2src would mark the first frame after it regained the signal with discont. 

We could make the signal loss not fatal and just post a warning/info message on the bus. Dunno if it would make sense to have a timeout and error out if the signal is not regained after some time.
Comment 3 Louis 2012-07-08 21:02:10 UTC
I'd vote for any solution that makes the loss of signal non-fatal.  As long as there is a message on the bus, the app can substitute the v4l source with any other means of generating a fallback signal.

I'm not sure of a clean way in the current system to catch the v4l errors that are posted and intercept the EOS before the pipeline comes down (though there's likely a better way I missed).  If the v4l source can handle the logic of detecting re-availability of the device, that would be very useful.
Comment 4 Stefan Sauer (gstreamer, gtkdoc dev) 2012-07-09 15:46:57 UTC
See also #667724
Comment 5 Edward Hervey 2014-01-29 14:02:07 UTC
Actually it would be nice to detect as early as possible that a frame is lost in v4l2src ... and notify that with a GAP event (on which videorate can react quickly).
Comment 6 Nicolas Dufresne (ndufresne) 2014-03-09 00:47:18 UTC
Review of attachment 205374 [details] [review]:

This patch seems to depend on another one not included, is that right ? Also, as Edward mention, we need to generate GAP event, not frames.
Comment 7 Nicolas Dufresne (ndufresne) 2014-03-09 00:53:06 UTC
(In reply to comment #5)
> Actually it would be nice to detect as early as possible that a frame is lost
> in v4l2src ... and notify that with a GAP event (on which videorate can react
> quickly).

Any idea, design in mind ? I'm myself unsure how signal lost is detected, and if this is only for antenna base sources ? Do we get an event from v4l2, or simply the streams stops (or never starts).

In the second scenario, I may propose having deadline timer the get reset on each frames, and send gaps event if frame did not arrive on expected time. It might solve some issues with cameras, as they often take takes ages to come up and don't respect the configured framerate (or maybe I'm mixing stuff here, just tell me if so).
Comment 8 Nicolas Dufresne (ndufresne) 2014-03-29 20:30:49 UTC
Ping ?  Can anyone clarify what definition of signal lost we are talking about ?  It is signal-lost in the ATSC/DVB sense, or is it more like, someone has pulled out the USB plug ?

For tracking devices, we have DeviceMonitor now, though I feel swapping element and dropping erros should be done outside of the src (e.g. camerabin ?)
Comment 9 Tim-Philipp Müller 2014-07-02 07:50:33 UTC
it's signal lost in the sense that when you have a capture card the input source is not actually generating a signal. Maybe because the cable got plugged or the device got switched off or whatever reason.

Someone pulling out the capture device from the USB plug is a different scenario, that should just generate a normal error.

I'm not sure if v4l2src should really be in the business of generating black frames, although it should notify the loss of signal and when it's re-established again.
Comment 10 Nicolas Dufresne (ndufresne) 2014-07-03 01:44:57 UTC
I can't find anything in V4L2 API that would clearly let us know when signal is lost. Can anyone explain what drivers would do to let the application know ?
Comment 11 Tim-Philipp Müller 2014-07-04 08:13:01 UTC
I think it's status field and V4L2_IN_ST_NO_SIGNAL, but not 100% sure. I am pretty sure signalling for this exists in some form even if it's not necessarily implemented by all drivers, I've seen it used somewhere in a product.
Comment 12 Nicolas Dufresne (ndufresne) 2014-10-29 20:16:06 UTC
This seems to come out of VIDIOC_ENUMINPUT call, which we probe once and we ignore the status field where this could be set. I do see yet how this can be detected, Anthony are you still willing to work on that ? Are you willing to post the entire patchset ?
Comment 13 anthony 2014-11-03 20:07:36 UTC
I can work on it. 
I can looking for in Vl2 API to know if it can return an information when the signal is lost but I can not to tell you when I will work on it.
I hope to have time next week to propose a patch with this feature.
Comment 14 Nicolas Dufresne (ndufresne) 2014-12-09 21:14:31 UTC
Any news ?
Comment 15 anthony 2014-12-10 21:41:51 UTC
(In reply to comment #14)
> Any news ?

sorry I have not started yet. I am trying to see what we can do.
V4l2.VIDIOC_ENUMINPUT can be used and then recover the status to see if there is signal.
For the GAP event can we define how long before videorate duplicates the last frame?
Comment 16 Florent Thiéry 2015-01-09 17:11:30 UTC
Indeed, the "official" way of querying the device for signal presence is currently V4l2.VIDIOC_ENUMINPUT.status (see http://www.linuxtv.org/downloads/v4l-dvb-apis/vidioc-enuminput.html -- which provides details about the cause, see http://www.linuxtv.org/downloads/v4l-dvb-apis/vidioc-enuminput.html#input-status). However, i believe this may be driver-dependant, which means we may not rely (only) on this. All devices we had our hands on did expose that, but may only return 0 when signal is present and not fully provide the exact reason. I don't know if all the drivers do implement the status properly though, any counter examples would be interesting.

However, in some cases the device may become unresponding (e.g. you plug a USB webcam out during capture), so no more querying. The only reliable thing is whether open() and dqbuf ioctl does not return -1.

Overall, the first big step is indeed to make the loss of signal non-fatal, i.e. what we have done in our current patch
* v4l2 ioctl node opening should be done with the O_NONBLOCK flag (http://linuxtv.org/downloads/v4l-dvb-apis/func-open.html) so that the whole plugin does not freeze when a call blocks (otherwise it will wait until the generic kernel 120s (!!!) timeout /proc/sys/kernel/hung_task_timeout_secs)
* v4l2src does not emit EOS in case of error, but either pushes a generated picture (scenario #1) or pushes down an event (scenario #2)
* keeps trying to restart capture by closing and re-opening the device node every N seconds (defined by the 'retry-delay' property in our implementation)
* whenever the device opening or dqbuf succeeds again, push the new buffers and stop the workaround (stop generating pictures or push down an event)

Regarding the vent (custom meta or GAP event), since you can't really predict how long the no-signal will be, is it okay if you push several contiguous GAP events with a constant duration (e.g. defined after the retry-delay property) ?

I agree with separating the frame generation in videorate or a dedicated element so that it can also be used with decklinksrc, dv1394src, ... As long as they emit GAP events.

The mitigation strategy defined in the target element should be configurable among:
* duplicate the last known frame (could be something not so nice, like an incomplete buffer, and will require a copy per buffer or keep the previous buffer allocated)
* black picture, or even color-selectable, with an optional user-defined text, or even text containing details on the failure (e.g. unsupported input, out-of-range...)
* a png picture ?
* ...

Here is a WIP list of known issues that we want to mitigate on a generic manner:
* signal unplugged during capture (dqbuf fails)
* no signal when starting capture (requires defining a default caps negotiation strategy -- will not work with devices/drivers that don't support scaling)
* v4l2 device disappears (USB unplugged example)
* v4l2 device driver crashes and stops responding (buggy hardware / overheating hardware)
* random "temporary" kernel errors, which require to restart the capture after some "quiet time"
* override device-provided fake frames by the user-defined strategy (some devices actually do it) by querying V4l2.VIDIOC_ENUMINPUT.status 

Open questions:
* what about compressed streams ?
* is this feature generic enough to integrate into videorate or is a separate element better ?
Comment 17 Nicolas Dufresne (ndufresne) 2015-01-09 18:49:27 UTC
(In reply to comment #16)
> However, in some cases the device may become unresponding (e.g. you plug a USB
> webcam out during capture), so no more querying. The only reliable thing is
> whether open() and dqbuf ioctl does not return -1.

Indeed, as soon as USB device get unplugegd, ENODEV is returned and an error is posted.

> 
> Overall, the first big step is indeed to make the loss of signal non-fatal,
> i.e. what we have done in our current patch
> * v4l2 ioctl node opening should be done with the O_NONBLOCK flag
> (http://linuxtv.org/downloads/v4l-dvb-apis/func-open.html) so that the whole
> plugin does not freeze when a call blocks (otherwise it will wait until the
> generic kernel 120s (!!!) timeout /proc/sys/kernel/hung_task_timeout_secs)

Would be nice to provide some clarification, since we should never be doing an block ioctl(), as we poll before action. If an ioctl() blocks after polling succeed, then it's likely a driver bug.

Note that NONBLOCK was commented out in 2006, would be nice if someone could dig up associated bug and figure-out why before going this way:

https://bugzilla.gnome.org/show_bug.cgi?id=338818

> * v4l2src does not emit EOS in case of error, but either pushes a generated
> picture (scenario #1) or pushes down an event (scenario #2)
> * keeps trying to restart capture by closing and re-opening the device node
> every N seconds (defined by the 'retry-delay' property in our implementation)
> * whenever the device opening or dqbuf succeeds again, push the new buffers and
> stop the workaround (stop generating pictures or push down an event)

In my opinion, this is too much for an element. The element could instead provide (if not already) ways to let application handle it. I remember Youness Alaoui that propsed an API to transparently handle WebCAMs going away, flipping it with a black frame generator and letting application dynamically select a new one, or the same if it came back. This is all outdated, but if it was possible when he did it, it should still be possible now. Except now we have new discovery API (GstDeviceMonitor) which should help.


> Here is a WIP list of known issues that we want to mitigate on a generic
> manner:
> * signal unplugged during capture (dqbuf fails)

As there is no event from the device, application can grab the device FD and poll for it. If signal is lost, it would swap the source, and keep polling, when it comes back, it would swap again.

> * no signal when starting capture (requires defining a default caps negotiation
> strategy -- will not work with devices/drivers that don't support scaling)

Not sure what happen, will probably lead to no-negotiated on certain devices. Application can catch the error on sync bus, remove it. We could have a way to poll on the element while it's in READY state. This will always remain specific to this element.

> * v4l2 device disappears (USB unplugged example)

That will throw an error, application can remove this error and swap.

> * v4l2 device driver crashes and stops responding (buggy hardware / overheating
> hardware)

This case might be unrecoverable, no luck, fix your driver. Per experience, this often lead to the lost of a CPU and all attached process becomes zomby.

> * random "temporary" kernel errors, which require to restart the capture after
> some "quiet time"

Will error out, same thing.

> * override device-provided fake frames by the user-defined strategy (some
> devices actually do it) by querying V4l2.VIDIOC_ENUMINPUT.status 

Should be application side, with may be property to help.

> 
> Open questions:
> * what about compressed streams ?

There is nothing you can do, unless you are allowed to swap with something that encode.

> * is this feature generic enough to integrate into videorate or is a separate
> element better ?

I don't think it's inline with videorate role in my opinion.
Comment 18 Nicolas Dufresne (ndufresne) 2015-03-02 13:58:28 UTC
I just fixed the bug title so we need what need to be

*** This bug has been marked as a duplicate of bug 745441 ***