GNOME Bugzilla – Bug 621289
[RFC] Dynamic element factories
Last modified: 2016-05-25 22:46:13 UTC
The v4l2 media controller kernel interface is quite dynamic. It allows to query the hardware topology. If we map that to gstreamer we can register proxy-elements for each camera component (sensor-src, scaler, filters). The problem here is that one needs to open the media controller device and enumerate it - it is not static. It is kind of similar to the plugin dependency mechanism, but we can't just cehck this from the file-system. It would be ugly if we need udev scripts to update some topology dump in the file-system when the hardware changes, so the the plugin dependency mechanism could act on it. Any better idea?
Are there any docs for that interface? I don't think we want to support dynamic factories, because that'd add yet another thing autopluggers need to be aware of. Isn't the current method of using NULL=>READY or caps negotiation sufficient?
Some links: http://www.linuxtv.org/downloads/plumbers2008/hverkuil_v4l2.pdf http://lwn.net/Articles/352622/ benjamin, how does it relate to the state-change or caps? Lets assume you have /dev/mc0. You open that, enumerate the hardware and you learn that the device is called HyperCam and has a Sensor, a Scaler plus 5 postprocessing effects. You can queury formats, linking constraints etc. Now if /dev/mc0 is provided by a usb device it disappear as you unplug it. If you plug three of them you get /dev/mc0, /dev/mc1 and /dev/mc2 - all of them possibly with different capabilities. So if we have a gstv4l2mc plugin, How and when would that register element-factories to the registry? The gst_plugin_add_dependency() dependency mechanism can probably be used for tracking /dev/mcX. It means that plugin_init would open all of them and register all factories. It needs to be robust enough to handle those disappearing anyway. If a new one is plugged at runtime of an app, it won't be recognised (maybe the app will have to watch /dev/mc and run gst_update_registry()). In this case gstv4l2mc_plugin_init needs to be careful to not re-register existing GTypes ...
Aren't element factories registered unconditionally? I mean, I get an alsasink no matter if I do have a sound card or not and esdsink no matter if esd is running or even installed. Element factories enumerate the features of installed plugins, they are in no way related to the provided features of the hardware. And I don't get why that suddenly needs to change for v4l.
I'm not convinced it's a good idea to use a pipeline processing model to describe a system that doesn't fully support pipeline processing. I think it will cause no end in confusion and impedence mismatch. And, as mentioned in the pdf, V4L doesn't have a history of clean drivers that all work the same. As Company said, if we go this way, we should have a bunch of v4l-related element factories. That way, when they are connected together in a certain way, they can automatically do the right thing in hardware. Sort of like the video/x-cairo stuff.
Yes, we can easily register a v4l2mcsrc and ~sink. But I am not convinced about having a v4l2mctranform eleemnt that can be a scaler, red-eye-reducation, denoiser, depending on what device node you pass to the device-property. Problems I see are: - GstElemntDetails - you want different description and classification - GObject properties - how to control the setting of the specific element?
Then have separate v4lmcscaler, v4lmcredeyereduction, v4lmcdenoiser elements.
Yeah, this looks like a problem that we solved with ffmpeg, LADSPA or other plugin frameworks already and where we should look at those solutions for inspiration instead of inventing new APIs.
Stefan: so what problem are you trying to solve here exactly? Making sure that the plugin is re-scanned/inited/updated at startup when the hardware topology changes, because the mechanisms used for ffmpeg/ladspa etc. (ie. re-read if certain directories or files change) don't apply here?
Tim, it is probably the only thing we can do right now. Unfortunately it would mean that at every startup of a gstreamer app, we would iterate over media-controller nodes and rescan them :/ So altogether I was not sure if people would be happy with plugins that interact with hardware devices at plugin_init time. So if everyone is happy with that, then I'll try it that way.
Well, if the choice is between doing this every gst_init() or doing this once in plugin_init() when the plugin is scanned, then I'd vote for the latter. I'm still not really sure if these things need to be exposed at the element level though, as something that's part of the pipeline topology. It seems quite v4l2 specific, and I wonder if it wouldn't be sufficient to expose interesting functionality via interfaces or so (apply this effect, apply that effect, etc.).
you should not start probing hardware in the plugin_init IMO. Better do it like we do everywhere else: unconditionally register elements with a defined API that may or may not work depending on the hardware that is available.
(In reply to comment #11) > you should not start probing hardware in the plugin_init IMO. Better do it like > we do everywhere else: unconditionally register elements with a defined API > that may or may not work depending on the hardware that is available. Then we need to keep this open until we have a better idea. You cannot register such elements, as there is no defined list of possible ones. The hardware the v4l media controller API targets is a combination of capture hardware + a image signal processor. Such ISP have loadable firware images (those might be static and/or monotitic). Features realised on the ISP can be almost anything. In practise the theoreticaly available feature don't change often. Thus such a plugin could localy cache and aggregate the features. plugin_init would only rescan if it finds new nodes. Having features registererd to gstreamer and the device beeing unplugged can be gracefully handled. The local cache would probably need an expiration policy to handle hardware that is never replugged. Still this would mean probing hardware in plugin_init (like we probe ladspa plugins in plugin_init).
I'm a bit concerned as to exposing that level of details in GStreamer because no dataflow will happen between such elements (it's all tunneled). It would make sense though to have the "final" devices supported, and maybe expose configuration via those devices (ex: a v4l2 output which has media controller capabilities and on which you can enable/disable filtering capabilities in the underlying MC device configuration).
Nicolas exposed something similar for the v4l2*dec by scanning for /dev/video* devices on the first start and binding those devices to the gstreamer registry. Which causes the scanning to then happen once per boot at most for this plugin. And the MediaController devices then just operate like regular v4l2 devices. It seems there is not much apetite to expose all of the MC API through GStreamer.