GNOME Bugzilla – Bug 796921
Hardware accelerated {en,de}coder classification and probing support
Last modified: 2018-10-12 16:37:37 UTC
It would be great if an app can probe for hardware-accelerated {en,de}coding support for a given format before using encodebin and decodebin. Given a video and/or audio configuration (containing at least a mime-type and then audio/video specific fields such as bitrate, samplerate, dimensions, framerate), the API would probe the registry for supported encoders and/or decoders. I would need such API so as to support the media-capabilities[0] spec in WebKit. Servo would also be a possible consumer of such API. [0] https://wicg.github.io/media-capabilities/#mediacapabilitiesinfo-smooth
The idea I had back when I was actively working on Farstream is to do 2 things: 1. Add a "Hardware" string to the klass string in the elementclass, so we can know which ones are hardware backed Then the second step has 2 options: 2a. Then you need to make sure that setting the element to READY fails if the hardware is not present or not available. Then you can just iterate the registry for elements with Video/Decoder/Hardware (as an example) and set them to ready one by one.. 2b. The other option is to do what v4l2 has been doing and only make the elements show up if they actually exist and then we can skip the second step and only have a flag in the element's klass.
*** Bug 782741 has been marked as a duplicate of this bug. ***
Another think to keep in mind is that using VAAPI or OMX it doesn't necessary mean that the backend is using hardware accelerated functions. For example the old bellagio (I don't know if there's a replacement), which implemented OpenMAX IL with ffmpeg. There's nothing in VAAPI to forbids a ffmpeg-based backend. Nonetheless, for sake of simplification, we could just ignore this possible configuration.
(In reply to Víctor Manuel Jáquez Leal from comment #3) > Another think to keep in mind is that using VAAPI or OMX it doesn't > necessary mean that the backend is using hardware accelerated functions. For > example the old bellagio (I don't know if there's a replacement), which Yes. Tizonia, which has software component. > implemented OpenMAX IL with ffmpeg. There's nothing in VAAPI to forbids a > ffmpeg-based backend. > > Nonetheless, for sake of simplification, we could just ignore this possible > configuration. Agreed.
(In reply to Nicolas Dufresne (ndufresne) from comment #4) > (In reply to Víctor Manuel Jáquez Leal from comment #3) > > Another think to keep in mind is that using VAAPI or OMX it doesn't > > necessary mean that the backend is using hardware accelerated functions. For > > example the old bellagio (I don't know if there's a replacement), which > > Yes. Tizonia, which has software component. For those that use "generated" elements from a discovery or config (like omx), I think they can set the string appropriately if it becomes a problem.
For gst-omx it's easy, we have a configuration file.
Thanks for the comments, it's an interesting discussion :) I'm not very worried about the classification part at this point. What is less clear to me is to know if decoding/encoding will be smooth (1) for a given element and possibly how power efficient (2) the process will be. I'm not sure where to put the boundary between the application and the library. (1) smooth meaning: the decoder is able to decode at a pace that allows a smooth playback. For an encoder, pace that allows encoding frames at the same pace as they are sent to the encoder. For instance in WebKit we could possibly assume that if a hardware-decoder is found for a given media format then decoding will be smooth if zero-copy support is enabled. Regarding power-efficiency, the spec recommends to rely on the Battery-status spec implementation. Also worth a mention, some devices can have limited capabilities, for instance we can't assume a Raspberry Pi will smoothly decode 4K or even full-HD at >30fps.
Limits for "real time" capabilities are quite hard to achieve in a generic way as they are bound by multiple things: memory (max frame size/max dpb size), processing power (max macroblocks/s etc).. And it's hard to do it in a generic way, especially that as far as I know, none of the underlying APIs offer that information. My guess is that you want to be able to return them as RFC 4281/6381 codecs ? I think we could easily add the maximum level to the caps.. The problem is where to get that information? From some kind of configuration file? I guess the only benefit to putting it at the GStreamer level instead of being application specific is that you can share it between different applications?
(In reply to Olivier Crête from comment #8) > Limits for "real time" capabilities are quite hard to achieve in a generic > way as they are bound by multiple things: memory (max frame size/max dpb > size), processing power (max macroblocks/s etc).. And it's hard to do it in > a generic way, especially that as far as I know, none of the underlying APIs > offer that information. > > My guess is that you want to be able to return them as RFC 4281/6381 codecs ? Yes something like that. > I think we could easily add the maximum level to the caps.. The problem is > where to get that information? From some kind of configuration file? Probably yes, as you say the underlying APIs don't provide those informations. > I guess > the only benefit to putting it at the GStreamer level instead of being > application specific is that you can share it between different applications? Right, I was mentioning this because at least 2 applications would potentially be consuming such "probing" API. For now I think I'll leave this aside. Let's first have support for hardware-enabled classification in the elements.
Created attachment 373854 [details] [review] RFC patch for new factory type classifier
Review of attachment 373854 [details] [review]: ::: gst/gstelementfactory.h @@ +112,3 @@ * @GST_ELEMENT_FACTORY_TYPE_MEDIA_SUBTITLE: Elements handling subtitle media types * @GST_ELEMENT_FACTORY_TYPE_MEDIA_METADATA: Elements handling metadata media types + * @GST_ELEMENT_FACTORY_TYPE_MEDIA_HARDWARE: Elements interacting with hardware devices I wonder if we shouldn't add: "Elements interacting with hardware devices, you may need to put the in "READY" to test if the hardware is present in the system".
Yes, good idea!
Created attachment 373870 [details] [review] patch for new factory type classifier
Comment on attachment 373870 [details] [review] patch for new factory type classifier Looks good to me >+ * @GST_ELEMENT_FACTORY_TYPE_MEDIA_HARDWARE: Elements interacting with hardware devices, you may need to put the element in "READY" state to test if the hardware is present in the system Could you also add an " (Since: 1.16)" there in the end? (Won't be "officially" picked up, but still good to mention it).
Comment on attachment 373870 [details] [review] patch for new factory type classifier Thanks! This patch is pushed now: commit ca4b61c55562a4b74f241fe54cf1e5639a2aea25 (HEAD -> master, origin/master, origin/HEAD) Author: Philippe Normand <philn@igalia.com> Date: Fri Oct 5 12:19:46 2018 +0200 gstelementfactory: Add MEDIA_HARDWARE klass classifier The Harware factory type classifier allows elements (decoders and encoders, mostly) to advertize they rely on hardware devices to perform encoding or decoding operations. This classifier can be used by applications to filter and select only the elements that use hardware devices, for instance to ensure zero-copy support is enabled for a specific pipeline. https://bugzilla.gnome.org/show_bug.cgi?id=796921
I started adding the new classifier in gst-vaapi and in the v4l2 plugins. Should I leave this bug open and attach new patches here or create other bugzilla entries?
> I started adding the new classifier in gst-vaapi and in the v4l2 plugins. > Should I leave this bug open and attach new patches here or create other > bugzilla entries? Is that something that needs reviewing? I think you can just push it?
(In reply to Tim-Philipp Müller from comment #17) > > I started adding the new classifier in gst-vaapi and in the v4l2 plugins. > > Should I leave this bug open and attach new patches here or create other > > bugzilla entries? > > Is that something that needs reviewing? I think you can just push it? Until now, those are mechanical and boring changes indeed :) Let's close this then and I'll follow-up in separate bugs for non-trivial changes.
Created attachment 373911 [details] [review] follow-up
Comment on attachment 373911 [details] [review] follow-up Worksforme.
Review of attachment 373911 [details] [review]: otherwise lgtm ::: gst/gstelementfactory.h @@ -144,3 @@ #define GST_ELEMENT_FACTORY_TYPE_MEDIA_SUBTITLE ((GstElementFactoryListType)(G_GUINT64_CONSTANT (1) << 52)) #define GST_ELEMENT_FACTORY_TYPE_MEDIA_METADATA ((GstElementFactoryListType)(G_GUINT64_CONSTANT (1) << 53)) -#define GST_ELEMENT_FACTORY_TYPE_MEDIA_HARDWARE ((GstElementFactoryListType)(G_GUINT64_CONSTANT (1) << 54)) Remove it from `docs/gst/gstreamer-sections.txt`.
Right, also adding GST_ELEMENT_FACTORY_KLASS_HARDWARE there.
Created attachment 373912 [details] [review] follow-up
Review of attachment 373912 [details] [review]: lgtm
commit a2832689adce1e12a30ddba9abefdfa88c692e65 (HEAD -> master, origin/master, origin/HEAD) Author: Philippe Normand <philn@igalia.com> Date: Fri Oct 12 15:34:45 2018 +0100 gstelementfactory: Remove MEDIA_HARDWARE FactoryType Using the MEDIA_ classifier prefix was inappropriate. It is sufficient to specify the additional klass name that element can set in their metadata. (follow-up of commit ca4b61c55562a4b74f241fe54cf1e5639a2aea25) https://bugzilla.gnome.org/show_bug.cgi?id=796921