GNOME Bugzilla – Bug 785712
FEI: Implement VAEntryPointFEI based h264 encoder
Last modified: 2017-09-01 17:48:08 UTC
FEI(Flexible Encoding Infrastructure) is an extension to VA API which allows applications to have more control over different phases of encoding and trade off quality for speed with their on IPs. More Details here: https://bugzilla.gnome.org/show_bug.cgi?id=784667 This bug report is to track the development of a new element "vaapih264feienc" which will be using VAEntryPointFEI for H264 encode. Why a new element for FEI?: FEI is useful in very advanced applications. I would say, it makes sense to use FEI, If we implement an encoder itself(not just the plugin integration) which is something we haven't experimented with GStreamer yet (I believe). Example use cases: --write custom BitRateControl algorithms, --tweak/alter the motion vector and macroblock prediction per macroblock basis for each frame encode. --analyze the vme distortion and make more intelligent prediction decisions Adding all these into existing vaapih264enc will make the code too complicated and it could be difficult to debug. Another point is that FEI is still under heavy development and only supported in Intel SKL platform for now. So better keep a separate code base for the new encoder and may be in future we can merge with vaapih264encoder(if needed).
Here is the initial implementation which I would like to bring upstream (Only after the driver patch integration) https://cgit.freedesktop.org/~sree/gstreamer-vaapi/log/?h=fei-dev This is more or less like a prototype because this is not exposing any public APIs for application to set n number of FEI attributes/input-buffers. The element "vaapih264feienc" will bring 17 new properties which are not available in existing vaapih264enc (this is not including the heavy input buffers MV and MBCode): disable-fei : Disable Flexible Encoding Infrasturcture flags: readable, writable Boolean. Default: false stats-out : Enable stats out for fei flags: readable, writable Boolean. Default: true num-mvpredict-l0 : Indicate how many predictors should be used for l0,only valid if MVPredictor input enabled flags: readable, writable Unsigned Integer. Range: 0 - 3 Default: 0 num-mvpredict-l1 : Indicate how many predictors should be used for l1,only valid if MVPredictor input enabled flags: readable, writable Unsigned Integer. Range: 0 - 3 Default: 0 search-window : Specify one of the predefined search path flags: readable, writable Enum "GstVaapiFeiH264SearchWindow" Default: 0, "none" (0): none - not use predefined search window (1): tiny - 4 SUs 24x24 window diamond search (2): small - 9 SUs 28x28 window diamond search (3): diamond - 16 SUs 48x40 window diamond search (4): large diamond - 32 SUs 48x40 window diamond search (5): exhaustive - 48 SUs 48x40 window full search (6): horizon diamond - 16 SUs 64x32 window diamond search (7): horizon large diamond - 32 SUs 64x32 window diamond search (8): horizon exhaustive - 48 SUs 64x32 window full search len-sp : This value defines number of search units in search path flags: readable, writable Unsigned Integer. Range: 1 - 63 Default: 32 search-path : Specify search path flags: readable, writable Enum "GstVaapiFeiH264SearchPath" Default: 0, "full" (0): full - full search path (1): diamond - diamond search path ref-width : Width of search region in pixel, must be multiple of 4 flags: readable, writable Unsigned Integer. Range: 4 - 64 Default: 32 ref-height : Height of search region in pixel, must be multiple of 4 flags: readable, writable Unsigned Integer. Range: 4 - 32 Default: 32 submbpart-mask : defines the bit-mask for disabling sub mb partition flags: readable, writable Flags "GstVaapiFeiH264SubMbPartMask" Default: 0x00000000, "enable all" (0x00000000): enable all - enable all subpartitions (0x00000002): 16x16 submb part disabled - 16x16 sub-macroblock disabled (0x00000004): 16x8 submb part disabled - 2x(16x8) sub-macroblock within 16x16 disabled (0x00000008): 8x16 submb part disabled - 2x(8x16) sub-macroblock within 16x16 disabled (0x00000010): 8x8 submb part disabled - 1x(8x8) sub-partition for 4x(8x8) within 16x16 disabled (0x00000020): 8x4 submb part disabled - 2x(8x4) sub-partition for 4x(8x8) within 16x16 disabled (0x00000040): 4x8 submb part disabled - 2x(4x8) sub-partition for 4x(8x8) within 16x16 disabled (0x00000080): 4x4 submb part disabled - 4x(4x4) sub-partition for 4x(8x8) within 16x16 disabled subpel-mode : Sub pixel precision for motion estimation flags: readable, writable Enum "GstVaapiFeiH264SubPelMode" Default: 0, "integer" (0): integer - integer mode searching (1): half - half-pel mode searching (3): quarter - quarter-pel mode searching intrapart-mask : Specifies which Luma Intra partition is enabled/disabled forintra mode decision flags: readable, writable Flags "GstVaapiFeiH264IntraPartMask" Default: 0x00000000, "enable all" (0x00000000): enable all - enable all intra mode (0x00000001): intra16x16 disabled - luma_intra_16x16 disabled (0x00000002): intra8x8 disabled - luma_intra_8x8 disabled (0x00000004): intra4x4 disabled - luma_intra_4x4 disabled intra-sad : Specifies distortion measure adjustments usedin the motion search SAD comparison for intra MB flags: readable, writable Enum "GstVaapiFeiH264SadMode" Default: 0, "none" (0): none - none transform adjusted (2): haar - Haar transform adjusted inter-sad : Specifies distortion measure adjustments usedin the motion search SAD comparison for inter MB flags: readable, writable Enum "GstVaapiFeiH264SadMode" Default: 0, "none" (0): none - none transform adjusted (2): haar - Haar transform adjusted adaptive-search : Enable adaptive search flags: readable, writable Boolean. Default: false multi-predL0 : Enable multi prediction for ref L0 list, when set neighbor MV will be usedas predictor, no neighbor MV will be used otherwise flags: readable, writable Boolean. Default: false multi-predL1 : Enable multi prediction for ref L1 list, when set neighbor MV will be usedas predictor, no neighbor MV will be used otherwise flags: readable, writable Boolean. Default: false fei-mode : Functional mode of FEI Encoding flags: readable, writable Flags "GstVaapiFeiMode" Default: 0x00000004, "ENC_PAK" (0x00000001): ENC - ENC Mode (0x00000002): PAK - PAK Mode (0x00000004): ENC_PAK - ENC_PAK Mode
Now the more important stuff: How an application can feed input buffers(not the rawdata buffers, but FEI input buffers) to this new element? Two applications are implemented in gstreamer-vaapi/test directory: 1: test-fei-enc-out : showcasing how to output MV,MBCode and Distortion 2: test-fei-enc-in: showcasing how to feed MVPredictor, QP, MbContorl buffers for VME operation These applications are heavily based on the internal apis in GST-libs/gst/vaapi, so basically not useful for external standalone applications. There is a new fei specific meta: GstVaapiFeiVideoMeta which can be used to share FEI buffers between FEI/VAAPI elements (we need to add more fei elements in future eg: a preenc for doing analysis/get_mv/get_distortion on each raw frame). Manin concern is how to provide per-frame input control to the application? Do we really need to provide such control? May be to add signals(half a dozen of signals) for feeding fei attributes in per frame basis?
I have been reviewing the code and it is quite messy. I would not like to merge it until I reviewed and fixed it. Otherwise it will be a maintenance nightmare.
(In reply to Víctor Manuel Jáquez Leal from comment #3) > I have been reviewing the code and it is quite messy. I would not like to > merge it until I reviewed and fixed it. Otherwise it will be a maintenance > nightmare. HI Victor, Thanks for reviewing. Few of the things which could be helpful while reviewing: 1: It might be difficult to get the full picture of this code until we sort of the implementation of PreEnc functionality (I am working on this). 2: There is code redundancy (for eg: both ENC only and PAK only abstracts, some structures like GstVaapiFeiInfoToPakH264 etc).But this is based on required future development. a) Stuff Custom BRC algorithm in the ENC+PAK mode b) There is a useful transcoding scenario. We can extend the *decoder* to output MVs and a different form of MBCode so that there is a possibility to do second transcoding with out doing VME. 3) a pipeline like this vaapih264feienc fei-mode=ENC ! vaapih264feienc fei-mode=PAK I would prefer an incremental approach, get the initial code base in (with low ranked element of course) then work on this to improve. Let's talk in IRC :)
The original patches were pushed "as is" since the re-factor is taking more time than expected :( In a second iteration the refactored code will be merged. * 9f98a02a FEI: Add test applications to showcase fei use case * 5750bd78 FEI: plugin: Add vaapih264feienc element * a46ad6b5 FEI: plugin: Add virtual methods to base encode * 0fdef0e3 FEI: plugin: Add fei specific video meta * 7e0d5b93 FEI: libs: Add FEI encoder * de2e4cd9 FEI: libs: Add fei codec objects to GstVaapiEncPicture * 132eacde FEI: libs: Add fei codec objects in codedbufferproxy * d5542fa8 FEI: libs: Add fei codec objects to surface proxy * d0d9f5e2 FEI: Add codec objects for fei usecase * 0b91ebeb FEI: libs: add H264 fei specific utility functions * 94483de4 FEI: libs: Add virtual method for secondary context creation. * ac1de3d3 FEI: libs: make sure the default context creation works as expected. * 7d77cdd9 FEI: libs: Add FEI functional mode configuration * 6bbe7992 FEI: libs: Add FEI Entrypoint mapping * 1d287ef8 FEI: Add support for FEI conditional build
Thanks for pushing those. As we discussed before, let's keep the FEI as experimental until we get a clear picture of the whole infrastructure. We might need more refactoring once the PreEnc support reaches in master. Also, the HEVC FEI APIs are under review. We have to think about common FEI design(may be a common fei base class) which can incorporate with other codecs APIs. Hopefully, we can have discussion at GstConf :)