After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 785712 - FEI: Implement VAEntryPointFEI based h264 encoder
FEI: Implement VAEntryPointFEI based h264 encoder
Status: RESOLVED FIXED
Product: GStreamer
Classification: Platform
Component: gstreamer-vaapi
git master
Other Linux
: Normal enhancement
: 1.13.1
Assigned To: GStreamer Maintainers
GStreamer Maintainers
Depends on:
Blocks: 784667
 
 
Reported: 2017-08-02 00:43 UTC by sreerenj
Modified: 2017-09-01 17:48 UTC
See Also:
GNOME target: ---
GNOME version: ---



Description sreerenj 2017-08-02 00:43:35 UTC
FEI(Flexible Encoding Infrastructure) is an extension to VA API which allows applications to have more control over different phases of encoding and trade off quality for speed with their on IPs.

More Details here: https://bugzilla.gnome.org/show_bug.cgi?id=784667

This bug report is to track the development of a new element "vaapih264feienc" which will be using VAEntryPointFEI for H264 encode.

Why a new element for FEI?:
 
FEI is useful in very advanced applications. I would say, it makes sense to use FEI, If we implement an encoder itself(not just the plugin integration) which is something we haven't experimented with GStreamer yet (I believe).

Example use cases: 
--write custom BitRateControl algorithms,
--tweak/alter the motion vector and macroblock prediction per macroblock basis for each frame encode.
--analyze the vme distortion and make more intelligent prediction decisions 

Adding all these into existing vaapih264enc will make the code too complicated and it could be difficult to debug.

Another point is that FEI is still under heavy development and only supported in Intel SKL platform for now. So better keep a separate code base for the new encoder and may be in future we can merge with vaapih264encoder(if needed).
Comment 1 sreerenj 2017-08-02 00:53:05 UTC
Here is the initial implementation which I would like to bring upstream (Only after the driver patch integration) 
https://cgit.freedesktop.org/~sree/gstreamer-vaapi/log/?h=fei-dev

This is more or less like a prototype because this is not exposing any public APIs for application to set n number of FEI attributes/input-buffers.

The element "vaapih264feienc" will bring 17 new properties which are not available in existing vaapih264enc (this is not including the heavy input buffers MV and MBCode):

  disable-fei         : Disable Flexible Encoding Infrasturcture
                        flags: readable, writable
                        Boolean. Default: false
  stats-out           : Enable stats out for fei
                        flags: readable, writable
                        Boolean. Default: true
  num-mvpredict-l0    : Indicate how many predictors should be used for l0,only valid if MVPredictor input enabled
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 3 Default: 0 
  num-mvpredict-l1    : Indicate how many predictors should be used for l1,only valid if MVPredictor input enabled
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 3 Default: 0 
  search-window       : Specify one of the predefined search path
                        flags: readable, writable
                        Enum "GstVaapiFeiH264SearchWindow" Default: 0, "none"
                           (0): none             - not use predefined search window
                           (1): tiny             - 4 SUs 24x24 window diamond search
                           (2): small            - 9 SUs 28x28 window diamond search
                           (3): diamond          - 16 SUs 48x40 window diamond search
                           (4): large diamond    - 32 SUs 48x40 window diamond search
                           (5): exhaustive       - 48 SUs 48x40 window full search
                           (6): horizon diamond  - 16 SUs 64x32 window diamond search
                           (7): horizon large diamond - 32 SUs 64x32 window diamond search
                           (8): horizon exhaustive - 48 SUs 64x32 window full search
  len-sp              : This value defines number of search units in search path
                        flags: readable, writable
                        Unsigned Integer. Range: 1 - 63 Default: 32 
  search-path         : Specify search path
                        flags: readable, writable
                        Enum "GstVaapiFeiH264SearchPath" Default: 0, "full"
                           (0): full             - full search path
                           (1): diamond          - diamond search path
  ref-width           : Width of search region in pixel, must be multiple of 4
                        flags: readable, writable
                        Unsigned Integer. Range: 4 - 64 Default: 32 
  ref-height          : Height of search region in pixel, must be multiple of 4
                        flags: readable, writable
                        Unsigned Integer. Range: 4 - 32 Default: 32 
  submbpart-mask      : defines the bit-mask for disabling sub mb partition
                        flags: readable, writable
                        Flags "GstVaapiFeiH264SubMbPartMask" Default: 0x00000000, "enable all"
                           (0x00000000): enable all       - enable all subpartitions
                           (0x00000002): 16x16 submb part disabled - 16x16 sub-macroblock disabled
                           (0x00000004): 16x8 submb part disabled - 2x(16x8) sub-macroblock within 16x16 disabled
                           (0x00000008): 8x16 submb part disabled - 2x(8x16) sub-macroblock within 16x16 disabled
                           (0x00000010): 8x8 submb part disabled - 1x(8x8) sub-partition for 4x(8x8) within 16x16 disabled
                           (0x00000020): 8x4 submb part disabled - 2x(8x4) sub-partition for 4x(8x8) within 16x16 disabled
                           (0x00000040): 4x8 submb part disabled - 2x(4x8) sub-partition for 4x(8x8) within 16x16 disabled
                           (0x00000080): 4x4 submb part disabled - 4x(4x4) sub-partition for 4x(8x8) within 16x16 disabled
  subpel-mode         : Sub pixel precision for motion estimation
                        flags: readable, writable
                        Enum "GstVaapiFeiH264SubPelMode" Default: 0, "integer"
                           (0): integer          - integer mode searching
                           (1): half             - half-pel mode searching
                           (3): quarter          - quarter-pel mode searching
  intrapart-mask      : Specifies which Luma Intra partition is enabled/disabled forintra mode decision
                        flags: readable, writable
                        Flags "GstVaapiFeiH264IntraPartMask" Default: 0x00000000, "enable all"
                           (0x00000000): enable all       - enable all intra mode
                           (0x00000001): intra16x16 disabled - luma_intra_16x16 disabled
                           (0x00000002): intra8x8 disabled - luma_intra_8x8 disabled
                           (0x00000004): intra4x4 disabled - luma_intra_4x4 disabled
  intra-sad           : Specifies distortion measure adjustments usedin the motion search SAD comparison for intra MB
                        flags: readable, writable
                        Enum "GstVaapiFeiH264SadMode" Default: 0, "none"
                           (0): none             - none transform adjusted
                           (2): haar             - Haar transform adjusted
  inter-sad           : Specifies distortion measure adjustments usedin the motion search SAD comparison for inter MB
                        flags: readable, writable
                        Enum "GstVaapiFeiH264SadMode" Default: 0, "none"
                           (0): none             - none transform adjusted
                           (2): haar             - Haar transform adjusted
  adaptive-search     : Enable adaptive search
                        flags: readable, writable
                        Boolean. Default: false
  multi-predL0        : Enable multi prediction for ref L0 list, when set neighbor MV will be usedas predictor, no neighbor MV will be used otherwise
                        flags: readable, writable
                        Boolean. Default: false
  multi-predL1        : Enable multi prediction for ref L1 list, when set neighbor MV will be usedas predictor, no neighbor MV will be used otherwise
                        flags: readable, writable
                        Boolean. Default: false
  fei-mode            : Functional mode of FEI Encoding
                        flags: readable, writable
                        Flags "GstVaapiFeiMode" Default: 0x00000004, "ENC_PAK"
                           (0x00000001): ENC              - ENC Mode
                           (0x00000002): PAK              - PAK Mode
                           (0x00000004): ENC_PAK          - ENC_PAK Mode
Comment 2 sreerenj 2017-08-02 01:12:29 UTC
Now the more important stuff: How an application can feed input buffers(not the rawdata buffers, but FEI input buffers) to this new element?

Two applications are implemented in gstreamer-vaapi/test directory:

1: test-fei-enc-out : showcasing how to output MV,MBCode and Distortion

2: test-fei-enc-in: showcasing how to feed MVPredictor, QP,  MbContorl buffers for VME operation

These applications are heavily based on the internal apis in GST-libs/gst/vaapi, so basically not useful for external standalone applications.

There is a new fei specific meta: GstVaapiFeiVideoMeta which can be used to share FEI  buffers between FEI/VAAPI elements (we need to add more fei elements in future eg: a preenc for doing analysis/get_mv/get_distortion on each raw frame).

Manin concern is how to provide per-frame input control to the application? Do we really need to provide such control? May be to add signals(half a dozen of signals) for feeding fei attributes in per frame basis?
Comment 3 Víctor Manuel Jáquez Leal 2017-08-15 16:00:12 UTC
I have been reviewing the code and it is quite messy. I would not like to merge it until I reviewed and fixed it. Otherwise it will be a maintenance nightmare.
Comment 4 sreerenj 2017-08-15 17:52:54 UTC
(In reply to Víctor Manuel Jáquez Leal from comment #3)
> I have been reviewing the code and it is quite messy. I would not like to
> merge it until I reviewed and fixed it. Otherwise it will be a maintenance
> nightmare.

HI Victor,
Thanks for reviewing.

Few of the things which could be helpful while reviewing:

1: It might be difficult to get the full picture of this code until we sort of the implementation of PreEnc functionality (I am working on this).

2: There is code redundancy (for eg: both ENC only and PAK only abstracts, some structures like GstVaapiFeiInfoToPakH264 etc).But this is based on required future development.
a) Stuff Custom BRC algorithm in the ENC+PAK mode
b) There is a useful transcoding scenario. We can extend the *decoder* to output MVs and a different form of MBCode so that there is a possibility to do second transcoding with out doing VME.
3) a pipeline like this vaapih264feienc fei-mode=ENC ! vaapih264feienc fei-mode=PAK

I would prefer an incremental approach, get the initial code base in (with low ranked element of course) then work on this to improve.


Let's talk in IRC :)
Comment 5 Víctor Manuel Jáquez Leal 2017-09-01 11:32:18 UTC
The original patches were pushed "as is" since the re-factor is taking more time than expected :(

In a second iteration the refactored code will be merged.


* 9f98a02a FEI: Add test applications to showcase fei use case
* 5750bd78 FEI: plugin: Add vaapih264feienc element
* a46ad6b5 FEI: plugin: Add virtual methods to base encode
* 0fdef0e3 FEI: plugin: Add fei specific video meta
* 7e0d5b93 FEI: libs: Add FEI encoder
* de2e4cd9 FEI: libs: Add fei codec objects to GstVaapiEncPicture
* 132eacde FEI: libs: Add fei codec objects in codedbufferproxy
* d5542fa8 FEI: libs: Add fei codec objects to surface proxy
* d0d9f5e2 FEI: Add codec objects for fei usecase
* 0b91ebeb FEI: libs: add H264 fei specific utility functions
* 94483de4 FEI: libs: Add virtual method for secondary context creation.
* ac1de3d3 FEI: libs: make sure the default context creation works as expected.
* 7d77cdd9 FEI: libs: Add FEI functional mode configuration
* 6bbe7992 FEI: libs: Add FEI Entrypoint mapping
* 1d287ef8 FEI: Add support for FEI conditional build
Comment 6 sreerenj 2017-09-01 17:48:08 UTC
Thanks for pushing those.
As we discussed before, let's keep the FEI as experimental until we get a clear picture of the whole infrastructure. We might need more refactoring once the PreEnc support reaches in master. Also, the HEVC FEI APIs are under review. We have to think about common FEI design(may be a common fei base class) which can incorporate with other codecs APIs.

Hopefully, we can have discussion at GstConf :)