Created attachment 268978 [details] [review]
MVC profiles definition

Created attachment 268979 [details] [review]
Isolate the bitstream writer from encoder

Created attachment 268980 [details]
Support subset sps bitstream writing

Created attachment 268983 [details] [review]
Wrap the ref and reorder frame list management

Created attachment 268987 [details] [review]
 Encode two views and generate MVC bitstream

Created attachment 269669 [details] [review]
Add MVC profiles mapping to driver profiles

Created attachment 269671 [details] [review]
Add MVC subset sps bit stream writting

Created attachment 269672 [details] [review]
Wrap the ref and reorder frame list management

Created attachment 269673 [details] [review]
MVC enc: Encode two views and generate MVC bitstream

Review of attachment 269669 [details] [review]:

Would be good to add a VA_VERSION check to maintain the backward compatibility.
But yes, the mvc-patches are still hanging in libva-staging branch. I think Gwenole can add it later once libva has mvc support in master.

Review of attachment 269671 [details] [review]:

Patch seems to be fine for me other than missing mvc_vui_extension() from the subset sps.

Review of attachment 269672 [details] [review]:

Seems to be fine for me

Review of attachment 269673 [details] [review]:

I have noticed a couple of issues:

-- enabling max-bframes >=1 causing the pipeline to hang in all occurrences.
-- have to store the subsetSPS also, which we need to generate the codec_data.
-- minor typo : warning message at the end of add_packed_mvc_sequence_header() should be GST_WARNING ("failed to write Subset SPS NAL unit");

@Gwenole: 

It seems that the indentation has messed-up in the gstvaapiencoder_h264.c. It has a mess-up of upstream gstreamer indentation and gstreamer-vaapi indentation :)
May be it is the time to apply gst-indent on all gstreamer-vaapi files :)

Created attachment 273871 [details] [review]
encoder: h264: Fix the pipeline hang while doing mvc encoding with b-frames

Created attachment 273978 [details] [review]
encoder: h264: store subset sps to generate the codec-data

Store the SubsetSPS nal unit which we need for MVC specific codec_data generation.

Created attachment 273979 [details] [review]
add MVCDecoderConfigurationRecord as codec_data

This patch is in "not for commit" state at the moment. We need to confirm few other things before enabling this. I am just submitting this patch to show the exact format of mvc_codec_data other than the normal avc_codec_data.

--Do we really need to provide codec_data separately for base_view and sub_view here? Then we need some major changes in gst-vaapi plugin implementation. Because the get_codec_data() won't get invoked second time by encoder plugins unless the input buffer format get changed.

--Do we just need to provide one MVCDecoderConfigurationRecord(mvc_codec_data) instead of avc_codec_data for the whole stream? Then also we need some major changes with in vaapiencoder_h264 to generate and store the  subset sps together with baseview. Because at the moment we are generating subset_sps only when view_idx changes to 1, which means for the non-baseview. So the first invocation of get_codec_data() (which get invoked before pushing first encoded base_view frame) can't generate the mvc-codec-data.

I have not formally checked the MP4 spec. I believe the records are different, so this probably means to keep codec_data in different buffers? I think we actually need to check the various muxers (mkv, ts, mp4) and how likely/easy it would be to extend them to support the MVC records. On the other hand, it's common to look for "codec-data" only.

What about handling "codec-data" as a buffer list instead of a unique buffer? Though, the difficulty here would be a way to identify AVC vs. MVC vs. anything else.

(In reply to comment #18)
As per the ISO Base file spec, we are supposed to provide two configuration records since we have both avc compatible base view and mvc specific non-base-view. 

As per mp4 spec:
If an MVC elementary stream contains a usable AVC compatible base view, then an AVC visual sample entry(‘avc1’ or ‘avc2’) shall be used. Here, the entry shall contain initially an AVC Configuration Box, possibly followed by an MVC Configuration Box as defined below. The AVC Configuration Box documents the Profile, Level and Parameter Set information pertaining to the AVC compatible base view as defined by the AVCDecoderConfigurationRecord. The MVC Configuration Box documents the Profile, Level and Parameter Set information pertaining to the entire stream containing the non-base views as defined by the MVCDecoderConfigurationRecord, stored in the MVCConfigurationBox.

A buffer-list might be the right option. Would be good to add some of the upstream guys also to the cc list.

commit 73355a680f87d9cc56e7c165fb4e1904c5feb166
Author: Sreerenj Balachandran <sreerenj.balachandran@intel.com>
Date:   Wed May 7 00:09:19 2014 +0300

    encoder: h264: store subset sps to generate the codec-data
    
    Store the SubsetSPS nal unit which we need for MVC specific
    codec_data generation.

commit f0924ba0d1839405f8a5dfdda6ff7e29379a20ec
Author: Sreerenj Balachandran <sreerenj.balachandran@intel.com>
Date:   Wed May 7 00:08:33 2014 +0300

    encoder: h264: fix MVC pipeline hang while encoding with B-frames.
    
    Since we are encoding each view independently from each other, we
    need a higher number of pre-allocated surfaces to be used as the
    reconstructed frames. For Stereo High profile encoding, this means
    to effectively double the number of frames to be stored in the DPB.

commit 7bdf3fa177dffa5be4c5dd079aace0c5368d5277
Author: Li Xiaowei <xiaowei.a.li@intel.com>
Date:   Mon Feb 17 15:51:43 2014 +0800

    encoder: h264: add initial support for H.264 Stereo High profile.
    
    Add initial support for Subset SPS, Prefix NAL and Slice Extension NAL
    for non-base-view streams encoding, and the usual SPS, PPS and Slice
    NALs for base-view encoding.
    
    The H.264 Stereo High profile encoding mode will be turned on when the
    "num-views" parameter is set to 2. The source (raw) YUV frames will be
    considered as Left/Right view, alternatively.
    
    Each of the two views has its own frames reordering pool and reference
    frames list management system. Inter-view references are not supported
    yet, so the views are encoded independently from each other.
    
    Signed-off-by: Li Xiaowei <xiaowei.a.li@intel.com>
    [limited to Stereo High profile per the definition of MAX_NUM_VIEWS]
    Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>

commit 5323570a353425930596755dbaf20af6b063e9a7
Author: Li Xiaowei <xiaowei.a.li@intel.com>
Date:   Mon Feb 17 11:10:26 2014 +0800

    encoder: h264: wrap pools for refs and frames reordering.
    
    Create structures to maintain the reference frames list (RefPool) and
    frames reordering (ReorderPool) logic.
    
    This is a prerequisite for H.264 MVC support.
    
    Signed-off-by: Li Xiaowei <xiaowei.a.li@intel.com>

commit 4beee390e5a1bbf04f4bfe710ef044dd1e0c52e7
Author: Li Xiaowei <xiaowei.a.li@intel.com>
Date:   Fri Feb 14 15:33:15 2014 +0800

    encoder: h264: add provisional support for subset SPS headers.
    
    Add provisions to write subset SPS headers to the bitstream in view
    to supporting the H.264 MVC specification.
    
    This assumes the libva "staging" branch is in use.
    
    Signed-off-by: Li Xiaowei <xiaowei.a.li@intel.com>

commit fb7ecb15e27a89525569503d627478f9d90df561
Author: Li Xiaowei <xiaowei.a.li@intel.com>
Date:   Wed Dec 18 13:47:32 2013 +0800

    utils: add H.264 MVC profiles.
    
    Add "MultiView High" and "Stereo High" definitions.
    
    Signed-off-by: Li Xiaowei <xiaowei.a.li@intel.com>
    [require VA-API >= 0.35.2 for MVC profiles]
    Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>