Currently, libva only support Y210 pixel format(422,10bit), need support this format in gstreamer side. I am cooking a patch to support this.

Created attachment 374021 [details] [review]
patch

This patch depends on a patch on gst-plugin-base which add support for format Y210.
https://bugzilla.gnome.org/show_bug.cgi?id=797267

@Victor, could you help to review both of them if you have the authority? Thanks.

And here is still a know issue that will cause GPU hang when decode main-422 10bit steam. This should be the media-driver's issue, which also happen when decode main-444 streams. I will report it to media-driver after this patch merged.

Created attachment 374024 [details] [review]
patch

@Fei,

I'll try to look at them this weekend, during the hackfest in Edinburgh.

(In reply to Víctor Manuel Jáquez Leal from comment #5)
> @Fei,
> 
> I'll try to look at them this weekend, during the hackfest in Edinburgh.

No problem, thanks Victor.

@Victor, any comments? :)

Review of attachment 374024 [details] [review]:

overall it looks good, just a couple library version guards.

Also, Nicolas already merged the color space definition patches.

::: gst-libs/gst/vaapi/gstvaapisurface.h
@@ +82,3 @@
   GST_VAAPI_CHROMA_TYPE_RGB16,
+  GST_VAAPI_CHROMA_TYPE_YUV420_10BPP,
+  GST_VAAPI_CHROMA_TYPE_YUV422_10BPP

nitpick: add a comma at the end, thus if a new chroma type is added, wouldn't be need to modify the previous line as in this case.

::: gst-libs/gst/vaapi/gstvaapiutils.c
@@ +403,3 @@
       break;
+    case VA_RT_FORMAT_YUV422_10:
+      chroma_type = GST_VAAPI_CHROMA_TYPE_YUV422_10BPP;

VA_RT_FORMAT_YUV422_10 was defined in VA version of 1.4 so you must guard this code around a 

#if VA_CHECK_VERSION(1,4,0)

@@ +454,3 @@
       break;
+    case GST_VAAPI_CHROMA_TYPE_YUV422_10BPP:
+      format = VA_RT_FORMAT_YUV422_10;

same here

@Fei,

I don't have hardware where to try this change so I'll have to trust in your testing :/

And thanks for the mug! :)

Created attachment 374115 [details] [review]
patch

(In reply to Víctor Manuel Jáquez Leal from comment #8)
> Review of attachment 374024 [details] [review] [review]:
> 
> overall it looks good, just a couple library version guards.
> 
> Also, Nicolas already merged the color space definition patches.
> 
> ::: gst-libs/gst/vaapi/gstvaapisurface.h
> @@ +82,3 @@
>    GST_VAAPI_CHROMA_TYPE_RGB16,
> +  GST_VAAPI_CHROMA_TYPE_YUV420_10BPP,
> +  GST_VAAPI_CHROMA_TYPE_YUV422_10BPP
> 
> nitpick: add a comma at the end, thus if a new chroma type is added,
> wouldn't be need to modify the previous line as in this case.
> 
> ::: gst-libs/gst/vaapi/gstvaapiutils.c
> @@ +403,3 @@
>        break;
> +    case VA_RT_FORMAT_YUV422_10:
> +      chroma_type = GST_VAAPI_CHROMA_TYPE_YUV422_10BPP;
> 
> VA_RT_FORMAT_YUV422_10 was defined in VA version of 1.4 so you must guard
> this code around a 
> 
> #if VA_CHECK_VERSION(1,4,0)
> 
> @@ +454,3 @@
>        break;
> +    case GST_VAAPI_CHROMA_TYPE_YUV422_10BPP:
> +      format = VA_RT_FORMAT_YUV422_10;
> 
> same here

Thanks Victor for your reviewing. I checked libva, VA_RT_FORMAT_YUV422_10 actually was defined in version 1.2.0. :)

@fei

Do you have a stream with this format?

Just to check the bailout if the hardware doesn't support it, also when libva is older than 1.2.0

Created attachment 374130 [details]
streams

(In reply to Víctor Manuel Jáquez Leal from comment #12)
> @fei
> 
> Do you have a stream with this format?
> 
> Just to check the bailout if the hardware doesn't support it, also when
> libva is older than 1.2.0

Hi Victor, upload to attachment. And the streams come from ITU:
https://www.itu.int/net/itu-t/sigdb/menu.aspx

pushe attachment 374115 [details] [review] as commit 63800487 - libs: dec: h265: support decode for main-10-422 10bit streams.