Title changed. Other video codecs like MPEG-4 and VP8 fail too. Only audio files play.

As you can see, your hardware only support MPEG2 decoding. For other codecs you should use software decoders as in gst-libav.

can you upload a debug log with GST_DEBUG=vaapi*:5 ??

what happens with this command:

gst-play-1.0 test.mkv --videosink=clutterautovideosink ??

(In reply to Víctor Manuel Jáquez Leal from comment #2)
> As you can see, your hardware only support MPEG2 decoding. For other codecs
> you should use software decoders as in gst-libav.

gst-play-1.0 also fails to play other codecs, but Totem plays them fine. Tested with MPEG-4 SP and VP8. Weird.

> 
> can you upload a debug log with GST_DEBUG=vaapi*:5 ??

Will do.

> 
> what happens with this command:
> 
> gst-play-1.0 test.mkv --videosink=clutterautovideosink ??

$ gst-play-1.0 test.mkv --videosink=clutterautovideosink

** (gst-play-1.0:1200): WARNING **: Couldn't create specified video sink 'clutterautovideosink'
Press 'k' to see a list of keyboard shortcuts.
Now playing /home/marcos/Downloads/test.mkv
libva info: VA-API version 0.39.4
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
libva info: VA-API version 0.39.4
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
Redistribute latency...
Redistribute latency...
wl_surface@7: error 2: Failed to create a texture for surface 7
ERROR Internal error: could not render surface for file:///home/marcos/Downloads/test.mkv
ERROR debug information: gstvaapisink.c(1457): gst_vaapisink_show_frame_unlocked (): /GstPlayBin:playbin/GstPlaySink:playsink/GstBin:vbin/GstVaapiSink:vaapisink0
Reached end of play list.

Created attachment 341482 [details]
gst-play-1.0 debug log

GST_DEBUG_NO_COLOR=1 GST_DEBUG=vaapi*:5 gst-play-1.0 test.mkv

Ok, you don't have installed clutterautovideosink. Totem should be getting other cluttersink.

But I confirmed, but I don't know where is the problem: either in vaapisink or in mutter, which fails to upload the surface into a texture.

*** Bug 783172 has been marked as a duplicate of this bug. ***

The problem is that mutter, in wayland, only imports EGL textures with color formats RGB and RGBA, but vaapisink asks to import a EGL texture with EGL_TEXTURE_Y_UV_WL.

As far as I understand, mutter cannot import these textures because COGL cannot handle them either.

In the case of weston wayland compositor, there is not problems with vaapisink.

My ideal solution would be to add support for this textures in cogl and mutter.

What we can do internally to gstreamer-vaapi? 

1\ reduce the rank of vaapisink
2\ figure out a way to internally convert the NV12 image into RGB.

(In reply to Víctor Manuel Jáquez Leal from comment #7)
> The problem is that mutter, in wayland, only imports EGL textures with color
> formats RGB and RGBA, but vaapisink asks to import a EGL texture with
> EGL_TEXTURE_Y_UV_WL.
> 
> As far as I understand, mutter cannot import these textures because COGL
> cannot handle them either.

That is not entirely correct. Weston synthesize this format using a shader. In the end, you have two textures, Y as an R8, and UV as a GR88 and a shader is used. The same action is done by cluttersink, which is using clutter/COGL, so there is no reason why you can't implement that directly inside gnome-shell/mutter. Plus, the code already exist.

Let's keep this bug about mutter only (or more generally when using waylandsink with NV12 videos in compositors that don't support NV12). 

Totem's VAAPI problems in Wayland are covered by bug 783169 and bug 784369.

By the way, this will work with VAAPI under Gnome on Wayland:
$ env GST_GL_WINDOW=wayland gst-play-1.0 --videosink glimagesink

The first part works around the gl/vaapi plugin bug 783169. The second part works around this bug 775698 as well as bug 784369. :)

Hm, anyone upgrading to latest Ubuntu will hit this issue!

It is breaking all gst-launch pipelines in Ubuntu 17.10 since it using gnome-shell/wayland as default now.

I am raising the severity.

@Victor: I don't think internal conversion of render_surface to RGB is going to help in this case because we need to get the Wayland buffer anyway and unfortunately vaGetSurfaceBufferWl() doesn't have RGB support.

I think the issue with gst-play/gst-launch in Ubuntu 17.10 is actually a regression in the existing DMA-buf support mentioned here:

https://bugzilla.gnome.org/show_bug.cgi?id=784369#c12

From Ubuntu's perspective we don't mind the regression in the short term because that's what it took to deliver working VAAPI support in Totem. And Totem using clutter-gst doesn't care about DMA-buf yet (bug 759209).

For general users the default video player (Totem) working efficiently in the default session (Wayland) is vastly more important than other use cases. But eventually we'd like to not compromise at all.

That issue doesn't have a bug for it yet so please do open one on Launchpad (because the offending patch and the problem theoretically only exists in Ubuntu right now).

(In reply to sreerenj from comment #11)
> Hm, anyone upgrading to latest Ubuntu will hit this issue!
> 
> It is breaking all gst-launch pipelines in Ubuntu 17.10 since it using
> gnome-shell/wayland as default now.
> 
> I am raising the severity.

Ubuntu now uses Wayland/Mutter by default.

And the "bug" is in Mutter. If you try Weston, it works.

But rather than bug it is a missing feature in mutter that needs to be developed.

(In reply to Víctor Manuel Jáquez Leal from comment #14)
> (In reply to sreerenj from comment #11)
> > Hm, anyone upgrading to latest Ubuntu will hit this issue!
> > 
> > It is breaking all gst-launch pipelines in Ubuntu 17.10 since it using
> > gnome-shell/wayland as default now.
> > 
> > I am raising the severity.
> 
> Ubuntu now uses Wayland/Mutter by default.
> 
> And the "bug" is in Mutter. If you try Weston, it works.
> 
> But rather than bug it is a missing feature in mutter that needs to be
> developed.

Yup, I did some investigation/testing too.

Wondering what we should do with this since it breaking most of the gst-launch pipelines. Adding BGRx support for vaGetSurfaceBufferWl() in intel-vaapi-driver and doing an internal conversion in vaapisink is the workaround I can think of (I don't like it though). Or degrade vaapisink rank and promote glimagesink usage which is of course what we need in future.

After looking at this, I'm leaning to demote vaapisink's rank, to avoid autoplug.
Though, I'm worried for non-wayland users, they might see performance degradation.

*** Bug 793376 has been marked as a duplicate of this bug. ***

> After looking at this, I'm leaning to demote vaapisink's rank, to avoid
> autoplug.

Let's do it?


> Though, I'm worried for non-wayland users, they might see performance
> degradation.

My understanding was vaapisink is primarily for debugging/testing purposes, and perhaps for specialised applications that know that they will be using VAAPI. No "normal" application will need vaapisink to be autoplugged, will it?

Better poor performance than doesn't work.

I'd like this to be fixed for 1.14, it's been showing up in distros, which looks really bad.

Created attachment 368273 [details] [review]
vaapi: register vaapisink as marginal on wayland

vaapsink, when used with the Intel VA-API driver, tries to display
surfaces with format NV12, which are handled correctly by
Weston. Nonetheless, COGL cannot display YUV surfaces, making fail
pipelines on mutter.

This shall be solved either by COGL or by making the driver to paint
RGB surfaces. In the meanwhile, let's just demote vaapisink as
marginal when the Wayland environment is detected, no matter if it is
Weston.

Review of attachment 368273 [details] [review]:

lgtm.

Is there any plan to move "glimagesink" as "primary rank" element?

(In reply to Tim-Philipp Müller from comment #18)
> My understanding was vaapisink is primarily for debugging/testing purposes,
> and perhaps for specialised applications that know that they will be using
> VAAPI. No "normal" application will need vaapisink to be autoplugged, will
> it?

Isn't VAAPI the only interface to get hardware-accelerated video on intel GPUs? My understanding is that VAAPI is required to use codec-specific hardware such as for H.264 or H.265/HEVC.

(In reply to Christian Stadelmann from comment #23)
> (In reply to Tim-Philipp Müller from comment #18)
> > My understanding was vaapisink is primarily for debugging/testing purposes,
> > and perhaps for specialised applications that know that they will be using
> > VAAPI. No "normal" application will need vaapisink to be autoplugged, will
> > it?
> 
> Isn't VAAPI the only interface to get hardware-accelerated video on intel
> GPUs? My understanding is that VAAPI is required to use codec-specific
> hardware such as for H.264 or H.265/HEVC.

Yes, but this is just about the display sink, which can be replaced by glimagesink.

(In reply to Nicolas Dufresne (stormer) from comment #24)
> (In reply to Christian Stadelmann from comment #23)
> > 
> > Isn't VAAPI the only interface to get hardware-accelerated video on intel
> > GPUs? My understanding is that VAAPI is required to use codec-specific
> > hardware such as for H.264 or H.265/HEVC.
> 
> Yes, but this is just about the display sink, which can be replaced by
> glimagesink.

Ok, that makes sense. Thanks!

(In reply to Tim-Philipp Müller from comment #18)
> > After looking at this, I'm leaning to demote vaapisink's rank, to avoid
> > autoplug.
> 
> Let's do it?
> 
> 
> > Though, I'm worried for non-wayland users, they might see performance
> > degradation.
> 
> My understanding was vaapisink is primarily for debugging/testing purposes,
> and perhaps for specialised applications that know that they will be using
> VAAPI. No "normal" application will need vaapisink to be autoplugged, will
> it?

I think so.
Everything through dmabuf + glimagesink should be the default.

BTW, vaapisink was giving slightly better performance(both cpu utilization and memory) than glimgesink last time when I tested.

The patch in comment #20 doesn't sound ideal. In fact I fear it might degrade things. We have a set of patches in Ubuntu that makes everything work properly. Let me find what you need...

Oh, sorry. This seems to be tied up in discussions we've already had above, and in bug 783169, bug 784369 and bug 740753.

So yeah, it works in Ubuntu using those patches but is still suboptimal and not yet accepted by upstream. It will be a while (weeks/months) before I have time to dive back into video acceleration work, sorry.

Clearly my memory of all things VAAPI has faded. However, I do remember that lack of support for NV12 by mutter is a red herring. Yes it would be nice for efficiency but it's also not necessary. Because (gstreamer-)vaapi can already decode to rgb32. So this could just be a negotiation failure of the sink to tell the vaapi side what format(s) it supports.

Waiting for mutter (and all compositors) to support NV12 is probably also a bad idea that doesn't scale. That's handing off responsibility for the problem to be fixed N times, where N (number of compositors) is unbound going into the future.

Surely we should just be asking vaapi to decode to a buffer using a format that the target sink definitely supports (where "supports" means it's also checked that with the current compositor). It seems to be working that way in Ubuntu, so whether you take my patches or build your own, it is at least possible to get the pipeline working with vaapi.

Keeping vaapi in the picture is important, because modern videos along the lines of 4K/HDR/10-bit are too complex to decode in software. So vaapi should not be considered the optional component anymore. Only the choice of a more efficient or more compatible buffer format should be optional, and negotiated.

(In reply to Daniel van Vugt from comment #27)
> The patch in comment #20 doesn't sound ideal. In fact I fear it might
> degrade things. We have a set of patches in Ubuntu that makes everything
> work properly. Let me find what you need...

I'm sorry, but VAAPI on Ubuntu fails miserably for many users. That is a recurrent report we get, and everyone's solution is to simply remove it from their OS.

(In reply to Daniel van Vugt from comment #29)
> Waiting for mutter (and all compositors) to support NV12 is probably also a
> bad idea that doesn't scale. That's handing off responsibility for the
> problem to be fixed N times, where N (number of compositors) is unbound
> going into the future.

It remains the I420, YUY2 and NV12 support is being planned for gnome-shell. Though, viewporter is a bigger priority. Meanwhile, if you have DMABuf, note that the following pipeline would work (on Intel), so similar thing could be achieved:

  ... ! glupload ! glcolorscale ! gldownload ! waylandsink

This will download as DMABuf.

> 
> Surely we should just be asking vaapi to decode to a buffer using a format
> that the target sink definitely supports (where "supports" means it's also
> checked that with the current compositor). It seems to be working that way
> in Ubuntu, so whether you take my patches or build your own, it is at least
> possible to get the pipeline working with vaapi.

That requires playbin3. Unless you have ported to playbin3 in Ubuntu, it should not be possible.

> 
> Keeping vaapi in the picture is important, because modern videos along the
> lines of 4K/HDR/10-bit are too complex to decode in software. So vaapi
> should not be considered the optional component anymore. Only the choice of
> a more efficient or more compatible buffer format should be optional, and
> negotiated.

We know this, but we are completely slave of the stability of VAAPI internals. Note that threading has been added to videoconvert to push back this limit a little longer. The stability also needs to be proven cross platform (not only Intel).

I filed a bug in intel-vaapi-driver requesting support for RGB, though VPP shall be used to convert the frames to RGB first.

https://github.com/intel/intel-vaapi-driver/issues/357

Attachment 368273 [details] pushed as c653cb5 - vaapi: register vaapisink as marginal on wayland

Victor,
Yes it probably was VPP converting to 32-bit. Not ideal, but OK.

Nicolas,
> I'm sorry, but VAAPI on Ubuntu fails miserably for many users.
I have asked Ubuntu users directly, AND I monitor all the bug reports. If what you say is true then nobody is reporting the problem. I only know of it breaking last week, due to a Mesa regression (https://bugs.freedesktop.org/show_bug.cgi?id=105013). It's worked perfectly for most of our users for the past few months.

Don't you try for a second to suggest that you communicate more with Ubuntu users than I do.

I'm an Ubuntu user and the first thing I had to do after installing 17.10 is uninstall vaapi. Totem couldn't play any video at all (black screen). I didn't report because it was so major I'm was sure you already had thousands of reports. Haven't tested since the release,  maybe it got fixed already.

What's all this talk about Ubuntu about? I don't think there's anything particularly problematic about Ubuntu and gst-vaapi? People have problems with gst-vaapi on lots of different distros/configurations.

From a GStreamer perspective it's sadly still the case that gst-vaapi makes us look a bit bad because it just doesn't work well enough across the board out of the box, with the apps and sinks people actually use (i.e. not vaapisink). The fault may be with other layers in the stack of course, but from an end-user perspective that doesn't matter.

(In reply to Tim-Philipp Müller from comment #36)
> People have problems with gst-vaapi on lots of different distros/configurations.

Bug 779762 is an example.

Xavier,
Please log a bug in launchpad and subscribe 'vanvugt' to it. I would like to find out what that black screen problem is.

Tim,
The dominant use case for us (Ubuntu desktop team) is with Totem. And prior to https://bugs.freedesktop.org/show_bug.cgi?id=105013 that worked well with vaapi in both Xorg and Wayland sessions. Although that assumes you also had patches for these installed to fix the Wayland case:
  https://bugzilla.gnome.org/show_bug.cgi?id=784369
  https://bugzilla.gnome.org/show_bug.cgi?id=783169