After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 775698 - vaapisink: wayland: mutter cannot import surface into a texture
vaapisink: wayland: mutter cannot import surface into a texture
Status: RESOLVED FIXED
Product: GStreamer
Classification: Platform
Component: gstreamer-vaapi
1.10.x
Other Linux
: High blocker
: 1.13.1
Assigned To: GStreamer Maintainers
GStreamer Maintainers
P3
: 783172 793376 (view as bug list)
Depends on: 705514
Blocks: 748634
 
 
Reported: 2016-12-06 11:17 UTC by Marcos Mello
Modified: 2018-10-13 16:08 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
gst-play-1.0 debug log (61.75 KB, text/plain)
2016-12-06 14:53 UTC, Marcos Mello
  Details
vaapi: register vaapisink as marginal on wayland (1.60 KB, patch)
2018-02-12 17:01 UTC, Víctor Manuel Jáquez Leal
committed Details | Review

Description Marcos Mello 2016-12-06 11:17:33 UTC
Running GNOME 3.22 + Wayland (Arch), with gstreamer 1.10.2 and libva 1.7.3, I get this for MPEG-2/AC3 MKV file (extracted from DVD):

$ gst-play-1.0 test.mkv 
Press 'k' to see a list of keyboard shortcuts.
Now playing /home/marcos/Downloads/test.mkv
libva info: VA-API version 0.39.4
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
libva info: VA-API version 0.39.4
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
Redistribute latency...
Redistribute latency...     
wl_surface@7: error 2: Failed to create a texture for surface 7
ERROR Internal error: could not render surface for file:///home/marcos/Downloads/test.mkv
ERROR debug information: gstvaapisink.c(1457): gst_vaapisink_show_frame_unlocked (): /GstPlayBin:playbin/GstPlaySink:playsink/GstBin:vbin/GstVaapiSink:vaapisink0
Reached end of play list.

(Totem fails too)

I have an old Intel GM45 chip:

$ vainfo
libva info: VA-API version 0.39.4
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
vainfo: VA-API version: 0.39 (libva 1.7.3)
vainfo: Driver version: Intel i965 driver for Intel(R) GM45 Express Chipset - 1.7.3
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointVLD

Under Xorg session it works:

$ gst-play-1.0 test.mkv 
Press 'k' to see a list of keyboard shortcuts.
Now playing /home/marcos/Downloads/test.mkv
libva info: VA-API version 0.39.4
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
libva info: VA-API version 0.39.4
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
Redistribute latency...
Redistribute latency...
0:00:05.1 / 0:00:05.2       
Reached end of play list.
Comment 1 Marcos Mello 2016-12-06 11:33:26 UTC
Title changed. Other video codecs like MPEG-4 and VP8 fail too. Only audio files play.
Comment 2 Víctor Manuel Jáquez Leal 2016-12-06 12:48:37 UTC
As you can see, your hardware only support MPEG2 decoding. For other codecs you should use software decoders as in gst-libav.

can you upload a debug log with GST_DEBUG=vaapi*:5 ??

what happens with this command:

gst-play-1.0 test.mkv --videosink=clutterautovideosink ??
Comment 3 Marcos Mello 2016-12-06 14:51:16 UTC
(In reply to Víctor Manuel Jáquez Leal from comment #2)
> As you can see, your hardware only support MPEG2 decoding. For other codecs
> you should use software decoders as in gst-libav.

gst-play-1.0 also fails to play other codecs, but Totem plays them fine. Tested with MPEG-4 SP and VP8. Weird.

> 
> can you upload a debug log with GST_DEBUG=vaapi*:5 ??

Will do.

> 
> what happens with this command:
> 
> gst-play-1.0 test.mkv --videosink=clutterautovideosink ??

$ gst-play-1.0 test.mkv --videosink=clutterautovideosink

** (gst-play-1.0:1200): WARNING **: Couldn't create specified video sink 'clutterautovideosink'
Press 'k' to see a list of keyboard shortcuts.
Now playing /home/marcos/Downloads/test.mkv
libva info: VA-API version 0.39.4
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
libva info: VA-API version 0.39.4
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
Redistribute latency...
Redistribute latency...
wl_surface@7: error 2: Failed to create a texture for surface 7
ERROR Internal error: could not render surface for file:///home/marcos/Downloads/test.mkv
ERROR debug information: gstvaapisink.c(1457): gst_vaapisink_show_frame_unlocked (): /GstPlayBin:playbin/GstPlaySink:playsink/GstBin:vbin/GstVaapiSink:vaapisink0
Reached end of play list.
Comment 4 Marcos Mello 2016-12-06 14:53:40 UTC
Created attachment 341482 [details]
gst-play-1.0 debug log

GST_DEBUG_NO_COLOR=1 GST_DEBUG=vaapi*:5 gst-play-1.0 test.mkv
Comment 5 Víctor Manuel Jáquez Leal 2016-12-06 15:23:47 UTC
Ok, you don't have installed clutterautovideosink. Totem should be getting other cluttersink.

But I confirmed, but I don't know where is the problem: either in vaapisink or in mutter, which fails to upload the surface into a texture.
Comment 6 Víctor Manuel Jáquez Leal 2017-05-29 08:40:34 UTC
*** Bug 783172 has been marked as a duplicate of this bug. ***
Comment 7 Víctor Manuel Jáquez Leal 2017-05-29 16:10:34 UTC
The problem is that mutter, in wayland, only imports EGL textures with color formats RGB and RGBA, but vaapisink asks to import a EGL texture with EGL_TEXTURE_Y_UV_WL.

As far as I understand, mutter cannot import these textures because COGL cannot handle them either.

In the case of weston wayland compositor, there is not problems with vaapisink.

My ideal solution would be to add support for this textures in cogl and mutter.

What we can do internally to gstreamer-vaapi? 

1\ reduce the rank of vaapisink
2\ figure out a way to internally convert the NV12 image into RGB.
Comment 8 Nicolas Dufresne (ndufresne) 2017-05-29 16:29:33 UTC
(In reply to Víctor Manuel Jáquez Leal from comment #7)
> The problem is that mutter, in wayland, only imports EGL textures with color
> formats RGB and RGBA, but vaapisink asks to import a EGL texture with
> EGL_TEXTURE_Y_UV_WL.
> 
> As far as I understand, mutter cannot import these textures because COGL
> cannot handle them either.

That is not entirely correct. Weston synthesize this format using a shader. In the end, you have two textures, Y as an R8, and UV as a GR88 and a shader is used. The same action is done by cluttersink, which is using clutter/COGL, so there is no reason why you can't implement that directly inside gnome-shell/mutter. Plus, the code already exist.
Comment 9 Daniel van Vugt 2017-07-07 08:12:37 UTC
Let's keep this bug about mutter only (or more generally when using waylandsink with NV12 videos in compositors that don't support NV12). 

Totem's VAAPI problems in Wayland are covered by bug 783169 and bug 784369.
Comment 10 Daniel van Vugt 2017-07-07 08:15:57 UTC
By the way, this will work with VAAPI under Gnome on Wayland:
$ env GST_GL_WINDOW=wayland gst-play-1.0 --videosink glimagesink

The first part works around the gl/vaapi plugin bug 783169. The second part works around this bug 775698 as well as bug 784369. :)
Comment 11 sreerenj 2017-10-31 20:58:35 UTC
Hm, anyone upgrading to latest Ubuntu will hit this issue!

It is breaking all gst-launch pipelines in Ubuntu 17.10 since it using gnome-shell/wayland as default now.

I am raising the severity.
Comment 12 sreerenj 2017-10-31 21:54:05 UTC
@Victor: I don't think internal conversion of render_surface to RGB is going to help in this case because we need to get the Wayland buffer anyway and unfortunately vaGetSurfaceBufferWl() doesn't have RGB support.
Comment 13 Daniel van Vugt 2017-11-01 03:36:00 UTC
I think the issue with gst-play/gst-launch in Ubuntu 17.10 is actually a regression in the existing DMA-buf support mentioned here:

https://bugzilla.gnome.org/show_bug.cgi?id=784369#c12

From Ubuntu's perspective we don't mind the regression in the short term because that's what it took to deliver working VAAPI support in Totem. And Totem using clutter-gst doesn't care about DMA-buf yet (bug 759209).

For general users the default video player (Totem) working efficiently in the default session (Wayland) is vastly more important than other use cases. But eventually we'd like to not compromise at all.

That issue doesn't have a bug for it yet so please do open one on Launchpad (because the offending patch and the problem theoretically only exists in Ubuntu right now).
Comment 14 Víctor Manuel Jáquez Leal 2017-11-01 15:52:06 UTC
(In reply to sreerenj from comment #11)
> Hm, anyone upgrading to latest Ubuntu will hit this issue!
> 
> It is breaking all gst-launch pipelines in Ubuntu 17.10 since it using
> gnome-shell/wayland as default now.
> 
> I am raising the severity.

Ubuntu now uses Wayland/Mutter by default.

And the "bug" is in Mutter. If you try Weston, it works.

But rather than bug it is a missing feature in mutter that needs to be developed.
Comment 15 sreerenj 2017-11-01 18:35:43 UTC
(In reply to Víctor Manuel Jáquez Leal from comment #14)
> (In reply to sreerenj from comment #11)
> > Hm, anyone upgrading to latest Ubuntu will hit this issue!
> > 
> > It is breaking all gst-launch pipelines in Ubuntu 17.10 since it using
> > gnome-shell/wayland as default now.
> > 
> > I am raising the severity.
> 
> Ubuntu now uses Wayland/Mutter by default.
> 
> And the "bug" is in Mutter. If you try Weston, it works.
> 
> But rather than bug it is a missing feature in mutter that needs to be
> developed.

Yup, I did some investigation/testing too.

Wondering what we should do with this since it breaking most of the gst-launch pipelines. Adding BGRx support for vaGetSurfaceBufferWl() in intel-vaapi-driver and doing an internal conversion in vaapisink is the workaround I can think of (I don't like it though). Or degrade vaapisink rank and promote glimagesink usage which is of course what we need in future.
Comment 16 Víctor Manuel Jáquez Leal 2017-11-15 12:07:39 UTC
After looking at this, I'm leaning to demote vaapisink's rank, to avoid autoplug.
Though, I'm worried for non-wayland users, they might see performance degradation.
Comment 17 Víctor Manuel Jáquez Leal 2018-02-12 11:07:34 UTC
*** Bug 793376 has been marked as a duplicate of this bug. ***
Comment 18 Tim-Philipp Müller 2018-02-12 11:24:23 UTC
> After looking at this, I'm leaning to demote vaapisink's rank, to avoid
> autoplug.

Let's do it?


> Though, I'm worried for non-wayland users, they might see performance
> degradation.

My understanding was vaapisink is primarily for debugging/testing purposes, and perhaps for specialised applications that know that they will be using VAAPI. No "normal" application will need vaapisink to be autoplugged, will it?

Better poor performance than doesn't work.
Comment 19 Nicolas Dufresne (ndufresne) 2018-02-12 16:22:15 UTC
I'd like this to be fixed for 1.14, it's been showing up in distros, which looks really bad.
Comment 20 Víctor Manuel Jáquez Leal 2018-02-12 17:01:11 UTC
Created attachment 368273 [details] [review]
vaapi: register vaapisink as marginal on wayland

vaapsink, when used with the Intel VA-API driver, tries to display
surfaces with format NV12, which are handled correctly by
Weston. Nonetheless, COGL cannot display YUV surfaces, making fail
pipelines on mutter.

This shall be solved either by COGL or by making the driver to paint
RGB surfaces. In the meanwhile, let's just demote vaapisink as
marginal when the Wayland environment is detected, no matter if it is
Weston.
Comment 21 sreerenj 2018-02-12 20:46:32 UTC
Review of attachment 368273 [details] [review]:

lgtm.
Comment 22 sreerenj 2018-02-12 20:48:57 UTC
Is there any plan to move "glimagesink" as "primary rank" element?
Comment 23 Christian Stadelmann 2018-02-12 21:10:26 UTC
(In reply to Tim-Philipp Müller from comment #18)
> My understanding was vaapisink is primarily for debugging/testing purposes,
> and perhaps for specialised applications that know that they will be using
> VAAPI. No "normal" application will need vaapisink to be autoplugged, will
> it?

Isn't VAAPI the only interface to get hardware-accelerated video on intel GPUs? My understanding is that VAAPI is required to use codec-specific hardware such as for H.264 or H.265/HEVC.
Comment 24 Nicolas Dufresne (ndufresne) 2018-02-12 21:17:37 UTC
(In reply to Christian Stadelmann from comment #23)
> (In reply to Tim-Philipp Müller from comment #18)
> > My understanding was vaapisink is primarily for debugging/testing purposes,
> > and perhaps for specialised applications that know that they will be using
> > VAAPI. No "normal" application will need vaapisink to be autoplugged, will
> > it?
> 
> Isn't VAAPI the only interface to get hardware-accelerated video on intel
> GPUs? My understanding is that VAAPI is required to use codec-specific
> hardware such as for H.264 or H.265/HEVC.

Yes, but this is just about the display sink, which can be replaced by glimagesink.
Comment 25 Christian Stadelmann 2018-02-12 21:21:14 UTC
(In reply to Nicolas Dufresne (stormer) from comment #24)
> (In reply to Christian Stadelmann from comment #23)
> > 
> > Isn't VAAPI the only interface to get hardware-accelerated video on intel
> > GPUs? My understanding is that VAAPI is required to use codec-specific
> > hardware such as for H.264 or H.265/HEVC.
> 
> Yes, but this is just about the display sink, which can be replaced by
> glimagesink.

Ok, that makes sense. Thanks!
Comment 26 sreerenj 2018-02-12 22:10:15 UTC
(In reply to Tim-Philipp Müller from comment #18)
> > After looking at this, I'm leaning to demote vaapisink's rank, to avoid
> > autoplug.
> 
> Let's do it?
> 
> 
> > Though, I'm worried for non-wayland users, they might see performance
> > degradation.
> 
> My understanding was vaapisink is primarily for debugging/testing purposes,
> and perhaps for specialised applications that know that they will be using
> VAAPI. No "normal" application will need vaapisink to be autoplugged, will
> it?

I think so.
Everything through dmabuf + glimagesink should be the default.

BTW, vaapisink was giving slightly better performance(both cpu utilization and memory) than glimgesink last time when I tested.
Comment 27 Daniel van Vugt 2018-02-13 01:49:41 UTC
The patch in comment #20 doesn't sound ideal. In fact I fear it might degrade things. We have a set of patches in Ubuntu that makes everything work properly. Let me find what you need...
Comment 28 Daniel van Vugt 2018-02-13 02:00:49 UTC
Oh, sorry. This seems to be tied up in discussions we've already had above, and in bug 783169, bug 784369 and bug 740753.

So yeah, it works in Ubuntu using those patches but is still suboptimal and not yet accepted by upstream. It will be a while (weeks/months) before I have time to dive back into video acceleration work, sorry.
Comment 29 Daniel van Vugt 2018-02-13 03:23:23 UTC
Clearly my memory of all things VAAPI has faded. However, I do remember that lack of support for NV12 by mutter is a red herring. Yes it would be nice for efficiency but it's also not necessary. Because (gstreamer-)vaapi can already decode to rgb32. So this could just be a negotiation failure of the sink to tell the vaapi side what format(s) it supports.

Waiting for mutter (and all compositors) to support NV12 is probably also a bad idea that doesn't scale. That's handing off responsibility for the problem to be fixed N times, where N (number of compositors) is unbound going into the future.

Surely we should just be asking vaapi to decode to a buffer using a format that the target sink definitely supports (where "supports" means it's also checked that with the current compositor). It seems to be working that way in Ubuntu, so whether you take my patches or build your own, it is at least possible to get the pipeline working with vaapi.

Keeping vaapi in the picture is important, because modern videos along the lines of 4K/HDR/10-bit are too complex to decode in software. So vaapi should not be considered the optional component anymore. Only the choice of a more efficient or more compatible buffer format should be optional, and negotiated.
Comment 30 Nicolas Dufresne (ndufresne) 2018-02-13 13:12:48 UTC
(In reply to Daniel van Vugt from comment #27)
> The patch in comment #20 doesn't sound ideal. In fact I fear it might
> degrade things. We have a set of patches in Ubuntu that makes everything
> work properly. Let me find what you need...

I'm sorry, but VAAPI on Ubuntu fails miserably for many users. That is a recurrent report we get, and everyone's solution is to simply remove it from their OS.
Comment 31 Nicolas Dufresne (ndufresne) 2018-02-13 13:20:04 UTC
(In reply to Daniel van Vugt from comment #29)
> Waiting for mutter (and all compositors) to support NV12 is probably also a
> bad idea that doesn't scale. That's handing off responsibility for the
> problem to be fixed N times, where N (number of compositors) is unbound
> going into the future.

It remains the I420, YUY2 and NV12 support is being planned for gnome-shell. Though, viewporter is a bigger priority. Meanwhile, if you have DMABuf, note that the following pipeline would work (on Intel), so similar thing could be achieved:

  ... ! glupload ! glcolorscale ! gldownload ! waylandsink

This will download as DMABuf.

> 
> Surely we should just be asking vaapi to decode to a buffer using a format
> that the target sink definitely supports (where "supports" means it's also
> checked that with the current compositor). It seems to be working that way
> in Ubuntu, so whether you take my patches or build your own, it is at least
> possible to get the pipeline working with vaapi.

That requires playbin3. Unless you have ported to playbin3 in Ubuntu, it should not be possible.

> 
> Keeping vaapi in the picture is important, because modern videos along the
> lines of 4K/HDR/10-bit are too complex to decode in software. So vaapi
> should not be considered the optional component anymore. Only the choice of
> a more efficient or more compatible buffer format should be optional, and
> negotiated.

We know this, but we are completely slave of the stability of VAAPI internals. Note that threading has been added to videoconvert to push back this limit a little longer. The stability also needs to be proven cross platform (not only Intel).
Comment 32 Víctor Manuel Jáquez Leal 2018-02-13 17:30:10 UTC
I filed a bug in intel-vaapi-driver requesting support for RGB, though VPP shall be used to convert the frames to RGB first.

https://github.com/intel/intel-vaapi-driver/issues/357
Comment 33 Víctor Manuel Jáquez Leal 2018-02-13 17:48:50 UTC
Attachment 368273 [details] pushed as c653cb5 - vaapi: register vaapisink as marginal on wayland
Comment 34 Daniel van Vugt 2018-02-14 02:14:57 UTC
Victor,
Yes it probably was VPP converting to 32-bit. Not ideal, but OK.

Nicolas,
> I'm sorry, but VAAPI on Ubuntu fails miserably for many users.
I have asked Ubuntu users directly, AND I monitor all the bug reports. If what you say is true then nobody is reporting the problem. I only know of it breaking last week, due to a Mesa regression (https://bugs.freedesktop.org/show_bug.cgi?id=105013). It's worked perfectly for most of our users for the past few months.

Don't you try for a second to suggest that you communicate more with Ubuntu users than I do.
Comment 35 Xavier Claessens 2018-02-14 12:01:01 UTC
I'm an Ubuntu user and the first thing I had to do after installing 17.10 is uninstall vaapi. Totem couldn't play any video at all (black screen). I didn't report because it was so major I'm was sure you already had thousands of reports. Haven't tested since the release,  maybe it got fixed already.
Comment 36 Tim-Philipp Müller 2018-02-14 12:32:47 UTC
What's all this talk about Ubuntu about? I don't think there's anything particularly problematic about Ubuntu and gst-vaapi? People have problems with gst-vaapi on lots of different distros/configurations.

From a GStreamer perspective it's sadly still the case that gst-vaapi makes us look a bit bad because it just doesn't work well enough across the board out of the box, with the apps and sinks people actually use (i.e. not vaapisink). The fault may be with other layers in the stack of course, but from an end-user perspective that doesn't matter.
Comment 37 Strangiato 2018-02-14 12:42:02 UTC
(In reply to Tim-Philipp Müller from comment #36)
> People have problems with gst-vaapi on lots of different distros/configurations.

Bug 779762 is an example.
Comment 38 Daniel van Vugt 2018-02-15 01:54:19 UTC
Xavier,
Please log a bug in launchpad and subscribe 'vanvugt' to it. I would like to find out what that black screen problem is.

Tim,
The dominant use case for us (Ubuntu desktop team) is with Totem. And prior to https://bugs.freedesktop.org/show_bug.cgi?id=105013 that worked well with vaapi in both Xorg and Wayland sessions. Although that assumes you also had patches for these installed to fix the Wayland case:
  https://bugzilla.gnome.org/show_bug.cgi?id=784369
  https://bugzilla.gnome.org/show_bug.cgi?id=783169