Yes, I can reproduce as well *sometimes* - The back area is actually the shadows mutter adds to server-side decorations with the X11 backend.

 - It seems fairly random - Moving the window might make the issue go away.
 - It seems to affect some apps more than others (FF, TB whereas I cannot reproduce with xterm for example)
 - It doesn't seem to be HW/driver related

(In reply to Olivier Fourdan from comment #1)
>  - It seems to affect some apps more than others (FF, TB whereas I cannot

Actually it seems to affect gtk2 and Qt based apps and I think this could be related to the use of the _NET_WM_SYNC protocol v1 by these toolkits.

As a test, if I disable support for _NET_WM_SYNC in mutter, I cannot reproduce the problem anymore (but obviously resizes are a lot less smooth)

(In reply to Olivier Fourdan from comment #2)
> Actually it seems to affect gtk2 and Qt based apps and I think this could be
> related to the use of the _NET_WM_SYNC protocol v1 by these toolkits.

Well, not just v1 actually, forcing X11 backend on gtk3-demo and resizing service-side decorated windows can exhibit the problem as well.

And I cannot reproduce with _NET_WM_SYNC disabled ("window->disable_sync = TRUE;" in meta_window_x11_update_sync_request_counter() from window-x11.c)

*** Bug 770367 has been marked as a duplicate of this bug. ***

So on Xorg, forcing sync eliminates the black box flickering but causes it on Xwayland?
Could this be a Xayland bug as suggested in bug 77036?

It's two different things imho. _NET_WM_SYNC is not to avoid black box flickering of the drop shadows on X11 (as these are outside the client toplevel window), it's to synchronize frames redraw and avoid resizing faster than the client can cope.

For reference, see Owen's description of the the extended synchronisation version of _NET_WM_SYNC used by gtk+ and mutter.

It could be a Xwayland bug, possibly, but so far I haven't really found hard evidence of that.

Besides, on X11/Xwayland with server side decorations, mutter draws the drop shadows underneath the windows.

Quick update - I have been looking into this lately to see if we could use Pekka's proposal of adding an _XWAYLAND_ALLOW_COMMITS property to tell Xwayland when to commit the surface:

  https://lists.x.org/archives/xorg-devel/2016-November/051913.html
  https://lists.x.org/archives/xorg-devel/2016-December/051947.html

and

  https://patchwork.freedesktop.org/series/16610/

Before so, I tried to understand what would be causing the back border issue in the first place and traced it down to meta_window_actor_update_opaque_region().

And it's not called from the same code path when using an application that uses _NET_WM_SYNC protocol (e.g. gtk-demo from gtk2) and one who doesn't (e.g. plain old xterm).

With XSync protocol:

  meta_window_x11_update_sync_request_counter()
    meta_compositor_sync_updates_frozen()
      meta_window_actor_sync_updates_frozen()
        meta_window_actor_set_updates_frozen()
          meta_window_actor_thaw()
              meta_window_actor_handle_updates()
              check_needs_reshape()
                meta_window_actor_update_opaque_region();

Without XSync protocol:

  clutter_clock_dispatch()
    master_clock_update_stages()
      _clutter_run_repaint_functions()
        meta_window_actor_pre_paint()
          meta_window_actor_handle_updates()
            check_needs_reshape()
              meta_window_actor_update_opaque_region();

Created attachment 341942 [details] [review]
[PATCH] MetaWindowActor: Update shape in pre-paint

When dealing with clients who support the _NET_WM_SYNC protocol, the
shape update is done from meta_compositor_sync_updates_frozen(), but tha
can leave (on Xwayland primarily) visual traces of the previous hsape as
being cleared by a resize, causing ugly black border flickers.

Avoid the issue by updating the shape in pre_paint() as with other
clients which do not support _NET_WM_SYNC.
--
Note: that (simple) patch seems to avoid the issue, without needing additional synchronization mechanism.

Created attachment 341943 [details] [review]
[PATCH] MetaWindowActor: Remove uneeded test

The is_frozen() function tests for freeze_count >0, so no need to test
it twice.
--
Note: Trivial simplification, while at it...

Review of attachment 341942 [details] [review]:

It breaks synchronization on X11 so it's not suitable in the general case

Also experience this on Fedora 25 (Gnome 3.22 on Wayland).
i5-2410M with Sandybridge integrated graphics.
Is there any way to help?

I'm also experiencing this issue with applications such as Firefox and Qt Creator, but not with i.e Blender and Chromium and most pre-insalled applications. I'm using Fedora 25 with Gnome 3.22.3 on Wayland, running on an Intel i5-2500 with its own integrated graphics.

testing 3.26 beta... this bug is still happening.

@Olivier Testing your patch with mutter 3.26.2, it does not resolve the problem. It makes the black artifacts outside the windows disappear, however the right and the bottom edge of the window flicker badly inside, black and otherwise, I would say that aspect is worse than before.

Anyway I noticed snapping / tiling windows got very smooth in 3.26. I can not see black flickering when tiling to window to half, or dragging it to the top and let it maximize.

Would it be possible for you to look into that issue again? This is a very visible issue and makes the Wayland experience bad, especially for users who do not know the reasons.

Pointed to this bug by Jonas Ådahl on the GNOME Shell mailing list.

We seem to see something similar when starting VirtualBox.  In VirtualBox we need for various reasons to know what the usable area of each screen is - that translates as the largest rectangle which fits on that screen without obscuring any panels or similar.  We find that out by mapping a maximised window on each screen at start-up and checking its position and dimensions.  To prevent disturbance to the user we create the window fully translucent, or if the X server does not support that we use XShape and leave only a single pixel visible.  On GNOME Shell Wayland edition (what is the right way to say that?) this unfortunately flashes a black rectangle up on the screen in the area which should be translucent.

(If anyone is interested I explained the why of this on the mailing list[1].)  I see this on Ubuntu 17.04 and 18.04 Wayland session, and it has been reported on Fedora too.  Please let me know if I should test any patch, including previous ones on this bug.

[1] https://mail.gnome.org/archives/gnome-shell-list/2018-February/msg00001.html

(In reply to Michael Thayer from comment #15)
> Pointed to this bug by Jonas Ådahl on the GNOME Shell mailing list.
> 
> We seem to see something similar when starting VirtualBox.  In VirtualBox we
> need for various reasons to know what the usable area of each screen is -
> that translates as the largest rectangle which fits on that screen without
> obscuring any panels or similar.  We find that out by mapping a maximised
> window on each screen at start-up and checking its position and dimensions.
> To prevent disturbance to the user we create the window fully translucent,
> or if the X server does not support that we use XShape and leave only a
> single pixel visible.  On GNOME Shell Wayland edition (what is the right way
> to say that?) this unfortunately flashes a black rectangle up on the screen
> in the area which should be translucent.

XShape might be a different issue than this one (I suspect this is related to ARGB here), but maybe you can provide a simple reproducer that does the same thing as Virtualbox?

It probably wasn't helpful of me to mention XShape here at all.  It is only used if ARGB is not available, so is probably not relevant here.  I will see if our GUI developer has time to write up a simpler reproduction scenario.  He is a (cross-platform) Qt developer rather than an X11/Wayland specialist, so is it alright if it uses Qt?

(In reply to Michael Thayer from comment #17)
> It probably wasn't helpful of me to mention XShape here at all.  It is only

Yeap, I understand, jus wanted to avoid mixing potentially different issues :)

> used if ARGB is not available, so is probably not relevant here.  I will see
> if our GUI developer has time to write up a simpler reproduction scenario. 
> He is a (cross-platform) Qt developer rather than an X11/Wayland specialist,
> so is it alright if it uses Qt?

Of course it's all right :) I was planning to try with gtk/gdk, but which toolkit the client is using is probably irrelevant (unless the issue lies in the toolkit).

One thing to consider is that, from what I found, this is affecting x11 clients using the _NET_WM_SYNC protocol (for example, I don't see any of these with, say, xterm), and both gtk+ and Qt use _NET_WM_SYNC.

Created attachment 368140 [details]
Test Qt application which demonstrates the problem

This is what our GUI developer provided.  Hope it is helpful.  Build instructions inside the ReadMe.txt.

That really doesn't look like the same issue at all, the entire window is opaque and black, as if it wasn't using transparency at all.

Does it look like something I should open a new bug for?

Created attachment 368144 [details]
ARGB window using cairo

ARGB works fine here, see this example using plain X11 and cairo.

The Qt example works with weston, so yes, different issue than this one.

(FWIW, trying to get xprop of the transparent Qt mapped window (the one shat shows up black), all I get is “WM_NAME(STRING) = "mutter guard window"”)

The “black border” effects comes from the frame window resize.

The gdk_window_move_resize() in meta_ui_frame_move_resize() is what causes the border to become solid black, before it's repainted by the compositor.

With basic X11 clients, the delay between the gdk_window_move_resize() and the shadow being redrawn is so short that it cannot be seen, but when the X11 client is using a sync mechnaism, the update of the content is delayed until the client has finished updating its content, and in this case the black border is seen for a short time.

That explains why the black border shows most with clients with using NET_WM_SYNC protocol and server-side decorations.

As an experiment, commenting out the gdk_window_move_resize() in meta_ui_frame_move_resize() makes the problem disappear entirely:

  https://gitlab.gnome.org/GNOME/mutter/blob/master/src/ui/frames.c#L656

whereas commenting out the call to build_and_scan_frame_mask() in meta_window_actor_update_shape_region() makes the black border visible continuously:

  https://gitlab.gnome.org/GNOME/mutter/blob/master/src/compositor/meta-window-actor.c#L1699

So yes, bug 757104 is a good idea, setting a different bg color on the gdk window frame changes the color seen during resize, so the black seen comes from the gdk_window background color, before it's eventually repainted by the compositor.

(and nope, using a alpha channel of 0.0 in the bg color doesn't avoid the issue)

Created attachment 368368 [details] [review]
Debug patch to exhibit the issue

This patch does:

 - Set the gdk_window frame gb_color to yellow
 - Set the metawindow actor color to red
 - Removes the repaint to black in meta_frames_draw()

The idea is to show the yellow/red blinking.

To reproduce

 1. Build mutter with this patch
 2. Run mutter nested:
    $ jhbuild run mutter --wayland --nested
 3. Run gtk3-demo “Application Class” with x11 backend under Wayland:
    $ DISPLAY=:1 GDK_BACKEND=x11 gtk3-demo --run application_demo
 4. Resize the “Application Class” window (that one uses server-side decorations, so decorated by mutter)

Now what does this show:

 1. The initial color (yellow with this patch) comes from the gdk_window.
 2. meta_frames_draw() clears the window to black, that doesn't help!
 3. Using an alpha of 0.0 for the GDK window background makes the color black again, as if the mutter's own gdk_window used for the frame wasn't composited...

So I reckon the problem is because in Wayland, we update the surface when a new buffer is received.

So what happens is, gdk_window_move_resize() clears the frame background, a new buffer is attached once the Xwindow is resized, MetaSurfaceActorWayland repaints using that buffer with the black frame, while repaint of the shadows still awaits for the sync of the client, thus delaying the repaint of the shadows. In the mean time, the black shadows have been seen.

Or something around these lines...

I mean, I think this is purely a mutter compositor issue.

Created attachment 368569 [details] [review]
[PATCH 1/2] window: Add set_frozen method

This is actually only meaningful for Xwayland as this translates as
_XWAYLAND_ALLOW_COMMITS.

Created attachment 368571 [details] [review]
[PATCH 2/2] window: Fix sync issue causing back borders

Using _xwayland_allow_commits() and meta_window_updates_are_frozen().

Comment on attachment 368571 [details] [review]
[PATCH 2/2] window: Fix sync issue causing back borders

https://gitlab.gnome.org/GNOME/mutter/merge_requests/32

let's track this on gitlab only.

latest link is here:

https://gitlab.gnome.org/GNOME/mutter/merge_requests/53