Did some research (for a different bug) and stumbled upon this gem:
Window shadow that Windows WM draws by its own? It's actually a separate layered window! If you fire up Process Hacker, you can see it in the list of the W32 window objects created by your process. The class of that window is, tellingly, called SysShadow.

Which means that WM doesn't count window shadow towards the size of a window because the shadow is not a part of a window to begin with.

This is good news in the sense that it should be possible to do what the WM does - create a new layered window for each toplevel and draw the shadow there.

That leaves the resize grip area, which is smaller.

Created attachment 317985 [details] [review]
Leaky experimental separate shadow window patch

Hacked together this thing. Note that it tries to break abstraction layers, leaks memory and probably isn't very efficient.

Instead of drawing window shadow as part of the window, GTK draws the shadow on a separate surface that it then gives to GDK. GDK (or, rather, its W32 backend) creates a separate layered no-input window and copies the shadow onto it. GDK also ensures that the shadow window's position and size change with the position and size of the main window.

One obvious bug that i haven't looked into is that the main window should be drawn with no shadow. It's probably as easy as commenting out the gtk_draw_background() call for it though.

I've converted all window moves into DeferWindowPos, so both windows are constantly aligned when moved. If the process is slowed down, moving is choppy, but windows never go out of sync.

Resizing is quite another story. Grabbing an edge and dragging makes GDK plunge into a subloop where it is constantly processing drag events and calls GTK context to do normal processing once every 10ms. Because of that (or some other factors) resizing doesn't look very good. This is actually not limited to my changes, vanilla CSD windows have this as well. If you run, say, gtk-demo.exe --gdk-debug=EVENTS, and try to resize the window, it will be just hard-cropped (when sizing down) or nothing will seem to happen until you stop moving the cursor (when sizing up). AFAIU, GTK never gets to redraw the window contents, and GDK can't do that for us, so it does nothing. --gdk-debug=EVENTS creates a delay in processing that makes the lag obvious (otherwise it can be just choppy, but generally passable).

This is a problem for non-CSD windows as well, but it's barely noticeable: because window edges are drawn by the WM, it's the contents that can't be updated fast enough (so there's some beating and tearing near the edge) instead of the edge and the shadow. This is observable if you run, say, Explorer, and try resizing its window quickly (since there's no way to easily slow it down, like --gdk-debug).

This experimental patch breaks the GTK/GDK border in a few places, and the problem described above makes it more obvious why that is necessary - to draw window border neatly, especially during sizing events, WM (or GDK, when GDK does CSD) needs to be able to redraw the border quickly and without breaking out of the sizing loop. Unless that capability is given to GDK somehow, it wouldn't be possible do clean resizes.

Also, because GDK can't draw the shadow by itself, the window is initially presented without a shadow. Resizing or moving it will make it appear (because there's currently no other way for GTK to tell GDK that it needs to update the shadow window; the shadow window can't be updated initially, because it takes a while to get to draw the shadow for the first time).

On the technical side, i've had to use a W32 cairo function to create the shadow surface (in GTK), because it's the only way to get a surface that has a DC. A DC is needed to update the layered window (we can't just draw on it, AFAICS). This is unlike how the normal drawing works, where GTK renders everything into an (image?) surface and then uses cairo to blit that surface onto the window surface (which is tied to window DC).

Anyway, with the shadow in another window, the main window only has 10-pixel empty border (this is what margin Adwaita asks for). This could be further reduced by, say, dynamically adjusting margin when window edge is near the edge of the screen (because why would you need wide margins when you can just move your mouse pointer to the edge - as long as the margin is at least 1 pixel thick, you'll hit it). Note that Windows 10 has 5-pixel margin everywhere except the top edge (top edge has resize grip area inside the titlebar), and, accordingly, you can't make a window wider than the screen width minus 10 pixels (making it taller than the screen is also impossible, but snapping actually helps there). They do, however, have manual snapping (winkey + arrow) and auto snapping, so the only case when this quirk is clearly noticeable is when you're trying to resize a window to be wider than the screen, which is a rare case.

Anyway, that's that. I think this needs to be discussed before any further progress can be done.

The upside of this work is that i've got some experience with layered windows now. Maybe i'll be able to port GDK to layered windows completely (bug 748872) at some point.

Now that bug 748872 and bug 761629 have solutions (still pending review), i see two ways to fix this bug:

1) As planned, make a separate non-input shadow window, the way W32 window manager does.

With layered windows it now should (hopefully; needs testing) be possible to move the window and its shadow in sync, which would fix the only major side-effect of having a separate window for shadow.

As the experimental patch shows, this might require some tinkering with the GTK/GDK barrier.

2) Go back to the things that bug 761629 left unimplemented and build on top of that.

Since we now control the whole resize process, it will be possible to just provide different sizing constrains, meaning that it will be possible to resize a CSD window to cover the whole desktop (and maybe even multiple desktops, depending on how it will be implemented; by us) visually, despite the fact that the underlying window will actually be larger than that. Same for movement - now that AeroSnap is not in play, nothing prevents the user from moving a window top the top edge of the screen visually, even though an invisible part of it will move past the edge.

This will probably not require any API changes, as GDK is already notified of the shadow geometry by GTK (we just couldn't use that info before).

Although, screenshots or other things that work with the window directly will still reveal that it's bigger than it looks.

Some updates on this: bug 763013 adds shadow size calculation support and fixes some of the things described here.

However, the underlying window is still larger than it should be.

I did another separate shadow patch as an experiment, this time doing the separation in GDK, internally (since GDK knows shadow size, that came to be possible). I also relaxed some of my visual quality constrains after realizing that layered windows might not survive the eventual move to GSK (which, if ebassi is to be believed, means compositing everything with GL, which conflicts with layered windows), so it might be OK if shadow doesn't too very good while window is being resized.

Anyway, splitting off the shadow in GDK looks interesting, but is ultimately pointless: window size remains the same.

I've tried to implement a function that also tells GDK the *grip* border size (along with the shadow size), and which GDK would use to reduce window size to minimally-possible. That eventually led nowhere - it's extremely difficult to keep GTK (and generic GDK) code thinking that window size is WxH, when in reality the underlying native window is actually smaller (by "shadow size" minus "grip size"). The surface where you have to convert back and forth is just too large to maintain such a lie.

Therefore to actually reduce window size by excluding the decorative bits, GTK has to know. I'll try to make something more presentable out of my first patch for this bug, as this is the direction my efforts are currently taking me in.

To put this in context, here's some stuff that was discussed in IRC at various points, but never made it here:

1) For layered windows, Windows WM does preliminary hit-testing via alpha value. Alpha == 0 means that this pixel is transparent, and the window never gets any mouse clicks on this pixel, no matter what i do. Alpha > 0 means that this pixel is not transparent, and the window gets the mouse clicks normally (see (2)).
For non-layered windows there's no alpha (there's a window shape region check, but i don't think that GDK uses GDI window shape regions), so window always gets a click that falls into its rectangle, and (2) always applies.

2) WM_NCHITTEST was thought to be responsible for hit-testing and transparency, but careful reading of the MSDN (as well as testing some code) shows that once a window gets a click, that's it. WM_NCHITTEST is to distinguish between subwindows of a toplevel HWND. There's no way to pass the click to a toplevel window below it, if that window is in another process (or another *thread* for that matter). In fact, WM_NCHITTEST is sent when the cursor just moves over a window, so it is hypothetical in its nature (i.e. "if i were to click at this point, which part of your window would i have hit?"). Only MS knows how clicks are actually processed, and we have no control over this.

3) Windows are much bigger than user sees, by some 26-29 pixels (depending on which edge we're looking at; values are from the top of my head). This is because Adwaita uses some kind of very generic Gaussian-blur-based shadow drawing process that doesn't give it much control over the form of the slope the shadow alpha has: resulting shadow rather quickly (in roughly 10 pixels) goes into near-invisibility, and then tapers off from alpha=0.07 to alpha=0.00 over the course of some 10-15 pixels more.
As per (1), this makes window receive clicks on the alpha-almost-0-but-not-0-yet areas, which is highly confusing and annoying for users, because that stuff is supposed to be click-through.
I don't see Adwaita or the shadow-drawing process improving much in this regard, so it is imperative that shadow is split off into a separate window.

4) Shadow should get split off, but grip probably shouldn't (at least i have hard time coming up with a plan to also split the grip area into separate toplevel that doesn't involve a *lot* of super-crazy code). This is not ideal, but grip is see-through, not click-through anyway, and MS seems to be unable (as of Windows 10) to split it off either, and grip area is way smaller than the whole shadow, so i'd leave it at that.
Layered windows will need alpha in grip areas to be set to non-0 (because of (1)). That is currently not a problem due to (3), but not all themes are Adwaita, so this should be fixed eventually (or not? Just tell theme writers that shadow should cover the margins at least?).

We're moving to gitlab! As part of this move, we are moving bugs to NEEDINFO if they haven't seen activity in more than a year. If this issue is still important to you and still relevant with GTK+ 3.22 or master, please reopen it and we will migrate it to gitlab.

As announced a while ago, we are migrating to gitlab, and bugs that haven't seen activity in the last year or so will be not be migrated, but closed out in bugzilla.

If this bug is still relevant to you, you can open a new issue describing the symptoms and how to reproduce it with gtk 3.22.x or master in gitlab:

https://gitlab.gnome.org/GNOME/gtk/issues/new