Created attachment 197298 [details] [review]
screenshot: compensate invisible window borders using frame extents

In order to grab a screenshot of a window we currently peek at the WM
decorated X window size and coordinates, and cut a pixbuf from the root
window using those dimensions.

This unfortunately breaks with invisible window borders, as the WM
window is actually bigger than expected, resulting in an unwanted frame
around the image. Mutter now stores the real dimensions of the window
decorated frame in _NET_FRAME_EXTENTS, so we can just use that
information to get the actual region to cut.

Mutter also stopped setting the XShape bounding box on rounded corners,
as they're now properly antialiased. Our code that takes the XShape
rectangles and makes the area outside of their bounding box transparent
also needs to be updated to handle the case where the WM decorated window
size is different from the frame extents.

Tested with GNOME Shell/Mutter and Metacity.

Review of attachment 197298 [details] [review]:

I find it silly that gdk_window_get_frame_extents doesn't give you the raw _NET_FRAME_EXTENTS. However, it's not hard to get the raw frame extents:

  GtkBorder extents;
  GdkRectangle client_rect, frame_rect;

  extents.left   = client_rect.x - frame_rect.x;
  extents.top    = client_rect.y - frame_rect.y;
  extents.right  = frame_rect.width - client_rect.width - extents.left;
  extents.bottom = frame_rect.height - client_rect.height - extents.top;

::: gnome-screenshot/screenshot-utils.c
@@ +804,3 @@
               rec_y = rectangles[i].y;
+              rec_width = rectangles[i].width - 2 * (real_coords.x - wm_real_coords.x);
+              rec_height = rectangles[i].height - 2 * (real_coords.y - wm_real_coords.y);

There is no guarantee that mutter will extend both sides with the same amount of invisible border. In fact, the height here is wrong, since mutter will use the titlebar height for the top and 1px for the bottom.

Created attachment 197305 [details] [review]
screenshot: compensate invisible window borders using frame extents

In order to grab a screenshot of a window we currently peek at the WM
decorated X window size and coordinates, and cut a pixbuf from the root
window using those dimensions.

This unfortunately breaks with invisible window borders, as the WM
window is actually bigger than expected, resulting in an unwanted frame
around the image. Mutter now stores the real dimensions of the window
decorated frame in _NET_FRAME_EXTENTS, so we can just use that
information to get the actual region to cut.

Mutter also stopped setting the XShape bounding box on rounded corners,
as they're now properly antialiased. Our code that takes the XShape
rectangles and makes the area outside of their bounding box transparent
also needs to be updated to handle the case where the WM decorated window
size is different from the frame extents.

Tested with GNOME Shell/Mutter and Metacity.

(In reply to comment #2)

> There is no guarantee that mutter will extend both sides with the same amount
> of invisible border. In fact, the height here is wrong, since mutter will use
> the titlebar height for the top and 1px for the bottom.

Okay, makes sense; I attached an updated patch which should fix the issues mentioned in your review, and also fixes the position of the drawn cursor to take invisible borders into account.

Attachment 197305 [details] pushed as 68530e4 - screenshot: compensate invisible window borders using frame extents

Pushed to master after r-t approval.