Created attachment 80920 [details]
kill metacity in 5 easy steps

Created attachment 80945 [details] [review]
Stop this killing metacity

What's happening here is a bug in gdk, I suppose. If you try to call gdk_window_begin_paint_rect with an insanely huge rectangle, it dies with BadAlloc, as you know from your own program; since metacity uses gtk to paint the windows, it itself will attempt to call gdk_window_begin_paint_rect with an insanely huge rectangle, and die with BadAlloc in the process. (You'll need to set METACITY_SYNC=1 to see this.) That's why other WMs don't have a problem here: they don't use gdk.

So, what can we do about it? I don't know. The attached is a patch to make metacity complain and not decorate the window if it can't do so because of an X error when calling gdk_window_begin_paint_rect. I don't know whether this is the best solution. Elijah, Havoc, others-- any thoughts?

s/paint the windows/decorate the windows/, you know what I meant.

One possible fix would be to clip the painting to the screen extents (there is a race condition here - the window could move in relation to the screen while drawing - maybe one fix for that would be to clip to a bit larger area than the screen, and skip the clipping for windows smaller than the screen, so only large windows that get moved a lot would repaint incorrectly)

I think metacity might already draw the frame as four stripes on the four sides instead of one big rect, which was also a past attempt to fix this bug if so...

Created attachment 81243 [details] [review]
Clamp the rectangle to an area slightly larger than the screen

Good plan. So, here's a patch which clamps the rectangle values not to stray >200px outside the edges of the screen.

While I was working on that, I noticed that we carry out all these calculations for every drawable and then throw them away 1/3 of the time when GDK_IS_WINDOW is false. I refactored the code a little bit to only do these calculations when GDK_IS_WINDOW is true.

Results: The kill.py program no longer kills metacity. Also, window decoration is slightly faster (about 0.1ms for a small simple window, but it all adds up).

Without the patch, metacity-theme-viewer says:

Drew 100 frames in 0.07 client-side seconds (0.7 milliseconds per frame) and 0.493933 seconds wall clock time including X server resources (4.93933 milliseconds per frame)

With the patch:

Drew 100 frames in 0.08 client-side seconds (0.8 milliseconds per frame) and 0.486864 seconds wall clock time including X server resources (4.86864 milliseconds per frame)

I haven't looked very closely at the patch, but considering the source and the long list of patches needing review...  :)

Great, thanks. Committed.

Assuming Thomas meant to mark this bug fixed too, since he didn't ask for testing/verification.  ;-)