Created attachment 170725 [details] [review]
Patch

This patch does a couple things:
 * Changes the return value of gdk_screen_get_default_colormap() to return an RGBA colormap that is different from the one returned by gdk_screen_get_rgba_colormap().  This is intended to be used to build a mask of the regions of the window which were explicitly asked to be RGBA by the application.  We can then use this as an optimization hint for the compositing manager.
 * It installs an X error trap handler that checks for BadMatch errors.  When it finds one, it attempts to reset the window's colormap using gdk_screen_get_system_colormap().  This fixes all the crashers that I am aware of, except for one which I was unable to verify (Nautilus when using nvidia TwinView).

TODO:
 * Build the optimization region (this probably also needs some coordination with the theme system if we want to allow themes to specify RGBA colors), and export that to the compositing manager.

http://www.mail-archive.com/wm-spec-list@gnome.org/msg00740.html

For discussion on wm-spec-list of the coordination between GTK+ and the compositing manager.  This needs to be picked up and resolved still.

Review of attachment 170725 [details] [review]:

::: gdk/x11/gdkwindow-x11.c
@@ +821,3 @@
+                                            0, private->depth, class, xvisual,
+                                            xattributes_mask, &xattributes);
+    }

BadMatch can be returned for many mismatches, not just colormaps. Some ideas:

- only install the trap when the colormap is rgba

- only try again if the colormap is rgba


Also, with recent error trap improvements, you no longer need to flush

*** Bug 516885 has been marked as a duplicate of this bug. ***

*** Bug 317361 has been marked as a duplicate of this bug. ***

Created attachment 267008 [details] [review]
gdk: enable rgba visuals by default

Unless GDK_PREFER_RGBA=0 is set.

Review of attachment 267008 [details] [review]:

I'm fighting a (losing) battle to keep all environment variables interpreted by gdk/gtk listed in the docs.
Would be great to add this one to the list.

::: gdk/x11/gdkvisual-x11.c
@@ +286,3 @@
     }
 
+  /* prefer the RGBA visual unless CSD is off */

The comment is a bit misleading, now that this is a separate variable.

@@ +289,3 @@
+  if (x11_screen->rgba_visual)
+    {
+      const gchar *csd_env = g_getenv ("GDK_PREFER_RGBA");

As is the variable name here.

Looking at some of the duped bugs, there's been an environment variable named GTK_RGBA_VISUAL. I think I'd prefer to either stay with the old name, or use 
something shorter, like

GDK_RGBA=0

Please note that my patches on duplicate bug #317361 go slightly beyond this: they also sets an additional class on menu windows to be able to construct themes with rounded corners on menus. (OTOH, perhaps that isn't necessary in current Gtk - I was looking at 3.8.)

Attachment 267008 [details] pushed as a2ccb66 - gdk: enable rgba visuals by default

This needs to be reverted, because it causes emacs (and other apps using libX11 directly) to receive a BadMatch trying to create a Render Picture with depth 24.

I'm not sure we can let emacs dictate the direction of GTK+. Do you know of other examples where this is happening ?

(In reply to comment #11)
> I'm not sure we can let emacs dictate the direction of GTK+. Do you know of
> other examples where this is happening ?

No, but:
1) gtk+ is supposed to be ABI/API stable, and that implies that apps that were running on an older version should keep running as the library is updated
2) emacs is kind of special because it's used by a good percentage of the gtk+/gnome developers, and a working developer environment is a prerequisite for having a direction in gtk+ at all

Btw, the crash is actually somewhere in Xft, so it's possible that all apps using Xft are affected.

(In reply to comment #12)
> (In reply to comment #11)
> > I'm not sure we can let emacs dictate the direction of GTK+. Do you know of
> > other examples where this is happening ?
> 
> No, but:
> 1) gtk+ is supposed to be ABI/API stable, and that implies that apps that were
> running on an older version should keep running as the library is updated

while this is nice and good, if an application is mixing Xlibs and GDK calls then there should be no expectation of things working flawlessly across released of GTK. going behind the back of the toolkit falls underneath the expectation of "undefined behaviour". we can fix the toolkit, but we should fix the application as well, as it happened in the past (Eclipse, SWT, etc.).

Of course, we'll try to make this work as much as possible - I would personally be open to a strcmp(g_get_prgname(), "emacs") level hack to keep this working.

Poking at the emacs code a little made me lose almost all hope... emacs is picking a visual entirely unrelated to the visual used by GTK+, just from looking at X resources and falling back to the default visual of the X display.

      dpyinfo->visual = DefaultVisualOfScreen (screen);


So, this is simply not true if the window is not created with that visual:

/* This is the Visual which frame F is on.  */
#define FRAME_X_VISUAL(f) FRAME_X_DISPLAY_INFO (f)->visual


In the short term, setting GDK_RGBA=0 in the environment may be the best workaround.

this has been reverted

nevertheless, csd windows are using an rgba visual by default, so this is essentially a solved problem.