gnome-shell isn't involved until the application establishes a connection to the display server, at which point it's too late. I doubt there is a fix for this (other than manually passing GDK_BACKEND=x11 to force the X11 backend), but let's see what the GTK+ developers think.

sorry for mixing up gnome-shell and gnome-session. the real command i used is:
  dbus-run-session -- gnome-shell --display-server --wayland

Given that X11 is available in Wayland (using Xwayland), gtk+ apps which support both Wayland and X11 must prefer Wayland over X11 when both are available, otherwise X11 would be always selected even on Wayland....

strange. i thought there is only two variables which decide what to do: DISPLAY and WAYLAND_DISPLAY. if both is set, put it on WAYLAND_DISPLAY. if only one is set, put it there.

how would i send a gnome-terminal to another DISPLAY?

the startup is slightly different now acc to florian's suggestion in bug774774. i start gnome_shell now on tty0 with:
  export QT_QPA_PLATFORM=wayland
  XDG_SESSION_TYPE=wayland dbus-run-session gnome-session
  (WAYLAND_DISPLAY=wayland-0, DISPLAY=:0)
on tty1 i start X with:
  startx 
  (WAYLAND_DISPLAY not set, DISPLAY=:1)
on tty2, i only log into command line and set the DISPLAY to do "remote":
  (DISPLAY=:1)
along the lines documented here: https://wiki.archlinux.org/index.php/GNOME#Manually. 

test with totem: started on tty0, it ends up on tty0, started on tty1 it ends up on tty0. started on tty2 it dumps a core.

test with gnome-terminal: no matter where started, it ends up on tty0. it says "warning: DESKTOP_STARTUP_ID not set and no fallback available" on tty1, and tty2.

i forgot, if nothing is set, no WAYLAND_DISPLAY, no DISPLAY, put it on wayland-0. this is valid for applications running on both, wayland and X. for applications only supporting wayland this might be different by only considering WAYLAND_DISPLAY.

olivier, can you please elaborate how i can send a gnome terminal to a remote X if it does not respect the dispaly variable please? currently i need to install xterm to do this.

gnome-terminal respects the X11 display value, but gnome-terminal is a bit special in the sense that it runs as a daemon so that particular daemon cannot inherit from the DISPLAY value that you set when invoking gnome-terminal (being started before you set env. the variable...)

So for example, this works:

 1. $ # Run an X11 display nested on :10 for example
    $ Xephyr :10 &

 2. $  # Run gnome-terminal-server using X11 backedn on DISPLAY :10 that we just started
    $ (GDK_BACKEND=x11 DISPLAY=:10 /usr/libexec/gnome-terminal-server --app-id my.foo.Terminal &)

 3. $ # Run gnome-terminal on the same display :10 using X11 backend connecting to the g-t-s tath we just started.
    $ (GDK_BACKEND=x11 DISPLAY=:10 gnome-terminal --app-id my.foo.Terminal)

I have to agree that this is almost entirely due to the way gnome-terminal does its client-server stuff

Well, comment 4 says this also affects totem, and I'd actually expect this to affect every application that is dbus activated.

(The reporter says he uses startx, but should note that only a gdm-initiated gnome-session is supported by gnome-terminal. But this doesn't matter here since) The problem there is that there is only one session dbus, and that session dbus' activation environment carries the DISPLAY variable that g-t-server gets and where it will open its windows on.

Now g-t-server does have some code to forward the DISPLAY from the client (gnome-terminal) to g-t-server and try to open the window on that display, but evidently it's broken (since it's not been tested in a long time). And even if it was made to work again, the problem is that as soon as a display vanishes, g-t-server will abort since gdk_display_close() still does not work after all these years.

One possible solution would be to start one g-t-server per DISPLAY (and not take the DISPLAY from the dbus activation environment anymore), but that doesn't work with dbus activation (since the app id/well-known name would have to be derived from the display name, and dbus activation doesn't support this...

as designed, see https://gitlab.gnome.org/GNOME/gtk/issues/1390#note_344758