When working with one, large primary monitor and a couple smaller monitors, one tends to focus on the primary monitor. It is there that a web browser, terminal, editor and day-to-day work is performed. The outlying monitors tend to be reference, chat systems, email, etc. Those things are only monitored and aren't often used nor toggled between. When changing between desktops, those chat windows sometimes take priority over the primary display.

Let's consider the setup of two monitors, each with a limited number of apps:

* Primary Monitor
    * Desktop 1: full-screen web browser, with debug toolbars open.
    * Desktop 2: Four terminal windows

* Secondary Monitor
    * Single, spanned desktop: chat client.

If you focus the chat client then the web browser, you can alt-tab as expected. You leave desktop 1 with the web browser focused and go to desktop 2.

On Desktop 2 you type a bit in several terminals. A chat message comes in and you select the chat program on the Secondary Monitor to reply, then go back to typing in terminals. You essentially bounce back and forth between two or three terminals.

When I use hotkeys to go back to Desktop 1 I would assume that the browser is still selected because that's how I left it. Instead, the chat window becomes selected.

I don't know how the most recently used window is tracked, but it would be wonderful if it was tracked per-desktop.