This might get flagged as a justification for off-line upgrades.
</drive-by-comment>

I have managed to hit this a few times when I randomly build a few components (say the gtk+ stack) from master (without jhbuild) in a way that it leaks into the system's prefix. I would be happy if it didn't happen, but I also accept the risks.

Note that GNOME doesn't necessarily need to be *fully* functional after an upgrade (though that'd be nice); it does, however, need to be functional enough to unlock the session.  (For the rest, ideally it could display a "you need to restart your session" notification or similar.)

You really shouldn't be doing online upgrades. Besides all the race conditions in there, your system should always be running one version or another, not half-and-half.

(In reply to Jasper St. Pierre from comment #3)
> You really shouldn't be doing online upgrades. Besides all the race
> conditions in there, your system should always be running one version or
> another, not half-and-half.

In theory, all the services in my session should have been the old version, unless something new gets started or something gets restarted.  Which doesn't explain why the unlock failed.

In theory, yes. In practice, we fork/exec several helper binaries during the unlock / authentication system, simply because of how PAM works.

(In reply to Jasper St. Pierre from comment #5)
> In theory, yes. In practice, we fork/exec several helper binaries during the
> unlock / authentication system, simply because of how PAM works.

The interface to those helper binaries could be made forward-compatible, though; they should be incredibly simple.

Sure. You can say all those things. I can also say "don't run two different versions of GNOME on your system".

We try to make things forward-compatible, we really do, but here we apparently failed. Note that you're skipping over a stable version in the middle.

From my perspective, we have a bug that we can really only reproduce once per system, and it's resolved by simply rebooting your computer so that you're running the new version of GNOME.

So unless someone gives me some detailed instructions about how to reproduce that doesn't involve me reinstalling my OS every time, it's going to be a bit tough for me to even figure out what happened.

(In reply to Jasper St. Pierre from comment #7)
> Sure. You can say all those things. I can also say "don't run two different
> versions of GNOME on your system".

I didn't.  I ran one version, then upgraded to a new version.

I don't expect GNOME to be fully functional after an upgrade.  There's a reason why there are packages designed to provide "you need to restart" notifications.  However, it doesn't seem unreasonable to expect GNOME to have enough functionality to allow unlocking the screen to *see* that notification and act on it, and to gracefully close programs and save work rather than having to abnormally terminate the session.

> We try to make things forward-compatible, we really do, but here we
> apparently failed. Note that you're skipping over a stable version in the
> middle.
> 
> From my perspective, we have a bug that we can really only reproduce once
> per system, and it's resolved by simply rebooting your computer so that
> you're running the new version of GNOME.
> 
> So unless someone gives me some detailed instructions about how to reproduce
> that doesn't involve me reinstalling my OS every time, it's going to be a
> bit tough for me to even figure out what happened.

Virtual machines and kvm's -snapshot option should make it much easier to repeatedly reproduce.

I just encountered this problem again, and in addition to the repeated authentication failure messages, I also saw a message on the lock screen matching this one I later found in logs:

gnome-session[551]: (gnome-shell:956): Gjs-WARNING **: JS ERROR: Failed to start verification for user: Gio.DBusError: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: Type of message, '(sssssssb)', does not match expected type '(sssssssbb)'

This occurred after the following package upgrades prior to locking the screen:

[UPGRADE] gir1.2-gtk-3.0:amd64 3.18.1-1 -> 3.18.1-2
[UPGRADE] libgail-3-0:amd64 3.18.1-1 -> 3.18.1-2
[UPGRADE] libgtk-3-0:amd64 3.18.1-1 -> 3.18.1-2
[UPGRADE] libgtk-3-bin:amd64 3.18.1-1 -> 3.18.1-2
[UPGRADE] libgtk-3-common:amd64 3.18.1-1 -> 3.18.1-2
[UPGRADE] gdm3:amd64 3.14.2-2 -> 3.18.0-1
[UPGRADE] gir1.2-gdm3:amd64 3.14.2-2 -> 3.18.0-1
[UPGRADE] libgdm1:amd64 3.14.2-2 -> 3.18.0-1

I'd guess the upgrade of gdm3 caused this issue.

(In reply to Josh Triplett from comment #0)
> gnome-shell needs to be able to unlock the session after a GNOME upgrade.

Unfortunately we do not support online upgrades anymore, not for several years now, for reasons that have been discussed at length on mailing lists. Upgrading from one GNOME version to another while GNOME is running is simply not expected to work, and it almost always goes wrong in my experience. If you really want to upgrade via command line, I would log out, switch to a virtual terminal, do the upgrade, then reboot. If Debian really wants to support this, I would track issues in downstream bugs.

Anyway, although this wasn't true at the time you filed this bug, we now expect users to upgrade via GNOME Software, offline, where issues like this will never occur.

(In reply to Debarshi Ray from comment #1)
> This might get flagged as a justification for off-line upgrades.
> </drive-by-comment>

Prescient. ;)

(In reply to Michael Catanzaro from comment #10)
> (In reply to Josh Triplett from comment #0)
> > gnome-shell needs to be able to unlock the session after a GNOME upgrade.
> 
> Unfortunately we do not support online upgrades anymore

I find it really disappointing that being locked out of the current session is not considered a bug.  I can understand not catching every instance where upgrading leads to misbehavior, but locking the user out of their session entirely seems unreasonable.

Our perspective is that upgrading GNOME while it's running is unreasonable... honestly, I'm really surprised you thought that might work. :(

Since our upgrade tool, GNOME Software, ensures this never happens, it's really a Debian problem, sorry.

(In reply to Michael Catanzaro from comment #12)
> Our perspective is that upgrading GNOME while it's running is
> unreasonable... honestly, I'm really surprised you thought that might work.
> :(

GNOME has (or had) a built-in mechanism that provided a notification when an upgrade may potentially require restarting the session or rebooting.  And such a requirement seems fine.  My only expectation was that it should be possible, after upgrading GNOME, to 1) unlock the session, 2) see that notification if any, and 3) log out / reboot.

> Since our upgrade tool, GNOME Software, ensures this never happens, it's
> really a Debian problem, sorry.

That hasn't always been the case; online upgrades have worked before, and the previous incarnation of software management tools built on PackageKit supported that.

(In reply to Josh Triplett from comment #13)
> (In reply to Michael Catanzaro from comment #12)
> > Since our upgrade tool, GNOME Software, ensures this never happens, it's
> > really a Debian problem, sorry.
> 
> That hasn't always been the case; online upgrades have worked before, and
> the previous incarnation of software management tools built on PackageKit
> supported that.

They supported it in the sense that they applied heuristics to figure out which components would be affected by the update and needed restarting. But those heuristics were never perfect and missed a lot of stuff (plugins, DBus APIs, parameters of helper binaries etc.), which is why they have been abandoned in favor of offline updates. For what it's worth, this issue is likely one of those cases that were never caught by the old heuristics either ...


> My only expectation was that it should be
> possible, after upgrading GNOME, to 1) unlock the session

That means you are expecting that all internal APIs involved in unlocking stay fully backward compatible over multiple stable GNOME versions (because your expectation is not about GNOME x -> GNOME x+1, but about Debian x -> Debian x+1). Sorry, but we do not support mixing different versions of core components, lest mixing binaries from different versions of the same component. If you really want to support that in Debian, it's up to you to make it work - at least "make sure components from Debian X keep working with components from Debian X+1" is less complex than "make sure components from GNOME X keep working with components from any previous GNOME versions".

Or tell users to use loginctl to unlock their session if they insist on updating from a live session despite warnings against it ...

> That means you are expecting that all internal APIs involved in unlocking stay fully backward compatible over multiple stable GNOME versions

Or that gnome-shell, specifically, know how to make a few calls to a range of older versions, or know how to unlock the screen without making those calls.  Or that some emergency fallback exists that can run a PAM transaction and then spawn loginctl.  Or, failing all of that, that gnome-shell could catch exceptions during screen unlock and display something meaningful to the user that tells them they need to log in at the console and run some command to unlock the screen.

> Or tell users to use loginctl to unlock their session if they insist on updating from a live session despite warnings against it ...

Users run "apt-get dist-upgrade" or similar as they always do; most users don't see any such warnings.  The release notes talk about not upgrading from a session run by gdm due to that possibly getting terminated, but that doesn't even happen anymore.  The release notes don't talk about not upgrading from a GNOME session in general.

(In reply to Josh Triplett from comment #15)
> Or that gnome-shell, specifically, know how to make a few calls to a range
> of older versions, 
wouldn't help. the gnome-shell you're running is old, not new. I think what's probably happening here is the old GDM is still running but it runs a new gdm-session-worker helper process for reauthentication.  The new session worker expects a slightly different dbus api so it balks (just a guess).  if that's what's happening, it's because of a change in a private api.  I guess i could have changed org.gnome.DisplayManager.Worker to org.gnome.DisplayManager.Worker2 and handled both from gdm-session.c but that's a lot of baggage for an unsupported configuration (and how long would that baggage have to linger?).

> or know how to unlock the screen without making those
> calls.  Or that some emergency fallback exists that can run a PAM
> transaction and then spawn loginctl.  
having a fall back implementation is a bad idea.  it means duplicated code that rarely gets run. That code will bitrot and then one day when it does get run by someone it won't work.  Having rarely tested, bolted on, fall back code for unlocking your session is a maintenance and security nightmare.

> Or, failing all of that, that
> gnome-shell could catch exceptions during screen unlock and display
> something meaningful to the user that tells them they need to log in at the
> console and run some command to unlock the screen.
At the point the user has to do that, we've (and they've) already lost.  We should have told them not to do the live update in the first place !  it's not like unlock is the only thing that breaks...it breaks firefox and a host of other things. Sure a lot of stuff happens to keep working mostly, and some stuff only breaks in subtle ways that are hard to attribute to the upgrade, but I think there may only be one or two desktop applications in the wild that explicitly support getting upgraded while being run.

i guess one thing that might be worthwhile to implement is making GDM restartable without taking the sessions it's managing.