GNOME Bugzilla – Bug 794369
ssh-agent errors with concurrent SSH connections
Last modified: 2018-03-19 11:37:50 UTC
Version: 3.28 Since the update from 3.26 to 3.28 I'm getting lot's of errors like this: mr update: /home/michael/git/filemanager-actions sign_and_send_pubkey: signing failed: communication with agent failed mbiebl@git.gnome.org: Permission denied (publickey,gssapi-keyex,gssapi-with-mic). fatal: Could not read from remote repository. I use "mr" to update multiple git repositories. My .mrconfig has "jobs = 5", i.e. there can be up to 5 concurrent SSH processes. If I set that to "jobs = 1", the problem goes away. So I suspect that there is an issue with concurrent access to the SSH agent. If I run ssh-agent manually (say on the console via `eval ssh-agent`; ssh-add ..) I do not get these errors.
Running with GIT_SSH_COMMAND="ssh -v", I get mr update: /home/michael/git/gnome-logs OpenSSH_7.6p1 Debian-4, OpenSSL 1.0.2n 7 Dec 2017 debug1: Reading configuration data /home/michael/.ssh/config debug1: /home/michael/.ssh/config line 17: Applying options for *.gnome.org debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug1: Connecting to git.gnome.org [209.132.180.184] port 22. debug1: Connection established. debug1: identity file /home/michael/.ssh/id_rsa.2 type 0 debug1: key_load_public: No such file or directory debug1: identity file /home/michael/.ssh/id_rsa.2-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Debian-4 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3 debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000 debug1: Authenticating to git.gnome.org:22 as 'mbiebl' debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: diffie-hellman-group-exchange-sha256 debug1: kex: host key algorithm: ssh-rsa debug1: kex: server->client cipher: aes128-ctr MAC: umac-64@openssh.com compression: none debug1: kex: client->server cipher: aes128-ctr MAC: umac-64@openssh.com compression: none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<3072<8192) sent debug1: got SSH2_MSG_KEX_DH_GEX_GROUP debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: got SSH2_MSG_KEX_DH_GEX_REPLY debug1: Server host key: ssh-rsa SHA256:hgcTgLDMDgXKNFlp9ymYPDc2fDVX3R5hW7hc9fBWoiQ debug1: Host 'git.gnome.org' is known and matches the RSA host key. debug1: Found key in /home/michael/.ssh/known_hosts:307 debug1: rekey after 4294967296 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey after 4294967296 blocks debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Next authentication method: gssapi-keyex debug1: No valid Key exchange context debug1: Next authentication method: gssapi-with-mic debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000) debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000) debug1: Next authentication method: publickey debug1: Offering public key: RSA SHA256:qc9VPWoSDep5sLRmvBQEVccmCHXxlkvWfs7E8/9VVvY /home/michael/.ssh/id_rsa.2 debug1: Server accepts key: pkalg ssh-rsa blen 277 debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: No more authentication methods to try. mbiebl@git.gnome.org: Permission denied (publickey,gssapi-keyex,gssapi-with-mic). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. mr update: command failed The relevant config from ~/.ssh/config Host *.gnome.org User mbiebl IdentityFile ~/.ssh/id_rsa.2
Not sure if it's related, but I see a lot of those error messages in the journal: Mär 15 23:58:10 pluto gnome-keyring-daemon[1555]: couldn't read from client: invalid packet size 0 Mär 15 23:58:14 pluto gnome-keyring-daemon[1555]: couldn't read from client: invalid packet size 0 Mär 15 23:58:19 pluto gnome-keyring-daemon[1555]: couldn't read from client: invalid packet size 0 Mär 15 23:58:24 pluto gnome-keyring-daemon[1555]: couldn't read from client: invalid packet size 0
And also Mär 16 00:04:05 pluto gnome-keyring-daemon[1557]: couldn't handle client request: Stream has outstanding operation
As well as Mär 16 00:05:30 pluto gnome-keyring-daemon[1557]: GLib-GIO: g_input_stream_clear_pending: assertion 'G_IS_INPUT_STREAM (stream)' failed Mär 16 00:05:30 pluto gnome-keyring-daemon[1557]: couldn't handle client request: Error receiving data: Socket operation on non-socket
Indeed, I hadn't thought of that case. As gnome-keyring internally keeps only one connection to ssh-agent, the fix would be either: - serialize the transactions so they are not intermixed on a single connection - actually open multiple connection to ssh-agent While I guess the former would be simpler, the latter would be robuster. I will come up with a fix soon.
Created attachment 369775 [details] [review] ssh-agent: Make EOF handling robuster
Created attachment 369776 [details] [review] ssh-agent: Allow opening multiple connections to inferior ssh-agent Previously, it keeps only one connection to the inferior ssh-agent process. That prevented simultaneous access to gnome-keyring's ssh-agent service. With this patch, it always opens a new connection to the inferior ssh-agent process when a new client connects.
The patch in comment 6 just makes diagnostics better (i.e. not printing misleading "invalid packet size 0"). The other patch should fix the problem in the way described in comment 5.
Thanks, the problem with the multiple connections seems to be gone and I can now run "mr up" with "jobs = 5". I still get a lot of error/warning messages though: Mär 16 14:06:51 pluto gnome-keyring-daemon[9019]: couldn't read from client: connection closed by peer $ journalctl -b | grep gnome-keyring-daemon | grep peer | wc -l 187
Created attachment 369782 [details] [review] ssh-agent: Make EOF handling robuster -- This should really suppress the warning.
The warning messages are gone, thanks.
Attachment 369776 [details] pushed as a19966b - ssh-agent: Allow opening multiple connections to inferior ssh-agent Attachment 369782 [details] pushed as 0db5a03 - ssh-agent: Make EOF handling robuster