GNOME Bugzilla – Bug 629184
gvfs-1.6.3 using openssh-5.6 fails to open sftp with "Error reading from Unix: Input/output error"
Last modified: 2010-09-27 15:15:10 UTC
Steps to replicate: 1. Install gvfs-1.6.3 2. Install openssh-5.6 3. gvfs-mount sftp://1.2.3.4/ Expected results: A prompt asking for a password. Actual results: Error mounting location: Error reading from Unix: Input/output error Additional details The same error is returned when mounting through nautilus, so this isn't an issue with gvfs-mount. I've started trying to debug the problem. It occurs with both publickey and password based authentication. Downgrading to openssh-5.5 immediately solves the problem, and everything returns to normal (without any dbus restarts, or additional messing around). I've done some additional debugging and discovered that the problem doesn't occur when stepping through gvfsd-sftp's handle_login function under gdb, suggesting that it's a timing issue of some kind. An strace of the processes showed that both openssh 5.5 and 5.6 start-up and write out a password prompt. I noticed that 5.6 now opens /proc/<pid>/fd and closes one more file descriptor that in 5.5, which is the best guess I've got as to where the problem might be. Otherwise the system calls made by 5.6 and 5.5 seem pretty much the same. I'm happy to help debug/test out patches/provide information, but I don't know what else to look for... Additional links: http://bugs.gentoo.org/335316 http://bugs.archlinux.org/task/20724
how did you manage to run gdb on gvfsd-sftp?
I moved /usr/libexec/gvfsd-sftp to /usr/libexec/gvfsd-sftp.bin. Then I made a small script that ran: #!/bin/bash gdbserver localhost:12345 /usr/libexec/gvfsd-sftp.bin "$@" Then before running the mount command, I started gdb watching for the remote endpoint with: gdb > file /usr/libexec/gvfsd-sftp.bin > b do_handle > target remote localhost:12345 That'll wait until the remote endpoint kicks in, and then let you continue until you hit the predefined breakpoint (do_handle in this instance).
If you set the breakpoint at gvfsbackendsftp.c:895, you see that g_input_stream_read at line 891 returns -1 which results in the failure
A workaround is to remove line 75 > #define USE_PTY 1 Or somehow not define HAVE_GRANTPT via the build system. I don't know how portable this is but it works fine on Arch Linux at least.
I tried simply removing the #define, but now get: $ gvfs-mount sftp://192.168.0.2 Error mounting location: Permission denied Oddly, this did seem to work on a reinstall immediately after patching, but once the machine's been restarted, it doesn't work. I can't tell if that's a timing issue, or if it's a consolekit/policykit issue, or whether it's just something else that's always been there. I'll try and do a bit more debugging later on, but if anyone has any ideas, I'm happy to test out patches, etc...
Removing the define also breaks openssh-5.5_p1 (same error, permission denied), re-adding the define solves the problem again. I've still no idea what's causing it though...
Sorry for the spam, but I just determined that the permission denied error only occurs on password (keyboard-interactive) authentication, not on publickey authentication (this is how my initial tests succeeded, but after reboot when I got home they failed). So as a quick summary: 5.5 w/o USE_PTY keyboard-interactive fails, publickey works 5.5 with USE_PTY both publickey, keyboard-interactive work 5.6 w/o USE_PTY keyboard-interactive fails, publickey works 5.6 with USE_PTY both publickey, keyboard-interactive fail
Same here - only worked for me because of public key auth.
Shouldn't this whole thing be using something like libssh anyway?
Created attachment 170394 [details] [review] use poll() instead of select()
Comment on attachment 170394 [details] [review] use poll() instead of select() When using poll() instead of select(), one can see that there is a POLLHUP event on tty_fd which maybe causes select() to return too early. Using poll() and waiting for POLLIN works however - see attached patch.
Created attachment 170407 [details] [review] use poll() instead of select() unified diff Thanks very much Otto. I can confirm the patch works for me here (on both publickey and password based authentication). I've reformatted the patch as a unified diff for easier appplication to the source code.
Nice catch, Otto! I've been dealing with this in our downstream bug https://bugzilla.redhat.com/show_bug.cgi?id=629322 but never spotted the POLLHUP event. This patch works fine and affected code is only used for authentication, shouldn't affect real protocol or data transfers. About the TODO: I've looked into sshfs sources (2.2 release) and can see that pty_expect_loop() reads password prompt from ptyfd, just like we do. Moreover the patch is consistent with sshfs code.
Committed to master, included in 1.6.4 release. commit 4e8e85c9b86ac1fa8f333ffd30e5961f9b493bc1 Author: Tomas Bzatek <tbzatek@redhat.com> Date: Mon Sep 27 16:10:07 2010 +0200 sftp: Use poll() to cope with openssh-5.6 changes Patch by Otto Allmendinger. See bug 629184 for details