After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 155872 - Slow transfers with sftp
Slow transfers with sftp
Product: gnome-vfs
Classification: Deprecated
Component: Module: ssh/sftp
Other All
: Normal normal
: ---
Assigned To: Christian Neumair
gnome-vfs maintainers
: 142022 159420 (view as bug list)
Depends on:
Reported: 2004-10-19 18:45 UTC by Kristian Berg
Modified: 2006-07-13 16:10 UTC
See Also:
GNOME target: ---
GNOME version: 2.13/2.14

GnomeVFSXFer, sftp performance patch (8.87 KB, patch)
2006-03-12 15:13 UTC, Christian Neumair
reviewed Details | Review

Description Kristian Berg 2004-10-19 18:45:57 UTC
1. Open two nautilus windows.
2. Open a remote location with sftp:// in one
3. Drag and drop a file from local to remote.

I tested this between my laptop and dekstop box, both on the same 100mbit switch. Further I tested 
it between my desktop and a remote server available through a 10mbit network. 

Between laptop and desktop, it used 6 minutes on a 350mib file. Doing the same filetransfer in the 
commandline took 45 seconds, with speeds varying between 11mib/s to 7mib/s.

Between the remote server and the desktop, a 15mib file had an estimated transfertime of 30 
minutes. I didn't complete this, but instead user scp from the commandline and completed the 
transfer in 20 seconds.
Comment 1 Kristian Berg 2004-10-19 18:46:53 UTC
The same results was achieved running the test between two computers using ubuntu linux.
Comment 2 Sebastien Bacher 2005-02-03 13:59:22 UTC
*** Bug 159420 has been marked as a duplicate of this bug. ***
Comment 3 Sebastien Bacher 2005-02-03 14:00:24 UTC
from #159420:

"1) Open Nautilus
2) File > Connect to Server
3) Enter the connection details:
- Service type: SSH
- Host:
- Port: 22
- Name used for connection: my username

Then the connection succeded but when i drag and drop files:

File 670M
scp: 9.8M/s   (01:24 mins).
nautilus: 1.3M/s  (08:18 mins) 

File 25M
scp: 10.2M/s  (00:03 seconds)
nautilus: 1.3M/s  (00:40 seconds) 

I did not notice any problem for the FTP."
Comment 4 Mark Nelson 2005-02-06 04:37:09 UTC
I am seeing similar behavior using nautilus 2.8.2 and libgnomevfs 2.8.3-11 in
debian unstable.  Using either smb or sftp server connections, transfers seem to
be going roughly 10 times slower than scp from the commandline.  It's slow
enough that streaming video over a 55Mb/s wireless connection in totem is
causing skipping.
Comment 5 Christian Neumair 2005-05-02 07:56:16 UTC
Just some ideas:
My guess would be that inside the sftp module, buffer_check_alloc is
reallocating too often. atomic_io looks pretty straightforward - I can't hint at
any other culprit which is used at buffer_write/buffer_send time.
Comment 6 Christian Neumair 2005-05-09 16:25:54 UTC
*** Bug 142022 has been marked as a duplicate of this bug. ***
Comment 7 Paul Ionescu 2005-06-20 13:55:46 UTC
This is present in FC4 too
Comment 8 Christian Kellner 2005-06-23 12:14:52 UTC
Did anybody try to use gnomevfs-copy and see if this is also that slow? If yes
please report if now can you try? Thanks!
Comment 9 Zbigniew Chyla 2005-07-28 19:13:39 UTC
I've just tried copying single 133 MiB file between two hosts connected with
Fast Ethernet:
 - gnomevfs-copy: 115 seconds
 - scp: 21 seconds
So it looks like it has nothing to do with Nautilus, something is very slow at
gnome-vfs level.
(using gnome-vfs2 2.10.1-5 from Debian unstable)
Comment 10 Timo Saarinen 2005-08-12 08:29:55 UTC
Any operation I perform with Nautilus SFTP seem to be at least 10x slower than
in command-line SFTP. Using Ubuntu Linux 5.04. It's better to use shfs instead
of Gnome-vfs until it gets usable. I just can't wait 30 seconds for File dialog
to appear...
Comment 11 Timo Saarinen 2005-08-12 11:46:26 UTC
I cleaned invalid connections from the .gtk-boorkmarks and the slowness
disappeared. Tried to reproduce the phenomenon to send a separate bug raport but
didn't succeed.
Comment 12 Christof Krüger 2005-08-25 22:01:31 UTC
I am experiencing the same problem. I get ca. 350KB/s over a line that is
capable of 10Mbit (ca. 1.2MB/s). I get full speed when using scp or sftp directly.
Comment 13 Marcelo Coelho 2006-02-05 12:15:04 UTC
Happens with Gnome 2.12.2!
Comment 14 Christof Krüger 2006-02-14 00:01:54 UTC
I can still confirm that. Is this something just some people experience or is it a general problem? It is really a pity, because its nice to mount remote shares with ssh/sftp within nautilus, but its just too slow to be fun.

Is somebody working on this? I would really appreciate it.
Comment 15 Christian Neumair 2006-03-11 11:44:10 UTC has some discussion on this. It's the sftp client's fault. Marking as NOTGNOME.
Comment 16 Murray Cumming 2006-03-11 11:53:35 UTC
But the whole point of this bug is that Nautilus is slower than using the command line, which presumably also uses the "sftp client". Maybe I'm not understanding something.
Comment 17 Paul Ionescu 2006-03-11 14:56:40 UTC

At your link is discussed the speed between  scp/sftp and ftp/rcp, which is something else than this bug.
This bug is about the speed difference between scp/sftp and GNOME/Nautilus sftp/scp client, which is huge as seen in the description of this bug.
So, please reopen the bug, as it is not yet resolved and is not the fault of openssh because the "scp/sftp" works just fine at a higher speed than Nautilus.
Comment 18 Christian Neumair 2006-03-11 16:02:43 UTC
Sorry for the bugspam, I'll look into it.
Comment 19 Christian Neumair 2006-03-12 15:13:12 UTC
Created attachment 61131 [details] [review]
GnomeVFSXFer, sftp performance patch

Discussion under
Comment 20 Christian Kellner 2006-03-12 18:15:20 UTC
I have played with that idea of adjusting the blocksize in xfer quite some time ago (patch from June 2005) but that didn't give my any noticable performance (seb128 also help'ed with the testing). Maybe it's the sftp blocksize tweaking... Are you sure you are seeing "literally by orders of magnitude". The sftp command line program doesn't do much better then we do, and if you are right, that would be awesome. I am going to have a quick look at the patch now.
Comment 21 Christian Kellner 2006-03-12 18:31:23 UTC
The patch looks fine (just a quick look though) but I can't make any sense of that line:

info->io_block_size = max_req * default_req_len;

I mean default_req_len was an arbitrary value choosen not based on any RFC, I guess. At least I couldn't find that value in it. What I have found in it is this:

     "All servers SHOULD support
      packets of at least 34000 bytes (where the packet size refers to
      the full length, including the header above).  This should allow
      for reads and writes of at most 32768 bytes."
(That's why I am using this odd value in my patch from June 2005).

So I guess I didn't correctly implement the xfer part, where your patch looks way better than my one. But I am not sure about the block_io_size for sftp yet. Can you explain that a bit more. Your explanation didn't help too much why it should exactly be 8K * 16).

Thanks for the great work though, rocking.
Comment 22 Sebastien Bacher 2006-03-12 21:05:19 UTC
Ubuntu has about that which just got that comment:

"There is a patch to this bug upstream. I recompiled libgnomevfs2 with the provided patch and the transfer speed rose to the same level as scp command line tool and was actually faster then LUFS sftp filesystem.

This is an annoying bug with a quite simple patch. It would be nice if it would be applied for dapper.

Note: scp uses the sftp protocol, so it is the same thing."
Comment 23 Christian Neumair 2006-03-12 21:39:04 UTC
Re comment 20, comment 21:

> I have played with that idea of adjusting the blocksize in xfer quite some time
> ago (patch from June 2005) but
that didn't give my any noticable performance
> Are you sure you are seeing "literally by orders of magnitude"

Yes, I am seeing that, at least on a loopback connection. Just try it out :). The sftp authors were quite aware of the performance issues, so they allowed for parallelized transfers, i.e. you request some non-overlapping (in this case contiguous) chunks, and wait for all to finish, requesting all of them "in parallel" (they requests are fired at once). You pretend to the outside world (i.e. to the VFS) to have a big block size, because the GnomeVFS code will force you to process one I/O block at once, which in fact consists of multiple parallel transfers, where foreach you request a typical size.

> ...
> I can't make any sense of that line:
> info->io_block_size = max_req * default_req_len;
> ...
> (...) I am using (32768 bytes buffer) in my patch from June 2005 (...).

You are right that it is a bit arbitrary. It's important to realize that because the GnomeVFS API works block-wise and uses a pull mechanism, i.e. requests the data separately for each of the blocks, the parallelization within this single GnomeVFS request only works when you request more than the length you pick for one sftp request. If the length of the GnomeVFS IO block is smaller or equal to the sftp request (which is what you did) it will only be handled by one of them, i.e. the inner while loop in sftp-method.c:do_read will be left after the first request.

I've picked max_req * default_req_len because my naive guess is that when using too many parallel requests there is some sort of saturation, i.e. you can't gain arbitrarily much performance, and when the max. possible number of requests is pending, you won't gain much, because you have to wait until any of the other requests is finished. Whether max_req, which is the number of max. parallel requests, should be raised, I don't know, but limiting the number of requests is recommended by the client, and transferring chunks of 0.5 MB at once sounds like a good measure for all kinds of pipes.

Sidenote: default_req_len was probably picked like this because the sftp draft you quoted suggests that it is a sane lower boundary for any read/write requests, and the fact that openssh (some sort of reference implementation) uses it qualifies it for being right by experience ;). OpenSSH seems to seek and write to the output fd as the request responses arrive, but we can't really do that because GnomeVFS doesn't have a push Xfer API (which can't be properly mapped on non-seekable FSes), i.e. both read and write are granular.

Do you have any questions on the Xfer optimization? In short, they were neccessary to handle big differences in two block I/O sizes for the I/O source and destination better, for instance writes to sftp locations are also parallelized (cf. do_write), which wouldn't work if we just considered the source I/O size, which is typically 8192 bytes for local file systems.

Re comment 22:

The patch looks trivial, but the involved machinery is a bit tricky. I'd not apply it in the distro unless it is blessed by at least one GnomeVFS maintainer, because it could break and damage all your data, and eat your children.
Comment 24 Sebastien Bacher 2006-03-12 21:47:04 UTC
No worry, I've planned to wait to get the patch applied upstream. Maybe for 2.14.1 or for next cycle ...
Comment 25 Christian Neumair 2006-07-13 16:10:40 UTC
A slightly modified patch was committed by Alex in March. Closing.