GNOME Bugzilla – Bug 363400
Nautilus / gnomevfs very slow with xfs file system
Last modified: 2007-03-30 00:27:54 UTC
Please describe the problem: It takes a very long time to copy files to xfs filesystem in nautilus. It's quick if I use cp command or other filesystems. Slow means that i need about 3 minutes for a 650mb file instead of 15 seconds while copying onto a ext3 filesystem or via command line or midnight commander onto xfs. And the problem also exists when copying via gnomefvs-copy. Steps to reproduce: 1. make a xfs filesystem and mount it to /target 2. open nautilus, copy a big file from somewhere and paste it to /target 3. Actual results: Very slow file transfer. Expected results: Look into description. Does this happen every time? Yes. Other information: The problem also exists when copying via gnomefvs-copy.
If at all, it's a gnome-vfs problem. Reassigning. According to https://launchpad.net/distros/ubuntu/+source/gnome-vfs2/+bug/58672/comments/7 this also happens in FC6.
i have the same problem. please fix the bug.
I have the same problem too
The same problem in FC6 here, gnome-vfs 2.16.0
Same problem here on gentoo ppc64 linux, gnome-vfs-2.16.2.
I have the same problem with the ntfs-3g (for fuse) driver. Writing to the NTFS partition is very slow while it is fast using command line. What I noticed is that bigger the partition is, slower it will be. For a 30 mb file, it takes about 15-30 seconds on a 20gb partition and I stopped the copy action after 20 minutes waiting on a 116 gb partition. I have this problem on Ubuntu Edgy Eft (6.10).
me too... gentoo-linux on amd64 kernel 2.6.17-gentoo-r8 gnome vfs 2.14.2-r1 nautilus 2.14.1 well for a iso file of 230MB takes... hmm... 3 guns 'n roses songs...
I got same problem in openSUSE 10.2 RC1 It's very very slow(all here is XFS), even execute applications and cause high CPU percentage. Everything's slower than in KDE. VERY VERY SLOW！ 128M file cost 4-5min(several seconds in KDE).
http://blog.51sun.cn/wp-content/uploads/2006/11/Screenshot-Copying%20files-1.png This file just 120.4MB, 3:49 remaining????
Why it's still UNCONFIRMED??
I have fix this BUG temporary, it's in libgnome/gnome-xfs-xfer.c line 1228. I set forget_cache = 0; It works good now... May be one of the functions below takes a lone time in XFS... gnome_vfs_forget_cache (source_handle, total_bytes_read, bytes_read); retry = handle_error (&result, progress, error_mode, skip);
it's in libgnome/gnome-vfs-xfer.c sorry :P Just gnome_vfs_forget_cache() is slow. Please fix this BUG as soon as possible thanks!! !!-_-
Created attachment 77527 [details] Please delete. gnome-vfs slow in XFS filesystem copy operations. Just because of forget_cache()?
I confirm low performance for gnome-vfs ... ... when copying from smb/cifs over network to ext3. Transferrate on 100 MBit network is around 4 MB/s where it sould be at least 8 MB/s (testet via "smbget"). Additional Nautilus and gnome-vfs-daemo start to eat up to 50 percent CPU-time on my iBookG4 (running on lower speed of 600 Mhz). Ubuntu 6.10 2.6.17-10-powerpc Gnome 2.16 libgnomevfs2-common 2.16.1-0ubuntu4
Has anyone ever tried to profile this so we can get an idea on how to solve this?
I'm experiencing the same problem: A 650M iso file written in 10 minutes with gnome-vfs, 30 secs with a shell cp. System: ubuntu edgy 6.10 athlon xp 3200 libgnomevfs2-common 2.16.1-0ubuntu4 I'll try to profile, if i find the time. I wonder why this is still unconfirmed btw.
I tested the same system with kernel 2.6.15 and it works flawlessly, so i suppose it's an XFS problem, probably in the code involved in the fadvice POSIX_FADV_DONTNEED call made by gnome_vfs_forget_cache when working on a local filesystem.
Hi Francesco, I'm not convinced it's actually XFS specific what causes this problem (see comment #14). Got the bad feeling it's slow code then. By the way: gnome-vfs seems to be pretty slow on debian stable, too. Can anybody confirm that? Cheers
Hi, this is ntfs-3g upstream. I can confirm that quite many people reported major slowness which were gone if they copied files from the command line or they started to use another file manager.
@szaka: Thanks to that confirmation. Seeing no significant action from gnome upstream on this issue, i contacted the maintainer of gnome-vfs-*-packages on debian. Let's see what happens.
(In reply to comment #20) > @szaka: Thanks to that confirmation. > > Seeing no significant action from gnome upstream on this issue, i contacted the > maintainer of gnome-vfs-*-packages on debian. Let's see what happens. Thanks for trolling around. But as you try to tell "the bad gnome people don't do anything" I'm going to tell you my 2 cents: From Comment #17: > Has anyone ever tried to profile this so we can get an idea on how to solve > this? You're just fooling around over here, instead of starting sysprof and providing some real numbers that we (developers) can use.
Created attachment 79228 [details] Profiling data (sysprof file) of gnome-vfs-copy from an xfs partition to an xfs partition with barriers enabled
(In reply to comment #18) I wrote about this on the XFS mailing list and some more informations surfaced. I suspect that we are facing 2 different problems here, one XFS related and one of a different kind. Regarding XFS, i can add that it only affects XFS with barriers active, remounting the filesystem with barriers disabled makes the problem vanish (barriers became active by default with 2.6.17). I suggest you take a look at the XFS mailing list thread i started for more insights; i also run sysprof while using gnomevfs-copy and all time gets spent in kernel space, i am attaching the profiling data as a sysprof file. XFS mailing list thread: http://oss.sgi.com/archives/xfs/2006-12/msg00256.html > Hi Francesco, > > I'm not convinced it's actually XFS specific what causes this problem (see > comment #14). Got the bad feeling it's slow code then. > > By the way: gnome-vfs seems to be pretty slow on debian stable, too. Can > anybody confirm that? > > > Cheers >
Thanks a lot mate; just to be sure (as you don't specify it explicitly): your profile was generated without XFS {barriers,problems}, right?
Does strace reveal any difference? Once I was reported stat() flood.
Unfortunately this isn't pretty useful; can you please install the debug symbols for glib and gnome-vfs? (and is it possible to get some for the kernel?) Having function names and not just libraries doesn't give much insight.
According to http://oss.sgi.com/archives/xfs/2006-12/msg00289.html the repeated call of posix_fadvise is very likely to be the culprit. It is only called in streaming mode (i.e. if the file is bigger than 20 MB): /* When the total copy size is over this limit, we try to * forget the data cached for the files, in order to * lower VM pressure */ #define DROP_CACHE_SIZE_LIMIT (20*1024*1024) It will be called with POSIX_FADV_DONTNEED. Assuming both the source and the target block sizes match, it will be called for each block size big chunk that has been transferred for both the source and the target file. Note that the block size is always 8192 for local files (I don't know why), so we get many calls (~> 10^6 per file!). On a sidenote: I'm not sure why we don't use POSIX_FADV_NOREUSE instead of POSIX_FADV_DONTNEED, according to "man posix_fadvise" it looks more appropriate - I don't know how the OS will precisely react, though.
Created attachment 79237 [details] tracing nautilus/gnome-vfs-daemon while copying over smb network
Trolling was not what i had in mind. But your right. I should provide more info. So back to business. Sry for the long posting. Here my profiling: -- (I) Bug 363400 Problem: Nautilus / gnomevfs very slow with xfs file system Partial Confirm: Nautilus seems slow when using gnome-vfs to copy files over smb/cifs-network. (II) Profiling Test 1: How slow exactly ist gnome-vfs on which system? Performance: 335 MB over 100 Mbit from Samba smb-share (Version: Samba 3.0.14a-Debian) to localhost ext3 takes 3.10 minutes with nautilus. On second and third try transfer took between 2.40 and 2.10 minutes. smbget needs around 45 seconds. Tracking resources on localhost while copying over network: /usr/bin/top : PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15087 benny 16 0 102m 31m 22m S 51.2 6.2 0:19.87 nautilus 5490 benny 15 0 59236 6628 4932 S 15.6 1.3 0:19.32 gnome-vfs-daemo Ubuntu 6.10 (Edgy Eft) 2.6.17-10 Arch: ppc G4, 1,33 GHz , 512 RAM gnome 2.16.1 libgnome2-0 2.16.0-0ubuntu1 nautilus 2.16.1-0ubuntu3 gstreamer0.10-gnomevfs 0.10.10-1ubuntu1 libgnome2-vfs-perl 1.060-1 libgnomevfs2-0 2.16.1-0ubuntu4 libgnomevfs2-bin 2.16.1-0ubuntu4 libgnomevfs2-common 2.16.1-0ubuntu4 libgnomevfs2-dev 2.16.1-0ubuntu4 libgnomevfs2-extra 2.16.1-0ubuntu4 libgnomevfs2-ruby 0.15.0-1.1 Test 2: Is it slow on attached devices, too ? copying a large file from localhost (ext3) to a usb 2.0 pendrive (vfat) is on equal speed with nautilus and command line tool. Test 3: Is it slow on same disk? duplicate a large file using nautilus on local ext3 is as fast as using cp on commandline. Ergo: Tracking only Test 1 (II) Stack Tracing (see attachment stack_trace_be1)
Created attachment 79264 [details] tracing a big file transfer over smb to ext2 with additional dbg-flags Reading comment # 27 I decided to make another trace. This time with a big file. Maybe it's easier to see this way if my problem has something in common with the XFS trouble described above.
I'm attacching a profiling session made with oprofile of the same copy of a 120Mb file to an xfs partition with barriers enabled (ultra slow). I'll attach also a run with strace -c that shows that all the time is spent in fadvice64. If you need more specific info, please ask precisely for what you need, i'm not really fluent with profiling and such. Francesco
Created attachment 79270 [details] oprofile output while vfscopying a file to an xfs partition with barriers enabled
Created attachment 79271 [details] strace -c output of vfscopying a file to an xfs partition with barriers enabled
(In reply to comment #32) > Created an attachment (id=79270) [edit] > oprofile output while vfscopying a file to an xfs partition with barriers > enabled > Which command did you use exactly to get that profile ? opreport -l gnomevfs-copy will only show profiling information related to gnomevfs-copy and might be more helpful.
Created attachment 79292 [details] oprofile output while vfscopying a file to an xfs partition with barriers enabled Redone profiling with oprofile, commands ran are the following: opreport -p /lib/modules/2.6.17-10-generic/kernel/fs/xfs/ /usr/bin/gnomevfs-copy >> /tmp/report opreport -l -p /lib/modules/2.6.17-10-generic/kernel/fs/xfs/ /usr/bin/gnomevfs-copy >> /tmp/report opreport -c -l -p /lib/modules/2.6.17-10-generic/kernel/fs/xfs/ /usr/bin/gnomevfs-copy >> /tmp/report
(In reply to comment #33) > Created an attachment (id=79271) [edit] > strace -c output of vfscopying a file to an xfs partition with barriers enabled > I think you need copy a "very large" file and take a strace. and compare with below. This report with the forget_cache disabled and copying large (608MB) file. Process 7219 detached % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 60.47 0.393004 3 148591 read 37.27 0.242244 2 148545 write 2.23 0.014486 0 148558 gettimeofday 0.02 0.000111 4 28 fadvise64 0.01 0.000046 1 41 fstat64 0.01 0.000041 0 326 285 open 0.00 0.000010 10 1 fadvise64_64 0.00 0.000000 0 43 close 0.00 0.000000 0 1 execve 0.00 0.000000 0 1 time 0.00 0.000000 0 1 chmod 0.00 0.000000 0 1 utime 0.00 0.000000 0 4 1 access 0.00 0.000000 0 3 brk 0.00 0.000000 0 2 ioctl 0.00 0.000000 0 10 munmap 0.00 0.000000 0 2 uname 0.00 0.000000 0 6 mprotect 0.00 0.000000 0 1 _llseek 0.00 0.000000 0 1 sched_getparam 0.00 0.000000 0 1 sched_getscheduler 0.00 0.000000 0 2 sched_get_priority_max 0.00 0.000000 0 1 sched_get_priority_min 0.00 0.000000 0 2 poll 0.00 0.000000 0 3 rt_sigaction 0.00 0.000000 0 1 rt_sigprocmask 0.00 0.000000 0 1 pread64 0.00 0.000000 0 2 getcwd 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 82 mmap2 0.00 0.000000 0 27 16 stat64 0.00 0.000000 0 7 1 lstat64 0.00 0.000000 0 2 getuid32 0.00 0.000000 0 1 getgid32 0.00 0.000000 0 1 geteuid32 0.00 0.000000 0 1 getegid32 0.00 0.000000 0 2 setresuid32 0.00 0.000000 0 2 setresgid32 0.00 0.000000 0 1 chown32 0.00 0.000000 0 2 getdents64 0.00 0.000000 0 3 fcntl64 0.00 0.000000 0 2 futex 0.00 0.000000 0 1 set_thread_area 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 statfs64 0.00 0.000000 0 1 SYS_311 0.00 0.000000 0 1 socket 0.00 0.000000 0 1 connect 0.00 0.000000 0 1 send 0.00 0.000000 0 1 recvmsg ------ ----------- ----------- --------- --------- ---------------- 100.00 0.649942 446321 303 total
Created attachment 81500 [details] [review] Remove fadvise(), Because it has poor preformace with SOME filesystem.
Maybe it's worth investigating with the different hints that posix_fadvise() can take to see if any of them work better than the others?
I don't think the solution is to remove fadvise() calls entirely. The traces show that we're calling gnome_vfs_forget_cache() way too often in gnome-vfs-xfer.c:copy_file_data() - as much as once per 4KB transferred. I'm proposing a patch to forget the cache only once per 512KB transferred. See https://bugzilla.novell.com/show_bug.cgi?id=224527 for reference.
Created attachment 83090 [details] [review] Reduce the number of calls to fadvise() during data transfer operations.
Was this patch tested? Could we hope to see this bug fixed or it will be present in gnome 2.18 as well?
I've tested it on my own system (with ext3 and reiserfs) only, and it did not cause any problems that I'm aware of.
Just applied here (with xfs). Copying a 700mb file from the command line takes ~40s, with nautilus and patched gnome-vfs ~50s against 10m+ without the patch. No apparent problems, for now.
From email: That removes the size limit for when using fadvice. I don't think this is right. For small files we don't want to flush them from the cache, as that could cause performance issues (we might be copying a commonly used file).
No. The following condition: if (bytes_to_write == 0 && total_bytes_read - last_cache_drop_point >= DROP_CACHE_BATCH_SIZE) Will trigger for the first time once DROP_CACHE_BATCH_SIZE has been reached. fadvise() will be triggered at the end of each batch, not at the beginning.
Alex and I discussed the matter on the gnome-vfs mailing list, and we've agreed to keep the lower 20MB file size limit on fadvise() action. I'm attaching a revised patch.
Created attachment 85481 [details] [review] Reduce the number of calls to fadvise() during data transfer operations, revisited.
The last patch was approved by Alex Larsson and committed to trunk and gnome-2-18 branch.