GNOME Bugzilla – Bug 306026
Copy to DAV uses much memory
Last modified: 2008-09-06 18:54:40 UTC
When copying a file to a DAV-share (through either Nautilus or gnomevfs-copy) the entire file is read into memory and then uploaded to the server upon write-close. Never had a problem with small files, but copying a 1GB file consumes at least 1GB memory, which makes my desktop unusable until the copy is done. Copying the same file to a SMB-share works without using much memory. I've quickly looked at the code. The entire content of the file is collected in an bytearray in do_write() and written using neon during do_close(). I'll fix it if somebody will give me a slight hint: does neon at all support sending a large http-request in small increments?
Yes, you can use a request provider callback: void ne_set_request_body_provider(ne_request *req, off_t bodysize, ne_provide_body provider, void *ud) also with a 64bit LFS variant: void ne_set_request_body_provider64(ne_request *req, off64_t bodysize, ne_provide_body provider, void *ud) The format of the callback function is: /* "Pull"-based request body provider: a callback which is invoked to * provide blocks of request body on demand. * * Before each time the body is provided, the callback will be called * once with buflen == 0. The body may have to be provided >1 time * per request (for authentication retries etc.). * * For a call with buflen == 0, the callback must return zero on success * or non-zero on error; the session error string must be set on error. * For a call with buflen > 0, the callback must return: * <0 : error, abort request; session error string must be set. * 0 : ignore 'buffer' contents, end of body. * 0 < x <= buflen : buffer contains x bytes of body data. */ typedef ssize_t (*ne_provide_body)(void *userdata, char *buffer, size_t buflen); I hope this helps Tristan
Well actually there were some reasons I did do it that way. Let's see if I can recall some of them. The most obvious one is that you would need to special case the RANDOM access case form the !RANDOM case which of course is more pain in the ass. The other one is that I don't see any good and clean way to make use of that ne_set_request_body_provider functions together with the gnome_vfs_write (). Well actually I don't like the way we do the PUT transfers atm myself so if you find any good way to solve that I am more then willing to review the patch ;)
Random access is only relevant when getting/downloading and not when putting/uploading to the DAV-server, or did I miss something? :) I agree with you that there doesn't seem to exist an elegant way of gnome_vfs_write() using ne_set_request_body_provider(). libneon wants to pull data through a callback and we want to push data from gnome-vfs. The only way I see that these two interfaces talking together would be through tmp-files or maybe special sockets. What if we extended libneon with a ne_request_body_write(char* buffer, size_t buflen) interface that a client can use to push segments of data in between the connection open and connection close? Such an interface should be quite possible to implement and would solve our problem nicely. The dav-module of gnome-vfs could then reduce it's write() as something a simple as found in the samba-module. Would the libneon folks accept such an interface? Or does something similar already exist and I just missed it?
Any progress on this? The bug is still present in 2.14 and prevents one from copying files that are larger than the available memory... which could happen very easily and happened already more than once for me As such it's not really an enhancement but a major bug in the webdav implementation and the severity should be raised
Ubuntu bug about that: https://launchpad.net/distros/ubuntu/+source/gnome-vfs2/+bug/42720
*** Bug 349909 has been marked as a duplicate of this bug. ***
I have this problem with 2.20.
How does the DAV backend in GVFS handle large files?
gnome-vfs has been deprecated and superseded by gio/gvfs since GNOME 2.22, hence mass-closing many of the gnome-vfs requests/bug reports. This means that gnome-vfs is NOT actively maintained anymore, however patches are still welcome. If your reported issue is still valid for gio/gvfs, please feel free to file a bug report against glib/gio or gvfs. @Bugzilla mail recipients: query for gnome-vfs-mass-close to get rid of this notification noise all together. General further information: http://en.wikipedia.org/wiki/GVFS Reasons behind this decision are listed at http://www.mail-archive.com/gnome-vfs-list@gnome.org/msg00899.html