After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 306026 - Copy to DAV uses much memory
Copy to DAV uses much memory
Status: RESOLVED WONTFIX
Product: gnome-vfs
Classification: Deprecated
Component: Module: http
2.14.x
Other All
: Normal enhancement
: ---
Assigned To: gnome-vfs maintainers
gnome-vfs maintainers
: 349909 (view as bug list)
Depends on:
Blocks: 341837
 
 
Reported: 2005-05-31 11:14 UTC by Jesper Christensen
Modified: 2008-09-06 18:54 UTC
See Also:
GNOME target: ---
GNOME version: Unversioned Enhancement



Description Jesper Christensen 2005-05-31 11:14:30 UTC
When copying a file to a DAV-share (through either Nautilus or gnomevfs-copy)
the entire file is read into memory and then uploaded to the server upon
write-close.

Never had a problem with small files, but copying a 1GB file consumes at least
1GB memory, which makes my desktop unusable until the copy is done.

Copying the same file to a SMB-share works without using much memory.

I've quickly looked at the code. The entire content of the file is collected in
an bytearray in do_write() and written using neon during do_close().

I'll fix it if somebody will give me a slight hint: does neon at all support
sending a large http-request in small increments?
Comment 1 nadir 2005-06-14 06:38:23 UTC
Yes, you can use a request provider callback:

void ne_set_request_body_provider(ne_request *req, off_t bodysize,
				  ne_provide_body provider, void *ud)
also with a 64bit LFS variant:

void ne_set_request_body_provider64(ne_request *req, off64_t bodysize,
                                    ne_provide_body provider, void *ud)

The format of the callback function is:

/* "Pull"-based request body provider: a callback which is invoked to
 * provide blocks of request body on demand.
 *
 * Before each time the body is provided, the callback will be called
 * once with buflen == 0.  The body may have to be provided >1 time
 * per request (for authentication retries etc.).
 *
 * For a call with buflen == 0, the callback must return zero on success
 * or non-zero on error; the session error string must be set on error.
 * For a call with buflen > 0, the callback must return:
 *        <0           : error, abort request; session error string must be set.
 *         0           : ignore 'buffer' contents, end of body.
 *     0 < x <= buflen : buffer contains x bytes of body data.  */
typedef ssize_t (*ne_provide_body)(void *userdata, 
				   char *buffer, size_t buflen);

I hope this helps

Tristan
Comment 2 Christian Kellner 2005-06-14 07:29:14 UTC
Well actually there were some reasons I did do it that way. Let's see if I can
recall some of them. The most obvious one is that you would need to special case
the RANDOM access case form the !RANDOM case which of course is more pain in the
ass.
The other one is that I don't see any good and clean way to make use of that
ne_set_request_body_provider functions together with the gnome_vfs_write (). 

Well actually I don't like the way we do the PUT transfers atm myself so if you
find any good way to solve that I am more then willing to review the patch ;)
Comment 3 Jesper Christensen 2005-06-14 10:01:30 UTC
Random access is only relevant when getting/downloading and not when
putting/uploading to the DAV-server, or did I miss something? :)

I agree with you that there doesn't seem to exist an elegant way of
gnome_vfs_write() using ne_set_request_body_provider(). libneon wants to pull
data through a callback and we want to push data from gnome-vfs. The only way I
see that these two interfaces talking together would be through tmp-files or
maybe special sockets.

What if we extended libneon with a ne_request_body_write(char* buffer, size_t
buflen) interface that a client can use to push segments of data in between the
connection open and connection close?

Such an interface should be quite possible to implement and would solve our
problem nicely. The dav-module of gnome-vfs could then reduce it's write() as
something a simple as found in the samba-module.

Would the libneon folks accept such an interface? Or does something similar
already exist and I just missed it?
Comment 4 Sebastian Dröge (slomo) 2006-05-03 08:59:08 UTC
Any progress on this? The bug is still present in 2.14 and prevents one from copying files that are larger than the available memory... which could happen very easily and happened already more than once for me
As such it's not really an enhancement but a major bug in the webdav implementation and the severity should be raised
Comment 5 Sebastien Bacher 2006-05-03 09:14:43 UTC
Ubuntu bug about that: https://launchpad.net/distros/ubuntu/+source/gnome-vfs2/+bug/42720
Comment 6 Christian Kellner 2006-08-04 11:30:12 UTC
*** Bug 349909 has been marked as a duplicate of this bug. ***
Comment 7 Martin Schaaf 2008-03-03 00:35:36 UTC
I have this problem with 2.20.
Comment 8 Martin Ejdestig 2008-03-03 01:57:57 UTC
How does the DAV backend in GVFS handle large files?
Comment 9 André Klapper 2008-09-06 18:54:40 UTC
gnome-vfs has been deprecated and superseded by gio/gvfs since GNOME 2.22, hence mass-closing many of the gnome-vfs requests/bug reports. This means that gnome-vfs is NOT actively maintained anymore, however patches are still welcome.

If your reported issue is still valid for gio/gvfs, please feel free to file a bug report against glib/gio or gvfs.

@Bugzilla mail recipients: query for
      gnome-vfs-mass-close
to get rid of this notification noise all together.


General further information: http://en.wikipedia.org/wiki/GVFS 
Reasons behind this decision are listed at http://www.mail-archive.com/gnome-vfs-list@gnome.org/msg00899.html