There had been done some changes in 2.25 development cycle, like added the ability to set the refresh interval manually in calendar properties, support for getctag (bug #560185), but none of them both is working in a way you want. Main problem could be that the CalDAV backend downloads all items to local cache, and then operates on it. It doesn't download all items in one, only list of known items, then downloads all new or changed one-by-one. Thus it shouldn't timeout on an iCal file download.

How many events do you have, that it generates such a big report file?

These are really only two small calendars:

"...I have only 3-5 simultaneous users, 5 daily appointments
in 2 shared calendars. (the last) 60 days of appointments"

The iCal format itself is somewhat verbose, so a request for the whole calendar extent as a file will be large in bytes for even a small number of events (e.g. about 250).

It would be much better for Evolution's plugin to request only the extent of the current view from the CalDAV server.

I'm not sure whether we are understanding each other, I've such a feeling we don't.

The response on the REPORT command looks like this:
<?xml version="1.0" encoding="UTF-8"?>

<D:multistatus xmlns:D="DAV:">
  <D:response>
    <D:href>/dav/u/Calendar/18c6ec55-ab24-4855-b99e-454339115a75.ics</D:href>
    <D:propstat>
      <D:status>HTTP/1.1 200 OK</D:status>
      <D:prop>
        <D:getetag>"433958-1237479880000"</D:getetag>
      </D:prop>
    </D:propstat>
  </D:response>
  <D:response>
    <D:href>/dav/u/Calendar/44e49bff-b5b7-4ed8-9e6e-94a5fd3ce20e.ics</D:href>
    <D:propstat>
      <D:status>HTTP/1.1 200 OK</D:status>
      <D:prop>
        <D:getetag>"6201-1202315721000"</D:getetag>
      </D:prop>
    </D:propstat>
  </D:response>

That's not the whole iCalendar format, it's just list of known events on the server, with their etags, which are compared to those stored on a local machine, and if differ, or are new for the local machine, then those items are downloaded, not all in one, but one by one. It's approximately 273 bytes per event, and recurring events are supposed to be returned only once, not one per the occurrence.

Say you do not have any reoccurring appointments, thus every single event is an event (could be quite nice mess, I would say), so it means 5 * 5 * 60 = 1500 events in both calendars, 1500 * 273 = 409500B for each user. Huh, I wouldn't expect that. There should help the getctag here, and DaviCal supports it (I'm not sure since which version), together with the refresh interval option, as I mentioned above.

Though both are sort of workarounds for you, as I realized. No exact idea how to properly manage this, right now. (I'm afraid of recurring events abit too.)

I'm probably missing something, yes. I'm using the packaged davical-0.9.6.1.

Can you suggest a wget test request that would imitate what Evolution's caldav is calling from the server?

On the less popular of the two calendars, I get an 86Kb ical file at the calendar URL: 

$ wget --http-user=myuser --http-password=foo http://192.168.10.99/cal/caldav.php/remote01/office -O remote01.ical
--2009-04-02 14:16:41--  http://192.168.10.99/cal/caldav.php/remote01/office
Connecting to 192.168.10.99:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Connecting to 192.168.10.99:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 88476 (86K) [text/calendar]
Saving to: `remote01.ical'

100%[==============================================================>] 88,476      --.-K/s   in 0.03s   

2009-04-02 14:16:43 (2.67 MB/s) - `remote01.ical' saved [88476/88476]

$ grep -c 'BEGIN:VEVENT' remote01.ical 
210

On the larger calendar (with 60 days history davical patch), I simply get an http 500 error:

$ wget --http-user=myuser --http-password=foo http://192.168.10.99/cal/caldav.php/cdog/office --save-headers
--2009-04-02 14:08:44--  http://192.168.10.99/cal/caldav.php/cdog/office
Connecting to 192.168.10.99:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Connecting to 192.168.10.99:80... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2009-04-02 14:09:17 ERROR 500: Internal Server Error.

Both calendars work in Evolution, however. Perhaps I need to craft a request with an --header=STRING to indicate REPORT?

You set your version to 2.24.x, but the getctag version is in since 2.26.0 (when talking about stable version). It was added there on request from DAViCal developer, and it's working correctly in DAViCal also in some recent version of DAViCal (+/- time of commit of bug #560185).

I do not know about wget command for a report, but you can run evolution-data-server on console, with a caldav debug turned on, and get all the information sending between you and the server. It's something like this:

CALDAV_DEBUG=all /usr/libexec/evolution-data-server-2.24 &>caldav.log

Please note that you can have the file located somewhere else, and also that the log will not contain only CalDAV debug information, but also other messages printed on evolution-data-server console. Also, before running it, close evolution and do 'evolution --force-shutdown' to stop previous evolution-data-server.

I have only tested DAViCal with Ubuntu 8.10's Evolution-2.24. I will be updating to Ubuntu-9.04 and Evolution-2.26 on or about the release date April 20th. I may be able to test a KVM-based Ubuntu-9.04 this weekend if you think it's necessary or helpful.

I will test the Evolution-2.24 client with the data server as:

CALDAV_DEBUG=all /usr/libexec/evolution-data-server-2.24 &>caldav.log

This weekend, when the users won't be disturbed.

Otherwise, if I'm interpreting the response correctly, I may expect to see different calendar view polling behaviour with Evolution-2.26, and as a result may be able to remove my 60-days limit patch from my DAViCal instance.

If so that's very good news, and I could report back after April 20th.

I can see some possibilities, but none of them might fix all the related things:
a) do not maintain a local cache and do fetches on demand - it might cut
   bandwidth requirements drastically, though doesn't work with (also
   requested) offline support and with autoupdating.

b) do fetch all items at the beginning, but afterwards only those changed
   since last checkout. Sounds very promising, but doesn't work with deleted
   items in the past (in time before last check).

c) fetch only items in the range of (say) -1 .. +1 month from the actual, and
   extend this interval on demand. The issue is quite similar as in a)
   (possibly outdated local cache).

Note: Reading the RFC I realized there is no problem with recurring events, as I was afraid above.

I'm trying to find some reasonable compromise for this, but nothing will work fully here. Do I understand correctly what we are trying to do here, is to not fetch all items information on every single check on the server, but preferably not more than once per session? I suppose to do a full fetch on start, (to synchronize with a local cache for offline) and then listen for changes only. (where the most hard thing is with deleted items.)

(In reply to comment #6)
> if you think it's necessary or helpful.

It's helpful because of those two new features in evolution, and one in DAViCal (the getctag support).

> I will test the Evolution-2.24 client with the data server as:
> 
> CALDAV_DEBUG=all /usr/libexec/evolution-data-server-2.24 &>caldav.log

I just realized that the file is not populated fully until the process ends, though shouldn't be so bad.

> Otherwise, if I'm interpreting the response correctly, I may expect to see
> different calendar view polling behaviour with Evolution-2.26, and as a result
> may be able to remove my 60-days limit patch from my DAViCal instance.

With the getctag support on both sides, evolution and DAViCal, you should be able to remove that workaround on DAViCal side. What it'll do is that it'll fetch information about all items on start, and then only if something changes in the calendar. (Together with longer refresh interval it should be quite good for you.) Other thing I just think of is to store the getctag value in the cache, and do not fetch all items on start if it's same as before. This part is not there (yet), because I thought it's just a small optimization.

Created commit 9e2e454 in eds master. A little improve to store CTag in a cache, thus when it didn't changed, then the calendar will not be fetched again next start.

I know that I can receive changed objects from a server by something like this:
-------------------------------------------------------------------------
index 78002ca..2b72974 100644
--- a/calendar/backends/caldav/e-cal-backend-caldav.c
+++ b/calendar/backends/caldav/e-cal-backend-caldav.c
@@ -1105,6 +1105,15 @@ caldav_server_list_objects (ECalBackendCalDAV *cbdav, CalDAVObject **objs, int *
 			xmlSetProp (sn, (xmlChar *) "name", (xmlChar *) "VTODO");
 			break;
 	}
+
+	/*
+	node = xmlNewTextChild (sn, nscd, (xmlChar *) "prop-filter", NULL);
+	xmlSetProp (node, (xmlChar *) "name", (xmlChar *) "LAST-MODIFIED");
+	
+	sn = xmlNewTextChild (node, nscd, (xmlChar *) "time-range", NULL);
+	xmlSetProp (sn, (xmlChar *) "start", (xmlChar *) "20090525T000000Z");
+	*/
+
 	/* ^^^ add timerange for performance?  */
 
-------------------------------------------------------------------------
but it works only with DavICal from 3 servers I use for testing, thus that's not the best way (not talking about inability to notice deleted items).

The other option for you might be turning off the local cache, and really receive only those objects from the given time range, but it might be slow.

Yet another option might be to check for objects from the server within some time interval, like not longer than month/week ago, and update local cache based on that.

Both two later options requires some checkboxes and such in a calendar properties, and though the first is easy to write, ordinary users might have trouble understand it. The later is harder for me for proper wording. Also, with the later I'm not sure how it will work with recurring appointments, though probably without any issue. Possible offline usage (which is not done yet and will take some time) is less possible with any of those two options.

What do you think, Jeff?

I'm confused why getctag (bug #560185) doesn't solve this.
-
Check ctag (PROPFIND)
 - no change 
   - done
 - change
   - do REPORT (calendar-query)
     - Retrieve events who's etags have changed
     - Delete events no longer in response
     - calendar-multiget events client didn't have before

So the initial calendar-multiget might be big, but subsequent ones should be pretty small.

Instead of relying on the Apple proprietary extension, please consider implementing http://tools.ietf.org/html/draft-daboo-webdav-sync which DavICal, amongst other servers, supports. See http://wiki.davical.org/w/RFC_Compliance

I'm closing this. The 3.26.x has made some changes which basically require having local cache populated, thus some of the options suggested in comment #8 do not apply. The current behaviour is like this:

a) getctag, if did not change, done; otherwise:
b) getetag for events changed in +/- 5 days from today and download those
   new and changed
c) getetag for the whole calendar and download those new and changed and
   delete those removed

That getetag reads only etags, not whole components, which are read in the second step. The step b) is there to provide current events as soon as possible and only then refresh whole calendar content.

(In reply to Arnaud Quillaud from comment #10)
> Instead of relying on the Apple proprietary extension, please consider
> implementing http://tools.ietf.org/html/draft-daboo-webdav-sync which
> DavICal, amongst other servers, supports. See
> http://wiki.davical.org/w/RFC_Compliance

The thing is that getctag is supported by all the common servers users use, while the webdav sync extension may not be. You named DAViCal, but with that one was no problem.

Ah, I see that draft made it into RFC 6578 in 2012 [1], that's nice. By the way, it's also initiated by the Apple folks. Interestingly, Google and Yandex claim to support it (I only tried to get the supported-report-set, not whether it truly works), DAViCal too, as you wrote, but Yahoo! and Zimbra do not, neither for example blauCloud, which is an ownCloud instance (I only do not know which version). Let's wait a bit more till the RFC is even more widespread.

[1] https://tools.ietf.org/html/rfc6578