GNOME Bugzilla – Bug 215240
imap caching + broken server + concurrent access = breakage
Last modified: 2004-09-29 20:42:34 UTC
Please fill in this template when reporting a bug, unless you know what you are doing. Description of Problem: Evolution client does not update shared folder view when messages are deleted (expunged) from that shared folder by another imap client. Steps to reproduce the problem: 1. Make a shared emap folder your current folder 2. With another client/connection/user remove a message from that folder 3. The deleted message is still shown in the first evolution client. New messages and subsequent changes to that folder are not shown. Doing a "send / receive" makes no difference. You need to exit out of the evolution client and reenter to see the changes. Actual Results: Subsequent changes to shared folder (new messages etc) are not seen until evolution is restarted Expected Results: The removed messages should disappear, new messages should be shown. I.e. evolution should display the contents of the shared folder! How often does this happen? All the time - whenever a message is removed from the folder by another client. Additional Information: Cyrus 2.0.16 is the IMAP server. This is the redhat 6.2 Release Candidate binary (not a build from CVS or anything else).
We've played around with this a bit more today. It has something to do with the local evolution cache getting out of sync with the files on the server - possibly when some status is altered on the server by another user.
I just got bit by this bug today. It is a real nasty issue for someone migrating to Evolution. If you forget and leave evolution running, it will be reading from IMAP while you're writing with your other client. When your other client deletes messages, Evolution can get hopelessly confused, eventually dying with a core dump.
Is it possible to get any IMAP logs of what evolution is seeing? If the server isn't telling it what is changing for example, there's not much we can do. What server/version is it?
*** bug 222558 has been marked as a duplicate of this bug. ***
I guess I should have clarified in 22558 that the synchronization issue was there, however I never experienced a message being undeleted or marked as unread prior to 1.0. The IMAP server isn't going to notify the client that some other client changed something. It is up to the client to poll the server for the current status. Caching is great however the server is the authority not the cache. IMAP kinda reminds the programmer of this by requiring a fetch of the flags before you can store the flags. Haven't looked at evolution due to lack of time however I have seen some clients just issue the fetch, ignore the returned values, then issue the store to set the flags that they wanted, clearly broken behaviour.
uhm, have you read the IMAP spec? IMAP doesn't force that you fetch the flags before setting them. and yes, the server MUST tell a client when flags change. That is part of the spec. *** This bug has been marked as a duplicate of 222558 ***
Humm.. never mind, I won't get in the spec war since I don't have time to catch up on everything. However please fix one of these reports rather than marking them solved as duplicates of each other.
can someone attach a log of evolution-mail's output with CAMEL_VERBOSE_DEBUG=1 set. If the server isn't telling us of flag changes, its the server at fault. If it is but we're ignoring it, we probably can't fix it anyway as the code is so messy.
Kudos to the Ximian knowledge database that gave me the additional clue of having to start evolution-mail by hand in order to get the debugging info. As you requested, here is output from evolution-mail, and I think we have a crucial clue. What I was trying to do was demonstrate the following fault: With Evolution and another IMAP client running, delete a message using the other client, and watch the message REFUSE to disappear in evolution. Unfortunately, the first couple times I deleted and expunged with the other client, Evolution didn't exhibit the fault. Why? Because another message had come in WHILE I was doing this stuff. As I watched the status scroll by from Evolution, I noticed no IMAP transactions arriving spontaneously from the IMAP server while I deleted and expunged an item on 3 separate occasions. On the FINAL attempt, when no other mesage came in, Evolution exhibited my failure case: Receive incoming message. Confirm that both Evolution and other client (In this case WebMail) have noticed its arrival, and displayed header summary info. In the other client delete and expunge the message. Confirm Evolution has not noticed that deletion and expunge. Click on the summary header in Evolution. Confirm Evolution displays the mesage body (surprising). Delete the message Use ^E to expunge deletions. Notice that the message is back, undeleted in Evolution. Confirm that the other client knows of no such message. Theory: Some bit of cache cleanup happens if new mail comes in across another client deleting/expunging that does NOT happen ordinarily, but should. Mind you, that cache may be in evolution, or it may be in the IMAP server. Comments?
Created attachment 41182 [details] Debugging output from evolution-mail with CAMEL_DEBUG_VERBOSE=1
Oh an important detail I forgot to mention: The EXACT sequence of events I follow when I say "delete and expunge" is: click on header to view body use delete command use expunge command The WIERD thing is that evolution, when asked to show the body of the deleted and expunged message, SUCCEEDS. Is that because it prefetched it, or because the IMAP server is confused?
probably because evolution had it cached locally.
This bug is still present in Evolution 1.0.5
Hmm, from what i can see from that log, the server isn't telling us that the last message was expunged. e.g. sending : A00034 UID EXPUNGE 101035 received: * 225 EXPUNGE received: * 226 EXISTS * Here we have 226 messages, after number 225 was expunged. received: * 2 RECENT received: A00034 OK Completed * We then fetch the new message info, only 1 message: sending : A00035 UID FETCH 101037:* (FLAGS RFC822.SIZE BODY.PEEK[HEADER]) received: * 226 FETCH (FLAGS (\Recent) UID 101037 RFC822.SIZE 2898 BODY[HEADER] {1806} received: ) received: A00035 OK Completed sending : A00036 UID FETCH 101037 BODY.PEEK[] received: * 226 FETCH (UID 101037 BODY[] {2898} received: ) received: A00036 OK Completed sending : A00037 UID STORE 101037 +FLAGS.SILENT \Deleted received: * 225 EXISTS * then when we go to store we're told 225 now exist, however we haven't received any EXPUNGE response to tell us that the message was expunged. received: * 1 RECENT received: * 226 FETCH (FLAGS (\Recent \Deleted) UID 101037) received: A00037 OK Completed sending : A00038 UID EXPUNGE 101037 received: * 225 EXISTS received: * 1 RECENT * And when we actually do the expunge ourselves, we're not given any EXPUNGE response to tell us that the explicit expunge we wanted actually occured. So as far as I can tell, this is actually the server ... rfc2180 has some comments to make on this, in section 4.1 onwards. From what I can tell, the server should be sending a NO response if we try and store to the missing message. But it may be that the server is keeping a separate view for the other client (e.g. evolution), in which case the server isn't really doing anything wrong (rfc2180 suggests several variations of shared mailbox access). The reason you get the message anyway, is that the server hasn't said the message doesn't exist, and it gets the "cached" copy of the data anyway.
this is not a 1.2 bug, read the last comment, its probably a server bug, and there's nothing we can do about it.
I concurr that this is a server bug. It was most helpful knowing how to understand the server logs. At MIT we seem to have upgraded our IMAP server software, and the problem seems to have stopped occurring. (I'll do more rigorous testing later but it will be directed at our IMAP server folk rather than Evolution.) The explanation of the server logs and what the server should be doing will be most helpful, in the event that more rigorous testing still points up a problem. I'm an outsider so it would be inappropriate for me to touch the state of the bug. I'd say that it would be quite reasonable to resolve the bug to state NOTABUG. Thanks very much for looking into this and taking the time to explain how it goes.
FYI: I just checked with our network folks. Our IMAP server has not changed. But I reviewed our local modifications to evolution. We have a local change that turned off caching. (Actually it was a strange hack. We have home directories in AFS, and caching to AFS was slower than uncached IMAP service (not because AFS was slow, but because we were doing two copies over the net.). So we disabled caching when the evolution folder is in AFS.) My conclusion: There is a bug in our IMAP server. It does throw off IMAP caching. It stopped happening to me because we turned off caching locally.
yea, the current caching code really sucks it gets really slow since it doesn't split the files up into different directories evenly (it just dumps everything to the same directory) and large directories are generally really slow. we also cache headers when downloading the summary info, which we should probably do away with. If we don't do this, then we should get much better performance when loading a folder for the first time especially.
I guess we can close this then?