GNOME Bugzilla – Bug 446276
imap mails get lost with vague connection
Last modified: 2008-09-07 20:48:10 UTC
Please describe the problem: I noticed several times: If network connection is vague due to wireless lan and mails are moved from one imap folder to the other, sometimes they don't arrive at the destination folder but are deleted at the source folder. That also happens with mail filters moving mails to folders. Steps to reproduce: 1. move mails by drag and drop from one imap folder to the other imap folder (same server) Actual results: Expected results: Mails should be deleted if they have arrived properly. Does this happen every time? No. Other information:
well, how to do this technically?
Sorry, it's hard to reproduce, I can't properly. Maybe programming a rule that drops every second packet. Anyway, does Evolution check for proper arrival of the moved mails? I lost a lot of mails due to network problems.
(In reply to comment #2) > Anyway, does Evolution check for proper arrival of the moved mails? how should this technically work?
IMAP provides a COPY command which is what Evolution uses when copying messages between IMAP folders on the same server. If the COPY command returns success, then it is assumed that all of the messages were copied. At this point, Evolution then goes and marks all of those messages as deleted if you were doing a "move messages to folder" operation.
I see. So it's about a bug in the IMAP protocol. I mean it's hard to compare all copy actions for success myself.
well, it might not be... I don't know :) you'd probably have to get a wireshark trace to see for sure if evolution is correctly interpreting the response from the server, etc. might be that there are so many messages you are trying to mopve at a time that it has to break it into several COPY commands, and maybe not all are returning success?
This happens for me on evolution 2.11.90. I have set up message filters which move my mail from a remote IMAP server on arrival to my local IMAP server. The local IMAP server was terminated but evolution kept moving the mails so they were marked deleted on the remote server although they were not saved on the local one. This is a serious data loss problem. I understand that the situation is little bit different from the original bug report (there was only one server in the game) but it still might the same roots.
(In reply to comment #0) > If network connection is vague due to wireless lan and mails are moved from > one imap folder to the other, sometimes they don't arrive at the destination > folder but are deleted at the source folder. (In reply to comment #4) > IMAP provides a COPY command which is what Evolution uses when copying > messages between IMAP folders on the same server. If the COPY command returns > success, then it is assumed that all of the messages were copied. At this > point, Evolution then goes and marks all of those messages as deleted if you > were doing a "move messages to folder" operation. What *exactly* is the behavior, if the server response gets lost? Could the by default assumed response be "success", unless there is an actual "failure" response, by any chance? Note: Bug 436802 describes a similar issue. However, it most likely is *not* the same as this one, which clearly is about moving mail on one IMAP server only. That other bug is about moving local mail to the IMAP server. However, if the "delete" command is executed "if (! failue)", then this actually might be the same bug.
Following up from https://bugs.launchpad.net/ubuntu/+source/evolution/+bug/117896 I have uploaded a debug log with CAMEL_DEBUG=ALL turned on to: http://launchpadlibrarian.net/8880158/evo_debug.log
Created attachment 97330 [details] log - move 2 mesages to A4A folder but end up in trash More follow up from the ubuntu bug report: https://bugs.launchpad.net/ubuntu/+source/evolution/+bug/117896 I experienced the bug with moving mails ending up in the trash folder with my webmail.us account for the first time this morning. Previously I had only experienced that bug with my blueyonder account and was suspicious this was caused by a blueyonder server problem; I can rule this out now. log file attached.
the log shows Evolution performing a COPY command for 2 messages from INBOX to INBOX.A4 or whatever... afterwards, Evolution marks those 2 messages as \Deleted in the INBOX folder (they are not deleted in the A4 folder) This is all working as expected.
When I did that move operation the messages were removed from the source folder but did not show up in the destination folder. I found them in the trash folder! I would not call that working as expected. I actually managed to repeat this twice - the first time it happened Evo was running without debugging turned on (autostarted, session management). I undeleted the messages and they reappeared in the source folder. Then I restarted Evo with debug log and tried the exact same move operation with exactly the same results. I have since undeleted those two messages again and they are back in the source folder.
if the messages are not showing up in the destination folder (server returned success for the copy operation), then it is a server bug. It is expected that moved messages will ALSO appear in the Trash fodler as the Trash fodler shows all messages marked as deleted in all folders. Since IMAP does not have a MOVE command, "move" is implemented as a COPY and a delete - hence messages will appear in the Trash folder as well.
Created attachment 97368 [details] error dialog after failed move operation Well, funnily enough I thought I would try it again with istanbul recording the session to make absolutely sure I wasn't going mad. On this occasion I got an error dialog popup. First time I have ever seen this particular dialog. Presumably this confirms it is an error at the IMAP server? Not sure this answers the question of why I am getting this same problem with 2 different IMAP accounts/servers.
It would be interesting to know which IMAP servers these services use. Here it happens with dovecot IMAP server but it is more like the bug 436802 because the moves are between local folder and imap folder.
I believe webmail.us use dovecot. I have experienced this bug much more frequently with blueyonder.co.uk, I think they use exim and Microsoft SMTPSVC.
But exim is MTA and not IMAP server. Perhaps they use dovecot for IMAP as well? That might be significant then.
Ah, I didn't realise that. No other info from headers. Is there a way of scanning for this info? Or you could try yourself: imap4.blueyonder.co.uk 195.188.53.52 I have given up trying to use Evolution with blueyonder via IMAP because of this bug and also: http://bugzilla.gnome.org/show_bug.cgi?id=348496 http://bugzilla.gnome.org/show_bug.cgi?id=436802 None of these issues occur with thunderbird and I remain sceptical that this is simply a server issue. Doubtless there are server problems involved but Evo's ability to deal with them gracefully is questionable and I think this bug has been closed hastily.
neither of you seem to be reading what I wrote. There is no bug here. Not on the server, not on the client. Here's a little script to illustrate what is happening: User says to Evo: Please MOVE messages 5 & 6 from INBOX to INBOX/4_A4A Evo says to server: COPY messages 5 & 6 from INBOX to INBOX/4_A4A Server responds to Evo: OK, done. Evo says to server: OK, now delete the original messages 5 & 6 in INBOX Server responds to Evo: OK, done. User says to Evo: Please show me all messages marked for deletion Evo says to user: Sure, they are 5 & 6 from INBOX, ... from FolderFu, ... from FolderBar, ... From the logs provided, I see no evidence of any bug anywhere. here is the relevant portion of the log: sending : A00054 UID COPY 1498:1499 INBOX.4_A4A received: A00054 OK Copy completed. sending : A00055 SELECT INBOX.4_A4A received: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft) received: * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted. received: * 102 EXISTS received: * 2 RECENT received: * OK [UIDVALIDITY 1191525983] UIDs valid received: * OK [UIDNEXT 103] Predicted next UID received: A00055 OK [READ-WRITE] Select completed. You can see there that not only does the response to the COPY command return "OK" (which means successfully completed), but when we SELECT the INBOX.4_A4A folder afterward, there are 2 RECENT messages, suggesting that indeed, the COPY command was successful. Now, in order to complete the requested "MOVE" (remember: implemented as a COPY & mark-as-deleted), the log continues with: sending : A00060 SELECT INBOX received: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft wasblank2 $$$INDEXED### $Forwarded $MDNSent Junk $Label1 $Label2 $Label3 $Label4 $Label5 farm next pending perhaps important) received: * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft wasblank2 $$$INDEXED### $Forwarded $MDNSent Junk $Label1 $Label2 $Label3 $Label4 $Label5 farm next pending perhaps important \*)] Flags permitted. received: * 60 EXISTS received: * 0 RECENT received: * OK [UIDVALIDITY 1162242671] UIDs valid received: * OK [UIDNEXT 1500] Predicted next UID received: A00060 OK [READ-WRITE] Select completed. sending : A00061 FETCH 60 UID received: * 60 FETCH (UID 1499) received: A00061 OK Fetch completed. sending : A00062 UID STORE 1498 FLAGS.SILENT (\Answered \Deleted \Seen) received: A00062 OK Store completed. sending : A00063 UID STORE 1499 FLAGS.SILENT (\Deleted \Seen) received: A00063 OK Store completed. Command # 62 and 63 mark the original messages as deleted (hence why they show up in the Trash folder). However, they are not deleted in the INBOX/4_A4A folder, just in the INBOX folder (which is what you wanted). If you check your 4_A4A folder, you'll find that those 2 messages exist there AS WELL AS in the Trash folder. Expunging the Trash folder WILL NOT remove them from the 4_A4A folder. I hope this is now clear...
Thanks for the explanation. Unfortunately the messages involved were part of a very longwinded thread and from what you are saying it sounds like I have missed seeing them second time around when I had debugging turned on. If I have screwed up on this occasion then I can only apologize. That said, it does not automatically mean that previous instances of this bug I have noted with blueyonder's IMAP server were errors nor does it mean that other people reporting this issue have screwed up either. So I stand by my comment that you have closed this bug hastily. I think it would have been appropriate to wait a few days for other people experiencing this problem, in particular the original reporter, to respond to your comments. In fact people have already done so on the ubuntu bug report in light of your comments and they don't seem to agree with your assertion that this a server error. Besides people who have filed bug reports here and with Ubuntu there are also people on the mailing lists reporting lost mail when doing move operations to IMAP folders, eg: http://mail.gnome.org/archives/evolution-list/2007-August/msg00036.html I'll keep running Evo with debugging turned on and see if I can trigger this bug by moving messages into a test folder from time to time. I'll keep this test folder empty normally to eliminate any possibility of confusion.
well, it's not like this bug has been lost in the ether ;) if more evidence can be provided, I have no problem with it being reopened. for now, there is none... and all evidence submitted thus far shows everything working properly. a bug being closed doesn't mean it gets locked away in a large government warehouse along with the ark of the covenant, never to be seen again.
Created attachment 101696 [details] Log of moved mail
I have the same problem on gentoo evolution 2.12.2 i had seen the same error before on 2.10.x Mail server is dovecot. Evolution log shows some errors, and it does not seem to show that the mail got copied correctly (so it does not seem to be a mailserver eror) I attached the logfile ... (i replaced some foldernames with [...] in it) I opened evolution, then dragged the already selected mail into the "Trash" folder, (not the Evolution "Trash" but a just a folder with that name...), looked in the "Trash" folder, no mail there, then closed evolution. The mail is only in the real Trash, thus it got lost on the move.
Would it be possible to do this ? I drag 'n drop the mail "a" to the imap folder "b" for moving it = - Evolution makes a COPY of the mail "a" to the folder "b" - If the COPY command returns success, evolution makes a "refresh" of the folder "b" and makes a "search" on the mail "a" in the folder "b". - If the SEARCH returns success, it is assumed that the message was copied, and Evolution marks the initial message "a" as deleted. - If the COPY or SEARCH command doesn't return success, an evolution error message appears and the mail "a" isn't deleted (and lost). This bug's status is "Resolved" but I lost several mail with evolution IMAP folders.
This IS an Evolution BUG! I have done the SAME operation with the SAME machine, with the SAME server on the SAME connection, in Kmail and Thunderbird, and I have NEVER lost an email. UNTIL NOW! I have lost a bunch of valuable messages, becasue of THIS BUG. THIS MUST BE REOPEN.
By the Way, my case was moving Folders From the Server to the Local Folders.
Some people have this problem on Ubuntu (see https://bugs.launchpad.net/ubuntu/+source/evolution/+bug/117896), so do I. I paste what I have written there : I am experiencing this bug. It is a critical one to me too, because it can lead to the loss of many e-mails, and I have actually lost some before noticing there were still available when checking "show deleted messages". I tell you how I can reproduce it on my machine : I have a professional pop account, an imap account on Free.fr and I access my gmail account through imap. I want to switch to gmail so I copied all my message from the local folders to gmail. It worked almost perfectly (except some subfolders thad failed to be created but I think it is gmail related). Then I tried to copy my messages from the imap account to gmail : no mail were transfered. I then tried to move manually some mails from the imap Inbox to gmail : the messages simply disappeared and I had to check "show deleted messages" to recover them. Please at least prevent users to do such dangerous operations or warn them, mails are very important data. I use an up to date Hardy 64bits and I'm connected to the internet by ethernet. I attach a log obtained by opening evolution with "CAMEL_DEBUG=all evolution >& evo.log", trying to move one message from the imap account to gmail and closing evolution. I have not been able to see anything suspicious but maybe you will. Update : I have just tried all possible transfers with two messages I sent to myself and everything worked fine. I tried with another mail that wasn't from me and transfers from local to gmail and from imap to gmail failed. It seems to affect messages randomly, but an affected message would fail with a 100% probability. It seems that mails with images are more likely to fail... I hope I can help, thank you. Jim.
Created attachment 108786 [details] Log file
I'm the original submitter of the bug in Ubuntu. I reword it all shortly (because it is getting pretty old already): 1) this bug happened on Ubuntu only, on various Ubuntu official release versions (3 so far) 2) it happens with Dovecot and Courier-IMAP as server (on various Debian stable releases) 3) it happens mainly when using automatic filters in Evolution, but not only 4) it does *not* happen with other IMAP clients (Thunderbird on Windows being one) 5) it does *not* happen on Debian as client (from Etch to current Sid) 6) it is fully and easily reproducible for my users I submitted logs when happening, I described everything, including the exact versions of all softwares involved, and hardware power data. If I can contribute somewhat more, don't hesitate to ask. I'm a long-time GNOME, Evolution and Debian user and supporter and would not trust Evolution on Ubuntu anymore as long as this bug is not fixed reliably. Thanks
Let me add some more info: 1) I do not use (nor do my users) the local folders at all 2) The problem is that the e-mails are marked as deleted while they have not been successfully copied to their destination, and to keep disk usage reasonable, I'm configuring Evolution to purge Trash at exit, which is when the e-mails are actually lost. So, strictly speaking, Evolution is not loosing mails, it is just marking them as deleted without having any copy anywhere else 3) From what I see (and described), I really doubt 436802 is a distinct bug, as to me
The log from Jim is interesting... sending : B00006 APPEND INBOX (\Seen) {27029} received: + go ahead received: B00006 OK Success sending : B00007 SELECT INBOX received: * FLAGS (\Answered \Flagged \Draft \Deleted \Seen) received: * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] received: * OK [UIDVALIDITY 595374242] received: * 0 EXISTS received: * 0 RECENT received: * OK [UNSEEN 0] received: * OK [UIDNEXT 4364] received: B00007 OK [READ-WRITE] INBOX selected. (Success) notice that according to Jim's log, Evolution is appending a message (of size 27029 bytes) to INBOX. The server first responds with a '+' response, which tells the client to go ahead and dump the message to the socket, then Evolution does so (not in the log), and then the server replies back with "OK" suggesting that the message has been successfully appended. However... when Jim selects the INBOX, the server claims that there are no messages in INBOX. Seems to suggest a server bug in Jim's case :( To everyone else with this symptom, can you attach logs of the error occurring? The logs so far provided have shown only server bugs but since there are at least a handful of people having this problem (unlikely to all have identical IMAP servers?), there may be an Evolution bug yet, but it's impossible to tell without more information. thanks.
Actually, it is the gmail server. So I am surely not the only one to use it and maybe something can be made on the software side. With some messages, the bug is fully reproductible. Moreover I have already got the same issue with another "standard" imap account before. Maybe Evolution could check if the message is actually uploaded before deleting it? Is there any drawback to doing so? I attach the source of a message that I can't upload to gmail servers, (I can to my other imap account) either from a local folder or from a remote imap folder.
Created attachment 108814 [details] Message that I can't upload to gmail
it's a bit ridiculous to go to the trouble of checking if the message(s) are actually there when the server replies "success!". The complexity and overhead of doing so is also not trivial.
Yes I agree, but from the point of view of an user, it is completely unbelievable that the mail client could allow some mails to be lost. Of course everybody should make backups, but most of people trust their mail client (I used to). Moreover it is why I use IMAP : even if my computer dies my mails are always instantaneously available to me.
I tried with thunderbird and there is the same problem, so it is definitely not Evolution's fault. However, I am still thinking that at least a warning would be appropriate like Warning : You are about to move messages to an IMAP server. A bug in the server could prevent your messages from being uploaded. Please don't upload too much messages at a time and check that everything worked fine after the transfer. In case of a problem, you can restore your messages from the trash. With a checkbox to disable the warning. What do you think?
I use 1and1.fr IMAP server and I have the same problem than Jim's since few month : I lost some emails (using filters, or not), and I never encountered this problem with Evolution before. I see this bug's status is "resolved" and the issue is "notabug". So I suppose it's mean that it's a server bug ; a dovecot, gmail, 1and1... server bug. But the problem is still there : some users lost emails. And there is no proposed solution to resolve this. So, I think it would be fine for users to give them a list of compatible servers before they are going to install Evolution. In my case, I lost some emails, and I can't move others. With this bug's status I understand that my provider won't be supported by evolution.So I have to search an other email client, that would be compatible with my provider. But this bug is old (ten months and maybe more). And If I had read a list of compatible (or uncompatible) servers, I wouldn't lost any emails.
The problem also happens here. Some random mails get lost while dragged and dropped to another folder. It is hard to log since it only happens once or twice a day. I saw it with different mailservers (courier, cyrus, even uw-imap). It seems to happen more often if there are several tasks: - Move one message - While the message is still being moved (see status bar) move another message - One of the messages may be missing
Please see http://bugzilla.gnome.org/show_bug.cgi?id=509721 and check whether it still happens with the latest version. (Seems to work for me now)