GNOME Bugzilla – Bug 664637
Deleted messages appear in Inbox
Last modified: 2012-04-28 12:21:09 UTC
evolution-ews shows deleted messages in Inbox which can't be opened and which won't disappear with Send/Receive, one must individually delete each such message. To reproduce, close all Exchange mail clients, open Evolution, open an earlier message, send a test message to yourself, wait for it to arrive, then close Evolution while the earlier message is still opened. Go to OWA, open the test message, delete it, and log out from OWA. Now open Evolution again and notice how the test message is still in Inbox even after Send/Receive and if selecting it an error is displayed: "Unable to retrieve message. The specified object was not found in the store." Alternatively, if one opens the test message before closing Evolution, closes Evolution, and deletes it with OWA, then after closing OWA and starting Evolution the message appears in Inbox as if it had not been deleted. This is problematic for people who use OWA, after a while Evolution is showing a lot of non-existing messages in Inbox which must be individually deleted.
I can reproduce this too, though it seemed like working fine with Inbox. The bug #665065 can be also related, maybe the SyncFolderItems request doesn't do one more thing right here. I changed similar parts in evolution-mapi to this algorithm: a) folder opened for the first time (they are supposed to stay in memory once opened): a1) get list of all items with their last modification time a2) fetch summary for all new items and those to whom the last modification time changed (it is saved in the CamelMessageInfo in evo-mapi) a3) delete what wasn't found on the server, but is in local cache b) folder is refreshing again b1) if count of items is different from the last check do full update like in a) b2) else get changes since the last check and update only those items There is a chance that user will delete one message and add another, thus the counts in b1) will match, even the folder content will differ, but it will be fixed as soon as evolution is run again. I suppose that implementing FindItem (as mentioned in the above mentioned bug) and using similar approach to that from evo-mapi will fix this. But it seems like too much ews core work for me at the moment.
Marko, please could you run Evolution from the command line with EWS_DEBUG=2 set in the environment (run 'EWS_DEBUG=2 evolution'). Capture the output to a file and show it to me. The SyncFolderItems call is the right thing to do here; it's the equivalent of SELECT+QRESYNC in IMAP, or the Sync call in ActiveSync. It's basically "tell me what changed since *this* bookmark". Reverting to the old, slow method of listing *everything* in the folder, every time we open it (like IMAP without QRESYNC) is not a particularly attractive option. The server *should* tell us about deleted items, in its SyncFolderItems response. I'd like to see that traffic from your debug output, to make sure we're acting appropriately on that information. Please let me know the ItemId of the offending item. And ideally if you can capture the part where the new mail is *first* seen by Evo-EWS, that would help too.
(In reply to comment #2) > Reverting to the old, slow method of > listing *everything* in the folder, every time we open it (like IMAP without > QRESYNC) is not a particularly attractive option. Just a note on this, it doesn't do that always, only the first time and when count of items changed. Otherwise it fetches only changed items since the last test, which decreases amount of fetched information significantly. The Restriction in FindItem is there for this :)
Thanks for the offline debugging help with this. This issue is that we end up using a stale SyncKey for the folder in the following circumstance... -> SyncFolderItems (since SyncKey AAA) <- New mail 'foo', SyncKey now BBB. < Evo stores the new mail, but *fails* to store the new SyncKey because the folder summary wasn't marked as dirty > < Evo quits > < Mail 'foo' deleted by external means (IMAP, OWA, etc.) > < Evo restarts > -> SyncFolderItems (since SyncKey AAA) <- No changes, SyncKey now CCC. This, Evolution misses out on getting told about the deletion because its SyncFolderItems call after restart is asking "what changed since AAA?", when it *should* be asking "what changed since BBB?". I've committed a quick fix to mark the summary as dirty to ensure that the new sync key *does* get saved to the database, but that still leaves us with a race condition. A crash at the wrong time can still reproduce the incorrect behaviour, because we save the new messages and *then* we save the new SyncKey. If we were to do things the other way round (save SyncKey and *then* new messages) we'd still have a race condition in which we could miss out on changes if we crash after saving a new SyncKey but not all the changes that come with it. We need to do this *atomically*, which I think means removing the camel_folder_summary_save_to_db() call from camel_ews_utils_sync_created_items() and similar functions, and doing it just *once* for the whole set of changes. We should double-check that we don't have similar issues for ActiveSync and IMAP+QRESYNC (all of which would be my fault, so this is something of a note-to-sefl...)
(In reply to comment #4) > We should double-check that we don't have similar issues for ActiveSync and > IMAP+QRESYNC (all of which would be my fault, so this is something of a > note-to-sefl...) Do not be so bad on you. I realized I'm also failing in saving the summary changes in my recent eov-mapi changes. I had it dirty, but I didn't save it. I'm thinkingof adding "save_to_db" call ito its dispose function, to ensure that changes will be saved. The save_to_db is a void function when summary is not touched, thus should be correct.
*** Bug 669741 has been marked as a duplicate of this bug. ***
Downstream bug report about the same from 3.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=809794 Delete message in ews doesn't delete message on the folder.
I tried to reproduce this with 3.4.0, deleting message from my Exchange 2007 Inbox, and when I move to other folder (which is basically the moment when locally done changes are synchronized towards the server), the message is properly moved from Inbox to Delete Items. Getting rid of the Deleted Items messages is not that simple, based on the code, an Expunge should be done, while the Deleted Items messages to be permanently removed are supposed to be deleted again. It's slightly unexpected behaviour for Deleted Items folder, though for that the bug #653631 fits better. That said, I can delete messages from Inbox, and when I move to another folder then they are properly moved to Deleted Items, which is properly indicated in Exchanges OWA interface. Did anyone of you do anything differently, like didn't move to a different folder or any other thing?
I see issues on 3.4.1 (Fedora 17) with both moving and deleting of messages. When I check my mail using the Exchange 2007 web interface it still shows all the moved and/or deleted messages. Let me know what debug you need me to do.
I think the connection debugging, like what tells ews to the server and what it replies may help here. I'll contain many private information, thus feel free to send it only to me. Steps for the log: a) run evolution as this: $ EWS_DEBUG=2 evolution &>log.txt b) mark few messages and either move them into another folder or delete them c) move to a different folder d) move back, verify they are gone in evolution e) close evolution - logging is done f) verify in server's web interface that the message you moved or deleted are still in that folder, where evolution have them gone In d) you should see some folder updating activity in the status bar. The f) is to verify that the issue occurred.
Created attachment 212350 [details] evolution log
1. I deleted a few messages (in the folder 'deleted', they do not appear) 2. The two messages I moved to a folder 'deleted' manually (in outlook they came in the folder deleted)
Hmm, as far as I can tell, the log should contain DeleteItem request, but I cannot find it there. It is included in my log, with its response, but not before I left the folder and let is store (there is such activity shown in the status bar when the folder is stored). I see it in my log even when deleting few messages and closing evolution immediately. This is with Exchange 2007 server.
Created attachment 212580 [details] evolution delete log (delete 1 message)
Created attachment 212581 [details] evolution move log (successful move 1 message to 'delete folder')
Thanks for the update. The first log is similar to the previous one, no DeleteItem request. The second log contain multiple MoveItem requests, with one response and then also with folder synchronization, which received info about just moved message as part of Deleted items. I'm afraid the communication log as such cannot show more, I'll cook a debug patch for places which should be affected by this.
Created attachment 212592 [details] [review] ews patch for evolution-ews; After reading the code which affects this I realized the issue here, pity I didn't notice it earlier. The thing is that if the message changes its state, and is also deleted, then ews only updates states of the message, but doesn't delete it from the server. I was testing only with messages which were already downloaded on my machine, but I guess you deleted message which just arrived, thus it had changed its unread flag to read. This patch checks whether message with changed flags was also deleted, and if so, then it deletes it from the server as well. I built a test package for Fedora 17 with the patch included for you at: http://koji.fedoraproject.org/koji/taskinfo?taskID=4014537
Created commit d364eff in ews master (3.5.1+) Created commit 84b8028 in ews gnome-3-4 (3.4.2+)
> I built a test package for Fedora 17 with the patch included for you at: > http://koji.fedoraproject.org/koji/taskinfo?taskID=4014537 Testing this now, will it sync all previously deleted emails or is there a way to force a sync to delete these?
The patch seems to fix the issue for me (with 3.4.1 / Exchange 2010). Great work!
Seems patch work only in Inbox folder. I removed all messages in 'delete folder' under Evolution, but this not affected to Outlook.
It seems to work OK for all new moves/changes/deletions. It doesn't seem to sync re-sync out the existing issues so there's still issues. Is there a way to force a sync?
(In reply to comment #21) > Seems patch work only in Inbox folder. I removed all messages in 'delete > folder' under Evolution, but this not affected to Outlook. Ii didn't try this case, I'll check it and update the bug. (In reply to comment #22) > It seems to work OK for all new moves/changes/deletions. It doesn't seem to > sync re-sync out the existing issues so there's still issues. Is there a way to > force a sync? Do you see them with View->Show deleted messages? If yes, then the only way is to touch the message on flags, easiest probably by selecting all deleted emails, mark them as important (beware, it undeletes messages), then delete them and mark as unimportant. Then, when you move away from the folder, they will be deleted.
> Do you see them with View->Show deleted messages? If yes, then the only way is > to touch the message on flags, easiest probably by selecting all deleted > emails, mark them as important (beware, it undeletes messages), then delete > them and mark as unimportant. Then, when you move away from the folder, they > will be deleted. Yes, I can see them then. Is there a way to filter/sort by the deleted status? There's 1000s of them. Also tried "Empty Wastebasket" from the file menu and it didn't seem to have an effect. Not sure if this is the same bug or a different one.
(In reply to comment #24) > Yes, I can see them then. Is there a way to filter/sort by the deleted status? > There's 1000s of them. I thought of that, but I'm not aware of any, unfortunately. > Also tried "Empty Wastebasket" from the file menu and it didn't seem to have an > effect. Not sure if this is the same bug or a different one. There is filled bug #653631 for it, though the ews uses real folder "Deleted items", thus it would delete messages from there, not those marked as deleted from other folders.
(In reply to comment #23) > (In reply to comment #21) > > Seems patch work only in Inbox folder. I removed all messages in 'delete > > folder' under Evolution, but this not affected to Outlook. > > Ii didn't try this case, I'll check it and update the bug. Aah, I see it. Let's deal with this in bug #653631 too.
> Aah, I see it. Let's deal with this in bug #653631 too. Ok. How can I help?
(In reply to comment #25) > (In reply to comment #24) > > Yes, I can see them then. Is there a way to filter/sort by the deleted status? > > There's 1000s of them. > > I thought of that, but I'm not aware of any, unfortunately. Looking at the code, it seems like calling Expunge (Ctrl+E) in the folder should move deleted messages into Deleted Items folder.
(In reply to comment #27) > > Aah, I see it. Let's deal with this in bug #653631 too. > Ok. How can I help? Let's move to bug #653631, I have patch ready.
(In reply to comment #28) > Looking at the code, it seems like calling Expunge (Ctrl+E) in the folder > should move deleted messages into Deleted Items folder. Oops, actually, it'll delete those messages permanently, _without_ moving them to Deleted Items. My fault.
(In reply to comment #30) > (In reply to comment #28) > > Looking at the code, it seems like calling Expunge (Ctrl+E) in the folder > > should move deleted messages into Deleted Items folder. > > Oops, actually, it'll delete those messages permanently, _without_ moving them > to Deleted Items. My fault. So it seems. In the case where the mails had been moved to another folder it just deletes the old email in the original folder, correct?
(In reply to comment #31) > So it seems. In the case where the mails had been moved to another folder it > just deletes the old email in the original folder, correct? The expunge is run on the currently selected folder (when invoked by Ctrl+E), and all messages marked as deleted in that folder are deleted from the server permanently. You can also purge your local cache for messages at ~/.cache/evolution/mail/<ews-account-id>/ and let evolution fetch your messages from scratch, which will show you exact view from the server, thus those currently "deleted" messages in evolution will not be marked as deleted anymore.
Seems same problem have IMAPI connector. I also use Gmail account in Evolution. And when I has removed messages from inbox folder in evolution, they still displays in inbox folder over Gmail web interface.
(In reply to comment #33) > Seems same problem have IMAPI connector. I also use Gmail account in Evolution. > And when I has removed messages from inbox folder in evolution, they still > displays in inbox folder over Gmail web interface. It's a different issue, and the way how GMail implements their IMAP. They require a real trash folder to have this working properly (or you can move messages to their trash folder, instead of deleting them like usual). The IMAP provider (not IMAP+) offers setting real trash on the Defaults tab when editing the account. Such change requires evolution's restart.
I change "IMAP +" to "IMAP" and set up all folder then restart Evolution, but this not help to solve problem.
Created attachment 212936 [details] IMAP folder settings
(In reply to comment #35) > I change "IMAP +" to "IMAP" and set up all folder then restart Evolution, but > this not help to solve problem. This bug is to do with the exchange web services (evolution-ews) package and nothing to do with IMAP. Please report bugs to do with IMAP against the appropriate component and don't hi-jack a bug that has nothing to do with IMAP
We should really make that 'real trash' thing more generic, pushing much of the logic into the core camel code instead of the provider. Then it can more easily be used by other providers.
I agree with David Woodhouse. Regarding the evolution-ews to 'delete folder' also does not moved confirmed or rejected invitation to the meeting.