GNOME Bugzilla – Bug 571206
Index files on Maildir accounts are not updated on move
Last modified: 2009-08-12 11:35:15 UTC
Please describe the problem: When mail is moved from an mbox folder (e.g. the inbox) to a Maildir folder, the index in the maildir folder is not updated. As a result the mail becomes inaccessible to the user. Checking the physical directory on the disk shows that the mail is actually present but within Evolution it no longer shows up in either the origin or destination folder. Using the refresh and expunge options has no effect. Stopping and restarting Evolution also has no effect. I've marked this as "normal" but it really is a pretty serious problem as it means I have to keep all my email in the inbox to prevent it from vanishing. This behavior started when I updated to Fedora 10. Steps to reproduce: 1. Set up Evolution to use local Maildir storage 2. Copy mail from one of the Evolution mbox folders to a Maildir folder 3. Mail vanishes without a trace. Actual results: The moved mail is no longer visible or accessible to the user. Expected results: The moved mail should show up in the destination folder. Does this happen every time? Yes. 100% of the time. Other information: Maildir functionality has been seriously degrading in Evolution with each successive release for some time. See these two related bugs, which have also bitten me: 318808 (filter actions do not update index files on Maildir accounts) and 318515 (Maildir flags do not get updated). At this point Evolution is basically unusable for Maildir and I will be dropping it and moving to a less buggy program unless some improvement seems forthcoming.
I would look at it pretty soon [in a few days]
I just ran into this using 2.24.5. It doesn't happen often for me though.
Sorry, wrong bug. I ran into #573177. But I can confirm that this bug exists in 2.24.5, I just tried it and could reproduce it.
I've come up with a work around that makes Evolution just barely usable again. 1. start evolution and check mail as usual 2. allow filters to move incoming mail to other folders, where this evolution bug causes it vanish 3. close evolution 4. delete all *.ibex.index.data, *.ibex.index, *.cmeta from each maildir directory (a script helps make this less tedious) 5. restart evolution, it will build new indices which will be correct until the next time mail is moved from one folder to another So instead of leaving evolution running all day and letting it check mail automatically, it's now necessary to manually start it up and go through the above process each time I want to check my mail. Needless to say this is getting a bit tedious. Might I suggest an alternative solution would be to rename Evolution to "De-Evolution" since that seems to better describe it's functionality over the last year or so. Perhaps some DEVO branding and music to go along with that would be nice. :)
*** Bug 573177 has been marked as a duplicate of this bug. ***
Created attachment 132832 [details] [review] proposed eds patch for evolution-data-server; This helped me.
Created attachment 132895 [details] [review] same patch as 132832 based on 2.24.x branch I applied Milan's to the 2.24.x branch and can confirm that it fixes this problem. The patch didn't apply cleanly (the void warning fix already was added, some line changes), so I'm posting a clean copy for those who might want a fixed 2.24.x instead of upgrading to 2.26.
--- camel/providers/local/camel-maildir-summary.c (revision 10220) +++ camel/providers/local/camel-maildir-summary.c (working copy) @@ -294,7 +294,7 @@ static CamelMessageInfo *message_info_ne mdi->info.info.uid = camel_pstring_add (camel_folder_summary_next_uid_string(s), TRUE); /* handle 'duplicates' */ - info = camel_folder_summary_uid(s, uid); + info = camel_folder_summary_peek_info (s, uid); if (info) { d(printf("already seen uid '%s', just summarising instead\n", uid)); camel_message_info_free(mi); Milan, everything else in the patch is fine, except this. You free the MI you use and rely on the MI you get from summary. You need to own a ref, once you get from summmary, if you want to use it further. mdi = (CamelMaildirMessageInfo *)(mi = info); You convert that as mdi and use later on and return it. Means that you must ref it. Apart from it, its fine to commit.
(In reply to comment #8) > You convert that as mdi and use later on and return it. Means that you must ref > it. Are you sure? The very similar code is in mbox summary too. Here's the function of a question body: CamelMessageInfo * camel_folder_summary_peek_info (CamelFolderSummary *s, const char *uid) { CamelMessageInfo *info = g_hash_table_lookup(s->loaded_infos, uid); if (info) camel_message_info_ref(info); return info; }
Created commit 05fe41d.
Cool, good to a see a fix for this! Will the fix be applied to the existing 2.24.x version that's part of Fedora 10 or will I have to wait for Fedora 11 to get the fix? Looks like 2.24.5 is the latest available so far on F10.
(In reply to comment #7) > I applied Milan's to the 2.24.x branch and can confirm that it fixes this > problem. The patch didn't apply cleanly (the void warning fix already was > added, some line changes), so I'm posting a clean copy for those who might want > a fixed 2.24.x instead of upgrading to 2.26. I've used the patch a while now. The problem with new mail in maildir not showing up in the folder view has occurred again during in the meantime, so this patch doesn't fix that problem.
(In reply to comment #11) > Cool, good to a see a fix for this! Will the fix be applied to the existing > 2.24.x version that's part of Fedora 10 or will I have to wait for Fedora 11 to > get the fix? Looks like 2.24.5 is the latest available so far on F10. There is no planned update in 2.24 version. (In reply to comment #12) > I've used the patch a while now. The problem with new mail in maildir not > showing up in the folder view has occurred again during in the meantime, so > this patch doesn't fix that problem. Do you have any steps how to reproduce/trigger that? What were you doing? Like filters applied on the background and mails not shown in the active folder, though they were filtered there?
Created attachment 133412 [details] [review] test patch for evolution-data-server; Patrick, could you give a shot to this test patch, please? I believe you'll see any issue much quicker than me. Thanks in advance.
(In reply to comment #13) > (In reply to comment #12) > > I've used the patch a while now. The problem with new mail in maildir not > > showing up in the folder view has occurred again during in the meantime, so > > this patch doesn't fix that problem. > > Do you have any steps how to reproduce/trigger that? Nothing that triggers it reliably. > What were you doing? Update maildir with offlineimap, hit "send/receive" in Evolution. > Like > filters applied on the background and mails not shown in the active folder, > though they were filtered there? I have stopped using filters in Evolution. In this particular installation of Evolution there are no filters defined at all. I tried the "test patch" with 2.24.x. It applied almost cleanly and the two conflicts were easy enough to resolve manually. The problem occurred again today, despite using the recompiled libcamellocal.so with all of these patches applied. This is getting off-topic for this particular bug, unless you want to rename it into a more generic "maildir folder summary not updated" bug and reopen it. Let me know whether you want another bug reported opened... I'll also update to 2.26 at some point, but don't have time for it right now.
I hope we are still doing working on the same issue, thus reopening. I will try to reproduce on 2.26.1, so I will appreciate as much details what to do as possible. > Update maildir with offlineimap, hit "send/receive" in Evolution. What does this mean in reality? In which folder are you staying when hitting "Send/Receive" button and what are expected results on this, with what data, and what are in the UI? Basically, I assume you are staying on some other than the folder where was new message received, in the UI, say On This Computer/Inbox. One message came to 'new' directory of the maildir structure. Hitting Send/Receive will move this mail from 'new' to 'cur' directory, and will notify the evolution UI about one new message in the folder. (I use 'folder' for evo UI, and 'directory' for on-disk-state). a) The number of unread in a folder tree of this folder should increase b) when selecting the folder I should see the one new. c) the same when the folder is selected should result in new message shown too When does it stop to work for you? What do you do to fix it? Like evo restart is enough, reselecting folder is enough, or still forced to delete some evo files?
(In reply to comment #16) > I hope we are still doing working on the same issue, thus reopening. > > I will try to reproduce on 2.26.1, so I will appreciate as much details what to > do as possible. > > Update maildir with offlineimap, hit "send/receive" in Evolution. > > What does this mean in reality? In which folder are you staying when hitting > "Send/Receive" button and what are expected results on this, with what data, > and what are in the UI? I haven't paid much attention to which folder was displayed when hitting "send/receive". > Basically, I assume you are staying on some other than the folder where was new > message received, in the UI, say On This Computer/Inbox. I'm pretty sure it happens also when viewing the folder to which offlineimap has added mails. I'm confident about that because most of the time I am in my Inbox. > One message came to > 'new' directory of the maildir structure. Yes, I can see that offlineimap puts new mails there. > Hitting Send/Receive will move this > mail from 'new' to 'cur' directory, and will notify the evolution UI about one > new message in the folder. (I use 'folder' for evo UI, and 'directory' for > on-disk-state). > a) The number of unread in a folder tree of this folder should increase Correct. This seems to work reliably. > b) when selecting the folder I should see the one new. Usually it does, but not always. If it failed, then moving between folders doesn't fix it. If I mark an email in the affected folder as deleted and use "expunge", then the new and unread mails appear. > c) the same when the folder is selected should result in new message shown too So far I have not observed that it makes a difference whether the folder is selected or not. I'll pay more attention in the future to that. > When does it stop to work for you? Randomly, as far as I can tell. > What do you do to fix it? Either the "delete + expunge" trick or deleting folders.db while Evolution is stopped.
hrm, happened only once for me, when I enabled a maildir account with some unspecified folders.db content (probably quite old). But after some clicking here and there, closing evolution and so on, it started to work again. Patrick, can I try to produce some very chatty patch, which will monitor things it does in all your local providers, to try to catch what's going wrong?
(In reply to comment #18) > hrm, happened only once for me, when I enabled a maildir account with some > unspecified folders.db content (probably quite old). But after some clicking > here and there, closing evolution and so on, it started to work again. I have kept stats today and oaid more attention on when new mail is detected. It definitely failed three times today, out of ~30 times when new mail arrived (haven't counted all of those). I haven't found a relevant difference between using "send + receive" and switching folders (which also refreshes the index). It failed both when the INBOX was the current folder and when being outside the INBOX and then switching to it. > Patrick, can I try to produce some very chatty patch, which will monitor things > it does in all your local providers, to try to catch what's going wrong? Sure, but if you can, please make it against 2.24.x. I'm not on 2.26 yet and probably would have trouble applying an extensive patch against 2.26 onto 2.24.
Created attachment 133644 [details] eds-2-24 test patch for eds 2-24 there is nothing inventive in this test patch, I only merged those two previous, and added some fix into camel/providers/local/camel-local-folder.c:local_refresh_info Apart of that it's veeery chatty (probably log to file to ensure nothing will be missing), though I'm not sure whether I covered all the interested parts. Let's see. (I do not compile 2.24 eds any more, but I tried to apply the patch on those sources; I hope you'll be able to fix any typo I might produce.) Thanks for your help.
(In reply to comment #20) > Created an attachment (id=133644) [edit] > eds-2-24 test patch I'm running 2.24 with that patch now. Whenever I want check for mail I do: * enter INBOX * run offlineimap * switch from INBOX to empty Junk folder of the same account and back This just failed again. The output was the following, including the output generated from switching folders with no changes before calling offlineimap: (evolution:11753): camel-CRITICAL **: camel_message_info_free: assertion `mi != NULL' failed maildir_summary_sync: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:0 maildir_summary_check: going to reload because count:74 cachesize:72 finally count: 74 maildir_summary_check: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:0 pohly@pohly-MOBL:/work/evolution-svn/obj/evolution-data-server$ date && offlineimap -a Intel Mon May 4 13:37:42 CEST 2009 OfflineIMAP 6.0.3 Copyright (C) 2002 - 2008 John Goerzen <jgoerzen@complete.org> This software comes with ABSOLUTELY NO WARRANTY; see the file COPYING for details. This is free software, and you are welcome to distribute it under the conditions laid out in COPYING. ***** Processing account Intel Copying folder structure from IMAP to Maildir Establishing connection to imapmail.intel.com:993. Syncing INBOX: IMAP -> Maildir Copy message -2 Maildir[.] -> IMAP[INBOX], LocalStatus[.] Copy message -1 Maildir[.] -> IMAP[INBOX], LocalStatus[.] Deleting 108 messages (117257, 117261, 117262, 117264, 117267, 117271, 117272, 117279, 117280, 117281, 117283, 117284, 117287, 117288, 117290, 117294, 117301, 117315, 117320, 117332, 117333, 117348, 117353, 117354, 117360, 117361, 117388, 117391, 117398, 117399, 117403, 117406, 117412, 117413, 117414, 117415, 117416, 117417, 117418, 117419, 117422, 117423, 117426, 117427, 117429, 117431, 117433, 117435, 117451, 117444, 117446, 117448, 117449, 117450, 117452, 117453, 117454, 117455, 117458, 117459, 117461, 117462, 117463, 117464, 117465, 117466, 117467, 117469, 117473, 117475, 117476, 117483, 117484, 117486, 117488, 117489, 117490, 117492, 117493, 117495, 117497, 117498, 117506, 117518, 117522, 117524, 117525, 117527, 117528, 117529, 117533, 117534, 117540, 117544, 117545, 117548, 117558, 117563, 117077, 117080, 117084, 117491, 117494, 117496, 117222, 117237, 117241, 117243) in IMAP[INBOX], LocalStatus[.] Copy message 117567 IMAP[INBOX] -> Maildir[.], LocalStatus[.] Copy message 117569 IMAP[INBOX] -> Maildir[.], LocalStatus[.] ***** Finished processing account Intel [several mails had been moved out of the INBOX, some were marked as read, two new received] pohly@pohly-MOBL:/work/evolution-svn/obj/evolution-data-server$ [now leave INBOX, increases the "unread count" to 11] maildir_summary_check: going to reload because count:74 cachesize:72 finally count: 74 camel_maildir_summary_add: /home/pohly/.evolution/mail/imapmail.intel.com/./cur/1241437067_0.12949.pohly-MOBL,U=117570,FMD5=5058f1af8388633f609cadb75a75dc9d:2,S info_set_flags: on . camel_maildir_summary_add: /home/pohly/.evolution/mail/imapmail.intel.com/./cur/1241437067_1.12949.pohly-MOBL,U=117571,FMD5=5058f1af8388633f609cadb75a75dc9d:2,S camel_maildir_summary_add: /home/pohly/.evolution/mail/imapmail.intel.com/./cur/1241437072_1.12949.pohly-MOBL,U=117569,FMD5=5058f1af8388633f609cadb75a75dc9d:2,:2, info_set_flags: on . camel_maildir_summary_add: /home/pohly/.evolution/mail/imapmail.intel.com/./cur/1241437072_0.12949.pohly-MOBL,U=117567,FMD5=5058f1af8388633f609cadb75a75dc9d:2,:2, maildir_summary_check: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:1 (evolution:11753): camel-CRITICAL **: camel_message_info_free: assertion `mi != NULL' failed maildir_summary_sync: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:1 local_sync: on . maildir_summary_check: going to reload because count:76 cachesize:74 finally count: 76 maildir_summary_check: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:0 (evolution:11753): camel-CRITICAL **: camel_message_info_free: assertion `mi != NULL' failed maildir_summary_sync: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:0 maildir_summary_check: going to reload because count:76 cachesize:74 finally count: 76 maildir_summary_check: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:0 [after switching back into the INBOX, only 9 unread emails are shown and none of the two new emails] [now mark two emails as read and move them to another folder] info_set_flags: on . info_set_flags: on . thaw: on . maildir_summary_check: going to reload because count:76 cachesize:74 finally count: 76 maildir_summary_check: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:0 maildir_get_message: no change in . maildir_summary_check: going to reload because count:18104 cachesize:3 finally count: 18104 maildir_summary_check: /home/pohly/.evolution/mail/imap.linux.intel.com/Meta/Intel: changes:0 info_set_user_flag: on Meta/Intel maildir_summary_add: changed:1 mi:0x92c40120 maildir_append_message: notify changed in Meta/Intel info_set_flags: on . maildir_summary_check: going to reload because count:76 cachesize:74 finally count: 76 maildir_summary_check: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:0 maildir_get_message: no change in . maildir_summary_check: /home/pohly/.evolution/mail/imap.linux.intel.com/Meta/Intel: changes:0 info_set_user_flag: on Meta/Intel maildir_summary_add: changed:1 mi:0x92c40178 maildir_append_message: notify changed in Meta/Intel info_set_flags: on . thaw: on Meta/Intel thaw: on . [expunge INBOX] maildir_summary_check: going to reload because count:76 cachesize:74 finally count: 76 maildir_summary_check: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:0 (evolution:11753): camel-CRITICAL **: camel_message_info_free: assertion `mi != NULL' failed maildir_summary_sync: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:1 local_sync: on . maildir_summary_check: going to reload because count:74 cachesize:72 finally count: 74 camel_maildir_summary_add: /home/pohly/.evolution/mail/imapmail.intel.com/./cur/1240915762_0.10115.pohly-MOBL,U=117247,FMD5=5058f1af8388633f609cadb75a75dc9d:2,S maildir_summary_check: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:0 maildir_get_message: no change in . [two new emails show up in the INBOX] Does that help? The assertion `mi != NULL' is triggered when moving away from the INBOX into the Junk folder; it's stack back trace is:
+ Trace 215089
mdi in maildir_summary_sync() is also NULL when info is NULL, so other than this redundant camel_message_info_free() nothing is done. Not sure whether this mdi/info NULL case is supposed to happen.
(In reply to comment #21) > Does that help? Yup, thanks for the update. I think I'm getting some clues about the issue, but still not completely sure. What I think from the above, the issue triggers mostly when there are some removes in the folder, together with insertions (maybe also with flag changes from the outside of evolution). The log shows the "folder_changed" event was sent (those lines with "changed: 1"), but the message information couldn't be found, as shows the runtime warning within sync operation. Thus I guess it couldn't find the new message infos, thus it failed to update message list in the UI. I'm not sure why it reloads from DB twice, it should do that only once (within some time interval, which is 5 minutes). > mdi in maildir_summary_sync() is also NULL when info is NULL, so other than > this redundant camel_message_info_free() nothing is done. Not sure whether this > mdi/info NULL case is supposed to happen. It seems the code checks for non-NULL values on every place but the call for a camel_message_info_free. Even it' bad, then we know that something goes wrong here, which is fine. I'll try to simulate such steps here and let's see.
Bad. So I applied a test patch from comment #14 on my git master here, and tried to simulate offlineimap operations in a way I think they should work (adding new mail into 'new' folder (not the 'cur' nor 'tmp' folder) and deleting some mails from 'cur' folder under Inbox). None of these actions work as for you. I see those new mails, I do not see a runtime warning on the evolution console when some mails had been deleted. Even the same without the test patch. I'm afraid that some change which was committed into the former trunk only, not included in your stable 2.24.x version, influences this as that much. Or some breakage of your folders.db of the maildir account? Nothing else I can think of at the moment. I'm proposing to include a test patch in the sources anyway, as I believe it's a correct change.
(In reply to comment #23) > I'm afraid that some change which was committed into the former trunk only, not > included in your stable 2.24.x version, influences this as that much. Or some > breakage of your folders.db of the maildir account? I already deleted folders.db a few times to get back into a sane state. As I read on the mailing list, that would have removed meta information about emails that isn't stored anywhere else, but luckily I don't have that. > Nothing else I can think of > at the moment. Thanks for giving it a try. I guess I'll have to upgrade to 2.26 as soon as time permits and then report back when it still occurs.
Milan, I think you are right. But are you sure no locks etc are required ? I see the camel-mbox folder doing the same with locks. It is possible that you have two threads doing camel_local_summary_check, leading to corruption. But otherwise, the patch is ready to meet the git.
(In reply to comment #25) > But are you sure no locks etc are required ? I'm not sure, I copied some code from mbox and hoped it'll work :) What locks did you mean? I see they are not used always, as the 'sync' methods doesn't use it, but the 'append_message' uses it in mbox (the call of camel_local_folder_lock). For example maildir_append_message doesn't use it either before the patch. Strange?
Created attachment 137195 [details] [review] proposed eds patch ][ for evolution-data-server; Thinking of it, after our chat on IRC, I tend to believe the missing locks are causing trouble Patrick described above. Let's hope it :) This is quite similar patch as the test one, just added locking into it.
Patrick, can you give this a try?
(In reply to comment #28) > Patrick, can you give this a try? I'm still not on 2.26.x and the patch doesn't apply due to non-trivial changes ( maildir_append_message() looks different), so I can't test it right now. I definitely want to to upgrade to 2.26 as part of reinstalling a new laptop, but that hasn't arrived yet.
*** Bug 587259 has been marked as a duplicate of this bug. ***
The problem still occurs with the latest "master" branch, compiled today. Milan's patch still applies to that source, so I recompiled and will check whether it still fails.
(In reply to comment #31) > The problem still occurs with the latest "master" branch, compiled today. > Milan's patch still applies to that source, so I recompiled and will check > whether it still fails. Yes, it still fails even with the patch from comment #27: I just ran offlineimap, moved between folders, unread mail count was updated, but the new mails didn't show up in the mail list. Delete+expunge made them show up. Milan, how do we proceed with this? Any suggestions?
Created attachment 140331 [details] [review] proposed eds patch ]I[ for evolution-data-server; Only one suggestion, please try this one :) It seems to work fine for me. Maybe still some corner cases I'm not aware of there, but otherwise should work fine. This patch is again quite similar to the previous one, only difference is maildir_summary_check. I realized that the offlineimap (you forced me to install and try it :) ) is storing files to maildir folders with ":X" suffix, but we use UIDs within maildir without that, thus evo had been notified with a wrong UID.
Beh, this > + nm =strrchr (newname, ':'); should be this + nm = strrchr (newname, ':');
(In reply to comment #33) > Created an attachment (id=140331) [edit] > proposed eds patch ]I[ > > for evolution-data-server; > > Only one suggestion, please try this one :) I have used this for over a day now, no problems so far. It seems that this really fixes the problem. > It seems to work fine for me. Maybe still some corner cases I'm not aware of > there, but otherwise should work fine. This patch is again quite similar to the > previous one, only difference is maildir_summary_check. I realized that the > offlineimap (you forced me to install and try it :) Hey, it's worth it ;-) > is storing files to > maildir folders with ":X" suffix, but we use UIDs within maildir without that, > thus evo had been notified with a wrong UID. Hmm, I don't see that with offlineimap 6.1.2. Mails downloaded by offlineimap: new/ 1250062876_0.8211.pohly-mobl1,U=127666,FMD5=5058f1af8388633f609cadb75a75dc9d:2, 1250062877_0.8211.pohly-mobl1,U=127662,FMD5=5058f1af8388633f609cadb75a75dc9d:2, 1250062877_1.8211.pohly-mobl1,U=127664,FMD5=5058f1af8388633f609cadb75a75dc9d:2, 1250062877_2.8211.pohly-mobl1,U=127663,FMD5=5058f1af8388633f609cadb75a75dc9d:2, 1250062877_3.8211.pohly-mobl1,U=127665,FMD5=5058f1af8388633f609cadb75a75dc9d:2, What's the meaning of :X?
(In reply to comment #35) > > is storing files to > > maildir folders with ":X" suffix, but we use UIDs within maildir without that, > > thus evo had been notified with a wrong UID. > > Hmm, I don't see that with offlineimap 6.1.2. Was a misunderstanding, as explained on IRC. :X was meant as a placeholder for something like ":2," or ":2,S" (seen). That is indeed placed in the "new" folder by offlineimap. Given that the patch arguably improves the situation I suggest to accept it into 2.28. I can't comment on the code itself, though, because I know nothing about camel.
Created commit 25fa98c in eds master (2.29.1+) Created commit 7d0fc22 in eds gnome-2-28 (2.27.91+) Thanks for the testing.