After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 571206 - Index files on Maildir accounts are not updated on move
Index files on Maildir accounts are not updated on move
Status: RESOLVED FIXED
Product: evolution
Classification: Applications
Component: Mailer
2.24.x (obsolete)
Other All
: Normal normal
: ---
Assigned To: Milan Crha
Evolution QA team
evolution[maildir]
: 573177 587259 (view as bug list)
Depends on:
Blocks: 543389
 
 
Reported: 2009-02-10 17:12 UTC by Steve Rainwater
Modified: 2009-08-12 11:35 UTC
See Also:
GNOME target: ---
GNOME version: 2.23/2.24


Attachments
proposed eds patch (9.99 KB, patch)
2009-04-17 15:22 UTC, Milan Crha
committed Details | Review
same patch as 132832 based on 2.24.x branch (6.02 KB, patch)
2009-04-19 09:32 UTC, Patrick Ohly
reviewed Details | Review
test patch (8.69 KB, patch)
2009-04-27 12:35 UTC, Milan Crha
accepted-commit_now Details | Review
eds-2-24 test patch (19.46 KB, text/plain)
2009-04-30 10:17 UTC, Milan Crha
  Details
proposed eds patch ][ (8.68 KB, patch)
2009-06-22 18:47 UTC, Milan Crha
none Details | Review
proposed eds patch ]I[ (9.40 KB, patch)
2009-08-10 14:35 UTC, Milan Crha
committed Details | Review

Description Steve Rainwater 2009-02-10 17:12:21 UTC
Please describe the problem:
When mail is moved from an mbox folder (e.g. the inbox) to a Maildir folder, the index in the maildir folder is not updated. As a result the mail becomes inaccessible to the user. Checking the physical directory on the disk shows that the mail is actually present but within Evolution it no longer shows up in either the origin or destination folder. Using the refresh and expunge options has no effect. Stopping and restarting Evolution also has no effect. I've marked this as "normal" but it really is a pretty serious problem as it means I have to keep all my email in the inbox to prevent it from vanishing. This behavior started when I updated to Fedora 10.

Steps to reproduce:
1. Set up Evolution to use local Maildir storage
2. Copy mail from one of the Evolution mbox folders to a Maildir folder
3. Mail vanishes without a trace.


Actual results:
The moved mail is no longer visible or accessible to the user.

Expected results:
The moved mail should show up in the destination folder.

Does this happen every time?
Yes. 100% of the time.

Other information:
Maildir functionality has been seriously degrading in Evolution with each successive release for some time. See these two related bugs, which have also bitten me: 318808 (filter actions do not update index files on Maildir accounts) and 318515 (Maildir flags do not get updated). At this point Evolution is basically unusable for Maildir and I will be dropping it and moving to a less buggy program unless some improvement seems forthcoming.
Comment 1 Srinivasa Ragavan 2009-02-11 04:50:21 UTC
I would look at it pretty soon [in a few days]
Comment 2 Patrick Ohly 2009-03-10 14:09:32 UTC
I just ran into this using 2.24.5. It doesn't happen often for me though.
Comment 3 Patrick Ohly 2009-03-10 14:12:12 UTC
Sorry, wrong bug. I ran into #573177.

But I can confirm that this bug exists in 2.24.5, I just tried it and could reproduce it.
Comment 4 Steve Rainwater 2009-04-03 19:32:13 UTC
I've come up with a work around that makes Evolution just barely usable again.

1. start evolution and check mail as usual
2. allow filters to move incoming mail to other folders, where this evolution bug causes it vanish
3. close evolution
4. delete all *.ibex.index.data, *.ibex.index, *.cmeta from each maildir directory (a script helps make this less tedious)
5. restart evolution, it will build new indices which will be correct until the next time mail is moved from one folder to another

So instead of leaving evolution running all day and letting it check mail automatically, it's now necessary to manually start it up and go through the above process each time I want to check my mail. Needless to say this is getting a bit tedious. 

Might I suggest an alternative solution would be to rename Evolution to "De-Evolution" since that seems to better describe it's functionality over the last year or so. Perhaps some DEVO branding and music to go along with that would be nice. :)  
Comment 5 Milan Crha 2009-04-17 15:08:04 UTC
*** Bug 573177 has been marked as a duplicate of this bug. ***
Comment 6 Milan Crha 2009-04-17 15:22:25 UTC
Created attachment 132832 [details] [review]
proposed eds patch

for evolution-data-server;

This helped me.
Comment 7 Patrick Ohly 2009-04-19 09:32:53 UTC
Created attachment 132895 [details] [review]
same patch as 132832 based on 2.24.x branch

I applied Milan's to the 2.24.x branch and can confirm that it fixes this problem. The patch didn't apply cleanly (the void warning fix already was added, some line changes), so I'm posting a clean copy for those who might want a fixed 2.24.x instead of upgrading to 2.26.
Comment 8 Srinivasa Ragavan 2009-04-20 08:39:54 UTC
--- camel/providers/local/camel-maildir-summary.c	(revision 10220)
+++ camel/providers/local/camel-maildir-summary.c	(working copy)
@@ -294,7 +294,7 @@ static CamelMessageInfo *message_info_ne
 			mdi->info.info.uid = camel_pstring_add (camel_folder_summary_next_uid_string(s), TRUE);
 
 		/* handle 'duplicates' */
-		info = camel_folder_summary_uid(s, uid);
+		info = camel_folder_summary_peek_info (s, uid);
 		if (info) {
 			d(printf("already seen uid '%s', just summarising instead\n", uid));
 			camel_message_info_free(mi);

Milan, everything else in the patch is fine, except this. You free the MI you use and rely on the MI you get from summary. You need to own a ref, once you get  from summmary, if you want to use it further.

			mdi = (CamelMaildirMessageInfo *)(mi = info);

You convert that as mdi and use later on and return it. Means that you must ref it.

Apart from it, its fine to commit.
Comment 9 Milan Crha 2009-04-20 10:36:02 UTC
(In reply to comment #8)
> You convert that as mdi and use later on and return it. Means that you must ref
> it.

Are you sure? The very similar code is in mbox summary too. Here's the function of a question body:

CamelMessageInfo *
camel_folder_summary_peek_info (CamelFolderSummary *s, const char *uid)
{
	CamelMessageInfo *info = g_hash_table_lookup(s->loaded_infos, uid);

	if (info)
		camel_message_info_ref(info);
	return info;
}


Comment 10 Milan Crha 2009-04-24 17:46:58 UTC
Created commit 05fe41d.
Comment 11 Steve Rainwater 2009-04-24 18:44:44 UTC
Cool, good to a see a fix for this! Will the fix be applied to the existing 2.24.x version that's part of Fedora 10 or will I have to wait for Fedora 11 to get the fix? Looks like 2.24.5 is the latest available so far on F10.
Comment 12 Patrick Ohly 2009-04-26 08:31:09 UTC
(In reply to comment #7)
> I applied Milan's to the 2.24.x branch and can confirm that it fixes this
> problem. The patch didn't apply cleanly (the void warning fix already was
> added, some line changes), so I'm posting a clean copy for those who might want
> a fixed 2.24.x instead of upgrading to 2.26.

I've used the patch a while now. The problem with new mail in maildir not
showing up in the folder view has occurred again during in the meantime, so
this patch doesn't fix that problem.


Comment 13 Milan Crha 2009-04-27 12:08:48 UTC
(In reply to comment #11)
> Cool, good to a see a fix for this! Will the fix be applied to the existing
> 2.24.x version that's part of Fedora 10 or will I have to wait for Fedora 11 to
> get the fix? Looks like 2.24.5 is the latest available so far on F10.

There is no planned update in 2.24 version.

(In reply to comment #12)
> I've used the patch a while now. The problem with new mail in maildir not
> showing up in the folder view has occurred again during in the meantime, so
> this patch doesn't fix that problem.

Do you have any steps how to reproduce/trigger that? What were you doing? Like filters applied on the background and mails not shown in the active folder, though they were filtered there?
Comment 14 Milan Crha 2009-04-27 12:35:42 UTC
Created attachment 133412 [details] [review]
test patch

for evolution-data-server;

Patrick, could you give a shot to this test patch, please? I believe you'll see any issue much quicker than me. Thanks in advance.
Comment 15 Patrick Ohly 2009-04-28 09:29:48 UTC
(In reply to comment #13)
> (In reply to comment #12)
> > I've used the patch a while now. The problem with new mail in maildir not
> > showing up in the folder view has occurred again during in the meantime, so
> > this patch doesn't fix that problem.
> 
> Do you have any steps how to reproduce/trigger that?

Nothing that triggers it reliably.

> What were you doing?

Update maildir with offlineimap, hit "send/receive" in Evolution.

> Like
> filters applied on the background and mails not shown in the active folder,
> though they were filtered there?

I have stopped using filters in Evolution. In this particular installation of
Evolution there are no filters defined at all.

I tried the "test patch" with 2.24.x. It applied almost cleanly and the
two conflicts were easy enough to resolve manually. The problem occurred again
today, despite using the recompiled libcamellocal.so with all of these
patches applied.

This is getting off-topic for this particular bug, unless you want to rename
it into a more generic "maildir folder summary not updated" bug and reopen
it. Let me know whether you want another bug reported opened...

I'll also update to 2.26 at some point, but don't have time for it right
now.
Comment 16 Milan Crha 2009-04-28 11:32:34 UTC
I hope we are still doing working on the same issue, thus reopening.

I will try to reproduce on 2.26.1, so I will appreciate as much details what to do as possible.
> Update maildir with offlineimap, hit "send/receive" in Evolution.

What does this mean in reality? In which folder are you staying when hitting "Send/Receive" button and what are expected results on this, with what data, and what are in the UI?

Basically, I assume you are staying on some other than the folder where was new message received, in the UI, say On This Computer/Inbox. One message came to 'new' directory of the maildir structure. Hitting Send/Receive will move this mail from 'new' to 'cur' directory, and will notify the evolution UI about one new message in the folder. (I use 'folder' for evo UI, and 'directory' for on-disk-state).
a) The number of unread in a folder tree of this folder should increase
b) when selecting the folder I should see the one new.
c) the same when the folder is selected should result in new message shown too

When does it stop to work for you? What do you do to fix it? Like evo restart is enough, reselecting folder is enough, or still forced to delete some evo files?
Comment 17 Patrick Ohly 2009-04-28 12:11:25 UTC
(In reply to comment #16)
> I hope we are still doing working on the same issue, thus reopening.
> 
> I will try to reproduce on 2.26.1, so I will appreciate as much details what to
> do as possible.
> > Update maildir with offlineimap, hit "send/receive" in Evolution.
> 
> What does this mean in reality? In which folder are you staying when hitting
> "Send/Receive" button and what are expected results on this, with what data,
> and what are in the UI?

I haven't paid much attention to which folder was displayed when hitting "send/receive".

> Basically, I assume you are staying on some other than the folder where was new
> message received, in the UI, say On This Computer/Inbox.

I'm pretty sure it happens also when viewing the folder to which offlineimap
has added mails. I'm confident about that because most of the time I am in
my Inbox.

> One message came to
> 'new' directory of the maildir structure.

Yes, I can see that offlineimap puts new mails there.

> Hitting Send/Receive will move this
> mail from 'new' to 'cur' directory, and will notify the evolution UI about one
> new message in the folder. (I use 'folder' for evo UI, and 'directory' for
> on-disk-state).
> a) The number of unread in a folder tree of this folder should increase

Correct. This seems to work reliably.

> b) when selecting the folder I should see the one new.

Usually it does, but not always. If it failed, then moving between folders
doesn't fix it. If I mark an email in the affected folder as deleted and
use "expunge", then the new and unread mails appear.

> c) the same when the folder is selected should result in new message shown too

So far I have not observed that it makes a difference whether the folder
is selected or not. I'll pay more attention in the future to that.

> When does it stop to work for you?

Randomly, as far as I can tell.

> What do you do to fix it?

Either the "delete + expunge" trick or deleting folders.db while Evolution
is stopped.
Comment 18 Milan Crha 2009-04-29 19:02:11 UTC
hrm, happened only once for me, when I enabled a maildir account with some unspecified folders.db content (probably quite old). But after some clicking here and there, closing evolution and so on, it started to work again.

Patrick, can I try to produce some very chatty patch, which will monitor things it does in all your local providers, to try to catch what's going wrong?
Comment 19 Patrick Ohly 2009-04-29 20:56:54 UTC
(In reply to comment #18)
> hrm, happened only once for me, when I enabled a maildir account with some
> unspecified folders.db content (probably quite old). But after some clicking
> here and there, closing evolution and so on, it started to work again.

I have kept stats today and oaid more attention on when new mail is detected.

It definitely failed three times today, out of ~30 times when new mail arrived
(haven't counted all of those).

I haven't found a relevant difference between using "send + receive" and
switching folders (which also refreshes the index).

It failed both when the INBOX was the current folder and when being outside
the INBOX and then switching to it.

> Patrick, can I try to produce some very chatty patch, which will monitor things
> it does in all your local providers, to try to catch what's going wrong?

Sure, but if you can, please make it against 2.24.x. I'm not on 2.26 yet
and probably would have trouble applying an extensive patch against 2.26
onto 2.24.
Comment 20 Milan Crha 2009-04-30 10:17:34 UTC
Created attachment 133644 [details]
eds-2-24 test patch

for eds 2-24

there is nothing inventive in this test patch, I only merged those two previous, and added some fix into
camel/providers/local/camel-local-folder.c:local_refresh_info
Apart of that it's veeery chatty (probably log to file to ensure nothing will be missing), though I'm not sure whether I covered all the interested parts. Let's see. (I do not compile 2.24 eds any more, but I tried to apply the patch on those sources; I hope you'll be able to fix any typo I might produce.) Thanks for your help.
Comment 21 Patrick Ohly 2009-05-04 11:57:09 UTC
(In reply to comment #20)
> Created an attachment (id=133644) [edit]
> eds-2-24 test patch

I'm running 2.24 with that patch now. Whenever I want check for mail I do:
* enter INBOX
* run offlineimap
* switch from INBOX to empty Junk folder of the same account and back

This just failed again. The output was the following, including the output generated from switching folders with no changes before calling offlineimap:

(evolution:11753): camel-CRITICAL **: camel_message_info_free: assertion `mi != NULL' failed
maildir_summary_sync: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:0
maildir_summary_check: going to reload because count:74 cachesize:72 finally count: 74
maildir_summary_check: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:0

pohly@pohly-MOBL:/work/evolution-svn/obj/evolution-data-server$ date && offlineimap -a Intel
Mon May  4 13:37:42 CEST 2009
OfflineIMAP 6.0.3
Copyright (C) 2002 - 2008 John Goerzen <jgoerzen@complete.org>
This software comes with ABSOLUTELY NO WARRANTY; see the file
COPYING for details.  This is free software, and you are welcome
to distribute it under the conditions laid out in COPYING.
***** Processing account Intel
Copying folder structure from IMAP to Maildir
Establishing connection to imapmail.intel.com:993.
Syncing INBOX: IMAP -> Maildir
Copy message -2 Maildir[.] -> IMAP[INBOX], LocalStatus[.]
Copy message -1 Maildir[.] -> IMAP[INBOX], LocalStatus[.]
Deleting 108 messages (117257, 117261, 117262, 117264, 117267, 117271, 117272, 117279, 117280, 117281, 117283, 117284, 117287, 117288, 117290, 117294, 117301, 117315, 117320, 117332, 117333, 117348, 117353, 117354, 117360, 117361, 117388, 117391, 117398, 117399, 117403, 117406, 117412, 117413, 117414, 117415, 117416, 117417, 117418, 117419, 117422, 117423, 117426, 117427, 117429, 117431, 117433, 117435, 117451, 117444, 117446, 117448, 117449, 117450, 117452, 117453, 117454, 117455, 117458, 117459, 117461, 117462, 117463, 117464, 117465, 117466, 117467, 117469, 117473, 117475, 117476, 117483, 117484, 117486, 117488, 117489, 117490, 117492, 117493, 117495, 117497, 117498, 117506, 117518, 117522, 117524, 117525, 117527, 117528, 117529, 117533, 117534, 117540, 117544, 117545, 117548, 117558, 117563, 117077, 117080, 117084, 117491, 117494, 117496, 117222, 117237, 117241, 117243) in IMAP[INBOX], LocalStatus[.]
Copy message 117567 IMAP[INBOX] -> Maildir[.], LocalStatus[.]
Copy message 117569 IMAP[INBOX] -> Maildir[.], LocalStatus[.]
***** Finished processing account Intel

[several mails had been moved out of the INBOX, some were marked as read, two new received]

pohly@pohly-MOBL:/work/evolution-svn/obj/evolution-data-server$ 

[now leave INBOX, increases the "unread count" to 11]

maildir_summary_check: going to reload because count:74 cachesize:72 finally count: 74
camel_maildir_summary_add: /home/pohly/.evolution/mail/imapmail.intel.com/./cur/1241437067_0.12949.pohly-MOBL,U=117570,FMD5=5058f1af8388633f609cadb75a75dc9d:2,S
info_set_flags: on .
camel_maildir_summary_add: /home/pohly/.evolution/mail/imapmail.intel.com/./cur/1241437067_1.12949.pohly-MOBL,U=117571,FMD5=5058f1af8388633f609cadb75a75dc9d:2,S
camel_maildir_summary_add: /home/pohly/.evolution/mail/imapmail.intel.com/./cur/1241437072_1.12949.pohly-MOBL,U=117569,FMD5=5058f1af8388633f609cadb75a75dc9d:2,:2,
info_set_flags: on .
camel_maildir_summary_add: /home/pohly/.evolution/mail/imapmail.intel.com/./cur/1241437072_0.12949.pohly-MOBL,U=117567,FMD5=5058f1af8388633f609cadb75a75dc9d:2,:2,
maildir_summary_check: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:1

(evolution:11753): camel-CRITICAL **: camel_message_info_free: assertion `mi != NULL' failed
maildir_summary_sync: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:1
local_sync: on .
maildir_summary_check: going to reload because count:76 cachesize:74 finally count: 76
maildir_summary_check: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:0

(evolution:11753): camel-CRITICAL **: camel_message_info_free: assertion `mi != NULL' failed
maildir_summary_sync: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:0
maildir_summary_check: going to reload because count:76 cachesize:74 finally count: 76
maildir_summary_check: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:0

[after switching back into the INBOX, only 9 unread emails are shown and none of the two new emails]

[now mark two emails as read and move them to another folder]

info_set_flags: on .
info_set_flags: on .
thaw: on .
maildir_summary_check: going to reload because count:76 cachesize:74 finally count: 76
maildir_summary_check: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:0
maildir_get_message: no change in .
maildir_summary_check: going to reload because count:18104 cachesize:3 finally count: 18104
maildir_summary_check: /home/pohly/.evolution/mail/imap.linux.intel.com/Meta/Intel: changes:0
info_set_user_flag: on Meta/Intel
maildir_summary_add: changed:1 mi:0x92c40120
maildir_append_message: notify changed in Meta/Intel
info_set_flags: on .
maildir_summary_check: going to reload because count:76 cachesize:74 finally count: 76
maildir_summary_check: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:0
maildir_get_message: no change in .
maildir_summary_check: /home/pohly/.evolution/mail/imap.linux.intel.com/Meta/Intel: changes:0
info_set_user_flag: on Meta/Intel
maildir_summary_add: changed:1 mi:0x92c40178
maildir_append_message: notify changed in Meta/Intel
info_set_flags: on .
thaw: on Meta/Intel
thaw: on .

[expunge INBOX]

maildir_summary_check: going to reload because count:76 cachesize:74 finally count: 76
maildir_summary_check: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:0

(evolution:11753): camel-CRITICAL **: camel_message_info_free: assertion `mi != NULL' failed
maildir_summary_sync: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:1
local_sync: on .
maildir_summary_check: going to reload because count:74 cachesize:72 finally count: 74
camel_maildir_summary_add: /home/pohly/.evolution/mail/imapmail.intel.com/./cur/1240915762_0.10115.pohly-MOBL,U=117247,FMD5=5058f1af8388633f609cadb75a75dc9d:2,S
maildir_summary_check: /home/pohly/.evolution/mail/imapmail.intel.com/.: changes:0
maildir_get_message: no change in .

[two new emails show up in the INBOX]

Does that help?

The assertion `mi != NULL' is triggered when moving away from the INBOX into the Junk folder; it's stack back trace is:

  • #0 camel_message_info_free
    at ../../../evolution-data-server/camel/camel-folder-summary.c line 4343
  • #1 maildir_summary_sync
    at ../../../../../evolution-data-server/camel/providers/local/camel-maildir-summary.c line 821
  • #2 camel_local_summary_sync
    at ../../../../../evolution-data-server/camel/providers/local/camel-local-summary.c line 306
  • #3 local_sync
    at ../../../../../evolution-data-server/camel/providers/local/camel-local-folder.c line 517
  • #4 camel_folder_sync
    at ../../../evolution-data-server/camel/camel-folder.c line 306
  • #5 sync_folder_exec
    at ../../../evolution/mail/mail-ops.c line 1538
  • #6 mail_msg_proxy
    at ../../../evolution/mail/mail-mt.c line 520
  • #7 ??
    from /usr/lib/libglib-2.0.so.0
  • #8 ??
    from /usr/lib/libglib-2.0.so.0
  • #9 start_thread
    from /lib/tls/i686/cmov/libpthread.so.0
  • #10 clone
    from /lib/tls/i686/cmov/libc.so.6

mdi in maildir_summary_sync() is also NULL when info is NULL, so other than this redundant camel_message_info_free() nothing is done. Not sure whether this mdi/info NULL case is supposed to happen.
Comment 22 Milan Crha 2009-05-04 15:32:21 UTC
(In reply to comment #21)
> Does that help?

Yup, thanks for the update. I think I'm getting some clues about the issue, but still not completely sure. 

What I think from the above, the issue triggers mostly when there are some removes in the folder, together with insertions (maybe also with flag changes from the outside of evolution). The log shows the "folder_changed" event was sent (those lines with "changed: 1"), but the message information couldn't be found, as shows the runtime warning within sync operation. Thus I guess it couldn't find the new message infos, thus it failed to update message list in the UI.

I'm not sure why it reloads from DB twice, it should do that only once (within some time interval, which is 5 minutes).

> mdi in maildir_summary_sync() is also NULL when info is NULL, so other than
> this redundant camel_message_info_free() nothing is done. Not sure whether this
> mdi/info NULL case is supposed to happen.

It seems the code checks for non-NULL values on every place but the call for a camel_message_info_free. Even it' bad, then we know that something goes wrong here, which is fine.

I'll try to simulate such steps here and let's see.
Comment 23 Milan Crha 2009-05-07 13:27:07 UTC
Bad. So I applied a test patch from comment #14 on my git master here, and tried to simulate offlineimap operations in a way I think they should work (adding new mail into 'new' folder (not the 'cur' nor 'tmp' folder) and deleting some mails from 'cur' folder under Inbox). None of these actions work as for you. I see those new mails, I do not see a runtime warning on the evolution console when some mails had been deleted. Even the same without the test patch.

I'm afraid that some change which was committed into the former trunk only, not included in your stable 2.24.x version, influences this as that much. Or some breakage of your folders.db of the maildir account? Nothing else I can think of at the moment.

I'm proposing to include a test patch in the sources anyway, as I believe it's a correct change.
Comment 24 Patrick Ohly 2009-05-07 21:01:34 UTC
(In reply to comment #23)
> I'm afraid that some change which was committed into the former trunk only, not
> included in your stable 2.24.x version, influences this as that much. Or some
> breakage of your folders.db of the maildir account?

I already deleted folders.db a few times to get back into a sane state. As I read on the mailing list, that would have removed meta information about emails that isn't stored anywhere else, but luckily I don't have that.

> Nothing else I can think of
> at the moment.

Thanks for giving it a try. I guess I'll have to upgrade to 2.26 as soon as time permits and then report back when it still occurs.
Comment 25 Srinivasa Ragavan 2009-06-20 14:38:24 UTC
Milan, I think you are right. But are you sure no locks etc are required ? I see the camel-mbox folder doing the same with locks. It is possible that you have two threads doing camel_local_summary_check, leading to corruption. But otherwise, the patch is ready to meet the git.
Comment 26 Milan Crha 2009-06-22 10:50:38 UTC
(In reply to comment #25)
> But are you sure no locks etc are required ?

I'm not sure, I copied some code from mbox and hoped it'll work :) What locks did you mean? I see they are not used always, as the 'sync' methods doesn't use it, but the 'append_message' uses it in mbox (the call of camel_local_folder_lock).
For example maildir_append_message doesn't use it either before the patch. Strange?
Comment 27 Milan Crha 2009-06-22 18:47:45 UTC
Created attachment 137195 [details] [review]
proposed eds patch ][

for evolution-data-server;

Thinking of it, after our chat on IRC, I tend to believe the missing locks are causing trouble Patrick described above. Let's hope it :)
This is quite similar patch as the test one, just added locking into it.
Comment 28 Srinivasa Ragavan 2009-06-23 04:27:50 UTC
Patrick, can you give this a try?
Comment 29 Patrick Ohly 2009-06-23 06:52:25 UTC
(In reply to comment #28)
> Patrick, can you give this a try?

I'm still not on 2.26.x and the patch doesn't apply due to non-trivial changes (
maildir_append_message() looks different), so I can't test it right now.

I definitely want to to upgrade to 2.26 as part of reinstalling a new laptop, but that hasn't arrived yet.
Comment 30 Milan Crha 2009-06-29 08:28:27 UTC
*** Bug 587259 has been marked as a duplicate of this bug. ***
Comment 31 Patrick Ohly 2009-08-09 21:05:31 UTC
The problem still occurs with the latest "master" branch, compiled today. Milan's patch still applies to that source, so I recompiled and will check whether it still fails.


Comment 32 Patrick Ohly 2009-08-10 07:57:16 UTC
(In reply to comment #31)
> The problem still occurs with the latest "master" branch, compiled today.
> Milan's patch still applies to that source, so I recompiled and will check
> whether it still fails.

Yes, it still fails even with the patch from comment #27: I just ran offlineimap, moved between folders, unread mail count was updated, but the new mails didn't show up in the mail list. Delete+expunge made them show up.

Milan, how do we proceed with this? Any suggestions?



Comment 33 Milan Crha 2009-08-10 14:35:47 UTC
Created attachment 140331 [details] [review]
proposed eds patch ]I[

for evolution-data-server;

Only one suggestion, please try this one :)
It seems to work fine for me. Maybe still some corner cases I'm not aware of there, but otherwise should work fine. This patch is again quite similar to the previous one, only difference is maildir_summary_check. I realized that the offlineimap (you forced me to install and try it :) ) is storing files to maildir folders with ":X" suffix, but we use UIDs within maildir without that, thus evo had been notified with a wrong UID.
Comment 34 Milan Crha 2009-08-10 14:37:39 UTC
Beh, this
> +				nm =strrchr (newname, ':');
should be this
+				nm = strrchr (newname, ':');

Comment 35 Patrick Ohly 2009-08-12 07:44:32 UTC
(In reply to comment #33)
> Created an attachment (id=140331) [edit]
> proposed eds patch ]I[
> 
> for evolution-data-server;
> 
> Only one suggestion, please try this one :)

I have used this for over a day now, no problems so far. It seems that this really fixes the problem.

> It seems to work fine for me. Maybe still some corner cases I'm not aware of
> there, but otherwise should work fine. This patch is again quite similar to the
> previous one, only difference is maildir_summary_check. I realized that the
> offlineimap (you forced me to install and try it :)

Hey, it's worth it ;-)

> is storing files to
> maildir folders with ":X" suffix, but we use UIDs within maildir without that,
> thus evo had been notified with a wrong UID.

Hmm, I don't see that with offlineimap 6.1.2.

Mails downloaded by offlineimap:
new/
1250062876_0.8211.pohly-mobl1,U=127666,FMD5=5058f1af8388633f609cadb75a75dc9d:2,
1250062877_0.8211.pohly-mobl1,U=127662,FMD5=5058f1af8388633f609cadb75a75dc9d:2,
1250062877_1.8211.pohly-mobl1,U=127664,FMD5=5058f1af8388633f609cadb75a75dc9d:2,
1250062877_2.8211.pohly-mobl1,U=127663,FMD5=5058f1af8388633f609cadb75a75dc9d:2,
1250062877_3.8211.pohly-mobl1,U=127665,FMD5=5058f1af8388633f609cadb75a75dc9d:2,

What's the meaning of :X?
Comment 36 Patrick Ohly 2009-08-12 08:13:20 UTC
(In reply to comment #35)
> > is storing files to
> > maildir folders with ":X" suffix, but we use UIDs within maildir without that,
> > thus evo had been notified with a wrong UID.
> 
> Hmm, I don't see that with offlineimap 6.1.2.

Was a misunderstanding, as explained on IRC. :X was meant as a placeholder for something like ":2," or ":2,S" (seen). That is indeed placed in the "new" folder by offlineimap.

Given that the patch arguably improves the situation I suggest to accept it into 2.28. I can't comment on the code itself, though, because I know nothing about camel.
Comment 37 Milan Crha 2009-08-12 11:35:15 UTC
Created commit 25fa98c in eds master (2.29.1+)
Created commit 7d0fc22 in eds gnome-2-28 (2.27.91+)

Thanks for the testing.