GNOME Bugzilla – Bug 782844
Folder changes not always saved
Last modified: 2017-09-19 16:00:46 UTC
After upgrading to 3.24 some messages (especially when they should be moved to a subfolder by an Evolution filter) are somehow received multiple times, which results in a lot of duplicates in the subfolder _and_ in Trash (how do they end up there - I have no idea). I think this is somehow related to message filters, as I don't observe that issue in accounts where there's no filtering. On the other hand, for two subfolders for which I've set up Evolution filtering (GCC and GDB mailing lists), I can sometimes "receive" the same message up to 20 times (with ~half of the messages ending up in Trash). This is probably related to filtering in general, as I observe some (but a lot less) duplicated messages which were matched by server-side filters (which includes some messages classified by server as SPAM).
Thanks for a bug report. this would be good to retest with the change from bug #782096 and if it'll help, then I'd mark this either as a duplicate of yours bug #782842 or the former bug report. Are you able to build evolution-data-server with the change from bug #782096 included, please? I can help to build a package for Fedora, but not for any other distribution.
I'll build all evolution packages from the repo and report back. I'm using Arch Linux, so I guess using the AUR packages should be exactly what is needed here: https://aur.archlinux.org/packages/evolution-git https://aur.archlinux.org/packages/evolution-data-server-git/
(In reply to Freddie Chopin from comment #2) > I'll build all evolution packages from the repo and report back. I'm using > Arch Linux, so I guess using the AUR packages should be exactly what is > needed here: > > https://aur.archlinux.org/packages/evolution-git > https://aur.archlinux.org/packages/evolution-data-server-git/ It doesn't reference latest git master, but some very outdated version, at least for evolution-data-server. To make it as simple as possible, I would just pick the patch from the above-mentioned bug report, apply it to evolution-data-server, built it, install it, restart the machine and try to reproduce again. There is no need to rebuild evolution, neither to try latest git master, which is a development version, not a stable version.
> It doesn't reference latest git master, but some very outdated version, at least for evolution-data-server. It actually doesn't matter, but it is pretty confusing indeed. These packages build the very latest revision from the repo (or optionally the revision you choose), the version displayed on the website only references the last time the script was updated, which makes little sense for such scenario, but... (; Anyway - thanks for the hints, I'll try to rebuild just the data server package with the patch above and see whether this helps. I'm not sure when I will get the time to do it, but I'll report back as soon as possible.
Hello again! I've followed your advice and applied this patch - https://git.gnome.org/browse/evolution-data-server/commit/?id=8852b13 . This solved my other issue, but this one still persists. To reproduce I think it is enough to have a subfolder into which messages are moved by Evolution built-in filtering. I have that configured for two subfolders and all of "message multiplication" happens in these two folders _AND_ trash (the message appears multiple times in the target subfolder and in trash). If that is of any interest, I have that done for "gcc" and "gdb" mailing lists, with the filtering rule: Specific header - Sender - contains - "gcc-owner@gcc.gnu.org" Action: Move to Folder - <some-subfolder> Important note - this doesn't happen to all messages. Somehow some of them are received just once, while something like 10-20% are received multiple times (usually at least 3 in subfolder + 3 in trash). Let me know how can I help you narrow that problem down.
Thanks for the update. Maybe as the starter, turn on filter logging, as descried here [1], which will show which rules had been applied and eventually how many times. I do not see it above, but I suppose you've configured real Trash and Junk folders and the account is an IMAP account, right? [1] https://help.gnome.org/users/evolution/stable/mail-filters-not-working.html.en
I can see the same issue as described in comment #0. I think I'm seeing this issue ever since updating to 3.24. I am sure it did not happen with 3.22, but I have seen this issue earlier (3.18 or 3.20 or something like that, not sure and I cannot find the bug report either). Happens with 3.24.0, 3.24.1, 3.24.2. 3.24.3 is not old enough to have it reproduced (yet). (In reply to Milan Crha from comment #6) > Thanks for the update. Maybe as the starter, turn on filter logging, as > descried here [1], which will show which rules had been applied and > eventually how many times. I've done that and got ~80MB log file since ~1 month ago. It contains quite much sensitive data, can you provide a GPG key to encrypt it against? I only found a 10 years old key which is pretty weak (1024Bit DSA). I can send it by mail as it can be heavily compressed. > I do not see it above, but I suppose you've > configured real Trash and Junk folders and the account is an IMAP account, > right? Yes.
(In reply to Christian Stadelmann from comment #7) > I've done that and got ~80MB log file since ~1 month ago. The whole month is not needed for sure. I would need some clues about message(s) which cause that, like the Subject or even UID (right-click the message list table header and pick Add Column... where it can be drag&dropped to the header to add the column to the view). > ...can you provide a GPG key to encrypt it against? I only found a 10 years > old key which is pretty weak (1024Bit DSA). That's correct. It should be something about ID F3C36A0D, it's available on gpg.mit.edu site for example.
(In reply to Milan Crha from comment #8) > (In reply to Christian Stadelmann from comment #7) > > I've done that and got ~80MB log file since ~1 month ago. > > The whole month is not needed for sure. I would need some clues about > message(s) which cause that, like the Subject or even UID (right-click the > message list table header and pick Add Column... where it can be > drag&dropped to the header to add the column to the view). Funnily, the bugzilla notification for your comment was affected by this bug too. You should receive a mail from me.
Thanks for the log. I know my key is old and possibly weak (it's just a theory, isn't it? Links needed :) ), but I do not use it that often, thus I do not care that much. The log shows an interesting thing: At the top it says: > Reported 1 recent messages in 'xxx : Inbox' > 2017-06-26 09:46:10 - Applied filter .... > ... > Finished test of message uid:3130 subject:'...... as NOMATCH then it tries many other filters and ends with: > 2017-06-26 09:46:10 - Applied filter ... > ... > Finished test of message uid:3130 subject:... as MATCHED > Filter 'yyy' matched > > Action: Move to folder ... > Action: Stopped processing > Filter 'yyy' did not match > > Stopped processing per request then there was a quiet period, as you also indicated with [...], and then the log continued (I'm not sure whether the first line is correct, can it be a copy&paste error?): > Stopped processing per request > 2017-06-26 10:21:54 - Applied filter... > ... > Finished test of message uid:3130 subject:.... and then it repeats the same way as the first run. The interesting thing I see there is that there had been like 35 minutes gap between the first run and the second run (can be related to your refresh interval setting) and that both runs are for the same UID 3130, which means the message is the same in the Inbox on the server. My current question for the log is for that second run: is it really beginning with "Stopped processing per request", while the first run begins with "Reported 1 recent messages in 'xxx : Inbox'"? That's important, because the difference means whether the message with that UID had been known to IMAP or not. It obviously noticed it, but maybe it failed to make a note in its local summary for some reason.
(In reply to Milan Crha from comment #10) > Thanks for the log. I know my key is old and possibly weak (it's just a > theory, isn't it? Links needed :) ), but I do not use it that often, thus I > do not care that much. > The log shows an interesting thing: At the top it says: > Correctly. > then there was a quiet period, as you also indicated with [...], and then > the log continued (I'm not sure whether the first line is correct, can it be > a copy&paste error?): […] indicates that a few more messages were filtered. For all of them, filtering starts with "Reported 1 recent messages in …" and ends with "Stopped processing per request". > > Stopped processing per request this line belongs to a previous filter run on a different (unrelated) message. There was no empty line between that and the following line, that's why I copied both. > > 2017-06-26 10:21:54 - Applied filter... > > ... > > Finished test of message uid:3130 subject:.... > > and then it repeats the same way as the first run. > […] > My current question for the log is for that second run: is it really > beginning with "Stopped processing per request", while the first run begins > with "Reported 1 recent messages in 'xxx : Inbox'"? That's important, > because the difference means whether the message with that UID had been > known to IMAP or not. It obviously noticed it, but maybe it failed to make a > note in its local summary for some reason. That is not a copy&paste error, that's the way the original log looks like. And no, before the second run, there is no "Reported [number] recent messages in …".
Also, I noted that my IMAP account trash folders have messages marked as unread and not deleted. Maybe that is causing the duplicated messages? @Freddie Chopin: Do you also see not-deleted messages in your IMAP trash? In Evolution Preferences, Mail Account, Account Editor, Receiving Options, I have "Apply filters to new messages in all folders" enabled. Will disable this and instead enable "Apply filters to new messages in Inbox on this server" and hope it works around the issue.
Thanks for the update. I think I managed to reproduce this. I see on evolution console, when run it from a terminal, this warning: > (evolution:4972): camel-WARNING **: uid '90' vanished from folder 'xxx : INBOX' Then I see that the filter is tried once again, thus it runs basically twice, while the second run succeeds and finds the message in the local cache. I guess that for you the first run also finds the message in the cache, thus it's truly filtered twice. I do not have an explanation for the time shift in your log, though. I do have checked to apply messages for Inbox only, thus that option might not be fully related, but I'll try with enabled filtering for all folders too, just to be sure.
(In reply to Milan Crha from comment #13) > Thanks for the update. I think I managed to reproduce this. I see on > evolution console, when run it from a terminal, this warning: > > > (evolution:4972): camel-WARNING **: uid '90' vanished from folder 'xxx : INBOX' Just for the record: I don't have this warning in my syslogs, not even when going back 3 weeks with about 3-5 duplicated messages per day. > Then I see that the filter is tried once again, thus it runs basically > twice, while the second run succeeds and finds the message in the local > cache. I guess that for you the first run also finds the message in the > cache, thus it's truly filtered twice. I do not have an explanation for the > time shift in your log, though. Hoping to shed some light on that: * There is nothing in the syslog about evolution in that time, just the usual stuff from other applications. * On 09:49:33, 10:13:21/10:13:22, 10:13:23 and 10:21:54 other messages were filtered. * The last rule in my message filter is a catch-all rule moving emails to folder://local/Inbox.
Could you try to go to Defaults tab of Account Properties and set either the Junk folder or the Trash folder _not_ to be a real folder, please? It seems to me that it's one part of the issue. I'm still investigating the cause and this is only the first thing I spot, as my account has them both set to real folders. If you've either or both "unreal", then even better. :)
Created attachment 354546 [details] [review] proposed eds patch I'm not able to reproduce it with these evolution-data-server changes, but it'll be safer if you could give it a try first, before I commit. I made a test build for Fedora 26 [1], in case you use it, but I'm not able to create any similar test build for other distributions, I'm sorry. [1] https://koji.fedoraproject.org/koji/taskinfo?taskID=20201560
(In reply to Milan Crha from comment #15) > Could you try to go to Defaults tab of Account Properties and set either the > Junk folder or the Trash folder _not_ to be a real folder, please? It seems > to me that it's one part of the issue. I'm still investigating the cause and > this is only the first thing I spot, as my account has them both set to real > folders. If you've either or both "unreal", then even better. :) In one of my accounts affected (the one I received the bugzilla notification on which I sent you yesterday), both are set "unreal" already. On some accounts, both are set to "real". On some other accounts, both are set to "unreal" as the server does not have "real" trash folders. (In reply to Milan Crha from comment #16) > I made a test build for Fedora 26 [1], in case you use it, but I'm not able > to create any similar test build for other distributions, I'm sorry. Will try. Thanks for making it that simple ;)
I think the patch solves the issue. Since yesterday I had no duplicate messages. On the other hand I also didn't receive many "candidates" for duplication, but usually that would be enough. Thanks for your support Milan! > @Freddie Chopin: Do you also see not-deleted messages in your IMAP trash? Yes - as I reported earlier, I see suplicates (usually the same amount) in trash and in the target folder - for example 3 duplicates end up in trash and 3 more in the mailing list folder.
Thanks Freddie, let's wait a bit longer, and for a response from Christian as well.
(In reply to Milan Crha from comment #16) > I made a test build for Fedora 26 [1], in case you use it, but I'm not able > to create any similar test build for other distributions, I'm sorry. > > [1] https://koji.fedoraproject.org/koji/taskinfo?taskID=20201560 With this test build applied, I do not get any more messages being duplicated (as far as I can tell right now) and I do not see any more "unread" messages stuck in trash. Looks fine to me. Thank you!
Thanks for the update. I've been about to ask, but you've been quicker. Created commit 6f19112 in eds master (3.25.4+) Created commit 2c56c33 in eds gnome-3-24 (3.24.4+)
This might cause bug #787514