GNOME Bugzilla – Bug 324804
IMAP backend does not filter new messages automatically
Last modified: 2013-09-13 00:52:11 UTC
Please describe the problem: I have three IMAP accounts on two servers. I have specified "receiving options" -> "automatically check for new mail every <1> minutes". I also specified "check for new messages in all folders" and "apply filters to new messages in INBOX on this server" and "automatically synchronize remote mail locally". The final goal is to get all mail from all three accounts into several folders of one imap account. I have made filters to make this happen and sometimes it works, but most of the time, I need to go to the various inboxes, type ^A and ^Y and then it works. So it works, but not automatically Steps to reproduce: See above. Actual results: Nothing ;-) Expected results: Automatic moving to other IMAP folders. Does this happen every time? Yep. Other information: Nope.
Hi Erik, Thanks for the bug report. Before i delve into the issue, let me give you a little backgroud as to how filters work in Evolution. Evolution filters(in IMAP) only those messages which have a "Recent" flag on them. This means, Only those messages which are 'downloaded' from the server for the first time from anywhere. If a message has been downloaded already using another mail-client or an imap deamon locally or using a web interface, then the messages are *not* recent anymore. And wont be filtered. So the question i would like to ask you is, are you doing any such thing? Hope am clear. Get back in case you need more info on this.
Thanks for your response. Evolution is the only application that I use to fetch these accounts, one instance at a time. Some additional background that may be helpful: two of the accounts are on a (remote, using openvpn) linux server that has dovecot running, one account is (local) on an nt4 exchange server. It's the last account that has the problem almost always. I just checked, when enabling the account, all works well initially. It goes wrong when mail arrives later, it's left in the imap's inbox instead of being filtered.
The 'last account' you mentioned in Comment #2. Are you using the Exchange connector to access the account or using IMAP on the Exchange servers??
I am using a setup with only IMAP. We are using the exchange server of NT4, afaik evolution does not support that directly using the connector.
Can you do this: export CAMEL_VERBOSE_DEBUG=1 on the command line and run evolution (from the command line). This would give you a lot of traces, put them in a file and then mail to me (since it will contain your folder listing and stuff). Also do operations like getting new messages and filtering. Capture the traces of the entire session and then send it. would be useful in tackling the problem. Thanks
Created attachment 56805 [details] Debug output.
can you do that again after disabling all other IMAP accounts? makes it really hard to follow the debug output with 3+ IMAP servers being connected to at the same time... thanks.
Created attachment 56850 [details] Something like this?
I've noticed the same problem on my machine. It happened both with evolution 2.2.2, and now with evolution 2.4.2.1 (I just upgraded yesterday). I have an IMAP connection to an Exchange server. Evolution is configured to check for mail every 3 minutes, and any mail that passes through my sorting filters should get caught by a final catch-all filter which plays a notification sound. Most of the time these filters work, especially when I first start evolution and get a whole bunch of overlapping sound bites for the morning's new mail. But as the day goes on, I occasionally check back and see that new mail has arrived which didn't trigger the audio notification filter. The first few times I thought I must have been away from my desk or had my volume turned down, but I've been paying more attention to that, and the last few times I know I was at my desk with the volume turned up. Also, there were a few messages which should have been sorted that weren't. Applying the filters manually took care of them. I'll see if I can get some additional debugging info for you.
Created attachment 57959 [details] Camel verbose debug output I caught the debug output and saw the problem had happened again within the last hour. I received two messages: the one entitled "Member Update 1/20-1/22" should have triggered the "Member Update" filter (which deletes it), and the one entitled "CDW Order Confirmation" should have triggered the "normal" filter (which plays an audio alert). Neither event occurred.
I just recently installed evolution 2.5.92. The problem still occurs in that version -- some messages get filtered (particularly on start-up), but other messages don't.
Adding me to the CC list, as I'm seeing this as well in evolution 2.6.0: no filters get applied to IMAP. My situation: dovecot as IMAP server, 1 single IMAP account, using only Evolution to connect to it.
I use IMAP to access an Exchange Server and apply filters to the IMAP INBOX using Evolution 2.6.0 on Fedora Core 5. I have noticed that the filters only get applied automatically if I do not have "Automatically check for new mail" selected on the "Receiving Options" page of the Account Editor. If I have this option selected then the automatic application of filters stops. Could it be that the automatic check for new mail is causing the Unseen flag to be removed and hence the filter system does not believe there is any new mail to filter?
I'm having the same problem. Running evolution 2.6.0 on Fedora-core-5 with courier-imap-4.0.6-2 as the server on a RedHat Enterprise box. New email arrives automatically into my INBOX, but is not filtered.
I just upgraded to evolution 2.6.1. Now I have another problem: at least one of my filters (the spam filter -- which is the most important one) is not working at all. The filter is simple. Our company's mail server tags suspected spam with the prefix "[SPAM] - " on the subject line. The rule is "if Subject starts with '[SPAM] - ' then Delete, Stop Processing". This was working (when filters were working at all) in 2.5.92. Now it skips that filter and falls through to the last filter, which plays an alert message.
Created attachment 65694 [details] Camel verbose for evolution 2.6.1 applying the wrong filter.
Please note that I have this problem only using a Exchange 2000 imap server. Using dovecot, it doesn't show. So it actually might be an Exchange bug.
Might be an Exchange bug... but might not be. That brings up additional questions: what is Exchange doing in IMAP that's different from dovecot? Do the differences still fall within the RFC specifications? Even if Exchange is doing something naughty, should (can) we implement some sort of workaround in Evolution to handle it? What's a good way for me to find out what the IMAP transactions are between Evolution and Exchange?
I can report that my situation has improved dramatically. Evolution filters all incoming mail now from my IMAP server. I'm running evolution 2.6.2 on fedora-core-5 and my IMAP server is courier-imap-4.0.6-2 on a RedHat Enterprise box. evolution-webcal-2.4.1-3.4 evolution-2.6.2-1.fc5.5 evolution-data-server-1.6.2-1.fc5.1 A few of my filters stopped working after upgrading to 2.6.2. I deleted the non-working filter and attempted to re-add only to find that I actually had two filters with the same name. Apparently evolution-2.6.2 is more picky about that than 2.6.0 was.
i have been noticing similar problems described here. currently running evolution 2.6.3 on FC5. this is just to add myself to the cc list since its really a pain to manully invoke filtering actions. one instance i noticed was the high-order filters are skipped and filtered by low-order ones. in this case, the high-order filter should have filtered it. so the consequence is that mails are moved to wrong places
Bug #356092 is clearly a duplicate of this bug, and proposes a patch. I tested this patch and confirms it solves the issue for me, though I've no idea if this is the right way to fix the issue. Other bugs related to the "recent" flag let me think the best way would probably to just not use it. It seems that it really not is a reliable way to find out which messages in a mailbox were recently arrived from the point of view of the current session. As specified in RFC3501, it is clear that this flag is useless as soon as multiple concurrnet accesses are made on a same account. In any case, looking at how many people are in Cc of the multiple bugs filled for this issue, it would be great to at least see a workaround (like the patch mentioned above) in an upcoming evo release.
*** Bug 230470 has been marked as a duplicate of this bug. ***
*** Bug 356092 has been marked as a duplicate of this bug. ***
*** Bug 317439 has been marked as a duplicate of this bug. ***
Discussion on irc show that this bug seems to be hit by many people (questions on mailing list etc.). Since this really is a disfunction of an essential feature (filtering with IMAP, I'm marking bug as "major" severity. I'm also attaching patch proposed in 356092 by Viktor Kojouharov to this bug.
Created attachment 79887 [details] [review] patch making camel-folder only look at messages marked "added" and ignoring if some messages are marked "recent" the patch works for me (two days of highly intensive use), but I'm unable to evaluate possible collateral damage
@veerapuram: comment #21, patch at bug 356092?
I'd just like to add that while the patch may not be the best way to fix the problem, it does work. From the time I posted bug 356092, I've seen to flaws resulting from it.
viktor: "I've seen to flaws resulting from it." --> maybe you meant "_no_ flaws resulting from it" ?
sorry about that. should be "no" flaws.
more than just the IMAP backend uses that camel-folder.c code to filters new messages, so this patch is broken. I would suggest the following alternative: have a checkbox in the UI akin to "Treat all previously unknown messages as new messages" if checked, add all new messages (\Recent or not) to the CamelFolderChangeInfo as 'recent' (in addition to 'added' I think) so that the camel-folder code filters them. of course, you will have to do a quick check to see if anything else uses CamelFolderChangeInfo's recent_uids and how it uses them to make sure doing it the way I suggested won't break something else.
While it is true that the patch is not a correct solution (by the way, it doesn't even apply anymore since e-d-s 2.12.x), I don't think a fix requiring checking a box in the UI is a good solution either. Evolution should IMHO work nicely out of the box for such an important protocol as IMAP. It should just NOT rely on the Recent flag for IMAP, that has a broken semantic. Having to CTRL-Y on top messages in my INBOX every 5 minutes, is really a pain. I really hope that a fix can be found for Evolution IMAP support with parallel accesses to an account. (I would be tempted to add the following general comment / rant : Evolution still has to come a long way to become a good IMAP client. IMHO the Gnome application for email should strive to have very good IMAP support. It may be that Novell's focus/priority is on Groupwise, but weak IMAP support for The Gnome mail application is really deceiving... Sorry for ranting without contributing to Gnome with much more that bug reports, but well, after month of fighting hard against myself to not switch to Thunderbird, I need to somehow express my feelings )
The checkbox suggestion was in case someone wiped their cache and had 100,000 emails in their INBOX, you wouldn't necessarily want all of them to be re-filtered, would you? It could take days (ok, maybe slight exaggeration... but w/ some imap servers *cough*like mine*cough* it might actually take days) It's really not that hard to add such a checkbox... Then, in the evolution-data-server/camel/providers/imap/ source tree, in camel-imap-folder.c:imap_update_summary() where the imap code adds recent uids to the CamelFolderChangeInfo based on the \Recent flag, make the logic ALSO check the state of the checkbox option to decide if it should be added to the recent list. e.g. if (\Recent flag is set OR option is enabled) add the uid to the CamelFolderChangeInfo as a recent message fwiw, if you read thru the imap code (or, likely any other folder code) you'll notice that the original patch will have side-effects - when you manually move messages from folder A to Folder B, uids will be added to the CamelFolderChangeInfo::uids_added array and then the original patch would go and filter them. That's broken. There are other problems as well, such as appending messages to a folder, etc. But I think that example is sufficient to show that the original patch was flawed. On a positive note, the original patch idea wasn't completely without merit, as my suggestion is basically just a slight variation of it. You still want to use the CamelFolderChangeInfo::uids_recent array in the camel-folder.c code that has the filtering logic, but you don't have to limit yourself to just adding items to it that are marked \Recent by the server. Anyways, looking forward to a new patch!
iven that the \Recent flag is useless (as soon as more than one client accesses the mailbox), I would expect Evolution to just NOT use it. Starting from this, I would expect Evolution to only look at messages ids to know if they were recently added or not. E.g. when the IMAP provides a message or message header, check if we have a message with the same id in our header cache, if yes it is not a recently added message, and if no consider it as a recently added message. I obviously don't know anything about the imap provider internals, so I may be proposing something hard or impossible to do. So my question would be : has something like this being considered already ? [ I don't know what other IMAP clients do in this respect, but Thunderbird doesn't seem to take into account the \Recent flag, or at least doesn't seem impact by such a bug as this one].
... o.O That is in effect what my suggestion above was all about... Except that instead of using message-ids, it uses uids, which are far easier to check for :) Anyways, my previous comment explained exactly how to fix this problem...
there is a patch at bug 356092. can someone please review that?
Created attachment 104647 [details] [review] proposed eds patch for evolution-data-server; As discussed with Jeff, something like this?
Milan, I feel you are just right. But we may have to do it as a configurable thing. No UI or anything atm. I feel that your patch way should be the default. If some users want the old way.. then they should be able to go that way also. For now, I think we can just go about is getenv based thing. If something like FILTER_RECENT environment is set go the old way other wise yours.. May be a short term thing.. which we may revisit with UI or drop totally during 2.23.x.
Committed to trunk. Committed revision 8469. The committed version is slightly modified from the attached: if ((mi->info.flags & CAMEL_IMAP_MESSAGE_RECENT) != 0 || getenv ("FILTER_RECENT") == NULL)
*** Bug 494639 has been marked as a duplicate of this bug. ***
Milan, I saw a issue here with this today. My UID validity expired and the next thing Evolution did was applied filters to all my mails in my INBOX. Reason, we clear the summary if the validity is gone. So all the mail summaries are downloaded and filters are applied. I'm so poor, since I had a archiving filter, that is now downloading a second copy of my INBOX.
srag, so adding check for unread flag too? The previous patch logic was exactly like this, when loading into summary, apply filters. If it's possible to know we upload to the summary because of expire thing, then it can be added there too. What is your idea?
Milan, I definitely have no idea on how to solve this. Fejj any thought?
I'd say that the easiest (imperfect, but likely good enough) fix for this would be to only add the uid to the recent array if the SEEN flag has not been set.
Created attachment 105991 [details] [review] proposed eds patch ][ for evolution-data-server; but it will take us back where we were, do not you think? It will also stop filtering on the other machine, I guess. I discussed it on IRC with srag, and the less-pain way for this is probably this one. I added special parameter used to indicate if summary is recreated because of some reason (actually only because of invalidity), and if so, then filters are applied to only recent messages, like before. It has some possible issues like not applied filters on the other machine when one open it and it realizes invalid summary, but this is probably the best thing we can do here. btw, I wasn't able to test this properly, because I do not know how to persuade my server to report invalid summary for me.
I'm so confused here. Should we really bother about this? Just because I have a filter that archives messages? How does other apps behave here.
<mcrha> srag_, the filters, you said you are confused there, if I recall correctly. Jeff was talking about /Seen flag, but if we will rely on it, then it's almost same as the /Recent flag, because first Evo instance will ease /Recent and probably also add /Seen, (or erase /Unseen, that's the same I think). <srag_> yeah <srag_> mcrha, I think it is right the current way <srag_> my filter is sort of a hack to archive <srag_> otherwise.. double filtering is something right out of what we have designed <srag_> our approach can have double filtering.. may be in different machine <srag_> even if it happens on the same machine, it should have the same results * srag_ is convinced that the impl is right :) <mcrha> srag_, ok, and what with your issue then :) will we omit my last improve patch? <srag_> mcrha, I think so <mcrha> srag_, fine, will you update bug? <srag_> mcrha, I think we need to omit my issue and see for what users feel out there and go with the majority <srag_> mcrha, you can just paste the irc logs on the bug and close it :) <srag_> I would like to see what users say here..
Created attachment 113419 [details] [review] proposed eds patch for evolution-data-server; This should help for the expire situation.
I downgraded after my archive filters broke, and I opened bug 527567 for the problem. Applying filters to "new" mail that not only does the IMAP server indicate has been seen, but is already flagged as having been read, is rather brain-dead.
*** Bug 492468 has been marked as a duplicate of this bug. ***