GNOME Bugzilla – Bug 336076
IMAP mail checking is prohibitively slow; uses SELECT and fetches flags instead of just STATUS
Last modified: 2013-08-19 14:53:08 UTC
Please describe the problem: Older versions of evolution would use STATUS to check for new mail in other folders. They were still quite inefficient because they'd do this for _all_ folders rather than only the folders for which it was necessary (bug #336074), but that was easily patched. Nowadays, however, Evolution seems to want to fetch every single header for every single mail in the IMAP store before it will operate. When I first started Evolution 2.6 it took two HOURS to start up, as it fetched all the headers. That is just not usable. Even when I came back in the morning and all the headers had been downloaded, Evolution 2.6 still had surprises in store for me. Each time it tried to check for new mail, it issued a SELECT for each folder it was checking, then fetched all the flags (and any new headers) for that folder. What was a three-line transaction (STATUS, untagged response, command completion) for each folder is now many thousands of lines of FLAGS responses. It's a gratuitous waste of time and bandwidth, and makes Evolution fairly much unusable. Evolution should go back to using STATUS to check how many unread mails are in the folders it wants to check for new mail. Steps to reproduce: Actual results: Expected results: Does this happen every time? Other information:
Created attachment 62036 [details] [review] Proof of concept This proof-of-concept patch makes Evolution use STATUS for any but the currently-selected folder. It isn't working correctly yet, but I think it's the way to go. It currently has two problems: Firstly, the GUI code is probably using camel_folder_get_unread_message_count() to get the unread message count, and that's ignoring the summary info which my patch updates -- it's making it up from the cached message index. We should probably fix that in imap_getv(), and then the message counts should work. Secondly, the check against imap_store->current_folder isn't the right thing to do. What we actually want to do is use SELECT if it's a folder for which the GUI actually cares about the message index, and use STATUS only when the GUI cares only about the unseen count. The GUI can have more than one folder 'open' at a time, in multiple windows, and it can use some folders as source for vFolders, etc. -- all these folders would fall into the first category where we should use 'SELECT'. We should probably add a folder flag to show that the GUI has a folder 'open', and then the code in my patch can use SELECT (or preferably EXAMINE) for any folder which is 'open', and STATUS for any folder which is not.
I've filed this with severity 'major' because it's actually preventing me from switching to Fedora Core 5 on most of my machines. I cannot live with a mailer which requires a multi-megabyte download just to start up. With the combination of this and bug #336074, Evolution just isn't usable for me on my DSL line -- let alone when I'm on GPRS.
This is just a me too message, this problem bugs me as well, although it helps somewhat to disable "Enable mailing list detection required for some filter and vFolder rules" on every folder I have (50+, took me a while to go to each and uncheck this checkbox in the properties Window). Having to use evolution through a 512kbit/sec line with a large pile of mails is a challenge with Evolution 2.4 and up.
Confirming. Regression.
Balazs, The only option I see in each folder's properties window is 'Copy folder content locally for offline operation'. Where should I be looking for the option you're referring to?
To clarify the severity of this problem, I should give some numbers... I've previously reported that it took Evolution two hours to start up when I first started it (http://david.woodhou.se/what-evo-did-to-my-bandwidth-graph-elided.png). That was using the _uplink_ of a DSL line. You can clearly see the initial download and then the constant abuse of bandwidth for the next few days, with a short period on the second afternoon while it was stopped and I was experimenting with the above patch. I tested this again, on a machine closer to the IMAP server -- it actually has a gigabit connection to the server. This time it 'only' took 23 minutes to start up Evolution, and it downloaded half a gigabyte of headers. Note that both of the above were measured _with_ my patch for bug #336074 already applied -- otherwise it would have been handling thirteen times as many folders.
Created attachment 62104 [details] Properties dialog for IMAP folders This screenshot shows the option I was referring to. I'm using the IMAP4rev1 camel backend.
*** Bug 322596 has been marked as a duplicate of this bug. ***
Commenter from the dupe (which was filed first, thank you ;) ). Yeah, this looks like the same issue, and is indeed more informative about why it's happening. I'd just like to add my datapoint: evolution is essentially unusable for me in its current state. I think we can all agree that a 30 second wait time from startup to knowing which folders have what amount of unread mail, let alone a 10minute-2hour wait time, is unacceptable considering how well evolution 2.2 behaved in this regard. For those of us with large numbers of large folders, this is a huge regression.
Balazs, please note that the IMAP4rev1 named backend is deprecated, unmaintained and shouldn't have been available in the first place. It is considered experimental and has a few serious issues. See bug 324118. Jeremy, thanks for the quick response. :-) Priority Urgent. Target Milestone set. Regression.
BTW, David wrote he's using Fedora 5 (just as I am). Maybe it's worth noting that Fedora removed IMAP4rev1 from Evolution back in January and Fedora 5 ships with only one "IMAP" backend available for users. Bug 324118 seems somewhat less bad thanks to this (few less users bugging Karsten about the broken backend).
It's ironic that we actually disabled the IMAP4rev1 back end in Fedora because we considered it to be totally unusable.... because it was showing exactly the same behaviour described in this bug; downloading all headers in all folders on startup, and all flags in all folders on mail-check :) That was https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=167574
I'd guess that it's not the individual imap backends that are the problem here, I'd wager that the front-end is the culprit here. I'll admit to not having had a look yet, tho.
Given how heavy the network usage is while the new mail indicators are being updated, I tend to believe it's not (entirely) the front-end's fault, at least not the GUI elements.
what I meant by blaming the front-end is that the front-end is requesting all that info from the backend unnecessarily.
Jeff, please be specific. You mean that the front-end should not be calling camel_folder_refresh_info() for the folders in question? What should it be doing instead, if it only needs the unseen and message counts rather than all the headers &c? (In fact, the front end should _never_ need all the headers in one go. The IMAP server can do sorting and threading for you, so in the common case you can build the tree which represents the mails in the folder index without actually seeing all the headers anyway. You only actually _need_ to fetch headers for a given mail when the user scrolls to it in the message list and it becomes visible, so you need to display the Subject, From and Date headers. That's a separate issue though, although it arises from the same design error.)
I ran into some code the other day that left me wondering if this behaviour was a bugfix for accurate message counts when folders contain messages marked as Junk and possibly also \Deleted It is desirable to reduce the total/unread counts to only include those not marked as Junk (same for messages marked as \Deleted) See camel-folder.c:folder_getv() if ((flags & (CAMEL_MESSAGE_SEEN|CAMEL_MESSAGE_DELETED|CAMEL_MESSAGE_JUNK)) == 0) unread++; You can't get a similarly accurate count from the IMAP server using STATUS commands, unfortunately. So I believe this change to have been made on purpose to fix total/unread message counts. Your patch would re-break that
While it may break that, I would wager that it is more important to have usable mail than to hide deleted/junk messages. Introducing code that causes an enormous delay in getting to mail is not good. I'd rather see the message count code backed out until such time that it can be accomplished without the penalty that it has introduced.
You've merely reprased one of my existing 'TODO' items -- see the discussion about folders which the GUI side considers to be 'open'. Any folder which is currently being display, which the GUI is using as source for vFolders, or which it's doing its weird 'Junk' stuff in, would still be treated the slow way. That's not the optimal case anyway, and it's not the case I'm interested in improving.
Jesse: users have historically cared more about message counts being accurate than speed (yes, it boggles my mind too, but they do). The instant anything breaks message counts, we'll get flooded with bug reports about it. But not many people are complaining about the slowness which has been around for a YEAR now. In fact, before David Woodhouse reported this bug, there weren't any complaints about it... I don't think backing out the message count fix is the best approach here. Also note that there's no other way to get those counts since the IMAP protocol does not allow any other way of getting that information.
Perhaps users (like me) were silently abandoning evolution when it took far far too long to get their email. I only started filing bugs because I was prompted to after ranting about the sorry state of Evolution in my blog. So I file bugs. If upstream really cares more about just informative looks rather than core function, perhaps all is lost. :/
> Jesse: users have historically cared more about message counts being accurate than speed Please please *please* don't mark this WONTFIX. Evolution is easilly my favorite mail reader in terms of UI and integration with other data, but I haven't used it in months because it's simply too slow to get anything done. FWIW, if there's "deleted" email that's still in a folder which I haven't expunged or marked as read, I don't really care whether it's included in the new mail count. This is mail that the user is either going to be expunging or moving to a trash folder immediatly anyways. Besides, isn't deleted mail immediatly marked as "not-new", (if not explicitly "read")? Likewise if a folder contains "junk" that wasn't so obviously spam to simply delete right away, I want to know that a folder has it, not have it silently lost with no visual indication of its delivery to a folder. How else can I know there's a message I need to check for a possible false-positive spam detection? Jeffrey: Can you provide some links to bugs where other users have complained about inaccurate mail counts that this fix would conflict with? I'd be very interested to see what sort of use case would have a user want deleted/junk mail that's still new and not expunged/moved-to-trash not included in their message counts. > In fact, before David Woodhouse reported this bug, there weren't any complaints about it... Actually my original bug which was marked as a duplicate of this (on account of this having better information), bug# 322596, was filed and recieved some discussion approximately 5 months ago. Perhaps the bugzilla needs to back-date bugs to the date of their oldest marked duplicate?
It was reported before I found it -- bug 322596. It just wasn't tracked down to a specific change in behaviour. Fedora users didn't see it in Evolution 2.4 because we never shipped Evolution 2.4 -- we went straight from 2.2 in FC4 to 2.6 in FC5. And then it became apparent.
> FWIW, if there's "deleted" email that's still in a folder which I haven't > expunged or marked as read, I don't really care whether it's included in the > new mail count. Well, then you're an odd-ball :) Every other user cares about this... > This is mail that the user is either going to be expunging or > moving to a trash folder immediatly anyways. No, if it's marked as \Deleted then it is already showing up in Trash and not in your INBOX (or whatever folder). You don't mark a message as \Deleted and then move that message to Trash. Marking it as \Deleted already makes it show up in Trash (and likely hides it from the normal mailbox view unless you turned that feature off). Same goes when marking messages as Junk, they disappear from the folder view and are shown instead in the Junk folder. It just so happens that in both cases, they are still in your INBOX folder - just hidden from view. Now, say you have 10 messages in your INBOX folder - you hit Delete for, say, 3 of them. You also mark 5 messages as Junk. That will leave you with 2 messages VIEWABLE in your INBOX. With the aformentioned fix (which is currently in Evo), Evolution will tell the user "you have 2 messages in this folder". If that fix were reverted, it would tell the user they have 10 messages. Well, clearly the user is going to start freaking out because they can't find the other 8 messages. Do you understand why that fix is important now? > Besides, isn't deleted mail > immediatly marked as "not-new", (if not explicitly "read")? Only if it was marked as \Deleted BY Evolution, if the user switches to other mailers (like uses Webmail or some other mailer at work/home), then this doesn't hold true.
(In reply to comment #24) > > No, if it's marked as \Deleted then it is already showing up in Trash and not > in your INBOX (or whatever folder). You don't mark a message as \Deleted and > then move that message to Trash. Marking it as \Deleted already makes it show > up in Trash (and likely hides it from the normal mailbox view unless you turned > that feature off). Same goes when marking messages as Junk, they disappear from > the folder view and are shown instead in the Junk folder. It just so happens > that in both cases, they are still in your INBOX folder - just hidden from > view. > > Now, say you have 10 messages in your INBOX folder - you hit Delete for, say, 3 > of them. You also mark 5 messages as Junk. That will leave you with 2 messages > VIEWABLE in your INBOX. With the aformentioned fix (which is currently in Evo), > Evolution will tell the user "you have 2 messages in this folder". If that fix > were reverted, it would tell the user they have 10 messages. Well, clearly the > user is going to start freaking out because they can't find the other 8 > messages. > > Do you understand why that fix is important now? > No, but this clearly points out why psuedo showing them in Trash or Junk while not actually moving them is a bad design. Mark deleted messages with a line through it, mark Junk simarly, but don't just hide it from view. If Evo crashes (as it often does) before it can actually make the move, now you've got a difference in what evo says is in your inbox vs the next app or webmail. This kind of deception is just poor design, and you shouldn't FURTHER break the mailer to prop up this poor design.
Jesse, there's a patch in bug 253110 which fixes the unwanted Trash and Junk folders -- I don't think it gains us much to rant about that here.
Its not that I don't want them, they are fine, but don't fake like you're going to move the message but don't actually move it. If you're going to move the message, either move it or mark it visibly as scheduled to move, don't just hide it. Then your count is accurate as you can clearly see all the messages the count says there is.
"that's like, your opinion, man."
(In reply to comment #28) > "that's like, your opinion, man." Well that was a useless comment. In the meantime, this bug is still open, and Gnome's premier groupware app is being laughed at.
this bug can't be fixed
Given that other clients perform orders of magnitude better than evolution in this regard, this bug certainly can be fixed. There was a good reason for this being marked urgent/major, as described in the earlier comments, and to me this indicates that it certainly should be fixed. Please reconsider this.
Evolution has features those other mailers don't. Unfortunately, that means Evolution needs more data. This of course makes it slower. If there was any way to fix it without removing functionality, I would, but there isn't.
What feature is that? If you're refering to the way that evolution keeps deleted mail in the folder it was deleted from instead of moving it to the trash, I have to say I seriously do not see the value in this feature. I'd be interested to hear someone's explaination of how this is valuable. If that's not the one, then please expand on what feature you're referring to?
Ask david woodhouse why that's useful, he apparently refuses to use KMail because they immediately move the message to a physical Trash folder.
Although I'd prefer to have them moved to the Trash folder, that's not even neccesarry. Just marking them as read whenever they're marked as deleted (which evolution appears to already do) is sufficient to remove the message from the new-mail count, is it not?
no, it is not sufficient because the user may alternate between clients (or use multiple at once even) and the other client may not mark something read when it marks them as deleted.
It seems that the situation of a mixed-client environment (especially when very few other clients in my experience leave deleted messages in the folder they're found in) is a much rarer edge-case than that of just having lots of mail to read.
we've gotten numerous complaints in the past about unread counts being off due to a user having used some other client and marking unread messages as deleted. this is a real scenario that cannot be ignored either. I do agree that this slowness sucks, but there's no way to fix it without breaking something else. catch22. I really think this just has to be left alone until a redesign of the ui is done such that this code isn't needed any longer.
One would assume that usability (IE its too damn slow to use) would come before aesthetics such as the message count.
Would it be difficult to check wheter visible unread message count differs from unread count reported by server (if the user can spot it and report, Evolution should know this by itself)? Would it be difficult to issue the whole folder re-check only in situation when these numbers differ? As I understand it, this seemingly simple task requires a major rewrite, am I right?
Slow and _expensive_. I did update my machines to FC5, since I couldn't really let Evolution hold me back from other things, and my fix for #336074 does at least manage to reduce the problem to a dull roar on my DSL line. Looking at the bandwidth stats provided by my ISP, it seems as if Evolution is going to be solely responsible for me reaching my 3GB 'daytime' download allocation this month; any usage _more_ than that will be chargable. http://david.woodhou.se/CBUK13861537.png shows it -- the near-constant traffic there is Evolution. The red dots every 12 minutes are Evolution in the office, which is uploading the same set of flags for every mail in every (active) folder, every 12 minutes.
Jesse: unread counts is also a usability issue. trust me, anytime they break in any way shape or form, we get nagged until it's fixed again. Leszek: there's no way to get a comparison without SELECTing the folder and scanning. A rewrite won't solve it because it would still have the same demands. The functionality would have to be modified in such a way as to remove the demands, but that's also likely to make user's unhappy (a lot of users really like vTrash-like folders - myself included, it means I can undelete a message in my Trash folder and it ends up back in the original folder AND in the same offset in the original folder - there's NO way to do that without vTrash). so again, catch22.
We're talking about an issue that annoys a few people vs one that makes evolution completely unusable and potentially costs them a substantial amount of money in service fees/fines. If we have to choose one of the above problems to live with until a major UI rewrite that properly addresses these issues, there's no question in my mind which one needs to be fixed and which one needs to be tolerated. The severity of this is huge. The only reason you won't see alot of nagging about this one is that the long time evolution users who are affected by this will switch to other clients, if they haven't already like me.
Reopening, and marking dependant on bug #206061 ("allow normal, non-vFolder Trash and Junk folder"). Once evolution behaves sanely wrt trash & junk folders (where 'sanely' means 'interoperates with any other client in the world), this behavior can be backed out without issue. I'd suggest in fact that this behavior be backed out immediatly given its severity, and a message count bug be re-opened, and marked dependant on #206061 as well, as it'll also fix that issue once and for all. As for the undeleted-mails going back where they came from, that's trickier. It seems recording the source folder information somewhere else would accomplish this without introducing bandwidth issues, and might even be standardized to a point where other clients could interoperate with that feature.
*** This bug has been marked as a duplicate of 206061 ***
marking as dependant on 206061 rather than as a duplicate of.
As of current development version, 2.7.x, a fix has been committed that fetches only the required header during a mail-fetch. So, I guess, that fixes the "slowness" part of the bug.
"only the required headers" would be none of them, if all we want is to know how many unread messages there are in the folder. shres had a patch based on my proof of concept which attempted to do this, by fetching full headers only for those folders which _needed_ it, by virtue of being source for vFolders.
No appreciable change on my end in eds 1.8.1. Evolution is still an order of magnitude worse than its competition here, and largely unusable on my IMAP account.
The scary thing is, even with this regression Evolution doesn't get the message counts right. If you use another IMAP client to read messages, then even if you change away from the folder and back again Evolution doesn't notice that the messages have become read.
Just thinking.... Do Select + Flag fetches for Visible folders and Folders that VFolders need and for rest Just STATUS should be fine? Fejj, is there any other conditions that needs to be addressed? Bug #336074 is fixed already and also note that there is a bug/patch in progress to have a option to disable vfolders (Don't remember the bug number)
Was there ever any response from Fejj? Still seeing this behavior with eds 2.22.2 in ubuntu hardy. Due for a target milestone update, since 1.7 was missed :)
instead of doing STATUS, maybe you could SELECT and then do a UID SEARCH to get accurate counts? that would probably be faster than doing a full summary fetch.
(This is not "urgent", adjusting priority)
As a more than 2 year old regression, it's at least "high". Further, for several commenting users this is blocking evolution mail being at all usable, which according to the documentation of the priority field puts it firmly into urgent. That said, changing the priority around won't change the fact that there's apparently no interest in resolving this issue, despite several proposals of how to address it. If there's any additional feedback or information I can provide, I'm happy to. But if it's not going to get fixed, let's mark it WONTFIX so people affected by this can finally move on and find another client that better suits their needs.
After being away for six weeks, it has taken me a full week to get back up and running with Evolution. I suspect that some of the issue is my 30 filters (I am looking at using procmail on the server instead of Evolution filters to fix that), and 30 email lists I subscribe to, but it also appears that I had to deal with this issue as well, as it took all night to upload my new message headers. Slow is not acceptable. Please think about this issue!
Me too. The only reason I'm using Evolution anymore is because I am required to use the Exchange connection rather than IMAP.
3 years on... Confirmed that evolution 2.26.1 on a fresh install still takes about 20 minutes on my IMAP account just to tell me which folders have new messages in them, at about 200kbytes/second. It spends several minutes "Fetching summary information for new messages" in each of my archival mail folders before even looking for a new-message count in the later folder where I actually receive mail. Since I have several thousand emails archived going back almost 10 years, evolution takes a HUGE amount of bandwidth and memory/local-storage loading message information it will never use. Even the few folders I have that are active with a few hundred messages from a mailinglist make evolution essentialy unusable. There's been some progress in bug http://bugzilla.gnome.org/show_bug.cgi?id=206061 . How does the development there impact this bug's resolution? Is there work required here to take advantage of that? Is that work ongoing? Does anyone have a currently feasible workaround for this bug, or is it still a significant blocker to evolution adoption?
From a fresh install, evolution still takes over 20 minutes to tell me what folders require attention, where Thunderbird takes seconds to get message counts. As a rule, by the time this process is finished, the counts are out of date... The bulk of this time is spent getting headers for large archival/historical folders, which are accessed only occasionally, including the large, normal, trash/spam folder that other clients use. How bad is this issue for gmail users, where the 7gb account size can mean a truly enormous number of messages in a small number of folders? In a case like that, asking the IMAP server for the message count instead of examining each message individually should be a *HUGE* performance difference... Incrementing versions.
A subsequent run takes about 2 minutes to update the new-message count on all folders. I should point out that this isn't on dial-up or anything, it's a normal wifi connection on campus.
Heh, you should try it on a 2G GPRS connection. Even if I _do_ want to pay for all the extra traffic that Evolution generates, I usually end up firing up pine to read my email while I'm waiting for Evolution to start up.
How about first doing STATUS (UNSEEN UIDNEXT) and then: a) unseen=0 -> there are 0 unseen messages, no need to select the mailbox b) unseen and uidnext have the same values as in previous check -> unlikely that anything changed, do nothing c) finally as fallback do the slow SELECT + flag fetching. And like Jeffrey mentioned, perhaps use UID SEARCH to get the count instead of fetching flags.
Still present in 2.29.x. Incrementing versions again. "Normal" priority is described as "Either a fairly straightforward workaround exists or the functionality is not very important and/or not frequently used." There's no workaround for unusably slow performance with gmail. If evolution devs really want to say that compatability with gmail is "not very important and/or not frequently used", then I guess the priority could go down again. For now i'm setting it back up to 'high', since the definition for 'normal' doesn't really fit. This is especially disasterous for ubuntu users, since evolution is the way the desktop notices new mail. Meaning just to know whether or not you need to check your mail can takes > 20 minutes. Assuming that the connections don't die before then, most people seem to give up and stop using evolution entirely before then.
*** Bug 269686 has been marked as a duplicate of this bug. ***
Closing as OBSOLETE since this was filed against the legacy IMAP backend which was retired in 3.8. Reopen if this issue is still present in Evolutoin 3.8 or later.