GNOME Bugzilla – Bug 269686
IMAP operations are sometimes very slow
Last modified: 2010-03-16 18:47:58 UTC
Please fill in this template when reporting a bug, unless you know what you are doing. Description of Problem: IMAP (SSL) operations are unbearably slow compared to 1.4. Steps to reproduce the problem: 1. Open Evolution with an IMAP SSL account with ~20 folders and thousands of messages in each. 2. Change folder, or click on a new message or go online after having been offline 3. Wait.... Actual Results: Evolution takes several minutes before it is usable again. It "hangs" in "Retrieving message..." and "Storing folder XYZ...", and "Refreshing folder..." Expected Results: Evolution should not take that long. 1.4 did not. How often does this happen? Every time Additional Information: I'm running an up-to-date Debian unstable system. This slowness is really strange, as I cannot fathom what is going on. It seems my uw-imapd server is working hard, but I don't know what it is doing. /Mikael
nothing has changed that would affect speed. if you can prove otherwise (diffs?), then you can attach the proof and reopen.
Here's another report of this, displaying the exact same symptoms: http://www.redhat.com/archives/fedora-test-list/2004-November/msg01300.html https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=139145 I could try to produce a similar log, but I doubt there would be anything more interesting in mine. Please do not ignore this problem. Thanks, /Mikael
looks like it's "slow" because he has a lot of folders. we can't speed up the server from evolution so I fail to see how this is our problem. without specifics, there's nothing we can do.
Ok, I wont argue with the closing of the bug. But where did you get the "many folders" from? He said he had 290 messages in his inbox and it was still slow when switching from local:Drafts to his IMAP INBOX. He said nothing about the number of folders. And still, I have the very same problem with around 20 folders (not "many" I hope). And a behaviour that is very different from 1.4. Should I reinstall 1.4 and do comparative timings? The result will be what I have said here: 1.4: response time from immediate to 5-10 secs 2.0: response time 1-2 minutes for the tasks I mentioned above. Please give me some hints what I need to do to convince you there is a real problem. Discuss it on evoluion-hackers? /Mikael
diff the imap code bwtween 1.4 and 2.0 and show me what changes are making it slow. or do some actual profiling to find where this "slowness" is telling me it's slow doesn't help at all, especially when afaict, it's faster than 1.4, not slower. I also fail to see how ssl will make it oodles slower, unless the ssl libs themselves are what is so slow - in which case, not our fault because we just use an external ssl lib.
SSL was only mentioned FYI, I don't think it affects the behavior.
Related? http://bugzilla.gnome.org/show_bug.cgi?id=247760
Jeff, ok: Your proud of your modifications, but your rude comments are contra productive and cannot deny the fact, that for some setups became incredibly slow. Ignoring the problem doesn't solve it. Deactivating "Check for new messages in all folders" improved performance dramatically, alltough I'd still consider folder scanning to be far too slow. To reproduce this setup some IMAP account, create some (nested) folders and populate some with alot of messages >1000. It's nearly impossible to work with Evolution in this constellation.
I too have noticed a MAJOR slowdown when I upgraded from RH9 and Evolution 1.4 to FC3 and Evolution 2.0.2. I wasn't sure it was Evolution or some other library/OS issue, but everything else seems fine so far. I have around 190 folders with thousands of emails. Under 1.4 this would take around 15 to load at startup. Not it takes over 2 minutes. Nothing has changed on the server (wu-imap 2001a-10) I set CAMEL_VERBOSE_DEBUG=1 and noticed that every single mailfolder has an LSUB called for it: ---------------------snip--------------------- received: * LSUB (\NoSelect) "/" mail/WT_clients received: * LSUB () "/" mail/sent-mail-aug-2004 received: * LSUB (\NoSelect) "/" mail/PEOPLE received: * LSUB (\NoSelect) "/" mail/PEOPLE received: * LSUB (\NoSelect) "/" mail/PEOPLE received: * LSUB (\NoSelect) "/" mail/WT_clients received: * LSUB (\NoSelect) "/" mail/PEOPLE received: * LSUB (\NoSelect) "/" mail/PEOPLE received: * LSUB (\NoSelect) "/" mail/WT_clients received: * LSUB (\NoSelect) "/" mail/PEOPLE received: * LSUB (\NoSelect) "/" mail/WT_clients received: * LSUB (\NoSelect) "/" mail/WT_clients received: * LSUB () "/" mail/security_incidents received: * LSUB () "/" mail/sent-mail-oct-2004 received: * LSUB () "/" mail/Politics received: * LSUB (\NoSelect) "/" mail/PEOPLE received: * LSUB (\NoSelect) "/" mail/PEOPLE received: * LSUB (\NoSelect) "/" mail/WT_clients received: * LSUB () "/" mail/WT_status_reports received: * LSUB () "/" mail/WT_management_team received: * LSUB (\NoSelect) "/" mail/PEOPLE received: * LSUB () "/" mail/AAA_TEST received: * LSUB (\NoSelect) "/" mail/PEOPLE received: * LSUB (\NoSelect) "/" mail/WT_clients received: * LSUB () "/" mail/sent-mail-jul-2004 received: * LSUB (\NoSelect) "/" mail/PEOPLE received: * LSUB (\NoSelect) "/" mail/Vendors received: * LSUB (\NoSelect) "/" mail/PEOPLE received: * LSUB () "/" mail/Trash received: A00005 OK LSUB completed sending : A00006 LSUB "" "mail/sent-mail/%" received: A00006 OK LSUB completed sending : A00007 LSUB "" "mail/ARCHIVE/%" received: A00007 OK LSUB completed sending : A00008 LSUB "" "mail/co-loc/%" received: * LSUB () "/" mail/co-loc/att received: * LSUB () "/" mail/co-loc/kiva received: * LSUB () "/" mail/co-loc/bluemarble received: * LSUB () "/" mail/co-loc/uunet received: * LSUB () "/" mail/co-loc/RackSpace -------------------snip-------------- Is this normal? This seems to be the vast majority of "hang time" during startup. thanks, -Brian
I must have been half asleep when I wrote that last comment! What I meant to say was: "would take around 15 *seconds* to load at startup. Now it takes over 2 minutes."
I noticed the delays are more infrequent when I increase the time between checking of new mail (previously 2 min, now 10 min). The "check for new mail in all folders" is unchecked, though. Seems evo does some very unnecessary things when checking for new mail. Thunderbird only taes a second to do that.
the imap code is being replaced so presuming no longer an issue (new imap code is a lot faster)
Great to hear! Any ideas when this code will be released? 2.0.4 or 2.2?
Reopening. IMAP4rev1 is just as slow in my tests. Just doing Send/Receive takes ~10-15 mins. It spends the time "Scanning folders..." even though "check for new mail in all folders" is unchecked. Thunderbird takes seconds. Debian unstable with newly uploaded 2.2.1.1. Sorry... :-( /Mikael
Brian: you could significantly speed up your IMAP performance by removing duplicate entries from your ~/.mailboxlist file on the IMAP server (since you are using uw.imapd). You have the same folders subscribed a bunch of times over and over, that is probably a huge portion of the time taken to scan your folders. evolution is *supposed* to scan the folders at send&receive time so that we can remove folders which have been deleted and/or add folders to the tree that may have been added by another client (as silly as it sounds to you and me, some people seem to like to use multiple IMAP clients at the same time and get upset when folder trees and other things don't match exactly) FWIW, I also have a bunch of folders each with oodles of mail in them and I don't experience any added slowness. anyways, yes, it does turn out that IMAP4rev1 code is slower than the old IMAP code in this respect - my suggestion: don't use IMAP4rev1 (likely it's going away anyway since I'm no longer working on it and have left the evolution team)
Can we tell Evo 2.2 to use the old IMAP code? Is this a setting somewhere?
in the account settings, make sure "serevr type" is set to "IMAP" instead of "IMAP4rev1" (IMAP4rev1 is the new code)
its definitely slower in 2.x because we ask for the list twice at startup ...
I have been getting similar timeouts with Evo 2.0.4 ("Retrieving message..." and "Storing folder XYZ...", and "Refreshing folder..."). Sadly, I use IMAP to a Lotus server (which is still miles better than using the Lotus client). I have roughly 100 messages in my Inbox, and I have "check all folders" disabled. The server capabilities appear to be: IMAP4rev1 AUTH=PLAIN LITERAL+ NAMESPACE QUOTA UIDPLUS I did some snooping on the network and found the following behavior: 183 37.677022 10.98.3.207 162.49.67.15 IMAP Request: A61343 SELECT DNTG 184 37.679595 162.49.67.15 10.98.3.207 IMAP Response: * 2 EXISTS 185 37.679670 10.98.3.207 162.49.67.15 IMAP Request: A61344 FETCH 2 UID 186 37.682039 162.49.67.15 10.98.3.207 IMAP Response: * 2 FETCH (UID 2) 187 37.682142 10.98.3.207 162.49.67.15 IMAP Request: A61345 STATUS INBOX (MESSAGES UNSEEN) 188 37.691087 162.49.67.15 10.98.3.207 IMAP Response: * 84 EXISTS 189 37.691155 10.98.3.207 162.49.67.15 IMAP Request: A61346 UID FETCH 3:* (FLAGS RFC822.SIZE INTERNALDATE BODY.PEEK[HEADER.FIELDS.NOT (RECEIVED)]) 659 57.691600 162.49.67.15 10.98.3.207 IMAP Response: * OK WORKING 1111 87.691569 162.49.67.15 10.98.3.207 IMAP Response: * OK WORKING 1331 96.545551 162.49.67.15 10.98.3.207 IMAP Response: A61346 OK FETCH completed 1333 96.545733 10.98.3.207 162.49.67.15 IMAP Request: A61347 SELECT INBOX 1334 96.553980 162.49.67.15 10.98.3.207 IMAP Response: * 84 EXISTS Note that the Lotus server is kind enough to let me know every 20 seconds that it is "WORKING" on something. The UID FETCH takes almost a full minute in this case. So, certainly, something ridiculous is up with the server, but the client request could be the root of it as well. First, the client SELECTs another one of my mail folders "DNTG". Second, it does a UID FETCH for 3:* which takes 59 seconds. Finally, the response comes back and _then_ the client SELECTs the Inbox folder. This seems a bit off, as I would expect it to SELECT Inbox and then UID Fetch for Inbox contents. This timeout happens repeatably. For comparison, I did a dump of Outlook's (sorry) conversation with the Lotus IMAP server. For reference, I never get _any_ hangs/timeouts when using Outlook with the same server under the same network conditions. 137 14.545447 10.98.3.91 162.49.67.15 IMAP Request: 9y38 SELECT "INBOX" 138 14.553795 162.49.67.15 10.98.3.91 IMAP Response: * 119 EXISTS 139 14.554433 10.98.3.91 162.49.67.15 IMAP Request: uibo UID FETCH 36075:* (BODY.PEEK[HEADER.FIELDS (References X-Ref X-Priority X-MSMail-Priority X-MSOESRec Newsgroups)] ENVELOPE RFC822.SIZE UID FLAGS INTERNALDATE) <a ton of good data follows instantaneously> Note that Outlook selects Inbox before the fetch, and it is also using a number that looks a lot more like a UID than a sequence number. I have seen some comments for similar Evo problems that suggest that the UID FETCH that Evo does should really just be a simple FETCH, given the range of message numbers it is using. I haven't studied IMAP enough to validate this, but it is another possible avenue to follow. Somehow, UID FETCH 3:* is much slower for this server with its current dataset than UID FETCH 36075:* is with the same dataset. So, while the Lotus IMAP server is clearly insane, the Evolution client could be exacerbating this with a couple slightly dubious commands.
the last is obviously a server performance issue with a complex fetch request, if you have the source you could try altering camel-imap-store.c:connect_to_server to set force_imap4 = TRUE as a default, which will use a simpler request, although it looks to me like the server is just slow. your assessment about uid fetch is incorrect. of course uid fetch 3:* is slower than uid fetch 36075:* - it is asking for more information. but it only asks for information it hasn't already retrieved. we definitely only use uids. the listing thing, is changed entirely in 2.4, as are a few other things; it's about as good as it's going to get for a while, but should be better than all other 2.x versions.
adding perf keyword
retargetting from 2.3 to 2.5
removing old 2.5 target milestone and retargetting to future - sorry.
Sounds like this is #336076, which already has alot of discussion about this behavior. Dupe?
Jeremy, I couldn't tell if it's a dupe, as the code has been rewritten a few times since my original report. Actually, I think the bug I saw has actually been fixed (though evo is still a lot slower than thunderbird). Feel free to mark as dupe or close.
*** This bug has been marked as a duplicate of bug 336076 ***