After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 269686 - IMAP operations are sometimes very slow
IMAP operations are sometimes very slow
Status: RESOLVED DUPLICATE of bug 336076
Product: evolution
Classification: Applications
Component: Mailer
2.2.x (obsolete)
Other All
: Normal enhancement
: Future
Assigned To: evolution-mail-maintainers
Evolution QA team
Depends on:
Blocks:
 
 
Reported: 2004-11-18 23:44 UTC by Mikael Nilsson
Modified: 2010-03-16 18:47 UTC
See Also:
GNOME target: ---
GNOME version: ---



Description Mikael Nilsson 2004-11-18 23:44:22 UTC
Please fill in this template when reporting a bug, unless you know what you
are doing.
Description of Problem:
IMAP (SSL) operations are unbearably slow compared to 1.4.

Steps to reproduce the problem:
1. Open Evolution with an IMAP SSL account with ~20 folders and thousands
of messages in each.
2. Change folder, or click on a new message or go online after having been
offline
3. Wait.... 

Actual Results:
Evolution takes several minutes before it is usable again. It "hangs" in
"Retrieving message..." and "Storing folder XYZ...", and "Refreshing folder..."


Expected Results:
Evolution should not take that long. 1.4 did not.

How often does this happen? 
Every time

Additional Information:
I'm running an up-to-date Debian unstable system.

This slowness is really strange, as I cannot fathom what is going on. It
seems my uw-imapd server is working hard, but I don't know what it is doing.

/Mikael
Comment 1 Jeffrey Stedfast 2004-11-19 17:23:41 UTC
nothing has changed that would affect speed. if you can prove
otherwise (diffs?), then you can attach the proof and reopen.
Comment 2 Mikael Nilsson 2004-11-19 18:17:56 UTC
Here's another report of this, displaying the exact same symptoms:

http://www.redhat.com/archives/fedora-test-list/2004-November/msg01300.html

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=139145

I could try to produce a similar log, but I doubt there would be
anything more interesting in mine.

Please do not ignore this problem. Thanks,

/Mikael
Comment 3 Jeffrey Stedfast 2004-11-19 18:34:49 UTC
looks like it's "slow" because he has a lot of folders. we can't speed
up the server from evolution so I fail to see how this is our problem.

without specifics, there's nothing we can do.
Comment 4 Mikael Nilsson 2004-11-19 19:22:30 UTC
Ok, I wont argue with the closing of the bug. 

But where did you get the "many folders" from? He said he had 290
messages in his inbox and it was still slow when switching from
local:Drafts to his IMAP INBOX. He said nothing about the number of
folders.

And still, I have the very same problem with around 20 folders (not
"many" I hope). And a behaviour that is very different from 1.4.
Should I reinstall 1.4 and do comparative timings? The result will be
what I have said here:

1.4: response time from immediate to 5-10 secs
2.0: response time 1-2 minutes

for the tasks I mentioned above.

Please give me some hints what I need to do to convince you there is a
real problem. Discuss it on evoluion-hackers?

/Mikael
Comment 5 Jeffrey Stedfast 2004-11-19 19:32:41 UTC
diff the imap code bwtween 1.4 and 2.0 and show me what changes are
making it slow.

or do some actual profiling to find where this "slowness" is

telling me it's slow doesn't help at all, especially when afaict, it's
faster than 1.4, not slower.

I also fail to see how ssl will make it oodles slower, unless the ssl
libs themselves are what is so slow - in which case, not our fault
because we just use an external ssl lib.
Comment 6 Mikael Nilsson 2004-11-19 20:01:31 UTC
SSL was only mentioned FYI, I don't think it affects the behavior.
Comment 7 Mikael Nilsson 2004-11-19 20:09:37 UTC
Related?

http://bugzilla.gnome.org/show_bug.cgi?id=247760
Comment 8 Mathias Hasselmann (IRC: tbf) 2004-12-02 03:28:28 UTC
Jeff, ok: Your proud of your modifications, but your rude comments are
contra productive and cannot deny the fact, that for some setups
became incredibly slow. Ignoring the problem doesn't solve it.

Deactivating "Check for new messages in all folders" improved
performance dramatically, alltough I'd still consider folder scanning
to be far too slow.

To reproduce this setup some IMAP account, create some (nested)
folders and populate some with alot of messages >1000. It's nearly
impossible to work with Evolution in this constellation.
Comment 9 Brian Eric Bothwell 2004-12-10 05:13:00 UTC
I too have noticed a MAJOR slowdown when I upgraded from RH9 and
Evolution 1.4 to FC3 and Evolution 2.0.2.  I wasn't sure it was
Evolution or some other library/OS issue, but everything else seems
fine so far.  

I have around 190 folders with thousands of emails.  Under 1.4 this
would take around 15 to load at startup.  Not it takes over 2 minutes.
 Nothing has changed on the server (wu-imap 2001a-10)

I set CAMEL_VERBOSE_DEBUG=1 and noticed that every single mailfolder
has an LSUB called for it:

---------------------snip---------------------
received: * LSUB (\NoSelect) "/" mail/WT_clients
received: * LSUB () "/" mail/sent-mail-aug-2004
received: * LSUB (\NoSelect) "/" mail/PEOPLE
received: * LSUB (\NoSelect) "/" mail/PEOPLE
received: * LSUB (\NoSelect) "/" mail/PEOPLE
received: * LSUB (\NoSelect) "/" mail/WT_clients
received: * LSUB (\NoSelect) "/" mail/PEOPLE
received: * LSUB (\NoSelect) "/" mail/PEOPLE
received: * LSUB (\NoSelect) "/" mail/WT_clients
received: * LSUB (\NoSelect) "/" mail/PEOPLE
received: * LSUB (\NoSelect) "/" mail/WT_clients
received: * LSUB (\NoSelect) "/" mail/WT_clients
received: * LSUB () "/" mail/security_incidents
received: * LSUB () "/" mail/sent-mail-oct-2004
received: * LSUB () "/" mail/Politics
received: * LSUB (\NoSelect) "/" mail/PEOPLE
received: * LSUB (\NoSelect) "/" mail/PEOPLE
received: * LSUB (\NoSelect) "/" mail/WT_clients
received: * LSUB () "/" mail/WT_status_reports
received: * LSUB () "/" mail/WT_management_team
received: * LSUB (\NoSelect) "/" mail/PEOPLE
received: * LSUB () "/" mail/AAA_TEST
received: * LSUB (\NoSelect) "/" mail/PEOPLE
received: * LSUB (\NoSelect) "/" mail/WT_clients
received: * LSUB () "/" mail/sent-mail-jul-2004
received: * LSUB (\NoSelect) "/" mail/PEOPLE
received: * LSUB (\NoSelect) "/" mail/Vendors
received: * LSUB (\NoSelect) "/" mail/PEOPLE
received: * LSUB () "/" mail/Trash
received: A00005 OK LSUB completed
sending : A00006 LSUB "" "mail/sent-mail/%"
received: A00006 OK LSUB completed
sending : A00007 LSUB "" "mail/ARCHIVE/%"
received: A00007 OK LSUB completed
sending : A00008 LSUB "" "mail/co-loc/%"
received: * LSUB () "/" mail/co-loc/att
received: * LSUB () "/" mail/co-loc/kiva
received: * LSUB () "/" mail/co-loc/bluemarble
received: * LSUB () "/" mail/co-loc/uunet
received: * LSUB () "/" mail/co-loc/RackSpace
-------------------snip--------------

Is this normal?  This seems to be the vast majority of "hang time"
during startup.

thanks,

-Brian
Comment 10 Brian Eric Bothwell 2004-12-11 20:35:05 UTC
I must have been half asleep when I wrote that last comment!  What I
meant to say was:

"would take around 15 *seconds* to load at startup.  Now it takes over
2 minutes."
Comment 11 Mikael Nilsson 2005-01-12 10:49:19 UTC
I noticed the delays are more infrequent when I increase the time
between checking of new mail (previously 2 min, now 10 min). The
"check for new mail in all folders" is unchecked, though.

Seems evo does some very unnecessary things when checking for new
mail. Thunderbird only taes a second to do that.
Comment 12 Jeffrey Stedfast 2005-01-12 17:53:32 UTC
the imap code is being replaced so presuming no longer an issue (new
imap code is a lot faster)
Comment 13 Mikael Nilsson 2005-01-12 22:06:17 UTC
Great to hear!

Any ideas when this code will be released? 2.0.4 or 2.2?
Comment 14 Mikael Nilsson 2005-04-09 12:05:19 UTC
Reopening.

IMAP4rev1 is just as slow in my tests. Just doing Send/Receive takes
~10-15 mins. It spends the time "Scanning folders..." even though
"check for new mail in all folders" is unchecked.

Thunderbird takes seconds.

Debian unstable with newly uploaded 2.2.1.1.

Sorry... :-(

/Mikael
Comment 15 Jeffrey Stedfast 2005-06-20 13:35:58 UTC
Brian: you could significantly speed up your IMAP performance by removing
duplicate entries from your ~/.mailboxlist file on the IMAP server (since you
are using uw.imapd). You have the same folders subscribed a bunch of times over
and over, that is probably a huge portion of the time taken to scan your folders.

evolution is *supposed* to scan the folders at send&receive time so that we can
remove folders which have been deleted and/or add folders to the tree that may
have been added by another client

(as silly as it sounds to you and me, some people seem to like to use multiple
IMAP clients at the same time and get upset when folder trees and other things
don't match exactly)

FWIW, I also have a bunch of folders each with oodles of mail in them and I
don't experience any added slowness.

anyways, yes, it does turn out that IMAP4rev1 code is slower than the old IMAP
code in this respect - my suggestion: don't use IMAP4rev1 (likely it's going
away anyway since I'm no longer working on it and have left the evolution team)
Comment 16 Thomas Cameron 2005-06-20 15:40:31 UTC
Can we tell Evo 2.2 to use the old IMAP code?  Is this a setting somewhere?
Comment 17 Jeffrey Stedfast 2005-06-20 15:48:53 UTC
in the account settings, make sure "serevr type" is set to "IMAP" instead of
"IMAP4rev1" (IMAP4rev1 is the new code)
Comment 18 Not Zed 2005-06-24 17:24:03 UTC
its definitely slower in 2.x because we ask for the list twice at startup ...

Comment 19 Payton R. White 2005-06-30 22:04:29 UTC
I have been getting similar timeouts with Evo 2.0.4 ("Retrieving message..." and
"Storing folder XYZ...", and "Refreshing folder...").  Sadly, I use IMAP to a
Lotus server (which is still miles better than using the Lotus client).  I have
roughly 100 messages in my Inbox, and I have "check all folders" disabled.

The server capabilities appear to be:

		IMAP4rev1 AUTH=PLAIN LITERAL+ NAMESPACE QUOTA UIDPLUS

I did some snooping on the network and found the following behavior:

    183 37.677022   10.98.3.207           162.49.67.15          IMAP    
Request: A61343 SELECT DNTG
    184 37.679595   162.49.67.15          10.98.3.207           IMAP    
Response: * 2 EXISTS
    185 37.679670   10.98.3.207           162.49.67.15          IMAP    
Request: A61344 FETCH 2 UID
    186 37.682039   162.49.67.15          10.98.3.207           IMAP    
Response: * 2 FETCH (UID 2)
    187 37.682142   10.98.3.207           162.49.67.15          IMAP    
Request: A61345 STATUS INBOX (MESSAGES UNSEEN)
    188 37.691087   162.49.67.15          10.98.3.207           IMAP    
Response: * 84 EXISTS
    189 37.691155   10.98.3.207           162.49.67.15          IMAP    
Request: A61346 UID FETCH 3:* 
           (FLAGS RFC822.SIZE INTERNALDATE BODY.PEEK[HEADER.FIELDS.NOT (RECEIVED)])
    659 57.691600   162.49.67.15          10.98.3.207           IMAP    
Response: * OK WORKING
   1111 87.691569   162.49.67.15          10.98.3.207           IMAP    
Response: * OK WORKING
   1331 96.545551   162.49.67.15          10.98.3.207           IMAP    
Response: A61346 OK FETCH completed
   1333 96.545733   10.98.3.207           162.49.67.15          IMAP    
Request: A61347 SELECT INBOX
   1334 96.553980   162.49.67.15          10.98.3.207           IMAP    
Response: * 84 EXISTS


Note that the Lotus server is kind enough to let me know every 20 seconds that
it is "WORKING" on something.  The UID FETCH takes almost a full minute in this
case.  So, certainly, something ridiculous is up with the server, but the client
request could be the root of it as well.  First, the client SELECTs another one
of my mail folders "DNTG".  Second, it does a UID FETCH for 3:* which takes 59
seconds.  Finally, the response comes back and _then_ the client SELECTs the
Inbox folder.  This seems a bit off, as I would expect it to SELECT Inbox and
then UID Fetch for Inbox contents.  This timeout happens repeatably.

For comparison, I did a dump of Outlook's (sorry) conversation with the Lotus
IMAP server.  For reference, I never get _any_ hangs/timeouts when using Outlook
with the same server under the same network conditions.
 
 
    137 14.545447   10.98.3.91            162.49.67.15          IMAP    
Request: 9y38 SELECT "INBOX"
    138 14.553795   162.49.67.15          10.98.3.91            IMAP    
Response: * 119 EXISTS
    139 14.554433   10.98.3.91            162.49.67.15          IMAP    
Request: uibo UID FETCH 36075:* 
           (BODY.PEEK[HEADER.FIELDS (References X-Ref X-Priority
X-MSMail-Priority X-MSOESRec Newsgroups)] 
           ENVELOPE RFC822.SIZE UID FLAGS INTERNALDATE)
		<a ton of good data follows instantaneously>

Note that Outlook selects Inbox before the fetch, and it is also using a number
that looks a lot more like a UID than a sequence number.  I have seen some
comments for similar Evo problems that suggest that the UID FETCH that Evo does
should really just be a simple FETCH, given the range of message numbers it is
using.  I haven't studied IMAP enough to validate this, but it is another
possible avenue to follow.  Somehow, UID FETCH 3:* is much slower for this
server with its current dataset than UID FETCH 36075:* is with the same dataset.

So, while the Lotus IMAP server is clearly insane, the Evolution client could be
exacerbating this with a couple slightly dubious commands.
Comment 20 Not Zed 2005-08-04 07:04:12 UTC
the last is obviously a server performance issue with a complex fetch request,
if you have the source you could try altering
camel-imap-store.c:connect_to_server to set force_imap4 = TRUE as a default,
which will use a simpler request, although it looks to me like the server is
just slow.

your assessment about uid fetch is incorrect.  of course uid fetch 3:* is slower
than uid fetch 36075:* - it is asking for more information.  but it only asks
for information it hasn't already retrieved.  we definitely only use uids.

the listing thing, is changed entirely in 2.4, as are a few other things; it's
about as good as it's going to get for a while, but should be better than all
other 2.x versions.
Comment 21 André Klapper 2005-11-08 01:09:52 UTC
adding perf keyword
Comment 22 André Klapper 2006-01-09 02:04:54 UTC
retargetting from 2.3 to 2.5
Comment 23 André Klapper 2006-03-22 01:29:44 UTC
removing old 2.5 target milestone and retargetting to future - sorry.
Comment 24 Jeremy Nickurak 2008-03-09 18:28:56 UTC
Sounds like this is #336076, which already has alot of discussion about this behavior. Dupe?
Comment 25 Mikael Nilsson 2008-03-09 19:01:21 UTC
Jeremy, I couldn't tell if it's a dupe, as the code has been rewritten a few times since my original report. Actually, I think the bug I saw has actually been fixed (though evo is still a lot slower than thunderbird). Feel free to mark as dupe or close.
Comment 26 Jeremy Nickurak 2010-03-16 18:47:58 UTC

*** This bug has been marked as a duplicate of bug 336076 ***