After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 537415 - evolution-exchange and gnome-keyring-daemon cause 100% CPU usage
evolution-exchange and gnome-keyring-daemon cause 100% CPU usage
Status: RESOLVED FIXED
Product: evolution-data-server
Classification: Platform
Component: Contacts
2.24.x (obsolete)
Other All
: Normal normal
: ---
Assigned To: Matthew Barnes
Ximian Connector QA
evolution[passwords]
: 539220 540268 (view as bug list)
Depends on:
Blocks:
 
 
Reported: 2008-06-09 14:44 UTC by Mike Vitale
Modified: 2013-09-14 16:52 UTC
See Also:
GNOME target: ---
GNOME version: 2.21/2.22


Attachments
Proposed patch (1.73 KB, patch)
2008-06-26 18:53 UTC, Matthew Barnes
committed Details | Review

Description Mike Vitale 2008-06-09 14:44:00 UTC
Please describe the problem:
When replying/forwarding/creating a new email with an Exchange account enabled (whether the email is "from" the Exchange account or not), and the Exchange Global Address List option is enabled, CPU usage spikes to 100%.

Steps to reproduce:
1. Create/Enable an Exchange mail account
2. Enable the Global Address List option
3. Click the "Create new Mail" button.


Actual results:
CPU usage jumps to 100%.

Expected results:
CPU usage not to spike.

Does this happen every time?
Yes

Other information:
Reference https://bugs.launchpad.net/ubuntu/+source/evolution-exchange/+bug/236171
Comment 1 Srinivasa Ragavan 2008-06-09 17:47:33 UTC
Mike, do you have GAL enabled for caching? Can you rlick on GAL and properties and enable for offline use and restart evo/eds/exchange?
Comment 2 Mike Vitale 2008-06-09 20:45:38 UTC
I'm afraid that I don't know where you're asking me to look to be able to right-click on GAL.  I assume that GAL is Global Address List.  The only place I can see anything about that GAL is in the Edit | Preferences | Autocomplete area, and that's a checkbox.  Nothing to right-click upon.  See http://www.mikevitale.com/images/EvoOptions.png.  If you mean somewhere/something else for GAL, please provide steps to get to what you're asking about.

Thanks.
Comment 3 thikrat 2008-06-09 21:30:04 UTC
I can confirm this bug numerous times, althought I must admit, I could not make the correlation between composing a message and having this happen, it definately does happen ALL the time.
Comment 4 Srinivasa Ragavan 2008-06-10 03:07:40 UTC
Go to addressbook. rclick on GAL book and go to properties :)
Comment 5 Mike Vitale 2008-06-10 13:16:29 UTC
I have performed these steps:

View | Window | Contacts
(Note: at this point, CPU usage spikes to 100%)
Right-clicked "Global Address List" from my Exchange email box and selected Properties
Selected the "Copy book content locally for offline operation"
Closed Evolution and run "evolution --force-shutdown" from a terminal window
Restarted Evolution

After running through these steps, whenever I either enter the Contacts window or start writing a new email, CPU usage still spikes.
Comment 6 Mike Vitale 2008-06-10 13:36:22 UTC
Follow-up:
Anytime I am "Working Online" after performing the above steps (IE, the "Copy content locally..." checkbox is checked), my CPU usage spikes.  If I'm offline, CPU is fine.  If I'm online, and just reading already-downloaded emails (whether on my local IMAP server, or from my work's Exchange server), my CPU is pegged.
Comment 7 Srinivasa Ragavan 2008-06-10 16:15:29 UTC
Mike, when you restart after marking for offine, start the contacts view and click gal and wait for the window to show "Search for Contact" in the top pane.

It means you have cached, it should be fast after that.

Can you retry and tell me ?
Comment 8 Mike Vitale 2008-06-10 16:39:29 UTC
I went into the GAL in Contacts View, clicked "Copy book content locally...", closed Evo, ran evolution --force-shutdown, restarted Evo, and went back to the GAL in the Contacts View.  I then took my dog for a walk to give it time to complete the operation.  I was gone at least 10 minutes.  When I returned, my CPU usage was still at 100%.  Top shows:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                            
17647 mike      20   0  141m  62m  19m S 19.6  3.1   0:56.99 evolution                                          
17658 mike      20   0 65632  14m 9820 S 19.3  0.7   0:46.50 evolution-excha                                    
 5742 mike      20   0 10588 7584 1916 R 17.6  0.4   6:42.98 gconfd-2                                           

as the top 3 processes.

Screenshot of what Evo looked like after my return is at http://www.mikevitale.com/images/Evo-GAL.png .

-Mike
Comment 9 Srinivasa Ragavan 2008-06-11 03:55:59 UTC
Looks like it is caching. Let it complete and first time, it takes more time. You need to wait till it shows "Search for Contacts" in the top pane.
Comment 10 Mike Vitale 2008-06-11 19:26:18 UTC
There's a whopping 50 people in my entire company.  It takes more than 10 minutes to download 50 names and addresses?  Come on.

Have you even bothered to look at the stack trace I posted on the bug report I originally started on Ubuntu's Launchpad?
Comment 11 Srinivasa Ragavan 2008-06-12 03:10:54 UTC
Ok. You got just 50 people at GAL. It shouldn't be slow.

BTW the trace didn't have any symbols and of no use to me. Mike do you build fomr source? In which case, I can debug it out on a hacking session with you.
Comment 12 Mike Vitale 2008-06-12 03:38:04 UTC
Build from source?  Er, no, sorry.  I'm on Ubuntu, and for the second stacktrace included in the bug, I apt-get installed the evolution-dbgsym package, and followed instructions from Sebastien Bacher to connect GDB to my evolution process to get debugging symbols.
Comment 13 Srinivasa Ragavan 2008-06-12 03:47:01 UTC
Mike, got it.

Can you run evolution-exchange-storage process on a separate console with E2K_DEBUG=4 ?

That can tell, what is going on at the back. Seems like GAL wasn't there at all in the trace.
Comment 14 Srinivasa Ragavan 2008-06-12 03:48:41 UTC
Start exchange storate and then start evolution
Comment 15 Mike Vitale 2008-06-12 16:20:05 UTC
Interesting output from starting:
/usr/lib/evolution/2.22/evolution-exchange-storage --gdk-debug=E2K_DEBUG=4
is:

impl_GNOME_Evolution_Addressbook_BookFactory_getBook
 + gal://mvitale;auth=NTLM@mail.mycompany.com/gal
 => 0x80f5660
impl_GNOME_Evolution_Addressbook_Book_open (0x80f5660)
offlin==============
impl_GNOME_Evolution_Addressbook_BookFactory_getBook
 + exchange://mvitale;auth=NTLM@mail.mycompany.com/;personal/Contacts
 => 0x80f5690
impl_GNOME_Evolution_Addressbook_Book_open (0x80f5690)
authenticate_user(0x80f6538, 0x80f5660, mvitale, mypassword, plain/password)
  
  
  
  
  
  
  
  
authenticate_user(0x80f6538, 0x80f5660, mvitale, mypassword, plain/password)
authenticate_user(0x80f6538, 0x80f5660, mvitale, mypassword, plain/password)
authenticate_user(0x80f6538, 0x80f5660, mvitale, mypassword, plain/password)
authenticate_user(0x80f6538, 0x80f5660, mvitale, mypassword, plain/password)


(And then the "authenticate_user()" line repeats ad infinitum until I ^C the process.)

-Mike
Comment 16 Matthew Barnes 2008-06-19 17:44:36 UTC
I know I've seen this busy loop before but I'm having trouble reproducing it on demand.

Mike, would you mind trying a debugging procedure so we can take a peek at what your evolution-exchange-storage process is doing when it starts arguing with the keyring daemon?

   1) Make sure you have debugging symbols installed for evolution-data-server
      and evolution-exchange.  I think the Ubuntu package names are something
      like evolution-data-server-dbg and evolution-exchange-dbg, right?

   2) Run Evolution and trigger the problem as you've described here.

   3) Open a terminal and run this command to obtain the process ID (pid) of
      the evolution-exchange-storage process:

          ps ax | grep evolution-exchange-storage

      It's the first number in the result (ignore the line that says 'grep',
      if you see one).

   4) Connect to the running evolution-exchange-storage process like so:

          gdb --pid <pid-of-evolution-exchange-storage>

      You'll see a whole bunch of messages fly by and eventually get a
      (gdb) prompt.  The rest of the instructions are GDB commands.

   5) (gdb) break ep_keyring_lookup_passwords

   6) (gdb) continue

      The program should continue running and then hopefully stop again
      almost immediately at the ep_keyring_lookup_passwords function.

   7) (gdb) backtrace

      Post the results of this command here, but first try to read through
      the gibberish a bit and look for words like "exchange" or "gal".  That
      should tell you that you're peeking at the right spot.  If you don't
      see those words, try repeating steps 6 and 7 a few more times.
Comment 17 Mike Vitale 2008-06-20 15:21:53 UTC
Matthew,

I have attempted the steps you mentioned, with the following information/results/whatever described here.

1) I installed the "evolution-exchange-dbg evolution-dbg evolution-data-server-dbg" packages.  For complete reference:

[root@oftheother ~]$ dpkg -l | grep evolution
ii  evolution                                  2.22.2-0ubuntu1.3                                  groupware suite with mail client and organiz
ii  evolution-common                           2.22.2-0ubuntu1.3                                  architecture independent files for Evolution
ii  evolution-data-server                      2.22.2-0ubuntu2                                    evolution database backend server
ii  evolution-data-server-common               2.22.2-0ubuntu2                                    architecture independent files for Evolution
ii  evolution-data-server-dbg                  2.22.2-0ubuntu2                                    evolution database backend server with debug
ii  evolution-dbg                              2.22.2-0ubuntu1.3                                  debugging symbols for Evolution
ii  evolution-exchange                         2.22.2-0ubuntu1                                    Exchange plugin for the Evolution groupware 
ii  evolution-exchange-dbg                     2.22.2-0ubuntu1                                    Exchange plugin for Evolution with debugging
ii  evolution-webcal                           2.21.92-0ubuntu1                                   webcal: URL handler for GNOME and Evolution
ii  libebook1.2-9                              2.22.2-0ubuntu2                                    Client library for evolution address books
ii  libecal1.2-7                               2.22.2-0ubuntu2                                    Client library for evolution calendars
ii  libedata-book1.2-2                         2.22.2-0ubuntu2                                    Backend library for evolution address books
ii  libedata-cal1.2-6                          2.22.2-0ubuntu2                                    Backend library for evolution calendars
ii  libedataserver1.2-9                        2.22.2-0ubuntu2                                    Utility library for evolution data servers
ii  libedataserverui1.2-8                      2.22.2-0ubuntu2                                    GUI utility library for evolution data serve
ii  libexchange-storage1.2-3                   2.22.2-0ubuntu2                                    Backend library for evolution calendars
ii  openoffice.org-evolution                   1:2.4.1-1ubuntu1                                   Evolution Addressbook support for OpenOffice

2) I ran "/usr/lib/evolution/2.22/evolution-exchange-storage --gdk-debug=E2K_DEBUG=4" in one terminal window, and "evolution" in another.

3) I clicked on the "New Email" button, which triggers 100% CPU usage.
3a) In my terminal window for evolution-exchange-storage with debug, the following line was repeated ad infinitum:

authenticate_user(0x813e880, 0x80f5890, mvitale, mypassword, plain/password)

4) I attached GDB to the pid of my evolution-exchange-storage process.
4a) CPU usage dropped to normal levels at this point.
4b) "authenticate_user()" messages stopped in the terminal.

5) I set the breakpoint at ep_keyring_lookup_passwords.

6) I "continue"d in GDB.
6a) CPU usage spiked back to 100% at this point.
6b) The program did not "hopefully stop again almost immediately" -- not even a couple minutes later.
6c) "authenticate_user()" messages resumed in the terminal.

7) Since the program didn't stop/fail/reach the breakpoint, there is no backtrace to provide.

For reference:
(gdb) break ep_keyring_lookup_passwords
Breakpoint 1 at 0xb79f62b6: file e-passwords.c, line 390.

What's next?

-Mike
Comment 18 Matthew Barnes 2008-06-20 16:56:08 UTC
Thanks for trying.  I'm trying to isolate where the busy loop is occurring in evolution-exchange-storage, and apparently ep_keyring_lookup_passwords() is not part of that loop as I had thought.  By the way, the behavior you described in steps 4 - 6 can be explained by the fact that connecting to a process with GDB temporarily pauses it until you tell it to continue.  That's why the 100% CPU usage and the "authenticate_user()" messages stopped.

Still trying to reproduce this for myself without any luck.  I believe it to be some kind of keyring configuration issue, whereby gnome-keyring-daemon is failing to process requests from Evolution and Evolution is not handling those failures correctly.  But I need to see the busy loop in action to confirm that.

Bug #539220 has a stacktrace similar to what I was expecting to see here, showing both authenticate_user() and ep_keyring_lookup_passwords() in Thread 1.
Comment 19 Mike Vitale 2008-06-20 17:03:42 UTC
How about this:

[mike@oftheother ~]$ gdb --pid 4477
[... snip ...]
(gdb) break authenticate_user
Breakpoint 1 at 0x8067ec4: file e-book-backend-gal.c, line 2228.
(gdb) continue
Continuing.

Thread 3056831376 (LWP 4566)

  • #0 authenticate_user
    at e-book-backend-gal.c line 2228
  • #1 e_book_backend_authenticate_user
    at e-book-backend.c line 379
  • #2 impl_GNOME_Evolution_Addressbook_Book_authenticateUser
    at e-data-book.c line 91
  • #3 _ORBIT_skel_small_GNOME_Evolution_Addressbook_Book_authenticateUser
    at Evolution-DataServer-Addressbook-common.c line 60
  • #4 ??
    from /usr/lib/libORBit-2.so.0
  • #5 ORBit_OAObject_invoke
    from /usr/lib/libORBit-2.so.0
  • #6 ORBit_small_invoke_adaptor
    from /usr/lib/libORBit-2.so.0
  • #7 ??
    from /usr/lib/libORBit-2.so.0
  • #8 ??
    from /usr/lib/libORBit-2.so.0
  • #9 giop_thread_queue_process
    from /usr/lib/libORBit-2.so.0
  • #10 ??
    from /usr/lib/libORBit-2.so.0
  • #11 ??
    from /usr/lib/libglib-2.0.so.0
  • #12 ??
    from /usr/lib/libglib-2.0.so.0
  • #13 start_thread
    from /lib/tls/i686/cmov/libpthread.so.0
  • #14 clone
    from /lib/tls/i686/cmov/libc.so.6

Comment 20 Mike Vitale 2008-06-20 17:13:54 UTC
Or, perhaps, a better backtrace after installing the following packages:
libglib2.0-0-dbgsym
liborbit2-dbgsym

(gdb) break authenticate_user
Breakpoint 1 at 0x8067ec4: file e-book-backend-gal.c, line 2228.
(gdb) continue
Continuing.

Thread 3048594320 (LWP 22405)

  • #0 authenticate_user
    at e-book-backend-gal.c line 2228
  • #1 e_book_backend_authenticate_user
    at e-book-backend.c line 379
  • #2 impl_GNOME_Evolution_Addressbook_Book_authenticateUser
    at e-data-book.c line 91
  • #3 _ORBIT_skel_small_GNOME_Evolution_Addressbook_Book_authenticateUser
    at Evolution-DataServer-Addressbook-common.c line 60
  • #4 ORBit_POAObject_invoke
    at poa.c line 1142
  • #5 ORBit_OAObject_invoke
    at orbit-adaptor.c line 338
  • #6 ORBit_small_invoke_adaptor
    at orbit-small.c line 844
  • #7 ORBit_POAObject_handle_request
    at poa.c line 1351
  • #8 ORBit_POAObject_invoke_incoming_request
    at poa.c line 1421
  • #9 giop_thread_queue_process
    at giop.c line 771
  • #10 giop_request_handler_thread
    at giop.c line 481
  • #11 g_thread_pool_thread_proxy
    at /build/buildd/glib2.0-2.16.3/glib/gthreadpool.c line 265
  • #12 g_thread_create_proxy
    at /build/buildd/glib2.0-2.16.3/glib/gthread.c line 635
  • #13 start_thread
    from /lib/tls/i686/cmov/libpthread.so.0
  • #14 clone
    from /lib/tls/i686/cmov/libc.so.6

Comment 21 Matthew Barnes 2008-06-20 21:21:07 UTC
Thanks Mike, that helps a little.  What I really need to know is what's happening once it reaches authenticate_user(), and what path is it taking to cause it to keep coming back to that function.

Not sure what other breakpoints to suggest trying just yet.  I'll look into that a bit and get back to you.

Also, and apologies if you've already mentioned this elsewhere, but exactly what version of gnome-keyring are you using?
Comment 22 Mike Vitale 2008-06-20 21:42:07 UTC
My current version is:

[root@oftheother ~]$ dpkg -l | grep gnome-keyring
ii  gnome-keyring                              2.22.2-0ubuntu1                                    GNOME keyring services (daemon and tools)
ii  libgnome-keyring0                          2.22.2-0ubuntu1                                    GNOME keyring services library

I originally noticed this problem with the 2.22.1-1ubuntu1 version, which is what I got when upgrading to Ubuntu Hardy.  If you haven't seen yet, I originally reported this bug in Launchpad at https://bugs.launchpad.net/ubuntu/+source/evolution-exchange/+bug/236171.  It's migrated here, since here is "upstream".

Any further assistance I can be, let me know.

-Mike
Comment 23 Mike Vitale 2008-06-24 02:36:19 UTC
Please see the original bug report on Launchpad for potentially new information pertaining to this issue.
Comment 24 Matthew Barnes 2008-06-25 10:56:33 UTC
Awesome.  That's the magic combination:

  - Enable autocompletion on the Exchange account.
  - Do not specify a Global Catalog Server.

As soon as I opened a new composer window, the evolution and keyring-daemon processes took off.  *Now* I can get to the bottom of this.  Thanks Mike!
Comment 25 Matthew Barnes 2008-06-26 17:29:54 UTC
I traced the problem to load_source_auth_cb() in e-book-auth-util.c, which gets called after we attempt to authenticate with the server.  In this case, with no Global Catalog server name given, we're not even getting that far.  The error I get back is E_BOOK_ERROR_REPOSITORY_OFFLINE.

I think we're handling error codes improperly here, but I'd like someone more familiar with EBook to verify this.  The logic goes something like this:

   if error_code != OK:
       if error_code == CANCELLED:
           /* try anonymous connection, if supported */
       else if error_code == INVALID_SERVER_VERSION:
           /* ignore the error and proceed */
       else:
           /* re-authenticate */

Now, looking at the list of possible error codes, I only see a few where re-authenticating seems like the right move.  It certainly should not be a catch-all as it is presently.

To me it looks like E_BOOK_ERROR_AUTHENTICATION_FAILED and E_BOOK_ERROR_AUTHENTICATION_REQUIRED are the only codes that should trigger a re-authentication.  Certainly not E_BOOK_ERROR_REPOSITORY_OFFLINE, anyway.  But then what's the correct action for the rest of the error codes, if not re-authentication?  This is where I need advice.
Comment 26 Matthew Barnes 2008-06-26 18:53:14 UTC
Created attachment 113481 [details] [review]
Proposed patch

Severs the busy loop by only re-authenticating if we get an AUTHENTICATION_FAILED or AUTHENTICATION_REQUIRED error code.  Otherwise we just pass the error code on to the provided EBookCallback and let it deal with it, which I /think/ is the right thing to do.
Comment 27 Srinivasa Ragavan 2008-06-30 08:40:01 UTC
Looks Ok to me. Ross do you have any suggestions ? I haven't test it thought, looks good as I see the code. If a tester can confirm it improves, we can take it.
Comment 28 Srinivasa Ragavan 2008-06-30 09:30:03 UTC
s/thought/though
Comment 29 Matthew Barnes 2008-06-30 18:24:48 UTC
*** Bug 539220 has been marked as a duplicate of this bug. ***
Comment 30 Matthew Barnes 2008-07-07 23:51:23 UTC
*** Bug 540268 has been marked as a duplicate of this bug. ***
Comment 31 Tom Astleitner 2008-07-08 09:08:28 UTC
I've had pretty serious problems with the described situation, as you can read in Bug 540268, Evo became absolutely useless. Now I've applied the patch and everything looks perfect again.

I'm using latest SVN which is Evo 2.35.5.

Thanks for that!
Comment 32 Matthew Barnes 2008-07-08 14:52:23 UTC
Based on comment #31 and my own testing, the patch works and there seems to be no obvious regressions.  Committing this to trunk for 2.23.5 (revision 9098).