GNOME Bugzilla – Bug 644817
Slow addressbook cache
Last modified: 2012-01-09 17:54:13 UTC
Whenever the addressbook factory replicates the GAL for a large organization, the system will have between 1-2 cores completely maxed out, and both evolution and the addressbook factory will use circa 0.5 GB _resident_ memory. The cores will remain maxed out for approximately 10 minutes, during which evolution is either semi-functional or non-functional (depending on whether it's blocking on waiting to hear back from the factory). After the 10 minutes, evolution returns to normal though the memory is never released. While I don't have any solid evidence, I think much of this may be the result of using xml to store/query the "cached" results. On this client, my current addressbook xml file is ~40MB, for example. Any reason xml was selected here, as opposed to bdb or maybe sqlite?
Basically because structures provided by evolution-data-server are using XML. Please note of [1] too. When eds will provide better caches, then we can start to use it. I'm moving this to evolution-data-server for now. [1] http://mail.gnome.org/archives/evolution-hackers/2011-March/msg00030.html
It would be good if you could get a backtrace of the running factory (with evolution-data-server and evolution-mapi debug info packages installed), to see where it is so slow. Thanks in advance.
The main thread always seems to be waiting on a poll whenever i interrupt it, but there's usually a therad that looks something like this:
+ Trace 226336
Created attachment 204786 [details] gdb backtrace of e-addressbook-factory during issue This is a gdb "thread apply all bt" of the e-addressbook-factory process, taken while experiencing the high system load. I notice it because there is nearly constant disk write activity and everything else slows to a crawl. Our GAL has over 84,000 entries.
Thanks for the update. I was just about to tell that newer evolution-mapi uses SQLite for its cache, but I see from your backtrace that you are actually using it, and this is stuck in SQLite. I'll see whether there is anything which can avoid this slowness.
Created attachment 204893 [details] [review] eds patch for evolution-data-server; Adding new API e_book_backend_sqlitedb_lock_updates(), e_book_backend_sqlitedb_unlock_updates(), which basically begins/ends transaction. Also changed inner transaction locking mechanism, to avoid deadlocks and nested transactions on the SQLite DB.
Created attachment 204894 [details] [review] ema patch for evolution-mapi; Using the new API, to avoid too often commits to SQLite database. There are still committed updates as they go, only not more often than a minute between each commit.
Created commit 6cba5c1 in eds master (3.3.4+) Created commit 9e7bbb6 in ema master (3.3.4+)