After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 680211 - [IMAP] Memory usage increases on each folder select
[IMAP] Memory usage increases on each folder select
Status: RESOLVED FIXED
Product: evolution
Classification: Applications
Component: Mailer
3.4.x (obsolete)
Other Linux
: Normal normal
: ---
Assigned To: evolution-mail-maintainers
Evolution QA team
evolution[imap]
Depends on:
Blocks:
 
 
Reported: 2012-07-18 21:51 UTC by Martin Erik Werner
Modified: 2012-08-01 07:00 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
valgrind massif output from evolution session, exited at 4G mem usage total (77.18 KB, application/x-gzip)
2012-07-19 13:28 UTC, Martin Erik Werner
  Details
massif log for evolution with debug symbols, much shorter session (73.10 KB, application/x-gzip)
2012-07-20 03:36 UTC, Martin Erik Werner
  Details
Command line output for first memcheck run, ended up crashing evolution (1.44 KB, application/octet-stream)
2012-07-20 03:38 UTC, Martin Erik Werner
  Details
First memcheck run, ended up crashing evolution, short session (551.38 KB, application/x-gzip)
2012-07-20 03:39 UTC, Martin Erik Werner
  Details
More detailed memcheck $ valgrind --tool=memcheck --leak-check=full --show-reachable=yes evolution (963.05 KB, application/x-gzip)
2012-07-20 04:18 UTC, Martin Erik Werner
  Details
quick start-stop memcheck with removed /usr/lib/evolution/3.4/modules/libevolution-module-plugin-python.so (952.80 KB, application/x-gzip)
2012-07-20 11:59 UTC, Martin Erik Werner
  Details
eds patch (7.66 KB, patch)
2012-07-20 16:00 UTC, Milan Crha
committed Details | Review

Description Martin Erik Werner 2012-07-18 21:51:08 UTC
I've been seeing evolution munching up to about 3.6G memory quite frequently, if left running for half a day or so. As far as I can tell there is no definite trigger for it.

My setup consists of 3 gmail imap accounts plus one hotmail POP account, the inboxes/trees are fairly large and stored locally.

I've run evolution with valgrind's massif tool, and exited it when the whole thing was using around 4G of memory. The output file is attached.

If I recall correctly, I never saw this issue on version 3.2 so I would assume it to be a 3.4-regression.
Comment 1 Milan Crha 2012-07-19 09:51:09 UTC
Thanks for a bug report. I see the valgrind log didn't make it in, probably because it's too large?
Comment 2 Milan Crha 2012-07-19 11:00:29 UTC
One more question, are the GMail accounts IMAP or IMAP+ type?
Comment 3 Martin Erik Werner 2012-07-19 13:28:14 UTC
Created attachment 219224 [details]
valgrind massif output from evolution session, exited at 4G mem usage total

Ah, here it is compressed.

My three gmail accounts are set to "IMAP" (and not "IMAP+").

4G is what massif was using at the time of exit, it appears evolution sat at around 1.5G at that time if I read the log correctly.
Comment 4 Milan Crha 2012-07-19 20:41:12 UTC
I just did these two commits [1][2] for 3.4.4+. The first is mostly for imapx code, but part of it is generic for all providers.

I looked around your massif log and it seems you do not have installed debuginfo packages for evolution-data-server and evolution, which would give more detailed information about place of the code, though even without that it seems like the most allocated memory comes when you move between folders on your IMAP account, when it reloads all saved messages in there. There is some auto-flush of loaded infos each 5 minutes, but maybe it doesn't apply here, for example if you use any search folders, which hold reference to each subfolder being used in it.

Does valgrind memcheck log show any lost memory, if you try? (The best with installed debuginfo.) I'll test here tomorrow too.

[1] http://git.gnome.org/browse/evolution-data-server/commit/?h=gnome-3-4&id=d7d9ce00ad
[2] http://git.gnome.org/browse/evolution-data-server/commit/?h=gnome-3-4&id=b5c292e13
Comment 5 Martin Erik Werner 2012-07-20 03:36:59 UTC
Created attachment 219307 [details]
massif log for evolution with debug symbols, much shorter session
Comment 6 Martin Erik Werner 2012-07-20 03:38:07 UTC
Created attachment 219308 [details]
Command line output for first memcheck run, ended up crashing evolution
Comment 7 Martin Erik Werner 2012-07-20 03:39:02 UTC
Created attachment 219309 [details]
First memcheck run, ended up crashing evolution, short session
Comment 8 Martin Erik Werner 2012-07-20 04:18:33 UTC
Created attachment 219310 [details]
More detailed memcheck $ valgrind --tool=memcheck --leak-check=full --show-reachable=yes evolution
Comment 9 Martin Erik Werner 2012-07-20 04:40:49 UTC
Some observations:

It appears that just switching back and forth from viewing folders will increase the memory usage steadily, in my case a folder with ~9k messages (from fedora-devel/testing lists mainly) adds ~8MB of memory usage *each time* it is viewed, the same seems to be true for smaller folders, with a smaller memory chunk.

Also doing a send/receive seems to add on 30MB to evolution's memory usage (even if just refreshing repeatedly). This *might* be the total of all my individual folders' memory-add as per above? I have _not_ tested and counted it.

So might it simply be that it loads folders to memory regardless if they were loaded previously?
Comment 10 Milan Crha 2012-07-20 07:01:48 UTC
Thanks for the update. The steps are appreciated, it's easier to know where to focus. From the valgrind memcheck log we definitely lead the chapter. Though I'm not seeing the same on my machine for some reason. Do you know whether your distribution patches evolution-data-server, namely the part of imap camel provider? Also, see the top of your valgrind log, there are issues from Python about invalid frees - it can do basically anything if not run under valgrind (probably if you remove libevolution-mdule-python-plugin from /usr/lib/evolution/3.4/modules, then the next start there will be no issues reported from it. Though that's most likely unrelated to this.

 18,632,880 bytes in 18,485 blocks are possibly lost in loss record 19,605 of 19,606
    at 0x4C270FE: memalign (vg_replace_malloc.c:694)
    by 0x4C271A7: posix_memalign (vg_replace_malloc.c:835)
    by 0x6CBCC2E: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3)
    by 0x6D046B2: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3)
    by 0x6D04715: g_slice_alloc0 (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3)
    by 0x8080649: camel_folder_summary_content_info_new (in /usr/lib/libcamel-1.2.so.33.0.0)
    by 0x8083EDC: content_info_from_db (in /usr/lib/libcamel-1.2.so.33.0.0)
    by 0x3030F0FB: content_info_from_db (in /usr/lib/evolution-data-server/camel-providers/libcamelimap.so)
    by 0x808380A: perform_content_info_load_from_db (in /usr/lib/libcamel-1.2.so.33.0.0)
    by 0x8083D9E: camel_read_mir_callback (in /usr/lib/libcamel-1.2.so.33.0.0)
    by 0x83BA1CE: sqlite3_exec (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
    by 0x806B6A3: cdb_sql_exec (in /usr/lib/libcamel-1.2.so.33.0.0)
 
 50,696,368 bytes in 24,949 blocks are possibly lost in loss record 19,606 of 19,606
    at 0x4C270FE: memalign (vg_replace_malloc.c:694)
    by 0x4C271A7: posix_memalign (vg_replace_malloc.c:835)
    by 0x6CBCC2E: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3)
    by 0x6D046B2: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3)
    by 0x6D04715: g_slice_alloc0 (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3)
    by 0x808001F: camel_message_info_new (in /usr/lib/libcamel-1.2.so.33.0.0)
    by 0x8084025: message_info_from_db (in /usr/lib/libcamel-1.2.so.33.0.0)
    by 0x3030EF9B: message_info_from_db (in /usr/lib/evolution-data-server/camel-providers/libcamelimap.so)
    by 0x8083D23: camel_read_mir_callback (in /usr/lib/libcamel-1.2.so.33.0.0)
    by 0x83BA1CE: sqlite3_exec (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
    by 0x806B6A3: cdb_sql_exec (in /usr/lib/libcamel-1.2.so.33.0.0)
    by 0x806DD82: camel_db_select (in /usr/lib/libcamel-1.2.so.33.0.0)

though your gl driver "leaks" a bit too (it's the third most leaking place):
 9,453,584 bytes in 1 blocks are possibly lost in loss record 19,604 of 19,606
    at 0x4C28BED: malloc (vg_replace_malloc.c:263)
    by 0x19D4F5F1: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
    by 0x19BD4314: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
    by 0x19BE8EC4: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
    by 0x19BDC7AC: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
    by 0x19C1F56F: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
    by 0x19C1F6EC: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
    by 0x18C11FCE: ??? (in /usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so.1.2)
    by 0x18BECC26: ??? (in /usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so.1.2)
    by 0x18BECEA9: glXCreateNewContext (in /usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so.1.2)
    by 0xC2159F9: ??? (in /usr/lib/x86_64-linux-gnu/libcogl.so.9.1.1)
    by 0xC1D3792: cogl_display_setup (in /usr/lib/x86_64-linux-gnu/libcogl.so.9.1.1)
Comment 11 Martin Erik Werner 2012-07-20 11:59:07 UTC
Created attachment 219320 [details]
quick start-stop memcheck with removed /usr/lib/evolution/3.4/modules/libevolution-module-plugin-python.so

I'm using Debian Testing at the moment, and the only patch which are added to e-data-server is a gettext one, which I'm guessing is unrelated:
http://anonscm.debian.org/viewvc/pkg-evolution/unstable/evolution-data-server/debian/patches/20_gettext_intltool.patch?view=markup

removing /usr/lib/evolution/3.4/modules/libevolution-module-plugin-python.so made no difference to the memory behaviour. But gets rid of the python errors.
Comment 12 Milan Crha 2012-07-20 12:38:47 UTC
Aaah, I got it, the trick is to use real trash/junk folders for the IMAP account. With them set I see the similar leaks like you (see below). With virtual junk/trash (not an option for GMail), the folders are never freed, as they are added to virtual junk/trash, thus it doesn't leak like here. I even got similar (but not the same) runtime warnings on console.

 4,004,506 (1,785,000 direct, 2,219,506 indirect) bytes in 10,625 blocks are definitely lost in loss record 24,407 of 24,407
    at 0x4A0884D: malloc (vg_replace_malloc.c:263)
    by 0x3B1644D2BE: g_malloc (in /usr/lib64/libglib-2.0.so.0.3200.3)
    by 0x3B164616B1: g_slice_alloc (in /usr/lib64/libglib-2.0.so.0.3200.3)
    by 0x3B16461C05: g_slice_alloc0 (in /usr/lib64/libglib-2.0.so.0.3200.3)
    by 0x7380370: camel_message_info_new (camel-folder-summary.c:4338)
    by 0x7377802: message_info_from_db (camel-folder-summary.c:536)
    by 0x2AE7935D: message_info_from_db (camel-imap-summary.c:223)
    by 0x737C0D0: camel_read_mir_callback (camel-folder-summary.c:2381)
    by 0x7B3A52E: sqlite3_exec (sqlite3.c:91949)
    by 0x73617A9: cdb_sql_exec (camel-db.c:408)
    by 0x7362AE0: camel_db_select (camel-db.c:988)
    by 0x7364A6A: camel_db_read_message_info_records (camel-db.c:1958)
    by 0x737B4A9: cfs_reload_from_db (camel-folder-summary.c:2142)
    by 0x737B677: camel_folder_summary_prepare_fetch_all (camel-folder-summary.c:2201)
    by 0x2AE61300: imap_rescan (camel-imap-folder.c:1222)
    by 0x2AE6005B: camel_imap_folder_selected (camel-imap-folder.c:730)
    by 0x2AE60723: imap_refresh_info_sync (camel-imap-folder.c:923)
    by 0x738B7AA: camel_folder_refresh_info_sync (camel-folder.c:3927)
    by 0x123D50C1: refresh_folders_exec (mail-send-recv.c:1044)
    by 0x6096628: mail_msg_proxy (mail-mt.c:423)
    by 0x3B1646AB41: ??? (in /usr/lib64/libglib-2.0.so.0.3200.3)
    by 0x3B1646A344: ??? (in /usr/lib64/libglib-2.0.so.0.3200.3)
    by 0x3DDD007D13: start_thread (in /usr/lib64/libpthread-2.15.so)
    by 0x3DDCCF199C: clone (in /usr/lib64/libc-2.15.so)
Comment 13 Martin Erik Werner 2012-07-20 13:41:18 UTC
I tested applying the two commits you linked above to evolution-data-server but the memory-adding behaviour stayed the same (both for send/recieve and folder-switching).
Comment 14 Milan Crha 2012-07-20 15:39:05 UTC
Yup, that's true. I'm currently finishing fix for this, which I'll attach shortly.
Comment 15 Milan Crha 2012-07-20 16:00:12 UTC
Created attachment 219334 [details] [review]
eds patch

for evolution-data-server;

With set real trash and junk folder each folder which is newly opened is also properly freed, with its summary as well, but the message infos the summary had in loaded_infos were not freed at all, thus they made the leaks. The patch is also fixing critical warnings, it was from cfs_try_release_memory() for me, due to live summary, but gone folder (summary can live longer than folder, though folder basically owns summary). I also fixed other leaks I saw in my valgrind log.
Comment 16 Milan Crha 2012-07-20 16:10:09 UTC
Created commit a86de47 in eds master (3.5.5+)
Created commit 6cfd76b in eds gnome-3-4 (3.4.4+)
Comment 17 Milan Crha 2012-07-20 16:16:29 UTC
I also made a little change in evolution itself [1], but that's not that serious as the leaks from eds.

[1] http://git.gnome.org/browse/evolution/commit/?h=gnome-3-4&id=bb1817a
Comment 18 Martin Erik Werner 2012-07-20 23:17:35 UTC
I've tested both your patches latest patches on 3.4.3, they do not fix the issue with (folder-viewing & send/receive) increasing memory for me, maybe those memleaks are unrelated to this?
Comment 19 Martin Erik Werner 2012-07-20 23:21:47 UTC
I've tested both your latest patches on evolution & e-d-s 3.4.3, they do not fix the issue with folder-viewing & send/receive increasing memory for me, maybe those patched memleaks are unrelated to this?
Comment 20 Milan Crha 2012-07-23 08:47:56 UTC
They did fix it for me. Could you get a valgrind log for a session, please? 

Please use this command:
   $ G_SLICE=always-malloc valgrind --leak-check=full --num-callers=50 \
     evolution &>log.txt
Make sure you've installed debuginfo packages (or you compile with debug symbols on), thus the backtrace is more useful. Thanks in advance.