GNOME Bugzilla – Bug 680211
[IMAP] Memory usage increases on each folder select
Last modified: 2012-08-01 07:00:30 UTC
I've been seeing evolution munching up to about 3.6G memory quite frequently, if left running for half a day or so. As far as I can tell there is no definite trigger for it. My setup consists of 3 gmail imap accounts plus one hotmail POP account, the inboxes/trees are fairly large and stored locally. I've run evolution with valgrind's massif tool, and exited it when the whole thing was using around 4G of memory. The output file is attached. If I recall correctly, I never saw this issue on version 3.2 so I would assume it to be a 3.4-regression.
Thanks for a bug report. I see the valgrind log didn't make it in, probably because it's too large?
One more question, are the GMail accounts IMAP or IMAP+ type?
Created attachment 219224 [details] valgrind massif output from evolution session, exited at 4G mem usage total Ah, here it is compressed. My three gmail accounts are set to "IMAP" (and not "IMAP+"). 4G is what massif was using at the time of exit, it appears evolution sat at around 1.5G at that time if I read the log correctly.
I just did these two commits [1][2] for 3.4.4+. The first is mostly for imapx code, but part of it is generic for all providers. I looked around your massif log and it seems you do not have installed debuginfo packages for evolution-data-server and evolution, which would give more detailed information about place of the code, though even without that it seems like the most allocated memory comes when you move between folders on your IMAP account, when it reloads all saved messages in there. There is some auto-flush of loaded infos each 5 minutes, but maybe it doesn't apply here, for example if you use any search folders, which hold reference to each subfolder being used in it. Does valgrind memcheck log show any lost memory, if you try? (The best with installed debuginfo.) I'll test here tomorrow too. [1] http://git.gnome.org/browse/evolution-data-server/commit/?h=gnome-3-4&id=d7d9ce00ad [2] http://git.gnome.org/browse/evolution-data-server/commit/?h=gnome-3-4&id=b5c292e13
Created attachment 219307 [details] massif log for evolution with debug symbols, much shorter session
Created attachment 219308 [details] Command line output for first memcheck run, ended up crashing evolution
Created attachment 219309 [details] First memcheck run, ended up crashing evolution, short session
Created attachment 219310 [details] More detailed memcheck $ valgrind --tool=memcheck --leak-check=full --show-reachable=yes evolution
Some observations: It appears that just switching back and forth from viewing folders will increase the memory usage steadily, in my case a folder with ~9k messages (from fedora-devel/testing lists mainly) adds ~8MB of memory usage *each time* it is viewed, the same seems to be true for smaller folders, with a smaller memory chunk. Also doing a send/receive seems to add on 30MB to evolution's memory usage (even if just refreshing repeatedly). This *might* be the total of all my individual folders' memory-add as per above? I have _not_ tested and counted it. So might it simply be that it loads folders to memory regardless if they were loaded previously?
Thanks for the update. The steps are appreciated, it's easier to know where to focus. From the valgrind memcheck log we definitely lead the chapter. Though I'm not seeing the same on my machine for some reason. Do you know whether your distribution patches evolution-data-server, namely the part of imap camel provider? Also, see the top of your valgrind log, there are issues from Python about invalid frees - it can do basically anything if not run under valgrind (probably if you remove libevolution-mdule-python-plugin from /usr/lib/evolution/3.4/modules, then the next start there will be no issues reported from it. Though that's most likely unrelated to this. 18,632,880 bytes in 18,485 blocks are possibly lost in loss record 19,605 of 19,606 at 0x4C270FE: memalign (vg_replace_malloc.c:694) by 0x4C271A7: posix_memalign (vg_replace_malloc.c:835) by 0x6CBCC2E: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3) by 0x6D046B2: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3) by 0x6D04715: g_slice_alloc0 (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3) by 0x8080649: camel_folder_summary_content_info_new (in /usr/lib/libcamel-1.2.so.33.0.0) by 0x8083EDC: content_info_from_db (in /usr/lib/libcamel-1.2.so.33.0.0) by 0x3030F0FB: content_info_from_db (in /usr/lib/evolution-data-server/camel-providers/libcamelimap.so) by 0x808380A: perform_content_info_load_from_db (in /usr/lib/libcamel-1.2.so.33.0.0) by 0x8083D9E: camel_read_mir_callback (in /usr/lib/libcamel-1.2.so.33.0.0) by 0x83BA1CE: sqlite3_exec (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) by 0x806B6A3: cdb_sql_exec (in /usr/lib/libcamel-1.2.so.33.0.0) 50,696,368 bytes in 24,949 blocks are possibly lost in loss record 19,606 of 19,606 at 0x4C270FE: memalign (vg_replace_malloc.c:694) by 0x4C271A7: posix_memalign (vg_replace_malloc.c:835) by 0x6CBCC2E: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3) by 0x6D046B2: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3) by 0x6D04715: g_slice_alloc0 (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3) by 0x808001F: camel_message_info_new (in /usr/lib/libcamel-1.2.so.33.0.0) by 0x8084025: message_info_from_db (in /usr/lib/libcamel-1.2.so.33.0.0) by 0x3030EF9B: message_info_from_db (in /usr/lib/evolution-data-server/camel-providers/libcamelimap.so) by 0x8083D23: camel_read_mir_callback (in /usr/lib/libcamel-1.2.so.33.0.0) by 0x83BA1CE: sqlite3_exec (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6) by 0x806B6A3: cdb_sql_exec (in /usr/lib/libcamel-1.2.so.33.0.0) by 0x806DD82: camel_db_select (in /usr/lib/libcamel-1.2.so.33.0.0) though your gl driver "leaks" a bit too (it's the third most leaking place): 9,453,584 bytes in 1 blocks are possibly lost in loss record 19,604 of 19,606 at 0x4C28BED: malloc (vg_replace_malloc.c:263) by 0x19D4F5F1: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so) by 0x19BD4314: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so) by 0x19BE8EC4: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so) by 0x19BDC7AC: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so) by 0x19C1F56F: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so) by 0x19C1F6EC: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so) by 0x18C11FCE: ??? (in /usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so.1.2) by 0x18BECC26: ??? (in /usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so.1.2) by 0x18BECEA9: glXCreateNewContext (in /usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so.1.2) by 0xC2159F9: ??? (in /usr/lib/x86_64-linux-gnu/libcogl.so.9.1.1) by 0xC1D3792: cogl_display_setup (in /usr/lib/x86_64-linux-gnu/libcogl.so.9.1.1)
Created attachment 219320 [details] quick start-stop memcheck with removed /usr/lib/evolution/3.4/modules/libevolution-module-plugin-python.so I'm using Debian Testing at the moment, and the only patch which are added to e-data-server is a gettext one, which I'm guessing is unrelated: http://anonscm.debian.org/viewvc/pkg-evolution/unstable/evolution-data-server/debian/patches/20_gettext_intltool.patch?view=markup removing /usr/lib/evolution/3.4/modules/libevolution-module-plugin-python.so made no difference to the memory behaviour. But gets rid of the python errors.
Aaah, I got it, the trick is to use real trash/junk folders for the IMAP account. With them set I see the similar leaks like you (see below). With virtual junk/trash (not an option for GMail), the folders are never freed, as they are added to virtual junk/trash, thus it doesn't leak like here. I even got similar (but not the same) runtime warnings on console. 4,004,506 (1,785,000 direct, 2,219,506 indirect) bytes in 10,625 blocks are definitely lost in loss record 24,407 of 24,407 at 0x4A0884D: malloc (vg_replace_malloc.c:263) by 0x3B1644D2BE: g_malloc (in /usr/lib64/libglib-2.0.so.0.3200.3) by 0x3B164616B1: g_slice_alloc (in /usr/lib64/libglib-2.0.so.0.3200.3) by 0x3B16461C05: g_slice_alloc0 (in /usr/lib64/libglib-2.0.so.0.3200.3) by 0x7380370: camel_message_info_new (camel-folder-summary.c:4338) by 0x7377802: message_info_from_db (camel-folder-summary.c:536) by 0x2AE7935D: message_info_from_db (camel-imap-summary.c:223) by 0x737C0D0: camel_read_mir_callback (camel-folder-summary.c:2381) by 0x7B3A52E: sqlite3_exec (sqlite3.c:91949) by 0x73617A9: cdb_sql_exec (camel-db.c:408) by 0x7362AE0: camel_db_select (camel-db.c:988) by 0x7364A6A: camel_db_read_message_info_records (camel-db.c:1958) by 0x737B4A9: cfs_reload_from_db (camel-folder-summary.c:2142) by 0x737B677: camel_folder_summary_prepare_fetch_all (camel-folder-summary.c:2201) by 0x2AE61300: imap_rescan (camel-imap-folder.c:1222) by 0x2AE6005B: camel_imap_folder_selected (camel-imap-folder.c:730) by 0x2AE60723: imap_refresh_info_sync (camel-imap-folder.c:923) by 0x738B7AA: camel_folder_refresh_info_sync (camel-folder.c:3927) by 0x123D50C1: refresh_folders_exec (mail-send-recv.c:1044) by 0x6096628: mail_msg_proxy (mail-mt.c:423) by 0x3B1646AB41: ??? (in /usr/lib64/libglib-2.0.so.0.3200.3) by 0x3B1646A344: ??? (in /usr/lib64/libglib-2.0.so.0.3200.3) by 0x3DDD007D13: start_thread (in /usr/lib64/libpthread-2.15.so) by 0x3DDCCF199C: clone (in /usr/lib64/libc-2.15.so)
I tested applying the two commits you linked above to evolution-data-server but the memory-adding behaviour stayed the same (both for send/recieve and folder-switching).
Yup, that's true. I'm currently finishing fix for this, which I'll attach shortly.
Created attachment 219334 [details] [review] eds patch for evolution-data-server; With set real trash and junk folder each folder which is newly opened is also properly freed, with its summary as well, but the message infos the summary had in loaded_infos were not freed at all, thus they made the leaks. The patch is also fixing critical warnings, it was from cfs_try_release_memory() for me, due to live summary, but gone folder (summary can live longer than folder, though folder basically owns summary). I also fixed other leaks I saw in my valgrind log.
Created commit a86de47 in eds master (3.5.5+) Created commit 6cfd76b in eds gnome-3-4 (3.4.4+)
I also made a little change in evolution itself [1], but that's not that serious as the leaks from eds. [1] http://git.gnome.org/browse/evolution/commit/?h=gnome-3-4&id=bb1817a
I've tested both your patches latest patches on 3.4.3, they do not fix the issue with (folder-viewing & send/receive) increasing memory for me, maybe those memleaks are unrelated to this?
I've tested both your latest patches on evolution & e-d-s 3.4.3, they do not fix the issue with folder-viewing & send/receive increasing memory for me, maybe those patched memleaks are unrelated to this?
They did fix it for me. Could you get a valgrind log for a session, please? Please use this command: $ G_SLICE=always-malloc valgrind --leak-check=full --num-callers=50 \ evolution &>log.txt Make sure you've installed debuginfo packages (or you compile with debug symbols on), thus the backtrace is more useful. Thanks in advance.