GNOME Bugzilla – Bug 313846
gam_server leaks fantastically large amounts of memory, sometimes
Last modified: 2006-06-03 15:22:28 UTC
Opened at http://bugzilla.ubuntu.com/show_bug.cgi?id=13449 [breezy] gamin has leaked 1.5gigs in the past 3 days I have 2 gigs of ram and I noticed that for some reason 700megs of my swap space was used. This confused me so I started looking for answers. desrt 6333 0.0 44.6 1486744 927136 ? S Aug10 0:14 /usr/lib/gamin/gam_server Default config, so I'm guessing inotify (valgrind complains about unknown syscall 291, which is an inotify syscall). Version: 0.1.5-0ubuntu1. I've been valgrinding it all night (since I filed the bug) but valgrind hasn't produced any output. I'm going to see if Kjartan can give me some hints. A contributing factor to this bug might be bug 313845 (but Kjartan says that it only leaks a little bit -- not 1.5GB).
Hey, I have seen this once. Over a night 400 mb's of ram was leaked. Valgrind wasn't fruitful for me. It seems to only happen occasionally. Any info you can get would really help.
BTW, Are you using inotify?
Okay, I more carefully read the report. If gamin is complaining about the unknown syscall, you need to upgrade. Valgrind 3.0 includes support for the inotify system calls.
Have you been able to repeat this? I have tried for the last few days with no luck.
Davyd has reproduced this a couple of times, as has someone I ran into on the #ubuntu help channel. It seems that it only happens to people running breezy, but keep in mind a couple of things: 1) I'm told that Ubuntu doesn't vendor patch gamin 2) Breezy is the only distribution exposing large numbers of people to newer versions of gamin.
I need _any_ information about this leak. It has only happened once to me, and I've been running gamin under valgrind for 3 days. If anyone can get a valgrind log when this is happening, this will get fixed.
Created attachment 51699 [details] Valgrind log Attaching valgrind log with both reachable and lost data. ==31808== LEAK SUMMARY: ==31808== definitely lost: 17600 bytes in 550 blocks. ==31808== indirectly lost: 9557 bytes in 552 blocks. ==31808== possibly lost: 0 bytes in 0 blocks. ==31808== still reachable: 365678 bytes in 531 blocks. ==31808== suppressed: 36 bytes in 1 blocks.
I've just noticed that this is probably useless as there are ??? in fundamental places... I'll build a debug gamin shortly.
Created attachment 51700 [details] [review] Valgrind log with symbols This log has debugging symbols in. The leak isn't as large as the previous log but by comparing both it should be possible to find the problem. I'm a little worried about the large amount of data which is still reachable somehow...
==2943== 262144 bytes in 1 blocks are still reachable in loss record 140 of 140 ==2943== at 0x1B900896: malloc (vg_replace_malloc.c:149) ==2943== by 0x1B93DE8F: g_malloc (gmem.c:137) ==2943== by 0x805264C: gam_inotify_read_handler (gam_inotify.c:1013) ==2943== by 0x1B95E29B: ??? (giounix.c:162) ==2943== by 0x1B93748D: g_main_context_dispatch (gmain.c:1934) ==2943== by 0x1B93A495: ??? (gmain.c:2565) ==2943== by 0x1B93A782: g_main_loop_run (gmain.c:2769) ==2943== by 0x804B8A7: main (gam_server.c:424) the problem is that I can't find a malloc in gam_inotify_read_handler() and it's now far away from line 1013 (around line 700 in CVS). There is a g_malloc'ed buffer around line 1050 in CVS, it's a static local variable (and hence can't be cleaned up, bad style !) but it's allocated once. the problem is that gam_server doesn't deallocate runtime variable on exit but I don't see a clear leak which could grow to hundred of megabytes. We need to have cleanup on exit which will then allow us to track what may be runtime leaks. Still I don't see a clear potential candidate for the large amount of memory leak you pointed out. Daniel
Yes, I agree with daniel, I can't see any code that could be leaking as much as the bug report says. I have gone over the inotify code very carefully, and I have paired all allocations with frees. I have also spent days with gam_server running under valgrind, the only leaks I can find are from gam_server not cleaning up the subscription level. It wouldn't be hard to add cleanup callbacks to each gam_inotify.c, and have them called. Moving the static local variable to a static global one would allow it to be free'd when gam_server is exiting.
I guess the problem was there. Subscriptions blocks were never freed when running under inotify. With dnotify and poll this was freed at the monitored tree level which the changes from John disconnected from. ==2943== 13470 (9184 direct, 4286 indirect) bytes in 287 blocks are definitely lost in loss record 138 of 140 ==2943== at 0x1B901B78: calloc (vg_replace_malloc.c:279) ==2943== by 0x1B93DEFD: g_malloc0 (gmem.c:154) ==2943== by 0x804A921: gam_subscription_new (gam_subscription.c:74) ==2943== by 0x8050106: gam_connection_data (gam_connection.c:358) ==2943== by 0x804F0BE: gam_client_conn_read (gam_channel.c:283) ==2943== by 0x1B95E29B: ??? (giounix.c:162) ==2943== by 0x1B93748D: g_main_context_dispatch (gmain.c:1934) ==2943== by 0x1B93A495: ??? (gmain.c:2565) ==2943== by 0x1B93A782: g_main_loop_run (gmain.c:2769) ==2943== by 0x804B8A7: main (gam_server.c:424) I hope this is fixed in CVS, Daniel
Created attachment 53625 [details] Bug is _not_ resolved. I know this bug is marked "Resolved", but it isn't. Behold, a screenshot [attached] of Gnome's System Monitor running on Ubuntu 5.10 with gamin using 939Mb of memory. Would a log file be useful, and if so, where would I find it?
Ubuntu 5.10 shipped without this fix. Please take this up in ubuntu's bugzilla. AFAIK, they decided not to use it because 1) gamin can (?) crash with this patch applied. 2) this leak is extremely rare.
Thanks for clarifying that for me. I'll post on their bugzilla, then.
I'm sorry to say this, but this error is apparently still present in gamin-0.1.7... I'm running Gentoo, so the fix should be implemented (?)... Today, gam_server was using more than 550MB's of swap, and over 150 MB's RAM.Considering the total of 512 MB's of RAM on my system, that is quite a large chunk... I'll start running debug, and see if I come up with something
I haven't been able to debug gam_server, but I have some more information about the problem... It seems to be random how fast the leak builds up. I have had to kill gam_server several times over the last two weeks, but the buildup has differed from 50-60 MB's over the course of a week, to the last kill where it was at 30 MB's in just four-five hours. Looking at the console where I started the x-session, an error message keeps repeating itself several times per second: End from FAM server connection invalid length 24902 invalid length 24902 invalid length 24902 I don't know if this is related to the memory leak, but...