GNOME Bugzilla – Bug 586498
leaks hundreds of megabytes of memory when re-scanning library
Last modified: 2009-06-22 01:07:26 UTC
Hello! I'm using Banshee 1.5 from git and notice that when I re-scan my library, Banshee memory grows by hundreds of megabytes, in a linear fashion as it scans, until the scan is done. My current library is about 8,000 songs and 50GB. It starts at around 40-60MB in usage, and after the re-scan ends at around 530MB for me of Resident memory as reported by top and gnome-system-monitor. It doesn't seem like it should use this much memory ever at all, and not releasing it afterwards is problematic as well. After having it running for 30-60 minutes the memory has not been released. This is very repeatable; I can scan the library and Banshee will always use large amounts of memory and never give it back until closed. I've filed this against the Importing component for lack of a better guess. Also, if this is repeatable by others it should probably be considered for 1.6 and as a potential Karmic blocker. Let me know if any other debugging information is needed from me!
Thanks for the bug report. This particular bug has already been reported into our bug tracking system, but please feel free to report any further bugs you find. *** This bug has been marked as a duplicate of 548633 ***
Thanks Alexander, although I'm not sure if there is enough information to determine that this is a duplicate. I was aware of the metadata leak bug, but was told in #banshee that it was a different issue. Assuming that a rescan does a metadata refresh, we don't have any way to know without comparing a metadata refresh to a rescan on the same library, if the metadata leak accounts for ALL the leakage of a rescan, some, or only a tiny portion. Unless you have some way to know, I'd say we should un-dupe this for now. Is there a way for me to trigger a metadata refresh? If so I can compare them a couple of times and give some results and hopefully determine if the leakage is all in the metadata refresh or if the rescanning is leaking more.
(In reply to comment #2) > Thanks Alexander, although I'm not sure if there is enough information to > determine that this is a duplicate. I was aware of the metadata leak bug, but > was told in #banshee that it was a different issue. > > Assuming that a rescan does a metadata refresh, we don't have any way to know > without comparing a metadata refresh to a rescan on the same library, if the > metadata leak accounts for ALL the leakage of a rescan, some, or only a tiny > portion. Unless you have some way to know, I'd say we should un-dupe this for > now. > Indeed, sorry for confusion. I will re-open the bug. > Is there a way for me to trigger a metadata refresh? If so I can compare them a > couple of times and give some results and hopefully determine if the leakage is > all in the metadata refresh or if the rescanning is leaking more. > You can temporarily bump CURRENT_METADATA_VERSION in BansheeDbFormatMigrator.cs, or reduce the value of 'MetadataVersion' key in CoreConfiguration table using sqlite3.
Thanks Alexander, sounds easy enough. I'll give it a shot and report back the results which should hopefully tell us if it is a dupe or not! I guess I hope it is, as it means one problem to solve instead of two.
Okay I've done some testing thanks to alexkay's assistance. I did three metadata refreshes (by decrementing the MetadataVersion value) and three rescans, restarting Banshee each time. The results were very consistent. They do appear to be separate issues; the rescan leaks 50% more than just a metadata. For all these results I refer to what gnome-system-monitor reports for both Resident and Writable memory as they are identical in all cases. Banshee starts out consistently at 40MB of RAM, and after a metadata refresh goes to 350M, for a difference of 310M or 40Kb per track (8000 total). A rescan takes it from 40M to 500M, which is a 460M increase, or an extra 150M leak compared to just a metadata refresh, or 20Kb more per track (60kb - 40kb for metadata refresh). I'll add the metadata specific information to that report, and confirm it. In conclusion, a metadata refresh leaks 40Kb per track and a rescan leaks an additional 20kb. Anyone have any clues as to what is being leaked here?
Created attachment 137100 [details] [review] Dispose the SqliteCommand after it's executed With this patch, re-scanning takes 10x less memory on a small library of 130 songs (0.8-2.0 MiB per re-scan vs 10-15 MiB without the patch). It also reduces memory consumption during metadata refresh. Mike, could you try it out on your database to see how it compares to the numbers you reported?
This patch is a great improvement if not complete fix for rescanning! On a rescan the memory quickly jumps on 10-12MB but this may just may extra resources being allocated for the rescanning resources. It doesn't seem to grow at all over the actual rescan process and sequential rescans show that the memory doesn't increase any more after the first time like it did before the patch. It might not hurt to look and see if something is stored/cached at the beginning of rescan time that could disposed of though, to perhaps eliminate the 12MB or so, as rescanning seems fairly common.
Thanks Mike, confirming.
Created attachment 137105 [details] [review] Now with a regression test
Good work tracking this down ! Alex, please commit.
Committed, thanks!