After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 586498 - leaks hundreds of megabytes of memory when re-scanning library
leaks hundreds of megabytes of memory when re-scanning library
Status: RESOLVED FIXED
Product: banshee
Classification: Other
Component: Importing
git master
Other Linux
: Normal normal
: 1.x
Assigned To: Banshee Maintainers
Banshee Maintainers
Depends on:
Blocks:
 
 
Reported: 2009-06-20 22:29 UTC by Mike Rooney
Modified: 2009-06-22 01:07 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Dispose the SqliteCommand after it's executed (3.83 KB, patch)
2009-06-21 06:22 UTC, Alexander Kojevnikov
none Details | Review
Now with a regression test (5.19 KB, patch)
2009-06-21 08:29 UTC, Alexander Kojevnikov
accepted-commit_now Details | Review

Description Mike Rooney 2009-06-20 22:29:23 UTC
Hello! I'm using Banshee 1.5 from git and notice that when I re-scan my library, Banshee memory grows by hundreds of megabytes, in a linear fashion as it scans, until the scan is done. My current library is about 8,000 songs and 50GB. It starts at around 40-60MB in usage, and after the re-scan ends at around 530MB for me of Resident memory as reported by top and gnome-system-monitor.

It doesn't seem like it should use this much memory ever at all, and not releasing it afterwards is problematic as well. After having it running for 30-60 minutes the memory has not been released.

This is very repeatable; I can scan the library and Banshee will always use large amounts of memory and never give it back until closed.

I've filed this against the Importing component for lack of a better guess. Also, if this is repeatable by others it should probably be considered for 1.6 and as a potential Karmic blocker.

Let me know if any other debugging information is needed from me!
Comment 1 Alexander Kojevnikov 2009-06-20 23:47:52 UTC
Thanks for the bug report. This particular bug has already been reported into our bug tracking system, but please feel free to report any further bugs you find.


*** This bug has been marked as a duplicate of 548633 ***
Comment 2 Mike Rooney 2009-06-20 23:58:09 UTC
Thanks Alexander, although I'm not sure if there is enough information to determine that this is a duplicate. I was aware of the metadata leak bug, but was told in #banshee that it was a different issue.

Assuming that a rescan does a metadata refresh, we don't have any way to know without comparing a metadata refresh to a rescan on the same library, if the metadata leak accounts for ALL the leakage of a rescan, some, or only a tiny portion. Unless you have some way to know, I'd say we should un-dupe this for now.

Is there a way for me to trigger a metadata refresh? If so I can compare them a couple of times and give some results and hopefully determine if the leakage is all in the metadata refresh or if the rescanning is leaking more.
Comment 3 Alexander Kojevnikov 2009-06-21 00:06:16 UTC
(In reply to comment #2)
> Thanks Alexander, although I'm not sure if there is enough information to
> determine that this is a duplicate. I was aware of the metadata leak bug, but
> was told in #banshee that it was a different issue.
> 
> Assuming that a rescan does a metadata refresh, we don't have any way to know
> without comparing a metadata refresh to a rescan on the same library, if the
> metadata leak accounts for ALL the leakage of a rescan, some, or only a tiny
> portion. Unless you have some way to know, I'd say we should un-dupe this for
> now.
> 

Indeed, sorry for confusion. I will re-open the bug.

> Is there a way for me to trigger a metadata refresh? If so I can compare them a
> couple of times and give some results and hopefully determine if the leakage is
> all in the metadata refresh or if the rescanning is leaking more.
> 

You can temporarily bump CURRENT_METADATA_VERSION in BansheeDbFormatMigrator.cs, or reduce the value of 'MetadataVersion' key in CoreConfiguration table using sqlite3.
Comment 4 Mike Rooney 2009-06-21 00:10:01 UTC
Thanks Alexander, sounds easy enough. I'll give it a shot and report back the results which should hopefully tell us if it is a dupe or not! I guess I hope it is, as it means one problem to solve instead of two.
Comment 5 Mike Rooney 2009-06-21 01:25:33 UTC
Okay I've done some testing thanks to alexkay's assistance. I did three metadata refreshes (by decrementing the MetadataVersion value) and three rescans, restarting Banshee each time. The results were very consistent.

They do appear to be separate issues; the rescan leaks 50% more than just a metadata. For all these results I refer to what gnome-system-monitor reports for both Resident and Writable memory as they are identical in all cases. Banshee starts out consistently at 40MB of RAM, and after a metadata refresh goes to 350M, for a difference of 310M or 40Kb per track (8000 total).

A rescan takes it from 40M to 500M, which is a 460M increase, or an extra 150M leak compared to just a metadata refresh, or 20Kb more per track (60kb - 40kb for metadata refresh).

I'll add the metadata specific information to that report, and confirm it. In conclusion, a metadata refresh leaks 40Kb per track and a rescan leaks an additional 20kb. Anyone have any clues as to what is being leaked here?

Comment 6 Alexander Kojevnikov 2009-06-21 06:22:48 UTC
Created attachment 137100 [details] [review]
Dispose the SqliteCommand after it's executed

With this patch, re-scanning takes 10x less memory on a small library of 130 songs (0.8-2.0 MiB per re-scan vs 10-15 MiB without the patch). It also reduces memory consumption during metadata refresh.

Mike, could you try it out on your database to see how it compares to the numbers you reported?
Comment 7 Mike Rooney 2009-06-21 07:23:09 UTC
This patch is a great improvement if not complete fix for rescanning! On a rescan the memory quickly jumps on 10-12MB but this may just may extra resources being allocated for the rescanning resources. It doesn't seem to grow at all over the actual rescan process and sequential rescans show that the memory doesn't increase any more after the first time like it did before the patch.

It might not hurt to look and see if something is stored/cached at the beginning of rescan time that could disposed of though, to perhaps eliminate the 12MB or so, as rescanning seems fairly common.
Comment 8 Alexander Kojevnikov 2009-06-21 07:25:27 UTC
Thanks Mike, confirming.
Comment 9 Alexander Kojevnikov 2009-06-21 08:29:26 UTC
Created attachment 137105 [details] [review]
Now with a regression test
Comment 10 Bertrand Lorentz 2009-06-21 19:56:51 UTC
Good work tracking this down !
Alex, please commit.
Comment 11 Alexander Kojevnikov 2009-06-22 01:07:26 UTC
Committed, thanks!