GNOME Bugzilla – Bug 335178
Beagle using ~1 gig of memory
Last modified: 2006-04-26 17:53:16 UTC
Please describe the problem: I installed beagle 0.2.3 and updated to cvs which differed with one update for Gaim 2.0 logs by Lukas Lipka. I ran that for a few days and it was using normal amounts of memory. I'm not sure if this is related, could just be a coincidence, but when I started "camorama" which is a local webcam application, firefox crashed, and then my system started getting extremely slow. After about a minute camorama started and then I checked out ps aux. Turned out to be slow due to beagle's memory usage skyrocketed. zeo@compaq:~/Downloads/progs/gnomecvs$ ps auxw | grep beagle zeo 661 2.8 50.2 658548 258968 ? Sl Mar18 47:20 beagled --debug /usr/lib/beagle/BeagleDaemon.exe --replace --bg zeo 4889 0.8 2.3 56300 12084 ? Sl 07:57 6:49 beagled-helper --debug /usr/lib/beagle/IndexHelper.exe Steps to reproduce: Actual results: Expected results: Does this happen every time? Other information:
Created attachment 61583 [details] process file usage
Can you attach the logs from ~/.beagle/Log ?
Bugzilla wouldn't let me attach it becuase it was too big so I Uploaded the archive to here: http://z.narutochaos.com/BeagleLogs.tar.gz
This has also happened to me although I did not have to run beagle for several days. I had run with BEAGLE_EXERCISE_THE_DOG=1 to build the index then shutdown beagle and started normally. After crawling the directories was complete my system was idle. I came back to it to check memory usage and it was at 695,000K.
You can find my logs at http://www.patdouble.com/Beagle-highmemusage.log.tbz2
Same problem here, although i don't think i have to run any special application. The change in memory consumption definitely occured between 0.2.3 et 0.2.2 for me.
Same problem here, with beagled 0.2.3 Started beagled without special arguments. After some time (about 40 minutes), evolution wouldn't start anymore because system was out of memory.
*** Bug 335560 has been marked as a duplicate of this bug. ***
(In reply to comment #8) > *** Bug 335560 has been marked as a duplicate of this bug. *** > This bug has an attachment by the poster. Might be helpful in debugging.
I found this entry in jow shaw's blog http://joeshaw.org/2006/03/17/386 It seems to be the same error reported here....
According to the posting: "So if you update to the latest Mono in SVN, this issue is fixed. But since most people aren’t running it yet, I also committed a workaround to Beagle in CVS. It’s included in the 0.2.3 release I announced earlier today." Everyone here is running beagle 0.2.3, so either the bug didn't get fixed completely or it's a new issue.
The blog entry is for something a little different I think. It was this work that was used to fix a memory leak in 0.2.2.1 and included in 0.2.3. Actually the problem is in mono, which has been fixed but not released IIRC, but a workaround also included in beagle 0.2.3. The problem we're seeing here is a very fast jump in memory usage, I've seen it go to 1.7GB before I killed it. Its happened only a few minutes after starting beagle, or sometimes not at all during the length of time I was logged in (several hours).
The most common issue has been fixed in CVS, so you guys must be seeing something different. If you can run beagled with --debug-memory, it will log some additional memory information to disk. It would be helpful if you could attach that info to the bug. It would also be helpful if you can narrow it down to one particular backend which causes this. You can do that by listing the backends with "beagled --list-backends" and then starting beagled with --allow-backend and/or --deny-backend. Early candidates to try are "files" and "evolutionmail". A lot of times this depends on the data you have on your computers, and isn't easily reproduceable here, so I need your help with this.
I am doing it right now. I am using EXCERCISE_THE_DOG to acelerate it
Created attachment 61809 [details] logs with --memory-debug (alejandro vera) I did as you asked. beagled --debug-memory --allow-backend files All was being indexed and beagle-index-info showed that were ~2000 files indexed.. Then i opend a gedit and typed some text in it (without saving it). then i saw a "stack trace" in the screen and a lot of garbage in current-Beagle. I look at beagle-index-info and only 87 files were indexed... do you think it is something related or is other bug??? the stacktrace was in the error output so it is not in the log. I paste it here ================================================================= Got a SIGSEGV while executing native code. This usually indicates a fatal error in the mono runtime or one of the native libraries used by your application. ================================================================= Stacktrace: in (wrapper managed-to-native) Beagle.Util.ExifEntry:exif_entry_get_value (System.Runtime.InteropServices.HandleRef,byte[],int) <0x4> in (wrapper managed-to-native) Beagle.Util.ExifEntry:exif_entry_get_value (System.Runtime.InteropServices.HandleRef,byte[],int) <0xfffffe6b> in Beagle.Util.ExifEntry:get_Value () (at /home/jose/devel/pkg-mono/build-area/beagle-0.2.3/Util/ExifData.cs:708) in Beagle.Util.ExifData:LookupFirstValue (Beagle.Util.ExifTag) (at /home/jose/devel/pkg-mono/build-area/beagle-0.2.3/Util/ExifData.cs:857) in Beagle.Filters.FilterJpeg:PullImageProperties () (at /home/jose/devel/pkg-mono/build-area/beagle-0.2.3/Filters/FilterJpeg.cs:118) in Beagle.Filters.FilterImage:DoPullProperties () (at /home/jose/devel/pkg-mono/build-area/beagle-0.2.3/Filters/FilterImage.cs:72) in Beagle.Daemon.Filter:Open (System.IO.FileSystemInfo) (at /home/jose/devel/pkg-mono/build-area/beagle-0.2.3/beagled/Filter.cs:496) in Beagle.Daemon.Filter:Open (string) (at /home/jose/devel/pkg-mono/build-area/beagle-0.2.3/beagled/Filter.cs:527) in Beagle.Daemon.FilterFactory:FilterIndexable (Beagle.Indexable,Beagle.Daemon.TextCache,Beagle.Daemon.Filter&) (at /home/jose/devel/pkg-mono/build-area/beagle-0.2.3/beagled/FilterFactory.cs:278) in Beagle.Daemon.LuceneIndexingDriver:Flush_Unlocked (Beagle.Daemon.IndexerRequest) (at /home/jose/devel/pkg-mono/build-area/beagle-0.2.3/beagled/LuceneIndexingDriver.cs:259) in Beagle.Daemon.LuceneIndexingDriver:Flush (Beagle.Daemon.IndexerRequest) (at /home/jose/devel/pkg-mono/build-area/beagle-0.2.3/beagled/LuceneIndexingDriver.cs:90) in Beagle.IndexHelper.RemoteIndexerExecutor:Execute (Beagle.RequestMessage) (at /home/jose/devel/pkg-mono/build-area/beagle-0.2.3/beagled/IndexHelper/RemoteIndexerExecutor.cs:69) in Beagle.Daemon.ConnectionHandler:HandleConnection () (at /home/jose/devel/pkg-mono/build-area/beagle-0.2.3/beagled/Server.cs:275) in (wrapper delegate-invoke) System.MulticastDelegate:invoke_void () <0x139> in Beagle.Util.ExceptionHandlingThread:ThreadStarted () (at /home/jose/devel/pkg-mono/build-area/beagle-0.2.3/Util/ExceptionHandlingThread.cs:54) in (wrapper delegate-invoke) System.MulticastDelegate:invoke_void () <0xffffff90> in (wrapper runtime-invoke) System.Object:runtime_invoke_void (object,intptr,intptr,intptr) <0x1aed99f> Native stacktrace: /usr/lib/libmono.so.0(mono_handle_native_sigsegv+0xeb) [0xa7e506ab] /usr/lib/libmono.so.0 [0xa7e1158d] [0xffffe440] /lib/tls/libc.so.6(__dcgettext+0x3d) [0xa7bd753d] /usr/lib/libexif.so.12(exif_entry_get_value+0x2478) [0xa574e0d8] [0xa58274e1] [0xa58272d4] [0xa5826732] [0xa582581d] [0xa58250a9] [0xa5824b21] [0xa582490b] [0xa581cc04] [0xa58eedb0] [0xa58edd5b] [0xa58fc3af] [0xa5d81a94] [0xa63410c8] [0xa63411d2] [0xa63410c8] [0xa6341029] /usr/lib/libmono.so.0 [0xa7e2e9a0] /usr/lib/libmono.so.0(mono_runtime_invoke+0x33) [0xa7e935e3] /usr/lib/libmono.so.0(mono_runtime_delegate_invoke+0x46) [0xa7e948d6] /usr/lib/libmono.so.0 [0xa7ec7366] /usr/lib/libmono.so.0 [0xa7f14832] /usr/lib/libmono.so.0(GC_start_routine+0x63) [0xa7f303b3] /lib/tls/libpthread.so.0 [0xa7d20ced] /lib/tls/libc.so.6(__clone+0x5e) [0xa7c85d7e] 060322 1949499255 11575 Beagle DEBUG: Caught ResponseMessageException: Document element did not appear. Line 1, position 1. 060322 1949499257 11575 Beagle DEBUG: Exception was unexpected. Details: Beagle.ResponseMessageException: Document element did not appear. Line 1, position 1. ---> System.Xml.XmlException: Document element did not appear. Line 1, position 1. 060322 1949499257 11575 Beagle DEBUG: in <0x00193> System.Xml.XmlTextReader:Read () 060322 1949499257 11575 Beagle DEBUG: in <0x00060> System.Xml.XmlReader:MoveToContent () 060322 1949499257 11575 Beagle DEBUG: in <0x00019> System.Xml.Serialization.XmlSerializationReaderInterpreter:ReadRoot () 060322 1949499257 11575 Beagle DEBUG: in <0x0005d> System.Xml.Serialization.XmlSerializer:Deserialize (System.Xml.Serialization.XmlSerializationReader reader) 060322 1949499257 11575 Beagle DEBUG: in <0x00040> System.Xml.Serialization.XmlSerializer:Deserialize (System.Xml.XmlReader xmlReader) 060322 1949499257 11575 Beagle DEBUG: in <0x00037> System.Xml.Serialization.XmlSerializer:Deserialize (System.IO.Stream stream) 060322 1949499257 11575 Beagle DEBUG: in [0x000cf] (at /home/jose/devel/pkg-mono/build-area/beagle-0.2.3/BeagleClient/Client.cs:365) Beagle.Client:Send (Beagle.RequestMessage request)--- End of inner exception stack trace --- 060322 1949499257 11575 Beagle DEBUG: 060322 1949499257 11575 Beagle DEBUG: in [0x00112] Beagle.Client:Send (Beagle.RequestMessage request) 060322 1949499257 11575 Beagle DEBUG: in [0x0000e] (at /home/jose/devel/pkg-mono/build-area/beagle-0.2.3/BeagleClient/Message.cs:172) Beagle.RequestMessage:Send () 060322 1949499257 11575 Beagle DEBUG: in [0x00037] (at /home/jose/devel/pkg-mono/build-area/beagle-0.2.3/beagled/RemoteIndexer.cs:146) Beagle.Daemon.RemoteIndexer:SendRequest (Beagle.Daemon.RemoteIndexerRequest request)
Created attachment 61811 [details] the same test but without text edition I re-ran the test.. this time i didnt do anything except leave beagle to do his work same result: the index went for 4800 -> 48 files here is the log just after "Done crawling!!!"
alejandro: In the attachments, you gzipped the links and not the files :-). tar-gz the files the links point to and attach them.
I have probles to upload archives in my email... anyway this is a corrupted jpeg that causes baeagle to crash http://gamercafe.cl/image.tgz I saw in the logs that that image was the last entry in IndexHelper and then, a new IndexHelper file was created.. I created another account, put that image in the home direcotry and run beagle.. it crashed a second later if i try to open it in eog, it crashes to.. i can open it with GIMP it is not a personal file, so you can see it :D .. is a couple of friends and her sister... i hope it is helpfull Sorry for the targeziped links :D
At last, I reproduced the error Once i remove the bad jpeg, i deleted .beagled and started again. all my files were idexed: 25005 :D Then i started beagled --debug-memory (all the backends) and waited Then, some minutes later i used the deskbar-applet to start a program (last.fm player), but just in that moment my box stoped and started to crawl! Deskbar has a live-beagle plugin and i saw my query "last" in the beagle log... I managed to open a virtual console and stop beagle All the logs are in http://gamercafe.cl/beagle-logs-all.tgz (700k ~) I think the important part is this 060322 2112286837 18476 Beagle DEBUG: Memory usage: VmSize=161.0 MB, VmRSS=100.4 MB, GC.GetTotalMemory=28856320 060322 2112336849 18476 Beagle DEBUG: Memory usage: VmSize=161.0 MB, VmRSS=100.5 MB, GC.GetTotalMemory=38776832 060322 2112387091 18476 Beagle DEBUG: Memory usage: VmSize=161.0 MB, VmRSS=100.5 MB, GC.GetTotalMemory=36884480 060322 2112407162 18476 Beagle DEBUG: Parsed query 'las' as text_query 060322 2112415619 18476 Beagle DEBUG: Parsed query 'last' as text_query 060322 2112437475 18476 Beagle DEBUG: Memory usage: VmSize=773.6 MB, VmRSS=116.4 MB, GC.GetTotalMemory=569651200 060322 2112437477 18476 Beagle DEBUG: VmSize too large --- shutting down 060322 2112437502 18476 Beagle DEBUG: CancelIfBlocking Beagle.Daemon.ConnectionHandler
Created attachment 61848 [details] beagle logs with memory bloat Here are logs from a beagle run with --debug-memory. I did not limit the backend in this one. It finished crawling and them some time afterwards ate the memory.
Pat: these lines are very suspicious: 060323 0733560804 19989 Beagle DEBUG: Memory usage: VmSize=145.6 MB, VmRSS=62.5 MB, GC.GetTotalMemory=34615296 060323 0733568325 19989 Beagle DEBUG: *** Add '/home/double/Mail/inbox/cur' '1143055374.17006.QfgNy:2,S' (file) 060323 0733570352 19989 Beagle DEBUG: *** Add '/home/double/Mail/trash/tmp' '1143120593.17006.t8LbY:2,S' (file) 060323 0733570685 19989 Beagle DEBUG: *** Move '/home/double/Mail/trash/tmp' '1143120593.17006.t8LbY:2,S' -> '/home/double/Mail/trash/cur' '1143120593.17006.t8LbY:2,S' (file) 060323 0733570965 19989 Beagle DEBUG: checking if /home/double/Mail/inbox is a maildir ? 060323 0733570967 19989 Beagle DEBUG: Removing mail:/home/double/Mail/inbox/cur/1143120593.17006.t8LbY 060323 0733570969 19989 Beagle DEBUG: *** Remove '/home/double/Mail/inbox/cur' '1143120593.17006.t8LbY' (file) 060323 0733571360 19989 Beagle DEBUG: *** Add '/home/double/Mail' '.trash.index.ids' (file) 060323 0733571363 19989 Beagle DEBUG: *** Add '/home/double/Mail' '.trash.index.ids' (file) 060323 0733571366 19989 Beagle DEBUG: *** Add '/home/double/Mail' '.trash.index' (file) 060323 0734011793 19989 Beagle DEBUG: Memory usage: VmSize=670.1 MB, VmRSS=144.0 MB, GC.GetTotalMemory=563539968 Are you using maildir in there, approximately how many files are in subdirectories there?
Yes, mostly maildir. Here's the file counts for each dir under /home/double/Mail: [double@ee-pdouble pts/2] for DIR in `find . -maxdepth 1 -type d`; do echo -n ${DIR} " "; find ${DIR} -type f | wc -l; done . 4338 ./Heartland 2 ./Misc 48 ./Palm 0 ./.i2rd.directory 87 ./i2rd 1 ./Suspend2Dev 448 ./Gentoo 0 ./Lists 0 ./.Gentoo.directory 655 ./inbox 289 ./trash 43 ./TechSupport 0 ./People 0 ./.TechSupport.directory 51 ./.People.directory 34 ./.Suspend2Users.directory 0 ./Federalist 844 ./.BibleThumper.directory 149 ./Suspend2Users 526 ./TalentBank 65 ./sent-mail 149 ./.Palm.directory 207 ./.TalentBank.directory 11 ./outbox 0 ./.IFB-Preachers.directory 0 ./nagios@patdouble.com 0 ./.Suspend2Dev.directory 0 ./.Lists.directory 147 ./IFB-Preachers 495 I cleaned about 2600+ entries from my trash earlier today, that may be important. Do you want the output from ls -lR ?
In comment #18 i posted an image that causes beagle to crash.. should i open a new bug for it??
(In reply to comment #22) > Yes, mostly maildir. Here's the file counts for each dir under > /home/double/Mail: Sorry to interrupt. Pat, is /home/double/Mail excluded from the filesystem indexing (using beagle-config or beagle-settings you can control which directories the _filesystem _backend should not index) ?
(In reply to comment #23) > In comment #18 i posted an image that causes beagle to crash.. should i open a > new bug for it?? As far as I can see, libexif crashed on that image. I doubt beagle (read mono) can do anything if external libraries crash, but thats just my guess. Does any other libexif using application (e.g. jhead) handle it properly ?
1) Thanks for the info, i can't read mono stacktrace so well... i'm going to try to file a bug on libexif 2) Today i started beagle a lot of times and every time it crashed some minutes later. Then, some hours ago i turned off the beagle-live plugin of deskbar-applet and started beagle. Until now i haven't experienced any crash. Coul'd be related to python-bindings? I remeber that for me the crashes started a couple of weeks ago, the same when i installed d-a.
> 2) Today i started beagle a lot of times and every time it crashed some minutes > later. Then, some hours ago i turned off the beagle-live plugin of > deskbar-applet and started beagle. Until now i haven't experienced any crash. > Coul'd be related to python-bindings? I remeber that for me the crashes started > a couple of weeks ago, the same when i installed d-a. Deskbar uses beagle API. If some beagle API can crash beagle, thats probably bad too. Does beagle crash using other ways of query like beagle-search or (command line) beagle-query ? If you suspect the crash has to do with the "live" query feature of deskbar, beagle-query has an option to specify live query, you can use that (beagle-search already uses live query).
I tried lots of times the search.exe and beagle-query. I tried creating a text field with my search pattern and it appeard in the results. All ok.. then i started beagle-live in d-a and tried a cuople of query.. the first showed a lot of 060323 2355317584 08719 Beagle DEBUG: Caught an exception sending response; socket shut down: Write failure 060323 2355317591 08719 Beagle DEBUG: (128) Waiting for 8 workers... 060323 2355317592 08719 Beagle DEBUG: waiting for HandleConnection (372) 060323 2355317593 08719 Beagle DEBUG: waiting for HandleConnection (305) 060323 2355317594 08719 Beagle DEBUG: waiting for HandleConnection (306) 060323 2355317595 08719 Beagle DEBUG: waiting for HandleConnection (303) 060323 2355317595 08719 Beagle DEBUG: waiting for HandleConnection (304) 060323 2355317596 08719 Beagle DEBUG: waiting for HandleConnection (334) 060323 2355317597 08719 Beagle DEBUG: waiting for HandleConnection (301) 060323 2355317598 08719 Beagle DEBUG: waiting for HandleConnection (342) and then beagled died with 060323 2355317739 08719 Beagle DEBUG: Caught an exception sending response; socket shut down: Write failure 060323 2355317741 08719 Beagle DEBUG: (134) Waiting for 2 workers... 060323 2355317742 08719 Beagle DEBUG: waiting for HandleConnection (305) 060323 2355317743 08719 Beagle DEBUG: waiting for HandleConnection (306) 060323 2355317791 08719 Beagle DEBUG: Caught an exception sending response; socket shut down: Write failure 060323 2355317802 08719 Beagle DEBUG: Caught an exception sending response; socket shut down: Write failure 060323 2355317805 08719 Beagle DEBUG: Exiting 060323 2355318066 08719 Beagle DEBUG: Leaving BeagleDaemon.Main 060323 2355323085 08719 Beagle DEBUG: Live ExceptionHandlingThread: EHT 08724 (static):Void LogMemoryUsage() Again i have problems to upload... the logs are here http://gamercafe.cl/beagle-log-live.tgz
(In reply to comment #24) > (In reply to comment #22) > > Yes, mostly maildir. Here's the file counts for each dir under > > /home/double/Mail: > > Sorry to interrupt. Pat, is /home/double/Mail excluded from the filesystem > indexing (using beagle-config or beagle-settings you can control which > directories the _filesystem _backend should not index) ? > No, it is not excluded. Would having two backends processing the same files (in this case 'files' and 'kmail') cause problems? Or are the number of files too much for the files backend? I am sure I have other directories with many files as well, but just not changing as fast as the maildir. I will try to exclude my mail from the files backend and see if the memory problem happens again.
(In reply to comment #21) > Pat: these lines are very suspicious: > > 060323 0733560804 19989 Beagle DEBUG: Memory usage: VmSize=145.6 MB, VmRSS=62.5 > MB, GC.GetTotalMemory=34615296 > 060323 0733568325 19989 Beagle DEBUG: *** Add '/home/double/Mail/inbox/cur' > '1143055374.17006.QfgNy:2,S' (file) > 060323 0733570352 19989 Beagle DEBUG: *** Add '/home/double/Mail/trash/tmp' > '1143120593.17006.t8LbY:2,S' (file) > 060323 0733570685 19989 Beagle DEBUG: *** Move '/home/double/Mail/trash/tmp' > '1143120593.17006.t8LbY:2,S' -> '/home/double/Mail/trash/cur' > '1143120593.17006.t8LbY:2,S' (file) > 060323 0733570965 19989 Beagle DEBUG: checking if /home/double/Mail/inbox is a > maildir ? > 060323 0733570967 19989 Beagle DEBUG: Removing > mail:/home/double/Mail/inbox/cur/1143120593.17006.t8LbY > 060323 0733570969 19989 Beagle DEBUG: *** Remove '/home/double/Mail/inbox/cur' > '1143120593.17006.t8LbY' (file) > 060323 0733571360 19989 Beagle DEBUG: *** Add '/home/double/Mail' > '.trash.index.ids' (file) > 060323 0733571363 19989 Beagle DEBUG: *** Add '/home/double/Mail' > '.trash.index.ids' (file) > 060323 0733571366 19989 Beagle DEBUG: *** Add '/home/double/Mail' > '.trash.index' (file) > 060323 0734011793 19989 Beagle DEBUG: Memory usage: VmSize=670.1 MB, > VmRSS=144.0 MB, GC.GetTotalMemory=563539968 > > Are you using maildir in there, approximately how many files are in > subdirectories there? > Crashed again. This time I resumed from suspend (using suspend 2.2.1 http://suspend2.net) and did a mail check. Since I haven't checked my mail for around 12 hours there was lots of mail and the KMail filters sent it to several different mailboxes. In the log I find this, and the last lines are processing my maildir. Sorry, no --debug-memory, I believe the KDE 'kerry' search frontend started beagled itself: 060324 0438345378 14889 Beagle WARN: Caught exception calling DoQuery on 'KMail' 060324 0438346116 14889 Beagle WARN: Caught exception calling DoQuery on 'KMail' 060324 0438346118 14889 Beagle WARN: Caught exception calling DoQuery on 'KMail' 060324 0438346126 14889 Beagle WARN: Caught exception calling DoQuery on 'KMail' 060324 0438346130 14889 Beagle WARN: Caught exception calling DoQuery on 'KMail' 060324 0438346132 14889 Beagle WARN: Caught exception calling DoQuery on 'KMail' 060324 0438346135 14889 Beagle WARN: Caught exception calling DoQuery on 'KMail' 060324 0438346138 14889 Beagle WARN: Caught exception calling DoQuery on 'KMail' 060324 0438346128 14889 Beagle WARN: Caught exception calling DoQuery on 'KMail'
Created attachment 61911 [details] logs from mp3 file scan and concurrent search OK, here's another log just before the memory bloat occurs. I have excluded /home/double/Mail (my mail directories) from the filesystem scan, although I did not rebuild the index and it seems it is still processing inotify updates to the maildirs, but not crawling them. Anyway, what happened is that I loaded a playlist in amarok which caused beagle to get inotify events and process them. During this a search was executed. I did not initiate it, but I am using kerry, and I think it periodically updates your current search, so that probably caused it. The log is a 'tail -f current-*'. You can see normal memory usage, an exception in searching, and then the memory bloat. Interesting beagle says it is going to exit, but I have to kill it before it takes over my system. Hope this helps. Perhaps combining a search with file updates is causing the problem. The previous log I posted shows exceptions searching using the KMail backend while processing inotify events.
Joe, In all the crashers, I notice that some kind of live-notification is invloved. Was there any similar bug fixed in beagle-search recently ? Maybe kerry and desktop-search authors need to fix their live-notification handler. Damn ... the bug started memory problems and now its also infected with fatal crashers. > No, it is not excluded. Would having two backends processing the same files (in > this case 'files' and 'kmail') cause problems? Or are the number of files too Dont know if it can cause any problem, never tried. Just wanted to be safe. I have more mails in kmail than yours and I do suspend computers overnight and in the morning open computer with quite a few mails. There arent any general problems ... probably some specific case, some sneaky corner case.
dbera: The only major crasher in beagle-search that was recently fixed, was a case when we tried generating a thumbnail in gnome thumbnail factory for a hit with an empty mimetype.
Well... I have not experimented any other crash since i turned off the beagle-live in deskbar applet. I have EXCERCISE_THE_DOG=1 and I suspend my laptop all the time. It seems that is the live-query what is crashing beagle....
The memory bloat has nothing to do with deskbar-applet, though queries from it do make the situation worse. Within a couple of minutes of starting beagled v0.23 the memory usage surges to well over 100Mb.
For those of you using deskbar-applet, do you get the same sort of memory explosion if you try running a query with "beagle-query --live-query" ?
(In reply to comment #36) > For those of you using deskbar-applet, do you get the same sort of memory > explosion if you try running a query with "beagle-query --live-query" ? I believe the live-query related problem is causing the crashes, not the memory problem. I was going to suggest opening another bug for crashes under live-query, but since the problem happens with kerry/deskbar-applet and not beagle-search, I am not sure whether this should be investigated by beagle people. IMO, beagle exposes API for public use and if any of it can crash beagle, then beagle has to be safeguarded against that.
Memory bloat again. Here's a partial log from tail -f current-*. Running beagled --debug-memory --bg (all backends). It had been done crawling some time ago, and was pretty much quiet, not many file changes. I decided to convert some mbox files in KMail to maildir. I create a new maildir and then copy the messages from the mbox folder to the maildir in KMail. I copied approx. 600 messages and then the memory bloat. I was running "beagle-query --live-query except", I figured "expect" would be a common word but not too common. kerry was not running (I have no other search GUI installed for beagle). ==> current-Beagle <== 060325 2130444491 20763 Beagle DEBUG: Memory usage: VmSize=140.4 MB, VmRSS=45.5 MB, GC.GetTotalMemory=29237248 ==> current-IndexHelper <== 060325 2130446780 26419 IndexH DEBUG: -file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.1uJOw:2,S 060325 2130446844 26419 IndexH DEBUG: +file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.1uJOw:2,S 060325 2130457524 26419 IndexH DEBUG: -file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.inSnb:2,S 060325 2130457581 26419 IndexH DEBUG: +file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.inSnb:2,S 060325 2130468519 26419 IndexH DEBUG: -file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.Us3cZ:2,S 060325 2130468574 26419 IndexH DEBUG: +file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.Us3cZ:2,S 060325 2130479775 26419 IndexH DEBUG: -file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.b0oVB:2,S 060325 2130479874 26419 IndexH DEBUG: +file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.b0oVB:2,S ==> current-Beagle <== 060325 2130491803 20763 Beagle DEBUG: Not adding task for already running task: /home/double/Mail/Jokes 060325 2130491807 20763 Beagle DEBUG: Not adding task for already running task: /home/double/Mail/Jokes 060325 2130494494 20763 Beagle DEBUG: Memory usage: VmSize=140.4 MB, VmRSS=45.5 MB, GC.GetTotalMemory=34578432 ==> current-IndexHelper <== 060325 2130491411 26419 IndexH DEBUG: -file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.NeakB:2,S 060325 2130491603 26419 IndexH DEBUG: +file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.NeakB:2,S 060325 2130496774 26419 IndexH DEBUG: Helper Size: VmRSS=15.5 MB, size=1.29, 7.2% 060325 2130502470 26419 IndexH DEBUG: -file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.dofB2:2,S 060325 2130502575 26419 IndexH DEBUG: +file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.dofB2:2,S 060325 2130503169 26419 IndexH DEBUG: -file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.xBwFZ:2,S 060325 2130503231 26419 IndexH DEBUG: +file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.xBwFZ:2,S 060325 2130504183 26419 IndexH DEBUG: -file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.rxMJi:2,S 060325 2130504243 26419 IndexH DEBUG: +file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.rxMJi:2,S 060325 2130505230 26419 IndexH DEBUG: -file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.vnYzN:2,S 060325 2130505283 26419 IndexH DEBUG: +file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.vnYzN:2,S 060325 2130506177 26419 IndexH DEBUG: -file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.7aLv5:2,S 060325 2130506244 26419 IndexH DEBUG: +file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.7aLv5:2,S 060325 2130507520 26419 IndexH DEBUG: -file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.WHh41:2,S 060325 2130507635 26419 IndexH DEBUG: +file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.WHh41:2,S 060325 2130508534 26419 IndexH DEBUG: -file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.tipWr:2,S 060325 2130508640 26419 IndexH DEBUG: +file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.tipWr:2,S 060325 2130527537 26419 IndexH DEBUG: Helper Size: VmRSS=15.9 MB, size=1.32, 8.1% 060325 2130536459 26419 IndexH DEBUG: -file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.rR2mg:2,S 060325 2130536511 26419 IndexH DEBUG: +file:///home/double/Mail/HeartlandBBC2/cur/1143343725.2903.rR2mg:2,S ==> current-Beagle <== 060325 2130543115 20763 Beagle DEBUG: checking if /home/double/Mail/Jokes2 is a maildir ? 060325 2130546434 20763 Beagle DEBUG: Memory usage: VmSize=664.9 MB, VmRSS=265.7 MB, GC.GetTotalMemory=561377280 060325 2130546435 20763 Beagle DEBUG: VmSize too large --- shutting down
I have tried again a scenario that causes memory bloat. I had the computer suspended for around 18 hours, with beagled running all backends, but no kerry or beagle-query running live queries. I resumed, got many emails, and no memory bloat. Seems that something in the live-query may cause memory bloat.
Since i remove the beagle-live option from deskbar-applet beagle has been nice with me. I have never had another memory bloat. Maybe this bug is exposed too in the python-beagle bindings.
Hi All, After reading the previous comments, I think this memory leak is caused by live-query. When i did a normal search from within beagle-search nothing happens. Memory usage stays low. When I do a search with either deskbar-applet or nautilus (with beagle), beagled starts to leak memory.
jeremy: Does beagle-query --live-query cause a leak to happen? I am trying to determine if it's any live query or only ones done through libbeagle (the C interface).
Never mind, I was just able to trigger this with deskbar-applet and the beagle-live backend. I'm looking into it.
Hi Joe, If you do find the source of the problem and have a patch for it, do attach the patch here if you require additional testing. I will happily test it out for you. I'm running Ubuntu Dapper. Cheers! ;)
Created attachment 62421 [details] [review] Patch which might fix this. Try this patch out and see if it fixes the problem. With it, I've been unable to duplicate the memory problem, and the occasional exceptions I was also getting have gone away as well.
Hi Joe, I am unable to test this patch against CVS HEAD as I am getting "symbol lookup failed" errors from /usr/lib/beagle/libbeagleglue.so saying that it can't find XScreenSaverExtension. I've running Ubuntu Dapper. However, I did a test against the official 0.2.3 release and have been unable to reproduce the memory problem with both deskbar-applet and nautilus (with beagle integration). This patch works great. ;)
I am using kerry and applied the patch to beagle 0.2.3. I have not had a problem yet and I did run the situation in which most every time (if not every time) I would get the memory bloat. Thanks!!
Is this patch going mainstream? i hope it is, because i _need_ beagle :D Thanks Joe
Created attachment 62449 [details] logs when beagle prematurely stopped crawling Joe, here are the logs using beagle 0.2.3 and your patch. There were some exceptions and then beagled stopped crawling. I do not know if it is related. I had a search open in kerry during this. I am trying again.
(In reply to comment #46) > I am unable to test this patch against CVS HEAD as I am getting "symbol lookup > failed" errors from /usr/lib/beagle/libbeagleglue.so saying that it can't find > XScreenSaverExtension. I've running Ubuntu Dapper. Are you running latest CVS? I think I've already fixed this XScreenSaverExtension issue.
Joe, I have an update on my previous note of beagle stopped crawling. I erased my .beagle directory and tried to rebuild. The patched beagle 0.2.3 stopped twice. I then reverted the patch and attempted to rebuild from a fresh .beagle and it finished. The only exceptions I got was from the IndexHelper, there were about 5 of these: 060331 1920184419 23661 IndexH WARN EX: Entagged.Audioformats.Exceptions.CannotReadException: Entagged.Audioformats.Except ions.CannotReadException: Error: could not synchronize to first mp3 frame I had many exceptions with the patched beagle both from beagle and the index helper. I have the logs if you want them, the completed index is 2.4M so I don't think I can upload it here.
Hi Lukas, I just updated from CVS HEAD. So far no problems with the XScreenSaverExtension issue. ;) Cheers!
Ok, I just checked this patch into CVS. Going to close as FIXED. Pat: I removed the part of the patch that would throw that exception and stop your indexing, so that should be fixed now. There is also an unrelated bug in that your Kopete buddy list isn't being parsed correctly. If you would, please file a separate bug about that. The mp3 exception would also appear to be a separate bug. If you could also file that, that would be appreciated.
Happened to me again with latest CVS as of yesterday morning... I did not have --debug-memory on. I am updating from CVS now and will run with --debug-memory.
Found this... Question: if I have to kill beagled because it won't shutdown using beagle-shutdown, will that cause index corruption? 060406 1338241464 21041 Beagle ERROR: Caught exception executing Inotify callbacks 060406 1338241484 21041 Beagle ERROR EX: System.ArgumentException: length 060406 1338241484 21041 Beagle ERROR EX: in [0x00247] System.Array:Copy (System.Array sourceArray, Int32 sourceIndex, Syste m.Array destinationArray, Int32 destinationIndex, Int32 length) 060406 1338241484 21041 Beagle ERROR EX: in [0x00020] (at /var/tmp/portage/beagle-999/work/beagle/beagled/Lucene.Net/Index/ TermBuffer.cs:97) Lucene.Net.Index.TermBuffer:Set (Lucene.Net.Index.TermBuffer other) 060406 1338241484 21041 Beagle ERROR EX: in [0x00039] (at /var/tmp/portage/beagle-999/work/beagle/beagled/Lucene.Net/Index/ SegmentTermEnum.cs:128) Lucene.Net.Index.SegmentTermEnum:Next () 060406 1338241484 21041 Beagle ERROR EX: in [0x00040] (at /var/tmp/portage/beagle-999/work/beagle/beagled/Lucene.Net/Index/ SegmentTermEnum.cs:167) Lucene.Net.Index.SegmentTermEnum:ScanTo (Lucene.Net.Index.Term term) 060406 1338241484 21041 Beagle ERROR EX: in [0x00009] (at /var/tmp/portage/beagle-999/work/beagle/beagled/Lucene.Net/Index/ TermInfosReader.cs:166) Lucene.Net.Index.TermInfosReader:ScanEnum (Lucene.Net.Index.Term term) 060406 1338241484 21041 Beagle ERROR EX: in [0x0008e] (at /var/tmp/portage/beagle-999/work/beagle/beagled/Lucene.Net/Index/ TermInfosReader.cs:157) Lucene.Net.Index.TermInfosReader:Get (Lucene.Net.Index.Term term) 060406 1338241484 21041 Beagle ERROR EX: in [0x00007] (at /var/tmp/portage/beagle-999/work/beagle/beagled/Lucene.Net/Index/ SegmentReader.cs:341) Lucene.Net.Index.SegmentReader:DocFreq (Lucene.Net.Index.Term t) 060406 1338241484 21041 Beagle ERROR EX: in [0x00013] (at /var/tmp/portage/beagle-999/work/beagle/beagled/Lucene.Net/Index/ MultiReader.cs:221) Lucene.Net.Index.MultiReader:DocFreq (Lucene.Net.Index.Term t) 060406 1338241484 21041 Beagle ERROR EX: in [0x00007] (at /var/tmp/portage/beagle-999/work/beagle/beagled/Lucene.Net/Search /IndexSearcher.cs:186) Lucene.Net.Search.IndexSearcher:DocFreq (Lucene.Net.Index.Term term) 060406 1338241484 21041 Beagle ERROR EX: in [0x00003] (at /var/tmp/portage/beagle-999/work/beagle/beagled/Lucene.Net/Search /Similarity.cs:317) Lucene.Net.Search.Similarity:Idf (Lucene.Net.Index.Term term, Lucene.Net.Search.Searcher searcher) 060406 1338241484 21041 Beagle ERROR EX: in [0x00032] (at /var/tmp/portage/beagle-999/work/beagle/beagled/Lucene.Net/Search /TermQuery.cs:63) Lucene.Net.Search.TermQuery+TermWeight:.ctor (Lucene.Net.Search.TermQuery enclosingInstance, Lucene.Net.S earch.Searcher searcher) 060406 1338241484 21041 Beagle ERROR EX: in [0x00002] (at /var/tmp/portage/beagle-999/work/beagle/beagled/Lucene.Net/Search /TermQuery.cs:166) Lucene.Net.Search.TermQuery:CreateWeight (Lucene.Net.Search.Searcher searcher) 060406 1338241484 21041 Beagle ERROR EX: in [0x0000a] (at /var/tmp/portage/beagle-999/work/beagle/beagled/Lucene.Net/Search /Query.cs:98) Lucene.Net.Search.Query:Weight (Lucene.Net.Search.Searcher searcher) 060406 1338241484 21041 Beagle ERROR EX: in [0x00003] (at /var/tmp/portage/beagle-999/work/beagle/beagled/Lucene.Net/Search /IndexSearcher.cs:262) Lucene.Net.Search.IndexSearcher:Search (Lucene.Net.Search.Query query, Lucene.Net.Search.Filter filt er, Lucene.Net.Search.HitCollector results) 060406 1338241484 21041 Beagle ERROR EX: in [0x00029] (at /var/tmp/portage/beagle-999/work/beagle/beagled/FileSystemQueryab le/LuceneNameResolver.cs:141) Beagle.Daemon.FileSystemQueryable.LuceneNameResolver:GetNameInfoById (Guid id) 060406 1338241484 21041 Beagle ERROR EX: in [0x00053] (at /var/tmp/portage/beagle-999/work/beagle/beagled/FileSystemQueryab le/FileSystemQueryable.cs:1023) Beagle.Daemon.FileSystemQueryable.FileSystemQueryable:AddFile (Beagle.Daemon.FileSystemQuer yable.DirectoryModel dir, System.String name) 060406 1338241484 21041 Beagle ERROR EX: in [0x00074] (at /var/tmp/portage/beagle-999/work/beagle/beagled/FileSystemQueryab le/FileSystemQueryable.cs:1420) Beagle.Daemon.FileSystemQueryable.FileSystemQueryable:HandleAddEvent (System.String directo ry_name, System.String file_name, Boolean is_directory) 060406 1338241484 21041 Beagle ERROR EX: in [0x00116] (at /var/tmp/portage/beagle-999/work/beagle/beagled/FileSystemQueryab le/InotifyBackend.cs:126) Beagle.Daemon.FileSystemQueryable.InotifyBackend:OnInotifyEvent (Watch watch, System.String path, System.String subitem, System.String srcpath, EventType type) 060406 1338241484 21041 Beagle ERROR EX: in (wrapper delegate-invoke) System.MulticastDelegate:invoke_void_Inotify/Watch_st ring_string_string_Inotify/EventType (Beagle.Util.Inotify/Watch,string,string,string,Beagle.Util.Inotify/EventType) 060406 1338241484 21041 Beagle ERROR EX: in [0x0010d] (at /var/tmp/portage/beagle-999/work/beagle/Util/Inotify.cs:638) Beag le.Util.Inotify:SendEvent (Beagle.Util.WatchInfo watched, System.String filename, System.String srcpath, EventType mask)
Created attachment 62912 [details] log snippet of memory bloat Here's another log. I was able to reproduce by: 1. BEAGLE_EXERCISE_THE_DOG=1 schedtool -D -e beagled --bg --debug-memory --autostarted 2. After done crawling, load a playlist in amaroK which sends inotify events for all mp3 files 3. Immediately do a query for 'kovpn' I am going to try to reproduce again and if I can I will run again with --profile=default:heap to see if I can get a heap dump.
Pat: killing it could corrupt the index, but it shouldn't be a big problem. If your index was messed up before I committed the patch to fix this, though, that might still be a problem. Have you nuked your ~/.beagle recently? If you're doing profiling, using heap-buddy would be much preferred to mono's built-in heap profiler. You can get it from here: http://blog.trowbridge.org/index.php?p=49 Then you can just attach the outfile (or put it up on the web) and I can do analysis on it.
I removed .beagle yesterday morning and rebuilt, I had installed the CVS version before that. I am using heap-buddy right now, waiting for the crawl to finish. I saw the --heap-buddy option in beagled and figured it was the preferred way. I will definately have to post on a web site, it's already >80M.
Created attachment 63317 [details] logs when beagle ate all memory I am using beagle 2.4 and libbeagle 2.4 The problem is present again, but now it happens randomly. In fact i was not doing a search when the beagle starts to eat all memory
I had this happen again as well, but not when running with --heap-buddy. I cannot get it to happen. However, I have to pin beagled to a specific processor when using heap-buddy, it does not like SMP. Perhaps SMP is triggering the problem? Joe, you said earlier this is a problem in mono. I thinking that the workaround you added does not cover all the triggers so I have updated to mono 1.1.13.6 and will run without heap-buddy to see if it happens.
I am using mono 1.1.13.4 Can it be the problem??
alejandro: you're seeing the same thing as Pat. It's being handled more specifically in bug 338165.
Just occurred with mono 1.1.13.6. Joe, should we move this discussion to bug 338165 ?
The primary cause of this (see bug 338165) is fixed in CVS and will go into 0.2.6. Afterward, if you see memory issues, please file a new bug for it, as it will probably be a different issue than the one we've been tracking here (the "length" exception one).