Created attachment 102063 [details]
More info

Can you attach /home/nx/.beagle/Log/2007-12-30-09-18-13-BeagleExceptions and /home/nx/.beagle/Log/2007-12-30-09-18-13-BeagleConsole ?

Also do you do a lot of querying or the beagled is pretty much inactive after initial crawl ? Last question, next time you start beagled can you start with only the files backend (--backend Files) and see if the same situation occurs. I will run beagled overnight and try to reproduce this.

I'll attach 2007-12-30-09-18-13-BeagleException, the other file (2007-12-30-09-18-13-BeagleConsole) is empty.

ATM I mostly use beagle using the deskbar search. But not more than perhaps 10 queries a day I think, at least in recent days. So beagled should be mostly inactive. I did not use the pc much from dec 28 to jan 2, and there certainly were no big changes in the file system.

Created attachment 102071 [details]
2007-12-30-09-18-13-BeagleExceptions

Thanks. There was a small leak (really small, I think) in Evolution Data Server backend which I spotted in the BeagleExceptions file. I checked that in (r4365), there should not be any more exceptions now.

But that problem cannot cause such a memory blowup.

Which version of deskbar are you running ?

There is a possibility that the problem is how deskbar is querying beagle or in our python-bindings; you can test if that is the case by quitting deskbar - memory usage should drop. As you said there wasnt much indexing activity, I am trying to think of the possibilities related to querying.

It would also be helpful if you could run beagled with the --debug-memory flag, and attach the regular Beagle log (not BeagleExceptions) to the bug.  That way we can see if the memory is exploding at once or whether it's a slow leak.

Thanks guys. I will update to trunk now and let beagled run with --debug-memory. Currently Gnome-system-monitor shows beagled using about 60mb, when I checked it last yesterday evening it was still below 30.

I did not use deskbar yesterday I think, but I have removed it from the panel now for the time being. Would be really nice to be able to track this problem down.

(In reply to comment #8)
> Thanks guys. I will update to trunk now and let beagled run with
> --debug-memory.

Thanks. For the record, I ran beagled overnight with the usual options, i.e. all backends running, though this is a test gutsy machine and does not have all kinds of data, and its still using the same 40MB RSS it was when crawling finished. I ran a script which periodically queried beagle using beagle-query and called beagle-status, so there is no problem in usual search and status querying.

I will test the memory usage with deskbar sometime later today (whatever version comes default with gutsy).

It is highly possibly that some kind of event, e.g. new data or removal of some data, is causing the problem in your case. The log file with --debug-memory will hopefully point out what is wrong.

Ahh... one more thing, after you think crawling is over, "beagle-info --status" should show no tasks. Do you see that ?

I'm running a test right now, will report back tomorrow. Deskbar is disabled for now and I won't to other queries.

The "beagle-info --status" shows this:

Pending Tasks:
Scheduler queue is empty.

Future Tasks:
Maintenance 0 (1/4/2008 12:52:43 PM)
Optimize PidginIndex
Hold until 1/5/2008 12:52:33 PM

Maintenance 0 (1/4/2008 3:24:35 PM)
Optimize FileSystemIndex
Hold until 1/5/2008 1:00:51 PM

Maintenance 0 (1/4/2008 3:31:23 PM)
Optimize TomboyIndex
Hold until 1/5/2008 12:43:08 PM

Maintenance 0 (1/4/2008 4:18:36 PM)
Optimize EvolutionDataServerIndex
Hold until 1/5/2008 12:32:54 PM

Ok, here is more information:

beagled --fg --debug-memory > beagle.log started at 12:20 with 9.7mb:
$ ps ux | grep beagle
nx        8360  8.0  0.9  59804 19112 pts/0    SNl+ 12:20   0:00 beagled /opt/gnome2.22/lib/beagle/BeagleDaemon.exe --fg --debug-memory
nx       11906  0.0  0.0   2980   760 pts/2    R+   12:21   0:00 grep beagle

"Scheduler queue is empty" at 01:10 with 24.7mb (this is a very common value for this state on my system):
$ ps ux | grep beagle
nx        8360  0.8  1.8  96260 38128 pts/0    SNl+ 12:20   0:31 beagled /opt/gnome2.22/lib/beagle/BeagleDaemon.exe --fg --debug-memory
nx       27533  0.6  1.0  64276 21760 pts/0    SNl+ 12:22   0:21 beagled-helper /opt/gnome2.22/lib/beagle/IndexHelper.exe

After running for about 20 hours, g-s-m shows 74.7mb in use:
nx        8360  0.0  4.3 147460 89308 pts/0    SNl+ Jan04   0:49 beagled /opt/gnome2.22/lib/beagle/BeagleDaemon.exe --fg --debug-memory
nx       27533  0.2  1.1  73036 23648 pts/0    SNl+ Jan04   2:44 beagled-helper /opt/gnome2.22/lib/beagle/IndexHelper.exe

This time I checked memory usage very often and I can say that it grew slowly but nearly at a linear rate. Before I had the impression that this effect only kicked in after a longer period of time, but this looks like as if it grew from the start.

I will also attach the lsof output and the log file.

Created attachment 102200 [details]
2007-01-04 lsof

Created attachment 102201 [details]
2007-01-04 log

There was no BeagleExceptions log this time.

I did a bunch of experiments yesterday, sending lots of random queries, querying using deskbar, several file operations but nothing seems to trigger anything. The memory usage now whatever it was when it started!
The only two differences from yours is that apart from Files, Opera and a couple more, the other backends do not have data to index. They are still running. And also I am using mono-1.2.4.

I checked your log file and you are right, the memory seems to gradually increase all the way. I am still trying to figure out what could be happening.

I'm running with --backend Files since the morning and the memory is staying exactly at 13.9mb for hours now. So I guess the files backend is not to blame.

Backends I have enabled: EvolutionDataServer, Files, Pidgin, Tomboy, applications, documentation.

Ahh... excellent find. Give it a few more hours and if you are satisfied (given the rate at which memory usage was increasing in the logfile, it should be easy to figure out if the Files backend is leaking) then could you individually try Evo..., Pidgin, Tomboy (ordered in the order of likelihood of leaking). applications and documentation are static indexes so they dont matter.

Another round of tests done: running with only files seems to be very stable memory-wise. Running only Pidgin for 3 hours, beagled grew about 0.2MB, but there have been some new messages to index during that time, so that seems to be quite stable, too.

Then I ran with EvolutionDataServer only. Again, about 3 hours. The initial memory usage after properly starting up was 17.7MB and it steadily grew to 24.5MB in gnome-system-monitor. The output of ps at the beginning and end:

nx         485 16.7  0.8  57364 17552 pts/0    SNl+ 17:33   0:00 beagled /opt/gnome2.22/lib/beagle/BeagleDaemon.exe --fg --backend EvolutionDataServer

nx         485  0.0  1.8  95092 37452 pts/0    SNl+ 17:33   0:03 beagled /opt/gnome2.22/lib/beagle/BeagleDaemon.exe --fg --backend EvolutionDataServer

Is this what we are looking for? I have started beagle again with all my usual backends but EDS disabled. I'll monitor this for a bit now and see if the process grows again.

Joe, please advice (see below).

(In reply to comment #17)
> Then I ran with EvolutionDataServer only. Again, about 3 hours. The initial
> memory usage after properly starting up was 17.7MB and it steadily grew to
> 24.5MB in gnome-system-monitor. The output of ps at the beginning and end:
> 
> nx         485 16.7  0.8  57364 17552 pts/0    SNl+ 17:33   0:00 beagled
> /opt/gnome2.22/lib/beagle/BeagleDaemon.exe --fg --backend EvolutionDataServer
> 
> nx         485  0.0  1.8  95092 37452 pts/0    SNl+ 17:33   0:03 beagled
> /opt/gnome2.22/lib/beagle/BeagleDaemon.exe --fg --backend EvolutionDataServer
> 
> Is this what we are looking for?

Thats a 20MB! increase in RSS. Looks like you nailed it. Maybe EDS backend requires some memory upfront but continously requiring more and more memory does not look right to me. Unfortunately I have never used EDS so I cant quite jump into the debugging bandwagon.

Something I forget to mention: While I do have some contacts, calendars and tasks,  most of the time I only use the mail component of evolution. So contacts/calendars/tasks (what the EDS backend is all about, right?) did not even change in the last weeks or so.

Beagle without EvolutionDataServer enabled gives me the following results:

When sheduler queue is empty:
nx       21583  1.2  1.1  65232 23596 pts/0    SNl+ 20:17   0:30 beagled /opt/gnome2.22/lib/beagle/BeagleDaemon.exe --fg

After about 14 hours:
nx       21583  0.0  1.3  69776 27328 pts/0    SNl+ Jan05   0:32 beagled /opt/gnome2.22/lib/beagle/BeagleDaemon.exe --fg

So this is not nearly as bad... and it nearly did not grow at all during the last 8 hours where I was asleep. Perhaps there's another small problem.

(In reply to comment #20)
> So this is not nearly as bad... and it nearly did not grow at all during the
> last 8 hours where I was asleep. Perhaps there's another small problem.

I doubt it will grow much from there. Probably the index optimization happened during that time which I have observed to increase the RSS a bit.

Yes... all I wanted to say that the EDS backend is to blame for the huge memory usage. Is there some way I can help track this down any further?

I asked another EDS backend user and he said there is no leak on his machine even with large uptime. This kind of suggests that the problem might not be easily reproducible.

I took an uneducated look at the EDS backend source and found nothing suspicious. Also nothing serious changed in the last few months which could have added this bug. Maybe its a bug in the evolution-sharp library.

Which evo-sharp version are you using ? I will see if I can get in touch with the evo-sharp dev and eds-backend authors to get them look at this.

Is it possible for you to install heap-shot ? http://www.mono-project.com/HeapShot
It is very easy to build. If you can, please build, install it and run beagled (with only eds backend) with --heap-shot. Then, once initial crawling is over, send a "kill -PROF <pid-of-beagled>". It will create an outfile*.omap file with the snapshot of the memory at that point. Then, let it run some more and once you see its memory usage is high enough, send another PROF signal and collect the second *.omap file. Send me those files, I might be able to find out the difference from them. Its really a complicated setup :( but that is the best I can suggest.

Ok, I'll try heapshot. Btw, this are the versions I use for testing:

mono: 1.2.6
beagle: trunk
evolution-data-server: 2.21.5
evolution-sharp: 0.15.4

Fine, I think I have a nice set of test data now. Beagle growing form 31480 RSS to 69008 in about 12 hours... 

Included is a small file where I document some states with ps and beagle-status output, as well as the two omaps. I have more of those, if you need them, just say so.

I'll send you the data in private now.

Wonderful. Can you send me two more from the middle ?

Looks like we are sure about the EDS backend thing. Changing summary.

Yep, as per our mail and IRC conversations, it is indeed a leak in evolution-sharp bindings.  More specifically, a regression of what was fixed couple of years ago.  Hope this can help Johnny to fix it. http://varadhan.blogspot.com/2005/08/showstopper.html :)

I made this depend on an EDS bug http://bugzilla.gnome.org/show_bug.cgi?id=509536 

BTW, Johnny, Varadhan, is it possible to have any workaround ? E.g. by using the previous ical-to-ecal-array routine (which was fixed for leaks by the our mighty Varadhan ;-). Say even we are ready to P/Invoke directly into libecal ?

FYI, there is now a patch in the linked EDS bug.

Fixed in evo-sharp 0.15.92

Are you ahead of time? SVN only seems to know e-sharp 0.15.91...

Forgot to mention, I was time travelling and was 12 hours ahead of earth time :). So yes ... "Fixed in evo-sharp " and will be soon available in "0.15.92".

evolution-sharp 0.15.92 released. Thanks for the info / testing .. you guyz rock !!