GNOME Bugzilla – Bug 576456
backend fails to start: can't remove read-only lock files when index is recreated
Last modified: 2009-05-27 20:07:53 UTC
Please describe the problem: I found 2.5M of numbered files in /home/udo/.beagle/Indexes/ThunderbirdIndex/ToIndex. I cleaned them out (finding a kernel bug on the way) and restarted the indexing of Thunderbird info. Now the index shows that items are added but in /home/udo/.beagle/Indexes/ThunderbirdIndex/ToIndex the number of files grows and grows. Steps to reproduce: 1. Do index Thunderbird 2. wait 3. see ~/.beagle/Indexes/ThunderbirdIndex/ToIndex Actual results: A lot of files in there Expected results: They are supposed to be cleaned up after being indexed? Does this happen every time? So far: yes Other information: beagle-0.3.9-2.fc10.x86_64 beagle-gnome-0.3.9-2.fc10.x86_64 beagle-firefox-0.3.9-2.fc10.x86_64 beagle-thunderbird-0.3.9-2.fc10.x86_64 libbeagle-0.3.5-1.fc9.x86_64 Fedora 10 on x86_64
[udo@surfplank2 ToIndex]$ pwd /home/udo/.beagle/Indexes/ThunderbirdIndex/ToIndex [udo@surfplank2 ToIndex]$ ls|wc -l 28302 (yes, I deleted a few but now I let it run)
It still fills up. Thousands of files. This cannot be correct.
[udo@surfplank2 ~]$ cd .beagle/Indexes/ThunderbirdIndex/ToIndex/ [udo@surfplank2 ToIndex]$ ls|wc 1099763 1099763 7695400
Don't remember ... but I remember some people saying beagle is broken with recent thunderbird (it was some API incompatibility in the extension). It is hard to guess what could be wrong. The following information might be helpful ... * shutdown beagle ($ beagle-shutdown) * remove .beagle/Indexes/ThunderbirdIndex/ * start beagled from a terminal as $ beagled --fg --debug --backend Thunderbird * Wait till a few files start accumulating in ToIndex * Stop beagled (send ctrl-c or run beagle-shutdown) The terminal output might have some error or exception - if you spot them, let me know. Otherwise you can attach beagle log file ~/.beagle/Log/current-* (it might contain full paths to your mailboxes).
Fedora 10: $ rpm -qa|grep beagle beagle-0.3.9-2.fc10.x86_64 beagle-gnome-0.3.9-2.fc10.x86_64 beagle-firefox-0.3.9-2.fc10.x86_64 beagle-thunderbird-0.3.9-2.fc10.x86_64 libbeagle-0.3.5-1.fc9.x86_64 $ rpm -qa|grep thunderbird thunderbird-2.0.0.21-1.fc10.x86_64 beagle-thunderbird-0.3.9-2.fc10.x86_64 So far nothing strange happened. Started beagle, started thunderbird. ToIndex remained at 0. Stopped beagle after a while. Restarted beagle. ToIndex files came down to 0...
If I had to guess, I suspect one of the files to be indexed caused enough errors to disable the backend and then the queue started filling up like crazy. That would probably explain why after nuking the ThunderbirdIndex directory things seem to be working ok now. The logs prior to deleting the directory might help confirm this.
> So far nothing strange happened. Started beagle, started thunderbird. ToIndex > remained at 0. Stopped beagle after a while. Restarted beagle. ToIndex files > came down to 0... Tch tch bad bad ... keep it running though. We need to reproduce the problem - Joe could be on track here. Could you attach a tarball of your log files ~/.beagle/Log - or if they are too large, just ~/.beagle/Log/*Exception[s] file(s)?
This morning I found files: $ ls ToIndex/|wc -l 66723 so: $ grep -i exc *|grep -i thund 2009-04-29-16-00-14-Beagle:20090429 16:00:17.2511 04048 Beagle ERROR EX: Caught exception while instantiating Thunderbird backend 2009-04-29-16-00-14-Beagle:20090429 16:00:17.2511 04048 Beagle ERROR EX: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.UnauthorizedAccessException: Access to the path "/home/udo/.beagle/Indexes/ThunderbirdIndex/Locks/lucene-50b7226e60ec5b3d3c19b0d6ab053118-write.lock" is denied. 2009-04-29-16-00-14-BeagleExceptions:20090429 16:00:17.2511 04048 Beagle ERROR EX: Caught exception while instantiating Thunderbird backend 2009-04-29-16-00-14-BeagleExceptions:20090429 16:00:17.2511 04048 Beagle ERROR EX: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.UnauthorizedAccessException: Access to the path "/home/udo/.beagle/Indexes/ThunderbirdIndex/Locks/lucene-50b7226e60ec5b3d3c19b0d6ab053118-write.lock" is denied. 2009-04-29-16-03-39-Beagle:20090429 16:03:39.7417 08073 Beagle ERROR EX: Caught exception while instantiating Thunderbird backend 2009-04-29-16-03-39-Beagle:20090429 16:03:39.7417 08073 Beagle ERROR EX: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.UnauthorizedAccessException: Access to the path "/home/udo/.beagle/Indexes/ThunderbirdIndex/Locks/lucene-50b7226e60ec5b3d3c19b0d6ab053118-write.lock" is denied. 2009-04-29-16-03-39-BeagleExceptions:20090429 16:03:39.7417 08073 Beagle ERROR EX: Caught exception while instantiating Thunderbird backend 2009-04-29-16-03-39-BeagleExceptions:20090429 16:03:39.7417 08073 Beagle ERROR EX: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.UnauthorizedAccessException: Access to the path "/home/udo/.beagle/Indexes/ThunderbirdIndex/Locks/lucene-50b7226e60ec5b3d3c19b0d6ab053118-write.lock" is denied. 2009-04-29-16-41-59-Beagle:20090429 16:42:01.7007 15503 Beagle ERROR EX: Caught exception while instantiating Thunderbird backend 2009-04-29-16-41-59-Beagle:20090429 16:42:01.7007 15503 Beagle ERROR EX: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.UnauthorizedAccessException: Access to the path "/home/udo/.beagle/Indexes/ThunderbirdIndex/Locks/lucene-50b7226e60ec5b3d3c19b0d6ab053118-write.lock" is denied. 2009-04-29-16-41-59-BeagleExceptions:20090429 16:42:01.7007 15503 Beagle ERROR EX: Caught exception while instantiating Thunderbird backend 2009-04-29-16-41-59-BeagleExceptions:20090429 16:42:01.7007 15503 Beagle ERROR EX: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.UnauthorizedAccessException: Access to the path "/home/udo/.beagle/Indexes/ThunderbirdIndex/Locks/lucene-50b7226e60ec5b3d3c19b0d6ab053118-write.lock" is denied. 2009-05-01-15-00-39-Beagle:20090501 15:00:41.2974 04063 Beagle ERROR EX: Caught exception while instantiating Thunderbird backend 2009-05-01-15-00-39-Beagle:20090501 15:00:41.2974 04063 Beagle ERROR EX: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.UnauthorizedAccessException: Access to the path "/home/udo/.beagle/Indexes/ThunderbirdIndex/Locks/lucene-50b7226e60ec5b3d3c19b0d6ab053118-write.lock" is denied. 2009-05-01-15-00-39-BeagleExceptions:20090501 15:00:41.2974 04063 Beagle ERROR EX: Caught exception while instantiating Thunderbird backend 2009-05-01-15-00-39-BeagleExceptions:20090501 15:00:41.2974 04063 Beagle ERROR EX: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.UnauthorizedAccessException: Access to the path "/home/udo/.beagle/Indexes/ThunderbirdIndex/Locks/lucene-50b7226e60ec5b3d3c19b0d6ab053118-write.lock" is denied. current-Beagle:20090501 15:00:41.2974 04063 Beagle ERROR EX: Caught exception while instantiating Thunderbird backend current-Beagle:20090501 15:00:41.2974 04063 Beagle ERROR EX: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.UnauthorizedAccessException: Access to the path "/home/udo/.beagle/Indexes/ThunderbirdIndex/Locks/lucene-50b7226e60ec5b3d3c19b0d6ab053118-write.lock" is denied. current-BeagleExceptions:20090501 15:00:41.2974 04063 Beagle ERROR EX: Caught exception while instantiating Thunderbird backend current-BeagleExceptions:20090501 15:00:41.2974 04063 Beagle ERROR EX: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.UnauthorizedAccessException: Access to the path "/home/udo/.beagle/Indexes/ThunderbirdIndex/Locks/lucene-50b7226e60ec5b3d3c19b0d6ab053118-write.lock" is denied. $ ls -l /home/udo/.beagle/Indexes/ThunderbirdIndex/Locks total 4 -r-------- 1 udo udo 6 2009-05-01 14:56 lucene-50b7226e60ec5b3d3c19b0d6ab053118-write.lock I do think the may 1 exceptions are most important.
So I shutdown beagle. I removed the lock. I restarted beagle. Now the number of files in ToIndex is going down again.
$ ls ToIndex/|wc -l 0 So: what about the lock, is it really the problem?
Yeah ... this is a problem which was hidden until people started using mono 2.0. It was mostly fixed before 0.3.9 - at least all the cases that were reported till then were fixed. The thunderbird and firefox backend has unveiled two more cases ... there is really no short term workaround other than removing the lock file (actually, just make the lock file read-write and it will also work).
BTW, can you attach the log file 2009-04-29-16-00-14-Beagle ... though there is a problem, I still don't quite see who is triggering it.
Created attachment 133840 [details] 2009-04-29-16-00-14-Beagle
The errors on the 29th could be due to me cleaning out, restarting, etc. Not sure.
This also happens to the Files backend. Is this the same bug? If you "kill -9" all the beagle-related processes (a bad idea, of course, but to illustrate the problem ...), then you get stale lock-files in ~/.beagle/Indexes/FileSystemIndex/Locks/. These prevent any new beagle from starting: 20090515 16:46:19.5273 11431 Beagle ERROR EX: Caught exception while instantiating Files backend 20090515 16:46:19.5273 11431 Beagle ERROR EX: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.UnauthorizedAccessException: Access to the path "/home/john/.beagle/Indexes/FileSystemIndex/Locks/lucene-61ceec39efda4799d7169ec0e9bc60b5-write.lock" is denied. For the average user, this means that if there is one power-failure in the house, then beagle is disabled forever. Beagle: 0.3.9 Mono: Mono 2.0.1 Ubuntu 9.0.4
I think you are right, the two bugs are very similar. Is there patch with a fix that we could test? (make lockfile rw)
Fixed in trunk - make the lockfile rw. Get the patch from the link below. I am pretty sure this would fix the problem, but in case it doesn't, please reopen. Thanks. http://git.gnome.org/cgit/beagle/commit/?id=8590dcd1cd7533226a9091ae0d1f039415f22cf5