This is likely something that will have to be done by someone in the community.
 Setting milestone as such.

One concern I have with using the thunderbird API to index mail:

If it goes through indexing an IMAP account, presumably the API transparently
handles downloading mail from the server? That way, beagle would be indexing
'network' data without realising it, which might not be what we want.

Also worth checking out this forum post:
http://forums.mozillazine.org/viewtopic.php?p=1891642

If the final comment about GDS requiring thunderbird to be open is true then
that makes it even less viable. Would be good if someone could check up on this.

(In reply to comment #2)
> One concern I have with using the thunderbird API to index mail:
> 
> If it goes through indexing an IMAP account, presumably the API transparently
> handles downloading mail from the server? That way, beagle would be indexing
> 'network' data without realising it, which might not be what we want.

Actually, could it be configurable? I would like to search for messages on my desktop using my laptop (using IMAP). BTW, is it possible using evolution?

Created attachment 63262 [details] [review]
Thunderbird Backend

This patch will add basic Thunderbird support to beagle

This patch (partially) works for me: indexing of POP accounts works and searching works. What does not work is:
- indexing of IMAP accounts
- indexig/searching of some attachments in mails
- opening documenta (emails) - nothing happens when clicked
- opening attachments - when opening images from attachments, beagle-search calls gimp-remote with "mailbox:////home/user/.thunderbird/...", which gimp can not open. Other attachments are also called with "mailbox:" filetype.

But overall, at least I can search my mails now. Thanks!
[Archlinux-0.7.1: mono-1.1.14, beagle-0.2.4, thunderbird-1.5]

(In reply to comment #6)
> This patch (partially) works for me: indexing of POP accounts works and
> searching works. What does not work is:
> - indexing of IMAP accounts
> - indexig/searching of some attachments in mails
> - opening documenta (emails) - nothing happens when clicked
> - opening attachments - when opening images from attachments, beagle-search
> calls gimp-remote with "mailbox:////home/user/.thunderbird/...", which gimp can
> not open. Other attachments are also called with "mailbox:" filetype.
> 
> But overall, at least I can search my mails now. Thanks!
> [Archlinux-0.7.1: mono-1.1.14, beagle-0.2.4, thunderbird-1.5]
> 

Most of these issues are just unimplemented features. More information about this is available on the mailinglist. 

I've spent some time today fixing some of the unimplemented stuff. IMAP url scheme is now correct . I also added support for subfolders in conjunction with fixing the IMAP scheme. Opening attachments now works in the same way as the evolution backend, by opening the parent email. 

When I wrote the search-ui patch I used "thunderbird" as command when starting thunderbird, this is probably the reason why you can't open any mails. I've changed this command to "thunderbird-bin" instead, which is more correct ("thunderbird" is a gentoo specific script I just found out).

I would also appreciate if you could upload a log file when you're running with the thunderbird backend, just to help me figure out why your IMAP mails aren't indexed.

I will also attach a new patch with today's work.

Created attachment 63320 [details] [review]
Thunderbird Backend improvements

Looks very good, I have a few comments.

* I'd like to see Mork.cs moved into the Util directory, in case we want to parse other Mork files.

* In ThunderbirdIndexableGenerator, instead of using Directory.GetFileSystemEntries(), use our own DirectoryWalker class.  It's a lot faster for large directories and results in fewer allocations.  Plus, it exposes an enumerable interface.

* You don't need to lock mork_files in AddMorkFile(), since that code will only ever be called from one thread (and you don't lock it anywhere else either)

* I'd recommend running StartWorker() in a thread in IndexDataCheck()

* In your update patch, you have "mailbox = database.Filename.Substring (account.Directory.Length+1);"  Could this be replaced with "mailbox = Path.GetFileName (database.FileName);" ?

* When concatenating strings that contain a bunch of parts, like the Uri you create in NewMailUri(), it's more readable (and often more efficient) to use String.Format().  Can you make that change?

The main difference between this and the Evolution backend is that you're indexing per-account, whereas the Evo mail backend does it per-folder.  This means that in your backend, an entire folder has to be indexed before the next one even gets touched.  I'm not sure which is better, but I thought it was worth pointing out.

(In reply to comment #9)
> Looks very good, I have a few comments.
> 
> * I'd like to see Mork.cs moved into the Util directory, in case we want to
> parse other Mork files.

Fixed.

> * In ThunderbirdIndexableGenerator, instead of using
> Directory.GetFileSystemEntries(), use our own DirectoryWalker class.  It's a
> lot faster for large directories and results in fewer allocations.  Plus, it
> exposes an enumerable interface.

Fixed.

> * You don't need to lock mork_files in AddMorkFile(), since that code will only
> ever be called from one thread (and you don't lock it anywhere else either)

Don't even know why I did that, but it's fixed now.

> * I'd recommend running StartWorker() in a thread in IndexDataCheck()

Could you give me an example on this? Can't find anything about this in the evolution backend nor the kmail backend.

> * In your update patch, you have "mailbox = database.Filename.Substring
> (account.Directory.Length+1);"  Could this be replaced with "mailbox =
> Path.GetFileName (database.FileName);" ?

No, Path.GetFileName() won't work. In order to support subfolders, we need the path to each mailbox relative to the directory we are indexing (the account directory).

> * When concatenating strings that contain a bunch of parts, like the Uri you
> create in NewMailUri(), it's more readable (and often more efficient) to use
> String.Format().  Can you make that change?

Sure. Fixed in the patch.

> The main difference between this and the Evolution backend is that you're
> indexing per-account, whereas the Evo mail backend does it per-folder.  This
> means that in your backend, an entire folder has to be indexed before the next
> one even gets touched.  I'm not sure which is better, but I thought it was
> worth pointing out.

I'm not sure which is the best one either, but I think I will have to change some aspects in the current design in order to get the automatic re-indexing of modified/created/deleted files working after the initial indexing has completed (one of the things mentioned on the mailing list).

I will attach an updated patch that combines the other two patches and fixes the things mentioned here.

Created attachment 63337 [details] [review]
Updated Thunderbird Backend

Trying the update thunderbird backend (attachment 63337 [details] [review]), with beagle-2.4 (-r2 for any gentoo users out there), I get the following warnings and it doesn't appear as though my mail is getting indexed.  This happens on both of the boxes I've tested it with:

Warn: Caught exception in DoTaskReal
Warn:         Tag: /home/mike/.thunderbird/dliskvne.default/ImapMail/mail.messagingengine.com
Warn:     Creator:
Warn: Description:
Warn:    Priority: Delayed (0)
Warn: System.UriFormatException: Invalid URI: The hostname could not be parsed.
in [0x00787] System.Uri:Parse (System.String uriString)
in [0x00007] (at /var/tmp/portage/mono-1.1.13.4/work/mono-1.1.13.4/mcs/class/System/System/Uri.cs:982) System.Uri:ParseUri ()
in [0x00001] (at /var/tmp/portage/mono-1.1.13.4/work/mono-1.1.13.4/mcs/class/System/System/Uri.cs:976) System.Uri:Parse ()
in [0x0005e] (at /var/tmp/portage/mono-1.1.13.4/work/mono-1.1.13.4/mcs/class/System/System/Uri.cs:160) System.Uri:.ctor (System.String uriString, Boolean dontEscape)
in [0x00003] (at /var/tmp/portage/mono-1.1.13.4/work/mono-1.1.13.4/mcs/class/System/System/Uri.cs:111) System.Uri:.ctor (System.String uriString)
in (wrapper remoting-invoke-with-check) System.Uri:.ctor (string)
in [0x00084] (at /var/tmp/portage/beagle-0.2.4-r2/work/beagle-0.2.4/beagled/ThunderbirdQueryable/ThunderbirdIndexer.cs:129) Beagle.Daemon.ThunderbirdQueryable.ThunderbirdIndexer:NewMailUri (Beagle.Daemon.ThunderbirdQueryable.ThunderbirdAccount account, System.String mailbox, System.String id)
in [0x00009] (at /var/tmp/portage/beagle-0.2.4-r2/work/beagle-0.2.4/beagled/ThunderbirdQueryable/ThunderbirdIndexer.cs:142) Beagle.Daemon.ThunderbirdQueryable.ThunderbirdIndexer:MorkMailToIndexable (Beagle.Daemon.ThunderbirdQueryable.ThunderbirdAccount account, Beagle.Daemon.ThunderbirdQueryable.ThunderbirdMail mail, GMime.Message message, System.String mailbox)
in [0x000ad] (at /var/tmp/portage/beagle-0.2.4-r2/work/beagle-0.2.4/beagled/ThunderbirdQueryable/ThunderbirdIndexableGenerator.cs:143) Beagle.Daemon.ThunderbirdQueryable.ThunderbirdIndexableGenerator:GetNextIndexable ()
in [0x00043] (at /var/tmp/portage/beagle-0.2.4-r2/work/beagle-0.2.4/beagled/LuceneQueryable.cs:573) Beagle.Daemon.LuceneQueryable+AddGeneratorTask:DoTaskReal ()
in [0x00054] (at /var/tmp/portage/beagle-0.2.4-r2/work/beagle-0.2.4/Util/Scheduler.cs:267) Beagle.Util.Scheduler+Task:DoTask ()

If there's any further information or testing I can do to help, please let me know.  I've been holding off using beagle because of the lack of thunderbird support, and I'd love to get this working!  Thanks for all your great work Pierre!

Right, I just figured out that the problem lies in NewMailUri.  For IMAP servers you've used account.Name and unfortunately this can contain spaces (as mine does).   I don't know C# well enough, but is there some kind of urlencoding function that could turn spaces in %20, and if so would this have an impact on where-ever this value is used?  Once again, many thanks for producing this backend, it's been much in demand!  5:)

(In reply to comment #13)
> Right, I just figured out that the problem lies in NewMailUri.  For IMAP
> servers you've used account.Name and unfortunately this can contain spaces (as
> mine does).   I don't know C# well enough, but is there some kind of
> urlencoding function that could turn spaces in %20, and if so would this have
> an impact on where-ever this value is used?  Once again, many thanks for
> producing this backend, it's been much in demand!  5:)
> 

Yes, I actually know about this problem already. I'm parsing the wrong thing from prefs.fs. Currently I'm reading the "name", but I should read "userName@hostname", where usernName and hostname are two settings saved separately in pref.js. I will work on this tonight.

Created attachment 63688 [details] [review]
Inotify, IMAP uri and beagle-search improvements

With this patch, most of the automatic re-indexing code is in place. Some improvements to the IMAP uri has been made as well. If a Thunderbird window was running when trying to open a mail in beagle-search, nothing would happen. To solve this, the included patch will make beagle-search open a new Thunderbird window each time a mail is opened.

Created attachment 63689 [details] [review]
Inotify, IMAP uri and beagle-search improvements

Oops.. mixed up the patches first time. Sorry...

That's fixed it right up Pierre, thanks!  If I haven't mentioned it before, brilliant work!!!!  5:)

A couple points on the latest version:
* When openning the message in thunderbird, it appears to open the message received just prior to the indexed message.  
* Currently some messages still seem to have difficulty indexing (no subject or from address), but that's been greatly reduced by the current patch.

Knowing however that this is a work in progress, I'm just gonna sit tight on those (and a few random exceptions).  If you need any help trying to figure out what's causing any of them, gimme a shout and I'll get on it!  Thanks again for all your efforts!

(In reply to comment #17)
> That's fixed it right up Pierre, thanks!  If I haven't mentioned it before,
> brilliant work!!!!  5:)
> 
Awsome! Then I can assume that the uri is somewhat correct now. Thanks btw.

> A couple points on the latest version:
> * When openning the message in thunderbird, it appears to open the message
> received just prior to the indexed message.  
Humm, this is not to good. I'll have to dig deeper into this issue.

> * Currently some messages still seem to have difficulty indexing (no subject or
> from address), but that's been greatly reduced by the current patch.
> 
There's some small issues when performing "partial indexing" that resembles your issue a lot. Partial indexing is used when complete mails aren't availble on your hard drive or if a mail can't be fully indexed for some reason. This type of indexing is just "fallback" and full indexing is what you really want. 

Have you enabled "Make the messages in my Inbox available when I am working offline" in Edit->Account Settings...->(some account)->Offline  & Disk Space? If not: try this and let beagle re-index your mails (this will allow for full indexing). The re-indexing part should be done automatically with my latest patch (if beagle is running), but I would recommend you shutdown beagle, remove the old thunderbird index (remove ~/.beagle/Indexes/ThunderbirdIndex) to make sure no old uri's are left from earlier patches and start beagle again.

> Knowing however that this is a work in progress, I'm just gonna sit tight on
> those (and a few random exceptions).  If you need any help trying to figure out
> what's causing any of them, gimme a shout and I'll get on it!  Thanks again for
> all your efforts!
> 
It would be nice if you could upload your latest log files for beagle. I'm particularly interested in those exceptions.

Created attachment 63714 [details]
Beagle exceptions

> Have you enabled "Make the messages in my Inbox available when I am working
> offline" in Edit->Account Settings...->(some account)->Offline  & Disk Space?
> If not: try this and let beagle re-index your mails (this will allow for full
> indexing). The re-indexing part should be done automatically with my latest
> patch (if beagle is running), but I would recommend you shutdown beagle, remove
> the old thunderbird index (remove ~/.beagle/Indexes/ThunderbirdIndex) to make
> sure no old uri's are left from earlier patches and start beagle again.

Yes, I've got all my folders set for offline use, so I should have complete records of all my emails on my harddisk.  There have been maybe a couple of occasions in the past where during the download of a large message thunderbird crashed, so perhaps it's possible that thunderbird broke once during the download of a large message, but if so I've got no idea how to locate it?

I've also been cleaning out my entire .beagle directory when trying out new patches to ensure there's nothing left over from previous attempts.  As I said, many of the partially indexed emails are now appearing fully indexed, but a few still appear without subjects/from addresses.

> It would be nice if you could upload your latest log files for beagle. I'm
> particularly interested in those exceptions.
> 
Please find the recent exceptions attached.  If you need further logs, just let me know and I'll cut out the relevant chunks and post them here...  5:)

Just a very quick note this morning, it's possible that it's only indexing certain mails as out by one, since I seem to have hit the exact email this morning.  My guess would the be that during the indexing a "bad" email was discovered for some reason, couldn't be parsed properly, and then threw the rest of the count out by about one.  Sorry it's so vague, I'll try and do more specific testing when I can, but this week's a little busier than last...

I regret to add that on the box suffering the exceptions, if the beagle daemon is left for many hours it eventually can cause periods of 100% CPU usage (mostly IOWait) that stop most other system activity.  These periods seem to be exacerbated by starting thunderbird, or having thunderbird running.  They again seem to be occurring much less frequently than they had before the inotify, but they're still a cause for concern.  At the moment I'm assuming they're associated with the exceptions previously mentioned and so I'm gonna keep beagle running in anticipation of the next patch.  I'm hooked, keep it up!  5:)

Created attachment 64751 [details] [review]
ThunderbirdQueryable r26

* Added an index cache that will allow for faster indexing and less problems when running Thunderbird 24/7 (less load, faster indexing etc.)
* Adding/removing Thunderbird accounts during beagle runtime now works
* Updated the IMAP uri generation a bit. It now tries to figure out the delimiter used by server.
* Some bugfixes

Just to confirm, this works well for me.  Not a single exception yet in the beagle logs and it's been going since last night.  Still have a couple emails that couldn't be indexed (Email ? from ?), and that's after removing the .beagle dir before use.  I haven't figured out the similarity between the messages that's causing the failure, but I'll investigate it further when I have a chance.  Meanwhile, fantastic work Pierre!  This really makes beagle usable with thunderbird, keep it up!!!  5:)

Hello
I am trying to download the latest patch "ThunderbirdQueryable r26" but I get some binary file (garbage?) when I try to download it. Is it just me?

Jost Diederichs: the patch is surely .tar or .tar.gz or .gz
try:
tar xvfz <file>
tar xvfj <file>
gunzip <file>

yes, it is .gz, thanks

Ok, I run this with a few other patches still pending in a repo at

https://kubasik.net/svn/beagle

for any and all interested.

I have beagle running on Mandriva Cooker with this patch applied. However it doesn't seem to index my mails at all - ThunderbirdIndex gets created, but it remains empty. The only message in logs mentioning Thunderbird is this:

2006-05-22-19-51-44-IndexHelper:060522 1951452093 05677 IndexH DEBUG: Optimizing ThunderbirdIndex
2006-05-22-19-51-44-IndexHelper:060522 1951452105 05677 IndexH DEBUG: ThunderbirdIndex optimized in .00s

Is there something special required how to set the thing up? Is it required to re-create the TB accounts (I hope not)? 

I have my mail downloaded from several POP3 accounts and all stored in Local Folders (one common set, not an extra set per account). 

Jan

Try running this from the command line, then restarting the beagle daemon

beagle-config daemon AllowBackend Thunderbird

I have done it and it makes no difference. 
I have tested also this:

$ beagled --backend +Thunderbird --indexing-test-mode

The command exits right away after printing some text about loading config, so Beagle thinks that everything is indexed already :(

(In reply to comment #28)
> I have beagle running on Mandriva Cooker with this patch applied. However it
> doesn't seem to index my mails at all - ThunderbirdIndex gets created, but it
> remains empty. The only message in logs mentioning Thunderbird is this:
> 2006-05-22-19-51-44-IndexHelper:060522 1951452093 05677 IndexH DEBUG:
> Optimizing ThunderbirdIndex
> 2006-05-22-19-51-44-IndexHelper:060522 1951452105 05677 IndexH DEBUG:
> ThunderbirdIndex optimized in .00s

This is not the Thunderbird backend itself though, these things are called from and managed by the daemon.

I'm quite interested in the log file, since there are debug data that doesn't indicate it's origin (all debug data the Thunderbird backend produces doesn't necessary contain the word "Thunderbird"). My guess is that Thunderbird in Mandriva is using a different location for storing it's data. The binary downloaded from mozilla.org uses ~/.thunderbird and the version shipped with Ubuntu uses ~/.mozilla-thunderbird. So this could be the case. I'll try to create a more generic detection of this later on, to make the backend guess the path in a better way. But some more debug data would help a lot.

> Is there something special required how to set the thing up? Is it required to
> re-create the TB accounts (I hope not)? 

There should not be any other configuration necessary.

> I have my mail downloaded from several POP3 accounts and all stored in Local
> Folders (one common set, not an extra set per account). 

This should work fine.

I finally have some spare time, so I'll start working on the code again and fix some of the remaining issues. I have a version that seems to fix the issue Mike is having (that the backend is failing to index some mails) and some other things are on my TODO as well. Btw, Kevin. Did you forget to do "svn add" on all the files related to the Thunderbird backend before commit? They are not available in the svn repository.

Hey, yeah, I just re-added the files, check out the latest revision from

https://kubasik.net/svn/beagle

If you want, e-mail me an encrypted password request and I'll hook you up with an account (just for fun and revision control until we get this in CVS)

I'm compiling heap-buddy as we speak and wanna do some performance profiling on the thunderbird backend, as atm a long run can get mem usage waaaaay up there. I post my results back here in a bit.

Overall I'm psyched to see this progressing, I was thinking about some way to handle remapping of mails when their positions change, maybe we could consider adding an internal header to each mail? Like a X-Beagle-UID: that we can remap to the thunderbird uri that we need. 

I dunno, I'm just brainstorming at the moment, I've just noticed that reindexing of large folders takes forever, and for about an hour, opening mails can reference the wrong mails. Ok, I'll make this more coherent when I get home, but before you get hacking I wanted to at least throw those 2 considerations out there.

Cool, I'll send you a name and a password. I'm using subversion as well to keep track of my updates - perhaps it's possible to merege them?

Btw, I've removed the IndexCache included in the lastest patch available here and replaced it with a LuceneAccess-class that's _much_ more effecient. It queries the Lucene database directly and only re-index mails that needs to be re-indexed (it vaguely based on LuceneNameResolver in the Filesystem queryable).

(In reply to comment #31)
> I'm quite interested in the log file, since there are debug data that doesn't
> indicate it's origin (all debug data the Thunderbird backend produces doesn't
> necessary contain the word "Thunderbird"). 

The files are attached. I have wiped ~/.beagle and reindexed from scratch. The Thunderbird index is empty, the backend is on. 

> My guess is that Thunderbird in
> Mandriva is using a different location for storing it's data. The binary
> downloaded from mozilla.org uses ~/.thunderbird

Mandriva has the data in ~/.thunderbird as well. In my case it is ~/.thunderbird/e629edx1.default

Created attachment 66075 [details]
Beagle logs from Mandriva

Logs from a clean run of Beagle on Mandriva Cooker (broken Thunderbird patch)

(In reply to comment #35)
> Created an attachment (id=66075) [edit]
> Beagle logs from Mandriva
> 
> Logs from a clean run of Beagle on Mandriva Cooker (broken Thunderbird patch)
> 
Hmm, the file that I actually need is almost empty (the one named <date>-Beagle). Could you run beagle again and pass "--debug" when starting it to make sure it starts in debug mode (which it should anyway)? As I said, I only need the file <date>-Beagle, the other files aren't necessary for now.

Well, the --debug flag was used, it just didn't print anything more. If you look into the other log detailing the progress of the indexer, you will see that the debug was on - there are tons of lines prefixed DEBUG detailing which files were indexed. This is not there if I do not pass --debug to beagle.

Anyway, I can do it again and upload the log.

(In reply to comment #37)
> Well, the --debug flag was used, it just didn't print anything more. If you
> look into the other log detailing the progress of the indexer, you will see
> that the debug was on - there are tons of lines prefixed DEBUG detailing which
> files were indexed. This is not there if I do not pass --debug to beagle.

You are absolutely right. It's very strange it didn't output anything to the *-Beagle file.

> Anyway, I can do it again and upload the log.

Please do, but make sure there's anything useful in that particular file before attaching anything here. 

If it doesn't work, try running beagle in the foreground and pipe the output to a file (or something like that). The daemon should pass some data to stdout when running with --fg IMHO.

It is possible that Mandriva patches their package to lower the default log level in the daemon (but not the helper) by default.  You might want to check into that, because that log is very short.

SUSE does something similar with 10.1, but it works better: logs are limited in the helper as well, and passing in --debug correctly overrides the default.

(In reply to comment #39)
> It is possible that Mandriva patches their package to lower the default log
> level in the daemon (but not the helper) by default.  You might want to check
> into that, because that log is very short.
> 
> SUSE does something similar with 10.1, but it works better: logs are limited in
> the helper as well, and passing in --debug correctly overrides the default.
> 

I just checked the code for Mandriva Cooker and you are right. They are applying a patch that changes the log level to "LogLevel.Error" instead of "LogLevel.Debug". But this can be solved by exporting BEAGLE_DEBUG_FLAG_IS_SET:

export BEAGLE_DEBUG_FLAG_IS_SET=1

This should produce much more debug data.

I have done this, however nothing changed. It seems that the .spew patch hardwires the log level to Error only. 

I am contacting the Mandriva maintainer of Beagle to drop/fix this patch.

The patch we're using in SUSE is here:

http://primates.ximian.com/~joe/beagle-log-level-warn.patch

I'd recommend using that rather than hardcoding it to Error.

Created attachment 66205 [details]
Beagle logs from Mandriva (with debug)

This is the current-Beagle log after full reindexation with --debug and BEAGLE_DEBUG_FLAG_IS_SET=1. The Thunderbird backed seems to run but fails to find anything to index.

Same here, the backend is running but fails to find anything to index.

Debug: Starting backend: 'Thunderbird'
Debug: Starting Thunderbird backend
Debug: Nothing to crawl.
Debug: Done crawling.
Debug: Thunderbird backend done in 0.141044s.

I have the emails in a non-default directory which can be part of the problem. Following Kevin's advice I modified ThunderbirdQueryable.cs to include my Mail path:

dirs.Add (Path.Combine (PathFinder.HomeDir, ".thunderbird"));
dirs.Add (Path.Combine (PathFinder.HomeDir, ".mozilla-thunderbird"));
dirs.Add (Path.Combine (PathFinder.HomeDir, "/media/hda5/_wip/Mail"));

Is this right?

My mails are in the default ~/.thunderbird directory, but beagle still doesn't find them :(

Just installed beagle with Thunderbird Backend in other computer and created a new email account to run some tests.

Storing emails in the default directory works for me there. Some problems though.

-The content of some emails don't get indexed. Don't know why.
-Words with special characters like accents (á,é,í,ó,ú - as in camión), local characters (ñ - as in España(Spain)), don't get indexed when in the Body of the message. When in the Subject, they are indexed. This is very important for languages other than english.
-Problems with attachments. 
*After receiving an email with an attachment, the indexing seems to get stuck sometimes (at least with a .doc file and no backend for .doc files). Maybe.
*The content of mails with attachments is not indexed.

Let me know if you need more info or some concrete test.

(In reply to comment #44)
> Same here, the backend is running but fails to find anything to index.
> 
> Debug: Starting backend: 'Thunderbird'
> Debug: Starting Thunderbird backend
> Debug: Nothing to crawl.
> Debug: Done crawling.
> Debug: Thunderbird backend done in 0.141044s.
> 
> I have the emails in a non-default directory which can be part of the problem.
> Following Kevin's advice I modified ThunderbirdQueryable.cs to include my Mail
> path:
> 
> dirs.Add (Path.Combine (PathFinder.HomeDir, ".thunderbird"));
> dirs.Add (Path.Combine (PathFinder.HomeDir, ".mozilla-thunderbird"));
> dirs.Add (Path.Combine (PathFinder.HomeDir, "/media/hda5/_wip/Mail"));
> 
> Is this right?
> 

Just add
dirs.Add ("/media/hda5/_wip/Mail"); 

Since this is not in your home directory. The other code uses some env-variable fun to determine your home directory, and then appends .thunderbird or .mozilla-thunderbird to it.

(In reply to comment #45)
> My mails are in the default ~/.thunderbird directory, but beagle still doesn't
> find them :(
> 

Is there any chance you started this profile/accounts on an older version of thunderbird and have upgraded? I'm not sure if profile.ini contains anything really personal, but if it doesn't, could you attach yours here?

(In reply to comment #48)
> Is there any chance you started this profile/accounts on an older version of
> thunderbird and have upgraded? I'm not sure if profile.ini contains anything
> really personal, but if it doesn't, could you attach yours here?
> 

Yes, I have my profile for quite some time already (Thunderbird 1.0.x, now 1.5.x).

Here is ~/.thunderbird/profiles.ini:

[General]
StartWithLastProfile=1

[Profile0]
Name=default
IsRelative=1
Path=e629edx1.default
Default=1

(In reply to comment #47)
> (In reply to comment #44)
> > Same here, the backend is running but fails to find anything to index.
> > 
> > Debug: Starting backend: 'Thunderbird'
> > Debug: Starting Thunderbird backend
> > Debug: Nothing to crawl.
> > Debug: Done crawling.
> > Debug: Thunderbird backend done in 0.141044s.
> > 
> > I have the emails in a non-default directory which can be part of the problem.
> > Following Kevin's advice I modified ThunderbirdQueryable.cs to include my Mail
> > path:
> > 
> > dirs.Add (Path.Combine (PathFinder.HomeDir, ".thunderbird"));
> > dirs.Add (Path.Combine (PathFinder.HomeDir, ".mozilla-thunderbird"));
> > dirs.Add (Path.Combine (PathFinder.HomeDir, "/media/hda5/_wip/Mail"));
> > 
> > Is this right?
> > 
> 
> Just add
> dirs.Add ("/media/hda5/_wip/Mail"); 
> 
> Since this is not in your home directory. The other code uses some env-variable
> fun to determine your home directory, and then appends .thunderbird or
> .mozilla-thunderbird to it.
> 

Did that and still the same. Nothing to crawl.
Furthermore, I created a new account and leaved the files in the default folder but nothing happens...

On closing thunderbird though, some errors.

emrys@ubuntu:~$ beagled --fg --allow-backend Thunderbird
Debug: Starting Beagle Daemon (version 0.2.7)
Debug: Running on Mono 1.1.13.6
Debug: Command Line: /usr/local/lib/beagle/BeagleDaemon.exe --fg --allow-backend Thunderbird

(/usr/local/lib/beagle/BeagleDaemon.exe:6225): Gdk-WARNING **: locale not supported by Xlib

(/usr/local/lib/beagle/BeagleDaemon.exe:6225): Gdk-WARNING **: cannot set locale modifiers
Debug: Established a connection to the X server
Debug: Starting main loop
Debug: Starting messaging server
Debug: Starting QueryDriver
Debug: Found 0 backends in /usr/local/lib/beagle/Backends/EvolutionBackends.dll
Debug: Found index helper at /usr/local/lib/beagle/beagled-index-helper
Debug: Found 1 backends in /usr/local/lib/beagle/BeagleDaemonLib.dll
Debug: Reading mapping from filters
Debug: Found 0 user-configured static indexes..
Debug: Waiting 60 seconds before starting queryables
Debug: Starting Scheduler thread
Debug: Starting Inotify threads
Debug: Loading Beagle.Util.Conf+IndexingConfig from indexing.xml
Debug: Loading Beagle.Util.Conf+DaemonConfig from daemon.xml
Debug: Loading Beagle.Util.Conf+SearchingConfig from searching.xml
Debug: Loading Beagle.Util.Conf+NetworkingConfig from networking.xml
Debug: Loading Beagle.Util.Conf+WebServicesConfig from webservices.xml
Debug: Daemon initialization finished after 1.00s
Debug: Starting queryables
Debug: Starting backend: 'Thunderbird'
Debug: Starting Thunderbird backend
Debug: Nothing to crawl.
Debug: Done crawling.
Debug: Thunderbird backend done in 0.275903s.
Warn: Error: Caught exception executing Inotify callbacks
Error: System.FormatException: Index (zero based) must be greater than or equal to zero and less than the size of the argument list.
in <0x004e3> System.String:FormatHelper (System.Text.StringBuilder result, IFormatProvider provider, System.String format, System.Object[] args)
in <0x0002b> System.String:Format (IFormatProvider provider, System.String format, System.Object[] args)
in <0x0000f> System.String:Format (System.String format, System.Object[] args)
in <0x00012> System.IO.TextWriter:Write (System.String format, System.Object[] arg)
in <0x00014> System.IO.TextWriter:WriteLine (System.String format, System.Object[] arg)
in <0x0002e> System.IO.SynchronizedWriter:WriteLine (System.String format, System.Object[] value)
in [0x00072] (at /home/emrys/beagle-svn/Util/Log.cs:203) Beagle.Util.Log:WriteLine (LogLevel level, System.String format, System.Object[] args, System.Exception ex)
in [0x00004] (at /home/emrys/beagle-svn/Util/Log.cs:300) Beagle.Util.Log:Warn (System.String message, System.Object[] args)
in [0x00002] (at /home/emrys/beagle-svn/Util/Logger.cs:80) Beagle.Util.Logger:Warn (System.String message, System.Object[] args)
in [0x00022] (at /home/emrys/beagle-svn/beagled/ThunderbirdQueryable/ThunderbirdIndexer.cs:227) Beagle.Daemon.ThunderbirdQueryable.ThunderbirdIndexer:UpdateAccounts ()
in [0x00096] (at /home/emrys/beagle-svn/beagled/ThunderbirdQueryable/ThunderbirdIndexer.cs:306) Beagle.Daemon.ThunderbirdQueryable.ThunderbirdIndexer:OnInotifyEvent (Watch watch, System.String path, System.String subitem, System.String srcpath, EventType type)
in (wrapper delegate-invoke) System.MulticastDelegate:invoke_void_Inotify/Watch_string_string_string_Inotify/EventType (Beagle.Util.Inotify/Watch,string,string,string,Beagle.Util.Inotify/EventType)
in [0x0010d] (at /home/emrys/beagle-svn/Util/Inotify.cs:638) Beagle.Util.Inotify:SendEvent (Beagle.Util.WatchInfo watched, System.String filename, System.String srcpath, EventType mask)

The biggest problem, as it seems, is the method that parses prefs.js file, which contains account information (the file can be found in ~/.thunderbird/randomname.default/). I'm working on this and will hopefully update Kevin's SVN tree with my latest work at the end of this week. I will include more debug information in this patch to make it easier to find and solve all these basic problems. Other things will have to wait, for now.

Just so this is logged here as well, I have noticed that the thunderbird backend has ceased to index my IMAP accounts, I'm looking into its profile.ini parsing, but I'll wait for your commit before I make any changes.

ps. Sorry that the SVN has been offline the past couple of days, some major thunderstorms have given me more than my share of headaches with power/phone/net.

If you want to see the progress on the ThunderbirdBackend, check out my svn repo at

https://kubasik.net/svn/beagle

While the attached patch works, its getting pretty outdated.

Is Pierre still updating the patches here, or are the patches being mirrored across to this subversion repository?  I've currently got custom ebuilds under gentoo that I can quite easily slot new patches into, but having to diff source builds is going to make the task more cumbersome, so is there anyone who'd be kind enough to post simple patchsets against the official source releases please?  5:)

Almost all the work is currently going on in the SVN repo, however, we are planning a 0.2.7 release soon, and after that, we will probably merge the Thunderbird stuff into HEAD. At which point the whole thing is moot.

However, I have kept all the CVS directories intact in the SVN repo just for this purpose, if you checkout the repo and type cvs diff -u, you will get an official patch against the current HEAD.

Ok cool, I'll give that a go, thanks!  5:)

Created attachment 67879 [details] [review]
Closer to final patch

Heres a very much updated version of the thunderbird backend... like by about 1203983 times... 

NOTE! You must configure with --enable-thunderbird if you want to build it.

Be sure to check the latest patch out.  I've spent lots of time working on mork improvements, speed improvements and better "up to date" indexing. Support has also been extended to support RSS feeds, address book contacts and of course e-mails (which already existed before). NNTP and MoveMail are in the works as well, but are still far from complete.

Since this patch will most likely hit HEAD very shortly, it would be nice with some feedback on it so the most outstanding bugs (that I still haven't find) can be fixed. Please try it out and report any bugs.

Do you also have the patch for configure.am ?

Created attachment 67931 [details] [review]
configure.in fix

This is an addition to the previous patch that will fix configure.in.

Created attachment 68977 [details] [review]
Release Candidate 1

Should be good to commit this, probably still needs some cleanups here and there, but its like 98% awesome. So huge props to Pierre, and hopefully you all will like it!

That patch from #61 is broken, the files that are not in the original beagle are missing. Please add -N to diff next time.

Created attachment 69116 [details] [review]
Release Candidate 2

Ok, this on actaully has all the files. I caught the mistake on the mailing list patch, but forgot to upload this one here *blush*

I don't have any major problems with the patch.  The code which touches Lucene makes me feel a little unclean, but I am hoping it will become unnecessary when we move to a mutable metadata store.

My comments:

There are a handful of things that need to be cleaned up, like stray commented-out lines like this one in Tiles/MailAttachment.cs:

+			//Console.WriteLine (str);

I don't like the way MailMessage.GetClientProcess() works, with the fact that it populates part of the Arguments array but not all of it.  I'd like to see the code refactored to handle populating the array fully.

Shouldn't StorageBase be an abstract class?

In MailIndexableGenerator.MailToIndexable, you get a GMime.Message in there for constructing the indexable.  You need to dispose the message when you're done with it.  They are large unmanaged resources, and the GC has no way of knowing that and so a ton of extra memory will be used.

In Database.ReadAccounts(), you call Database.GetDelimiter() and pass in a new string[] as the args.  It'd be cleaner and avoid an allocation if you declared the parameters in GetDelimiter() as params.

I'm not sure why a string array is being created in one of the functions and passed as an argument to String.Format() in Database.NewUri().  It isn't necessary.  That block of code also doesn't follow the style guidelines (although by and large most of the rest of the code does, thanks).

RSS.cs should probably be named Rss.cs for naming consistency with other parts of the code.

All the changes to Filter.cs should be backed out.

I think for now all the changes to FilterMail.cs should also be backed out.  We probably will want to start snippeting mails, but I'm not sure it's a good idea quite yet.  We need to test this on extremely large volumes of mail to make sure that it'll be feasible.

ContactViewer/Makefile.in and Makefile shouldn't be checked in, they're autogenerated files; Makefile.am should be.

In the contact viewer, having a menu option to open a Mork database is pretty scary from a usability standpoint.  Couldn't we autodetect the locations?

(In reply to comment #64)
> I don't have any major problems with the patch.  The code which touches Lucene
> makes me feel a little unclean, but I am hoping it will become unnecessary when
> we move to a mutable metadata store.
> 
I've mostly based this code on the LuceneNameResolver found in FileSystemQueryable. But it would be nice to replace it with something better.

> There are a handful of things that need to be cleaned up, like stray
> commented-out lines like this one in Tiles/MailAttachment.cs:
> 
> +                       //Console.WriteLine (str);
> 
> I don't like the way MailMessage.GetClientProcess() works, with the fact that
> it populates part of the Arguments array but not all of it.  I'd like to see
> the code refactored to handle populating the array fully.
> 
I've removed all this code in RC3 because it was never meant to be used in practice, only as testing code to make sure Thunderbird loaded correct mail etc. 

The reason the last field in the array is left empty is because it is meant to be "reserved" for the uri. I guess a uri parameter to GetClientProcess () could fix this. But wouldn't it be better to implement a more flexible way to add third-party support through an XML resource or something?

> Shouldn't StorageBase be an abstract class?
> 
Yes it should and now it is.

> In MailIndexableGenerator.MailToIndexable, you get a GMime.Message in there for
> constructing the indexable.  You need to dispose the message when you're done
> with it.  They are large unmanaged resources, and the GC has no way of knowing
> that and so a ton of extra memory will be used.
> 
Sould be fixed now.

> In Database.ReadAccounts(), you call Database.GetDelimiter() and pass in a new
> string[] as the args.  It'd be cleaner and avoid an allocation if you declared
> the parameters in GetDelimiter() as params.
> 
Indeed. Fixed.

> I'm not sure why a string array is being created in one of the functions and
> passed as an argument to String.Format() in Database.NewUri().  It isn't
> necessary.  That block of code also doesn't follow the style guidelines
> (although by and large most of the rest of the code does, thanks).
> 
Uhm, yeah... That's not how it supposed to be... Fixed this.

> RSS.cs should probably be named Rss.cs for naming consistency with other parts
> of the code.
> 
Yepp. Renamed this file and NNTP.cs to Nntp.cs as well.

> All the changes to Filter.cs should be backed out.
> 
> I think for now all the changes to FilterMail.cs should also be backed out.  We
> probably will want to start snippeting mails, but I'm not sure it's a good idea
> quite yet.  We need to test this on extremely large volumes of mail to make
> sure that it'll be feasible.
> 
Removed these changes as well. This is Kevin's code so he might have something to add here?

> ContactViewer/Makefile.in and Makefile shouldn't be checked in, they're
> autogenerated files; Makefile.am should be.
> 
I think it just slipped through when Kevin made the patch. Removed it anyhow.

> In the contact viewer, having a menu option to open a Mork database is pretty
> scary from a usability standpoint.  Couldn't we autodetect the locations?
> 
The location is auto-detected when loading a file. I should have removed this Open-menu before the code was committed. Removed now though.

Created attachment 69325 [details] [review]
Release Candidate 3

(In reply to comment #65)
> (In reply to comment #64)
> > I don't have any major problems with the patch.  The code which touches Lucene
> > makes me feel a little unclean, but I am hoping it will become unnecessary when
> > we move to a mutable metadata store.
> > 
> I've mostly based this code on the LuceneNameResolver found in
> FileSystemQueryable. But it would be nice to replace it with something better.
> 
> > There are a handful of things that need to be cleaned up, like stray
> > commented-out lines like this one in Tiles/MailAttachment.cs:
> > 
> > +                       //Console.WriteLine (str);
> > 
> > I don't like the way MailMessage.GetClientProcess() works, with the fact that
> > it populates part of the Arguments array but not all of it.  I'd like to see
> > the code refactored to handle populating the array fully.
> > 
> I've removed all this code in RC3 because it was never meant to be used in
> practice, only as testing code to make sure Thunderbird loaded correct mail
> etc. 
> 
> The reason the last field in the array is left empty is because it is meant to
> be "reserved" for the uri. I guess a uri parameter to GetClientProcess () could
> fix this. But wouldn't it be better to implement a more flexible way to add
> third-party support through an XML resource or something?
> 
> > Shouldn't StorageBase be an abstract class?
> > 
> Yes it should and now it is.
> 
> > In MailIndexableGenerator.MailToIndexable, you get a GMime.Message in there for
> > constructing the indexable.  You need to dispose the message when you're done
> > with it.  They are large unmanaged resources, and the GC has no way of knowing
> > that and so a ton of extra memory will be used.
> > 
> Sould be fixed now.
> 
> > In Database.ReadAccounts(), you call Database.GetDelimiter() and pass in a new
> > string[] as the args.  It'd be cleaner and avoid an allocation if you declared
> > the parameters in GetDelimiter() as params.
> > 
> Indeed. Fixed.
> 
> > I'm not sure why a string array is being created in one of the functions and
> > passed as an argument to String.Format() in Database.NewUri().  It isn't
> > necessary.  That block of code also doesn't follow the style guidelines
> > (although by and large most of the rest of the code does, thanks).
> > 
> Uhm, yeah... That's not how it supposed to be... Fixed this.
> 
> > RSS.cs should probably be named Rss.cs for naming consistency with other parts
> > of the code.
> > 
> Yepp. Renamed this file and NNTP.cs to Nntp.cs as well.
> 
> > All the changes to Filter.cs should be backed out.
> > 
> > I think for now all the changes to FilterMail.cs should also be backed out.  We
> > probably will want to start snippeting mails, but I'm not sure it's a good idea
> > quite yet.  We need to test this on extremely large volumes of mail to make
> > sure that it'll be feasible.
> > 
> Removed these changes as well. This is Kevin's code so he might have something
> to add here?
> 
This wasn't supposed to be in the final patch *blush* I just found my note about all the things I meant to remove before this was ready for CVS..
> > ContactViewer/Makefile.in and Makefile shouldn't be checked in, they're
> > autogenerated files; Makefile.am should be.
> > 
> I think it just slipped through when Kevin made the patch. Removed it anyhow.
> 
> > In the contact viewer, having a menu option to open a Mork database is pretty
> > scary from a usability standpoint.  Couldn't we autodetect the locations?
> > 
> The location is auto-detected when loading a file. I should have removed this
> Open-menu before the code was committed. Removed now though.
> 
This was in that note, as well as the stuff in FilterMail =/ sorry. Thought I had this ready, but I guess I wasn't paying that much attention...

Either way, hope this looks good :)

(In reply to comment #66)
> Created an attachment (id=69325) [edit]
> Release Candidate 3

Something appears to be wrong with the patch (probably the mime type).  It's listed as a patch but it's a bunch of binary gunk when I try to download it.  If it is compressed, can you fix the mime type on it appropriately?

Created attachment 69330 [details] [review]
Uncompressed RC 3

Ok, this one is decompressed

Yes, the patch was compressed with gzip to save some space. Shouldn't the bugzilla automatically manage to figure the correct mimetype out?

Unfortunatly, when you click patch, bugzilla automaticaly assumes the mime-type text/plain =/.

Only thing in this latest patch is that the MONO_GAC_PREFIX part of beagle-contactviewer.in should be removed.

I think it's fine to go in.  Pierre, do you have a CVS account?

(In reply to comment #72)
> Only thing in this latest patch is that the MONO_GAC_PREFIX part of
> beagle-contactviewer.in should be removed.

Shouldn't be too hard to fix.

> I think it's fine to go in.  Pierre, do you have a CVS account?
> 

Cool! No, I don't have a CVS account.

Pierre: Can you sign up for one?  The info on how to do so is here:

http://developer.gnome.org/doc/policies/accounts/requesting.html

Please use me as a reference.  I'd greatly prefer if you could maintain the backend directly going forward.

Ok, I committed this, please open new bugs as they arise. 

Kevin/Pierre,
  A quick question. ContactViewer is a GUI and so needs the Glade# and other GUI libs. So, ideally it should only be enabled under ENABLE_GUI. I think the thunderbird backend can still be used w/out contactviewer, but probably it wont be terribly helpful. So for users building with --disable-gui, should the thunderbird backend be totally disabled or only contact_viewer disabled ?
 (I am temporarily patching the CVS to re-enable building with --disable-gui.)

- dBera

This is probably the way to go dependency-wise (in case it should be possible to build the daemon w/o GTK# some day), but wouldn't it be better to disable contact indexing if the contact viewer isn't enabled? Without the contact viewer it's not possible to view contacts which probably would end up with pointless bug reports.

Well, considering all max's new metadata work, I think we will want to always leave the mail/contact list backends active for those purposes, but maybe we should just add an intelligent default. (like if no glade# we just open the thunderbird address book when the contact is clicked on? its not perfect, but at least were pointing in the right direction...)

As long as "something" happens when a user selects a contact, I will be happy. Maybe check if beagle-contactviewer exists in PATH and run it if it does, otherwise fallback to "thunderbird -addressbook" (basically what you are suggesting)?

That sounds peachy to me, I can try to make it happen tonight, or if someone else wants to get on it before then, feel free.