Hello,

I have the same problem here on Ubuntu 7.10 with Tracker 6.3

Patch proposed (among others) to the mailing-list [1].

Mathieu

[1] http://mail.gnome.org/archives/tracker-list/2007-December/msg00084.html

Thunderbird support is broken since we rewrote the indexer modules API, and we only reimplemented the evolution module.

There is the old code available in the svn, and some contributor can help to implement a new module. 

I change the subject of the bug and move it to component "new formats".

As I wasn't able to find any up to date information about the current status and as this seems to be the newest post, I would like to ask kindly what's about the Thunderbird integration.
Is it possible to index with tracker my Thunderbird mails and dates (using the addon lightning)? I really would love it. If it isn't possible by default, is there anywhere a current Thunderbird addon working under Ubuntu?

I'm using tracker 0.6.6 shipped with Xubuntu 8.10 but would also be able to switch to 0.6.90 or use the latest svn checkout.

Simon

Can the reporter of this bug set the _version_ so we can target bugs to work on more easily please. We can also have an idea about if these bugs are likely to be obsolete too.

If you don't set the version, we are less likely to look at them.

Sorry for the spam, but I don't want to say this 88 times on each bug :)

(In reply to comment #0)
> Hello,
> 
> I'm running debian "unstable" with thunderbird (aka icedove) installed.  I
> would like to index my email but unfortunatly, the option is disabled in the
> tracker preferences dialog box.  I'm guessing it might be because thunderbird
> has been renamed to icedove on debian.
> 
> Marek 
> 
> PS I'm running tracker 6.3

I would love to have my thunderbird email indexed too, however, we don't support this right now. The preferences dialog used to have an option for it because it was going to be supported at some point. That's not actually happened though as yet.

Is anyone interested in taking on such a patch? It would rock if that was the case! :)

Hello,

It's a killer feature for Tracker, happily Thunderbird 3 has an integrated indexer. But it will be more practice to have a search center like tracker.

Thanks for your good work.

LGDN.

I would also really like to have this feature.  For me it's TB 3.1 and tracker whatever...

Thanks matt

And what about it...?

Any news about Thunderbird support in Tracker ?

Yes actually :)

Adrien has done a superb job of making the latest TB work (v5 and above only unfortunately). The work is available here:

  http://git.mymadcat.com/index.php/p/thunderbird-tracker/source/tree/master/

There are plans to merge it to the Tracker code base soon. It may make it into the next tracker 0.11.x release. Not sure it will be in 0.10.x yet.

CCing Adrien.

Note that the current version works with both Tracker 0.10 and 0.11, if you want to give it a try. If you want to report issues, don't forget to CC me on the bug :)

(In reply to comment #11)
> Yes actually :)
> 
> Adrien has done a superb job of making the latest TB work (v5 and above only
> unfortunately). The work is available here:
> 
>   http://git.mymadcat.com/index.php/p/thunderbird-tracker/source/tree/master/
OK, I installed it in Shredder (Thunderbird-trunck 7.0a1). I can access the preference panel (scan delay is 0 ms) so it should be ok.

Now, how do I tell tracker to use it ?

I haven't tested 7.0 yet, I hope it works... You should see in the statusbar of the main window a small label "Indexing" with a count of messages indexed. This count should decrease as the indexing goes one (known bug: sometimes it stops, restart TB and it'll pick up where it stopped).
From there, you can use tracker-needle (in tracker 0.10) to search mail, Martyn can confirm if needle in 0.10 supports mail search.
Note that I now have to add a way to tell TB to display a mail when "opened" from needle... As you see, it's still a work in progress :)
Feel free to drop by on IRC if something does not work.

(In reply to comment #14)
> You should see in the statusbar of the main window a small label "Indexing" with a count of messages indexed.
It says "Cannot initialize Tracker"

tracker-info --version says it is Tracker 0.8.17.

tracker-needle is not installed, and cannot be found in the repositories for Ubuntu natty 64 bits.

Oh, as precised in comment 12, you need Tracker 0.10 or 0.11 (0.10 is stable, 0.8 is old stable, not maintained anymore) for it to work. There might be a PPA somewhere with recent packages, again, maybe Martyn knows (I'm not a Ubuntu user myself).

(In reply to comment #16)
> Oh, as precised in comment 12, you need Tracker 0.10 or 0.11 (0.10 is stable,
> 0.8 is old stable, not maintained anymore) for it to work.
Done, I upgraded it to version 0.10.3 (via PPA https://launchpad.net/~matthaeus123/+archive/ppa/)

The new version is running fine, it looks like: I can find documents based on in-file searches.

In Thunderbird, after a slow decreasing counter (As mentioned, it often freeze so I restarted Thunderbird a lot of times), the error message is now replaced by "Not indexing".

Now, I have some questions about the global tracker setup:
	- should I add Thunderbird's profile into Tracker's source list of directories for indexing, or loading the addon in Thunderbird is enough?
	- how does tracker react to encrypted email? Does it index the decrypted email or does it skip it?
	- is there a way to tell Tracker not to index some IMAP repositories (like the trash, the junk, and some non-relevant mailing list folders)?
	- does Tracker indexes the attached files in the emails?
	- when I delete an email, how long does it take for Tracker to update its database accordingly?
	- when searching for something basic (like my name) in the emails, the date are incorrect. How can I fix it? See the example below :

___
 $ tracker-search --emails Clauzel
Courriels:
  1970-01-15T15:56:47Z, (null)
  1970-01-16T02:19:18Z, :)
  1970-01-16T02:19:19Z, :)
  1970-01-16T01:34:36Z, 2 avril Re: Opération 1er avril
  1970-01-16T01:34:37Z, 2 avril Re: Opération 1er avril
  1970-01-16T01:35:26Z, 2 avril Re: Opération 1er avril
___

The results I get look correct: the emails indeed contain what I am searching for. So it looks very promising to me. Good job!

OK, never heard of this PPA, but whatever works :)
This freeze thing is really annoying, that said here it indexes around 5-10 mails per second, and I have some ideas on how to make it faster (just not implemented yet). I so far have strictly no idea of why the indexing gets stuck, any particular sign you might observe and that might help debugging is welcome :)

To answer your questions:
- No, you should not add TB's dir to Tracker's conf. The TB plugin inserts the mails directly into Tracker, so there is no need to tell the FS miner to go there too.
- Encrypted mails, good question... I use TB's functions to get the mail contents, I have no idea what it returns for encrypted emails.
- There is no blacklisting of directories so far. Adding a list of folders to skip might indeed make sense, I'll have to bake a small UI for that.
- I wouldn't expect attached files to be indexed properly at this stage, since I do no MIME parsing at all. I think the TB function I use only returns the plain text content, ie what you'd see in UI. Again, one feature worth investigating.
- Assuming you're using git master, new/moved/deleted emails should be indexed immediatly, ie for those events we prepend to the work queue instead of appending. That of course supposes indexing didn't get stuck :/
- About the date: not sure what you're seeing here... Can you run this command:
  tracker-sparql -q 'SELECT ?date ?title {?m nmo:receivedDate ?date ; nmo:messageSubject ?title ; fts:match "Clauzel*"}'
  and check if the dates are correct there? There's also a known bug that I always store the date as "received date", whereas for "Sent" folder I should use "sent date".

Thanks for the feedback, that helps a lot :)

(In reply to comment #18)

> - Encrypted mails, good question... I use TB's functions to get the mail
> contents, I have no idea what it returns for encrypted emails.
I cannot help you on this, except by giving my opinion: encrypted emails must not be indexed. As they are supposed to be private, exposing them would be Bad.


> - About the date: not sure what you're seeing here... Can you run this command:
>   tracker-sparql -q 'SELECT ?date ?title {?m nmo:receivedDate ?date ;
> nmo:messageSubject ?title ; fts:match "Clauzel*"}'
>   and check if the dates are correct there? There's also a known bug that I
> always store the date as "received date", whereas for "Sent" folder I should
> use "sent date".
The result I get is:
___
$ tracker-sparql -q 'SELECT ?date ?title {?m nmo:receivedDate ?date ; nmo:messageSubject ?title ; fts:match "Clauzel*"}'
Résultats:
  1970-01-16T02:00:01Z, Acteurs de l'éco
  1970-01-16T02:00:54Z, Confirmation de prise en compte
  [...]
___

So I would say that it is the same result: bogus date. In Thunderbird, all the emails' date are OK.

Problem found: new emails are not indexed by Tracker, unless I restart Thunderbird. Then I can see "XX items remaining", and only when the indexing is complete can I search for content inside the new emails.

The expected behaviour would be on-the-fly indexing of incoming emails.

(In reply to comment #19)
> (In reply to comment #18)
> 
> > - Encrypted mails, good question... I use TB's functions to get the mail
> > contents, I have no idea what it returns for encrypted emails.
> I cannot help you on this, except by giving my opinion: encrypted emails must
> not be indexed. As they are supposed to be private, exposing them would be Bad.
> 

Actually, does TB even deal with encrypted mails natively, or do you use enigmail?

> 
> > - About the date: not sure what you're seeing here... Can you run this command:
> >   tracker-sparql -q 'SELECT ?date ?title {?m nmo:receivedDate ?date ;
> > nmo:messageSubject ?title ; fts:match "Clauzel*"}'
> >   and check if the dates are correct there? There's also a known bug that I
> > always store the date as "received date", whereas for "Sent" folder I should
> > use "sent date".
> The result I get is:
> ___
> $ tracker-sparql -q 'SELECT ?date ?title {?m nmo:receivedDate ?date ;
> nmo:messageSubject ?title ; fts:match "Clauzel*"}'
> Résultats:
>   1970-01-16T02:00:01Z, Acteurs de l'éco
>   1970-01-16T02:00:54Z, Confirmation de prise en compte
>   [...]
> ___
> 
> So I would say that it is the same result: bogus date. In Thunderbird, all the
> emails' date are OK.

Hmm interesting I'll have to check that

(In reply to comment #20)
> Problem found: new emails are not indexed by Tracker, unless I restart
> Thunderbird. Then I can see "XX items remaining", and only when the indexing is
> complete can I search for content inside the new emails.
> 
> The expected behaviour would be on-the-fly indexing of incoming emails.

The git master of the plugin was doing it at least with TB 5, I'll have to check with 7. Will comment on this bug when I get more info.

(In reply to comment #21)
> (In reply to comment #19)
> > (In reply to comment #18)
> > 
> > > - Encrypted mails, good question... I use TB's functions to get the mail
> > > contents, I have no idea what it returns for encrypted emails.
> > I cannot help you on this, except by giving my opinion: encrypted emails must
> > not be indexed. As they are supposed to be private, exposing them would be Bad.
> > 
> 
> Actually, does TB even deal with encrypted mails natively, or do you use
> enigmail?
I use Enigmail, version 1.2a2pre
I don't know how it works internally.

> > So I would say that it is the same result: bogus date. In Thunderbird, all the
> > emails' date are OK.
> 
> Hmm interesting I'll have to check that
Good luck :)

OK, about the message contents: apparently when we get the email, the contents are not yet available. I'll have to check how I can listen to the "contents are now available" flag.

Oh, and the dates were completely busted here too, since in JS you create Date objects from milliseconds, not seconds, which I assumed :) That's fixed in master.

(In reply to comment #24)

> Oh, and the dates were completely busted here too, since in JS you create Date
> objects from milliseconds, not seconds, which I assumed :) That's fixed in
> master.
I did a pull and installed the new version. Then I did a hard reset on Tracker (tracker-control --hard-reset).

Thunderbird says it is indexing my emails. I have the feeling that the process is slower than the first time, and more subject to freezes. But I don't have evidences, just this feeling.

The good news is that the new version works fine: the dates are now correct.
___
$ tracker-search --emails Clauzel
Courriels:
  2011-06-30T09:21:11Z, Re: à propos du modèle de traces
___

Good job! :)

It seems that I did found a way to prevent indexing freezes.

In the Trackerbird preference panel, I did setup the delay to 1000. Then I closed Thunderbird and restarted it. Then I left Thunderbird alone, not touching it. The counter went quickly from 3k to zero, at a rate of about 10 emails per second.

I verified the content of the index with several searches, and tracker did find the appropriate emails.

However, after quitting and restarting a few time Thunderbird, it started again to index the 3k emails in my profile. And this time, freezing a lot, and sometime just saying "Not indexing" but at the next restart resuming to index.

But Tracker's database still contain the emails, because I can any of them even if Thunderbird is saying it is indexing them.

Hmm, so I would also be misusing the timers in javascript? :p
About reindexing, what the plugin does is that it has an internal cache to know what messages is has already indexed (a sqlite db in TB's extensions dir). Unless that cache got corrupted somehow, it shouldn't start reindexing from scratch. I'll see how I can change the timer mechanism in the queue so that it's more reliable.

I have a new strange problem: the indexing counter is stalled to 56. Whatever I do (cleaning tracker's database and removing trackerbird's database), at the of the indexing process I am always coming back to having the counter showing « 56 items remaining »

Stalled indexing should be fixed now, brave testers, you can git pull and give feedback :)

I fixed a quite bad memory issue with TB indexing, and anyway the plugin is now in Tracker's source tree... So I guess we can finally close this bug. Among the still known issues is the fact that the prefs window is sort of broken, you can't pick what folders to index, etc. but let's track that in different bugs.

See commit e62ada3c57253de173b4357cbf81e0afbf4bce46 for TB support.