After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 762226 - tracker broken if sqlite 3.11.0 is installed.
tracker broken if sqlite 3.11.0 is installed.
Status: RESOLVED FIXED
Product: tracker
Classification: Core
Component: FTS
1.7.x
Other Linux
: Normal critical
: ---
Assigned To: tracker-general
tracker-general
: 762577 (view as bug list)
Depends on:
Blocks:
 
 
Reported: 2016-02-17 21:49 UTC by Hussam Al-Tayeb
Modified: 2016-03-01 16:45 UTC
See Also:
GNOME target: ---
GNOME version: ---



Description Hussam Al-Tayeb 2016-02-17 21:49:58 UTC
hello. tracker 1.7.3 is completely broken for me. "tracker search something" ould not get search results, GDBus.Errorrg.freedesktop.Tracker1.SparqlError.Internal: unknown tokenizer: TrackerTokenizer

http://pastebin.com/raw/ZPWWicLd this is what journald says

this is under sqlite 3.11
downgrading to 3.10.2 fixes this.

This is only if the initial database was created while sqlite 3.11 is installed(fresh settings/profile).
Comment 1 Hussam Al-Tayeb 2016-02-17 22:56:07 UTC
I email the upstream Sqlite developer, Richard Hipp. He replied:

"Go to https://www.sqlite.org/releaselog/3_11_0.html and read the
single bullet under "Backwards Compatibility".  I think it applies to
you.  By fixing this, you will also fix a security issue in your
application."


So if I understand him correctly, tracker needs to be modified to not need  SQLITE_ENABLE_FTS3_TOKENIZER to function correctly?
Comment 2 Carlos Garnacho 2016-02-18 17:22:00 UTC
We can't just do that without dropping several of our FTS features. The right fix is updating to fts5, which offers a different method to implement the custom tokenizer we need.

The only thing you can do in the mean time is compiling your sqlite with SQLITE_ENABLE_FTS3_TOKENIZER, none of those security concerns apply to Tracker, although they might concern other applications...
Comment 3 Hussam Al-Tayeb 2016-02-24 22:22:32 UTC
FTS doesn't very well in tracker at the moment anyway. 
If I do 'tracker search MaryPoppins', tracker shows the resultant document with the word MaryPoppins highlighted in red.

Then if I delete a word MaryPoppins from that text document, tracker updates and now if I run 'tracker search MaryPoppins', tracker shows the document that used to have the word MaryPoppins but without the actual word highlighted in red.

It shouldn't show results at all.

Will updating to FTS5 fix this?
Comment 4 Carlos Garnacho 2016-02-26 02:04:37 UTC
*** Bug 762577 has been marked as a duplicate of this bug. ***
Comment 5 Matthias Clasen 2016-02-26 13:22:22 UTC
As discussed on irc a few days ago: the options here are to either port tracker to fts5 or ship a bundled sqlite with fts3 enabled (which is safe in trackers case). Carlos is looking into the first, failing that, will do the second.
Comment 6 Hussam Al-Tayeb 2016-02-26 13:58:29 UTC
If the first option fails, another option is to detect if the needed feature is enabled in sqlite like firefox does 
https://github.com/mozilla/gecko-dev/blob/aurora/configure.in#L6675
and print an error saying "you need so and so in sqlite to compile tracker".
Comment 7 Debarshi Ray 2016-02-26 14:11:04 UTC
(In reply to Hussam Al-Tayeb from comment #6)
> If the first option fails, another option is to detect if the needed feature
> is enabled in sqlite like firefox does 
> https://github.com/mozilla/gecko-dev/blob/aurora/configure.in#L6675
> and print an error saying "you need so and so in sqlite to compile tracker".

If I correctly understood the problem (and what Carlos said), we cannot recommend people (eg., distributors) to enable SQLITE_ENABLE_FTS3_TOKENIZER because it can have security implications for other applications.
Comment 8 Carlos Garnacho 2016-02-26 14:26:18 UTC
(In reply to Debarshi Ray from comment #7)
> (In reply to Hussam Al-Tayeb from comment #6)
> > If the first option fails, another option is to detect if the needed feature
> > is enabled in sqlite like firefox does 
> > https://github.com/mozilla/gecko-dev/blob/aurora/configure.in#L6675
> > and print an error saying "you need so and so in sqlite to compile tracker".
> 
> If I correctly understood the problem (and what Carlos said), we cannot
> recommend people (eg., distributors) to enable SQLITE_ENABLE_FTS3_TOKENIZER
> because it can have security implications for other applications.

Yes. That's exactly it. I'm targeting 1.7.3+1/3.19.91 for the fix.


(In reply to Hussam Al-Tayeb from comment #3)
> FTS doesn't very well in tracker at the moment anyway. 
> If I do 'tracker search MaryPoppins', tracker shows the resultant document
> with the word MaryPoppins highlighted in red.
> 
> Then if I delete a word MaryPoppins from that text document, tracker updates
> and now if I run 'tracker search MaryPoppins', tracker shows the document
> that used to have the word MaryPoppins but without the actual word
> highlighted in red.
> 
> It shouldn't show results at all.
> 
> Will updating to FTS5 fix this?

Well... I've seen why this may happen, however it will not get fixed because we update to fts5, but because this code is being revisited. No need to file a different bug this time, but otherwise it's better to not feature creep bugs :).
Comment 9 Michael Catanzaro 2016-02-26 15:49:00 UTC
(In reply to Carlos Garnacho from comment #8)
> Yes. That's exactly it. I'm targeting 1.7.3+1/3.19.91 for the fix.

Monday? That's fast....

Since this affects F23, we'll either have to be that fast, or else revert the recent sqlite3 upgrade.
Comment 10 Carlos Garnacho 2016-02-26 16:05:11 UTC
(In reply to Michael Catanzaro from comment #9)
> (In reply to Carlos Garnacho from comment #8)
> > Yes. That's exactly it. I'm targeting 1.7.3+1/3.19.91 for the fix.
> 
> Monday? That's fast....

Considering I mostly work on Tracker during weekends, that's plenty of time :P. I've got the fts5 work partially done, too... it basically depends with how comfortable I'm with the patch considering this late stage (by all accounts, it will be big).

> 
> Since this affects F23, we'll either have to be that fast, or else revert
> the recent sqlite3 upgrade.

As Matthias said in comment #5, my options are either fts5, or including an static copy of sqlite for the time being. So there's a fallback plan even if the fts5 option doesn't pan out yet. Thing is, sqlite >= 3.11 is not going away...

I should at least add a configure time check across latest stables, too...
Comment 11 Michael Catanzaro 2016-02-26 23:05:17 UTC
Renaming this bug to reflect the fact that tracker breaks also for upgrades to sqlite 3.11.0.
Comment 12 Carlos Garnacho 2016-02-29 13:22:28 UTC
I've at the moment pushed the FTS5 approach to https://git.gnome.org/browse/tracker/log/?h=wip/fts5

Despite the big change (most of that is deleting our internal FTS3/4 module copy, and adding a FTS5 one) I could make the change seamless enough, things should just work after installing and letting tracker-store restart.

Also I were able to iron out all API and SQL level incompatibilities, this should be completely transparent to apps using libtracker-sparql.

The only caveat is that the old fts table is left stale in the database. In order to drop it we need to register the tokenizer we created the table with, and we can't do that because we lack fts3_tokenizer(?,?)... It is worth pointing out that this table info consist just of tokenization info (the text itself is stored elsewhere), so at worst it'll mean a few megs wasted in ~/.cache/tracker/meta.db.
Comment 13 Hussam Al-Tayeb 2016-02-29 14:13:00 UTC
I tried compiling from wip/fts5 branch.

In file included from tracker-fts-tokenizer.c:30:0:
../../src/libtracker-data/tracker-data.h:43:34: fatal error: tracker-sparql-query.h: No such file or directory
compilation terminated.

That file doesn't exist in the source tree.
Comment 14 Carlos Garnacho 2016-02-29 15:18:16 UTC
That file is generated from vala sources, I've fixed the include that required its existence too early.
Comment 15 Matthias Clasen 2016-03-01 16:31:00 UTC
moving off the tracker since the main issue is fixed
Comment 16 Carlos Garnacho 2016-03-01 16:45:59 UTC
wip/fts5 is now in master, and released as 1.7.4