GNOME Bugzilla – Bug 762226
tracker broken if sqlite 3.11.0 is installed.
Last modified: 2016-03-01 16:45:59 UTC
hello. tracker 1.7.3 is completely broken for me. "tracker search something" ould not get search results, GDBus.Errorrg.freedesktop.Tracker1.SparqlError.Internal: unknown tokenizer: TrackerTokenizer http://pastebin.com/raw/ZPWWicLd this is what journald says this is under sqlite 3.11 downgrading to 3.10.2 fixes this. This is only if the initial database was created while sqlite 3.11 is installed(fresh settings/profile).
I email the upstream Sqlite developer, Richard Hipp. He replied: "Go to https://www.sqlite.org/releaselog/3_11_0.html and read the single bullet under "Backwards Compatibility". I think it applies to you. By fixing this, you will also fix a security issue in your application." So if I understand him correctly, tracker needs to be modified to not need SQLITE_ENABLE_FTS3_TOKENIZER to function correctly?
We can't just do that without dropping several of our FTS features. The right fix is updating to fts5, which offers a different method to implement the custom tokenizer we need. The only thing you can do in the mean time is compiling your sqlite with SQLITE_ENABLE_FTS3_TOKENIZER, none of those security concerns apply to Tracker, although they might concern other applications...
FTS doesn't very well in tracker at the moment anyway. If I do 'tracker search MaryPoppins', tracker shows the resultant document with the word MaryPoppins highlighted in red. Then if I delete a word MaryPoppins from that text document, tracker updates and now if I run 'tracker search MaryPoppins', tracker shows the document that used to have the word MaryPoppins but without the actual word highlighted in red. It shouldn't show results at all. Will updating to FTS5 fix this?
*** Bug 762577 has been marked as a duplicate of this bug. ***
As discussed on irc a few days ago: the options here are to either port tracker to fts5 or ship a bundled sqlite with fts3 enabled (which is safe in trackers case). Carlos is looking into the first, failing that, will do the second.
If the first option fails, another option is to detect if the needed feature is enabled in sqlite like firefox does https://github.com/mozilla/gecko-dev/blob/aurora/configure.in#L6675 and print an error saying "you need so and so in sqlite to compile tracker".
(In reply to Hussam Al-Tayeb from comment #6) > If the first option fails, another option is to detect if the needed feature > is enabled in sqlite like firefox does > https://github.com/mozilla/gecko-dev/blob/aurora/configure.in#L6675 > and print an error saying "you need so and so in sqlite to compile tracker". If I correctly understood the problem (and what Carlos said), we cannot recommend people (eg., distributors) to enable SQLITE_ENABLE_FTS3_TOKENIZER because it can have security implications for other applications.
(In reply to Debarshi Ray from comment #7) > (In reply to Hussam Al-Tayeb from comment #6) > > If the first option fails, another option is to detect if the needed feature > > is enabled in sqlite like firefox does > > https://github.com/mozilla/gecko-dev/blob/aurora/configure.in#L6675 > > and print an error saying "you need so and so in sqlite to compile tracker". > > If I correctly understood the problem (and what Carlos said), we cannot > recommend people (eg., distributors) to enable SQLITE_ENABLE_FTS3_TOKENIZER > because it can have security implications for other applications. Yes. That's exactly it. I'm targeting 1.7.3+1/3.19.91 for the fix. (In reply to Hussam Al-Tayeb from comment #3) > FTS doesn't very well in tracker at the moment anyway. > If I do 'tracker search MaryPoppins', tracker shows the resultant document > with the word MaryPoppins highlighted in red. > > Then if I delete a word MaryPoppins from that text document, tracker updates > and now if I run 'tracker search MaryPoppins', tracker shows the document > that used to have the word MaryPoppins but without the actual word > highlighted in red. > > It shouldn't show results at all. > > Will updating to FTS5 fix this? Well... I've seen why this may happen, however it will not get fixed because we update to fts5, but because this code is being revisited. No need to file a different bug this time, but otherwise it's better to not feature creep bugs :).
(In reply to Carlos Garnacho from comment #8) > Yes. That's exactly it. I'm targeting 1.7.3+1/3.19.91 for the fix. Monday? That's fast.... Since this affects F23, we'll either have to be that fast, or else revert the recent sqlite3 upgrade.
(In reply to Michael Catanzaro from comment #9) > (In reply to Carlos Garnacho from comment #8) > > Yes. That's exactly it. I'm targeting 1.7.3+1/3.19.91 for the fix. > > Monday? That's fast.... Considering I mostly work on Tracker during weekends, that's plenty of time :P. I've got the fts5 work partially done, too... it basically depends with how comfortable I'm with the patch considering this late stage (by all accounts, it will be big). > > Since this affects F23, we'll either have to be that fast, or else revert > the recent sqlite3 upgrade. As Matthias said in comment #5, my options are either fts5, or including an static copy of sqlite for the time being. So there's a fallback plan even if the fts5 option doesn't pan out yet. Thing is, sqlite >= 3.11 is not going away... I should at least add a configure time check across latest stables, too...
Renaming this bug to reflect the fact that tracker breaks also for upgrades to sqlite 3.11.0.
I've at the moment pushed the FTS5 approach to https://git.gnome.org/browse/tracker/log/?h=wip/fts5 Despite the big change (most of that is deleting our internal FTS3/4 module copy, and adding a FTS5 one) I could make the change seamless enough, things should just work after installing and letting tracker-store restart. Also I were able to iron out all API and SQL level incompatibilities, this should be completely transparent to apps using libtracker-sparql. The only caveat is that the old fts table is left stale in the database. In order to drop it we need to register the tokenizer we created the table with, and we can't do that because we lack fts3_tokenizer(?,?)... It is worth pointing out that this table info consist just of tokenization info (the text itself is stored elsewhere), so at worst it'll mean a few megs wasted in ~/.cache/tracker/meta.db.
I tried compiling from wip/fts5 branch. In file included from tracker-fts-tokenizer.c:30:0: ../../src/libtracker-data/tracker-data.h:43:34: fatal error: tracker-sparql-query.h: No such file or directory compilation terminated. That file doesn't exist in the source tree.
That file is generated from vala sources, I've fixed the include that required its existence too early.
moving off the tracker since the main issue is fixed
wip/fts5 is now in master, and released as 1.7.4