After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 785883 - tracker search doesn't work with sqlite 3.20.0 due to changes in fts5 interfaces
tracker search doesn't work with sqlite 3.20.0 due to changes in fts5 interfaces
Status: RESOLVED FIXED
Product: tracker
Classification: Core
Component: General
1.99.x
Other Linux
: Normal blocker
: ---
Assigned To: tracker-general
tracker-general
: 785876 786130 (view as bug list)
Depends on:
Blocks:
 
 
Reported: 2017-08-06 13:24 UTC by jaha
Modified: 2017-08-22 12:04 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
libtracker-fts: Adapt to incompatible change in sqlite 3.20 (1.78 KB, patch)
2017-08-06 16:43 UTC, Carlos Garnacho
none Details | Review
libtracker-fts: Adapt to incompatible change in sqlite 3.20 (1.79 KB, patch)
2017-08-06 21:13 UTC, Carlos Garnacho
committed Details | Review
build: Mandate that sqlite3 >= 3.20.0 has fts5 builtin (2.08 KB, patch)
2017-08-22 10:43 UTC, Carlos Garnacho
committed Details | Review

Description jaha 2017-08-06 13:24:18 UTC
After upgrading from sqlite 3.19.3 to 3.20.0 tracker can no longer find files, which makes searching in the Activities overview of Gnome Shell not show results from Files.

Running `tracker search -f` shows files are indexed fine but running `tracker search -f something` returns: "Could not get search results, no such tokenizer: TrackerTokenizer".

Doing `tracker reset --soft` didn't help. Downgrading to sqlite 3.19.3 fixed the issue.

Tested with both tracker version 1.12.1 and 1.99.1. Also tested with rebuilding them from source after installing sqlite 3.20.0. Same issue with both.

Operating system: Arch Linux
Architecture: x86_64
Desktop environment: GNOME Shell 3.24.3
Comment 1 Carlos Garnacho 2017-08-06 16:43:25 UTC
Thanks for the heads up, sqlite indeed made an incompatible change on their fts5 interfaces. I'm attaching (and pushing after I get to test with sqlite 3.20) a patch that should cater for the old and new way of doing things.
Comment 2 Carlos Garnacho 2017-08-06 16:43:47 UTC
Created attachment 357061 [details] [review]
libtracker-fts: Adapt to incompatible change in sqlite 3.20

From that version on, we must use the brand new sqlite3_bind_pointer()
interface to access the fts5 interface pointer. Adapt to it without
giving up older versions.
Comment 3 jaha 2017-08-06 19:44:43 UTC
Could you also provide a backport to tracker 1.12.1?

You provided another patch for what looks to be a possibly related bug https://bugzilla.gnome.org/show_bug.cgi?id=785876 earlier today. I tried the patch you provided there on suggestion of downstream but that patch doesn't solve my issue.
Comment 4 Hussam Al-Tayeb 2017-08-06 20:46:10 UTC
I am getting a build failure with the patch.

https://paste.gnome.org/psnsukuxe (it expects 5 arguments).
Comment 5 Carlos Garnacho 2017-08-06 21:00:46 UTC
(In reply to Jake Dane from comment #3)
> Could you also provide a backport to tracker 1.12.1?

Yeah, had that in mind, the patch should apply mostly as is.

> 
> You provided another patch for what looks to be a possibly related bug
> https://bugzilla.gnome.org/show_bug.cgi?id=785876 earlier today. I tried the
> patch you provided there on suggestion of downstream but that patch doesn't
> solve my issue.

Right, I didn't know of the sqlite 3.20.0 incompatibility back then, if debian also updated to sqlite 3.20.0 it makes sense for him to have this error. The pushed fix still makes sense though, because that database interface plugged by the patch there had no FTS set up, so this error was certainly and legitimately possible.

(In reply to Hussam Al-Tayeb from comment #4)
> I am getting a build failure with the patch.
> 
> https://paste.gnome.org/psnsukuxe (it expects 5 arguments).

Meh, was deceived by the recipe at http://sqlite.org/fts5.html#extending_fts5 , seeing the api docs it indeed takes 5 arguments. I'm updating the patch.
Comment 6 Carlos Garnacho 2017-08-06 21:13:47 UTC
Created attachment 357074 [details] [review]
libtracker-fts: Adapt to incompatible change in sqlite 3.20

From that version on, we must use the brand new sqlite3_bind_pointer()
interface to access the fts5 interface pointer. Adapt to it without
giving up older versions.
Comment 7 jaha 2017-08-07 05:50:03 UTC
That patch worked for me on tracker 1.12.1. First upgrading to sqlite 3.20.0 and afterwards installing tracker recompiled with this patch made tracker do file searches again. Thank you ever so much for the quick response and patch!
Comment 8 Carlos Garnacho 2017-08-07 14:43:29 UTC
Thanks for testing! After ensuring all is correct I'm pushing to
master/tracker-1.12, I'm sorting out new releases for both versions
today.

Attachment 357074 [details] pushed as 4aadfc9 - libtracker-fts: Adapt to incompatible change in sqlite 3.20
Comment 9 Carlos Garnacho 2017-08-12 15:47:54 UTC
*** Bug 786130 has been marked as a duplicate of this bug. ***
Comment 10 Carlos Garnacho 2017-08-22 10:33:26 UTC
Reopening. There are still problems because the fts5 module from sqlite 3.20.0 (as shipped in Tracker) is incompatible with sqlite >= 3.20.0, and fts5 modules generated from sqlite >= 3.20 are backwards incompatible with sqlite < 3.20.

As I refuse to ship 2 copies of fts5 for pre/post 3.20.0, I plan to mandate that sqlite >= 3.20.0 must have fts5 enabled itself, and preserve the internal copy to keep the requirement optional for sqlite < 3.20.0.

Distros were eager to push API/ABI-incompatible sqlite 3.20.0 quickly to users, they'll hopefully be as eager to do --enable-fts5 on it.
Comment 11 Carlos Garnacho 2017-08-22 10:34:51 UTC
*** Bug 785876 has been marked as a duplicate of this bug. ***
Comment 12 Carlos Garnacho 2017-08-22 10:37:19 UTC
(In reply to Carlos Garnacho from comment #10)
> Reopening. There are still problems because the fts5 module from sqlite
> 3.20.0 (as shipped in Tracker) is incompatible with sqlite >= 3.20.0, and
  ^ That should read < 3.20.0

I should also mention, the change shall affect 1.12 and 2.x
Comment 13 Carlos Garnacho 2017-08-22 10:43:41 UTC
Created attachment 358126 [details] [review]
build: Mandate that sqlite3 >= 3.20.0 has fts5 builtin

That sqlite3 version brought incompatibilities in the fts5 interface,
fts5 modules generated from < 3.20.0 trees won't work with sqlite3
>= 3.20.0, and viceversa.

Since it's not feasible nor desirable to include an additional fts5
module copy nor an embedded sqlite3 amalgamation copy, mandate that
SQLite was compiled with --enable-fts5 if >= 3.20.0 is found.

The current copy of the fts5 module is kept around for sqlite3 < 3.20.0,
so we don't make this a mandatory requirement for earlier versions.
Comment 14 Carlos Garnacho 2017-08-22 12:04:07 UTC
Pushed to git, I am doing new 1.12 and 1.99 releases.

Attachment 358126 [details] pushed as 84c8cc2 - build: Mandate that sqlite3 >= 3.20.0 has fts5 builtin