After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 786132 - Tracker-CRITICAL Could not execute sparql: ... Constraint error
Tracker-CRITICAL Could not execute sparql: ... Constraint error
Status: RESOLVED OBSOLETE
Product: tracker
Classification: Core
Component: Miners
1.12.x
Other Linux
: Normal normal
: ---
Assigned To: tracker-general
tracker-general
Depends on: 789249
Blocks:
 
 
Reported: 2017-08-10 23:31 UTC by Mel
Modified: 2021-05-26 22:24 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
libtracker-miner: Check directory updates before dropping mtime info (1.92 KB, patch)
2017-09-25 10:54 UTC, Carlos Garnacho
committed Details | Review
libtracker-miner: Always fallback to URN query when queueing file (3.15 KB, patch)
2017-09-25 10:54 UTC, Carlos Garnacho
committed Details | Review
applications: Ignore broken symlinks (3.05 KB, patch)
2017-09-25 10:56 UTC, Carlos Garnacho
committed Details | Review
tracker-miner-fs: Explicitly ignore non-native mounts (1.38 KB, patch)
2017-09-25 10:57 UTC, Carlos Garnacho
committed Details | Review

Description Mel 2017-08-10 23:31:15 UTC
Hi all,

I'm going through all my tracker errors and this is the 3rd type of errors I've come across. I've noticed a lot of messages such as this in my .xsession-errors file:

(tracker-miner-fs:936): Tracker-CRITICAL **:   (Sparql buffer) Error in task 21 (file:///home/mel/pics/nen.jpg) of the array-update: UNIQUE constraint failed: nie:DataObject.nie:url (strerror of errno (not necessarily related): No such file or directory)

(tracker-miner-fs:936): Tracker-CRITICAL **: Could not execute sparql: UNIQUE constraint failed: nie:DataObject.nie:url (strerror of errno (not necessarily related): No such file or directory)

These errors seem to happen after I've deleted/moved files in question and then rebooted. For some reason, tracker-miner-fs ends up reporting hundreds of  (critical) error messages.

Not sure how to fix this... I've deleted cache files and that seems to help.

Thanks.
Comment 1 Carlos Garnacho 2017-08-12 15:35:13 UTC
Thanks for the bug report. Is that 'pics' folder your XDG_PICTURES_DIR? Did you add it to the set of indexed folders yourself? Did you modify the XDG folders or Tracker configuration in a significant way around these changes?

It seems pretty clear that tracker-miner-fs stumbles on leftover data when reindexing a previously indexed folder from scratch, but I haven't seen so far able to see how could this happen, since all deletes are recursive and atomic. Even in the reboot scenario, worst that should happen is that Tracker deals with remaining 'move' operations from before the reboot as 'remove+create'.

(In reply to Mel from comment #0)
> Not sure how to fix this... I've deleted cache files and that seems to help.

That's great, but unfortunate for me :). Having done "tracker info file:///home/mel/pics/nen.jpg" before the rest could have helped a great deal.

If you find some steps to reproduce reliably, that'd be great (no need to reboot, "tracker daemon --terminate" and "tracker daemon --start" would result in the same effect).
Comment 2 Bastien Nocera 2017-08-13 12:40:36 UTC
I have a fair number of "UNIQUE constraint failed" messages on my system.

One related to an SMB mountpoint:
Aug 13 14:18:21 classic tracker-miner-f[12923]: Could not find parent node for URI:'smb://diskstation.local/bastien/'
Aug 13 14:18:21 classic tracker-miner-f[12923]: NOTE: URI theme may be outside scheme expected, for example, expecting 'file://' when given 'http://' prefix.
Aug 13 14:18:21 classic tracker-miner-f[12923]: Could not find parent node for URI:'smb://diskstation.local/privatedata/'
Aug 13 14:18:21 classic tracker-miner-f[12923]: NOTE: URI theme may be outside scheme expected, for example, expecting 'file://' when given 'http://' prefix.
Aug 13 14:18:21 classic dbus-daemon[1689]: [session uid=1000 pid=1689] Activating via systemd: service name='org.freedesktop.Tracker1.Miner.Extract' unit='tracker-extract.service' requested by ':1.14639' (uid=1000 pid=12868 comm="tracker daemon --start " label="unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023")
Aug 13 14:18:21 classic systemd[1674]: Starting Tracker metadata extractor...
Aug 13 14:18:21 classic tracker-miner-f[12923]: Could not initialize currently active mount points: GDBus.Error:org.freedesktop.Tracker1.SparqlError.Internal: UNIQUE constraint failed: nie:DataObject.nie:url (strerror of errno (not necessarily related): Resource temporarily unavailable)

Not sure why it needs to go poke there when it's not listed in the tracker directories to search.

One related to applications:
Aug 13 14:18:21 classic tracker-miner-a[12950]:   (Sparql buffer) Error in task 2 (file:///usr/share/applications/firefox.desktop) of the array-update: UNIQUE constraint failed: nie:DataObject.nie:url (strerror of errno (not necessarily related): Resource temporarily unavailable)
Aug 13 14:18:21 classic tracker-miner-a[12950]: Could not execute sparql: UNIQUE constraint failed: nie:DataObject.nie:url (strerror of errno (not necessarily related): Resource temporarily unavailable)
Aug 13 14:18:21 classic tracker-miner-a[12950]:   (Sparql buffer) Error in task 3 (file:///usr/share/applications/moserial.desktop) of the array-update: UNIQUE constraint failed: nie:DataObject.nie:url (strerror of errno (not necessarily related): Resource temporarily unavailable)
Aug 13 14:18:21 classic tracker-miner-a[12950]: Could not execute sparql: UNIQUE constraint failed: nie:DataObject.nie:url (strerror of errno (not necessarily related): Resource temporarily unavailable)
Aug 13 14:18:21 classic tracker-miner-a[12950]: Couldn't properly parse desktop file 'file:///home/hadess/.local/share/applications/epiphany-tvcatchup-6e3e6c1a5454bcfeeb604f92bc78c7d7c231a3b4.desktop': 'Couldn't load desktop file:'/home/hadess/.local/share/applications/epiphany-tvcatchup-6e3e6c1a5454bcfeeb604f92bc78c7d7c231a3b4.desktop''

That last one is a dangling symlink. The other 2 exist.
Comment 3 Carlos Garnacho 2017-09-25 10:52:23 UTC
FWIW, I've discovered one possible source of these errors which I have fixes for (as well as some of the warnings in comment #2). I also have fixes so tracker-miner-fs doesn't stumble anymore on inconsistent leftover data.

(In reply to Bastien Nocera from comment #2)
> I have a fair number of "UNIQUE constraint failed" messages on my system.
> 
> One related to an SMB mountpoint:
> Aug 13 14:18:21 classic tracker-miner-f[12923]: Could not find parent node
> for URI:'smb://diskstation.local/bastien/'
> Aug 13 14:18:21 classic tracker-miner-f[12923]: NOTE: URI theme may be
> outside scheme expected, for example, expecting 'file://' when given
> 'http://' prefix.
> Aug 13 14:18:21 classic tracker-miner-f[12923]: Could not find parent node
> for URI:'smb://diskstation.local/privatedata/'
> Aug 13 14:18:21 classic tracker-miner-f[12923]: NOTE: URI theme may be
> outside scheme expected, for example, expecting 'file://' when given
> 'http://' prefix.
> Aug 13 14:18:21 classic dbus-daemon[1689]: [session uid=1000 pid=1689]
> Activating via systemd: service
> name='org.freedesktop.Tracker1.Miner.Extract' unit='tracker-extract.service'
> requested by ':1.14639' (uid=1000 pid=12868 comm="tracker daemon --start "
> label="unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023")
> Aug 13 14:18:21 classic systemd[1674]: Starting Tracker metadata extractor...
> Aug 13 14:18:21 classic tracker-miner-f[12923]: Could not initialize
> currently active mount points:
> GDBus.Error:org.freedesktop.Tracker1.SparqlError.Internal: UNIQUE constraint
> failed: nie:DataObject.nie:url (strerror of errno (not necessarily related):
> Resource temporarily unavailable)
> 
> Not sure why it needs to go poke there when it's not listed in the tracker
> directories to search.

Tracker does just try to update the tracker:Volume information on mounts (besides checking they contain/are inside an indexed folder). This includes creating a FileDataObject for the mountpoint uri, but it's not like it's trying to index it.

But it's slightly bad anyway, internally TrackerFileSystem is a tree with a single uri scheme, which tracker-miner-fs make it to be file:///. It just shouldn't be mixing file:/// uris with anything else.

> 
> One related to applications:
> Aug 13 14:18:21 classic tracker-miner-a[12950]:   (Sparql buffer) Error in
> task 2 (file:///usr/share/applications/firefox.desktop) of the array-update:
> UNIQUE constraint failed: nie:DataObject.nie:url (strerror of errno (not
> necessarily related): Resource temporarily unavailable)
> Aug 13 14:18:21 classic tracker-miner-a[12950]: Could not execute sparql:
> UNIQUE constraint failed: nie:DataObject.nie:url (strerror of errno (not
> necessarily related): Resource temporarily unavailable)
> Aug 13 14:18:21 classic tracker-miner-a[12950]:   (Sparql buffer) Error in
> task 3 (file:///usr/share/applications/moserial.desktop) of the
> array-update: UNIQUE constraint failed: nie:DataObject.nie:url (strerror of
> errno (not necessarily related): Resource temporarily unavailable)

Hmmm, I can't quite find an explanation for these from tracker-miner-apps though. If you still have the offending data (i.e. didn't do a tracker reset -r), could you provide the output of "tracker info /usr/share/applications/firefox.desktop" and/or moserial.desktop?

> Aug 13 14:18:21 classic tracker-miner-a[12950]: Could not execute sparql:
> UNIQUE constraint failed: nie:DataObject.nie:url (strerror of errno (not
> necessarily related): Resource temporarily unavailable)
> Aug 13 14:18:21 classic tracker-miner-a[12950]: Couldn't properly parse
> desktop file
> 'file:///home/hadess/.local/share/applications/epiphany-tvcatchup-
> 6e3e6c1a5454bcfeeb604f92bc78c7d7c231a3b4.desktop': 'Couldn't load desktop
> file:'/home/hadess/.local/share/applications/epiphany-tvcatchup-
> 6e3e6c1a5454bcfeeb604f92bc78c7d7c231a3b4.desktop''

I can see this one happening, but AFAICS it's not associated to the constraint errors. Also have a fix for it.
Comment 4 Carlos Garnacho 2017-09-25 10:54:34 UTC
Created attachment 360337 [details] [review]
libtracker-miner: Check directory updates before dropping mtime info

file_notifier_traverse_tree() has the side effect of deleting mtime data,
as it's presumably not needed anymore, except here. Check whether the
directory was updated before notifying of the files inside, so we can
thoroughly check for deleted content.
Comment 5 Carlos Garnacho 2017-09-25 10:54:41 UTC
Created attachment 360338 [details] [review]
libtracker-miner: Always fallback to URN query when queueing file

Tracker may end up with nfo:FileDataObject prior to handling monitor
events. Be it leftover data from previous bugs, explicit "tracker index"
calls, or data from some other application.

As we can't be really sure of the data consistence, always fallback to
a URN query so we don't break nie:url UNIQUE constraint (inverse functional
property in SPARQL parlance).
Comment 6 Carlos Garnacho 2017-09-25 10:56:57 UTC
Created attachment 360339 [details] [review]
applications: Ignore broken symlinks

We still do need to query mtime from the symlink itself in order to keep
filesystem accounting happy, so wait for ENOENT while doing
g_key_file_load_from_file() in order to figure out we are dealing with
a broken link (or a no longer existing file, which is also possible). This
avoids spewing any warnings with those.
Comment 7 Carlos Garnacho 2017-09-25 10:57:06 UTC
Created attachment 360340 [details] [review]
tracker-miner-fs: Explicitly ignore non-native mounts

tracker-miner-fs does rely in a few places to index local content
exclusively. Make it explicitly ignore non-native mounts, they won't
be indexed anyway, but we can spare trying to create the tracker:Volume
and mount root's nfo:FileDataObject for these.

Fixes nie:url constraint errors and other warnings when trying to deal
with those.
Comment 8 Carlos Garnacho 2017-09-29 12:53:04 UTC
Attachment 360337 [details] pushed as d139123 - libtracker-miner: Check directory updates before dropping mtime info
Attachment 360338 [details] pushed as de8691a - libtracker-miner: Always fallback to URN query when queueing file
Comment 9 Carlos Garnacho 2017-09-29 13:01:28 UTC
Attachment 360339 [details] pushed as eb32c86 - applications: Ignore broken symlinks
Attachment 360340 [details] pushed as e9f35b1 - tracker-miner-fs: Explicitly ignore non-native mounts
Comment 10 Bastien Nocera 2017-10-04 12:07:47 UTC
What's the easiest way for me to test this out? (Are the patches in a release and/or in a Fedora package?)
Comment 11 Zach Ploskey 2017-10-04 12:12:06 UTC
(In reply to Carlos Garnacho from comment #9)
> Attachment 360339 [details] pushed as eb32c86 - applications: Ignore broken
> symlinks
> Attachment 360340 [details] pushed as e9f35b1 - tracker-miner-fs: Explicitly
> ignore non-native mounts

It looks like these two commits are not in the repository? I get "bad object id".
Comment 12 Carlos Garnacho 2017-10-04 15:18:15 UTC
(In reply to Bastien Nocera from comment #10)
> What's the easiest way for me to test this out? (Are the patches in a
> release and/or in a Fedora package?)

I released 1.10.6 and 1.12.4 last Fri with this stuff backported, and was planning to do 2.0.x releases this week (I'm obviously late...).

If you don't mind building tracker from git, should be matter of doing:
killall -15 tracker-miner-fs; ~/jhbuild/libexec/tracker-miner-fs

After building, same for tracker-miner-apps.

And anyway, as per comment #3, I'd appreciate "tracker info" output for your firefox/moserial desktop files :).


(In reply to Zach Ploskey from comment #11)
> (In reply to Carlos Garnacho from comment #9)
> > Attachment 360339 [details] pushed as eb32c86 - applications: Ignore broken
> > symlinks
> > Attachment 360340 [details] pushed as e9f35b1 - tracker-miner-fs: Explicitly
> > ignore non-native mounts
> 
> It looks like these two commits are not in the repository? I get "bad object
> id".

Those commit IDs belong to the tracker-miners repo, which BZ seems confused about:

https://git.gnome.org/browse/tracker-miners/commit/?id=eb32c86
https://git.gnome.org/browse/tracker-miners/commit/?id=e9f35b1
Comment 13 Carlos Garnacho 2017-10-20 14:41:03 UTC
The patches helped for some places where this could be "legitimately" happening (still a bug, but due to tracker-miner-fs being clumsy on very specific situations and trying to insert the same data twice).

I however figured out there's more left, and very random ones, here's why:

https://bugzilla.gnome.org/show_bug.cgi?id=789249
Comment 14 Sam Thursfield 2021-05-26 22:24:29 UTC
GNOME is going to shut down bugzilla.gnome.org in favor of gitlab.gnome.org.
As part of that, we are mass-closing older open tickets in bugzilla.gnome.org
which have not seen updates for a longer time (resources are unfortunately
quite limited so not every ticket can get handled).

If you can still reproduce the situation described in this ticket in a recent
and supported software version, then please follow
  https://wiki.gnome.org/GettingInTouch/BugReportingGuidelines
and create a new enhancement request ticket at
  https://gitlab.gnome.org/GNOME/tracker/-/issues/

Thank you for your understanding and your help.