After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 749262 - functions-tracker: Fix collate order of test output
functions-tracker: Fix collate order of test output
Product: tracker
Classification: Core
Component: General
Other All
: Normal normal
: ---
Assigned To: tracker-general
Depends on:
Reported: 2015-05-12 14:03 UTC by Iain Lane
Modified: 2015-07-05 13:10 UTC
See Also:
GNOME target: ---
GNOME version: ---

functions-tracker: Fix collate order (1.23 KB, patch)
2015-05-12 14:03 UTC, Iain Lane
none Details | Review
functions-tracker: Fix collate order (1.58 KB, patch)
2015-07-03 22:35 UTC, Carlos Garnacho
none Details | Review

Description Iain Lane 2015-05-12 14:03:31 UTC
These tests fail in a locale like this:

laney@wily> HOME=$(mktemp -d) LANG=C.UTF-8 LANGUAGE=C.UTF-8 locale

(/home/laney/dev/gnome/jhbuild/checkout/tracker/tests/libtracker-data/.libs/lt-tracker-sparql:4787): Tracker-ERROR **: --- /home/laney/d
ev/gnome/jhbuild/checkout/tracker/tests/libtracker-data/functions/functions-tracker-1.out   2015-05-11 11:46:17.481670200 +0100
+++ -   2015-05-12 14:47:54.770091602 +0100
@@ -1,7 +1,7 @@
 "optional for third"

R) FROM (SELECT * FROM (SELECT t6_g."4_u", t5_g."1_u", t5_g."3_u", t5_g."2_u" FROM (SELECT * FROM (SELECT t4_g."3_u", t3_g."1_u", t3_g."
2_u" FROM (SELECT * FROM (SELECT t1_g."1_u", t2_g."2_u" FROM (SELECT "example:A1"."ID" AS "1_u" FROM "example:A" AS "example:A1") AS t1_
g LEFT JOIN (SELECT "example:A_example:s2"."ID" AS "1_u", "example:A_example:s2"."example:s" AS "2_u" FROM "example:A_example:s" AS "exa
mple:A_example:s2") AS t2_g ON t1_g."1_u" = t2_g."1_u")) AS t3_g LEFT JOIN (SELECT "example:A_example:t3"."ID" AS "1_u", "example:A_exam
ple:t3"."example:t" AS "3_u" FROM "example:A_example:t" AS "example:A_example:t3") AS t4_g ON t3_g."1_u" = t4_g."1_u")) AS t5_g LEFT JOI
N (SELECT "example:A_example:u4"."ID" AS "1_u", "example:A_example:u4"."example:u" AS "4_u" FROM "example:A_example:u" AS "example:A_exa
mple:u4") AS t6_g ON t5_g."1_u" = t6_g."1_u")) ORDER BY COALESCE("4_u" COLLATE TRACKER, "3_u" COLLATE TRACKER, "2_u" COLLATE TRACKER, ? 
2015-05-11 11:46:17.481670200 +0100
+++ -   2015-05-12 14:47:54.770091602 +0100
@@ -1,7 +1,7 @@
 "optional for third"
Trace/breakpoint trap (core dumped)
Comment 1 Iain Lane 2015-05-12 14:03:35 UTC
Created attachment 303258 [details] [review]
functions-tracker: Fix collate order

In at least en_GB.UTF-8 and C.UTF-8, these tests fail because the output is
actually in a different order.
Comment 2 Martyn Russell 2015-05-31 13:02:31 UTC
Hi Iain,

I thought we forced the locale to C locale for unit tests... Main reason being, we can't guarantee the outcome in the user's locale (which is random of course).

By C locale, I don't mean C.UTF8 - saying that, I'd not seen that locale until your bug report here.

Aleksander, any comment to add here?
Comment 3 Aleksander Morgado 2015-06-01 06:29:19 UTC
We did force locale in all unit tests, as collation unit tests shouldn't depend on the user locale... but didn't we enforce the locale to be en_US.UTF-8? I truly don't remember.

If the locale isn't enforced in the unit test, it should be.
Comment 4 Iain Lane 2015-06-01 08:21:36 UTC
You do for this one, but not for the others.

&, e.g.

So I'm not sure why you don't see the problem. Maybe you don't have en_US.utf8 generated on your system? I /think/ that setlocale is supposed to return NULL in this case - you could check for that & fail or skip.

If you want, you could have the testsuite generate and use the locale, like this:

  localedef -c -i en_US -f UTF-8 .../locale/en_US.utf8

and then set $LOCPATH to the right directory in the environment of the test process.

I think the patch on this bug will be necessary.
Comment 5 Carlos Garnacho 2015-07-03 22:34:04 UTC
From my investigation, this not due to locale, but due to a change in sqlite (somewhere between 3.8.7 and I dare to say it's a fix rather than a bug, which we were conveniently hiding in testcases. 

If you see the stored outputs for these tests, they do match BINARY collation as defined in

 "optional for third"

Where uppercase goes before lowercase just because A<a, it seems that it's what sqlite used to default for functions like COALESCE, despite us specifying everything as COLLATE TRACKER.

With newer sqlite, the resulting ordering is:

 "optional for third"

Which matches our collation function ("Text" there is not moved last, it's moved after lowercase t, if we had a "Default" string there, it would be on the 2nd position, not the next-to-last). This also matches the same ordering we'd get if we were sorting by properties directly, and not their COALESCE() result.

So, this time the tests were actually hiding a bug... I'm pushing the patch (with reworded commit log) to master.
Comment 6 Carlos Garnacho 2015-07-03 22:35:43 UTC
The following fix has been pushed:
0165650 functions-tracker: Fix collate order
Comment 7 Carlos Garnacho 2015-07-03 22:35:48 UTC
Created attachment 306783 [details] [review]
functions-tracker: Fix collate order

Somewhere between sqlite 3.8.7 and, sqlite seemed to fix
a bug on ORDER BY SOMESTRINGFUNCTION() clauses, where it would
invariably apply the BINARY collator, despite tracker specifying
its own one.

This is something that 2 libtracker-data tests were indirectly
testing, where queries are sorted by tracker:coalesce(), these now
happen to fail because the stored results inadvertently had the wrong
Comment 8 Iain Lane 2015-07-05 13:10:41 UTC
Thanks Carlos for your investigation!