GNOME Bugzilla – Bug 749262
functions-tracker: Fix collate order of test output
Last modified: 2015-07-05 13:10:41 UTC
These tests fail in a locale like this: laney@wily> HOME=$(mktemp -d) LANG=C.UTF-8 LANGUAGE=C.UTF-8 locale LANG=C.UTF-8 LANGUAGE=C.UTF-8 LC_CTYPE="C.UTF-8" LC_NUMERIC="C.UTF-8" LC_TIME="C.UTF-8" LC_COLLATE="C.UTF-8" LC_MONETARY="C.UTF-8" LC_MESSAGES="C.UTF-8" LC_PAPER="C.UTF-8" LC_NAME="C.UTF-8" LC_ADDRESS="C.UTF-8" LC_TELEPHONE="C.UTF-8" LC_MEASUREMENT="C.UTF-8" LC_IDENTIFICATION="C.UTF-8" LC_ALL=C.UTF-8 (/home/laney/dev/gnome/jhbuild/checkout/tracker/tests/libtracker-data/.libs/lt-tracker-sparql:4787): Tracker-ERROR **: --- /home/laney/d ev/gnome/jhbuild/checkout/tracker/tests/libtracker-data/functions/functions-tracker-1.out 2015-05-11 11:46:17.481670200 +0100 +++ - 2015-05-12 14:47:54.770091602 +0100 @@ -1,7 +1,7 @@ -"Text" "default" "optional" "optional for third" "other" "purposes" "testing" +"Text" R) FROM (SELECT * FROM (SELECT t6_g."4_u", t5_g."1_u", t5_g."3_u", t5_g."2_u" FROM (SELECT * FROM (SELECT t4_g."3_u", t3_g."1_u", t3_g." 2_u" FROM (SELECT * FROM (SELECT t1_g."1_u", t2_g."2_u" FROM (SELECT "example:A1"."ID" AS "1_u" FROM "example:A" AS "example:A1") AS t1_ g LEFT JOIN (SELECT "example:A_example:s2"."ID" AS "1_u", "example:A_example:s2"."example:s" AS "2_u" FROM "example:A_example:s" AS "exa mple:A_example:s2") AS t2_g ON t1_g."1_u" = t2_g."1_u")) AS t3_g LEFT JOIN (SELECT "example:A_example:t3"."ID" AS "1_u", "example:A_exam ple:t3"."example:t" AS "3_u" FROM "example:A_example:t" AS "example:A_example:t3") AS t4_g ON t3_g."1_u" = t4_g."1_u")) AS t5_g LEFT JOI N (SELECT "example:A_example:u4"."ID" AS "1_u", "example:A_example:u4"."example:u" AS "4_u" FROM "example:A_example:u" AS "example:A_exa mple:u4") AS t6_g ON t5_g."1_u" = t6_g."1_u")) ORDER BY COALESCE("4_u" COLLATE TRACKER, "3_u" COLLATE TRACKER, "2_u" COLLATE TRACKER, ? COLLATE TRACKER)' 2015-05-11 11:46:17.481670200 +0100 +++ - 2015-05-12 14:47:54.770091602 +0100 @@ -1,7 +1,7 @@ -"Text" "default" "optional" "optional for third" "other" "purposes" "testing" +"Text" Trace/breakpoint trap (core dumped)
Created attachment 303258 [details] [review] functions-tracker: Fix collate order In at least en_GB.UTF-8 and C.UTF-8, these tests fail because the output is actually in a different order.
Hi Iain, I thought we forced the locale to C locale for unit tests... Main reason being, we can't guarantee the outcome in the user's locale (which is random of course). By C locale, I don't mean C.UTF8 - saying that, I'd not seen that locale until your bug report here. Aleksander, any comment to add here?
We did force locale in all unit tests, as collation unit tests shouldn't depend on the user locale... but didn't we enforce the locale to be en_US.UTF-8? I truly don't remember. If the locale isn't enforced in the unit test, it should be.
You do for this one, but not for the others. https://git.gnome.org/browse/tracker/tree/tests/libtracker-data/tracker-sparql-test.c#n371 &, e.g. https://git.gnome.org/browse/tracker/tree/tests/libtracker-sparql/tracker-test.c#n397 So I'm not sure why you don't see the problem. Maybe you don't have en_US.utf8 generated on your system? I /think/ that setlocale is supposed to return NULL in this case - you could check for that & fail or skip. If you want, you could have the testsuite generate and use the locale, like this: localedef -c -i en_US -f UTF-8 .../locale/en_US.utf8 and then set $LOCPATH to the right directory in the environment of the test process. I think the patch on this bug will be necessary.
From my investigation, this not due to locale, but due to a change in sqlite (somewhere between 3.8.7 and 3.8.10.2). I dare to say it's a fix rather than a bug, which we were conveniently hiding in testcases. If you see the stored outputs for these tests, they do match BINARY collation as defined in https://www.sqlite.org/datatype3.html#collation "Text" "default" "optional" "optional for third" "other" "purposes" "testing" Where uppercase goes before lowercase just because A<a, it seems that it's what sqlite used to default for functions like COALESCE, despite us specifying everything as COLLATE TRACKER. With newer sqlite, the resulting ordering is: "default" "optional" "optional for third" "other" "purposes" "testing" "Text" Which matches our collation function ("Text" there is not moved last, it's moved after lowercase t, if we had a "Default" string there, it would be on the 2nd position, not the next-to-last). This also matches the same ordering we'd get if we were sorting by properties directly, and not their COALESCE() result. So, this time the tests were actually hiding a bug... I'm pushing the patch (with reworded commit log) to master.
The following fix has been pushed: 0165650 functions-tracker: Fix collate order
Created attachment 306783 [details] [review] functions-tracker: Fix collate order Somewhere between sqlite 3.8.7 and 3.8.10.2, sqlite seemed to fix a bug on ORDER BY SOMESTRINGFUNCTION() clauses, where it would invariably apply the BINARY collator, despite tracker specifying its own one. This is something that 2 libtracker-data tests were indirectly testing, where queries are sorted by tracker:coalesce(), these now happen to fail because the stored results inadvertently had the wrong order.
Thanks Carlos for your investigation!