Thanks for a bug report. I do not know what you mean with "so much", because the log you showed me yesterday contained a single test failure on a timeout, probably when trying to access some D-Bus service.

(I replied on IRC with more explanation of the test logs.)

Using your slightly modified script from:

https://anonscm.debian.org/viewvc/pkg-evolution/unstable/evolution-data-server/debian/tests/installed-tests?revision=3183&view=co

I get a 100% success for the first time, when running under Wayland. Running under X11 I get one failure, which I understand it being due to the second run, because the error is:

> /EBookClient/Self/Set: 
> (/usr/libexec/evolution-data-server/installed-tests/test-book-client-self:6556):
> client-test-utils-ERROR **: add contact sync: Conflicting UIDs found in added
> contacts
> FAIL: evolution-data-server/test-book-client-self.test (Child process killed
> by signal 5)

That test can be fixed to delete the contact first, or delete it after the Get test, or something like that, but it's not the case for you, your run failed with a different test, I do not recall which it was.

To be honest, I noticed one more failed test, something with calendar, but it showed only once, out of 3 runs. The error and test were:

> /ECalClient/GetRevisionView/Sync: 
> (/usr/libexec/evolution-data-server/installed-tests/test-cal-client-revision-
> view:29560): test-cal-client-revision-view-ERROR **: remove object sync:
> Message recipient disconnected from message bus without replying
> FAIL: evolution-data-server/test-cal-client-revision-view.test (Child
> process killed by signal 5)

That said, the tests can show some "bad" timing issues or something like that, but in a rather unreliable way. At least on my side. I will try to test this more carefully, in case I manage to catch anything useful.

With added further debugging the /EBookClient/Self/Set test error reported from the backend was:
   UNIQUE constraint failed: folder_id.uid
which seems to me like the local data not being removed as they should, because when I run the test manually several times it never shows this error and just succeeds.

Created attachment 359291 [details] [review]
eds patch for test-book-client-self

for evolution-data-server;

This fixes the issue I see when the test is run against installed services. The difference from `make check` is that the local book is used, instead of a temporary, which can eventually contain the "simple-1" contact already, which causes the failure.

I noticed that gnome-desktop-testing-runner has a --parallel option. Could you try to run it with --parallel=1, to avoid parallelism, because eds tests are not meant to be run in parallel. If it'll still fail, could you try to run the failing test only, whether it can be reproduces as a single run, please? You can do that when specifying it on the gnome-desktop-testing-runner command line, like instead of only "evolution-data-server" you'll specify "evolution-data-server/test-to-run". As I said above, the only failure I had been able to reproduce was about this single test, for which I have the patch.

I think those 2 suggests from comment 5 (disabling parallel and the patch) help. Ubuntu's autopkgtest infrastructure is overwhelmed for the next couple days so I can't test there yet.

When I did an autopkgtest run locally (on my laptop with "traditional" hard drive), I recorded 2 errors.

Error 1
=======
PASS: evolution-data-server/test-cal-cache-getters.test
Running test: evolution-data-server/test-cal-client-get-free-busy.test
/ECalClient/GetFreeBusy/Sync: **
test-cal-client-get-free-busy:ERROR:/<<PKGBUILDDIR>>/tests/libecal/client/test-cal-client-get-free-busy.c:130:test_get_free_busy_sync: assertion failed: (freebusy_data)
FAIL: evolution-data-server/test-cal-client-get-free-busy.test (Child process killed by signal 6)
Running test: evolution-data-server/test-cal-client-get-object-list.test
/ECalClient/GetObjectList/Sync: OK

Error 2
=======
PASS: evolution-data-server/test-intervaltree.test
Running test: evolution-data-server/test-cal-client-get-revision.test
/ECalClient/GetRevision: test-cal-client-get-revision-Message: Passed cycle, revision before '2017-09-07T00:13:07.779365Z(1)' revision after '2017-09-07T00:13:07.780356Z(2)'
test-cal-client-get-revision-Message: Passed cycle, revision before '2017-09-07T00:13:07.780356Z(2)' revision after '2017-09-07T00:13:07.780913Z(3)'
test-cal-client-get-revision-Message: Passed cycle, revision before '2017-09-07T00:13:07.780913Z(3)' revision after '2017-09-07T00:13:07.781490Z(4)'
test-cal-client-get-revision-Message: Passed cycle, revision before '2017-09-07T00:13:07.781490Z(4)' revision after '2017-09-07T00:13:07.782070Z(5)'
test-cal-client-get-revision-Message: Passed cycle, revision before '2017-09-07T00:13:07.782070Z(5)' revision after '2017-09-07T00:13:07.782711Z(6)'
test-cal-client-get-revision-Message: Passed cycle, revision before '2017-09-07T00:13:07.782711Z(6)' revision after '2017-09-07T00:13:07.783308Z(7)'
test-cal-client-get-revision-Message: Passed cycle, revision before '2017-09-07T00:13:07.783308Z(7)' revision after '2017-09-07T00:13:07.783996Z(8)'
test-cal-client-get-revision-Message: Passed cycle, revision before '2017-09-07T00:13:07.783996Z(8)' revision after '2017-09-07T00:13:07.784704Z(9)'
test-cal-client-get-revision-Message: Passed cycle, revision before '2017-09-07T00:13:07.784704Z(9)' revision after '2017-09-07T00:13:07.785449Z(10)'
**
test-cal-client-get-revision:ERROR:/<<PKGBUILDDIR>>/tests/libecal/client/test-cal-client-get-revision.c:56:get_revision_compare_cycle: assertion failed: (strcmp (revision_before, revision_after) != 0)
FAIL: evolution-data-server/test-cal-client-get-revision.test (Child process killed by signal 6)
Running test: evolution-data-server/test-ecal-get-ldap-attribute.test

I understand it as once I commit the above patch I can also close this bug report. Note that some issues are caused by D-Bus notifications/signals delivery, which is out of hands of the application by all means.

Created commit a7c2b51 in eds master (3.27.1+)
Created commit c49bdae in eds gnome-3-26 (3.26.1+)