GNOME Bugzilla – Bug 668186
make check does not run the test suite (which doesn't pass when it is run)
Last modified: 2012-04-19 14:47:41 UTC
Today I noticed that, although there are some beautiful-looking GTest-based test cases in the tests/ directory, `make check` does not run them. `make test -C tests` runs them, and the majority of them do not pass (which is unsurprising). 2025 % make test -C tests make: Entering directory `/home/wjt/src/tp/empathy/tests' gtester -o test-report.xml -k --verbose empathy-irc-server-test empathy-irc-network-test empathy-irc-network-manager-test empathy-chatroom-test empathy-chatroom-manager-test empathy-parser-test empathy-live-search-test empathy-tls-test TEST: empathy-irc-server-test... (pid=4331) /irc-server/new: OK /irc-server/property-change: OK /irc-server/modified-signal: OK PASS: empathy-irc-server-test TEST: empathy-irc-network-test... (pid=4332) /irc-network/new: OK /irc-network/property-change: OK /irc-network/modified-signal: OK /irc-network/add-server: OK /irc-network/modified-signal-because-of-server: OK /irc-network/set-server-position: OK PASS: empathy-irc-network-test TEST: empathy-irc-network-manager-test... (pid=4333) /irc-network-manager/add: OK /irc-network-manager/load-global-file: ** ERROR:empathy-irc-network-manager-test.c:96:test_load_global_file: assertion failed: (g_slist_length (networks) == 4) FAIL GTester: last random seed: R02S4b9d6f1ba4a05fbe0ccc5a9b1ba277fe (pid=4334) /irc-network-manager/remove: ** ERROR:empathy-irc-network-manager-test.c:197:test_empathy_irc_network_manager_remove: assertion failed: (result) FAIL GTester: last random seed: R02Sd3d98e11f84aea8009a946bae217bfae (pid=4335) /irc-network-manager/load-user-file: ** ERROR:test-helper.c:69:copy_xml_file: assertion failed: (result) FAIL GTester: last random seed: R02S704eae02af33f3c13e2ce2fcb9feb2ea (pid=4336) /irc-network-manager/load-both-files: ** ERROR:empathy-irc-network-manager-test.c:350:test_load_both_files: assertion failed: (g_slist_length (networks) == 5) FAIL GTester: last random seed: R02S3a64f13f303976d58c6fe08c74016af9 (pid=4337) /irc-network-manager/modify-user-file: ** ERROR:test-helper.c:69:copy_xml_file: assertion failed: (result) FAIL GTester: last random seed: R02S13d4f8c5f32dfd4e015f51dceab6c6cd (pid=4338) /irc-network-manager/modify-both-files: ** ERROR:test-helper.c:69:copy_xml_file: assertion failed: (result) FAIL GTester: last random seed: R02S2662aab1fe89a7d767982cafda087158 (pid=4339) /irc-network-manager/find-network-by-address: ** ERROR:empathy-irc-network-manager-test.c:777:test_empathy_irc_network_manager_find_network_by_address: assertion failed: (network != NULL) FAIL GTester: last random seed: R02S62f3be407f2d91335ecd6e1e6f33d8b0 (pid=4340) /irc-network-manager/no-modify-with-empty-user-file: OK FAIL: empathy-irc-network-manager-test TEST: empathy-chatroom-test... (pid=4341) PASS: empathy-chatroom-test TEST: empathy-chatroom-manager-test... (pid=4342) PASS: empathy-chatroom-manager-test TEST: empathy-parser-test... (pid=4343) /parsers: Gtk-WARNING **: Theme file for default has no name FAIL GTester: last random seed: R02S27156f2428e3c16ac83db0b744cf4e1a (pid=4344) FAIL: empathy-parser-test TEST: empathy-live-search-test... (pid=4345) /live-search: OK PASS: empathy-live-search-test TEST: empathy-tls-test... (pid=4346) /tls/certificate_basics: ** ERROR:empathy-tls-test.c:258:mock_tls_certificate_new_and_register: assertion failed (error == NULL): Failed to open file '': No such file or directory (g-file-error-quark, 4) FAIL GTester: last random seed: R02S8e39572e9ef2b3b85b34126be6d13134 (pid=4348) /tls/certificate_verify_success_with_pkcs11_lookup: ** ERROR:empathy-tls-test.c:258:mock_tls_certificate_new_and_register: assertion failed (error == NULL): Failed to open file '': No such file or directory (g-file-error-quark, 4) FAIL GTester: last random seed: R02S98852d9bbc3d5c54267e2725ae6faedd (pid=4350) /tls/certificate_verify_success_with_full_chain: ** ERROR:empathy-tls-test.c:258:mock_tls_certificate_new_and_register: assertion failed (error == NULL): Failed to open file '': No such file or directory (g-file-error-quark, 4) FAIL GTester: last random seed: R02Sb340c2d9640c8e4411832f1c96810b17 (pid=4352) /tls/certificate_verify_root_not_found: ** ERROR:empathy-tls-test.c:258:mock_tls_certificate_new_and_register: assertion failed (error == NULL): Failed to open file '': No such file or directory (g-file-error-quark, 4) FAIL GTester: last random seed: R02S76406efd6a7213402d413c24605c1519 (pid=4354) /tls/certificate_verify_root_not_anchored: ** ERROR:empathy-tls-test.c:258:mock_tls_certificate_new_and_register: assertion failed (error == NULL): Failed to open file '': No such file or directory (g-file-error-quark, 4) FAIL GTester: last random seed: R02Sc9d4e48fabf89bcad325ea94af66fabf (pid=4356) ** ERROR:empathy-tls-test.c:258:mock_tls_certificate_new_and_register: assertion failed (error == NULL): Failed to open file '': No such file or directory (g-file-error-quark, 4) /tls/certificate_verify_identities_invalid: FAIL GTester: last random seed: R02S6af29f738b60cc74cf42e6c1ed745b8d (pid=4358) /tls/certificate_verify_uses_reference_identities: ** ERROR:empathy-tls-test.c:258:mock_tls_certificate_new_and_register: assertion failed (error == NULL): Failed to open file '': No such file or directory (g-file-error-quark, 4) FAIL GTester: last random seed: R02S235ae18fe2aa3d195e25a3c1e0d27c98 (pid=4360) FAIL: empathy-tls-test make: *** [test] Error 1 make: Leaving directory `/home/wjt/src/tp/empathy/tests'
The certificate tests are failing as well for me but not empathy-irc-network-manager-test. Here is a patch to start debugging why.
Created attachment 205613 [details] [review] empathy-irc-network-manager-test: use more specific assertions They are more useful for debugging when the test is failing.
Created attachment 205614 [details] [review] empathy-tls-test: use g_assert_(no_)error()
Attachment 205613 [details] pushed as 695ff49 - empathy-irc-network-manager-test: use more specific assertions Attachment 205614 [details] pushed as 871b0cf - empathy-tls-test: use g_assert_(no_)error()
I opened bug #668261 about my TLS test failure. Your failures are because EMPATHY_SRCDIR isn't defined when running the tests so they fail to find some files. That's weird because tests/Makefile.am contains TESTS_ENVIRONMENT = EMPATHY_SRCDIR=@abs_top_srcdir@ Any Makefile ninja having an idea?
Got it! But we should first fix bug #668261 if we don't want to break "make distcheck".
Created attachment 205618 [details] [review] Hook tests on 'make check' instead of 'make test' This is how it's supposed to be done and so has the nice side effect of actually using TESTS_ENVIRONMENT and so fix tests if EMPATHY_SRCDIR is not defined.
Created attachment 212329 [details] [review] tls-test: Use a proper PKCS#11 mock module for testing * In empathy-tls-test.c use a proper PKCS#11 mock module, which we can place certificate anchors and pinned certificates in. * Fix EmpathyTlsVerifier to check for server trust assertions not client ones, this was a bug highlighted by fixing tests.
This problem has been fixed in the development version. The fix will be available in the next major software release. Thank you for your bug report.