GNOME Bugzilla – Bug 720713
Support running specific unittests with make check
Last modified: 2015-02-07 16:47:31 UTC
In an effort to make the GI codebase a bit friendlier to rapid unittest driven hacking, a good start would be using some form of test discovery and test specification. The idea is to have the ability to single out of specific test cases for a make check run. In PyGObject it is very nice to be able to run: $ make check TEST_NAMES=test_gobject.foo.bar and have a quickly executed test for a specific function being worked on. One possibility is to use pythons test discovery, but it might be tricky to single out specific cases when there is also automake test harness usage and check-local hooks used in a make check run. But perhaps it would be a matter of supporting two variables for test specification: $ make check AMTESTS="" PYTESTS=test_scanner As an alternative it might be possible to fully use the automake test harness for everything:: http://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html I lack expertise with AM so it would be good to hear from others who might have ideas about this? It is also unclear if it is possible to achieve the level of granularity needed to support per-test method runs in Python with automake.
Created attachment 264780 [details] [review] tests: Use Automake test harness for scanner diff tests Add gi-tester script for dispatching various scanner diff tests based on their target suffix. Hook the dispatcher script into the Automake test harness which is passed the diff test target name. Remove usage of check-local which is always run regardless of whether or not the TESTS variable is set from the command line. This allows for both the ability to use the harnesses XFAIL_TESTS variable for expected failures as well as testing of targets individually, for example: cd tests/scanner make check TESTS=GtkFrob-1.0.gir make check TESTS=Regress-1.0-Python Add Headeronly-1.0-expected.gir for validating the results of the header only gir creation test.
Created attachment 264781 [details] [review] tests: Use EXTRA_LTLIBRARIES instead of check_LTLIBRARIES Update tests/Makefile.am and tests/scanner/Makefile.am to use EXTRA_LTLIBRARIES which minimizes the building of libraries to only dependencies of specific tests when using: make check TESTS="..."
Created attachment 264782 [details] [review] tests: Use Automake test harness for warning tests Update warning tests to use the Automake test harness per-header file check. This allows for quicker more granular testing, for example: cd tests/warn make check TESTS=return-gobject.h
Created attachment 264783 [details] [review] tests: Refactor test_parser and test_patterns to use unittest.TestSuite Update both test_parser.py and test_patterns.py to dynamically generate TestCase sub-classes rather than modify a statically defined one. Use unittest.TestSuite to queue up the generated test cases via the "load_tests" hook. Use underscores instead of periods as a seperator for dynamically generated test cases and methods to match Python identifiers. Beyond general cleanup, these changes will help the ability to specify Python tests individually.
Created attachment 264784 [details] [review] tests: Use Python unittest module parameterization for annotationparser Add Python unittest module parameterization support to gi-tester through the use of a "TESTARGS" variable. Use the Automake test harness along with gi-tester for running the Python annotationparser tests. Add tests/scanner/annotationparser/Makefile.am and localize testing rules and EXTRA_DIST setup to this file. Specific Python tests can now be run as follows: cd tests/scanner/annotationparser make check TESTS=test_patterns.py TESTARGS=TestTagValueStability
Created attachment 264785 [details] [review] tests: Use Automake test harness instead of gtester Switch both cmph-bdz-test and gthash_test to use the Automake test harness. Use EXTRA_PROGRAMS and EXTRA_LTLIBRARIES along with CLEANUP to allow lazy dependency evaluation of testing targets.
Created attachment 264786 [details] [review] tests: Use Automake test harness for tests/offsets Replace usage of check-local with the Automake test harness. Change check_ variable usage to EXTRA_ in order to avoid building these dependencies with targeted testing using TESTS= in parent directories.
Notes on patches: The gi-tester script in the first patch states a problem I ran into with target names including relative paths back to srcdir for documentation directory diffs. This seems like it might be a bug in Automake but I am unsure. The script has a workaround. Targeting specific tests with the TESTS environment from a parent directory doesn't work well. You must first cd into the directory where the test target lives then run: make check TESTS="..." (as exemplified in various commit messages). Furthermore, if there are sub-directories within the targeted tests directory, automake will complain when TESTS is specified (but the targeted test is still run correctly). Most of this work could also be considered somewhat precursory for using the parallel test harness or tap output if that is desired. I verified "make distcheck" runs successfully after each of these patches.
Created attachment 264792 [details] [review] tests: Use Python unittest module parameterization for annotationparser Fixed path for running Python test as a module from a VPATH build dir. cd tests/scanner/annotationparser make check TESTS=test_patterns.py TESTARGS=TestTagValueStability
Created attachment 264960 [details] [review] tests: Use Automake test harness instead of gtester Updated to not use EXTRA_* when appending to CLEANFILES because these variables are "global" and might be modified in other includes.
Review of attachment 264780 [details] [review]: Makes sense. Thanks for the excellent patch and commit message!
Review of attachment 264781 [details] [review]: Makes sense. I didn't know about EXTRA_ before this.
Review of attachment 264782 [details] [review]: Having an execution binary in TESTS_ENVIRONMENT is deprecated, right? Although we do that elsewhere, so...ok
Review of attachment 264783 [details] [review]: I'm not familiar enough with the Python unittest framework to really review this, but the commit message makes sense, and the code looks reasonable.
Review of attachment 264786 [details] [review]: ::: tests/offsets/Makefile.am @@ +50,3 @@ +TESTS = gitestoffsets +TESTS_ENVIRONMENT = \ + GI_TYPELIB_PATH=:$(top_builddir) ./gitestoffsets$(EXEEXT) offsets.compiled offsets.introspected && \ Won't we be running gitestoffsets twice? I guess that doesn't really hurt, but if we wanted to fix this we'd probably have to push the "diff" invocation down into gitestoffsets.c.
Review of attachment 264960 [details] [review]: Looks fine.
Attachment 264781 [details] pushed as c981aa7 - tests: Use EXTRA_LTLIBRARIES instead of check_LTLIBRARIES Attachment 264782 [details] pushed as beb7a56 - tests: Use Automake test harness for warning tests Attachment 264960 [details] pushed as 4e88f0e - tests: Use Automake test harness instead of gtester
Comment on attachment 264783 [details] [review] tests: Refactor test_parser and test_patterns to use unittest.TestSuite Attachment 264783 [details] pushed as 42bb69a - tests: Refactor test_parser and test_patterns to use unittest.TestSuite
(In reply to comment #13) > Review of attachment 264782 [details] [review]: > > Having an execution binary in TESTS_ENVIRONMENT is deprecated, right? Although > we do that elsewhere, so...ok I can update all of this in another pass to use the Parallel Test Harness if that is desired. Would we need to add "parallel-tests" to AM_INIT_AUTOMAKE and make sure we have a minimum version which supports it as well? (In reply to comment #15) > Review of attachment 264786 [details] [review]: > > ::: tests/offsets/Makefile.am > @@ +50,3 @@ > +TESTS = gitestoffsets > +TESTS_ENVIRONMENT = \ > + GI_TYPELIB_PATH=:$(top_builddir) ./gitestoffsets$(EXEEXT) offsets.compiled > offsets.introspected && \ > > Won't we be running gitestoffsets twice? It's a bit hacky but the harness appends gitestoffsets to TEST_ENVIRONMENT which ends with "... && echo". So it is only run once but the program name is echoed at the end :) I didn't have any ideas beyond this to get it working. I guess we could also use a custom bash script runner if my solution seems too hacky.
Review of attachment 264792 [details] [review]: A little sad at further recursive make usage, but this otherwise looks good. Thanks!
Comment on attachment 264792 [details] [review] tests: Use Python unittest module parameterization for annotationparser Attachment 264792 [details] pushed as f27aff2 - tests: Use Python unittest module parameterization for annotationparser
Created attachment 265004 [details] [review] tests: Update Python stanza to change directories before running tests Ensure PYTHONPATH uses absolute paths for top_builddir and top_srcdir. Use pushd/popd before and after running tests with "python -m unittest...". This ensures TESTARGS can be used from within a vpath build directory.
Review of attachment 264786 [details] [review]: Ah...I see. Looks OK then, thanks!
Review of attachment 265004 [details] [review]: One comment, feel free to commit with or without change. ::: tests/gi-tester @@ +46,3 @@ + ${PYTHON} -m unittest -v "${modulename}.${TESTARGS}" + res=$? + popd (cd $(dirname $targetname) && ${PYTHON} -m unittest -v "${modulename}.${TESTARGS}") exit $? is easier than pushd/popd and this "res" variable, I think.
Attachment 264786 [details] pushed as 80b54bd - tests: Use Automake test harness for tests/offsets Attachment 265004 [details] pushed as 8787aec - tests: Update Python stanza to change directories before running tests
(In reply to comment #24) > Review of attachment 265004 [details] [review]: > (cd $(dirname $targetname) && ${PYTHON} -m unittest -v > "${modulename}.${TESTARGS}") > exit $? > > is easier than pushd/popd and this "res" variable, I think. Indeed. I went with this and additionally replace all the backtick usage. I'm still learning bash ;) Also added an additional comment regarding the tailing echo in TESTS_ENVIRONMENT so it doesn't throw future readers off.
[Mass-moving gobject-introspection tickets to its own Bugzilla product - see bug 708029. Mass-filter your bugmail for this message: introspection20150207 ]