GNOME Bugzilla – Bug 726832
Add installed tests using --enable-installed-tests switch
Last modified: 2014-03-25 11:10:54 UTC
Created attachment 272575 [details] [review] Initial tests for shortcuts in Evolution The following patches are adding a minimal set of tests for evolution via UI. These tests are verifying UI state using dogtail [1] / behave [2] / installed tests [3] and shall be executed on gnome-continuous [4] The tests are split into two parts: 1) Scenarios in *.feature files using gherkin [5] language to describe steps to be performed. This scenarios are also useful for manual testing 2) Step definitions for dogtail in steps/*.py files. These are dogtail instructions to operate on running program These tests are supposed to fail: <Alt>A in Calendar view doesn't trigger 'Actions' menu - I will file a bug on that after 3.12 release How to run those tests: 1) compile evolution with --enable-installed-tests switch 2) run 'gnome-desktop-testing-runner evolution' to execute all tests or 'gnome-desktop-testing-runner evolution/general_shortcuts' for one test of general shortcuts. Notes: 1) the tests wipe all settings, data and cache for evolution and GOA, so DON'T RUN THEM ON YOUR MACHINE. Do the backups or use VMs 2) Please don't close this bug after first patches are committed - I will keep attaching more tests after review [1] http://fedorahosted.org/dogtail [2] https://pypi.python.org/pypi/behave [3] https://wiki.gnome.org/action/show/Initiatives/GnomeGoals/InstalledTests [4] https://wiki.gnome.org/Projects/GnomeContinuous [5] https://github.com/cucumber/cucumber/wiki/Gherkin
Created attachment 272576 [details] [review] Installed tests infrastructure bits
Thanks for the patches. I have only one request, the installed tests should be uninstalled when one invokes "make uninstall", which doesn't happen currently (I ran it at the root of the evolution's checkout). By the way, do you have commit rights to evolution's git? It might be easier for you to commit additional tests directly, than to wait for a "review" (I do not expect there is much to be tested on this, because it doesn't touch evolution's code directly/at all).
Created attachment 272827 [details] [review] Installed tests infrastructure bits - with uninstall (In reply to comment #2) > Thanks for the patches. I have only one request, the installed tests should be > uninstalled when one invokes "make uninstall", which doesn't happen currently > (I ran it at the root of the evolution's checkout). Added uninstall procedure in this patch > By the way, do you have commit rights to evolution's git? Yes, I do. > It might be easier for you to commit additional tests directly, > than to wait for a "review" (I do > not expect there is much to be tested on this, because it doesn't touch > evolution's code directly/at all). I think I'll commit the test bits myself - and in case any code/build procedure will require changes I'll file a bug.
Review of attachment 272827 [details] [review]: Looks better, thanks. Please commit to master, both patches as one single commit, and make sure you'll write the warning about account removal and so on in the commit message. Please commit any further tests directly, without review.
By the way, evolution-data-server allows to run factories (and source registry) with a different XDG base directory, which allows to run tests "in a different folder", without touching real user data. This approach might be better to use in these tests as well.
Pushed as https://git.gnome.org/browse/evolution/commit/?id=399b1a6 (In reply to comment #5) > By the way, evolution-data-server allows to run factories (and source registry) > with a different XDG base directory, which allows to run tests "in a different > folder", without touching real user data. This approach might be better to use > in these tests as well. Right, I think I could improve it, though usually we use one-time machines/VMs for this tests to make sure we're simulating real-world user actions
(In reply to comment #6) > Right, I think I could improve it, though usually we use one-time machines/VMs > for this tests to make sure we're simulating real-world user actions Makes sense. Either do the "run as user" optional (false by default), or skip it entirely, I'll keep it up to you.