GNOME Bugzilla – Bug 728188
Add installed tests
Last modified: 2021-06-18 15:53:02 UTC
Created attachment 274277 [details] [review] Initial tests The following patches are adding a minimal set of tests for Files via UI. These tests are verifying various actions in nautilus using dogtail [1] / behave [2] / installed tests [3] and shall be executed on gnome-continuous [4] The tests are split into two parts: 1) Scenarios in *.feature files using gherkin [5] language to describe steps to be performed. This scenarios are also useful for manual testing 2) Step definitions for dogtail in steps/*.py files. These are dogtail instructions to operate on running program How to run those tests: 1) compile nautilus with --enable-installed-tests switch 2) run 'gnome-desktop-testing-runner nautilus' to execute all tests or 'gnome-desktop-testing-runner nautilus/create_folder' for one test of folder creation. [1] http://fedorahosted.org/dogtail [2] https://pypi.python.org/pypi/behave [3] https://wiki.gnome.org/action/show/Initiatives/GnomeGoals/InstalledTests [4] https://wiki.gnome.org/Projects/GnomeContinuous [5] https://github.com/cucumber/cucumber/wiki/Gherkin
Created attachment 274278 [details] [review] Installed tests infrastructure bits
Review of attachment 274278 [details] [review]: Looks good
Review of attachment 274277 [details] [review]: Overall, looks pretty good. Thanks for working on this! I left some comments below. ::: installed-tests/common_steps.py @@ +18,3 @@ + + +def wait_until(my_lambda, element, timeout=30, period=0.25): Seems unused? @@ +56,3 @@ + + +def limit_execution_time_to( Seems unused? @@ +173,3 @@ +def test_app_started(context, app): + # Dogtail seems to cache applications list + # So we should wait for exception here Not sure what this comment means; could you expand? Why is it fine to receive a GLib.GError here? ::: installed-tests/steps/steps.py @@ +86,3 @@ + typeText(name) + keyCombo("<Enter>") + sleep(0.1) Why sleeping here? @@ +148,3 @@ + drag(source_center, dest_center, button=2) + context.app.menuItem('%s Here' % action).click() + sleep(0.1) Why do you need to sleep here? @@ +153,3 @@ +@step(u'Go to Trash') +def go_to_trash(context): + context.execute_steps(u'* Click "Trash" in sidebar') Could maybe more robustly use go_to_directory(trash:///)? It's fine either way by me, so feel free to adjust to your liking. @@ +165,3 @@ +def empty_trash(context): + context.app.child('Empty').click() + context.app.child('Empty Trash').click() Why two instructions here? @@ +172,3 @@ +def rename_file_to(context, source, dest): + context.execute_steps(u'* Open context menu for "%s"' % source) + context.app.child(roleName='window').findChildren(lambda x: x.roleName=='menu item' and 'Rename' in x.name)[0].click() Could maybe more robustly use F2 for this instead? I don't feel strongly either way, so feel free to keep it as-is if you disagree.
Review of attachment 274277 [details] [review]: ::: installed-tests/common_steps.py @@ +18,3 @@ + + +def wait_until(my_lambda, element, timeout=30, period=0.25): This file is a library of useful procedures, so we should keep it (this one will be useful for expecting the element to show up / hide / change state etc. @@ +56,3 @@ + + +def limit_execution_time_to( Ditto, useful function for some steps in the future @@ +173,3 @@ +def test_app_started(context, app): + # Dogtail seems to cache applications list + # So we should wait for exception here Dogtail stores a list of applications (in root.applications() function) but doesn't seem to update them regularly, so the comment states that we're trying to find the required app from scratch. GLib.GErrors may occur during at-spi calls due to various reasons - timeout from dbus, at-spi daemon crashing etc. - we shouldn't give up on accidental error. ::: installed-tests/steps/steps.py @@ +86,3 @@ + typeText(name) + keyCombo("<Enter>") + sleep(0.1) We're waiting for textbox to react on Enter and become ineditable. This typing is a workaround for inaccessible textbox, actually, ideally we'd check for textbox state here. @@ +148,3 @@ + drag(source_center, dest_center, button=2) + context.app.menuItem('%s Here' % action).click() + sleep(0.1) We're waiting for context menu to dissappear (we have to be careful here as we don't know which step will be the next one and cannot guarantee that the next step will wait until menu is hidden) @@ +153,3 @@ +@step(u'Go to Trash') +def go_to_trash(context): + context.execute_steps(u'* Click "Trash" in sidebar') This way it will also verify that 'Trash' is present on the sidebar, this will allow us to check more in less amount of scenarios Though it might be a good idea to verify that navigating to trash:/// will also open Trash - probably in a separate scenario @@ +165,3 @@ +def empty_trash(context): + context.app.child('Empty').click() + context.app.child('Empty Trash').click() This clicks 'Empty' button on toolbar and then 'Empty Trash' button in confirmation dialog (TODO: find this toolbar and the dialog accordingly to make sure we don't click some other button) @@ +172,3 @@ +def rename_file_to(context, source, dest): + context.execute_steps(u'* Open context menu for "%s"' % source) + context.app.child(roleName='window').findChildren(lambda x: x.roleName=='menu item' and 'Rename' in x.name)[0].click() Shortcuts are known to work correctly, though menus might be disabled / removed etc. - let's choose the most problematic approach
Created attachment 280507 [details] [review] Extended tests for nautilus This is an extension of Vadim's tests. It covers also View features and some general actions. Some renaming was needed, co take a look in the patch for more details. This patch is applicable after Vadim's patches.
Review of attachment 280507 [details] [review]: Looks good, though got several nitpicks: ::: Makefile.am @@ +61,2 @@ @BEHAVE_INSTALLED_TESTS_RULE@ +INSTALLED_TESTS=general_feature view_feature files_feature \ Not sure if its a good idea. Continuous has a limit of 5 mins per each test, so we might hit this barrier if we supply less tests ::: installed-tests/general.feature @@ +39,3 @@ + Scenario: GApplication menu - Help + * Click "Help" in GApplication menu + Then Help titled "Files, folders & search" is shown I'd prefer to make this test optional - nautilus can be built without help support. We can enable this test in makefile is help is being built ::: installed-tests/steps/steps.py @@ +195,3 @@ + assert item in menu_items + +@step(u'Click "{item}" in GApplication menu') This is a duplicate of a step from common_steps:194 (as a side note: all GApplication menu should better be in common_steps.py)
Review of attachment 280507 [details] [review]: Comments inside, new patch is comming ::: Makefile.am @@ +61,2 @@ @BEHAVE_INSTALLED_TESTS_RULE@ +INSTALLED_TESTS=general_feature view_feature files_feature \ Ok, understand, so let's list them all separately ::: installed-tests/general.feature @@ +39,3 @@ + Scenario: GApplication menu - Help + * Click "Help" in GApplication menu + Then Help titled "Files, folders & search" is shown For now, this test will not be listed in Makefile.am ::: installed-tests/steps/steps.py @@ +195,3 @@ + assert item in menu_items + +@step(u'Click "{item}" in GApplication menu') I'll move all GApplication relevant steps to common_steps. But just for the record, this is not duplicate, not in its content itself. My reimplementation solves missing feature in Dogtail for handling Unicode texts in GApplication menu. I've already posted a patch for Dogtail, but it's not in Dogtail's upstream yet
Created attachment 280520 [details] [review] Extended tests for nautilus v2 Comments fixed, new version
Created attachment 280675 [details] [review] Extended tests for nautilus v2.1 minor change in patch - moved start of try-except statement in nautilus_window_is_not_displayed() function.
ping?
Vadim, can you take a look?
If the tests work, we should just add them
After fixing the conflicts I tried to run those but it doesn't work. It says: Caught exception during testing: Failed to execute child process "behave" (No such file or directory) Also I guess we will need to add a dependency to gnome-desktop-testing.
you need python-behave
The tests seem to be failing with the latest versions of used tools (dogtail is the first problem). I'll take a look at this in a few days and provide updated patches against the latest master branch.
https://bugzilla.gnome.org/show_bug.cgi?id=763865 is one of the blockers (block dogtail -> this tests as well). In the meantime, I'm working on updating and stabilizing the tests themselves (the GUI and functionality changes slightly).
Review of attachment 280675 [details] [review]: As per last comment, marking as needs-work until behave etc. works fine. I'm looking forward to have this anyway!
GNOME is going to shut down bugzilla.gnome.org in favor of gitlab.gnome.org. As part of that, we are mass-closing older open tickets in bugzilla.gnome.org which have not seen updates for a longer time (resources are unfortunately quite limited so not every ticket can get handled). If you can still reproduce the situation described in this ticket in a recent and supported software version of Files (nautilus), then please follow https://wiki.gnome.org/GettingInTouch/BugReportingGuidelines and create a new ticket at https://gitlab.gnome.org/GNOME/nautilus/-/issues/ Thank you for your understanding and your help.