After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 728188 - Add installed tests
Add installed tests
Status: RESOLVED OBSOLETE
Product: nautilus
Classification: Core
Component: general
3.12.x
Other Linux
: Normal enhancement
: ---
Assigned To: Nautilus Maintainers
Nautilus Maintainers
Depends on: 763865 764080
Blocks:
 
 
Reported: 2014-04-14 15:14 UTC by Vadim Rutkovsky
Modified: 2021-06-18 15:53 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Initial tests (25.50 KB, patch)
2014-04-14 15:14 UTC, Vadim Rutkovsky
reviewed Details | Review
Installed tests infrastructure bits (6.87 KB, patch)
2014-04-14 15:15 UTC, Vadim Rutkovsky
accepted-commit_now Details | Review
Extended tests for nautilus (20.76 KB, patch)
2014-07-11 14:30 UTC, Martin Simon
needs-work Details | Review
Extended tests for nautilus v2 (21.43 KB, patch)
2014-07-11 15:39 UTC, Martin Simon
none Details | Review
Extended tests for nautilus v2.1 (21.43 KB, patch)
2014-07-14 21:17 UTC, Martin Simon
needs-work Details | Review

Description Vadim Rutkovsky 2014-04-14 15:14:29 UTC
Created attachment 274277 [details] [review]
Initial tests

The following patches are adding a minimal set of tests for Files
via UI. These tests are verifying various actions in nautilus using dogtail [1] /
behave [2] /
installed tests [3] and shall be executed on gnome-continuous [4]

The tests are split into two parts:
 1) Scenarios in *.feature files using gherkin [5] language to describe steps
to be performed. This scenarios are also useful for manual testing
 2) Step definitions for dogtail in steps/*.py files. These are dogtail
instructions to operate on running program


How to run those tests:
 1) compile nautilus with --enable-installed-tests switch
 2) run 'gnome-desktop-testing-runner nautilus' to execute all tests or
'gnome-desktop-testing-runner nautilus/create_folder' for one test of
folder creation.

[1] http://fedorahosted.org/dogtail
[2] https://pypi.python.org/pypi/behave
[3] https://wiki.gnome.org/action/show/Initiatives/GnomeGoals/InstalledTests
[4] https://wiki.gnome.org/Projects/GnomeContinuous
[5] https://github.com/cucumber/cucumber/wiki/Gherkin
Comment 1 Vadim Rutkovsky 2014-04-14 15:15:05 UTC
Created attachment 274278 [details] [review]
Installed tests infrastructure bits
Comment 2 Cosimo Cecchi 2014-07-08 22:02:40 UTC
Review of attachment 274278 [details] [review]:

Looks good
Comment 3 Cosimo Cecchi 2014-07-08 23:03:47 UTC
Review of attachment 274277 [details] [review]:

Overall, looks pretty good. Thanks for working on this!
I left some comments below.

::: installed-tests/common_steps.py
@@ +18,3 @@
+
+
+def wait_until(my_lambda, element, timeout=30, period=0.25):

Seems unused?

@@ +56,3 @@
+
+
+def limit_execution_time_to(

Seems unused?

@@ +173,3 @@
+def test_app_started(context, app):
+    # Dogtail seems to cache applications list
+    # So we should wait for exception here

Not sure what this comment means; could you expand? Why is it fine to receive a GLib.GError here?

::: installed-tests/steps/steps.py
@@ +86,3 @@
+        typeText(name)
+    keyCombo("<Enter>")
+    sleep(0.1)

Why sleeping here?

@@ +148,3 @@
+    drag(source_center, dest_center, button=2)
+    context.app.menuItem('%s Here' % action).click()
+    sleep(0.1)

Why do you need to sleep here?

@@ +153,3 @@
+@step(u'Go to Trash')
+def go_to_trash(context):
+    context.execute_steps(u'* Click "Trash" in sidebar')

Could maybe more robustly use go_to_directory(trash:///)? It's fine either way by me, so feel free to adjust to your liking.

@@ +165,3 @@
+def empty_trash(context):
+    context.app.child('Empty').click()
+    context.app.child('Empty Trash').click()

Why two instructions here?

@@ +172,3 @@
+def rename_file_to(context, source, dest):
+    context.execute_steps(u'* Open context menu for "%s"' % source)
+    context.app.child(roleName='window').findChildren(lambda x: x.roleName=='menu item' and 'Rename' in x.name)[0].click()

Could maybe more robustly use F2 for this instead? I don't feel strongly either way, so feel free to keep it as-is if you disagree.
Comment 4 Vadim Rutkovsky 2014-07-09 09:08:41 UTC
Review of attachment 274277 [details] [review]:

::: installed-tests/common_steps.py
@@ +18,3 @@
+
+
+def wait_until(my_lambda, element, timeout=30, period=0.25):

This file is a library of useful procedures, so we should keep it (this one will be useful for expecting the element to show up / hide / change state etc.

@@ +56,3 @@
+
+
+def limit_execution_time_to(

Ditto, useful function for some steps in the future

@@ +173,3 @@
+def test_app_started(context, app):
+    # Dogtail seems to cache applications list
+    # So we should wait for exception here

Dogtail stores a list of applications (in root.applications() function) but doesn't seem to update them regularly, so the comment states that we're trying to find the required app from scratch.

GLib.GErrors may occur during at-spi calls due to various reasons - timeout from dbus, at-spi daemon crashing etc. - we shouldn't give up on accidental error.

::: installed-tests/steps/steps.py
@@ +86,3 @@
+        typeText(name)
+    keyCombo("<Enter>")
+    sleep(0.1)

We're waiting for textbox to react on Enter and become ineditable. This typing is a workaround for inaccessible textbox, actually, ideally we'd check for textbox state here.

@@ +148,3 @@
+    drag(source_center, dest_center, button=2)
+    context.app.menuItem('%s Here' % action).click()
+    sleep(0.1)

We're waiting for context menu to dissappear (we have to be careful here as we don't know which step will be the next one and cannot guarantee that the next step will wait until menu is hidden)

@@ +153,3 @@
+@step(u'Go to Trash')
+def go_to_trash(context):
+    context.execute_steps(u'* Click "Trash" in sidebar')

This way it will also verify that 'Trash' is present on the sidebar, this will allow us to check more in less amount of scenarios

Though it might be a good idea to verify that navigating to trash:/// will also open Trash - probably in a separate scenario

@@ +165,3 @@
+def empty_trash(context):
+    context.app.child('Empty').click()
+    context.app.child('Empty Trash').click()

This clicks 'Empty' button on toolbar and then 'Empty Trash' button in confirmation dialog (TODO: find this toolbar and the dialog accordingly to make sure we don't click some other button)

@@ +172,3 @@
+def rename_file_to(context, source, dest):
+    context.execute_steps(u'* Open context menu for "%s"' % source)
+    context.app.child(roleName='window').findChildren(lambda x: x.roleName=='menu item' and 'Rename' in x.name)[0].click()

Shortcuts are known to work correctly, though menus might be disabled / removed etc. - let's choose the most problematic approach
Comment 5 Martin Simon 2014-07-11 14:30:20 UTC
Created attachment 280507 [details] [review]
Extended tests for nautilus

This is an extension of Vadim's tests. It covers also View features and some general actions. Some renaming was needed, co take a look in the patch for more details. This patch is applicable after Vadim's patches.
Comment 6 Vadim Rutkovsky 2014-07-11 14:44:10 UTC
Review of attachment 280507 [details] [review]:

Looks good, though got several nitpicks:

::: Makefile.am
@@ +61,2 @@
 @BEHAVE_INSTALLED_TESTS_RULE@
+INSTALLED_TESTS=general_feature view_feature files_feature \

Not sure if its a good idea. Continuous has a limit of 5 mins per each test, so we might hit this barrier if we supply less tests

::: installed-tests/general.feature
@@ +39,3 @@
+  Scenario: GApplication menu - Help
+    * Click "Help" in GApplication menu
+    Then Help titled "Files, folders & search" is shown

I'd prefer to make this test optional - nautilus can be built without help support. We can enable this test in makefile is help is being built

::: installed-tests/steps/steps.py
@@ +195,3 @@
+    assert item in menu_items
+
+@step(u'Click "{item}" in GApplication menu')

This is a duplicate of a step from common_steps:194 (as a side note: all GApplication menu should better be in common_steps.py)
Comment 7 Martin Simon 2014-07-11 15:39:24 UTC
Review of attachment 280507 [details] [review]:

Comments inside, new patch is comming

::: Makefile.am
@@ +61,2 @@
 @BEHAVE_INSTALLED_TESTS_RULE@
+INSTALLED_TESTS=general_feature view_feature files_feature \

Ok, understand, so let's list them all separately

::: installed-tests/general.feature
@@ +39,3 @@
+  Scenario: GApplication menu - Help
+    * Click "Help" in GApplication menu
+    Then Help titled "Files, folders & search" is shown

For now, this test will not be listed in Makefile.am

::: installed-tests/steps/steps.py
@@ +195,3 @@
+    assert item in menu_items
+
+@step(u'Click "{item}" in GApplication menu')

I'll move all GApplication relevant steps to common_steps. But just for the record, this is not duplicate, not in its content itself. My reimplementation solves missing feature in Dogtail for handling Unicode texts in GApplication menu. I've already posted a patch for Dogtail, but it's not in Dogtail's upstream yet
Comment 8 Martin Simon 2014-07-11 15:39:52 UTC
Created attachment 280520 [details] [review]
Extended tests for nautilus v2

Comments fixed, new version
Comment 9 Martin Simon 2014-07-14 21:17:17 UTC
Created attachment 280675 [details] [review]
Extended tests for nautilus v2.1

minor change in patch - moved start of try-except statement in nautilus_window_is_not_displayed() function.
Comment 10 Carlos Soriano 2015-10-15 12:16:49 UTC
ping?
Comment 11 Carlos Soriano 2016-03-01 18:49:32 UTC
Vadim, can you take a look?
Comment 12 Matthias Clasen 2016-03-01 20:06:02 UTC
If the tests work, we should just add them
Comment 13 Carlos Soriano 2016-03-02 09:34:05 UTC
After fixing the conflicts I tried to run those but it doesn't work. It says:
Caught exception during testing: Failed to execute child process "behave" (No such file or directory)

Also I guess we will need to add a dependency to gnome-desktop-testing.
Comment 14 Matthias Clasen 2016-03-02 15:08:06 UTC
you need python-behave
Comment 15 Martin Simon 2016-03-03 16:30:35 UTC
The tests seem to be failing with the latest versions of used tools (dogtail is the first problem). I'll take a look at this in a few days and provide updated patches against the latest master branch.
Comment 16 Martin Simon 2016-03-18 12:01:52 UTC
https://bugzilla.gnome.org/show_bug.cgi?id=763865 is one of the blockers (block dogtail -> this tests as well).

In the meantime, I'm working on updating and stabilizing the tests themselves (the GUI and functionality changes slightly).
Comment 17 Carlos Soriano 2017-08-07 10:55:43 UTC
Review of attachment 280675 [details] [review]:

As per last comment, marking as needs-work until behave etc. works fine.
I'm looking forward to have this anyway!
Comment 18 André Klapper 2021-06-18 15:53:02 UTC
GNOME is going to shut down bugzilla.gnome.org in favor of gitlab.gnome.org.
As part of that, we are mass-closing older open tickets in bugzilla.gnome.org
which have not seen updates for a longer time (resources are unfortunately
quite limited so not every ticket can get handled).

If you can still reproduce the situation described in this ticket in a recent
and supported software version of Files (nautilus), then please follow
  https://wiki.gnome.org/GettingInTouch/BugReportingGuidelines
and create a new ticket at
  https://gitlab.gnome.org/GNOME/nautilus/-/issues/

Thank you for your understanding and your help.