GNOME Bugzilla – Bug 725219
Tests fail on build.gnome.org
Last modified: 2014-06-19 22:19:12 UTC
Created attachment 270380 [details] [review] tests: use correct app name Here are several patches to stabilize tests
Created attachment 270381 [details] [review] Make sure we kill gnome-software after each test
Created attachment 270382 [details] [review] tests: close About dialog using Esc
Review of attachment 270380 [details] [review]: You sure that's right? org.gnome.Software is (or at least should be) the app name.
(In reply to comment #3) > Review of attachment 270380 [details] [review]: > > You sure that's right? org.gnome.Software is (or at least should be) the app > name. Yes, tested via gnome-desktop-testing-runner locally. This is an a11y app name, which by default is a binary name
Created attachment 271591 [details] [review] Open Software Sources dialog This test case tries to open Software Sources dialog using the app's super menu on top bar and checks if the dialog is correctly open. Since the dialog is empty, it only checks its label.
Created attachment 271592 [details] [review] Test: Details page This patch adds a test case to test Details page. It opens GIMP details after searching for it and checks buttons in the page. At the end it clicks to "Go Back" button to test its function.
Review of attachment 271591 [details] [review]: Looks good to me
Review of attachment 271592 [details] [review]: +1 from me, though some nitpicking: doDelay(4) is generally bad idea, but I guess we can rewrite this as soon as we port this to unittest/BDD ::: tests/basic.py @@ +111,3 @@ + assert (history_details_button.getState().contains(pyatspi.STATE_VISIBLE)) + + if install_button.getState().contains(pyatspi.STATE_SHOWING): we can use dogtail shorcut to simplify this: if install_button.showing:
Created attachment 271597 [details] [review] Use g_set_prgname to set nice a11y name
Review of attachment 271597 [details] [review]: Looks good to me.
Review of attachment 270380 [details] [review]: the dogtail docs don't say much about the name argument, so if vadim says this is what works, then I guess we should believe him.
Are there any more patches ready to push or are they all upstream now?
Created attachment 277903 [details] [review] Fix for missing gimp app in tests for Details page Gimp is searched in Details page test case. The gimp is not present in gnome-continuous. This patch changes the searched app to geary, instead. Also ugly access to widget state has been changed.
Review of attachment 277903 [details] [review]: Looks good to me, except that the test now started failing on rawhide for me with this patch. Apparently the name differs in metadata ("Geary" vs "Geary Mail"). Can we use something else for that test, maybe Polari?
By the way, any idea why basic.py takes 30 seconds to start here? gnome-software window shows up, then it does something and finally spits out this warning before going on with the tests: Warning: ./basic.py:22: The requested widget could not be focused: "gnome-software" application: run('gnome-software')
(In reply to comment #14) >Apparently the name differs in metadata ("Geary" vs "Geary Mail"). Right, it should be 'Geary' (http://build.gnome.org/continuous/buildmaster/builds/2014/06/05/0/integrationtest/work-gnome-continuous-x86_64-runtime/screenshot-141.png) >Can we use something else for that test, maybe Polari? This patch was made specifically for GNOME Continuous, which has only Geary metadata there. The best option would be providing a dummy appdata in Software itself, so we wouldn't depend on other app changes
Sure, sounds good with the edit in that case.
I went ahead and pushed this with the Geary Mail -> Geary change. Thanks for the patch! Attachment 277903 [details] pushed as 6edcedc - tests: Details page - fixed
Created attachment 278724 [details] [review] [PATCH] Automated tests extended and ported to Behave. I ported the tests to behave style and also extended them a little. Now, they basically cover all the functionality without actual installing/removing applications, because that is not as straightforward in GnomeContinuous. This test is also blocked by lack of sample data, so please wait with merging. I have prepared some sample data and just need to figure out how to update Makefiles and so on to make them installed properly. These tests are tested using jhbuild and gnome-desktop-testing-runner. I'm ready for comments.
Review of attachment 278724 [details] [review]: Fails to build with: Makefile:898: *** missing separator. Stop. We also need to include https://git.gnome.org/browse/evolution/tree/m4/behave-installed-tests.m4
Review of attachment 278724 [details] [review]: ::: Makefile.am @@ +68,3 @@ +@BEHAVE_INSTALLED_TESTS_RULE@ +INSTALLED_TESTS=basic_tests We have a timeout limit of 5 minutes on test execution time in gnome-desktop-test-runner, so when I applied the patch it timed out. This should be split into several tests (add per-scenario tags here instead) This patch also removes 'actions_test' - this should also be ported to behave
Created attachment 278789 [details] [review] Automated tests extended and ported to Behave. Thanks, Vadim, for you comments. I forgot to add m4/behave-installed-tests.m4, because that in repo .gitignore and I missed it. Now it should be fine. Also, I split the one test set into several more according to pages where tests are executed. There is still one test called "basic_test" which basically consists from common tests (GApplication menu of gnome-software and starting/closing app). The GApplications tests were defined in actions.py, so it's actually ported as well.
Created attachment 278790 [details] [review] Automated tests extended and ported to Behave. Sorry, I attached the old patch. This is the correct one.
I've filed a bug to missing sample data. https://bugzilla.gnome.org/show_bug.cgi?id=731941