GNOME Bugzilla – Bug 721818
generating and extending InstalledTests
Last modified: 2021-06-05 16:29:36 UTC
Right now we have a custom "applicationtest" that starts/stop each app. I've been thinking lately though that InstalledTests has been so successful as a framework that I'd like to push more testing that way. We could add some code that *generates* InstalledTests. For example, we could statically inspect the disk, loop over /usr/share/applications, and generate a test named: gnome-continuous/${appname}/start-stop The reason I mention this is Matthias specifically mentioned RTL as something that we should test. It'd be easy then to do: gnome-continuous/${appname}/start-stop-rtl Or perhaps we want to boot the whole OS with LANG=he_IL.UTF-8? We could also generate many other types of tests. Say that we had a test that entered/exited the shell overview. We could run that test at different resolutions too.
InstalledTests could use two new features: * The ability to request a screenshot (likely custom to Continuous for now) * A way to export test result data; for example, if an installed test creates a directory "test-results", we grab that from the guest.
as mentioned in the discussion, having a screenshot api and a way to export test results (including those shots) would be very useful to just generate more screenshots, so we can get more than just the initial screen - it could be combined with a11y-driven tests that 'click through' some of the app ui.
Created attachment 269815 [details] [review] Store test results in a separate dir
Created attachment 274008 [details] [review] Store results in installed-test-results dir in workdir
(In reply to comment #4) > Created an attachment (id=274008) [details] [review] > Store results in installed-test-results dir in workdir This patch also solves test failures (Child process killed by signal 13), which happen due to ratelimiting being off. After it gets applied we can update the UI to turn failed test names into a links to a test logs
Review of attachment 274008 [details] [review]: Cool, this looks good to me.
Review of attachment 274008 [details] [review]: Pushed as https://git.gnome.org/browse/gnome-continuous/commit/?id=ccf06f4bca88f9d8e9df86833a98c0d40ff2472e
gnome-continuous is not under active development anymore. Its codebase has been archived: https://gitlab.gnome.org/Archive/gnome-continuous Closing all its open tickets as part of housekeeping.