GNOME Bugzilla – Bug 318531
Support dogtail scripts in GNOME tinderbox
Last modified: 2008-10-18 10:24:35 UTC
As discussed in the QA BOF at the GNOME summit, it would be good to get automated testing support into a tinderbox system for GNOME: http://mail.gnome.org/archives/gnome-bugsquad/2005-October/msg00000.html
Created attachment 53313 [details] [review] Patch which adds pyspi and dogtail to gnome-2-14.modules pyspi and dogtail don't yet build cleanly in a jhbuild environment. Would applying this patch break things for everyone? IIRC jhbuild only tries to build the modules (and dependencies thereof) that you specify, so is this applyable now?
Created attachment 53314 [details] Proposed new moduleset to hold automated test cases for GNOME 2.14 Attached file is a proposed structure for holding tests. I've kept the name ("gnome-2.14-testsuite") agnostic wrt which test framework is used (Dogtail/LDTP); tests written in either/both framework could be contained within.
Looks like gnome bugzilla didn't get the filename of attachment 53314 [details]. It's meant to be named "gnome-2.14-testsuite.modules", which hopefully makes the final paragraph of my last comment make more sense.
Created attachment 53315 [details] Updated version of gnome-2.14-testsuite.modules I forgot to specify the cvsroot for gnome in the new moduleset
Marking as an enhancement. Shouldn't this bug be filed on jhbuild, anyway?
Good point. AFAIK pyspi and dogtail don't quite build yet in a jhbuild environment; see comment #1 Perhaps we should commit the modules anyway, since I don't think it affects people who aren't working with dogtail, and it'll make it easier to fix the problems. Reassigning to "jhbuild" product to get feedback from them.
Dave, don't wait around for someone to give you permission. Just add it to jhbuild.
Didn't realise that there were patches on this bug. Feel free to commit attachment 53313 [details] [review]. As far as a test suite goes, wouldn't it make more sense to include the tests with the associated modules, so that they can get run with "make check"? That way, if a tinderbox is set to run "make check", they'll call the tests. (it also means that the tests will be run on "make distcheck", which couldn't hurt).
(i) Re: attachment 53313 [details] [review]: I've committed it as it's a step in the right direction, and AIUI shouldn't break anything. However at the moment I can't build pyspi under jhbuild. IIRC if I build without bootstrapping, the module doesn't know where it should be installed (perhaps we need to transition the project to the GNU autotools?), and if I build with bootstrapping it fails earlier since it needs Pyrex (perhaps add this as an explicit dependency in the jhbuild file) (ii) re including the tests inline in the modules: IIRC our view at the QA BOF at the 2005 gnome summit was that we wanted to allow "out-of-band" tests. There are a few reasons for wanting to allow out-of-band tests: (a) not all tests relate to a specific module. Think of integration tests such as starting a session and measuring startup time, clicking on every launcher in the panel and checking that every app starts, that help works in every app, etc. Basically anywhere involving more than one app. (b) "inline" tests that require dogtail or LDTP would add a dependency on dogtail/LDTP to each module as a whole (and hence on e.g. python) and I think neither of our testing projects are mature enough for the various gnome modules to want that. (c) LDTP is storing its tests in a separate repository anyway, and so I'm guessing that it's going to be easier for them if jhbuild is set up to allow out-of-band tests (d) putting my Red Hat hat on (I suspect this view is shared by other distributions packaging GNOME) we'd greatly prefer tests to be in a separate tarball from the source tarball, since it means we can run tests on a package in a separate environment from our build environment, without having to continually do rebuilds, or having to hack the tests into a separate module. Of course nothing rules out individual module maintainers from adding dogtail/LDTP test case into their own modules, what I want is an out-of-band location for them to live Does this make more sense?
Created attachment 70037 [details] Updated moduleset, following the new DTD, for gnome-2.16
Dave: I commited last year SoC which added support for LDTP and Dogtail tests. It works like this: <testmodule id="dogtail-gedit-test" type="dogtail"> <branch module="dogtail-tests/trunk/gedit" /> <dependencies> <dep package="dogtail"/> <dep package="gedit"/> </dependencies> <testedmodules> <tested package="gedit"/> </testedmodules> </testmodule> This + a moduleset with modules from dogtail-tests/ and I believe this bug could be fixed. What do you think?
So, is this bug now fixed?
I believe it to be fixed, marked it so.