After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 318531 - Support dogtail scripts in GNOME tinderbox
Support dogtail scripts in GNOME tinderbox
Status: RESOLVED FIXED
Product: jhbuild
Classification: Infrastructure
Component: module sets
unspecified
Other Linux
: Normal enhancement
: ---
Assigned To: James Henstridge
Jhbuild QA
Depends on: 318534 318535 318537
Blocks:
 
 
Reported: 2005-10-11 04:18 UTC by Dave Malcolm
Modified: 2008-10-18 10:24 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Patch which adds pyspi and dogtail to gnome-2-14.modules (1.13 KB, patch)
2005-10-11 04:22 UTC, Dave Malcolm
committed Details | Review
Proposed new moduleset to hold automated test cases for GNOME 2.14 (390 bytes, text/xml)
2005-10-11 04:27 UTC, Dave Malcolm
  Details
Updated version of gnome-2.14-testsuite.modules (512 bytes, text/plain)
2005-10-11 05:26 UTC, Dave Malcolm
  Details
Updated moduleset, following the new DTD, for gnome-2.16 (561 bytes, application/xml)
2006-08-01 16:49 UTC, Dave Malcolm
  Details

Description Dave Malcolm 2005-10-11 04:18:31 UTC
As discussed in the QA BOF at the GNOME summit, it would be good to get
automated testing support into a tinderbox system for GNOME:
http://mail.gnome.org/archives/gnome-bugsquad/2005-October/msg00000.html
Comment 1 Dave Malcolm 2005-10-11 04:22:32 UTC
Created attachment 53313 [details] [review]
Patch which adds pyspi and dogtail to gnome-2-14.modules

pyspi and dogtail don't yet build cleanly in a jhbuild environment.
Would applying this patch break things for everyone?  IIRC jhbuild only tries
to build the modules (and dependencies thereof) that you specify, so is this
applyable now?
Comment 2 Dave Malcolm 2005-10-11 04:27:49 UTC
Created attachment 53314 [details]
Proposed new moduleset to hold automated test cases for GNOME 2.14

Attached file is a proposed structure for holding tests.

I've kept the name ("gnome-2.14-testsuite") agnostic wrt which test framework
is used (Dogtail/LDTP); tests written in either/both framework could be
contained within.
Comment 3 Dave Malcolm 2005-10-11 04:35:50 UTC
Looks like gnome bugzilla didn't get the filename of attachment 53314 [details].  It's
meant to be named "gnome-2.14-testsuite.modules", which hopefully makes the
final paragraph of my last comment make more sense.
Comment 4 Dave Malcolm 2005-10-11 05:26:01 UTC
Created attachment 53315 [details]
Updated version of gnome-2.14-testsuite.modules

I forgot to specify the cvsroot for gnome in the new moduleset
Comment 5 Zack Cerza 2005-11-16 17:59:32 UTC
Marking as an enhancement. Shouldn't this bug be filed on jhbuild, anyway?
Comment 6 Dave Malcolm 2005-11-16 18:52:37 UTC
Good point.

AFAIK pyspi and dogtail don't quite build yet in a jhbuild environment; see
comment #1   Perhaps we should commit the modules anyway, since I don't think it
affects people who aren't working with dogtail, and it'll make it easier to fix
the problems.

Reassigning to "jhbuild" product to get feedback from them.
Comment 7 Murray Cumming 2005-12-05 15:23:35 UTC
Dave, don't wait around for someone to give you permission. Just add it to jhbuild. 
Comment 8 James Henstridge 2005-12-06 07:19:59 UTC
Didn't realise that there were patches on this bug.  Feel free to commit
attachment 53313 [details] [review].

As far as a test suite goes, wouldn't it make more sense to include the tests
with the associated modules, so that they can get run with "make check"?  That
way, if a tinderbox is set to run "make check", they'll call the tests.

(it also means that the tests will be run on "make distcheck", which couldn't hurt).
Comment 9 Dave Malcolm 2005-12-07 20:01:40 UTC
(i) Re:  attachment 53313 [details] [review]:
I've committed it as it's a step in the right direction, and AIUI shouldn't
break anything.

However at the moment I can't build pyspi under jhbuild.  IIRC if I build
without bootstrapping, the module doesn't know where it should be installed
(perhaps we need to transition the project to the GNU autotools?), and if I
build with bootstrapping it fails earlier since it needs Pyrex (perhaps add this
as an explicit dependency in the jhbuild file)

(ii) re including the tests inline in the modules:
IIRC our view at the QA BOF at the 2005 gnome summit was that we wanted to allow
"out-of-band" tests.

There are a few reasons for wanting to allow out-of-band tests:
  (a) not all tests relate to a specific module.  Think of integration tests
such as starting a session and measuring startup time, clicking on every
launcher in the panel and checking that every app starts, that help works in
every app, etc.  Basically anywhere involving more than one app. 
  (b) "inline" tests that require dogtail or LDTP would add a dependency on
dogtail/LDTP to each module as a whole (and hence on e.g. python) and I think
neither of our testing projects are mature enough for the various gnome modules
to want that.
  (c) LDTP is storing its tests in a separate repository anyway, and so I'm
guessing that it's going to be easier for them if jhbuild is set up to allow
out-of-band tests
  (d) putting my Red Hat hat on (I suspect this view is shared by other
distributions packaging GNOME) we'd greatly prefer tests to be in a separate
tarball from the source tarball, since it means we can run tests on a package in
a separate environment from our build environment, without having to continually
do rebuilds, or having to hack the tests into a separate module.

Of course nothing rules out individual module maintainers from adding
dogtail/LDTP test case into their own modules, what I want is an out-of-band
location for them to live 

Does this make more sense?
Comment 10 Dave Malcolm 2006-08-01 16:49:02 UTC
Created attachment 70037 [details]
Updated moduleset, following the new DTD, for gnome-2.16
Comment 11 Frederic Peters 2007-04-14 17:17:06 UTC
Dave: I commited last year SoC which added support for LDTP and Dogtail tests.

It works like this:

  <testmodule id="dogtail-gedit-test" type="dogtail">
    <branch module="dogtail-tests/trunk/gedit" />
    <dependencies>
      <dep package="dogtail"/>
      <dep package="gedit"/>
    </dependencies>
    <testedmodules>
      <tested package="gedit"/>
    </testedmodules>
  </testmodule>

This + a moduleset with modules from dogtail-tests/ and I believe this bug could be fixed.  What do you think?
Comment 12 Murray Cumming 2007-05-01 15:29:51 UTC
So, is this bug now fixed?
Comment 13 Frederic Peters 2008-10-18 10:24:35 UTC
I believe it to be fixed, marked it so.