GNOME Bugzilla – Bug 349114
Integrate LDTP tests into jhbuild
Last modified: 2007-04-14 17:08:27 UTC
Need to integrate LDTP test cases into jhbuild/tinderbox/jhautobuild.
Created attachment 69845 [details] [review] patch to add `test' command and add ldtp support to jhbuild I am attaching a patch for the same. tests can be run in 2 ways: 1. Building the individual packages: ``jhbuild buildone ldtp-gedit-test'' 2. using the new test command: ``jhbuild test gedit'' this will run all test cases which describe gedit as a tested package. Also the test cases will run in an Xvfb environment.
For completeness, here is a copy of my first review, sent by email A minor thing, I would prefer module imports to each be on their own lines, especially when mixing standard modules and jhbuild modules. Another minor thing (I'm reading the diff linearly), avoid space changes on empty lines, so we get a cleaner diff. Question, make_option for noxvfb, isn't it duplicated ? (it appears both in commands/base.py and in command/test.py) frontends/autobuild.by: - unused global modules dict - lazy loading, load_tests should only be called the first time the value is necessary (end_phase when phase == 'test') (and self.modules would then be stored) - s/compress_date/compress_data/, just a typo - use os.path.join(xx, yy) instead of xx+os.sep+yy modtypes/testmodule.py: - why inherit from AutogenModule ? wouldn't Package be enough ? If you want to use AutogenModule, don't duplicate code. - _check_ldtp_log_file & friends, is all this work necessary, formatting a status string and stuff ? Or would it be enough to look for a <pass> node with 0 as value ?
The patch attached is the modified one.
More comments: - Xvfb, Debian has a xvfb-run script to start Xvfb, execute a command in it and close Xvfb, perhaps it would be best to get an helper script of that sort instead of doing all the work inside testmodule.py ? - display :1 is hardcoded while in many places it will already be used (xvfb-run dynamically looks for an available display) - a new Xvfb session doesn't allow for external connections (at least in Debian default configuration), needs to play with xauth (xvfb-run does this). Alternative is to set XAUTHORITY like this, before self._start_xvfb os.environ['XAUTHORITY'] = '/tmp/jhbuild.xauthority' os.system('xauth add "%s" "." "%s"' % (display, random.random())) - it may be necessary to reset LANG/LC_ALL/etc. to C for test scripts to run properly - in do_dogtail_test, e.returnvalue should be e.returncode.
Created attachment 70364 [details] [review] Fixed the display number selection (based on the xvfb-run script) and the XAUTHORITY environment As for the environment being defaulted to `C', I think jhbuild/frontends/autobuild.py already does this: for k in os.environ.keys(): if k.startswith('LC_'): os.environ[k] = 'C' otherwise isnt it safe to assume that the user already has set the locale settings correctly? (After all he is already running a Gnome session)
modtypes/testmodule.py - getting display number: - why not os.listdir() instead of os.walk() - or doing it like debian xvfb-run while True: if not os.path.exists(...): break servernum += 1 - xauthority - my fault, but xauth last arg (mcookie) should be in hex, md5.md5(...).hexdigest() frontends/autobuild.py - could _upload_logfile be cut out of ldtp stuff, for easier dogtail (and more) integration ? something like: def _upload_ldtp_logfile(self, module): test_module = self.modules.modules[module] src_dir = test_module.get_srcdir() if not os.path.exists (os.path.join(src_dir, 'run.xml')): return logfile = test_module.get_ldtp_log_file(os.path.join(src_dir,'run.xml')) if not os.path.exists (logfile): return self._upload_logfile(module, logfile, 'application/x-ldtp+xml') def _upload_logfile(self, module, logfile, mimetype): logfd = open(logfile) basename = os.path.basename(logfile) self.server.attach_file(self.build_id, module, 'test', basename, compress_data(logfd.read()), mimetype) logfd.close()
Created attachment 70558 [details] [review] I have modified the changed you had commented on I was wondering, you were looking at a general <attachments> tag in the moduleset, maybe we could convert _upload_logfile() to a _upload_file() which will be able to handle binary files too. We could treat the logfiles as binary too and process it based on the mime type at the server. If we were to upload the binary files, I guess we will have to convert it based on base64 and then upload it? Any views on this?
Let's keep generic attachment support for later. I patched another copy of jhbuild with your previous patch and there is a new issue; I had to define xvfbargs in defaults.jhbuildrc (empty at first, then overriden to get a 24-bit visual). Also, modtypes/testmodule.py lacks a get_builddir, used for falling back to shell in interactive method, it is ok to define it to return self.get_srcdir(...).
I changed patch status to none so James notices it must be reviewed. James ?
James, can you look at this please? Thanks.
Created attachment 74121 [details] [review] Fix a typo (screenum->screennum) in dogtail support This fixes a one-character typo in the patch (screenum -> screennum). I've tested the fixed patch and works correctly and easily with dogtail.
James, can you please look at this patch and review it so that we can get it in? Thanks.
I think it would be nice to have tests integrated into application sources and run just by "make check". Otherwise it would be really hard to keep tests in sync with application.
There is already support for make check in jhbuild. Also, having tests in a different module is not mandatory, we could support LDTP/Dogtail tests integrated into application sources as well. There are comments from David Malcolm wrt this issue: http://mail.gnome.org/archives/dogtail-devel-list/2006-August/msg00001.html I quote the relevant part: Whether tests should live with the software they test, or in a separate module is a much-debated area. I think we should allow modules to have tests live inside them, if possible: an example of a module with inline dogtail tests is Frysk ( http://sourceware.org/frysk/ ) where the tests for the java-gnome GUI are deeply embedded within the rest of the test suite: http://sources.redhat.com/cgi-bin/cvsweb.cgi/frysk-gui/frysk/gui/test/dogtail_scripts/?cvsroot=frysk The caveat is that such a module can't be within the dependency tree for dogtail, or you'll run into circular dependency problems (I think). With my "distribution" hat on I have a preference for out-of-module test suites since it makes packaging easier: I can run the tests on an installed package on a live distribution, without having to deal with a build system. Though obviously this is heading off-topic for a tinderbox-style setup.
Created attachment 85943 [details] [review] LDTP and Dogtail jhbuild support I updated to trunk and fixed a few stylistic issues; could someone check it works correctly (I don't have a 2.14 system to execute those tests), command is: jhbuild -m gnome-2.14-test test gedit To LDTP folks: care to submit a gnome-2.14-test.modules for 2.18 or 2.20 ? (which should probably be named gnome-tests-2.20.modules).
It still needs updated module sets for newer GNOME versions. 2007-04-14 Frederic Peters <fpeters@0d.be> * jhbuild/commands/base.py, jhbuild/frontends/autobuild.py, jhbuild/frontends/terminal.py, jhbuild/frontends/tinderbox.py, jhbuild/moduleset.py, jhbuild/main.py, jhbuild/config.py, jhbuild/defaults.jhbuildrc, modulesets/gnome-2.14.modules, modulesets/gnome-2.14-test.modules: integrate LDTP and Dogtail tests, most of the work done by Prashanth Mohan during Google SoC 2006. (Closes: #308352 and #349114)