GNOME Bugzilla – Bug 698935
Some installed tests improvements
Last modified: 2013-05-10 11:39:45 UTC
Yes, this installed tests thing is nice to get integration tests and automated smoketests and stuff like that, but it's kind of annoying if you're working on the code and constantly breaking it, as you generally don't want to install the new gjs and crash your shell at the next login. Anyway, these are a few improvements I'd find useful to have.
Created attachment 242536 [details] [review] Allow configuring the installed-tests directory from the toplevel This way one can get the test suite without calling configure manually, which is useful in a jhbuild environment with complex configure calls.
Created attachment 242537 [details] [review] Don't run gjs-tests twice on make check Automake already invokes check-local if defined, we don't need to depend on it.
Created attachment 242539 [details] [review] Allow running installed tests with make check If installed tests are enabled, run them when invoking make check on the toplevel.
Review of attachment 242536 [details] [review]: The problem with this is that it will require an already-built version of gjs in your buildroot. That's the normal case for people using jhbuild over time of course, but it would break the build from scratch if we added --enable-installed-tests to the moduleset. I had been planning to add support for installed tests to jhbuild; e.g. it could detect for each module it built whether there's an installed-tests subdirectory, and if so after the normal build, go over each of those and build. That'd hopefully be not too hard. This is kind of what the gnome-ostree build system does, except it just has a list of modules with installed tests. I completely understand how the installed-tests subdirectory approach is annoying at the moment for developers. And it will also be traumatic for many existing distributions since they'd have to basically make a separate source package for the tests with a BuildDepends on the main package. On the other hand, it is clearly independent of the main source. The configure.ac for gjs/installed-tests is nice and tiny. So...that's not a "no", but let's discuss here. Would jhbuild improvements be enough here for you?
Well, ideally, there is a single command that rebuilds everything and runs the test suite on the new executables and libraries, without affecting the system or jhbuild installation, just like "make check" used too. If that's not possible, even having a jhbuild command that runs the tests after installation would be ok.
Review of attachment 242537 [details] [review]: Looks good, thanks!
Comment on attachment 242537 [details] [review] Don't run gjs-tests twice on make check Attachment 242537 [details] pushed as 2b82afd - Don't run gjs-tests twice on make check
Created attachment 243505 [details] [review] installed-tests: all is a reserved keyword in automake all is the default target, so you cannot have a program named like that.
I'm considering changing gjs to the --enable-installed-tests model, because it's less painful for other extant build tools (jhbuild, rpm/dpkg).
Review of attachment 243505 [details] [review]: Ah, right, thanks. I think I'd prefer "alltests" or something but I'm fine with unit too. Up to you.
Ok, I did this: https://git.gnome.org/browse/gjs/commit/?id=1f1293247653a36533639b5c6762fa968e6909b6 Let's do any further patches as a separate bug.
Review of attachment 242539 [details] [review]: There is already a name for "test my just-installed tree", and it's "make installcheck". It's rarely-implemented, but quite useful. It looks as though this will fail when using a $(DESTDIR), though. libdbus' test/Makefile.am does all the variations: * normal GNU check: tests in the $(builddir) against libraries in the $(builddir) * normal GNU installcheck: tests in the $(builddir) against libraries in the $(DESTDIR)$(prefix) * "installed tests" install tests in the $(prefix) that will use libraries in the $(prefix) and in addition, * if DESTDIR is empty, installcheck runs the installed tests libdbus doesn't currently distinguish between black-box and white-box tests, it just installs everything that can pass, even if it doesn't really make sense (e.g. tests that are statically linked to a libdbus variant, so running them with the installed libdbus has no practical effect). This is mainly because the --enable-installed-tests infrastructure was originally written for a project (Maemo) that only ran regression tests as installed tests, and didn't routinely run "make check".