After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 698935 - Some installed tests improvements
Some installed tests improvements
Product: gjs
Classification: Bindings
Component: general
Other Linux
: Normal normal
: ---
Assigned To: gjs-maint
Depends on:
Reported: 2013-04-26 10:28 UTC by Giovanni Campagna
Modified: 2013-05-10 11:39 UTC
See Also:
GNOME target: ---
GNOME version: ---

Allow configuring the installed-tests directory from the toplevel (1.59 KB, patch)
2013-04-26 10:28 UTC, Giovanni Campagna
reviewed Details | Review
Don't run gjs-tests twice on make check (737 bytes, patch)
2013-04-26 10:28 UTC, Giovanni Campagna
committed Details | Review
Allow running installed tests with make check (1.18 KB, patch)
2013-04-26 10:38 UTC, Giovanni Campagna
none Details | Review
installed-tests: all is a reserved keyword in automake (1.66 KB, patch)
2013-05-07 16:11 UTC, Giovanni Campagna
accepted-commit_now Details | Review

Description Giovanni Campagna 2013-04-26 10:28:03 UTC
Yes, this installed tests thing is nice to get integration tests and automated smoketests and stuff like that, but it's kind of annoying if you're working on the code and constantly breaking it, as you generally don't want to install the new gjs and crash your shell at the next login.
Anyway, these are a few improvements I'd find useful to have.
Comment 1 Giovanni Campagna 2013-04-26 10:28:24 UTC
Created attachment 242536 [details] [review]
Allow configuring the installed-tests directory from the toplevel

This way one can get the test suite without calling configure manually,
which is useful in a jhbuild environment with complex configure
Comment 2 Giovanni Campagna 2013-04-26 10:28:36 UTC
Created attachment 242537 [details] [review]
Don't run gjs-tests twice on make check

Automake already invokes check-local if defined, we don't need
to depend on it.
Comment 3 Giovanni Campagna 2013-04-26 10:38:17 UTC
Created attachment 242539 [details] [review]
Allow running installed tests with make check

If installed tests are enabled, run them when invoking make check
on the toplevel.
Comment 4 Colin Walters 2013-04-26 10:50:57 UTC
Review of attachment 242536 [details] [review]:

The problem with this is that it will require an already-built version of gjs in your buildroot.  That's the normal case for people using jhbuild over time of course, but it would break the build from scratch if we added --enable-installed-tests to the moduleset.

I had been planning to add support for installed tests to jhbuild; e.g. it could detect for each module it built whether there's an installed-tests subdirectory, and if so after the normal build, go over each of those and build.  That'd hopefully be not too hard.  This is kind of what the gnome-ostree build system does, except it just has a list of modules with installed tests.

I completely understand how the installed-tests subdirectory approach is annoying at the moment for developers.  And it will also be traumatic for many existing distributions since they'd have to basically make a separate source package for the tests with a BuildDepends on the main package.  On the other hand, it is clearly independent of the main source.  The for gjs/installed-tests is nice and tiny.

So...that's not a "no", but let's discuss here.  Would jhbuild improvements be enough here for you?
Comment 5 Giovanni Campagna 2013-04-26 12:38:32 UTC
Well, ideally, there is a single command that rebuilds everything and runs the test suite on the new executables and libraries, without affecting the system or jhbuild installation, just like "make check" used too.
If that's not possible, even having a jhbuild command that runs the tests after installation would be ok.
Comment 6 Colin Walters 2013-04-26 13:43:50 UTC
Review of attachment 242537 [details] [review]:

Looks good, thanks!
Comment 7 Giovanni Campagna 2013-05-07 15:56:52 UTC
Comment on attachment 242537 [details] [review]
Don't run gjs-tests twice on make check

Attachment 242537 [details] pushed as 2b82afd - Don't run gjs-tests twice on make check
Comment 8 Giovanni Campagna 2013-05-07 16:11:48 UTC
Created attachment 243505 [details] [review]
installed-tests: all is a reserved keyword in automake

all is the default target, so you cannot have a program named
like that.
Comment 9 Colin Walters 2013-05-07 16:15:40 UTC
I'm considering changing gjs to the --enable-installed-tests model, because it's less painful for other extant build tools (jhbuild, rpm/dpkg).
Comment 10 Colin Walters 2013-05-07 16:21:24 UTC
Review of attachment 243505 [details] [review]:

Ah, right, thanks.  I think I'd prefer "alltests" or something but I'm fine with unit too.  Up to you.
Comment 11 Colin Walters 2013-05-09 23:07:02 UTC
Ok, I did this:

Let's do any further patches as a separate bug.
Comment 12 Simon McVittie 2013-05-10 11:39:45 UTC
Review of attachment 242539 [details] [review]:

There is already a name for "test my just-installed tree", and it's "make installcheck". It's rarely-implemented, but quite useful.

It looks as though this will fail when using a $(DESTDIR), though.

libdbus' test/ does all the variations:

* normal GNU check: tests in the $(builddir) against libraries in the $(builddir)

* normal GNU installcheck: tests in the $(builddir) against libraries in the $(DESTDIR)$(prefix)

* "installed tests" install tests in the $(prefix) that will use libraries in the $(prefix)

and in addition,

* if DESTDIR is empty, installcheck runs the installed tests

libdbus doesn't currently distinguish between black-box and white-box tests, it just installs everything that can pass, even if it doesn't really make sense (e.g. tests that are statically linked to a libdbus variant, so running them with the installed libdbus has no practical effect). This is mainly because the --enable-installed-tests infrastructure was originally written for a project (Maemo) that only ran regression tests as installed tests, and didn't routinely run "make check".