GNOME Bugzilla – Bug 313997
make jhbuild eleventy billion times faster
Last modified: 2007-12-16 12:11:46 UTC
Right now, jhbuild's default build mode always reruns the install step, even if there were no changes. This is quite slow on my laptop. But the default mode of operation generally doesn't work, because many modules need to have autogen.sh rerun (the -a) option. This appears to be mostly autotools suckage. So if I pass the -a flag, then every module gets the configure/build/install, even if there were no changes. This is very, very, very slow. I don't want to use -n, because I *do* want to update to the latest GNOME. I just don't want to wait while all these modules that have already been built are needlessly rebuilt. Hence the patch I will attach shortly, which our lab testing has revealed makes jhbuild approximately eleventy billion times faster for the "run latest GNOME" case (assuming you have a relatively recent previous installation). With this patch, it might make sense for the default mode of operation to become --autogen, so things work more consistently. I bet many fewer people will use -n now too.
Created attachment 51015 [details] [review] really: eleventy billion
So the basic algorithm here is: if CVS doesn't say it updated any files, don't rebuild the module If a module's dependencies get rebuilt though, shouldn't the module be rebuilt to see if it still works with the new version of the dep, even if it hasn't changed itself? Working out whether a module really needs to be rebuilt would be a lot easier with tree-wide revision numbers ...
That's a good point. Now, we will still get the new dep installed, so any binary breakage (ABI or internal) will be detected. However it is true that e.g. if a module header breaks, we will not detect it unless something in the stack depending on it is recompiled. My opinion on that is that the tinderbox should just always do full builds (I provided a --force-build option), and we can catch breakage that way. Every developer doesn't need to be doing full rebuilds though. Those that want to can also set the force_build option too.
Thinking about this a bit more, with this patch, jhbuild operates much like how many GNOME distributors do. None that I know of rebuild the entire stack when a dependent module changes. So having jhbuild operate in this way will let us catch classes of bugs (e.g. binary incompatibility) that would not be caught before by jhbuild users, but *would* be hit by binary GNOME distributors. That seems quite valuable to me.
Conversely, not rebuilding when dependencies changes can hide source API breakages. Also, it looks like this would cause problems if "cvs update" was run by anyone other than jhbuild -- the code seems to assume "no changes since last update" is the same as "no changes since last build", which is not necessarily the same thing.
I agreed it could hide source breakage; but you don't feel the tinderbox and or having some people using --force-build is sufficient for addressing that? However, some modules do always require rebuilds of dependent applications since they're not API/ABI stable; e.g. libwnck, dbus, etc. We can have an additional flag in the module XML marking them as "unstable", meaning a change in them requires rebuild of dependent applications. I can work on adding support for this if you agree. As for running "cvs update" outside of jhbuild; sure...but why would you do that? The main scenario for that is a module you're actively hacking on; in that case, why wouldn't you also run "make && make install" manually too? Or alternatively, we make 'jhbuild buildone' always do a build. For extra points we have 'jhbuild buildone' look in the current directory for the module name. In any case, the semantics here should be easy for GNOME hackers to understand, I think in general they will appreciate the tradeoffs involved in the speedup, and if they don't, there is always the force-build option, right?
Comment on attachment 51015 [details] [review] really: eleventy billion Rejecting due to problematic method of determining if tree has changed (it should be storing some fingerprint of the tree state when building, and testing if that fingerprint has changed next time). I'm still not convinced that this behaviour is desirable though.
This patch ROCKS! Thanks Colin! I did notice a few issues: o --force-build didn't work with buildone o printing cvs update conflicts didn't work o not all calls of buildscript.execute were updated despite API change I'll add a patch which fixes those issues in a minute (yes, and I'll have it marked as rejected since it doesn't change James' fundamental objection). That patch also avoids rebuilding merely due to translation updates; I know that's probably not something that could be accepted but the patch is already rejected anyway, right? ;-) Do you happen to have any other updates to this patch as well?
Created attachment 57861 [details] [review] Fixed a couple bugs
Created attachment 65483 [details] cvs-wc-version.py I think it would be useful to provide 3 policies for builds here: 1. build everything requested (current behaviour) 2. build only modules that have changed 3. build modules that have changed, or their dependencies have changed. As I said in a previous comment, part of my problem with the patch is its method of determining if a module has changed. The attached script provides a way of generating an ID for a CVS working copy state. If the ID differs after a "cvs update" from the ID recorded from the previous build, then the tree has changed since the last build. While it may look expensive, it should be fairly cheap right after a "cvs update", since all the files it accesses should still be in the buffer cache. For other version control systems, we can use the following: Subversion: a combination of the "URL" and "Last Changed Rev" values output by "svn info". Arch: `baz tree-id` Bzr: The revision ID for the tree. Tarballs: some combination of the tarball version and the set of patches applied (possibly hashed like the cvs ID). GIT: contents of ./.git/HEAD, maybe? Darcs: collection of patch IDs, probably hashed.
Finally I decided to tackle this. from defaults.jhbuildrc: # policy for modules to build, options are: # - all: build everything requested (default behaviour) # - updated: build only modules that have changed # - updated-deps: build modules that have changed, or their dependencies # have changed. build_policy = 'all' 2007-12-16 Frederic Peters <fpeters@0d.be> * jhbuild/config.py, jhbuild/defaults.jhbuildrc, jhbuild/modtypes/__init__.py, jhbuild/modtypes/autotools.py, jhbuild/modtypes/distutils.py, jhbuild/modtypes/mesa.py, jhbuild/modtypes/mozillamodule.py, jhbuild/modtypes/perl.py, jhbuild/versioncontrol/arch.py, jhbuild/versioncontrol/bzr.py, jhbuild/versioncontrol/darcs.py, jhbuild/versioncontrol/git.py, jhbuild/versioncontrol/hg.py: added possibility to skip modules that have not been updated (with/without considering dependencies). This is driven by a new build_policy configuration option, possible values are 'all' (default behaviour, build everything), 'updated' (build only modules that have changed) and 'updated-deps' (build modules that have changed, or their dependencies have changed). (closes: #313997)
*** Bug 348688 has been marked as a duplicate of this bug. ***