After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 313997 - make jhbuild eleventy billion times faster
make jhbuild eleventy billion times faster
Product: jhbuild
Classification: Infrastructure
Component: general
Other Linux
: Normal enhancement
: ---
Assigned To: James Henstridge
Jhbuild QA
: 348688 (view as bug list)
Depends on:
Reported: 2005-08-20 01:44 UTC by Colin Walters
Modified: 2007-12-16 12:11 UTC
See Also:
GNOME target: ---
GNOME version: ---

really: eleventy billion (14.88 KB, patch)
2005-08-20 01:45 UTC, Colin Walters
rejected Details | Review
Fixed a couple bugs (17.46 KB, patch)
2006-01-22 18:12 UTC, Elijah Newren
rejected Details | Review (1.81 KB, text/plain)
2006-05-15 10:21 UTC, James Henstridge

Description Colin Walters 2005-08-20 01:44:31 UTC
Right now, jhbuild's default build mode always reruns the install step, even if
there were no changes.  This is quite slow on my laptop.
But the default mode of operation generally doesn't work, because many modules
need to have rerun (the -a) option.  This appears to be mostly
autotools suckage.

So if I pass the -a flag, then every module gets the configure/build/install,
even if there were no changes.  This is very, very, very slow.

I don't want to use -n, because I *do* want to update to the latest GNOME.  I
just don't want to wait while all these modules that have already been built are
needlessly rebuilt.

Hence the patch I will attach shortly, which our lab testing has revealed makes
jhbuild approximately eleventy billion times faster for the "run latest GNOME"
case (assuming you have a relatively recent previous installation).

With this patch, it might make sense for the default mode of operation to become
--autogen, so things work more consistently.  I bet many fewer people will use
-n now too.
Comment 1 Colin Walters 2005-08-20 01:45:18 UTC
Created attachment 51015 [details] [review]
really: eleventy billion
Comment 2 James Henstridge 2005-08-20 02:47:03 UTC
So the basic algorithm here is:
 if CVS doesn't say it updated any files, don't rebuild the module

If a module's dependencies get rebuilt though, shouldn't the module be rebuilt
to see if it still works with the new version of the dep, even if it hasn't
changed itself?

Working out whether a module really needs to be rebuilt would be a lot easier
with tree-wide revision numbers ...
Comment 3 Colin Walters 2005-08-20 03:09:27 UTC
That's a good point.  Now, we will still get the new dep installed, so any
binary breakage (ABI or internal) will be detected.  However it is true that
e.g. if a module header breaks, we will not detect it unless something in the
stack depending on it is recompiled.

My opinion on that is that the tinderbox should just always do full builds (I
provided a --force-build option), and we can catch breakage that way.  Every
developer doesn't need to be doing full rebuilds though.  Those that want to can
also set the force_build option too.
Comment 4 Colin Walters 2005-08-20 03:15:59 UTC
Thinking about this a bit more, with this patch, jhbuild operates much like how
many GNOME distributors do.  None that I know of rebuild the entire stack when a
dependent module changes.  So having jhbuild operate in this way will let us
catch classes of bugs (e.g. binary incompatibility) that would not be caught
before by jhbuild users, but *would* be hit by binary GNOME distributors.  That
seems quite valuable to me.
Comment 5 James Henstridge 2005-08-20 16:07:16 UTC
Conversely, not rebuilding when dependencies changes can hide source API breakages.

Also, it looks like this would cause problems if "cvs update" was run by anyone
other than jhbuild -- the code seems to assume "no changes since last update" is
the same as "no changes since last build", which is not necessarily the same thing.
Comment 6 Colin Walters 2005-08-20 20:08:17 UTC
I agreed it could hide source breakage; but you don't feel the tinderbox and or
having some people using --force-build is sufficient for addressing that?  

However, some modules do always require rebuilds of dependent applications since
they're not API/ABI stable; e.g. libwnck, dbus, etc.  We can have an additional
flag in the module XML marking them as "unstable", meaning a change in them
requires rebuild of dependent applications.  I can work on adding support for
this if you agree.

As for running "cvs update" outside of jhbuild; sure...but why would you do
that?  The main scenario for that is a module you're actively hacking on; in
that case, why wouldn't you also run "make && make install" manually too?
Or alternatively, we make 'jhbuild buildone' always do a build.  For extra
points we have 'jhbuild buildone' look in the current directory for the module name.

In any case, the semantics here should be easy for GNOME hackers to understand,
I think in general they will appreciate the tradeoffs involved in the speedup,
and if they don't, there is always the force-build option, right?
Comment 7 James Henstridge 2005-10-24 07:24:49 UTC
Comment on attachment 51015 [details] [review]
really: eleventy billion

Rejecting due to problematic method of determining if tree has changed (it
should be storing some fingerprint of the tree state when building, and testing
if that fingerprint has changed next time).

I'm still not convinced that this behaviour is desirable though.
Comment 8 Elijah Newren 2006-01-22 18:11:41 UTC
This patch ROCKS!  Thanks Colin!

I did notice a few issues:
  o --force-build didn't work with buildone
  o printing cvs update conflicts didn't work
  o not all calls of buildscript.execute were updated despite API change
I'll add a patch which fixes those issues in a minute (yes, and I'll have it marked as rejected since it doesn't change James' fundamental objection).  That patch also avoids rebuilding merely due to translation updates; I know that's probably not something that could be accepted but the patch is already rejected anyway, right?  ;-)

Do you happen to have any other updates to this patch as well?
Comment 9 Elijah Newren 2006-01-22 18:12:30 UTC
Created attachment 57861 [details] [review]
Fixed a couple bugs
Comment 10 James Henstridge 2006-05-15 10:21:24 UTC
Created attachment 65483 [details]

I think it would be useful to provide 3 policies for builds here:
 1. build everything requested (current behaviour)
 2. build only modules that have changed
 3. build modules that have changed, or their dependencies have changed.

As I said in a previous comment, part of my problem with the patch is its method of determining if a module has changed.

The attached script provides a way of generating an ID for a CVS working copy state.  If the ID differs after a "cvs update" from the ID recorded from the previous build, then the tree has changed since the last build.

While it may look expensive, it should be fairly cheap right after a "cvs update", since all the files it accesses should still be in the buffer cache.

For other version control systems, we can use the following:

Subversion: a combination of the "URL" and "Last Changed Rev" values output by "svn info".

Arch: `baz tree-id`

Bzr: The revision ID for the tree.

Tarballs: some combination of the tarball version and the set of patches applied (possibly hashed like the cvs ID).

GIT: contents of ./.git/HEAD, maybe?

Darcs: collection of patch IDs, probably hashed.
Comment 11 Frederic Peters 2007-12-16 12:10:36 UTC
Finally I decided to tackle this.

from defaults.jhbuildrc:
 # policy for modules to build, options are:
 #  - all: build everything requested (default behaviour)
 #  - updated: build only modules that have changed
 #  - updated-deps: build modules that have changed, or their dependencies
 #    have changed.
 build_policy = 'all'

2007-12-16  Frederic Peters  <>

        * jhbuild/, jhbuild/defaults.jhbuildrc, 
        jhbuild/modtypes/, jhbuild/modtypes/,
        jhbuild/modtypes/, jhbuild/modtypes/,
        jhbuild/modtypes/, jhbuild/modtypes/,
        jhbuild/versioncontrol/, jhbuild/versioncontrol/,
        jhbuild/versioncontrol/, jhbuild/versioncontrol/,
        jhbuild/versioncontrol/ added possibility to skip modules that
        have not been updated (with/without considering dependencies).  This
        is driven by a new build_policy configuration option, possible values
        are 'all' (default behaviour, build everything), 'updated' (build only
        modules that have changed) and 'updated-deps' (build modules that have
        changed, or their dependencies have changed).  (closes: #313997)
Comment 12 Frederic Peters 2007-12-16 12:11:46 UTC
*** Bug 348688 has been marked as a duplicate of this bug. ***