GNOME Bugzilla – Bug 670790
should have an option to erase checkout after install
Last modified: 2021-05-17 15:52:48 UTC
after installing the module (or failing to do so), jhbuild should have an option to erase the checkout directory. this would let you have checkoutdir on a tmpfs, assuming the size of the largest build did not exceed available memory. that could substantially decrease build time and/or disk wear. the option would also be useful for avoiding the huge number of problems caused by things like autotool's inability to deal with files being moved as sort of a before-the-fact "wipe directory and start over". this would lead to increased build reliability and determinism. obviously it wouldn't be a good idea to turn this on without a dvcs mirror directory enabled as well.
Created attachment 208422 [details] [review] tmpfs Here's a patch that pretty much does what I want. It's obviously in no shape to commit. In order for this to work you need a line in fstab like so: none /home/desrt/jhbuild/tmpfs tmpfs user,exec,noatime 0 0 Unmounting each time is sort of nice because it's probably faster than manually 'rm -rf' all the contents and it's sure to do a 'complete' job. On the other hand, it can cause problems. If you're in the tmpfs directory (even as a cwd in a shell) then the umount will fail. The other approach to this problem is as I originally stated: just nuke the checkout dir when done with it.
Created attachment 208423 [details] [review] Add 'rm_after' config option This allows the checkout directory to be removed after a build completes. On the balance, I believe this approach to be more useful in general, and since it still addresses my usecase, it's probably the better option.
Comment on attachment 208423 [details] [review] Add 'rm_after' config option DO NOT TRY THIS PATCH get_srcdir() returns the toplevel checkout dir for meta packages and this runs 'rm -rf' on that. This clearly needs a bit more work. Glad it wasn't my homedir :)
Created attachment 208427 [details] [review] Add 'rm_after' config option This allows the checkout directory to be removed after a build completes. This is implemented by making the existing 'wipedir' functionality (used for the "wipe directory and start over" functionality) public and calling it after the build if 'rm_after' is set. This should be much safer...
"jhbuild should have an option to erase the checkout directory." For tarballs, erasing the checkout makes sense. For git modules, we should really just do: git clean -dfx; git reset --hard origin/master Downloading all of the e.g. gtk+ git repository after a failed build is a large hit. The problem with implementing this the matrix of [tarball/git/hg/cvs] x [autotools/cmake/waf] But like 95% of modules are tarball/autotools or git/autotools, so if we can just special case that somehow maybe with a hacky if isinstance() somewhere it'd be a win.
git clean doesn't solve my usecase: building on tmpfs. I really want the space back. With the dvcs mirror, it's not a problem.
(In reply to comment #6) > git clean doesn't solve my usecase: building on tmpfs. I really want the space > back. > > With the dvcs mirror, it's not a problem. Oh, with dvcs mirror, I see. Patch looks reasonable to me then.
I was actually considering adding a full phase for this: "cleanup" (which usually does nothing). Do you think that would be appropriate, or overkill?
(In reply to comment #8) > I was actually considering adding a full phase for this: "cleanup" (which > usually does nothing). Do you think that would be appropriate, or overkill? That might be an interesting approach. If we also allowed running 'git clean -dfx' there I'd probably use it.
Or honestly maybe the current 'clean' target is just broken and we should repurpose it for this. See 48ed4e4bb9d9914429888f8431bb2dcdc0dbfe8f
So here's my feelings on a few issues (thinking a bit bigger picture): - 'make clean' is almost useless in terms of building-fresh. It will basically only erase the things that would already have been rebuilt anyway on the basis of timestamps. Particularly, it won't fix any of the issues caused by needing to rerun autogen (eg: because someone moved a source file into deprecated/). - keeping source trees around as the basis for future builds is pretty much useless. if none of the dependencies changed then our default behaviour is not to rebuild them anyway, and if any dependency has changed then probably the newly-installed headers will cause everything to rebuild from scratch again anyway. the situation could be improved here in two ways: 1) avoid touching the timestamp while installing headers if the existing header has exactly the same content and 2) using ccache. - dvcs mirror and tarballdir should be on by default (see bug 669439) - out-of-tree builds are probably a complete waste of time for what we're doing here -- dvcs_mirror gives us almost the same thing, but better in every way and without the need to constantly file bugs against projects that aren't working properly. basically: i'm sick of fighting automake. - in terms of reliability and determinism in builds i think we'd be best to recommend to people that they should always rm -rf their checkoutdir and prefix before starting a full-gnome jhbuild. there are just too many weird issues that pop out due to having old files lying around while trying to build. i'm not sure if we should consider these bugs in individual gnome packages anymore, because fixing them is difficult and because jhbuild is the only situation in which they are really encountered.
(In reply to comment #11) > So here's my feelings on a few issues (thinking a bit bigger picture): > > - 'make clean' is almost useless in terms of building-fresh. It will > basically only erase the things that would already have been rebuilt > anyway on the basis of timestamps. Particularly, it won't fix any > of the issues caused by needing to rerun autogen (eg: because someone > moved a source file into deprecated/). Agree. > - keeping source trees around as the basis for future builds is pretty > much useless. Except - one of the whole points of jhbuild is for the developer to actually hack on a module. Clearly deleting their source tree is going to be problematic. What's your suggested story here? > - dvcs mirror and tarballdir should be on by default (see bug 669439) Probably, yeah. > - out-of-tree builds are probably a complete waste of time for what we're > doing here -- dvcs_mirror gives us almost the same thing, but better in > every way and without the need to constantly file bugs against projects > that aren't working properly. basically: i'm sick of fighting automake. Mmm...I'd like to at least annotate things with the Build API saying whether they build in-tree or out-of-tree (there are some projects out there that ONLY build out of tree). http://people.gnome.org/~walters/docs/build-api.txt (Eventually I'll get jhbuild to implement the API...)
(In reply to comment #12) > > - keeping source trees around as the basis for future builds is pretty > > much useless. > > Except - one of the whole points of jhbuild is for the developer to actually > hack on a module. Clearly deleting their source tree is going to be > problematic. What's your suggested story here? I've found that jhbuild is pretty bad for this, actually. What I do instead (and something I'd like to talk to you more about in general) is use a system of layers. What jhbuild installs for me is always more or less stock upstream stuff. If I had obvious fixes (ie: build fixes) -- stuff that will go upstream really really soon, then I sometimes do these here just to get it building again. In general, I never do 'real work' here, though. Instead, I have code/ that contains the git checkouts of things I actually hack on. I install those to ~/local. Then I do a layering approach with my LD_LIBRARY_PATH, PKG_CONFIG_PATH, XDG_DATA_DIRS and so on. For example, my LD_LIBRARY_PATH: /home/desrt/local/lib:/home/desrt/jhbuild/install/lib I routinely nuke ~/jhbuild/install and ~/jhbuild/checkout (and redo the build) and I also routinely nuke ~/local/ (and type again 'make install' from my code/ checkouts). This system works insanely well for me. I have my system stuff (/usr) plus stock upstream bleeding edge GNOME (via jhbuild, ~/jhbuild/install/) plus the stuff I'm actually working on (~/local). I have three separate gnome-terminal profiles: 1) stock gnome-terminal with no extra environment setup 2) jhbuild shell 3) what I call 'lcl' which is jhbuild shell layered with my ~/local After a lot of messing around, this is the only way to fly for me. I think modifying jhbuild checkout trees and then having jhbuild come along and try to automatically rebase your changes on top of upstream stuff is just asking for pain. ps: my desire to be able to randomly install things into yet another library path (~/local) and then remove them again just as easily as if they were never there is why I so desperately despise DT_RPATH (in case you were wondering). We're getting quite off topic here, but maybe it helps to see my motivations...
Review of attachment 208427 [details] [review]: Thank you for the patch. This option doesn't make sense with buildroot set, as you stated. JHBuild should error and alert the user if both buildroot and rm_after is set. Or implement buildroot support. A nitpick, I don't like the name 'rm_after'. I'd prefer something a little more descriptive like remove_checkout_dir. ::: jhbuild/frontends/buildscript.py @@ +212,3 @@ + (self.config.rm_after == 'success' and not failed)): + module.wipedir(self) + The above 4 lines need to be before self.end_module. Because wipedir failures will be attributed to the correct module. Important in tinderbox mode.
-- GitLab Migration Automatic Message -- This bug has been migrated to GNOME's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.gnome.org/GNOME/jhbuild/-/issues/127.