GNOME Bugzilla – Bug 647231
Add "workdir", use make install DESTDIR=, and clean .la files
Last modified: 2011-07-10 03:42:16 UTC
Work around automake installing this garbage, see: https://live.gnome.org/GnomeShell/RemovingLaFiles
Created attachment 185555 [details] [review] autotools: Remove .la files post-install
Review of attachment 185555 [details] [review]: There is a end_phase hook defined in the base class buildscript.BuildScript that I believe is well suited to this task. Something like: def end_phase(self, module, phase, error): if not error and phase == 'install': _clean_la_files(...) Also maybe handle the OSError exception, and ignore it.
(In reply to comment #2) > Review of attachment 185555 [details] [review]: > > There is a end_phase hook defined in the base class buildscript.BuildScript > that I believe is well suited to this task. That hook is intended for frontend subclasses to override for presentation purposes; it has little to do with post processing the install. Admittedly the autotools module isn't quite right either since a waf or cmake build could also install .la files. So, let's make a new postprocess.py in modtypes.
Created attachment 186631 [details] [review] Add a "workdir" (defaults to $prefix/jhbuild), change PackageDB to use it For a future change to use "make install DESTDIR=$workdir", which in turn will enable things like "jhbuild uninstall". This patch adds an explicit concept in jhbuild of a directory for build state called "workdir", distinct from the installation tree. We previously had something like this in the $prefix/share/jhbuild directory, which is where it put the packagedb.xml file. Add a workdir variable, and move packagedb.xml in there.
Created attachment 186632 [details] [review] autotools: Use make install DESTDIR= Like most "package managers", use DESTDIR= to create a temporary installation tree, from which we can generate a file manifest.
Created attachment 186633 [details] [review] Clean .la files post-install These do not serve any useful purpose (supposing one's software system supports "uninstall"), and are actively harmful in other scenarios; see https://live.gnome.org/GnomeShell/RemovingLaFiles
Created attachment 186848 [details] [review] Add a "workdir" (defaults to $prefix/jhbuild), change PackageDB to use it Reattaching for ordering
Created attachment 186849 [details] [review] autotools: Use make install DESTDIR= Much more extensive now; tested and working
Created attachment 186850 [details] [review] Clean .la files post-install Rebased on top of earlier commits
Review of attachment 186849 [details] [review]: ::: jhbuild/modtypes/autotools.py @@ +70,1 @@ The autotools modtype has been used for many modules that offered something close to "configure; make; make install" but were definitely not real autotools; this is the reason for many of the attributes and I think we need another one to set "supports_install_destir" to False. ::: jhbuild/utils/packagedb.py @@ +34,3 @@ return time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime(tm)) +def _get_text_content(node): I'd like to take this opportunity to switch this module from xml.dom.minidom to ElementTree.
Created attachment 187238 [details] [review] scripts: Clean up Makefile The append pattern ends up being a lot less of a mess.
Created attachment 187239 [details] [review] Add --binpkg option for "buildone" This moves us architecturally even closer to dpkg/rpm; we now have the option to use the "fakeroot" command to generate a binary tarball of the results.
Review of attachment 186849 [details] [review]: ::: jhbuild/modtypes/__init__.py @@ +34,3 @@ SkipToEnd, UndefinedRepositoryError from jhbuild.utils.sxml import sxml +from jhbuild.utils.fileutils import move_directory_tree Where is move_directory_tree used?
Review of attachment 186848 [details] [review]: With the move from $prefix/share/jhbuild to $prefix/jhbuild won't everyone's builds be marked as not done, forcing a complete rebuild? Considering that, why not leave as $prefix/share/jhbuild?
(In reply to comment #14) > Review of attachment 186848 [details] [review]: > > With the move from $prefix/share/jhbuild to $prefix/jhbuild won't everyone's > builds be marked as not done, forcing a complete rebuild? I could move the existing packagedb file though easily enough. > Considering that, why > not leave as $prefix/share/jhbuild? Well, I want to make explicit that this directory is "build metadata", and not part of the final OS image. So actually, maybe calling it _jhbuild or .jhbuild would make sense.
(In reply to comment #13) > Review of attachment 186849 [details] [review]: > > ::: jhbuild/modtypes/__init__.py > @@ +34,3 @@ > SkipToEnd, UndefinedRepositoryError > from jhbuild.utils.sxml import sxml > +from jhbuild.utils.fileutils import move_directory_tree > > Where is move_directory_tree used? It's not, good catch; it was a leftover from earlier code.
Created attachment 188185 [details] [review] Add a "workdir" (defaults to $prefix/jhbuild), change PackageDB to use it Migrate packagedb.xml from old location
(In reply to comment #10) > > The autotools modtype has been used for many modules that offered something > close to "configure; make; make install" but were definitely not real > autotools; this is the reason for many of the attributes and I think we need > another one to set "supports_install_destir" to False. So, ultimately we need this functionality, so I think the approach here is - patch the module to make it work (and send a patch upstream even!) To jump the gun a bit, I think this is the right approach with most build metadata. I've started working on it: http://people.gnome.org/~walters/docs/build-api.txt > ::: jhbuild/utils/packagedb.py > @@ +34,3 @@ > return time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime(tm)) > > +def _get_text_content(node): > > I'd like to take this opportunity to switch this module from xml.dom.minidom to > ElementTree. Hmm, do you have familiarity with it? I know the DOM API basically from working on the web, but it's been a while since i've used ElementTree.
Review of attachment 188185 [details] [review]: I would like the config option to be called destdir. It is clear what destdir is because of convention. With workdir its not clear.
(In reply to comment #19) > Review of attachment 188185 [details] [review]: > > I would like the config option to be called destdir. It is clear what destdir > is because of convention. With workdir its not clear. Well it's not destdir in the sense that it's the literal value of make install DESTDIR=. A subdirectory of it is used per-package, and it also holds the packagedb.xml. For people familiar with the random packaging systems out there, maybe the best term is "buildroot": http://rpm5.org/docs/api/buildroot.html Debian calls it "build-tree": http://www.debian.org/doc/manuals/maint-guide/dreq.en.html#targets
Created attachment 188348 [details] [review] Make --binpkg put packages in $workdir/deploy/$pkgname-$arch-gnome-$os.tar.bz2 Rather than having the name be supplied like --binpkg=foo.tar.bz2, instead we choose a name, which makes a lot more sense for autobuild scenarios. The 'deploy' name and the $arch-gnome-$os style both come from Bitbake.
(In reply to comment #18) > (In reply to comment #10) > > > > The autotools modtype has been used for many modules that offered something > > close to "configure; make; make install" but were definitely not real > > autotools; this is the reason for many of the attributes and I think we need > > another one to set "supports_install_destir" to False. > > So, ultimately we need this functionality, so I think the approach here is - > patch the module to make it work (and send a patch upstream even!) But patching is expensive, both on the spot and longer term, in maintenance, if not accepted upstream; thus not something I want to force on people creating jhbuild modulesets. > > ::: jhbuild/utils/packagedb.py > > @@ +34,3 @@ > > return time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime(tm)) > > > > +def _get_text_content(node): > > > > I'd like to take this opportunity to switch this module from xml.dom.minidom to > > ElementTree. > > Hmm, do you have familiarity with it? I know the DOM API basically from > working on the web, but it's been a while since i've used ElementTree. I have some familiarity with it, from its use in library-web; it's quite simple and will result in code that is much more readable than what you have now. <http://effbot.org/zone/element.htm>
Comment on attachment 187239 [details] [review] Add --binpkg option for "buildone" I don't see binary packages as a primary goal for jhbuild, it could go in as an additional command (like there was an experimental "jhbuild debuild" to produce .deb packages). Also if jhbuild was to produce binary packages, I would really prefer to have them be .rpm or .deb, instead of creating tarballs without any metadata.
(In reply to comment #23) > (From update of attachment 187239 [details] [review]) > I don't see binary packages as a primary goal for jhbuild, Let me better explain my high level goal. For GNOME 3.0, networking ended up being almost a total disaster because *very* few people were actually running NetworkManager 0.9. The reason for that is simply that most developers were on older OSes - and even Fedora 15 was blocked on NetworkManager 0.9 because of KDE, creating a bizarre situation where GNOME development is blocked on KDE - totally broken. We need to do a better job of building *ALL* of GNOME, and that includes system components. jhbuild is a good approach for that - with these patches, one can unpack the tarball on a root filesystem and it will have the correct permissions etc., which is very important for things like policykit that have setuid files. > it could go in as an > additional command (like there was an experimental "jhbuild debuild" to produce > .deb packages). I'm fine with an additional command instead of --bindir. > Also if jhbuild was to produce binary packages, I would really prefer to have > them be .rpm or .deb, instead of creating tarballs without any metadata. I could look at that, but what metadata are you thinking of specifically? What value is .rpm/.deb adding over the tarballs?
(In reply to comment #22) > > But patching is expensive, both on the spot and longer term, in maintenance, if > not accepted upstream; thus not something I want to force on people creating > jhbuild modulesets. Do you have a concrete example of a module that used AutogenModule but didn't support make install DESTDIR= ?
(In reply to comment #24) > (In reply to comment #23) > > (From update of attachment 187239 [details] [review] [details]) > > I don't see binary packages as a primary goal for jhbuild, > > Let me better explain my high level goal. For GNOME 3.0, networking ended up > being almost a total disaster because *very* few people were actually running > NetworkManager 0.9. The reason for that is simply that most developers were on > older OSes - and even Fedora 15 was blocked on NetworkManager 0.9 because of > KDE, creating a bizarre situation where GNOME development is blocked on KDE - > totally broken. > > We need to do a better job of building *ALL* of GNOME, and that includes system > components. jhbuild is a good approach for that - with these patches, one can > unpack the tarball on a root filesystem and it will have the correct > permissions etc., which is very important for things like policykit that have > setuid files. I don't think we should encourage people to unpack tarballs over their root filesystem. jhbuild was created to allow the isolation of development, not to interfere with the system. In the case of NetworkManager, or other modules that are considered too low level and not proper for jhbuild, binary packages should be provided. > > Also if jhbuild was to produce binary packages, I would really prefer to have > > them be .rpm or .deb, instead of creating tarballs without any metadata. > > I could look at that, but what metadata are you thinking of specifically? What > value is .rpm/.deb adding over the tarballs? For example they would have their own version numbers, and bug reports against distributions won't involve overwritten binaries.
(In reply to comment #24) > it will have the correct permissions etc., which is very important for things > like policykit that have setuid files. In my view, this is a big problem with JHBuild. I'd love to see this fixed. I do agree with Frederic, JHBuild shouldn't interfere with distro packages, but provide aim to provide a flawless sandbox.
(In reply to comment #26) > I don't think we should encourage people to unpack tarballs over their root > filesystem. jhbuild was created to allow the isolation of development, not to > interfere with the system. I totally understand that - but jhbuild is also the GNOME build system, and I'm trying to extend it to be (part of) an OS builder. I'm not saying we'd recommend to people unpacking tarballs over their root filesystems; rather, as a first cut, I intend the binary tarballs to be overlaid on a filesystem image built in another way. I have working code for "another way", I'll post about it when I'm able to integrate jhbuild binaries on top of it. So basically, my plan is not to ask GNOME testers and developers to unpack tarballs over their root, but to install an official GNOME jhbuild image. (Or use virtualization of course). > For example they would have their own version numbers, and bug reports against > distributions won't involve overwritten binaries. So, yeah I'm definitely not opposed to this, but I see it as a different direction.
you know, you can use OBS to build all packages and even installable image (from git source, if you want) It was exactly what I did for GNOME 3 live image.
(In reply to comment #29) > you know, you can use OBS to build all packages and even installable image > (from git source, if you want) > > It was exactly what I did for GNOME 3 live image. To be clear, I'm going to propose GNOME align with Yocto: http://www.yoctoproject.org/ OBS and Yocto have different tradeoffs - but the compelling advantage of Yocto for GNOME is that we have *direct* control over all the source, and ultimately our destiny. Yocto, like jhbuild, is designed to run on a single developer machine. OBS is another world - anyone who wants to make changes has to have a Novell account for example, and in reality is pretty tied to OpenSUSE. In the big picture, I see GNOME releasing a fast moving operating system where the underlying Unix gunk is treated as an implementation detail primarily - I want to see GNOME as the place where things like systemd land *first*, and then spread out. It's harder to do that being tied to OBS. Most importantly OBS implicitly ties us in to an RPM-based story for upgrading and management - and honestly it's totally broken. No one knows how to support upgrading apps in a race-free way with zypper/yum/whatever-your-favorite-wget wrapper is. Every time I see yum rip out files from under a running Firefox and break it, it makes me sad. GNOME will not be that broken. So - not opposed in any way to OBS as a short term tactical choice. But I'd also like to improve the official GNOME build system as part of an answer for building an OS.
you are obviously misinformed : -OBS isn't hardcoded to RPM, it handles DEB fine too -there is plan to switch away from Novell account to openID and regarding GNOME OS, there is still no decision about it at all in GNOME..
(In reply to comment #31) > you are obviously misinformed : > -OBS isn't hardcoded to RPM, it handles DEB fine too Both of them are equally stupid =) > and regarding GNOME OS, there is still no decision about it at all in GNOME.. Sure, but it's easier to decide when we have at least a prototype to show. Anyways - can we not block jhbuild patches on this?
Colin: I'm trying to understand the approach you're proposing, but I guess we're lacking some context here :-) Keep in mind that I hate the fact that we have many distros all doing the same work to build packages of the same software, so I'm not really fond of the current way things are done. (oh, and the OBS people are working hard to address the perception that OBS is tied to openSUSE -- because it's really a perception issue here) (In reply to comment #30) > In the big picture, I see GNOME releasing a fast moving operating system where > the underlying Unix gunk is treated as an implementation detail primarily - I > want to see GNOME as the place where things like systemd land *first*, and then > spread out. It's harder to do that being tied to OBS. Not sure why this is harder. > Most importantly OBS implicitly ties us in to an RPM-based story for upgrading > and management - and honestly it's totally broken. (s/rpm/package/) > No one knows how to support > upgrading apps in a race-free way with zypper/yum/whatever-your-favorite-wget > wrapper is. Every time I see yum rip out files from under a running Firefox > and break it, it makes me sad. GNOME will not be that broken. But this is not related to package management, this is related to how Firefox gets broken on upgrades anyway if you don't restart it. Why would something not based on packages be able to handle that better? The only solution is to not remove the old version at all, as far as I know. And you could do this with packages if you really want. What I'm worried about is that it sounds like we'd also be doing some kind of "packaging" work in GNOME (even if automatic), adding even more to the duplication already existing.
(In reply to comment #33) > Colin: I'm trying to understand the approach you're proposing, but I guess > we're lacking some context here :-) Keep in mind that I hate the fact that we > have many distros all doing the same work to build packages of the same > software, so I'm not really fond of the current way things are done. Of course. > (oh, and the OBS people are working hard to address the perception that OBS is > tied to openSUSE -- because it's really a perception issue here) I dunno...anyways I don't want to get into a long discussion about OBS tradeoffs - I'm really not stopping anyone from making OBS images. > (In reply to comment #30) > > But this is not related to package management, this is related to how Firefox > gets broken on upgrades anyway if you don't restart it. Why would something not > based on packages be able to handle that better? The only solution is to not > remove the old version at all, as far as I know. And you could do this with > packages if you really want. The solution is independent mechanisms for OS and application upgrades. Or at the very least, the system needs to be aware of the difference - and everyone writing a "package manager" believes upgrades are simply solving a dependency graph and running wget+untar, when the reality is way harder than that. > What I'm worried about is that it sounds like we'd also be doing some kind of > "packaging" work in GNOME (even if automatic), adding even more to the > duplication already existing. We already are - that's what the jhbuild moduleset is. I want GNOME to be a place to drain the pointless duplication too, as much as you do, if not more. Yocto has the compelling advantage here of being significantly more "vendor neutral" than anything else. Basically I believe we should leave packages to distributions. Ultimately a distribution would be an app you'd install on top of GNOME OS. I have a long writeup of all of this, but please - can we not block my jhbuild patches on this?
(In reply to comment #34) > I have a long writeup of all of this, but please - can we not block my jhbuild > patches on this? They're not blocked, perhaps this bug could get back to its summary, enhancing jhbuild so it tracks the list of installed files, and can ignore .la files. And the discussion about GNOME OS and distributions can happen in other channels. OBS, jhdebuild, those patches, there are many ways to get GNOME into the hand of developers, and I don't want jhbuild to assert a position at the moment. (eck, even jhdebuild which is my own code was never merged).
Created attachment 188502 [details] [review] Add a "top_builddir" (defaults to $prefix/_jhbuild), change PackageDB to use it For a future change to use "make install DESTDIR=", which in turn will enable things like "jhbuild uninstall". This patch adds an explicit concept in jhbuild of a directory for build state called "top_builddir", distinct from the installation tree. We previously had something like this in the $prefix/share/jhbuild directory, which is where it put the packagedb.xml file. Move packagedb.xml in there, and migrate an old DB file.
Created attachment 188503 [details] [review] autotools: Use make install DESTDIR= Like most "package managers", use DESTDIR= to create a temporary installation tree, from which we can generate a file manifest. Add a <manifest> node to PackageDB which contains the contents of a build. Then implement "uninstall" using this. A big advantage now is that we don't need to do a build in order to do an uninstall, and uninstall is independent of version.
Created attachment 188504 [details] [review] Clean .la files post-install These do not serve any useful purpose (supposing one's software system supports "uninstall"), and are actively harmful in other scenarios; see https://live.gnome.org/GnomeShell/RemovingLaFiles
Created attachment 188505 [details] [review] scripts: Clean up Makefile The append pattern ends up being a lot less of a mess.
The last patch is actually a preparatory cleanup for something else; if we care I can move it to a different bug. The GNOME OS prep work is moved to bug 650995 (which depends on this). This bug is now just "kill .la files and make uninstall more reliable".
Created attachment 188535 [details] [review] Add a "top_builddir" (defaults to $prefix/_jhbuild), change PackageDB to use it Fix a typo (top_buildroot -> top_builddir)
Created attachment 188536 [details] [review] autotools: Use make install DESTDIR= Reattaching for ordering
Created attachment 188537 [details] [review] Clean .la files post-install Reattaching for ordering
Created attachment 188538 [details] [review] distutils: support DESTDIR install This follows from the similar autotools work.
Created attachment 188539 [details] [review] cmake: Support make install DESTDIR= See similar patch for autotools.
Created attachment 188540 [details] [review] Remove 'ant' modtype Nothing in gnome-world-3.2 uses it, and it never worked in the first place since 'install' did nothing.
Created attachment 188541 [details] [review] waf: Support install with DESTDIR Similar to autotools.
Comment on attachment 188505 [details] [review] scripts: Clean up Makefile Nevermind this patch for now.
Comment on attachment 188536 [details] [review] autotools: Use make install DESTDIR= Review from comment 10 still holds (should use ElementTree).
Created attachment 189831 [details] [review] packagedb: Port to ElementTree It's generally cleaner and smaller code.
This series OK to commit now that we have the ElementTree port?
I wanted to dedicate time to test it but as things are currently, it's unlikely that I can give proper testing. Is this possible to commit it without the "remove ant" part, as we wait for an answer in bug 652666?
This does not appear to work when starting without an old packagedb file: IOError: [Errno 2] No such file or directory: '/tmp/dirktmp/new/INSTALL/_jhbuild/packagedb.xml' Sometimes people read XML -- maybe just for debugging. Is there an easy way to add newlines after each record in the new packagedb.xml, nowadays? (That's when I stopped using ET; when I did not find a trivial way to do that)
(In reply to comment #52) > I wanted to dedicate time to test it but as things are currently, it's unlikely > that I can give proper testing. I'll be available of course to help with any fallout. > Is this possible to commit it without the > "remove ant" part, as we wait for an answer in bug 652666? Fine by me. Thanks!
(In reply to comment #53) > This does not appear to work when starting without an old packagedb file: > IOError: [Errno 2] No such file or directory: > '/tmp/dirktmp/new/INSTALL/_jhbuild/packagedb.xml' I have a fix for that I forgot to rebase into the tree. I'll push it when I push the rest of the branch. > Sometimes people read XML -- maybe just for debugging. Is there an easy way to > add newlines after each record in the new packagedb.xml, nowadays? (That's > when I stopped using ET; when I did not find a trivial way to do that) Yes =( I'll look into this.
Attachment 188535 [details] pushed as 12f8c87 - Add a "top_builddir" (defaults to $prefix/_jhbuild), change PackageDB to use it Attachment 188536 [details] pushed as 092053e - autotools: Use make install DESTDIR= Attachment 188537 [details] pushed as 841b7fa - Clean .la files post-install Attachment 188538 [details] pushed as 40c1459 - distutils: support DESTDIR install Attachment 188539 [details] pushed as 9ea83ee - cmake: Support make install DESTDIR= Attachment 188541 [details] pushed as 76dbb7f - waf: Support install with DESTDIR Attachment 189831 [details] pushed as 10d1a97 - packagedb: Port to ElementTree
Thank you for your patches and working on make JHBuild better. I modified the code to handle the OSError as per comment 2. I also removed the print as it breaks i10n, tinderbox & quiet mode There are few other incorrect prints in your other commits, I'll try to fix them too. Help welcome too. http://git.gnome.org/browse/jhbuild/commit/?id=e83c49582b22176f450179d02500f722e44dfa64
(In reply to comment #57) > Thank you for your patches and working on make JHBuild better. > > I modified the code to handle the OSError as per comment 2. > I also removed the print as it breaks i10n, tinderbox & quiet mode > There are few other incorrect prints in your other commits, I'll try to fix > them too. Help welcome too. The other prints as far as I can see are just for my debug code in if __name__ == '__main__' code. Basically so one can run: python ./jhbuild/utils/packagedb /path/to/_jhbuild/packagedb.xml
I'm hitting this issue and I've been told it's because of your changes. $ jhbuild buildone telepathy-glib Traceback (most recent call last):
+ Trace 227552
jhbuild.main.main(sys.argv[1:])
rc = jhbuild.commands.run(command, config, args, help=lambda: print_help(parser))
return cmd.execute(config, args, help)
return self.run(config, options, args, help)
check_bootstrap_updateness(config)
p_version = packagedb.entries.get(module.name)[0]
Created attachment 190439 [details] [review] base: Fix lookup of package version Packages are no longer tuples but objects, so use the "version" property. While we have the patient open, avoid looking up in the dictionary twice.
Btw, please open new bugs for issues like this instead of reviving this mega-bug from the dead =)
Attachment 190439 [details] pushed as 8b0fd88 - base: Fix lookup of package version
*** Bug 632964 has been marked as a duplicate of this bug. ***
Comment on attachment 188540 [details] [review] Remove 'ant' modtype Committed. http://git.gnome.org/browse/jhbuild/commit/?id=8db8d874daa030a0ca6a550f94726f3f6cf96dc0 See also: http://git.gnome.org/browse/jhbuild/commit/?id=aefcb2fb7d53cdd1f0dcc83d122d988afeb94c21