GNOME Bugzilla – Bug 650995
Add binpkg option
Last modified: 2012-04-24 21:26:38 UTC
Depends on bug 647231
Created attachment 188508 [details] [review] Add binpkg option This moves us architecturally even closer to dpkg/rpm; we now have the option to use the "fakeroot" command to generate a binary tarball of the results. This step is useful for multiple reasons; I want it for implementing GNOME OS, and it also is useful for developers who currently do 'sudo make install'.
From bug 647231 comment 23, > I don't see binary packages as a primary goal for jhbuild, it could go in as an > additional command (like there was an experimental "jhbuild debuild" to produce > .deb packages). > > Also if jhbuild was to produce binary packages, I would really prefer to have > them be .rpm or .deb, instead of creating tarballs without any metadata.
(In reply to comment #2) > From bug 647231 comment 23, > > > I don't see binary packages as a primary goal for jhbuild, it could go in as an > > additional command (like there was an experimental "jhbuild debuild" to produce > > .deb packages). > > > > Also if jhbuild was to produce binary packages, I would really prefer to have > > them be .rpm or .deb, instead of creating tarballs without any metadata. For reference what Yocto does is go through one of dpkg/rpm/ipkg, but it doesn't force their usage afterwards if you don't want it; i.e. they can just be an intermediate artifact. I *could* do something similar to this. Would a patch then that generated a .rpm be OK?
Created attachment 189858 [details] [review] Add makerpms option We generate RPMs like: jhbuild-$(modname).x86_64.rpm This is a slightly nicer replacement for people who currently use "sudo make install". For one, it doesn't totally override the packaging system, though in practice you'll have to install with --force. What this is also useful for is that the GNOME autobuilders can produce reference binaries.
Just wondering. Should this use xz instead of bz2 to be consistent with the recent move to xz/ustar in gnome ftpadmin?
The 'jhbuild doesn't set correct owner on install files' problem annoys me when sandboxing GNOME. Are you working towards the default/recommended method is to generate rpms of the JHBuilt packages, then ask user to install via package manager? With this approach, after the user completes a jhbuild build, I'd expect a single command to upgrade/install the fresh rpms. ie: sudo yum update jhbuild-gnome Do you agree? If so, how would this be done? What about users on non-rpm distros?
I've been thinking about this bug. So in the big picture, very few people actually run jhbuild regularly. And that includes me! There are multiple reasons - first, it's slow/not fun to debug. I've been working on fixing that. However even if jhbuild compiled everything instantly and never failed, at the end of the day when you log in, you can't $#@!$@ use it because you can't connect to your wireless network because you need PolicyKit privileges and the agent helper in your jhbuild root isn't setuid. (There are other, similar big issues like pulseaudio not being able to connect, etc.) Should we just give up on jhbuild and have people go through native packages? So the problem with that is that jhbuild is supposed to be about *development*, you're supposed to be able to change/test/break/change etc., and the whole while have a (supposedly) stable system to go back to. After you install jhbuild, your computer still has a distribution root. If we just said "ok, let's just tell people to use distribution packages" then basically it's impossible to develop GNOME without hacking on spec files and similar shit. That's not a good place to be in - and more importantly, one big problem with this is that if we want to preserve the "revert to stable" aspect, we can't actually use the existing spec files, because the resulting packages would conflict! So we need parallel installation, and that implies not using (or at least not *needing*) distribution package build rules. Also, at least RPM/Debian packages aren't focused on development at all. They're frankly just painful (and yes, I have written and used both). Let me propose a plan here in the next comment, before this one gets lost to a Firefox crash or something.
JHBuild sysroot: == Pick a Prefix == We pick a --prefix (for purposes of discussion, let's call it /gnome). == The setuid helper binary == jhbuild is modified to install a setuid helper binary "jhbuild-install-helper". When you first install jhbuild with "make install", we pop up a PolicyKit dialog to make it setuid root and mode u+s. And executable only by root (i.e. chmod 0, and add an ACL to also allow the current uid access (yes I just came up with this idea, and tested it - it works). We could also install a system service I guess, but that would require distro-specific knowledge (which in turn implies jhbuild being a distro package, which i wouldn't mind *too* much but...) == The jhbuild root is now root-owned == Like Debian, jhbuild will be modified to use "fakeroot" to generate a tarball that looks like what we expect. The install helper accepts commands, one of which is e.g. "overlay /path/to/foo-tar.gz". That will overlay the new "package" over top. == .tar.gz versus .deb/.rpm == We don't generate .deb/.rpm because 1) We don't care about runtime dependencies 2) We don't execute arbitrary scripts - jhbuild has centralized "postinst" handling that is based on inspection of contents == gnome.org binaries == We can generate GNOME/Linux binaries targeted with a --prefix=/gnome on our build servers. What would be cool is if jhbuild understood how to overlay local builds on top of this.
I am not sure about the existence of a "GNOMELinux binaries target", the thing we have is LSB and is probably not enough. Let me outline the ideas I had when thinking about jhbuild producing binary packages: - generate .rpm/.deb, as they are just tarballs + metadata, - name package jhbuild-$moduleset-$module, ex: jhbuild-3.2-gnome-documents, to avoid conflicting with distro packages, - and install them to a specific directory, (ex: /opt/gnome-3.2/) - create the required metadata from moduleset and other jhbuild means (version number, dependencies, a list of required system packages can be obtained from sysdeps); at the moment this means we will generate $n$ packages, $n$ being the list of distributions we want to support as a base system) - ignore specific bits of distro policies when justified and appropriate, - generate the required number of repositories, (ex: fedora-15/gnome-3.0, or debian-squeeze/gnome-3.2), with packages built with a reliance on those system's base packages.
(In reply to comment #9) > I am not sure about the existence of a "GNOMELinux binaries target", the thing > we have is LSB and is probably not enough. If we said LSB I think it'd help get us *very* far, but it's hard to know until we have more practical experience. > Let me outline the ideas I had when thinking about jhbuild producing binary > packages: > > - generate .rpm/.deb, as they are just tarballs + metadata, + arbitrary shell scripts. It's actually the "scripts are embedded in each package" part that I find really annoying and stupid about RPM/.deb, and I'm far happier with our centralized automatic postinst handling in jhbuild. So for jhbuild binaries, I'd like to keep using jhbuild's triggers, and not attempt to use "distro" triggers (since we can't anyways, because they are probably hardcoded for the main root). > - name package jhbuild-$moduleset-$module, ex: jhbuild-3.2-gnome-documents, > to avoid conflicting with distro packages, Yes. > - and install them to a specific directory, (ex: /opt/gnome-3.2/) > - create the required metadata from moduleset and other jhbuild means (version > number, dependencies, a list of required system packages can be obtained from > sysdeps); at the moment this means we will generate $n$ packages, $n$ being the > list of distributions we want to support as a base system) I think it'd be far simpler to simply omit runtime dependencies from the binaries. Basically I am focusing on one problem at a time here - the present problem is permissions. It's really the dependencies (i.e. the arbitrary package names, as well as versions) that form a "distro"; just sharing a tar wrapper like RPM doesn't make them compatible. And it's the many distros and versions of them that will be an ongoing maintenance nightmare. For someone who wants that project - OBS might be an interesting approach. I don't want that project =) The runtime dependencies gain us **very** little. The canonical example of an implicit dependency we have in jhbuild right now is XLib. So with a "have native dependencies approach", you could install a minimal Debian system and choose to install the GNOME jhbuild packages directly and they pull in xlib. With my approach there would simply be a linker error when you tried to log in, and we would immediately tell you "Install a full Debian/GNOME base first" and be done. So bottom line - .deb/.rpm buy us little if we treat them as just tarball+metadata. That's not to say I am totally against it - but they create false expectations that I don't think we can live up to. If for example we're using jhbuild for triggers (as above) then you can't actually use RPM to install them, since jhbuild needs to run triggers after.
In the big picture, a reasonable question one might as as we proceed towards making jhbuild more powerful is "what is the difference between GNOME (jhbuild) and a "distro""? I've been thinking about that, and I have a pretty simple answer: In GNOME, we will never have runtime dependencies. We expect everything we specify to be installed, period. We leave stuff like: 1) Dynamically deconstructing the operating system (apt-get remove pulseaudio) 1a) Dynamically replacing/parallel installing the operating system (apt-get install XXDE) 2) Install a minimal server image, bootstrap to a desktop (apt-get install gnome-desktop) 3) Having no idea how to upgrade firefox (or apps in general) to distros as a value add.
bug 654872 touches a lot of similar code, marking that as a dependency.
Also there are two important aspects of the current jhbuild workflow I think it's important to preserve: 1) You have your system apps (/usr/share/applications) 2) You have your user data ($HOME) So even if we have /gnome root-owned etc., we'd still be setting XDG_DATA_DIRS=/gnome/share/:/usr/share for example. Where things get is if we extend the /gnome root to include gdm, we'd have to be very careful that $HOME is still the system /home (or whatever). Which probably means we can't do gdm this way initially.
When it comes to building binaries that can be hosted on gnome.org and shared, I think we'll be much better off looking closely at alex' glick2 efforts.
(In reply to comment #14) > When it comes to building binaries that can be hosted on gnome.org and shared, > I think we'll be much better off looking closely at alex' glick2 efforts. Ah...glick2 is maybe something for apps, but I don't see how it makes sense to use for the operating system. Are you thinking something like a 100MB "gnome" executable that you can download and log into GDM? While in theory that sounds easy enough - there are some issues like how do you avoid redownloading 100+ MB every time we do a new build on gnome.org (how glick interacts with updates is something I haven't seen alex discuss well). Also it'd suck to regenerate a huge glick image executable every time one does a *local* build. Efficiency of compilation and installation is actually important to me, because it's part of the edit-compile-debug cycle that I do hundreds of times a day.
Here are some random thoughts from me about jhbuild, gnome-os and bundles: I think bundles is a great way to *deploy* *apps*. So, for end-users this would be the primary way desktop applications are installed. However, there are some areas bundles don't really work out. First, its for deployment. For instance, during development you don't want to recreate bundles each time you want to run your app. So, our app development story has to be slightly different, with a bundle popping out at the end. Secondly, its for apps. Its certainly possible to go all the way with bundling, like rPath is doing where you bundle everything your app requires, including the kernel into a VM image. The other extreme is the current distro model where you bundle nothing and make everything depend on everything else. I think a reasonable mid point between these two is that we do ship an OS image, which is a single entity (i.e. guaranteed set of ABIs, no optional parts) that contains a bootable, usable base system where you can log in, get on the network and start apps. Anything above this is done via bundles. Of course, there will always be problematic areas in a setup like this. For instance, you can't ship out-of-tree x drivers or printer drivers as bundles. So, you always have to have something like /usr/local being writable for local system "packages". Another problem is that some things are not bundleable, because they rely on being a singleton. For instance, if we create a gnome-os 1.0 i think its unlikely we'd want gconfd in the supported ABI, as its deprecated, but we can't really have apps bundle gconfd, because it depends on having a single instance of it running. So, at some level I think we need bundles being services too, such that you have to install the gconf bundle for some old apps to run. Similarly, you might want the kde runtime to be a service bundle. (Of course, service bundles don't include lowlevel things like libraries, just the services). So, we have a hierarchy of things that replace "packages". 1) Base OS image (in two parts, supported ABI and desktop shell) 2) Locally installed system-level stuff in /usr/local 3) Service bundles 4) App bundles Now, jhbuild is a development tool primarily, but it could also be a great tool to construct an OS like this. It could create the base os image, and it could help you build bundles (automatically knowing what are in the base os and bundling other dependencies). Creating bundles is done by building your libraries and apps with a known runtime prefix (/opt/bundle or something) and the installing them in some directory, then you run whatever post-install scripts you want. Then a tool just creates the bundle from that. However, for optimal behaviour we want to use exactly the same binaries in bundles that have shared dependencies (like the same library), because then glick can share the disk i/o and the pagecache memory for these two files. So, ideally we'd construct some set of binary representation of the dependencies of apps and pick pre-build versions of the dependencies when creating the final bundles. So, it would be cool if jhbuild supported some representation of a binary build for a specific dependency.
For interested observers, one set of ideas now lives in: http://git.gnome.org/browse/hacktree I'd also be totally fine with someone else just "picking" a distribution and maintaining git builds of packages for it, and suggesting that to developers. Most likely that'd be OpenSUSE+OBS, but it's up to the person. But I will never put my own name on an upgrade process that rips files away while the OS is running - in other words, I think rpm/dpkg are an evolutionary dead end on the client, at least. Maybe server people like the flexibility and are willing to accept the danger.
Yeah, let's forget about doing this in jhbuild.