GNOME Bugzilla – Bug 732395
Build the kernel as a subcomponent
Last modified: 2014-07-17 17:57:05 UTC
These patches build the kernel using the configuration from: http://git.fishsoup.net/cgit/gnome-continuous-kernel/ (Will be moved into GNOME Git if there's a consensus that this is a good idea.) The advantages of this are: - We aren't dependent on rebasing Yocto to pick up new kernel versions; e.g., we'll have the ability to git kdbus support on our schedule. - We build a mainline kernel, rather than a Yocto branch of the kernel - We can follow a kernel branch and catch performance and functionality regressions as they happen. The disadvantages are: - Building the kernel is slow - on my laptop it takes about 30 minutes to build. (The config stripping I did reduced this time by about 50% - probably another 50% is possible.) - We may not want to be on the pointy end of the stick for kernel regressions. - Less of an incentive to keep rebasing Yocto, and falling behind to an old Yocto is a bad idea. Tagging to a particular version is always possible. It might be also possible to have some sort of minimum-successful-build-frequency option. The fact that the kernel is *not* done by lots of people pushing to a shared repository should help a bit in terms of continually building the kernel. The repository name "gnome-continuous-kernel" is because I wanted to emphasize that it wasn't a general purpose kernel - it has stuff like wifi and webcam support stripped out. I don't *need* this for hardware testing - modifying the yocto kernel config is also possible - but it does make testing graphics performance more realistic.
Created attachment 279500 [details] [review] Add a facility for a 'subcheckout' A subcheckout is a bit like a submodule - it's a directory inside a parent module that is checked out from version control. But instead of being configured in git, it is configured in manifest.json - the reason for this is to allow us to rebuild the outer module whenever the inner module changes, rather than having a fixed version of the inner module. This will be used for the kernel build where the outer module has our configuration and build scripts, and the inner module is the actual upstream kernel tree.
Created attachment 279501 [details] [review] Build the kernel as a component. Move building the kernel from part of the Yocto base to a component (kernel configuration lives in a new gnome-continuous-kernel git module.) The advantage of this is that we can include an upstream kernel branch in our regression testing and don't need to wait for a complete new version of Yocto to pick up kernel changes.
Created attachment 279502 [details] [review] Switch to an external kernel build Don't include the kernel or kernel-derived headers in gnomeos trees; include bc in the devel tree since recent kernels depend on it for building. (Earlier kernels used perl for arbitrary precision calculations.)
Review of attachment 279500 [details] [review]: "component": "none" feels awkward to me. Wonder if we should have something like a secondary "nonbuilt-components"? Or alternatively, literally include it, like: {"src": "git:git://git.fishsoup.net/gnome-continuous-kernel", "component": "minimal", "child-component": {"src": "kernel:stable/linux-stable", "name": "linux", "branch": "linux-3.14.y" }} Basically we don't want any of the build logic for kernel, just the git mirroring, so the git mirroring could handle "child-component" specially? (Debate whether it should be an array "child-components", but if there's not a use case it seems like it would just add complexity). BTW I am realizing the "component" verb really should have been "tree". I'm OK with this patch - we can push it as is too.
Review of attachment 279501 [details] [review]: Looks right, marking ACN after previous commit.
Review of attachment 279502 [details] [review]: It's cool it's that simple =)
Now that I've said this I'm a little worried about ongoing maintenance. I can clearly see that to do any real hardware testing, tracking an active kernel would be really useful. It also really excites me to have the opportunity to bring GNOME development and Linux kernel development closer together. But I am worried about the ongoing maintenance here. You said you cloned the Fedora configuration, and I'd try to be sure that we can track changes to that configuration over time and particpate in it. Missing some of them can cause weird issues: https://bugs.freedesktop.org/show_bug.cgi?id=75512#c1 Some of them are security-relevant too. Also, what about linux-firmware? Do you think we're going to need that? I'd like to avoid it if we can.
(In reply to comment #7) > Now that I've said this I'm a little worried about ongoing maintenance. I can > clearly see that to do any real hardware testing, tracking an active kernel > would be really useful. > > It also really excites me to have the opportunity to bring GNOME development > and Linux kernel development closer together. > > But I am worried about the ongoing maintenance here. You said you cloned the > Fedora configuration, and I'd try to be sure that we can track changes to that > configuration over time and particpate in it. Hmm, not impossible, but downsides include: * The Fedora kernel has a *lot* of options on. Striping out options we don't need reduced build time by a lot on my laptop - iirc from 40 minutes to 25 minutes. There's more that could be done - the 'make defconfig' kernel built in 5 minutes on the same laptop! - though we do want some expensive subsystems that aren't in the default kernel - btrfs and xfs, nouveau and radeon drivers. * The Fedora kernel has a *lot* of options on - this makes it really hard to understand its configuration - there is a ton of ancient / ARM-specific / etc. hardware listed explicitly in the config files. The kernel configuration system is pretty robust - the Fedora spec files themselves list drivers that have since been removed, mark things as built-in that end up getting built as modules due to dependencies and vice-versa, etc. So I'm not worried much about build breakage. There's certainly some possibility of missing kernel options that affect performance or are required for operation, but my instinct for now is to keep a stripped down set for sanity and to avoid slowing down the operation of GNOME Continuous too much. We could go back to the Fedora kernel configuration at any time if maintenance does end up being a problem. > Missing some of them can cause weird issues: > https://bugs.freedesktop.org/show_bug.cgi?id=75512#c1 > > Some of them are security-relevant too. > > Also, what about linux-firmware? Do you think we're going to need that? I'd > like to avoid it if we can. The only driver that I know that we'll need firmware for is the Radeon driver. But we do want Radeon in our test matrix, so I don't think we can avoid firmware long term. Could we download firmware directly to the controller, merge it into the tree there, and not put it on gnome.org? Conceivably possible, but the need for the firmware in the initrd means we'd then have to rebuild the initrd, and it gets uglier. And just listing benchmarks of GNOME running with Radeon graphics might gets us on the wrong side of someone thinks we shouldn't promote hardware that requires downloading non-free software onto it... In the end, we aren't providing software for end-users to use on their systems and the choice of whether to include Radeon drivers and firmware is really up to the people who do provide that software.
Created attachment 280094 [details] [review] Add support for "child components" A child component is a bit like a submodule - it's a directory inside a parent module that is checked out from version control. But instead of being configured in git, it is configured in manifest.json - the reason for this is to allow us to rebuild the outer module whenever the inner module changes, rather than having a fixed version of the inner module. Child components must have unique names globally within manifest.json - this is because they are treated like main components in certain contexts: in particular names of child components can be passed in the 'fetchComponents' argument of 'resolve' and child components appear independently in the bdiff output. This will be used for the kernel build where the outer module has our configuration and build scripts, and the inner module is the actual upstream kernel tree.
Created attachment 280095 [details] [review] Build the kernel as a component. Move building the kernel from part of the Yocto base to a component (kernel configuration lives in a new gnome-continuous-kernel git module.) The advantage of this is that we can include an upstream kernel branch in our regression testing and don't need to wait for a complete new version of Yocto to pick up kernel changes.
Redone with your suggestion - it's a fair bit more complicated, but avoids the "component": "none" thing.
Review of attachment 280094 [details] [review]: This was more complex than I thought it would be; sorry about that. ::: src/js/snapshot.js @@ +37,3 @@ + let childName = childComponents[j]['name']; + if (r[childName]) + throw new Error("Duplicate component name " + chidlName); Typo, "chidl" ::: src/js/tasks/task-resolve.js @@ +91,3 @@ + print("Describe cache miss for " + component['name']); + revision = Vcs.describeVersion(mirrordir, tagOrBranch); + modifiedCache = true; This variable needs to be a return value.
Review of attachment 280095 [details] [review]: Looks sane to me.
Attachment 280094 [details] pushed as 95095be - Add support for "child components" Attachment 280095 [details] pushed as 2545c85 - Build the kernel as a component.