GNOME Bugzilla – Bug 732394
Add a hwtest tree and image
Last modified: 2014-07-17 17:56:21 UTC
The hwtest tree is an operating system tree used for the hardware testing project to: * Act as the controller machine * Temporarily be booted to upgrade a target machine * Boot from USB to install a controller or target machine The actual operation of the hardware testing project lives in the gnome-hwtest module - these patches define the build infrastructure for the tree.
Created attachment 279492 [details] [review] Add a "hwtest" tree Add two additional component types - "minimal" and "hwtest". "minimal" components are needed to boot a basic text-only command line environment. The "hwtest" tree will include the "minimal" components plus some extra components (like DHCP and iSCSI servers) that are needed for the controller and update images for the GNOME Hardware Testing Project.
Created attachment 279493 [details] [review] Mark components as "minimal" Mark components that will be needed for both the normal runtime and the hardware testing controller as "minimal". GJS is built twice - once normally and once with GTK+ and Cairo disabled as "minimal-only".
Created attachment 279494 [details] [review] Add components needed by gnome-hwtest Populate the hwtest tree with the external components needed for the GNOME Hardware Testing Project. These are: gnome-hwtest: The GNOME Hardware testing scripts (included in minimal since we want to have the subset used for running the tests in the runtime image) gjs-minimal: a version of GJS built without GTK+ and Cairo support - gnome-hwtest uses this for its runtime. iPXE: Used to network boot with iSCSI support GRUB: Used to find the partition we want to boot into dnsmasq: Used as a dhcp server for exporting boot parameters from the controller to the target machines. lighttpd: Used to export files to the target machines from the controller tgt: Used to export boot images over iSCSI (It turns out this is a somewhat obsolete daemon - the kernel support for iSCSI targets plus targetcli would be the modern way to do it. tgt works for our needs, but it could be switched over to targetcli later.) open-iscsi: Used to mount iscsi volumes on the target machine. socat: Used for exporting logs from the target to the controller.
Created attachment 279495 [details] [review] Reenable building video drivers for hardware Go back to building the Mesa and Xorg drivers for Intel, Radeon, and Nouveau.
Created attachment 279496 [details] [review] hwtest: Set default target to gnome-hwtest.target When creating the hwtest tree, we want the default systemctl target to be gnome-hwtest.target, so that we can start the appropriate activity (install, controller, update)
Created attachment 279497 [details] [review] Hack: always install the bootloader for the hwtest image We need to boot the hwtest image once read-write so that the right subdirs of /var are there when booted read-only. Always installing the bootloader accomplishes this.
Created attachment 279498 [details] [review] Add modules for Hardware Testing dosfstools: used when creating the grub-based boot image parted: used when installing and when creating boot iamges
Review of attachment 279496 [details] [review]: I was just scanning over these patches and noticed this is going to conflict with https://git.gnome.org/browse/gnome-ostree-integration/commit/?id=3ff32e19066110ed03a17bcaec61800021276eb6 Probably what we really want here is the Continuous equivalent of https://github.com/projectatomic/rpm-ostree/commit/227137ed413548465cbd1e6de62f7b9c79470eb4 We could revert the gnome-ostree-integration change if Continuous had such a syntax to set the default target per-tree.
Review of attachment 279492 [details] [review]: So I have to admit that I was pretty happy with Continuous that it showed that it could do *two* trees, but there was still no dependency resolution - nothing "package" like. The radical simplicity of manifest.json I felt was just a huge breath of fresh air compared to a traditional package distribution. However as I am now attempting to use the underlying OSTree technology within the context of a traditional package-based distribution, if one wants to ship many OSTree trees, I'm feeling like it'd be hard to get away from more fine grained model with dependencies. I think we could probably live with two new trees, but also it seems clear to me it wouldn't expand much farther without becoming unmanageable. Particularly when you get into things like "OK, I need debuginfo for minimal"... Baserock has "strata", or layers. If we have the constraint that e.g. "runtime" is a strict superset of "minimal", it would make sense to me to split up the component list that way. All of those words later, this patch is pretty simple, and I'm ok with it.
Review of attachment 279493 [details] [review]: Looks right, although the gjs update is really in the next patch.
Review of attachment 279494 [details] [review]: Ok, you put a ton of work into this obviously but I'm finding myself uncertain here. This server we're building here is going to be pulling the same kernel and gjs we're trying to test, right? Is that really what we want? If we defer a bit, how is it updated? What we've ended up with things like jhbuild is taking a stable base, and adding layered unstable/development components. That's how I maintain Continuous right now on top of RHEL6, and it's how many/most GNOME developers hack on "their" components. Isn't that what we want for a testing server too? I mean the scripts could be distribution-independent, like jhbuild is, so you could set up a hwtest server on Debian or RHEL or whatever. I may be completely misunderstanding the architecture, please correct me if so =)
Review of attachment 279495 [details] [review]: OK. Though I haven't looked; are we going to need any firmware?
Review of attachment 279497 [details] [review]: ::: src/js/tasks/task-builddisks.js @@ +129,3 @@ + // read-only and bootloader installation, by booting the disk read-write, has + // the positive side-effect of letting systemd-tempfiles run and create the + // necessary directories in /var that we depend on. Can't we just mount a tmpfs on /var?
Review of attachment 279498 [details] [review]: Sure.
(In reply to comment #11) > Review of attachment 279494 [details] [review]: > > Ok, you put a ton of work into this obviously but I'm finding myself uncertain > here. > > This server we're building here is going two be pulling the same kernel and gjs > we're trying to test, right? Is that really what we want? If we defer a bit, > how is it updated? The actual tests that get run are not in the hwtest tree - just the code that handles the pull-from-build.gnome.org / update-target-system / fire off tests process. What I'm planning is that the image running on the controller (which is the same image that targets are booted with to upgrade) will be sitting on a reference that we make manually on build.gnome.org after testing. Say: gnome-continuous/hwtest-stable/x86_64 > What we've ended up with things like jhbuild is taking a stable base, and > adding layered unstable/development components. That's how I maintain > Continuous right now on top of RHEL6, and it's how many/most GNOME developers > hack on "their" components. You do make a good point, which I hadn't necessarily considered, which is that since gnome-continuous is built as a unit, any change to any component of the hwtest image requires upgrading to the latest version of every other one. To make a one-line change to the scripts in the gnome-hwtest module, the latest version of every other dependent module has to be pulled in - kernel, systemd, gjs, ostree, etc. But on the other hand, these are such fundamental modules that if they are broken for an extended period of time, then we're in pretty big trouble. My basic feeling is that the entire hwtest image, acting as a controller, or acting as a image to boot to upgrade the target, has two states: Working Not working and we can distinguish between them with a bit of testing... I don't see a case where have to spend a lot of time restabilizing after an upgrade - where we fix one bug and another pops up. And I don't think the controller scripts in gnome-hwtest will need upgrading very often - I have a TODO list for the project, and nothing on it requires changes to the controller scripts. So while you've identified a problem with the model, it's one that I think that won't be a huge problem in practice. If we hit bad problems in this area, we could do for other modules what I did for gjs where there are two separate builds of gjs in gnome-continuous - one for the runtime, one for hwtest. In the case of gjs they are different only in configure options - whether GTK+ and Cairo are enabled - but different tags are possible as well. (Or we could go to a more complicated system of layers and dependencies, but that seems like a lot of work.) > Isn't that what we want for a testing server too? I mean the scripts could be > distribution-independent, like jhbuild is, so you could set up a hwtest server > on Debian or RHEL or whatever. > > I may be completely misunderstanding the architecture, please correct me if so > =) I really view the software running on a controller machine more as an operating system we define than as a service: * Controller machines are basically appliances - they need to run unattended in a lab or under a desk somewhere, and when we do retag the stable hwtest tree, upgrade themselves and keep on going. So I want the entire upgrade process to be defined and consistent. * Supporting the network boot of target machines requires a suite of services running on privileged ports - DHCP, etc. It's not just one service you start running. * We also need an operating system to boot on target machines to upgrade them (the basic rule is that we keep on going in an automated fashion even if a gnome-continuous build locks up hard.) This can simply be the same OS as the controller OS. This operating system could be be built on something other than gnome-continuous - Fedora, Debian, etc. Advantages of a gnome-continuous base are: * Someone hacking on this doesn't need to learn another operating system - they already need to learn how gnome-continuous works. * The upgrade story for the controller appliance is very similar to what ostree strives to achieve - and gnome-continuous has very good ostree integration already. (For self-preservation, I also would not look forward to redebugging the network boot process with initramfs-tools rather than Dracut to move to a Debian base! ;-)
Review of attachment 279496 [details] [review]: Hmm, since we don't have any tree configuration, I think this is basically identical in spirit to that rpm-ostree commit, but hardcoded. The basic conflict between this and the ostree-integration patch is that makes the link in /etc/systemd/system and this makes the link in /usr/lib/systemd/system - otherwise it seems like it would just cleanly override it. (gnome-ostree-integration setting a OS-wide setting out of 'make install' seems a bit hackish to me.) Would you consider this patch OK (at least as a stop-gap until some future point when per-tree configuration is added) if I just switched it to make the link in /usr/etc rather than in /usr/lib?
(In reply to comment #16) > gnome-ostree-integration setting a OS-wide setting out of 'make > install' seems a bit hackish to me.) Well given the original design of Continuous (build a singular OS from source), it makes sense to me. Similar choices like NetworkManager being enabled, using gdm are hard coded there. Of course now hwtest is moving away from that a bit and more towards something where we have multiple tree variants, and that takes us closer to where we want per-tree defaults. > Would you consider this patch OK (at least as a stop-gap until some future > point when per-tree configuration is added) if I just switched it to make the > link in /usr/etc rather than in /usr/lib? You'd need to unlink the existing one first right? If so that seems reasonable to me - the default is graphical, we're overriding it.
Created attachment 280301 [details] [review] hwtest: Set default target to gnome-hwtest.target When creating the hwtest tree, we want the default systemctl target to be gnome-hwtest.target, so that we can start the appropriate activity (install, controller, update)
Attachment 279492 [details] pushed as 5f3b4e6 - Add a "hwtest" tree Attachment 279493 [details] pushed as 32cba74 - Mark components as "minimal" Attachment 279494 [details] pushed as ac85e32 - Add components needed by gnome-hwtest Attachment 279495 [details] pushed as becc437 - Reenable building video drivers for hardware Attachment 280301 [details] pushed as 8a144bd - hwtest: Set default target to gnome-hwtest.target