GNOME Bugzilla – Bug 658912
Integrate with LAVA
Last modified: 2011-10-29 15:12:46 UTC
Linaro Automated Validation Architecture is a useful piece of infrastructure for testing on multiple hardware platforms, and we should be making the most of this resource. This bug tracks the effort to integrate the two systems. Current tasks as of 13/09/2011 include: * Create insanity-dumpresults-json, which produces an easily-parsed format inspired by LAVA's bundle format (just needs rebasing). * Add support in gst-media-test for reading INSANITY_TEST_FOLDERS from a python file and running tests on all subdirs. * Bug #658495 - samplemedia.linaro.org integration * Make sure that expected failures don't cause bogus reruns. * Make sure that insanity-dumpresults-json can correctly mark expected failures as such. Progress and review requests will be made in comments or other bugs which block this one.
http://cgit.collabora.com/git/user/alsuren/gst-qa-system.git/log/?h=lava-658912 The first few commits are pretty self explanatory. Some of the insanity-dumpresults commits were previously ++'d by Edward, but never merged, and I did some cleanup (and squashed/re-split using git citool for reviewability) so I think it'd be simpler to just re-review them. "Scenario: auto-aggregation of checklists" is a hack, and I'll happily delete it, but it helps me to quickly find extra failures from the output of rm testrun.db; (INSANITY_DEBUG=3 PYTHONPATH=$HOME/src/gst-qa-system bin/gst-media-test --settings web/settings.py --tests full-gnlfilesource-scenario) 2>&1 | grep --colour=auto "[0e])"
Also, I have commits entitled: Move tests to within insanity package This avoids namespace collisions with other generically named "test" packages. more samplemedia expected failures I think that it was previously failing the typefind for this file, and therefore skipping the subtests. install dumpresults-json that I discovered were needed while I was testing. I will try to rebase them by the time you get around to reviewing. If I don't manage it, they're all insta-reviewable, so I will tack them on when I fix any review comments you might have.
added more cruft to the end of the branch
Merged to master and packaged in my PPA == Steps to test using schroot == $ sudo debootstrap --arch i386 --verbose natty /srv/chroot/natty-i386 \ http://archive.ubuntu.com/ubuntu # This isn't a secure chroot: it's a scratch area with enough # information to make ubuntu behave itself and reduce the number # of moving parts in the test system. # A malicious program would be able to break out easily. $ sudo mount -o bind /proc /srv/chroot/natty-i386/proc $ sudo mount -o bind /dev /srv/chroot/natty-i386/dev $ sudo mount -o bind /sys /srv/chroot/natty-i386/sys $ echo " [natty-i386] personality=linux32 root-users=$USER" | sudo tee --append /etc/schroot/schroot.conf $ sudo cp /etc/shadow /etc/passwd /etc/group /etc/resolv.conf /etc/hosts \ /srv/chroot/natty-i386/etc/ $ sudo mkdir /srv/chroot/natty-i386/$HOME $ sudo chown $USER /srv/chroot/natty-i386/$HOME # Now chroot in and run the tests. $ (cd && schroot -p -c natty-i386 bash) $ sudo apt-get install python-software-properties python-setuptools $ sudo apt-add-repository http://archive.ubuntu.com/ubuntu universe $ sudo apt-add-repository ppa:alsuren/linaro $ sudo apt-get update $ bzr branch lp:~alsuren/lava-test/insanity $ cd insanity $ setup.py develop $ lava-test install insanity $ lava-test dashboard setup $http://$SERVER/launch-control/ $ lava-test run insanity # and copy-paste $RESULT below $ lava-test dashboard put /anonymous/$STREAM/ $RESULT