GNOME Bugzilla – Bug 700823
Add integration test suite
Last modified: 2017-09-22 14:06:43 UTC
As discussed by email/IRC already, I'm currently working on an integration test suite for NetworkManager. This uses the mac80211_hwsim kernel module and Linux' veth devices, hostapd, and dnsmasq to provide simulated wifi and ethernet hardware with routers/APs behind them. With that we can create arbitrary configurations of APs with and without encryption, 802.11a/b/g, simulate hotplugging and suspend, and so on. All these are already working. The current version is at http://bazaar.launchpad.net/~network-manager/network-manager/ubuntu/view/head:/debian/tests/network_test_base.py http://bazaar.launchpad.net/~network-manager/network-manager/ubuntu/view/head:/debian/tests/nm if you want to take a first look and comment. We currently run these against the 0.9.8.0 Ubuntu package (with a few patches) in Jenkins: https://jenkins.qa.ubuntu.com/view/Saucy/view/AutoPkgTest/job/saucy-adt-network-manager/ Once we square away some prerequisites (I'll link the bug dependencies), I'll integrate them into "make check"/"make installcheck" (when being run as root), and submit a git patch.
Hi Martin, I'm delighted to see you working on this. I'll certainly find time for looking at it later and I'd also like to talk to LNST developers about their view, too. But this looks pretty much amazing already. Btw, we already differentiate classic and root tests, the latter potentially affecting the runtime network status (trying to avoid it as much as possible). I'm curious whether new development for kernel 3.9 and 3.10 could help you running the tests with non-root UID. I believe the kernel namespaces are moving towards a very flexible framework for stuff like that.
(In reply to comment #1) > I'm delighted to see you working on this. I'll certainly find time for looking > at it later If you have questions, feel free to ping me on IRC ("pitti" on freenode and GNOME networks). > Btw, we already differentiate classic and root tests, the latter potentially > affecting the runtime network status (trying to avoid it as much as possible). For now, I SIGKILL the running "production" NM daemon before running tests, as that keeps the "real" devices connected. But most of the time I just do the entire testing in a VM. But for a "sudo make check" this is indeed something which we need to automate, it just hasn't been my priority just yet. > I'm curious whether new development for kernel 3.9 and 3.10 could help you > running the tests with non-root UID. I believe the kernel namespaces are moving > towards a very flexible framework for stuff like that. Sounds interesting, I'll read about that. But if nothing else, I currently need root to stop the system NM and to modprobe the mac80211_hwsim modules; users should not be able to do either. Thanks!
Created attachment 245435 [details] [review] Add integration test suite This provides the current version of the test as git formatted patch, with proper make integration. I added a test/README which explains what the test suite does, that it can be run on production systems, and how to call the various modes (build tree vs. installed, single test vs. whole suite, interactive, etc.). Dependencies: * Packages: python3, gir1.2-networkmanager-1.0, hostapd, iw * Optional: python3-dbusmock (https://pypi.python.org/pypi/python-dbusmock) for running the suspend tests * A fixed mac80211_hwsim module which registers a proper driver; this needs http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=9ea92 (it's not yet in 3.9 I believe); I can walk you through quickly compiling this module only with the patch, or we add a temporary workaround to NM to ignore a missing "driver" attribute for a mac80211_hwsim device. * The patch from bug 701112 to avoid "sudo make check" failing early in the existing tests (you can locally apply it without pushing for now, of course). These run quite reliably for me in a VM (both "installcheck" against the installed 0.9.8.0 as well as "check" against the build tree), but I often get timeouts when I run them on my workstation. If you get a timeout, please try to re-run the suite (or the individual test), we still need to debug those (it could be that I make wrong assumptions in the tests, or they are an actual bug). I'd appreciate if you could try them on a current Fedora, to uncover any hidden/missing dependencies, proper error messages when they are missing, Debian/Ubuntu specific assumptions about paths, and so on. Please also let me know if anything that the tests do is unclear; I'd like to add enough comments to them for another developer to understand them, as eventually these should grow regression tests for bugs, new features, etc. Thanks, Martin
Created attachment 245436 [details] [review] Add integration test suite Argh, still had some uncommitted amendments to test/README.
BTW, I see relatively little technical overlap between this and LNST (that's just the creation of veth devices, which is only some 3 lines of code). Also, there doesn't seem to be a conceptual overlap: LNST is for testing network communication and higher-level protocols between machines (real or VMs), i. e. anything on the IP level and higher; while this test suite is for configuring network devices on one local machine (hardware and MAC layers, we can't test proper data communication other than RA and DHCP). So the NM tests are conceptually lower in the stack than the LNST tests, and they nicely complement each other.
(In reply to comment #5) > BTW, I see relatively little technical overlap between this and LNST (that's > just the creation of veth devices, which is only some 3 lines of code). Also, > there doesn't seem to be a conceptual overlap: LNST is for testing network > communication and higher-level protocols between machines (real or VMs), i. e. > anything on the IP level and higher; Thanks for good summary. Yes, LNST specializes itself on multi-host test configurations and virtual/physical agnosticism. > while this test suite is for configuring > network devices on one local machine (hardware and MAC layers, we can't test > proper data communication other than RA and DHCP). Actually with dummy devices it could be possible to do proper RA/DHCP on localhost even without virtualazation/containers. > So the NM tests are > conceptually lower in the stack than the LNST tests, and they nicely complement > each other. Thanks for the information. I will get back to the testsuite later.
(In reply to comment #6) > > while this test suite is for configuring > > network devices on one local machine (hardware and MAC layers, we can't test > > proper data communication other than RA and DHCP). > > Actually with dummy devices it could be possible to do proper RA/DHCP on > localhost even without virtualazation/containers. For clarity, we *can* do RA/DHCP with these virtual devices, that's part of the tests. We just can't do anything TCP based, as those don't talk in terms of interface names any more; maybe there is a clever trick to do that too, of course, but I don't think doing that is important for testing NetworkManager itself.
(In reply to comment #7) > For clarity, we *can* do RA/DHCP with these virtual devices, that's part of the > tests. My bad. Was answering a number of other things... > We just can't do anything TCP based, as those don't talk in terms of > interface names any more; maybe there is a clever trick to do that too, of > course, but I don't think doing that is important for testing NetworkManager > itself. OK to me.
/me pokes thaller to look at this stuff
This branch offers one solution to bug 749374. While it has patches/code ready, it still needs some consensus and work to see which integration testing we add to upstream. I let this bug unblock 'nm-review' for now... but it shall not be forgotten.
FTR, we've maintained and run these tests on the Ubuntu side in the past two years; they naturally evolved a bit, got adjusted to NM 0.9.10. So the attached patches are out of date and shouldn't be applied as-is; the structure hasn't changed, but the two files referenced in the description should be taken out of the current bzr packaging branch, not from the attached patches any more.
I think this is obsolete by now. Closing. There is https://github.com/NetworkManager/NetworkManager-ci where our (previously Red Hat internal) CI tests are. They can run on upstream CentosCI, and there is a vagrant script to run it on Ubuntu as well (in virtual machines). The aim is to make this independent of RHEL/Fedora (and it mostly is). Somebody can take it, and try to run in it's own infrastructure. Patches/improvements are very welcome.