GNOME Bugzilla – Bug 558133
NM nukes certain routes on DHCP lease renewal, breaks VPN
Last modified: 2008-11-07 13:58:14 UTC
Please describe the problem: When NM first activates an interface, routes might be added. For instance, the OpenVPN plugin will add a host route to the VPN router with the same next hop as the default route (prior to it changing due to VPN activation), and activating my wlan adds a Zeroconf route to 169.254.0.0/16. My WLAN uses DHCP. When the lease is renewed, these extra routes are lost. This breaks OpenVPN completely since the host can no longer communicate with the VPN router (if the default route, or the prefix the VPN router resides in, is routed to the tunnel device). In the Ubuntu bug tracker there is also a report from a guy which see the same thing happening with a host route added by the VPNC parts of NM, so I assume this happens somewhere in the core of NM, not in the VPN plugins. Steps to reproduce: 1. Activate a connection to a [W]LAN using DHCP. 2. Wait for the DHCP lease to be renewed. Actual results: After step 1, a route to the Zeroconf network (169.254/16) is present in the kernel's routing table. After step 2, it has vanished. Expected results: The route to the Zeroconf network should be present after step 2 as well. Does this happen every time? Yes. Other information: I've observed the bug on a day-fresh Ubuntu Intrepid, running NM version 0.7~~svn20081018t105859-0ubuntu1. According to the changelog, this is upstream BZR version 3648, but I think that version number is applicable only to Ubuntu's BZR repo. The time stamp in the version number maps exactly to SVN rev 4191. URL to the sibling bug in Launchpad: https://bugs.launchpad.net/bugs/288703
Created attachment 121445 [details] Contents of syslog from when I reproduced it. What happened here, was that I: 1) Configured my DSL CPE to have a lease time of 600 seconds, 2) Reset the WLAN connection by clicking the network name in the list of available WLANs, 3) Waited for the setup to complete, then ran "ip r | logger -t ip", 4) Waited for DHCP lease renewal, then ran "ip r | logger -t ip".
Is the _only_ connection in-use the WLAN connection? ie you do not have a cable plugged in at the time?
I've seen this both on my laptop, which is using WLAN only (eth0 is disconnected), and on my desktop PC which are only using wired ethernet (no wireless network interface card is present). Both are running Ubuntu Intrepid. If you want I can try with both wired and wireless ethernet active on my laptop, too. Tore
I tested it in Fedora 10 beta (completely updated) too, by the way, but I was unable to reproduce it there: 1) Fedora doesn't seem to add the Zeroconf link-local route by default so I could not monitor it for going AWOL, and even if it had been added I still would have been unable to confirm it, because: 2) Fedora doesn't appear to renew DHCP leases when they expire, for some reason. This looks like a separate bug to me. The lack of lease renewals does in any case prevent this bug from manifesting itself, even if it is present. Tore
I found out that the Zeroconf route is added by avahi-autoipd. Not sure if it cooperates with NM or not, but if I deinstall all avahi-* packages, the route does not show up anymore. Presumably this isn't installed by default on F10. However the host route to the VPN gateway is still removed upon DHCP lease renewal like before... Tore
On the Fedora 10 beta, can you run Nm from a terminal (with /usr/sbin/NetworkManager --no-daemon) and wait until the DHCP renewal, then attach the logs here? dhclient does renew: DHCPREQUEST on eth2 to 192.168.1.1 port 67 DHCPACK from 192.168.1.1 bound to 192.168.1.101 -- renewal in 57 seconds. DHCPREQUEST on eth2 to 192.168.1.1 port 67 DHCPACK from 192.168.1.1 bound to 192.168.1.101 -- renewal in 53 seconds. Need to see what yours does.
Okay, I'll attach output in a sec. It gets the lease just fine and says it will renew it, but it never does. This pretty much says it all: [tore@wrath ~]$ date ; cat /var/run/dhclient-eth0.lease ma. 03. nov. 18:43:50 +0100 2008 lease { interface "eth0"; fixed-address 10.0.0.53; option subnet-mask 255.255.255.0; option routers 10.0.0.1; option dhcp-lease-time 600; option dhcp-message-type 5; option domain-name-servers 217.13.7.140,217.13.4.24; option dhcp-server-identifier 10.0.0.1; option domain-name "lan"; renew 1 2008/11/03 18:32:19; rebind 1 2008/11/03 18:36:36; expire 1 2008/11/03 18:37:51; } Tore
Created attachment 121888 [details] Output from NetworkManager --no-daemon
Created attachment 121890 [details] Output from sealert -l c319651c-b204-4bc0-b828-d5de301a24c8 I just noticed the following message appear in the messages log as NM got the initial DHCP lease, thought maybe it could be relevant: setroubleshoot: SELinux is preventing dhclient (dhcpc_t) "read" to ./nm-dhclient-eth0.conf (var_run_t). For complete SELinux messages. run sealert -l c319651c-b204-4bc0-b828-d5de301a24c8 I've attached the output from this command. I also tried the suggested workaround (after changing directory to /var/run): restorecon -v './nm-dhclient-eth0.conf' But it did not help (the message still appeared after restarting NM). I suspect that is due to NM apparantly re-creating this file every time it is started or something, and that I therefore applied the workaround to the old, now deleted, version of the file. But I'm a SELinux newbie so I really have no idea. I'll try to see if I can figure out if it is possible to disable it completely. Tore
Bingo! With SELinux disabled (echo 0 > /selinux/enforce), the DHCP lease gets renewed as expected. I see you're involved in the packaging of NM for Fedora, do you want me to go to their bug tracker and submit another report, or will this one here suffice? Now I can also confirm that the original bug (routes going AWOL when the DHCP lease is renewed) does _not_ occur in F10, so that's something specific to Ubuntu (or the SVN version or NM that they're shipping). Tore
Some more info - after a yum update & reboot, the DHCP renewal stuff works fine. Which confuses me as none of the packages that was upgraded had anything to do with SELinux nor NM: Nov 03 18:48:20 Updated: pulseaudio-libs-0.9.13-6.fc10.x86_64 Nov 03 18:48:20 Updated: pulseaudio-core-libs-0.9.13-6.fc10.x86_64 Nov 03 18:48:21 Updated: pulseaudio-0.9.13-6.fc10.x86_64 Nov 03 18:48:21 Updated: pulseaudio-module-x11-0.9.13-6.fc10.x86_64 Nov 03 18:48:21 Updated: pulseaudio-libs-glib2-0.9.13-6.fc10.x86_64 Nov 03 18:48:21 Updated: ksysguardd-4.1.2-10.fc10.x86_64 Nov 03 18:48:26 Updated: amarok-1.94-2.fc10.x86_64 Nov 03 18:48:26 Updated: pulseaudio-utils-0.9.13-6.fc10.x86_64 Nov 03 18:48:28 Updated: solar-kde-theme-0.1.12-1.fc10.noarch Nov 03 18:48:28 Updated: kde-settings-kdm-4.1-4.20081031svn.fc10.noarch Nov 03 18:48:28 Updated: fedora-release-9.93-1.noarch Nov 03 18:48:28 Updated: kde-settings-4.1-4.20081031svn.fc10.noarch Nov 03 18:48:30 Updated: initscripts-8.85-1.x86_64 Nov 03 18:48:30 Updated: kde-settings-pulseaudio-4.1-4.20081031svn.fc10.noarch Nov 03 18:48:30 Updated: rpm-libs-4.6.0-0.rc1.7.x86_64 Nov 03 18:48:30 Updated: kdebase-workspace-libs-4.1.2-10.fc10.x86_64 Nov 03 18:48:40 Updated: kdebase-workspace-4.1.2-10.fc10.x86_64 Nov 03 18:48:41 Updated: rpm-4.6.0-0.rc1.7.x86_64 Nov 03 18:48:41 Updated: rpm-python-4.6.0-0.rc1.7.x86_64 Nov 03 18:48:45 Updated: system-config-users-1.2.81-1.fc10.noarch Nov 03 18:49:02 Updated: rpm-4.6.0-0.rc1.7.x86_64 ...but somehow it seems fixed. Oh well... Tore
About this bug being Ubuntu-specific: I take it back. I just saw the host route disappear on F10 beta (fully updated this time). I'll be attaching the full /var/log/messages (from when NM was started to when the route disappeared) in a second. Routing table prior to lease renewal: [tore@wrath ~]$ ip r $VPN_GW_ADDRESS via 10.0.0.1 dev eth0 proto static 10.8.0.5 dev tun0 proto kernel scope link src 10.8.0.6 10.8.0.1 via 10.8.0.5 dev tun0 proto static 10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.53 metric 1 default via 10.8.0.5 dev tun0 proto static After lease renewal: [tore@wrath ~]$ ip r 10.8.0.5 dev tun0 proto kernel scope link src 10.8.0.6 10.8.0.1 via 10.8.0.5 dev tun0 proto static 10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.53 metric 1 default via 10.8.0.5 dev tun0 proto static My latest suspicion is that this bug only occurs if the VPN connection was started before the DHCP lease was renewed for the first time. I'll try to confirm that. Tore
Created attachment 121903 [details] messages log file from VPN problem occuring on F10
My suspicion turned out to be correct. If the VPN tunnel is set up before the first renewal of the lease, the host route is removed. If it is after the first renewal, it's left intact. I hadn't noticed this before becuase starting the VPN connection is usually one of the very first things I do after logging in. It seems NM does a lot more magic during the first renewal than on subsequent ones, compare: dhclient: DHCPREQUEST on eth0 to 10.0.0.1 port 67 dhclient: DHCPACK from 10.0.0.1 dhclient: bound to 10.0.0.53 -- renewal in 287 seconds. NetworkManager: <info> DHCP: device eth0 state changed bound -> renew NetworkManager: <info> address 10.0.0.53 NetworkManager: <info> prefix 24 (255.255.255.0) NetworkManager: <info> gateway 10.0.0.1 NetworkManager: <info> nameserver '217.13.7.140' NetworkManager: <info> nameserver '217.13.4.24' NetworkManager: <info> domain name 'lan' avahi-daemon[2442]: Withdrawing address record for 10.0.0.53 on eth0. avahi-daemon[2442]: Leaving mDNS multicast group on interface eth0.IPv4 with address 10.0.0.53. avahi-daemon[2442]: Interface eth0.IPv4 no longer relevant for mDNS. avahi-daemon[2442]: Joining mDNS multicast group on interface eth0.IPv4 with address 10.0.0.53. avahi-daemon[2442]: New relevant interface eth0.IPv4 for mDNS. avahi-daemon[2442]: Registering new address record for 10.0.0.53 on eth0.IPv4. NetworkManager: <info> Policy set 'Auto eth0' (eth0) as default for routing and DNS. with: dhclient: DHCPREQUEST on eth0 to 10.0.0.1 port 67 dhclient: DHCPACK from 10.0.0.1 dhclient: bound to 10.0.0.53 -- renewal in 256 seconds. Those blocks are what is being logged to /var/log/messages during the first renewal and then any subsequent renewals. I could see no difference between the log messages from a first renewal with VPN active and from one without (except that the default policy set was 'VPN-tilkobling 1' - Norwegian for 'VPN connection 1'). I'll include them here anyway in case I missed something: dhclient: DHCPREQUEST on eth0 to 10.0.0.1 port 67 dhclient: DHCPACK from 10.0.0.1 NetworkManager: <info> DHCP: device eth0 state changed bound -> renew NetworkManager: <info> address 10.0.0.53 NetworkManager: <info> prefix 24 (255.255.255.0) NetworkManager: <info> gateway 10.0.0.1 NetworkManager: <info> nameserver '217.13.7.140' NetworkManager: <info> nameserver '217.13.4.24' NetworkManager: <info> domain name 'lan' dhclient: bound to 10.0.0.53 -- renewal in 239 seconds. avahi-daemon[2442]: Withdrawing address record for 10.0.0.53 on eth0. avahi-daemon[2442]: Leaving mDNS multicast group on interface eth0.IPv4 with address 10.0.0.53. avahi-daemon[2442]: Interface eth0.IPv4 no longer relevant for mDNS. avahi-daemon[2442]: Joining mDNS multicast group on interface eth0.IPv4 with address 10.0.0.53. avahi-daemon[2442]: New relevant interface eth0.IPv4 for mDNS. avahi-daemon[2442]: Registering new address record for 10.0.0.53 on eth0.IPv4. NetworkManager: <info> Policy set 'VPN-tilkobling 1' (tun0) as default for routing and DNS. Subsequent renewals also look identical regardless of VPN being active or not. Are you now able to reproduce the problem? (I hope so!) Tore
Fixed in svn r4277.