I can confirm this bug, it occurs with the very latest Ubuntu Intrepid NM0.7 SVN packages.  It is a regression from NM0.6, which redirected the default route just fine.  OpenVPN run from the command line also redirects the default route just fine, the problem appears only when using NM to handle the OpenVPN tunnel.

There's a lot more debug info in the Launchpad bug tracker, see https://bugs.launchpad.net/ubuntu/+source/network-manager-openvpn/+bug/269071

Tore

To route all traffic over the VPN (which is the function of redirect-gateway), one of the following must be true:

1) the openvpn server should not push down routes

or

2) You check "Ignore automatically obtained routes" in the connection editor for that connection, and you do not enter any custom routes

Then, all traffic will be routed over the VPN.  You also must have at least NM svn r4181 or later as well as NetworkManager-openvpn r4170 or later.

Please confirm that you have these versions or later of NM (I don't have any idea how the Ubuntu package versions map to SVN revision numbers, that's a question for the Ubuntu maintainer of NM), and also that either one of the two above are true.

It may be that Ubuntu included the NM-openvpn patch attached in http://bugzilla.gnome.org/show_bug.cgi?id=549196 (fixed in a more correct manner in svn revisions mentioned above), but that patch would negate the fixes in NM svn r4181 and later.

Hello,

The OpenVPN server has pushed down the default route (and only that), which worked with no further troubles with NM 0.6, and has always been working when OpenVPN was invoked directly outside of NM's control.  There hasn't been any change in server configuration either, so it's definetively a change in behaviour starting with NM 0.7.

However, when playing around a bit more I checked the «Ignore automatically obtained routes» setting you mentioned and started the connection, and lo and behold!  The default route is now added as expected!  So it seems the code itself is working, only that the setting does exactly the opposite of what it says it will do:  When the «ignore» setting was deactivated (default), NM ignored the route anyway.  When I told it to ignore VPN server routes by activating the setting, the route was respected and installed into my routing table.

I bet it's a really simple bug to fix, you probably just need to invert the logic in the code.  The default setting is sensible (the one in the GUI, that is, not the current default behaviour of ignoring routes), so that should probably left as-is (then it will be compatible with NM 0.6 behaviour).

Tore

Tore: any chance you could find out exactly what environment variables openvpn is executing its scripts with?  Doing something like "env > /tmp/openvpn-env.txt" at the top of the default openvpn script, and doing a connection, might work.

It may be that the server is sending the default route as an actual route, and the NM-openvpn plugin should detect that and simply not pass that route along to NetworkManager.

Created attachment 121299 [details]
Debug output from nm-openvpn-auth-dialog

Dan,

I could not find any scripts contained within Ubuntu's network-manager-openvpn package.  I suspect you might refer to the binaries in /usr/lib/network-manager-openvpn/?  I renamed all three of them so they got the suffix ".real", and replaced the original file names with the following script:

#! /bin/sh
echo args: "$@" >> /tmp/`basename $0`.log
env >> /tmp/`basename $0`.log
echo >> /tmp/`basename $0`.log
exec "$0".real "$@"

I then connected and disconnected the VPN service, once with the «Ignore auomatically obtained routes set, and once with it unset.  The files resulting from those attempts were identical (I checked with md5sum), so I'll only attach one set of them.  I've also replaced some data in them that could conceivably be used to identify the location of the OpenVPN server with strings enclosed in angle brackets.  Apologies for the paranoia...  Anyway, the scripts were only run when I activated the VPN service, no data was added to the debug files when I deactivated the VPN connection, at least.  I hope this helps you nail the bug, but do tell me if you think I could provide further information.

The bug manifested itself as expected, by the way;  when «Ignore automatically obtained routes» was unset, the default route given by the VPN router was ignored and.  When it was set, the route was not ignored, and installed to the routing table.

Your last comment confused me a bit, though.  What do you mean by an «actual» route, and how does it differ from a, uhm, «non-actual» route?

Tore

Created attachment 121300 [details]
Debug output from nm-openvpn-service

Created attachment 121301 [details]
Debug output from nm-openvpn-service-openvpn-helper

Thanks for the output!  Can you figure out from your admin (or if you are the admin) whether you're using --redirect-gateway on the server?  I don't think it is.

Your VPN server _is_ sending one route, but it's a host route to the DNS server, which is pretty useless when the VPN is intended to be the default route.

And BTW, re: https://bugs.launchpad.net/ubuntu/+source/network-manager-openvpn/+bug/269071/comments/20

It's not that NM starts respecting the OpenVPN routes when you check <<Ignore>>.  It does completely ignore the OpenVPN provided static routes.  But of course NM does routing setup of it's own, and that's where the confusion apparently is.

If the OpenVPN server sent static routes, it cannot mean that it wants the VPN to be the default route, because otherwise it wouldn't have sent pointless static routes.  Sending static routes is pointless when the VPN is supposed to be the default route, because any traffic destined for those static routes would already be routed over the default route anyway.  Thus they are redundant.

No matter what the VPN servers say, NM will always add a host route to the VPN gateway's external IP address, and should add a host route to the local DHCP server if DHCP is in-use on the interface which the VPN tunnel is tied to.

Redirect-gateway is in use.  The entire OpenVPN server configuration file is as follows (I left in a comment from the default file that seems relevant):

local <OpenVPN router IP address>
port 1194
proto udp
dev tun
ca x509/all.pem
cert /etc/ssl/certs/openvpn-cert.pem
key /etc/ssl/certs/openvpn-NOPASS_key.pem
dh x509/dh1024.pem
# Configure server mode and supply a VPN subnet
# for OpenVPN to draw client addresses from.
# The server will take 10.8.0.1 for itself,
# the rest will be made available to clients.
# Each client will be able to reach the server
# on 10.8.0.1. Comment this line out if you are
# ethernet bridging. See the man page for more info.
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "redirect-gateway"
push "dhcp-option DOMAIN <employer's domain name>"
push "dhcp-option DNS 10.8.0.1"
keepalive 10 120
comp-lzo
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
verb 3

This is OpenVPN 2.0-1sarge4 by the way.  There's no command line options that seems to be relevant (just config and pid file location and such things).

The static route you're seeing is something special to OpenVPN, I think.  Maybe it always sends it regardless of it being explicitly configured or not (it's not in my employer's case, as you can see).

When you have routes present to 10.8.0.1/32 and 0.0.0.0/0 at the same time, you're right that they are redundant.  But do I understand you correctly that nm-openvpn tries to optimise away the redundancy, but screws up and end up removing the route to 0.0.0.0/0 instead of the one to 10.8.0.0/32?  The latter could probably have been optimised away with no problems since it is covered within the 0.0.0.0/0 route, but the opposite is not true.

Anyway, in my opinion nm-openvpn should not really care about the redundancy and install all the received routes regardless of them being overlapping or not.  This appears to be the behaviour of both NM 0.6 and the standalone OpenVPN client, by the way.  The IP stack in the kernel will have no problem with this, it'll select the route with the longest matching prefix anyway.

Although probably not common, it could also be desired to send a host route as well as a default route in some situations, imagine for instance that I already had a route to 10.0.0.0/8 in my routing table via a nexthop on the LAN segment to which eth0 was connected.  If I then connect OpenVPN and it installs a new default route to 0.0.0.0/0, traffic to 10.8.0.1 will still not be routed through the tunnel since 10.0.0.0/8 is more specific than 0.0.0.0/0.  In that case you might want to have the OpenVPN server send a route to 10.8.0.1/32 in addition to the default route to ensure that traffic to 10.8.0.1 is also tunneled over the VPN connection.  I assume something like this is the reason for sending the 10.8.0.1 route unconditionally in the first place.

Have I also understood you correctly that if I select the «Ignore» option, nm-openvpn will ignore all received routes (both 0.0.0.0/0 and 10.8.0.1/32), and furthermore conclude that since there's no routes there at all, the tunnel doesn't make much sense, so better add the default route anyway since that's probably what was intended?  (In other words, if I removed «push "redirect-gateway"» from the server configuration file, the default route would still be redirected over the tunnel if the «Ignore» option was selected?)

Tore

(In reply to comment #11)
> The static route you're seeing is something special to OpenVPN, I think.  Maybe
> it always sends it regardless of it being explicitly configured or not (it's
> not in my employer's case, as you can see).

Could be special, but it also could be a result of passing "--route-noexec".  NM passes this because _NM_ controls the routing table, and having OpenVPN install the routes itself could conflict with other device routes or even your manual overrides of what OpenVPN sends.  Just to repeat: there are policy decisions made _after_ openvpn runs, which openvpn is not a part of, because openvpn doesn't know about the overall network config of the entire machine, or your overrides of its behavior.

> When you have routes present to 10.8.0.1/32 and 0.0.0.0/0 at the same time,
> you're right that they are redundant.  But do I understand you correctly that
> nm-openvpn tries to optimise away the redundancy, but screws up and end up
> removing the route to 0.0.0.0/0 instead of the one to 10.8.0.0/32?  The latter
> could probably have been optimised away with no problems since it is covered
> within the 0.0.0.0/0 route, but the opposite is not true.

nm-openvpn doesn't try to optimise away anything.  OpenVPN isn't sending the 0.0.0.0/0 route at all, in fact there's no way for an openvpn script to know that the server was configured with redirect-gateway, because openvpn does not pass that option to scripts in any form.  Even if it did, the vpn connection is still subject to NM and user policy applied after openvpn has run.

The _only_ route that openvpn passes to the script is 10.8.0.1/10.8.0.5/32.  There is no 0.0.0.0 route passed to nm-openvpn.

> Anyway, in my opinion nm-openvpn should not really care about the redundancy
> and install all the received routes regardless of them being overlapping or
> not.  This appears to be the behaviour of both NM 0.6 and the standalone
> OpenVPN client, by the way.  The IP stack in the kernel will have no problem
> with this, it'll select the route with the longest matching prefix anyway.

NM will install the received routes, provided they are not overridden by user policy or another higher priority connection.  The problem with what you say here is that openvpn does not indicate any default route because --redirect-gateway is not indicated to nm-openvpn in any way.

Even if it was, the default route may be taken by a higher priority device or manual user overrides.  Just because openvpn says it wants the default route doesn't mean it will get the default route, because there could be other devices and VPNs connected at the same time.

> Although probably not common, it could also be desired to send a host route as
> well as a default route in some situations, imagine for instance that I already
> had a route to 10.0.0.0/8 in my routing table via a nexthop on the LAN segment
> to which eth0 was connected.  If I then connect OpenVPN and it installs a new
> default route to 0.0.0.0/0, traffic to 10.8.0.1 will still not be routed
> through the tunnel since 10.0.0.0/8 is more specific than 0.0.0.0/0.  In that
> case you might want to have the OpenVPN server send a route to 10.8.0.1/32 in
> addition to the default route to ensure that traffic to 10.8.0.1 is also
> tunneled over the VPN connection.  I assume something like this is the reason
> for sending the 10.8.0.1 route unconditionally in the first place.

Correct, but because openvpn cannot ever know the full network situation of your machine at a given time, you may also have a 10.8.0.1 that you want to access on the _other_ connected network.  When you start overlapping networks things will get both quite confusing and quite unclear, no matter what openvpn wants to do with host routes.

> Have I also understood you correctly that if I select the «Ignore» option,
> nm-openvpn will ignore all received routes (both 0.0.0.0/0 and 10.8.0.1/32),

Again, openvpn does not send a 0.0.0.0/0 route to nm-openvpn at all.  There is no indication that openvpn wishes the client to create a default route through the VPN.

> and furthermore conclude that since there's no routes there at all, the tunnel
> doesn't make much sense, so better add the default route anyway since that's
> probably what was intended?  (In other words, if I removed «push
> "redirect-gateway"» from the server configuration file, the default route
> would still be redirected over the tunnel if the «Ignore» option was
> selected?)

If the the VPN service does not push any routes to the client, and you have not manually overridden the routes that the VPN server sends by either (a) entering your own in the connection editor, or (b) checking <<Ignore>>, then the VPN will receive the default route.

I'm probably going to adjust NetworkManager code to ignore /32 routes in the default route calculation code, which will fix your case without you having to change the VPN server configuration.

svn r4212 will ignore /32 vpn-provided routes when determining whether the VPN connection should be the default route.

Thanks for spelling it out for me, now everything is much clearer to me!  :-)  This is what had me confused:

> Again, openvpn does not send a 0.0.0.0/0 route to nm-openvpn at all.  There is
> no indication that openvpn wishes the client to create a default route through
> the VPN.

I had no idea this was the case, I assumed that OpenVPN passed 0.0.0.0/0 on to NM if redirect-gateway was in use, and that NM attempted to do some (wrong) route coalescing magic that caused it to be ignored.  I was arguing against attempting doing any such thing, which was pointless as:

> NM will install the received routes, provided they are not overridden by user
> policy or another higher priority connection.

is a very sensible approach that I was trying to advocate, apologies for getting you wrong on this count.

Ignoring /32s will most likely solve my problem.  But it feels like a workaround, the _real_ bug here is that OpenVPN doesn't pass the 0.0.0.0/0 route on to NM when redirect-gateway is in use, agreed?

I wonder if this changed recently...  It used to work with NM 0.6, but I don't know if that's due to OpenVPN introducing this bug or that the route calculation in NM 0.6 worked differently (both OpenVPN and NM 0.6 was upgraded at the same time, going from Ubuntu Hardy to Intrepid).

By the way, «--route-noexec» is not in use.  So it's probably a built-in thing.

Oh, and one more thing.  Are you developing on NM core, or just OpenVPN?  I've got another problem with NM, see, that it throws out some routes on DHCP lease removal.  This includes the host-route added to the VPN router, and if that's combined with a setup where the default route points to the VPN tunnel you end up in a catch-22 where you can't really communicate with the VPN router any longer.  If you feel like taking a look at it, see <https://bugs.launchpad.net/bugs/288703>.  I'll be happy to submit a new bug report in this bug tracker for that issue should you prefer it.

Thanks for your help!  :-)

Tore

(In reply to comment #14)
> Ignoring /32s will most likely solve my problem.  But it feels like a
> workaround, the _real_ bug here is that OpenVPN doesn't pass the 0.0.0.0/0
> route on to NM when redirect-gateway is in use, agreed?

Yes, but even if it did, NM would still override the default route through the openvpn tunnel if needed, based on user-defined static routes, higher priority connections, etc.

> I wonder if this changed recently...  It used to work with NM 0.6, but I don't
> know if that's due to OpenVPN introducing this bug or that the route
> calculation in NM 0.6 worked differently (both OpenVPN and NM 0.6 was upgraded
> at the same time, going from Ubuntu Hardy to Intrepid).

NM 0.6 wasn't as capable as 0.7 is, didn't handle multiple devices, couldn't allow users to do static IP addressing or manual routing, etc.  The 0.6 and 0.7 code work differently.

> By the way, «--route-noexec» is not in use.  So it's probably a built-in
> thing.

NM-openvpn passes --route-noexec to openvpn on the command line to ensure that openvpn doesn't touch the routing table itself.

> Oh, and one more thing.  Are you developing on NM core, or just OpenVPN?  I've

Both.

> got another problem with NM, see, that it throws out some routes on DHCP lease
> removal.  This includes the host-route added to the VPN router, and if that's
> combined with a setup where the default route points to the VPN tunnel you end
> up in a catch-22 where you can't really communicate with the VPN router any

What exact SVN version of NetworkManager are you using?  I tested this out yesterday with 2m lease times on the wifi router, with vpnc connection.  I could not get any routes to disappear, mainly becuase the connection was continually renewed after the first renewal, and NM doesn't do anything if the DHCP state doesn't change.

But a new bug would be good, as long as you're using svn r4181 or later.  I don't actually know what SVN revision asac is pulling into the Ubuntu repos so you may have to find out from him.  r4181 contains some routing fixes that might have fixed this.

Dan

> Yes, but even if it did, NM would still override the default route through the
> openvpn tunnel if needed, based on user-defined static routes, higher priority
> connections, etc.

Understood.  I would have expected that, anyway.

> NM-openvpn passes --route-noexec to openvpn on the command line to ensure that
> openvpn doesn't touch the routing table itself.

Aha.  I assumed (without actually checking :-) that --route-noexec was a server thing.

> But a new bug would be good, as long as you're using svn r4181 or later.  I
> don't actually know what SVN revision asac is pulling into the Ubuntu repos so
> you may have to find out from him.  r4181 contains some routing fixes that
> might have fixed this.

I believe I'm using r4191, and have submitted a new bug report:

http://bugzilla.gnome.org/show_bug.cgi?id=558133

Over and out from this one, see you in that one...

Thanks again for fixing this one so quickly!

Tore