After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 759892 - Regression: server pushes redirect-gateway but client does no longer sets default route
Regression: server pushes redirect-gateway but client does no longer sets def...
Status: RESOLVED FIXED
Product: NetworkManager
Classification: Platform
Component: VPN: openvpn
1.0.x
Other Linux
: Normal major
: ---
Assigned To: NetworkManager maintainer(s)
NetworkManager maintainer(s)
: 759949 (view as bug list)
Depends on:
Blocks:
 
 
Reported: 2015-12-26 23:22 UTC by dimitris
Modified: 2016-01-06 09:42 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Successful log (63.04 KB, text/plain)
2016-01-04 16:56 UTC, Beniamino Galvani
  Details
Failed log (55.15 KB, text/plain)
2016-01-04 16:59 UTC, Beniamino Galvani
  Details
My openvpn server configuration (10.31 KB, text/x-chdr)
2016-01-04 17:42 UTC, Beniamino Galvani
  Details
core: fix failure to configure routes with IPv4 peer-addresses (2.31 KB, patch)
2016-01-04 18:33 UTC, Thomas Haller
none Details | Review

Description dimitris 2015-12-26 23:22:27 UTC
On Fedora 23, original report at
https://bugzilla.redhat.com/show_bug.cgi?id=1294309

Server is configured with:

push		"redirect-gateway"

Client plugin-generated config says:

[ipv4]
dns-search=
method=auto

With VPN inactive, default route is:

default via 192.168.1.1 dev wlp3s0  proto static  metric 600 

Connecting VPN, journal entries include:

Dec 26 15:16:40 vimes NetworkManager[1404]: <info>  IPv4 configuration:
Dec 26 15:16:40 vimes NetworkManager[1404]: <info>    Internal Gateway: 10.24.24.5
Dec 26 15:16:40 vimes NetworkManager[1404]: <info>    Internal Address: 10.24.24.6
Dec 26 15:16:40 vimes NetworkManager[1404]: <info>    Internal Prefix: 32
Dec 26 15:16:40 vimes NetworkManager[1404]: <info>    Internal Point-to-Point Address: 10.24.24.5
Dec 26 15:16:40 vimes NetworkManager[1404]: <info>    Maximum Segment Size (MSS): 0
Dec 26 15:16:40 vimes NetworkManager[1404]: <info>    Static Route: 10.24.24.1/32   Next Hop: 10.24.24.5
Dec 26 15:16:40 vimes NetworkManager[1404]: <info>    Forbid Default Route: no

but later:

Dec 26 15:16:40 vimes NetworkManager[1404]: <warn>  platform-linux: do-add-ip4-route: failure adding ip4-route '6: 0.0.0.0/0 50': Unspecific failure (1)
Dec 26 15:16:40 vimes NetworkManager[1404]: <warn>  default-route: failed to add default route 0.0.0.0/0 via 10.24.24.5 dev 6 metric 50 mss 0 src vpn with effective metric 50

So I'm left with the original default routing.

This is a recent regression, I'd say within the last month or so.
Comment 1 dimitris 2015-12-27 00:03:26 UTC
Adding:

topology subnet

to the server configuration mostly fixes this; Default route is now

default via 10.24.24.1 dev tun0  proto static  metric 50 
default via 192.168.1.1 dev wlp3s0  proto static  metric 600 

It would be even better if there was a way to completely make the original default route disappear.  The rationale is that, if my VPN server is pushing a default route, it's because I want to treat the local network as hostile.  Leaving that default route in the table, even at a higher metric, can lead to traffic leaking out of the VPN if the tunnel is dropped.  At that point I would prefer to have no connectivity instead of untrusted connectivity.

Note that I'm using "redirect-route", not "redirect-route def1", in the serve config.

I do seem to remember older versions of NetworkManager-openvpn plugin doing exactly that - completely removing the original default route as long as the VPN tunnel was up.  Running the openvpn process from the command line, instead of managed through NM-openvpn, still does the same.  Is it possible to get the old behavior back?

Should I pursue this with a separate RFE, or is this issue good to track this?
Comment 2 5GS1G3aSwS54Vgv 2015-12-28 21:12:54 UTC
I can confirm this problem with version 1.0.10 on Manjaro. The VPN connects but isn't set as the default route so the real IP address is still visible, but if you use the openvpn command on it's own (without using NetworkManager-openvpn) then the VPN is set as the default route and everything works correctly.
Comment 3 Thomas Haller 2015-12-29 12:26:33 UTC
(In reply to dimitris from comment #0)
> On Fedora 23, original report at

> Dec 26 15:16:40 vimes NetworkManager[1404]: <info>  IPv4 configuration:
> Dec 26 15:16:40 vimes NetworkManager[1404]: <info>    Internal Gateway:
> 10.24.24.5
> Dec 26 15:16:40 vimes NetworkManager[1404]: <info>    Internal Address:
> 10.24.24.6
> Dec 26 15:16:40 vimes NetworkManager[1404]: <info>    Internal Prefix: 32
> Dec 26 15:16:40 vimes NetworkManager[1404]: <info>    Internal
> Point-to-Point Address: 10.24.24.5
> Dec 26 15:16:40 vimes NetworkManager[1404]: <info>    Maximum Segment Size
> (MSS): 0
> Dec 26 15:16:40 vimes NetworkManager[1404]: <info>    Static Route:
> 10.24.24.1/32   Next Hop: 10.24.24.5
> Dec 26 15:16:40 vimes NetworkManager[1404]: <info>    Forbid Default Route:
> no
> 
> but later:
> 
> Dec 26 15:16:40 vimes NetworkManager[1404]: <warn>  platform-linux:
> do-add-ip4-route: failure adding ip4-route '6: 0.0.0.0/0 50': Unspecific
> failure (1)
> Dec 26 15:16:40 vimes NetworkManager[1404]: <warn>  default-route: failed to
> add default route 0.0.0.0/0 via 10.24.24.5 dev 6 metric 50 mss 0 src vpn
> with effective metric 50


NM wants to configure a default-route, but it fails to do so. Can you attach a logfile with DEBUG enabled?

You can do that by editing /etc/NetworkManager/NetworkManager.conf

  [logging]
  level=DEBUG

then restart NM and reproduce.

(the logfile should not contain any sensitive data, but you might want to check that before posting).
Comment 4 Thomas Haller 2015-12-29 12:44:17 UTC
(In reply to dimitris from comment #1)
> Adding:
> 
> topology subnet
> 
> to the server configuration mostly fixes this; Default route is now
> 
> default via 10.24.24.1 dev tun0  proto static  metric 50 
> default via 192.168.1.1 dev wlp3s0  proto static  metric 600 
> 
> It would be even better if there was a way to completely make the original
> default route disappear.  The rationale is that, if my VPN server is pushing
> a default route, it's because I want to treat the local network as hostile. 
> Leaving that default route in the table, even at a higher metric, can lead
> to traffic leaking out of the VPN if the tunnel is dropped.  At that point I
> would prefer to have no connectivity instead of untrusted connectivity.

The "metric 600" route is overruled by the "metric 50" route. So, as long as VPN is up, it doesn't matter.

Once VPN goes down, there is only the wlp3s0 device left. As you configured wlp3s0 to have a default route (never-default=no), NM will assign the default route to that device.

If you want wlp3s0 not to have a default-route at all, configure it as never-default=yes.

You basically ask NM to sometimes assign a default-route to wlp3s0 and sometimes not (... when VPN ~unexpectedly~ goes down?). It is not clear how that would work.


> Note that I'm using "redirect-route", not "redirect-route def1", in the
> serve config.


> I do seem to remember older versions of NetworkManager-openvpn plugin doing
> exactly that - completely removing the original default route as long as the
> VPN tunnel was up.

Yes, that changed. But I don't think it matters, as also an older NM would have ~restored~ the default-route via wlp3s0 once VPN goes down (otherwise it'd be a bug). So that didn't protect you any better.

> Running the openvpn process from the command line,
> instead of managed through NM-openvpn, still does the same.  Is it possible
> to get the old behavior back?

Running openvpn client outside of NM is entirely different, because then openvpn (or your scripts) are in charge of configuring routes and addresses. NM's plugin runs openvpn with --route-noexec and --ifconfig-noexec and does the IP configuration itself.
Comment 5 Thomas Haller 2015-12-29 12:51:15 UTC
(In reply to Thomas Haller from comment #4)
> (In reply to dimitris from comment #1)

> You basically ask NM to sometimes assign a default-route to wlp3s0 and
> sometimes not (... when VPN ~unexpectedly~ goes down?). It is not clear how
> that would work.

maybe related to RFE from bug 749376 ...
Comment 6 Beniamino Galvani 2016-01-01 23:39:48 UTC
A bisection indicates commit df4e5357521d ("platform: fix event_handler_read_netlink_one() wrongly returning with nothing to read") as the culprit. But at first glance the change looks right; this needs more investigation.
Comment 7 Beniamino Galvani 2016-01-04 16:56:56 UTC
Created attachment 318218 [details]
Successful log

Successful log with trace enabled, commit df4e5357521d^
Comment 8 Beniamino Galvani 2016-01-04 16:59:48 UTC
Created attachment 318219 [details]
Failed log

Commit df4e5357521 (or master, I'm not sure).
Comment 9 Beniamino Galvani 2016-01-04 17:42:44 UTC
Created attachment 318223 [details]
My openvpn server configuration

It's basically the default configuration changed for password authentication.
Comment 10 Thomas Haller 2016-01-04 17:52:55 UTC
in attachment 318219 [details], we have a /32 peer-to-peer address:

adding or updating IPv4 address: 10.8.0.58/32 lft forever pref forever lifetime 21-0[4294967295,4294967295] ptp 10.8.0.57 dev 32 src unknown


for which kernel adds the following route:

added: 10.8.0.57/32 via 0.0.0.0 dev 32 metric 0 mss 0 src rtprot-kernel scope link pref-src 10.8.0.58



then, route-manager deletes that route, because for the /32 p2p address we wrongly assume that the device-route is 10.8.0.58/32, instead of 10.8.0.57/32:


http://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/src/nm-ip4-config.c?id=a232f885c233dbe4f2489c7a9691bd0b8c9816c5#n301

(on master that is handled properly via http://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/src/nm-ip4-config.c?id=0e164ac6076c855ea7263bdf2d31b3ec682c5a1f#n312



as route-manager deletes the 10.8.0.57/32 route, adding later a route like
10.8.0.1/32 via 10.8.0.57 dev 32 metric 50 mss 0 src vpn
0.0.0.0/0 via 10.8.0.57 dev 32 metric 50 mss 0 src vpn
fails.




As to why that bisects to df4e5357521d is a bit strange.


the proper fix would be to backport the proper handling of Ipv4 addresses.
Comment 11 Thomas Haller 2016-01-04 18:33:03 UTC
Created attachment 318224 [details] [review]
core: fix failure to configure routes with IPv4 peer-addresses

As in the case of OpenVPN, we might add an address like:
  10.8.0.58/32 ptp 10.8.0.57

In this case, kernel will automatically add a device-route like:
  10.8.0.57/32 via 0.0.0.0 dev 32 metric 0 mss 0 src rtprot-kernel scope link pref-src 10.8.0.58

nm_ip4_config_commit() checks all IP addresses to figure out
the present device-routes. Then the routes are synced by NMRouteManager.
Due to a bug, we would not consider the peer-address, but the local-address
and configure a route 10.8.0.58/32, instead of 10.8.0.57/32.

That stays mostly unnoticed, because usually the peer and the local-address are
in the same subnet, so that there is no difference (/32 is an example of the
peer-address being in a different subnet).

It also seems that previously due to a bug fixed by df4e5357521 this
issue didn't surface. Probably because we would not notice the 10.8.0.57/32
right away and thus nm_route_manager_ip4_route_sync() would not wrongly delete
it.

https://bugzilla.gnome.org/show_bug.cgi?id=759892

Also:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=809195
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=809494
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=809526
https://bugs.archlinux.org/task/47535
https://bugzilla.redhat.com/show_bug.cgi?id=1294309
https://mail.gnome.org/archives/networkmanager-list/2015-December/msg00059.html
Comment 12 Dan Williams 2016-01-04 18:50:27 UTC
Looks legit to me.
Comment 13 Beniamino Galvani 2016-01-05 08:45:08 UTC
(In reply to Thomas Haller from comment #11)
> Created attachment 318224 [details] [review] [review]
> core: fix failure to configure routes with IPv4 peer-addresses

Works for me, the default route is now added correctly.
Comment 14 Beniamino Galvani 2016-01-05 09:04:43 UTC
I also tested master branch and it seems not affected.
Comment 16 Thomas Haller 2016-01-06 09:42:34 UTC
*** Bug 759949 has been marked as a duplicate of this bug. ***