After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 763204 - VPN split DNS gets broken after suspend/resume.
VPN split DNS gets broken after suspend/resume.
Status: RESOLVED OBSOLETE
Product: NetworkManager
Classification: Platform
Component: IP and DNS config
unspecified
Other Linux
: Normal normal
: ---
Assigned To: NetworkManager maintainer(s)
NetworkManager maintainer(s)
Depends on:
Blocks:
 
 
Reported: 2016-03-07 06:09 UTC by Nikolay Martynov
Modified: 2020-11-12 14:25 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Proposed patch to fix the problem (2.55 KB, patch)
2016-03-07 06:09 UTC, Nikolay Martynov
none Details | Review
resume log (73.13 KB, text/x-log)
2016-03-09 05:05 UTC, Nikolay Martynov
  Details
Log before 'unmanaged' state (8.16 KB, text/x-log)
2016-03-10 04:17 UTC, Nikolay Martynov
  Details

Description Nikolay Martynov 2016-03-07 06:09:42 UTC
Created attachment 323230 [details] [review]
Proposed patch to fix the problem

I've got a company's VPN server I connect to. When I connect to it I get several DNS entries for split DNS configuration and this works fine.

After suspend/resume cycle I reconnect to VPN and see those split DNS entries added along with entries for VPN's DNS servers that do not have domain attached to it (i.e. not split). Disconnecting and reconnecting to VPN doesn't solve the problem: those non split entries gets stuck until I restart NetworkManager.

I think I've tracked down the problem and will attach the patch.
Comment 1 Beniamino Galvani 2016-03-07 13:31:03 UTC
(In reply to Nikolay Martynov from comment #0)

> This patch addresses the problem by making sure old values in 'priv->ip{4,6}_{vpn,device}_config'
> are removed from 'priv->configs' before new values are applyed.

Hi, thanks for reporting and debugging the problem. The patch looks
correct to me, but I think that it may hide another bug... when the
VPN reconnects, NM should have called vpn_connection_deactivated() ->
nm_dns_manager_remove_ip4_config() on the old connection to clear
@ip4_vpn_config and remove the old configuration from the list.

Instead this didn't happen... can you please paste NM debug logs (or,
if they contain sensitive data, try to understand why the old
connection doesn't clear the DNS configuration)?
Comment 2 Nikolay Martynov 2016-03-07 15:47:40 UTC
Hi.

I'll try to gather this info for you.

I would like to point out that problem existed on Ubuntu 14.04. And back then NM didn't try to reconnect VPN after suspend/resume - i.e. VPN connection just got disconnected on resume, but yet those dns entries poped up somehow. And 14.04 uses NM 0.9.8.8.

Unfortunately I do not have 14.04 running to test my patch against it but I think there that is was the same problem.

Now, to your original statement: I think what happens is that `nm_dns_manager_add_ip4_config` is somehow called twice during connection lifetime. So event though `nm_dns_manager_remove_ip4_config` is being called on deactivation it removes dns settings only for last call to `nm_dns_manager_add_ip4_config`. And I think this is not VPN specific problem, it's just for VPN it happens to matter more (at least in my case).

I'll try to gather logs info to prove this theory.
Comment 3 Nikolay Martynov 2016-03-08 04:27:50 UTC
I think this log shows things would be added twice:

NetworkManager[30808]: <debug> [1457410704.663987] [dns-manager/nm-dns-manager.c:1076] nm_dns_manager_begin_updates(): (vpn_connection_activated): queueing DNS updates (1)
NetworkManager[30808]: <debug> [1457410704.664215] [dns-manager/nm-dns-manager.c:1076] nm_dns_manager_begin_updates(): (update_routing_and_dns): queueing DNS updates (2)
NetworkManager[30808]: <debug> [1457410704.664477] [dns-manager/nm-dns-manager.c:1094] nm_dns_manager_end_updates(): (nm_dns_manager_end_updates): DNS configuration did not change
NetworkManager[30808]: <debug> [1457410704.664712] [dns-manager/nm-dns-manager.c:1098] nm_dns_manager_end_updates(): (update_routing_and_dns): no DNS changes to commit (1)
NetworkManager[30808]: <debug> [1457410704.664950] [dns-manager/nm-dns-manager.c:1094] nm_dns_manager_end_updates(): (nm_dns_manager_end_updates): DNS configuration did not change
NetworkManager[30808]: <debug> [1457410704.665180] [dns-manager/nm-dns-manager.c:1098] nm_dns_manager_end_updates(): (vpn_connection_activated): no DNS changes to commit (0)

Please let me know if you'd like me to do more detailed logging.
Comment 4 Beniamino Galvani 2016-03-08 08:22:15 UTC
(In reply to Nikolay Martynov from comment #3)
> I think this log shows things would be added twice:

This part of logs only shows a single invocation of
vpn_connection_activated(), which calls:

 - nm_dns_manager_begin_updates()
 - nm_dns_manager_add_ip4_config()
 - nm_dns_manager_add_ip6_config()
 - update_routing_and_dns()
 - nm_dns_manager_end_updates()

and so it seems to me that nm_dns_manager_add_ip4_config()
is being called only once.

BTW, which NM version are you using?
Comment 5 Nikolay Martynov 2016-03-09 05:05:50 UTC
Created attachment 323471 [details]
resume log

Please find resume log attached.

Note: this log is taken with patch applied, but I've also added some log statements:

- 'nm_dns_manager_add_ip4_config(): (<dev>): starting' - this is when nm_dns_manager_add_ip4_config function is entered
- 'nm_dns_manager_add_ip4_config(): (<dev>): replacing priv->ip4_vpn_config' - this is when config gets replaced (i.e. non-NULL overwritten) - this is the place when without this patch bug would happen.

Hope this helps. Please let me know if there is any other debugging info you might need.

Oh, and version. Currently I use 1.0.4 (Ubuntu 15.10), but same problem existed as early as Ubuntu 14.04 (0.9.8.8, probably), and maybe earlier.
Comment 6 Beniamino Galvani 2016-03-09 13:03:42 UTC
If I understand correctly, there is a VPN over a bond over
wifi+ethernet? I have some troubles understanding what's going on, in
particular at lines:

nm-openvpn[19237]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper --tun -- tun0 1500 1542 X.X.X.X X.X.X.X restart
<info>  (tun0): device state change: activated -> unmanaged (reason 'removed') [100 10 36]
<debug> [1457498384.750504] [nm-manager.c:296] active_connection_remove(): Assumed connection disconnected. Deleting generated connection 'tun0' (56e8ad46-5861-4902-aadc-92f5e768e265)
nm-openvpn[19237]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper --tun -- tun0 1500 1542 X.X.X.X X.X.X.X init

I don't understand why the 'tun0' is considered a 'assumed generated'
connection, since it's supposed to be a VPN started by NM. Can you
please describe your setup? Also, do you have any chance to test this
with a recent build from git? Thanks!
Comment 7 Nikolay Martynov 2016-03-09 14:23:01 UTC
Hi.

  Yes, I have bond interface over ethernet and wifi, but I think problem exists even if I remove that bonding. Otherwise I do not really have anything special. Would having more logs enabled somehow help?

> I don't understand why the 'tun0' is considered a 'assumed generated'
connection, since it's supposed to be a VPN started by NM.
Can it be somehow that NM forgets that it created it after resume?

  I'm running Ubuntu and they have things patched... Is there any sort documentation on how I can run NM from master on Ubuntu?
Comment 8 Nikolay Martynov 2016-03-10 04:17:46 UTC
Created attachment 323586 [details]
Log before 'unmanaged' state

This is piece of log that hopefully can help diagnosing tun0 going into unmanaged state. It looks like openvpn client is doing something there that may be causing it.
Comment 9 Nikolay Martynov 2016-03-21 01:38:22 UTC
Hi.

I was curious: is there some problem with proposed patch or is there any other debugging information that I can provide?
Comment 10 Beniamino Galvani 2016-03-21 12:57:36 UTC
(In reply to Nikolay Martynov from comment #9)
> Hi.
> 
> I was curious: is there some problem with proposed patch or is there any
> other debugging information that I can provide?

Hi, sorry for the delay. I'm trying to understand the strange behavior wrt the VPN connection; this line is suspicious:

Mar  8 23:39:44 yyyyy-laptop nm-openvpn[19237]: NOTE: Pulled options changed on restart, will need to close and reopen TUN/TAP device.

and probably is what is causing NM to consider the connection as assumed. We should investigate this more after the 1.2 release.

That said, the patch looks ok to me. What do others think about it?
Comment 11 Thomas Haller 2016-05-29 12:01:31 UTC
I think the patch is not correct.


NMPolicy, the user of NMDnsManager, must properly match nm_dns_manager_add_ip4_config() calls with nm_dns_manager_remove_ip4_config().

Thus, a "add" shall only add the new config, it should not "remove" an existing best-config that is replaced by a different one.

Even when setting a new best-config, the previous best-config should still be tracked -- not forgotten. It should only be unset as being the best-config and the new config shall become the best-config.


For example, you have two interfaces, wlan0 is active (and best-config).
Now you activate eth0, which has higher priority and becomes the new best-config. Still, the config from wlan0 is relevant and should not be dropped.



If you experience leaks or wrong behaviors, then it is because NMPolicy forgot to call remove() for a certain config. That would be the bug then.
Comment 12 Lyude 2016-10-12 16:35:46 UTC
Hi! Went to file a bug for this and found this. I'm still able to see this issue with Fedora 24, NetworkManager-vpnc-1:1.2.2-1.fc24.
Since you both seem to be Red Hatters I can give you a bit more useful info though: this problem is actually reproducible using our own VPN (assuming you set it up using vpnc of course).
Comment 13 Beniamino Galvani 2016-10-13 09:27:46 UTC
(In reply to Lyude from comment #12)
> Hi! Went to file a bug for this and found this. I'm still able to see this
> issue with Fedora 24, NetworkManager-vpnc-1:1.2.2-1.fc24.
> Since you both seem to be Red Hatters I can give you a bit more useful info
> though: this problem is actually reproducible using our own VPN (assuming
> you set it up using vpnc of course).

Probably you're experiencing this bug [1], which needs an update of dnsmasq.

Regarding the original problem, we did many changes to the DNS code since this bug was filed and I think the issue should be solved now.

Nikolay, can you please check if the bug is fixed on latest version from git? Thanks!

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1373485
Comment 14 Nikolay Martynov 2016-10-16 02:02:19 UTC
With current version of NM in Ubuntu 16.04 version I think there is no vpn autoreconnect, but after suspend/resume cycle DNS seems to be still split. So I cannot reproduce the problem there anymore, but maybe because autoreconnect just doesn't happen.
Unfortunately I might not be able to reproduce it on git version due to time constraints and large amount of patching ubuntu tends to do to NM. I'm planning to update to 16.10 soon and will retest there and report results.
Comment 15 André Klapper 2020-11-12 14:25:44 UTC
bugzilla.gnome.org is being shut down in favor of a GitLab instance. 
We are closing all old bug reports and feature requests in GNOME Bugzilla which have not seen updates for a long time.

If you still use NetworkManager and if you still see this bug / want this feature in a recent and supported version of NetworkManager, then please feel free to report it at https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/

Thank you for creating this report and we are sorry it could not be implemented (workforce and time is unfortunately limited).