(In reply to Mike Auty from comment #0)
> Systemd recently released a resolvconf implementation that ties into its DNS
> resolver.  Systemd handles the ordering of the interfaces and construction
> of /etc/resolv.conf.  The downside is that it only makes use of DNS
> information for interfaces that exist, and networkmanager always adds its
> DNS information to an interface called NetworkManager. 

This sounds strange to me. Can you please attach NetworkManager logs at TRACE level when a connection gets activated? (Set level=TRACE in the [logging] section of /etc/NetworkManager/NetworkManager.conf and restart the service).

The TRACE log contains the following for dns-mgr:

Jul 09 08:22:24 plasma NetworkManager[2134]: <debug> [1531120944.9155] dns-mgr: (device_ip6_config_changed): DNS configuration changed
Jul 09 08:22:24 plasma NetworkManager[2134]: <debug> [1531120944.9155] dns-mgr: (device_ip6_config_changed): committing DNS changes (0)
Jul 09 08:22:24 plasma NetworkManager[2134]: <debug> [1531120944.9155] dns-mgr: update-dns: updating resolv.conf
Jul 09 08:22:24 plasma NetworkManager[2134]: <trace> [1531120944.9155] dns-mgr: config:      100 best    v4 wlp2s0           : 192.168.188.254
Jul 09 08:22:24 plasma NetworkManager[2134]: <trace> [1531120944.9155] dns-mgr: config:      100 default v6 wlp2s0           : fe80::9e97:26ff:fe7a:85f6
Jul 09 08:22:24 plasma NetworkManager[2134]: <debug> [1531120944.9155] dns-mgr: update-dns: updating plugin systemd-resolved
Jul 09 08:22:24 plasma NetworkManager[2134]: <info>  [1531120944.9157] dns-mgr: Writing DNS information to /sbin/resolvconf
Jul 09 08:22:24 plasma NetworkManager[2134]: Unknown interface 'NetworkManager': No such device
Jul 09 08:22:24 plasma NetworkManager[2134]: <warn>  [1531120944.9227] dns-mgr: resolvconf failed with status 256
Jul 09 08:22:24 plasma NetworkManager[2134]: <trace> [1531120944.9228] dns-mgr: update-resolv-conf: write internal file /run/NetworkManager/resolv.conf succeeded
Jul 09 08:22:24 plasma NetworkManager[2134]: <trace> [1531120944.9229] dns-mgr: current configuration: [{'nameservers': <['192.168.188.254']>, 'domains': <['lan']>, 'interface': <'wlp2s0'>, 'priority': <100>, '>
Jul 09 08:22:24 plasma NetworkManager[2134]: <warn>  [1531120944.9229] dns-mgr: could not commit DNS changes: resolvconf failed with status 256

This makes perfect sense, because the value passed to resolvconf -a is hardcoded as shown at https://github.com/NetworkManager/NetworkManager/blob/master/src/dns/nm-dns-manager.c#L710

It cannot ever run anything other than "resolvconf -a NetworkManager" meaning the information is always sent about the non-existant "NetworkManager" network interface.  This works with other resolvconf systems because they construct a directory structure regardless of actual interfaces, and then build the resolv.conf file based on that.  Systemd-239 doesn't do that apparently.  5:(

It should be noted that to appropriately use systemd's DNS resolver, which now supports resolvconf, the /etc/resolv.conf is not a symlink to networkmanager's one, but is a symlink to /run/systemd/resolve/resolv.conf as managed by systemd-239 instead.  The TRACE log shows that systemd's resolvconf is rejecting the interface information because NetworkManager hasn't provided the actual interface that the information is associated with when it populates it.

As I mentioned, I haven't been able to find a clear specification for resolvconf, just the code for openresolv which happens to support non-existent interfaces.  As such, it's hard to say whether systemd or NetworkManager is the one not adhereing to the specification.

I understand the rationale for the single DNS value (being able to have total control over the DNS entries and ordering), but it breaks with systemd-239's DNS resolvconf code, which also wants total control over DNS.  It doesn't matter to me which one has control, but I suspect systemd will be far less flexible than NetworkManager might be (given their stance on /proc/sys/net/ipv4/ip_forward, which they started ignoring since they felt they should control all aspects of networking, ancient precedent or not).  I don't really care which one wins in having total control, but I'd like it if they could both interoperate on the same system together...  5:S

In NetworkManager, you can configure either main.dns=resolvconf or main.dns=systemd-resolved.

main.dns=resolvconf works the way it does for a long time, and it's unclear that NetworkManager would do anything wrong by setting `-a NetworkManager`.

It seems, the bug is in the integration of resolvconf when used in combination with systemd-resolved.



You say,
> Systemd recently released a resolvconf implementation that ties into its DNS resolver.  

and I assume you are referring to https://github.com/systemd/systemd/pull/8296


Note the comment

    Of course, resolvconf(8) is a tool with multiple backends, in our
    implementation systemd-resolved is the only backend.

in https://github.com/systemd/systemd/pull/8296/commits/088c136384edb1d121ed40a6d3ecf4b7abade169


If you use the systemd-resolved's compat implementation, which only supports systemd-resolved as backend, then it seems there is very little benefit in confguring NetworkManager with

  [main]
  dns=resolvconf

and you should instead tell NetworkManager, to directly use dns=systemd-resolved.



Regardless, the integration here should be improved. But it looks like missing functionality of systemd's resolveconf.

Thanks for the quick response.  5:)  I think dns=resolvconf is the default, but as long as there's a path for people to use systemd-resolved (and this bug helps them find the need to configure NetworkManager differently) then it's all good.  Happy to close this if you'd like?

if you don't configure "main.dns" in NetworkManager.conf at all (which should be the default on your system), then the usage of systemd-resolved should be automatically detected because /etc/resolv.conf is a symlink to one of the well-known paths (like /usr/lib/systemd/resolv.conf).

Optimally, neither your distribution nor the admin needs to configure anything here...

When you start NetworkManager with level=TRACE, it logs the configuration at the beginning... (search for "CONFIG:"). What does it say about the [main] settings?

Hmmm, interesting so the configuration is always as follows (just the logging entry):

Jul 09 21:31:45 plasma systemd[1]: Starting Network Manager...
Jul 09 21:31:45 plasma NetworkManager[6552]: <debug> [1531168305.9867] monotonic timestamp started counting 1.553663573 seconds ago with an offset of 215.0 seconds to CLOCK_BOOTTIME (local time is 2018-07-09 21>
Jul 09 21:31:45 plasma NetworkManager[6552]: <info>  [1531168305.9867] NetworkManager (version 1.10.10) is starting... (after a restart)
Jul 09 21:31:45 plasma NetworkManager[6552]: <info>  [1531168305.9867] Read config: /etc/NetworkManager/NetworkManager.conf (etc: logging.conf)
Jul 09 21:31:45 plasma NetworkManager[6552]: <debug> [1531168305.9868] CONFIG: config-data[0x564a61054820]: 1 groups
Jul 09 21:31:45 plasma NetworkManager[6552]: <debug> [1531168305.9868] CONFIG: 
Jul 09 21:31:45 plasma NetworkManager[6552]: <debug> [1531168305.9868] CONFIG: [logging]
Jul 09 21:31:45 plasma NetworkManager[6552]: <debug> [1531168305.9868] CONFIG:   level=TRACE
Jul 09 21:31:45 plasma NetworkManager[6552]: <debug> [1531168305.9868] config: state: successfully read state file "/var/lib/NetworkManager/NetworkManager.state"

So the config is definitely empty.
After a restart of the NetworkManger service, it looks like it did a "soft-restart" because I got the following

ul 09 21:31:46 plasma NetworkManager[6552]: <trace> [1531168306.1249] dns-mgr[0x564a6109e950]: creating...
Jul 09 21:31:46 plasma NetworkManager[6552]: <info>  [1531168306.1251] dns-mgr[0x564a6109e950]: init: dns=systemd-resolved, rc-manager=resolvconf, plugin=systemd-resolved
Jul 09 21:31:46 plasma NetworkManager[6552]: <trace> [1531168306.1252] dns-mgr[0x564a6109e950]: current configuration: @aa{sv} []

And that operated as expected, however after a full reboot I got the following lines instead:

Jul 09 21:42:13 plasma NetworkManager[3634]: <trace> [1531168933.6238] dns-mgr[0x5565a619c970]: creating...
Jul 09 21:42:13 plasma NetworkManager[3634]: <info>  [1531168933.6239] dns-mgr[0x5565a619c970]: init: dns=default, rc-manager=resolvconf
Jul 09 21:42:13 plasma NetworkManager[3634]: <trace> [1531168933.6239] dns-mgr[0x5565a619c970]: current configuration: @aa{sv} []

This apparently doesn't use the systemd-resolved mechanism, and is why I was running into problems.  My symlink for resolv.conf is as follows:

plasma ~ # ls -la /etc/resolv.conf                                      
lrwxrwxrwx 1 root root 36 Jun 26 14:34 /etc/resolv.conf -> /var/run/systemd/resolve/resolv.conf

I guess this means the symlink detection isn't spotting my link as valid for systemd-resolved?  I'm happy to provide more detailed trace logs of my NetworkManager startup if they'll help, just lemme know if you need them...

Ah, my bad.  Checking the code, it seems that it's because I made the symlink myself and used /var/run instead of just /run (/var/run being a symlink to /run), and it does a string comparison to check.  Sorry for all the trouble!  5:S  

If there's anything that could be fixed, it's for the symlink check to follow symlinks/check the canonical path, but I would expect distributions to handle the symlink properly (rather than people manually setting it) so I'm fine for this to be closed.  Sorry again!  5:S

(In reply to Mike Auty from comment #7)

> If there's anything that could be fixed, it's for the symlink check to
> follow symlinks/check the canonical path, 

I think that already happens, for the most part.

See [1] and also commit [2].

First, it tries plain readlink (but that doesn't match in your case, because it expects /run not /var/run).

Then it tries following the link, but might not match during boot, because there is a race with resolved creating that file.


One possible solution would be to also check for /var/run, but it seems to me, the user should either:
 - explicitly configure main.dns=systemd-resolved in NetworkManager.conf
 - or preferably, set /etc/resolv.conf to one of the understood paths -- that 
   implies not using /var/run.
   Note that resolved itself assigns special meaning if you have 
   /etc/resolv.conf symlink to certain paths. See `man systemd-resolved`. But 
   the systemd manual page also does not mention the "/var/run" variants as 
   valid.


[1] https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/src/dns/nm-dns-manager.c?id=c866df799735a46689cb53e06c42b09a904062b6#n1845

[2] https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=e09503dcc43039905018b26304bfe94287211aa6


I think there is nothing left to do. Closing. Please reopen, if something is missing. Thanks!!

(In reply to Thomas Haller from comment #8)

> See [1] and also commit [2].

> [2]
> https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/
> ?id=e09503dcc43039905018b26304bfe94287211aa6

should have been

[2] https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=fae84b16f850c8a9d6a5f5c4bee1c162debb8dec