What you describe is how it is already supposed to work.

I think the WantedBy=multi-user.target just means "*if* NM-w-o gets enabled, then multi-user.target requires it", but it does not imply on its own that NM-w-o should be enabled.

This is how NetworkManager-wait-online.service for the current versions of systemd could IMO look like:

[Unit]
Description=Network Manager Wait Online
Requisite=NetworkManager.service
After=NetworkManager.service
Before=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/nm-online -s -q --timeout=30

[Install]
WantedBy=network-online.target

It starts after NetworkManager.service and delays network-online.target and can be enabled for network-online.target. The goal is to enable it by default without affecting case #1 where there are no services requesting network-online.target and without affecting any other services not dependant on network-online.target in case #2.

(In reply to comment #1)
> What you describe is how it is already supposed to work.

What I describe means you could enable NetworkManager-wait-online.service without affecting services that don't depend on network-online.target as well as the overall boot time in case #1 where no such services are enabled.

That would mean you could enable NetworkManager-wait-online.service on all installations without the well known negative effects (provided that other services are configured correctly).

> I think the WantedBy=multi-user.target just means "*if* NM-w-o gets enabled,
> then multi-user.target requires it", but it does not imply on its own that
> NM-w-o should be enabled.

WantedBy=multi-user.target actually means NetworkManager-wait-online.service shouldn't be enabled by default to avoid unnecessary boot time delays.

From systemd documentation:

    Units that strictly require a configured network connection should pull in network-online.target (via a Wants= type dependency) and order themselves after it. This target unit is intended to pull in a service that delays further execution until the network is sufficiently set up. What precisely this requires is left to the implementation of the network managing service.

    Note the distinction between this unit and network.target. This unit is an active unit (i.e. pulled in by the consumer rather than the provider of this functionality) and pulls in a service which possibly adds substantial delays to further execution. In contrast, network.target is a passive unit (i.e. pulled in by the provider of the functionality, rather than the consumer) that usually does not delay execution much. Usually, network.target is part of the boot of most systems, while network-online.target is not, except when at least one unit requires it. Also see Running Services After the Network is up for more information.

    All mount units for remote network file systems automatically pull in this unit, and order themselves after it. Note that networking daemons that simply provide functionality to other hosts generally do not need to pull this in.

Source: http://www.freedesktop.org/software/systemd/man/systemd.special.html#network-online.target

I discussed the issue with systemd developers (and they discussed it among themselves) and they brought up an issue in systemd's support for initscripts. They support the LSB headers where they translate the $network dependency to systemd's network.target but some services rather expect to have a fully working network in which case $network should rather translate to network-online.target.

So far they added this to their TODO list:

http://cgit.freedesktop.org/systemd/systemd/commit/?id=d20850cbf4545715340580c179cf316005d53905

It doesn't make any change for the default configuration where NetworkManager-wait-online.service is currently off by default. The only affected situation is when the administrator already called *systemctl enable* for this service.

There are a couple of possible solution for this issue, which are up to the systemd project. One is to just translate $network to network-online.target instead of network.target. The other is to add a separate $network-online target and let the distributions change their initscripts.

Note that this issue also doesn't affect distributions that disable systemd's initscript support entirely. Gentoo is an example.

Right now the NM-w-o systemd unit is doing what Lennart told us to do with it, back when network-online was created for systemd 201.  If he's got suggestions for how to change it so that we can enable it by default, I'm all ears.

Part of his suggestion was the "Before: network.target" that is current in the NM-w-o unit file.  So if that bit is wrong, I'd like to get confirmation from systemd people what really should be there.

If there is a way to achieve (1) and (2) from your first comment, that would be fantastic because then we could enable it by default, or at least not have to babysit whether NM-w-o is enabled by default or not.

Right now, I think no matter if anything cares about NM-w-o, your startup blocks if you have enabled it.  Like your (1) I really want a way to enable it by default, but only *run* it if something requires it.

Some references:

1) bug about modifying NM-wait-online.service to work with network-online.target:

https://bugzilla.redhat.com/show_bug.cgi?id=787314#c37

(and all the other comments).  This comment is the original source of the "Before=network.target network-online.target" bits.  Would be good to get clarification from Lennart or Kay or somebody if the network.target bit there is wrong.

2) original systemd mailing list thread about adding Nm-wait-online in the first place:

http://lists.freedesktop.org/archives/systemd-devel/2011-March/001692.html

(In reply to comment #6)
> Some references:

Thanks!

> 1) bug about modifying NM-wait-online.service to work with
> network-online.target:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=787314#c37

Reading Lennart's suggestions, that looks a bit more sophisticated, as in theory it should provide both #1 and #2 and on top of it (3) still allow NM-wait-online.service to block network.target when enabled for multi-user.target.

So, let's check it (on Gentoo with a clean install from git master):

$ equery f networkmanager | grep NetworkManager-wait-online
/etc/init.d/NetworkManager-wait-online
/usr/lib/systemd/system/NetworkManager-wait-online.service

And it looks like even the .service files are installed by the following command in the ebuild.

for i in data/*.service; do
    systemd_dounit $i || die
done

And the current Fedora rawhide package:

# rpm -ql NetworkManager | grep NetworkManager-wait-online
/usr/lib/systemd/system/NetworkManager-wait-online.service
/usr/lib/systemd/system/network-online.target.wants/NetworkManager-wait-online.service

So the first problem upstream is that we don't seem to be installing the necessary files. That should be fixed now that systemd is becoming the most common init and service manager on distributions where you can expect NetworkManager.

I think we should fix that first and then (1) verify it works as expected and (2) verify that other services in a distribution comply.

I believe the documentation implies that a service that *needs* to (try to) wait for network connectivity should include:

Wants=network-online.target
After=network-online.target

For backwards compatibility this could be:

Wants=network.target network-online.target
After=network.target network-online.target

This is how I modified the Fedora's aiccu package.

(In reply to comment #5)
> Right now the NM-w-o systemd unit is doing what Lennart told us to do with it,
> back when network-online was created for systemd 201.  If he's got suggestions
> for how to change it so that we can enable it by default, I'm all ears.

With the above in mind, it would be nice to propagate this from Fedora to upstream and handle it in the build system for 0.9.10. Any suggestions?

> Part of his suggestion was the "Before: network.target" that is current in the
> NM-w-o unit file.

AFAIK Before=network.target is for backwards compatibility and apart from covering bugs it doesn't do any harm. We might want to remove it *later*.

> So if that bit is wrong, I'd like to get confirmation from
> systemd people what really should be there.
> 
> If there is a way to achieve (1) and (2) from your first comment, that would be
> fantastic because then we could enable it by default, or at least not have to
> babysit whether NM-w-o is enabled by default or not.

AFAIK Lennart's way was intended to achieve exactly that, we just need to upstream it and verify that it works properly. Still there's an issue with LSB initscript support that $network is translated as network.target, not network-online.target and there's no way to trigger network-online.target but that's a separate issue in systemd.

Just temporarily fixed it in my Gentoo ebuild:

https://github.com/okias/ixit/commit/339410bfb0f51bc398fafae762fed6294d54a22e

I performed a couple of tests with the following service file:

# cat /etc/systemd/system/testonline.service 
[Unit]
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/bin/true

The commands used to test it:

# systemctl daemon-reload
# systemctl stop network-online.target
# systemctl start  testonline.service

It seems to work as expected:

# journalctl -f
...
May 06 14:26:44 mole systemd[1]: Starting Network Manager Wait Online...
May 06 14:26:44 mole systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
May 06 14:26:44 mole systemd[1]: Started Network Manager Wait Online.
May 06 14:26:44 mole systemd[1]: Starting Network is Online.
May 06 14:26:44 mole systemd[1]: Reached target Network is Online.
May 06 14:26:44 mole systemd[1]: Starting testonline.service...
May 06 14:26:44 mole systemd[1]: Started testonline.service.

(The devices.list issue seems to be unrelated.)

Now it looks like apart from upstreaming the systemd configuration, the only issue is to change .service files for affected services to actually use network-online.target.

(In reply to comment #8)
> [Unit]
> Wants=network-online.target
> After=network-online.target

For the sake of completeness, the following (for compatibility) works just as well:

Wants=network.target network-online.target
After=network.target network-online.target

Just a quick note that systemd-networkd is adding support for network-online upstream:

http://cgit.freedesktop.org/systemd/systemd/commit/units/systemd-etworkd-wait-online.service.in?id=3a67e927e3be7efb8edf314a31aa4f8f5cba4f53

Hi guys,

Thanks Pavel for looking into this issue.

I have now pushed a change to systemd [0], so there is no longer a problem with the LSB compatibility. If a legacy script requires $network, it will now order itself after network-online.target, and also pull in that target (which was the missing part).

That should mean that you can change your service file as in comment #2, and enable it by default.

As Pavel noted I added a wait-online service to networkd as well, and the aim here is that it will block if networks are managed by networkd, but get out of the way if the networks are managed by NetworkManager (or something else). I haven't checked if your wait-online service works similarly, but it would be cool I think, as then both could be enabled unconditionally without getting in each other's way.

Cheers,

Tom

[0]: <http://cgit.freedesktop.org/systemd/systemd/commit/?id=0404c609f399b2092a3de52eef9d75b0dc12e94c>

(In reply to comment #11)
> As Pavel noted I added a wait-online service to networkd as well, and the aim
> here is that it will block if networks are managed by networkd, but get out of
> the way if the networks are managed by NetworkManager (or something else). I
> haven't checked if your wait-online service works similarly, but it would be
> cool I think, as then both could be enabled unconditionally without getting in
> each other's way.

Yeah, NetworkManager-wait-online.service doesn't actually wait for the network to be up, it just waits for NetworkManager to declare that it is done with its startup-time network operations. If you have no connections configured in NM, NM will declare that startup is complete right away, and NetworkManager-wait-online will exit immediately.

(In reply to comment #12)
> (In reply to comment #11)
> > As Pavel noted I added a wait-online service to networkd as well, and the aim
> > here is that it will block if networks are managed by networkd, but get out of
> > the way if the networks are managed by NetworkManager (or something else). I
> > haven't checked if your wait-online service works similarly, but it would be
> > cool I think, as then both could be enabled unconditionally without getting in
> > each other's way.
> 
> Yeah, NetworkManager-wait-online.service doesn't actually wait for the network
> to be up, it just waits for NetworkManager to declare that it is done with its
> startup-time network operations. If you have no connections configured in NM,
> NM will declare that startup is complete right away, and
> NetworkManager-wait-online will exit immediately.

Great! Thanks for confirming.

Created attachment 277062 [details] [review]
patch to upstream the symbolic link used in Fedora

This alternative patch ports the Fedora way (originally suggested by lennart) to the upstream autotools based build system. This is enough to give all distributions what Fedora already uses.

Note that systemd-networkd currently uses the approach I suggested originally in this bug report. I believe both tools should switch to the same solution in the long term, whichever solution it is.

(In reply to comment #13)
> (In reply to comment #12)
> > If you have no connections configured in NM,
> > NM will declare that startup is complete right away, and
> > NetworkManager-wait-online will exit immediately.
> 
> Great! Thanks for confirming.

When NetworkManager is not running at all, NetworkManager-wait-online.target also fails immediately.

The way NetworkManager and NetworkManager-wait-online is not perfect, though, as you cannot choose which connections should be waited for (see bug #700651). That causes the following issues:

1) network-online.target often doesn't wait for connections that you expect to be up.

2) network-online.target may wait for a connection that you do need to be up at all.

The first issue can be worked around by using another tool (e.g. systemd-networkd) to bring up all requested connections *before* network-online.target. Legacy /etc/init.d/network sort of works out of the box as well, as it only finishes when the network is configured. But that's another story.

(In reply to comment #14)
> Created an attachment (id=277062) [details] [review]
> patch to upstream the symbolic link used in Fedora
> 
> This alternative patch ports the Fedora way (originally suggested by lennart)
> to the upstream autotools based build system. This is enough to give all
> distributions what Fedora already uses.
> 
> Note that systemd-networkd currently uses the approach I suggested originally
> in this bug report. I believe both tools should switch to the same solution in
> the long term, whichever solution it is.

I think the lines should move up inside
  if HAVE_SYSTEMD
  ...
  endif


and I would add quotes around the path names and a trailing slash after "network-online.target.wants/"



and maybe:

if [ "$(DESTDIR)" = "" ]; then
  ln -s ../NetworkManager-wait-online.service \
     "$(DESTDIR)$(systemdsystemunitdir)"/network-online.target.wants/
else
  ln -s "$(systemdsystemunitdir)"/NetworkManager-wait-online.service \
     "$(DESTDIR)$(systemdsystemunitdir)"/network-online.target.wants/
fi



The approach looks right

(In reply to comment #16)
> and maybe:
> 
> if [ "$(DESTDIR)" = "" ]; then
>   ln -s ../NetworkManager-wait-online.service \
>      "$(DESTDIR)$(systemdsystemunitdir)"/network-online.target.wants/
> else
>   ln -s "$(systemdsystemunitdir)"/NetworkManager-wait-online.service \
>      "$(DESTDIR)$(systemdsystemunitdir)"/network-online.target.wants/
> fi

This is incorrect. The DESTDIR variable is used to install into an alternate root filesystem so the files can become part of a package. It shouldn't affect the data nor link targets.

(In reply to comment #15)
> (In reply to comment #13)
> > (In reply to comment #12)
> > > If you have no connections configured in NM,
> > > NM will declare that startup is complete right away, and
> > > NetworkManager-wait-online will exit immediately.
> > 
> > Great! Thanks for confirming.
> 
> When NetworkManager is not running at all, NetworkManager-wait-online.target
> also fails immediately.
> 
> The way NetworkManager and NetworkManager-wait-online is not perfect, though,
> as you cannot choose which connections should be waited for (see bug #700651).
> That causes the following issues:
> 
> 1) network-online.target often doesn't wait for connections that you expect to
> be up.
> 
> 2) network-online.target may wait for a connection that you do need to be up at
> all.
> 
> The first issue can be worked around by using another tool (e.g.
> systemd-networkd) to bring up all requested connections *before*
> network-online.target. Legacy /etc/init.d/network sort of works out of the box
> as well, as it only finishes when the network is configured. But that's another
> story.

Which connections would you expect to be up already, that are not waited for when using NM-w-o?

I believe we still disagree on ONBOOT-type stuff like in bug #700651, but that doesn't have anything to do with #1 above.  Bug #700651 *does* have to do with #2 though.  But lets take this one step at a time...

(In reply to comment #18)
> (In reply to comment #15)
> > 2) network-online.target may wait for a connection that you do not need to be up at
> > all.

Note that network-online.target has been deprecated as long as it has existed, as discussed in http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/. Assuming that the network will be fully configured at boot time and then will never change is very 1990s; modern software should just cope with dynamic networking changes, in which case it doesn't matter if it's started before or after the network comes up.

(In reply to comment #18)
> Which connections would you expect to be up already, that are not waited for
> when using NM-w-o?

That is described in closed bug #700651.

> But lets take this one step at a time...

Sure. Just wanted to keep Tom informed about related information.

(In reply to comment #19)
> Note that network-online.target has been deprecated as long as it has existed,
> as discussed in http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/.

Not a single occurance of 'network-online.target' there. Refer to the official systemd documentation instead and to links provided by Tom Gundersen that prove 'network-online.target' is actively supported by systemd and systemd-networkd.

> modern software should just cope with dynamic networking changes

In this bugzilla ticket we are specifically discussing support for software that is not 'modern' by that definition. After all it's the only reason to keep NetworkManager-wait-online.service around.

(In reply to comment #20)
> > Note that network-online.target has been deprecated as long as it has existed,
> > as discussed in http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/.
> 
> Not a single occurance of 'network-online.target' there. Refer to the official
> systemd documentation instead and to links provided by Tom Gundersen that prove
> 'network-online.target' is actively supported by systemd and systemd-networkd.

OK, "deprecated" was the wrong word. It's fully supported, but it's only needed because of other software that is considered to be broken.

(Yes, it never mentions network-online.target, but it tells people they shouldn't depend on network.target, which obviously implies not depending on network-online.target as well.)

> > modern software should just cope with dynamic networking changes
> 
> In this bugzilla ticket we are specifically discussing support for software
> that is not 'modern' by that definition.

That comment was in the context of your objection that "network-online.target may wait for a connection that you do not need to be up".

Yes, if service A depends on connection X, but not connection Y, then obviously you don't want it to have to wait for connection Y to come up. But adding an ONBOOT-vs-autoactivate distinction doesn't solve that in the general case, because eventually you'll install service B, which *does* require connection Y to be started at boot, and then service A would get slowed down again.

The only way to fix the problem in the general case is for service A to not depend on network-online.target. And given that, it doesn't make sense to add lots of additional special bonus features to NetworkManager-wait-online to solve particular edge cases.

(In reply to comment #21)
> But adding an ONBOOT-vs-autoactivate distinction doesn't solve that in the general case

I started the related bug #700651 for that purpose. This bug report is about providing the already working functionality present on Fedora upstream.

(In reply to comment #15)
> The way NetworkManager and NetworkManager-wait-online is not perfect, though,
> as you cannot choose which connections should be waited for (see bug #700651).

Tom,

I realized that while feature request described in bug #700651 has been marked as CLOSED/WONTFIX, it has been actually implemented (without the auto=yes onboot=no use case described in the implementation proposal). I reopened bug #700651 to handle an issue described in a comment there.

Not counting that issue, the network-online.target would work as expected with NetworkManager is a patch similar to the one attached here would be applied.

Created attachment 277584 [details] [review]
patch to upstream the symbolic link used in Fedora

commit afd30f5f3bdb1d1ba75e4526afa90cec0e399acf
Author: Pavel Šimerda <psimerda@redhat.com>
Date:   Fri May 23 14:56:32 2014 +0200

    implement systemd network-online.target properly (bgo #728965)
    
    Make network-online.target depend on NetworkManager-wait-online.service
    just as is done in Fedora. This makes network-online.target work with
    NetworkManager as described in systemd documentation.
    
    An alternative way would be to use a combination of setting
    Install.WantedBy to network-online.target and enabling the service by
    default. This alternative approach is currently used by
    systemd-networkd.
    
    https://bugzilla.gnome.org/show_bug.cgi?id=728965
    
    Acked-By: Dan Williams <dcbw@redhat.com>