I cannot reproduce this on Fedora 23.

I put
                                                   
proc            /proc           proc    defaults,hidepid=2        0       0 

to /etc/fstab and rebooted.


afterwards, I didn't see any issues (especially not with nmcli).


Can you provide more information?

Fedora 23 has version 1.0.10, so now we know the regression is between this version and mine (1.1.92).

I don't know which information to provide. You already have the version of N-M, the version of all the dependencies (in the Debian report), as well as my kernel version (idem). So, what kind of information could help you track the problem?

I was using newest master (1.1.92+).

Probably the D-Bus request is rejected by NetworkManager daemon during polkit authentication. Could you verify whether it works if you disable policy-kit by setting /etc/NetworkManager/NetworkManager.conf to have:

[main]
auth-polkit=false

(and restart NetworkManager).

Thanks for your help.

Unfortunately disabling polkit does not change anything. NM and polkitd both run as root, so they don't have any restrictions. This most probably happen on the client side.

Between the two versions OK/NOK, I see NM switched to GDbus, so this is probably related to the change in behavior.

As for dbus in general with hidepid=2, I've got no problems with pulseaudio (pavucontrol) or fcitx for example. And as for communication with system components, upowerd (upower -m) or udisk2 (udisksctl info) also work perfectly. But hey may not use GDbus.

Strange, that disabling polkit didn't help.

The only part in the deamon that reads /proc/$PID is nm_utils_get_start_time_for_pid():
https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/src/nm-core-utils.c?id=9498ea507eb1d5042736c4351337e91b9c13bdf6#n448

It's used to identify processes that we want to kill. So, that might be problematic.
Another use is to send the start-time as "unix-process" subject to polkit.
https://www.freedesktop.org/software/polkit/docs/master/eggdbus-interface-org.freedesktop.PolicyKit1.Authority.html#eggdbus-struct-Subject
This sounds like hidepic could break authentication via polkit.


Anyway, if you say disabling polkid didn't help, it is either in GDBus or in the client.

Can you run `strace nmcli g`, maybe it shows something interesting


Thanks.

Even with hidepid=1 you are not allowed to look into /proc/$PID directories. Nevertheless you can stat() them and get the uid/gid of the process, but… not the start time unfortunately.

It is also possible to have a special GID excluded from this hiding (for identd for example). Nevertheless, on Debian as it is, polkitd and NM are running as root and do not have such restrictions anyway.

I'm attaching two "nmcli g" straces, one working, the other not.

Created attachment 325406 [details]
strace of a working "nmcli g"

Created attachment 325407 [details]
strace of a not-working "nmcli g"

nmcli is working for me on a Debian VM where /proc is mounted with hidepid=2:

# cat /etc/debian_version
stretch/sid

# mount | grep ^proc
proc on /proc type proc (rw,relatime,hidepid=2)

# nmcli --version
nmcli tool, version 1.1.93

# nmcli c
NAME                UUID                                  TYPE            DEVICE
Wired connection 1  1c93eeb7-da0b-4f7a-8b7f-b3cafd6593b0  802-3-ethernet  eth0

I tried to analyze the strace logs but couldn't find anything. Do you see anything useful in the journal?

You're stracing the wrong process. From an strace of the NetworkManager daemon process while trying to connect with nmcli:

open("/proc/14095/stat", O_RDONLY)      = -1 ENOENT (No such file or directory)

The key being that 14095 is the PID of nmcli. The auth process for NetworkManager clients tries to get the start time of the client process [1], and it attempts to do so by accessing /proc/PID/stat [2]. The default systemd service for NetworkManager drops most capabilities [3], including the ability to access files in /proc for processes outside of its own process group.

Solution for systemd systems is to add SYS_CAP_PTRACE to CapabilityBoundingSet in NetworkManager.service. Non-systemd systems shouldn't have this issue (as NetworkManager won't be running with bound capabilities) and don't need any fix.


[1] https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/src/nm-auth-subject.c#n360
[2] https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/src/nm-core-utils.c#n457
[3] https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/data/NetworkManager.service.in#n15

(In reply to Patrick Burroughs (Celti) from comment #10)
> You're stracing the wrong process. From an strace of the NetworkManager
> daemon process while trying to connect with nmcli:
> 
> open("/proc/14095/stat", O_RDONLY)      = -1 ENOENT (No such file or
> directory)
> 
> The key being that 14095 is the PID of nmcli. The auth process for
> NetworkManager clients tries to get the start time of the client process
> [1], and it attempts to do so by accessing /proc/PID/stat [2]. The default
> systemd service for NetworkManager drops most capabilities [3], including
> the ability to access files in /proc for processes outside of its own
> process group.
> 
> Solution for systemd systems is to add SYS_CAP_PTRACE to
> CapabilityBoundingSet in NetworkManager.service. Non-systemd systems
> shouldn't have this issue (as NetworkManager won't be running with bound
> capabilities) and don't need any fix.
> 
> 
> [1]
> https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/src/nm-auth-
> subject.c#n360
> [2]
> https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/src/nm-core-
> utils.c#n457
> [3]
> https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/data/
> NetworkManager.service.in#n15

Thanks Patrick. Good catch. I was on that track, but I missed that even when disabling polkit-authentication, we would still try to create the NMAuthSubject object and fail.

Strange enough, I could not reproduce this on Fedora 23, although starting NM via systemd and restricting capabilities.

I don't see an alternative to extending the capabilities with SYS_CAP_PTRACE, do you? I assume you tested that and it works for you?

Yep, tested and working on Arch Linux with networkmanager 1.1.94-1 and systemd 229-3.

The only alternative solution I'm aware of is to mount procfs with the gid option and add that group to SupplementaryGroups in the service unit, but AFAIK Arch is the only distro to have a standardised group/gid for that (proc:26).

As for reproducibility on F23, I suspect that with the way systemd mounts /proc early in boot, simply adding hidepid=2 to fstab may not work. Try remounting /proc with the option after you've booted and seeing NetworkManager breaks then.

Created attachment 326543 [details] [review]
auth: allow failures to read process start_time from /proc/$PID/stat

When mounting /proc with hide_pid, we might fail to read the
start-time of the process from /proc/$PID/stat. In this case,
we should just accept a start-time of zero.

On the other side of authentication, polkit will accept a zero
start-time.

nm_utils_get_start_time_for_pid() has other uses in NetworkManager,
mostly when killing a process from a PIDFILE or during
nm_utils_kill_process_sync(). In both cases, this will only succeed
if we try to kill a process that runs a root.
For processes started by the current instance, we don't care about the
PIDFILE and use nm_utils_kill_child_?sync() -- so there is no problem
there.

I can only partly reproduce the problem. Thus, it is not clear that attachment 326543 [details] [review] is a right fix.


When I mount with hide_pid, polkit will always reject the authentication request -- regardless of whether NM reads the start-time or not.

When I start NM with auth-polkit=no, the authentication is always bypassed and succeeds (contrary to comment 4).


So, it's not clear how this patch would help.

(In reply to Thomas Haller from comment #15)
> 
> So, it's not clear how this patch would help.

(but if somebody affected could try it, it would be great. Thanks).

Patch tested, works flawlessly. No need for CAP_SYS_PTRACE or adding to the hidepid group.

(In reply to Patrick Burroughs (Celti) from comment #17)
> Patch tested, works flawlessly. No need for CAP_SYS_PTRACE or adding to the
> hidepid group.

Thank you for testing.


Applied a slightly modified version of the patch.

master:
https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=3d505b3f87c9cb9bfdc9b9a1fc67f57330701d03

nm-1-2:
https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=4f06ae603e268f237d439afe3f3e7e662a0c2727

I can confirm this patch fix this issue.

Thanks you very much :-).