GNOME Bugzilla – Bug 738694
New [connectivity] config section does not respect system proxies
Last modified: 2020-11-12 14:31:26 UTC
I recently upgraded to Fedora 21 and got NetworkManager-0.9.10.0-5.git20140704.fc21.x86_64. After the upgrade I began to get word from our enterprise support personnel that I was appearing at the top of their lists in their blocked traffic reports. I finally tracked down the source of the errant traffic, the [connectivity] section defined in /etc/NetworkManager/conf.d/20-connectivity-fedora.conf, which contains: [connectivity] uri=http://fedoraproject.org/static/hotspot.txt response=OK interval=300 In my Network settings in GNOME I have "Manual" proxies for HTTP and HTTPS defined under "Network proxy". NetworkManager is not respecting these proxies when it attempts to verify connectivity in accordance with the [connectivity] section. This also resulted in my network icon in the GNOME system status menu showing a boxed question mark "[?]" instead of an ethernet icon. Now with interval set to 0, instead of "[?]", I get what might be an RJ-45 jack icon, but it mostly looks like a barely visible box with a white ellipsis ("...") inside it. It would seem to me that NetworkManager's [connectivity] feature should respect the system-wide proxy settings of the logged in user. Thanks!
The connectivity checking checks whether you are connected to the Internet, to a Hotspot, or to a local network only. Some UI show different icons depending on your connectivity status. In your company network you can apparently not access the internet, only HTTP via a proxy. The connectivity check fails correctly -- you have no internet there. Using the HTTP proxy would test the wrong thing. If you dislike connectivity checking, just disable it in the configuration. See `man NetworkManager.config` Actually, the file /etc/NetworkManager/conf.d/20-connectivity-fedora.conf comes from the packaged "NetworkManager-config-connectivity-fedora" package. This package only installs the mentioned config file. Remove this package, if you don't want it.
(In reply to comment #1) > In your company network you can apparently not access the internet, only HTTP > via a proxy. The connectivity check fails correctly -- you have no internet > there. Using the HTTP proxy would test the wrong thing. Presumably ports other than 80 either don't require the proxy, or else use some other proxy. But at any rate, the point of the indicator is not to indicate connectivity in the most abstract possible sense, it's to indicate whether the user is likely to be able to do the things they want to do. The problem here is that it's difficult for NetworkManager to get access to your proxy settings. (It would need to figure out the address of your D-Bus session bus so it could connect to it and query dconf... and I'm not sure dbus-daemon/dconf would respond to requests from the wrong user anyway, even if it was root.) Maybe NM should ask gnome-shell to do the connectivity test itself? Or maybe gnome-shell should ignore the connectivity stuff if you have a proxy configured. Anyway, for now, your best bet is to uninstall the connectivity-checking package, as Thomas said.
(In reply to comment #2) > (In reply to comment #1) > > In your company network you can apparently not access the internet, only HTTP > > via a proxy. The connectivity check fails correctly -- you have no internet > > there. Using the HTTP proxy would test the wrong thing. > > Presumably ports other than 80 either don't require the proxy, or else use some > other proxy. But at any rate, the point of the indicator is not to indicate > connectivity in the most abstract possible sense, it's to indicate whether the > user is likely to be able to do the things they want to do. > > > The problem here is that it's difficult for NetworkManager to get access to > your proxy settings. (It would need to figure out the address of your D-Bus > session bus so it could connect to it and query dconf... and I'm not sure > dbus-daemon/dconf would respond to requests from the wrong user anyway, even if > it was root.) Also, a proxy can be configured via different means. E.g. $http_proxy environment variable comes to mind. Applications can overwrite the setting. Different user sessions can have different settings. The requirement to check connectivity based on being able to proxy HTTP seems ill defined to me. We could redefine this bug to an RFE extending connectivity-checking. In that case, we could call out to a DBUS activated "NetworkManager-connectivity-check.service" that can be implemented according to the users needs. Otherwise, lets close this bug.
> Also, a proxy can be configured via different means. E.g. $http_proxy > environment variable comes to mind. Applications can overwrite the setting. > Different user sessions can have different settings. The requirement to check > connectivity based on being able to proxy HTTP seems ill defined to me. FWIW, the GNOME settings affect the environment variables session-wide. About the only things that don't instantaneously reflect those settings are opened shells. > We could redefine this bug to an RFE extending connectivity-checking. In that > case, we could call out to a DBUS activated > "NetworkManager-connectivity-check.service" that can be implemented according > to the users needs. There should perhaps be a UI to control the [connectivity] config section? It took a fair amount of digging to locate the NetworkManager.config man page once the new behavior exhibited itself.
(In reply to comment #1) > The connectivity checking checks whether you are connected to the Internet, to > a Hotspot, or to a local network only. Some UI show different icons depending > on your connectivity status. Some hotspots and local LANs hand out automatic proxy configurations (WPAD), which GNOME and dhclient currently have poor support for. This is maybe an opinion-based thing, but I think proxy connectivity is still connectivity, even if in a limited sense. > In your company network you can apparently not access the internet, only HTTP > via a proxy. The connectivity check fails correctly -- you have no internet > there. Using the HTTP proxy would test the wrong thing. Yes, I work in a secure environment. However, I respectfully disagree that using a proxy to check connectivity is the wrong test. Connectivity is connectivity. Transparent NAT isn't the One True Connectivity(tm). > If you dislike connectivity checking, just disable it in the configuration. > See `man NetworkManager.config` I'm a Linux expert, so discovering that wasn't so much a problem for me. Maybe Grandma won't ever be behind a secured network or a proxy, so maybe you guys should debate amongst yourselves as to whether there should be a UI for these settings. NetworkManager is obviously mimicking a Windows feature here, and this feature is obviously early in development for Linux-land. But, for the record, Windows can traverse a proxy in a secure network to verify Internet connectivity. > Actually, the file /etc/NetworkManager/conf.d/20-connectivity-fedora.conf comes > from the packaged "NetworkManager-config-connectivity-fedora" package. This > package only installs the mentioned config file. Remove this package, if you > don't want it. Yes, I figured all that out. However, gnome-shell-3.14.0-2.fc21.x86_64 and fedora-release-workstation-21-0.14.noarch require the existence of NetworkManager-config-connectivity-fedora. I've already opened a copy of this bug at Red Hat's bugzilla. That is a probably a better place to address these concerns. I'll make sure to go mention this there.
> Presumably ports other than 80 either don't require the proxy, or else use some > other proxy. But at any rate, the point of the indicator is not to indicate > connectivity in the most abstract possible sense, it's to indicate whether the > user is likely to be able to do the things they want to do. My secure workplace forbids all ports being routed beyond our enterprise network. External access is available for HTTP & HTTPS. This user wants to access HTTP and HTTPS via a proxy. :) If that is your argument, perhaps a much lighter-weight check should be done than an HTTP fetch of a resource and a comparison of its returned contents. Testing SYN/ACK/FIN would be enough under those assumptions.
See also: https://bugzilla.redhat.com/show_bug.cgi?id=1184406 https://bugzilla.gnome.org/show_bug.cgi?id=750260
Glad to see this bug get bumped recently. Maybe your work and thoughts on this issue can help inform Mozilla towards fix of this bug of theirs (only fixed on OS X & Windows, not fixed on modern Linux desktops): https://bugzilla.mozilla.org/show_bug.cgi?id=654579
The previous comment makes me wonder if this old bug has seen any love recently? I was just trying with NM 1.10.6, and the problem still occurs there - even though a proxy is configured, NM tries to access the Web directly, and fails. Notably this occurs even if the proxy settings have been set explictly for the active connetion using nmcli: > $ nmcli con show 'Wired connection 1' | grep proxy > proxy.method: auto > proxy.browser-only: no > proxy.pac-url: http://192.168.1.5/wpad.dat > proxy.pac-script: -- (did this both for the wired connection and the Wifi). So, while figuring out the proxy settings for the user session from dbus might be complicated (comment 3), it should be doable to respect explicitly configured per-connection proxy settings, no? I have a router here that would act like a captive portal for any restricted client, which causes the highly annoying captive portal popup to be shown all the time. If the client just respected the proxy, this wouldn't need to happen. Of course I can simply disable the connectivity check, but that's not what I want. When I'm somewhere else, using other networks without proxies, I'd like the connectivity check to work.
> Some hotspots and local LANs hand out automatic proxy configurations (WPAD), which GNOME and dhclient currently have poor support for. In the meantime NetworkManager got proxy support to help with getting proxy settings via DHCP (it requires pacrunner for that). > Notably this occurs even if the proxy settings have been set explictly for the active connetion using nmcli: > > > $ nmcli con show 'Wired connection 1' | grep proxy The proxy setting of the profile instructs NetworkManager to push the proxy configuration to pacrunner. There, the proxy settings will be available to applications that request them (or use libproxy with the pacrunner backend). If you don't have pacrunner running, this has no effect. Just like setting connection.zone has no effect if you don't run firewalld. Which both is fine, then the feature just isn't available.
I went now through some efforts to package pacrunner for openSUSE, but I have had no success so far, as far as the connectivity test is concerned. It appears that the proxy settings are pushed to pacrunner, but are not used in the connectivity test. Is this expected? This is NetworkManager-1.10.6-lp151.8.1.x86_64 It seems that NM doesn't even wait for the pacrunner call to finish before it does the connectivity check. > Jan 03 23:35:00 pallas.mittagstun.de NetworkManager[8792]: <debug> [1578090900.4885] pacrunner: call[0x55e56ee34f80]: send: new config ({'Interface': <'eth0'>, 'Method': <'auto'>, 'URL': <'http://192.168.1.5/wpad.dat'>, 'BrowserOnly': <false>},) > Jan 03 23:35:00 pallas.mittagstun.de NetworkManager[8792]: <debug> [1578090900.4885] connectivity: (eth0) sending request to 'http://conncheck.opensuse.org' > Jan 03 23:35:00 pallas.mittagstun.de NetworkManager[8792]: <info> [1578090900.4995] connectivity: (eth0) response did not match expected response 'NetworkManager is online'; assuming captive portal. > Jan 03 23:35:00 pallas.mittagstun.de NetworkManager[8792]: <debug> [1578090900.5013] pacrunner: name owner appeared (:1.58) > Jan 03 23:35:00 pallas.mittagstun.de NetworkManager[8792]: <debug> [1578090900.5023] pacrunner: call[0x55e56ee34f80]: sent (/org/pacrunner/configuration2) > Jan 03 23:35:00 pallas.mittagstun.de pacrunner[6219]: done 0x559b19b76170
Created attachment 374247 [details] journal ouput with nm debug messages Full journal from which I took the excerpts in the previous comment.
Thomas, I obviously misunderstood what you wanted to say in comment 10. NM has functionality to access the org.pacrunner.Manager interface, but it lacks code to access org.pacrunner.Client. The latter would be necessary to actually figure out the proxy to use for the connectivity check. So, one *might* implement org.pacrunner.Client:FindProxyForURL() calls to check whether proxy settings would be needed for the connectivity check. For people with NM coding experience (I don't qualify) it should be rather trivial to add such a FindProxyForURL() call, as all the infrastructure is basically in place. I wonder whether there are any plans to do this, or if respective patches would be taken.
NetworkManager will (still) not use the proxy to do the connectivity check. That is intended. Sorry for not being clear about that. I thought that was clear. In comment 10 I merely pointed out that in the meantime NM supports configuring proxies (but for a different purpose, unrelated to connectivity checks). "connectivity" check does not check whether you can tunnel some protocols via some proxy to reach some websites. It checks whether it can reach the connectivity check URL directly. The effect is, that if you can only reach the (a small part of) the internet via a proxy, then connectivity check will tell you that you are not connected to the internet. I think NetworkManager is giving you the correct answer here. --- Currently the connectivity check can only be enabled/disabled system-wide. Maybe it would sense to have this more flexible, although it seems over engineered to me. What might make sense is force the connectivity check per-profile basis. It would make sense to force/pretend connectivity check result on certain devices if, - you are never/always connected the the internet on that device - you don't want NM to perform connectivity checks on the device (to save overhead or whatever). - you have a different definition of what "online" means. E.g. when being behind a proxy. Consulting the proxy for the connectivity checking by default and always (not configurably) seems wrong to me. However, would might possible to add a per-profile setting to opt-in to the requested behaviour here. But I don't think such an option is worth implementing. I think the "pretend-always-global" and "pretend-always-local" options would be sufficient. This is basically about adding new API that solves these use-cases.
Unfortunately, pacrunner (and hence NetworkManager's proxy support) is not widely used. I guess, proxies are just not popular anymore. @martin, aside this specific issue about connectivity checking, how did it work for you? I'd be actually interested in feedback about the general proxy feature. Note that NetworkManager-1.10.6-lp151.8.1.x86_64 is really old by now :(
(In reply to Thomas Haller from comment #14) > "connectivity" check does not check whether you can tunnel some protocols > via some proxy to reach some websites. It checks whether it can reach the > connectivity check URL directly. > > The effect is, that if you can only reach the (a small part of) the internet > via a proxy, then connectivity check will tell you that you are not > connected to the internet. I think NetworkManager is giving you the correct > answer here. I respectfully disagree. To the the vast majority of users being "connected" means that "the internet" is reachable via http(s). Most users I know don't care whether or not they connect "directly" - they care whether they are able to connect *at all*. The NM connectivity check uses a single HTTP URL to represent "the internet" and test connectivity. That's obviously a weak test which could be very easily fooled... for practical purposes, it should check what most users care about (see above). So, if this connectivity URL is configured to be in the general internet, and a proxy is required to access this URL, the test should be done via the proxy. You may argue that users *should* care about proxies for privacy and content filtering reasons, or because they may want to use other ports besides http(s). That's a valid concern, but I'd suggest that it should be represented differently, e.g. by distinguishing "connected directly" and "connected via proxy". > Currently the connectivity check can only be enabled/disabled system-wide. > Maybe it would sense to have this more flexible, although it seems over > engineered to me. > > What might make sense is force the connectivity check per-profile basis. It > would make sense to force/pretend connectivity check result on certain > devices if, > > - you are never/always connected the the internet on that device > > - you don't want NM to perform connectivity checks on the device (to save > overhead or whatever). > > - you have a different definition of what "online" means. E.g. when being > behind a proxy. I don't think "connectivity" should have different meanings for different devices. But I guess I do have a different definition of "online" :-) Perhaps we could simply add a config option for that? [connectivity] uri=... use_proxy=[yes|no] I'm fine with defaulting to "use_proxy=no". > Consulting the proxy for the connectivity checking by default and always > (not configurably) seems wrong to me. However, would might possible to add a > per-profile setting to opt-in to the requested behaviour here. But I don't > think such an option is worth implementing. I think the > "pretend-always-global" and "pretend-always-local" options would be > sufficient. I'm not sure I understand the intended semantics of "pretend-always-{local,global}". Would it mean the same as use_proxy={yes,no} ? (In reply to Thomas Haller from comment #15) > Unfortunately, pacrunner (and hence NetworkManager's proxy support) is not > widely used. I guess, proxies are just not popular anymore. I'm not sure about that. My guess is that few Linux developers (have to) work behind proxies. Wrt pacrunner, most distributions seem to favor libproxy because (unlike pacrunner) it supports GNOME/KDE-based configuration, and the pacrunner backend for libproxy has never been upstreamed. Anyway, I think what's written in https://wiki.gnome.org/Projects/NetworkManager/Proxies is still valid. It's *GREAT* that NM can now update pacrunner's settings. IMO NM + pacrunner + libproxy with pacrunner backend is the way to go, if we ever want to reach user friendly configuration on par with other OSes. > @martin, aside this specific issue about connectivity checking, how did it > work for you? I'd be actually interested in feedback about the general proxy > feature. AFAICS it worked fine, apart from the connectivity check :-) > Note that NetworkManager-1.10.6-lp151.8.1.x86_64 is really old by now :( I know. It's what we have in Leap 15.1. I don't think more recent NM differs much wrt proxy support though.
> I respectfully disagree. I thought so :) > I'm not sure I understand the intended semantics of "pretend-always-{local,global}". Would it mean the same as use_proxy={yes,no} ? No, that would be then a 3rd choice: pretend-always-{local,global} + use-proxy. The semantics of "pretend-always-*" would be that NM doesn't actually perform any connectivity checking on the devices. It would assume that the result is fixed. Clearly the name "pretend-always-*" was confusing. That was just to explain the potential feature. But the behavior would be as such. I don't think system wide configuration (in NetworkManager.conf) for this would cut it. If it isn't per-profile (and thus per-device), it doesn't seem useful. > Wrt pacrunner, most distributions seem to favor libproxy because (unlike > pacrunner) it supports GNOME/KDE-based configuration, and the pacrunner backend > for libproxy has never been upstreamed. Yes, applications should talk directly to pacrunner but via libproxy. I think libproxy backend should be upstream already. It may be only the distro not integrating them correctly... > It's *GREAT* that NM can now update pacrunner's settings. IMO NM + pacrunner + libproxy with pacrunner backend is the way to go, if we ever want to reach user friendly configuration on par with other OSes. Fully agree. > > I'd be actually interested in feedback about the general proxy > > feature. > > AFAICS it worked fine, apart from the connectivity check :-) ACK. Thanks.
(In reply to Thomas Haller from comment #17) > Yes, applications should talk directly to pacrunner but via libproxy. I > think libproxy backend should be upstream already. It may be only the distro > not integrating them correctly... It's not upstream yet. I just made a new attempt, https://github.com/libproxy/libproxy/pull/105.
(In reply to Thomas Haller from comment #17) > I don't think system wide configuration (in NetworkManager.conf) for this > would cut it. If it isn't per-profile (and thus per-device), it doesn't seem > useful. Hm, maybe I'm getting you wrong once more, but IMO: - the actual proxy settings required to reach the outer world are a property of the network, and thus should depend on the profile; - the definition of being "online" (i.e. "use_proxy") is a user preference, and thus belongs in NetworkManager.conf. No?
bugzilla.gnome.org is being shut down in favor of a GitLab instance. We are closing all old bug reports and feature requests in GNOME Bugzilla which have not seen updates for a long time. If you still use NetworkManager and if you still see this bug / want this feature in a recent and supported version of NetworkManager, then please feel free to report it at https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/ Thank you for creating this report and we are sorry it could not be implemented (workforce and time is unfortunately limited).