After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 784415 - net-misc/networkmanager fails to connect to APs whose SSID can't be decoded with system locale
net-misc/networkmanager fails to connect to APs whose SSID can't be decoded w...
Status: RESOLVED FIXED
Product: NetworkManager
Classification: Platform
Component: Wi-Fi
1.8.x
Other Linux
: Normal normal
: ---
Assigned To: NetworkManager maintainer(s)
NetworkManager maintainer(s)
Depends on:
Blocks: nm-review
 
 
Reported: 2017-07-01 08:57 UTC by Leonard Lausen
Modified: 2017-09-12 21:51 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Screenshot (104.89 KB, image/png)
2017-07-01 08:57 UTC, Leonard Lausen
  Details
[PATCH 1/2] libnm-core: cache the current character encoding (10.71 KB, patch)
2017-09-11 08:41 UTC, Beniamino Galvani
none Details | Review
[PATCH 2/2] libnm-core: also consider LC_ALL and LC_CTYPE to guess SSID charset (1.07 KB, patch)
2017-09-11 08:41 UTC, Beniamino Galvani
none Details | Review

Description Leonard Lausen 2017-07-01 08:57:46 UTC
Created attachment 354757 [details]
Screenshot

networkmanager fails to decode Chinese AP names and therefore fails to connect to the AP as during the connection request it seems to ask for a different name than how the AP actually is called.

Output from nmcli listing WiFi networks:

➜  nmcli dev wifi
*  SSID                            MODE   CHAN  RATE       SIGNAL  BARS  SECURITY
   »ç¿ì³ª½ºÆ÷Ã÷¸¶»çÁö5G            Infra  149   54 Mbit/s  94      ▂▄▆█  --
   »ç¿ì³ª½ºÆ÷Ã÷¸¶»çÁö              Infra  12    54 Mbit/s  69      ▂▄▆_  --
   »ç¿ì³ª °æ¶ô5G                   Infra  149   54 Mbit/s  67      ▂▄▆_  --
   AndroidHotspot4865              Infra  6     54 Mbit/s  62      ▂▄▆_  WPA2
   HP-Print-89-Officejet Pro 8640  Infra  1     54 Mbit/s  55      ▂▄__  WPA2
   »ç¿ì³ª °æ¶ô                     Infra  11    54 Mbit/s  45      ▂▄__  --
   AMIGO_123456                    Infra  1     54 Mbit/s  44      ▂▄__  WPA1 WPA2


Note for example the correct name of the third and sixth network in above list would be "混快唱 版愕" and "混快唱 版愕5G". They are correctly shown on my Android phone and I can also connect to these networks using my Android phone.

The wrongly decoded network names are also shown in the KDE applet. I am attaching a screenshot.

Trying to connect to one of these networks via the KDE applet will lead to the following error in the NetworkManager log:

NetworkManager[2364]: <info>  [1498896910.6776] audit: op="connection-add-activate" pid=2561 uid=1000 result="fail" reason="802-11-wireless.ssid: connection does not match access point"

networkmanager is built with the following use flags:

 * Found these USE flags for net-misc/networkmanager-1.8.0:
 U I
 - - abi_x86_32         : 32-bit (x86) libraries
 - - audit              : Enable support for Linux audit subsystem using sys-process/audit
 + + bluetooth          : Enable Bluetooth Support
 - - connection-sharing : Use net-dns/dnsmasq and net-firewall/iptables for connection sharing
 + + dhclient           : Use dhclient from net-misc/dhcp for getting ip
 - - dhcpcd             : Use net-misc/dhcpcd for getting ip
 - - elogind            : Use sys-auth/elogind for session tracking
 - - gnutls             : Add support for net-libs/gnutls (TLS 1.0 and SSL 3.0 support)
 + + introspection      : Add support for GObject based introspection
 - - json               : Enable JSON validation via dev-libs/jansson in libnm.
 + + modemmanager       : Enable support for mobile broadband devices using net-misc/modemmanager
 + + ncurses            : Add ncurses support (console display library)
 + + nss                : Use dev-libs/nss for cryptography
 - - ofono              : Use net-misc/ofono for telephony support.
 + + ppp                : Enable support for mobile broadband and PPPoE connections using net-dialup/ppp
 - - resolvconf         : Use net-dns/openresolv for managing DNS information
 + + systemd            : Enable use of systemd-specific libraries and features like socket activation or session tracking
 - - teamd              : Enable Teamd control support
 - - test               : Workaround to pull in packages needed to run with FEATURES=test. Portage-2.1.2 handles this internally, so don't set it in make.conf/package.use anymore
 - - vala               : Enable bindings for dev-lang/vala
 + + wext               : Enable support for the deprecated Wext (Wireless Extensions) API; needed for some older drivers (e.g. ipw2200, ndiswrapper)
 + + wifi               : Enable support for wifi and 802.1x security using net-wireless/wpa_supplicant


I have not experienced such bugs with using NetworkManager on other distributions. Though on other distributions I used older versions of NetworkManager. There is no problem with displaying and entering Chinese characters in other programs, including the NetworkManager KDE frontend.

I am attaching the output of locale on my system:

➜ locale
LANG=en_US.UTF-8
LC_CTYPE=zh_CN.UTF-8
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE=C
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Downstream bug: https://bugs.gentoo.org/show_bug.cgi?id=623242
Comment 1 Thomas Haller 2017-07-17 10:42:32 UTC
In which encoding is the SSID? Try for example:

 $ busctl --system call --verbose org.freedesktop.NetworkManager /org/freedesktop org.freedesktop.DBus.ObjectManager GetManagedObjects 

and search for 

   OBJECT_PATH "/org/freedesktop/NetworkManager/AccessPoint/

and see the Ssid bytes.



In general, if your LANG indicates that the locale is UTF-8, nmcli doesn't know any other encoding to try. It might would work, if you set the correct locale.
Comment 2 Leonard Lausen 2017-07-17 11:51:23 UTC
I believe 

> busctl --system call --verbose org.freedesktop.NetworkManager /org/freedesktop org.freedesktop.DBus.ObjectManager GetManagedObjects 

will only contain the networks that the system has already been connected to, is that correct? At least I have done a short test, creating a new AP on my phone and couldn't find it with above command (after making sure it is visible in nmcli output).

But to test your assumption of encoding issues I created an AP on my phone with the name "你好" and the name was correctly displayed on my computer. I.e. the phone seems to send the SSID in UTF-8 encoding.

Unfortunately I have no idea about the encoding of the SSIDs mentioned in the bug report. I just tried encoding above string from nmcli to bytes and then decoding it with every combination of the standard encodings known to python [1], checking if the output contains one of the expected characters, but was not successful.

Currently I do not have access to the AP, as they are run at a local shopping mall but I can check back next time I happen to be there. In the meantime it would be great if you can suggest me how to get the ssid bytes mentioned in your previous comment, given that the busctl command seems not to include networks that have not been connected to yet.

However, given that my Android phone correctly decodes the problematic SSIDs I would expect NetworkManager to do that as well. I.e. the bug in this case may be that NetworkManager does not try other encodings if the decoding with the system locale fails.
I am not sure if this is a feature of my Chinese phone or part of AOSP but can test that as well with a phone running AOSP coming weekend. If it is part of AOSP, NetworkManager may adopt the same strategy for decoding SSID names as AOSP (given one can check the strategy in the AOSP sources). What do you think?

[1] https://docs.python.org/3/library/codecs.html#standard-encodings
Comment 3 Beniamino Galvani 2017-07-19 16:14:08 UTC
(In reply to Leonard Lausen from comment #2)
> I believe 
> 
> > busctl --system call --verbose org.freedesktop.NetworkManager /org/freedesktop org.freedesktop.DBus.ObjectManager GetManagedObjects 
> 
> will only contain the networks that the system has already been connected
> to, is that correct? At least I have done a short test, creating a new AP on
> my phone and couldn't find it with above command (after making sure it is
> visible in nmcli output).

If the AP is present in the nmcli output, it must also be in the
D-Bus dump. You should see something like:

  DICT_ENTRY "oa{sa{sv}}" {
           OBJECT_PATH "/org/freedesktop/NetworkManager/AccessPoint/179";
                                   ...
                                   DICT_ENTRY "sv" {
                                           STRING "Ssid";
                                           VARIANT "ay" {
                                                   ARRAY "y" {
                                                           BYTE 100;
                                                           BYTE 100;
                                                           BYTE 100;
                                                           BYTE 100;
                                                   };
                                           };
                                   };
Comment 4 Leonard Lausen 2017-07-30 08:39:48 UTC
So I found that the AP name is encoded in GB 18030:

➜  ~ ipython            
Python 3.6.1 (default, Jul 19 2017, 13:27:38)    
Type 'copyright', 'credits' or 'license' for more information                                     
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.                               

In [1]: import codecs   

In [2]: a = [ 187,      
   ...:  231,           
   ...:  191,           
   ...:  236,           
   ...:  179,           
   ...:  170,           
   ...:  199,           
   ...:  209,           
   ...:  189,           
   ...:  196,           
   ...:  180,           
   ...:  231,           
   ...:  53,            
   ...:  71             
   ...: ]               

In [3]: [codecs.decode(bytes(a), c) for c in ["gb2312" , "gbk", "gb18030"]]                       
Out[3]: ['荤快唱茄侥寸5G', '荤快唱茄侥寸5G', '荤快唱茄侥寸5G']                                    

Please do let me know if you would like to see the whole log of the busctl command.

Please also note that these bytes are not valid utf-8, which leads to networkmanager decoding routine returning gibberish, making any connection attempt to the network impossible. To fix this bug, at least networkmanager should return the AP name as a string of hex digits in case decoding fails, making connection attempts to the affected networks possible.

Arguably better would be to fallback to the "GB 18030" codec for decoding the bytes. Alternatively an ordered list of fallback codecs could be specified in a configuration file (or at build time) that NetworkManager would try when failing to decode with the system locale. I believe that is what my Chinese phone is doing. The latter option could default to "GB 18030", but could be changed by users if they live in a different area of the world.
Comment 5 Beniamino Galvani 2017-09-05 13:40:34 UTC
(In reply to Leonard Lausen from comment #4)
> Arguably better would be to fallback to the "GB 18030" codec for decoding
> the bytes. Alternatively an ordered list of fallback codecs could be
> specified in a configuration file (or at build time) that NetworkManager
> would try when failing to decode with the system locale. I believe that is
> what my Chinese phone is doing. The latter option could default to "GB
> 18030", but could be changed by users if they live in a different area of
> the world.

This is exactly what nmcli does: it tries UTF-8 and falls back to a list of encodings depending on the LANG= environment variable [1].

Is the SSID displayed correctly if you set LANG=zh_CN or LANG=zh_CN.gb18030 ?

[1] https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/libnm-core/nm-utils.c?id=a47153f5b8a114ce1f0d226b33d0ad6d6ec7f150#n76
Comment 6 Leonard Lausen 2017-09-05 14:07:48 UTC
Thanks for the reference. Unfortunately I don't have access to the AP currently to test with a different LANG variable, but let's just assume the the decoding would work in that case.

The main bug however remains, which is that NetworkManager fails to connect to the network if it can't decode the AP name. We can't expect that the user will always have the right LANG setting for the surrounding APs. NetworkManager needs to this case gracefully. I.e. it should still allow connecting, perhaps displaying the MAC address of the AP in the UI.

On a sidenote, shouldn't LC_CTYPE be checked instead of LANG?
Comment 7 Thomas Haller 2017-09-05 16:02:28 UTC
As far as NetworkManager (the deamon) is concerned, the SSID is only a binary blob, like on the physical wire(less).

It's up the the client (in this case nmcli) to present that in some textual form.

nmcli in this case, guesses the encoding already. It tries UTF8, and falls back to the encoding from LANG. What else could it do?
Comment 8 Beniamino Galvani 2017-09-05 16:10:00 UTC
(In reply to Thomas Haller from comment #7)
> As far as NetworkManager (the deamon) is concerned, the SSID is only a
> binary blob, like on the physical wire(less).
> 
> It's up the the client (in this case nmcli) to present that in some textual
> form.
> 
> nmcli in this case, guesses the encoding already. It tries UTF8, and falls
> back to the encoding from LANG. What else could it do?

Also, 'nmcli -f all device wifi list' will show the hex SSID for each AP and its BSSID, which can be used to connect with 'nmcli device wifi connect <BSSID>'.
Comment 9 Beniamino Galvani 2017-09-05 19:24:15 UTC
I tried nm-applet (not the KDE one) and it can connect to non-UTF8 SSIDs, so I think it's a problem of the specific client (KDE applet).
Comment 10 Leonard Lausen 2017-09-06 06:27:55 UTC
Thanks. I will report the bug to KDE.

For nmcli then the only problem remaining is that LC_CTYPE should be used instead of LANG. This follows from: https://www.gnu.org/savannah-checkouts/gnu/libc/manual/html_node/Locale-Categories.html

> LC_CTYPE
> This category applies to [...] Character Set Handling.

> LANG 
> If this environment variable is defined, its value specifies the locale to use
> for all purposes except as overridden by the variables above. 

The reasoning is that while a user may need to work with characters encoded in e.g. gb18030, it doesn't mean this user is proficient or willing to switch the interface language to zh_CN. Currently it seems nmcli requires to switch the interface language (i.e. LANG=zh_CN.UTF-8) to allow decoding of AP names using gb18030.

I have set LC_CTYPE=zh_CN.UTF-8, but LANG=en_US.UTF-8 so I can confirm that either nmcli ignores LC_CTYPE or that there is some other bug which prevents the decoding to work..
Comment 11 Leonard Lausen 2017-09-06 06:49:59 UTC
This is the KDE bug: https://bugs.kde.org/show_bug.cgi?id=384410
Comment 12 Beniamino Galvani 2017-09-11 08:41:03 UTC
Created attachment 359511 [details] [review]
[PATCH 1/2] libnm-core: cache the current character encoding
Comment 13 Beniamino Galvani 2017-09-11 08:41:56 UTC
Created attachment 359512 [details] [review]
[PATCH 2/2] libnm-core: also consider LC_ALL and LC_CTYPE to guess SSID charset
Comment 14 Thomas Haller 2017-09-11 11:59:39 UTC
patches lgtm