In which encoding is the SSID? Try for example:

 $ busctl --system call --verbose org.freedesktop.NetworkManager /org/freedesktop org.freedesktop.DBus.ObjectManager GetManagedObjects 

and search for 

   OBJECT_PATH "/org/freedesktop/NetworkManager/AccessPoint/

and see the Ssid bytes.



In general, if your LANG indicates that the locale is UTF-8, nmcli doesn't know any other encoding to try. It might would work, if you set the correct locale.

I believe 

> busctl --system call --verbose org.freedesktop.NetworkManager /org/freedesktop org.freedesktop.DBus.ObjectManager GetManagedObjects 

will only contain the networks that the system has already been connected to, is that correct? At least I have done a short test, creating a new AP on my phone and couldn't find it with above command (after making sure it is visible in nmcli output).

But to test your assumption of encoding issues I created an AP on my phone with the name "你好" and the name was correctly displayed on my computer. I.e. the phone seems to send the SSID in UTF-8 encoding.

Unfortunately I have no idea about the encoding of the SSIDs mentioned in the bug report. I just tried encoding above string from nmcli to bytes and then decoding it with every combination of the standard encodings known to python [1], checking if the output contains one of the expected characters, but was not successful.

Currently I do not have access to the AP, as they are run at a local shopping mall but I can check back next time I happen to be there. In the meantime it would be great if you can suggest me how to get the ssid bytes mentioned in your previous comment, given that the busctl command seems not to include networks that have not been connected to yet.

However, given that my Android phone correctly decodes the problematic SSIDs I would expect NetworkManager to do that as well. I.e. the bug in this case may be that NetworkManager does not try other encodings if the decoding with the system locale fails.
I am not sure if this is a feature of my Chinese phone or part of AOSP but can test that as well with a phone running AOSP coming weekend. If it is part of AOSP, NetworkManager may adopt the same strategy for decoding SSID names as AOSP (given one can check the strategy in the AOSP sources). What do you think?

[1] https://docs.python.org/3/library/codecs.html#standard-encodings

(In reply to Leonard Lausen from comment #2)
> I believe 
> 
> > busctl --system call --verbose org.freedesktop.NetworkManager /org/freedesktop org.freedesktop.DBus.ObjectManager GetManagedObjects 
> 
> will only contain the networks that the system has already been connected
> to, is that correct? At least I have done a short test, creating a new AP on
> my phone and couldn't find it with above command (after making sure it is
> visible in nmcli output).

If the AP is present in the nmcli output, it must also be in the
D-Bus dump. You should see something like:

  DICT_ENTRY "oa{sa{sv}}" {
           OBJECT_PATH "/org/freedesktop/NetworkManager/AccessPoint/179";
                                   ...
                                   DICT_ENTRY "sv" {
                                           STRING "Ssid";
                                           VARIANT "ay" {
                                                   ARRAY "y" {
                                                           BYTE 100;
                                                           BYTE 100;
                                                           BYTE 100;
                                                           BYTE 100;
                                                   };
                                           };
                                   };

So I found that the AP name is encoded in GB 18030:

➜  ~ ipython            
Python 3.6.1 (default, Jul 19 2017, 13:27:38)    
Type 'copyright', 'credits' or 'license' for more information                                     
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.                               

In [1]: import codecs   

In [2]: a = [ 187,      
   ...:  231,           
   ...:  191,           
   ...:  236,           
   ...:  179,           
   ...:  170,           
   ...:  199,           
   ...:  209,           
   ...:  189,           
   ...:  196,           
   ...:  180,           
   ...:  231,           
   ...:  53,            
   ...:  71             
   ...: ]               

In [3]: [codecs.decode(bytes(a), c) for c in ["gb2312" , "gbk", "gb18030"]]                       
Out[3]: ['荤快唱茄侥寸5G', '荤快唱茄侥寸5G', '荤快唱茄侥寸5G']                                    

Please do let me know if you would like to see the whole log of the busctl command.

Please also note that these bytes are not valid utf-8, which leads to networkmanager decoding routine returning gibberish, making any connection attempt to the network impossible. To fix this bug, at least networkmanager should return the AP name as a string of hex digits in case decoding fails, making connection attempts to the affected networks possible.

Arguably better would be to fallback to the "GB 18030" codec for decoding the bytes. Alternatively an ordered list of fallback codecs could be specified in a configuration file (or at build time) that NetworkManager would try when failing to decode with the system locale. I believe that is what my Chinese phone is doing. The latter option could default to "GB 18030", but could be changed by users if they live in a different area of the world.

(In reply to Leonard Lausen from comment #4)
> Arguably better would be to fallback to the "GB 18030" codec for decoding
> the bytes. Alternatively an ordered list of fallback codecs could be
> specified in a configuration file (or at build time) that NetworkManager
> would try when failing to decode with the system locale. I believe that is
> what my Chinese phone is doing. The latter option could default to "GB
> 18030", but could be changed by users if they live in a different area of
> the world.

This is exactly what nmcli does: it tries UTF-8 and falls back to a list of encodings depending on the LANG= environment variable [1].

Is the SSID displayed correctly if you set LANG=zh_CN or LANG=zh_CN.gb18030 ?

[1] https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/libnm-core/nm-utils.c?id=a47153f5b8a114ce1f0d226b33d0ad6d6ec7f150#n76

Thanks for the reference. Unfortunately I don't have access to the AP currently to test with a different LANG variable, but let's just assume the the decoding would work in that case.

The main bug however remains, which is that NetworkManager fails to connect to the network if it can't decode the AP name. We can't expect that the user will always have the right LANG setting for the surrounding APs. NetworkManager needs to this case gracefully. I.e. it should still allow connecting, perhaps displaying the MAC address of the AP in the UI.

On a sidenote, shouldn't LC_CTYPE be checked instead of LANG?

As far as NetworkManager (the deamon) is concerned, the SSID is only a binary blob, like on the physical wire(less).

It's up the the client (in this case nmcli) to present that in some textual form.

nmcli in this case, guesses the encoding already. It tries UTF8, and falls back to the encoding from LANG. What else could it do?

(In reply to Thomas Haller from comment #7)
> As far as NetworkManager (the deamon) is concerned, the SSID is only a
> binary blob, like on the physical wire(less).
> 
> It's up the the client (in this case nmcli) to present that in some textual
> form.
> 
> nmcli in this case, guesses the encoding already. It tries UTF8, and falls
> back to the encoding from LANG. What else could it do?

Also, 'nmcli -f all device wifi list' will show the hex SSID for each AP and its BSSID, which can be used to connect with 'nmcli device wifi connect <BSSID>'.

I tried nm-applet (not the KDE one) and it can connect to non-UTF8 SSIDs, so I think it's a problem of the specific client (KDE applet).

Thanks. I will report the bug to KDE.

For nmcli then the only problem remaining is that LC_CTYPE should be used instead of LANG. This follows from: https://www.gnu.org/savannah-checkouts/gnu/libc/manual/html_node/Locale-Categories.html

> LC_CTYPE
> This category applies to [...] Character Set Handling.

> LANG 
> If this environment variable is defined, its value specifies the locale to use
> for all purposes except as overridden by the variables above. 

The reasoning is that while a user may need to work with characters encoded in e.g. gb18030, it doesn't mean this user is proficient or willing to switch the interface language to zh_CN. Currently it seems nmcli requires to switch the interface language (i.e. LANG=zh_CN.UTF-8) to allow decoding of AP names using gb18030.

I have set LC_CTYPE=zh_CN.UTF-8, but LANG=en_US.UTF-8 so I can confirm that either nmcli ignores LC_CTYPE or that there is some other bug which prevents the decoding to work..

This is the KDE bug: https://bugs.kde.org/show_bug.cgi?id=384410

Created attachment 359511 [details] [review]
[PATCH 1/2] libnm-core: cache the current character encoding

Created attachment 359512 [details] [review]
[PATCH 2/2] libnm-core: also consider LC_ALL and LC_CTYPE to guess SSID charset

patches lgtm

Applied to master:

https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=28a062748180534445b3996862c60de330d48dbf
https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=789f8a6b51ebdd7cb6ba5c902c02e5a44a71ed68