GNOME Bugzilla – Bug 576838
Handle non-ASCII chars in NTLM and basic authentication
Last modified: 2018-09-21 16:02:03 UTC
For Evolution Exchange to correctly work with usernames with umlauts, please consider applying these two patches. They * make NTLM negotiate Unicode instead of OEM, according to http://davenport.sourceforge.net/ntlm.html. * encode the username for Basic auth as ISO-8859-1 first. This is expected by some servers, and Firefox does it as well. The patches are against libsoup2.4-2.4.1, but should be easily apply to the current version. If you do apply these, please mention that development was sponsored by ITOMIG GmbH and the City of Böblingen.
Created attachment 131420 [details] [review] Uncode in NTLM
Created attachment 131421 [details] [review] LATIN-1 in basic auth
On the latin-1-in-basic auth, is there any reason you encode the username but not the password? (What does Firefox do?) For NTLM, I'm always wary about making changes to that code, because I don't have any Microsoft servers to test it against. If you could provide tcpdump output of using this to authenticate against Exchange, that would be great. (Or E2K_DEBUG if that prints the NTLM headers... I don't remember if it does). Also, are there other bug reports about this anywhere that you know of?
About latin1 and password – yes, you are right. Both should be encoded (or just the concatenation, to save a call to g_convert()) :-) About NTLM: I’m not at my work place right now, and it’ll take a while until I’m there again... I did see the NTLM headers with E2K_DEBUG, and base64-decoding them looked good, and it worked. Are you looking for something specific, or just a proof that I tested the patch? :-) I did not find related reports, but it seems that most people just don’t use non-ascii-characters in login names, which is probably a good idea anyways...
(In reply to comment #4) > About NTLM: I’m not at my work place right now, and it’ll take a while > until I’m there again... I did see the NTLM headers with E2K_DEBUG, and > base64-decoding them looked good, and it worked. Are you looking for something > specific, or just a proof that I tested the patch? :-) Well, like, if this ends up breaking someone else, having debug output would let me compare your results against theirs, and have a better idea of what's wrong etc. Also, I might try to update tests/ntlm-test.c. > I did not find related reports, but it seems that most people just don’t use > non-ascii-characters in login names, which is probably a good idea anyways... Yeah, back when I was hacking on evolution-exchange, we always tested against users with non-ASCII *names*, but never non-ASCII usernames or passwords...
Ok, I’ll try to provide debug output when I get around to it. Thanks so far!
Created attachment 131903 [details] E2K_DEBUG log Here is the E2K_DEBUG log of evolution, logging in with a username of "töst".
Dan, ping.
Dan, it's a very old bug, awaiting for your response, can you please give a shot sometime ?
Created attachment 199594 [details] [review] SoupAuthBasic: allow (some) non-ASCII usernames/passwords Rebased, cleaned up. As compared to Joachim's original patch, this assumes the input to soup_auth_authenticate() is UTF-8 (like almost all other GNOME APIs). Probably still needs some tests added to auth-test
Created attachment 199595 [details] [review] SoupAuthManagerNTLM: allow non-ASCII usernames/passwords As above, assumes UTF-8 input. I fixed up ntlm-test to still pass. I have no way of testing that this really works though... also, I need to investigate the interaction with the /usr/bin/ntlm_auth code...
OK, Milan pointed me to some Red Hat lab machines I could use to test a while back. There was still one case that didn't work (IIRC, NTLM with passwords containing ISO-8859-1 characters worked, but passwords with ISO-8859-2 characters didn't, even though it was UTF-8 so that shouldn't have mattered [but perhaps depends on the server's codepage in some way?]). But anyway, these patches at least add more cases that work, and if there are still some cases that don't work, we can figure them out in the future committed to master Attachment 199594 [details] pushed as d350aed - SoupAuthBasic: allow (some) non-ASCII usernames/passwords Attachment 199595 [details] pushed as 67824e6 - SoupAuthManagerNTLM: allow non-ASCII usernames/passwords
(In reply to comment #12) > OK, Milan pointed me to some Red Hat lab machines I could use to test > a while back. There was still one case that didn't work (IIRC, > NTLM with passwords containing ISO-8859-1 characters worked, but > passwords with ISO-8859-2 characters didn't, even though it was UTF-8 > so that shouldn't have mattered That's because you're not using g_convert() where it matters for the password, which is in soup_ntlm_nt_hash(). Instead, you still have the naïve version: while (*password) { *p++ = *password++; *p++ = '\0'; } That is a correct conversion from ISO8859-1 to UCS2LE, so if you actually passed an ISO8859-1 password in (with a single byte 0xef for ï, for example), then it's expected that it would have worked. But if you passed in the password as UTF-8, it shouldn't have worked if there were any non-ASCII characters. Something else is weird.
I believe you meant to reopen this, not mark it QA-approved. (Yay non-obvious state names.)
(In reply to comment #10) > Created an attachment (id=199594) [details] [review] > SoupAuthBasic: allow (some) non-ASCII usernames/passwords > > Rebased, cleaned up. As compared to Joachim's original patch, this > assumes the input to soup_auth_authenticate() is UTF-8 (like almost > all other GNOME APIs). This patch (commit d350aedcf0) broke Basic auth for me against our corporate ActiveSync servers. They require the password to be in UTF-8. As does my local copy of squid, it seems. Perhaps the world has moved on since 2009 and people are starting to use UTF-8 consistently? See also: http://stackoverflow.com/questions/7242316/what-encoding-should-i-use-for-http-basic-authentication https://datatracker.ietf.org/doc/draft-ietf-httpauth-basicauth-enc/
FWIW curl and Chrome send UTF-8 for Basic authentication. Firefox sends ISO8859-1 with a twist... it actually just sends the low 8 bits of every character, which happens to work out OK for ISO8859-1. So if you type a U+2639 WHITE FROWNING FACE ☹, it'll just send 0x39 '9'. Apparently (according to the second answer in the stackoverflow URL I gave above) IE just uses the system code page, and Safari sends ISO8859-1. Squid does *not* appear to accept the RFC2047-encoded scheme that is mentioned there (I wrapped my password with =?utf-8?q?...? and it wasn't accepted). Joachim (if you're still here), what servers were you testing against that required ISO8859-1, and are they still stuck in the dark ages or do they now work with UTF-8?
(In reply to comment #16) > Squid does *not* appear to accept the RFC2047-encoded scheme that is mentioned > there (I wrapped my password with =?utf-8?q?...? and it wasn't accepted). Yeah, I don't think that I-D got much traction, unfortunately. > Joachim (if you're still here), what servers were you testing against that > required ISO8859-1, and are they still stuck in the dark ages or do they now > work with UTF-8? The problem is that if the password was originally created via the web (as opposed to be looked up in AD or something), and the web server just accepts the password as a byte string without trying to interpret it (which is absolutely what it should be doing if you ignore charset issues), then it's not a question of the server being "stuck in the dark ages". It's a question of whether the web browser you're using now sends the same byte string as the web browser you were using when you set the password. (In reply to comment #15) > This patch (commit d350aedcf0) broke Basic auth for me against our corporate > ActiveSync servers. They require the password to be in UTF-8. As does my local > copy of squid, it seems. Out of curiosity, what is the charset on the 401/407 page that they send?
(In reply to comment #17) > (In reply to comment #16) > > Squid does *not* appear to accept the RFC2047-encoded scheme that is mentioned > > there (I wrapped my password with =?utf-8?q?...? and it wasn't accepted). > > Yeah, I don't think that I-D got much traction, unfortunately. There was an I-D about explicitly stating the character set. But the RFC2047-encoding thing is an interpretation of the *existing* standard. One which makes sense, I suppose, but doesn't seem to have been shared by anyone actually writing any code :) > The problem is that if the password was originally created via the web (as > opposed to be looked up in AD or something), and the web server just accepts > the password as a byte string without trying to interpret it Is that a common scenario? If the password was created on the web, I'd usually expect the login to happen "on the web" too — with forms, rather than HTTP authentication. I think I've only ever seen HTTP authentication used when the password has been set through other means. > Out of curiosity, what is the charset on the 401/407 page that they send? Squid just says 'Content-type: text/html', so that's the default ISO8859-1 I think. And the ActiveSync service just gives my 'Content-Length: 0' :) No clues there, I'm afraid.
I note that the original service which required ISO8859-1 was evolution-exchange, which is long dead. Microsoft services these days do seem to use UTF-8 in Basic auth (which is where I noticed the problem). I'd recommend that libsoup stay out of it and refrain from doing any conversion, and let the caller provide the username/password in the form they want it to appear on the wire.
(In reply to comment #19) > I'd recommend that libsoup stay out of it and refrain from doing any > conversion, and let the caller provide the username/password in the form they > want it to appear on the wire. Yeah... I'd take a patch for this I guess
[mass-moving all "UNCONFIRMED" libsoup bugs to "NEW" after disabling the "UNCONFIRMED" status for this product now that bugzilla.gnome.org allows that. bugspam-libsoup-20150210]
-- GitLab Migration Automatic Message -- This bug has been migrated to GNOME's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.gnome.org/GNOME/libsoup/issues/21.