For reference, from mail/e-mail-autoconfig.c:

/* XXX Thoughts on RFC 6186: Use of SRV Records for Locating Email
 *                           Submission/Access Services
 *
 *     RFC 6186 specifies using SRV DNS lookups to aid in automatic
 *     configuration of mail accounts.  While it may be tempting to
 *     implement the RFC here (I was tempted at least), upon closer
 *     examination I find the RFC to be insufficient.
 *
 *     An SRV DNS lookup only provides a host name and port number.
 *     The RFC assumes the account's user name can be derived from
 *     the email address, and suggests probing the mail server for
 *     a valid user name by actually attempting authentication,
 *     first with the user's full email address and then falling
 *     back to only the local part.
 *
 *     I'm uncomfortable with this for a number of reasons:
 *
 *     1) I would prefer the user have a chance to manually review
 *        the settings before transmitting credentials of any kind,
 *        since DNS responses can be spoofed.
 *
 *     2) Authentication at this phase would require asking for
 *        a password either before or during auto-configuration.
 *        Asking before assumes a password-based authentication
 *        mechanism is to be used, which is not always the case,
 *        and asking during may raise the user's suspicion about
 *        what's going on behind the scenes (it would mine).
 *
 *     3) For better or worse, our architecture isn't really built
 *        to handle authentication at this stage.  EMailSession is
 *        wired into too many other areas to be reused here without
 *        risking unwanted side-effects, therefore it would require
 *        a custom CamelSession subclass with an authenticate_sync()
 *        implementation similar to EMailSession.
 *
 * While the technical limitations of (3) could be overcome, my concerns
 * in (1) and (2) still stand.  I think for the time being a better solution
 * is to have an administrator script on api.gnome.org that compares the host
 * and port settings in each clientConfig file to the _imap._tcp, _pop3._tcp,
 * and _submission._tcp SRV records for that service provider (if available)
 * to help ensure the static XML content remains accurate.  It would also be
 * instructive to track how many service providers even implement RFC 6186.
 *
 * Recording my thoughts here for posterity. -- mbarnes

Some thoughts on comment 1...

(1) DNSSEC is becoming more common, and any RFC6186 lookup validated with DNSSEC really ought to be trustworthy. For any lookup which is *not* so validated, perhaps we could permit authentication only using non-plaintext methods such as CRAM-MD5, until/unless the user confirms the selection of IMAP/SMTP servers.

(2) We've fixed most things already to cope with authentication requests occurring on demand; I don't think this should be too much of a problem. Regarding the 'user's suspicion' that's mostly solved by letting the user know what's going on, surely? I imagine this would have a UI where you see something like..

Checking which protocol to use for dwmw2@example.com...

 ⟳ IMAP service:
   Looking up _tcp._imaps.example.com

 ⟳ Exchange Web Services:
   Requesting https://autodiscover.example.com/autodiscover.xml

 X Checking for ActiveSync:
   No response from https://autodiscover.example.com/autodiscover.xml

You'd have checkboxes or something to *stop* each type of lookup, as well as to skip the whole thing and configure it manually.

When a given protocol needs a (username and/or) password, it could either pop up a text entry box *in* its own part of the UI, or if it's going to pop up a modal dialog box (hint: don't) then it needs to explicitly state what it's asking for.

We are still going to want special cases for things like Google though, because we use them to add the matching calendar and addressbook. Unless we're going to ditch that in favour of GOA? Didn't I just find that direct-in-evolution Google addressbook is utterly broken these days anyway, because it needs OAUTH and that only works with GOA? (Or does that just mean that what we really need is an even *more* special case for Google, if we don't want to depend on GOA?)

Created attachment 321484 [details]
srv.c

Here's a little test program which tries to dig information from SRV records for various services. The first line contains a command line to build & run it. It can be used as a starting point.

Some servers which can be tried: icloud.com, google.com, googlemail.com (the two Google servers return different values).

Some more test domains from popular email providers:

IMAP/SMTP SRVs only:
* comcast.net
* mail.ru
* qq.com

All SRVs:
* fastmail.com
* gmail.com
* me.com
* yandex.ru

CalDAV/CardDAV SRVs only:
* aol.com
* gmx.com
* yahoo.com
* ymail.com

Most of these  either operate under different names around the world, or offer many domain variations. As far as I can tell, their SRV records are the same for these secondary domains as for their main domains. Someone with access to a large database of email addresses could mine it for popular email domains for further testing.

I made some initial work on this. I made all needed what the bug summary asks for, that is, I made server settings lookup extensible. It's only a beginning, it doesn't do everything I wanted to see there, but I've some semi-proof of concept with added EWS code, which looks promising. The code currently doesn't ask for a password, which means the EWS code is void.

I will extend this further, even I'm closing this bug report.

I'm not sure how it'll look like yet, possibly not like suggested in (2) of comment #2, because with many config lookups it would make UI overcomplicated, but we'll see. I can imagine that more changes will be required even to the just added code. The time will show.

I think that the New->Account (not New->Mail Account, though it doesn't exist yet, it's buried in Edit->Accounts->Add->...) will have a password entry or something like that for the Collection account (also doesn't exist yet), though this won't work for OAuth2. The worse the password dialog will be shown for some temporary ESource and as David said, it'll clearly claim why it wants it.

Created commit 62e5d0cd16 in evo master (3.25.90+)
Created commit_7bb721c93b in ews master (3.25.90+) [1]

[1] https://git.gnome.org/browse/evolution-ews/commit/?id=7bb721c93b

Passwords... obviously EWS should normally work *without* a password at all; I don't know of any servers which support EWS but *not* Kerberos/NTLM single-sign-on. 

But in the general case, we want to avoid prompting users for a password which isn't going to work anyway, because they don't have an account on that server at all.

Do we want a first pass of all the back ends' auto-configure methods *without* a password first? Some of them may indicate failure; others may indicate that they need a password. We do a second pass of those which wanted a password, only after the first pass has failed to do a passwordless discovery for *all* back ends.

I'm really not sure how to deal with it at the moment. I only know that evolution-ews currently fails when you try to autodiscover without password, it does that with a runtime warning. Whether it's after it tried without password or not I didn't look for, I'll investigate it later.

I thought of mimicking what GOA does, as it has just an entry for the password and that's all.

Please don't propagate the mis-design of asking for a password in advance, even when it isn't needed. We've already fixed that in a few places already, to make SSO work properly. :)

It doesn't even work right for all cases where passwords *are* needed, if the password in question is time-based like TOTP.

You need to ask for passwords interactively, IF and WHEN they are needed. Storing that password for later can be an *option*, and obviously the account-provisioning tools that we don't have yet (see http://david.woodhou.se/setup-evolution.sh and weep) will want a way to 'provision' a password if they *do* know they'll need it. But in general, don't assume you'll need one.

Actually, that's a design request for this auto-configure feature that I hadn't really made explicit — please make it possible for "provisioning scripts" like that nasty hack I just showed, to make use of this feature too — including creating the resulting account without resorting to creating source files manually, if at all possible!

(In reply to David Woodhouse from comment #9)
> including creating the resulting account without resorting to creating
> source files manually, if at all possible!

Yes, it's doable now. You just create your own configuration module and add to it something like e-ews-config-lookup.c/.h, which I added in the above EWS commit, except you won't probably call the autodiscover code, but will just fill the EConfigLookupResultSimple accordingly and pass it to EConfigLookup. You can derive from the EConfigLookupResultSimple and provide your own, more complicated, too, this one is just to make things easy when they can be easy (your script touches multiple .source files, which makes things harder, but still possible (e_config_lookup_get_source() can be used)). On the other hand, such module requires runtime dependencies, while the script is more or less simple.

You can use also eds' autoconfiguration feature (bug #773156), though it requires to provide your own .source files.

I made a use of the SRV records with the below commit. The real benefit will be implemented for 3.28, unfortunately, because there's a UI freeze (among others) starting on Monday and I'm surely not able to make all of that on time for it.

Created commit a6bf868a6af703b in evo master (3.25.90+)

I made this happen similarly as outlined in comment #2 for Edit->Accounts->New->Collection Account, as part of bug #760329.