OK, I have no better solution.

There are multiple other bugs about this, but I can't immediately find them.

Looks like we already use something similar, via soup_ltd_domain_is_public_suffix:

https://publicsuffix.org/

It's maintained in libsoup, where it hasn't been updated in three years:

https://git.gnome.org/browse/libsoup/tree/data/effective_tld_names.dat

It carries an explicit warning that it's wrong to use it to decide whether to auto-search:

"""Some people use the PSL to determine what is a valid domain name and what isn't. This is dangerous, particularly in these days where new gTLDs are arriving at a rapid pace, if your software does not regularly receive PSL updates, because it will erroneously think new gTLDs are not valid. The DNS is the proper source for this information. If you must use it for this purpose, please do not bake static copies of the PSL into your software with no update mechanism."""

Indeed, the list in libsoup has not been updated in three years.

(In reply to Michael Catanzaro from comment #2)
> Looks like we already use something similar, via
> soup_ltd_domain_is_public_suffix:

Twiddled that, I mean soup_tld_domain_is_public_suffix()

(In reply to Michael Catanzaro from comment #2)
> Indeed, the list in libsoup has not been updated in three years.

Bug #769650

(In reply to Michael Catanzaro from comment #2)
> The DNS is the proper source for this information.

You know, this should have been really blindingly obvious... but I didn't think of that.

*** Bug 771990 has been marked as a duplicate of this bug. ***

Created attachment 368225 [details] [review]
Add check for dns name resolution

Patch adds dns name resolution check.

*** Bug 741294 has been marked as a duplicate of this bug. ***

Review of attachment 368225 [details] [review]:

::: embed/ephy-embed-utils.c
@@ +207,3 @@
+  resolver = g_resolver_get_default ();
+
+  list = g_resolver_lookup_by_name (resolver, address, NULL, NULL);

We can't hang the UI process to do a DNS lookup. This needs to use g_resolver_lookup_by_name_async.

And that's going to complicate everything, because then we need to turn ephy_embed_utils_address_is_valid into an asynchronous function. That will make it much harder to call. Fortunately, it's only used in one place: ephy_embed_utils_normalize_or_autosearch_address. But then we have to turn that into an async function, and it's used in a couple more places (ephy-location-controller.c, ephy-notebook.c, ephy-web-view-test.c.) This might take some effort to get right.

Lastly, we have one privacy problem: this is a dangerous information leak if the user is using certain SOCKS proxies where DNS resolution is expected to be performed by the proxy rather than locally (e.g. tor). In that case, it's better to fail and force the user to manually type in the URI scheme "https://" rather than make a DNS request. So we should check if a proxy is in use before doing this. And that also has to be done asynchronously. Look in Source/WebCore/platform/network/soup/DNSSoup.cpp for an example of how to do this.

@@ +233,3 @@
            g_regex_match (get_non_search_regex (), address, 0, NULL) ||
            is_public_domain (address) ||
+           is_bang_search (address) ||

Probably should check is_bang_search at the top, after "scheme ||", since that's less work than checking for a file on disk, regex matching, parsing the public suffic list, or making a DNS query.

@@ +234,3 @@
            is_public_domain (address) ||
+           is_bang_search (address) ||
+           is_dns_resolvable (address);

I guess you've checked to make sure this code is only ever reached when the address looks like a website that's missing from the public suffix list, and not a search term, right? We don't want to be sending the user's search terms out in plaintext to the DNS server.

(In reply to Michael Catanzaro from comment #9)
> In that case, it's
> better to fail and force the user to manually type in the URI scheme
> "https://" rather than make a DNS request.

I wonder how Firefox and Chrome handle this.

(In reply to Michael Catanzaro from comment #10)
> I wonder how Firefox and Chrome handle this.

This is what we need to examine and figure out.

(In reply to Michael Catanzaro from comment #11)
> This is what we need to examine and figure out.

We need to understand this to reenable DNS prefetch in the location entry, as well, which is probably quite important for performance (bug #661455)....

Current findings:
 - Firefox stops prefetch as soon as a proxy is in use
 - Resolving flow:
   - Special check for .onion: if true, UNKNOWN_HOST
   - Whitelist check if it is a valid hostname, if not UNKNOWN_HOST
   - Try to convert to IP
   - if no match could be found:
     - start dns lookup (async)

-- GitLab Migration Automatic Message --

This bug has been migrated to GNOME's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.gnome.org/GNOME/epiphany/issues/282.