After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 796364 - Consider whether Safe Browsing requests to threatLists.list should be rate-limited
Consider whether Safe Browsing requests to threatLists.list should be rate-li...
Status: RESOLVED WONTFIX
Product: epiphany
Classification: Core
Component: General
git master
Other Linux
: Normal normal
: ---
Assigned To: Epiphany Maintainers
Epiphany Maintainers
Depends on:
Blocks:
 
 
Reported: 2018-05-23 17:31 UTC by Michael Catanzaro
Modified: 2018-05-24 02:20 UTC
See Also:
GNOME target: ---
GNOME version: ---



Description Michael Catanzaro 2018-05-23 17:31:47 UTC
We have a fairly high number of Safe Browsing API errors. Almost all the errors are HTTP 429 ("too many requests") for the google.security.safebrowsing.v4.SafeBrowsing.ListThreatLists endpoint. We should implement better client-side rate limiting to ensure Epiphany doesn't spam Google with repeated requests. Should make sure the other requests are rate-limited as well.
Comment 1 Michael Catanzaro 2018-05-23 17:42:33 UTC
BTW, I expect this problem will disappear once bug #796324 is resolved, but we should have a client side limit anyway.
Comment 2 Gabriel Ivașcu 2018-05-23 18:48:17 UTC
Epiphany implements the back-off algorithm described at https://developers.google.com/safe-browsing/v4/request-frequency#back-off-mode for its requests (the back-off period is stored in the local database so it's even persistent across different runs). However, as the above link says, the back-off protocol is applied only to fullHashes.find and threatListUpdates.fetch methods which are part of the Update API. This doesn't include the threatLists.list method (https://developers.google.com/safe-browsing/v4/reference/rest/v4/threatLists/list), which apparently falls into the "other requests" category. Having said this, I'd say that we can do a slight change in our code to apply the back-off mode to threatLists.list too, but I reckon it won't matter that much since 10,000 requests per day is a fairly low limit which will be hit regardless.
Comment 3 Michael Catanzaro 2018-05-24 02:20:56 UTC
Hm, I do see that Google's documentation says the rate limiting applies to the functions that you have already implemented it for, and not for threatLists.list. I guess that's fine, then.

I guess threatLists.list is probably only called too often currently because the quota is being hit, so the browser keeps trying again? In that case, getting the quota fixed should reduce our error rate.