GNOME Bugzilla – Bug 796364
Consider whether Safe Browsing requests to threatLists.list should be rate-limited
Last modified: 2018-05-24 02:20:56 UTC
We have a fairly high number of Safe Browsing API errors. Almost all the errors are HTTP 429 ("too many requests") for the google.security.safebrowsing.v4.SafeBrowsing.ListThreatLists endpoint. We should implement better client-side rate limiting to ensure Epiphany doesn't spam Google with repeated requests. Should make sure the other requests are rate-limited as well.
BTW, I expect this problem will disappear once bug #796324 is resolved, but we should have a client side limit anyway.
Epiphany implements the back-off algorithm described at https://developers.google.com/safe-browsing/v4/request-frequency#back-off-mode for its requests (the back-off period is stored in the local database so it's even persistent across different runs). However, as the above link says, the back-off protocol is applied only to fullHashes.find and threatListUpdates.fetch methods which are part of the Update API. This doesn't include the threatLists.list method (https://developers.google.com/safe-browsing/v4/reference/rest/v4/threatLists/list), which apparently falls into the "other requests" category. Having said this, I'd say that we can do a slight change in our code to apply the back-off mode to threatLists.list too, but I reckon it won't matter that much since 10,000 requests per day is a fairly low limit which will be hit regardless.
Hm, I do see that Google's documentation says the rate limiting applies to the functions that you have already implemented it for, and not for threatLists.list. I guess that's fine, then. I guess threatLists.list is probably only called too often currently because the quota is being hit, so the browser keeps trying again? In that case, getting the quota fixed should reduce our error rate.