After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 770656 - gdata: Query Google Drive in smaller chunks instead of getting everything all at once
gdata: Query Google Drive in smaller chunks instead of getting everything all...
Status: RESOLVED FIXED
Product: gnome-online-miners
Classification: Applications
Component: general
3.20.x
Other All
: Normal normal
: ---
Assigned To: GNOME Online Miners maintainer(s)
GNOME Online Miners maintainer(s)
: 733642 741472 (view as bug list)
Depends on:
Blocks:
 
 
Reported: 2016-08-31 16:25 UTC by Debarshi Ray
Modified: 2016-09-02 08:22 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
gdata: Consolidate the return sites (1.42 KB, patch)
2016-08-31 16:28 UTC, Debarshi Ray
committed Details | Review
gdata: Use a separate GError variable for local errors (1.81 KB, patch)
2016-08-31 16:28 UTC, Debarshi Ray
committed Details | Review
gdata: Page through the GDataDocumentsFeed (3.25 KB, patch)
2016-08-31 16:29 UTC, Debarshi Ray
committed Details | Review

Description Debarshi Ray 2016-08-31 16:25:57 UTC
There can be a very large number of documents in a Google Drive. Getting all of them at once is not a good idea because there can be an upper bound on how long the response from the server can be. This is also what the GVfs backend does:
https://git.gnome.org/browse/gvfs/tree/daemon/gvfsbackendgoogle.c#n539

I am not sure about the best error handling strategy. GVfs only treats the failure to get the first page as fatal. Failure to get any subsequent page terminates the operation but the failure is not reported to the caller. This is also what I have implemented, but I am open to better ideas.
Comment 1 Debarshi Ray 2016-08-31 16:28:26 UTC
Created attachment 334539 [details] [review]
gdata: Consolidate the return sites
Comment 2 Debarshi Ray 2016-08-31 16:28:59 UTC
Created attachment 334540 [details] [review]
gdata: Use a separate GError variable for local errors
Comment 3 Debarshi Ray 2016-08-31 16:29:27 UTC
Created attachment 334541 [details] [review]
gdata: Page through the GDataDocumentsFeed
Comment 4 Debarshi Ray 2016-08-31 16:32:04 UTC
*** Bug 741472 has been marked as a duplicate of this bug. ***
Comment 5 Debarshi Ray 2016-08-31 16:32:12 UTC
*** Bug 733642 has been marked as a duplicate of this bug. ***
Comment 6 Debarshi Ray 2016-08-31 16:33:52 UTC
See the duplicate bug 733642 for concerns about performance and exceeding the server's response limit.
Comment 7 Debarshi Ray 2016-09-02 08:22:38 UTC
I have pushed these to master so that I can release 3.20.1. Let me know if I have missed something.