GNOME Bugzilla – Bug 61263
Experiment to speed up IMAP folder scannning
Last modified: 2009-08-15 18:40:50 UTC
This is related to bug #60785. Recently I have been getting *very* slow ping times to my IMAP server (up to 2 seconds--and it's about 2 miles away!). Balsa takes minutes to start up, and then dies an instant after displaying the main window (firing it up immediately, it never dies the same way--weird!). So...how to speed up the folder scanning when a round-trip message takes a second or two? Balsa asks the server to list all subfolders of all mailboxes, and current code *waits for the response* for one folder before asking about the next. This isn't necessary! We need the list of folders at one depth in the tree, to know where to look for the next level, but we can start several requests at once. This patch, which is strictly experimental (that is, it works for me, but ymmv), sends multiple requests, picking up responses as they come, and waits for everything to come back only when absolutely necessary--in this case, after scanning all folders at a given level. It builds a list of outstanding requests and checks them off as they are completed. Affected files: - libbalsa/folder-scanners.c: let the libmutt code that the current series of requests is finished - libmutt/imap/browse.c: much rewriting - libmutt/browser.h: add seq_list field to struct browser_state
Created attachment 5703 [details] [review] the patch
Broke somewhat for me on uw-imap. It created 7 nameless folders with no leafs under the root. This happened on 2 diferent machines with diferent versions of uw-imapd. Parsed the existing folders correctly as far as i can tell. Worked with myrealbox.com (novell server)
Created attachment 5836 [details] [review] Another experiment
Hazardous content--highly experimental--yhbw. The attachment labeled `Another experiment' reflects much debate about the imap protocol, especially about `subscribed' mailboxes and the various servers' idiosyncracies. It's very fast when asked to list `subscribed folders only'. The strategy in this case is to use LSUB prefix/* to see *all* of the subscribed folders with a single command. As each response is received, a LIST path/% command is issued, where `path' is the path to the latest folder with the last level removed; this ensures that we get a LIST response for the current folder, and the flags are ORed with those, if any, on the original LSUB response. The folder tree is then built, with synthetic nodes to fill in any missing layers in the hierarchy. There's also some record-keeping and other tweaking to avoid redundant commands. The net result is that network traffic and latency is minimized: commands are reduced by up to 90%, and many overlap; there are two responses for each subscribed folder, plus one for any unsubscribed folder in the same parent folder as a subscribed folder. It's hard to see how to reduce this count. The strategy for `all folders' is more cautious: use LIST path/% to descend the tree recursively, one level at a time, rather than LIST prefix/*. This is to avoid: (i) endless responses when the mailstore has circular references; and (ii) voluminous responses when the user has, for instance, the whole of usenet news in the namespace. Again, the commands are overlapped as far as possible, to reduce latency. However, LIST path/% is issued for every leaf node, with no responses naturally, and this generates many command/response sequences. Affected files are much as for the previous attempt, with some minor changes to src/mailbox-node.c to cope more sensibly with the case of a nonnull prefix that's a real mailbox.
Created attachment 5908 [details] [review] patch against cvs of Oct 25 '01
The above patch implements the same changes, wrt current (Oct 25 '01) cvs.
Is this ready for CVS?
I used this code for a while with a Cyrus server with no problems, and the strategy was designed to work with uw-imap also, but I don't know if anyone tested that. I think it has some real benefits, but... Carlos mentioned something about getting asynch imap i/o in libmutt a week or two ago. If something more general is in the works, I'd suggest waiting for it. These changes make browsing work quite differently for Balsa than for mutt, so there may be some difficulties down the road keeping things in sync. I guess there are some issues here that I don't feel competent to comment on.
The following patch is a cleaned-up version of an earlier experiment, which browses the folder tree only far enough to find out whether the viewable nodes have children. It works by restricting the folder browsing to depth 1 (below the current node), with further browsing triggered by exposing more of the tree. The depth can be specified through the prefs page, so setting it to 7 would reproduce the current behavior, while setting it to 1 (the lowest value) gives the fastest start with the least browsing. The option is on the `Startup' page, and applies to any IMAP account. It should perhaps be server-specific, and located in the folder-conf dialog; I chose not to put it there, as that dialog is getting pretty crowded, but that could be fixed. A corresponding update to help/C/balsa.sgml is included (with some other minor IMAP changes). Files: help/C/balsa.sgml: documentation. libbalsa/folder-scanners.c: add state.scanned--set it, pass it to the handlers; libbalsa/folder-scanners.h: change handler prototype. libmutt/browser.h: new member `scanned' in struct browser_state. src/balsa-app.[hc]: add balsa_app.imap_scan_depth src/balsa-mblist.[hc]: change "tree-expand" callback to connect-after; unrelated cleanup of balsa_mblist_repopulate; add balsa_mblist_scan_children, for appending subtrees when needed; add mblist_scan_mailbox_node as the corresponding public interface. src/mailbox-node.[hc]: add mbnode->scanned, and set it; pass balsa_app.imap_scan_depth to libbalsa_scanner_imap_dir; call mblist_scan_mailbox_node. src/pref-manager.c: manage balsa_app.imap_scan_depth. src/save-restore.c: ditto.
Created attachment 6131 [details] [review] patch against current cvs
Attachment 6131 [details] is a bit untidy, with some code transplanted from mailbox-node.c to balsa-mblist.c. The following is cleaner, and handles rescanning properly. It does delete a `g_return_' test that doesn't seem necessary--if I'm missing something, please let me know!
Created attachment 6161 [details] [review] Patch against current cvs
Attachment 6161 [details] commited, thanks. This hopefully resolves the issue(?).
I close this report.
balsa-1.3.5 and 2.0.0 released, closing this report for good.