GNOME Bugzilla – Bug 748636
[IMAPx] Disable LIST-EXTENDED extension for Cyrus servers
Last modified: 2015-10-08 07:12:02 UTC
My initial set-up was two accounts, a Courier IMAP server and a Cyrus IMAP server (with some shared folders that sit outside of the INBOX namespace). Both of them set to use IMAP idle. BACKGROUND: Evolution 2.32.x and earlier worked great! Evolution 3.8.x and 3.10.x were OK, but needed a patch to work with the shared folders, however, they had frequent issues with showing Shared Folders and then telling me they didn't exist when I tried copying in to them or opening them - this appears to have been resolved in 3.12.x along with the need for the patch. OUTLINE OF CURRENT ISSUE: The IMAP connections appear to 'freeze' and then suddenly reconnect 10's of minutes later. Sometimes during these reconnects the Shared folders disappear from the UI. I've tried starting evolution with the following "CAMEL_DEBUG=imapx:io evolution >& evolution_cyrus_only.log" and found that with IMAP idle enabled the following occurred: 1) I use email 'normally' for approx 10 minutes (reading / writing / moving and deleting messages). 2) I leave evolution alone and carry on with other work (around 10:50am) - at this point the log shows: [imapx:B] I/O: '+ idling' [imapx:A] I/O: 'A00087 IDLE' [imapx:A] I/O: '+ entering idle mode' 3) Emails continue to arrive in my other IMAP client (Aquamail on Android also using IMAP IDLE functionality). 4) At 11:13 more data appears in the log: [imapx:B] Data read failed with error 'Error receiving data: Connection timed out' [imapx:A] Data read failed with error 'Error receiving data: Connection timed out' [imapx:B] I/O: '' [imapx:A] I/O: '* OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE AUTH=LOGIN AUTH=PLAIN SASL-IR] mta1.zednax.com Cyrus IMAP v2.4.17 server ready' [imapx:A] I/O: 'A00088 LOGIN ...' ... 5) The UI updates with new emails, but the Shared folder hsave disappeared (the shared folders are reported as * LIST (\Subscribed \HasChildren) "." "Shared Folders" (CHILDINFO (SUBSCRIBED))^M - there is also the INBOX.* namespace which doesn't disappear. ---- To try and simplify things I temporarily disabled IMAP idle, lowered the checking for email setting to every 2 minutes and disabled one of the accounts (the Courier IMAP account). The behaviour then becomes: 1) Evolution first opens; new emails are shown and the Shared Folders are present. 2) After approx. 2 minutes the shared folders disappear and the logs show: [imapx:A] I/O: 'A00045 LIST "" INBOX RETURN (CHILDREN SUBSCRIBED)' [imapx:A] I/O: '* LIST (\Subscribed \HasChildren) "." INBOX (CHILDINFO (SUBSCRIBED))^M A00045 OK Completed (0.014 secs 614 calls)' [imapx:A] Ignoring timeout error, nothing was waiting (original error: Socket I/O timed out)
Thanks for a bug report. What is your exact evolution(-data-server) version, please? What you've described looks to me as bug #743533 and commit [1], which didn't make it into 3.12.11. By the way, the set connection timeout is 90 seconds, thus using shorter automatic update interval will avoid the timeout issue. Could you verify that the both changes fix the issue for you, please? [1] https://git.gnome.org/browse/evolution-data-server/commit/?id=94c9a1f27c42
I was running 3.12.11 I have: 1) reduced the check interval to 1 minute. 2) Installed 3.16.1 of both Evolution and evolution-data-server In both cases the original issue still occurs (after the first mail check the shared folders disappear). I note in the bug that the user had offline sync of remote folders set - I didn't; so have tried setting that option, but it makes no difference and the shared folders still disappear for me. Sorry...
Thanks for the update. 3.16.1 definitely worked properly for me. I'll retest with my cyrus server to see whether I'll be able to reproduce it here.
I tried this with my setup and I see a difference in cyrus and dovecot (I do not have a courier server). Evolution's IMAPx asks both (almost) identically, it is: > A00005 LIST "" "Public.Groups.*" RETURN (CHILDREN SUBSCRIBED STATUS (MESSAGES UNSEEN UIDVALIDITY UIDNEXT HIGHESTMODSEQ) SPECIAL-USE) > * LIST (\Subscribed \HasNoChildren) "." Public.Groups.group-2 > * LIST (\HasNoChildren \UnMarked) "." Public.Groups.group-1 > A00005 OK List completed. in case of Dovecot and in case of cyrus: > A00021 LIST "" "Shared Folders.*" RETURN (CHILDREN SUBSCRIBED) > * LIST (\HasNoChildren) "." "Shared Folders.shared.folder > A00021 OK Completed (0.000 secs 1 calls) Notice the difference in the List response flags, namely the \Subscribed flag. That is crucial for evolution. As you can see, I'm subscribed for Public.Groups.group-2, but not for Public.Groups.group-1 on the dovecot server. The cyrus pretends the Shared Folders.shared.folder is not subscribed, thus evolution works as it. Interestingly, instead of using the LIST-EXTENDED extension, doing it the old way of using LIST and LSUB (the second to get list of only subscribed folders) cyrus returns folders correctly. > A00005 LSUB "" "Shared Folders.*" > * LSUB () "." "Shared Folders.shared.folder" > A00005 OK Completed (0.000 secs 1 calls) There is not needed the \Subscribed flag, because the LSUB itself means that the listed folders are subscribed. I can disable the LIST-EXTENDED extension for cyrus servers, but the right fix would be to let the cyrus developers know to fix the response.
That's great; I'm going to see if we can migrate to Dovecot in the short term. Would you like me to file a bug report with the Cyrus devs or will you do that? Thanks!!
So I did so, the LIST-EXTENDED is now disabled for Cyrus servers. The difference can be seen when you enter Manage Subscriptions, the shared folder, if subscribed, will be marked as subscribed, while without the fix the shared folder will be always shown as unsubscribed. Created commit e68fa19 in eds master (3.17.2+) Created commit adf18c4 in eds gnome-3-16 (3.16.2+)
(In reply to Alex Masidlover from comment #5) > I'm going to see if we can migrate to Dovecot in the short term. It seems a bit drastic to me :) If you can, then try to apply the above commit and compile evolution-data-server, whether it'll help you the same as it did for me (then you might not need to change your server infrastructure). > Would you like me to file a bug report with the Cyrus devs or will you do > that? I just committed a workaround into the IMAPx code (see above). I do not have any account in their bugzilla, thus I'll appreciate if you could take care of it.
I think the patch applied ok: lamech evolution-data-server-3.16.1 # patch -p1 < list_extended_fix.patch patching file camel/providers/imapx/camel-imapx-server.c Hunk #2 succeeded at 3420 (offset -6 lines). Hunk #3 succeeded at 8145 (offset -35 lines). Hunk #4 succeeded at 8332 (offset -35 lines). Hunk #5 succeeded at 8355 (offset -35 lines). patching file camel/providers/imapx/camel-imapx-utils.c However, the Shared Folders are still disappearing. I've tried grepping the CAMEL DEBUG output for LSUB, but it seems I was getting LSUB commands in the debugs before I applied the patch as well as after, so I'm not really sure what I'm looking for. I'm also still seeing timeout errors even with the check set to every minute: B00132 OK LSUB completed' [imapx:A] Ignoring timeout error, nothing was waiting (original error: Socket I/O timed out) [imapx:A] Ignoring timeout error, nothing was waiting (original error: Socket I/O timed out) [imapx:B] I/O: 'B00133 STATUS INBOX (MESSAGES UNSEEN UIDVALIDITY UIDNEXT HIGHESTMODSEQ)' [imapx:B] I/O: 'B00134 STATUS INBOX (MESSAGES UNSEEN UIDVALIDITY UIDNEXT)' [imapx:B] I/O: '* STATUS INBOX (MESSAGES 26 UIDNEXT 501962 UIDVALIDITY 1307287298 UNSEEN 1 HIGHESTMODSEQ 599678)^M B00133 OK Completed' [imapx:B] I/O: '* STATUS "INBOX" (MESSAGES 2464 UIDNEXT 18106 UIDVALIDITY 1120750628 UNSEEN 1)^M B00134 OK STATUS Completed.' [imapx:A] Ignoring timeout error, nothing was waiting (original error: Socket I/O timed out) [imapx:A] Ignoring timeout error, nothing was waiting (original error: Socket I/O timed out)
Some further info - I've migrated to Dovecot over the weekend; which solved my disappearing shared folders, however I discovered at the last minute that I'd not set the main user namespace to INBOX/ on the server. Once I added the prefix=INBOX/ setting the folders started disappearing again - I have therefore removed it.
(In reply to Alex Masidlover from comment #8) > However, the Shared Folders are still disappearing. I've tried grepping the > CAMEL DEBUG output for LSUB, but it seems I was getting LSUB commands in the > debugs before I applied the patch as well as after, so I'm not really sure > what I'm looking for. Does the LSUB return the folders you see disappearing? > I'm also still seeing timeout errors even with the check set to every minute: > > [imapx:A] Ignoring timeout error, nothing was waiting (original error: > Socket I/O timed out) These timeouts are expected and correct. (In reply to Alex Masidlover from comment #9) > Some further info - I've migrated to Dovecot over the weekend; which solved > my disappearing shared folders, however I discovered at the last minute that > I'd not set the main user namespace to INBOX/ on the server. Once I added > the prefix=INBOX/ setting the folders started disappearing again - I have > therefore removed it. Is that server setting done with the Dovecot server, or with the Cyrus?
> Does the LSUB return the folders you see disappearing? My reading of the log was that it did, yes. > Is that server setting done with the Dovecot server, or with the Cyrus? Dovecot, I'd retired the Cyrus server by that point.
I just realized that I missed one place here, which I fixed with the below change. The thing is that unauthenticated CAPABILITY contains the 'Cyrus' "keyword", but the authenticated doesn't, while the IMAPx checked only in that authenticated response, thus this didn't work well. I do not know why it did seem to work here though. In any case, it's corrected now and explains the trouble you had. A workaround would be to run Folder->Subscriptions and subscribe to the Inbox folder in the Cyrus server (I do not know why they have it unsubscribed by default). Created commit bf3dad9 in eds master (3.19.1+) Created commit 132467c in eds gnome-3-18 (3.18.1+)