This seems to work fine for me. I modified my account to have all "Check for new..." unchecked, exited and run evolution again, sent an email to my account, waited a minute, hit "send & receive" and the Inbox I wasn't in yet had been updated with one new message. I'm at commit 4a9bf4d on evo, but I didn't see any related changes between this and 2.28.1 in file where I would expect it
(mail-send-recv.c).

Created attachment 149538 [details]
debug patch

for evolution;

when you press send/receive it prints which accounts it's working with, and what it does with that. There is a line
> mail_send_receive: updating on imap:...
for my IMAP account configured as you descried above.

It still happens on both my home system and work system (just verified, and tested at work with the latest git code updated this morning).

I'll try the patch; thanks Milan.

OK, here's the deal.  First a few things that might be different about my config than yours.  In addition to my IMAP account I have a few accounts created with different email addresses, but with the mail protocol of "None"; I just use these to select different "From:" addresses when I send mail.  Second, I can't contact my IMAP server directly from behind my company's firewall.  So, I create a tunnel using SSH to another system:

ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=15 -N -T -L 40993:<realserver>:993 psmith@<intermediateserver>

Then I configure my IMAP server address as "localhost:40993", so it uses a specific port.  This all works fine with Evo 2.28.x and below, and I can get my mail if I select my INBOX folder (based on the debugging you provided, that uses a completely different path for retrieving the contents of the folder than "send/receive" does).


Here's what happens when I click send/receive:

(evolution:19026): evolution-mail-WARNING **: Couldn't get service: : No provider available for protocol ''
removing cache for  INBOX 732 0x25890a0
done .. now 1
removing cache for  INBOX 1160 0x2f91230
done .. now 0
CamelException.setv(0x7fff843a3d40, 302, 'No provider available for protocol ''')
build_dialog: adding 'imap://myimapaccount@localhost:40993/;command=ssh%20-C%20-l%20%25u%20%25h%20exec%20/usr/sbin/imapd;use_ssl=always;imap_custom_headers;filter_junk'
CamelException.setv(0x7fff843a3610, 302, 'No provider available for protocol ''')
build_dialog: skipping ''
CamelException.setv(0x7fff843a3610, 302, 'No provider available for protocol ''')
build_dialog: skipping ''
mail_send_receive: updating on imap://myimapaccount@localhost:40993/;command=ssh%20-C%20-l%20%25u%20%25h%20exec%20/usr/sbin/imapd;use_ssl=always;imap_custom_headers;filter_junk
get folder info online
get folder info offline
Thread 0 >
Get folder info(0x2654620:imap://myimapaccount@localhost:40993/, '<null>') =
  uri: imap://myimapaccount@localhost:40993/INBOX;noselect=yes
  full_name: INBOX
  flags: 00000000
    uri: imap://myimapaccount@localhost:40993/INBOX/Drafts
    full_name: INBOX/Drafts
    flags: 00000018
    uri: imap://myimapaccount@localhost:40993/INBOX/Sent
    full_name: INBOX/Sent
    flags: 00000018
    uri: imap://myimapaccount@localhost:40993/INBOX/Trash
    full_name: INBOX/Trash
    flags: 00000018
  uri: imap://myimapaccount@localhost:40993/.%23evolution/Trash
  full_name: .#evolution/Trash
  flags: 00000cf2
  uri: imap://myimapaccount@localhost:40993/.%23evolution/Junk
  full_name: .#evolution/Junk
  flags: 000010f2
< 0 >
build_dialog: adding 'imap://myimapaccount@localhost:40993/;command=ssh%20-C%20-l%20%25u%20%25h%20exec%20/usr/sbin/imapd;use_ssl=always;imap_custom_headers;filter_junk'
CamelException.setv(0x7fff843a3610, 302, 'No provider available for protocol ''')
build_dialog: skipping ''
CamelException.setv(0x7fff843a3610, 302, 'No provider available for protocol ''')
build_dialog: skipping ''
mail_send_receive: updating on imap://myimapaccount@localhost:40993/;command=ssh%20-C%20-l%20%25u%20%25h%20exec%20/usr/sbin/imapd;use_ssl=always;imap_custom_headers;filter_junk
get folder info online
get folder info offline
Thread 0 >
Get folder info(0x2654620:imap://myimapaccount@localhost:40993/, '<null>') =
  uri: imap://myimapaccount@localhost:40993/INBOX;noselect=yes
  full_name: INBOX
  flags: 00000000
    uri: imap://myimapaccount@localhost:40993/INBOX/Drafts
    full_name: INBOX/Drafts
    flags: 00000018
    uri: imap://myimapaccount@localhost:40993/INBOX/Sent
    full_name: INBOX/Sent
    flags: 00000018
    uri: imap://myimapaccount@localhost:40993/INBOX/Trash
    full_name: INBOX/Trash
    flags: 00000018
  uri: imap://myimapaccount@localhost:40993/.%23evolution/Trash
  full_name: .#evolution/Trash
  flags: 00000cf2
  uri: imap://myimapaccount@localhost:40993/.%23evolution/Junk
  full_name: .#evolution/Junk
  flags: 000010f2
< 0 >

So, it SEEMS to be doing something but yet no mail is retrieved.  Again, if I click on the INBOX folder right after I did this I actually downloaded 5 new emails.

I should add that on my system at home, where I have this same problem, I do NOT have to use the SSL tunnel hack but I still see this problem.  So, I don't think it has to do with that.  However I do have extra "None" protocol accounts on both systems so they're similar in that way.

OK, so it does something, as you said, but it fails to call the real update on your folder for some reason.
> mail_send_receive: updating on imap://myimapaccount@localhost:40993/;...
Those skipped are correct, as you use them for sending only. Let me add few more debug output to the patch (and maybe to IMAP code in eds as well).

OK Thanks Milan.  I'll wait for your enhanced debugging patch.

Alternatively, or additionally, if you have some pointers of where to wait and what to look for I can wield GDB directly and see what might be happening.

> Get folder info(0x2654620:imap://myimapaccount@localhost:40993/, '<null>') =
>   uri: imap://myimapaccount@localhost:40993/INBOX;noselect=yes
>   full_name: INBOX
>   flags: 00000000

aha, looking deeper in the code, I see that the Inbox is always refreshed only when it has set proper flags on the folder info. It should be 0, but (1 << 10), which is CAMEL_FOLDER_TYPE_INBOX. The flag about noselect=yes on the Inbox is also very strange to me.

It seems like your local IMAP server is returning \NoSelect on the Inbox, which might happen probably only when you are not subscribed to the Inbox itself. Could you check your subscription in Folder->Subscriptions for this account, please?

Huh.  I never even heard of that before, but sure enough the Inbox folder is not subscribed.  Why does this work with Evo 2.28?

So, I selected this folder, closed the subscriptions window, then "send/receive".

Oh no!!  Send/receive did work this time (I had a few more junk emails show up), but now all my folders except for "Junk" and "Trash" in that IMAP account have disappeared.

I reopened folder->subscriptions and everything is still checked.  I re-ran "Send/Receive" and my Inbox folder came back, but all my other folders are still missing.

Also, when I look in my "Inbox" folder, the ONLY thing shown there are the 7 new emails.  All the other contents of that folder are not shown.  If I look at the folder properties for the inbox, it says I have 752 total messages with 7 unread, but the summary only shows me the unread messages.

And none of my other folders are visible.

I went back again and unselected the Inbox folder but left the other subfolders of Inbox selected.  Again, my Inbox folder disappeared from the folder list.  I selected Send/Receive and it re-filtered my entire inbox, which took about 5 minutes.  But, my Inbox did not come back; I still see only Junk and Trash.

Not sure what to do now; I guess I'll delete this account and recreate it.

Hm.  I disabled the account (preferences -> mail accounts / unchecked "enabled") then re-enabled it, and everything came back.

Then I tried again to subscribe to the inbox, and this time my folders didn't disappear.

Now it seems to be working OK.

Bizarre!

I still don't know why that folder was not selected, and why it worked OK with Evo 2.28.

(In reply to comment #9)
> Huh.  I never even heard of that before, but sure enough the Inbox folder is
> not subscribed.  Why does this work with Evo 2.28?

I also didn't hear about that before, I realized that from the code. Why does it work in 2.28? Maybe some change either in camel or in camel/providers/imap in eds caused this. What I'm thinking also about is some weird corruption of store's folder cache and its information about subscribed folders. I believe that disable/enable account refreshes this local folder cache, which then is "repaired". I only believe it and I agree it's rather a guess.

Am I right that the folder was supposed to be subscribed? As if so, then we found the issue here (even not what caused that). I've not much idea how to hunt this without some reproducible steps, what do you think?

Whoops, it happened again: I clicked Send/Receive and all my folders disappeared again.  I then disabled/re-enabled the account and all the folders came back, BUT my INBOX only shows the 8 new emails that were downloaded, not all my emails.

As of now I haven't figured out how to get back all my email again; no amount of enabling/disabling or re-running Send/Receive shows them to me.  I can probably get them back by deleting and re-creating the account.  It seemed that last time they came back as part of some process which re-ran the filtering on all of them, so something made them all appear as new to the local Evo instance (but not new on the server; they were not considered new messages).


There's something really unhappy about Evo 2.29 when I have this Inbox folder subscribed, that's for sure.

Just for clarification, I have one of those weird IMAP server configs where my Inbox folder, where all my new mail appears, has all my other folders as children of it.  I don't know exactly how this happened and I don't know how to change it; I've seen various discussions on the list but I've not paid enough attention.  It seems to work fine for years, until this problem, so I've had no incentive to change it.  But maybe this kind of IMAP folder hierarchy is somehow unexpected by Evo?  Is there some information I can give about this that would help you (I did excise most of my personal folders from the log above so they're not shown--in retrospect that was probably not a good idea)?

I would guess no, Evolution isn't tight to any hierarchy and should support all hierarchies on the IMAP server, even that your with Inbox and couple subfolders of it. There might be something with that subscribe/unsubscribe. Maybe the server forces your Inbox to be unsubscribed? It's also possible to do. The local IMAP is just an SSH tunnel to real IMAP server, right? Do you what what the IMAP server it is?

Please move somewhere your ~/.evolution/mail/imap/<account>/folders.db file, together with .ev-store-summary (where is stored information about subscriptions and available folders), and then run Evolution and let it recreate this information from scratch. It might take some time, depending on your folder sizes.

Yesterday at work I got into an impossible state: all attempts to synchronize the folders resulted in an infinite timeout of some sort: Evo just sat there trying to load the folder.  I tried deleting the account from Evo, then recreating it, but that didn't help.  Then I deleted the account from Evo, then went in and deleted the contents of the ~/.evolution directory related to that account, but that didn't help either.  I couldn't even run the test to see what authorization modes were supported, when I created the account: it hung as well.  I'm still not sure what is going on there: I'll try again when I get back to work.  I have a suspicion its to do with the password as I was never asked for a password, even when recreating the account.

I can log into my ISP and looking around it seems that my ISP is running dovecot as the IMAP server.  It seems to be dovecot 1.1.11.

If you like I can give you an IMAP account on my server so you can try yourself; I don't know if this will be helpful or not.

I'll try some more things today.

OK, the infinite timeout was my fault: I forgot I needed to enable SSL to connect.  It's too bad I didn't get any kind of dialog or anything that might have led me in the right direction, but I guess that's the way it is; probably Evo doesn't get any useful error back either.

Anyway, I'm beginning to think we have two different issues: the original and this new one that causes the folders for this account to disappear.  I say that because when I recreated my account I realized that again the Junk filtering options had defaulted to "off" (is that a bug too?  Not sure).  When I went to turn them on I selected "Apply filters to new messages in INBOX on this server", and "Check new messages for Junk contents", and also "Only check for Junk messages in the INBOX folder", and when I clicked "OK" all my folders disappeared again, just as above!

I had to disable/re-enable the account again to get them back.

I did just notice that my top-level "INBOX" folder is subscribed now, so it might still be some strange interaction with subscriptions that's causing these folders to disappear.

(In reply to comment #14)
> If you like I can give you an IMAP account on my server so you can try
> yourself; I don't know if this will be helpful or not.

If possible then yes, I would like to try my environment with your server, running actual master of all gtkhtml, eds, evo, eex, ema. Please try to create there some similar folder structure as you have, at least the first level with the same count of subfolders, and few second level folders, so there will be some hierarchy. Please give me some commands how to have configured evo and running commands to have as close setup as you have. Maybe try if you can reproduce the issue on your machine with a new account as well, it seems you have with it quite many issues. Thanks in advance. (what about your quota on a home directory and disk free space?)

Hi Milan.  Sorry for the delay; I had vacation and lots of other things.

I'll email you some info privately on how to connect to my server with a guest account.  I was just able to reproduce the problem very easily, by having the Junk folder selected (I don't think this matters, I'm just being complete) then sending this account an email then waiting a few minutes to be sure it had arrived, then pressing Send/Receive.  It didn't show any new mail.  I selected INBOX, and it downloaded the email and showed it.

Thanks for the information and access there. I see very similar behaviour on actual master. I was tu summarize here, maybe open/find new bugs to have it in separate reports. So, we are having two issues here:
a) unresponsive IMAP Inbox - doesn't show new message and similar
b) disappearing folders on send/receive or other actions

ad a)
  RFC 3501 - section 7.2.2. says
      \Noselect
         It is not possible to use this name as a selectable mailbox.

I believe that all the issue is with that /NoSelect returned from the server on Inbox. Though after more investigation on this I realized that it's probably because of some issue with my patch to namespaces (I'm investigating more), because 2.28 does:
> received: A00001 OK Logged in.
> sending : A00002 NAMESPACE
> received: * NAMESPACE (("INBOX." ".")) NIL NIL
> received: A00002 OK Namespace completed.
> sending : A00003 LIST "" INBOX
> received: * LIST (\HasChildren) "." "INBOX"
> received: A00003 OK List completed.
> sending : A00004 LSUB "" INBOX
> received: * LSUB () "." "INBOX"
> received: A00004 OK Lsub completed.
> sending : A00005 LIST "" INBOX.
> received: * LIST (\HasChildren) "." "INBOX."
> received: A00005 OK List completed.

but the actual master does:
> received: A00001 OK Logged in.
> sending : A00002 NAMESPACE
> received: * NAMESPACE (("INBOX." ".")) NIL NIL
> received: A00002 OK Namespace completed.
> sending : A00003 LIST INBOX ""
> received: * LIST (\Noselect) "." "INBOX."
> received: A00003 OK List completed.
> sending : A00004 LIST "" {7+}

Note the opposite A00003 arguments. Let's see why it is doing so.

ad b) I'm not sure, but I think it will be better to have a separate bug report for it, where I believe there is one already, even I didn't search for it yet.

Created attachment 151856 [details] [review]
proposed eds patch

for evolution-data-server;

Could you try with this patch, please? It seems to work for me on your test account, but I would like to know how it'll be with your real data. It's for the unresponsive Inbox only. Couple notes:
a) please try without removing ~/.evolution/imap/<affected-account> because
   it should fix itself the first run, maybe the second. If not, then please
   erase it and make it fetch the information again.
b) The Inbox should have the icon as the one under On This Computer,
   with this bug exposed it showed it as a regular folder.
c) After initial fetch (or self repair), with all the 'Check for new messages'
   unchecked, the counts will not match, but as soon as the send/receive is
   finished the count on the Inbox is fixed and stays there even the next start

I can reproduce the problem with disappearing folders on unsubscribe in 2.28 too, it's enough to unsubscribe from Inbox and all subfolders gone too, but they are back the next start. I know it's a variation of your initial problem with disappearing on send/receive, but I cannot reproduce the issue with the above patch applied.

Hi Milan.  I rebuilt with your patch.  I did not delete anything under ~/.evolution.

Now my Send/Receive is working correctly.  Yay!

However, my IMAP Inbox icon is still a regular folder, not a special "Inbox" folder.  I should point out that I have created extra folders (using Evolution) in my account, and they show up as sub-folders to the Inbox (I do understand why this is).  But, it might bear on which icon we use?

Or maybe this will be resolved if I delete my imap directory and allow it to be recreated?  I'll try this.

One odd thing is I'm connecting to my Exchange 2007 using IMAP (not MAPI) as well, and in THAT account (a) all the folders are siblings of Inbox, not children of Inbox, and (b) the Inbox icon is correct (a fancy inbox icon, not a folder).

Hmm, after some investigation I realized I cannot do the hide on unsubscribe properly. The problem is that there is IMAP's special option "Show only subscribed folders", but it's so specific that noone else knows about such option, thus the UI part cannot rely on this. More than that, there is no difference for the store between not knowing the option and not having it checked.

The other option might be to let the provider itself decide whether invoke the "folder_unsubscribed" event or not, but that means the provider knows who all is listening to the signal and try to cheat listeners. It might not work because in evo is listening UI part of the folder tree and a folder cache for this event. All this together it doesn't seem to me doable at the moment, unless I overlooked something.

Maybe do just a special case for Inbox and not remove it from UI at all, even when it's not subscribed?

I looked at Folders->Subscriptions and for my Dovecot IMAP account, all my
subfolders WERE checked, but the top-level Inbox folder was NOT checked.  Even
so, Send/Receive did pull new messages into my Dovecot IMAP account.

To try to reproduce the missing folders problem I checked the Inbox box as
well, then ran Send/Receive.  It still pulled new email, and none of my folders
disappeared.  So, I cannot reproduce this behavior anymore either (so far).  I
don't know if it was this change, or something before this, because I've left
that setting off ever since I had this problem back in mid-Dec.  Also I've not
been using Send/Receive because it didn't work; I was just selecting the Inbox
to get new mail.

Created attachment 151866 [details]
folders

hmm, it probably didn't updated the flags there. The remove of local imap cache and refetching it (or even only .ev-store-summary files should fix it the next start. Maybe.

I didn't get the problem with exchange 2007 over IMAP. This is what I see, and I created also subfolders on Inbox in Evolution, so it should work. Maybe I just missed something.

Nonetheless, I'm happy the patch is working for you too, thus I'm committing it to master.

Created commit 4cf0921 in eds master (2.29.6+)

Deleting .ev_store_summary and restarting evolution gives me the magic Inbox icon.

Yay!

The change above made an old behaviour back, which is one cannot fully unsubscribe Inbox (if I understand the code properly). Anyway, even slightly confusing, I'm happy it's working for you again. I'm closing this bug report. Thanks for your help. (Feel free to delete the test account, thanks for it.)