Thanks for a bug report. I found the icons about connection state hard to understand myself (those beside account name in the folder tree). There are currently four states:
1) network unavailable (this is usually together with 'evolution offline')
2) network available, but destination host unreachable
3) host reachable, but the account being disconnected (offline)
4) host reachable and the account connected (online)
I think the 1) and 2) share the same icon.

You are right that evolution doesn't reconnect even when the network is available, it's done either manually (as you said) or when the periodic update is reached.

You are also right that it has downsides with the IDLE/notify features, which is unfortunate and would be nice to change. Ignoring (connection) errors after going online, that would make it work.

I'll think about this a bit, but no promises.

I noticed that my evolution connects the account after resume from suspend, but it doesn't run the IDLE, thus there are no notifications of the server changes. One part of the change comes to evolution-data-server, issuing NOOP after connect. That's to run the whole mechanism around IDLE in an expected way.

Created commit 782b862 in eds master (3.23.92+)
Created commit 7b16c31 in eds gnome-3-22 (3.22.6+)

I made also a change in evolution itself, because I noticed that this won't cover network connections enabled after VPN connect or such. I saw is misbehaving too, due to lost signal notifications or something around, but when the signal is received it is properly connected and the IDLE is running. As the change as itself is sort of debatable, I decided to push it only to the development version.

Created commit_487554d in evo master (3.23.92+) [1]

[1] https://git.gnome.org/browse/evolution/commit/?id=487554d

*** Bug 782428 has been marked as a duplicate of this bug. ***

I switched to F26 and Evo 3.24.2 and it still doesn't seem to be picking up new mail after resume. I'll give it a few more days before I reopen this.

Milan,

Unfortunately, this is not fixed. I just resumed F26 with 3.24.2 installed and the new mail was not picked up until I manually did it.

For whatever reason, I cannot reopen this bug in Gnome Bugzilla. Could you please reopen it?

Okay, I can reopen it. Only before that, could you try one thing for me, please? I'm wondering whether the mail account connected to the server and whether it eventually also run refresh. I do not think the IDLE start after new connection to the server is also notified about all messages "since I've been connected to the server last time", which can be weeks. To do so, please watch the folder tree on the left of the Mail view. Where there's the line with account name it also shows icons on the right showing whether the account is connected/disconnected and even when it's connecting there's a spinner spinning.

Before you test fully, please run evolution from a command line with an IMAPx debugging on. It would be like this:

   $ CAMEL_DEBUG=imapx:io,imapx:conman evolution

I would be interested only in the end of the log. Particularly with the part which is below messages about forced offline/online. The account doesn't connect immediately, it waits for ~5 seconds (to avoid flood of reconnections when there are many notifications about network changes in a row). Then the IMAPx account should probably run the IDLE, if it's configured to do so (Listen for server change notifications).

Here it is (note: the line of dashes is suspend/resume, the line of equal signs is me clicking send/receive after the email I sent after suspend did not show up automatically the folder it was stored after resume):

* 2 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1111895052] UIDs valid
* OK [UIDNEXT 21797] Predicted next UID
* OK [HIGHESTMODSEQ 70962] Highest
A00006 OK [READ-WRITE] Select completed (0.001 + 0.000 secs).'
[imapx:A] I/O: 'A00007 IDLE'
[imapx:A] I/O: '+ idling'
----------------------
evolution-shell-Message: Network disconnected.  Forced offline.
[imapx:*] Disconnecting all 1 connections
[imapx:A] I/O: ''
[imapx:A] I/O: 'DONE'
[imapx:A] I/O: ''
[imapx:A] I/O: ''
evolution-shell-Message: Connection established.  Going online.
[imapx:B] I/O: '* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE XLIST NOTIFY AUTH=PLAIN AUTH=GSSAPI] Dovecot ready.'
[imapx:B] I/O: 'B00008 AUTHENTICATE GSSAPI'

[..snip authentication stuff..]

[imapx:B] I/O: 'B00008 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE XLIST NOTIFY NOTIFY COMPRESS=DEFLATE] Logged in'
[imapx:B] I/O: 'B00009 NAMESPACE'
[imapx:B] I/O: '* NAMESPACE (("" "/")) NIL NIL
B00009 OK Namespace completed (0.001 + 0.000 secs).'
[imapx:B] I/O: 'B00010 ENABLE CONDSTORE QRESYNC'
[imapx:B] I/O: '* ENABLED CONDSTORE QRESYNC
B00010 OK Enabled (0.001 + 0.000 secs).'
[imapx:B] I/O: 'B00011 NOTIFY SET (selected (MessageNew (UID RFC822.SIZE RFC822.HEADER FLAGS) MessageExpunge FlagChange)) (personal (MessageNew MessageExpunge MailboxName SubscriptionChange))'
[imapx:B] I/O: 'B00011 OK NOTIFY completed (0.003 + 0.000 + 0.002 secs).'
[imapx:B] Created new connection 0x7fb0840380d0 (server:0x55c0dcb17950) for [null]; total connections 1
[imapx:B] I/O: 'B00012 SELECT INBOX (QRESYNC (1111895052 70962 20134:21381))'
[imapx:B] I/O: '* FLAGS (\Answered \Flagged \Deleted \Seen \Draft receipt-handled $has_cal $Forwarded)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft receipt-handled $has_cal $Forwarded \*)] Flags permitted.
* 2 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1111895052] UIDs valid
* OK [UIDNEXT 21797] Predicted next UID
* OK [HIGHESTMODSEQ 70962] Highest
B00012 OK [READ-WRITE] Select completed (0.002 + 0.000 + 0.001 secs).'
[imapx:B] I/O: 'B00013 NOOP'
[imapx:B] I/O: 'B00013 OK NOOP completed (0.001 + 0.000 secs).'
[imapx:B] I/O: 'B00014 IDLE'
[imapx:B] I/O: '+ idling'
================
[imapx:B] I/O: ''
[imapx:B] I/O: 'DONE'
[imapx:B] I/O: ''
[imapx:B] I/O: 'B00014 OK Idle completed (60.252 + 60.251 + 60.251 secs).'
[imapx:C] I/O: '* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE XLIST NOTIFY AUTH=PLAIN AUTH=GSSAPI] Dovecot ready.'
[imapx:C] I/O: 'C00015 AUTHENTICATE GSSAPI'

Thanks for the log. If I read it properly, then behaves as expected. It disconnected when gone offline, then went online and connected. It enabled NOTIFY (it's great your Dovecot supports it), and ran IDLE on Inbox. It will notice any newly received/changed/removed messages since it started the NOTIFY/IDLE, depending on the way the server sends the notifications.

What is missing here, if I understand it correctly, is a refresh after resume, but I'm unsure whether it's really to be done. It should depend on current options, like "Is refresh after start enabled? If yes, is automatic update enabled? If yes, check in all configured folders, just like Send/Receive does". It can be made simple by doing this for all accounts after evolution went online, but is that really what you want? I do not think I'm able to differentiate between suspend/resume and intermittent network outage (I'm not going to compare some times). In other words, once evolution will run automatic update it'll refresh the folders as usual.

Bye the way, which folder did the message arrive to? The only touched after resume was Inbox, which had 2 messages and no recent (aka new) message.

Anyway, we can discuss this in a new bug report, if you want to, because I believe this bug is working properly, at least from my point of view.

Created attachment 354266 [details]
Receiving options that are turned on

I attached receiving options that I have. Cannot see "refresh after start", but maybe I didn't check in the right place.

The message arrived in the folder other than inbox (I use sieve filtering on Dovecot side to sort out my messages as they come). But, even the inbox doesn't get refreshed. For instance, I just resumed my computer now and there were two new messages in inbox, but only after I manually did send/receive.

Personally, I would think that a one-off refresh should be run when network reconnects, if notify is not capable of letting Evo know what changed.

(In reply to Bojan Smojver from comment #10)

> Personally, I would think that a one-off refresh should be run when network
> reconnects, if notify is not capable of letting Evo know what changed.

Or, maybe we can have a receiving option to do this (if it doesn't already exist), just in case not everyone is happy with that as a default approach.

(In reply to Bojan Smojver from comment #11)

> Or, maybe we can have a receiving option to do this (if it doesn't already
> exist), just in case not everyone is happy with that as a default approach.

I looked through some other options in Mail (not receiving options of the account) and I found "Check for new messages in all active accounts" under "Start up". After that, everything seems to be working as expected.

So, please ignore me. Obviously, I didn't have the right option turned on. Sorry about the noise. :-(

(In reply to Bojan Smojver from comment #12)
> I looked through some other options in Mail (not receiving options of the
> account) and I found "Check for new messages in all active accounts" under
> "Start up".

There's also the other option I've been talking about, the "Check for new messages on start".

> After that, everything seems to be working as expected.
> 
> So, please ignore me. Obviously, I didn't have the right option turned on.

I would not tell that. That option only lets evolution run "periodic update" also for accounts which has that disabled, as you do (in Receiving Options, the very first option "Check for new messages every NNN minutes". You can have enabled either this, or the option from Preferences.

OK, I'm confused now. Are you saying that option to check for new messages in all active accounts on startup activates some kind of period check?

BTW, check new messages on start option alone does not make it work with notify on resume. It is turned on by default, but notify does not work unless I also turn on all accounts under startup.

Also, the periodic update in receiving options (which I have turned off) kinda defeats the purpose of notify (why would you want to keep polling when the server can tell you when there is stuff to pick up). And, at least under 3.22.x, actually makes notify not work at all.

BTW, there was also a fix in NOTIFY command put into Dovecot 2.2.31, which is still an RC:

https://dovecot.org/list/dovecot-news/2017-June/000349.html

Maybe that has something to do with all this. Not quite sure. Just FYI.

(In reply to Bojan Smojver from comment #12)
> Obviously, I didn't have the right option turned on.

Hmm, just had a bit of a failure with my "check all accounts on startup" setup. I did not get notified about one of the folders (coincidentally, my Fedora folder :-) getting new mail. Not quite sure why.

Anyhow, let's wait until Dovecot 2.2.31 lands in F26. There is apparently a fix related to NOTIFY there. Maybe that will make some kind of difference.

(In reply to Bojan Smojver from comment #14)
> OK, I'm confused now. Are you saying that option to check for new messages
> in all active accounts on startup activates some kind of period check?

No no, that option only means that the initial refresh (and only the initial refresh) will be done also for accounts which have the periodic check off. Thinking of it, I believe your usage of it is appropriate here. 

(In reply to Bojan Smojver from comment #16)
> Anyhow, let's wait until Dovecot 2.2.31 lands in F26. There is apparently a
> fix related to NOTIFY there. Maybe that will make some kind of difference.

I doubt it'll help. The NOTIFY is for notifications since that command had been started, that is, if you've a sequence of:
   - start Evolution
      - NOTIFY started
   - suspend
      - NOTIFY stopped
      - account disconnected
      - Evolution goes offline
   - resume
      - Evolution goes online
      - NOTIFY started

then you'll not be notified about messages received while Evolution was offline. It would be great, but NOTIFY is not for this kind of synchronization. There's not much difference between this offline-then-online state and exiting evolution.

While debugging bug #782844 I noticed that sometimes after start my IMAPx account does LIST followed by SELECT and then IDLE, but other times (not that often) it does only LIST and then nothing. I do not know how to reproduce it, but it is most likely what you face, that there is a connection, but not an IDLE thread, which receives changes both from IDLE itself and from NOTIFY (if server supports it). I'll check it further once I figure out the other bug report and let you know here. (I do not have set evolution to refresh after start, which is similar to what your setup does.)

You are right, Bojan. I missed some cases. One "sometimes" was for me when the first folder listing had been done only for "Templates", which didn't discover Inbox and then had Inbox missing, thus didn't run IDLE. This "sometimes" was related to thread interleaving. Another time was when going online after Evolution being started in offline mode. It didn't do the listing, thus it also didn't know Inbox, thus it didn't run IDLE. There had been added NOOP on Inbox after successful connection, to update Inbox and to eventually run IDLE, but this NOOP could be skipped, again due to not discovered Inbox yet. The changes below address all these cases.

I made a test build of evolution-data-server for Fedora 26 with this change [1], it contains changes for bug #782844 as well (I didn't want to split them, as I made a test build for that bug first). Give it a try, if you can, please.
[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=20206592

Created commit 0308c92 in eds master (3.25.4+)
Created commit 9f0ece6 in eds gnome-3-24 (3.24.4+)

(In reply to Milan Crha from comment #19)

> I made a test build of evolution-data-server for Fedora 26 with this change
> [1], it contains changes for bug #782844 as well (I didn't want to split
> them, as I made a test build for that bug first). Give it a try, if you can,
> please.
> [1] https://koji.fedoraproject.org/koji/taskinfo?taskID=20206592

Thanks for chasing this down. I'll try right away.

Do I leave the option to check all accounts on startup on or off?

(In reply to Bojan Smojver from comment #20)

> Do I leave the option to check all accounts on startup on or off?

To answer my own question, works when this option is on. Obviously, to replicate that case from comment 16, I'll have to run it for a while.

Right, you should use the settings you prefer, which is not to update after start at all, as it used to be before we begun follow up discussion here. You can test with various options on and off, but it adds to complexity.

(In reply to Milan Crha from comment #22)
> Right, you should use the settings you prefer, which is not to update after
> start at all, as it used to be before we begun follow up discussion here.

That still does not see mail that has arrived between suspend and resume. So, I'm guessing checking once on reconnect is required.

(In reply to Bojan Smojver from comment #23)

> That still does not see mail that has arrived between suspend and resume.
> So, I'm guessing checking once on reconnect is required.

Which, when I think about it, makes perfect sense.

For instance, let's say I suspend my system for 24 hours. If I then reconnect to the IMAP server, the server would have to keep some kind of state about my last connection in order to tell me what changed and notify me. I don't think this is how IMAP works. It can only tell me what changed since I last connected.

So, I have to run one check when the connection is established (i.e. the check all account on startup option) and then listen to notifications.

I'd say this is working as intended. I'm guessing your patches will probably make it more consistent (i.e. regarding what I was saying in comment 16).

Thank you for looking into this!

Thanks for the feedback.

(In reply to Bojan Smojver from comment #24)
> Which, when I think about it, makes perfect sense.

Yes, that's what I've been trying to express above, like at the end of
comment #17.

(In reply to Bojan Smojver from comment #16)

> Anyhow, let's wait until Dovecot 2.2.31 lands in F26. There is apparently a
> fix related to NOTIFY there. Maybe that will make some kind of difference.

Notify broken now with this release of Dovecot. This is with Evo/EDS 3.24.3-1. Interesting...

(In reply to Bojan Smojver from comment #26)
> (In reply to Bojan Smojver from comment #16)
> 
> > Anyhow, let's wait until Dovecot 2.2.31 lands in F26. There is apparently a
> > fix related to NOTIFY there. Maybe that will make some kind of difference.
> 
> Notify broken now with this release of Dovecot. This is with Evo/EDS
> 3.24.3-1. Interesting...

Also against the scratch build of EDS from this bug.

Just to be 100% clear, IDLE works with 2.2.31 (i.e. inbox gets new messages as they are delivered). But NOTIFY (i.e. delivery to any other folder) does not.

The conversation between Evo and Dovecot says:

[imapx:B] I/O: 'B00010 NOTIFY SET (selected (MessageNew (UID RFC822.SIZE RFC822.HEADER FLAGS) MessageExpunge FlagChange)) (personal (MessageNew MessageExpunge MailboxName SubscriptionChange))'
[imapx:B] I/O: 'B00010 OK NOTIFY completed (0.001 + 0.000 secs).'

But then it doesn't work. My guess is that in an effort to fix NOTIFY in 2.2.31, it was actually broken.

(In reply to Bojan Smojver from comment #26)
> Notify broken now with this release of Dovecot. This is with Evo/EDS
> 3.24.3-1. Interesting...

If it's about Dovecot version, then there's really nothing evolution(-data-server) can do about it.

(In reply to Bojan Smojver from comment #29)
> The conversation between Evo and Dovecot says:

Right, it enabled the NOTIFY, thus the server knows that it can do the notifications. Just to verify that the issue is not on the IMAPx side, when you've that logging on (CAMEL_DEBUG=imapx:io evolution) it should always end with an IDLE call, which makes sure that the IMAPx is reading from the stream, thus it's able to receive the notifications from the server as soon as they are sent. The log ends like:

[imapx:A] I/O: 'A00009 IDLE'
[imapx:A] I/O: '+ idling'
[imapx:A] I/O: '* OK Still here'

After this I added a new message to INBOX/atest folder from another instance, which resulted in:

[imapx:A] I/O: '* STATUS INBOX/atest (MESSAGES 9 UIDNEXT 219 UNSEEN 2 HIGHESTMODSEQ 233)'
[imapx:A] I/O: ''
[imapx:A] I/O: 'DONE'
[imapx:A] I/O: ''
[imapx:A] I/O: 'A00009 OK Idle completed.'
[imapx:A] I/O: 'A00010 STATUS INBOX/atest (MESSAGES UNSEEN UIDVALIDITY UIDNEXT HIGHESTMODSEQ)'

and so on (the INBOX/atest had been selected and refreshed). And when it had been finished the connection went back to:

[imapx:A] I/O: 'A00016 IDLE'
[imapx:A] I/O: '+ idling'
[imapx:A] I/O: '* OK Still here'

(In reply to Milan Crha from comment #30)

> If it's about Dovecot version, then there's really nothing
> evolution(-data-server) can do about it.

It was just an FYI for you. :-)

> Right, it enabled the NOTIFY, thus the server knows that it can do the
> notifications. Just to verify that the issue is not on the IMAPx side, when
> you've that logging on (CAMEL_DEBUG=imapx:io evolution) it should always end
> with an IDLE call, which makes sure that the IMAPx is reading from the
> stream, thus it's able to receive the notifications from the server as soon
> as they are sent.

Yeah, it does. In fact, inbox IDLE works (i.e. mail is delivered immediately), but any other folder is a no go.

Another FYI:

https://dovecot.org/pipermail/dovecot/2017-June/108498.html

So, yeah, NOTIFY got broken in 2.2.31.