Oh, and I forgot to mention, I've been testing this with evolution and evolution-data-server 3.12.5 (only versions available to me at the moment, but supposed to be sufficient for the follow-up of bug #729848)

Hmm, it partly makes sense. If evolution doesn't check the folder for new messages, then you do not see them there as well. But you are right, the automatic download should not be related to regular folder update only, but also whenever a new message is discovered.

I tried to reproduce this, but no luck, the newly received messages are downloaded shortly after I enter the folder. What I do:
a) have set a folder for offline sync, which is not Inbox
b) the account does only update of the Inbox
c) open Evolution, move to folder from a)
d) if new messages are recognized, they are also downloaded shortly after
   that, which is indicated in the status bar with the "Downloading new
   messages for offline mode" operation.

When I select any from these newly downloaded messages they are shown instantly.

Could you try to run evolution from a console with a command:
   $ CAMEL_DEBUG=imapx:io evolution
then enter the offending folder, please? I do not want to read the log, I'd like to ask you to check whether, after you open the folder and a new messages will be recognized (probably with a \Recent FETCH flag, but not necessarily), then the log will also contain messages' content, which means that they were downloaded.

Hi Milan,

I think I figured it out some more...

1) It actually works for newly received messages in those folders, but not for older messages from back when caching was broken (or from before a given folder was marked for autocaching). In other words, the caching is not retroactive. Shouldn't it be, ideally?

2) Regardless of the outcome of #1, an issue is that the error message mentioned above is misleading. The problem is not that we couldn't reach the server (of course not, Evolution is offline and it knows it!), the problem is that there is no cached copy of the message you're trying to access. So in that case, shouldn't the error infobar say something like (depending on whether you consider #1 a bug or not):

"The message you're trying to open is not present in Evolution's offline cache. This might be because this folder was not set to automatically synchronize for offline use, [or because the message predates the activation of that setting. In that case, you can use "File > Fetch messages for offline use" next time you are online to download the older messages into the cache.]*"


*: Notice how the section in [ ] is somewhat ridiculous to try to explain to the user... I'd be happier if #1 was fixed and the message was just ending with a pair of buttons instead: [Edit folder properties] and [Edit global account settings]. After all, part of the point of caching select folders is to be able to have access to all of their contents offline without worrying about disk space (because other "big folders" are not marked for offline use)

Damnit, my hypothesis from my previous comment seems to be wrong, there actually is a bug as initially reported in bug #729848, the problem is that it is a heisenbug.

Tonight in the bus I tried accessing mails in my inbox folder (which only contaìns 15 items) and I couldn't access most of them offline. Even though they had been opened before, even though during the day I actually forced evolution to cache those folders by doing a "File > Fetch messages for offline use"... I can't figure out the logic that causes this to occur or not.

In previous testing attempts, I tried various ways to "break" Evolution to trigger the bug, including "--force-shutdown" to ensure no processes remained. The only difference tonight is that sometime during the day I logged out and did a soft reboot ("systemctl isolate multi-user.target", followed by "systemctl isolate graphical.target") because I wanted to get a fresh system. At the end of the day, before leaving I just unplugged my ethernet cable and closed the lid, putting the computer to sleep. In the bus, I closed and started Evolution.

So I thought "ok this is probably some race condition, maybe because I have a solid state drive... in which case running Evolution in gdb (just to slow it down) would have good chances of working"... but no luck. Out of 15 messages in my inbox, while offline I could only view one of them (the last one). This was consistent across app restarts.

Trying to run CAMEL_DEBUG=imapx:io evolution, I got nothing of value, only:

** (evolution:15502): CRITICAL **: categories_icon_theme_hack: assertion 'filename != NULL && *filename != '\0'' failed
evolution-shell-Message: Network disconnected.  Forced offline.
No bp log location saved, using default.
[000:000] Cpu: 6.37.5, x4, 2134Mhz, 3751MB
[000:000] Computer model: Not available
[000:000] Browser XEmbed support present: 1
[000:000] Browser toolkit is Gtk2.
[000:007] Using Gtk2 toolkit
No bp log location saved, using default.
[000:000] Cpu: 6.37.5, x4, 2134Mhz, 3751MB
[000:000] Computer model: Not available
** (evolution:15502): WARNING **: Shell not finalized on exit


...so I went all out and ran this instead: CAMEL_DEBUG="all" evolution &>logfile
The resulting log file is pretty huge and exceeds the 1M limit even compressed, so I'll mail it to you in the hope that it can be useful. All I did to generate that log file is to start Evolution while my network is down, try to open the 1st (oldest) message in the inbox, then try to open the last (newest) message in the inbox, then quit Evolution.

(In reply to comment #5)
> Trying to run CAMEL_DEBUG=imapx:io evolution, I got nothing of value

Makes sense. The imapx:io logs the communication between the server and the client. As you are offline, there is no communication between the two, thus nothing in the log.

> ...so I went all out and ran this instead: CAMEL_DEBUG="all" evolution
> &>logfile

The complete camel log is mostly useless, full of "random" debugging messages. Targeted camel logging is more efficient (the uncompressed log had more than 36MB).

More important question is, why, when being online, the message was not downloaded (and left) in the local cache. You mentioned all the possible occasions above:
a) it's an old message which was not downloaded yet
b) it's a new message, but its automatic download didn't happen for some
   reason, like the automatic download was stopped before it got to that
   message
c) there was some generic failure when viewing the message (similar to b), but
   aimed more for the a) case).

Did anything from the above happened to your Inbox with 15 messages? Did you see more than just that single available message from the Inbox before going offline? There was a bug report that the IMAPx cleared its local cache periodically, after several days. This had been disabled for Evolution-Data-Server 3.12.2, which you have.

Evolution doesn't download old messages on its own, but it can do it. It can be done either by aforementioned "File->Download messages for offline" or by "File->Work offline", which lets evolution know that you are going away from the network, and the evolution will eventually ask whether you want to download messages for offline. I think there is some limitation that the folder marked for offline might be visited during the run of evolution, otherwise it's skipped from the download before going offline, but I can be wrong.

The thing is that I miss some kind of a reproducer. All this works very simply, when evolution asks for the message, the IMAP mail provider looks into its cache first and if the message is there, then it is returned immediately. Otherwise an attempt to download it is done, which can eventually end with an error about the offline case.

Your proposal to change the error message looks fine to me. The current message is accurate, but too generic, thus I believe you that it looks misleading for regular users.

I changed the error message [1] for 3.13.8+ when the store is not connected and the error matches expected value, to a more descriptive version, though not exactly that yours. The more detailed description should go to the user manual.

[1] https://git.gnome.org/browse/evolution/commit/?id=a5513b3

Please tell me the targetted logging command you would like to get useful information for this?

Hypothesis A and B don't work as per my comment #5, it's the same messages that have been sitting in my inbox and worked offline less than 24 hours before and then stopped working offline. I'm inching towards C in my comment.

And looks like hypothesis C is either about some generic failure, or about the cache emptying itself for some arcane reason. The 15 messages I had in my inbox for which I could read only the last one offline yesterday night as per comment 5... I restarted my computer this morning, started Evolution with my network offline, and I can't read the 15th message that I was able to read offline yesterday night.

This kind of information (message had/had not been found in a local cache, or the cached message had been removed) is not logged anywhere within evolution. Evolution stores its local folder cache at
   ~/.cache/evolution/mail/<imap-account-uid>/...
Might it be that you have some periodical job to cleanup the ~/.cache ? It looks to me that it's something like that, all your observations suggest that. The IMAPx doesn't clean the cache, I use it too and the downloaded messages survive for a longer than a day, especially after the fix mentioned in comment #6 (a fix in eds 3.12.2+).

Could you try one thing, please? If we are talking about INBOX, then it has its locally downloaded files at:
   ~/.cache/evolution/mail/<imap-account-uid>/folders/INBOX/cur
divided into some subfolders. There can be stored multiple messages in them. The name of the message is the message ID as reported by the server.

When you view a certain message, then you might be able to find it in the cache, for example when searching for it by looking for a content with the same Message-ID header, which you can see in the message source (Ctrl+U), or simply for a text from the subject or such.

When you locate it, please make a copy of the whole
   ~/.cache/evolution/mail/<imap-account-uid>/folders/INBOX/cur
to your home, and then restart the machine (basically repeat the steps as you described in comment #9). Is, after the restart, but with evolution not running, the ...INBOX/cur the same as it was before the restart? Does anything changes after you run the evolution?

This is a stupid question, because evolution requires the same or higher version of evolution-data-server, but you do have the evolution-data-server version the same as the evolution has, right?

Yes, both Evolution and EDS are at version 3.12.5 on my machine (so at least the versions are in sync), and I do not have anything in my crontab ("crontab -e" is empty) nuking the cache (why would I ever want to do that? :)

I had quite a bit of trouble finding the relevant cache folder because I couldn't easily figure out the mail account's UID from looking at .config/evolution/sources/, which has 64 (!) items as it mixes everything (mail, contacts, calendars, signatures, certs) into one melting pot. I had to look at the stuff in ~/.config/evolution/mail/folders/et-exanded-folder* to get a hint as to which of those actually is the gmail account I'm looking for, which is "1210338597.14218.1@kiki"

So then I looked into  ~/.cache/evolution/mail/1210338597.14218.1@kiki/folders/INBOX/cur and found out that it has 64 folders (from what I can see in Nautilus)... all of which are empty. Why are there 64 subfolders in INBOX, and why are they all empty? That makes no sense to me.


~/.cache/evolution/mail/1210338597.14218.1@kiki/folders/INBOX/cur$ ls -a
.   00	02  04	06  08	0a  0c	0e  10	12  14	16  18	1a  1c	1e  20	22  24	26  28	2a  2c	2e  30	32  34	36  38	3a  3c	3e

..  01	03  05	07  09	0b  0d	0f  11	13  15	17  19	1b  1d	1f  21	23  25	27  29	2b  2d	2f  31	33  35	37  39	3b  3d	3f


Reactivating the network connection (for the first time in two days) to get a refresh of the inbox today, I then saw that 1 item was cached into folder "34", 8 items in folder 35, and 6 items in folder 36. This constitutes the "new mail" that showed up in INBOX (the total went up to 31, of which 15 are new mails that were therefore cached) The rest of subfolders remained empty. So I cut off the wifi (fn+F5), shut down the computer (and thus evolution), restarted, and the "new" mails can still be accessed offline after a reboot... but I suspect that won't be true in a day or so.



Tristan on IRC suggested that this might be some race condition in the "cache-reaper" module?

As different point of reference, my other IMAPx account does not seem to have this problem at all, but 1) it's not GMail  2) it has the global "Synchronize mail for offline use in all folders" switch enabled.

Ah and to address your question at the end of your comment 10: nah, the contents of the INBOX/cur/ folder did not change across the restart (at least if one is to judge on the number of items)

(In reply to comment #11)
> Tristan on IRC suggested that this might be some race condition in the
> "cache-reaper" module?

Not likely.  cache-reaper only touches leftover data from *deleted* accounts.

So, AFAICT (so far) it's probably not about the elapsed time - if I come back >24hrs later without having gone online, starting Evolution, it's still in the same state and it can still read the previously accessible cached messages.

However, when I go online and let Evolution sync up, then some older messages become inaccessible offline.

At least that's the gut feeling I get at the moment. Temporary demonstration video, if that helps: http://jeff.ecchi.ca/public/evolution-739650-2014-11-09.webm

(In reply to comment #11)
> So then I looked into 
> ~/.cache/evolution/mail/1210338597.14218.1@kiki/folders/INBOX/cur and
> found out that it has 64 folders (from what I can see in Nautilus)... all
> of which are empty. Why are there 64 subfolders in INBOX, and why are they
> all empty? That

The cache folders are not deleted, there are deleted only cached messages. I would remove everything (all folders) from the ~/.cache/.../folders/INBOX/cur , then start evolution in online and view some messages (they will be downloaded and will create new folders, but this time only those really used). 

(In reply to comment #14)
> However, when I go online and let Evolution sync up, then some older messages
> become inaccessible offline.

Hmm, let's see what we have.
- this is related to a GMail account configured in evolution using IMAPx
- it's only the INBOX set to cache messages for offline, not a whole account
  (it may not have any influence, as far as I can tell)
- as long as evolution is not run in online, the previously downloaded
  messages can be seen in offline without any issue
- once evolution is connected to the GMail the previously downloaded messages
  will stop showing in the next offline state, unless downloaded during this
  new online session

It looks like the GMail instructed a UID validity change for the folder for you for some reason, or something similar, which invalidates the local cache and thus gets rid of the downloaded messages. I tried my GMail account, but it doesn't have any huge traffic, thus I cannot reproduce it here (I didn't fully disconnect, I only switched evolution into offline and online).

What are your GMail options, specifically do you use "Quick Resync" and/or "Listen for server change notifications"?

I would like to ask you for a test:
a) run evolution in online, view a message in the GMail's Inbox
b) disable all but GMail account in Evolution
c) close evolution and run it in offline ($ evolution --offline)
d) verify you can see the message you was able to see previously
e) close evolution, run it as:
   $ CAMEL_DEBUG=imapx:io evolution &>log.txt
f) switch evolution to online
g) let it refresh the GMail account's Inbox
h) close evolution
i) run evolution in offline ($ evolution --offline)
j) check whether you can still see the same message as in d)

If you cannot see the message, then you managed to reproduce the issue and I'm interested in the log.txt, which will contain an information exchange between evolution and GMail. It will contain at least your email address, thus make sure you'll not expose anything private. If you are unsure, then feel free to send the log only to me, just name the bug number in the subject, otherwise I would miss it in my junk folder.

Thanks in advance.

> I would like to ask you for a test:

Still an issue in 3.16? If so, can you answer Milan's question?

Could it be that yours GMail Inbox lost the "Copy content locally for offline use" flag? I noticed that I can reproduce what you see only on folders which are not marked for offline, those I have set for offline behave properly, the messages are not "lost". I also thought that the GMail reported some kind of uidvalidity change, which means to forget everything in the folder and download its content from scratch. I noticed on the video that the folder content changed, some messages were removed from the Inbox, which can be related.

I agree with Andre, the best to try with 3.16.x (today's final 3.16 series release was 3.16.5). Considering also changes from bug #745545, which changed the IMAP internals significantly, the 3.18.0 would be even better.

nekohayo: ping - see two comments above

Closing this bug report as no further information has been provided. Please feel free to reopen this bug report if you can provide the information that was asked for in a previous comment.
Thanks!