Thanks for a bug report. That "Listen for server change notifications" option means using IDLE or NOTIFY (when either of them is supported by the server), in which case no periodic send/receive is usually required (it's good to be set in case other than Inbox is populated by the server, like due to server-side filtering, because IDLE works only for currently selected folder, not for any available). That's only a side note, it has not much to do with the issue.

I understood from your description that messages are properly delivered until something happened, currently only the time seems to be the clue, after which the folder stops updating for some reason. it can be that the folder is locked for some reason and some code left it locked either by an accident, or there is an ongoing (and eventually long) operation running in the background, which prevents folder update. As you download messages for offline use, it can be that the message download started with the folder locked and it's waiting until the message is downloaded. It can take "longer", if the message is large. Such operations are usually shown in the status bar. Anything ongoing can be also seen in a backtrace of running evolution. You can get the backtrace with command like this:
   $ gdb --batch --ex "t a a bt" -pid=`pidof evolution` &>bt.txt
Please check the bt.txt for any private information, like passwords, email address, server addresses,... I usually search for "pass" at least (quotes for clarity only). Ideally have installed debuginfo packages for evolution and evolution-data-server (no need for its dependencies, definitely not for webkitgtk4; something like:

   $ sudo dnf install evolution-data-server-debuginfo evolution-debuginfo \
        --enablerepo=updates-debuginfo

may do the trick, just make sure that the binary package version matches debuginfo package version precisely).

You can also run evolution with IMAPx debugging, like this:

   $ CAMEL_DEBUG=imapx:io evolution

which logs the raw communication between all IMAP servers and evolution on console. Using File->Send/Receive-><the account>, eventually right-clicking the folder in the folder tree on the left and choosing Refresh in the context menu will generate new content on the terminal.

Knowing the exact action which preceded the "folder doesn't update after refresh/send-receive" issue would help the most. It's possible the backtrace itself will show some ongoing operations.

By the way, in the Defaults tab, are the Junk and Trash folders set to be real folders on the server, or they are not, please?

Hi,

I think I’m hit by the same issue. Using Evolution 3.24.5 on Gentoo.

I will attach 2 backtraces as requested above. Saddly I have no CAMEL_DEBUG imapx log, It was not running with the debug at this time. I thought I had enabled debug symbols on evolution, but it seems I miss many symbols. But it may be useful ?

- First backtrace: I had the notification for one mail in my INBOX (the blue dot, and the dbus notification with the mail subject). But the mail does not shows up in the INBOX. Even after refreshing manually the folder, on after going offline and back online in evolution.

- Second backtrace, some minutes later, I get a second mail notification for a new mail in the same INBOX. Still no mail showing up in the IMBOX, neither the first, nor the second.

Now, I quit evolution => it hangs, and the popup for the force close shows up. I force close, do a « evolution --force-shutdown », then restart evolution. The missing mails shows up in the INBOX, and all is back normal.

Created attachment 358563 [details]
one mail missing

Created attachment 358565 [details]
two mails missing

I’ve been running with IMAPx debugging for days and not have this happen. I forgot to start Evolution with debugging once, and it happens in like five minutes. Will keep trying to catch it with debugging on.

Thanks for the backtrace. Even they do not show any symbol from
evolution-data-server and evolution (most of those with "??" might be them), then the difference between the two backtraces is that the first shows 4 threads reading data (or waiting for data) from the network, while the second backtrace shows two more. I do not know what exactly those threads are trying to do though, due to missing debug symbols.

(In reply to Guillaume Castagnino from comment #2)
> I had the notification for one mail in my INBOX (the blue dot, and the dbus
> notification with the mail subject). But the mail does not shows up in
> the INBOX.

This is a good clue. I suppose the notification comes from the evolution itself, which means that the underlying structures do know about the message, but the message list, for some reason, rejects to include the message in the view.

> Now, I quit evolution => it hangs, and the popup for the force close
> shows up.

Capture a backtrace of the hang the next time, please. By the way, is the hang of type "GUI doesn't update" or of type "UI updates, but it takes too long to quit"?

> I force close, do a « evolution --force-shutdown », then restart
> evolution. The missing mails shows up in the INBOX, and all is back normal.

Close of evolution is enough, as the mail part still runs within the evolution process only.

While playing with some debugging messages for the message list [1], I've got an idea. Maybe the folder structure being used by the MessageList is not the same which is used by the Mail Notification. It could happen in the past, though I thought it's already fixed. This thing won't show the backtrace, neither the added debug prints (they do print the actual CamelFolder pointer address, but there are no prints for the mail notification part).

How are your accounts configured? I know they are IMAP. The actual behaviour also depends on other settings, notably whether listening for server change notifications, whether using quick resync, whether both Trash and Junk folders are set as real folders on the server (the last thing is the most related to the guess of different CamelFolder structures being used by different parts of Evolution).

[1] https://git.gnome.org/browse/evolution/commit/?id=acadf85

(In reply to Daniel Aleksandersen from comment #5)
> I’ve been running with IMAPx debugging for days and not have this happen. I
> forgot to start Evolution with debugging once, and it happens in like five
> minutes. Will keep trying to catch it with debugging on.

Hmm, added debugging adds some lags, things take longer, which can mean the timing plays a role here as well.

(In reply to Milan Crha from comment #6)
> > Now, I quit evolution => it hangs, and the popup for the force close
> > shows up.
> 
> Capture a backtrace of the hang the next time, please. By the way, is the
> hang of type "GUI doesn't update" or of type "UI updates, but it takes too
> long to quit"?

I will try to get the backtrace next time. I will try to get all the debug symbols too.
This a hang of type UI is greyed out, stalled, and after some time a popup asking if I want to wait more oh force quit shows up.

> How are your accounts configured? I know they are IMAP. The actual behaviour
> also depends on other settings, notably whether listening for server change
> notifications, whether using quick resync, whether both Trash and Junk
> folders are set as real folders on the server (the last thing is the most
> related to the guess of different CamelFolder structures being used by
> different parts of Evolution).

I have 5 IMAP accounts, and one EWS. The IMAP that exibits the problem above is a Cyrus-imapd server, handling IDLE and quick resync, so both options are enabled. Sent/Trash and Junk folders are on the IMAP server:

[Imapx Backend]
FilterInbox=false
StoreChangesInterval=3
BatchFetchCount=500
CheckAll=true
CheckSubscribed=true
ConcurrentConnections=3
FetchOrder=ascending
FilterAll=false
FilterJunk=false
FilterJunkInbox=false
Namespace=
RealJunkPath=INBOX/Junk
RealTrashPath=INBOX/Trash
ShellCommand=ssh -C -l %u %h exec /usr/sbin/imapd
UseIdle=true
UseNamespace=false
UseQresync=true
UseRealJunkPath=true
UseRealTrashPath=true
UseShellCommand=false
UseSubscriptions=true
IgnoreOtherUsersNamespace=false
IgnoreSharedFoldersNamespace=false
UseMultiFetch=false

(In reply to Milan Crha from comment #6)
> Thanks for the backtrace. Even they do not show any symbol from
> evolution-data-server and evolution (most of those with "??" might be them),
> then the difference between the two backtraces is that the first shows 4
> threads reading data (or waiting for data) from the network, while the
> second backtrace shows two more. I do not know what exactly those threads
> are trying to do though, due to missing debug symbols.

I double checked: evolution IS build with debug symbols. So all "??" comes from evolutien-data-server that was stripped, not from evolution itself, if that may help until I get a better backtrace

(In reply to Guillaume Castagnino from comment #8)
> This a hang of type UI is greyed out, stalled,

Is it "stalled" like it doesn't change much, but when for example you move other window above this one, then it still paints properly, only all items are greyed there? If so, then it's a proper behaviour, evoltuion is waiting till all accounts stop working.

> and after some time a popup asking if I want to wait more oh force
> quit shows up.

Evolution has such dialog too, it tells that it's taking too long to quit (it's shown after a minute), then it has too buttons, one for "Keep Waiting", the other for "Quit Immediately". I guess you see this dialog. The system dialog can differ, depending on your desktop environment; it can be for example like "Wait" and "Force quit", I think. The difference is that evolution is stuck and doesn't update UI at all.

(In reply to Guillaume Castagnino from comment #9)
> I double checked: evolution IS build with debug symbols.

It's weird. The last thread, the Thread 1, in your backtraces looks like this:

> Thread 1 (Thread 0x7fd0b7488e40 (LWP 5404)):
> #0  0x00007fd0b271297d in poll () from /lib64/libc.so.6
> #1  0x00007fd0b523d716 in ?? () from /usr/lib64/libglib-2.0.so.0
> #2  0x00007fd0b523daa2 in g_main_loop_run () from /usr/lib64/libglib-2.0.so.0
> #3  0x00007fd0b5d23be5 in ?? () from /usr/lib64/libgtk-3.so.0
> #4  0x0000000000000000 in ?? ()

That #4 frame comes from evolution, from e-shell.c, main() function. Your gdb doesn't even know which file it comes from, with contrast of the other frames, where at least the library name is shown. Anyway, the backtrace would be useful only for the stuck state on quit.

I configured my main (and the most active) IMAP account the same way as you have it, let's see whether it'll be triggered here as well.

Created attachment 359043 [details]
six mails missing in directories

Here is a backtrace while missing 6 emails in 3 different subdirs, but _NOT_ INBOX, only subdirs.
As the previous time, I saw the mail notification.
I checked that the mails where missing using my webmail connected on the same imap.
I tried to refresh manually the concerned folders (F5 and right click -> refresh), but it did not change anything.

After restarting evolution, the missing mails become visible.
This time, no hang while stopping evolution, so no backtrace while stopping !


The previous backtrace was from my work configuration. This time it’s my home configuration, so there is only 2 IMAP configured, the same way at above.

Hope this may help !
Thanks

I still don’t run into this issue as long as I run with the debugging commands. I can still reproduce it within an hour or so if I run Evolution without debugging, but that isn’t all that useful.

Okay, The backtrace shows one IMAP IDLE thread, which is all fine. I've been afraid there's some pending operation in the background avoid the message list update, but it's not the case here.

Could be either of you able to connect to #evolution channel on GIMPNet (irc.gimp.org) IRC server once you get evolution on that side and ping me (mcrha), thus we could do some live gdb debugging, please? As before, make sure you've installed debuginfo packages for evolution-data-server and evolution, of the same version as the binary packages, because without it it won't work (I would like to set breakpoints into certain places in the code, print values of variables and results of some functions, and so on, all of which will require debuginfo to see what that part has set). Having a live debugging will be significantly easier than trying to do this through bugzilla.

Guillaume caught me on IRC and we realized that the local summary doesn't contain the messages, while it also reports the correct total count of messages, the same as on the server. There is also enabled mail notification in all folders (not only for Inbox) and there are no extra incoming filters set (using server side filtering).

There is also QResync enabled and I suspect it's casing the issue, but I forgot to enable the logging in gdb, thus bad luck for me. We'll try the next time when it happens, while it requires evolution to be run from terminal (without any other switch or debuggin variable), thus it will have a place to print the output to.

Created attachment 359282 [details]
debugging patch for eds

This is a debugging patch for eds. It doesn't do anything, it just prints information on places which I suspect as being related. The prints are going to be on the console of evolution, thus please run it from a terminal. It also prints only if an issue happens, or when a CamelStore is going to be freed (maybe the odd behaviour is related to changes for bug #770476, though I would not expect it from those changes reading), which should look like "store_dispose" line, immediately followed by "store_finalize" line for the same store. If it's not, then there is something going wrong.

Please, apply it to your sources (when saved as eds.patch, a call of:

   $ patch -p1 <../evo.patch

inside the evolution-data-server sources will do the trick) and recompile it (make && make install). It's enough to compile eds only, because the change doesn't change any public API.

(In reply to Milan Crha from comment #16)
> Please, apply it to your sources (when saved as eds.patch, a call of:
> 
>    $ patch -p1 <../evo.patch

It's meant to be:

   $ patch -p1 <../eds.patch

of course. (I noticed it right after pressing 'Submit' the attachment, I'm sorry.)

Created attachment 359357 [details]
debugging eds patch ][

This is an updated debugging patch. It contains the previous, thus revert it with:

   $ patch -p1 -R <../eds.patch

then save and apply this new one.

The changes being made:
a) when the in-memory versus on-the-disk mismatch happens, then it tries to
   save changes to the disk and the next time you enter the folder it should
   show those missing messages
b) it's more chatty. I tried to print things only when bad things happen, but
   it's not always possible, thus I see multiple lines like this one:
      camel_folder_summary_save: summ:0x7f7e5c1a91f0 no dirty found
   on console now. They will be interesting when it comes to the count mismatch.

For the record, the previous patch showed that there happened a mismatch between in-memory and on-the-disk saved message infos, where the in-memory did contain the newly recognized message. It had not been saved to the disk for some reason, while the summary itself showed it as not being changed, thus any attempt to save its content let to void operation.

Created attachment 359728 [details] [review]
eds semi-debugging patch

This can be applied on top of the previous patch. It is way more chatty, especially about the summaries being created and disposed/finalized (each dispose should be followed by finalize, if not, something goes wrong). It also says when the issue we debugged the last time (today) happens, which was that in save_to_db_cb() the passed-in message info had not set any 'summary'. This change also verifies that the summary set on the mesasge info matches the one which is saving the message info.

I thought there is some place where the IMAPx provider set NULL summary to the info, but I didn't find it. There is one place, but it's not used at all in the code.

Hi,

I did not forget, but since I installed the last patch on evolution, I still did not manage to reproduce the issue. Does the patch changes anything besides the debug?

But I have a lot of logs like this in the console output:

(evolution:14302): camel-WARNING **: save_to_db_cb: Summary of message info 'INBOX/Gentoo/Gentoo-Announce : [gentoo-announce] [ GLSA 201709-03 ] WebKitGTK+: Multiple vulnerabilities' is NULL, but expected 0x7f50042e0cf0

(evolution:14302): camel-WARNING **: save_to_db_cb: Summary of message info 'INBOX/Gentoo/Gentoo-Announce : [gentoo-announce] [ GLSA 201709-12 ] Perl: Race condition vulnerability' is NULL, but expected 0x7f50042e0cf0

(evolution:14302): camel-WARNING **: save_to_db_cb: Summary of message info 'INBOX/Gentoo/Gentoo-Announce : [gentoo-announce] [ GLSA 201709-02 ] Binutils: Multiple vulnerabilities' is NULL, but expected 0x7f50042e0cf0

The patch contains a possible fix and also some debug prints to help to narrow the issue down. One of those debug prints which confirm the issue is that you pasted. Thus it can still happen that the summary is NULL, weird.

Does any message above these also mention 'dispose' for '0x7f50042e0cf0', please?

Created attachment 360442 [details]
debug.txt.gz

There are such message, but not directly above. Quite far above in fact.

attached the full console log

Thanks. I see that it's not what I've been afraid of, which is good (the summary is alive, not half-freed and then resurrected). Furthermore, the Trash folder is just created and many of the messages in it do not have the summary set at all (there's that warning for).

The places I looked at in IMAPx code both set the summary, thus it should be there, but the messages in the log do not have set it. I'll try to figure out, once again, what the cause is. Seeing there the Trash folder, maybe it's after copying/moving messages between folders, which can be possible.

Created attachment 360528 [details]
eds patch with debugging ]I[

This is a patch for evolution-data-server, which fixes the issue (in particular, camel_folder_summary_remove_uid() could leak CamelMessageInfo, which had a side-effect of left CamelFolderSummary in a global CamelWeakRefGroup for its address, but when it had been freed the associated GWeakRef returned NULL, as expected, but the pointer should not be in the weak-ref-group in that time; when another summary had been created at the same memory address the weak-ref-group used the old object it had stored, thus the newly created CamelMessageInfo with the new summary returned NULL, instead of that new summary at the same address).

It applies to both 3.24.x and 3.26.x. Ideally run cmake with -DENABLE_BACKTRACES=ON and have installed elfutils-devel (development package of elfutils). The idea of the patch is that it is quiet until bad things happen. Thus once compiled, run evolution from a terminal and just let it be and use evolution as before. Look at the console from time to time, whether any new prints had been added. They may contain backtraces too, if eds had been configured with them and necessary libraries had been found. The most it should print on quit of evolution is this line:

   dump_left_groups: no groups left

which is when everything went properly.

A test build for Fedora 26 can be found here:
https://koji.fedoraproject.org/koji/taskinfo?taskID=22098034
I wrote at bug #786636 comment #14 what to do with it.

As noted in bug #786636, with respect of evolution-ews, the sync-states for affected folders should be reset, which causes refresh of the local cache. If unsure (it's also harder to find the right places), then just remove
   ~/.cache/evolution/mail/
and let evolution fill local cache from scratch. The downside is that it'll download everything again, which is time and bandwidth consuming for large accounts.

There is planned a 3.26.1 release on the next Monday, October 2nd. I would like to have this change (with removed debugging stuff) committed there and I'll also release 3.24.7 evolution-data-server with it. I would like to ask you to give this some time for testing and let me know in this bug report either on Friday or on Monday, ideally before noon CEST:

   $ date --date="Mon Oct 02 12:00:00 CEST 2017"

eventually to get the time for other than local timezone, like America/New_York:

   $ TZ="America/New_York" date --date="Mon Oct 02 12:00:00 CEST 2017"

, whether it worked for you, and/or what output you've got on the evolution console. Thanks in advance.

I applied the patch and ran E for 15 min, then exit. I saw this:
dump_left_groups: there left 4 groups
   obj:0x7fd1fc6cdd50 use_count:356 in-ref:0x7fd1fc6cdd50
   obj:0x7fd1fc6cdf90 use_count:229 in-ref:0x7fd1fc6cdf90
   obj:0x7fd1fc6cd8d0 use_count:96 in-ref:0x7fd1fc6cd8d0
   obj:0x7fd270008210 use_count:4004 in-ref:0x7fd270008210

So I guess there is something off?

(In reply to Joakim Tjernlund from comment #25)
> I applied the patch and ran E for 15 min, then exit. I saw this:
> dump_left_groups: there left 4 groups
>    obj:0x7fd1fc6cdd50 use_count:356 in-ref:0x7fd1fc6cdd50

Ah, I didn't mention it earlier, I'm sorry. It can eventually print something at the end from the "dump_left_groups", it's for cases when there's a memory leak, but the important thing is that the "in-ref" is a non-NULL memory pointer, which means that the summary is still alive. The "use_count" says how many message infos reference the folder summary of "in-ref". When everything is freed as it should, then there are no left groups. If you see any "in-ref" being NULL in the output, then bad things can happen, but it should also claim it before you close evolution.

Looking good, no problem so far. Did an exit and got this:
dump_left_groups: there left 5 groups
   obj:0x7f4c98009210 use_count:465 in-ref:0x7f4c98009210
   obj:0x7f4c981a17b0 use_count:109 in-ref:0x7f4c981a17b0
   obj:0x7f4c98009e70 use_count:5 in-ref:0x7f4c98009e70
   obj:0x21b1e70 use_count:230 in-ref:0x21b1e70
   obj:0x7f4c98009690 use_count:1752 in-ref:0x7f4c98009690
No NULL ptrs yet.

I accidentally reproduce it when copying a message from one IMAP account to another IMAP account (drag & drop), which claimed the issue:

camel_weak_ref_group_claim_summary_finalize: summary gone:0x7f12a80131a0, while still in the weak ref groups
   at camel_weak_ref_group_claim_summary_finalize() at camel-weak-ref-group.c:159
   by folder_summary_finalize() at camel-folder-summary.c:255
   ...

 and when I've been closing evolution it printed:

  dump_left_groups: there left 2 groups
     obj:0x7f12a80131a0 use_count:1 in-ref:(nil)
     obj:0x7f12140269d0 use_count:22631 in-ref:0x7f12140269d0

where the "in-ref:(nil)" confirms the issue. I'll extend the patch today, once I finish the other thing I work on, during which I faced it.

Created attachment 360658 [details]
eds patch with debugging IV

This is an updated patch. The issue described in my previous comment was caused by IMAP code calling camel_folder_summary_add() once conditionally, from:
   imapx_untagged_fetch() at camel-imapx-server.c:1414
and the other time, immediately after this call for me, from:
   camel_imapx_server_append_message_sync() at camel-imapx-server.c:4688
The later call was done unconditionally and eventually overwrote just added message info, without properly freeing it (and without removing the weak ref group reference on the summary for it).

This patch adds the correction for this, by making camel_folder_summary_add() smart enough to properly free what should be freed when this condition happens. The patch can print on console something like this now:
>  camel_folder_summary_add: 0x7fd1dd9a1a50: adding info:0x7fd1de128f30 which
>  is already there as:0x7fd1dc17cd90
which only claims that it had been hit, but otherwise it doesn't cause any harm and at the end is properly shown either no left groups or the groups with in-ref non-NULL.

Here can be found updated test build for Fedora 26:
https://koji.fedoraproject.org/koji/taskinfo?taskID=22143092

Thanks, I have rebuilt my eds now.
Will leave it on over the weekend.

BTW, I get from time to time:
Error while Downloading new messages for offline mode in “Infinera : Deleted Items”.

Failed to move message cache file from “(null)” to “/home/jocke/.cache/evolution/mail/1440067750.1759.3@gentoo-jocke/folders/Deleted Items/cur/29/c5e09ff9035c18ef573cba5060e86eb132d6b55b22df8cb0687730beb0ccfcd7”: Bad address

The folder name varies.

(In reply to Joakim Tjernlund from comment #30)
> BTW, I get from time to time...

That's new to me, unrelated to this report, seems like evolution-ews specific on the first look. It would worth investigate it too, but I do not want to steal this already lengthy bug report with other issues.

(In reply to Milan Crha from comment #24)
> There is planned a 3.26.1 release on the next Monday, October 2nd. I would
> like to have this change (with removed debugging stuff) committed there and
> I'll also release 3.24.7 evolution-data-server with it. I would like to ask
> you to give this some time for testing and let me know in this bug report
> either on Friday or on Monday, ideally before noon CEST:
> 
>    $ date --date="Mon Oct 02 12:00:00 CEST 2017"
> 
> eventually to get the time for other than local timezone, like
> America/New_York:
> 
>    $ TZ="America/New_York" date --date="Mon Oct 02 12:00:00 CEST 2017"
> 
> , whether it worked for you, and/or what output you've got on the evolution
> console. Thanks in advance.

Hi,

Just to notice that since I applied the last patch, I did not encounter the issue anymore (but it sometimes needed days to reproduce).
Just having some left group at exit:
dump_left_groups: there left 5 groups
   obj:0x7f5de0012360 use_count:576 in-ref:0x7f5de0012360
   obj:0x7f5de0012ad0 use_count:19 in-ref:0x7f5de0012ad0
   obj:0x173da60 use_count:3 in-ref:0x173da60
   obj:0x7f5de0012be0 use_count:704 in-ref:0x7f5de0012be0
   obj:0x173dd90 use_count:96 in-ref:0x173dd90



As a side effect of this patch, I noticed I huge speed improvement when moving/deleting (to a real trash, so basically it’s a move too) messages. The transient state where the moved/deleted mail is mark with a line-through is a lot shorter. Just like it was with eds 3.22.

Thanks for the update. It looks fine. I'll wait for Joakim with the release, but I'm confident that most of the issue should be fixed with the latest patch. I'll prepare a change for evoltuion-ews meanwhile, to auto-update folder content with the patched versions, thus also "invisible"/missing messages in those folders will be shown without a need to manual changes in underlying Evolution data.

(In reply to Milan Crha from comment #33)
> Thanks for the update. It looks fine. I'll wait for Joakim with the release,
> but I'm confident that most of the issue should be fixed with the latest
> patch. I'll prepare a change for evoltuion-ews meanwhile, to auto-update
> folder content with the patched versions, thus also "invisible"/missing
> messages in those folders will be shown without a need to manual changes in
> underlying Evolution data.

As far as I can tell, I have not lost any mails during the weekend so
I think we are good :)

Nice. Thank you all for the help with testing and patience. I'll do the commits and releases as promised.

Created commit 2a26cd15c in eds master (3.27.1+)
Created commit 9d7dad23a in eds gnome-3-26 (3.26.1+)
Created commit 3999a753a in eds gnome-3-24 (3.24.7+)

Created commit_f662bb0 in ews master (3.27.1+) [1]
Created commit_e526b08 in ews gnome-3-26 (3.26.1+)
Created commit_fcfc442 in ews gnome-3-24 (3.24.6+)

[1] https://git.gnome.org/browse/evolution-ews/commit/?id=f662bb0

*** Bug 786636 has been marked as a duplicate of this bug. ***