After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 693101 - IMAPx can vanish and redownload folder summary
IMAPx can vanish and redownload folder summary
Status: RESOLVED FIXED
Product: evolution
Classification: Applications
Component: Mailer
3.8.x (obsolete)
Other Linux
: High major
: ---
Assigned To: evolution-mail-maintainers
Evolution QA team
3.8.1
: 693078 (view as bug list)
Depends on: 704513
Blocks:
 
 
Reported: 2013-02-03 18:28 UTC by Adam Williamson
Modified: 2013-11-27 12:38 UTC
See Also:
GNOME target: ---
GNOME version: 3.7/3.8


Attachments
eds patch (2.20 KB, patch)
2013-02-18 14:27 UTC, Milan Crha
committed Details | Review
console output from the bug happening on a resume from suspend, with qresync disabled, imapx debugging output enabled (gzipped: too big for text attachment) (113.06 KB, application/x-compressed-tar)
2013-03-19 06:07 UTC, Adam Williamson
  Details
eds patch ][ (1.29 KB, patch)
2013-03-21 18:27 UTC, Milan Crha
committed Details | Review

Description Adam Williamson 2013-02-03 18:28:35 UTC
This has happened to me three times since updating from 3.6.x to 3.7.x, and never happened to me before. If I suspend then resume with Evo 3.7.4 running, it completely loses all the mail in my IMAP account (a Dovecot server) and has to reload all the headers from scratch. The server side isn't affected so far as I can tell, all the mails are there, Evo just 'forgets' that they are then goes out and gets them all again.

When I mentioned this in IRC mbarnes implied that problems with suspend/resume and Evo in general were known and I should just close Evo before suspending, but I've never had this problem before (been running Evo for many years) and it's happened every time I've suspended with 3.7.4, so it seems like something specific.
Comment 1 Matthew Barnes 2013-02-04 20:33:52 UTC
Marking this as an enhancement request for explicit suspend/resume support, since we've never had any.  If it worked in the past it worked by fluke, not by design.

Probably won't be able to get to this until at least 3.9.
Comment 2 Adam Williamson 2013-02-04 20:40:58 UTC
That seems pretty crazy, I mean, suspend/resume is expected functionality of GNOME - the GNOME 3 design pushes its use pretty heavily - and Evo is still the official GNOME mail client so far as I know, and this has worked for me for probably a decade. I'm sure you'd have heard a hell of a lot more yelling if suspend/resume was generally broken. It seems almost unacceptable in 2013 to say 'this core desktop application is not suspend/resume capable by design' - especially when the desktop it's a core part of is pushing the use of suspend/resume.
Comment 3 Matthew Barnes 2013-02-04 21:13:24 UTC
(In reply to comment #2)
> It seems almost unacceptable in 2013 to say 'this core desktop application
> is not suspend/resume capable by design' - especially when the desktop it's
> a core part of is pushing the use of suspend/resume.

That's why I'm proposing we support it for real by listening for sleeping() and resuming() signals from upower, and not relying on network status signals.

Granted it's long overdue.
Comment 4 Adam Williamson 2013-02-04 21:21:19 UTC
Is it too much to ask that someone put a half hour towards seeing if they can reproduce this specific failure and put a band-aid on it for now? It's really, really inconvenient for me, and if you're not going to fix it till 3.9, I'll probably wind up switching mail clients. I'm never going to remember to close evo every time I suspend, and waiting for it to reload 20,000 mails every time I resume doesn't seem like fun (though granted, it does it pretty fast these days).
Comment 5 Owen Taylor 2013-02-04 21:21:55 UTC
There does seem to be a new problem in recent versions where Evolution loses track of folder contents and refetches them. I've long had problems with suspend/resume but never had that particular problem before the last few months.
Comment 6 Matthias Clasen 2013-02-04 21:23:11 UTC
(In reply to comment #3)
> (In reply to comment #2)
> > It seems almost unacceptable in 2013 to say 'this core desktop application
> > is not suspend/resume capable by design' - especially when the desktop it's
> > a core part of is pushing the use of suspend/resume.
> 
> That's why I'm proposing we support it for real by listening for sleeping() and
> resuming() signals from upower, and not relying on network status signals.
> 
> Granted it's long overdue.

No, that would be a step backwards. You just need to deal with network going away at any time.
Comment 7 Ray Strode [halfline] 2013-02-04 21:23:21 UTC
FWIW, reacting to suspend/resume events is rarely ever right. These actions should be transparent to applications.  If the problem is "evolution loses all its folder contents when the network disconnects" then the fix should be "handle the network disappearing at inopportune moments better" not "do something to preemptively walk around the bug right before suspend"
Comment 8 Matthew Barnes 2013-02-04 22:29:06 UTC
(In reply to comment #6)
> No, that would be a step backwards. You just need to deal with network going
> away at any time.

We already do that, but back when we listened to NetworkManager directly we relied on its NMSleepMonitor [1] to broadcast a network state change just prior to suspend and just after resume.  GNetworkMonitor apparently doesn't do this.

Turns out NMSleepMonitor now prefers systemd, with upower as a fallback.

Fine, whatever.  I clarified the summary.

Point is, I'm merely proposing to reimplement NMSleepMonitor directly in Evolution-Data-Server.  I'd actually prefer it if GIO provided equivalent APIs, either as a GSleepMonitor class or just add "sleeping" and "resuming" signals to GApplication.  But doesn't sound like that proposal would get very far.


[1] http://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/src/nm-sleep-monitor.h
Comment 9 Ray Strode [halfline] 2013-02-04 23:02:04 UTC
but the bug isn't about suspend and resume is it? I mean the problem would happen if someone yanked the network cable for 10 minutes and then plugged it back in, right?
Comment 10 Adam Williamson 2013-02-04 23:49:11 UTC
In response to Ray (comment #9) - That may well be the case, it just looks like suspend/resume to the dumb testing monkey: I suspend, I walk away, I come back later, I resume, and all my mail's gone.

At Ray's suggestion I tested a very short suspend/resume cycle - suspend, wait five seconds, resume - and that doesn't trigger the bug. I'll try and find some time later tonight to just yank my network cable for 15 mins then replug, as he suggests, and see if that triggers the problem.

Owen noted on IRC that he's been seeing this issue too.
Comment 11 Matthew Barnes 2013-02-05 04:53:21 UTC
(In reply to comment #9)
> but the bug isn't about suspend and resume is it? I mean the problem would
> happen if someone yanked the network cable for 10 minutes and then plugged it
> back in, right?

No, Evo/EDS gets notified of network outages by way of GNetworkMonitor and can act accordingly.
Comment 12 Adam Williamson 2013-02-13 22:23:37 UTC
So for whatever it's worth, I'm pretty sure I saw this happen once even when I closed Evo before suspending. It could be that the bug is better described as 'Evo periodically loses all my folders and recreates them for some reason'.

Between this, https://bugzilla.redhat.com/show_bug.cgi?id=908143 , Google Calendar integration being apparently entirely broken, and a few other problems, I've moved to Thunderbird for now; current Rawhide Evo is just too broken to use (even for me, I have a pretty high tolerance for busted-ness).
Comment 13 Milan Crha 2013-02-14 19:52:54 UTC
Just happened to me too, no disconnect, no suspend. Only IMAPx.
Comment 14 Adam Williamson 2013-02-14 23:20:22 UTC
Yup, I am indeed using IMAPx.
Comment 15 Milan Crha 2013-02-18 14:27:17 UTC
Created attachment 236591 [details] [review]
eds patch

for evolution-data-server;

I thought this will be harder, but it turned out that the issue is with the cancel changes for imapx in master. The simplest reproducer is to move between folders quickly, thus there are piled "Searching for changed messages in..." operations in status bar, and then close Evolution. This calls cancel_all_jobs(), but it pretends that the job finished successfully, even it was cancelled instead. That way the refresh operation removes all the uids it didn't meet yet thinking they all are gone, for each piled folder refresh operation.

The patch makes sure the cancelled job is also recognized as finished with error, not otherwise.
Comment 16 Milan Crha 2013-02-18 14:28:22 UTC
Created commit ffb11c0 in eds master (3.7.91+)
Comment 17 André Klapper 2013-02-18 14:29:45 UTC
(In reply to comment #16)
> Created commit ffb11c0 in eds master (3.7.91+)

To make this clear: 3.6.x code is not affected?
Comment 18 Milan Crha 2013-02-18 17:35:02 UTC
(In reply to comment #17)
> To make this clear: 3.6.x code is not affected?

No, it isn't. The change which caused this was not done there, and even Adam claimed in comment #0 that this happened since he updated to 3.7.x.
Comment 19 Adam Williamson 2013-02-18 17:49:32 UTC
Yeah, my laptop's still running 3.6 and no problems there.
Comment 20 Adam Williamson 2013-03-01 13:41:40 UTC
[adamw@adam postfix]$ rpm -q evolution-data-server
evolution-data-server-3.7.90-3.fc19.x86_64

* Thu Feb 21 2013 Milan Crha <mcrha@redhat.com> - 3.7.90-3
- Add patch for GNOME bug #693101 (IMAPx vanishes folder summary)

...but I just hit this again on an overnight suspend. not fixed.
Comment 21 Milan Crha 2013-03-06 12:08:31 UTC
I'm unable to reproduce this with git master, which is currently pretty much like 3.7.91. I tried with my IMAP+ account which connects to a server accessible only through VPN. I waited long enough that the vpn connection was invalidated, then I woke-up the machine from suspend. The folder summary is left as is. I see error about connection above the message list, and the message list is empty, but once I reconnect the vpn and return back to the folder the message list is filled with all my 110K+ messages from summary like before, there is nothing redownloaded from the server like in case of pre-comment #15 patch. I tried to suspend the machine in various stages, once during "Opening folder" stage, another time when "Scanning for changed messages", but none of these led to vanish of a folder summary.

Is your environment similar to mine, or completely different? Could you try with 3.7.91, please? (It's part of rawhide since this Monday.) Maybe other changes around network change detection helped a bit here, though I'm pretty sure that during my testing the connection usually failed with timeout, due to vpn connection being down, but vpnc being still running.
Comment 22 Adam Williamson 2013-03-06 19:54:10 UTC
I'm using pure Rawhide. My server is my own IMAP server (Dovecot) on my local network, it is not behind a VPN.

I've seen this bug several times over the last few days, since re-opening it. It's definitely happening repeatedly here, though I still can't say precisely what triggers it. I'm back to mainly using Thunderbird, but I've been launching Evo since Thunderbird's 'all headers' view sucks, and often when I launch Evo, it comes up empty.

I have 3.7.91 installed and have since yesterday afternoon; I'll keep an eye out and see if this bug still happens (I can't recall with 100% certainty if I've seen it since yesterday afternoon yet).
Comment 23 Milan Crha 2013-03-07 07:59:14 UTC
Thanks. Let's try this: installed also debuginfo for evolution-data-server and evolution, then run evolution under gdb:

   $ gdb evolution --ex "b camel_folder_summary_remove_uids" --ex "b camel_folder_summary_clear" --ex r --ex "bt"

which adds two breakpoints, while I expect the first is triggered in this case, and prints a backtrace from where it is called. Evolution will be "frozen" when this happens, waiting in a gdb prompt. You can instruct gdb to simple continue:
   (gdb) c
and if it'll stop there again, then also print a backtrace:
   (gdb) bt
from where it is called. It may help to identify where it happened, and hopefully also why (it depends on the current session, to check what and where, but at least to know where it is I might be able to cook a concrete debug patch which will print state of the code and why got to that point).
Comment 24 Adam Williamson 2013-03-11 20:49:20 UTC
So I let Evo get to a state where the folders were in sync, rebooted, ran it with the command above, and before it even renders, I get:

Breakpoint 1, camel_folder_summary_remove_uids (summary=0x7be660, 
    uids=uids@entry=0x75e420) at camel-folder-summary.c:3465
3465	{
  • #0 camel_folder_summary_remove_uids
    at camel-folder-summary.c line 3465
  • #1 imapx_untagged_vanished
  • #2 imapx_untagged
    at camel-imapx-server.c line 2520
  • #3 imapx_step
    at camel-imapx-server.c line 2858
  • #4 parse_contents
    at camel-imapx-server.c line 6893
  • #5 imapx_parser_thread
    at camel-imapx-server.c line 6963
  • #6 g_thread_proxy
    at gthread.c line 798
  • #7 start_thread
    at pthread_create.c line 308
  • #8 clone
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 113
  • #0 camel_folder_summary_remove_uids
    at camel-folder-summary.c line 3465
  • #1 imapx_untagged_vanished
    at camel-imapx-server.c line 1637
  • #2 imapx_untagged
    at camel-imapx-server.c line 2520
  • #3 imapx_step
    at camel-imapx-server.c line 2858
  • #4 parse_contents
    at camel-imapx-server.c line 6893
  • #5 imapx_parser_thread
    at camel-imapx-server.c line 6963
  • #6 g_thread_proxy
    at gthread.c line 798
  • #7 start_thread
    at pthread_create.c line 308
  • #8 clone
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 113

Then I 'c', and Evo comes up looking okay. I'll leave it running now and see if I can catch it emptying.
Comment 25 Adam Williamson 2013-03-11 20:51:21 UTC
Oh, damn. Bugzilla intercepted my previous comment and did 'clever' things to it. I had text interspersed with the backtrace, Bugzilla. Don't fucking try and do clever things with it.

So, I ran Evo as instructed, and it printed 'Breakpoint 1...' as in comment #24 before the Evo window popped up at all. I hit 'return' then typed 'bt' and pasted in the results, which I think was just showing the same backtrace twice.
Comment 26 Ray Strode [halfline] 2013-03-11 21:23:54 UTC
Your comments are there if you click on Trace 231626
Comment 27 Adam Williamson 2013-03-12 05:58:31 UTC
Ooh, okay, so after running it all day, I went to the console and see it's hit a breakpoint. I don't know exactly when. Here's the requested output:

Breakpoint 1, camel_folder_summary_remove_uids (summary=0x7fffc04ab650, 
    uids=uids@entry=0x7fffb79ea8e0) at camel-folder-summary.c:3465
3465	{
Missing separate debuginfos, use: debuginfo-install google-talkplugin-3.15.2.0-1.x86_64 icedtea-web-1.3.1-5.fc19.x86_64
(gdb) c
Continuing.

(evolution:2377): GLib-GIO-WARNING **: Error on netlink socket: Error receiving message: No buffer space available
[New Thread 0x7fff42aa1700 (LWP 10897)]

Program received signal SIGPIPE, Broken pipe.

Thread 140734311831296 (LWP 10897)

  • #0 __libc_send
    at ../sysdeps/unix/sysv/linux/x86_64/send.c line 31
  • #1 pt_Send
    at ../../../mozilla/nsprpub/pr/src/pthreads/ptio.c line 1914
  • #2 ssl_DefSend
    at ssldef.c line 95
  • #3 ssl3_SendRecord
    at ssl3con.c line 2556
  • #4 ssl3_SendApplicationData
    at ssl3con.c line 2645
  • #5 ssl_SecureSend
    at sslsecur.c line 1237
  • #6 ssl_Send
    at sslsock.c line 2092
  • #7 write_to_prfd
    at camel-tcp-stream-raw.c line 380
  • #8 camel_stream_write
    at camel-stream.c line 158
  • #9 camel_stream_write
    at camel-stream.c line 158
  • #10 camel_stream_write_string
    at camel-stream.c line 265
  • #11 imapx_command_start
    at camel-imapx-server.c line 878
  • #12 imapx_command_start_next
    at camel-imapx-server.c line 1016
  • #13 imapx_command_queue
    at camel-imapx-server.c line 1294
  • #14 camel_imapx_job_run
    at camel-imapx-job.c line 216
  • #15 imapx_submit_job
    at camel-imapx-server.c line 3023
  • #16 camel_imapx_server_noop
    at camel-imapx-server.c line 7692
  • #17 imapx_store_noop_sync
    at camel-imapx-store.c line 1775
  • #18 camel_store_noop_sync
    at camel-store.c line 3155
  • #19 mail_msg_proxy
    at mail-mt.c line 426
  • #20 g_thread_pool_thread_proxy
    at gthreadpool.c line 309
  • #21 g_thread_proxy
    at gthread.c line 798
  • #22 start_thread
    at pthread_create.c line 308
  • #23 clone
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 113

What next?
Comment 28 Milan Crha 2013-03-12 09:08:41 UTC
Thanks for the update. You are right, comment #24 contains two same backtraces. Comment #27 contains backtrace of SIG_PIPE, I forgot gdb stops on these on its own. I'll provide a debug patch based on the comment #24's backtrace.
Comment 29 Milan Crha 2013-03-12 09:24:48 UTC
Aha, no patch is required to debug this, just do this:
   $ export CAMEL_DEBUG=imapx
   $ gdb evolution --ex "b camel_folder_summary_remove_uids" --ex "b
camel_folder_summary_clear" --ex r --ex "bt"

From the code reading it seems to me that your server reported to evolution that certain messages had been vanished [1]. From the RFC I guess the whole trouble is caused by QResync code, which doesn't seem to do the right thing here (it can be disabled in account preferences, in case you have it enabled).

[1] http://tools.ietf.org/html/rfc5162#section-3.6
Comment 30 Adam Williamson 2013-03-12 12:57:59 UTC
I recall qresync causing me lots of issues in an earlier release (3.4? 3.6?) before being fixed, so that sounds plausible; apparently it and dovecot (or at least my copy of dovecot) don't really get along.

I have run evo as described above - it's 'in sync' - and will wait for something exciting to happen!
Comment 31 Adam Williamson 2013-03-13 22:46:09 UTC
OK, it hit a break after a suspend/resume by the looks of it. Here's the output:

evolution-shell-Message: Network disconnected.  Forced offline.
[New Thread 0x7fff36aa1700 (LWP 16546)]
[New Thread 0x7fffc4ee1700 (LWP 16547)]
[Thread 0x7fffc4ee1700 (LWP 16547) exited]
[Thread 0x7fff36aa1700 (LWP 16546) exited]
evolution-shell-Message: Connection established.  Going online.
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Alsa'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Archives'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Archives.2013'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Barry'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Blog'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Blog.Backups'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Chix'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Chix.Techtalk'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Chix.WIFS'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Clarendon'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Damien'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Deleted Items'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Drafts'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.FDBugzilla'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Facebook'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.Board'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.Bodhi'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.Builds'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.Fedora-devel'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.Fedora-infrastructure-list'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.Fedora-python'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.Fedora-test'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.FedoraForum'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.Fusion'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.Fusion.Bugzilla'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.Fusion.Builds'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.LXDE'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.Marketing'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.Xfce'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.ambassadors'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.anaconda'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.arm'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.autoqa'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.autoqa.ambassadors'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.fedora-desktop'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.fedora-docs'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.fedora-news'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.i18n'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.kde'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.kickstart-list'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.qa-devel'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.releng'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Fedora.spins'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Gaym'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Gnome-shell'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.GnomeBugzilla'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.INBOX.Sent'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Jon'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Junk'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Lirc'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Mandriva'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Mandriva.Bugsquad'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Mandriva.Club-translators'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Mandriva.Cooker'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Mandriva.Engineering'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Mandriva.Maintainers'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Mandriva.PLF'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Mandriva.Security'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Mandriva.Triage'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Opensync'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Pembroke'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.RedHat'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.RedHat.Bugs'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.RedHat.Queue'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.RedHat.announce-list'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.RedHat.canada-list'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.RedHat.memo-list'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.RedHat.prod-dept'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.RedHat.qe-dept-list'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.RedHat.qe-fedora-list'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.RedHat.qe-staff-list'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.RedHat.remotees'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.RedHat.sst'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.RedHat.tech-list'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Sammy'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Sent'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Sent Items'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Skencil'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Systems'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Systems.HTPC'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Systems.Mailserver'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Systems.Webserver'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Systems.ircbounce'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Tcl'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.Trash'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.VanLUG'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.concordance-devel'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.eBay-Paypal'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.egroupware'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.mc'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.spam'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.spam.dupes'
[imapx:?] find_full: comparing namespace 'INBOX' to name 'INBOX.synce-devel'
[New Thread 0x7fff36aa1700 (LWP 16582)]
[New Thread 0x7fffc4ee1700 (LWP 16585)]
[New Thread 0x7fff362a0700 (LWP 16591)]
[imapx:A] Removed folder INBOX from connection folder list - op done 
[imapx:A] adding command, format = 'STATUS %f (MESSAGES UNSEEN UIDVALIDITY UIDNEXT HIGHESTMODSEQ)'
[imapx:A] got folder 'INBOX'
[imapx:?] looking up path INBOX -> INBOX
[imapx:A] completing command buffer is [64] 'STATUS INBOX (MESSAGES UNSEEN UIDVALIDITY UIDNEXT HIGHESTMODSEQ)'
[imapx:A] enqueue job 'STATUS INBOX (MESSAGES UNSEEN UIDVALIDITY UIDNEXT HIGHESTMODSEQ)'
[imapx:A] ** Starting next command
[imapx:A] camel_imapx_write: 'DONE
'
[imapx:A] Stopping idle after 33367 seconds
[imapx:A] waiting for idle to stop 
[imapx:A] - we're selected on 'INBOX/Facebook', current jobs?
[imapx:A] -    0 'IDLE'
[imapx:A] -- Checking job queue
[imapx:A] --  10 'STATUS'?
[imapx:A] --> starting 'STATUS'
[imapx:A] completing command buffer is [0] ''
[imapx:A] Starting command (active=2,) A112293 STATUS INBOX (MESSAGES UNSEEN UIDVALIDITY UIDNEXT HIGHESTMODSEQ)
[imapx:A] camel_imapx_write: 'A112293 STATUS INBOX (MESSAGES UNSEEN UIDVALIDITY UIDNEXT HIGHESTMODSEQ)
'
[imapx:A] adding command, format = '%s "" %s'
[imapx:A] got string 'LIST'
[imapx:A] got string '*'
[imapx:A] adding command, format = ' '
[imapx:A] adding command, format = 'RETURN (SUBSCRIBED)'
[imapx:A] completing command buffer is [31] 'LIST "" "*" RETURN (SUBSCRIBED)'
[imapx:A] enqueue job 'LIST "" "*" RETURN (SUBSCRIBED)'
[imapx:A] ** Starting next command
[imapx:A] - we're selected on 'INBOX/Facebook', current jobs?
[imapx:A] -    0 'IDLE'
[imapx:A] -   10 'STATUS'
[imapx:A] -- Checking job queue
[imapx:A] camel_imapx_read: -1
[imapx:A] * queueing job -80 'LIST'
[imapx:A] completing command buffer is [0] ''
[imapx:A] Starting command (active=3,) A112294 LIST "" "*" RETURN (SUBSCRIBED)
[imapx:A] camel_imapx_write: 'A112294 LIST "" "*" RETURN (SUBSCRIBED)
'
[imapx:A] ==== list or lsub completed ==== 
[Thread 0x7fff477fe700 (LWP 12522) exited]

Program received signal SIGPIPE, Broken pipe.

Thread 140736505718528 (LWP 12512)

  • #0 __libc_send
  • #1 pt_Send
    at ../../../mozilla/nsprpub/pr/src/pthreads/ptio.c line 1914
  • #2 ssl_DefSend
    at ssldef.c line 95
  • #3 ssl3_SendRecord
    at ssl3con.c line 2556
  • #4 SSL3_SendAlert
    at ssl3con.c line 2847
  • #5 ssl_SecureShutdown
    at sslsecur.c line 1087
  • #6 ssl_Shutdown
    at sslsock.c line 2017
  • #7 tcp_stream_raw_close
    at camel-tcp-stream-raw.c line 443
  • #8 camel_stream_close
    at camel-stream.c line 217
  • #9 camel_stream_close
  • #10 imapx_disconnect
    at camel-imapx-server.c line 7326
  • #11 imapx_server_dispose
    at camel-imapx-server.c line 7108
  • #12 g_object_unref
    at gobject.c line 2987
  • #13 g_signal_emit_valist
    at gsignal.c line 3232
  • #14 g_signal_emit
    at gsignal.c line 3370
  • #15 imapx_parser_thread
    at camel-imapx-server.c line 7014
  • #16 g_thread_proxy
    at gthread.c line 798
  • #17 start_thread
    at pthread_create.c line 308
  • #18 clone
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 113

Comment 32 Milan Crha 2013-03-14 18:31:27 UTC
Thanks, unfortunately the gdb stopped on the SIGPIPE, instead of our breakpoint. Let's try with QResync off, without gdb, with imapx debugging on, whether you'll see any folder vanished.
Comment 33 Milan Bouchet-Valat 2013-03-18 22:17:40 UTC
FWIW, the issue of redownloading folder summary happened to me several times with a very fragile network connection and Evo 3.4 and 3.6, maybe even 3.2. I've never used 3.7. (Just in case this helps finding the problematic code.)
Comment 34 Adam Williamson 2013-03-18 22:26:58 UTC
It has never happened to me before 3.7. Sorry Milan, got caught up in other bugs; I'll report back on comment #32 soon.
Comment 35 Adam Williamson 2013-03-19 06:05:26 UTC
So this just happened on a resume, with qresync disabled. Attaching the freaking _gigantic_ console output extract. So far as I can tell - if I scroll any further I'm going to get RSI - everything before and after this extract is just vast tracts of the "token '*'"..."token '" sequence with different token numbers.
Comment 36 Adam Williamson 2013-03-19 06:07:25 UTC
Created attachment 239218 [details]
console output from the bug happening on a resume from suspend, with qresync disabled, imapx debugging output enabled (gzipped: too big for text attachment)
Comment 37 Milan Crha 2013-03-19 18:25:54 UTC
After a chat with Adam on IRC, the issue on Adam's side is that all folders are vanished, not only content of one folder, which I thought is an issue here.
Comment 38 Milan Crha 2013-03-21 18:27:16 UTC
Created attachment 239487 [details] [review]
eds patch ][

for evolution-data-server;

We did some private testing with Adam, he currently uses patched eds which prints 'oops' in a situation when it would drop any of the folders - it prints couple more things. I'd like to get a confirmation from Adam too, though it just happened to me as well, all my folders gone, then added back when I moved between two networks, no suspend, just moved between networks. Thus I take a step forward and attach a patch which prevents this from happening.

The reason was that the sync_folders() function relied on returned data from fetch_folders_for_namespaces(), but that function can return incomplete set of folders in certain situations, which makes IMAPX think that the folders which are not part of the returned hash table are effectively gone. Properly checking for either error or if the operation was already cancelled makes this unfortunate folder list vanish fixed.
Comment 39 André Klapper 2013-03-21 18:49:15 UTC
Should this be something to get in for 3.8.0? If so please ask release-team@ for a hardcode freeze exception.
Comment 40 Adam Williamson 2013-03-21 18:53:42 UTC
Hey Milan - I haven't hit the problem with the patched e-d-s yet, mainly because I keep hitting intel_pstate hangs in the kernel I think, so I'm not running long enough to see it (I still mostly see this one when I suspend overnight). I'll try and catch it again soon.
Comment 41 Matthew Barnes 2013-03-21 19:01:19 UTC
(In reply to comment #39)
> Should this be something to get in for 3.8.0? If so please ask release-team@
> for a hardcode freeze exception.

I'd prefer the fix simmer for a bit in the master branch just to verify there's no unwanted side-effects, then backport it for 3.8.1.

Hacking on complex and delicate IMAP code during a final code freeze is just tempting fate.  I've learned that the hard way.
Comment 42 Milan Crha 2013-03-22 07:22:08 UTC
(In reply to comment #41)
> Hacking on complex and delicate IMAP code during a final code freeze is just
> tempting fate.  I've learned that the hard way.

I do not know if you read the right patch, but this one is pretty straightforward change, which definitely doesn't belong into "complex and delicate IMAP code", it's just a better error checking and nothing more.

(In reply to comment #39)
> Should this be something to get in for 3.8.0? If so please ask release-team@
> for a hardcode freeze exception.

Right, I wanted to do that after Adam's confirmation, even I saw that occurrence on my machine too. Pity Adam faces other bugs too.
Comment 43 Adam Williamson 2013-03-22 16:19:05 UTC
mcrha: sorry about that :/ I am trying to hit this bug (so I need to suspend!) and a kernel crash which is obstinately refusing to reproduce itself (so I need to stay running!) and transfer a bunch of data to my new NAS (so I need to stay running!) all at once. le sigh.
Comment 44 Matthew Barnes 2013-03-22 16:27:27 UTC
You could probably simulate a suspend (or the effects of a suspend that are relevant to Camel) by disabling networking, waiting a few minutes for the IMAP server's sockets to time out, and then reconnecting.
Comment 45 Adam Williamson 2013-03-22 16:34:53 UTC
I tried that back when we were still figuring out where the bug was, it didn't work, but then I only tried once. I guess I can use that as a whack-it-with-a-stick option if I haven't reproduced it soon. See under 'NAS transfer', though.
Comment 46 Bastien Nocera 2013-03-23 08:09:04 UTC
It's a regular occurrence on my laptop too. Would you have some F19 RPMs for me to test with?
Comment 47 Adam Williamson 2013-03-23 08:19:27 UTC
I believe http://koji.fedoraproject.org/koji/taskinfo?taskID=5143488 is the scratch build I'm using.
Comment 48 Adam Williamson 2013-03-23 08:20:11 UTC
And I believe right now mcrha wants us to be running with qresync disabled and saving the console output - the patched e-d-s should print something useful for him to the console.
Comment 49 Sjoerd Simons 2013-03-23 10:11:48 UTC
I've been testing with this patch for a day or so, going through several disconnects and suspends. I haven't yet hit the 
issue while before i did see it on a very regular basis. So things are looking positive.
Comment 50 Adam Williamson 2013-03-23 18:14:40 UTC
well the patch isn't supposed to _fix_ it, AIUI, just to print useful debug output...
Comment 51 Sjoerd Simons 2013-03-25 10:13:02 UTC
(In reply to comment #50)
> well the patch isn't supposed to _fix_ it, AIUI, just to print useful debug
> output...

The first patch did that, the second patch (from comment 38) is a proper bugfix in my reading
Comment 52 Milan Crha 2013-03-25 10:23:05 UTC
(In reply to comment #51)
> (In reply to comment #50)
> > well the patch isn't supposed to _fix_ it, AIUI, just to print useful debug
> > output...
> 
> The first patch did that, the second patch (from comment 38) is a proper bugfix
> in my reading

Right, while I shared a debug patch with Adam, the patch at comment #38 is strip from all the debugging stuff and holds only the relevant change, a fix.
I'm committing this for 3.8.1 shortly. Thanks to all for the testing.
Comment 53 Sjoerd Simons 2013-03-25 13:19:42 UTC
For reference i've added this patch to the Debian package. In my personal opinion/experience e-d-s without this patch simply isn't usable with imapx on a laptop, so ideally this should hit 3.8.0.
Comment 54 Milan Crha 2013-03-25 13:23:20 UTC
Created commit 4e908c9 in eds master (3.9.1+)
Created commit 7a52e0c in eds gnome-3-8 (3.8.1+)
Comment 55 Nirbheek Chauhan 2013-03-25 13:25:59 UTC
This is a very debilitating bug. Every single user of Evolution will hit this
bug at some point. I don't think I need to reiterate just how bad that is.

I've added this for Gentoo (thanks, sjoerd!), so we won't hit that, but in view of the bug's seriousness, why isn't this queued for 3.8?
Comment 56 Milan Crha 2013-03-25 14:33:29 UTC
(In reply to comment #55)
> This is a very debilitating bug. Every single user of Evolution will hit this
> bug at some point. I don't think I need to reiterate just how bad that is.

I didn't hit this bug myself, only after Adam told me/here. It just depends
on a usage, and my usage is obviously different from that Adam's or yours.
Comment 57 Adam Williamson 2013-03-25 15:07:20 UTC
mcrha: ooh, I somehow got out of sync and missed your fix. Thanks. I also would support having this in 3.8.0, but I can see either way.
Comment 58 Tobias Mueller 2013-06-10 08:48:16 UTC
*** Bug 693078 has been marked as a duplicate of this bug. ***
Comment 59 Adam Williamson 2013-08-03 15:48:54 UTC
This, or something very like it, has shown up again in 3.9.5 :( It was not happening until now in the 3.9 series.

Note that prior to 3.9.5 it would not not go back online when I came out of suspend automatically; it would still be in offline mode after each resume, and I'd have to manually put it in online mode. That is fixed in 3.9.5, but I've seen it lose the folder summary twice now.
Comment 60 Adam Williamson 2013-08-15 16:19:19 UTC
Just saw this again with 3.9.5-3.fc20...
Comment 61 Adam Williamson 2013-09-17 20:53:36 UTC
Now on evolution-3.9.91-2.fc20.x86_64 , will keep an eye on it.
Comment 62 Milan Crha 2013-11-27 12:36:02 UTC
The 3.10.x should be fine, according to bug #704513 comment #25. I'm closing this for now, but please feel to continue either here or at bug #704513.