After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 729546 - [IMAPx] Crash after large message download cancel
[IMAPx] Crash after large message download cancel
Status: RESOLVED FIXED
Product: evolution-data-server
Classification: Platform
Component: Mailer
3.12.x (obsolete)
Other Linux
: Normal critical
: ---
Assigned To: evolution-mail-maintainers
Evolution QA team
: 729997 730061 732173 732455 (view as bug list)
Depends on:
Blocks:
 
 
Reported: 2014-05-05 00:38 UTC by Ankur Sinha (FranciscoD)
Modified: 2014-07-01 09:37 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Complete multi thread backtrace (53.15 KB, text/x-log)
2014-05-14 05:45 UTC, Ankur Sinha (FranciscoD)
Details

Description Ankur Sinha (FranciscoD) 2014-05-05 00:38:21 UTC
[asinha@ankur-laptop  ~]$ rpm -qa \*evolution\*
evolution-debuginfo-3.12.1-1.fc20.x86_64
evolution-ews-3.12.1-1.fc20.x86_64
evolution-data-server-3.12.1-1.fc20.x86_64
evolution-help-3.12.1-1.fc20.noarch
evolution-3.12.1-1.fc20.x86_64

  • #0 g_queue_peek_head_link
    at gqueue.c line 579
  • #1 imapx_command_start
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #2 imapx_command_start_next
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #3 imapx_command_queue
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #4 imapx_command_fetch_message_done
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #5 imapx_step
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #6 imapx_ready_to_read
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #7 g_main_dispatch
    at gmain.c line 3064
  • #8 g_main_context_dispatch
    at gmain.c line 3663
  • #9 g_main_context_iterate
    at gmain.c line 3734
  • #10 g_main_loop_run
    at gmain.c line 3928
  • #11 imapx_parser_thread
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #12 g_thread_proxy
    at gthread.c line 764
  • #13 start_thread
    at pthread_create.c line 309
  • #14 clone
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 111


I've been running evolution in gdb to catch this crash, since abrt doesnt work with COPR stuff. I hope this stack trace is useful. The crash happens quite often, but quite randomly.
Comment 1 Ankur Sinha (FranciscoD) 2014-05-05 00:40:52 UTC
Another crash:

  • #0 g_queue_peek_head_link
    at gqueue.c line 579
  • #1 imapx_command_start
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #2 imapx_command_start_next
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #3 imapx_command_queue
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #4 imapx_command_fetch_message_done
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #5 imapx_command_start
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #6 imapx_command_start_next
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #7 imapx_command_queue
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #8 imapx_command_fetch_message_done
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #9 imapx_command_start
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #10 imapx_command_start_next
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #11 imapx_command_queue
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #12 imapx_command_fetch_message_done
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #13 imapx_command_start
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #14 imapx_command_start_next
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #15 imapx_step
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #16 imapx_ready_to_read
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #17 g_main_dispatch
    at gmain.c line 3064
  • #18 g_main_context_dispatch
    at gmain.c line 3663
  • #19 g_main_context_iterate
    at gmain.c line 3734
  • #20 g_main_loop_run
    at gmain.c line 3928
  • #21 imapx_parser_thread
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #22 g_thread_proxy
    at gthread.c line 764
  • #23 start_thread
    at pthread_create.c line 309
  • #24 clone
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 111

Comment 2 Ankur Sinha (FranciscoD) 2014-05-05 00:42:46 UTC
And another:

  • #0 camel_imapx_command_close
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #1 imapx_command_start
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #2 imapx_command_start_next
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #3 imapx_command_queue
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #4 imapx_command_fetch_message_done
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #5 imapx_step
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #6 imapx_ready_to_read
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #7 g_main_dispatch
    at gmain.c line 3064
  • #8 g_main_context_dispatch
    at gmain.c line 3663
  • #9 g_main_context_iterate
    at gmain.c line 3734
  • #10 g_main_loop_run
    at gmain.c line 3928
  • #11 imapx_parser_thread
    from /usr/lib64/evolution-data-server/camel-providers/libcamelimapx.so
  • #12 g_thread_proxy
    at gthread.c line 764
  • #13 start_thread
    at pthread_create.c line 309
  • #14 clone
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 111

Comment 3 André Klapper 2014-05-06 12:54:01 UTC
Please install the debug package for evolution-data-server so the traces have filename, functionname and linenumber information.
Comment 4 Ankur Sinha (FranciscoD) 2014-05-06 13:43:36 UTC
(In reply to comment #3)
> Please install the debug package for evolution-data-server so the traces have
> filename, functionname and linenumber information.

Apologies. I've installed the debuginfo package. I'll update the bug with fresh stack traces when evolution crashes next time.

Thanks,
Warm regards,
Ankur
Comment 5 Ankur Sinha (FranciscoD) 2014-05-08 01:22:46 UTC
I think this should be a proper back trace:


  • #0 g_queue_peek_head_link
    at gqueue.c line 579
  • #1 imapx_command_start
    at camel-imapx-server.c line 1413
  • #2 imapx_command_start_next
    at camel-imapx-server.c line 1786
  • #3 imapx_command_queue
    at camel-imapx-server.c line 1947
  • #4 imapx_command_fetch_message_done
    at camel-imapx-server.c line 5283
  • #5 imapx_completion
    at camel-imapx-server.c line 3622
  • #6 imapx_step
    at camel-imapx-server.c line 3665
  • #7 imapx_ready_to_read
    at camel-imapx-server.c line 7783
  • #8 g_main_dispatch
    at gmain.c line 3064
  • #9 g_main_context_dispatch
    at gmain.c line 3663
  • #10 g_main_context_iterate
    at gmain.c line 3734
  • #11 g_main_loop_run
    at gmain.c line 3928
  • #12 imapx_parser_thread
    at camel-imapx-server.c line 7869
  • #13 g_thread_proxy
    at gthread.c line 764
  • #14 start_thread
    at pthread_create.c line 309
  • #15 clone
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 111

Comment 6 Ankur Sinha (FranciscoD) 2014-05-08 01:26:32 UTC
Just had another crash:

  • #0 imapx_server_command_removed
    at camel-imapx-server.c line 623
  • #1 imapx_command_start_next
    at camel-imapx-server.c line 1785
  • #2 imapx_completion
    at camel-imapx-server.c line 3625
  • #3 imapx_step
    at camel-imapx-server.c line 3665
  • #4 imapx_ready_to_read
    at camel-imapx-server.c line 7783
  • #5 g_main_dispatch
    at gmain.c line 3064
  • #6 g_main_context_dispatch
    at gmain.c line 3663
  • #7 g_main_context_iterate
    at gmain.c line 3734
  • #8 g_main_loop_run
    at gmain.c line 3928
  • #9 imapx_parser_thread
    at camel-imapx-server.c line 7869
  • #10 g_thread_proxy
    at gthread.c line 764
  • #11 start_thread
    at pthread_create.c line 309
  • #12 clone
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 111

Comment 7 André Klapper 2014-05-11 09:00:59 UTC
Full traces welcome with all threads.
Comment 8 Ankur Sinha (FranciscoD) 2014-05-12 08:41:38 UTC
(In reply to comment #7)
> Full traces welcome with all threads.

Ran into a dnf debuginfo bug. Will get this done ASAP

https://bugzilla.redhat.com/show_bug.cgi?id=1096507
Comment 9 Milan Crha 2014-05-13 07:16:54 UTC
*** Bug 729997 has been marked as a duplicate of this bug. ***
Comment 10 Milan Crha 2014-05-13 07:18:30 UTC
Thanks for a bug report. The backtraces are fine. I didn't get any of it myself yet, unfortunately.
Comment 11 Ankur Sinha (FranciscoD) 2014-05-14 05:45:32 UTC
Created attachment 276509 [details]
Complete multi thread backtrace

Hi,

Please find the complete backtrace attached. It's quite large. It was another random crash. I hope they're all related.

Thanks,
Warm regards,
Ankur
Comment 12 Milan Crha 2014-05-14 10:39:49 UTC
(In reply to comment #11)
> Please find the complete backtrace attached. It's quite large. It was another
> random crash. I hope they're all related.

It's the same as comment #5. I think you can stop adding more backtraces here, they are repeating. More interesting would be to know your IMAP settings and the reason why this happened, but the later is hard to guess. I know the rough circumstances, it's when a certain folder is updating, but I do not see a reason why it does what it does (yet).
Comment 13 Milan Crha 2014-05-14 16:34:42 UTC
I just noticed that you are running Fedora with 3.12.1 of evolution-data-server. Could you update it to 3.12.2, once it'll be available, please? Just in case, because there had been done many changes between 3.12.1 and 3.12.2 in the IMAPx code, maybe some of them fixed this as a side effect.
Comment 14 Milan Crha 2014-05-14 16:59:51 UTC
(In reply to comment #13)
> I just noticed that you are running Fedora with 3.12.1 of
> evolution-data-server. Could you update it to 3.12.2, once it'll be available,
> please? Just in case, because there had been done many changes between 3.12.1
> and 3.12.2 in the IMAPx code, maybe some of them fixed this as a side effect.

Oops, no, I take this bag, the current git master version suffers of the same issue, I just managed to reproduce this locally.
Comment 15 Milan Crha 2014-05-15 11:09:56 UTC
The main issue with the GET_MESSAGE job was that it could be done in chunks, if the message was large enough. These chunks meant multiple commands of the job. Once the command ends there was scheduled another command, in a way of remembering GList link to it, but the the call of imapx_command_start_next() could be called recursively, as is shown in comment #1, thus reusing the same GList link, which could be freed in the upper call, which means a bang.

Another issue was with "tmp" files, when downloading messages. Cancelled jobs were "finished" immediately, which means that if there was waiting any other message download for this result, then it was released before the previous command completely finished, and reused the "tmp" file. The initial command them deleted the "tmp" file and the next command, now running, reported various errors, usually something like "Failed to copy the tmp file: No such file or directory".

Yet another issue could be (I guess this one, but the others are verified) with error "Empty cache file". That could happen when the "tmp" contained a 0-sized file. These can be there after above issue, I guess.

Yet another issue could be that some commands/jobs were starving in a queue, thus the UI could show something like "Retrieving message '3'...", but the imapx_parser_thread() was effectively idle and waiting for jobs.

There was also mentioned that job->commands property could be modified from multiple thread, to which I agree, thus I made it use memory barriers (by g_atomic_int_... functions).

I also added a nice debugging function "camel_imapx_store_dump_queue_status()", which shows what the store thinks is pending.

All this is committed in sources:

Created commit 2d4f35c in eds master (3.13.2+) [1]
Created commit 1479f26 in eds evolution-data-server-3-12 (3.12.3+)

[1] https://git.gnome.org/browse/evolution-data-server/commit/?id=2d4f35c
Comment 16 Milan Crha 2014-05-28 17:26:53 UTC
*** Bug 730061 has been marked as a duplicate of this bug. ***
Comment 17 Ankur Sinha (FranciscoD) 2014-06-11 02:59:14 UTC
Hi Milan,

Any chance these fixes could be pushed to the 3.12 COPR repository please? Evo is still crashing for me here since the repo is still at 3.12.2, and I can't find any information on when the repo will be updated to 3.12.3 - I can't test it out yet :/

Thanks,
Warm regards,
Ankur
Comment 18 Milan Crha 2014-06-11 07:11:18 UTC
The Fedora COPR should be updated by the end of this week, I guess, there were done changes on Monday, but the build may take some time, together with the change propagation and so on.
Comment 19 Ankur Sinha (FranciscoD) 2014-06-12 16:16:50 UTC
(In reply to comment #18)
> The Fedora COPR should be updated by the end of this week, I guess, there were
> done changes on Monday, but the build may take some time, together with the
> change propagation and so on.

Just received the update. No crashes yet. Thanks Milan! :)
Comment 20 Milan Crha 2014-06-26 17:51:03 UTC
*** Bug 732173 has been marked as a duplicate of this bug. ***
Comment 21 Milan Crha 2014-07-01 09:37:01 UTC
*** Bug 732455 has been marked as a duplicate of this bug. ***