After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 684238 - SoupConnection:state thread-safety issues
SoupConnection:state thread-safety issues
Status: RESOLVED FIXED
Product: libsoup
Classification: Core
Component: HTTP Transport
2.39.x
Other Linux
: Normal critical
: ---
Assigned To: libsoup-maint@gnome.bugs
libsoup-maint@gnome.bugs
Depends on:
Blocks:
 
 
Reported: 2012-09-17 17:32 UTC by Lionel Landwerlin
Modified: 2012-11-15 14:20 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
maybe-workaround (912 bytes, patch)
2012-09-28 12:25 UTC, Dan Winship
none Details | Review
SoupConnection: fix a race condition with non-keepalive messages (5.04 KB, patch)
2012-10-02 13:01 UTC, Dan Winship
committed Details | Review

Description Lionel Landwerlin 2012-09-17 17:32:05 UTC
Reproduced on 638aa6e28d708152cafe7ddbced952f5af2e6237 :

(plop:28931): libsoup-CRITICAL **: soup_socket_get_iostream: assertion `SOUP_IS_SOCKET (sock)' failed

(plop:28931): GLib-GObject-CRITICAL **: g_object_ref: assertion `G_IS_OBJECT (object)' failed
[Thread 0x7fffeccc3700 (LWP 28957) exited]

Program received signal SIGSEGV, Segmentation fault.

Thread 140737085699840 (LWP 28959)

  • #0 g_io_stream_get_input_stream
    at giostream.c line 221
  • #1 new_iostate
    at soup-message-io.c line 1017
  • #2 soup_message_io_client
    at soup-message-io.c line 1050
  • #3 soup_message_send_request
    at soup-message-client-io.c line 153
  • #4 soup_connection_send_request
    at soup-connection.c line 994
  • #5 soup_session_send_queue_item
    at soup-session.c line 1078
  • #6 process_queue_item
    at soup-session-sync.c line 257
  • #7 queue_message_thread
    at soup-session-sync.c line 304
  • #8 g_thread_proxy
    at gthread.c line 801
  • #9 start_thread
    at pthread_create.c line 304
  • #10 clone
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 112
  • #11 ??

Comment 1 Lionel Landwerlin 2012-09-19 14:51:58 UTC
Ok reproduced on top of master : 

Backtrace from first error message :

(aldgate-pc:18767): GLib-GIO-CRITICAL **: g_socket_condition_check: assertion `G_IS_SOCKET (socket)' failed

Program received signal SIGTRAP, Trace/breakpoint trap.

Thread 140737166194432 (LWP 18779)

  • #0 g_logv
    at gmessages.c line 967
  • #1 g_log
    at gmessages.c line 1003
  • #2 g_return_if_fail_warning
  • #3 g_socket_condition_check
    at gsocket.c line 3369
  • #4 soup_connection_get_state
    at soup-connection.c line 927
  • #5 soup_session_cleanup_connections
    at soup-session.c line 1100
  • #6 get_connection
    at soup-session-sync.c line 159
  • #7 process_queue_item
    at soup-session-sync.c line 242
  • #8 queue_message_thread
    at soup-session-sync.c line 304
  • #9 g_thread_proxy
    at gthread.c line 801
  • #10 start_thread
    at pthread_create.c line 304
  • #11 clone
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 112
  • #12 ??

Comment 2 Lionel Landwerlin 2012-09-19 14:53:51 UTC
Private structure from within soup_connection_get_state :

(gdb) p *priv        
$4 = {socket = 0x0, remote_uri = 0x2aea680, proxy_uri = 0x0, proxy_resolver = 0x0, use_gproxyresolver = 0, tlsdb = 0x0, ssl = 0, ssl_strict = 0, ssl_fallback = 0, async_context = 0x0, use_thread_context = 0, cur_item = 0x0, state = SOUP_CONNECTION_DISCONNECTED, 
  unused_timeout = 0, io_timeout = 0, idle_timeout = 0, idle_timeout_src = 0x0, reusable = 1}
Comment 3 Lionel Landwerlin 2012-09-27 15:49:10 UTC
Adding bt all :

(aldgate-pc:9010): GLib-GIO-CRITICAL **: g_socket_condition_check: assertion `G_IS_SOCKET (socket)' failed

Program received signal SIGTRAP, Trace/breakpoint trap.

Thread 40 (Thread 0x7fffecc3e700 (LWP 9054))

  • #0 __lll_lock_wait
    from /lib/x86_64-linux-gnu/libpthread.so.0
  • #1 _L_lock_1145
    from /lib/x86_64-linux-gnu/libpthread.so.0
  • #2 pthread_mutex_lock
    from /lib/x86_64-linux-gnu/libpthread.so.0
  • #3 g_mutex_lock
    at gthread-posix.c line 208
  • #4 connection_disconnected
    at soup-session.c line 1155
  • #5 _g_closure_invoke_va
    at gclosure.c line 840
  • #6 g_signal_emit_valist
    at gsignal.c line 3211
  • #7 g_signal_emit
    at gsignal.c line 3356
  • #8 soup_connection_disconnect
    at soup-connection.c line 882
  • #9 clear_current_item
    at soup-connection.c line 452
  • #10 soup_connection_set_state
    at soup-connection.c line 952
  • #11 soup_session_unqueue_item
    at soup-session.c line 1337
  • #12 process_queue_item
    at soup-session-sync.c line 275
  • #13 queue_message_thread
    at soup-session-sync.c line 304
  • #14 g_thread_proxy
    at gthread.c line 797
  • #15 start_thread
    from /lib/x86_64-linux-gnu/libpthread.so.0
  • #16 clone
    from /lib/x86_64-linux-gnu/libc.so.6
  • #17 ??

Comment 4 Dan Winship 2012-09-28 12:25:10 UTC
Created attachment 225328 [details] [review]
maybe-workaround

This should fix the specific case in the backtrace, though it's not a
complete fix.
Comment 5 Dan Winship 2012-10-02 13:01:09 UTC
Created attachment 225583 [details] [review]
SoupConnection: fix a race condition with non-keepalive messages

When a SoupSession sets a connection back to IDLE on a non-keepalive
message, the connection then disconnects itself. However, it had been
briefly setting its state to IDLE before disconnecting. Although this
wasn't supposed to be observable (it doesn't emit a notification), in
a SoupSessionSync, it could be observed by another thread, which might
then try to claim the connection to send another request on, causing
problems when the first thread then disconnected it.

Fix this by inlining clear_current_item() into
soup_connection_set_state() and reordering the code to not change
priv->state until after deciding whether or not it's going to
disconnect.
Comment 6 Dan Winship 2012-10-02 13:03:44 UTC
original actually broke more than it fixed. This patch should fix this specific race condition, and it small enough to be safe to 2.40.1; there are still other potential race conditions caused by the fact that SoupConnection:state can change when the session isn't holding conn_lock, but fixing the problem in general will take a lot more changes.

Can you test this patch to see if it fixes the crash for you?
Comment 7 Lionel Landwerlin 2012-10-02 13:08:32 UTC
Sure, thanks for that fix.
However I don't reproduce easily, so if it doesn't happen for a week it'll probably be good.
Comment 8 Milan Crha 2012-10-17 07:48:37 UTC
Similar downstream bug report from evoltuion-3.2.3 (rather old):
https://bugzilla.redhat.com/show_bug.cgi?id=867069

Thread 1 (Thread 0x7ff5c9427700 (LWP 6325))

  • #0 soup_session_cleanup_connections
    at soup-session.c line 1272
  • #1 run_queue
    at soup-session-async.c line 432
  • #2 idle_run_queue
    at soup-session-async.c line 465
  • #3 g_main_dispatch
    at gmain.c line 2441
  • #4 g_main_context_dispatch
    at gmain.c line 3011
  • #5 g_main_context_iterate
    at gmain.c line 3089
  • #6 g_main_loop_run
    at gmain.c line 3297
  • #7 e_ews_soup_thread
    at e-ews-connection.c line 796
  • #8 g_thread_create_proxy
    at gthread.c line 1962
  • #9 start_thread
    at pthread_create.c line 309
  • #10 clone
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 115

Comment 9 Lionel Landwerlin 2012-10-17 09:12:41 UTC
Hey there, I almost forgot to get back to that bug.
Basically I haven't had a single crash with the attached patch.
So I guess we can close the issue.

Thanks Dan.
Comment 10 Dan Winship 2012-10-18 16:51:02 UTC
committed the fix (but unfortunately forgot about this bug pre-2.40.1, so it will have to wait until 2.40.2). Leaving the bug open for the larger problem (comment 6)

Milan: that bug is using SoupSessionAsync, so it shouldn't be this bug (since you're not allowed to use an async session from multiple threads at once).
Comment 11 Milan Crha 2012-10-19 09:50:43 UTC
(In reply to comment #10)
> Milan: that bug is using SoupSessionAsync, so it shouldn't be this bug (since
> you're not allowed to use an async session from multiple threads at once).

Oh, it seemed similar to me, at least from the crashing function (comment #1) point of view. Looking into the backtrace itself, in comment #8, it is from EWS, and there had been done various fixes with thread-accessing libsoup's async session already, thus it should be fixed in 3.6.x of evolution-ews already, only between multiple other bugs.
Comment 12 Dan Winship 2012-11-15 14:20:06 UTC
(In reply to comment #6)
> there are still other
> potential race conditions caused by the fact that SoupConnection:state can
> change when the session isn't holding conn_lock, but fixing the problem in
> general will take a lot more changes.

This was fixed in master 2 weeks ago; I forgot that this bug was still open.