After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 712570 - Hang in g_threaded_socket_service_func
Hang in g_threaded_socket_service_func
Status: RESOLVED FIXED
Product: glib
Classification: Platform
Component: network
2.39.x
Other Linux
: Normal normal
: ---
Assigned To: gtkdev
gtkdev
Depends on:
Blocks:
 
 
Reported: 2013-11-18 10:19 UTC by Ross Lagerwall
Modified: 2014-12-07 06:56 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
gio: Prevent hang when finalizing GThreadedSocketService (2.83 KB, patch)
2014-10-28 21:29 UTC, Ross Lagerwall
committed Details | Review
program to reproduce the problem (1.09 KB, text/plain)
2014-10-28 21:33 UTC, Ross Lagerwall
  Details
gio/tests: Prevent hangs and aborts in socket-listener (2.10 KB, patch)
2014-11-30 22:21 UTC, Ross Lagerwall
committed Details | Review

Description Ross Lagerwall 2013-11-18 10:19:31 UTC
gvfs creates and frees a lot of GDBusServers.  Each of these has a GThreadedSocketService which makes use of a threadpool.  I noticed that sometimes threads and fds are leaked:

  37   Thread 0x7fd873b96700 (LWP 21849) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  36   Thread 0x7fd871ea4700 (LWP 21887) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  35   Thread 0x7fd8716a3700 (LWP 21896) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  34   Thread 0x7fd870ea2700 (LWP 21899) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  33   Thread 0x7fd85bfff700 (LWP 21998) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  32   Thread 0x7fd85b7fe700 (LWP 22002) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  31   Thread 0x7fd85affd700 (LWP 22008) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  30   Thread 0x7fd85a7fc700 (LWP 22011) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  29   Thread 0x7fd859ffb700 (LWP 22017) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  28   Thread 0x7fd8597fa700 (LWP 22025) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  27   Thread 0x7fd858ff9700 (LWP 22234) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  26   Thread 0x7fd837fff700 (LWP 22306) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  25   Thread 0x7fd8377fe700 (LWP 22362) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  24   Thread 0x7fd836ffd700 (LWP 22381) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  23   Thread 0x7fd8367fc700 (LWP 22454) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  22   Thread 0x7fd835ffb700 (LWP 22728) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  21   Thread 0x7fd8357fa700 (LWP 22731) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  20   Thread 0x7fd834ff9700 (LWP 22765) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  19   Thread 0x7fd817fff700 (LWP 22772) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  18   Thread 0x7fd8177fe700 (LWP 22914) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  17   Thread 0x7fd816ffd700 (LWP 22919) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  16   Thread 0x7fd8167fc700 (LWP 22922) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  15   Thread 0x7fd815ffb700 (LWP 22928) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  14   Thread 0x7fd8157fa700 (LWP 22931) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  13   Thread 0x7fd814ff9700 (LWP 23279) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  12   Thread 0x7fd7f7fff700 (LWP 23372) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  11   Thread 0x7fd7f77fe700 (LWP 23491) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  10   Thread 0x7fd7f6ffd700 (LWP 23527) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  9    Thread 0x7fd7f67fc700 (LWP 23593) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  8    Thread 0x7fd7f5ffb700 (LWP 23598) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  7    Thread 0x7fd7f57fa700 (LWP 24004) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  6    Thread 0x7fd7f4ff9700 (LWP 24062) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  5    Thread 0x7fd7e7fff700 (LWP 24068) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  4    Thread 0x7fd7e77fe700 (LWP 24168) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  3    Thread 0x7fd7e6ffd700 (LWP 24280) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  2    Thread 0x7fd7e67fc700 (LWP 24311) "pool" pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185

A backtrace of one of the threads:
  • #0 pthread_cond_wait
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S line 185
  • #1 g_cond_wait
    at gthread-posix.c line 753
  • #2 g_thread_pool_free
    at gthreadpool.c line 773
  • #3 g_threaded_socket_service_finalize
    at gthreadedsocketservice.c line 160
  • #4 g_object_unref
    at gobject.c line 3173
  • #5 g_threaded_socket_service_func
    at gthreadedsocketservice.c line 92
  • #6 g_thread_pool_thread_proxy
    at gthreadpool.c line 309
  • #7 g_thread_proxy
    at gthread.c line 798
  • #8 start_thread
    at pthread_create.c line 308
  • #9 clone
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 110
  • #10 ??


From what I can determine, g_threaded_socket_service_func() calls "g_object_unref (data->service);" which, if it is the last reference, causes g_threaded_socket_service_finalize() to be called which hangs because it waits for all tasks to be finished, but it is called from a task...

I'm not sure how best to fix this.
Comment 1 Ross Lagerwall 2014-06-22 13:33:32 UTC
I just noticed this again, attaching to gvfsd-trash:
(gdb) info threads
  Id   Target Id         Frame 
  11   Thread 0x7f06aeff2700 (LWP 1470) "gdbus" 0x00007f06b74bc9dd in poll () at ../sysdeps/unix/syscall-template.S:81
  10   Thread 0x7f06adf8e700 (LWP 1472) "gmain" 0x00007f06b74bc9dd in poll () at ../sysdeps/unix/syscall-template.S:81
  9    Thread 0x7f06acf8c700 (LWP 1474) "pool" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  8    Thread 0x7f069ffff700 (LWP 1475) "pool" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  7    Thread 0x7f06ad78d700 (LWP 1621) "pool" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  6    Thread 0x7f069f7fe700 (LWP 1627) "pool" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  5    Thread 0x7f069effd700 (LWP 1760) "pool" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  4    Thread 0x7f069e7fc700 (LWP 1766) "pool" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  3    Thread 0x7f069dffb700 (LWP 1776) "pool" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
* 2    Thread 0x7f069d7fa700 (LWP 1782) "pool" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  1    Thread 0x7f06b8f5b840 (LWP 1469) "gvfsd-trash" 0x00007f06b74bc9dd in poll () at ../sysdeps/unix/syscall-template.S:81

There are 8 hung threads.

(gdb) thread apply 2 3 4 bt


Comment 2 Ross Lagerwall 2014-10-28 21:29:07 UTC
Created attachment 289551 [details] [review]
gio: Prevent hang when finalizing GThreadedSocketService

If all users of a GThreadedSocketService release their references to the
service while a connection thread is running, the thread function will
release the last reference to the service which causes the finalize to
deadlock waiting for all threads to finish (because it's called from the
thread function).

To fix this, don't wait for all threads to finish in the service's
finalize method.  Since the threads hold a reference to the service,
finalize should only be called when all threads are finished running (or
have unrefed the service and are about to finish).
Comment 3 Ross Lagerwall 2014-10-28 21:33:01 UTC
Attached is a reproducer:

$ gcc -Wall -g -o test test.c `pkg-config --cflags --libs glib-2.0 gio-2.0`
$ ./test

In another terminal, run:
$ nc localhost 8888

Then wait 30 seconds and attach to the process with gdb.

(gdb) bt

Thread 2 (Thread 0x7f1419f79700 (LWP 16367))

  • #0 syscall
    from /usr/lib/libc.so.6
  • #1 g_cond_wait
    at gthread-posix.c line 1390
  • #2 g_thread_pool_free
    at gthreadpool.c line 776
  • #3 g_threaded_socket_service_finalize
    at gthreadedsocketservice.c line 159
  • #4 g_object_unref
    at gobject.c line 3170
  • #5 g_threaded_socket_service_func
    at gthreadedsocketservice.c line 91
  • #6 g_thread_pool_thread_proxy
    at gthreadpool.c line 307
  • #7 g_thread_proxy
    at gthread.c line 764
  • #8 start_thread
    from /usr/lib/libpthread.so.0
  • #9 clone
    from /usr/lib/libc.so.6

Comment 4 Ross Lagerwall 2014-10-28 21:33:33 UTC
Created attachment 289553 [details]
program to reproduce the problem
Comment 5 Dan Winship 2014-11-23 17:35:37 UTC
Pushed your patch, along with a test case based on the reproducer.
Thanks


Attachment 289551 [details] pushed as 9a6e01e - gio: Prevent hang when finalizing GThreadedSocketService
Comment 6 Ross Lagerwall 2014-11-23 20:15:12 UTC
Thanks!
Comment 7 Iain Lane 2014-11-30 19:17:05 UTC
I can make this testcase hang if I run it in a loop (initially detected by the Ubuntu CI machines where it seems more frequent; 3/6 runs of the installed-tests). bt goes:

Thread 2 (Thread 0x7f0a4dd65700 (LWP 27257))

  • #0 syscall
    at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
  • #1 g_mutex_lock_slowpath
    at /build/buildd/glib2.0-2.43.1/./glib/gthread-posix.c line 1314
  • #2 g_mutex_lock
    at /build/buildd/glib2.0-2.43.1/./glib/gthread-posix.c line 1338
  • #3 connection_cb
    at /build/buildd/glib2.0-2.43.1/./gio/tests/socket-listener.c line 67
  • #4 ffi_call_unix64
    at ../src/x86/unix64.S line 76
  • #5 ffi_call
    at ../src/x86/ffi64.c line 525
  • #6 g_cclosure_marshal_generic_va
    at /build/buildd/glib2.0-2.43.1/./gobject/gclosure.c line 1541
  • #7 _g_closure_invoke_va
    at /build/buildd/glib2.0-2.43.1/./gobject/gclosure.c line 831
  • #8 g_signal_emit_valist
    at /build/buildd/glib2.0-2.43.1/./gobject/gsignal.c line 3195
  • #9 g_signal_emit
    at /build/buildd/glib2.0-2.43.1/./gobject/gsignal.c line 3342
  • #10 g_threaded_socket_service_func
    at /build/buildd/glib2.0-2.43.1/./gio/gthreadedsocketservice.c line 87
  • #11 g_thread_pool_thread_proxy
    at /build/buildd/glib2.0-2.43.1/./glib/gthreadpool.c line 307
  • #12 g_thread_proxy
    at /build/buildd/glib2.0-2.43.1/./glib/gthread.c line 764
  • #13 start_thread
    at pthread_create.c line 309
  • #14 clone
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 111

Comment 8 Ross Lagerwall 2014-11-30 22:21:44 UTC
Created attachment 291849 [details] [review]
gio/tests: Prevent hangs and aborts in socket-listener

Fix two problems:
1) If g_socket_service_stop is called before the accept call is requeued,
then the reference count won't decrease and this code will hang forever:
  while (G_OBJECT (service)->ref_count == ref_count)
    g_main_context_iteration (NULL, TRUE);

2) Sometimes the testcase fails (maybe 1 in 200 times for me):
GLib-GIO:ERROR:socket-listener.c:73:connection_cb: assertion failed
(G_OBJECT (service)->ref_count == 2): (3 == 2)
Aborted (core dumped)

The problem is that depending on ordering, cancellation of the async
listener can require further main context iterations before it releases
the reference on the socket service. Furthermore, in some cases, it
requires at least one iteration.
Comment 9 Ross Lagerwall 2014-11-30 22:22:42 UTC
I have tested this for 20000 iterations without a hang or abort.
Comment 10 Ross Lagerwall 2014-12-07 06:55:28 UTC
Pushed to master as 7d9816934e6a233369a2df1282a49d1549d9fa0f. Please reopen if the test case hangs or fails.