I discovered a deadlock during some stress testing of our video streaming architecture that utilizes shmsink/shmsrc to stream video buffers between two processes.  The system in question was running with GStreamer 1.2.4, the version that ships with SuSE Linux Enterprise 12. A quick examination of git seems to indicate that the problematic code still exists in the current version of gst-plugins-bad.

In our tests, a shmsrc client repeatedly connects/disconnects at various intervals to receive video from the shmsink server pipeline.  At some point during the test, the shmsink element in the server process was observed to deadlock in gst_shm_sink_allocator_free().  The stack trace of the appropriate thread from gdb:

+ Trace 235894

The lock in question is the shmsink GST_OBJECT lock.  A closer examination in gdb revealed that the same thread already held a lock on the mutex.  Looking at the code, we found the initial GST_OBJECT_LOCK() being acquired at the start of gst_shm_sink_render(), but it isn't released until *after* a call to gst_memory_unref():

lines 714-721 in gstshmsink.c (exact line #s may vary depending on version):

    while (self->wait_for_connection && !self->clients) {
      g_cond_wait (&self->cond, GST_OBJECT_GET_LOCK (self));
      if (self->unlock) {
        gst_memory_unref (memory);
        GST_OBJECT_UNLOCK (self);
        return GST_FLOW_FLUSHING;
      }
    }

As can be seen in the stack trace, this gst_memory_unref() leads to calls to _gst_memory_free() and gst_shm_sink_allocator_free().  Inside of gst_shm_sink_allocator_free(), the code tries to acquire the shmsink GST_OBJECT_LOCK() again:

static void
gst_shm_sink_allocator_free (GstAllocator * allocator, GstMemory * mem)
{
  GstShmSinkMemory *mymem = (GstShmSinkMemory *) mem;

  if (mymem->block) {
    GST_OBJECT_LOCK (mymem->sink);
    sp_writer_free_block (mymem->block);
    GST_OBJECT_UNLOCK (mymem->sink);
    gst_object_unref (mymem->sink);
  }
  gst_object_unref (mem->allocator);

  g_slice_free (GstShmSinkMemory, mymem);
}


I *think* this deadlock can be resolved by moving the gst_memory_unref() call in gst_shm_sink_render() to after the GST_OBJECT_UNLOCK() call, as in the following diff:

@@ -714,8 +722,8 @@
     while (self->wait_for_connection && !self->clients) {
       g_cond_wait (&self->cond, GST_OBJECT_GET_LOCK (self));
       if (self->unlock) {
-        gst_memory_unref (memory);
         GST_OBJECT_UNLOCK (self);
+        gst_memory_unref (memory);
         return GST_FLOW_FLUSHING;
       }
     }

We are currently testing this change in our own rebuilt version of the plugin and wanted to raise this issue with the GStreamer team to make sure they were aware of the problem and to get feedback on whether the proposed fix was appropriate.