Created attachment 290499 [details] [review]
Test/Example and fix

The GstTask thread entry point looks like this:

static void
gst_task_func (GstTask * task)
{
...
  while (G_LIKELY (GET_TASK_STATE (task) != GST_TASK_STOPPED)) {
    if (G_UNLIKELY (GET_TASK_STATE (task) == GST_TASK_PAUSED)) {
      GST_OBJECT_LOCK (task);
      while (G_UNLIKELY (GST_TASK_STATE (task) == GST_TASK_PAUSED)) {
        g_rec_mutex_unlock (lock);

        GST_TASK_SIGNAL (task);
        GST_INFO_OBJECT (task, "Task going to paused");
        GST_TASK_WAIT (task);
        GST_INFO_OBJECT (task, "Task resume from paused");
        GST_OBJECT_UNLOCK (task);
        /* locking order.. */
        g_rec_mutex_lock (lock);

        GST_OBJECT_LOCK (task);
        if (G_UNLIKELY (GET_TASK_STATE (task) == GST_TASK_STOPPED)) {
          GST_OBJECT_UNLOCK (task);
          goto done;
        }
      }
      GST_OBJECT_UNLOCK (task);
    }

    task->func (task->user_data);
  }
done:
  g_rec_mutex_unlock (lock);

  GST_OBJECT_LOCK (task);
  task->thread = NULL;
...
}

From what I can see, the while (!stopped) and if (paused) is not guarded by the object lock. A task func that look like this:
my_task_func ()
{
  ...
  gst_pad_pause_task ();
}

will once in a while cause the taks func to reenter, because another thread may change the task state to stopped between while (!stopped) and if (paused).


I've attached a patch which adds a test that must be run like this for a couple of minutes (or hours, depending on SMP/threading):
GST_CHECKS=test_pause_stop_race make -C tests/check gst/gsttask.forever

Theres also a fix in the patch. Without the fix the deadlock will normally appear after a couple of million runs, in a 1-2minutes on my system. With the fix I've been running the test for a couple of hours without deadlocking.