osxvideosink should be lower than glimagesink IMHO, it's much less good. And the directshowsink should be lower than d3d and glimagesink I guess.

Our use case : we have a Qt app that runs on linux/windows/osx and needs a video sink that supports the VideoOverlay interface.

autovideosink works fine on linux and windows, but on osx we get an external window. Explicitly using osxvideosink works fine. 
I thought it was due to the NONE rank of osxvideosink and glimagesink not supporting VideoOverly. But I just checked and glimagesink does support VideoOverlay. Need to investigate more.

PS : the side note about dshowvideosink was just to highlight that the comment is wrong now...

Ok, please let us know what you find out :)

Generally I think we should review the ranks and fix them up as I said above though, in case you want to provide a patch for that.

With 1.12.1 we are seeing the same behavior as described here : http://gstreamer-devel.966125.n4.nabble.com/autovideosink-SIGSEGV-on-OS-X-Sierra-td4683057.html

For now, we are forcing the rank of osxvideosink to PRIMARY so it gets selected by autovideosink. We do that with :

    GstRegistry *reg = gst_registry_get();

    GstPluginFeature *feature = gst_registry_lookup_feature(reg, "osxvideosink");
    if (feature) {
        // raise rank of osxvideosink so it gets selected by autovideosink
        gst_plugin_feature_set_rank(feature, GST_RANK_PRIMARY);
        gst_object_unref(feature);
    }

Here are the video sink ranks in 1.12.2:

osxvideosink : MARGINAL

dshowvideosink : SECONDARY
d3dvideosink   : PRIMARY

ximagesink : SECONDARY
xvimagesink : PRIMARY

glimagesink : SECONDARY
glimagesinkelement : NONE

So except, for osx, there is always a platform specific video sink that outranks the gl sink.

osxvideosink is worse in every regard than glimagesink. It's doing GL things by itself in a minimal way.

Ok. It is not a problem to keep osxvideosink's rank to what it is.

We can always change it in our app if using a gl sink is not possible (which is our case when integrating using a non GL Qt widget).

If you are ok with it, please close this issue.

How can you integrate osxvideosink, but not glimagesink? In theory it should be possible to use both the same way from an application point of view. What is missing for glimagesink?

You are right : videotestsrc ! glimagesink works on windows.

I'll have to get more information from the person that helps on Mac (I don't own a Mac).

What I know, is that on Mac, "videotestsrc ! autovideosink" fails with the same error as described here: http://gstreamer-devel.966125.n4.nabble.com/autovideosink-SIGSEGV-on-OS-X-Sierra-td4683057.html

Here's the backtrace:

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007fffdcc05f46 libsystem_kernel.dylib`__semwait_signal + 10
    frame #1: 0x00007fffdcb8cb72 libsystem_c.dylib`nanosleep + 199
    frame #2: 0x000000010aeeabce libglib-2.0.0.dylib`g_usleep + 71
    frame #3: 0x000000010ac6cdf4 gst-launch-1.0`fault_spin at gst-launch.c:113
    frame #4: 0x000000010ac6cd03 gst-launch-1.0`fault_handler_sighandler(signum=11) at gst-launch.c:94
    frame #5: 0x00007fffdcce6b3a libsystem_platform.dylib`_sigtramp + 26
    frame #6: 0x000000010b3e6402 libgstgl-1.0.0.dylib`-[GstGLCAOpenGLLayer initWithGstGLContext:](self=0x00007f85c1584e50, _cmd="initWithGstGLContext:", parent_gl_context=0x0000000000000000) at gstglcaopengllayer.m:75
    frame #7: 0x000000010b3e358f libgstgl-1.0.0.dylib`gst_gl_window_cocoa_create_window(window_cocoa=0x00007f85c2848090) at gstglwindow_cocoa.m:179 [opt]
    frame #8: 0x00007fffdcaaa524 libdispatch.dylib`_dispatch_call_block_and_release + 12
    frame #9: 0x00007fffdcaa18fc libdispatch.dylib`_dispatch_client_callout + 8
    frame #10: 0x00007fffdcaaeaac libdispatch.dylib`_dispatch_main_queue_callback_4CF + 925
    frame #11: 0x00007fffc6f68d69 CoreFoundation`__CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 9
    frame #12: 0x00007fffc6f2a04d CoreFoundation`__CFRunLoopRun + 2221
    frame #13: 0x00007fffc6f29544 CoreFoundation`CFRunLoopRunSpecific + 420
    frame #14: 0x00007fffc6489ebc HIToolbox`RunCurrentEventLoopInMode + 240
    frame #15: 0x00007fffc6489bf9 HIToolbox`ReceiveNextEventCommon + 184
    frame #16: 0x00007fffc6489b26 HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter + 71
    frame #17: 0x00007fffc4a22a54 AppKit`_DPSNextEvent + 1120
    frame #18: 0x00007fffc519e7ee AppKit`-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 2796
    frame #19: 0x000000010aec7b68 libglib-2.0.0.dylib`cocoa_poll_func + 173
    frame #20: 0x000000010aec8a1a libglib-2.0.0.dylib`g_main_context_iterate + 337
    frame #21: 0x000000010aec8cc6 libglib-2.0.0.dylib`g_main_loop_run + 207
    frame #22: 0x000000010ac9c713 libgstreamer-1.0.0.dylib`gst_bus_poll(bus=0x00007f85c2012b20, events=GST_MESSAGE_ANY, timeout=0) at gstbus.c:1207
    frame #23: 0x000000010ac6b274 gst-launch-1.0`event_loop(pipeline=0x00007f85c20480b0, blocking=0, do_progress=1, target_state=GST_STATE_PLAYING) at gst-launch.c:555
    frame #24: 0x000000010ac6a920 gst-launch-1.0`main(argc=7, argv=0x00007fff54f965e8) at gst-launch.c:1156
    frame #25: 0x00007fffdcad7235 libdyld.dylib`start + 1
    frame #26: 0x00007fffdcad7235 libdyld.dylib`start + 1


initWithGstGLContext doesn't check that parent_gl_context is valid. Add that check makes things work here but I get some more errors:

gst-launch-1.0 -v videotestsrc is-live=1 ! queue ! autovideosink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...

(gst-launch-1.0:81940): GLib-GObject-WARNING **: invalid unclassed pointer in cast to 'GstGLWindow'

** (gst-launch-1.0:81940): CRITICAL **: gst_gl_window_get_context: assertion 'GST_IS_GL_WINDOW (window)' failed

(gst-launch-1.0:81940): GStreamer-CRITICAL **: gst_object_unref: assertion 'object != NULL' failed

** (gst-launch-1.0:81940): CRITICAL **: gst_gl_context_get_display: assertion 'GST_IS_GL_CONTEXT (context)' failed

** (gst-launch-1.0:81940): CRITICAL **: gst_gl_display_get_gl_api: assertion 'GST_IS_GL_DISPLAY (display)' failed

** (gst-launch-1.0:81940): CRITICAL **: gst_gl_context_new_wrapped: assertion '(display_api & available_apis) != GST_GL_API_NONE' failed

(gst-launch-1.0:81940): GStreamer-CRITICAL **: gst_object_unref: assertion 'object != NULL' failed
0:00:00.324579000 81940 0x7fa817f0e560 ERROR        glcaopengllayer gstglcaopengllayer.m:141:-[GstGLCAOpenGLLayer copyCGLContextForPixelFormat:]: failed to create wrapped context
Got context from element 'autovideosink0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayCocoa\)\ gldisplaycocoa0";
Setting pipeline to PLAYING ...
New clock: GstSystemClock

I suspect gst_gl_window_cocoa_create_window should take a new ref of the window and unref before returning to caller.

Also it seems that gst_gl_window_cocoa_create_window() is called before the context is set on the window. Not sure what's going on there :)

Created attachment 355693 [details] [review]
proposed patch

I created a very similar patch last night based on bug 782379. I've pushed that version as it's a little more complete.

For the actual problem in the title (having osxvideosink outrank glimagesink) I don't think that's something we want in upstream as osxvideosink is very outdated at this point and does some suboptimal things.  glimagesink is they way forward for a video sink on macOS.  Thus marking as WONFIX.