Hi Matthew, although on the face of if this may sound quite straightforward, there are some quite thorny issues that need to be considered here...

One thing is that OpenGL[ES] itself doesn't expose any handle for a GL context, it's up to the platform's window system integration layer to manage GL contexts and make them current for GL to run. This means we would either need to expose EGLContext, GLXContext and wgl's HGLRC etc or we'd need to come up with an abstract CoglGLContext for all of these which GStreamer would then need to support somehow.

Besides a handle to the context, it's worth nothing that we would also presumably need to expose the display handles that most window systems require to be able to do anything with a context and you may also need a handle to surface too, even if you are only doing offscreen rendering. The details of what you need to bind a context vary for different platforms.

The much more difficult issue (imho) than both of those though is that OpenGL is a very stateful api which isn't suited to having orthogonal components sharing access to one context. Cogl necessarily relies on being able to assume it knows what state it left the GL context in because otherwise every interaction with GL would have to somehow reset all GL state. The problem goes both ways though because the GL elements in gstreamer would have no idea what state the context is in when they start to do work, and Cogl would have no idea what changes a GL GStreamer element has made.

The best approach that we currently have for supporting GL integration with Cogl is via the CoglGLES2Context api, but this api has some caveats currently. Firstly it's currently only supported when running on a real gles2 driver, since there are glsl incompatibilities with full GL that we need to work around but haven't done that yet. It relies on creating a separate GL context to avoid trampling on Cogl's state tracking, but the context is set up so that textures can be shared in both directions. This api is based around a dispatch vtable for accessing the GLES 2.0 api that GStreamer would need to use instead of directly accessing libGLESv2 symbols since it relies on us being able to replace certain functions to make the integration possible. We provide a convenience libcogl-gles2 library that can be swapped in to replace libGLESv2 which may help. You can find an examples of using this integration api in cogl-gles-context.c and cogl-gles2-gears.c in cogl's examples/ directory.

Out of curiosity, I'd be interested to know what limitations you see with using cogl-gst compared to using gl based gstreamer elements? It might be that we can implement some missing capability in cogl-gst which I guess would be much easier than trying to reliably support sharing a single GL context. Another thing to consider here, is why are GL textures being used to communicate between elements? Ideally each combined video effects shouldn't require an intermediate GL texture for each effect, since that will cost memory bandwidth and performance. The way cogl-gst was designed, it should be possible to create effects that can hook in snippets into a single pipeline so that multiple fragment based effects can be applied in one draw-call with no intermediate textures.

I hope that gives enough information for you to see some of the issues involved here. It would be good to hear more about your specific use case so we can see if there might be other ways of making it work.

(In reply to comment #1)
> Hi Matthew, although on the face of if this may sound quite straightforward,
> there are some quite thorny issues that need to be considered here...
> 
> One thing is that OpenGL[ES] itself doesn't expose any handle for a GL context,
> it's up to the platform's window system integration layer to manage GL contexts
> and make them current for GL to run. This means we would either need to expose
> EGLContext, GLXContext and wgl's HGLRC etc or we'd need to come up with an
> abstract CoglGLContext for all of these which GStreamer would then need to
> support somehow.
> 
> Besides a handle to the context, it's worth nothing that we would also
> presumably need to expose the display handles that most window systems require
> to be able to do anything with a context and you may also need a handle to
> surface too, even if you are only doing offscreen rendering. The details of
> what you need to bind a context vary for different platforms.
> 
> The much more difficult issue (imho) than both of those though is that OpenGL
> is a very stateful api which isn't suited to having orthogonal components
> sharing access to one context. Cogl necessarily relies on being able to assume
> it knows what state it left the GL context in because otherwise every
> interaction with GL would have to somehow reset all GL state. The problem goes
> both ways though because the GL elements in gstreamer would have no idea what
> state the context is in when they start to do work, and Cogl would have no idea
> what changes a GL GStreamer element has made.

All of the previous points are already implemented by gst-plugins-gl and mechanisms exists within GStreamer for that sharing to occur.

The way I would see this happening is that we share contexts between cogl and GL GStreamer elements.  Combined with separate threads solves the problem of us trampling on each other's state.

> The best approach that we currently have for supporting GL integration with
> Cogl is via the CoglGLES2Context api, but this api has some caveats currently.
> Firstly it's currently only supported when running on a real gles2 driver,
> since there are glsl incompatibilities with full GL that we need to work around
> but haven't done that yet. It relies on creating a separate GL context to avoid
> trampling on Cogl's state tracking, but the context is set up so that textures
> can be shared in both directions. This api is based around a dispatch vtable
> for accessing the GLES 2.0 api that GStreamer would need to use instead of
> directly accessing libGLESv2 symbols since it relies on us being able to
> replace certain functions to make the integration possible. We provide a
> convenience libcogl-gles2 library that can be swapped in to replace libGLESv2
> which may help. You can find an examples of using this integration api in
> cogl-gles-context.c and cogl-gles2-gears.c in cogl's examples/ directory.

That sounds like an interesting idea.  I'm not sure on how we would integrate without introducing a dependency (soft or hard).  FWIW gst-plugins-gl uses getProcAddress to retreive all it's GL functions.

> Out of curiosity, I'd be interested to know what limitations you see with using
> cogl-gst compared to using gl based gstreamer elements? It might be that we can
> implement some missing capability in cogl-gst which I guess would be much
> easier than trying to reliably support sharing a single GL context. Another
> thing to consider here, is why are GL textures being used to communicate
> between elements? Ideally each combined video effects shouldn't require an
> intermediate GL texture for each effect, since that will cost memory bandwidth
> and performance. The way cogl-gst was designed, it should be possible to create
> effects that can hook in snippets into a single pipeline so that multiple
> fragment based effects can be applied in one draw-call with no intermediate
> textures.

I have to say that I've never used the cogl-gst snippets at all although have thought of similar mechanisms for gst-plugins-gl.  The main disadvantage I see of using cogl-gst at the moment is the threading support.  The main reason that gst-plugins-gl is useful is that it is aware of the threads and can operate across thread boundaries between elements.  At the moment it does so by marshalling everything to a dedicated thread however the plan is too allow each GStreamer streaming thread to contain their own GL context that are shared together when needed.  It also attempts to solve more than just simple filter effects.  It also contains a couple of mixing elements (glmosaic, glvideomixer). and some effects that make use of lighting as well as bumpmaps/normalmaps.  There's even a greedy deinterlacer available.

> I hope that gives enough information for you to see some of the issues involved
> here. It would be good to hear more about your specific use case so we can see
> if there might be other ways of making it work.

Thanks for taking the time to reply.

(In reply to comment #2)
> (In reply to comment #1)
> > The much more difficult issue (imho) than both of those though is that OpenGL
> > is a very stateful api which isn't suited to having orthogonal components
> > sharing access to one context. Cogl necessarily relies on being able to assume
> > it knows what state it left the GL context in because otherwise every
> > interaction with GL would have to somehow reset all GL state. The problem goes
> > both ways though because the GL elements in gstreamer would have no idea what
> > state the context is in when they start to do work, and Cogl would have no idea
> > what changes a GL GStreamer element has made.
> 
> All of the previous points are already implemented by gst-plugins-gl and
> mechanisms exists within GStreamer for that sharing to occur.
> 
> The way I would see this happening is that we share contexts between cogl and
> GL GStreamer elements.  Combined with separate threads solves the problem of us
> trampling on each other's state.

Sorry, I don't understand how you can avoid trampling state only via some mechanism in gst-plugins-gl, unless you use a separate context?

I'm fairly sure that sharing a GL context is only going to be possible by using some kind of cooperative state management mechanism which Cogl would need to be a part of, due to the nature of the GL api. Thread boundaries don't implicitly provide any separation of state with GL, only contexts do, and the way a context can be accessed from different threads depends on the window system but is generally orthogonal to how context state is managed.

Maybe if would help if you could clarify a bit more the mechanisms in gst-plugins-gl, and perhaps point me to some of the code?

(In reply to comment #3)
> (In reply to comment #2)
> > (In reply to comment #1)
> > > The much more difficult issue (imho) than both of those though is that OpenGL
> > > is a very stateful api which isn't suited to having orthogonal components
> > > sharing access to one context. Cogl necessarily relies on being able to assume
> > > it knows what state it left the GL context in because otherwise every
> > > interaction with GL would have to somehow reset all GL state. The problem goes
> > > both ways though because the GL elements in gstreamer would have no idea what
> > > state the context is in when they start to do work, and Cogl would have no idea
> > > what changes a GL GStreamer element has made.
> > 
> > All of the previous points are already implemented by gst-plugins-gl and
> > mechanisms exists within GStreamer for that sharing to occur.
> > 
> > The way I would see this happening is that we share contexts between cogl and
> > GL GStreamer elements.  Combined with separate threads solves the problem of us
> > trampling on each other's state.
> 
> Sorry, I don't understand how you can avoid trampling state only via some
> mechanism in gst-plugins-gl, unless you use a separate context?
> 
> I'm fairly sure that sharing a GL context is only going to be possible by using
> some kind of cooperative state management mechanism which Cogl would need to be
> a part of, due to the nature of the GL api. Thread boundaries don't implicitly
> provide any separation of state with GL, only contexts do, and the way a
> context can be accessed from different threads depends on the window system but
> is generally orthogonal to how context state is managed.
> 
> Maybe if would help if you could clarify a bit more the mechanisms in
> gst-plugins-gl, and perhaps point me to some of the code?

We would precisely use a separate context that are shared across the thread boundary.  As for code, the main GL context class is http://cgit.freedesktop.org/gstreamer/gst-plugins-gl/tree/gst-libs/gst/gl/gstglcontext.c  There's also http://cgit.freedesktop.org/gstreamer/gst-plugins-gl/tree/tests/check/libs/gstglcontext.c for a couple of examples on how it works.

Hmm, that still sounds ambiguous.  By share I mean the OpenGL meaning, where two contexts can access each other's textures, shaders, etc.  The other case of sharing a single context across multiple elements from different gstreamer plugin libraries requires, as you said, some kind of cooperative state management.

In a GStreamer pipeline, the idea is to have an upstream element create their GL context as usual.  They then pass this downstream using the ALLOCATION query.  Downstream then would create a new GL context that is shared with upstream's context.  The other requirement for this to occur, is that the two GL contexts be in separate threads.  I'm reasonably happy for gst-plugins-gl to create a separate thread/task if needed in order to keep the two GL contexts separate.

More details are also available in https://bugzilla.gnome.org/show_bug.cgi?id=704809

Ah, right sorry I apparently overlooked the part in your original description where you said that you were intending on creating your own context, oops.

I've pushed a wip/expose-glx-egl-context branch and sent patches to the mailinglist (http://lists.freedesktop.org/archives/cogl/2014-March/001609.html) to start by exposing what I think you would want for EGL and GLX. Do you also support WGL too?

I'm afraid I just made the patch for master, and I guess you only really care about the 1.18 branch, but it should be easy to adapt to the 1.18 branch too.

It would be good to hear if you think these api additions would be enough?

Awesome, that looks like it will work wonderfully.  I'll give it a try later.

Yes we support WGL as well as Cocoa (CGL).

Cool, good to hear. Btw I've just pushed a wip/expose-glx-egl-context-1.18 branch that should give you the same apis for the cogl-1.18 branch. It would be good if you could test these.

Both branches now also have a patch to expose the WGL HGLRC context handle.

One potential way to get the context without adding any API to Cogl could be to use eglGetCurrentContext(). Cogl should currently always have the context bound so it should work. I suppose it's not really ideal though because we probably don't want to guarantee that Cogl always has the context bound in the main thread. For example it would be nice to leave the possibility open for Cogl to internally use a separate thread for rendering. It looks like WGL and GLX have equivalent functions.

I've hacked up cluttersink to share the gl context and display for the GLX case and it works.  EGL would require a little bit more work on our side to work but it should definitely be possible with the additional API.

One thing I noticed is that the texture seems to draw over the previous frames.  If I resize the window, it resets to the current frame and then continues drawing over the top.

Clearing the texture before copying does not help.

GNOME is going to shut down bugzilla.gnome.org in favor of gitlab.gnome.org.
As part of that, we are mass-closing older open tickets in bugzilla.gnome.org
which have not seen updates for a longer time (resources are unfortunately
quite limited so not every ticket can get handled).

If you can still reproduce the situation described in this ticket in a recent
and supported software version of cogl, then please follow
  https://wiki.gnome.org/GettingInTouch/BugReportingGuidelines
and create a ticket at
  https://gitlab.gnome.org/GNOME/cogl/-/issues/

Thank you for your understanding and your help.