After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 310775 - libvisual element needs to support opengl
libvisual element needs to support opengl
Status: RESOLVED FIXED
Product: GStreamer
Classification: Platform
Component: gst-plugins-gl
git master
Other Linux
: Normal enhancement
: 0.10.3
Assigned To: GStreamer Maintainers
GStreamer Maintainers
: 310696 (view as bug list)
Depends on:
Blocks:
 
 
Reported: 2005-07-18 16:07 UTC by Andy Wingo
Modified: 2012-05-10 21:51 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
libvisual plugin for gst-plugins-gl (32.66 KB, patch)
2009-05-15 01:59 UTC, Jonathan Matthew
none Details | Review
libvisual plugin for gst-plugins-gl (32.41 KB, patch)
2011-11-24 16:26 UTC, Stefan Sauer (gstreamer, gtkdoc dev)
none Details | Review
libvisual_gl_lv_gltest now works (37.31 KB, patch)
2011-11-25 16:12 UTC, Julien Isorce
committed Details | Review
snapshot of libvisual_gl_lv_gltest (21.05 KB, image/png)
2011-12-05 10:27 UTC, Julien Isorce
  Details
libvisual-gl: full compatibility with projectM-libvisual if libprojectM >= 2.0.1 (4.05 KB, patch)
2012-01-03 16:43 UTC, Julien Isorce
none Details | Review
gst-launch-0.10 audiotestsrc ! libvisual_gl_projectM ! "video/x-raw-gl, width=800, height=600" ! gleffects effect=tunnel ! gldownload ! pngenc ! filesink location=sample2.png (685.40 KB, image/png)
2012-01-03 16:49 UTC, Julien Isorce
  Details
libvisual-gl: full compatibility with projectM-libvisual if libprojectM >= 2.0.1 (6.88 KB, patch)
2012-01-17 15:42 UTC, Julien Isorce
none Details | Review

Description Andy Wingo 2005-07-18 16:07:53 UTC
We need to support opengl-rendering libvisual plugins. The first level would be
giving it an off-screen opengl context, grabbing pixels into buffers and pushing
them out as normal video frames. The best, though, would be for gl-capable sinks
to return subclasses of GstBuffer from their bufferalloc functions, and for
gl-capable renderers to peel the gl context off the buffer directly.
Comment 1 Andy Wingo 2005-07-18 16:09:52 UTC
Since we don't have gl-capable sinks in 0.9 thi bug is only for the first level
(stealing the gl context's pixels).
Comment 2 Andy Wingo 2005-07-18 16:10:07 UTC
*** Bug 310696 has been marked as a duplicate of this bug. ***
Comment 3 Julien MOUTTE 2006-03-06 22:46:11 UTC
There's no nice way to do that with libvisual. IIRC libvisual works with SDL to do opengl visualisation rendering, so what we could do is write another libvisual based GStreamer element called libvisualsink that would take audio buffers and render using SDL.

That would allow for OpenGL visualisation actors to be used.

Do we really need to have a bug for that ?
Comment 4 Jonathan Matthew 2006-05-08 14:08:03 UTC
I've implemented support for opengl libvisual visualisations using the following hackery:

- added a 'glcontext' interface that gl-based sinks can implement to allow other elements (and maybe applications) to request that they make their gl context current in the calling thread.  I thought this was better than passing the context back from the sink to the rendering element and (eventually) having #ifdefs everywhere for calls to glXMakeCurrent, wglMakeCurrent, etc.

- added a new content type, video/x-opengl-texture-id to glimagesink's caps. Buffers with this content type contain only an opengl texture id that the sink is to use to draw the frame.  Hypothetical intermediate elements (gltextoverlay?) could also draw on the pbuffer.

- created a variant of the libvisual plugin (I intend to merge the code back in at some point, but haven't gotten around to it yet) that creates a framebuffer (using GL_EXT_framebuffer_object) and convinces libvisual to render into it.  For each frame, it then pushes a video/x-opengl-texture-id buffer containing the framebuffer's texture id.

This all seems to work fairly well, but it's a bit ugly in places, particularly how video/x-opengl-texture-id buffers are handled.  I've tested all the gl-based visualisations that come with libvisual (but not projectM, as far as I recall).  

I haven't looked in to getting playbin to handle it properly (not trying to run the vis plugin output through ffmpegcolorspace, for a start).  It'd be nice to get it to automatically construct a gl-capable video sink if the vis plugin has video/x-opengl-texture-id caps, but I'd be happy enough if it required the application to do that for itself.
Comment 5 Viktor Kojouharov 2006-12-21 17:18:18 UTC
what's the status of this bug? the last comment seems to be quite old, is there any progress on it?
Comment 6 Stewart Adam 2007-11-17 16:10:34 UTC
Any news?
Comment 7 Andy Wingo 2008-01-12 20:42:25 UTC
Unassigning myself -- a proper solution will use schleef's new gl buffer foo
Comment 8 Jonathan Matthew 2009-05-15 01:59:34 UTC
Created attachment 134676 [details] [review]
libvisual plugin for gst-plugins-gl

This more or less works.  Haven't really tested with projectM as it seems to fall back to software rendering on this machine, and all the other libvisual GL plugins are fairly boring.

Obvious problems: shares 80% of its code with the libvisual plugin in -base, does awful hackery with GL matrices to give the libvisual actor the rendering state it expects.
Comment 9 Julien Isorce 2009-07-27 09:06:04 UTC
This bug should be moved to gst-plugins-gl.


Comment 10 Jan Schmidt 2009-07-27 10:42:03 UTC
Oh strange. I completely missed your patch. I have a similar one here in a dev branch. I should compare to see if I missed something :)
Comment 11 Julien Isorce 2009-09-03 09:20:15 UTC
I tested the patch a little bit.
Seems to work.
There is some jerk sometimes and some darkness. 
Less with a mp3 than audiotestsrc.
I could try to find what's happen later.

Anyway, I think libgstgl (gst-plugins-gl/gst-libs/gst/gl should be build as a shared library (for now it's static)

So instead of having:
libgstopengl.so
libgstlibvisualgl.so

We should have:
libgstgl-0.10.so (and deploy gstglbuffer.h)
libgstopengl.so
libgstlibvisualgl.so
Comment 12 Matthias Klumpp 2010-07-28 14:48:44 UTC
Any news about this? Why was the patch not yet applied?
Comment 13 Julien Isorce 2010-07-28 16:11:45 UTC
Hi, I was wrong in 'Comment 11'. There is already a libgstgl-0.10.so. So there is no blocking thing.

I will try to test the patch again to check if it still work.
Comment 14 Matthias Klumpp 2010-12-30 16:21:53 UTC
Did it work?
Is there a chance to get these patches included in GStreamer? (There are many people asking for projectM visualisations in Rhythmbox etc. so if there's nothing blocking, I think it would be a great feature to have)
Thanks!
Comment 15 Stefan Sauer (gstreamer, gtkdoc dev) 2011-11-23 21:15:08 UTC
This patch needs a bit of updating. 

gst-plugins-gl/ext/libvisual:(git:master)> make
  CC     libgstlibvisualgl_la-visual-gl.lo
cc1: warnings being treated as errors
visual-gl.c: In function ‘gst_visual_gl_class_init’:
visual-gl.c:215: error: implicit declaration of function ‘gst_element_class_set_details’
visual-gl.c: In function ‘gst_visual_gl_change_state’:
visual-gl.c:818: error: too many arguments to function ‘gst_gl_display_create_context’
make: *** [libgstlibvisualgl_la-visual-gl.lo] Error 1

Julien, do you have an updated patch?
Comment 16 Julien Isorce 2011-11-23 21:44:25 UTC
Hi. no.
Comment 17 Stefan Sauer (gstreamer, gtkdoc dev) 2011-11-24 16:20:50 UTC
I've updated the patch to make things build, but it does not work for me. I am getting these elements:

libvisual-gl:  libvisual_gl_projectM: libvisual projectM
libvisual-gl:  libvisual_gl_nastyfft: Libvisual NastyFFT plugin
libvisual-gl:  libvisual_gl_madspin: libvisual madspin port
libvisual-gl:  libvisual_gl_lv_gltest: libvisual GL analyser
libvisual-gl:  libvisual_gl_lv_flower: libvisual Pseudotoad flower, yello


But none of them work for me:

gst-launch filesrc location=$HOME/Music/1.mp3 ! decodebin2 ! tee name=t ! queue ! audioconvert ! libvisual_gl_madspin ! glimagesink t. ! queue ! pulsesink

=> just a white window

gst-launch filesrc location=$HOME/Music/1.mp3 ! decodebin2 ! tee name=t ! queue ! audioconvert ! libvisual_gl_madspin ! gldownload ! xvimagesink t. ! queue ! pulsesink

=> just a black window

When taking a peek into the debug log, it looks like it is doing some work though:
0:00:04.406503587  4151      0x1d2c180 DEBUG              libvisual visual-gl.c:663:gst_visual_gl_chain:<visualglmadspin0> chain function called
0:00:04.406522029  4151      0x1d2c180 DEBUG              libvisual visual-gl.c:687:gst_visual_gl_chain:<visualglmadspin0> Input buffer has 1152 samples, time=4231836734
0:00:04.406536852  4151      0x1d2c180 DEBUG              libvisual visual-gl.c:694:gst_visual_gl_chain:<visualglmadspin0> processing buffer
0:00:04.406549847  4151      0x1d2c180 DEBUG              libvisual visual-gl.c:697:gst_visual_gl_chain:<visualglmadspin0> avail now 10224
0:00:04.406563102  4151      0x1d2c180 DEBUG              libvisual visual-gl.c:582:get_buffer:<visualglmadspin0> allocating output buffer with caps video/x-raw-gl, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1
Comment 18 Stefan Sauer (gstreamer, gtkdoc dev) 2011-11-24 16:26:33 UTC
Created attachment 202069 [details] [review]
libvisual plugin for gst-plugins-gl

This patch needs quiet a bit of more work. Besides the before mentioned functional issues it needs a bit of stylish work, e.g.:
* use GST_BOILERPLATE
* use _base_init (or comment why its not used)
Comment 19 Julien Isorce 2011-11-24 16:33:46 UTC
I'll try to debug it.
Comment 20 Julien Isorce 2011-11-25 16:12:19 UTC
Created attachment 202144 [details] [review]
libvisual_gl_lv_gltest now works

gst-inspect-0.10:

libvisual_gl_lv_flower: libvisual libvisual Pseudotoad flower, yellow rose of texas plugin v.0.1
  libvisual_gl_lv_gltest: libvisual libvisual GL analyser plugin v.0.1
  libvisual_gl_madspin: libvisual libvisual madspin port plugin v.0.1
  libvisual_gl_nastyfft: libvisual Libvisual NastyFFT plugin plugin v.0.5

libvisual_gl_lv_gltest: ok
libvisual_gl_nastyfft: almost
libvisual_gl_madspin: can see something but that's not what is expected
libvisual_gl_lv_flower: only can see first frame

We could commit the patch to start fixing pending problems from there.

A question: "SUBDIRS_EXT" in gst-plugins-gl/Makefile.am should be replaced by "ext" ?
Comment 21 Stefan Sauer (gstreamer, gtkdoc dev) 2011-11-25 17:10:14 UTC
Yes, I'd say commit it. And yes, $(SUBDIRS_EXT) should jst be ext in the patch.
Comment 22 Stefan Sauer (gstreamer, gtkdoc dev) 2011-11-25 18:19:08 UTC
Hmm, it still does not work here (intel gfx). Will try on nvidia later.
Comment 23 Stefan Sauer (gstreamer, gtkdoc dev) 2011-11-25 22:26:33 UTC
Tried a few more times on the intel gfx (also turned 3d desktop effects off), no luck. Now tried the same on nvideo (closed driver) and I am getting:

gst-launch-0.10: xcb_io.c:507: _XReply: Assertion `!dpy->xcb->reply_data' failed.
Comment 24 Stefan Sauer (gstreamer, gtkdoc dev) 2011-11-28 09:39:51 UTC
(In reply to comment #23)
> Tried a few more times on the intel gfx (also turned 3d desktop effects off),
> no luck. Now tried the same on nvideo (closed driver) and I am getting:
> 
> gst-launch-0.10: xcb_io.c:507: _XReply: Assertion `!dpy->xcb->reply_data'
> failed.

^^ this is on ati (fglrx), but there are other issues with the driver. Julien, please push this.
Comment 25 Stefan Sauer (gstreamer, gtkdoc dev) 2011-11-28 09:40:15 UTC
Comment on attachment 202144 [details] [review]
libvisual_gl_lv_gltest now works

please push with the 'ext' fix.
Comment 26 Julien Isorce 2011-11-28 11:18:06 UTC
Hi, I cannot until Thursday 01. I have not the environnement here to do that. You can do it if you have want. Julien
Comment 27 Stefan Sauer (gstreamer, gtkdoc dev) 2011-11-28 11:21:07 UTC
I pushed the patch with minor correction. Most of the gl elements work for me on nvidia. two issues:
* flower becomes white after a couple of frames
* projectM stays black

Julien, what gfx/driver card do you use?
Comment 28 Stefan Sauer (gstreamer, gtkdoc dev) 2011-11-28 11:29:00 UTC
Comment on attachment 202144 [details] [review]
libvisual_gl_lv_gltest now works

with smal mofifications.
Comment 29 Julien Isorce 2011-12-05 10:27:36 UTC
Created attachment 202796 [details]
snapshot of libvisual_gl_lv_gltest

OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: GeForce GT 330/PCI/SSE2
OpenGL version string: 3.3.0 NVIDIA 280.13

Linux version 3.0.0-13-generic-pae (buildd@rothera) (gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) ) #22-Ubuntu SMP Wed Nov 2 15:17:35 UTC 2011
Comment 30 Julien Isorce 2011-12-16 16:29:12 UTC
Some updates and for log:

There was a bug in libvisual_gl_lv_flower. It's fixed in Trunk since today ( http://libvisual.svn.sourceforge.net/viewvc/libvisual/trunk/libvisual-plugins/plugins/actor/pseudotoad_flower/main.c?view=log )

To know what is expected to show from libvisual_gl_lv_flower you have to build libvisual from Trunk:

svn co https://libvisual.svn.sourceforge.net/svnroot/libvisual/trunk

Build libvisual and install libvisual-plugins. Run trunk/examples/client/rebuild_sdl (use 0.4 or replace by 0.5)
and run ./lv-standalone lv_flower

Actually you will have something visible with 0.4 but not the exact result.

Well, even after the fix, it still does not work with gstgl. It needs more work.

Same result with projectM:

gst-inspect-0.10 libvisual-gl

  libvisual_gl_lv_flower: libvisual libvisual Pseudotoad flower, yellow rose of texas plugin v.0.1
  libvisual_gl_lv_gltest: libvisual libvisual GL analyser plugin v.0.1
  libvisual_gl_madspin: libvisual libvisual madspin port plugin v.0.1
  libvisual_gl_nastyfft: libvisual Libvisual NastyFFT plugin plugin v.0.5
  libvisual_gl_projectM: libvisual libvisual projectM plugin v.1.1

You can also run: ./lv-standalone projectM
or: qmmp sample.mp3  then right clic to select projectM visualisation
Comment 31 Julien Isorce 2011-12-22 14:41:53 UTC
commit 23b17f6d59e2c79897201724269a46cc5c709376
Author: Julien Isorce <julien.isorce@gmail.com>
Date:   Thu Dec 22 15:30:38 2011 +0100

    libvisual-gl: add minimal support to libvisual plugins that uses Framebuffer objects
    
    Fix bug #310775
    
    gst-launch audiotestsrc ! libvisual_gl_projectM ! glimagesink is working
    but for now you cannot append any other opengl filters between
    libvisual_gl_projectM and glimagesink because our FBO is turned OFF.
    
    It would require that libvisual allows to split rendering between
    pass1,2,3... and final rendering. In order to unbind our FBO before
    the passN, and then rebind it just before the final libvisual rendering.


----------------------


Try: gst-launch audiotestsrc ! libvisual_gl_projectM ! glimagesink
:)
Comment 32 Stefan Sauer (gstreamer, gtkdoc dev) 2011-12-23 17:54:23 UTC
Awesome, it works on my intel gfx system now. Switchign the window size still does not renegotiate. Thanks for the updates!
Comment 33 Julien Isorce 2012-01-03 16:43:47 UTC
Created attachment 204506 [details] [review]
libvisual-gl: full compatibility with projectM-libvisual if libprojectM >= 2.0.1

(In reply to comment #32)
> Switchign the window size still does not renegotiate.

Because our FBO is unbinded (see last commit). So the rendering from libvisual_gl_projectM is directly done into the opengl back buffer instead of our FBO. So it's not possible to chain FBOs in this case. The resizing pb is a consequence of that.

It's not correct to unbind it before the projectM rendering but I only do it to see something.
projectM does the rendering in several passes (it uses its own FBOs).

The correct way is to patch libprojectM. Before starting rendering it has to detect if a FBO is currently binded. If yes it has to keep the fbo's identifier in mind. Then it has to rebind it just before the final pass (when libprojectM is rendering to the back buffer):

 -1: detect if a fbo is currently binded <- patch libprojectM to add this step
 -2: projectM binds its own fbo to render into it (So it cancels step 1) and does the redering (=pass1).
 -3: bind our fbo  <- patch libprojectM to add this step
 -4: projectM's final rendering (no fbo)
  
Here I submitted a patch to projectM bug tracker that contains those requirements:
https://sourceforge.net/tracker/?func=detail&aid=3468910&group_id=104201&atid=637262

If projectM accepts the patch then the following 0001-libvisual-gl-full-compatibility-with-projectM-libvis.patch attached patch should be enough.
Comment 34 Julien Isorce 2012-01-03 16:49:58 UTC
Created attachment 204509 [details]
gst-launch-0.10 audiotestsrc ! libvisual_gl_projectM ! "video/x-raw-gl, width=800, height=600" ! gleffects effect=tunnel ! gldownload ! pngenc ! filesink location=sample2.png

If the 2 patches (1 for libprojectM and 1 for gst-plugins-gl/, see comment #33) then the pipeline

gst-launch-0.10 audiotestsrc ! libvisual_gl_projectM ! "video/x-raw-gl, width=800, height=600" ! gleffects effect=tunnel ! glimagesink

would work.

See attached sample2.png
Comment 35 Matthias Klumpp 2012-01-03 17:44:39 UTC
Hi!
Your projectM patch looks fine to me, I'll apply it if nobody else has objections.
Thanks!
Comment 36 Julien Isorce 2012-01-17 15:42:11 UTC
Created attachment 205454 [details] [review]
libvisual-gl: full compatibility with projectM-libvisual if libprojectM >= 2.0.1
Comment 37 Julien Isorce 2012-04-03 12:17:03 UTC
(In reply to comment #35)
> Hi!
> Your projectM patch looks fine to me, I'll apply it if nobody else has
> objections.
> Thanks!

Hi, the patch is here: https://sourceforge.net/tracker/?func=detail&aid=3468910&group_id=104201&atid=637262
Comment 38 Matthias Klumpp 2012-04-03 12:26:19 UTC
Okay, I asked the original projectM authors about this patch, they wanted to look at it, but nothing happened yet.
I looked at this patch too and it looks fine to me, as well as it is running without problems for months now, so I will apply this patch now :)
Thanks for your work!
Comment 39 Julien Isorce 2012-04-04 14:25:16 UTC
commit a08f746435165461cfb74fc7db967d4d4d913d2d
Author: Julien Isorce <julien.isorce@gmail.com>
Date:   Wed Apr 4 16:22:40 2012 +0200

    libvisual-gl: full compatibility with projectM-libvisual if libprojectM >= 2.0.1
    
    Fix bug #310775




Try:

gst-launch-0.10 audiotestsrc ! libvisual_gl_projectM ! "video/x-raw-gl,width=800, height=600" ! gleffects effect=tunnel ! glimagesink

:)