GNOME Bugzilla – Bug 310775
libvisual element needs to support opengl
Last modified: 2012-05-10 21:51:17 UTC
We need to support opengl-rendering libvisual plugins. The first level would be giving it an off-screen opengl context, grabbing pixels into buffers and pushing them out as normal video frames. The best, though, would be for gl-capable sinks to return subclasses of GstBuffer from their bufferalloc functions, and for gl-capable renderers to peel the gl context off the buffer directly.
Since we don't have gl-capable sinks in 0.9 thi bug is only for the first level (stealing the gl context's pixels).
*** Bug 310696 has been marked as a duplicate of this bug. ***
There's no nice way to do that with libvisual. IIRC libvisual works with SDL to do opengl visualisation rendering, so what we could do is write another libvisual based GStreamer element called libvisualsink that would take audio buffers and render using SDL. That would allow for OpenGL visualisation actors to be used. Do we really need to have a bug for that ?
I've implemented support for opengl libvisual visualisations using the following hackery: - added a 'glcontext' interface that gl-based sinks can implement to allow other elements (and maybe applications) to request that they make their gl context current in the calling thread. I thought this was better than passing the context back from the sink to the rendering element and (eventually) having #ifdefs everywhere for calls to glXMakeCurrent, wglMakeCurrent, etc. - added a new content type, video/x-opengl-texture-id to glimagesink's caps. Buffers with this content type contain only an opengl texture id that the sink is to use to draw the frame. Hypothetical intermediate elements (gltextoverlay?) could also draw on the pbuffer. - created a variant of the libvisual plugin (I intend to merge the code back in at some point, but haven't gotten around to it yet) that creates a framebuffer (using GL_EXT_framebuffer_object) and convinces libvisual to render into it. For each frame, it then pushes a video/x-opengl-texture-id buffer containing the framebuffer's texture id. This all seems to work fairly well, but it's a bit ugly in places, particularly how video/x-opengl-texture-id buffers are handled. I've tested all the gl-based visualisations that come with libvisual (but not projectM, as far as I recall). I haven't looked in to getting playbin to handle it properly (not trying to run the vis plugin output through ffmpegcolorspace, for a start). It'd be nice to get it to automatically construct a gl-capable video sink if the vis plugin has video/x-opengl-texture-id caps, but I'd be happy enough if it required the application to do that for itself.
what's the status of this bug? the last comment seems to be quite old, is there any progress on it?
Any news?
Unassigning myself -- a proper solution will use schleef's new gl buffer foo
Created attachment 134676 [details] [review] libvisual plugin for gst-plugins-gl This more or less works. Haven't really tested with projectM as it seems to fall back to software rendering on this machine, and all the other libvisual GL plugins are fairly boring. Obvious problems: shares 80% of its code with the libvisual plugin in -base, does awful hackery with GL matrices to give the libvisual actor the rendering state it expects.
This bug should be moved to gst-plugins-gl.
Oh strange. I completely missed your patch. I have a similar one here in a dev branch. I should compare to see if I missed something :)
I tested the patch a little bit. Seems to work. There is some jerk sometimes and some darkness. Less with a mp3 than audiotestsrc. I could try to find what's happen later. Anyway, I think libgstgl (gst-plugins-gl/gst-libs/gst/gl should be build as a shared library (for now it's static) So instead of having: libgstopengl.so libgstlibvisualgl.so We should have: libgstgl-0.10.so (and deploy gstglbuffer.h) libgstopengl.so libgstlibvisualgl.so
Any news about this? Why was the patch not yet applied?
Hi, I was wrong in 'Comment 11'. There is already a libgstgl-0.10.so. So there is no blocking thing. I will try to test the patch again to check if it still work.
Did it work? Is there a chance to get these patches included in GStreamer? (There are many people asking for projectM visualisations in Rhythmbox etc. so if there's nothing blocking, I think it would be a great feature to have) Thanks!
This patch needs a bit of updating. gst-plugins-gl/ext/libvisual:(git:master)> make CC libgstlibvisualgl_la-visual-gl.lo cc1: warnings being treated as errors visual-gl.c: In function ‘gst_visual_gl_class_init’: visual-gl.c:215: error: implicit declaration of function ‘gst_element_class_set_details’ visual-gl.c: In function ‘gst_visual_gl_change_state’: visual-gl.c:818: error: too many arguments to function ‘gst_gl_display_create_context’ make: *** [libgstlibvisualgl_la-visual-gl.lo] Error 1 Julien, do you have an updated patch?
Hi. no.
I've updated the patch to make things build, but it does not work for me. I am getting these elements: libvisual-gl: libvisual_gl_projectM: libvisual projectM libvisual-gl: libvisual_gl_nastyfft: Libvisual NastyFFT plugin libvisual-gl: libvisual_gl_madspin: libvisual madspin port libvisual-gl: libvisual_gl_lv_gltest: libvisual GL analyser libvisual-gl: libvisual_gl_lv_flower: libvisual Pseudotoad flower, yello But none of them work for me: gst-launch filesrc location=$HOME/Music/1.mp3 ! decodebin2 ! tee name=t ! queue ! audioconvert ! libvisual_gl_madspin ! glimagesink t. ! queue ! pulsesink => just a white window gst-launch filesrc location=$HOME/Music/1.mp3 ! decodebin2 ! tee name=t ! queue ! audioconvert ! libvisual_gl_madspin ! gldownload ! xvimagesink t. ! queue ! pulsesink => just a black window When taking a peek into the debug log, it looks like it is doing some work though: 0:00:04.406503587 4151 0x1d2c180 DEBUG libvisual visual-gl.c:663:gst_visual_gl_chain:<visualglmadspin0> chain function called 0:00:04.406522029 4151 0x1d2c180 DEBUG libvisual visual-gl.c:687:gst_visual_gl_chain:<visualglmadspin0> Input buffer has 1152 samples, time=4231836734 0:00:04.406536852 4151 0x1d2c180 DEBUG libvisual visual-gl.c:694:gst_visual_gl_chain:<visualglmadspin0> processing buffer 0:00:04.406549847 4151 0x1d2c180 DEBUG libvisual visual-gl.c:697:gst_visual_gl_chain:<visualglmadspin0> avail now 10224 0:00:04.406563102 4151 0x1d2c180 DEBUG libvisual visual-gl.c:582:get_buffer:<visualglmadspin0> allocating output buffer with caps video/x-raw-gl, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1
Created attachment 202069 [details] [review] libvisual plugin for gst-plugins-gl This patch needs quiet a bit of more work. Besides the before mentioned functional issues it needs a bit of stylish work, e.g.: * use GST_BOILERPLATE * use _base_init (or comment why its not used)
I'll try to debug it.
Created attachment 202144 [details] [review] libvisual_gl_lv_gltest now works gst-inspect-0.10: libvisual_gl_lv_flower: libvisual libvisual Pseudotoad flower, yellow rose of texas plugin v.0.1 libvisual_gl_lv_gltest: libvisual libvisual GL analyser plugin v.0.1 libvisual_gl_madspin: libvisual libvisual madspin port plugin v.0.1 libvisual_gl_nastyfft: libvisual Libvisual NastyFFT plugin plugin v.0.5 libvisual_gl_lv_gltest: ok libvisual_gl_nastyfft: almost libvisual_gl_madspin: can see something but that's not what is expected libvisual_gl_lv_flower: only can see first frame We could commit the patch to start fixing pending problems from there. A question: "SUBDIRS_EXT" in gst-plugins-gl/Makefile.am should be replaced by "ext" ?
Yes, I'd say commit it. And yes, $(SUBDIRS_EXT) should jst be ext in the patch.
Hmm, it still does not work here (intel gfx). Will try on nvidia later.
Tried a few more times on the intel gfx (also turned 3d desktop effects off), no luck. Now tried the same on nvideo (closed driver) and I am getting: gst-launch-0.10: xcb_io.c:507: _XReply: Assertion `!dpy->xcb->reply_data' failed.
(In reply to comment #23) > Tried a few more times on the intel gfx (also turned 3d desktop effects off), > no luck. Now tried the same on nvideo (closed driver) and I am getting: > > gst-launch-0.10: xcb_io.c:507: _XReply: Assertion `!dpy->xcb->reply_data' > failed. ^^ this is on ati (fglrx), but there are other issues with the driver. Julien, please push this.
Comment on attachment 202144 [details] [review] libvisual_gl_lv_gltest now works please push with the 'ext' fix.
Hi, I cannot until Thursday 01. I have not the environnement here to do that. You can do it if you have want. Julien
I pushed the patch with minor correction. Most of the gl elements work for me on nvidia. two issues: * flower becomes white after a couple of frames * projectM stays black Julien, what gfx/driver card do you use?
Comment on attachment 202144 [details] [review] libvisual_gl_lv_gltest now works with smal mofifications.
Created attachment 202796 [details] snapshot of libvisual_gl_lv_gltest OpenGL vendor string: NVIDIA Corporation OpenGL renderer string: GeForce GT 330/PCI/SSE2 OpenGL version string: 3.3.0 NVIDIA 280.13 Linux version 3.0.0-13-generic-pae (buildd@rothera) (gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) ) #22-Ubuntu SMP Wed Nov 2 15:17:35 UTC 2011
Some updates and for log: There was a bug in libvisual_gl_lv_flower. It's fixed in Trunk since today ( http://libvisual.svn.sourceforge.net/viewvc/libvisual/trunk/libvisual-plugins/plugins/actor/pseudotoad_flower/main.c?view=log ) To know what is expected to show from libvisual_gl_lv_flower you have to build libvisual from Trunk: svn co https://libvisual.svn.sourceforge.net/svnroot/libvisual/trunk Build libvisual and install libvisual-plugins. Run trunk/examples/client/rebuild_sdl (use 0.4 or replace by 0.5) and run ./lv-standalone lv_flower Actually you will have something visible with 0.4 but not the exact result. Well, even after the fix, it still does not work with gstgl. It needs more work. Same result with projectM: gst-inspect-0.10 libvisual-gl libvisual_gl_lv_flower: libvisual libvisual Pseudotoad flower, yellow rose of texas plugin v.0.1 libvisual_gl_lv_gltest: libvisual libvisual GL analyser plugin v.0.1 libvisual_gl_madspin: libvisual libvisual madspin port plugin v.0.1 libvisual_gl_nastyfft: libvisual Libvisual NastyFFT plugin plugin v.0.5 libvisual_gl_projectM: libvisual libvisual projectM plugin v.1.1 You can also run: ./lv-standalone projectM or: qmmp sample.mp3 then right clic to select projectM visualisation
commit 23b17f6d59e2c79897201724269a46cc5c709376 Author: Julien Isorce <julien.isorce@gmail.com> Date: Thu Dec 22 15:30:38 2011 +0100 libvisual-gl: add minimal support to libvisual plugins that uses Framebuffer objects Fix bug #310775 gst-launch audiotestsrc ! libvisual_gl_projectM ! glimagesink is working but for now you cannot append any other opengl filters between libvisual_gl_projectM and glimagesink because our FBO is turned OFF. It would require that libvisual allows to split rendering between pass1,2,3... and final rendering. In order to unbind our FBO before the passN, and then rebind it just before the final libvisual rendering. ---------------------- Try: gst-launch audiotestsrc ! libvisual_gl_projectM ! glimagesink :)
Awesome, it works on my intel gfx system now. Switchign the window size still does not renegotiate. Thanks for the updates!
Created attachment 204506 [details] [review] libvisual-gl: full compatibility with projectM-libvisual if libprojectM >= 2.0.1 (In reply to comment #32) > Switchign the window size still does not renegotiate. Because our FBO is unbinded (see last commit). So the rendering from libvisual_gl_projectM is directly done into the opengl back buffer instead of our FBO. So it's not possible to chain FBOs in this case. The resizing pb is a consequence of that. It's not correct to unbind it before the projectM rendering but I only do it to see something. projectM does the rendering in several passes (it uses its own FBOs). The correct way is to patch libprojectM. Before starting rendering it has to detect if a FBO is currently binded. If yes it has to keep the fbo's identifier in mind. Then it has to rebind it just before the final pass (when libprojectM is rendering to the back buffer): -1: detect if a fbo is currently binded <- patch libprojectM to add this step -2: projectM binds its own fbo to render into it (So it cancels step 1) and does the redering (=pass1). -3: bind our fbo <- patch libprojectM to add this step -4: projectM's final rendering (no fbo) Here I submitted a patch to projectM bug tracker that contains those requirements: https://sourceforge.net/tracker/?func=detail&aid=3468910&group_id=104201&atid=637262 If projectM accepts the patch then the following 0001-libvisual-gl-full-compatibility-with-projectM-libvis.patch attached patch should be enough.
Created attachment 204509 [details] gst-launch-0.10 audiotestsrc ! libvisual_gl_projectM ! "video/x-raw-gl, width=800, height=600" ! gleffects effect=tunnel ! gldownload ! pngenc ! filesink location=sample2.png If the 2 patches (1 for libprojectM and 1 for gst-plugins-gl/, see comment #33) then the pipeline gst-launch-0.10 audiotestsrc ! libvisual_gl_projectM ! "video/x-raw-gl, width=800, height=600" ! gleffects effect=tunnel ! glimagesink would work. See attached sample2.png
Hi! Your projectM patch looks fine to me, I'll apply it if nobody else has objections. Thanks!
Created attachment 205454 [details] [review] libvisual-gl: full compatibility with projectM-libvisual if libprojectM >= 2.0.1
(In reply to comment #35) > Hi! > Your projectM patch looks fine to me, I'll apply it if nobody else has > objections. > Thanks! Hi, the patch is here: https://sourceforge.net/tracker/?func=detail&aid=3468910&group_id=104201&atid=637262
Okay, I asked the original projectM authors about this patch, they wanted to look at it, but nothing happened yet. I looked at this patch too and it looks fine to me, as well as it is running without problems for months now, so I will apply this patch now :) Thanks for your work!
commit a08f746435165461cfb74fc7db967d4d4d913d2d Author: Julien Isorce <julien.isorce@gmail.com> Date: Wed Apr 4 16:22:40 2012 +0200 libvisual-gl: full compatibility with projectM-libvisual if libprojectM >= 2.0.1 Fix bug #310775 Try: gst-launch-0.10 audiotestsrc ! libvisual_gl_projectM ! "video/x-raw-gl,width=800, height=600" ! gleffects effect=tunnel ! glimagesink :)