GNOME Bugzilla – Bug 704222
gldownload: Support for multi-planar format for GLES
Last modified: 2016-05-15 08:32:39 UTC
The attached patch in theory adds support for multi-planar formats for GLES2, by using a NV extension that was added officially to GLES3. Unfortunately this crashes here when reading the second/third fbo colorbuffer attachment... no idea yet why :) Keeping it here as a reference if anybody else has time to work on this before me. Also missing: multi-planar formats should only be supported if GLES3 or the NV extension is available.
Created attachment 249158 [details] [review] 0001-gldownload-Add-support-for-downloading-multi-planar-.patch
Review of attachment 249158 [details] [review]: The reason it crashes is that GL_COLOR_ATTACHMENTi (i != 0) is defined to be 0 in gst-libs/gst/gl/gstgles2.h. Changing that to the correct value makes it work properly. I've been thinking about adding a code generator that generates the enums, typedefs and function definitions we need. Recently, Khronos announced their xml registry and code generator which on first glance looks like what we would need. See https://cvs.khronos.org/svn/repos/ogl/trunk/doc/registry/public/api/readme.pdf. ::: gst-libs/gst/gl/glprototypes/gles3opengl.h @@ -54,3 @@ - (GLsizei n, const GLenum *bufs)) -GST_GL_EXT_END () - This should stay here. GST_GL_API_GLES3 is correct. It is the GL api that the function is core in. If that fails it should try the NV extension.
Created attachment 249740 [details] [review] 0001-gldownload-Add-support-for-downloading-multi-planar-.patch Ah, was really something stupid then :) It doesn't crash anymore at least, but produces just zeroes for all planes here.
(In reply to comment #3) > Created an attachment (id=249740) [details] [review] > 0001-gldownload-Add-support-for-downloading-multi-planar-.patch > > Ah, was really something stupid then :) It doesn't crash anymore at least, but > produces just zeroes for all planes here. Works for me :) Pipeline: videotestsrc ! gleffects effect=0 ! video/x-raw,format=I420 ! xvimagesink (YV12 works too)
Also, don't change the GST_GL_API_GLES3 to GLES2. That value is what API the function is core in. The code will check for the extension version if the GL api is not 'new' enough.
(In reply to comment #4) > (In reply to comment #3) > > Created an attachment (id=249740) [details] [review] [details] [review] > > 0001-gldownload-Add-support-for-downloading-multi-planar-.patch > > > > Ah, was really something stupid then :) It doesn't crash anymore at least, but > > produces just zeroes for all planes here. > > Works for me :) > > Pipeline: videotestsrc ! gleffects effect=0 ! video/x-raw,format=I420 ! > xvimagesink > (YV12 works too) Only green for me, I guess something else is broken here then :) Or are you sure that you disabled desktop GL? (In reply to comment #5) > Also, don't change the GST_GL_API_GLES3 to GLES2. That value is what API the > function is core in. The code will check for the extension version if the GL > api is not 'new' enough. If I kept it at 3 the extensions were not found, and gl->DrawBuffers for example was NULL
(In reply to comment #6) > (In reply to comment #4) > > (In reply to comment #3) > > > Created an attachment (id=249740) [details] [review] [details] [review] [details] [review] > > > 0001-gldownload-Add-support-for-downloading-multi-planar-.patch > > > > > > Ah, was really something stupid then :) It doesn't crash anymore at least, but > > > produces just zeroes for all planes here. > > > > Works for me :) > > > > Pipeline: videotestsrc ! gleffects effect=0 ! video/x-raw,format=I420 ! > > xvimagesink > > (YV12 works too) > > Only green for me, I guess something else is broken here then :) Or are you > sure that you disabled desktop GL? 0:00:00.104579779 3997 0x244d770 INFO glwindow gstglwindow.c:503:_create_context_gles2: GL_VERSION: OpenGL ES 2.0 Mesa 9.2.0 (git-6b53e2b) 0:00:00.104591862 3997 0x244d770 INFO glwindow gstglwindow.c:505:_create_context_gles2: GL_SHADING_LANGUAGE_VERSION: OpenGL ES GLSL ES 1.0.16 0:00:00.104605341 3997 0x244d770 INFO glwindow gstglwindow.c:506:_create_context_gles2: GL_VENDOR: nouveau 0:00:00.104617633 3997 0x244d770 INFO glwindow gstglwindow.c:507:_create_context_gles2: GL_RENDERER: Gallium 0.4 on NV98 and configured with --disable-opengl > (In reply to comment #5) > > Also, don't change the GST_GL_API_GLES3 to GLES2. That value is what API the > > function is core in. The code will check for the extension version if the GL > > api is not 'new' enough. > > If I kept it at 3 the extensions were not found, and gl->DrawBuffers for > example was NULL What platform is this on? can you try es2_info (glxinfo for GLES2)? or try a GST_DEBUG=gl*:9 log
(In reply to comment #7) > > (In reply to comment #5) > > > Also, don't change the GST_GL_API_GLES3 to GLES2. That value is what API the > > > function is core in. The code will check for the extension version if the GL > > > api is not 'new' enough. > > > > If I kept it at 3 the extensions were not found, and gl->DrawBuffers for > > example was NULL > > What platform is this on? can you try es2_info (glxinfo for GLES2)? or try a > GST_DEBUG=gl*:9 log Does this help? EGL_VERSION = 1.4 (DRI2) EGL_VENDOR = Mesa Project EGL_EXTENSIONS = EGL_MESA_drm_image EGL_WL_bind_wayland_display EGL_KHR_image_base EGL_KHR_image_pixmap EGL_KHR_image EGL_KHR_gl_renderbuffer_image EGL_KHR_surfaceless_context EGL_KHR_create_context EGL_NOK_swap_region EGL_NOK_texture_from_pixmap EGL_NV_post_sub_buffer EGL_CLIENT_APIS = OpenGL OpenGL_ES OpenGL_ES2 OpenGL_ES3 GL_VERSION: OpenGL ES 3.0 Mesa 9.1.4 GL_RENDERER: Mesa DRI Intel(R) Sandybridge Mobile GL_EXTENSIONS: GL_EXT_blend_minmax, GL_EXT_multi_draw_arrays, GL_EXT_texture_filter_anisotropic, GL_EXT_texture_compression_dxt1, GL_EXT_texture_format_BGRA8888, GL_OES_compressed_ETC1_RGB8_texture, GL_OES_depth24, GL_OES_element_index_uint, GL_OES_fbo_render_mipmap, GL_OES_mapbuffer, GL_OES_rgb8_rgba8, GL_OES_standard_derivatives, GL_OES_stencil8, GL_OES_texture_3D, GL_OES_texture_npot, GL_OES_EGL_image, GL_OES_depth_texture, GL_OES_packed_depth_stencil, GL_EXT_texture_type_2_10_10_10_REV, GL_OES_get_program_binary, GL_APPLE_texture_max_level, GL_EXT_read_format_bgra, GL_NV_fbo_color_attachments, GL_OES_vertex_array_object, GL_ANGLE_texture_compression_dxt3, GL_ANGLE_texture_compression_dxt5, GL_EXT_texture_rg, GL_EXT_unpack_subimage, GL_NV_draw_buffers, GL_NV_read_buffer, GL_EXT_map_buffer_range, GL_OES_depth_texture_cube_map, GL_EXT_color_buffer_float
Created attachment 249767 [details] [review] gles3 is a strict superset of gles2 This should fix your issue. However it doesn't explain why it didn't pick up the extension version instead. :S
Review of attachment 249767 [details] [review]: Well, why would it ever dwelve into the GLES3 API if the API selected was only GLES2? Or is the only difference then that it doesn't try the non-extension variants? Maybe that code should be changed to always try the extension and normal variants? Anyway: objdump -T /usr/lib/x86_64-linux-gnu/libGLESv2.so.2 | grep NV 0000000000004900 g DF .text 0000000000000000 Base glDrawBuffersNV 0000000000004140 g DF .text 0000000000000000 Base glReadBufferNV ::: gst-libs/gst/gl/gstglapi.h @@ +65,3 @@ GST_GL_API_OPENGL3 = (1 << 1), GST_GL_API_GLES = (1 << 15), GST_GL_API_GLES2 = (1 << 16), Shouldn't GLES2 be subset of GLES then? And OPENGL3 of OPENGL? @@ +66,3 @@ GST_GL_API_GLES = (1 << 15), GST_GL_API_GLES2 = (1 << 16), + GST_GL_API_GLES3 = GST_GL_API_GLES2 + (1 << 17), Instead of a plus, make it the binary operation it actually is: | :) Makes it more intuitive IMHO
s/subset/superset/ of course
(In reply to comment #10) > Review of attachment 249767 [details] [review]: > > Well, why would it ever dwelve into the GLES3 API if the API selected was only > GLES2? Or is the only difference then that it doesn't try the non-extension > variants? Maybe that code should be changed to always try the extension and > normal variants? It already tries all extension variants, regardless of GL vs GLES selected. And it already tries the non-extension variant if it can't find the function in the core GL api specified (the GST_GL_API_GLES2 you kept wanting to change). All of this logic is in gstglfeature.c. It makes more sense to me in code :). > Anyway: > objdump -T /usr/lib/x86_64-linux-gnu/libGLESv2.so.2 | grep NV > 0000000000004900 g DF .text 0000000000000000 Base glDrawBuffersNV > 0000000000004140 g DF .text 0000000000000000 Base glReadBufferNV If you run with GST_DEBUG=default:9 it should show some debug about the function retrieval. Specifically whether it could find the extension in the extension list and whether it failed to get a function (and thus the whole extension). > ::: gst-libs/gst/gl/gstglapi.h > @@ +65,3 @@ > GST_GL_API_OPENGL3 = (1 << 1), > GST_GL_API_GLES = (1 << 15), > GST_GL_API_GLES2 = (1 << 16), > > Shouldn't GLES2 be subset of GLES then? And OPENGL3 of OPENGL? No because there were some functions removed in the GLES -> GLES2 transition and OPENGL3 effectively means GL 3 minus the deprecated bits (fixed function pipeline). If you have GL 3 with the compatibility profile, that would be OPENGL3 | OPENGL. Yes, the OPENGL and OPENGL3 functions overlap and we will have to accommodate that when we ask for a GL3 forward compatible context. > @@ +66,3 @@ > GST_GL_API_GLES = (1 << 15), > GST_GL_API_GLES2 = (1 << 16), > + GST_GL_API_GLES3 = GST_GL_API_GLES2 + (1 << 17), > > Instead of a plus, make it the binary operation it actually is: | :) Makes it > more intuitive IMHO ok :)
Well, it was obviously not detected because "read_buffer" was not in the extensions list at the end.
Created attachment 249878 [details] [review] 0001-gldownload-Add-support-for-downloading-multi-planar-.patch Still gives me only green though
(In reply to comment #14) > Created an attachment (id=249878) [details] [review] > 0001-gldownload-Add-support-for-downloading-multi-planar-.patch > > Still gives me only green though Works for me under nouveau and the software renderer (llvmpipe). Does running it with LIBGL_ALWAYS_SOFTWARE=1 make it work? Also try setting MESA_DEBUG=1 and see what you get. Try some others http://www.mesa3d.org/envvars.html
LIBGL_ALWAYS_SOFTWARE=1 makes it work, MESA_DEBUG=1 gives lots of warnings :) Setting pipeline to PAUSED ... libEGL warning: failed to create a pipe screen for i965 Mesa warning: couldn't open libtxc_dxtn.so, software DXTn compression/decompression unavailable Pipeline is PREROLLING ... Mesa: User error: GL_INVALID_ENUM in glEnable(GL_TEXTURE_2D) Mesa: User error: GL_INVALID_OPERATION in glReadPixels(invalid format GL_LUMINANCE and/or type GL_UNSIGNED_BYTE) Mesa: User error: GL_INVALID_OPERATION in glReadPixels(invalid format GL_LUMINANCE and/or type GL_UNSIGNED_BYTE) Mesa: User error: GL_INVALID_OPERATION in glReadPixels(invalid format GL_LUMINANCE and/or type GL_UNSIGNED_BYTE) Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock Mesa: User error: GL_INVALID_ENUM in glEnable(GL_TEXTURE_2D) Mesa: User error: GL_INVALID_OPERATION in glReadPixels(invalid format GL_LUMINANCE and/or type GL_UNSIGNED_BYTE) Mesa: User error: GL_INVALID_OPERATION in glReadPixels(invalid format GL_LUMINANCE and/or type GL_UNSIGNED_BYTE) Mesa: User error: GL_INVALID_OPERATION in glReadPixels(invalid format GL_LUMINANCE and/or type GL_UNSIGNED_BYTE) [...]
Seems that GLES only supports GL_ALPHA/RGB/RGBA http://www.khronos.org/opengles/sdk/docs/man/xhtml/glReadPixels.xml format: Specifies the format of the pixel data. The following symbolic values are accepted: GL_ALPHA, GL_RGB, and GL_RGBA.
Maybe more luck here with GL_EXT_texture_rg here
(In reply to comment #16) > LIBGL_ALWAYS_SOFTWARE=1 makes it work, MESA_DEBUG=1 gives lots of warnings :) > > Setting pipeline to PAUSED ... > libEGL warning: failed to create a pipe screen for i965 > Mesa warning: couldn't open libtxc_dxtn.so, software DXTn > compression/decompression unavailable > Pipeline is PREROLLING ... > Mesa: User error: GL_INVALID_ENUM in glEnable(GL_TEXTURE_2D) > Mesa: User error: GL_INVALID_OPERATION in glReadPixels(invalid format > GL_LUMINANCE and/or type GL_UNSIGNED_BYTE) > Mesa: User error: GL_INVALID_OPERATION in glReadPixels(invalid format > GL_LUMINANCE and/or type GL_UNSIGNED_BYTE) > Mesa: User error: GL_INVALID_OPERATION in glReadPixels(invalid format > GL_LUMINANCE and/or type GL_UNSIGNED_BYTE) > Pipeline is PREROLLED ... > Setting pipeline to PLAYING ... > New clock: GstSystemClock > Mesa: User error: GL_INVALID_ENUM in glEnable(GL_TEXTURE_2D) > Mesa: User error: GL_INVALID_OPERATION in glReadPixels(invalid format > GL_LUMINANCE and/or type GL_UNSIGNED_BYTE) > Mesa: User error: GL_INVALID_OPERATION in glReadPixels(invalid format > GL_LUMINANCE and/or type GL_UNSIGNED_BYTE) > Mesa: User error: GL_INVALID_OPERATION in glReadPixels(invalid format > GL_LUMINANCE and/or type GL_UNSIGNED_BYTE) > [...] Interesting... I only get the GL_INVALID_ENUM error. Seems the intel drivers are stricter. (In reply to comment #18) > Maybe more luck here with GL_EXT_texture_rg here Agreed.
(In reply to comment #19) > > Interesting... I only get the GL_INVALID_ENUM error. Seems the intel drivers > are stricter. Why do we get this INVALID_ENUM btw? :) > (In reply to comment #18) > > Maybe more luck here with GL_EXT_texture_rg here > > Agreed. Only that it will cause so many special cases everywhere in the code... but should be the best solution overall, it's also faster.
(In reply to comment #20) > (In reply to comment #19) > > > > > Interesting... I only get the GL_INVALID_ENUM error. Seems the intel drivers > > are stricter. > > Why do we get this INVALID_ENUM btw? :) Well, run it in a debugger and you can break on _mesa_error to find out :). I just did that and it has to do with calling glEnable(GL_TEXTURE_RECTANGLE). we define GL_TEXTURE_RECTANGLE to be GL_TEXTURE_2D in GLES2 and enable/disable with that value is an error (in GLES2).
Just a note I discovered, in GLES2, GL_RGBA/GL_UNSIGNED_BYTE are the only formats officially supported by glReadPixels (GLES3 has the RGB and ALPHA variants as well). So it will be difficult to download multi-planer formats with GLES.
Not sure to follow, but GLES3 mention GL_LUMINANCE in the format description (and seems to depend on GL_IMPLEMENTATION_COLOR_READ_FORMAT) https://www.khronos.org/opengles/sdk/docs/man3/ and https://www.khronos.org/opengles/sdk/docs/man3/html/glReadPixels.xhtml (I do not remember if it was really working with gstgl-0.10 http://cgit.freedesktop.org/gstreamer/gst-plugins-gl/tree/gst-libs/gst/gl/gstgldisplay.c?h=0.10#n3613 )
commit 001b92ba129a3d263dd43eabece172b365a3f075 Author: Matthew Waters <ystreet00@gmail.com> Date: Wed May 21 21:47:45 2014 +1000 gl/memory: implement GL_EXT_texture_rg support Which is used by default over the Luminance formats due to it being color renderable with fbos (and deprecation/removal with GL 3.x). https://bugzilla.gnome.org/show_bug.cgi?id=729750 https://bugzilla.gnome.org/show_bug.cgi?id=704222 https://bugzilla.gnome.org/show_bug.cgi?id=728890
(In reply to comment #23) > Not sure to follow, but GLES3 mention GL_LUMINANCE in the format description > (and seems to depend on GL_IMPLEMENTATION_COLOR_READ_FORMAT) > https://www.khronos.org/opengles/sdk/docs/man3/ and > https://www.khronos.org/opengles/sdk/docs/man3/html/glReadPixels.xhtml Download requires glReadPixels support for the number of components we are attempting to download. The actual GLES2/3 spec only allows the format to be RGBA/UNSIGNED_BYTE or the implementation/framebuffer specific read format. If we can get a gl format that fits then it will work however there are no guarentees as to the format and thus whether it will work. > (I do not remember if it was really working with gstgl-0.10 > http://cgit.freedesktop.org/gstreamer/gst-plugins-gl/tree/gst-libs/gst/gl/gstgldisplay.c?h=0.10#n3613 > )
1- So nothing we can do more ? Can we close this bug ? 2- Also should we open a bug to detect GLES3 ? Because I tested gst-launch-1.0 videotestsrc ! gleffects ! "video/x-raw,format=I420" ! xvimagesink and it fails here http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/gst-libs/gst/gl/gstglcolorconvert.c#n977 because USING_GLES2 is true. 3- And it does not accept GST_GL_API=gles3: glcontext gstglcontext.c:606:_create_context_gles2: GL_VERSION: OpenGL ES 3.0 4- is attachment 249278 [details] [review] obsolete due to your recent commits ? should we push attachment 749767 ? Thx!
(In reply to comment #26) > 1- So nothing we can do more ? Can we close this bug ? We could try and parse the GL_IMPLEMENTATION_COLOR_READ format/type and work with the format it produces (which could be framebuffer attachment types specific) but other than that, nothing we can do. > 2- Also should we open a bug to detect GLES3 ? Because I tested gst-launch-1.0 > videotestsrc ! gleffects ! "video/x-raw,format=I420" ! xvimagesink and it fails > here > http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/gst-libs/gst/gl/gstglcolorconvert.c#n977 > because USING_GLES2 is true. It will fail the same way in GLES3 as in GLES2 so not relevant. > 3- And it does not accept GST_GL_API=gles3: > > glcontext gstglcontext.c:606:_create_context_gles2: GL_VERSION: OpenGL ES 3.0 Yea, that us not detecting GLES3 at all. > 4- is attachment 249278 [details] [review] obsolete due to your recent commits ? should we push > attachment 749767 ? 249278 is obsolete :). I'll push 749767 > Thx!
Actually, I don't think that we should expose GLES3 through an api flag but through a version check like Desktop GL 1.x/2.x.
commit 9ea15579ec5402c52e995aaed85347358d859a68 Author: Matthew Waters <matthew@centricular.com> Date: Thu Mar 31 19:38:12 2016 +1100 glmemory: add checking the read implementation format/type on gles2 platforms By default, reading GL_RED or GL_RG us unsupported by glReadPixels unless exposed through GL_COLOR_READ_IMPLEMENTATION_FORMAT/TYPE. This allows downloading multiple-planar video frames where possible.