GNOME Bugzilla – Bug 747297
gl: live rpicamsrc/omxh264dec/glimagesink pipeline fails
Last modified: 2016-09-29 01:40:35 UTC
This fails on my RPi: gst-launch-1.0 rpicamsrc preview=0 fullscreen=0 ! queue ! h264parse ! omxh264dec ! glimagesink sync=false On that pipeline we end up creating 3 GstGLBufferPools with unlimited buffer size (max_buffers=0). At some point one of the pools errors out, fails to create an EGLImage, likely because resources are exhausted. It's a regression from 1.4.x. Perhaps we should limit max_buffers for the EGL/RPI scenario, or use less buffer pools?
Created attachment 309363 [details] [review] workaround I use this here but I think this might not be the right solution :/
Why are there three buffer pools? I can only think of two that would be created (glimagesinkelement and omxh264enc).
Hum I reported this bug 4 months ago, maybe it was different back then :) I'll check again what's going on.
4 different pools are created currently:
+ Trace 235338
Hi I think you are trying to display video that is bigger than your physical display; on my test setup (raspbian 8, gstreamer 1.4.4-2.1rpi4rpi1g, 1280x800 display): gst-launch-1.0 rpicamsrc preview=0 fullscreen=0 ! queue ! h264parse ! omxh264dec ! glimagesink sync=false eglCreateImageKHR: failed to create image for buffer 0x2 target 12465 error 0x300c However, the following pipeline does work gst-launch-1.0 rpicamsrc preview=0 fullscreen=0 ! video/x-h264\,\ width\=\(int\)1280\,\ height\=\(int\)800\,\ framerate\=\(fraction\)30/1 ! queue ! h264parse ! omxh264dec ! glimagesink sync=false You didn't report the error messages you were seeing, was it similar to this one ? eglCreateImageKHR: failed to create image for buffer 0x2 target 12465 error 0x300c
Yes, same error: eglCreateImageKHR: failed to create image for buffer 0x2 target 12465 error 0x300c 0:00:01.147965729 284 0x73202090 ERROR eglimagememory gsteglimagememory.c:701:gst_egl_image_memory_setup_buffer: Failed to create EGLImage 0:00:01.472037447 284 0x1a883b0 ERROR default gstglutils.c:80:gst_gl_context_check_framebuffer_status: GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT ERROR: from element /GstPipeline:pipeline0/GstGLImageSinkBin:glimagesinkbin0/GstGLColorConvertElement:glcolorconvertelement0: Failed to convert video buffer Additional debug info: gstglcolorconvertelement.c(217): gst_gl_color_convert_element_prepare_output_buffer (): /GstPipeline:pipeline0/GstGLImageSinkBin:glimagesinkbin0/GstGLColorConvertElement:glcolorconvertelement0 Execution ended after 0:00:00.904222291
And with a capsfilter setting the size to 1280x800 it indeed works.
Great :) Did you succeed to have the pipeline run fuildly ? I didn't, and i don't know why because the CPU isn't starved.
Well I think the queue introduces some lag, apart from that it is smooth. However I can see rendering artifacts in the bottom of the picture: https://dl.dropboxusercontent.com/u/280087/2016-01-04%2010.35.11.mp4 This is on a RPi2 btw.
Getting back at the bug itself, should glimagesink check the physical display size before accepting to create the context ? Does it make sense to have a gles context that is bigger than the display size (on other plaforms for instance) ?
"gles context that is bigger than the display size" - what does this mean? It is possible and sometimes desirable to have textures that are bigger than the display size if that's what you mean? Artificially limiting them is not workable. 0x300c == EGL_BAD_PARAMETER.
(In reply to Matthew Waters (ystreet00) from comment #11) > "gles context that is bigger than the display size" - what does this mean? I meant a pi plugged into a 1280x800 hdmi monitor; when booting, the GPU initializes to 1280x800 (because of the edid); the pipeline Philippe described automatically negociates to the source default resolution, which is 1920x1080, and glimagesink apparently fails to create the context in these conditions. @philippe, what is the resolution of the monitor in your case ? I'm still to find a way to simulate this, and i'm not entirely certain it is the problem because setting the pi hdmi_mode to low resolution (640x480) does not reproduce the issue. It could also be a GPU memory limitation (since omx and rpicamsrc are both using the gpu). I'll try to find more about the issue conditions
Okay, it is related to gpu memory indeed; setting gpu_mem to 256 solves it, nothing related to display resolution after all (sorry). Lowering the resolution avoided the gpu memory exhaustion. Is there any way to read the gpu memory/usage ? 3 buffers of 1920x1080 RGBA are 190 MB, which is larger than the free memory available with gpu_mem set to 128. With gst 1.4.4 (raspbian): pi@rpicam:~ $ /opt/vc/bin/vcgencmd get_mem gpu gpu=128M pi@rpicam:~ $ sudo LD_LIBRARY_PATH=/opt/vc/lib /opt/vc/bin/vcdbg reloc | grep 'free' free list at 0x1ea05d20 106M free memory in 1 free block(s) largest free block is 106M bytes 0x18000000: free 106M pi@rpicam:~ $ gst-launch-1.0 rpicamsrc preview=0 fullscreen=0 ! queue ! h264parse ! omxh264dec ! glimagesink sync=false Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... Got context from element 'glimagesink0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0"; Setting pipeline to PLAYING ... New clock: GstSystemClock eglCreateImageKHR: failed to create image for buffer 0x4 target 12465 error 0x300c pi@rpicam:~ $ /opt/vc/bin/vcgencmd get_mem gpu gpu=256M pi@rpicam:~ $ sudo LD_LIBRARY_PATH=/opt/vc/lib /opt/vc/bin/vcdbg reloc | grep 'free' free list at 0x1d088c20 208M free memory in 3 free block(s) largest free block is 171M bytes 0x10000000: free 234M 0x1ea05d40: free 576 pi@rpicam:~ $ gst-launch-1.0 rpicamsrc preview=0 fullscreen=0 ! queue ! h264parse ! omxh264dec ! glimagesink sync=false Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... Got context from element 'glimagesink0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0"; Setting pipeline to PLAYING ... New clock: GstSystemClock ^Chandling interrupt. Interrupt: Stopping pipeline ... Execution ended after 0:00:04.404496325 Interestingly, the error is not the same with 1.6.2 (arch): [alarm@alarmpi ~]$ /opt/vc/bin/vcgencmd get_mem gpu gpu=256M [alarm@alarmpi ~]$ gst-launch-1.0 rpicamsrc preview=0 fullscreen=0 ! queue ! h264parse ! omxh264dec ! glimagesink sync=false [alarm@alarmpi ~]$ /opt/vc/bin/vcgencmd get_mem gpu gpu=128M [alarm@alarmpi ~]$ gst-launch-1.0 rpicamsrc preview=0 fullscreen=0 ! queue ! h264parse ! omxh264dec ! glimagesink sync=false Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... Got context from element 'sink': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0"; Setting pipeline to PLAYING ... New clock: GstSystemClock ERROR: from element /GstPipeline:pipeline0/GstGLImageSinkBin:glimagesinkbin0/GstGLColorConvertElement:glcolorconvertelement0: Failed to convert video buffer Additional debug info: gstglcolorconvertelement.c(219): gst_gl_color_convert_element_prepare_output_buffer (): /GstPipeline:pipeline0/GstGLImageSinkBin:glimagesinkbin0/GstGLColorConvertElement:glcolorconvertelement0 Execution ended after 0:00:00.825575899 Setting pipeline to PAUSED ... Setting pipeline to READY ... Setting pipeline to NULL ... Freeing pipeline ...
(In reply to Florent Thiery from comment #13) > Okay, it is related to gpu memory indeed; setting gpu_mem to 256 solves it, > nothing related to display resolution after all (sorry). Lowering the > resolution avoided the gpu memory exhaustion. Is there any way to read the > gpu memory/usage ? > > 3 buffers of 1920x1080 RGBA are 190 MB, which is larger than the free memory > available with gpu_mem set to 128. That's huge! We should avoid using useless Frame Buffer Objects in the glimagesink bin: gluploadelement ! glcolorconvert ! glimagesinkelement (Indeed a FBO by itself uses memory in addition to the pool of textures/eglimages) This could help: https://bugzilla.gnome.org/show_bug.cgi?id=730927 . Thierry could you check if glcolorconvert is really set to passthrough mode ? At least this one should not do any conversion. Also please check it does not create any FBO, even if it is put in passthrough. You can hack glimageisink(bin) by replacing glcolorconvert with identity. You can look for theses traces: GST_TRACE ("creating FBO dimensions:%ux%u", width, height); GST_LOG ("copying memory %p, tex %u into texture %i", src, src->tex_id, tex_id); GST_INFO ("Context, EXT_framebuffer_object supported: yes"); (I searched where gl->GenFramebuffers is called)
Note, I never seem video playback working at 128 mb on the Pi, shall we close this bug ?
(In reply to Nicolas Dufresne (stormer) from comment #15) > Note, I never seem video playback working at 128 mb on the Pi, shall we > close this bug ? gst-launch-1.0 videotestsrc pattern=2 ! video/x-raw\,\ format\=\(string\)YUY2\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)30/1 ! fakesink sync=false silent=false -v Well, it does work, quite slowly at 1080p (~3.2 fps), but it does. What do you mean by "working" ? ;)
(In reply to Florent Thiéry from comment #12) > (In reply to Matthew Waters (ystreet00) from comment #11) > > "gles context that is bigger than the display size" - what does this mean? > > I meant a pi plugged into a 1280x800 hdmi monitor; when booting, the GPU > initializes to 1280x800 (because of the edid); the pipeline Philippe > described automatically negociates to the source default resolution, which > is 1920x1080, and glimagesink apparently fails to create the context in > these conditions. > > @philippe, what is the resolution of the monitor in your case ? 1080p But with my gpu_mem set to 100MB I can only get 1280x800. As you state, raising gpu_mem allows to reach 1920x1080 but it's rather slow here.
(In reply to Julien Isorce from comment #14) > That's huge! We should avoid using useless Frame Buffer Objects in the > glimagesink bin: gluploadelement ! glcolorconvert ! glimagesinkelement > (Indeed a FBO by itself uses memory in addition to the pool of > textures/eglimages) After checking, the large memory amount does not seem to be allocated by gstgl alone (at least not in 1.6.2), because with a small amount of reserved memory (gpu_mem=64) i can display upto 2049x1000 YUY2 (glvideoconvert used) or 10000x1000 RGBA (glvideoconvert apparently in passthrough). GST_DEBUG=glmemory*:5,3 gst-launch-1.0 videotestsrc pattern=2 num-buffers=1 ! video/x-raw\,\ format\=\(string\)YUY2\,\ width\=\(int\)2049\,\ height\=\(int\)1000 ! glimagesink 0:00:00.557045000 327 0xb32f20 DEBUG glmemory gstglmemory.c:669:_gl_mem_init: new GL texture context:<glcontextegl0> memory:0xb29670 format:1 dimensions:2049x1000 stride:4100 size:4100000 0:00:00.855408000 327 0xaff780 FIXME glshader gstglshader.c:543:gst_gl_shader_compile: vertex shader info log: 0:00:00.879219000 327 0xaff780 ERROR default gstglutils.c:79:gst_gl_context_check_framebuffer_status: GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT 0:00:00.881039000 327 0xaff780 WARN default gstglutils.c:426:gst_gl_context_set_error: GL framebuffer status incomplete 0:00:00.882494000 327 0xb32f20 WARN glconvertelement gstglcolorconvertelement.c:219:gst_gl_color_convert_element_prepare_output_buffer:<glcolorconvertelement0> error: Failed to convert video buffer I can make it crash also as RGBA, but the error is not the same: [root@rpitest ~]# GST_DEBUG=glmemory*:5,3 gst-launch-1.0 videotestsrc pattern=2 num-buffers=1 ! video/x-raw\,\ format\=\(string\)RGBA\,\ width\=\(int\)11000\,\ height\=\(int\)10000 ! glimagesink 0:00:00.546419000 300 0xc35f20 DEBUG glmemory gstglmemory.c:669:_gl_mem_init: new GL texture context:<glcontextegl0> memory:0xc2c670 format:4 dimensions:11000x10000 stride:44000 size:440000000 0:00:00.548851000 300 0xc35f20 WARN glbufferpool gstglbufferpool.c:300:gst_gl_buffer_pool_alloc:<glbufferpool1> Could not create GL Memory 0:00:00.550773000 300 0xc35f20 WARN bufferpool gstbufferpool.c:300:do_alloc_buffer:<glbufferpool1> alloc function failed 0:00:00.552435000 300 0xc35f20 WARN bufferpool gstbufferpool.c:333:default_start:<glbufferpool1> failed to allocate buffer 0:00:00.554225000 300 0xc35f20 ERROR bufferpool gstbufferpool.c:531:gst_buffer_pool_set_active: What happens with philippe's pipeline is that too many elements in the pipeline rely on the GPU (rpicamsrc, maybe omx), which results in the gpu memory being exhausted quite quickly. The culprit are most probably within rpicamsrc or omxh264dec. > This could help: https://bugzilla.gnome.org/show_bug.cgi?id=730927 . > > Thierry could you check if glcolorconvert is really set to passthrough mode ? At least this one should not do any conversion. Also please check it does I think that my RGBA test checks that, otherwise it would crash with a smaller buffer. > not create any FBO, even if it is put in passthrough. You can hack > glimageisink(bin) by replacing glcolorconvert with identity. > > You can look for theses traces: > GST_TRACE ("creating FBO dimensions:%ux%u", width, height); Indeed, no FBO is created with the RGBA case; 1 is created with the YUY2 case > GST_LOG ("copying memory %p, tex %u into texture %i", src, src->tex_id, > tex_id); > GST_INFO ("Context, EXT_framebuffer_object supported: yes"); > (I searched where gl->GenFramebuffers is called)
(In reply to Nicolas Dufresne (stormer) from comment #15) > Note, I never seem video playback working at 128 mb on the Pi, shall we > close this bug ? It rings some bells, I remember that it required to set 256 mb for the gpu. The default (128) was not working for 1080p but I have been always surprised of this. (In reply to Florent Thiéry from comment #16) > gst-launch-1.0 videotestsrc pattern=2 ! video/x-raw\,\ > format\=\(string\)YUY2\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ > framerate\=\(fraction\)30/1 ! fakesink sync=false silent=false -v > > Well, it does work, quite slowly at 1080p (~3.2 fps), but it does. What do > you mean by "working" ? ;) You get 3 fps with this above pipeline ? That's due do videotestsrc not using SIMD through orc since by default it is disabled on arm, see http://cgit.freedesktop.org/gstreamer/orc/tree/configure.ac#n169 . The distro you are using would have to enable it manually. (It was enabled there https://git.collabora.com/cgit/raspberry-pi/ sometime ago but I could not find it anymore) I remember I could have +25fps with videotestsrc ! ximagesink for 720p when liborc is enabled on arm. Same with omxh264dec ! ximagesink with 720p with gst-omx from https://git.collabora.com/cgit/raspberry-pi/gst-omx.git/. (In reply to Florent Thiéry from comment #18) > (In reply to Julien Isorce from comment #14) > > That's huge! We should avoid using useless Frame Buffer Objects in the > > glimagesink bin: gluploadelement ! glcolorconvert ! glimagesinkelement > > (Indeed a FBO by itself uses memory in addition to the pool of > > textures/eglimages) > > After checking, the large memory amount does not seem to be allocated by > gstgl alone (at least not in 1.6.2), because with a small amount of reserved > memory (gpu_mem=64) i can display upto 2049x1000 YUY2 (glvideoconvert used) > or 10000x1000 RGBA (glvideoconvert apparently in passthrough). omxh264dec will also use a lot of gpu mem since this mem is used by the HW decoder. How much gpu mem is used if you only do omxh264dec ! .. ! ximagesink (no gl) with 1080p ?
> It rings some bells, I remember that it required to set 256 mb for the gpu. > The default (128) was not working for 1080p but I have been always surprised > of this. Default is 64 (http://elinux.org/RPiconfig#Memory) > (In reply to Florent Thiéry from comment #16) > > gst-launch-1.0 videotestsrc pattern=2 ! video/x-raw\,\ > > format\=\(string\)YUY2\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ > > framerate\=\(fraction\)30/1 ! fakesink sync=false silent=false -v > > > > Well, it does work, quite slowly at 1080p (~3.2 fps), but it does. What do > > you mean by "working" ? ;) > > You get 3 fps with this above pipeline ? That's due do videotestsrc not > using SIMD through orc since by default it is disabled on arm, see > http://cgit.freedesktop.org/gstreamer/orc/tree/configure.ac#n169 Oh, i thought i checked that videotestsrc pattern=2 was not the bottleneck, but you're right. I did the tests again and here's my approach, please let me know if it sounds valid : - timing the buffer generation alone using gst-launch-1.0 videotestsrc num-buffers=100 pattern=2 ! video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)1920\,\ height\=\(int\)1080 ! fakesink sync=false i measure 2.57 fps - because of that i dump into a gdp file: gst-launch-1.0 videotestsrc num-buffers=30 pattern=2 ! video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)1920\,\ height\=\(int\)1080 ! gdppay ! filesink location=/tmp/30buf.gdp (takes 12.4s) - i ensure the file is loaded in the fs cache gst-launch-1.0 filesrc location=/tmp/30buf.gdp ! fakesink sync=false gives out 2.5s execution time - last, i run gst-launch-1.0 filesrc location=/tmp/30buf.gdp ! gdpdepay ! fpsdisplaysink text-overlay=false sync=false -v, which took 5.2s and gives 5.58 average fps Here are my results (with gpu_mem=64 -- i checked that changing gpu_mem does not impact performance at all), what am i missing here (i'm pretty far from 25 fps) ? NB: this is a pi B+, not a pi 2 Resolution Colorspace Average fps 640x480 I420 24,31 640x480 YUY2 23,7 640x480 RGBA 16,82 1280x720 I420 9,87 1280x720 YUY2 8,72 1280x720 RGBA 5,79 1920x1080 I420 5,58 1920x1080 YUY2 4,58 1920x1080 RGBA 2,65 > The distro you are using would have to enable it manually. (It was enabled there > https://git.collabora.com/cgit/raspberry-pi/ sometime ago but I could not > find it anymore) > I remember I could have +25fps with videotestsrc ! ximagesink for 720p when > liborc is enabled on arm. Same with omxh264dec ! ximagesink with 720p with > gst-omx from https://git.collabora.com/cgit/raspberry-pi/gst-omx.git/. Ok. Do you think my approach with working from a gdp dump is a good way of benchmarking (and doesn't rely on videotestsrc performance) ? > omxh264dec will also use a lot of gpu mem since this mem is used by the HW > decoder. > How much gpu mem is used if you only do omxh264dec ! .. ! ximagesink (no gl) > with 1080p ? During gst-launch-1.0 filesrc location=/tmp/30buf.gdp ! gdpdepay ! omxh264enc ! fakesink i get 24 MB of used GPU memory (using sudo LD_LIBRARY_PATH=/opt/vc/lib /opt/vc/bin/vcdbg reloc | grep 'free')
Getting back at the precise issue this ticket is about, GPU memory-wise, for 1080p: - rpicamsrc takes 43 MB - omxh264dec takes 28 MB - omxh264enc takes 23 MB - glimagesink takes 59 MB for I420, 46 MB for RGBA, 61 MB for YUY2 Which means that for philippe's pipeline, we have 130 MB of required GPU memory, hence over 128 MB. It's possible to avoid it by raising the amount of gpu_mem to e.g. 256. Before closing this issue, isn't it possible to detect that the problem is in fact a memory error "gst_gl_buffer_pool_alloc Could not create GL Memory" and not "gst_gl_context_set_error: GL framebuffer status incomplete", which is less explanatory ?
(In reply to Florent Thiéry from comment #21) > Getting back at the precise issue this ticket is about, GPU memory-wise, for > 1080p: > - rpicamsrc takes 43 MB > - omxh264dec takes 28 MB > - omxh264enc takes 23 MB > - glimagesink takes 59 MB for I420, 46 MB for RGBA, 61 MB for YUY2 > > Which means that for philippe's pipeline, we have 130 MB of required GPU > memory, hence over 128 MB. It's possible to avoid it by raising the amount > of gpu_mem to e.g. 256. Great analysis. Worth to mention that for RGBA, 1920*1080*4 is around 8 MB. So if glimagesink uses 1 FBO and a pool a size 2, it should be around 24MB. So it sounds to me that glimagesink(bin) is using 2 FBO and more textures and or/eglimage. At least filesrc ! qtdemux ! h264parse ! omxh264dec ! glimagesink should work for 1080p and 128 MB. I also think rpicamsrc ! omxh264enc should work for 1080p and 128 MB of gpu mem. > > Before closing this issue, isn't it possible to detect that the problem is > in fact a memory error "gst_gl_buffer_pool_alloc Could not create GL Memory" > and not "gst_gl_context_set_error: GL framebuffer status incomplete", which > is less explanatory ? Good idea!. Maybe we could catch the gl error "GL_OUT_OF_MEMORY" with glGetError. At least in Mesa it seems to be set, see http://cgit.freedesktop.org/mesa/mesa/tree/src/mesa/main/fbobject.c#n2636 (search for GL_OUT_OF_MEMORY)
(In reply to Julien Isorce from comment #22) > (In reply to Florent Thiéry from comment #21) > Great analysis. > Worth to mention that for RGBA, 1920*1080*4 is around 8 MB. So if > glimagesink uses 1 FBO and a pool a size 2, it should be around 24MB. So it > sounds to me that glimagesink(bin) is using 2 FBO and more textures and > or/eglimage. You're missing the triple back buffer which is effectively another 3 frames (plus possibly depth/stencil buffers). Plus it's more like 3 GL buffers that are used at startup. > Good idea!. Maybe we could catch the gl error "GL_OUT_OF_MEMORY" with > glGetError. At least in Mesa it seems to be set, see > http://cgit.freedesktop.org/mesa/mesa/tree/src/mesa/main/fbobject.c#n2636 > (search for GL_OUT_OF_MEMORY) glGetError() implies a synchronisation which significantly kills performance.
(In reply to Matthew Waters (ystreet00) from comment #23) > (In reply to Julien Isorce from comment #22) > > (In reply to Florent Thiéry from comment #21) > > Great analysis. > > Worth to mention that for RGBA, 1920*1080*4 is around 8 MB. So if > > glimagesink uses 1 FBO and a pool a size 2, it should be around 24MB. So it > > sounds to me that glimagesink(bin) is using 2 FBO and more textures and > > or/eglimage. > > You're missing the triple back buffer which is effectively another 3 frames > (plus possibly depth/stencil buffers). > > Plus it's more like 3 GL buffers that are used at startup. Ah right, make sense :) (that's why RGB565 would be useful for the whole chain) > > > Good idea!. Maybe we could catch the gl error "GL_OUT_OF_MEMORY" with > > glGetError. At least in Mesa it seems to be set, see > > http://cgit.freedesktop.org/mesa/mesa/tree/src/mesa/main/fbobject.c#n2636 > > (search for GL_OUT_OF_MEMORY) > > glGetError() implies a synchronisation which significantly kills performance. Can we try to call "glGetError()" only if GL framebuffer status is detected incomplete ?
Even getting the framebuffer status is fishy for performance.
(In reply to Matthew Waters (ystreet00) from comment #25) > Even getting the framebuffer status is fishy for performance. Well, in that case the "gst_gl_context_set_error: GL framebuffer status incomplete" error is critical and shuts down the pipeline; are there cases when this error is non-critical (hence, why care of the performance impact)?
Checking framebuffer completeness is only slightly more developer orientated than glGetError() and falls into the same category of usage. Retrieving state from GL is orders of magnitude slower than sending state.
This is a system configuration issue with not having enough gpu_mem available. Closing