GNOME Bugzilla – Bug 788116
android: glimagesink/amcvideodec unable to retain and save last-sample
Last modified: 2018-11-03 14:14:03 UTC
I have a pipeline like this: decoder ! glupload ! glocolorconvert ! glimagesink to take a snapshot I get the last-sample from glimagesink and then save it as jpeg with a pipeline like this: appsrc ! gldownload ! videoconvert ! jpegenc ! filesink the snapshot can be generated even after the decoding pipeline is finalized, this should be ok since last-sample returns a reference to the sample and an user can keep this reference until needed this works fine with a sw decoder such as avdec_h264 but if I use an hw decoder such as amcvideodec and you try to generate a snapshot after the decoding pipeline is setted to NULL then the snapshot pipeline will hang after changing state from NULL to READY, probably after the pipeline is finalized gldownload cannot get a context or access to the glmemory. This seems a bug ot at least should be documented.
You have to share the GL context between both pipeline for this to work. Did you do so ?
(In reply to Nicolas Dufresne (stormer) from comment #1) > You have to share the GL context between both pipeline for this to work. Did > you do so ? Thanks Nicolas, I'm not doing anything to share the GL context, I'll try to understand how to share the GstGLContext that I can get from glimagesink with the one used in gldownload (is gst_gl_context_set_shared_with the right way?), anyway seems really strange that if I use avdec_* encoder all is fine while if I use amcvideodec I have the reported issue, in both case the sample I get from glimagesink has feature "memory:GLMemory" so I supposed that a different behaviour was a bug, maybe I'm wrong
I added the code to share the gl context to my app based on this: http://ystreet00.blogspot.it/2015/09/gstreamer-16-and-opengl-contexts.html and tested on linux, it works as with no context sharing unluckily after adding gstreamer-gl-1.0 deps my android project it does not compile anymore In file included from /home/nicola/gst-android/armv7/include/gstreamer-1.0/gst/gl/gstglapi.h:81:0, from /home/nicola/gst-android/armv7/include/gstreamer-1.0/gst/gl/gstgl_fwd.h:26, from /home/nicola/gst-android/armv7/include/gstreamer-1.0/gst/gl/gl.h:29, from /home/nicola/gst-android/armv7/include/gstreamer-1.0/gst/gl/gstglcontext.h:26, from /home/nicola/AndroidStudioProjects/MediaLibrary/app/src/main/jni/media-player.h:6, from /home/nicola/AndroidStudioProjects/MediaLibrary/app/src/main/jni/media-manager.c:3: /home/nicola/gst-android/armv7/include/gstreamer-1.0/gst/gl/glprototypes/gstgl_compat.h:40:18: error: conflicting types for 'GLsync' typedef gpointer GLsync; I'll retry with an older ndk or I'll download the sample to cpu before closing the pipeline as workaround
(In reply to Nicola from comment #0) > I have a pipeline like this: > > decoder ! glupload ! glocolorconvert ! glimagesink > > to take a snapshot I get the last-sample from glimagesink and then save it > as jpeg with a pipeline like this: > > appsrc ! gldownload ! videoconvert ! jpegenc ! filesink > > the snapshot can be generated even after the decoding pipeline is finalized, > this should be ok since last-sample returns a reference to the sample and an > user can keep this reference until needed > > this works fine with a sw decoder such as avdec_h264 but if I use an hw > decoder such as amcvideodec and you try to generate a snapshot after the > decoding pipeline is setted to NULL then the snapshot pipeline will hang > after changing state from NULL to READY, > > probably after the pipeline is finalized gldownload cannot get a context or > access to the glmemory. > > This seems a bug ot at least should be documented. This is problematic as android's decoders don't actually give us access to the actual buffer contents in zerocopy mode. We only hold one end of a queue that is updated into the same texture. This means that any GstBuffer reference effectively always points to the same texture and thus always contains the latest frame. That frame is entirely controlled by amc so whether it is available or not after destruction of the decoder is up to android. (In reply to Nicolas Dufresne (stormer) from comment #1) > You have to share the GL context between both pipeline for this to work. Did > you do so ? Yes, that is also true but there is also another snag with android that the external texture is only valid in a single GL context and must be released from all other GL contexts before it is usable in another GL context. I thought there was a bug already open about this however I can't find it. (In reply to Nicola from comment #2) > (In reply to Nicolas Dufresne (stormer) from comment #1) > > You have to share the GL context between both pipeline for this to work. Did > > you do so ? > > Thanks Nicolas, I'm not doing anything to share the GL context, > > I'll try to understand how to share the GstGLContext that I can get from > glimagesink with the one used in gldownload (is > gst_gl_context_set_shared_with the right way?), > > anyway seems really strange that if I use avdec_* encoder all is fine while > if I use amcvideodec I have the reported issue, > > in both case the sample I get from glimagesink has feature "memory:GLMemory" > so I supposed that a different behaviour was a bug, maybe I'm wrong "memory:GLMemory" is not enough information to go on. What are the full caps? There's a texture-target field in the caps which is extremely important as well. (In reply to Nicola from comment #3) > I added the code to share the gl context to my app based on this: > > http://ystreet00.blogspot.it/2015/09/gstreamer-16-and-opengl-contexts.html > > and tested on linux, it works as with no context sharing > > unluckily after adding gstreamer-gl-1.0 deps my android project it does not > compile anymore > > /home/nicola/gst-android/armv7/include/gstreamer-1.0/gst/gl/glprototypes/ > gstgl_compat.h:40:18: error: conflicting types for 'GLsync' > typedef gpointer GLsync; > > I'll retry with an older ndk or I'll download the sample to cpu before > closing the pipeline as workaround This means that you're building and app against a different android api than gstreamer was built with. This can be mitigated by modifying the defines in gstreamer/prefix/usr/include/gstreamer-1.0/gst/gl/gstglconfig.h to match you expected android api.
(In reply to Matthew Waters (ystreet00) from comment #4) > (In reply to Nicola from comment #0) > > I have a pipeline like this: > > > > decoder ! glupload ! glocolorconvert ! glimagesink > > > > to take a snapshot I get the last-sample from glimagesink and then save it > > as jpeg with a pipeline like this: > > > > appsrc ! gldownload ! videoconvert ! jpegenc ! filesink > > > > the snapshot can be generated even after the decoding pipeline is finalized, > > this should be ok since last-sample returns a reference to the sample and an > > user can keep this reference until needed > > > > this works fine with a sw decoder such as avdec_h264 but if I use an hw > > decoder such as amcvideodec and you try to generate a snapshot after the > > decoding pipeline is setted to NULL then the snapshot pipeline will hang > > after changing state from NULL to READY, > > > > probably after the pipeline is finalized gldownload cannot get a context or > > access to the glmemory. > > > > This seems a bug ot at least should be documented. > > This is problematic as android's decoders don't actually give us access to > the actual buffer contents in zerocopy mode. We only hold one end of a > queue that is updated into the same texture. This means that any GstBuffer > reference effectively always points to the same texture and thus always > contains the latest frame. That frame is entirely controlled by amc so > whether it is available or not after destruction of the decoder is up to > android. thanks for you explaination, indeed I was unable to find a workaround, I disabled this feature on android if hw decoding is used, while it works fine, for example, on iOS with hw decoding > > (In reply to Nicolas Dufresne (stormer) from comment #1) > > You have to share the GL context between both pipeline for this to work. Did > > you do so ? > > Yes, that is also true but there is also another snag with android that the > external texture is only valid in a single GL context and must be released > from all other GL contexts before it is usable in another GL context. > > I thought there was a bug already open about this however I can't find it. > > (In reply to Nicola from comment #2) > > (In reply to Nicolas Dufresne (stormer) from comment #1) > > > You have to share the GL context between both pipeline for this to work. Did > > > you do so ? > > > > Thanks Nicolas, I'm not doing anything to share the GL context, > > > > I'll try to understand how to share the GstGLContext that I can get from > > glimagesink with the one used in gldownload (is > > gst_gl_context_set_shared_with the right way?), > > > > anyway seems really strange that if I use avdec_* encoder all is fine while > > if I use amcvideodec I have the reported issue, > > > > in both case the sample I get from glimagesink has feature "memory:GLMemory" > > so I supposed that a different behaviour was a bug, maybe I'm wrong > > "memory:GLMemory" is not enough information to go on. What are the full > caps? There's a texture-target field in the caps which is extremely > important as well. the caps seems the same, for sw decoding I have: video/x-raw(memory:GLMemory), format=(string)RGBA, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1, texture-target=(string)2D while for hw decoding I have: video/x-raw(memory:GLMemory), width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1, format=(string)RGBA, texture-target=(string)2D > > (In reply to Nicola from comment #3) > > I added the code to share the gl context to my app based on this: > > > > http://ystreet00.blogspot.it/2015/09/gstreamer-16-and-opengl-contexts.html > > > > and tested on linux, it works as with no context sharing > > > > unluckily after adding gstreamer-gl-1.0 deps my android project it does not > > compile anymore > > > > /home/nicola/gst-android/armv7/include/gstreamer-1.0/gst/gl/glprototypes/ > > gstgl_compat.h:40:18: error: conflicting types for 'GLsync' > > typedef gpointer GLsync; > > > > I'll retry with an older ndk or I'll download the sample to cpu before > > closing the pipeline as workaround > > This means that you're building and app against a different android api than > gstreamer was built with. This can be mitigated by modifying the defines in > gstreamer/prefix/usr/include/gstreamer-1.0/gst/gl/gstglconfig.h to match you > expected android api. I confirm that sharing the context does not fix the issue
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/616.