GNOME Bugzilla – Bug 747707
applemedia: expose GstCoreMediaMeta to app
Last modified: 2018-11-03 13:34:07 UTC
An OS X and iOS app might use appsink to get resulting GstBuffers from a pipeline containing applemedia elements. Furthermore, it can wish to access (and retain) the underlying CMSampleBufferRef or CMPixelBufferRef, as they're needed: 1) to work with AVFoundation, Core Video and Core Image 2) for GL rendering done by app How can we expose some of the applemedia stuff (e.g. coremediabuffer.h) as API?
We should first of all merge the two metas into one, and I think these also shouldn't be metas but instead custom GstMemory. Otherwise this is a really good idea, and we could add a libgstcoremedia library in gst-plugins-bad for that. It could also contain things like a GstClock based on the CoreMedia clock.
As discussed on IRC, GstCoreVideoMemory should not be the only kind of object retaining the Apple objects -- since: a) there are multiple GstCoreVideoMemory for planes -- which one you look into? b) when passing a CMSampleBuffer / CVPixelBuffer between supporting elements (determined via meta feature-caps), we can avoid populating the GstBuffer with GstCoreVideoMemory objects at all We might unify the metas, possibly keeping a variant of GstCoreMediaMeta whose sample_buf can be NULL?
avfassetsrc: * output: CMSampleBuffer -> CMBlockBuffer (if audio) -> CVImageBuffer (if video) avfvideosrc: * output: CMSampleBuffer -> CVImageBuffer vtenc: * input: CVImageBuffer * output: CMSampleBuffer vtdec: * input: CMSampleBuffer * output: CVImageBuffer avsamplevideosink: * input: CMSampleBuffer - - - CVImageBuffer, in addition to CVPixelBuffer, can also be CVOpenGLBuffer (Mac-only) and CVOpenGL(ES)Texture. Since, we don't populate GstBuffer with GstMemory when configured with "meta:GstCoreVideoBuffer" caps, we don't actually care -- we just pass the CVImageBufferRef to the framework.
Created attachment 302783 [details] [review] applemedia: refactor into library Preliminary patch moving GstCoreMediaMeta, GstCoreVideoMeta and GstCoreVideoMemory into libgstapplemedia-1.0. APPLYING: Applies on top of bug 747216, i.e. https://github.com/ikonst/gst-plugins-bad/tree/corevideomemory REMARKS: As said before, the rationale for naming it "applemedia" (and not "coremedia") is the significance of Apple technologies. The words "core" and "media" are just too generic. I've tried enabling it for GObject-Introspection, and that's something I'd like to do, but it wasn't trivial so it'll be a separate patch. I can't think of other parts of "applemedia" which should turn into "API".
Created attachment 302927 [details] [review] applemedia: refactor into library UPDATE: sys/applemedia/Makefile.am: remove headers from noinst_HEADERS
Created attachment 305436 [details] [review] [1/3] applemedia: rename Core(Video,Media)BufferMeta fields Unify field names between the two metas.
Created attachment 305437 [details] [review] [2/3] applemedia: add Apple to GstCore(Media,Video)BufferMeta Yes, this is a very noisy patch.
Created attachment 305438 [details] [review] [3/3] applemedia: refactor into library
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/238.