Sounds like a good idea.

Someone wants to write a patch for this? :)

Created attachment 317314 [details] [review]
build a QML extension plugin exposing GstGLVideoItem

Created attachment 317317 [details] [review]
tests: fix warning in qml example

There seemed to be an old reference to a no longer existing element. Not sure how this was intended to look like initially.

Review of attachment 317314 [details] [review]:

This looks good.

::: ext/qt/Makefile.am
@@ +88,3 @@
+
+libglvideoitemplugin_la_LDFLAGS =  -module -avoid-version
+libglvideoitemplugin_la_LIBTOOLFLAGS = --tag=disable-static

This probably isn't going to work very well on android and iOS which require static plugins.  I'm not sure what the process is there for extension plugins.

> This probably isn't going to work very well on android and iOS which require
> static plugins.  I'm not sure what the process is there for extension
> plugins.

Do gstreamer builds for android require that?

For a 'normal' Qt application targeting Android platform and using 3rd party qml plugins imho the process is

1) Build the plugin with the Qt build-tools for the android_armv7 platform.
2) Package the plugin as ANDROID_EXTRA_PLUGIN (qmake wording)
3) Buildit

Looking at how we package e.g. the qml-box2d plugin in GCompris-qt for android (https://quickgit.kde.org/?p=gcompris.git&a=blob&h=731d968737e7f7ba15cf988be1b29caf46dd01c8&hb=826d26b0c89070a47065790b887de4c44b9cbd03&f=CMakeLists.txt#l321) I find linker flags like the following generated for building this 3rd party module:

LFLAGS        = --sysroot=/opt/android-ndk//platforms/android-9/arch-arm/ -Wl,-rpath=/mnt/ltmp/Qt5.5.1-android/5.5/android_armv7/lib -Wl,--no-undefined -Wl,-z,noexecstack -shared

Therefore from a Qt PoV I don't think it's necessary to link the *qml* plugin statically on Android. And the gstreamer-part only needs to cope with the *gst* plugin, no?

I have no idea of iOS, though.

If we'd really to link the plugin statically, afaik the registering of the qml type probably needs to be done manually on app-layer (that's what I see other plugins doing then).

Review of attachment 317314 [details] [review]:

::: ext/qt/Makefile.am
@@ +88,3 @@
+
+libglvideoitemplugin_la_LDFLAGS =  -module -avoid-version
+libglvideoitemplugin_la_LIBTOOLFLAGS = --tag=disable-static

So on Android and iOS Qt is using shared libraries as plugins?

::: ext/qt/glvideoitemplugin.cc
@@ +31,3 @@
+{
+  Q_ASSERT(uri == QLatin1String("org.freedesktop.gstreamer.GLVideoItem"));
+  qmlRegisterType<QtGLVideoItem> (uri, 1, 0, "GstGLVideoItem");

The easiest on those platforms would probably be to statically link the GStreamer plugin into the Qt plugin shared library, and then initialize the GStreamer plugin from here. However that would then work different as with the other GStreamer plugins on those platforms.

The alternative would be that the application still needs to initialize the GStreamer plugin before being able to use the QML plugin, which our build system stuff for these platforms is automatically doing. If you use that. That's what your changes now are doing AFAIU.

I had a first look at how the android build-env for gstreamer works and understand your concerns better -- not having found a single dir containing .so files :-)

If you want to keep that direction I'd try to add a way to build and initialize the qml-plugin statically, which corresponds to your 2nd alternative, slomo. We wouldn't have won much then compared to the status quo.

Again from the Qt perspective I'd keep the split between 

1. a statically linked libgstqtsink.a supposed to be linked like the rest of the gstreamer-on-android world
2. a dynamically linked qml plugin/glvideoitemplugin.so that could be used by a Qt application in the Qt way

What would be the use-case for GstGLVideoSink on android? The user wants to render a decoded video directly into this Qt-Item, i.e. integrate GStreamer-based decoding in a QML scene. Then he would probably build a Qt/QML application in the first place (using GStreamer under the hood) and use the Qt tools (qmake/cmake) to create an apk-bundle. In most of the cases (unless tools like ministro are used) the apk would be a standalone app. Then the glvideoitemplugin.so would perfectly well integrate into the Qt build-process and being packaged as the other qml plugins are. Also the gstreamer_android.so would need to be bundled into the Qt-app.

Don't understand enough about the internals of gstreamer apps on android to see how this would look like in detail.

Would be nice to have a testbed for that. Will have a look at how cerbero works ...

There not yet anything from -bad/ext/qt/ in the latest android tarballs, is it?

I think keeping this split as you suggested makes sense, that way it should work as expected by Qt and GStreamer at the same time.


The Qt plugin is not included in the Android binaries as special work is needed to make everything compileable. Matthew would know the details.
And also because Qt is a huge dependency. It would probably make sense to provide this as a separate binary package, also for the other platforms.

Android is easy and works out of the box if one compiles gst-plugins-bad against a Qt-Android build.  The iOS build requires special build magic to compile against an iOS Qt involving the modification of Qt's pkg-config and libtool files.

I tried to get a POC running, hacking around some issues I came across: Cross-built GStreamer for Android with adjusted cerbero recipes plus some hackish handwork linking together a shared qml-plugin from shared Qt- and static gstreamer-libs and modfied tests/examples/qt/qml/play.pro to make it build for Android. This was tricky and I'm no longer convinced my initial idea of building a shared qml module for android is a good one :-)

Some of my issues might be simply caused by my ignorance so let me sum them up and pls correct me where you know better:

- Cerbero: Did not find a way to inject 'foreign' PKG_CONFIG_PATH into the cerbero build-env when linking against installed Qt-for-android. Hacked it into the cerbero-code. Is there a way?

- autotools: I did not find a way to build both static *and* shared libs (for the qml-module) in one go during cross-building a gst-stack for android with cerbero. This might be due to my poor knowledge of autotools, though. This for now would be a showstopper for me building a shared qml plugin for android.

- cerbero: When manually resolving the dependencies for the qt-module from the .la file the tools.mk from cerbero produced a syntactically wrong linking command. Hacked around this in tools.mk. Once we agreed upon how to build the qml module and this is still an issue, I'd come up with a patch for that.

- gst-plugins-bad: iiuc the qtsink plugin is badly named atm. The .so file should probably be named as the plugin ("libgstqt.so" instead of "libgstqtsink.so") otherwise the ndk-build generates wrong code with the GST_PLUGIN_STATIC_DECLARE macros. Right?

- Having modfied the tests/examples/qt/qml/ to build for android, bundling in a gstreamer_android.so I finally ran into an Android error:

E/BufferQueue( 1855): [org.freedesktop.gstreamer.play/org.qtproject.qt5.android.bindings.QtActivity] dequeueBuffer: SurfaceTexture has been abandoned!
E/[EGL-ERROR]( 2729): void __egl_platform_dequeue_buffer(egl_surface*):1271: failed to dequeue buffer from native window (0x535dc6e8); err = -19, buf = 0x52262974
E/BufferQueue( 1855): [org.freedesktop.gstreamer.play/org.qtproject.qt5.android.bindings.QtActivity] setBufferCount: SurfaceTexture has been abandoned!
E/SurfaceTextureClient( 2729): ISurfaceTexture::setBufferCount(0) returned No such device

Not sure if this is a gst- or a qt/qml-problem. Have not yet been digging deeper ...

To sum up: Building the qml-module as a shared one for android seems to be hard work if possible at all. Then, on iOS from what I googled about it static linking is mandatory. Given that, I tend to agree with you that on (these) mobile platforms linking everything statically might be the best way to go. Ok?

Still my above problem needs to be solved: how to inject the Qt-dependcies in a cerbero-build. Unless you plan to build Qt *inside* the cerbero env -- would you!?? -- the pkg-config deps would need to be injected from outside the build-env. How do you plan to handle this?

(In reply to Holger Kaelberer from comment #11)
> - Cerbero: Did not find a way to inject 'foreign' PKG_CONFIG_PATH into the
> cerbero build-env when linking against installed Qt-for-android. Hacked it
> into the cerbero-code. Is there a way?

I guess the best here would be to have a Qt recipe that downloads the binaries from their website and then places them somewhere in the cerbero dist directory. Then pkg-config would directly find it.

Alternatively you could adjust the PKG_CONFIG_PATH from the gst-plugins-bad recipe, however that would require knowing somehow where Qt was placed manually before.

> - autotools: I did not find a way to build both static *and* shared libs
> (for the qml-module) in one go during cross-building a gst-stack for android
> with cerbero. This might be due to my poor knowledge of autotools, though.
> This for now would be a showstopper for me building a shared qml plugin for
> android.

That's why there is a gst-plugins-bad-1.0.recipe and gst-plugins-bad-1.0-static.recipe :) Building both at once is not trivial with autotools unfortunately.

> - cerbero: When manually resolving the dependencies for the qt-module from
> the .la file the tools.mk from cerbero produced a syntactically wrong
> linking command. Hacked around this in tools.mk. Once we agreed upon how to
> build the qml module and this is still an issue, I'd come up with a patch
> for that.

How was it syntactically wrong?

> - gst-plugins-bad: iiuc the qtsink plugin is badly named atm. The .so file
> should probably be named as the plugin ("libgstqt.so" instead of
> "libgstqtsink.so") otherwise the ndk-build generates wrong code with the
> GST_PLUGIN_STATIC_DECLARE macros. Right?

Yes

> - Having modfied the tests/examples/qt/qml/ to build for android, bundling
> in a gstreamer_android.so I finally ran into an Android error:
> 
> E/BufferQueue( 1855):
> [org.freedesktop.gstreamer.play/org.qtproject.qt5.android.bindings.
> QtActivity] dequeueBuffer: SurfaceTexture has been abandoned!
> E/[EGL-ERROR]( 2729): void __egl_platform_dequeue_buffer(egl_surface*):1271:
> failed to dequeue buffer from native window (0x535dc6e8); err = -19, buf =
> 0x52262974
> E/BufferQueue( 1855):
> [org.freedesktop.gstreamer.play/org.qtproject.qt5.android.bindings.
> QtActivity] setBufferCount: SurfaceTexture has been abandoned!
> E/SurfaceTextureClient( 2729): ISurfaceTexture::setBufferCount(0) returned
> No such device
> 
> Not sure if this is a gst- or a qt/qml-problem. Have not yet been digging
> deeper ...

I'd say that's a problem in our plugin. Can you create another bug for that?
 
> To sum up: Building the qml-module as a shared one for android seems to be
> hard work if possible at all. Then, on iOS from what I googled about it
> static linking is mandatory. Given that, I tend to agree with you that on
> (these) mobile platforms linking everything statically might be the best way
> to go. Ok?

What are Qt developers doing on Android usually? Dynamic or statically linked plugins? I'd say we should try (if possible) to do whatever developers on those platforms are used to.
Otherwise for a static plugin, we could probably initialize that during plugin_init() too for Qt?

> Still my above problem needs to be solved: how to inject the Qt-dependcies
> in a cerbero-build. Unless you plan to build Qt *inside* the cerbero env --
> would you!?? -- the pkg-config deps would need to be injected from outside
> the build-env. How do you plan to handle this?

See above :) Building Qt is also an option, but it's absolutely non-trivial on non-Linux platforms... or at least was 3 years ago. I'd just create a recipe with their binaries.

Qt should not be enabled by default though, it should be an variant.

(In reply to Sebastian Dröge (slomo) from comment #12)
> (In reply to Holger Kaelberer from comment #11)
> > - Cerbero: Did not find a way to inject 'foreign' PKG_CONFIG_PATH into the
> > cerbero build-env when linking against installed Qt-for-android. Hacked it
> > into the cerbero-code. Is there a way?
> 
> I guess the best here would be to have a Qt recipe that downloads the
> binaries from their website and then places them somewhere in the cerbero
> dist directory. Then pkg-config would directly find it.

From a GStreamer PoV probably the cleanest solution. Then the Qt developer needs to assure to use a Qt SDK that is compatible with the one used to build GStreamers qml plugin.

> 
> Alternatively you could adjust the PKG_CONFIG_PATH from the gst-plugins-bad
> recipe, however that would require knowing somehow where Qt was placed
> manually before.

Thanks, works now ;-)

> 
> > - autotools: I did not find a way to build both static *and* shared libs
> > (for the qml-module) in one go during cross-building a gst-stack for android
> > with cerbero. This might be due to my poor knowledge of autotools, though.
> > This for now would be a showstopper for me building a shared qml plugin for
> > android.
> 
> That's why there is a gst-plugins-bad-1.0.recipe and
> gst-plugins-bad-1.0-static.recipe :) Building both at once is not trivial
> with autotools unfortunately.

So you mean first cross build gst-plugins-bad-1.0-static.recipe and then into the same dist gst-plugins-bad-1.0.recipe?

> 
> > - cerbero: When manually resolving the dependencies for the qt-module from
> > the .la file the tools.mk from cerbero produced a syntactically wrong
> > linking command. Hacked around this in tools.mk. Once we agreed upon how to
> > build the qml module and this is still an issue, I'd come up with a patch
> > for that.
> 
> How was it syntactically wrong?

It appended something like "Qt5Gui Qt5Core ... " instead of "-lQt5Gui -lQt5Core ... "

> 
> > - gst-plugins-bad: iiuc the qtsink plugin is badly named atm. The .so file
> > should probably be named as the plugin ("libgstqt.so" instead of
> > "libgstqtsink.so") otherwise the ndk-build generates wrong code with the
> > GST_PLUGIN_STATIC_DECLARE macros. Right?
> 
> Yes
> 
> > - Having modfied the tests/examples/qt/qml/ to build for android, bundling
> > in a gstreamer_android.so I finally ran into an Android error:
> > 
> > E/BufferQueue( 1855):
> > [org.freedesktop.gstreamer.play/org.qtproject.qt5.android.bindings.
> > QtActivity] dequeueBuffer: SurfaceTexture has been abandoned!
> > E/[EGL-ERROR]( 2729): void __egl_platform_dequeue_buffer(egl_surface*):1271:
> > failed to dequeue buffer from native window (0x535dc6e8); err = -19, buf =
> > 0x52262974
> > E/BufferQueue( 1855):
> > [org.freedesktop.gstreamer.play/org.qtproject.qt5.android.bindings.
> > QtActivity] setBufferCount: SurfaceTexture has been abandoned!
> > E/SurfaceTextureClient( 2729): ISurfaceTexture::setBufferCount(0) returned
> > No such device
> > 
> > Not sure if this is a gst- or a qt/qml-problem. Have not yet been digging
> > deeper ...
> 
> I'd say that's a problem in our plugin. Can you create another bug for that?

Will do once there is a clean way to reproduce it.

>  
> > To sum up: Building the qml-module as a shared one for android seems to be
> > hard work if possible at all. Then, on iOS from what I googled about it
> > static linking is mandatory. Given that, I tend to agree with you that on
> > (these) mobile platforms linking everything statically might be the best way
> > to go. Ok?
> 
> What are Qt developers doing on Android usually?

The QML plugins shipped with Qt are bundled as shared libs.

> Dynamic or statically
> linked plugins? I'd say we should try (if possible) to do whatever
> developers on those platforms are used to.

I don't know how to link that using cerbero: Manually I linked all gstreamer dependcies statically and only the Qt ones dynamically, which is probably the best option. Otherwise, if you link everything dynamically you'd need to bundle all shared dependencies in your apk!

How would the first option be done in a cerbero build? The static libs would need to be available in the shared build. Would you first build a complete xxx-static dist, then, on top, a shared stack?

> Otherwise for a static plugin, we could probably initialize that during
> plugin_init() too for Qt?

That is how it was done so far.

(In reply to Holger Kaelberer from comment #13)
> 
> > Dynamic or statically
> > linked plugins? I'd say we should try (if possible) to do whatever
> > developers on those platforms are used to.
> 
> I don't know how to link that using cerbero: Manually I linked all gstreamer
> dependcies statically and only the Qt ones dynamically, which is probably
> the best option. Otherwise, if you link everything dynamically you'd need to
> bundle all shared dependencies in your apk!
> 
> How would the first option be done in a cerbero build? The static libs would
> need to be available in the shared build. Would you first build a complete
> xxx-static dist, then, on top, a shared stack?

We always build the shared and static recipes, first the shared ones and then the static ones. The shared ones are also providing the static *libraries*, while the static ones only provide the static *plugins*.

Created attachment 319606 [details] [review]
qt: Fix build for android

Created attachment 319607 [details] [review]
qt: build a QML extension plugin exposing GstGLVideoItem

Changes:

- Unifies gst plugin naming by renaming the .so file qtsink --> qt
- Now builds a static plugin for android

Created attachment 319608 [details] [review]
tests: fix warning in qml example

I tried building a shared qml plugin (in the gst-plugins-bad-1.0-static recipe), but libtool drove me crazy by ignoring my linker flags. Therefore updated the patches with a static variant for android (and possibly ios too).

This way one can use Q_IMPORT_PLUGIN in an application to initialize the plugin (cf. changes in main.cpp).

I can't test on iOS, but it should work as for Android, I think.

Some thoughts about plugin naming: ATM the plugin is more or less named as the type it exposes:

Plugin: org.freedesktop.gstreamer.GLVideoItem
Type: GstGLVideoItem

It could make sense to choose the name more general and thus allow to add other types to it. This of course depends on what you plan with respect to Qt-integration. To propose a few:

org.freedesktop.gstreamer.GstQt/GstQml/GStreamerQml (semantically completely open), 
org.freedesktop.gstreamer.GstVideo (supposed to contain video-related plugins)
<your-proposals>

Also, especially if you plan to add more gtreamer qml types, it would simplify adoption for Qt app developers to stay close to QtMultimedia's types and interfaces (http://doc.qt.io/qt-5/qtmultimedia-qmlmodule.html). 

Plugin: org.freedesktop.gstreamer.GstMultimedia
Type: GstVideoOutput/VideoOutput

... just some thoughts ;-)

commit 0ae665e35c54c10ed6246eb1bbe01f64ed87dadc
Author: Holger Kaelberer <holger.k@elberer.de>
Date:   Sun Jan 24 15:47:12 2016 +0100

    tests: fix warning in qml example
    
    https://bugzilla.gnome.org/show_bug.cgi?id=756082

commit 0f865b316c50662df1bca8c0933396da0eb3bbb4
Author: Holger Kaelberer <holger.k@elberer.de>
Date:   Sun Jan 24 15:42:32 2016 +0100

    qt: Fix build for android
    
    https://bugzilla.gnome.org/show_bug.cgi?id=756082

How should we proceed with the missing parts here?

Regarding cerbero integration, I don't have enough time to look into this atm, sorry.

-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/issues/228.