GNOME Bugzilla – Bug 796534
Wrapped subproject gl-headers breaks Continuous
Last modified: 2018-06-19 10:04:56 UTC
The gl-headers subproject is not a Git sub-module, but it's left to Meson to download; see commit: https://cgit.freedesktop.org/gstreamer/gst-plugins-base/commit/?id=82c43ff9a32f1c1d113cd6e1e8d046b7ab64422d (reverted in master) This requires that the build happens in an environment with network connectivity; Continuous, like Flatpak or Buildstream, downloads modules first, then builds them inside a sandbox with no network connection, for security and reproducibility reasons. If gl-headers were a Git sub-module, build environments like Continuous would be able to download the project beforehand. Additionally, the subproject is cloned regardless of whether it's going to be used. It would be better to check if the GL headers are available, and only then use a subproject with them.
I think we should avoid unconditional/required submodules, and I also think that we should avoid git submodules. I would like us to move away from git submodules eventually. I don't know if we can easily make this conditional as an optional fallback in this case. Will have to discuss with Matt. Alternatively, I wonder if we could just drop the subproject and put the headers underneath gst-libs/gst/gl/ directly instead.
(In reply to Tim-Philipp Müller from comment #1) > I think we should avoid unconditional/required submodules, and I also think > that we should avoid git submodules. I would like us to move away from git > submodules eventually. > Moving away from git submodules requires implementing a meson feature that will download wraps ahead of time. > I don't know if we can easily make this conditional as an optional fallback > in this case. Will have to discuss with Matt. > We could make it windows-only, which would solve this. > Alternatively, I wonder if we could just drop the subproject and put the > headers underneath gst-libs/gst/gl/ directly instead. We discussed this earlier, and a subproject is better because then we don't have to duplicate it between -base and gst-build.
(In reply to Nirbheek Chauhan from comment #2) > (In reply to Tim-Philipp Müller from comment #1) > > I think we should avoid unconditional/required submodules, and I also think > > that we should avoid git submodules. I would like us to move away from git > > submodules eventually. > > > > Moving away from git submodules requires implementing a meson feature that > will download wraps ahead of time. This won't help Continuous. By the time we invoke Meson, we're already inside a sandbox, because the version of Meson we use is part of that sandbox.
(In reply to Emmanuele Bassi (:ebassi) from comment #3) > (In reply to Nirbheek Chauhan from comment #2) > > (In reply to Tim-Philipp Müller from comment #1) > > > I think we should avoid unconditional/required submodules, and I also think > > > that we should avoid git submodules. I would like us to move away from git > > > submodules eventually. > > > > > > > Moving away from git submodules requires implementing a meson feature that > > will download wraps ahead of time. > > This won't help Continuous. > > By the time we invoke Meson, we're already inside a sandbox, because the > version of Meson we use is part of that sandbox. What part of Continuous does the git submodule update? Why can't that also run `meson wrap --download-all` or whatever?
> > Alternatively, I wonder if we could just drop the subproject and put the > > headers underneath gst-libs/gst/gl/ directly instead. > > We discussed this earlier, and a subproject is better because then we don't > have to duplicate it between -base and gst-build. Not sure I understand. Duplicate what? My suggestion was to move the headers directly into the gst-plugins-base source code as a subdirectory.
(In reply to Nirbheek Chauhan from comment #4) > > By the time we invoke Meson, we're already inside a sandbox, because the > > version of Meson we use is part of that sandbox. > > What part of Continuous does the git submodule update? The same that runs `git pull`; if `git submodule status` returns a list of modules, it'll run `git submodule update` on any sub-module. This is similar to how flatpak-builder works. > Why can't that also > run `meson wrap --download-all` or whatever? Unlike Git, Meson is not installed on the host OS. Again, building modules in a sandbox assumes you only have the tool necessary to check out the source code installed on your host; anything else, from the compiler toolchain to the build system itself, is part of the SDK inside the sandbox.
This also broke gnome-build-meta.
A few options 1. Keep the thing and update builders to download meson subprojects before entering the build environment. 2. git submodule it up 3. embed the project into -base directly at the same location under subprojects/ (doesn't work where -base is already a subproject, a-la gst-build) 4. make subprojects optional in meson like dependency fallbacks so that if a subproject cannot be found/built, it can be skipped/ignored. Build people don't like 1 cause it means downloading extra code at configure time and doesn't really work with tarballs and building on a non-network-connected computer. 3. means duplicating the embedded copy in -base and any (meson) parent project which is suboptimal. 4. involves dealing with meson which really only leaves 2. as a simple viable option.
1 would require fixing all three of gnome-continuous, gnome-build-meta/buildstream, and flatpak-builder.
Having Khronos gl headers in gst-plugins-base/gst-libs/gst/gl is a good idea. The build system should try to use the gl headers provided by the system/platform first. And make sure our copy of gl headers is not installable.
1) As I see it we can't ever have it unconditional, that's just a no-go. If someone downloads a tarball they should be able to compile it without network access (without specifying extra options). 2) I would like to get rid of git submodules (esp. the common one), so this would at best be a temporary solution. Also, this would probably (didn't check) have the same probs as 1) if it's unconditional as I don't think meson will dist the submodule (but didn't check, as I said). 3) that sounds iffy, why can't we put it underneath gst-libs/gst/gl/gl-headers again? 4) I think making it optional is best. We could hide it behind an option that's off by default and has 'inherit: true' and then we can add the same option to gst-build and default to true there. (I hope that works anyway). It needs to be an opt-in IMHO.
Correction: I actually think just putting it under gst-libs/gst/gl/ is best and sync it up every now and then, but there may be reasons against that aiui, in which case (4) is next-best imo :)
Michael, Emmanuele: the builders use meson --wrap-mode=nodownload already right?
Um, I doubt it. Should be easy to change, though, at least for BuildStream and presumably Continuous. I'm not sure about flatpak-builder, since that one kinda does want to download stuff, but maybe that's fine for it to do, anyway.
Here's a commit that works if wrap-mode=nodownload is passed to meson :) commit cfa5d64ff1c1356b7a5b1c998aa9949f72d23a03 Author: Matthew Waters <matthew@centricular.com> Date: Mon Jun 18 22:24:13 2018 +1000 gl: use dependency fallbacks to get at a possible gl-headers subproject We would make the subproject conditional on the wrap-mode=nodownload however get_option('wrap-mode') or similar is not available from meson.build files as the wrap-mode is meant to be used automatically. Instead use the dependency fallback mechanism to get at the subproject where possible i.e. when downloading is allows and only add the compat includes when we have a valid internal dependency from the gl-headers subproject. https://bugzilla.gnome.org/show_bug.cgi?id=796534
It even seems to work for the 'git clone fails' case as well according to continuous.