After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 797316 - Implement fetch-bootstrap and bootstrap --offline
Implement fetch-bootstrap and bootstrap --offline
Status: RESOLVED FIXED
Product: GStreamer
Classification: Platform
Component: cerbero
unspecified
Other All
: Normal normal
: git master
Assigned To: GStreamer Maintainers
GStreamer Maintainers
Depends on:
Blocks:
 
 
Reported: 2018-10-22 00:14 UTC by Nirbheek Chauhan
Modified: 2018-10-22 15:18 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
cerbero: Split recipe tarball class into a base class (6.12 KB, patch)
2018-10-22 00:14 UTC, Nirbheek Chauhan
committed Details | Review
cerbero: Add new option --offline to bootstrap (6.88 KB, patch)
2018-10-22 00:14 UTC, Nirbheek Chauhan
committed Details | Review
cerbero: Add fetch() and extract() to bootstrapper (3.03 KB, patch)
2018-10-22 00:14 UTC, Nirbheek Chauhan
committed Details | Review
cerbero: Split all bootstrappers into fetch/extract/start (16.34 KB, patch)
2018-10-22 00:15 UTC, Nirbheek Chauhan
none Details | Review
cerbero: New command fetch-bootstrap (4.87 KB, patch)
2018-10-22 00:15 UTC, Nirbheek Chauhan
committed Details | Review
cerbero/windows: Don't install msys-wget (1.09 KB, patch)
2018-10-22 00:15 UTC, Nirbheek Chauhan
committed Details | Review
cerbero/hacks: Ensure that zipfile sets permission on extract (1.50 KB, patch)
2018-10-22 02:34 UTC, Nirbheek Chauhan
committed Details | Review
cerbero: Split all bootstrappers into fetch/extract/start (16.35 KB, patch)
2018-10-22 15:14 UTC, Nirbheek Chauhan
committed Details | Review

Description Nirbheek Chauhan 2018-10-22 00:14:39 UTC
Mirroring fetch-package and package --offline, this allows you to download all resources needed for bootstrap into local_sources, and then bootstrap without needing network access.

This also means that we now verify the checksums of all toolchain binaries and extra binaries we download.
Comment 1 Nirbheek Chauhan 2018-10-22 00:14:46 UTC
Created attachment 373984 [details] [review]
cerbero: Split recipe tarball class into a base class

We will reuse this base class for download tarballs during bootstrap.
Comment 2 Nirbheek Chauhan 2018-10-22 00:14:51 UTC
Created attachment 373985 [details] [review]
cerbero: Add new option --offline to bootstrap

It hasn't been hooked up everywhere yet, the next commits will do that
Comment 3 Nirbheek Chauhan 2018-10-22 00:14:56 UTC
Created attachment 373986 [details] [review]
cerbero: Add fetch() and extract() to bootstrapper

This separates each bootstrapper into three distinct stages:

1. Fetch network resources into self.config.local_sources
2. Extract/copy network resources into the correct location
3. Run arbitrary commands that do not need network access
Comment 4 Nirbheek Chauhan 2018-10-22 00:15:01 UTC
Created attachment 373987 [details] [review]
cerbero: Split all bootstrappers into fetch/extract/start

Now when you run bootstrap --offline, the bootstrappers will
not require network access.

All URLs downloaded in the fetch() step now point to locations with
fixed contents and are checksummed.

FIXME: On Windows mingw-get is still run which uses the network
Comment 5 Nirbheek Chauhan 2018-10-22 00:15:06 UTC
Created attachment 373988 [details] [review]
cerbero: New command fetch-bootstrap

This fetches all network resources required for running bootstrap,
including build-tools recipe sources, toolchain binaries, etc.
Comment 6 Nirbheek Chauhan 2018-10-22 00:15:10 UTC
Created attachment 373989 [details] [review]
cerbero/windows: Don't install msys-wget

We don't use this for downloading, and we don't want any recipes or
build systems to shell out to this and download anything from the
network. Everything needed for building must be downloaded in the
fetch stage.
Comment 7 Nirbheek Chauhan 2018-10-22 02:34:27 UTC
Created attachment 373990 [details] [review]
cerbero/hacks: Ensure that zipfile sets permission on extract

Fix a years-old bug in Python's zipfile module:
https://bugs.python.org/issue15795

Without this, the extracted Android NDK does not have executable
permissions on any binaries and is unusable.
Comment 8 Sebastian Dröge (slomo) 2018-10-22 10:17:52 UTC
Review of attachment 373987 [details] [review]:

::: cerbero/bootstrap/__init__.py
@@ +38,3 @@
+    'android-ndk-r16-darwin-x86_64.zip': '589a567c4fc84f5f7219cae56d199b63059033e08512b6d8132739c90ef9483d',
+    'android-ndk-r16-windows-x86_64.zip': 'fb4ce0bbcb927dc1bb507d3dbb1aef0f243e5184b3dd5d956a450bcf4b0808df',
+    'android-ndk-r16-windows-x86.zip': 'aaa52b9e8b283be1249376f7373704687a5b13106575f613378e6d44c663f782',

Shouldn't these be in the specific bootstrappers where also the URLs are defined?

::: cerbero/bootstrap/windows.py
@@ +42,3 @@
+
+# Extra binary dependencies
+GNOME_FTP = 'https://ftp.gnome.org/pub/gnome/binaries/win32/'

Shouldn't this be downloads.gnome.org?
Comment 9 Nirbheek Chauhan 2018-10-22 15:14:00 UTC
Created attachment 374004 [details] [review]
cerbero: Split all bootstrappers into fetch/extract/start

Now when you run bootstrap --offline, the bootstrappers will
not require network access.

All URLs downloaded in the fetch() step now point to locations with
fixed contents and are checksummed.

FIXME: On Windows mingw-get is still run which uses the network
Comment 10 Nirbheek Chauhan 2018-10-22 15:17:34 UTC
Thanks for the review!

Attachment 373984 [details] pushed as 5aa2303 - cerbero: Split recipe tarball class into a base class
Attachment 373985 [details] pushed as ded0eb5 - cerbero: Add new option --offline to bootstrap
Attachment 373986 [details] pushed as 63e1b6d - cerbero: Add fetch() and extract() to bootstrapper
Attachment 373988 [details] pushed as 55cba13 - cerbero: New command fetch-bootstrap
Attachment 373989 [details] pushed as 968343e - cerbero/windows: Don't install msys-wget
Attachment 373990 [details] pushed as 75b669e - cerbero/hacks: Ensure that zipfile sets permission on extract
Attachment 374004 [details] pushed as 3996812 - cerbero: Split all bootstrappers into fetch/extract/start