GNOME Bugzilla – Bug 797316
Implement fetch-bootstrap and bootstrap --offline
Last modified: 2018-10-22 15:18:03 UTC
Mirroring fetch-package and package --offline, this allows you to download all resources needed for bootstrap into local_sources, and then bootstrap without needing network access. This also means that we now verify the checksums of all toolchain binaries and extra binaries we download.
Created attachment 373984 [details] [review] cerbero: Split recipe tarball class into a base class We will reuse this base class for download tarballs during bootstrap.
Created attachment 373985 [details] [review] cerbero: Add new option --offline to bootstrap It hasn't been hooked up everywhere yet, the next commits will do that
Created attachment 373986 [details] [review] cerbero: Add fetch() and extract() to bootstrapper This separates each bootstrapper into three distinct stages: 1. Fetch network resources into self.config.local_sources 2. Extract/copy network resources into the correct location 3. Run arbitrary commands that do not need network access
Created attachment 373987 [details] [review] cerbero: Split all bootstrappers into fetch/extract/start Now when you run bootstrap --offline, the bootstrappers will not require network access. All URLs downloaded in the fetch() step now point to locations with fixed contents and are checksummed. FIXME: On Windows mingw-get is still run which uses the network
Created attachment 373988 [details] [review] cerbero: New command fetch-bootstrap This fetches all network resources required for running bootstrap, including build-tools recipe sources, toolchain binaries, etc.
Created attachment 373989 [details] [review] cerbero/windows: Don't install msys-wget We don't use this for downloading, and we don't want any recipes or build systems to shell out to this and download anything from the network. Everything needed for building must be downloaded in the fetch stage.
Created attachment 373990 [details] [review] cerbero/hacks: Ensure that zipfile sets permission on extract Fix a years-old bug in Python's zipfile module: https://bugs.python.org/issue15795 Without this, the extracted Android NDK does not have executable permissions on any binaries and is unusable.
Review of attachment 373987 [details] [review]: ::: cerbero/bootstrap/__init__.py @@ +38,3 @@ + 'android-ndk-r16-darwin-x86_64.zip': '589a567c4fc84f5f7219cae56d199b63059033e08512b6d8132739c90ef9483d', + 'android-ndk-r16-windows-x86_64.zip': 'fb4ce0bbcb927dc1bb507d3dbb1aef0f243e5184b3dd5d956a450bcf4b0808df', + 'android-ndk-r16-windows-x86.zip': 'aaa52b9e8b283be1249376f7373704687a5b13106575f613378e6d44c663f782', Shouldn't these be in the specific bootstrappers where also the URLs are defined? ::: cerbero/bootstrap/windows.py @@ +42,3 @@ + +# Extra binary dependencies +GNOME_FTP = 'https://ftp.gnome.org/pub/gnome/binaries/win32/' Shouldn't this be downloads.gnome.org?
Created attachment 374004 [details] [review] cerbero: Split all bootstrappers into fetch/extract/start Now when you run bootstrap --offline, the bootstrappers will not require network access. All URLs downloaded in the fetch() step now point to locations with fixed contents and are checksummed. FIXME: On Windows mingw-get is still run which uses the network
Thanks for the review! Attachment 373984 [details] pushed as 5aa2303 - cerbero: Split recipe tarball class into a base class Attachment 373985 [details] pushed as ded0eb5 - cerbero: Add new option --offline to bootstrap Attachment 373986 [details] pushed as 63e1b6d - cerbero: Add fetch() and extract() to bootstrapper Attachment 373988 [details] pushed as 55cba13 - cerbero: New command fetch-bootstrap Attachment 373989 [details] pushed as 968343e - cerbero/windows: Don't install msys-wget Attachment 373990 [details] pushed as 75b669e - cerbero/hacks: Ensure that zipfile sets permission on extract Attachment 374004 [details] pushed as 3996812 - cerbero: Split all bootstrappers into fetch/extract/start