GNOME Bugzilla – Bug 796953
Add option to enable shallow fetch optimization
Last modified: 2018-11-03 10:24:28 UTC
See commit, this is a proposal to add a shallow fetch optimization option to cerbero global configuration. This reduces fetch time both initially and when we check for updates.
Created attachment 373322 [details] [review] Add option to enable shallow fetch optimization When use_shallow_fetch=True is set in your cerbero.cbc file, cerbero will attempt to fetch only the requested tag or branch from the git server, without any history. This speeds up the fetch operation, though it only works well know references (branches and tags). Git protocol 2 might allow further optimization in the future.
Review of attachment 373322 [details] [review]: ::: cerbero/build/source.py @@ +215,3 @@ + commit = self.config.recipe_commit(self.name) or self.commit + if self.config.use_shallow_fetch and commit.startswith('origin/'): + git.fetch_ref(self.repo_dir, 'origin', commit[7:], fail=False) Why limit it to "origin/"? What's wrong with doing it for specific commits or tags (note: tags don't specify the remote!), or other remotes than origin? ::: recipes/taglib.recipe @@ -8,3 @@ version = '1.11.1' stype = SourceType.TARBALL - url = 'https://taglib.org/releases/taglib-%(version)s.tar.gz' This should be a separate commit
Oops, the taglib thing is totally accidental, I'll remove/split in next update. It's possible someone already fixed this taglib issue, I need to check. About the origin/, this is because git shallow is only supported on branches and tags (announced commit). This is a limitation of the git protocol 1, I haven't tested, but this should work with git protocol 2. Though it's too early, all the servers we use need to be updated.
I need to check for tags (not tested yet). In theory, because you can have multiple remotes in cerbero (an ancient use case from the original SDK), you could have clashing tags, which would lead to some ambiguity error. I've seen that once elsewhere, a bad release, and someone moved the tag / force push. But this is rare. I guess we need to improve a bit that change, since atm it means the shallow can only work on origin branches. Not that useful for CI after all.
(In reply to Nicolas Dufresne (ndufresne) from comment #4) > I need to check for tags (not tested yet). In theory, because you can have > multiple remotes in cerbero (an ancient use case from the original SDK), you > could have clashing tags, which would lead to some ambiguity error. > > I guess we need to improve a bit that change, since atm it means the shallow > can only work on origin branches. Not that useful for CI after all. A nice way to do this might be to have a function `self._ref_can_shallow` (or similar) that splits on '/' and checks whether the first part of that is `in self.remotes`. That should cover branches.
Agreed, but I'm starting to think I'll just drop that optimization for now and reconsider if the new git protocol can do what I really want it to do.
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/cerbero/issues/71.