GNOME Bugzilla – Bug 538507
Jhbuild should be able to use bzr and git mirrors
Last modified: 2009-04-23 07:09:10 UTC
Repository URL's can be overridden so that users can check out as the user they plan to commit against. There is no way to specify an alternative repository type if you wanted to use bzr, git or hg instead of svn, though.
Created attachment 112801 [details] [review] One way we could do this SVN already has some special casing for when dealing with bzr and git (as i understand it, to automatically use bzr-svn and git-svn against svn.gnome.org). Here we want to utilize ready made mirrors, so I ignored this existing support. There is already a way to override the href property of a repository. I considered a way to let the config override the repository type. This would be global, and I don't want to force users to download the whole history of multiple projects when they might only be hacking on one or two. The attached patch reads <mirror> nodes from the moduleset XML. Extra repository objects are created for each mirror, and they are attached to the mirrors property of the parent "master" repository. With this in place, the patch adds mirrors to svn.gnome.org: <repository type="svn" name="svn.gnome.org" href="http://svn.gnome.org/svn/"> <mirror type="bzr" href="http://bzr-mirror.gnome.org/bzr/" /> <mirror type="git" href="http://git-mirror.gnome.org/git/" /> </repository> It adds two new configuration options, mirror_policy and module_mirror_policy. Better names are probably needed. Intended use is: mirror_policy = "git" module_mirror_policy["conduit"] = "bzr" Here, jhbuild will try to use Git mirrors if they are available, but for conduit it will prefer the Bazaar mirror. The patch needs to pass config into get_branch, which has been modified to check mirror policy and use a repository from .mirrors if it is appropriate. I also had to patch bzr.py. I'm not sure if it is the right patch to make, but without it the bzr code will sometimes try to checkout from the wrong part of bzr-mirror (e.g. from /bzr/$module instead of /bzr/$module/trunk. More work is to follow for Git support, but I need to get git-mirror working first :-) So here it is in its current state. Very much open to ideas on other ways of approaching this and suggestions for cleaning/simplifying the implementation.
Okay, I had started with the global alternative, so it was possible to do something like this: module_repositories = { 'svn.gnome.org': { 'type': 'bzr', 'href': 'http://bzr-mirror.gnome.org/', } } The thing is Git users (from what I could tell) would often like JHBuild to point to *their* mirror, not an official Git repository; modifying the moduleset is not an option for their usecase. So there is more thinking to do, probably going with your patch then adding the possibility to override the mirror for a specific module; or I could keep telling them to just create their own moduleset. Second issue you pointed it already, the Bzr support doesn't expect svn-like directory layout; and adding trunk_path is not enough, as the nice thing about jhbuild is also to have knowledge about the branch to checkout; perhaps the BzrRepository could simply have a "svn-like-directory-layout" option which would do the magic.
Created attachment 114598 [details] [review] support for a bzr playground Support for a bzr playground, just like described on http://live.gnome.org/Bazaar/DemoMachine It is somehow related to the bzr-mirror patch, they should be merged. To test: svn_program = 'bzr'; jhbuild updateone baobab
Created attachment 121964 [details] [review] Update my first patch to jhbuild svn Updated to apply cleanly to latest jhbuild. Only one (minor) conflict, and added mirrors to gnome 2.26 modulesets.
Comments on initial review: I can see benefit to a .jhbuildrc, but prefer putting it in .modules so everyone benefits from it easily. Re: svn-like switch, given how much variety svn in theory has i dont think an svn-like switch is the right answer.. (consider gnomemm). I think default-branch='' is a better name perhaps? My view is that we aren't infering anything about the structure of branches, just providing a default branch for if a fully specified location isnt provided... ? *goes to check*
Created attachment 121965 [details] [review] Updated to use git:// path for git-mirror
Frederic! Since it seems unlikely that we move to a DCVS anytime soon, it would be really great if you can review and commit this asap for you.
Created attachment 121999 [details] [review] Updated the .rnc and .dtd I think the updated .rnc and .dtd are OK, not sure how to test... Is this OK to commit, and then we'll discuss refinements (for example, when cloning git-mirror there are steps we can perform to prepare the repo so that its user can push back to svn. And we have the bzr playground stuff too).
Created attachment 122018 [details] [review] Fix git support for git-svn branches Was previously unable to checkout a branch if it was in git as origin/svn/branchname, which i think is the common case for git-svn stuff. So it now handles branches of svn.gnome.org stuff from git-mirror correctly.
Initial work has hit SVN. Note that it won't do anything smart if you have existing checkouts... To try it out, svn up your jhbuild and edit your .jhbuildrc to have mirror_policy='git' (Or 'bzr'. hg to follow.) Seems to work fine here, if you try it out a "works great here" would help me stay sane :P Work on bzr support for branches will follow. As will better support for local overriding ("i want to use my gittorious branch of x, y and z" use case).
Created attachment 122083 [details] [review] Fix git support for updating checkouts of branches zeenix pointed out a couple of problems: *** Checking out gnome-common *** [5/35] git fetch git stash save jhbuild-build No local changes to save git checkout gnome-2-24 error: pathspec 'gnome-2-24' did not match any file(s) known to git. *** error during stage checkout of gnome-common: ########## Error running git checkout gnome-2-24 *** [5/35] This can happen in the _update case because jhbuild just does "git checkout branch". However, if that branch doesnt exist locally it needs to do "git checkout -b branch origin/svn/branch". The patch adds a try except to do this (and uses self.branchname rather than hardcoding origin/svn/branch). *** Checking out gtk+ *** [18/38] git fetch git stash save jhbuild-build Saved working directory and index state "On gtk-2-14: jhbuild-build" HEAD is now at bb09f3f 2008-11-05 Richard Hult <richard@imendio.com> git checkout gtk-2-14 Already on "gtk-2-14" git rebase origin gtk-2-14 First, rewinding head to replay your work on top of it... Applying Plug a memory leak error: patch failed: modules/other/gail/ChangeLog:1 error: modules/other/gail/ChangeLog: patch does not apply error: patch failed: modules/other/gail/gailtreeview.c:3667 error: modules/other/gail/gailtreeview.c: patch does not apply Using index info to reconstruct a base tree... Falling back to patching base and 3-way merge... Auto-merged modules/other/gail/ChangeLog CONFLICT (content): Merge conflict in modules/other/gail/ChangeLog Failed to merge in the changes. Patch failed at 0001. When you have resolved this problem run "git rebase --continue". If you would prefer to skip this patch, instead run "git rebase --skip". To restore the original branch and stop rebasing run "git rebase --abort". *** error during stage checkout of gtk+: ########## Error running git rebase origin gtk-2-14 *** [18/38] This happens because jhbuild is doing "git rebase origin branch". I think the correct thing to do is "git rebase origin/svn/branch branch". Actually, branch is optional because we already have branch checked out. Attached patch uses self.branchname instead of hardcoding origin/svn/branch.
I can't review the git side of this; as for the proposed patch, I am confident you know what you do and you tested it, so please commit.
For people trying this out, if you override repos in your .jhbuildrc like: repos["svn.gnome.org"] = "svn+ssh://svn.gnome.org/svn/" that will break with mirror_policy, because jhbuild will end up trying to do git clone svn+ssh://svn.gnome.org/svn/glib or something. So just remove that line from your .jhbuildrc. For the developers, it would be nice if there was some way to get commitable-to checkouts, like you can with normal svn with the repos override above. According to http://live.gnome.org/GitForGnomeDevelopers, you just need to do: git svn init -s svn+ssh://$SVNUSERNAME@svn.gnome.org/svn/modulename --prefix=origin/svn/ git svn rebase after cloning, and then "git svn dcommit" will work. Might be a nice addition.
In response to the first point, I wonder if we can do anything about this in jhbuild. Not applying the override when using the mirror would be nice perhaps? That setting is for all svn.gnome.org modules and if there is another full git mirror that the user wants to use instead then someone is doing something wrong somewhere... So repo= should only affect the original definition, not the mirrors. In response to the second point, i agree. I've been thinking about how to do the same thing without adding hacky spaghetti to jhbuild (after all, we'll move to $DVCS within the next year and this tweak will not apply). I would apply a heuristic such as: if has origin/svn/trunk: # this is a git-svn repo if "svn-remote" in .git/config: # has been init'd else: # hasn't been inited And use the original url as the svn-remote location. Could use the url from repos[] = 'svn+ssh://svn.gnome.org/' as the one passed to git-svn.
Well, Git migration and stuff, it has been a fun ride, but this bug can certainly be marked as obsolete now.