After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 613033 - Setup new master.gnome.org
Setup new master.gnome.org
Status: RESOLVED OBSOLETE
Product: sysadmin
Classification: Infrastructure
Component: Other
unspecified
Other All
: Normal normal
: ---
Assigned To: GNOME Sysadmins
GNOME Sysadmins
: 613023 (view as bug list)
Depends on:
Blocks:
 
 
Reported: 2010-03-16 12:04 UTC by Olav Vitters
Modified: 2013-11-21 14:55 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Files added to /ftp in the last 60 days (270.46 KB, text/plain)
2011-03-01 09:22 UTC, Olav Vitters
Details
Tarballs in sources/ not matching my regexp (12.79 KB, text/plain)
2011-03-01 23:06 UTC, Olav Vitters
Details
maintainers per directory in sources (21.09 KB, text/plain)
2011-03-08 20:41 UTC, Olav Vitters
Details

Description Olav Vitters 2010-03-16 12:04:53 UTC
Currently many people have access to a shell account on master.gnome.org just to run install-module.

Though people should be trusted before allowing shell access, the more people who have shell, the more likely things can go wrong. Further, as everyone has access to the 'real' ftp location, things can get messy (see sources/gazpacho: it has binaries in there). Further, some modules use strange version numbers. :-(

I propose setting up a master.gnome.org which:
1. Has a ChrootDirectory
2. Only allows sftp using "internal-sftp" ForceCommand
   ==> No more shell access!
   ==> Can this be done in a authorized_keys file? Or do we need a sshd_config Match command? IIRC not possible in RHEL5
3. Access is solely determined from .doap files
4. Lives as a VM on combobox

>>> ChrootDirectory <<<
On master.gnome.org there should be a directory structure which looks like (chrooted):
/
/sources
/binaries/linux
/binaries/mac
/binaries/win32
/binaries/win64
/errors

People do NOT get access to the real /ftp/pub/GNOME unless they have specific needs.

Either via cron or via inotify a new version of the install-module script is called. This script validates every new file and places it in the correct location in /ftp/pub/GNOME. The script should be somewhat smart: Placing a file in sources directory means it will go into the sources directory. However: sources files MUST end with .tar.gz or .tar.bz2. However, perhaps some files are allowed to be stored in just /. Not sure yet.

Files placed in the root will be determined automatically.

In case there was an error validating the file it is moved to /errors. On success the file is removed from the chroot.

Informing of user:
User receives an email *always*. Subject field should be clear.

This script should live in sysadmin-bin Git module (currently it resides in releng). I have a partial script in /usr/local/bin/py-install-module on window. The new Python version will NOT work exactly like the existing version.


>>> ftpadmin group and shell access <<<
Everyone is removed from this group. People who really have specific needs will have to request access again. This will result in people *NOT* having shell anymore. I expect a big backlash with this, especially as some people are hosting various things in their ~/public_html.

>>> DOAP files and access <<<
Currently anyone can upload anything they want. In future:
1. Any release will be emailed'ed to *all* the maintainers (as determined from the .doap files)
2. Any access change will be emailed to all the current/previous maintainers
3. When gaining master.gnome.org access, user should get an email with instructions for master.gnome.org

>>> Modules not in git.gnome.org <<<
Not sure what to do. There have to be exceptions for either people (release team) or modules (gstreamer). Perhaps hard-code modules? (gstreamer, intltool). New group perhaps?

>>> VM on combobox <<<
Need a public IP address
Comment 1 Olav Vitters 2010-03-16 12:05:29 UTC
*** Bug 613023 has been marked as a duplicate of this bug. ***
Comment 2 Christer Edwards 2011-02-03 20:45:07 UTC
VM has been built on RHEL6.

1cpu, 512M, 5G HDD.
Comment 3 Olav Vitters 2011-02-27 21:15:14 UTC
https://github.com/seb-m/pyinotify
https://github.com/seb-m/pyinotify/blob/master/python2/examples/autocompile.py

Planning to simplify the Python version of install-module. Remove RSS feed generation and so on. Matching every feature is taking too much time and preventing me from making good progress.

Regarding shell access, there is a 'webusers' group which we could use to allow people to SSH to whatever people.gnome.org is hosted upon.
Comment 4 Owen Taylor 2011-02-28 21:26:56 UTC
When I send out release announcements, I include the sha256sums of the tarballs that install-module creates or installs. I generally do that, by

 install-module
 cat /ftp/<path-mentioned-by-install-module>/gnome-shell-2.99.99.sha256sum

Would be nice if they were just printed out.
Comment 5 Olav Vitters 2011-02-28 21:43:40 UTC
<owen> (I'm sort of skeptical about the whole tarball release process, really. It seems a bit like a big dance for not a whole lot of point other than maybe gentoo, when the git tag should be thing that really matters and is a few bytes, but thats a different matter)
<owen> bkor: non-maintainer uploads?
<bkor> yes
<bkor> think
<bkor> 1) send an email when a non-maintainer uploads to the maintainers
<bkor> 2) have a special group of people who can upload anything
<bkor> cannot decide between 1 and 2
<owen> bkor: I think I'd favor 1) ... less overhead of maintaining that group
Comment 6 Olav Vitters 2011-02-28 21:44:20 UTC
<owen> bkor: Does some sort of 'gnomepkg' script make sense?
<owen> You put it in your ~/bin and do 'gnomepkg upload module.tar.gz' and it does the sftp and tells youthe shasums?
<bkor> hmm, sounds nice
<bkor> I was wondering about interrupts when uploading via sftp
<bkor> gnomepkg could upload a sha256sum as well
Comment 7 Olav Vitters 2011-03-01 09:22:02 UTC
Created attachment 182165 [details]
Files added to /ftp in the last 60 days

Should cater for:
1. Release team
2. Teams in general (though nobody uploaded anything in the last 60 days)
3. Source tarballs
4. Binaries
Comment 8 Olav Vitters 2011-03-01 16:20:04 UTC
Thought about it some more. I think we cannot avoid allowing some people to log in directly to change /ftp[1] . As such, sysadmins should be able to login normally as well. Due to the current problems with window, I'm going to first setup the new master.gnome.org and follow the current setup on window.

For hosting your own site in ~/public_html:
1) master.gnome.org VM shouldn't have any webserver IMO
2) don't want any scripts running on hosts where people might not had a shell yet

Result:
1) ~/public_html shouldn't have any ability to execute scripts by default
2) all default AddHandlers should be removed
3) e.g. no .php/.cgi AddHandler by default
4) want to move people.gnome.org to webapps

Note:
Also considered splitting people.gnome.org from gnomeweb-wml. Still think it is better not to allow anything to affect the /ftp directory.

[1] release team, perhaps others.. want to really limit the number of people having access
Comment 9 Olav Vitters 2011-03-01 23:01:37 UTC
Wondering about using xdelta3 for binary diff files.

> xdelta3 -e -B 200000000 -P 67108864 -9 -S djw -s gtk+-2.18.6.tar.bz2 gtk+-2.18.7.tar.bz2 gtk.6.7-7.xd3

> ls -l gtk+-2.18.6.tar.bz2 gtk+-2.18.7.tar.bz2 gtk.6.7-7.xd3
> -rw-rw-r-- 1 olav olav 18137401 Jan 12  2010 gtk+-2.18.6.tar.bz2
> -rw-rw-r-- 1 olav olav 18304767 Feb 13  2010 gtk+-2.18.7.tar.bz2
> -rw-rw-r-- 1 olav olav   336837 Mar  1 23:23 gtk.6.7-7.xd3

Difference in the tar.bz2 files:
  167366

xdelta3 file:
  336837

This diffs png files, etc. Could also do some validations on this file.

Note: xdelta3 (un)compresses the bz2 file. So it'll diff the tar and compress again. Likely breaking checksumming in the process.
Comment 10 Olav Vitters 2011-03-01 23:06:27 UTC
Created attachment 182218 [details]
Tarballs in sources/ not matching my regexp

Tarballs in sources directory which do not match the following regexp:

^(?P<module>.*)[_-](?P<version>([0-9]+[\.\-])*[0-9]+)?\.(?P<format>tar.*)$

I don't want to allow too much in the regexp. I basically hate these 'pre1' etc things, because then I cannot do a sane version sort. I need such a version sort to allow diffs between e.g. module-2.91.20 and module-3.0.0.

Think the regexp is good enough. Might want to allow version-1. E.g. module-2.91.20-1.. though I think some enforcement is fine.
Comment 11 Olav Vitters 2011-03-01 23:08:48 UTC
PS: 231 mismatches out of 35484 tarballs (0.7%). Dia and java-gnome often release such versions.
Comment 12 Olav Vitters 2011-03-01 23:31:06 UTC
Determining valid file:
 - module name matches a DOAP file (LDAP actually)
 - existing directory (gstreamer, external dependencies on GNOME)

What to do with module vs tarball name mismatches?

If maintainers cannot be determined, whom to inform? Release-team I guess.
Comment 13 Olav Vitters 2011-03-06 21:16:22 UTC
(In reply to comment #4)
> When I send out release announcements, I include the sha256sums of the tarballs
> that install-module creates or installs. I generally do that, by
> 
>  install-module
>  cat /ftp/<path-mentioned-by-install-module>/gnome-shell-2.99.99.sha256sum

Current Python version of install-module:
> $ py-install-module /ftp/pub/GNOME/sources/gtk+/2.99/gtk+-2.99.3.tar.bz2
> Checking for info about /ftp/pub/GNOME/sources/gtk+/2.99/gtk+-2.99.3.tar.bz2, done
> ERROR: gtk+-2.99.3.tar.bz2 already exists in the archive!
> DEBUG: Continuing anyway in debug mode
> Checking consistency /ftp/pub/GNOME/sources/gtk+/2.99/gtk+-2.99.3.tar.bz2: done
>       Module: gtk+
>      Version: 2.99.3   (previous version: 2.99.2)
>  Destination: /ftp/pub/GNOME/sources/gtk+/2.99/
>·
> Install gtk+? [Y/n] y
> Creating new files:
>  - Checking previous tarball, done
>  - NEWS, done (diff, 138 lines)
>  - ChangeLog, done (diff, 4637 lines)
>  - Copying tar.bz2, done
>  - Creating tar.gz from tar.bz2: ......................., done
>  - Creating sha256sum, done
> DEBUG: Not removing temporary directory: /tmp/install_module5UDLBp
> Doing notifications:
>  - Informing ftp-release-list, done
>  - Triggering GNOME library update, done
>  - Adding new version to GNOME Bugzilla, ignored (debug mode)
>  - Triggering ftp.gnome.org update, ignored (debug mode)
>·
> Your tarball will appear in the following location on ftp.gnome.org:
>·
>   http://download.gnome.org/sources/gtk+/2.99/
>·
> It is important to retain the trailing slash for compatibility with
> broken http clients, and to use http as it is less taxing on the server.
>·
> 34955946bb41d951bc54ffd462e23e9cd1a0f5d8c73fc0e42614ac77c07d740e  gtk+-2.99.3.changes
> e5dcc884794e2cb232d132a8feeabccbc3218cefaeb9397e154fe091f1e09a02  gtk+-2.99.3.news
> 03dd37fd89fe0f0cea688cb077192b5ff95325b66d53d724c7511d36b6e90496  gtk+-2.99.3.tar.bz2
> b456ab2b90c550cca9596bc5a939eb87dc4d0af3d502c180ec2fa4718a0ced61  gtk+-2.99.3.tar.gz


Script is not ready yet, but hope to complete it soon. See sysadmin-bin module for the script (py-install-module).
Note: Requires RHEL6 (needs Python 2.6, etc)
Comment 14 Olav Vitters 2011-03-07 21:54:30 UTC
Working on http://git.gnome.org/browse/sysadmin-bin/tree/run-rrsync-or-special-cmd

Then working on enabling something like:
> rsync -av gtk+-2.12.7.tar.xz master.gnome.org:sources
> ssh master.gnome.org install-module sources/gtk+-2.12.7.tar.xz

Above is annoying to type in (plus not sure if rsync options are perfect), but we could provide a 'gnomepkg' script to ease the typing.

Theoretically this should be possible. Just lots of things to think about.

Benefits:
 * rrsync secures the file transfer (though chroot would be more ideal)
 * separate install-module step, so you can follow its progress

Obviously, the install-module options would be limited. Not the full options available to 'ftpadmin' users.

My only thoughts atm are on how I:
 1. Can have one group for /ftp/scratch (the rrsync location)
 2. Have another group for install-module.

I think I'll go for some hack involving:
>  sudo /usr/local/bin/install-module --sudo -- /ftp/scratch/sources/gtk+-2.12.7.tar.xz

Maybe directory sticky bits?

Need to create a new group for the automatic (and restricted) maintainers (the ones not in ftpadmin).
Comment 15 Olav Vitters 2011-03-07 23:20:37 UTC
(In reply to comment #14)
> Then working on enabling something like:
> > rsync -av gtk+-2.12.7.tar.xz master.gnome.org:sources
> > ssh master.gnome.org install-module sources/gtk+-2.12.7.tar.xz

The following will be used instead:
> ssh master.gnome.org install-module

And install-module will just determine all the files and ask you which ones you want to have installed. I'll also allow you to delete files in the prompt (pretty annoying with just rsync).

Going to make install-module smart enough to sort all the new files by version. This when you're moving towards the GNOME infrastructure and have loads of files to upload.

Nice thing about using sudo is that the user is still the same, only the group changed. More secure IMO. Also, I can use the current effective userid to determine which files are new.

> My only thoughts atm are on how I:
>  1. Can have one group for /ftp/scratch (the rrsync location)
>  2. Have another group for install-module.
[..]
> Maybe directory sticky bits?

That seems to work. Files uploaded by my testuser have the ftpadmin group thanks to the sticky bit :)

> Need to create a new group for the automatic (and restricted) maintainers (the
> ones not in ftpadmin).

Created an ftpbasic group for this. Going to make the DOAP handling script automatically add/remove maintainers from this group.
Comment 16 Olav Vitters 2011-03-07 23:48:42 UTC
(In reply to comment #15)
> Created an ftpbasic group for this. Going to make the DOAP handling script
> automatically add/remove maintainers from this group.

done. about 482 people are in this group (!).. excludes my testuser (need to add him back)

hrm, ftpadmin has 357 people.. so.. not too much of a difference :P
Comment 17 Olav Vitters 2011-03-08 20:41:52 UTC
Created attachment 182867 [details]
maintainers per directory in sources

tried to link the different directories in download.gnome.org/sources to maintainers.

didn't always match up.. though a lot of old stuff
Comment 18 Olav Vitters 2011-03-19 19:35:19 UTC
Shell version is as good as finished...

Note: notifications don't work in below example as install-module installs to an alternate directory.

$ ftpadmin install ~/delme-0.10.3.tar.bz2 
WARNING: Running in DEBUG MODE!
Gathering information and sorting on version: ., done
Preparing installation of delme-0.10.3.tar.bz2:
 - Checking consistency: ..., done
 - NEWS, done (new file)
 - ChangeLog, done (new file)
 - Copying tar.bz2, done
 - Creating tar.gz from tar.bz2: ...., done
 - Creating sha256sum, done

      Module: delme
     Version: 0.10.3   (previous version: N/A)
 Destination: /ftp/tmp/sources/delme/0.10/

Install delme? [Y/n] y
Installing delme-0.10.3.tar.bz2:
 - Moving files: ...., done
 - Updating LATEST-IS, done
 - Updating known versions, done
Doing notifications:

Please report any problems to:
https://bugzilla.gnome.org/enter_bug.cgi?product=sysadmin
Comment 19 Andrea Veri 2013-11-21 14:55:38 UTC
The GNOME Infrastructure Team is currently migrating its bug / issue tracker away from Bugzilla to Request Tracker and therefore all the currently open bugs have been closed and marked as OBSOLETE.

The following move will also act as a cleanup for very old and ancient tickets that were still living on Bugzilla. If your issue still hasn't been fixed as of today please report it again on the relevant RT queue.

More details about the available queues you can report the bug against can be found at https://wiki.gnome.org/Sysadmin/RequestTracker.

Thanks for your patience,

the GNOME Infrastructure Team