Also, if we support tags (bug 727564) it seems clear to me that we should also allow enumerating the available tags.

For 729388 we are going to need a single file which contains at a minimum the branch names and the commits they point to, if we want to support chaining like this:

* Retrieve metalink via TLS, resolves to target url + checksum
* Retrieve target URL via HTTP (or ftp), verify checksum

bug 729388

Ok, let's make this a generic property bag of a{sv}.  That will maximize our future flexibility.

Let's call it the repository "summary".  It will appear as repo/summary.

Additionally, it should support inline GPG signatures.  We can detect whether the content starts with ----BEGIN PGP...---- and automatically verify/unwrap it.

Example code: https://github.com/wgwoods/fedup/blob/master/fedup/download.py#L366
Which is basically using gpgme under the hood.

Do we want to support detached signatures as well for this file?

The a{sv} will contain at least the following keys:

* commits: a(ayay) - array of pairs of commit checksum to commit - this is an optimization for when multiple refs point to the same commit.  Commits are in sorted order by checksum, to allow bsearch.
* refs: a{say} - mapping from name to commit id

Created attachment 283979 [details] [review]
fetcher: Close request body on error paths

Otherwise, we're potentially holding up subsequent requests.

I was hitting this when testing the metalink code, where we want to
continue doing more fetches after hitting a 404.

Created attachment 283980 [details] [review]
Bump GIO dependency to 2.38 for GTask

I'd like to use GTask, so bump the dependency.

Created attachment 283981 [details] [review]
Add repository "summary" file and metalink support

For Fedora and potentially other distributions which use globally
distributed mirrors, metalink is a popular solution to redirect
clients to a dynamic set of mirrors.

In order to make metalink work though, it needs *one* file which can
be checksummed.  (Well, potentially we could explode all refs into the
metalink.xml, but that would be a lot more invasive, and a bit weird
as we'd end up checksumming the checksum file).

This commit adds a new command:

$ ostree summary -u

To regenerate the summary file.  Can only be run by one process at a
time.

After that's done, the metalink can be generated based on it, and the
client fetch code will parse and load it.

TODO:
 - Decide whether this file can contain an inline signature, or should
   be duplicated
 - Add size of commit object

Also, we should use the size metadata to limit request sizes smaller than the 10MiB limit.

Also, we should obey maxconnections.

Comment on attachment 283979 [details] [review]
fetcher: Close request body on error paths

Attachment 283979 [details] pushed as c08a450 - fetcher: Close request body on error paths

(In reply to comment #9)
> Also, we should obey maxconnections.

Though...that element was designed for package systems; let's see if people object to us being more aggressive by default.  If they do, we can change things.

James approved this on IRC.

Attachment 283980 [details] pushed as 3571418 - Bump GIO dependency to 2.38 for GTask
Attachment 283981 [details] pushed as f8f5da2 - Add repository "summary" file and metalink support