So our aggressive use of fsync() is unwarranted; the bug turned out to 
be

https://bugzilla.redhat.com/show_bug.cgi?id=1099237

I think the simplest plan is to instead of calling fsync(), keep track of the filesystems we're writing to (ordinarily, just /boot and /).  Then just syncfs() them in the same place we're calling sync() right now.

There's two places where we fdatasync() - pulling objects, and during the checkout.

I think we could use syncfs() by default for pulls with a scheme like this:
- Anything stored in objects/ is known to be fdatasync()d
- During pull, we leave objects in tmp/, and don't fsync them.
- When pull for an object is complete, rename them to tmpobject-$checksum.$type, leaving them in the tmpdir.
- This means that for metadata, we will have to change the load_object() calls inside pull to look for temporary objects as well
- When pull is complete, we syncfs() and renameat() all of the temporary objects into place


Then the next phase is checkouts.  Where we do what I was describing in comment #1.

We could optimize a pull-deploy phase by only doing exactly one syncfs(), and renaming all of the objects into place after that.

I've implemented this logic here:

https://github.com/giuseppe/ostree/commit/b13e13babda9a83309e1435554dff82636304df6

can this be closed?

Yep, we can close this now I think.

Note there is some work on an XFS "fsync many things" API:

http://oss.sgi.com/archives/xfs/2014-06/msg00214.html

We might be able to make use of that if we're on XFS instead of the big `syncfs` hammer.