GNOME Bugzilla – Bug 762515
Endianness not handled for static deltas
Last modified: 2016-02-26 13:21:26 UTC
When creating the variants for the commits we're very careful to always use GUINT64_FROM_BE() and GUINT32_FROM_BE() when we're creating any variant data with non-byte data. This is because the on-disk format is designed to be big-endian. However, for static deltas we're just shoving data in as-is. So, things like file uid,gid,mode as well as sizes and versions are in native endianness, and we never store the endianness anywhere in the file.
Ugh, yes. That was clearly screwed up. I think to dig ourselves out, the first step is going to be adding an ostree.endian metadata key so we can at least infer the endianness for new static deltas. Then try to determine if we can infer endianness from existing deltas via a heuristic. Then finally, change the deltas code to determine the endianness, and use the appropriate accessor macros. We might consider changing the deltas to generate BE by default as well as an option...but I think in practice most users are LE, so that would be the thing that would break them.
Colin: Please put any metadata created by ostree after the user supplied metadata. I used the first metadata element to get a nice sniffable file-format header for the xdg-app bundles.
Ah...that's an interesting hack. But sure, we can support that. https://github.com/GNOME/ostree/pull/191
I think i misread the code a bit. Interestingly it *does* use GUINT32_TO_BE() on the uid/gid/mode, and GUINT64_TO_BE on the timestamp. However, it doesn't in a bunch of places: The fallback object compressed and uncompressed size. The delta part compressed and uncompressed size. The delta part version (Which unfortunately is 0 atm, otherwise we could have used it to detect endianness, but it also means that current deltas are correct wrt endianness) So, unless I'm missing some part this is mainly about the sizes. Unfortunately its not unrealistic that sizes are > 32bit in size, and at that size it becomes hard to guess if they are little or big endian. That said, the sizes are purely informative, and will not cause a corrupt pull, right?
(In reply to Alexander Larsson from comment #4) > So, unless I'm missing some part this is mainly about the sizes. > Unfortunately its not unrealistic that sizes are > 32bit in size, and at > that size it becomes hard to guess if they are little or big endian. I'd say it's pretty unrealistic to have a single > 4GiB file in OSTree, not to mention more than one. > That said, the sizes are purely informative, and will not cause a corrupt > pull, right? Yeah, AFAICS. Anyways I'm working on this.