GNOME Bugzilla – Bug 719902
video: Multi-byte-per-component video formats, endianness and component order
Last modified: 2018-11-03 11:26:47 UTC
Currently ARGB64 and AYUV64 are defined in native-endianness, while I420_10BE/LE have different variants for the different endianness. Should we make this more consistent, deprecate the ARGB64 format and add two new ones for it? I'm also going to add a ARGB64_F16 video format soonish, for which the same question would apply. Currently I only need it in native endianness, but it's a question of consistency. Also should we add variants for the different component orders? E.g. a RGBA64, BGRA64, ABGR64? Variants with x instead of A? Or do we expect elements to do the conversion themselves to the correct order? What I currently would need would be a RGBA64_F16 to allow zerocopy, but that would of course be inconsistent with ARGB64 :)
Note that in the most generic form this would be an explosion of video formats. 4 (ARGB, ABGR, RGBA, BGRA) times 2 (A vs x) times 2 (BE vs LE) video formats for one component data type (i.e. 16 bit unsigned integer). So I would basically add 32 new video formats (ARGB U16 and F32), and 2 new for AYUV (AYUV64_BE and AYUV64_LE).
Or you can just add the ones that actually exist in files?
These are clearly processing or transport formats more than anything else, no? I think only having one "native" variant and requiring variants with other byte orders or pixel packing to be re-packed into the native one is perfectly fine for now. Can always add more later when there's really a use case for it or some pressing performance issue.
The problem I see here is the existing I420_10BE/LE format for example, which should not exist according to what Tim said in comment 3 as we always get it in native endianness. So ideally those would become deprecated in favor of a I420_10 (and same for the GBR, I422 and others there). So you would propose to add a native endianness variant of ARGB64_F16 then as it's in line with ARGB64. And later when needed we add a ARGB64_F16_LE/BE, or RGBA_F16 or ...? I think we're missing some general rules here :)
You don't have to add variants for ARGB, ABGR, RGBA, BGRA, these formats have bit depths of 8 and so ARGB is uniquely defined in all endiannesses. For the ones with component sizes > 8 (I420_10, ..) you need to know how to read the component and there is a flag in the info that describes this. For RGB16, we always have LE endianness (but we don't read it like this currently, we read it in native endianness..) The endianness is of course essential for exchanging raw formats between architectures of different endianness. If there is no I420_10LE and BE variant, it would be impossible to exchange this format between LE and BE machines. If there is a format in the wild that specifies the endianness explicitly, I would add a variant for it or else you would not be able to handle this on some architectures. To exchange raw formats, I would again only make those formats that you find in real files and not care about exchanging raw formats. For the weird ones you either need to convert them to something portable or (when possible) exchange for the equivalent formats (ARGB -> BGRA), IMHO. I also think I420_10LE and BE is a bit overkill, we should probably have kept the native ones only.
(In reply to comment #5) > You don't have to add variants for ARGB, ABGR, RGBA, BGRA, these formats have > bit depths of 8 and so ARGB is uniquely defined in all endiannesses. Agreed, I never suggested that but was talking about ARGB64 here :) > For the ones with component sizes > 8 (I420_10, ..) you need to know how to > read the component and there is a flag in the info that describes this. For > RGB16, we always have LE endianness (but we don't read it like this currently, > we read it in native endianness..) The flag is per VideoFormatInfo though, not something you can change via the caps or GstVideoInfo. So it's a fixed endianness per GstVideoFormat. Should we change the flag for RGB16/15 to be set depending on the host endianness btw? > The endianness is of course essential for exchanging raw formats between > architectures of different endianness. If there is no I420_10LE and BE variant, > it would be impossible to exchange this format between LE and BE machines. > > If there is a format in the wild that specifies the endianness explicitly, I > would add a variant for it or else you would not be able to handle this on some > architectures. > > To exchange raw formats, I would again only make those formats that you find in > real files and not care about exchanging raw formats. For the weird ones you > either need to convert them to something portable or (when possible) exchange > for the equivalent formats (ARGB -> BGRA), IMHO. > > I also think I420_10LE and BE is a bit overkill, we should probably have kept > the native ones only. Makes sense but what does this mean in the end for my case? You would add a ARGB64_F16 (for consistency with ARGB64) and the only user of it so far would reorder the bytes? I'm fine with that btw, I just think we should define this somewhere instead of deciding on something different next time someone comes up with such a thing ;) If I had a time machine I would make the I420_10LE/BE things a I420_16 (without LE/BE) btw as there's not much advantage in having 10 bits... other than that we have to add variants for 9, 11, 12, ... 16 bits too now :)
If nobody complains, I'll: 1) Make the endianness flag for RGB16/15 depending on the native endianness 2) Add ARGB64_F16 (native endianness) 3) Maybe: Deprecate endianness-aware variants of I420_10* and GBR_10* and replace them by a native endianness variant? Note that libav has LE/BE variants of these. And also variants for 9, 11 and 12 bits.
(In reply to comment #7) > If nobody complains, I'll: > > 1) Make the endianness flag for RGB16/15 depending on the native endianness Yes, at least > 2) Add ARGB64_F16 (native endianness) How about we add (and only add) ARGB64_F16LE on little endian and ARGB64_F16BE on big endian at least we can then later add the other endianness without causing trouble? We could do GST_VIDEO_FORMAT_ARGB_F64 = _GST_VIDEO_FORMAT_NE(ARGB_F64) like we do with audio. > 3) Maybe: Deprecate endianness-aware variants of I420_10* and GBR_10* and > replace them by a native endianness variant? Note that libav has LE/BE variants > of these. And also variants for 9, 11 and 12 bits. also note that ffmpeg only does native audio formats while we list all LE/BE variants
(In reply to comment #8) > > 2) Add ARGB64_F16 (native endianness) > > How about we add (and only add) ARGB64_F16LE on little endian and ARGB64_F16BE > on big endian at least we can then later add the other endianness without > causing trouble? That will break gobject-introspection in interesting ways, otherwise I like it :) > We could do GST_VIDEO_FORMAT_ARGB_F64 = _GST_VIDEO_FORMAT_NE(ARGB_F64) like we > do with audio. Yes > > 3) Maybe: Deprecate endianness-aware variants of I420_10* and GBR_10* and > > replace them by a native endianness variant? Note that libav has LE/BE variants > > of these. And also variants for 9, 11 and 12 bits. > > also note that ffmpeg only does native audio formats while we list all LE/BE > variants What do you mean with that?
> > also note that ffmpeg only does native audio formats while we list all LE/BE > > variants > > What do you mean with that? I mean that we would go the opposite path of ffmpeg, they specify all video formats with LE/BE and audio formats only in native endianness.
That's true, yes. Do you think that's a problem? We can still add the endianness-aware versions later if needed without too much effort.
(In reply to comment #11) > That's true, yes. Do you think that's a problem? We can still add the > endianness-aware versions later if needed without too much effort. I think it's fine to only add the endianness that we currently need but we should do it in such a way that we can easily and naturally add the other endianness later and while we're at it, make it possible to exchange formats between machines with different endiannesses (so endianness needs to be in the format name somewhere). If it would be possible to only define one of ARGB_F16LE or ARGB_F16BE (depending on the endianness with an enum value mapping ARGB_F16 to the native endian version) that would be ideal. I think that's what we do for audio now.
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/issues/96.