GNOME Bugzilla – Bug 738914
video: Add a linear/affine transformation matrix meta
Last modified: 2018-11-03 11:31:45 UTC
+++ This bug was initially created as a clone of Bug #679522 +++ In Quicktime/MP4 we can get an arbitrary transformation matrix that should be applied to the video before displaying. In other contexts we also get that, or sometimes just a rotation angle which could be expressed like this too. There should probably be a meta that contains such an arbitrary 3x3 transformation matrix, can be negotiated inside the pipeline and implemented by sinks or other elements. Additionally there might be constraints on what elements are implementing, e.g. some might only implement n*90° rotations and flips (videoflip element), which might be something that should be possible to negotiate. Also an open question is what should happen with transformations that produce a non-rectangle output (e.g. rotations or shears). I would say they should create a bigger frame with black borders, but what also would be valid is cropping everything that goes over the frame size.
Is this fixed by Commit: 0b98ed32ce9db48089787f32c13c1a3db3ac36f3 URL: http://cgit.freedesktop.org/gstreamer/gst-plugins-base/commit/?id=0b98ed32ce9db48089787f32c13c1a3db3ac36f3 Author: Matthew Waters <matthew@centricular.com> Date: Mon Nov 2 23:12:19 2015 +1100 videometa: add GstVideoAffineTransformationMeta Adds a simple 4x4 affine transformations meta for passing arbitrary transformations on buffers. Based on patch by Matthieu Bouron https://bugzilla.gnome.org/show_bug.cgi?id=731791 ?
The merged affine transform is not really "negotiated" as describe. Element needs to support any value in the 4x4 matrix, or not support it at all. That basically mean, videoscale cannot advertise support for the scaling matrix, and videoflip cannot advertise support for the flipping matrix. So there is no pipeline that can guaranty a ISOMP4 with an AffineMatrixMeta can be rendered unless you have GL (I didn't see that patches for that either, but I might have missed it). So unless we have an element (software and GL) that can handle that meta, I'm not sure this solution helps in enabling the matrix on ISOMP4, as that would likely break things. Best effort for those matrix is kind of ugly. We also need to figure-out and document the interaction with other metas. An example, when the transform is from ISOMP4, it's clear it should be applied before placing the text, but if the transform was added later by an element, then maybe it should be applied later. How do we define the ordering of all meta that are applied on a buffer ? I think more work is needed, the Meta is just a peace of the puzzled thrown in.
That commit provides the data to pass around. The negotiation is still a little up in the air especially for other elements having limited support for a certain set of transformations. GL will be able to use the meta very soon in certain elements (it's needed for proper android zerocopy).
As for software elements wanting to support the meta, there are a couple of options. 1. Don't ;) 2. Add new meta's that can be up-converted to the 4x4 transformation meta. 3. Attempt to decompose the 4x4 matrix into the separate transformations (very hard) 4. Add multiple transformations to the buffer for each scale/rotate/crop/etc (where supported) and have an id system to describe the order 5. Add a meta ordering meta where other meta's can be added to a buffer and ordered based on some priority system to say before/after memory map/access, other meta operations, etc. Maybe the meta tag system could be used for this (have not seen it used successfully for much yet)?
Sorry if I'm being very naive here, but wouldn't it be possible to have flags for the most common type of transformations so we could have something close to number 3? What are the types of transformations that we need to negotiate? 1. none 2. rotation 3. scaling 4. any
5. crop 6. translate 7. shear? 8. perspective? The problem per-se with 3 is that it's very hard to go backwards from the matrix to a specific transformation in some cases (especially when operations are composed).
Would it make sense for qtdemux to first try to decompose the matrix into common simple elementary operations (scale & 90°/180°/270° rotations & flip), and if it fails, then use the whole matrix meta? Then use simple operation meta if possible and full matrix meta as a last resort. If you have GL, then you get full functionality, if you don't then you get a negotiation failure as the software elements (or non-GL hardware) will support the simple operations. I'm going to assume that 99.99% of all files only do simple operations (mostly 90° rotations).
That's certainly a possibility.
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/issues/136.