GNOME Bugzilla – Bug 779213
buffer: Add GstReferenceTimestampMeta
Last modified: 2017-05-12 09:44:29 UTC
See commit message
Created attachment 346701 [details] [review] buffer: Add GstReferenceTimestampMeta This is a meta that generically allows to attach additional reference timestamps to a buffer, that don't have to relate to the pipeline clock in any way. Examples of this could be an NTP timestamp when the media was captured, a frame counter on the capture side or the (local) UNIX timestamp when the media was captured.
Created attachment 346702 [details] [review] decklinkvideo/audiosrc: Add GstReferenceTimestampMeta with the stream time to each buffer This is basically a frame counter provided by the driver and it's advancing at the speed of the HDMI/SDI input. Having this available on each buffer allows to know what constant-framerate-based timestamp each frame is corresponding to and can be used e.g. to write out files accordingly without having the local pipeline clock timestamps used.
Created attachment 346703 [details] [review] buffer: Add GstReferenceTimestampMeta This is a meta that generically allows to attach additional reference timestamps to a buffer, that don't have to relate to the pipeline clock in any way. Examples of this could be an NTP timestamp when the media was captured, a frame counter on the capture side or the (local) UNIX timestamp when the media was captured.
Review of attachment 346703 [details] [review]: Interesting, my early concern would be if you endup with two of these? E.g., v4l has a original capture timestamp we could expose (though exposing a clock for this one would do the same), and a frame counter.
(In reply to Nicolas Dufresne (stormer) from comment #4) > Review of attachment 346703 [details] [review] [review]: > > Interesting, my early concern would be if you endup with two of these? E.g., > v4l has a original capture timestamp we could expose (though exposing a > clock for this one would do the same), and a frame counter. Maybe this needs some differentiation then :)
So how about timestamp/x-original-capture-time and timestamp/x-stream for these two? However in the Decklink case they're both the same :)
Now, on top, how do we express if that timestamp is CLOCK_MONOTIC, or something unknown, if it's timestamp of the first line being capture, or the last one ? (or who knows, something in the middle). In V4L2, there is flags you can add to further describe the timestamps.
Maybe we could instead have reference to what driver produce the timestamp, and then copy the flags as-is ?
We could add parameters to the caps for those for example, yes. What is it all in the v4l2 API, can you summarize?
Summary of the V4L2 time features: struct v4l2_buffer: timestamp; used to produce timestamp timecode; I believe should be mappable to timecode meta flags; Specify what fields of v4l2_buffer are are valid and extra meaning The timestamp flags are: ***V4L2_BUF_FLAG_TIMESTAMP_MASK; Allow to extract the timestamp format (an enum inside the bitmask) Valid values are one of: V4L2_BUF_FLAG_TIMESTAMP_UNKNOWN; Was set on legacy driver with no evident clock id correlation, in theory it should be real-time or monotonic. (it's the value 0, so basically drivers that haven been updated will have that by default). V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC; All new and maintained driver capture driver have this flag, since timestamp are derived to the monotonic clock V4L2_BUF_FLAG_TIMESTAMP_COPY; this is used by color converter, decoder, encoder etc. to indicate that the timestamp has simply been copied from input to output (even if there is re-ordering happening). ***V4L2_BUF_FLAG_TSTAMP_SRC_MASK This is another enum inside the bitmask, which indicate the moment the timestamp is taken. Values are: V4L2_BUF_FLAG_TSTAMP_SRC_EOF The timestamp was taken at the End of Frame. This is the default, it is also used when the timestamp has been software generated. This happens when the last pixel has been received over some serial bus. V4L2_BUF_FLAG_TSTAMP_SRC_SOE Timestamp taken at Start of Exposure. Only valid for capture device, will be set on some UVC devices, no other driver uses it (other then vivid). References: https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/buffer.html#struct-v4l2-buffer https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/buffer.html#buffer-flags
Oh, and forgot one v4l2_buffer field: sequence, Set by the driver, counting the frames (not fields!) in sequence. This field is set for both input and output devices.
Review of attachment 346703 [details] [review]: ::: gst/gstbuffer.h @@ +662,3 @@ + /*< public >*/ + GstCaps *reference; + GstClockTime timestamp, duration; Maybe we want to allow for DTS != PTS here also? Nicolas?
So from the V4L2 point of view, what would be relevant here is 1) timestamp/x-monotonic (V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC) 2) timestamp/x-realtime (optionally could contain information like timezone?) All other things seem unneeded. EOF/SOE does not apply for us really, all timestamps in GStreamer should be SOE? Do you see any need for addition to the meta, other than defining the reference a bit better and giving more examples?
Created attachment 349229 [details] [review] decklinkvideo/audiosrc: Add GstReferenceTimestampMeta with the stream time to each buffer This is basically a frame counter provided by the driver and it's advancing at the speed of the HDMI/SDI input. Having this available on each buffer allows to know what constant-framerate-based timestamp each frame is corresponding to and can be used e.g. to write out files accordingly without having the local pipeline clock timestamps used.
Nicolas, do you have any suggestions how we could define something common here? I don't care about the actual frame counter for now, that can be added later. But in decklink we have something like a "stream time", in v4l2 we have the monotonic system time and "real time", and the frame counter. These are all different things it seems. The frame counter (in time units) would seem equivalent to the decklink stream time though. Maybe we should just not define this one (and mention NTP/PTP as examples), and let decklink use timestamp/x-decklink-stream-time instead? And V4L2 do their own custom things too? These meanings are highly dependent on the driver in the end anyway, and you probably always need to know what driver was used to make any use of it. Ideas?
I think timestamp/x-<driver>-<desc> is safe. When you need this meta, you are already no doing something generic I think. And then, we can add per driver field we need.
Created attachment 351704 [details] [review] buffer: Add GstReferenceTimestampMeta This is a meta that generically allows to attach additional reference timestamps to a buffer, that don't have to relate to the pipeline clock in any way. Examples of this could be an NTP timestamp when the media was captured, a frame counter on the capture side or the (local) UNIX timestamp when the media was captured.
Created attachment 351705 [details] [review] decklinkvideo/audiosrc: Add GstReferenceTimestampMeta with the stream time to each buffer This is basically a frame counter provided by the driver and it's advancing at the speed of the HDMI/SDI input. Having this available on each buffer allows to know what constant-framerate-based timestamp each frame is corresponding to and can be used e.g. to write out files accordingly without having the local pipeline clock timestamps used.
Comment on attachment 351704 [details] [review] buffer: Add GstReferenceTimestampMeta Attachment 351704 [details] pushed as 7d930b8 - buffer: Add GstReferenceTimestampMeta
Attachment 351705 [details] pushed as 5dc1039 - decklinkvideo/audiosrc: Add GstReferenceTimestampMeta with the stream time to each buffer