GNOME Bugzilla – Bug 690564
Design documentation is out of date WRT GstSegment
Last modified: 2015-04-06 01:25:19 UTC
Created attachment 231999 [details] Notes on GstSegment fields and functions part-synchronisation.txt, part-clocks.txt and part-segments.txt still refer to GstSegment fields only present in Gstreamer 0.10 but removed for Gstreamer 1.0. This includes the field "accum" - I believe this has been replaced by "base" and abs_rate which can be derived from rate and applied_rate. Also GST_SEEK_TYPE_CUR has been removed. It seems from the Gstreamer 1.0 porting guide that GstSegment underwent a sanitization process between 0.10 and 1.0 so it would be good if the documentation could reflect that. I have been attempting to understand GstSegment in 1.0 (having had no exposure to GstSegment in 0.10). I have attached my notes so far from reading gstsegment.c and working though the different transformations that the included functions do. I'm still trying to work things out so hopefully I'll be able to make the notes more coherent in the future.
Posting notes inline for easier access: Field overview ============== s.base is total elapsed clock time spent in previous segments (base == 0 means it is the first segment). e.g. the running time at which the segment started. running time (and thus s.base) is reset every time there is a flushing seek. s.start is the stream time of the start of the segment (e.g. the lower bound of the stream that should actually be displayed as part of this segment in stream time) s.stop is the stream time of the end of the segment (e.g. the upper bound of the stream that should actually be displayed as part of this segment in stream time) s.offset is non-zero when a seek that didn't involve updating the play position has occurred (e.g. a play speed adjustment). In which case it's the amount of the segment (in stream time) that has already been played at the time the segment was updated. If we are doing a seek which involves adjusting the play position (e.g. not just adjusting the playback rate) s.position is set to the start of the segment in stream time (or end in the case of reverse playback), so s.position is the stream time that the position query would return at the beginning of the segment (segment time == 0). !!! Is position the stream time at which we want one of these transformations to occur??? In which case it would only be interesting when we are making one of these GstSegment transformations rather than in any other situation. This would also make sense for duration. !!! ???s.duration is the duration of the stream (e.g. is what would be returned by gst_element_query_duration??? ???s.time is the stream time at which position and duration were sampled??? s.time - After a seek s.time = s.start - seems to have something to do with applied_rates as against rate. Relationships between segment fields and different clocks ========================================================= If the above is true then the transformations between the different clocks are: running time -> stream time f(s, x) = (x - s.base) * rate + s.start stream time -> running time f(s, x) = (x - s.start) / rate + s.base stream time -> clock time ? f(s, x) = (x - s.start) / rate + s.base + element.base_time These transformations do not include s.time or s.offset so must be incomplete in some way. Methods ======= gst_segment_to_running_time --------------------------- input: Stream Time output: Running time Forward play (rate > 0): f(s, x) = (p - s.start - s.offset)/s.rate + base Backward play (rate <= 0): f(s, x) = (p - s.stop + s.offset)/s.rate + base gst_segment_to_position ----------------------- Nominally the inverse of gst_segment_to_running_time but doesn't involve s.offset for some reason? input: Running time output: Stream time Forward (s.rate > 0) f(s, x) = start + (x - s.base) * s.rate Backward (s.rate <= 0) f(s, x) = stop + (x - s.base) * s.rate gst_segment_to_stream_time -------------------------- Seems to have something to do with timestamp transformations when a rate transformation has already been applied to embedded video data PTS? or buffer timestamps? input: Not sure (same time as in buffer timestamps) output: stream time Forward (s.applied_rate > 0) f(s, x) = s.time + (x-s.start) * s.applied_rate Backward (s.applied_rate <= 0) f(s, x) = s.time - (x-s.start) * s.applied_rate
Forgot to mention: Tim was helpful on IRC and I got some of my understanding from discussing with him. He asked me to create this bug.
(In reply to comment #1) > Posting notes inline for easier access: > > s.start is the stream time of the start of the segment (e.g. the lower bound of > the stream that should actually be displayed as part of this segment in stream > time) > > s.stop is the stream time of the end of the segment (e.g. the upper bound of > the stream that should actually be displayed as part of this segment in stream > time) s.start and s.stop are _not_ stream-time but the first and last valid buffer timestamp for the segment. > > If we are doing a seek which involves adjusting the play position (e.g. not > just adjusting the playback rate) s.position is set to the start of the segment > in stream time (or end in the case of reverse playback), so s.position is the > stream time that the position query would return at the beginning of the > segment (segment time == 0). There are two use cases for the GstSegment structure. One is to track the current progress in a media file, like what a demuxer or source might do. And the other is to send as an event in-band to notify downstream elements of the running-time of buffers. Position and duration are only used when tracking the current progress in a media file. > > !!! Is position the stream time at which we want one of these transformations > to occur??? In which case it would only be interesting when we are making one > of these GstSegment transformations rather than in any other situation. This > would also make sense for duration. !!! No, it is purely to track current progress for a demuxer. > > ???s.duration is the duration of the stream (e.g. is what would be returned by > gst_element_query_duration??? It is used by a demuxer to track current progress and the demuxer would also use it to answer the duration query. > > ???s.time is the stream time at which position and duration were sampled??? > > s.time - After a seek s.time = s.start - seems to have something to do with > applied_rates as against rate. s.time is the stream-time associated with the timestamp s.start. > > Relationships between segment fields and different clocks > ========================================================= > If the above is true then the transformations between the different clocks are: > > running time -> stream time > > f(s, x) = (x - s.base) * rate + s.start > > stream time -> running time > > f(s, x) = (x - s.start) / rate + s.base > > stream time -> clock time ? > > f(s, x) = (x - s.start) / rate + s.base + element.base_time > > These transformations do not include s.time or s.offset so must be incomplete > in some way. These transformations are wrong, stream-time != timestamp. > > Methods > ======= > > gst_segment_to_running_time > --------------------------- > > input: Stream Time No, these methods take a buffer timestamp as input, not a stream-time > output: Running time > > Forward play (rate > 0): > > f(s, x) = (p - s.start - s.offset)/s.rate + base > > Backward play (rate <= 0): Rate == 0 is not allowed, it must be done by pausing the pipeline. > > f(s, x) = (p - s.stop + s.offset)/s.rate + base > > gst_segment_to_position > ----------------------- > > Nominally the inverse of gst_segment_to_running_time but doesn't involve > s.offset for some reason? This could be a bug. > > input: Running time > output: Stream time No, output is a timestamp in the segment, not a stream-time > > Forward (s.rate > 0) > > f(s, x) = start + (x - s.base) * s.rate > > Backward (s.rate <= 0) > > f(s, x) = stop + (x - s.base) * s.rate > > gst_segment_to_stream_time > -------------------------- > > Seems to have something to do with timestamp transformations when a rate > transformation has already been applied to embedded video data PTS? or buffer > timestamps? gives the position in the stream, a value between 0 and the duration of the stream. > > input: Not sure (same time as in buffer timestamps) Input is a buffer timestamp. > output: stream time > > Forward (s.applied_rate > 0) > > f(s, x) = s.time + (x-s.start) * s.applied_rate > > Backward (s.applied_rate <= 0) applied rate can't be 0 > > f(s, x) = s.time - (x-s.start) * s.applied_rate I'll try to update and clarify the docs a little.
commit bc54ec2677ac13ed186d371afff716ea63cd6048 Author: Wim Taymans <wim.taymans@collabora.co.uk> Date: Fri Dec 21 10:09:30 2012 +0100 docs: update synchronization document a little
Created attachment 232044 [details] Diagram showing GstSegment members
Created attachment 232045 [details] Raster version of diagram illustrating GstSegment members I've created a diagram based on what you've described. Does that look correct? I'm unsure how offset fits in there.
Notes from IRC: <willmanley> wtay: I've created a diagram of the GstSegment members: http://bugzilla-attachments.gnome.org/attachment.cgi?id=232045 <willmanley> does this look correct? <wtay> willmanley, It looks ok to me <wtay> willmanley, http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/section-time-overview.html <willmanley> I'd seen that but I was a little confused as I didn't realize that there was also buffer timestamp time in addition to stream, running and clock time. It's much clearer to me now, thanks. <wtay> ok, maybe I can merge this into the document <willmanley> wtay, I'm still a little unsure about offset though <willmanley> in your updated documentation you say "s.offset: an offset to apply to S.start or S.stop. This is the amount that has already been elapsed in the segment." <wtay> willmanley, the reason is that when you are using the segment for tracking the current playing position and you do a seek that only changes the rate, you don't want to modify the start/stop in the segment <willmanley> right <willmanley> ok <wtay> but you still want to have the rate change happen from the current position <willmanley> but that still qualifies as a new segment right? e.g. you would receieve GST_EVENT_SEGMENT_DONE and then GST_EVENT_SEGMENT <wtay> you would and the demuxer uses the GstSegment structure to generate the event <willmanley> And the offset is expressed in timestamp time <wtay> yes <wtay> a value between start and stop <willmanley> wtay, Thanks for your help, I'll see if I can write some of this stuff up and submit it as a patch
> <willmanley> wtay, Thanks for your help, I'll see if I can write some of this > stuff up and submit it as a patch Will, are you still going to make a patch ?
Let's close this bug, I don't see anything really actionable in here, it's more like a stream of consciousness of someone trying to figure out how segments are supposed to work :) I've pushed this now: commit c8f680e395a2520b231c2ce8b38d131005f5e2f1 Author: Tim-Philipp Müller <tim@centricular.com> Date: Sat Apr 4 19:29:51 2015 +0100 segment: small docs addition https://bugzilla.gnome.org/show_bug.cgi?id=690564 commit 199c8ba561cb641a86b12fd607dbbb3e73bc8661 Author: Tim-Philipp Müller <tim@centricular.com> Date: Sat Apr 4 18:18:03 2015 +0100 docs: design: fix some 0.10-isms in GstSegment docs 1) segment.accum -> segment.base 2) Refer to GstSegment members as S.foo instead of NS.foo, the event is now called a segment event rather than newsegment event. 3) There's no more abs_rate field in GstSegment, and there never was an abs_applied_rate field. https://bugzilla.gnome.org/show_bug.cgi?id=690564 Feel free to re-open if you have any further patches/improvements, thanks!