GNOME Bugzilla – Bug 108297
repeat field for GstBuffer
Last modified: 2004-12-22 21:47:04 UTC
GstBuffer might need a 'repeat' field. This means that the data in the buffer should be played N times. Normally, this would be 1, but in some cases, it can be more than one. Imagine an AVI file where the index refers to the same location within the file more than once subsequentially. Or imagine a video capture device (v4l*src) losing a frame (or needing to insert one) and wanting to duplicate the next one to correct for this. Currently, the buffer is just _push()'ed twice, but this is insufficient since you possibly add extra CPU cycles which aren't necessarily needed. An MJPEG encoder, for example, would currently encode the same buffer N times, while in the future, it would just encode it once and leave the repeat field as it is. An MPEG encoder, on the other hand, would process the frame N times (since the MPEG structure mostly doesn't allow for this type of behaviour, unless we're using I-frame only MPEG). Anyway, lots of text, the idea is simple: can we please get a repeat field in GstBuffer?
We talked about this, but I'll present my reasons against it, which are mostly maintenance/understandability reasons. 1. increased complexity Buffers are a core structure of GStreamer and should as such be kept as simple as possible. People learning about buffers should easily understand it and the code handling buffers should not be too complex. 2. little gain The only advantage of such a change would be increased performance. While performance increases are a nice thing to have, I think we shouldn't implement it before we are stable. 3. handling of this feature The only workable long-term solution would be for every plugin to check if it has to duplicate a buffer, which is lots of code and overhead in lots of plugins. The solution of "let the scheduler do this job" doesn't appeal to me either, because schedulers are hard enough as it is and they should not care about the data they pass. 4. significant work It's lots of time to implement this correctly everywhere, since it's quite a big change. Proposed solution in the video case: In IRC the suggestion was to set the timestamp accordingly on the next sent frame, so every plugin can detect if a frame was dropped and react accordingly. Additionally it could be possible to specify a caps property "constant framerate" wo indicate that this element needs a constant framerate. So it should be possible to tackle this problem without changes to GstBuffer but with better specifying the mimetype(s) "video/*". And it'd be still lots of work to do.
I'd like to make one comment to the last argument (timestamp as indicator of dupped frames): that wouldn't be any different if the repeat field was implemented, it'd just be an *extra* measure for *constant* framerate streams to measure this. So both would be possible simultanoeusly.
One alternative is for the source to push buffers pointing to the same data multiple times. If the sink does not understand this little trick, it will use the buffers the same as if the buffers pointed to different data. If it does understand this trick, it will recognize that the data for subsequent buffers is identical. Of course, this requires a bit of overhead, since both the source and sink need to have an additional buffer referenced at all times, thereby increasing average memory usage. But any method will require additional memory usage.
I don't think that increasing performance for corner cases is an interesting use of developer time right now. A typical DVD mpeg stream that has been converted from film has ~30% repeated frames, though. Now that we have a duration field, can that be used to the same effect? I'm concerned that "sparse" video streams will just end up being a lot of work to support for plugins that are really pretty simple, like timeoverlay. I'd like to tag this bug "thoughts for 1.2" (or 0.10), and close it.
I'll mark this as unfixed for now... When I've got a clear picture of whether we really need this next to the timings, I'll reopen it.